The following changes since commit 474f3938d79ab36b9231c9ad3b5a9314c2aeacde:

Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-jun-21-2019' into staging (2019-06-21 15:40:50 +0100)

https://github.com/XanClic/qemu.git tags/pull-block-2019-06-24

for you to fetch changes up to ab5d4a30f7f3803ca5106b370969c1b7b54136f8:

iotests: Fix 205 for concurrent runs (2019-06-24 16:01:40 +0200)

Block patches:

- The SSH block driver now uses libssh instead of libssh2

- The VMDK block driver gets read-only support for the seSparse

subformat

- Various fixes

---

v2:

- Squashed Pino's fix for pre-0.8 libssh into the libssh patch

Klaus Birkelund Jensen (1):

nvme: do not advertise support for unsupported arbitration mechanism

Max Reitz (1):

iotests: Fix 205 for concurrent runs

Pino Toscano (1):

ssh: switch from libssh2 to libssh

Sam Eiderman (3):

vmdk: Fix comment regarding max l1_size coverage

vmdk: Reduce the max bound for L1 table size

vmdk: Add read-only support for seSparse snapshots

Vladimir Sementsov-Ogievskiy (1):

blockdev: enable non-root nodes for transaction drive-backup source

configure | 65 +-

block/Makefile.objs | 6 +-

block/ssh.c | 652 ++++++++++--------

block/vmdk.c | 372 +++++++++-

blockdev.c | 2 +-

hw/block/nvme.c | 1 -

.travis.yml | 4 +-

block/trace-events | 14 +-

docs/qemu-block-drivers.texi | 2 +-

.../dockerfiles/debian-win32-cross.docker | 1 -

.../dockerfiles/debian-win64-cross.docker | 1 -

tests/docker/dockerfiles/fedora.docker | 4 +-

tests/docker/dockerfiles/ubuntu.docker | 2 +-

tests/docker/dockerfiles/ubuntu1804.docker | 2 +-

tests/qemu-iotests/059.out | 2 +-

tests/qemu-iotests/134 | 9 +

tests/qemu-iotests/134.out | 10 +

tests/qemu-iotests/205 | 2 +-

tests/qemu-iotests/207 | 54 +-

tests/qemu-iotests/207.out | 2 +-

20 files changed, 823 insertions(+), 384 deletions(-)

2.21.0

From: Klaus Birkelund Jensen <klaus@birkelund.eu>

The device mistakenly reports that the Weighted Round Robin with Urgent

Priority Class arbitration mechanism is supported.

It is not.

Signed-off-by: Klaus Birkelund Jensen <klaus.jensen@cnexlabs.com>

Message-id: 20190606092530.14206-1-klaus@birkelund.eu

Acked-by: Maxim Levitsky <mlevitsk@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

hw/block/nvme.c | 1 -

1 file changed, 1 deletion(-)

diff --git a/hw/block/nvme.c b/hw/block/nvme.c

--- a/hw/block/nvme.c

+++ b/hw/block/nvme.c

@@ -XXX,XX +XXX,XX @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp)

n->bar.cap = 0;

NVME_CAP_SET_MQES(n->bar.cap, 0x7ff);

NVME_CAP_SET_CQR(n->bar.cap, 1);

- NVME_CAP_SET_AMS(n->bar.cap, 1);

NVME_CAP_SET_TO(n->bar.cap, 0xf);

NVME_CAP_SET_CSS(n->bar.cap, 1);

NVME_CAP_SET_MPSMAX(n->bar.cap, 4);

2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

We forget to enable it for transaction .prepare, while it is already

enabled in do_drive_backup since commit a2d665c1bc362

"blockdev: loosen restrictions on drive-backup source node"

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Message-id: 20190618140804.59214-1-vsementsov@virtuozzo.com

Reviewed-by: John Snow <jsnow@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

---

blockdev.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/blockdev.c b/blockdev.c

--- a/blockdev.c

+++ b/blockdev.c

@@ -XXX,XX +XXX,XX @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)

assert(common->action->type == TRANSACTION_ACTION_KIND_DRIVE_BACKUP);

backup = common->action->u.drive_backup.data;

- bs = qmp_get_root_bs(backup->device, errp);

+ bs = bdrv_lookup_bs(backup->device, backup->device, errp);

if (!bs) {

2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

Commit b0651b8c246d ("vmdk: Move l1_size check into vmdk_add_extent")

extended the l1_size check from VMDK4 to VMDK3 but did not update the

default coverage in the moved comment.

The previous vmdk4 calculation:

(512 * 1024 * 1024) * 512(l2 entries) * 65536(grain) = 16PB

The added vmdk3 calculation:

(512 * 1024 * 1024) * 4096(l2 entries) * 512(grain) = 1PB

Adding the calculation of vmdk3 to the comment.

In any case, VMware does not offer virtual disks more than 2TB for

vmdk4/vmdk3 or 64TB for the new undocumented seSparse format which is

not implemented yet in qemu.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>

Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>

Reviewed-by: Liran Alon <liran.alon@oracle.com>

Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>

Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>

Message-id: 20190620091057.47441-2-shmuel.eiderman@oracle.com

Reviewed-by: yuchenlin <yuchenlin@synology.com>

Reviewed-by: Max Reitz <mreitz@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

block/vmdk.c | 11 ++++++++---

1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c

--- a/block/vmdk.c

+++ b/block/vmdk.c

@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,

return -EFBIG;

if (l1_size > 512 * 1024 * 1024) {

- /* Although with big capacity and small l1_entry_sectors, we can get a

+ /*

+ * Although with big capacity and small l1_entry_sectors, we can get a

* big l1_size, we don't want unbounded value to allocate the table.

- * Limit it to 512M, which is 16PB for default cluster and L2 table

- * size */

+ * Limit it to 512M, which is:

+ * 16PB - for default "Hosted Sparse Extent" (VMDK4)

+ * cluster size: 64KB, L2 table size: 512 entries

+ * 1PB - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)

+ * cluster size: 512B, L2 table size: 4096 entries

+ */

error_setg(errp, "L1 size too big");

return -EFBIG;

2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

512M of L1 entries is a very loose bound, only 32M are required to store

the maximal supported VMDK file size of 2TB.

Fixed qemu-iotest 59# - now failure occures before on impossible L1

table size.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>

Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>

Reviewed-by: Liran Alon <liran.alon@oracle.com>

Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>

Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>

Message-id: 20190620091057.47441-3-shmuel.eiderman@oracle.com

Reviewed-by: Max Reitz <mreitz@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

block/vmdk.c | 13 +++++++------

tests/qemu-iotests/059.out | 2 +-

2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c

--- a/block/vmdk.c

+++ b/block/vmdk.c

@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,

error_setg(errp, "Invalid granularity, image may be corrupt");

return -EFBIG;

}

- if (l1_size > 512 * 1024 * 1024) {

+ if (l1_size > 32 * 1024 * 1024) {

/*

* Although with big capacity and small l1_entry_sectors, we can get a

* big l1_size, we don't want unbounded value to allocate the table.

- * Limit it to 512M, which is:

- * 16PB - for default "Hosted Sparse Extent" (VMDK4)

- * cluster size: 64KB, L2 table size: 512 entries

- * 1PB - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)

- * cluster size: 512B, L2 table size: 4096 entries

+ * Limit it to 32M, which is enough to store:

+ * 8TB - for both VMDK3 & VMDK4 with

+ * minimal cluster size: 512B

+ * minimal L2 table size: 512 entries

+ * 8 TB is still more than the maximal value supported for

+ * VMDK3 & VMDK4 which is 2TB.

*/

error_setg(errp, "L1 size too big");

return -EFBIG;

diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out

--- a/tests/qemu-iotests/059.out

+++ b/tests/qemu-iotests/059.out

@@ -XXX,XX +XXX,XX @@ Offset Length Mapped to File

0x140000000 0x10000 0x50000 TEST_DIR/t-s003.vmdk

=== Testing afl image with a very large capacity ===

-qemu-img: Can't get image size 'TEST_DIR/afl9.IMGFMT': File too large

+qemu-img: Could not open 'TEST_DIR/afl9.IMGFMT': L1 size too big

*** done

2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

Until ESXi 6.5 VMware used the vmfsSparse format for snapshots (VMDK3 in

QEMU).

This format was lacking in the following:

* Grain directory (L1) and grain table (L2) entries were 32-bit,

allowing access to only 2TB (slightly less) of data.

* The grain size (default) was 512 bytes - leading to data

fragmentation and many grain tables.

* For space reclamation purposes, it was necessary to find all the

grains which are not pointed to by any grain table - so a reverse

mapping of "offset of grain in vmdk" to "grain table" must be

constructed - which takes large amounts of CPU/RAM.

The format specification can be found in VMware's documentation:

https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf

In ESXi 6.5, to support snapshot files larger than 2TB, a new format was

introduced: SESparse (Space Efficient).

This format fixes the above issues:

* All entries are now 64-bit.

* The grain size (default) is 4KB.

* Grain directory and grain tables are now located at the beginning

of the file.

+ seSparse format reserves space for all grain tables.

+ Grain tables can be addressed using an index.

+ Grains are located in the end of the file and can also be

addressed with an index.

- seSparse vmdks of large disks (64TB) have huge preallocated

headers - mainly due to L2 tables, even for empty snapshots.

* The header contains a reverse mapping ("backmap") of "offset of

grain in vmdk" to "grain table" and a bitmap ("free bitmap") which

specifies for each grain - whether it is allocated or not.

Using these data structures we can implement space reclamation

efficiently.

* Due to the fact that the header now maintains two mappings:

* The regular one (grain directory & grain tables)

* A reverse one (backmap and free bitmap)

These data structures can lose consistency upon crash and result

in a corrupted VMDK.

Therefore, a journal is also added to the VMDK and is replayed

when the VMware reopens the file after a crash.

Since ESXi 6.7 - SESparse is the only snapshot format available.

Unfortunately, VMware does not provide documentation regarding the new

seSparse format.

This commit is based on black-box research of the seSparse format.

Various in-guest block operations and their effect on the snapshot file

were tested.

The only VMware provided source of information (regarding the underlying

implementation) was a log file on the ESXi:

/var/log/hostd.log

Whenever an seSparse snapshot is created - the log is being populated

with seSparse records.

Relevant log records are of the form:

[...] Const Header:

[...] constMagic = 0xcafebabe

[...] version = 2.1

[...] capacity = 204800

[...] grainSize = 8

[...] grainTableSize = 64

[...] flags = 0

[...] Extents:

[...] Header : <1 : 1>

[...] JournalHdr : <2 : 2>

[...] Journal : <2048 : 2048>

[...] GrainDirectory : <4096 : 2048>

[...] GrainTables : <6144 : 2048>

[...] FreeBitmap : <8192 : 2048>

[...] BackMap : <10240 : 2048>

[...] Grain : <12288 : 204800>

[...] Volatile Header:

[...] volatileMagic = 0xcafecafe

[...] FreeGTNumber = 0

[...] nextTxnSeqNumber = 0

[...] replayJournal = 0

The sizes that are seen in the log file are in sectors.

Extents are of the following format: <offset : size>

This commit is a strict implementation which enforces:

* magics

* version number 2.1

* grain size of 8 sectors (4KB)

* grain table size of 64 sectors

* zero flags

* extent locations

Additionally, this commit proivdes only a subset of the functionality

offered by seSparse's format:

* Read-only

* No journal replay

* No space reclamation

* No unmap support

Hence, journal header, journal, free bitmap and backmap extents are

unused, only the "classic" (L1 -> L2 -> data) grain access is

implemented.

However there are several differences in the grain access itself.

Grain directory (L1):

* Grain directory entries are indexes (not offsets) to grain

tables.

* Valid grain directory entries have their highest nibble set to

0x1.

* Since grain tables are always located in the beginning of the

file - the index can fit into 32 bits - so we can use its low

part if it's valid.

Grain table (L2):

* Grain table entries are indexes (not offsets) to grains.

* If the highest nibble of the entry is:

0x0:

The grain in not allocated.

The rest of the bytes are 0.

0x1:

The grain is unmapped - guest sees a zero grain.

The rest of the bits point to the previously mapped grain,

see 0x3 case.

0x2:

The grain is zero.

0x3:

The grain is allocated - to get the index calculate:

((entry & 0x0fff000000000000) >> 48) |

((entry & 0x0000ffffffffffff) << 12)

* The difference between 0x1 and 0x2 is that 0x1 is an unallocated

grain which results from the guest using sg_unmap to unmap the

grain - but the grain itself still exists in the grain extent - a

space reclamation procedure should delete it.

Unmapping a zero grain has no effect (0x2 will not change to 0x1)

but unmapping an unallocated grain will (0x0 to 0x1) - naturally.

In order to implement seSparse some fields had to be changed to support

both 32-bit and 64-bit entry sizes.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>

Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>

Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>

Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>

Message-id: 20190620091057.47441-4-shmuel.eiderman@oracle.com

Signed-off-by: Max Reitz <mreitz@redhat.com>

---

block/vmdk.c | 358 ++++++++++++++++++++++++++++++++++++++++++++++++---

1 file changed, 342 insertions(+), 16 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c

index XXXXXXX..XXXXXXX 100644

--- a/block/vmdk.c

+++ b/block/vmdk.c

@@ -XXX,XX +XXX,XX @@ typedef struct {

uint16_t compressAlgorithm;

} QEMU_PACKED VMDK4Header;

+typedef struct VMDKSESparseConstHeader {

+ uint64_t magic;

+ uint64_t version;

+ uint64_t capacity;

+ uint64_t grain_size;

+ uint64_t grain_table_size;

+ uint64_t flags;

+ uint64_t reserved1;

+ uint64_t reserved2;

+ uint64_t reserved3;

+ uint64_t reserved4;

+ uint64_t volatile_header_offset;

+ uint64_t volatile_header_size;

+ uint64_t journal_header_offset;

+ uint64_t journal_header_size;

+ uint64_t journal_offset;

+ uint64_t journal_size;

+ uint64_t grain_dir_offset;

+ uint64_t grain_dir_size;

+ uint64_t grain_tables_offset;

+ uint64_t grain_tables_size;

+ uint64_t free_bitmap_offset;

+ uint64_t free_bitmap_size;

+ uint64_t backmap_offset;

+ uint64_t backmap_size;

+ uint64_t grains_offset;

+ uint64_t grains_size;

+ uint8_t pad[304];

+} QEMU_PACKED VMDKSESparseConstHeader;

+

+typedef struct VMDKSESparseVolatileHeader {

+ uint64_t magic;

+ uint64_t free_gt_number;

+ uint64_t next_txn_seq_number;

+ uint64_t replay_journal;

+ uint8_t pad[480];

+} QEMU_PACKED VMDKSESparseVolatileHeader;

+

#define L2_CACHE_SIZE 16

typedef struct VmdkExtent {

@@ -XXX,XX +XXX,XX @@ typedef struct VmdkExtent {

bool compressed;

bool has_marker;

bool has_zero_grain;

+ bool sesparse;

+ uint64_t sesparse_l2_tables_offset;

+ uint64_t sesparse_clusters_offset;

+ int32_t entry_size;

int version;

int64_t sectors;

int64_t end_sector;

int64_t flat_start_offset;

int64_t l1_table_offset;

int64_t l1_backup_table_offset;

- uint32_t *l1_table;

+ void *l1_table;

uint32_t *l1_backup_table;

unsigned int l1_size;

uint32_t l1_entry_sectors;

unsigned int l2_size;

- uint32_t *l2_cache;

+ void *l2_cache;

uint32_t l2_cache_offsets[L2_CACHE_SIZE];

uint32_t l2_cache_counts[L2_CACHE_SIZE];

@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,

* minimal L2 table size: 512 entries

* 8 TB is still more than the maximal value supported for

* VMDK3 & VMDK4 which is 2TB.

+ * 64TB - for "ESXi seSparse Extent"

+ * minimal cluster size: 512B (default is 4KB)

+ * L2 table size: 4096 entries (const).

+ * 64TB is more than the maximal value supported for