The following changes since commit 474f3938d79ab36b9231c9ad3b5a9314c2aeacde:

Merge remote-tracking branch 'remotes/amarkovic/tags/mips-queue-jun-21-2019' into staging (2019-06-21 15:40:50 +0100)

https://github.com/XanClic/qemu.git tags/pull-block-2019-06-24

for you to fetch changes up to ab5d4a30f7f3803ca5106b370969c1b7b54136f8:

iotests: Fix 205 for concurrent runs (2019-06-24 16:01:40 +0200)

Klaus Birkelund Jensen (1):

nvme: do not advertise support for unsupported arbitration mechanism

Max Reitz (1):

iotests: Fix 205 for concurrent runs

Pino Toscano (1):

ssh: switch from libssh2 to libssh

Sam Eiderman (3):

vmdk: Fix comment regarding max l1_size coverage

vmdk: Reduce the max bound for L1 table size

vmdk: Add read-only support for seSparse snapshots

Vladimir Sementsov-Ogievskiy (1):

blockdev: enable non-root nodes for transaction drive-backup source

configure | 65 +-

block/Makefile.objs | 6 +-

block/ssh.c | 652 ++++++++++--------

block/vmdk.c | 372 +++++++++-

blockdev.c | 2 +-

hw/block/nvme.c | 1 -

.travis.yml | 4 +-

block/trace-events | 14 +-

docs/qemu-block-drivers.texi | 2 +-

.../dockerfiles/debian-win32-cross.docker | 1 -

.../dockerfiles/debian-win64-cross.docker | 1 -

tests/docker/dockerfiles/fedora.docker | 4 +-

tests/docker/dockerfiles/ubuntu.docker | 2 +-

tests/docker/dockerfiles/ubuntu1804.docker | 2 +-

tests/qemu-iotests/059.out | 2 +-

tests/qemu-iotests/134 | 9 +

tests/qemu-iotests/134.out | 10 +

tests/qemu-iotests/205 | 2 +-

tests/qemu-iotests/207 | 54 +-

tests/qemu-iotests/207.out | 2 +-

20 files changed, 823 insertions(+), 384 deletions(-)

2.21.0

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

We forget to enable it for transaction .prepare, while it is already

enabled in do_drive_backup since commit a2d665c1bc362

"blockdev: loosen restrictions on drive-backup source node"

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Message-id: 20190618140804.59214-1-vsementsov@virtuozzo.com

Reviewed-by: John Snow <jsnow@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

blockdev.c | 2 +-

diff --git a/blockdev.c b/blockdev.c

--- a/blockdev.c

+++ b/blockdev.c

@@ -XXX,XX +XXX,XX @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)

assert(common->action->type == TRANSACTION_ACTION_KIND_DRIVE_BACKUP);

backup = common->action->u.drive_backup.data;

- bs = qmp_get_root_bs(backup->device, errp);

+ bs = bdrv_lookup_bs(backup->device, backup->device, errp);

if (!bs) {

return;

}

2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

Commit b0651b8c246d ("vmdk: Move l1_size check into vmdk_add_extent")

extended the l1_size check from VMDK4 to VMDK3 but did not update the

default coverage in the moved comment.

The previous vmdk4 calculation:

(512 * 1024 * 1024) * 512(l2 entries) * 65536(grain) = 16PB

The added vmdk3 calculation:

(512 * 1024 * 1024) * 4096(l2 entries) * 512(grain) = 1PB

Adding the calculation of vmdk3 to the comment.

In any case, VMware does not offer virtual disks more than 2TB for

vmdk4/vmdk3 or 64TB for the new undocumented seSparse format which is

not implemented yet in qemu.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>

Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>

Reviewed-by: Liran Alon <liran.alon@oracle.com>

Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>

Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>

Message-id: 20190620091057.47441-2-shmuel.eiderman@oracle.com

Reviewed-by: yuchenlin <yuchenlin@synology.com>

Reviewed-by: Max Reitz <mreitz@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

block/vmdk.c | 11 ++++++++---

1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c

--- a/block/vmdk.c

+++ b/block/vmdk.c

@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,

return -EFBIG;

}

if (l1_size > 512 * 1024 * 1024) {

- /* Although with big capacity and small l1_entry_sectors, we can get a

+ /*

+ * Although with big capacity and small l1_entry_sectors, we can get a

* big l1_size, we don't want unbounded value to allocate the table.

- * Limit it to 512M, which is 16PB for default cluster and L2 table

- * size */

+ * Limit it to 512M, which is:

+ * 16PB - for default "Hosted Sparse Extent" (VMDK4)

+ * cluster size: 64KB, L2 table size: 512 entries

+ * 1PB - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)

+ * cluster size: 512B, L2 table size: 4096 entries

+ */

error_setg(errp, "L1 size too big");

return -EFBIG;

}

2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

512M of L1 entries is a very loose bound, only 32M are required to store

the maximal supported VMDK file size of 2TB.

Fixed qemu-iotest 59# - now failure occures before on impossible L1

table size.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>

Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>

Reviewed-by: Liran Alon <liran.alon@oracle.com>

Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>

Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>

Message-id: 20190620091057.47441-3-shmuel.eiderman@oracle.com

Reviewed-by: Max Reitz <mreitz@redhat.com>

Signed-off-by: Max Reitz <mreitz@redhat.com>

---

block/vmdk.c | 13 +++++++------

tests/qemu-iotests/059.out | 2 +-

2 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c

index XXXXXXX..XXXXXXX 100644

--- a/block/vmdk.c

+++ b/block/vmdk.c

@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,

error_setg(errp, "Invalid granularity, image may be corrupt");

return -EFBIG;

}

- if (l1_size > 512 * 1024 * 1024) {

+ if (l1_size > 32 * 1024 * 1024) {

/*

* Although with big capacity and small l1_entry_sectors, we can get a

* big l1_size, we don't want unbounded value to allocate the table.

- * Limit it to 512M, which is:

- * 16PB - for default "Hosted Sparse Extent" (VMDK4)

- * cluster size: 64KB, L2 table size: 512 entries

- * 1PB - for default "ESXi Host Sparse Extent" (VMDK3/vmfsSparse)

- * cluster size: 512B, L2 table size: 4096 entries

+ * Limit it to 32M, which is enough to store:

+ * 8TB - for both VMDK3 & VMDK4 with

+ * minimal cluster size: 512B

+ * minimal L2 table size: 512 entries

+ * 8 TB is still more than the maximal value supported for

+ * VMDK3 & VMDK4 which is 2TB.

*/

error_setg(errp, "L1 size too big");

return -EFBIG;

diff --git a/tests/qemu-iotests/059.out b/tests/qemu-iotests/059.out

index XXXXXXX..XXXXXXX 100644

--- a/tests/qemu-iotests/059.out

+++ b/tests/qemu-iotests/059.out

@@ -XXX,XX +XXX,XX @@ Offset Length Mapped to File

0x140000000 0x10000 0x50000 TEST_DIR/t-s003.vmdk

=== Testing afl image with a very large capacity ===

-qemu-img: Can't get image size 'TEST_DIR/afl9.IMGFMT': File too large

+qemu-img: Could not open 'TEST_DIR/afl9.IMGFMT': L1 size too big

*** done

2.21.0

From: Sam Eiderman <shmuel.eiderman@oracle.com>

Until ESXi 6.5 VMware used the vmfsSparse format for snapshots (VMDK3 in

QEMU).

This format was lacking in the following:

* Grain directory (L1) and grain table (L2) entries were 32-bit,

allowing access to only 2TB (slightly less) of data.

* The grain size (default) was 512 bytes - leading to data

fragmentation and many grain tables.

* For space reclamation purposes, it was necessary to find all the

grains which are not pointed to by any grain table - so a reverse

mapping of "offset of grain in vmdk" to "grain table" must be

constructed - which takes large amounts of CPU/RAM.

The format specification can be found in VMware's documentation:

https://www.vmware.com/support/developer/vddk/vmdk_50_technote.pdf

In ESXi 6.5, to support snapshot files larger than 2TB, a new format was

introduced: SESparse (Space Efficient).

This format fixes the above issues:

* All entries are now 64-bit.

* The grain size (default) is 4KB.

* Grain directory and grain tables are now located at the beginning

of the file.

+ seSparse format reserves space for all grain tables.

+ Grain tables can be addressed using an index.

+ Grains are located in the end of the file and can also be

addressed with an index.

- seSparse vmdks of large disks (64TB) have huge preallocated

headers - mainly due to L2 tables, even for empty snapshots.

* The header contains a reverse mapping ("backmap") of "offset of

grain in vmdk" to "grain table" and a bitmap ("free bitmap") which

specifies for each grain - whether it is allocated or not.

Using these data structures we can implement space reclamation

efficiently.

* Due to the fact that the header now maintains two mappings:

* The regular one (grain directory & grain tables)

* A reverse one (backmap and free bitmap)

These data structures can lose consistency upon crash and result

in a corrupted VMDK.

Therefore, a journal is also added to the VMDK and is replayed

when the VMware reopens the file after a crash.

Since ESXi 6.7 - SESparse is the only snapshot format available.

Unfortunately, VMware does not provide documentation regarding the new

seSparse format.

This commit is based on black-box research of the seSparse format.

Various in-guest block operations and their effect on the snapshot file

were tested.

The only VMware provided source of information (regarding the underlying

implementation) was a log file on the ESXi:

/var/log/hostd.log

Whenever an seSparse snapshot is created - the log is being populated

with seSparse records.

Relevant log records are of the form:

[...] Const Header:

[...] constMagic = 0xcafebabe

[...] version = 2.1

[...] capacity = 204800

[...] grainSize = 8

[...] grainTableSize = 64

[...] flags = 0

[...] Extents:

[...] Header : <1 : 1>

[...] JournalHdr : <2 : 2>

[...] Journal : <2048 : 2048>

[...] GrainDirectory : <4096 : 2048>

[...] GrainTables : <6144 : 2048>

[...] FreeBitmap : <8192 : 2048>

[...] BackMap : <10240 : 2048>

[...] Grain : <12288 : 204800>

[...] Volatile Header:

[...] volatileMagic = 0xcafecafe

[...] FreeGTNumber = 0

[...] nextTxnSeqNumber = 0

[...] replayJournal = 0

The sizes that are seen in the log file are in sectors.

Extents are of the following format: <offset : size>

This commit is a strict implementation which enforces:

* magics

* version number 2.1

* grain size of 8 sectors (4KB)

* grain table size of 64 sectors

* zero flags

* extent locations

Additionally, this commit proivdes only a subset of the functionality

offered by seSparse's format:

* Read-only

* No journal replay

* No space reclamation

* No unmap support

Hence, journal header, journal, free bitmap and backmap extents are

unused, only the "classic" (L1 -> L2 -> data) grain access is

implemented.

However there are several differences in the grain access itself.

Grain directory (L1):

* Grain directory entries are indexes (not offsets) to grain

tables.

* Valid grain directory entries have their highest nibble set to

0x1.

* Since grain tables are always located in the beginning of the

file - the index can fit into 32 bits - so we can use its low

part if it's valid.

Grain table (L2):

* Grain table entries are indexes (not offsets) to grains.

* If the highest nibble of the entry is:

0x0:

The grain in not allocated.

The rest of the bytes are 0.

0x1:

The grain is unmapped - guest sees a zero grain.

The rest of the bits point to the previously mapped grain,

see 0x3 case.

0x2:

The grain is zero.

0x3:

The grain is allocated - to get the index calculate:

((entry & 0x0fff000000000000) >> 48) |

((entry & 0x0000ffffffffffff) << 12)

* The difference between 0x1 and 0x2 is that 0x1 is an unallocated

grain which results from the guest using sg_unmap to unmap the

grain - but the grain itself still exists in the grain extent - a

space reclamation procedure should delete it.

Unmapping a zero grain has no effect (0x2 will not change to 0x1)

but unmapping an unallocated grain will (0x0 to 0x1) - naturally.

In order to implement seSparse some fields had to be changed to support

both 32-bit and 64-bit entry sizes.

Reviewed-by: Karl Heubaum <karl.heubaum@oracle.com>

Reviewed-by: Eyal Moscovici <eyal.moscovici@oracle.com>

Reviewed-by: Arbel Moshe <arbel.moshe@oracle.com>

Signed-off-by: Sam Eiderman <shmuel.eiderman@oracle.com>

Message-id: 20190620091057.47441-4-shmuel.eiderman@oracle.com

Signed-off-by: Max Reitz <mreitz@redhat.com>

---

block/vmdk.c | 358 ++++++++++++++++++++++++++++++++++++++++++++++++---

1 file changed, 342 insertions(+), 16 deletions(-)

diff --git a/block/vmdk.c b/block/vmdk.c

index XXXXXXX..XXXXXXX 100644

--- a/block/vmdk.c

+++ b/block/vmdk.c

@@ -XXX,XX +XXX,XX @@ typedef struct {

uint16_t compressAlgorithm;

} QEMU_PACKED VMDK4Header;

+typedef struct VMDKSESparseConstHeader {

+ uint64_t magic;

+ uint64_t version;

+ uint64_t capacity;

+ uint64_t grain_size;

+ uint64_t grain_table_size;

+ uint64_t flags;

+ uint64_t reserved1;

+ uint64_t reserved2;

+ uint64_t reserved3;

+ uint64_t reserved4;

+ uint64_t volatile_header_offset;

+ uint64_t volatile_header_size;

+ uint64_t journal_header_offset;

+ uint64_t journal_header_size;

+ uint64_t journal_offset;

+ uint64_t journal_size;

+ uint64_t grain_dir_offset;

+ uint64_t grain_dir_size;

+ uint64_t grain_tables_offset;

+ uint64_t grain_tables_size;

+ uint64_t free_bitmap_offset;

+ uint64_t free_bitmap_size;

+ uint64_t backmap_offset;

+ uint64_t backmap_size;

+ uint64_t grains_offset;

+ uint64_t grains_size;

+ uint8_t pad[304];

+} QEMU_PACKED VMDKSESparseConstHeader;

+

+typedef struct VMDKSESparseVolatileHeader {

+ uint64_t magic;

+ uint64_t free_gt_number;

+ uint64_t next_txn_seq_number;

+ uint64_t replay_journal;

+ uint8_t pad[480];

+} QEMU_PACKED VMDKSESparseVolatileHeader;

+

#define L2_CACHE_SIZE 16

typedef struct VmdkExtent {

@@ -XXX,XX +XXX,XX @@ typedef struct VmdkExtent {

bool compressed;

bool has_marker;

bool has_zero_grain;

+ bool sesparse;

+ uint64_t sesparse_l2_tables_offset;

+ uint64_t sesparse_clusters_offset;

+ int32_t entry_size;

int version;

int64_t sectors;

int64_t end_sector;

int64_t flat_start_offset;

int64_t l1_table_offset;

int64_t l1_backup_table_offset;

- uint32_t *l1_table;

+ void *l1_table;

uint32_t *l1_backup_table;

unsigned int l1_size;

uint32_t l1_entry_sectors;

unsigned int l2_size;

- uint32_t *l2_cache;

+ void *l2_cache;

uint32_t l2_cache_offsets[L2_CACHE_SIZE];

uint32_t l2_cache_counts[L2_CACHE_SIZE];

@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,

* minimal L2 table size: 512 entries

* 8 TB is still more than the maximal value supported for

* VMDK3 & VMDK4 which is 2TB.

+ * 64TB - for "ESXi seSparse Extent"

+ * minimal cluster size: 512B (default is 4KB)

+ * L2 table size: 4096 entries (const).

+ * 64TB is more than the maximal value supported for

+ * seSparse VMDKs (which is slightly less than 64TB)

*/

error_setg(errp, "L1 size too big");

return -EFBIG;

@@ -XXX,XX +XXX,XX @@ static int vmdk_add_extent(BlockDriverState *bs,

extent->l2_size = l2_size;

extent->cluster_sectors = flat ? sectors : cluster_sectors;

extent->next_cluster_sector = ROUND_UP(nb_sectors, cluster_sectors);

+ extent->entry_size = sizeof(uint32_t);

if (s->num_extents > 1) {

extent->end_sector = (*(extent - 1)).end_sector + extent->sectors;

@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,

int i;

/* read the L1 table */

- l1_size = extent->l1_size * sizeof(uint32_t);

+ l1_size = extent->l1_size * extent->entry_size;

extent->l1_table = g_try_malloc(l1_size);

if (l1_size && extent->l1_table == NULL) {

return -ENOMEM;

@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,

goto fail_l1;

}

for (i = 0; i < extent->l1_size; i++) {

- le32_to_cpus(&extent->l1_table[i]);

+ if (extent->entry_size == sizeof(uint64_t)) {

+ le64_to_cpus((uint64_t *)extent->l1_table + i);

+ } else {

+ assert(extent->entry_size == sizeof(uint32_t));

+ le32_to_cpus((uint32_t *)extent->l1_table + i);

+ }

}

if (extent->l1_backup_table_offset) {

+ assert(!extent->sesparse);

extent->l1_backup_table = g_try_malloc(l1_size);

if (l1_size && extent->l1_backup_table == NULL) {

ret = -ENOMEM;

@@ -XXX,XX +XXX,XX @@ static int vmdk_init_tables(BlockDriverState *bs, VmdkExtent *extent,

}

extent->l2_cache =

- g_new(uint32_t, extent->l2_size * L2_CACHE_SIZE);

+ g_malloc(extent->entry_size * extent->l2_size * L2_CACHE_SIZE);

return 0;

fail_l1b:

g_free(extent->l1_backup_table);

@@ -XXX,XX +XXX,XX @@ static int vmdk_open_vmfs_sparse(BlockDriverState *bs,

return ret;

}

+#define SESPARSE_CONST_HEADER_MAGIC UINT64_C(0x00000000cafebabe)

+#define SESPARSE_VOLATILE_HEADER_MAGIC UINT64_C(0x00000000cafecafe)

+

+/* Strict checks - format not officially documented */

+static int check_se_sparse_const_header(VMDKSESparseConstHeader *header,

+ Error **errp)

+{

+ header->magic = le64_to_cpu(header->magic);

+ header->version = le64_to_cpu(header->version);

+ header->grain_size = le64_to_cpu(header->grain_size);

+ header->grain_table_size = le64_to_cpu(header->grain_table_size);

+ header->flags = le64_to_cpu(header->flags);

+ header->reserved1 = le64_to_cpu(header->reserved1);

+ header->reserved2 = le64_to_cpu(header->reserved2);

+ header->reserved3 = le64_to_cpu(header->reserved3);

+ header->reserved4 = le64_to_cpu(header->reserved4);

+

+ header->volatile_header_offset =

+ le64_to_cpu(header->volatile_header_offset);

+ header->volatile_header_size = le64_to_cpu(header->volatile_header_size);

+

+ header->journal_header_offset = le64_to_cpu(header->journal_header_offset);

+ header->journal_header_size = le64_to_cpu(header->journal_header_size);

+

+ header->journal_offset = le64_to_cpu(header->journal_offset);

+ header->journal_size = le64_to_cpu(header->journal_size);

+

+ header->grain_dir_offset = le64_to_cpu(header->grain_dir_offset);

+ header->grain_dir_size = le64_to_cpu(header->grain_dir_size);

+

+ header->grain_tables_offset = le64_to_cpu(header->grain_tables_offset);

+ header->grain_tables_size = le64_to_cpu(header->grain_tables_size);

+

+ header->free_bitmap_offset = le64_to_cpu(header->free_bitmap_offset);

+ header->free_bitmap_size = le64_to_cpu(header->free_bitmap_size);

+

+ header->backmap_offset = le64_to_cpu(header->backmap_offset);

+ header->backmap_size = le64_to_cpu(header->backmap_size);

+

+ header->grains_offset = le64_to_cpu(header->grains_offset);

+ header->grains_size = le64_to_cpu(header->grains_size);

+

+ if (header->magic != SESPARSE_CONST_HEADER_MAGIC) {

+ error_setg(errp, "Bad const header magic: 0x%016" PRIx64,

+ header->magic);

+ return -EINVAL;

+ }

+

+ if (header->version != 0x0000000200000001) {

+ error_setg(errp, "Unsupported version: 0x%016" PRIx64,

+ header->version);

+ return -ENOTSUP;

+ }

+

+ if (header->grain_size != 8) {

+ error_setg(errp, "Unsupported grain size: %" PRIu64,

+ header->grain_size);

+ return -ENOTSUP;

+ }

+

+ if (header->grain_table_size != 64) {

+ error_setg(errp, "Unsupported grain table size: %" PRIu64,

+ header->grain_table_size);

+ return -ENOTSUP;

+ }

+

+ if (header->flags != 0) {

+ error_setg(errp, "Unsupported flags: 0x%016" PRIx64,

+ header->flags);

+ return -ENOTSUP;

+ }

+

+ if (header->reserved1 != 0 || header->reserved2 != 0 ||

+ header->reserved3 != 0 || header->reserved4 != 0) {

+ error_setg(errp, "Unsupported reserved bits:"

+ " 0x%016" PRIx64 " 0x%016" PRIx64

+ " 0x%016" PRIx64 " 0x%016" PRIx64,

+ header->reserved1, header->reserved2,

+ header->reserved3, header->reserved4);

+ return -ENOTSUP;

+ }

+

+ /* check that padding is 0 */

+ if (!buffer_is_zero(header->pad, sizeof(header->pad))) {

+ error_setg(errp, "Unsupported non-zero const header padding");

+ return -ENOTSUP;

+ }

+

+ return 0;

+}

+

+static int check_se_sparse_volatile_header(VMDKSESparseVolatileHeader *header,

+ Error **errp)

+{

+ header->magic = le64_to_cpu(header->magic);

+ header->free_gt_number = le64_to_cpu(header->free_gt_number);

+ header->next_txn_seq_number = le64_to_cpu(header->next_txn_seq_number);

+ header->replay_journal = le64_to_cpu(header->replay_journal);

+

+ if (header->magic != SESPARSE_VOLATILE_HEADER_MAGIC) {

+ error_setg(errp, "Bad volatile header magic: 0x%016" PRIx64,

+ header->magic);

+ return -EINVAL;

+ }

+

+ if (header->replay_journal) {

+ error_setg(errp, "Image is dirty, Replaying journal not supported");

+ return -ENOTSUP;

+ }

+

+ /* check that padding is 0 */

+ if (!buffer_is_zero(header->pad, sizeof(header->pad))) {

+ error_setg(errp, "Unsupported non-zero volatile header padding");

+ return -ENOTSUP;

+ }

+

+ return 0;

+}

+

+static int vmdk_open_se_sparse(BlockDriverState *bs,

+ BdrvChild *file,

+ int flags, Error **errp)

+{

+ int ret;

+ VMDKSESparseConstHeader const_header;

+ VMDKSESparseVolatileHeader volatile_header;

+ VmdkExtent *extent;

+

+ ret = bdrv_apply_auto_read_only(bs,

+ "No write support for seSparse images available", errp);

+ if (ret < 0) {

+ return ret;

+ }

+

+ assert(sizeof(const_header) == SECTOR_SIZE);

+

+ ret = bdrv_pread(file, 0, &const_header, sizeof(const_header));

+ if (ret < 0) {

+ bdrv_refresh_filename(file->bs);

+ error_setg_errno(errp, -ret,

+ "Could not read const header from file '%s'",

+ file->bs->filename);

+ return ret;

+ }

+

+ /* check const header */

+ ret = check_se_sparse_const_header(&const_header, errp);

+ if (ret < 0) {

+ return ret;

+ }

+

+ assert(sizeof(volatile_header) == SECTOR_SIZE);

+

+ ret = bdrv_pread(file,

+ const_header.volatile_header_offset * SECTOR_SIZE,

+ &volatile_header, sizeof(volatile_header));

+ if (ret < 0) {

+ bdrv_refresh_filename(file->bs);

+ error_setg_errno(errp, -ret,

+ "Could not read volatile header from file '%s'",

+ file->bs->filename);

+ return ret;

+ }

+

+ /* check volatile header */

+ ret = check_se_sparse_volatile_header(&volatile_header, errp);

+ if (ret < 0) {

+ return ret;

+ }

+

+ ret = vmdk_add_extent(bs, file, false,

+ const_header.capacity,

+ const_header.grain_dir_offset * SECTOR_SIZE,

+ 0,

+ const_header.grain_dir_size *

+ SECTOR_SIZE / sizeof(uint64_t),

+ const_header.grain_table_size *

+ SECTOR_SIZE / sizeof(uint64_t),

+ const_header.grain_size,

+ &extent,

+ errp);

+ if (ret < 0) {

+ return ret;

+ }

+

+ extent->sesparse = true;

+ extent->sesparse_l2_tables_offset = const_header.grain_tables_offset;

+ extent->sesparse_clusters_offset = const_header.grains_offset;

+ extent->entry_size = sizeof(uint64_t);

+

+ ret = vmdk_init_tables(bs, extent, errp);

+ if (ret) {

+ /* free extent allocated by vmdk_add_extent */

+ vmdk_free_last_extent(bs);

+ }

+

+ return ret;

+}

+

static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,

QDict *options, Error **errp);

@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,

* RW [size in sectors] SPARSE "file-name.vmdk"

* RW [size in sectors] VMFS "file-name.vmdk"

* RW [size in sectors] VMFSSPARSE "file-name.vmdk"

+ * RW [size in sectors] SESPARSE "file-name.vmdk"

*/

flat_offset = -1;

matches = sscanf(p, "%10s %" SCNd64 " %10s \"%511[^\n\r\"]\" %" SCNd64,

@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,

if (sectors <= 0 ||

(strcmp(type, "FLAT") && strcmp(type, "SPARSE") &&

- strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE")) ||

+ strcmp(type, "VMFS") && strcmp(type, "VMFSSPARSE") &&

+ strcmp(type, "SESPARSE")) ||

(strcmp(access, "RW"))) {

continue;

}

@@ -XXX,XX +XXX,XX @@ static int vmdk_parse_extents(const char *desc, BlockDriverState *bs,

return ret;

}

extent = &s->extents[s->num_extents - 1];

+ } else if (!strcmp(type, "SESPARSE")) {

+ ret = vmdk_open_se_sparse(bs, extent_file, bs->open_flags, errp);

+ if (ret) {

+ bdrv_unref_child(bs, extent_file);

+ return ret;

+ }

+ extent = &s->extents[s->num_extents - 1];

} else {

error_setg(errp, "Unsupported extent type '%s'", type);

bdrv_unref_child(bs, extent_file);

@@ -XXX,XX +XXX,XX @@ static int vmdk_open_desc_file(BlockDriverState *bs, int flags, char *buf,

if (strcmp(ct, "monolithicFlat") &&

strcmp(ct, "vmfs") &&

strcmp(ct, "vmfsSparse") &&

+ strcmp(ct, "seSparse") &&

strcmp(ct, "twoGbMaxExtentSparse") &&

strcmp(ct, "twoGbMaxExtentFlat")) {

error_setg(errp, "Unsupported image type '%s'", ct);

@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,

{

unsigned int l1_index, l2_offset, l2_index;

int min_index, i, j;

- uint32_t min_count, *l2_table;

+ uint32_t min_count;

+ void *l2_table;

bool zeroed = false;

int64_t ret;

int64_t cluster_sector;

+ unsigned int l2_size_bytes = extent->l2_size * extent->entry_size;

if (m_data) {

m_data->valid = 0;

@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,

if (l1_index >= extent->l1_size) {

return VMDK_ERROR;

}

- l2_offset = extent->l1_table[l1_index];

+ if (extent->sesparse) {

+ uint64_t l2_offset_u64;

+

+ assert(extent->entry_size == sizeof(uint64_t));

+

+ l2_offset_u64 = ((uint64_t *)extent->l1_table)[l1_index];

+ if (l2_offset_u64 == 0) {

+ l2_offset = 0;

+ } else if ((l2_offset_u64 & 0xffffffff00000000) != 0x1000000000000000) {

+ /*

+ * Top most nibble is 0x1 if grain table is allocated.

+ * strict check - top most 4 bytes must be 0x10000000 since max

+ * supported size is 64TB for disk - so no more than 64TB / 16MB

+ * grain directories which is smaller than uint32,

+ * where 16MB is the only supported default grain table coverage.

+ */

+ return VMDK_ERROR;

+ } else {

+ l2_offset_u64 = l2_offset_u64 & 0x00000000ffffffff;

+ l2_offset_u64 = extent->sesparse_l2_tables_offset +

+ l2_offset_u64 * l2_size_bytes / SECTOR_SIZE;

+ if (l2_offset_u64 > 0x00000000ffffffff) {

+ return VMDK_ERROR;

+ }

+ l2_offset = (unsigned int)(l2_offset_u64);

+ }

+ } else {

+ assert(extent->entry_size == sizeof(uint32_t));

+ l2_offset = ((uint32_t *)extent->l1_table)[l1_index];

+ }

if (!l2_offset) {

return VMDK_UNALLOC;

}

@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,

extent->l2_cache_counts[j] >>= 1;

}

}

- l2_table = extent->l2_cache + (i * extent->l2_size);

+ l2_table = (char *)extent->l2_cache + (i * l2_size_bytes);

goto found;

}

}

@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,

min_index = i;

}

}

- l2_table = extent->l2_cache + (min_index * extent->l2_size);

+ l2_table = (char *)extent->l2_cache + (min_index * l2_size_bytes);

BLKDBG_EVENT(extent->file, BLKDBG_L2_LOAD);

if (bdrv_pread(extent->file,

(int64_t)l2_offset * 512,

l2_table,

- extent->l2_size * sizeof(uint32_t)

- ) != extent->l2_size * sizeof(uint32_t)) {

+ l2_size_bytes

+ ) != l2_size_bytes) {

return VMDK_ERROR;

}

@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,

extent->l2_cache_counts[min_index] = 1;

found:

l2_index = ((offset >> 9) / extent->cluster_sectors) % extent->l2_size;

- cluster_sector = le32_to_cpu(l2_table[l2_index]);

- if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {

- zeroed = true;

+ if (extent->sesparse) {

+ cluster_sector = le64_to_cpu(((uint64_t *)l2_table)[l2_index]);

+ switch (cluster_sector & 0xf000000000000000) {

+ case 0x0000000000000000:

+ /* unallocated grain */

+ if (cluster_sector != 0) {

+ return VMDK_ERROR;

+ }

+ break;

+ case 0x1000000000000000:

+ /* scsi-unmapped grain - fallthrough */

+ case 0x2000000000000000:

+ /* zero grain */

+ zeroed = true;

+ break;

+ case 0x3000000000000000:

+ /* allocated grain */

+ cluster_sector = (((cluster_sector & 0x0fff000000000000) >> 48) |

+ ((cluster_sector & 0x0000ffffffffffff) << 12));

+ cluster_sector = extent->sesparse_clusters_offset +

+ cluster_sector * extent->cluster_sectors;

+ break;

+ default:

+ return VMDK_ERROR;

+ }

+ } else {

+ cluster_sector = le32_to_cpu(((uint32_t *)l2_table)[l2_index]);

+

+ if (extent->has_zero_grain && cluster_sector == VMDK_GTE_ZEROED) {

+ zeroed = true;

+ }

}

if (!cluster_sector || zeroed) {

if (!allocate) {

return zeroed ? VMDK_ZEROED : VMDK_UNALLOC;

}

+ assert(!extent->sesparse);

if (extent->next_cluster_sector >= VMDK_EXTENT_MAX_SECTORS) {

return VMDK_ERROR;

@@ -XXX,XX +XXX,XX @@ static int get_cluster_offset(BlockDriverState *bs,

m_data->l1_index = l1_index;

m_data->l2_index = l2_index;

m_data->l2_offset = l2_offset;

- m_data->l2_cache_entry = &l2_table[l2_index];

+ m_data->l2_cache_entry = ((uint32_t *)l2_table) + l2_index;

}

}

*cluster_offset = cluster_sector << BDRV_SECTOR_BITS;

@@ -XXX,XX +XXX,XX @@ static int vmdk_pwritev(BlockDriverState *bs, uint64_t offset,

if (!extent) {

return -EIO;

}

+ if (extent->sesparse) {

+ return -ENOTSUP;

+ }

offset_in_cluster = vmdk_find_offset_in_cluster(extent, offset);

n_bytes = MIN(bytes, extent->cluster_sectors * BDRV_SECTOR_SIZE

- offset_in_cluster);

--

2.21.0

From: Pino Toscano <ptoscano@redhat.com>

Rewrite the implementation of the ssh block driver to use libssh instead

of libssh2. The libssh library has various advantages over libssh2:

- easier API for authentication (for example for using ssh-agent)

- easier API for known_hosts handling

- supports newer types of keys in known_hosts

Use APIs/features available in libssh 0.8 conditionally, to support

older versions (which are not recommended though).

Adjust the iotest 207 according to the different error message, and to

find the default key type for localhost (to properly compare the

fingerprint with).

Contributed-by: Max Reitz <mreitz@redhat.com>

Adjust the various Docker/Travis scripts to use libssh when available

instead of libssh2. The mingw/mxe testing is dropped for now, as there

are no packages for it.

Signed-off-by: Pino Toscano <ptoscano@redhat.com>

Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Acked-by: Alex Bennée <alex.bennee@linaro.org>

Message-id: 20190620200840.17655-1-ptoscano@redhat.com

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Message-id: 5873173.t2JhDm7DL7@lindworm.usersys.redhat.com

Signed-off-by: Max Reitz <mreitz@redhat.com>

configure | 65 +-

block/Makefile.objs | 6 +-

block/ssh.c | 652 ++++++++++--------

.travis.yml | 4 +-

block/trace-events | 14 +-

docs/qemu-block-drivers.texi | 2 +-

.../dockerfiles/debian-win32-cross.docker | 1 -

.../dockerfiles/debian-win64-cross.docker | 1 -

tests/docker/dockerfiles/fedora.docker | 4 +-

tests/docker/dockerfiles/ubuntu.docker | 2 +-

tests/docker/dockerfiles/ubuntu1804.docker | 2 +-

tests/qemu-iotests/207 | 54 +-

tests/qemu-iotests/207.out | 2 +-

13 files changed, 449 insertions(+), 360 deletions(-)

diff --git a/configure b/configure

index XXXXXXX..XXXXXXX 100755

--- a/configure

+++ b/configure

@@ -XXX,XX +XXX,XX @@ auth_pam=""

vte=""

virglrenderer=""

tpm=""

-libssh2=""

+libssh=""

live_block_migration="yes"

numa=""

tcmalloc="no"

@@ -XXX,XX +XXX,XX @@ for opt do

;;

--enable-tpm) tpm="yes"

;;

- --disable-libssh2) libssh2="no"

+ --disable-libssh) libssh="no"

;;

- --enable-libssh2) libssh2="yes"

+ --enable-libssh) libssh="yes"

;;

--disable-live-block-migration) live_block_migration="no"

;;

@@ -XXX,XX +XXX,XX @@ disabled with --disable-FEATURE, default is enabled if available:

coroutine-pool coroutine freelist (better performance)

glusterfs GlusterFS backend

tpm TPM support

- libssh2 ssh block device support

+ libssh ssh block device support

numa libnuma support

libxml2 for Parallels image format

tcmalloc tcmalloc support

@@ -XXX,XX +XXX,XX @@ EOF

fi

##########################################

-# libssh2 probe

-min_libssh2_version=1.2.8

-if test "$libssh2" != "no" ; then

- if $pkg_config --atleast-version=$min_libssh2_version libssh2; then

- libssh2_cflags=$($pkg_config libssh2 --cflags)

- libssh2_libs=$($pkg_config libssh2 --libs)

- libssh2=yes

+# libssh probe

+if test "$libssh" != "no" ; then

+ if $pkg_config --exists libssh; then

+ libssh_cflags=$($pkg_config libssh --cflags)

+ libssh_libs=$($pkg_config libssh --libs)

+ libssh=yes

else

- if test "$libssh2" = "yes" ; then

- error_exit "libssh2 >= $min_libssh2_version required for --enable-libssh2"

+ if test "$libssh" = "yes" ; then

+ error_exit "libssh required for --enable-libssh"

fi

- libssh2=no

+ libssh=no

fi

fi

##########################################

-# libssh2_sftp_fsync probe

+# Check for libssh 0.8

+# This is done like this instead of using the LIBSSH_VERSION_* and

+# SSH_VERSION_* macros because some distributions in the past shipped

+# snapshots of the future 0.8 from Git, and those snapshots did not

+# have updated version numbers (still referring to 0.7.0).

-if test "$libssh2" = "yes"; then

+if test "$libssh" = "yes"; then

cat > $TMPC <<EOF

-#include <stdio.h>

-#include <libssh2.h>

-#include <libssh2_sftp.h>

-int main(void) {

- LIBSSH2_SESSION *session;

- LIBSSH2_SFTP *sftp;

- LIBSSH2_SFTP_HANDLE *sftp_handle;

- session = libssh2_session_init ();

- sftp = libssh2_sftp_init (session);

- sftp_handle = libssh2_sftp_open (sftp, "/", 0, 0);

- libssh2_sftp_fsync (sftp_handle);

- return 0;

-}

+#include <libssh/libssh.h>

+int main(void) { return ssh_get_server_publickey(NULL, NULL); }

EOF

- # libssh2_cflags/libssh2_libs defined in previous test.

- if compile_prog "$libssh2_cflags" "$libssh2_libs" ; then

- QEMU_CFLAGS="-DHAS_LIBSSH2_SFTP_FSYNC $QEMU_CFLAGS"

+ if compile_prog "$libssh_cflags" "$libssh_libs"; then

+ libssh_cflags="-DHAVE_LIBSSH_0_8 $libssh_cflags"

fi

fi

@@ -XXX,XX +XXX,XX @@ echo "GlusterFS support $glusterfs"

echo "gcov $gcov_tool"

echo "gcov enabled $gcov"

echo "TPM support $tpm"

-echo "libssh2 support $libssh2"

+echo "libssh support $libssh"

echo "QOM debugging $qom_cast_debug"

echo "Live block migration $live_block_migration"

echo "lzo support $lzo"

@@ -XXX,XX +XXX,XX @@ if test "$glusterfs_iocb_has_stat" = "yes" ; then

echo "CONFIG_GLUSTERFS_IOCB_HAS_STAT=y" >> $config_host_mak

fi

-if test "$libssh2" = "yes" ; then

- echo "CONFIG_LIBSSH2=m" >> $config_host_mak

- echo "LIBSSH2_CFLAGS=$libssh2_cflags" >> $config_host_mak

- echo "LIBSSH2_LIBS=$libssh2_libs" >> $config_host_mak

+if test "$libssh" = "yes" ; then

+ echo "CONFIG_LIBSSH=m" >> $config_host_mak

+ echo "LIBSSH_CFLAGS=$libssh_cflags" >> $config_host_mak

+ echo "LIBSSH_LIBS=$libssh_libs" >> $config_host_mak

fi

if test "$live_block_migration" = "yes" ; then

diff --git a/block/Makefile.objs b/block/Makefile.objs

--- a/block/Makefile.objs

+++ b/block/Makefile.objs

@@ -XXX,XX +XXX,XX @@ block-obj-$(CONFIG_CURL) += curl.o

block-obj-$(CONFIG_RBD) += rbd.o

block-obj-$(CONFIG_GLUSTERFS) += gluster.o

block-obj-$(CONFIG_VXHS) += vxhs.o

-block-obj-$(CONFIG_LIBSSH2) += ssh.o

+block-obj-$(CONFIG_LIBSSH) += ssh.o

block-obj-y += accounting.o dirty-bitmap.o

block-obj-y += write-threshold.o

block-obj-y += backup.o

@@ -XXX,XX +XXX,XX @@ rbd.o-libs := $(RBD_LIBS)

gluster.o-cflags := $(GLUSTERFS_CFLAGS)

gluster.o-libs := $(GLUSTERFS_LIBS)

vxhs.o-libs := $(VXHS_LIBS)

-ssh.o-cflags := $(LIBSSH2_CFLAGS)

-ssh.o-libs := $(LIBSSH2_LIBS)

+ssh.o-cflags := $(LIBSSH_CFLAGS)

+ssh.o-libs := $(LIBSSH_LIBS)

block-obj-dmg-bz2-$(CONFIG_BZIP2) += dmg-bz2.o

block-obj-$(if $(CONFIG_DMG),m,n) += $(block-obj-dmg-bz2-y)

dmg-bz2.o-libs := $(BZIP2_LIBS)

diff --git a/block/ssh.c b/block/ssh.c

index XXXXXXX..XXXXXXX 100644

--- a/block/ssh.c

+++ b/block/ssh.c

@@ -XXX,XX +XXX,XX @@

#include "qemu/osdep.h"

-#include <libssh2.h>

-#include <libssh2_sftp.h>

+#include <libssh/libssh.h>

+#include <libssh/sftp.h>

#include "block/block_int.h"

#include "block/qdict.h"

@@ -XXX,XX +XXX,XX @@

#include "trace.h"

/*

- * TRACE_LIBSSH2=<bitmask> enables tracing in libssh2 itself. Note

- * that this requires that libssh2 was specially compiled with the

- * `./configure --enable-debug' option, so most likely you will have

- * to compile it yourself. The meaning of <bitmask> is described

- * here: http://www.libssh2.org/libssh2_trace.html

+ * TRACE_LIBSSH=<level> enables tracing in libssh itself.

+ * The meaning of <level> is described here:

+ * http://api.libssh.org/master/group__libssh__log.html

*/

-#define TRACE_LIBSSH2 0 /* or try: LIBSSH2_TRACE_SFTP */

+#define TRACE_LIBSSH 0 /* see: SSH_LOG_* */

typedef struct BDRVSSHState {

/* Coroutine. */

@@ -XXX,XX +XXX,XX @@ typedef struct BDRVSSHState {

/* SSH connection. */

int sock; /* socket */

- LIBSSH2_SESSION *session; /* ssh session */

- LIBSSH2_SFTP *sftp; /* sftp session */

- LIBSSH2_SFTP_HANDLE *sftp_handle; /* sftp remote file handle */

+ ssh_session session; /* ssh session */

+ sftp_session sftp; /* sftp session */

+ sftp_file sftp_handle; /* sftp remote file handle */

- /* See ssh_seek() function below. */