The following changes since commit ae49fbbcd8e4e9d8bf7131add34773f579e1aff7: Merge remote-tracking branch 'remotes/rth/tags/pull-tcg-20171025' into staging (2017-10-25 16:38:57 +0100) are available in the git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for you to fetch changes up to 4254d01ce4eec9a3ccf320d14e2da132b8ad4a51: Merge remote-tracking branch 'mreitz/tags/pull-block-2017-10-26' into queue-block (2017-10-26 15:02:40 +0200) ---------------------------------------------------------------- Block layer patches ---------------------------------------------------------------- Alberto Garcia (1): qcow2: Use BDRV_SECTOR_BITS instead of its literal value Eric Blake (24): block: Allow NULL file for bdrv_get_block_status() block: Add flag to avoid wasted work in bdrv_is_allocated() block: Make bdrv_round_to_clusters() signature more useful qcow2: Switch is_zero_sectors() to byte-based block: Switch bdrv_make_zero() to byte-based qemu-img: Switch get_block_status() to byte-based block: Convert bdrv_get_block_status() to bytes block: Switch bdrv_co_get_block_status() to byte-based block: Switch BdrvCoGetBlockStatusData to byte-based block: Switch bdrv_common_block_status_above() to byte-based block: Switch bdrv_co_get_block_status_above() to byte-based block: Convert bdrv_get_block_status_above() to bytes qemu-img: Simplify logic in img_compare() qemu-img: Speed up compare on pre-allocated larger file qemu-img: Add find_nonzero() qemu-img: Drop redundant error message in compare qemu-img: Change check_empty_sectors() to byte-based qemu-img: Change compare_sectors() to be byte-based qemu-img: Change img_rebase() to be byte-based qemu-img: Change img_compare() to be byte-based block: Align block status requests block: Reduce bdrv_aligned_preadv() rounding qcow2: Reduce is_zero() rounding qemu-io: Relax 'alloc' now that block-status doesn't assert Kevin Wolf (2): qemu-iotests: Test backing_fmt with backing node reference Merge remote-tracking branch 'mreitz/tags/pull-block-2017-10-26' into queue-block Max Reitz (8): qemu-img.1: Image invalidation on qemu-img commit iotests: Add test for dataplane mirroring iotests: Pull _filter_actual_image_size from 67/87 iotests: Filter actual image size in 184 and 191 qcow2: Emit errp when truncating the image tail qcow2: Fix unaligned preallocated truncation qcow2: Always execute preallocate() in a coroutine iotests: Add cluster_size=64k to 125 Peter Krempa (1): block: don't add 'driver' to options when referring to backing via node name include/block/block.h | 29 ++- include/block/block_int.h | 11 +- block.c | 3 +- block/blkdebug.c | 13 +- block/io.c | 334 ++++++++++++++++----------- block/mirror.c | 26 +-- block/qcow2-cluster.c | 2 +- block/qcow2.c | 116 ++++++---- qemu-img.c | 381 ++++++++++++++----------------- qemu-io-cmds.c | 13 -- block/trace-events | 2 +- qemu-img.texi | 9 +- tests/qemu-iotests/067 | 2 +- tests/qemu-iotests/074.out | 2 - tests/qemu-iotests/087 | 2 +- tests/qemu-iotests/125 | 7 +- tests/qemu-iotests/125.out | 480 +++++++++++++++++++++++++++++++++++---- tests/qemu-iotests/127 | 97 ++++++++ tests/qemu-iotests/127.out | 14 ++ tests/qemu-iotests/177 | 12 +- tests/qemu-iotests/177.out | 19 +- tests/qemu-iotests/184 | 3 +- tests/qemu-iotests/184.out | 6 +- tests/qemu-iotests/191 | 7 +- tests/qemu-iotests/191.out | 48 ++-- tests/qemu-iotests/common.filter | 6 + tests/qemu-iotests/group | 1 + 27 files changed, 1102 insertions(+), 543 deletions(-) create mode 100755 tests/qemu-iotests/127 create mode 100644 tests/qemu-iotests/127.out From: Peter Krempa <pkrempa@redhat.com> When referring to a backing file of an image via node name bdrv_open_backing_file would add the 'driver' option to the option list filling it with the backing format driver. This breaks construction of the backing chain via -blockdev, as bdrv_open_inherit reports an error if both 'reference' and 'options' are provided. $ qemu-img create -f raw /tmp/backing.raw 64M $ qemu-img create -f qcow2 -F raw -b /tmp/backing.raw /tmp/test.qcow2 $ qemu-system-x86_64 \ -blockdev driver=file,filename=/tmp/backing.raw,node-name=backing \ -blockdev driver=qcow2,file.driver=file,file.filename=/tmp/test.qcow2,node-name=root,backing=backing qemu-system-x86_64: -blockdev driver=qcow2,file.driver=file,file.filename=/tmp/test.qcow2,node-name=root,backing=backing: Could not open backing file: Cannot reference an existing block device with additional options or a new filename Signed-off-by: Peter Krempa <pkrempa@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ int bdrv_open_backing_file(BlockDriverState *bs, QDict *parent_options, goto free_exit; } - if (bs->backing_format[0] != '\0' && !qdict_haskey(options, "driver")) { + if (!reference && + bs->backing_format[0] != '\0' && !qdict_haskey(options, "driver")) { qdict_put_str(options, "driver", bs->backing_format); } -- 2.13.6 This changes test case 191 to include a backing image that has backing_fmt set in the image file, but is referenced by node name in the qemu command line. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- tests/qemu-iotests/191 | 3 ++- tests/qemu-iotests/191.out | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/191 b/tests/qemu-iotests/191 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/191 +++ b/tests/qemu-iotests/191 @@ -XXX,XX +XXX,XX @@ echo === Preparing and starting VM === echo TEST_IMG="${TEST_IMG}.base" _make_test_img $size -TEST_IMG="${TEST_IMG}.mid" _make_test_img -b "${TEST_IMG}.base" +IMGOPTS=$(_optstr_add "$IMGOPTS" "backing_fmt=$IMGFMT") \ + TEST_IMG="${TEST_IMG}.mid" _make_test_img -b "${TEST_IMG}.base" _make_test_img -b "${TEST_IMG}.mid" TEST_IMG="${TEST_IMG}.ovl2" _make_test_img -b "${TEST_IMG}.mid" diff --git a/tests/qemu-iotests/191.out b/tests/qemu-iotests/191.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/191.out +++ b/tests/qemu-iotests/191.out @@ -XXX,XX +XXX,XX @@ QA output created by 191 === Preparing and starting VM === Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 -Formatting 'TEST_DIR/t.IMGFMT.mid', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base +Formatting 'TEST_DIR/t.IMGFMT.mid', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.mid Formatting 'TEST_DIR/t.IMGFMT.ovl2', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.mid wrote 65536/65536 bytes at offset 1048576 -- 2.13.6 From: Eric Blake <eblake@redhat.com> Not all callers care about which BDS owns the mapping for a given range of the file. This patch merely simplifies the callers by consolidating the logic in the common call point, while guaranteeing a non-NULL file to all the driver callbacks, for no semantic change. The only caller that does not care about pnum is bdrv_is_allocated, as invoked by vvfat; we can likewise add assertions that the rest of the stack does not have to worry about a NULL pnum. Furthermore, this will also set the stage for a future cleanup: when a caller does not care about which BDS owns an offset, it would be nice to allow the driver to optimize things to not have to return BDRV_BLOCK_OFFSET_VALID in the first place. In the case of fragmented allocation (for example, it's fairly easy to create a qcow2 image where consecutive guest addresses are not at consecutive host addresses), the current contract requires bdrv_get_block_status() to clamp *pnum to the limit where host addresses are no longer consecutive, but allowing a NULL file means that *pnum could be set to the full length of known-allocated data. Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- include/block/block_int.h | 10 ++++++---- block/io.c | 49 ++++++++++++++++++++++++++--------------------- block/mirror.c | 3 +-- block/qcow2.c | 8 ++------ qemu-img.c | 10 ++++------ 5 files changed, 40 insertions(+), 40 deletions(-) diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ struct BlockDriver { int64_t offset, int bytes); /* - * Building block for bdrv_block_status[_above]. The driver should - * answer only according to the current layer, and should not - * set BDRV_BLOCK_ALLOCATED, but may set BDRV_BLOCK_RAW. See block.h - * for the meaning of _DATA, _ZERO, and _OFFSET_VALID. + * Building block for bdrv_block_status[_above] and + * bdrv_is_allocated[_above]. The driver should answer only + * according to the current layer, and should not set + * BDRV_BLOCK_ALLOCATED, but may set BDRV_BLOCK_RAW. See block.h + * for the meaning of _DATA, _ZERO, and _OFFSET_VALID. The block + * layer guarantees non-NULL pnum and file. */ int64_t coroutine_fn (*bdrv_co_get_block_status)(BlockDriverState *bs, int64_t sector_num, int nb_sectors, int *pnum, diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ int bdrv_make_zero(BdrvChild *child, BdrvRequestFlags flags) { int64_t target_sectors, ret, nb_sectors, sector_num = 0; BlockDriverState *bs = child->bs; - BlockDriverState *file; int n; target_sectors = bdrv_nb_sectors(bs); @@ -XXX,XX +XXX,XX @@ int bdrv_make_zero(BdrvChild *child, BdrvRequestFlags flags) if (nb_sectors <= 0) { return 0; } - ret = bdrv_get_block_status(bs, sector_num, nb_sectors, &n, &file); + ret = bdrv_get_block_status(bs, sector_num, nb_sectors, &n, NULL); if (ret < 0) { error_report("error getting block status at sector %" PRId64 ": %s", sector_num, strerror(-ret)); @@ -XXX,XX +XXX,XX @@ int64_t coroutine_fn bdrv_co_get_block_status_from_backing(BlockDriverState *bs, * beyond the end of the disk image it will be clamped; if 'pnum' is set to * the end of the image, then the returned value will include BDRV_BLOCK_EOF. * - * If returned value is positive and BDRV_BLOCK_OFFSET_VALID bit is set, 'file' - * points to the BDS which the sector range is allocated in. + * If returned value is positive, BDRV_BLOCK_OFFSET_VALID bit is set, and + * 'file' is non-NULL, then '*file' points to the BDS which the sector range + * is allocated in. */ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs, int64_t sector_num, @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs, int64_t total_sectors; int64_t n; int64_t ret, ret2; + BlockDriverState *local_file = NULL; - *file = NULL; + assert(pnum); + *pnum = 0; total_sectors = bdrv_nb_sectors(bs); if (total_sectors < 0) { - return total_sectors; + ret = total_sectors; + goto early_out; } if (sector_num >= total_sectors) { - *pnum = 0; - return BDRV_BLOCK_EOF; + ret = BDRV_BLOCK_EOF; + goto early_out; } if (!nb_sectors) { - *pnum = 0; - return 0; + ret = 0; + goto early_out; } n = total_sectors - sector_num; @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs, } if (bs->drv->protocol_name) { ret |= BDRV_BLOCK_OFFSET_VALID | (sector_num * BDRV_SECTOR_SIZE); - *file = bs; + local_file = bs; } - return ret; + goto early_out; } bdrv_inc_in_flight(bs); ret = bs->drv->bdrv_co_get_block_status(bs, sector_num, nb_sectors, pnum, - file); + &local_file); if (ret < 0) { *pnum = 0; goto out; } if (ret & BDRV_BLOCK_RAW) { - assert(ret & BDRV_BLOCK_OFFSET_VALID && *file); - ret = bdrv_co_get_block_status(*file, ret >> BDRV_SECTOR_BITS, - *pnum, pnum, file); + assert(ret & BDRV_BLOCK_OFFSET_VALID && local_file); + ret = bdrv_co_get_block_status(local_file, ret >> BDRV_SECTOR_BITS, + *pnum, pnum, &local_file); goto out; } @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs, } } - if (*file && *file != bs && + if (local_file && local_file != bs && (ret & BDRV_BLOCK_DATA) && !(ret & BDRV_BLOCK_ZERO) && (ret & BDRV_BLOCK_OFFSET_VALID)) { - BlockDriverState *file2; int file_pnum; - ret2 = bdrv_co_get_block_status(*file, ret >> BDRV_SECTOR_BITS, - *pnum, &file_pnum, &file2); + ret2 = bdrv_co_get_block_status(local_file, ret >> BDRV_SECTOR_BITS, + *pnum, &file_pnum, NULL); if (ret2 >= 0) { /* Ignore errors. This is just providing extra information, it * is useful but not necessary. @@ -XXX,XX +XXX,XX @@ out: if (ret >= 0 && sector_num + *pnum == total_sectors) { ret |= BDRV_BLOCK_EOF; } +early_out: + if (file) { + *file = local_file; + } return ret; } @@ -XXX,XX +XXX,XX @@ int64_t bdrv_get_block_status(BlockDriverState *bs, int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset, int64_t bytes, int64_t *pnum) { - BlockDriverState *file; int64_t sector_num = offset >> BDRV_SECTOR_BITS; int nb_sectors = bytes >> BDRV_SECTOR_BITS; int64_t ret; @@ -XXX,XX +XXX,XX @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset, assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE)); assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE) && bytes < INT_MAX); ret = bdrv_get_block_status(bs, sector_num, nb_sectors, &psectors, - &file); + NULL); if (ret < 0) { return ret; } diff --git a/block/mirror.c b/block/mirror.c index XXXXXXX..XXXXXXX 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -XXX,XX +XXX,XX @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) int io_sectors; unsigned int io_bytes; int64_t io_bytes_acct; - BlockDriverState *file; enum MirrorMethod { MIRROR_METHOD_COPY, MIRROR_METHOD_ZERO, @@ -XXX,XX +XXX,XX @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) ret = bdrv_get_block_status_above(source, NULL, offset >> BDRV_SECTOR_BITS, nb_chunks * sectors_per_chunk, - &io_sectors, &file); + &io_sectors, NULL); io_bytes = io_sectors * BDRV_SECTOR_SIZE; if (ret < 0) { io_bytes = MIN(nb_chunks * s->granularity, max_io_bytes); diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static bool is_zero_sectors(BlockDriverState *bs, int64_t start, uint32_t count) { int nr; - BlockDriverState *file; int64_t res; if (start + count > bs->total_sectors) { @@ -XXX,XX +XXX,XX @@ static bool is_zero_sectors(BlockDriverState *bs, int64_t start, if (!count) { return true; } - res = bdrv_get_block_status_above(bs, NULL, start, count, - &nr, &file); + res = bdrv_get_block_status_above(bs, NULL, start, count, &nr, NULL); return res >= 0 && (res & BDRV_BLOCK_ZERO) && nr == count; } @@ -XXX,XX +XXX,XX @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, offset += pnum * BDRV_SECTOR_SIZE) { int nb_sectors = MIN(ssize - offset, BDRV_REQUEST_MAX_BYTES) / BDRV_SECTOR_SIZE; - BlockDriverState *file; int64_t ret; ret = bdrv_get_block_status_above(in_bs, NULL, offset >> BDRV_SECTOR_BITS, - nb_sectors, - &pnum, &file); + nb_sectors, &pnum, NULL); if (ret < 0) { error_setg_errno(&local_err, -ret, "Unable to get block status"); diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) for (;;) { int64_t status1, status2; - BlockDriverState *file; nb_sectors = sectors_to_process(total_sectors, sector_num); if (nb_sectors <= 0) { @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) } status1 = bdrv_get_block_status_above(bs1, NULL, sector_num, total_sectors1 - sector_num, - &pnum1, &file); + &pnum1, NULL); if (status1 < 0) { ret = 3; error_report("Sector allocation test failed for %s", filename1); @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) status2 = bdrv_get_block_status_above(bs2, NULL, sector_num, total_sectors2 - sector_num, - &pnum2, &file); + &pnum2, NULL); if (status2 < 0) { ret = 3; error_report("Sector allocation test failed for %s", filename2); @@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num) n = MIN(s->total_sectors - sector_num, BDRV_REQUEST_MAX_SECTORS); if (s->sector_next_status <= sector_num) { - BlockDriverState *file; if (s->target_has_backing) { ret = bdrv_get_block_status(blk_bs(s->src[src_cur]), sector_num - src_cur_offset, - n, &n, &file); + n, &n, NULL); } else { ret = bdrv_get_block_status_above(blk_bs(s->src[src_cur]), NULL, sector_num - src_cur_offset, - n, &n, &file); + n, &n, NULL); } if (ret < 0) { return ret; -- 2.13.6 From: Eric Blake <eblake@redhat.com> Not all callers care about which BDS owns the mapping for a given range of the file, or where the zeroes lie within that mapping. In particular, bdrv_is_allocated() cares more about finding the largest run of allocated data from the guest perspective, whether or not that data is consecutive from the host perspective, and whether or not the data reads as zero. Therefore, doing subsequent refinements such as checking how much of the format-layer allocation also satisfies BDRV_BLOCK_ZERO at the protocol layer is wasted work - in the best case, it just costs extra CPU cycles during a single bdrv_is_allocated(), but in the worst case, it results in a smaller *pnum, and forces callers to iterate through more status probes when visiting the entire file for even more extra CPU cycles. This patch only optimizes the block layer (no behavior change when want_zero is true, but skip unnecessary effort when it is false). Then when subsequent patches tweak the driver callback to be byte-based, we can also pass this hint through to the driver. Tweak BdrvCoGetBlockStatusData to declare arguments in parameter order, rather than mixing things up (minimizing padding is not necessary here). Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/io.c | 57 +++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 41 insertions(+), 16 deletions(-) diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ int bdrv_flush_all(void) typedef struct BdrvCoGetBlockStatusData { BlockDriverState *bs; BlockDriverState *base; - BlockDriverState **file; + bool want_zero; int64_t sector_num; int nb_sectors; int *pnum; + BlockDriverState **file; int64_t ret; bool done; } BdrvCoGetBlockStatusData; @@ -XXX,XX +XXX,XX @@ int64_t coroutine_fn bdrv_co_get_block_status_from_backing(BlockDriverState *bs, * Drivers not implementing the functionality are assumed to not support * backing files, hence all their sectors are reported as allocated. * + * If 'want_zero' is true, the caller is querying for mapping purposes, + * and the result should include BDRV_BLOCK_OFFSET_VALID and + * BDRV_BLOCK_ZERO where possible; otherwise, the result may omit those + * bits particularly if it allows for a larger value in 'pnum'. + * * If 'sector_num' is beyond the end of the disk image the return value is * BDRV_BLOCK_EOF and 'pnum' is set to 0. * @@ -XXX,XX +XXX,XX @@ int64_t coroutine_fn bdrv_co_get_block_status_from_backing(BlockDriverState *bs, * is allocated in. */ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs, + bool want_zero, int64_t sector_num, int nb_sectors, int *pnum, BlockDriverState **file) @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs, if (ret & BDRV_BLOCK_RAW) { assert(ret & BDRV_BLOCK_OFFSET_VALID && local_file); - ret = bdrv_co_get_block_status(local_file, ret >> BDRV_SECTOR_BITS, + ret = bdrv_co_get_block_status(local_file, want_zero, + ret >> BDRV_SECTOR_BITS, *pnum, pnum, &local_file); goto out; } if (ret & (BDRV_BLOCK_DATA | BDRV_BLOCK_ZERO)) { ret |= BDRV_BLOCK_ALLOCATED; - } else { + } else if (want_zero) { if (bdrv_unallocated_blocks_are_zero(bs)) { ret |= BDRV_BLOCK_ZERO; } else if (bs->backing) { BlockDriverState *bs2 = bs->backing->bs; int64_t nb_sectors2 = bdrv_nb_sectors(bs2); + if (nb_sectors2 >= 0 && sector_num >= nb_sectors2) { ret |= BDRV_BLOCK_ZERO; } } } - if (local_file && local_file != bs && + if (want_zero && local_file && local_file != bs && (ret & BDRV_BLOCK_DATA) && !(ret & BDRV_BLOCK_ZERO) && (ret & BDRV_BLOCK_OFFSET_VALID)) { int file_pnum; - ret2 = bdrv_co_get_block_status(local_file, ret >> BDRV_SECTOR_BITS, + ret2 = bdrv_co_get_block_status(local_file, want_zero, + ret >> BDRV_SECTOR_BITS, *pnum, &file_pnum, NULL); if (ret2 >= 0) { /* Ignore errors. This is just providing extra information, it @@ -XXX,XX +XXX,XX @@ early_out: static int64_t coroutine_fn bdrv_co_get_block_status_above(BlockDriverState *bs, BlockDriverState *base, + bool want_zero, int64_t sector_num, int nb_sectors, int *pnum, @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status_above(BlockDriverState *bs, assert(bs != base); for (p = bs; p != base; p = backing_bs(p)) { - ret = bdrv_co_get_block_status(p, sector_num, nb_sectors, pnum, file); + ret = bdrv_co_get_block_status(p, want_zero, sector_num, nb_sectors, + pnum, file); if (ret < 0) { break; } @@ -XXX,XX +XXX,XX @@ static void coroutine_fn bdrv_get_block_status_above_co_entry(void *opaque) BdrvCoGetBlockStatusData *data = opaque; data->ret = bdrv_co_get_block_status_above(data->bs, data->base, + data->want_zero, data->sector_num, data->nb_sectors, data->pnum, @@ -XXX,XX +XXX,XX @@ static void coroutine_fn bdrv_get_block_status_above_co_entry(void *opaque) * * See bdrv_co_get_block_status_above() for details. */ -int64_t bdrv_get_block_status_above(BlockDriverState *bs, - BlockDriverState *base, - int64_t sector_num, - int nb_sectors, int *pnum, - BlockDriverState **file) +static int64_t bdrv_common_block_status_above(BlockDriverState *bs, + BlockDriverState *base, + bool want_zero, + int64_t sector_num, + int nb_sectors, int *pnum, + BlockDriverState **file) { Coroutine *co; BdrvCoGetBlockStatusData data = { .bs = bs, .base = base, - .file = file, + .want_zero = want_zero, .sector_num = sector_num, .nb_sectors = nb_sectors, .pnum = pnum, + .file = file, .done = false, }; @@ -XXX,XX +XXX,XX @@ int64_t bdrv_get_block_status_above(BlockDriverState *bs, return data.ret; } +int64_t bdrv_get_block_status_above(BlockDriverState *bs, + BlockDriverState *base, + int64_t sector_num, + int nb_sectors, int *pnum, + BlockDriverState **file) +{ + return bdrv_common_block_status_above(bs, base, true, sector_num, + nb_sectors, pnum, file); +} + int64_t bdrv_get_block_status(BlockDriverState *bs, int64_t sector_num, int nb_sectors, int *pnum, @@ -XXX,XX +XXX,XX @@ int64_t bdrv_get_block_status(BlockDriverState *bs, int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset, int64_t bytes, int64_t *pnum) { - int64_t sector_num = offset >> BDRV_SECTOR_BITS; - int nb_sectors = bytes >> BDRV_SECTOR_BITS; int64_t ret; int psectors; assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE)); assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE) && bytes < INT_MAX); - ret = bdrv_get_block_status(bs, sector_num, nb_sectors, &psectors, - NULL); + ret = bdrv_common_block_status_above(bs, backing_bs(bs), false, + offset >> BDRV_SECTOR_BITS, + bytes >> BDRV_SECTOR_BITS, &psectors, + NULL); if (ret < 0) { return ret; } -- 2.13.6 From: Eric Blake <eblake@redhat.com> In the process of converting sector-based interfaces to bytes, I'm finding it easier to represent a byte count as a 64-bit integer at the block layer (even if we are internally capped by SIZE_MAX or even INT_MAX for individual transactions, it's still nicer to not have to worry about truncation/overflow issues on as many variables). Update the signature of bdrv_round_to_clusters() to uniformly use int64_t, matching the signature already chosen for bdrv_is_allocated and the fact that off_t is also a signed type, then adjust clients according to the required fallout (even where the result could now exceed 32 bits, no client is directly assigning the result into a 32-bit value without breaking things into a loop first). Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- include/block/block.h | 4 ++-- block/io.c | 6 +++--- block/mirror.c | 7 +++---- block/trace-events | 2 +- 4 files changed, 9 insertions(+), 10 deletions(-) diff --git a/include/block/block.h b/include/block/block.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -XXX,XX +XXX,XX @@ int bdrv_get_flags(BlockDriverState *bs); int bdrv_get_info(BlockDriverState *bs, BlockDriverInfo *bdi); ImageInfoSpecific *bdrv_get_specific_info(BlockDriverState *bs); void bdrv_round_to_clusters(BlockDriverState *bs, - int64_t offset, unsigned int bytes, + int64_t offset, int64_t bytes, int64_t *cluster_offset, - unsigned int *cluster_bytes); + int64_t *cluster_bytes); const char *bdrv_get_encrypted_filename(BlockDriverState *bs); void bdrv_get_backing_filename(BlockDriverState *bs, diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ static void mark_request_serialising(BdrvTrackedRequest *req, uint64_t align) * Round a region to cluster boundaries */ void bdrv_round_to_clusters(BlockDriverState *bs, - int64_t offset, unsigned int bytes, + int64_t offset, int64_t bytes, int64_t *cluster_offset, - unsigned int *cluster_bytes) + int64_t *cluster_bytes) { BlockDriverInfo bdi; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_co_do_copy_on_readv(BdrvChild *child, struct iovec iov; QEMUIOVector local_qiov; int64_t cluster_offset; - unsigned int cluster_bytes; + int64_t cluster_bytes; size_t skip_bytes; int ret; int max_transfer = MIN_NON_ZERO(bs->bl.max_transfer, diff --git a/block/mirror.c b/block/mirror.c index XXXXXXX..XXXXXXX 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -XXX,XX +XXX,XX @@ static int mirror_cow_align(MirrorBlockJob *s, int64_t *offset, bool need_cow; int ret = 0; int64_t align_offset = *offset; - unsigned int align_bytes = *bytes; + int64_t align_bytes = *bytes; int max_bytes = s->granularity * s->max_iov; - assert(*bytes < INT_MAX); need_cow = !test_bit(*offset / s->granularity, s->cow_bitmap); need_cow |= !test_bit((*offset + *bytes - 1) / s->granularity, s->cow_bitmap); @@ -XXX,XX +XXX,XX @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) while (nb_chunks > 0 && offset < s->bdev_length) { int64_t ret; int io_sectors; - unsigned int io_bytes; + int64_t io_bytes; int64_t io_bytes_acct; enum MirrorMethod { MIRROR_METHOD_COPY, @@ -XXX,XX +XXX,XX @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) io_bytes = s->granularity; } else if (ret >= 0 && !(ret & BDRV_BLOCK_DATA)) { int64_t target_offset; - unsigned int target_bytes; + int64_t target_bytes; bdrv_round_to_clusters(blk_bs(s->target), offset, io_bytes, &target_offset, &target_bytes); if (target_offset == offset && diff --git a/block/trace-events b/block/trace-events index XXXXXXX..XXXXXXX 100644 --- a/block/trace-events +++ b/block/trace-events @@ -XXX,XX +XXX,XX @@ blk_co_pwritev(void *blk, void *bs, int64_t offset, unsigned int bytes, int flag bdrv_co_preadv(void *bs, int64_t offset, int64_t nbytes, unsigned int flags) "bs %p offset %"PRId64" nbytes %"PRId64" flags 0x%x" bdrv_co_pwritev(void *bs, int64_t offset, int64_t nbytes, unsigned int flags) "bs %p offset %"PRId64" nbytes %"PRId64" flags 0x%x" bdrv_co_pwrite_zeroes(void *bs, int64_t offset, int count, int flags) "bs %p offset %"PRId64" count %d flags 0x%x" -bdrv_co_do_copy_on_readv(void *bs, int64_t offset, unsigned int bytes, int64_t cluster_offset, unsigned int cluster_bytes) "bs %p offset %"PRId64" bytes %u cluster_offset %"PRId64" cluster_bytes %u" +bdrv_co_do_copy_on_readv(void *bs, int64_t offset, unsigned int bytes, int64_t cluster_offset, int64_t cluster_bytes) "bs %p offset %"PRId64" bytes %u cluster_offset %"PRId64" cluster_bytes %"PRId64 # block/stream.c stream_one_iteration(void *s, int64_t offset, uint64_t bytes, int is_allocated) "s %p offset %" PRId64 " bytes %" PRIu64 " is_allocated %d" -- 2.13.6 From: Eric Blake <eblake@redhat.com> We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Convert another internal function (no semantic change), and rename it to is_zero() in the process. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Fam Zheng <famz@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.c | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ finish: } -static bool is_zero_sectors(BlockDriverState *bs, int64_t start, - uint32_t count) +static bool is_zero(BlockDriverState *bs, int64_t offset, int64_t bytes) { int nr; int64_t res; + int64_t start; - if (start + count > bs->total_sectors) { - count = bs->total_sectors - start; + /* TODO: Widening to sector boundaries should only be needed as + * long as we can't query finer granularity. */ + start = QEMU_ALIGN_DOWN(offset, BDRV_SECTOR_SIZE); + bytes = QEMU_ALIGN_UP(offset + bytes, BDRV_SECTOR_SIZE) - start; + + /* Clamp to image length, before checking status of underlying sectors */ + if (start + bytes > bs->total_sectors * BDRV_SECTOR_SIZE) { + bytes = bs->total_sectors * BDRV_SECTOR_SIZE - start; } - if (!count) { + if (!bytes) { return true; } - res = bdrv_get_block_status_above(bs, NULL, start, count, &nr, NULL); - return res >= 0 && (res & BDRV_BLOCK_ZERO) && nr == count; + res = bdrv_get_block_status_above(bs, NULL, start >> BDRV_SECTOR_BITS, + bytes >> BDRV_SECTOR_BITS, &nr, NULL); + return res >= 0 && (res & BDRV_BLOCK_ZERO) && + nr * BDRV_SECTOR_SIZE == bytes; } static coroutine_fn int qcow2_co_pwrite_zeroes(BlockDriverState *bs, @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_pwrite_zeroes(BlockDriverState *bs, } if (head || tail) { - int64_t cl_start = (offset - head) >> BDRV_SECTOR_BITS; uint64_t off; unsigned int nr; assert(head + bytes <= s->cluster_size); /* check whether remainder of cluster already reads as zero */ - if (!(is_zero_sectors(bs, cl_start, - DIV_ROUND_UP(head, BDRV_SECTOR_SIZE)) && - is_zero_sectors(bs, (offset + bytes) >> BDRV_SECTOR_BITS, - DIV_ROUND_UP(-tail & (s->cluster_size - 1), - BDRV_SECTOR_SIZE)))) { + if (!(is_zero(bs, offset - head, head) && + is_zero(bs, offset + bytes, + tail ? s->cluster_size - tail : 0))) { return -ENOTSUP; } qemu_co_mutex_lock(&s->lock); /* We can have new write after previous check */ - offset = cl_start << BDRV_SECTOR_BITS; + offset = QEMU_ALIGN_DOWN(offset, s->cluster_size); bytes = s->cluster_size; nr = s->cluster_size; ret = qcow2_get_cluster_offset(bs, offset, &nr, &off); -- 2.13.6 From: Eric Blake <eblake@redhat.com> We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Change the internal loop iteration of zeroing a device to track by bytes instead of sectors (although we are still guaranteed that we iterate by steps that are sector-aligned). Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Fam Zheng <famz@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/io.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ int bdrv_pwrite_zeroes(BdrvChild *child, int64_t offset, */ int bdrv_make_zero(BdrvChild *child, BdrvRequestFlags flags) { - int64_t target_sectors, ret, nb_sectors, sector_num = 0; + int64_t target_size, ret, bytes, offset = 0; BlockDriverState *bs = child->bs; - int n; + int n; /* sectors */ - target_sectors = bdrv_nb_sectors(bs); - if (target_sectors < 0) { - return target_sectors; + target_size = bdrv_getlength(bs); + if (target_size < 0) { + return target_size; } for (;;) { - nb_sectors = MIN(target_sectors - sector_num, BDRV_REQUEST_MAX_SECTORS); - if (nb_sectors <= 0) { + bytes = MIN(target_size - offset, BDRV_REQUEST_MAX_BYTES); + if (bytes <= 0) { return 0; } - ret = bdrv_get_block_status(bs, sector_num, nb_sectors, &n, NULL); + ret = bdrv_get_block_status(bs, offset >> BDRV_SECTOR_BITS, + bytes >> BDRV_SECTOR_BITS, &n, NULL); if (ret < 0) { - error_report("error getting block status at sector %" PRId64 ": %s", - sector_num, strerror(-ret)); + error_report("error getting block status at offset %" PRId64 ": %s", + offset, strerror(-ret)); return ret; } if (ret & BDRV_BLOCK_ZERO) { - sector_num += n; + offset += n * BDRV_SECTOR_BITS; continue; } - ret = bdrv_pwrite_zeroes(child, sector_num << BDRV_SECTOR_BITS, - n << BDRV_SECTOR_BITS, flags); + ret = bdrv_pwrite_zeroes(child, offset, n * BDRV_SECTOR_SIZE, flags); if (ret < 0) { - error_report("error writing zeroes at sector %" PRId64 ": %s", - sector_num, strerror(-ret)); + error_report("error writing zeroes at offset %" PRId64 ": %s", + offset, strerror(-ret)); return ret; } - sector_num += n; + offset += n * BDRV_SECTOR_SIZE; } } -- 2.13.6 From: Eric Blake <eblake@redhat.com> We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Continue by converting an internal function (no semantic change), and simplifying its caller accordingly. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Fam Zheng <famz@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-img.c | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static void dump_map_entry(OutputFormat output_format, MapEntry *e, } } -static int get_block_status(BlockDriverState *bs, int64_t sector_num, - int nb_sectors, MapEntry *e) +static int get_block_status(BlockDriverState *bs, int64_t offset, + int64_t bytes, MapEntry *e) { int64_t ret; int depth; BlockDriverState *file; bool has_offset; + int nb_sectors = bytes >> BDRV_SECTOR_BITS; + assert(bytes < INT_MAX); /* As an optimization, we could cache the current range of unallocated * clusters in each file of the chain, and avoid querying the same * range repeatedly. @@ -XXX,XX +XXX,XX @@ static int get_block_status(BlockDriverState *bs, int64_t sector_num, depth = 0; for (;;) { - ret = bdrv_get_block_status(bs, sector_num, nb_sectors, &nb_sectors, - &file); + ret = bdrv_get_block_status(bs, offset >> BDRV_SECTOR_BITS, nb_sectors, + &nb_sectors, &file); if (ret < 0) { return ret; } @@ -XXX,XX +XXX,XX @@ static int get_block_status(BlockDriverState *bs, int64_t sector_num, has_offset = !!(ret & BDRV_BLOCK_OFFSET_VALID); *e = (MapEntry) { - .start = sector_num * BDRV_SECTOR_SIZE, + .start = offset, .length = nb_sectors * BDRV_SECTOR_SIZE, .data = !!(ret & BDRV_BLOCK_DATA), .zero = !!(ret & BDRV_BLOCK_ZERO), @@ -XXX,XX +XXX,XX @@ static int img_map(int argc, char **argv) length = blk_getlength(blk); while (curr.start + curr.length < length) { - int64_t nsectors_left; - int64_t sector_num; - int n; - - sector_num = (curr.start + curr.length) >> BDRV_SECTOR_BITS; + int64_t offset = curr.start + curr.length; + int64_t n; /* Probe up to 1 GiB at a time. */ - nsectors_left = DIV_ROUND_UP(length, BDRV_SECTOR_SIZE) - sector_num; - n = MIN(1 << (30 - BDRV_SECTOR_BITS), nsectors_left); - ret = get_block_status(bs, sector_num, n, &next); + n = QEMU_ALIGN_DOWN(MIN(1 << 30, length - offset), BDRV_SECTOR_SIZE); + ret = get_block_status(bs, offset, n, &next); if (ret < 0) { error_report("Could not read file metadata: %s", strerror(-ret)); -- 2.13.6 From: Eric Blake <eblake@redhat.com> We are gradually moving away from sector-based interfaces, towards byte-based. In the common case, allocation is unlikely to ever use values that are not naturally sector-aligned, but it is possible that byte-based values will let us be more precise about allocation at the end of an unaligned file that can do byte-based access. Changing the name of the function from bdrv_get_block_status() to bdrv_block_status() ensures that the compiler enforces that all callers are updated. For now, the io.c layer still assert()s that all callers are sector-aligned, but that can be relaxed when a later patch implements byte-based block status in the drivers. There was an inherent limitation in returning the offset via the return value: we only have room for BDRV_BLOCK_OFFSET_MASK bits, which means an offset can only be mapped for sector-aligned queries (or, if we declare that non-aligned input is at the same relative position modulo 512 of the answer), so the new interface also changes things to return the offset via output through a parameter by reference rather than mashed into the return value. We'll have some glue code that munges between the two styles until we finish converting all uses. For the most part this patch is just the addition of scaling at the callers followed by inverse scaling at bdrv_block_status(), coupled with the tweak in calling convention. But some code, particularly bdrv_is_allocated(), gets a lot simpler because it no longer has to mess with sectors. For ease of review, bdrv_get_block_status_above() will be tackled separately. Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- include/block/block.h | 17 +++++++++-------- block/io.c | 47 ++++++++++++++++++++++++++++++++++------------- block/qcow2-cluster.c | 2 +- qemu-img.c | 25 ++++++++++++++----------- 4 files changed, 58 insertions(+), 33 deletions(-) diff --git a/include/block/block.h b/include/block/block.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -XXX,XX +XXX,XX @@ typedef struct HDGeometry { #define BDRV_REQUEST_MAX_BYTES (BDRV_REQUEST_MAX_SECTORS << BDRV_SECTOR_BITS) /* - * Allocation status flags for bdrv_get_block_status() and friends. + * Allocation status flags for bdrv_block_status() and friends. * * Public flags: * BDRV_BLOCK_DATA: allocation for data at offset is tied to this layer @@ -XXX,XX +XXX,XX @@ typedef struct HDGeometry { * that the block layer recompute the answer from the returned * BDS; must be accompanied by just BDRV_BLOCK_OFFSET_VALID. * - * If BDRV_BLOCK_OFFSET_VALID is set, bits 9-62 (BDRV_BLOCK_OFFSET_MASK) - * represent the offset in the returned BDS that is allocated for the - * corresponding raw data; however, whether that offset actually contains - * data also depends on BDRV_BLOCK_DATA and BDRV_BLOCK_ZERO, as follows: + * If BDRV_BLOCK_OFFSET_VALID is set, bits 9-62 (BDRV_BLOCK_OFFSET_MASK) of + * the return value (old interface) or the entire map parameter (new + * interface) represent the offset in the returned BDS that is allocated for + * the corresponding raw data. However, whether that offset actually + * contains data also depends on BDRV_BLOCK_DATA, as follows: * * DATA ZERO OFFSET_VALID * t t t sectors read as zero, returned file is zero at offset @@ -XXX,XX +XXX,XX @@ int bdrv_has_zero_init_1(BlockDriverState *bs); int bdrv_has_zero_init(BlockDriverState *bs); bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs); bool bdrv_can_write_zeroes_with_unmap(BlockDriverState *bs); -int64_t bdrv_get_block_status(BlockDriverState *bs, int64_t sector_num, - int nb_sectors, int *pnum, - BlockDriverState **file); +int bdrv_block_status(BlockDriverState *bs, int64_t offset, + int64_t bytes, int64_t *pnum, int64_t *map, + BlockDriverState **file); int64_t bdrv_get_block_status_above(BlockDriverState *bs, BlockDriverState *base, int64_t sector_num, diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ int bdrv_pwrite_zeroes(BdrvChild *child, int64_t offset, */ int bdrv_make_zero(BdrvChild *child, BdrvRequestFlags flags) { - int64_t target_size, ret, bytes, offset = 0; + int ret; + int64_t target_size, bytes, offset = 0; BlockDriverState *bs = child->bs; - int n; /* sectors */ target_size = bdrv_getlength(bs); if (target_size < 0) { @@ -XXX,XX +XXX,XX @@ int bdrv_make_zero(BdrvChild *child, BdrvRequestFlags flags) if (bytes <= 0) { return 0; } - ret = bdrv_get_block_status(bs, offset >> BDRV_SECTOR_BITS, - bytes >> BDRV_SECTOR_BITS, &n, NULL); + ret = bdrv_block_status(bs, offset, bytes, &bytes, NULL, NULL); if (ret < 0) { error_report("error getting block status at offset %" PRId64 ": %s", offset, strerror(-ret)); return ret; } if (ret & BDRV_BLOCK_ZERO) { - offset += n * BDRV_SECTOR_BITS; + offset += bytes; continue; } - ret = bdrv_pwrite_zeroes(child, offset, n * BDRV_SECTOR_SIZE, flags); + ret = bdrv_pwrite_zeroes(child, offset, bytes, flags); if (ret < 0) { error_report("error writing zeroes at offset %" PRId64 ": %s", offset, strerror(-ret)); return ret; } - offset += n * BDRV_SECTOR_SIZE; + offset += bytes; } } @@ -XXX,XX +XXX,XX @@ int64_t bdrv_get_block_status_above(BlockDriverState *bs, nb_sectors, pnum, file); } -int64_t bdrv_get_block_status(BlockDriverState *bs, - int64_t sector_num, - int nb_sectors, int *pnum, - BlockDriverState **file) +int bdrv_block_status(BlockDriverState *bs, int64_t offset, int64_t bytes, + int64_t *pnum, int64_t *map, BlockDriverState **file) { - return bdrv_get_block_status_above(bs, backing_bs(bs), - sector_num, nb_sectors, pnum, file); + int64_t ret; + int n; + + assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); + assert(pnum); + /* + * The contract allows us to return pnum smaller than bytes, even + * if the next query would see the same status; we truncate the + * request to avoid overflowing the driver's 32-bit interface. + */ + bytes = MIN(bytes, BDRV_REQUEST_MAX_BYTES); + ret = bdrv_get_block_status_above(bs, backing_bs(bs), + offset >> BDRV_SECTOR_BITS, + bytes >> BDRV_SECTOR_BITS, &n, file); + if (ret < 0) { + assert(INT_MIN <= ret); + *pnum = 0; + return ret; + } + *pnum = n * BDRV_SECTOR_SIZE; + if (map) { + *map = ret & BDRV_BLOCK_OFFSET_MASK; + } else { + ret &= ~BDRV_BLOCK_OFFSET_VALID; + } + return ret & ~BDRV_BLOCK_OFFSET_MASK; } int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset, diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -XXX,XX +XXX,XX @@ static int discard_single_l2(BlockDriverState *bs, uint64_t offset, * cluster is already marked as zero, or if it's unallocated and we * don't have a backing file. * - * TODO We might want to use bdrv_get_block_status(bs) here, but we're + * TODO We might want to use bdrv_block_status(bs) here, but we're * holding s->lock, so that doesn't work today. * * If full_discard is true, the sector should not read back as zeroes, diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num) if (s->sector_next_status <= sector_num) { if (s->target_has_backing) { - ret = bdrv_get_block_status(blk_bs(s->src[src_cur]), - sector_num - src_cur_offset, - n, &n, NULL); + int64_t count = n * BDRV_SECTOR_SIZE; + + ret = bdrv_block_status(blk_bs(s->src[src_cur]), + (sector_num - src_cur_offset) * + BDRV_SECTOR_SIZE, + count, &count, NULL, NULL); + assert(ret < 0 || QEMU_IS_ALIGNED(count, BDRV_SECTOR_SIZE)); + n = count >> BDRV_SECTOR_BITS; } else { ret = bdrv_get_block_status_above(blk_bs(s->src[src_cur]), NULL, sector_num - src_cur_offset, @@ -XXX,XX +XXX,XX @@ static void dump_map_entry(OutputFormat output_format, MapEntry *e, static int get_block_status(BlockDriverState *bs, int64_t offset, int64_t bytes, MapEntry *e) { - int64_t ret; + int ret; int depth; BlockDriverState *file; bool has_offset; - int nb_sectors = bytes >> BDRV_SECTOR_BITS; + int64_t map; - assert(bytes < INT_MAX); /* As an optimization, we could cache the current range of unallocated * clusters in each file of the chain, and avoid querying the same * range repeatedly. @@ -XXX,XX +XXX,XX @@ static int get_block_status(BlockDriverState *bs, int64_t offset, depth = 0; for (;;) { - ret = bdrv_get_block_status(bs, offset >> BDRV_SECTOR_BITS, nb_sectors, - &nb_sectors, &file); + ret = bdrv_block_status(bs, offset, bytes, &bytes, &map, &file); if (ret < 0) { return ret; } - assert(nb_sectors); + assert(bytes); if (ret & (BDRV_BLOCK_ZERO|BDRV_BLOCK_DATA)) { break; } @@ -XXX,XX +XXX,XX @@ static int get_block_status(BlockDriverState *bs, int64_t offset, *e = (MapEntry) { .start = offset, - .length = nb_sectors * BDRV_SECTOR_SIZE, + .length = bytes, .data = !!(ret & BDRV_BLOCK_DATA), .zero = !!(ret & BDRV_BLOCK_ZERO), - .offset = ret & BDRV_BLOCK_OFFSET_MASK, + .offset = map, .has_offset = has_offset, .depth = depth, .has_filename = file && has_offset, -- 2.13.6 From: Eric Blake <eblake@redhat.com> We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Convert another internal function (no semantic change); and as with its public counterpart, rename to bdrv_co_block_status() and split the offset return, to make the compiler enforce that we catch all uses. For now, we assert that callers and the return value still use aligned data, but ultimately, this will be the function where we hand off to a byte-based driver callback, and will eventually need to add logic to ensure we round calls according to the driver's request_alignment then touch up the result handed back to the caller, to start permitting a caller to pass unaligned offsets. Note that we are now prepared to accepts 'bytes' larger than INT_MAX; this is okay as long as we clamp things internally before violating any 32-bit limits, and makes no difference to how a client will use the information (clients looping over the entire file must already be prepared for consecutive calls to return the same status, as drivers are already free to return shorter-than-maximal status due to any other convenient split points, such as when the L2 table crosses cluster boundaries in qcow2). Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/io.c | 124 ++++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 81 insertions(+), 43 deletions(-) diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ int64_t coroutine_fn bdrv_co_get_block_status_from_backing(BlockDriverState *bs, * BDRV_BLOCK_ZERO where possible; otherwise, the result may omit those * bits particularly if it allows for a larger value in 'pnum'. * - * If 'sector_num' is beyond the end of the disk image the return value is + * If 'offset' is beyond the end of the disk image the return value is * BDRV_BLOCK_EOF and 'pnum' is set to 0. * - * 'pnum' is set to the number of sectors (including and immediately following - * the specified sector) that are known to be in the same - * allocated/unallocated state. - * - * 'nb_sectors' is the max value 'pnum' should be set to. If nb_sectors goes + * 'bytes' is the max value 'pnum' should be set to. If bytes goes * beyond the end of the disk image it will be clamped; if 'pnum' is set to * the end of the image, then the returned value will include BDRV_BLOCK_EOF. * - * If returned value is positive, BDRV_BLOCK_OFFSET_VALID bit is set, and - * 'file' is non-NULL, then '*file' points to the BDS which the sector range - * is allocated in. + * 'pnum' is set to the number of bytes (including and immediately + * following the specified offset) that are easily known to be in the + * same allocated/unallocated state. Note that a second call starting + * at the original offset plus returned pnum may have the same status. + * The returned value is non-zero on success except at end-of-file. + * + * Returns negative errno on failure. Otherwise, if the + * BDRV_BLOCK_OFFSET_VALID bit is set, 'map' and 'file' (if non-NULL) are + * set to the host mapping and BDS corresponding to the guest offset. */ -static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs, - bool want_zero, - int64_t sector_num, - int nb_sectors, int *pnum, - BlockDriverState **file) -{ - int64_t total_sectors; - int64_t n; - int64_t ret, ret2; +static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs, + bool want_zero, + int64_t offset, int64_t bytes, + int64_t *pnum, int64_t *map, + BlockDriverState **file) +{ + int64_t total_size; + int64_t n; /* bytes */ + int64_t ret; + int64_t local_map = 0; BlockDriverState *local_file = NULL; + int count; /* sectors */ assert(pnum); *pnum = 0; - total_sectors = bdrv_nb_sectors(bs); - if (total_sectors < 0) { - ret = total_sectors; + total_size = bdrv_getlength(bs); + if (total_size < 0) { + ret = total_size; goto early_out; } - if (sector_num >= total_sectors) { + if (offset >= total_size) { ret = BDRV_BLOCK_EOF; goto early_out; } - if (!nb_sectors) { + if (!bytes) { ret = 0; goto early_out; } - n = total_sectors - sector_num; - if (n < nb_sectors) { - nb_sectors = n; + n = total_size - offset; + if (n < bytes) { + bytes = n; } if (!bs->drv->bdrv_co_get_block_status) { - *pnum = nb_sectors; + *pnum = bytes; ret = BDRV_BLOCK_DATA | BDRV_BLOCK_ALLOCATED; - if (sector_num + nb_sectors == total_sectors) { + if (offset + bytes == total_size) { ret |= BDRV_BLOCK_EOF; } if (bs->drv->protocol_name) { - ret |= BDRV_BLOCK_OFFSET_VALID | (sector_num * BDRV_SECTOR_SIZE); + ret |= BDRV_BLOCK_OFFSET_VALID; + local_map = offset; local_file = bs; } goto early_out; } bdrv_inc_in_flight(bs); - ret = bs->drv->bdrv_co_get_block_status(bs, sector_num, nb_sectors, pnum, + /* + * TODO: Rather than require aligned offsets, we could instead + * round to the driver's request_alignment here, then touch up + * count afterwards back to the caller's expectations. + */ + assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); + /* + * The contract allows us to return pnum smaller than bytes, even + * if the next query would see the same status; we truncate the + * request to avoid overflowing the driver's 32-bit interface. + */ + bytes = MIN(bytes, BDRV_REQUEST_MAX_BYTES); + ret = bs->drv->bdrv_co_get_block_status(bs, offset >> BDRV_SECTOR_BITS, + bytes >> BDRV_SECTOR_BITS, &count, &local_file); if (ret < 0) { - *pnum = 0; goto out; } + if (ret & BDRV_BLOCK_OFFSET_VALID) { + local_map = ret & BDRV_BLOCK_OFFSET_MASK; + } + *pnum = count * BDRV_SECTOR_SIZE; if (ret & BDRV_BLOCK_RAW) { assert(ret & BDRV_BLOCK_OFFSET_VALID && local_file); - ret = bdrv_co_get_block_status(local_file, want_zero, - ret >> BDRV_SECTOR_BITS, - *pnum, pnum, &local_file); + ret = bdrv_co_block_status(local_file, want_zero, local_map, + *pnum, pnum, &local_map, &local_file); + assert(ret < 0 || + QEMU_IS_ALIGNED(*pnum | local_map, BDRV_SECTOR_SIZE)); goto out; } @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs, ret |= BDRV_BLOCK_ZERO; } else if (bs->backing) { BlockDriverState *bs2 = bs->backing->bs; - int64_t nb_sectors2 = bdrv_nb_sectors(bs2); + int64_t size2 = bdrv_getlength(bs2); - if (nb_sectors2 >= 0 && sector_num >= nb_sectors2) { + if (size2 >= 0 && offset >= size2) { ret |= BDRV_BLOCK_ZERO; } } @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs, if (want_zero && local_file && local_file != bs && (ret & BDRV_BLOCK_DATA) && !(ret & BDRV_BLOCK_ZERO) && (ret & BDRV_BLOCK_OFFSET_VALID)) { - int file_pnum; + int64_t file_pnum; + int ret2; - ret2 = bdrv_co_get_block_status(local_file, want_zero, - ret >> BDRV_SECTOR_BITS, - *pnum, &file_pnum, NULL); + ret2 = bdrv_co_block_status(local_file, want_zero, local_map, + *pnum, &file_pnum, NULL, NULL); if (ret2 >= 0) { /* Ignore errors. This is just providing extra information, it * is useful but not necessary. @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status(BlockDriverState *bs, out: bdrv_dec_in_flight(bs); - if (ret >= 0 && sector_num + *pnum == total_sectors) { + if (ret >= 0 && offset + *pnum == total_size) { ret |= BDRV_BLOCK_EOF; } early_out: if (file) { *file = local_file; } + if (map) { + *map = local_map; + } + if (ret >= 0) { + ret &= ~BDRV_BLOCK_OFFSET_MASK; + } else { + assert(INT_MIN <= ret); + } return ret; } @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status_above(BlockDriverState *bs, BlockDriverState *p; int64_t ret = 0; bool first = true; + int64_t map = 0; assert(bs != base); for (p = bs; p != base; p = backing_bs(p)) { - ret = bdrv_co_get_block_status(p, want_zero, sector_num, nb_sectors, - pnum, file); + int64_t count; + + ret = bdrv_co_block_status(p, want_zero, + sector_num * BDRV_SECTOR_SIZE, + nb_sectors * BDRV_SECTOR_SIZE, &count, + &map, file); if (ret < 0) { break; } + assert(QEMU_IS_ALIGNED(count | map, BDRV_SECTOR_SIZE)); + ret |= map; + *pnum = count >> BDRV_SECTOR_BITS; if (ret & BDRV_BLOCK_ZERO && ret & BDRV_BLOCK_EOF && !first) { /* * Reading beyond the end of the file continues to read -- 2.13.6 From: Eric Blake <eblake@redhat.com> We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Convert another internal type (no semantic change), and rename it to match the corresponding public function rename. Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/io.c | 56 ++++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 18 deletions(-) diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ int bdrv_flush_all(void) } -typedef struct BdrvCoGetBlockStatusData { +typedef struct BdrvCoBlockStatusData { BlockDriverState *bs; BlockDriverState *base; bool want_zero; - int64_t sector_num; - int nb_sectors; - int *pnum; + int64_t offset; + int64_t bytes; + int64_t *pnum; + int64_t *map; BlockDriverState **file; - int64_t ret; + int ret; bool done; -} BdrvCoGetBlockStatusData; +} BdrvCoBlockStatusData; int64_t coroutine_fn bdrv_co_get_block_status_from_file(BlockDriverState *bs, int64_t sector_num, @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status_above(BlockDriverState *bs, /* Coroutine wrapper for bdrv_get_block_status_above() */ static void coroutine_fn bdrv_get_block_status_above_co_entry(void *opaque) { - BdrvCoGetBlockStatusData *data = opaque; + BdrvCoBlockStatusData *data = opaque; + int n = 0; + int64_t ret; - data->ret = bdrv_co_get_block_status_above(data->bs, data->base, - data->want_zero, - data->sector_num, - data->nb_sectors, - data->pnum, - data->file); + ret = bdrv_co_get_block_status_above(data->bs, data->base, + data->want_zero, + data->offset >> BDRV_SECTOR_BITS, + data->bytes >> BDRV_SECTOR_BITS, + &n, + data->file); + if (ret < 0) { + assert(INT_MIN <= ret); + data->ret = ret; + } else { + *data->pnum = n * BDRV_SECTOR_SIZE; + *data->map = ret & BDRV_BLOCK_OFFSET_MASK; + data->ret = ret & ~BDRV_BLOCK_OFFSET_MASK; + } data->done = true; } @@ -XXX,XX +XXX,XX @@ static int64_t bdrv_common_block_status_above(BlockDriverState *bs, BlockDriverState **file) { Coroutine *co; - BdrvCoGetBlockStatusData data = { + int64_t n; + int64_t map; + BdrvCoBlockStatusData data = { .bs = bs, .base = base, .want_zero = want_zero, - .sector_num = sector_num, - .nb_sectors = nb_sectors, - .pnum = pnum, + .offset = sector_num * BDRV_SECTOR_SIZE, + .bytes = nb_sectors * BDRV_SECTOR_SIZE, + .pnum = &n, + .map = &map, .file = file, .done = false, }; @@ -XXX,XX +XXX,XX @@ static int64_t bdrv_common_block_status_above(BlockDriverState *bs, bdrv_coroutine_enter(bs, co); BDRV_POLL_WHILE(bs, !data.done); } - return data.ret; + if (data.ret < 0) { + *pnum = 0; + return data.ret; + } + assert(QEMU_IS_ALIGNED(n | map, BDRV_SECTOR_SIZE)); + *pnum = n >> BDRV_SECTOR_BITS; + return data.ret | map; } int64_t bdrv_get_block_status_above(BlockDriverState *bs, -- 2.13.6 From: Eric Blake <eblake@redhat.com> We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Convert another internal function (no semantic change). Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/io.c | 61 ++++++++++++++++++++++++++++++------------------------------- 1 file changed, 30 insertions(+), 31 deletions(-) diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ static void coroutine_fn bdrv_get_block_status_above_co_entry(void *opaque) * * See bdrv_co_get_block_status_above() for details. */ -static int64_t bdrv_common_block_status_above(BlockDriverState *bs, - BlockDriverState *base, - bool want_zero, - int64_t sector_num, - int nb_sectors, int *pnum, - BlockDriverState **file) +static int bdrv_common_block_status_above(BlockDriverState *bs, + BlockDriverState *base, + bool want_zero, int64_t offset, + int64_t bytes, int64_t *pnum, + int64_t *map, + BlockDriverState **file) { Coroutine *co; - int64_t n; - int64_t map; BdrvCoBlockStatusData data = { .bs = bs, .base = base, .want_zero = want_zero, - .offset = sector_num * BDRV_SECTOR_SIZE, - .bytes = nb_sectors * BDRV_SECTOR_SIZE, - .pnum = &n, - .map = &map, + .offset = offset, + .bytes = bytes, + .pnum = pnum, + .map = map, .file = file, .done = false, }; @@ -XXX,XX +XXX,XX @@ static int64_t bdrv_common_block_status_above(BlockDriverState *bs, bdrv_coroutine_enter(bs, co); BDRV_POLL_WHILE(bs, !data.done); } - if (data.ret < 0) { - *pnum = 0; - return data.ret; - } - assert(QEMU_IS_ALIGNED(n | map, BDRV_SECTOR_SIZE)); - *pnum = n >> BDRV_SECTOR_BITS; - return data.ret | map; + return data.ret; } int64_t bdrv_get_block_status_above(BlockDriverState *bs, @@ -XXX,XX +XXX,XX @@ int64_t bdrv_get_block_status_above(BlockDriverState *bs, int nb_sectors, int *pnum, BlockDriverState **file) { - return bdrv_common_block_status_above(bs, base, true, sector_num, - nb_sectors, pnum, file); + int64_t ret; + int64_t n; + int64_t map; + + ret = bdrv_common_block_status_above(bs, base, true, + sector_num * BDRV_SECTOR_SIZE, + nb_sectors * BDRV_SECTOR_SIZE, + &n, &map, file); + if (ret < 0) { + *pnum = 0; + return ret; + } + assert(QEMU_IS_ALIGNED(n | map, BDRV_SECTOR_SIZE)); + *pnum = n >> BDRV_SECTOR_BITS; + return ret | map; } int bdrv_block_status(BlockDriverState *bs, int64_t offset, int64_t bytes, @@ -XXX,XX +XXX,XX @@ int bdrv_block_status(BlockDriverState *bs, int64_t offset, int64_t bytes, int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset, int64_t bytes, int64_t *pnum) { - int64_t ret; - int psectors; + int ret; + int64_t dummy; - assert(QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE)); - assert(QEMU_IS_ALIGNED(bytes, BDRV_SECTOR_SIZE) && bytes < INT_MAX); - ret = bdrv_common_block_status_above(bs, backing_bs(bs), false, - offset >> BDRV_SECTOR_BITS, - bytes >> BDRV_SECTOR_BITS, &psectors, + ret = bdrv_common_block_status_above(bs, backing_bs(bs), false, offset, + bytes, pnum ? pnum : &dummy, NULL, NULL); if (ret < 0) { return ret; } - if (pnum) { - *pnum = psectors * BDRV_SECTOR_SIZE; - } return !!(ret & BDRV_BLOCK_ALLOCATED); } -- 2.13.6 From: Eric Blake <eblake@redhat.com> We are gradually converting to byte-based interfaces, as they are easier to reason about than sector-based. Convert another internal type (no semantic change), and rename it to match the corresponding public function rename. Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/io.c | 68 ++++++++++++++++++++++---------------------------------------- 1 file changed, 24 insertions(+), 44 deletions(-) diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ early_out: return ret; } -static int64_t coroutine_fn bdrv_co_get_block_status_above(BlockDriverState *bs, - BlockDriverState *base, - bool want_zero, - int64_t sector_num, - int nb_sectors, - int *pnum, - BlockDriverState **file) +static int coroutine_fn bdrv_co_block_status_above(BlockDriverState *bs, + BlockDriverState *base, + bool want_zero, + int64_t offset, + int64_t bytes, + int64_t *pnum, + int64_t *map, + BlockDriverState **file) { BlockDriverState *p; - int64_t ret = 0; + int ret = 0; bool first = true; - int64_t map = 0; assert(bs != base); for (p = bs; p != base; p = backing_bs(p)) { - int64_t count; - - ret = bdrv_co_block_status(p, want_zero, - sector_num * BDRV_SECTOR_SIZE, - nb_sectors * BDRV_SECTOR_SIZE, &count, - &map, file); + ret = bdrv_co_block_status(p, want_zero, offset, bytes, pnum, map, + file); if (ret < 0) { break; } - assert(QEMU_IS_ALIGNED(count | map, BDRV_SECTOR_SIZE)); - ret |= map; - *pnum = count >> BDRV_SECTOR_BITS; if (ret & BDRV_BLOCK_ZERO && ret & BDRV_BLOCK_EOF && !first) { /* * Reading beyond the end of the file continues to read @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn bdrv_co_get_block_status_above(BlockDriverState *bs, * unallocated length we learned from an earlier * iteration. */ - *pnum = nb_sectors; + *pnum = bytes; } if (ret & (BDRV_BLOCK_ZERO | BDRV_BLOCK_DATA)) { break; } - /* [sector_num, pnum] unallocated on this layer, which could be only - * the first part of [sector_num, nb_sectors]. */ - nb_sectors = MIN(nb_sectors, *pnum); + /* [offset, pnum] unallocated on this layer, which could be only + * the first part of [offset, bytes]. */ + bytes = MIN(bytes, *pnum); first = false; } return ret; } /* Coroutine wrapper for bdrv_get_block_status_above() */ -static void coroutine_fn bdrv_get_block_status_above_co_entry(void *opaque) +static void coroutine_fn bdrv_block_status_above_co_entry(void *opaque) { BdrvCoBlockStatusData *data = opaque; - int n = 0; - int64_t ret; - ret = bdrv_co_get_block_status_above(data->bs, data->base, - data->want_zero, - data->offset >> BDRV_SECTOR_BITS, - data->bytes >> BDRV_SECTOR_BITS, - &n, - data->file); - if (ret < 0) { - assert(INT_MIN <= ret); - data->ret = ret; - } else { - *data->pnum = n * BDRV_SECTOR_SIZE; - *data->map = ret & BDRV_BLOCK_OFFSET_MASK; - data->ret = ret & ~BDRV_BLOCK_OFFSET_MASK; - } + data->ret = bdrv_co_block_status_above(data->bs, data->base, + data->want_zero, + data->offset, data->bytes, + data->pnum, data->map, data->file); data->done = true; } /* - * Synchronous wrapper around bdrv_co_get_block_status_above(). + * Synchronous wrapper around bdrv_co_block_status_above(). * - * See bdrv_co_get_block_status_above() for details. + * See bdrv_co_block_status_above() for details. */ static int bdrv_common_block_status_above(BlockDriverState *bs, BlockDriverState *base, @@ -XXX,XX +XXX,XX @@ static int bdrv_common_block_status_above(BlockDriverState *bs, if (qemu_in_coroutine()) { /* Fast-path if already in coroutine context */ - bdrv_get_block_status_above_co_entry(&data); + bdrv_block_status_above_co_entry(&data); } else { - co = qemu_coroutine_create(bdrv_get_block_status_above_co_entry, - &data); + co = qemu_coroutine_create(bdrv_block_status_above_co_entry, &data); bdrv_coroutine_enter(bs, co); BDRV_POLL_WHILE(bs, !data.done); } -- 2.13.6 From: Eric Blake <eblake@redhat.com> We are gradually moving away from sector-based interfaces, towards byte-based. In the common case, allocation is unlikely to ever use values that are not naturally sector-aligned, but it is possible that byte-based values will let us be more precise about allocation at the end of an unaligned file that can do byte-based access. Changing the name of the function from bdrv_get_block_status_above() to bdrv_block_status_above() ensures that the compiler enforces that all callers are updated. Likewise, since it a byte interface allows an offset mapping that might not be sector aligned, split the mapping out of the return value and into a pass-by-reference parameter. For now, the io.c layer still assert()s that all uses are sector-aligned, but that can be relaxed when a later patch implements byte-based block status in the drivers. For the most part this patch is just the addition of scaling at the callers followed by inverse scaling at bdrv_block_status(), plus updates for the new split return interface. But some code, particularly bdrv_block_status(), gets a lot simpler because it no longer has to mess with sectors. Likewise, mirror code no longer computes s->granularity >> BDRV_SECTOR_BITS, and can therefore drop an assertion about alignment because the loop no longer depends on alignment (never mind that we don't really have a driver that reports sub-sector alignments, so it's not really possible to test the effect of sub-sector mirroring). Fix a neighboring assertion to use is_power_of_2 while there. For ease of review, bdrv_get_block_status() was tackled separately. Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- include/block/block.h | 8 +++----- block/io.c | 55 ++++++++------------------------------------------- block/mirror.c | 18 ++++++----------- block/qcow2.c | 30 +++++++++++----------------- qemu-img.c | 49 +++++++++++++++++++++++++-------------------- 5 files changed, 57 insertions(+), 103 deletions(-) diff --git a/include/block/block.h b/include/block/block.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -XXX,XX +XXX,XX @@ bool bdrv_can_write_zeroes_with_unmap(BlockDriverState *bs); int bdrv_block_status(BlockDriverState *bs, int64_t offset, int64_t bytes, int64_t *pnum, int64_t *map, BlockDriverState **file); -int64_t bdrv_get_block_status_above(BlockDriverState *bs, - BlockDriverState *base, - int64_t sector_num, - int nb_sectors, int *pnum, - BlockDriverState **file); +int bdrv_block_status_above(BlockDriverState *bs, BlockDriverState *base, + int64_t offset, int64_t bytes, int64_t *pnum, + int64_t *map, BlockDriverState **file); int bdrv_is_allocated(BlockDriverState *bs, int64_t offset, int64_t bytes, int64_t *pnum); int bdrv_is_allocated_above(BlockDriverState *top, BlockDriverState *base, diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_co_block_status_above(BlockDriverState *bs, return ret; } -/* Coroutine wrapper for bdrv_get_block_status_above() */ +/* Coroutine wrapper for bdrv_block_status_above() */ static void coroutine_fn bdrv_block_status_above_co_entry(void *opaque) { BdrvCoBlockStatusData *data = opaque; @@ -XXX,XX +XXX,XX @@ static int bdrv_common_block_status_above(BlockDriverState *bs, return data.ret; } -int64_t bdrv_get_block_status_above(BlockDriverState *bs, - BlockDriverState *base, - int64_t sector_num, - int nb_sectors, int *pnum, - BlockDriverState **file) +int bdrv_block_status_above(BlockDriverState *bs, BlockDriverState *base, + int64_t offset, int64_t bytes, int64_t *pnum, + int64_t *map, BlockDriverState **file) { - int64_t ret; - int64_t n; - int64_t map; - - ret = bdrv_common_block_status_above(bs, base, true, - sector_num * BDRV_SECTOR_SIZE, - nb_sectors * BDRV_SECTOR_SIZE, - &n, &map, file); - if (ret < 0) { - *pnum = 0; - return ret; - } - assert(QEMU_IS_ALIGNED(n | map, BDRV_SECTOR_SIZE)); - *pnum = n >> BDRV_SECTOR_BITS; - return ret | map; + return bdrv_common_block_status_above(bs, base, true, offset, bytes, + pnum, map, file); } int bdrv_block_status(BlockDriverState *bs, int64_t offset, int64_t bytes, int64_t *pnum, int64_t *map, BlockDriverState **file) { - int64_t ret; - int n; - - assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); - assert(pnum); - /* - * The contract allows us to return pnum smaller than bytes, even - * if the next query would see the same status; we truncate the - * request to avoid overflowing the driver's 32-bit interface. - */ - bytes = MIN(bytes, BDRV_REQUEST_MAX_BYTES); - ret = bdrv_get_block_status_above(bs, backing_bs(bs), - offset >> BDRV_SECTOR_BITS, - bytes >> BDRV_SECTOR_BITS, &n, file); - if (ret < 0) { - assert(INT_MIN <= ret); - *pnum = 0; - return ret; - } - *pnum = n * BDRV_SECTOR_SIZE; - if (map) { - *map = ret & BDRV_BLOCK_OFFSET_MASK; - } else { - ret &= ~BDRV_BLOCK_OFFSET_VALID; - } - return ret & ~BDRV_BLOCK_OFFSET_MASK; + return bdrv_block_status_above(bs, backing_bs(bs), + offset, bytes, pnum, map, file); } int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset, diff --git a/block/mirror.c b/block/mirror.c index XXXXXXX..XXXXXXX 100644 --- a/block/mirror.c +++ b/block/mirror.c @@ -XXX,XX +XXX,XX @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) uint64_t delay_ns = 0; /* At least the first dirty chunk is mirrored in one iteration. */ int nb_chunks = 1; - int sectors_per_chunk = s->granularity >> BDRV_SECTOR_BITS; bool write_zeroes_ok = bdrv_can_write_zeroes_with_unmap(blk_bs(s->target)); int max_io_bytes = MAX(s->buf_size / MAX_IN_FLIGHT, MAX_IO_BYTES); @@ -XXX,XX +XXX,XX @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) } /* Clear dirty bits before querying the block status, because - * calling bdrv_get_block_status_above could yield - if some blocks are + * calling bdrv_block_status_above could yield - if some blocks are * marked dirty in this window, we need to know. */ bdrv_reset_dirty_bitmap_locked(s->dirty_bitmap, offset, @@ -XXX,XX +XXX,XX @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) bitmap_set(s->in_flight_bitmap, offset / s->granularity, nb_chunks); while (nb_chunks > 0 && offset < s->bdev_length) { - int64_t ret; - int io_sectors; + int ret; int64_t io_bytes; int64_t io_bytes_acct; enum MirrorMethod { @@ -XXX,XX +XXX,XX @@ static uint64_t coroutine_fn mirror_iteration(MirrorBlockJob *s) } mirror_method = MIRROR_METHOD_COPY; assert(!(offset % s->granularity)); - ret = bdrv_get_block_status_above(source, NULL, - offset >> BDRV_SECTOR_BITS, - nb_chunks * sectors_per_chunk, - &io_sectors, NULL); - io_bytes = io_sectors * BDRV_SECTOR_SIZE; + ret = bdrv_block_status_above(source, NULL, offset, + nb_chunks * s->granularity, + &io_bytes, NULL, NULL); if (ret < 0) { io_bytes = MIN(nb_chunks * s->granularity, max_io_bytes); } else if (ret & BDRV_BLOCK_DATA) { @@ -XXX,XX +XXX,XX @@ static void mirror_start_job(const char *job_id, BlockDriverState *bs, granularity = bdrv_get_default_bitmap_granularity(target); } - assert ((granularity & (granularity - 1)) == 0); - /* Granularity must be large enough for sector-based dirty bitmap */ - assert(granularity >= BDRV_SECTOR_SIZE); + assert(is_power_of_2(granularity)); if (buf_size < 0) { error_setg(errp, "Invalid parameter 'buf-size'"); diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ finish: static bool is_zero(BlockDriverState *bs, int64_t offset, int64_t bytes) { - int nr; - int64_t res; + int64_t nr; + int res; int64_t start; /* TODO: Widening to sector boundaries should only be needed as @@ -XXX,XX +XXX,XX @@ static bool is_zero(BlockDriverState *bs, int64_t offset, int64_t bytes) if (!bytes) { return true; } - res = bdrv_get_block_status_above(bs, NULL, start >> BDRV_SECTOR_BITS, - bytes >> BDRV_SECTOR_BITS, &nr, NULL); - return res >= 0 && (res & BDRV_BLOCK_ZERO) && - nr * BDRV_SECTOR_SIZE == bytes; + res = bdrv_block_status_above(bs, NULL, start, bytes, &nr, NULL, NULL); + return res >= 0 && (res & BDRV_BLOCK_ZERO) && nr == bytes; } static coroutine_fn int qcow2_co_pwrite_zeroes(BlockDriverState *bs, @@ -XXX,XX +XXX,XX @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, required = virtual_size; } else { int64_t offset; - int pnum = 0; + int64_t pnum = 0; - for (offset = 0; offset < ssize; - offset += pnum * BDRV_SECTOR_SIZE) { - int nb_sectors = MIN(ssize - offset, - BDRV_REQUEST_MAX_BYTES) / BDRV_SECTOR_SIZE; - int64_t ret; + for (offset = 0; offset < ssize; offset += pnum) { + int ret; - ret = bdrv_get_block_status_above(in_bs, NULL, - offset >> BDRV_SECTOR_BITS, - nb_sectors, &pnum, NULL); + ret = bdrv_block_status_above(in_bs, NULL, offset, + ssize - offset, &pnum, NULL, + NULL); if (ret < 0) { error_setg_errno(&local_err, -ret, "Unable to get block status"); @@ -XXX,XX +XXX,XX @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, } else if ((ret & (BDRV_BLOCK_DATA | BDRV_BLOCK_ALLOCATED)) == (BDRV_BLOCK_DATA | BDRV_BLOCK_ALLOCATED)) { /* Extend pnum to end of cluster for next iteration */ - pnum = (ROUND_UP(offset + pnum * BDRV_SECTOR_SIZE, - cluster_size) - offset) >> BDRV_SECTOR_BITS; + pnum = ROUND_UP(offset + pnum, cluster_size) - offset; /* Count clusters we've seen */ - required += offset % cluster_size + pnum * BDRV_SECTOR_SIZE; + required += offset % cluster_size + pnum; } } } diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) BlockDriverState *bs1, *bs2; int64_t total_sectors1, total_sectors2; uint8_t *buf1 = NULL, *buf2 = NULL; - int pnum1, pnum2; + int64_t pnum1, pnum2; int allocated1, allocated2; int ret = 0; /* return value - 0 Ident, 1 Different, >1 Error */ bool progress = false, quiet = false, strict = false; @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) } for (;;) { - int64_t status1, status2; + int status1, status2; nb_sectors = sectors_to_process(total_sectors, sector_num); if (nb_sectors <= 0) { break; } - status1 = bdrv_get_block_status_above(bs1, NULL, sector_num, - total_sectors1 - sector_num, - &pnum1, NULL); + status1 = bdrv_block_status_above(bs1, NULL, + sector_num * BDRV_SECTOR_SIZE, + (total_sectors1 - sector_num) * + BDRV_SECTOR_SIZE, + &pnum1, NULL, NULL); if (status1 < 0) { ret = 3; error_report("Sector allocation test failed for %s", filename1); @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) } allocated1 = status1 & BDRV_BLOCK_ALLOCATED; - status2 = bdrv_get_block_status_above(bs2, NULL, sector_num, - total_sectors2 - sector_num, - &pnum2, NULL); + status2 = bdrv_block_status_above(bs2, NULL, + sector_num * BDRV_SECTOR_SIZE, + (total_sectors2 - sector_num) * + BDRV_SECTOR_SIZE, + &pnum2, NULL, NULL); if (status2 < 0) { ret = 3; error_report("Sector allocation test failed for %s", filename2); goto out; } allocated2 = status2 & BDRV_BLOCK_ALLOCATED; + /* TODO: Relax this once comparison is byte-based, and we no longer + * have to worry about sector alignment */ + assert(QEMU_IS_ALIGNED(pnum1 | pnum2, BDRV_SECTOR_SIZE)); if (pnum1) { - nb_sectors = MIN(nb_sectors, pnum1); + nb_sectors = MIN(nb_sectors, pnum1 >> BDRV_SECTOR_BITS); } if (pnum2) { - nb_sectors = MIN(nb_sectors, pnum2); + nb_sectors = MIN(nb_sectors, pnum2 >> BDRV_SECTOR_BITS); } if (strict) { - if ((status1 & ~BDRV_BLOCK_OFFSET_MASK) != - (status2 & ~BDRV_BLOCK_OFFSET_MASK)) { + if (status1 != status2) { ret = 1; qprintf(quiet, "Strict mode: Offset %" PRId64 " block status mismatch!\n", @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) } } if ((status1 & BDRV_BLOCK_ZERO) && (status2 & BDRV_BLOCK_ZERO)) { - nb_sectors = MIN(pnum1, pnum2); + nb_sectors = DIV_ROUND_UP(MIN(pnum1, pnum2), BDRV_SECTOR_SIZE); } else if (allocated1 == allocated2) { if (allocated1) { ret = blk_pread(blk1, sector_num << BDRV_SECTOR_BITS, buf1, @@ -XXX,XX +XXX,XX @@ static void convert_select_part(ImgConvertState *s, int64_t sector_num, static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num) { - int64_t ret, src_cur_offset; - int n, src_cur; + int64_t src_cur_offset; + int ret, n, src_cur; convert_select_part(s, sector_num, &src_cur, &src_cur_offset); @@ -XXX,XX +XXX,XX @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num) n = MIN(s->total_sectors - sector_num, BDRV_REQUEST_MAX_SECTORS); if (s->sector_next_status <= sector_num) { + int64_t count = n * BDRV_SECTOR_SIZE; + if (s->target_has_backing) { - int64_t count = n * BDRV_SECTOR_SIZE; ret = bdrv_block_status(blk_bs(s->src[src_cur]), (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE, count, &count, NULL, NULL); - assert(ret < 0 || QEMU_IS_ALIGNED(count, BDRV_SECTOR_SIZE)); - n = count >> BDRV_SECTOR_BITS; } else { - ret = bdrv_get_block_status_above(blk_bs(s->src[src_cur]), NULL, - sector_num - src_cur_offset, - n, &n, NULL); + ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL, + (sector_num - src_cur_offset) * + BDRV_SECTOR_SIZE, + count, &count, NULL, NULL); } if (ret < 0) { return ret; } + n = DIV_ROUND_UP(count, BDRV_SECTOR_SIZE); if (ret & BDRV_BLOCK_ZERO) { s->status = BLK_ZERO; -- 2.13.6 From: Eric Blake <eblake@redhat.com> As long as we are querying the status for a chunk smaller than the known image size, we are guaranteed that a successful return will have set pnum to a non-zero size (pnum is zero only for queries beyond the end of the file). Use that to slightly simplify the calculation of the current chunk size being compared. Likewise, we don't have to shrink the amount of data operated on until we know we have to read the file, and therefore have to fit in the bounds of our buffer. Also, note that 'total_sectors_over' is equivalent to 'progress_base'. With these changes in place, sectors_to_process() is now dead code, and can be removed. Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-img.c | 38 +++++++++++--------------------------- 1 file changed, 11 insertions(+), 27 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static int64_t sectors_to_bytes(int64_t sectors) return sectors << BDRV_SECTOR_BITS; } -static int64_t sectors_to_process(int64_t total, int64_t from) -{ - return MIN(total - from, IO_BUF_SIZE >> BDRV_SECTOR_BITS); -} - /* * Check if passed sectors are empty (not allocated or contain only 0 bytes) * @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) goto out; } - for (;;) { + while (sector_num < total_sectors) { int status1, status2; - nb_sectors = sectors_to_process(total_sectors, sector_num); - if (nb_sectors <= 0) { - break; - } status1 = bdrv_block_status_above(bs1, NULL, sector_num * BDRV_SECTOR_SIZE, (total_sectors1 - sector_num) * @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) /* TODO: Relax this once comparison is byte-based, and we no longer * have to worry about sector alignment */ assert(QEMU_IS_ALIGNED(pnum1 | pnum2, BDRV_SECTOR_SIZE)); - if (pnum1) { - nb_sectors = MIN(nb_sectors, pnum1 >> BDRV_SECTOR_BITS); - } - if (pnum2) { - nb_sectors = MIN(nb_sectors, pnum2 >> BDRV_SECTOR_BITS); - } + + assert(pnum1 && pnum2); + nb_sectors = MIN(pnum1, pnum2) >> BDRV_SECTOR_BITS; if (strict) { if (status1 != status2) { @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) } } if ((status1 & BDRV_BLOCK_ZERO) && (status2 & BDRV_BLOCK_ZERO)) { - nb_sectors = DIV_ROUND_UP(MIN(pnum1, pnum2), BDRV_SECTOR_SIZE); + /* nothing to do */ } else if (allocated1 == allocated2) { if (allocated1) { + nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS); ret = blk_pread(blk1, sector_num << BDRV_SECTOR_BITS, buf1, nb_sectors << BDRV_SECTOR_BITS); if (ret < 0) { @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) } } } else { - + nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS); if (allocated1) { ret = check_empty_sectors(blk1, sector_num, nb_sectors, filename1, buf1, quiet); @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) if (total_sectors1 != total_sectors2) { BlockBackend *blk_over; - int64_t total_sectors_over; const char *filename_over; qprintf(quiet, "Warning: Image size mismatch!\n"); if (total_sectors1 > total_sectors2) { - total_sectors_over = total_sectors1; blk_over = blk1; filename_over = filename1; } else { - total_sectors_over = total_sectors2; blk_over = blk2; filename_over = filename2; } - for (;;) { + while (sector_num < progress_base) { int64_t count; - nb_sectors = sectors_to_process(total_sectors_over, sector_num); - if (nb_sectors <= 0) { - break; - } ret = bdrv_is_allocated_above(blk_bs(blk_over), NULL, sector_num * BDRV_SECTOR_SIZE, - nb_sectors * BDRV_SECTOR_SIZE, + (progress_base - sector_num) * + BDRV_SECTOR_SIZE, &count); if (ret < 0) { ret = 3; @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) assert(QEMU_IS_ALIGNED(count, BDRV_SECTOR_SIZE)); nb_sectors = count >> BDRV_SECTOR_BITS; if (ret) { + nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS); ret = check_empty_sectors(blk_over, sector_num, nb_sectors, filename_over, buf1, quiet); if (ret) { -- 2.13.6 From: Eric Blake <eblake@redhat.com> Compare the following images with all-zero contents: $ truncate --size 1M A $ qemu-img create -f qcow2 -o preallocation=off B 1G $ qemu-img create -f qcow2 -o preallocation=metadata C 1G On my machine, the difference is noticeable for pre-patch speeds, with more than an order of magnitude in difference caused by the choice of preallocation in the qcow2 file: $ time ./qemu-img compare -f raw -F qcow2 A B Warning: Image size mismatch! Images are identical. real 0m0.014s user 0m0.007s sys 0m0.007s $ time ./qemu-img compare -f raw -F qcow2 A C Warning: Image size mismatch! Images are identical. real 0m0.341s user 0m0.144s sys 0m0.188s Why? Because bdrv_is_allocated() returns false for image B but true for image C, throwing away the fact that both images know via lseek(SEEK_HOLE) that the entire image still reads as zero. From there, qemu-img ends up calling bdrv_pread() for every byte of the tail, instead of quickly looking for the next allocation. The solution: use block_status instead of is_allocated, giving: $ time ./qemu-img compare -f raw -F qcow2 A C Warning: Image size mismatch! Images are identical. real 0m0.014s user 0m0.011s sys 0m0.003s which is on par with the speeds for no pre-allocation. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-img.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) while (sector_num < progress_base) { int64_t count; - ret = bdrv_is_allocated_above(blk_bs(blk_over), NULL, + ret = bdrv_block_status_above(blk_bs(blk_over), NULL, sector_num * BDRV_SECTOR_SIZE, (progress_base - sector_num) * BDRV_SECTOR_SIZE, - &count); + &count, NULL, NULL); if (ret < 0) { ret = 3; error_report("Sector allocation test failed for %s", @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) goto out; } - /* TODO relax this once bdrv_is_allocated_above does not enforce + /* TODO relax this once bdrv_block_status_above does not enforce * sector alignment */ assert(QEMU_IS_ALIGNED(count, BDRV_SECTOR_SIZE)); nb_sectors = count >> BDRV_SECTOR_BITS; - if (ret) { + if (ret & BDRV_BLOCK_ALLOCATED && !(ret & BDRV_BLOCK_ZERO)) { nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS); ret = check_empty_sectors(blk_over, sector_num, nb_sectors, filename_over, buf1, quiet); -- 2.13.6 From: Eric Blake <eblake@redhat.com> During 'qemu-img compare', when we are checking that an allocated portion of one file is all zeros, we don't need to waste time computing how many additional sectors after the first non-zero byte are also non-zero. Create a new helper find_nonzero() to do the check for a first non-zero sector, and rebase check_empty_sectors() to use it. The new interface intentionally uses bytes in its interface, even though it still crawls the buffer a sector at a time; it is robust to a partial sector at the end of the buffer. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-img.c | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ done: } /* + * Returns -1 if 'buf' contains only zeroes, otherwise the byte index + * of the first sector boundary within buf where the sector contains a + * non-zero byte. This function is robust to a buffer that is not + * sector-aligned. + */ +static int64_t find_nonzero(const uint8_t *buf, int64_t n) +{ + int64_t i; + int64_t end = QEMU_ALIGN_DOWN(n, BDRV_SECTOR_SIZE); + + for (i = 0; i < end; i += BDRV_SECTOR_SIZE) { + if (!buffer_is_zero(buf + i, BDRV_SECTOR_SIZE)) { + return i; + } + } + if (i < n && !buffer_is_zero(buf + i, n - end)) { + return i; + } + return -1; +} + +/* * Returns true iff the first sector pointed to by 'buf' contains at least * a non-NUL byte. * @@ -XXX,XX +XXX,XX @@ static int check_empty_sectors(BlockBackend *blk, int64_t sect_num, int sect_count, const char *filename, uint8_t *buffer, bool quiet) { - int pnum, ret = 0; + int ret = 0; + int64_t idx; + ret = blk_pread(blk, sect_num << BDRV_SECTOR_BITS, buffer, sect_count << BDRV_SECTOR_BITS); if (ret < 0) { @@ -XXX,XX +XXX,XX @@ static int check_empty_sectors(BlockBackend *blk, int64_t sect_num, sectors_to_bytes(sect_num), filename, strerror(-ret)); return ret; } - ret = is_allocated_sectors(buffer, sect_count, &pnum); - if (ret || pnum != sect_count) { + idx = find_nonzero(buffer, sect_count * BDRV_SECTOR_SIZE); + if (idx >= 0) { qprintf(quiet, "Content mismatch at offset %" PRId64 "!\n", - sectors_to_bytes(ret ? sect_num : sect_num + pnum)); + sectors_to_bytes(sect_num) + idx); return 1; } -- 2.13.6 From: Eric Blake <eblake@redhat.com> If a read error is encountered during 'qemu-img compare', we were printing the "Error while reading offset ..." message twice; this was because our helper function was awkward, printing output on some but not all paths. Fix it to consistently report errors on all paths, so that the callers do not risk a redundant message, and update the testsuite for the improved output. Further simplify the code by hoisting the conversion from an error message to an exit code into the helper function, rather than repeating that logic at all callers (yes, the helper function is now less generic, but it's a net win in lines of code). Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-img.c | 19 +++++-------------- tests/qemu-iotests/074.out | 2 -- 2 files changed, 5 insertions(+), 16 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static int64_t sectors_to_bytes(int64_t sectors) /* * Check if passed sectors are empty (not allocated or contain only 0 bytes) * - * Returns 0 in case sectors are filled with 0, 1 if sectors contain non-zero - * data and negative value on error. + * Intended for use by 'qemu-img compare': Returns 0 in case sectors are + * filled with 0, 1 if sectors contain non-zero data (this is a comparison + * failure), and 4 on error (the exit status for read errors), after emitting + * an error message. * * @param blk: BlockBackend for the image * @param sect_num: Number of first sector to check @@ -XXX,XX +XXX,XX @@ static int check_empty_sectors(BlockBackend *blk, int64_t sect_num, if (ret < 0) { error_report("Error while reading offset %" PRId64 " of %s: %s", sectors_to_bytes(sect_num), filename, strerror(-ret)); - return ret; + return 4; } idx = find_nonzero(buffer, sect_count * BDRV_SECTOR_SIZE); if (idx >= 0) { @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) filename2, buf1, quiet); } if (ret) { - if (ret < 0) { - error_report("Error while reading offset %" PRId64 ": %s", - sectors_to_bytes(sector_num), strerror(-ret)); - ret = 4; - } goto out; } } @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) ret = check_empty_sectors(blk_over, sector_num, nb_sectors, filename_over, buf1, quiet); if (ret) { - if (ret < 0) { - error_report("Error while reading offset %" PRId64 - " of %s: %s", sectors_to_bytes(sector_num), - filename_over, strerror(-ret)); - ret = 4; - } goto out; } } diff --git a/tests/qemu-iotests/074.out b/tests/qemu-iotests/074.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/074.out +++ b/tests/qemu-iotests/074.out @@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 wrote 512/512 bytes at offset 512 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) qemu-img: Error while reading offset 0 of blkdebug:TEST_DIR/blkdebug.conf:TEST_DIR/t.IMGFMT: Input/output error -qemu-img: Error while reading offset 0: Input/output error 4 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824 Formatting 'TEST_DIR/t.IMGFMT.2', fmt=IMGFMT size=0 @@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT.2', fmt=IMGFMT size=0 wrote 512/512 bytes at offset 512 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) qemu-img: Error while reading offset 0 of blkdebug:TEST_DIR/blkdebug.conf:TEST_DIR/t.IMGFMT: Input/output error -qemu-img: Error while reading offset 0 of blkdebug:TEST_DIR/blkdebug.conf:TEST_DIR/t.IMGFMT: Input/output error Warning: Image size mismatch! 4 Cleanup -- 2.13.6 From: Eric Blake <eblake@redhat.com> Continue on the quest to make more things byte-based instead of sector-based. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-img.c | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static int64_t sectors_to_bytes(int64_t sectors) * an error message. * * @param blk: BlockBackend for the image - * @param sect_num: Number of first sector to check - * @param sect_count: Number of sectors to check + * @param offset: Starting offset to check + * @param bytes: Number of bytes to check * @param filename: Name of disk file we are checking (logging purpose) * @param buffer: Allocated buffer for storing read data * @param quiet: Flag for quiet mode */ -static int check_empty_sectors(BlockBackend *blk, int64_t sect_num, - int sect_count, const char *filename, +static int check_empty_sectors(BlockBackend *blk, int64_t offset, + int64_t bytes, const char *filename, uint8_t *buffer, bool quiet) { int ret = 0; int64_t idx; - ret = blk_pread(blk, sect_num << BDRV_SECTOR_BITS, buffer, - sect_count << BDRV_SECTOR_BITS); + ret = blk_pread(blk, offset, buffer, bytes); if (ret < 0) { error_report("Error while reading offset %" PRId64 " of %s: %s", - sectors_to_bytes(sect_num), filename, strerror(-ret)); + offset, filename, strerror(-ret)); return 4; } - idx = find_nonzero(buffer, sect_count * BDRV_SECTOR_SIZE); + idx = find_nonzero(buffer, bytes); if (idx >= 0) { qprintf(quiet, "Content mismatch at offset %" PRId64 "!\n", - sectors_to_bytes(sect_num) + idx); + offset + idx); return 1; } @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) } else { nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS); if (allocated1) { - ret = check_empty_sectors(blk1, sector_num, nb_sectors, + ret = check_empty_sectors(blk1, sector_num * BDRV_SECTOR_SIZE, + nb_sectors * BDRV_SECTOR_SIZE, filename1, buf1, quiet); } else { - ret = check_empty_sectors(blk2, sector_num, nb_sectors, + ret = check_empty_sectors(blk2, sector_num * BDRV_SECTOR_SIZE, + nb_sectors * BDRV_SECTOR_SIZE, filename2, buf1, quiet); } if (ret) { @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) nb_sectors = count >> BDRV_SECTOR_BITS; if (ret & BDRV_BLOCK_ALLOCATED && !(ret & BDRV_BLOCK_ZERO)) { nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS); - ret = check_empty_sectors(blk_over, sector_num, nb_sectors, + ret = check_empty_sectors(blk_over, + sector_num * BDRV_SECTOR_SIZE, + nb_sectors * BDRV_SECTOR_SIZE, filename_over, buf1, quiet); if (ret) { goto out; -- 2.13.6 From: Eric Blake <eblake@redhat.com> In the continuing quest to make more things byte-based, change compare_sectors(), renaming it to compare_buffers() in the process. Note that one caller (qemu-img compare) only cares about the first difference, while the other (qemu-img rebase) cares about how many consecutive sectors have the same equal/different status; however, this patch does not bother to micro-optimize the compare case to avoid the comparisons of sectors beyond the first mismatch. Both callers are always passing valid buffers in, so the initial check for buffer size can be turned into an assertion. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-img.c | 55 +++++++++++++++++++++++++++---------------------------- 1 file changed, 27 insertions(+), 28 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static int is_allocated_sectors_min(const uint8_t *buf, int n, int *pnum, } /* - * Compares two buffers sector by sector. Returns 0 if the first sector of both - * buffers matches, non-zero otherwise. + * Compares two buffers sector by sector. Returns 0 if the first + * sector of each buffer matches, non-zero otherwise. * - * pnum is set to the number of sectors (including and immediately following - * the first one) that are known to have the same comparison result + * pnum is set to the sector-aligned size of the buffer prefix that + * has the same matching status as the first sector. */ -static int compare_sectors(const uint8_t *buf1, const uint8_t *buf2, int n, - int *pnum) +static int compare_buffers(const uint8_t *buf1, const uint8_t *buf2, + int64_t bytes, int64_t *pnum) { bool res; - int i; + int64_t i = MIN(bytes, BDRV_SECTOR_SIZE); - if (n <= 0) { - *pnum = 0; - return 0; - } + assert(bytes > 0); - res = !!memcmp(buf1, buf2, 512); - for(i = 1; i < n; i++) { - buf1 += 512; - buf2 += 512; + res = !!memcmp(buf1, buf2, i); + while (i < bytes) { + int64_t len = MIN(bytes - i, BDRV_SECTOR_SIZE); - if (!!memcmp(buf1, buf2, 512) != res) { + if (!!memcmp(buf1 + i, buf2 + i, len) != res) { break; } + i += len; } *pnum = i; @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) int64_t total_sectors; int64_t sector_num = 0; int64_t nb_sectors; - int c, pnum; + int c; uint64_t progress_base; bool image_opts = false; bool force_share = false; @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) /* nothing to do */ } else if (allocated1 == allocated2) { if (allocated1) { + int64_t pnum; + nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS); ret = blk_pread(blk1, sector_num << BDRV_SECTOR_BITS, buf1, nb_sectors << BDRV_SECTOR_BITS); @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) ret = 4; goto out; } - ret = compare_sectors(buf1, buf2, nb_sectors, &pnum); - if (ret || pnum != nb_sectors) { + ret = compare_buffers(buf1, buf2, + nb_sectors * BDRV_SECTOR_SIZE, &pnum); + if (ret || pnum != nb_sectors * BDRV_SECTOR_SIZE) { qprintf(quiet, "Content mismatch at offset %" PRId64 "!\n", - sectors_to_bytes( - ret ? sector_num : sector_num + pnum)); + sectors_to_bytes(sector_num) + (ret ? 0 : pnum)); ret = 1; goto out; } @@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv) /* If they differ, we need to write to the COW file */ uint64_t written = 0; - while (written < n) { - int pnum; + while (written < n * BDRV_SECTOR_SIZE) { + int64_t pnum; - if (compare_sectors(buf_old + written * 512, - buf_new + written * 512, n - written, &pnum)) + if (compare_buffers(buf_old + written, + buf_new + written, + n * BDRV_SECTOR_SIZE - written, &pnum)) { ret = blk_pwrite(blk, - (sector + written) << BDRV_SECTOR_BITS, - buf_old + written * 512, - pnum << BDRV_SECTOR_BITS, 0); + (sector << BDRV_SECTOR_BITS) + written, + buf_old + written, pnum, 0); if (ret < 0) { error_report("Error while writing to COW image: %s", strerror(-ret)); -- 2.13.6 From: Eric Blake <eblake@redhat.com> In the continuing quest to make more things byte-based, change the internal iteration of img_rebase(). We can finally drop the TODO assertion added earlier, now that the entire algorithm is byte-based and no longer has to shift from bytes to sectors. Most of the change is mechanical ('num_sectors' becomes 'size', 'sector' becomes 'offset', 'n' goes from sectors to bytes); some of it is also a cleanup (use of MIN() instead of open-coding, loss of variable 'count' added earlier in commit d6a644bb). Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-img.c | 84 +++++++++++++++++++++++++------------------------------------- 1 file changed, 34 insertions(+), 50 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv) * the image is the same as the original one at any time. */ if (!unsafe) { - int64_t num_sectors; - int64_t old_backing_num_sectors; - int64_t new_backing_num_sectors = 0; - uint64_t sector; - int n; - int64_t count; + int64_t size; + int64_t old_backing_size; + int64_t new_backing_size = 0; + uint64_t offset; + int64_t n; float local_progress = 0; buf_old = blk_blockalign(blk, IO_BUF_SIZE); buf_new = blk_blockalign(blk, IO_BUF_SIZE); - num_sectors = blk_nb_sectors(blk); - if (num_sectors < 0) { + size = blk_getlength(blk); + if (size < 0) { error_report("Could not get size of '%s': %s", - filename, strerror(-num_sectors)); + filename, strerror(-size)); ret = -1; goto out; } - old_backing_num_sectors = blk_nb_sectors(blk_old_backing); - if (old_backing_num_sectors < 0) { + old_backing_size = blk_getlength(blk_old_backing); + if (old_backing_size < 0) { char backing_name[PATH_MAX]; bdrv_get_backing_filename(bs, backing_name, sizeof(backing_name)); error_report("Could not get size of '%s': %s", - backing_name, strerror(-old_backing_num_sectors)); + backing_name, strerror(-old_backing_size)); ret = -1; goto out; } if (blk_new_backing) { - new_backing_num_sectors = blk_nb_sectors(blk_new_backing); - if (new_backing_num_sectors < 0) { + new_backing_size = blk_getlength(blk_new_backing); + if (new_backing_size < 0) { error_report("Could not get size of '%s': %s", - out_baseimg, strerror(-new_backing_num_sectors)); + out_baseimg, strerror(-new_backing_size)); ret = -1; goto out; } } - if (num_sectors != 0) { - local_progress = (float)100 / - (num_sectors / MIN(num_sectors, IO_BUF_SIZE / 512)); + if (size != 0) { + local_progress = (float)100 / (size / MIN(size, IO_BUF_SIZE)); } - for (sector = 0; sector < num_sectors; sector += n) { - - /* How many sectors can we handle with the next read? */ - if (sector + (IO_BUF_SIZE / 512) <= num_sectors) { - n = (IO_BUF_SIZE / 512); - } else { - n = num_sectors - sector; - } + for (offset = 0; offset < size; offset += n) { + /* How many bytes can we handle with the next read? */ + n = MIN(IO_BUF_SIZE, size - offset); /* If the cluster is allocated, we don't need to take action */ - ret = bdrv_is_allocated(bs, sector << BDRV_SECTOR_BITS, - n << BDRV_SECTOR_BITS, &count); + ret = bdrv_is_allocated(bs, offset, n, &n); if (ret < 0) { error_report("error while reading image metadata: %s", strerror(-ret)); goto out; } - /* TODO relax this once bdrv_is_allocated does not enforce - * sector alignment */ - assert(QEMU_IS_ALIGNED(count, BDRV_SECTOR_SIZE)); - n = count >> BDRV_SECTOR_BITS; if (ret) { continue; } @@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv) * Read old and new backing file and take into consideration that * backing files may be smaller than the COW image. */ - if (sector >= old_backing_num_sectors) { - memset(buf_old, 0, n * BDRV_SECTOR_SIZE); + if (offset >= old_backing_size) { + memset(buf_old, 0, n); } else { - if (sector + n > old_backing_num_sectors) { - n = old_backing_num_sectors - sector; + if (offset + n > old_backing_size) { + n = old_backing_size - offset; } - ret = blk_pread(blk_old_backing, sector << BDRV_SECTOR_BITS, - buf_old, n << BDRV_SECTOR_BITS); + ret = blk_pread(blk_old_backing, offset, buf_old, n); if (ret < 0) { error_report("error while reading from old backing file"); goto out; } } - if (sector >= new_backing_num_sectors || !blk_new_backing) { - memset(buf_new, 0, n * BDRV_SECTOR_SIZE); + if (offset >= new_backing_size || !blk_new_backing) { + memset(buf_new, 0, n); } else { - if (sector + n > new_backing_num_sectors) { - n = new_backing_num_sectors - sector; + if (offset + n > new_backing_size) { + n = new_backing_size - offset; } - ret = blk_pread(blk_new_backing, sector << BDRV_SECTOR_BITS, - buf_new, n << BDRV_SECTOR_BITS); + ret = blk_pread(blk_new_backing, offset, buf_new, n); if (ret < 0) { error_report("error while reading from new backing file"); goto out; @@ -XXX,XX +XXX,XX @@ static int img_rebase(int argc, char **argv) /* If they differ, we need to write to the COW file */ uint64_t written = 0; - while (written < n * BDRV_SECTOR_SIZE) { + while (written < n) { int64_t pnum; - if (compare_buffers(buf_old + written, - buf_new + written, - n * BDRV_SECTOR_SIZE - written, &pnum)) + if (compare_buffers(buf_old + written, buf_new + written, + n - written, &pnum)) { - ret = blk_pwrite(blk, - (sector << BDRV_SECTOR_BITS) + written, + ret = blk_pwrite(blk, offset + written, buf_old + written, pnum, 0); if (ret < 0) { error_report("Error while writing to COW image: %s", -- 2.13.6 From: Eric Blake <eblake@redhat.com> In the continuing quest to make more things byte-based, change the internal iteration of img_compare(). We can finally drop the TODO assertions added earlier, now that the entire algorithm is byte-based and no longer has to shift from bytes to sectors. Most of the change is mechanical ('total_sectors' becomes 'total_size', 'sector_num' becomes 'offset', 'nb_sectors' becomes 'chunk', 'progress_base' goes from sectors to bytes); some of it is also a cleanup (sectors_to_bytes() is now unused, loss of variable 'count' added earlier in commit 51b0a488). Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-img.c | 124 ++++++++++++++++++++++++------------------------------------- 1 file changed, 48 insertions(+), 76 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static int compare_buffers(const uint8_t *buf1, const uint8_t *buf2, #define IO_BUF_SIZE (2 * 1024 * 1024) -static int64_t sectors_to_bytes(int64_t sectors) -{ - return sectors << BDRV_SECTOR_BITS; -} - /* * Check if passed sectors are empty (not allocated or contain only 0 bytes) * @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) const char *fmt1 = NULL, *fmt2 = NULL, *cache, *filename1, *filename2; BlockBackend *blk1, *blk2; BlockDriverState *bs1, *bs2; - int64_t total_sectors1, total_sectors2; + int64_t total_size1, total_size2; uint8_t *buf1 = NULL, *buf2 = NULL; int64_t pnum1, pnum2; int allocated1, allocated2; @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) bool progress = false, quiet = false, strict = false; int flags; bool writethrough; - int64_t total_sectors; - int64_t sector_num = 0; - int64_t nb_sectors; + int64_t total_size; + int64_t offset = 0; + int64_t chunk; int c; uint64_t progress_base; bool image_opts = false; @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) buf1 = blk_blockalign(blk1, IO_BUF_SIZE); buf2 = blk_blockalign(blk2, IO_BUF_SIZE); - total_sectors1 = blk_nb_sectors(blk1); - if (total_sectors1 < 0) { + total_size1 = blk_getlength(blk1); + if (total_size1 < 0) { error_report("Can't get size of %s: %s", - filename1, strerror(-total_sectors1)); + filename1, strerror(-total_size1)); ret = 4; goto out; } - total_sectors2 = blk_nb_sectors(blk2); - if (total_sectors2 < 0) { + total_size2 = blk_getlength(blk2); + if (total_size2 < 0) { error_report("Can't get size of %s: %s", - filename2, strerror(-total_sectors2)); + filename2, strerror(-total_size2)); ret = 4; goto out; } - total_sectors = MIN(total_sectors1, total_sectors2); - progress_base = MAX(total_sectors1, total_sectors2); + total_size = MIN(total_size1, total_size2); + progress_base = MAX(total_size1, total_size2); qemu_progress_print(0, 100); - if (strict && total_sectors1 != total_sectors2) { + if (strict && total_size1 != total_size2) { ret = 1; qprintf(quiet, "Strict mode: Image size mismatch!\n"); goto out; } - while (sector_num < total_sectors) { + while (offset < total_size) { int status1, status2; - status1 = bdrv_block_status_above(bs1, NULL, - sector_num * BDRV_SECTOR_SIZE, - (total_sectors1 - sector_num) * - BDRV_SECTOR_SIZE, - &pnum1, NULL, NULL); + status1 = bdrv_block_status_above(bs1, NULL, offset, + total_size1 - offset, &pnum1, NULL, + NULL); if (status1 < 0) { ret = 3; error_report("Sector allocation test failed for %s", filename1); @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) } allocated1 = status1 & BDRV_BLOCK_ALLOCATED; - status2 = bdrv_block_status_above(bs2, NULL, - sector_num * BDRV_SECTOR_SIZE, - (total_sectors2 - sector_num) * - BDRV_SECTOR_SIZE, - &pnum2, NULL, NULL); + status2 = bdrv_block_status_above(bs2, NULL, offset, + total_size2 - offset, &pnum2, NULL, + NULL); if (status2 < 0) { ret = 3; error_report("Sector allocation test failed for %s", filename2); goto out; } allocated2 = status2 & BDRV_BLOCK_ALLOCATED; - /* TODO: Relax this once comparison is byte-based, and we no longer - * have to worry about sector alignment */ - assert(QEMU_IS_ALIGNED(pnum1 | pnum2, BDRV_SECTOR_SIZE)); assert(pnum1 && pnum2); - nb_sectors = MIN(pnum1, pnum2) >> BDRV_SECTOR_BITS; + chunk = MIN(pnum1, pnum2); if (strict) { if (status1 != status2) { ret = 1; qprintf(quiet, "Strict mode: Offset %" PRId64 - " block status mismatch!\n", - sectors_to_bytes(sector_num)); + " block status mismatch!\n", offset); goto out; } } @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) if (allocated1) { int64_t pnum; - nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS); - ret = blk_pread(blk1, sector_num << BDRV_SECTOR_BITS, buf1, - nb_sectors << BDRV_SECTOR_BITS); + chunk = MIN(chunk, IO_BUF_SIZE); + ret = blk_pread(blk1, offset, buf1, chunk); if (ret < 0) { - error_report("Error while reading offset %" PRId64 " of %s:" - " %s", sectors_to_bytes(sector_num), filename1, - strerror(-ret)); + error_report("Error while reading offset %" PRId64 + " of %s: %s", + offset, filename1, strerror(-ret)); ret = 4; goto out; } - ret = blk_pread(blk2, sector_num << BDRV_SECTOR_BITS, buf2, - nb_sectors << BDRV_SECTOR_BITS); + ret = blk_pread(blk2, offset, buf2, chunk); if (ret < 0) { error_report("Error while reading offset %" PRId64 - " of %s: %s", sectors_to_bytes(sector_num), - filename2, strerror(-ret)); + " of %s: %s", + offset, filename2, strerror(-ret)); ret = 4; goto out; } - ret = compare_buffers(buf1, buf2, - nb_sectors * BDRV_SECTOR_SIZE, &pnum); - if (ret || pnum != nb_sectors * BDRV_SECTOR_SIZE) { + ret = compare_buffers(buf1, buf2, chunk, &pnum); + if (ret || pnum != chunk) { qprintf(quiet, "Content mismatch at offset %" PRId64 "!\n", - sectors_to_bytes(sector_num) + (ret ? 0 : pnum)); + offset + (ret ? 0 : pnum)); ret = 1; goto out; } } } else { - nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS); + chunk = MIN(chunk, IO_BUF_SIZE); if (allocated1) { - ret = check_empty_sectors(blk1, sector_num * BDRV_SECTOR_SIZE, - nb_sectors * BDRV_SECTOR_SIZE, + ret = check_empty_sectors(blk1, offset, chunk, filename1, buf1, quiet); } else { - ret = check_empty_sectors(blk2, sector_num * BDRV_SECTOR_SIZE, - nb_sectors * BDRV_SECTOR_SIZE, + ret = check_empty_sectors(blk2, offset, chunk, filename2, buf1, quiet); } if (ret) { goto out; } } - sector_num += nb_sectors; - qemu_progress_print(((float) nb_sectors / progress_base)*100, 100); + offset += chunk; + qemu_progress_print(((float) chunk / progress_base) * 100, 100); } - if (total_sectors1 != total_sectors2) { + if (total_size1 != total_size2) { BlockBackend *blk_over; const char *filename_over; qprintf(quiet, "Warning: Image size mismatch!\n"); - if (total_sectors1 > total_sectors2) { + if (total_size1 > total_size2) { blk_over = blk1; filename_over = filename1; } else { @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) filename_over = filename2; } - while (sector_num < progress_base) { - int64_t count; - - ret = bdrv_block_status_above(blk_bs(blk_over), NULL, - sector_num * BDRV_SECTOR_SIZE, - (progress_base - sector_num) * - BDRV_SECTOR_SIZE, - &count, NULL, NULL); + while (offset < progress_base) { + ret = bdrv_block_status_above(blk_bs(blk_over), NULL, offset, + progress_base - offset, &chunk, + NULL, NULL); if (ret < 0) { ret = 3; error_report("Sector allocation test failed for %s", @@ -XXX,XX +XXX,XX @@ static int img_compare(int argc, char **argv) goto out; } - /* TODO relax this once bdrv_block_status_above does not enforce - * sector alignment */ - assert(QEMU_IS_ALIGNED(count, BDRV_SECTOR_SIZE)); - nb_sectors = count >> BDRV_SECTOR_BITS; if (ret & BDRV_BLOCK_ALLOCATED && !(ret & BDRV_BLOCK_ZERO)) { - nb_sectors = MIN(nb_sectors, IO_BUF_SIZE >> BDRV_SECTOR_BITS); - ret = check_empty_sectors(blk_over, - sector_num * BDRV_SECTOR_SIZE, - nb_sectors * BDRV_SECTOR_SIZE, + chunk = MIN(chunk, IO_BUF_SIZE); + ret = check_empty_sectors(blk_over, offset, chunk, filename_over, buf1, quiet); if (ret) { goto out; } } - sector_num += nb_sectors; - qemu_progress_print(((float) nb_sectors / progress_base)*100, 100); + offset += chunk; + qemu_progress_print(((float) chunk / progress_base) * 100, 100); } } -- 2.13.6 From: Eric Blake <eblake@redhat.com> Any device that has request_alignment greater than 512 should be unable to report status at a finer granularity; it may also be simpler for such devices to be guaranteed that the block layer has rounded things out to the granularity boundary (the way the block layer already rounds all other I/O out). Besides, getting the code correct for super-sector alignment also benefits us for the fact that our public interface now has byte granularity, even though none of our drivers have byte-level callbacks. Add an assertion in blkdebug that proves that the block layer never requests status of unaligned sections, similar to what it does on other requests (while still keeping the generic helper in place for when future patches add a throttle driver). Note that iotest 177 already covers this (it would fail if you use just the blkdebug.c hunk without the io.c changes). Meanwhile, we can drop assertions in callers that no longer have to pass in sector-aligned addresses. There is a mid-function scope added for 'count' and 'longret', for a couple of reasons: first, an upcoming patch will add an 'if' statement that checks whether a driver has an old- or new-style callback, and can conveniently use the same scope for less indentation churn at that time. Second, since we are trying to get rid of sector-based computations, wrapping things in a scope makes it easier to group and see what will be deleted in a final cleanup patch once all drivers have been converted to the new-style callback. Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- include/block/block_int.h | 3 +- block/blkdebug.c | 13 ++++++++- block/io.c | 71 ++++++++++++++++++++++++++++++----------------- 3 files changed, 59 insertions(+), 28 deletions(-) diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ struct BlockDriver { * according to the current layer, and should not set * BDRV_BLOCK_ALLOCATED, but may set BDRV_BLOCK_RAW. See block.h * for the meaning of _DATA, _ZERO, and _OFFSET_VALID. The block - * layer guarantees non-NULL pnum and file. + * layer guarantees input aligned to request_alignment, as well as + * non-NULL pnum and file. */ int64_t coroutine_fn (*bdrv_co_get_block_status)(BlockDriverState *bs, int64_t sector_num, int nb_sectors, int *pnum, diff --git a/block/blkdebug.c b/block/blkdebug.c index XXXXXXX..XXXXXXX 100644 --- a/block/blkdebug.c +++ b/block/blkdebug.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn blkdebug_co_pdiscard(BlockDriverState *bs, return bdrv_co_pdiscard(bs->file->bs, offset, bytes); } +static int64_t coroutine_fn blkdebug_co_get_block_status( + BlockDriverState *bs, int64_t sector_num, int nb_sectors, int *pnum, + BlockDriverState **file) +{ + assert(QEMU_IS_ALIGNED(sector_num | nb_sectors, + DIV_ROUND_UP(bs->bl.request_alignment, + BDRV_SECTOR_SIZE))); + return bdrv_co_get_block_status_from_file(bs, sector_num, nb_sectors, + pnum, file); +} + static void blkdebug_close(BlockDriverState *bs) { BDRVBlkdebugState *s = bs->opaque; @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_blkdebug = { .bdrv_co_flush_to_disk = blkdebug_co_flush, .bdrv_co_pwrite_zeroes = blkdebug_co_pwrite_zeroes, .bdrv_co_pdiscard = blkdebug_co_pdiscard, - .bdrv_co_get_block_status = bdrv_co_get_block_status_from_file, + .bdrv_co_get_block_status = blkdebug_co_get_block_status, .bdrv_debug_event = blkdebug_debug_event, .bdrv_debug_breakpoint = blkdebug_debug_breakpoint, diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs, { int64_t total_size; int64_t n; /* bytes */ - int64_t ret; + int ret; int64_t local_map = 0; BlockDriverState *local_file = NULL; - int count; /* sectors */ + int64_t aligned_offset, aligned_bytes; + uint32_t align; assert(pnum); *pnum = 0; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs, } bdrv_inc_in_flight(bs); + + /* Round out to request_alignment boundaries */ + /* TODO: until we have a byte-based driver callback, we also have to + * round out to sectors, even if that is bigger than request_alignment */ + align = MAX(bs->bl.request_alignment, BDRV_SECTOR_SIZE); + aligned_offset = QEMU_ALIGN_DOWN(offset, align); + aligned_bytes = ROUND_UP(offset + bytes, align) - aligned_offset; + + { + int count; /* sectors */ + int64_t longret; + + assert(QEMU_IS_ALIGNED(aligned_offset | aligned_bytes, + BDRV_SECTOR_SIZE)); + /* + * The contract allows us to return pnum smaller than bytes, even + * if the next query would see the same status; we truncate the + * request to avoid overflowing the driver's 32-bit interface. + */ + longret = bs->drv->bdrv_co_get_block_status( + bs, aligned_offset >> BDRV_SECTOR_BITS, + MIN(INT_MAX, aligned_bytes) >> BDRV_SECTOR_BITS, &count, + &local_file); + if (longret < 0) { + assert(INT_MIN <= longret); + ret = longret; + goto out; + } + if (longret & BDRV_BLOCK_OFFSET_VALID) { + local_map = longret & BDRV_BLOCK_OFFSET_MASK; + } + ret = longret & ~BDRV_BLOCK_OFFSET_MASK; + *pnum = count * BDRV_SECTOR_SIZE; + } + /* - * TODO: Rather than require aligned offsets, we could instead - * round to the driver's request_alignment here, then touch up - * count afterwards back to the caller's expectations. - */ - assert(QEMU_IS_ALIGNED(offset | bytes, BDRV_SECTOR_SIZE)); - /* - * The contract allows us to return pnum smaller than bytes, even - * if the next query would see the same status; we truncate the - * request to avoid overflowing the driver's 32-bit interface. + * The driver's result must be a multiple of request_alignment. + * Clamp pnum and adjust map to original request. */ - bytes = MIN(bytes, BDRV_REQUEST_MAX_BYTES); - ret = bs->drv->bdrv_co_get_block_status(bs, offset >> BDRV_SECTOR_BITS, - bytes >> BDRV_SECTOR_BITS, &count, - &local_file); - if (ret < 0) { - goto out; + assert(QEMU_IS_ALIGNED(*pnum, align) && align > offset - aligned_offset); + *pnum -= offset - aligned_offset; + if (*pnum > bytes) { + *pnum = bytes; } if (ret & BDRV_BLOCK_OFFSET_VALID) { - local_map = ret & BDRV_BLOCK_OFFSET_MASK; + local_map += offset - aligned_offset; } - *pnum = count * BDRV_SECTOR_SIZE; if (ret & BDRV_BLOCK_RAW) { assert(ret & BDRV_BLOCK_OFFSET_VALID && local_file); ret = bdrv_co_block_status(local_file, want_zero, local_map, *pnum, pnum, &local_map, &local_file); - assert(ret < 0 || - QEMU_IS_ALIGNED(*pnum | local_map, BDRV_SECTOR_SIZE)); goto out; } @@ -XXX,XX +XXX,XX @@ early_out: if (map) { *map = local_map; } - if (ret >= 0) { - ret &= ~BDRV_BLOCK_OFFSET_MASK; - } else { - assert(INT_MIN <= ret); - } return ret; } -- 2.13.6 From: Eric Blake <eblake@redhat.com> Now that bdrv_is_allocated accepts non-aligned inputs, we can remove the TODO added in commit d6a644bb. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/io.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/block/io.c b/block/io.c index XXXXXXX..XXXXXXX 100644 --- a/block/io.c +++ b/block/io.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn bdrv_aligned_preadv(BdrvChild *child, } if (flags & BDRV_REQ_COPY_ON_READ) { - /* TODO: Simplify further once bdrv_is_allocated no longer - * requires sector alignment */ - int64_t start = QEMU_ALIGN_DOWN(offset, BDRV_SECTOR_SIZE); - int64_t end = QEMU_ALIGN_UP(offset + bytes, BDRV_SECTOR_SIZE); int64_t pnum; - ret = bdrv_is_allocated(bs, start, end - start, &pnum); + ret = bdrv_is_allocated(bs, offset, bytes, &pnum); if (ret < 0) { goto out; } - if (!ret || pnum != end - start) { + if (!ret || pnum != bytes) { ret = bdrv_co_do_copy_on_readv(child, offset, bytes, qiov); goto out; } -- 2.13.6 From: Eric Blake <eblake@redhat.com> Now that bdrv_is_allocated accepts non-aligned inputs, we can remove the TODO added in earlier refactoring. Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static bool is_zero(BlockDriverState *bs, int64_t offset, int64_t bytes) { int64_t nr; int res; - int64_t start; - - /* TODO: Widening to sector boundaries should only be needed as - * long as we can't query finer granularity. */ - start = QEMU_ALIGN_DOWN(offset, BDRV_SECTOR_SIZE); - bytes = QEMU_ALIGN_UP(offset + bytes, BDRV_SECTOR_SIZE) - start; /* Clamp to image length, before checking status of underlying sectors */ - if (start + bytes > bs->total_sectors * BDRV_SECTOR_SIZE) { - bytes = bs->total_sectors * BDRV_SECTOR_SIZE - start; + if (offset + bytes > bs->total_sectors * BDRV_SECTOR_SIZE) { + bytes = bs->total_sectors * BDRV_SECTOR_SIZE - offset; } if (!bytes) { return true; } - res = bdrv_block_status_above(bs, NULL, start, bytes, &nr, NULL, NULL); + res = bdrv_block_status_above(bs, NULL, offset, bytes, &nr, NULL, NULL); return res >= 0 && (res & BDRV_BLOCK_ZERO) && nr == bytes; } -- 2.13.6 From: Eric Blake <eblake@redhat.com> Previously, the alloc command required that input parameters be sector-aligned and clamped to 32 bits, because the underlying bdrv_is_allocated used a 32-bit parameter and asserted aligned inputs. But now that we have fixed block status to report a 64-bit bytes value, and to properly round requests on behalf of guests, we can pass any values, and can use qemu-io to add coverage that our rounding is correct regardless of the guest alignment constraints. Update iotest 177 to intentionally probe block status at unaligned boundaries as well as with a bytes value that does not map to 32-bit sectors, which also required tweaking the image prep to leave an unallocated portion to the image under test. Signed-off-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-io-cmds.c | 13 ------------- tests/qemu-iotests/177 | 12 ++++++++++-- tests/qemu-iotests/177.out | 19 ++++++++++++++----- 3 files changed, 24 insertions(+), 20 deletions(-) diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-io-cmds.c +++ b/qemu-io-cmds.c @@ -XXX,XX +XXX,XX @@ static int alloc_f(BlockBackend *blk, int argc, char **argv) if (offset < 0) { print_cvtnum_err(offset, argv[1]); return 0; - } else if (!QEMU_IS_ALIGNED(offset, BDRV_SECTOR_SIZE)) { - printf("%" PRId64 " is not a sector-aligned value for 'offset'\n", - offset); - return 0; } if (argc == 3) { @@ -XXX,XX +XXX,XX @@ static int alloc_f(BlockBackend *blk, int argc, char **argv) if (count < 0) { print_cvtnum_err(count, argv[2]); return 0; - } else if (count > INT_MAX * BDRV_SECTOR_SIZE) { - printf("length argument cannot exceed %llu, given %s\n", - INT_MAX * BDRV_SECTOR_SIZE, argv[2]); - return 0; } } else { count = BDRV_SECTOR_SIZE; } - if (!QEMU_IS_ALIGNED(count, BDRV_SECTOR_SIZE)) { - printf("%" PRId64 " is not a sector-aligned value for 'count'\n", - count); - return 0; - } remaining = count; sum_alloc = 0; diff --git a/tests/qemu-iotests/177 b/tests/qemu-iotests/177 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/177 +++ b/tests/qemu-iotests/177 @@ -XXX,XX +XXX,XX @@ echo "== setting up files ==" TEST_IMG="$TEST_IMG.base" _make_test_img $size $QEMU_IO -c "write -P 11 0 $size" "$TEST_IMG.base" | _filter_qemu_io _make_test_img -b "$TEST_IMG.base" -$QEMU_IO -c "write -P 22 0 $size" "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "write -P 22 0 110M" "$TEST_IMG" | _filter_qemu_io # Limited to 64k max-transfer echo @@ -XXX,XX +XXX,XX @@ $QEMU_IO -c "open -o $options,$limits blkdebug::$TEST_IMG" \ -c "discard 80000001 30M" | _filter_qemu_io echo +echo "== block status smaller than alignment ==" +limits=align=4k +$QEMU_IO -c "open -o $options,$limits blkdebug::$TEST_IMG" \ + -c "alloc 1 1" -c "alloc 0x6dffff0 1000" -c "alloc 127m 5P" \ + -c map | _filter_qemu_io + +echo echo "== verify image content ==" function verify_io() @@ -XXX,XX +XXX,XX @@ function verify_io() echo read -P 0 32M 32M echo read -P 22 64M 13M echo read -P $discarded 77M 29M - echo read -P 22 106M 22M + echo read -P 22 106M 4M + echo read -P 11 110M 18M } verify_io | $QEMU_IO -r "$TEST_IMG" | _filter_qemu_io diff --git a/tests/qemu-iotests/177.out b/tests/qemu-iotests/177.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/177.out +++ b/tests/qemu-iotests/177.out @@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=134217728 wrote 134217728/134217728 bytes at offset 0 128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728 backing_file=TEST_DIR/t.IMGFMT.base -wrote 134217728/134217728 bytes at offset 0 -128 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 115343360/115343360 bytes at offset 0 +110 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) == constrained alignment and max-transfer == wrote 131072/131072 bytes at offset 1000 @@ -XXX,XX +XXX,XX @@ wrote 33554432/33554432 bytes at offset 33554432 discard 31457280/31457280 bytes at offset 80000001 30 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +== block status smaller than alignment == +1/1 bytes allocated at offset 1 bytes +16/1000 bytes allocated at offset 110 MiB +0/1048576 bytes allocated at offset 127 MiB +110 MiB (0x6e00000) bytes allocated at offset 0 bytes (0x0) +18 MiB (0x1200000) bytes not allocated at offset 110 MiB (0x6e00000) + == verify image content == read 1000/1000 bytes at offset 0 1000 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) @@ -XXX,XX +XXX,XX @@ read 13631488/13631488 bytes at offset 67108864 13 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) read 30408704/30408704 bytes at offset 80740352 29 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -read 23068672/23068672 bytes at offset 111149056 -22 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 4194304/4194304 bytes at offset 111149056 +4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 18874368/18874368 bytes at offset 115343360 +18 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) Offset Length File 0 0x800000 TEST_DIR/t.IMGFMT 0x900000 0x2400000 TEST_DIR/t.IMGFMT 0x3c00000 0x1100000 TEST_DIR/t.IMGFMT -0x6a00000 0x1600000 TEST_DIR/t.IMGFMT +0x6a00000 0x400000 TEST_DIR/t.IMGFMT No errors were found on the image. *** done -- 2.13.6 From: Max Reitz <mreitz@redhat.com> qemu-img commit invalidates all images between base and top. This should be mentioned in the man page. Suggested-by: Ping Li <pingl@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Jeff Cody <jcody@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-img.texi | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/qemu-img.texi b/qemu-img.texi index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.texi +++ b/qemu-img.texi @@ -XXX,XX +XXX,XX @@ If the backing chain of the given image file @var{filename} has more than one layer, the backing file into which the changes will be committed may be specified as @var{base} (which has to be part of @var{filename}'s backing chain). If @var{base} is not specified, the immediate backing file of the top -image (which is @var{filename}) will be used. For reasons of consistency, -explicitly specifying @var{base} will always imply @code{-d} (since emptying an -image after committing to an indirect backing file would lead to different data -being read from the image due to content in the intermediate backing chain -overruling the commit target). +image (which is @var{filename}) will be used. Note that after a commit operation +all images between @var{base} and the top image will be invalid and may return +garbage data when read. For this reason, @code{-b} implies @code{-d} (so that +the top image stays valid). @item compare [-f @var{fmt}] [-F @var{fmt}] [-T @var{src_cache}] [-p] [-s] [-q] @var{filename1} @var{filename2} -- 2.13.6 From: Alberto Garcia <berto@igalia.com> BDRV_SECTOR_BITS is defined to be 9 in block.h (and BDRV_SECTOR_SIZE is calculated from that), but there are still a couple of places where we are using the literal value instead of the macro. Signed-off-by: Alberto Garcia <berto@igalia.com> Message-id: 20171009153856.20387-1-berto@igalia.com Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/qcow2.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags, s->cluster_bits = header.cluster_bits; s->cluster_size = 1 << s->cluster_bits; - s->cluster_sectors = 1 << (s->cluster_bits - 9); + s->cluster_sectors = 1 << (s->cluster_bits - BDRV_SECTOR_BITS); /* Initialise version 3 header fields */ if (header.version == 2) { @@ -XXX,XX +XXX,XX @@ static int64_t coroutine_fn qcow2_co_get_block_status(BlockDriverState *bs, bytes = MIN(INT_MAX, nb_sectors * BDRV_SECTOR_SIZE); qemu_co_mutex_lock(&s->lock); - ret = qcow2_get_cluster_offset(bs, sector_num << 9, &bytes, + ret = qcow2_get_cluster_offset(bs, sector_num << BDRV_SECTOR_BITS, &bytes, &cluster_offset); qemu_co_mutex_unlock(&s->lock); if (ret < 0) { -- 2.13.6 From: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> Message-id: 20170929170843.3711-1-mreitz@redhat.com Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/127 | 97 ++++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/127.out | 14 +++++++ tests/qemu-iotests/group | 1 + 3 files changed, 112 insertions(+) create mode 100755 tests/qemu-iotests/127 create mode 100644 tests/qemu-iotests/127.out diff --git a/tests/qemu-iotests/127 b/tests/qemu-iotests/127 new file mode 100755 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/127 @@ -XXX,XX +XXX,XX @@ +#!/bin/bash +# +# Test case for mirroring with dataplane +# +# Copyright (C) 2017 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator +owner=mreitz@redhat.com + +seq=$(basename $0) +echo "QA output created by $seq" + +here=$PWD +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_qemu + _cleanup_test_img + _rm_test_img "$TEST_IMG.overlay0" + _rm_test_img "$TEST_IMG.overlay1" +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and qemu instance handling +. ./common.rc +. ./common.filter +. ./common.qemu + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +IMG_SIZE=64K + +_make_test_img $IMG_SIZE +TEST_IMG="$TEST_IMG.overlay0" _make_test_img -b "$TEST_IMG" $IMG_SIZE +TEST_IMG="$TEST_IMG.overlay1" _make_test_img -b "$TEST_IMG" $IMG_SIZE + +# So that we actually have something to mirror and the job does not return +# immediately (which may be bad because then we cannot know whether the +# 'return' or the 'BLOCK_JOB_READY' comes first). +$QEMU_IO -c 'write 0 42' "$TEST_IMG.overlay0" | _filter_qemu_io + +# We cannot use virtio-blk here because that does not actually set the attached +# BB's AioContext in qtest mode +_launch_qemu \ + -object iothread,id=iothr \ + -blockdev node-name=source,driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG.overlay0" \ + -device virtio-scsi,id=scsi-bus,iothread=iothr \ + -device scsi-hd,bus=scsi-bus.0,drive=source + +_send_qemu_cmd $QEMU_HANDLE \ + "{ 'execute': 'qmp_capabilities' }" \ + 'return' + +_send_qemu_cmd $QEMU_HANDLE \ + "{ 'execute': 'drive-mirror', + 'arguments': { + 'job-id': 'mirror', + 'device': 'source', + 'target': '$TEST_IMG.overlay1', + 'mode': 'existing', + 'sync': 'top' + } }" \ + 'BLOCK_JOB_READY' + +# The backing BDS should be assigned the overlay's AioContext +_send_qemu_cmd $QEMU_HANDLE \ + "{ 'execute': 'block-job-complete', + 'arguments': { 'device': 'mirror' } }" \ + 'BLOCK_JOB_COMPLETED' + +_send_qemu_cmd $QEMU_HANDLE \ + "{ 'execute': 'quit' }" \ + 'return' + +wait=yes _cleanup_qemu + +# success, all done +echo '*** done' +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/127.out b/tests/qemu-iotests/127.out new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/127.out @@ -XXX,XX +XXX,XX @@ +QA output created by 127 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=65536 +Formatting 'TEST_DIR/t.IMGFMT.overlay0', fmt=IMGFMT size=65536 backing_file=TEST_DIR/t.IMGFMT +Formatting 'TEST_DIR/t.IMGFMT.overlay1', fmt=IMGFMT size=65536 backing_file=TEST_DIR/t.IMGFMT +wrote 42/42 bytes at offset 0 +42 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_READY", "data": {"device": "mirror", "len": 65536, "offset": 65536, "speed": 0, "type": "mirror"}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "mirror", "len": 65536, "offset": 65536, "speed": 0, "type": "mirror"}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -XXX,XX +XXX,XX @@ 124 rw auto backing 125 rw auto 126 rw auto backing +127 rw auto backing quick 128 rw auto quick 129 rw auto quick 130 rw auto quick -- 2.13.6 From: Max Reitz <mreitz@redhat.com> Tests 067 and 087 filter the actual image size because it depends on the host filesystem (and is not part of the respective test). Since this is generally true, we should have a common filter function for this, so let's pull out the sed line from both tests into such a function. Signed-off-by: Max Reitz <mreitz@redhat.com> Message-id: 20171009163456.485-2-mreitz@redhat.com Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Jeff Cody <jcody@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/067 | 2 +- tests/qemu-iotests/087 | 2 +- tests/qemu-iotests/common.filter | 6 ++++++ 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/067 b/tests/qemu-iotests/067 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/067 +++ b/tests/qemu-iotests/067 @@ -XXX,XX +XXX,XX @@ _filter_qmp_events() function run_qemu() { do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qmp | _filter_qemu \ - | sed -e 's/\("actual-size":\s*\)[0-9]\+/\1SIZE/g' \ + | _filter_actual_image_size \ | _filter_generated_node_ids | _filter_qmp_events } diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/087 +++ b/tests/qemu-iotests/087 @@ -XXX,XX +XXX,XX @@ function run_qemu() { do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qmp \ | _filter_qemu | _filter_imgfmt \ - | sed -e 's/\("actual-size":\s*\)[0-9]\+/\1SIZE/g' + | _filter_actual_image_size } size=128M diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.filter +++ b/tests/qemu-iotests/common.filter @@ -XXX,XX +XXX,XX @@ _filter_block_job_len() sed -e 's/, "len": [0-9]\+,/, "len": LEN,/g' } +# replace actual image size (depends on the host filesystem) +_filter_actual_image_size() +{ + sed -s 's/\("actual-size":\s*\)[0-9]\+/\1SIZE/g' +} + # replace driver-specific options in the "Formatting..." line _filter_img_create() { -- 2.13.6 From: Max Reitz <mreitz@redhat.com> Whenever the actual image size is not part of the test, it should be filtered as it depends on the host filesystem. Signed-off-by: Max Reitz <mreitz@redhat.com> Message-id: 20171009163456.485-3-mreitz@redhat.com Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Jeff Cody <jcody@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/184 | 3 ++- tests/qemu-iotests/184.out | 6 +++--- tests/qemu-iotests/191 | 4 ++-- tests/qemu-iotests/191.out | 46 +++++++++++++++++++++++----------------------- 4 files changed, 30 insertions(+), 29 deletions(-) diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/184 +++ b/tests/qemu-iotests/184 @@ -XXX,XX +XXX,XX @@ function do_run_qemu() function run_qemu() { do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qemu | _filter_qmp\ - | _filter_qemu_io | _filter_generated_node_ids + | _filter_qemu_io | _filter_generated_node_ids \ + | _filter_actual_image_size } _make_test_img 64M diff --git a/tests/qemu-iotests/184.out b/tests/qemu-iotests/184.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/184.out +++ b/tests/qemu-iotests/184.out @@ -XXX,XX +XXX,XX @@ Testing: "filename": "json:{\"throttle-group\": \"group0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"qcow2\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/t.qcow2\"}}}", "cluster-size": 65536, "format": "throttle", - "actual-size": 200704, + "actual-size": SIZE, "dirty-flag": false }, "iops_wr": 0, @@ -XXX,XX +XXX,XX @@ Testing: "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", - "actual-size": 200704, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ Testing: "virtual-size": 197120, "filename": "TEST_DIR/t.qcow2", "format": "file", - "actual-size": 200704, + "actual-size": SIZE, "dirty-flag": false }, "iops_wr": 0, diff --git a/tests/qemu-iotests/191 b/tests/qemu-iotests/191 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/191 +++ b/tests/qemu-iotests/191 @@ -XXX,XX +XXX,XX @@ echo === Check that both top and top2 point to base now === echo _send_qemu_cmd $h "{ 'execute': 'query-named-block-nodes' }" "^}" | - _filter_generated_node_ids + _filter_generated_node_ids | _filter_actual_image_size _send_qemu_cmd $h "{ 'execute': 'quit' }" "^}" wait=1 _cleanup_qemu @@ -XXX,XX +XXX,XX @@ echo === Check that both top and top2 point to base now === echo _send_qemu_cmd $h "{ 'execute': 'query-named-block-nodes' }" "^}" | - _filter_generated_node_ids + _filter_generated_node_ids | _filter_actual_image_size _send_qemu_cmd $h "{ 'execute': 'quit' }" "^}" wait=1 _cleanup_qemu diff --git a/tests/qemu-iotests/191.out b/tests/qemu-iotests/191.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/191.out +++ b/tests/qemu-iotests/191.out @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.base", "cluster-size": 65536, "format": "qcow2", - "actual-size": 397312, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.ovl2", "cluster-size": 65536, "format": "qcow2", - "actual-size": 200704, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "virtual-size": 197120, "filename": "TEST_DIR/t.qcow2.ovl2", "format": "file", - "actual-size": 200704, + "actual-size": SIZE, "dirty-flag": false }, "iops_wr": 0, @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.base", "cluster-size": 65536, "format": "qcow2", - "actual-size": 397312, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", - "actual-size": 200704, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "virtual-size": 197120, "filename": "TEST_DIR/t.qcow2", "format": "file", - "actual-size": 200704, + "actual-size": SIZE, "dirty-flag": false }, "iops_wr": 0, @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.base", "cluster-size": 65536, "format": "qcow2", - "actual-size": 397312, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.mid", "cluster-size": 65536, "format": "qcow2", - "actual-size": 397312, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "virtual-size": 393216, "filename": "TEST_DIR/t.qcow2.mid", "format": "file", - "actual-size": 397312, + "actual-size": SIZE, "dirty-flag": false }, "iops_wr": 0, @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.base", "cluster-size": 65536, "format": "qcow2", - "actual-size": 397312, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "virtual-size": 393216, "filename": "TEST_DIR/t.qcow2.base", "format": "file", - "actual-size": 397312, + "actual-size": SIZE, "dirty-flag": false }, "iops_wr": 0, @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.base", "cluster-size": 65536, "format": "qcow2", - "actual-size": 397312, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.ovl2", "cluster-size": 65536, "format": "qcow2", - "actual-size": 200704, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "virtual-size": 197120, "filename": "TEST_DIR/t.qcow2.ovl2", "format": "file", - "actual-size": 200704, + "actual-size": SIZE, "dirty-flag": false }, "iops_wr": 0, @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.base", "cluster-size": 65536, "format": "qcow2", - "actual-size": 397312, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.ovl2", "cluster-size": 65536, "format": "qcow2", - "actual-size": 200704, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.ovl3", "cluster-size": 65536, "format": "qcow2", - "actual-size": 200704, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "virtual-size": 197120, "filename": "TEST_DIR/t.qcow2.ovl3", "format": "file", - "actual-size": 200704, + "actual-size": SIZE, "dirty-flag": false }, "iops_wr": 0, @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.base", "cluster-size": 65536, "format": "qcow2", - "actual-size": 397312, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "virtual-size": 393216, "filename": "TEST_DIR/t.qcow2.base", "format": "file", - "actual-size": 397312, + "actual-size": SIZE, "dirty-flag": false }, "iops_wr": 0, @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2.base", "cluster-size": 65536, "format": "qcow2", - "actual-size": 397312, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "filename": "TEST_DIR/t.qcow2", "cluster-size": 65536, "format": "qcow2", - "actual-size": 200704, + "actual-size": SIZE, "format-specific": { "type": "qcow2", "data": { @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 1048576 "virtual-size": 197120, "filename": "TEST_DIR/t.qcow2", "format": "file", - "actual-size": 200704, + "actual-size": SIZE, "dirty-flag": false }, "iops_wr": 0, -- 2.13.6 From: Max Reitz <mreitz@redhat.com> bdrv_truncate() has an errp parameter which is always set when an error occurs. Let's use that instead of a plain strerror(). Signed-off-by: Max Reitz <mreitz@redhat.com> Message-id: 20171009155431.14093-1-mreitz@redhat.com Reviewed-by: Pavel Butsykin <pbutsykin@virtuozzo.com> Reviewed-by: Jeff Cody <jcody@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/qcow2.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset, return last_cluster; } if ((last_cluster + 1) * s->cluster_size < old_file_size) { - ret = bdrv_truncate(bs->file, (last_cluster + 1) * s->cluster_size, - PREALLOC_MODE_OFF, NULL); - if (ret < 0) { - warn_report("Failed to truncate the tail of the image: %s", - strerror(-ret)); - ret = 0; + Error *local_err = NULL; + + bdrv_truncate(bs->file, (last_cluster + 1) * s->cluster_size, + PREALLOC_MODE_OFF, &local_err); + if (local_err) { + warn_reportf_err(local_err, + "Failed to truncate the tail of the image: "); } } } else { -- 2.13.6 From: Max Reitz <mreitz@redhat.com> A qcow2 image file's length is not required to have a length that is a multiple of the cluster size. However, qcow2_refcount_area() expects an aligned value for its @start_offset parameter, so we need to round @old_file_size up to the next cluster boundary. Reported-by: Ping Li <pingl@redhat.com> Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1414049 Signed-off-by: Max Reitz <mreitz@redhat.com> Message-id: 20171009215533.12530-2-mreitz@redhat.com Cc: qemu-stable@nongnu.org Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Jeff Cody <jcody@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/qcow2.c | 1 + 1 file changed, 1 insertion(+) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_truncate(BlockDriverState *bs, int64_t offset, "Failed to inquire current file length"); return old_file_size; } + old_file_size = ROUND_UP(old_file_size, s->cluster_size); nb_new_data_clusters = DIV_ROUND_UP(offset - old_length, s->cluster_size); -- 2.13.6 From: Max Reitz <mreitz@redhat.com> Some qcow2 functions (at least perform_cow()) expect s->lock to be taken. Therefore, if we want to make use of them, we should execute preallocate() (as "preallocate_co") in a coroutine so that we can use the qemu_co_mutex_* functions. Signed-off-by: Max Reitz <mreitz@redhat.com> Message-id: 20171009215533.12530-3-mreitz@redhat.com Cc: qemu-stable@nongnu.org Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Jeff Cody <jcody@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- block/qcow2.c | 41 ++++++++++++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 7 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_set_up_encryption(BlockDriverState *bs, const char *encryptfmt, } +typedef struct PreallocCo { + BlockDriverState *bs; + uint64_t offset; + uint64_t new_length; + + int ret; +} PreallocCo; + /** * Preallocates metadata structures for data clusters between @offset (in the * guest disk) and @new_length (which is thus generally the new guest disk @@ -XXX,XX +XXX,XX @@ static int qcow2_set_up_encryption(BlockDriverState *bs, const char *encryptfmt, * * Returns: 0 on success, -errno on failure. */ -static int preallocate(BlockDriverState *bs, - uint64_t offset, uint64_t new_length) +static void coroutine_fn preallocate_co(void *opaque) { + PreallocCo *params = opaque; + BlockDriverState *bs = params->bs; + uint64_t offset = params->offset; + uint64_t new_length = params->new_length; BDRVQcow2State *s = bs->opaque; uint64_t bytes; uint64_t host_offset = 0; @@ -XXX,XX +XXX,XX @@ static int preallocate(BlockDriverState *bs, int ret; QCowL2Meta *meta; - if (qemu_in_coroutine()) { - qemu_co_mutex_lock(&s->lock); - } + qemu_co_mutex_lock(&s->lock); assert(offset <= new_length); bytes = new_length - offset; @@ -XXX,XX +XXX,XX @@ static int preallocate(BlockDriverState *bs, ret = 0; done: + qemu_co_mutex_unlock(&s->lock); + params->ret = ret; +} + +static int preallocate(BlockDriverState *bs, + uint64_t offset, uint64_t new_length) +{ + PreallocCo params = { + .bs = bs, + .offset = offset, + .new_length = new_length, + .ret = -EINPROGRESS, + }; + if (qemu_in_coroutine()) { - qemu_co_mutex_unlock(&s->lock); + preallocate_co(&params); + } else { + Coroutine *co = qemu_coroutine_create(preallocate_co, &params); + bdrv_coroutine_enter(bs, co); + BDRV_POLL_WHILE(bs, params.ret == -EINPROGRESS); } - return ret; + return params.ret; } /* qcow2_refcount_metadata_size: -- 2.13.6 From: Max Reitz <mreitz@redhat.com> Apparently it would be a good idea to test that, too. Signed-off-by: Max Reitz <mreitz@redhat.com> Message-id: 20171009215533.12530-4-mreitz@redhat.com Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Jeff Cody <jcody@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/125 | 7 +- tests/qemu-iotests/125.out | 480 ++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 437 insertions(+), 50 deletions(-) diff --git a/tests/qemu-iotests/125 b/tests/qemu-iotests/125 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/125 +++ b/tests/qemu-iotests/125 @@ -XXX,XX +XXX,XX @@ fi # in B CREATION_SIZE=$((2 * 1024 * 1024 - 48 * 1024)) +# 512 is the actual test -- but it's good to test 64k as well, just to be sure. +for cluster_size in 512 64k; do # in kB for GROWTH_SIZE in 16 48 80; do for create_mode in off metadata falloc full; do for growth_mode in off metadata falloc full; do - echo "--- growth_size=$GROWTH_SIZE create_mode=$create_mode growth_mode=$growth_mode ---" + echo "--- cluster_size=$cluster_size growth_size=$GROWTH_SIZE create_mode=$create_mode growth_mode=$growth_mode ---" - IMGOPTS="preallocation=$create_mode,cluster_size=512" _make_test_img ${CREATION_SIZE} + IMGOPTS="preallocation=$create_mode,cluster_size=$cluster_size" _make_test_img ${CREATION_SIZE} $QEMU_IMG resize -f "$IMGFMT" --preallocation=$growth_mode "$TEST_IMG" +${GROWTH_SIZE}K host_size_0=$(get_image_size_on_host) @@ -XXX,XX +XXX,XX @@ for GROWTH_SIZE in 16 48 80; do done done done +done # success, all done echo '*** done' diff --git a/tests/qemu-iotests/125.out b/tests/qemu-iotests/125.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/125.out +++ b/tests/qemu-iotests/125.out @@ -XXX,XX +XXX,XX @@ QA output created by 125 ---- growth_size=16 create_mode=off growth_mode=off --- +--- cluster_size=512 growth_size=16 create_mode=off growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=off growth_mode=metadata --- +--- cluster_size=512 growth_size=16 create_mode=off growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=off growth_mode=falloc --- +--- cluster_size=512 growth_size=16 create_mode=off growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=off growth_mode=full --- +--- cluster_size=512 growth_size=16 create_mode=off growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=metadata growth_mode=off --- +--- cluster_size=512 growth_size=16 create_mode=metadata growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=metadata growth_mode=metadata --- +--- cluster_size=512 growth_size=16 create_mode=metadata growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=metadata growth_mode=falloc --- +--- cluster_size=512 growth_size=16 create_mode=metadata growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=metadata growth_mode=full --- +--- cluster_size=512 growth_size=16 create_mode=metadata growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=falloc growth_mode=off --- +--- cluster_size=512 growth_size=16 create_mode=falloc growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=falloc growth_mode=metadata --- +--- cluster_size=512 growth_size=16 create_mode=falloc growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=falloc growth_mode=falloc --- +--- cluster_size=512 growth_size=16 create_mode=falloc growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=falloc growth_mode=full --- +--- cluster_size=512 growth_size=16 create_mode=falloc growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=full growth_mode=off --- +--- cluster_size=512 growth_size=16 create_mode=full growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=full growth_mode=metadata --- +--- cluster_size=512 growth_size=16 create_mode=full growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=full growth_mode=falloc --- +--- cluster_size=512 growth_size=16 create_mode=full growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=16 create_mode=full growth_mode=full --- +--- cluster_size=512 growth_size=16 create_mode=full growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 16384/16384 bytes at offset 2048000 16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=off growth_mode=off --- +--- cluster_size=512 growth_size=48 create_mode=off growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=off growth_mode=metadata --- +--- cluster_size=512 growth_size=48 create_mode=off growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=off growth_mode=falloc --- +--- cluster_size=512 growth_size=48 create_mode=off growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=off growth_mode=full --- +--- cluster_size=512 growth_size=48 create_mode=off growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=metadata growth_mode=off --- +--- cluster_size=512 growth_size=48 create_mode=metadata growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=metadata growth_mode=metadata --- +--- cluster_size=512 growth_size=48 create_mode=metadata growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=metadata growth_mode=falloc --- +--- cluster_size=512 growth_size=48 create_mode=metadata growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=metadata growth_mode=full --- +--- cluster_size=512 growth_size=48 create_mode=metadata growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=falloc growth_mode=off --- +--- cluster_size=512 growth_size=48 create_mode=falloc growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=falloc growth_mode=metadata --- +--- cluster_size=512 growth_size=48 create_mode=falloc growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=falloc growth_mode=falloc --- +--- cluster_size=512 growth_size=48 create_mode=falloc growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=falloc growth_mode=full --- +--- cluster_size=512 growth_size=48 create_mode=falloc growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=full growth_mode=off --- +--- cluster_size=512 growth_size=48 create_mode=full growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=full growth_mode=metadata --- +--- cluster_size=512 growth_size=48 create_mode=full growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=full growth_mode=falloc --- +--- cluster_size=512 growth_size=48 create_mode=full growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=48 create_mode=full growth_mode=full --- +--- cluster_size=512 growth_size=48 create_mode=full growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 49152/49152 bytes at offset 2048000 48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=off growth_mode=off --- +--- cluster_size=512 growth_size=80 create_mode=off growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=off growth_mode=metadata --- +--- cluster_size=512 growth_size=80 create_mode=off growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=off growth_mode=falloc --- +--- cluster_size=512 growth_size=80 create_mode=off growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=off growth_mode=full --- +--- cluster_size=512 growth_size=80 create_mode=off growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=metadata growth_mode=off --- +--- cluster_size=512 growth_size=80 create_mode=metadata growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=metadata growth_mode=metadata --- +--- cluster_size=512 growth_size=80 create_mode=metadata growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=metadata growth_mode=falloc --- +--- cluster_size=512 growth_size=80 create_mode=metadata growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=metadata growth_mode=full --- +--- cluster_size=512 growth_size=80 create_mode=metadata growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=falloc growth_mode=off --- +--- cluster_size=512 growth_size=80 create_mode=falloc growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=falloc growth_mode=metadata --- +--- cluster_size=512 growth_size=80 create_mode=falloc growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=falloc growth_mode=falloc --- +--- cluster_size=512 growth_size=80 create_mode=falloc growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=falloc growth_mode=full --- +--- cluster_size=512 growth_size=80 create_mode=falloc growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=full growth_mode=off --- +--- cluster_size=512 growth_size=80 create_mode=full growth_mode=off --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=full growth_mode=metadata --- +--- cluster_size=512 growth_size=80 create_mode=full growth_mode=metadata --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=full growth_mode=falloc --- +--- cluster_size=512 growth_size=80 create_mode=full growth_mode=falloc --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 @@ -XXX,XX +XXX,XX @@ wrote 2048000/2048000 bytes at offset 0 wrote 81920/81920 bytes at offset 2048000 80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) ---- growth_size=80 create_mode=full growth_mode=full --- +--- cluster_size=512 growth_size=80 create_mode=full growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=off growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=off growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=off growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=off growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=metadata growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=metadata growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=metadata growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=metadata growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=falloc growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=falloc growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=falloc growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=falloc growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=full growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=full growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=full growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=16 create_mode=full growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16384/16384 bytes at offset 2048000 +16 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=off growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=off growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=off growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=off growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=metadata growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=metadata growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=metadata growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=metadata growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=falloc growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=falloc growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=falloc growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=falloc growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=full growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=full growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=full growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=48 create_mode=full growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 49152/49152 bytes at offset 2048000 +48 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=off growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=off growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=off growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=off growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=off +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=metadata growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=metadata growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=metadata growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=metadata growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=metadata +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=falloc growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=falloc growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=falloc growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=falloc growth_mode=full --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=falloc +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=full growth_mode=off --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=full growth_mode=metadata --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=full growth_mode=falloc --- +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full +Image resized. +wrote 2048000/2048000 bytes at offset 0 +1.953 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 81920/81920 bytes at offset 2048000 +80 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +--- cluster_size=64k growth_size=80 create_mode=full growth_mode=full --- Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2048000 preallocation=full Image resized. wrote 2048000/2048000 bytes at offset 0 -- 2.13.6 The following changes since commit d9bbfea646e86426d549bd612cd9f91e49aa50c2: Merge remote-tracking branch 'remotes/riscv/tags/riscv-qemu-upstream-v8.2' into staging (2018-03-09 10:58:57 +0000) are available in the git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for you to fetch changes up to a1be5921e35dcf84ce9e3c9a5c3029cea530a60b: Merge remote-tracking branch 'mreitz/tags/pull-block-2018-03-09' into queue-block (2018-03-09 16:09:06 +0100) ---------------------------------------------------------------- Block layer patches ---------------------------------------------------------------- Alberto Garcia (8): qcow2: Generalize validate_table_offset() into qcow2_validate_table() qcow2: Check L1 table offset in qcow2_snapshot_load_tmp() qcow2: Check L1 table parameters in qcow2_expand_zero_clusters() qcow2: Check snapshot L1 tables in qcow2_check_metadata_overlap() qcow2: Check snapshot L1 table in qcow2_snapshot_goto() qcow2: Check snapshot L1 table in qcow2_snapshot_delete() qcow2: Make qemu-img check detect corrupted L1 tables in snapshots iotests: Tweak 030 in order to trigger a race condition with parallel jobs Daniel P. Berrangé (1): block: implement the bdrv_reopen_prepare helper for LUKS driver Eric Blake (1): iotests: Mark all tests executable Fam Zheng (2): iotests: Test creating overlay when guest running iotests: Skip test for ENOMEM error Kevin Wolf (38): block/qapi: Introduce BlockdevCreateOptions block/qapi: Add qcow2 create options to schema qcow2: Rename qcow2_co_create2() to qcow2_co_create() qcow2: Let qcow2_create() handle protocol layer qcow2: Pass BlockdevCreateOptions to qcow2_co_create() qcow2: Use BlockdevRef in qcow2_co_create() qcow2: Use QCryptoBlockCreateOptions in qcow2_co_create() qcow2: Handle full/falloc preallocation in qcow2_co_create() util: Add qemu_opts_to_qdict_filtered() test-qemu-opts: Test qemu_opts_append() test-qemu-opts: Test qemu_opts_to_qdict_filtered() qdict: Introduce qdict_rename_keys() qcow2: Use visitor for options in qcow2_create() block: Make bdrv_is_whitelisted() public block: x-blockdev-create QMP command file-posix: Support .bdrv_co_create file-win32: Support .bdrv_co_create gluster: Support .bdrv_co_create rbd: Fix use after free in qemu_rbd_set_keypairs() error path rbd: Factor out qemu_rbd_connect() rbd: Remove non-schema options from runtime_opts rbd: Pass BlockdevOptionsRbd to qemu_rbd_connect() rbd: Support .bdrv_co_create rbd: Assign s->snap/image_name in qemu_rbd_open() rbd: Use qemu_rbd_connect() in qemu_rbd_do_create() nfs: Use QAPI options in nfs_client_open() nfs: Support .bdrv_co_create sheepdog: QAPIfy "redundancy" create option sheepdog: Support .bdrv_co_create ssh: Use QAPI BlockdevOptionsSsh object ssh: QAPIfy host-key-check option ssh: Pass BlockdevOptionsSsh to connect_to_ssh() ssh: Support .bdrv_co_create file-posix: Fix no-op bdrv_truncate() with falloc preallocation block: Fail bdrv_truncate() with negative size qemu-iotests: Test qcow2 over file image creation with QMP qemu-iotests: Test ssh image creation over QMP Merge remote-tracking branch 'mreitz/tags/pull-block-2018-03-09' into queue-block Paolo Bonzini (6): qcow2: introduce qcow2_write_caches and qcow2_flush_caches qcow2: fix flushing after dirty bitmap metadata writes qcow2: make qcow2_do_open a coroutine_fn qed: make bdrv_qed_do_open a coroutine_fn block: convert bdrv_invalidate_cache callback to coroutine_fn block: convert bdrv_check callback to coroutine_fn Stefan Hajnoczi (1): qemu-iotests: fix 203 migration completion race qapi/block-core.json | 326 +++++++++++++++++++++++++- block/qcow2.h | 12 +- include/block/block.h | 2 + include/block/block_int.h | 13 +- include/qapi/qmp/qdict.h | 6 + include/qemu/option.h | 2 + block.c | 138 +++++++++++- block/create.c | 76 +++++++ block/crypto.c | 7 + block/file-posix.c | 93 +++++--- block/file-win32.c | 47 +++- block/gluster.c | 135 +++++++---- block/iscsi.c | 6 +- block/nfs.c | 244 +++++++++----------- block/parallels.c | 17 +- block/qcow2-bitmap.c | 4 +- block/qcow2-cluster.c | 24 +- block/qcow2-refcount.c | 52 ++++- block/qcow2-snapshot.c | 24 +- block/qcow2.c | 552 ++++++++++++++++++++++++++++----------------- block/qed-check.c | 1 + block/qed-table.c | 26 +-- block/qed.c | 66 ++++-- block/rbd.c | 403 +++++++++++++++++---------------- block/sheepdog.c | 321 ++++++++++++++++++-------- block/ssh.c | 290 +++++++++++++----------- block/vdi.c | 6 +- block/vhdx.c | 7 +- block/vmdk.c | 7 +- qobject/qdict.c | 34 +++ tests/check-qdict.c | 129 +++++++++++ tests/test-qemu-opts.c | 253 +++++++++++++++++++++ util/qemu-option.c | 42 +++- block/Makefile.objs | 2 +- tests/qemu-iotests/030 | 52 ++++- tests/qemu-iotests/030.out | 4 +- tests/qemu-iotests/049.out | 8 +- tests/qemu-iotests/059 | 5 +- tests/qemu-iotests/080 | 22 +- tests/qemu-iotests/080.out | 58 ++++- tests/qemu-iotests/096 | 0 tests/qemu-iotests/112.out | 4 +- tests/qemu-iotests/124 | 0 tests/qemu-iotests/129 | 0 tests/qemu-iotests/132 | 0 tests/qemu-iotests/136 | 0 tests/qemu-iotests/139 | 0 tests/qemu-iotests/148 | 0 tests/qemu-iotests/152 | 0 tests/qemu-iotests/153 | 8 +- tests/qemu-iotests/153.out | 7 +- tests/qemu-iotests/163 | 0 tests/qemu-iotests/203 | 15 +- tests/qemu-iotests/203.out | 5 + tests/qemu-iotests/205 | 0 tests/qemu-iotests/206 | 436 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/206.out | 209 +++++++++++++++++ tests/qemu-iotests/207 | 261 +++++++++++++++++++++ tests/qemu-iotests/207.out | 75 ++++++ tests/qemu-iotests/group | 2 + 60 files changed, 3574 insertions(+), 964 deletions(-) create mode 100644 block/create.c mode change 100644 => 100755 tests/qemu-iotests/096 mode change 100644 => 100755 tests/qemu-iotests/124 mode change 100644 => 100755 tests/qemu-iotests/129 mode change 100644 => 100755 tests/qemu-iotests/132 mode change 100644 => 100755 tests/qemu-iotests/136 mode change 100644 => 100755 tests/qemu-iotests/139 mode change 100644 => 100755 tests/qemu-iotests/148 mode change 100644 => 100755 tests/qemu-iotests/152 mode change 100644 => 100755 tests/qemu-iotests/163 mode change 100644 => 100755 tests/qemu-iotests/205 create mode 100755 tests/qemu-iotests/206 create mode 100644 tests/qemu-iotests/206.out create mode 100755 tests/qemu-iotests/207 create mode 100644 tests/qemu-iotests/207.out From: "Daniel P. Berrange" <berrange@redhat.com> If the bdrv_reopen_prepare helper isn't provided, the qemu-img commit command fails to re-open the base layer after committing changes into it. Provide a no-op implementation for the LUKS driver, since there is not any custom work that needs doing to re-open it. Signed-off-by: Daniel P. Berrange <berrange@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/crypto.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/block/crypto.c b/block/crypto.c index XXXXXXX..XXXXXXX 100644 --- a/block/crypto.c +++ b/block/crypto.c @@ -XXX,XX +XXX,XX @@ static void block_crypto_close(BlockDriverState *bs) qcrypto_block_free(crypto->block); } +static int block_crypto_reopen_prepare(BDRVReopenState *state, + BlockReopenQueue *queue, Error **errp) +{ + /* nothing needs checking */ + return 0; +} /* * 1 MB bounce buffer gives good performance / memory tradeoff @@ -XXX,XX +XXX,XX @@ BlockDriver bdrv_crypto_luks = { .bdrv_truncate = block_crypto_truncate, .create_opts = &block_crypto_create_opts_luks, + .bdrv_reopen_prepare = block_crypto_reopen_prepare, .bdrv_refresh_limits = block_crypto_refresh_limits, .bdrv_co_preadv = block_crypto_co_preadv, .bdrv_co_pwritev = block_crypto_co_pwritev, -- 2.13.6 From: Paolo Bonzini <pbonzini@redhat.com> They will be used to avoid recursively taking s->lock during bdrv_open or bdrv_check. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <1516279431-30424-7-git-send-email-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.h | 2 ++ block/qcow2-refcount.c | 28 ++++++++++++++++++++++++++++ block/qcow2.c | 20 ++++---------------- 3 files changed, 34 insertions(+), 16 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry, int qcow2_update_snapshot_refcount(BlockDriverState *bs, int64_t l1_table_offset, int l1_size, int addend); +int coroutine_fn qcow2_flush_caches(BlockDriverState *bs); +int coroutine_fn qcow2_write_caches(BlockDriverState *bs); int qcow2_check_refcounts(BlockDriverState *bs, BdrvCheckResult *res, BdrvCheckMode fix); diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -XXX,XX +XXX,XX @@ void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry, } } +int coroutine_fn qcow2_write_caches(BlockDriverState *bs) +{ + BDRVQcow2State *s = bs->opaque; + int ret; + ret = qcow2_cache_write(bs, s->l2_table_cache); + if (ret < 0) { + return ret; + } + + if (qcow2_need_accurate_refcounts(s)) { + ret = qcow2_cache_write(bs, s->refcount_block_cache); + if (ret < 0) { + return ret; + } + } + + return 0; +} + +int coroutine_fn qcow2_flush_caches(BlockDriverState *bs) +{ + int ret = qcow2_write_caches(bs); + if (ret < 0) { + return ret; + } + + return bdrv_flush(bs->file->bs); +} /*********************************************************/ /* snapshots and image creation */ diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_mark_clean(BlockDriverState *bs) s->incompatible_features &= ~QCOW2_INCOMPAT_DIRTY; - ret = bdrv_flush(bs); + ret = qcow2_flush_caches(bs); if (ret < 0) { return ret; } @@ -XXX,XX +XXX,XX @@ int qcow2_mark_consistent(BlockDriverState *bs) BDRVQcow2State *s = bs->opaque; if (s->incompatible_features & QCOW2_INCOMPAT_CORRUPT) { - int ret = bdrv_flush(bs); + int ret = qcow2_flush_caches(bs); if (ret < 0) { return ret; } @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_flush_to_os(BlockDriverState *bs) int ret; qemu_co_mutex_lock(&s->lock); - ret = qcow2_cache_write(bs, s->l2_table_cache); - if (ret < 0) { - qemu_co_mutex_unlock(&s->lock); - return ret; - } - - if (qcow2_need_accurate_refcounts(s)) { - ret = qcow2_cache_write(bs, s->refcount_block_cache); - if (ret < 0) { - qemu_co_mutex_unlock(&s->lock); - return ret; - } - } + ret = qcow2_write_caches(bs); qemu_co_mutex_unlock(&s->lock); - return 0; + return ret; } static BlockMeasureInfo *qcow2_measure(QemuOpts *opts, BlockDriverState *in_bs, -- 2.13.6 From: Paolo Bonzini <pbonzini@redhat.com> update_header_sync itself does not need to flush the caches to disk. The only paths that allocate clusters are: - bitmap_list_store with in_place=false, called by update_ext_header_and_dir - store_bitmap_data, called by store_bitmap - store_bitmap, called by qcow2_store_persistent_dirty_bitmaps and followed by update_ext_header_and_dir So in the end the central place where we need to flush the caches is update_ext_header_and_dir. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2-bitmap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -XXX,XX +XXX,XX @@ static int update_header_sync(BlockDriverState *bs) return ret; } - return bdrv_flush(bs); + return bdrv_flush(bs->file->bs); } static inline void bitmap_table_to_be(uint64_t *bitmap_table, size_t size) @@ -XXX,XX +XXX,XX @@ static int update_ext_header_and_dir(BlockDriverState *bs, return ret; } - ret = bdrv_flush(bs->file->bs); + ret = qcow2_flush_caches(bs); if (ret < 0) { goto fail; } -- 2.13.6 From: Paolo Bonzini <pbonzini@redhat.com> It is called from qcow2_invalidate_cache in coroutine context (incoming migration runs in a coroutine), so it's cleaner if metadata is always loaded from a coroutine. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <1516279431-30424-4-git-send-email-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.c | 46 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 41 insertions(+), 5 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options, return ret; } -static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags, - Error **errp) +/* Called with s->lock held. */ +static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, + int flags, Error **errp) { BDRVQcow2State *s = bs->opaque; unsigned int len, i; @@ -XXX,XX +XXX,XX @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags, } } - /* Initialise locks */ - qemu_co_mutex_init(&s->lock); bs->supported_zero_flags = header.version >= 3 ? BDRV_REQ_MAY_UNMAP : 0; /* Repair image if dirty */ @@ -XXX,XX +XXX,XX @@ static int qcow2_do_open(BlockDriverState *bs, QDict *options, int flags, return ret; } +typedef struct QCow2OpenCo { + BlockDriverState *bs; + QDict *options; + int flags; + Error **errp; + int ret; +} QCow2OpenCo; + +static void coroutine_fn qcow2_open_entry(void *opaque) +{ + QCow2OpenCo *qoc = opaque; + BDRVQcow2State *s = qoc->bs->opaque; + + qemu_co_mutex_lock(&s->lock); + qoc->ret = qcow2_do_open(qoc->bs, qoc->options, qoc->flags, qoc->errp); + qemu_co_mutex_unlock(&s->lock); +} + static int qcow2_open(BlockDriverState *bs, QDict *options, int flags, Error **errp) { + BDRVQcow2State *s = bs->opaque; + QCow2OpenCo qoc = { + .bs = bs, + .options = options, + .flags = flags, + .errp = errp, + .ret = -EINPROGRESS + }; + bs->file = bdrv_open_child(NULL, options, "file", bs, &child_file, false, errp); if (!bs->file) { return -EINVAL; } - return qcow2_do_open(bs, options, flags, errp); + /* Initialise locks */ + qemu_co_mutex_init(&s->lock); + + if (qemu_in_coroutine()) { + /* From bdrv_co_create. */ + qcow2_open_entry(&qoc); + } else { + qemu_coroutine_enter(qemu_coroutine_create(qcow2_open_entry, &qoc)); + BDRV_POLL_WHILE(bs, qoc.ret == -EINPROGRESS); + } + return qoc.ret; } static void qcow2_refresh_limits(BlockDriverState *bs, Error **errp) -- 2.13.6 From: Paolo Bonzini <pbonzini@redhat.com> It is called from qed_invalidate_cache in coroutine context (incoming migration runs in a coroutine), so it's cleaner if metadata is always loaded from a coroutine. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <1516279431-30424-5-git-send-email-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qed.c | 40 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/block/qed.c b/block/qed.c index XXXXXXX..XXXXXXX 100644 --- a/block/qed.c +++ b/block/qed.c @@ -XXX,XX +XXX,XX @@ static void bdrv_qed_init_state(BlockDriverState *bs) qemu_co_queue_init(&s->allocating_write_reqs); } -static int bdrv_qed_do_open(BlockDriverState *bs, QDict *options, int flags, - Error **errp) +/* Called with table_lock held. */ +static int coroutine_fn bdrv_qed_do_open(BlockDriverState *bs, QDict *options, + int flags, Error **errp) { BDRVQEDState *s = bs->opaque; QEDHeader le_header; @@ -XXX,XX +XXX,XX @@ out: return ret; } +typedef struct QEDOpenCo { + BlockDriverState *bs; + QDict *options; + int flags; + Error **errp; + int ret; +} QEDOpenCo; + +static void coroutine_fn bdrv_qed_open_entry(void *opaque) +{ + QEDOpenCo *qoc = opaque; + BDRVQEDState *s = qoc->bs->opaque; + + qemu_co_mutex_lock(&s->table_lock); + qoc->ret = bdrv_qed_do_open(qoc->bs, qoc->options, qoc->flags, qoc->errp); + qemu_co_mutex_unlock(&s->table_lock); +} + static int bdrv_qed_open(BlockDriverState *bs, QDict *options, int flags, Error **errp) { + QEDOpenCo qoc = { + .bs = bs, + .options = options, + .flags = flags, + .errp = errp, + .ret = -EINPROGRESS + }; + bs->file = bdrv_open_child(NULL, options, "file", bs, &child_file, false, errp); if (!bs->file) { @@ -XXX,XX +XXX,XX @@ static int bdrv_qed_open(BlockDriverState *bs, QDict *options, int flags, } bdrv_qed_init_state(bs); - return bdrv_qed_do_open(bs, options, flags, errp); + if (qemu_in_coroutine()) { + bdrv_qed_open_entry(&qoc); + } else { + qemu_coroutine_enter(qemu_coroutine_create(bdrv_qed_open_entry, &qoc)); + BDRV_POLL_WHILE(bs, qoc.ret == -EINPROGRESS); + } + BDRV_POLL_WHILE(bs, qoc.ret == -EINPROGRESS); + return qoc.ret; } static void bdrv_qed_refresh_limits(BlockDriverState *bs, Error **errp) -- 2.13.6 From: Paolo Bonzini <pbonzini@redhat.com> QED's bdrv_invalidate_cache implementation would like to reuse functions that acquire/release the metadata locks. Call it from coroutine context to simplify the logic. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <1516279431-30424-6-git-send-email-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- include/block/block_int.h | 3 ++- block.c | 41 +++++++++++++++++++++++++++++++++++++---- block/iscsi.c | 6 +++--- block/nfs.c | 6 +++--- block/qcow2.c | 7 +++++-- block/qed.c | 13 +++++-------- block/rbd.c | 6 +++--- 7 files changed, 58 insertions(+), 24 deletions(-) diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ struct BlockDriver { /* * Invalidate any cached meta-data. */ - void (*bdrv_invalidate_cache)(BlockDriverState *bs, Error **errp); + void coroutine_fn (*bdrv_co_invalidate_cache)(BlockDriverState *bs, + Error **errp); int (*bdrv_inactivate)(BlockDriverState *bs); /* diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ void bdrv_init_with_whitelist(void) bdrv_init(); } -void bdrv_invalidate_cache(BlockDriverState *bs, Error **errp) +static void coroutine_fn bdrv_co_invalidate_cache(BlockDriverState *bs, + Error **errp) { BdrvChild *child, *parent; uint64_t perm, shared_perm; @@ -XXX,XX +XXX,XX @@ void bdrv_invalidate_cache(BlockDriverState *bs, Error **errp) } QLIST_FOREACH(child, &bs->children, next) { - bdrv_invalidate_cache(child->bs, &local_err); + bdrv_co_invalidate_cache(child->bs, &local_err); if (local_err) { error_propagate(errp, local_err); return; @@ -XXX,XX +XXX,XX @@ void bdrv_invalidate_cache(BlockDriverState *bs, Error **errp) } bdrv_set_perm(bs, perm, shared_perm); - if (bs->drv->bdrv_invalidate_cache) { - bs->drv->bdrv_invalidate_cache(bs, &local_err); + if (bs->drv->bdrv_co_invalidate_cache) { + bs->drv->bdrv_co_invalidate_cache(bs, &local_err); if (local_err) { bs->open_flags |= BDRV_O_INACTIVE; error_propagate(errp, local_err); @@ -XXX,XX +XXX,XX @@ void bdrv_invalidate_cache(BlockDriverState *bs, Error **errp) } } +typedef struct InvalidateCacheCo { + BlockDriverState *bs; + Error **errp; + bool done; +} InvalidateCacheCo; + +static void coroutine_fn bdrv_invalidate_cache_co_entry(void *opaque) +{ + InvalidateCacheCo *ico = opaque; + bdrv_co_invalidate_cache(ico->bs, ico->errp); + ico->done = true; +} + +void bdrv_invalidate_cache(BlockDriverState *bs, Error **errp) +{ + Coroutine *co; + InvalidateCacheCo ico = { + .bs = bs, + .done = false, + .errp = errp + }; + + if (qemu_in_coroutine()) { + /* Fast-path if already in coroutine context */ + bdrv_invalidate_cache_co_entry(&ico); + } else { + co = qemu_coroutine_create(bdrv_invalidate_cache_co_entry, &ico); + qemu_coroutine_enter(co); + BDRV_POLL_WHILE(bs, !ico.done); + } +} + void bdrv_invalidate_cache_all(Error **errp) { BlockDriverState *bs; diff --git a/block/iscsi.c b/block/iscsi.c index XXXXXXX..XXXXXXX 100644 --- a/block/iscsi.c +++ b/block/iscsi.c @@ -XXX,XX +XXX,XX @@ static int iscsi_get_info(BlockDriverState *bs, BlockDriverInfo *bdi) return 0; } -static void iscsi_invalidate_cache(BlockDriverState *bs, - Error **errp) +static void coroutine_fn iscsi_co_invalidate_cache(BlockDriverState *bs, + Error **errp) { IscsiLun *iscsilun = bs->opaque; iscsi_allocmap_invalidate(iscsilun); @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_iscsi = { .create_opts = &iscsi_create_opts, .bdrv_reopen_prepare = iscsi_reopen_prepare, .bdrv_reopen_commit = iscsi_reopen_commit, - .bdrv_invalidate_cache = iscsi_invalidate_cache, + .bdrv_co_invalidate_cache = iscsi_co_invalidate_cache, .bdrv_getlength = iscsi_getlength, .bdrv_get_info = iscsi_get_info, diff --git a/block/nfs.c b/block/nfs.c index XXXXXXX..XXXXXXX 100644 --- a/block/nfs.c +++ b/block/nfs.c @@ -XXX,XX +XXX,XX @@ static void nfs_refresh_filename(BlockDriverState *bs, QDict *options) } #ifdef LIBNFS_FEATURE_PAGECACHE -static void nfs_invalidate_cache(BlockDriverState *bs, - Error **errp) +static void coroutine_fn nfs_co_invalidate_cache(BlockDriverState *bs, + Error **errp) { NFSClient *client = bs->opaque; nfs_pagecache_invalidate(client->context, client->fh); @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_nfs = { .bdrv_refresh_filename = nfs_refresh_filename, #ifdef LIBNFS_FEATURE_PAGECACHE - .bdrv_invalidate_cache = nfs_invalidate_cache, + .bdrv_co_invalidate_cache = nfs_co_invalidate_cache, #endif }; diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static void qcow2_close(BlockDriverState *bs) qcow2_free_snapshots(bs); } -static void qcow2_invalidate_cache(BlockDriverState *bs, Error **errp) +static void coroutine_fn qcow2_co_invalidate_cache(BlockDriverState *bs, + Error **errp) { BDRVQcow2State *s = bs->opaque; int flags = s->flags; @@ -XXX,XX +XXX,XX @@ static void qcow2_invalidate_cache(BlockDriverState *bs, Error **errp) options = qdict_clone_shallow(bs->options); flags &= ~BDRV_O_INACTIVE; + qemu_co_mutex_lock(&s->lock); ret = qcow2_do_open(bs, options, flags, &local_err); + qemu_co_mutex_unlock(&s->lock); QDECREF(options); if (local_err) { error_propagate(errp, local_err); @@ -XXX,XX +XXX,XX @@ BlockDriver bdrv_qcow2 = { .bdrv_change_backing_file = qcow2_change_backing_file, .bdrv_refresh_limits = qcow2_refresh_limits, - .bdrv_invalidate_cache = qcow2_invalidate_cache, + .bdrv_co_invalidate_cache = qcow2_co_invalidate_cache, .bdrv_inactivate = qcow2_inactivate, .create_opts = &qcow2_create_opts, diff --git a/block/qed.c b/block/qed.c index XXXXXXX..XXXXXXX 100644 --- a/block/qed.c +++ b/block/qed.c @@ -XXX,XX +XXX,XX @@ static int bdrv_qed_change_backing_file(BlockDriverState *bs, return ret; } -static void bdrv_qed_invalidate_cache(BlockDriverState *bs, Error **errp) +static void coroutine_fn bdrv_qed_co_invalidate_cache(BlockDriverState *bs, + Error **errp) { BDRVQEDState *s = bs->opaque; Error *local_err = NULL; @@ -XXX,XX +XXX,XX @@ static void bdrv_qed_invalidate_cache(BlockDriverState *bs, Error **errp) bdrv_qed_close(bs); bdrv_qed_init_state(bs); - if (qemu_in_coroutine()) { - qemu_co_mutex_lock(&s->table_lock); - } + qemu_co_mutex_lock(&s->table_lock); ret = bdrv_qed_do_open(bs, NULL, bs->open_flags, &local_err); - if (qemu_in_coroutine()) { - qemu_co_mutex_unlock(&s->table_lock); - } + qemu_co_mutex_unlock(&s->table_lock); if (local_err) { error_propagate(errp, local_err); error_prepend(errp, "Could not reopen qed layer: "); @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_qed = { .bdrv_get_info = bdrv_qed_get_info, .bdrv_refresh_limits = bdrv_qed_refresh_limits, .bdrv_change_backing_file = bdrv_qed_change_backing_file, - .bdrv_invalidate_cache = bdrv_qed_invalidate_cache, + .bdrv_co_invalidate_cache = bdrv_qed_co_invalidate_cache, .bdrv_check = bdrv_qed_check, .bdrv_detach_aio_context = bdrv_qed_detach_aio_context, .bdrv_attach_aio_context = bdrv_qed_attach_aio_context, diff --git a/block/rbd.c b/block/rbd.c index XXXXXXX..XXXXXXX 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -XXX,XX +XXX,XX @@ static BlockAIOCB *qemu_rbd_aio_pdiscard(BlockDriverState *bs, #endif #ifdef LIBRBD_SUPPORTS_INVALIDATE -static void qemu_rbd_invalidate_cache(BlockDriverState *bs, - Error **errp) +static void coroutine_fn qemu_rbd_co_invalidate_cache(BlockDriverState *bs, + Error **errp) { BDRVRBDState *s = bs->opaque; int r = rbd_invalidate_cache(s->image); @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_rbd = { .bdrv_snapshot_list = qemu_rbd_snap_list, .bdrv_snapshot_goto = qemu_rbd_snap_rollback, #ifdef LIBRBD_SUPPORTS_INVALIDATE - .bdrv_invalidate_cache = qemu_rbd_invalidate_cache, + .bdrv_co_invalidate_cache = qemu_rbd_co_invalidate_cache, #endif }; -- 2.13.6 From: Paolo Bonzini <pbonzini@redhat.com> Suggested-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-Id: <1516279431-30424-8-git-send-email-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- include/block/block_int.h | 5 +++-- block.c | 43 ++++++++++++++++++++++++++++++++++++++++--- block/parallels.c | 17 +++++++++++------ block/qcow2.c | 23 +++++++++++++++++++---- block/qed-check.c | 1 + block/qed-table.c | 26 +++++++++----------------- block/qed.c | 13 +++++++++---- block/vdi.c | 6 +++--- block/vhdx.c | 7 ++++--- block/vmdk.c | 7 ++++--- 10 files changed, 103 insertions(+), 45 deletions(-) diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ struct BlockDriver { * Returns 0 for completed check, -errno for internal errors. * The check results are stored in result. */ - int (*bdrv_check)(BlockDriverState *bs, BdrvCheckResult *result, - BdrvCheckMode fix); + int coroutine_fn (*bdrv_co_check)(BlockDriverState *bs, + BdrvCheckResult *result, + BdrvCheckMode fix); int (*bdrv_amend_options)(BlockDriverState *bs, QemuOpts *opts, BlockDriverAmendStatusCB *status_cb, diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ static void bdrv_delete(BlockDriverState *bs) * free of errors) or -errno when an internal error occurred. The results of the * check are stored in res. */ -int bdrv_check(BlockDriverState *bs, BdrvCheckResult *res, BdrvCheckMode fix) +static int coroutine_fn bdrv_co_check(BlockDriverState *bs, + BdrvCheckResult *res, BdrvCheckMode fix) { if (bs->drv == NULL) { return -ENOMEDIUM; } - if (bs->drv->bdrv_check == NULL) { + if (bs->drv->bdrv_co_check == NULL) { return -ENOTSUP; } memset(res, 0, sizeof(*res)); - return bs->drv->bdrv_check(bs, res, fix); + return bs->drv->bdrv_co_check(bs, res, fix); +} + +typedef struct CheckCo { + BlockDriverState *bs; + BdrvCheckResult *res; + BdrvCheckMode fix; + int ret; +} CheckCo; + +static void bdrv_check_co_entry(void *opaque) +{ + CheckCo *cco = opaque; + cco->ret = bdrv_co_check(cco->bs, cco->res, cco->fix); +} + +int bdrv_check(BlockDriverState *bs, + BdrvCheckResult *res, BdrvCheckMode fix) +{ + Coroutine *co; + CheckCo cco = { + .bs = bs, + .res = res, + .ret = -EINPROGRESS, + .fix = fix, + }; + + if (qemu_in_coroutine()) { + /* Fast-path if already in coroutine context */ + bdrv_check_co_entry(&cco); + } else { + co = qemu_coroutine_create(bdrv_check_co_entry, &cco); + qemu_coroutine_enter(co); + BDRV_POLL_WHILE(bs, cco.ret == -EINPROGRESS); + } + + return cco.ret; } /* diff --git a/block/parallels.c b/block/parallels.c index XXXXXXX..XXXXXXX 100644 --- a/block/parallels.c +++ b/block/parallels.c @@ -XXX,XX +XXX,XX @@ static coroutine_fn int parallels_co_readv(BlockDriverState *bs, } -static int parallels_check(BlockDriverState *bs, BdrvCheckResult *res, - BdrvCheckMode fix) +static int coroutine_fn parallels_co_check(BlockDriverState *bs, + BdrvCheckResult *res, + BdrvCheckMode fix) { BDRVParallelsState *s = bs->opaque; int64_t size, prev_off, high_off; @@ -XXX,XX +XXX,XX @@ static int parallels_check(BlockDriverState *bs, BdrvCheckResult *res, return size; } + qemu_co_mutex_lock(&s->lock); if (s->header_unclean) { fprintf(stderr, "%s image was not closed correctly\n", fix & BDRV_FIX_ERRORS ? "Repairing" : "ERROR"); @@ -XXX,XX +XXX,XX @@ static int parallels_check(BlockDriverState *bs, BdrvCheckResult *res, prev_off = off; } + ret = 0; if (flush_bat) { ret = bdrv_pwrite_sync(bs->file, 0, s->header, s->header_size); if (ret < 0) { res->check_errors++; - return ret; + goto out; } } @@ -XXX,XX +XXX,XX @@ static int parallels_check(BlockDriverState *bs, BdrvCheckResult *res, if (ret < 0) { error_report_err(local_err); res->check_errors++; - return ret; + goto out; } res->leaks_fixed += count; } } - return 0; +out: + qemu_co_mutex_unlock(&s->lock); + return ret; } @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_parallels = { .bdrv_co_writev = parallels_co_writev, .supports_backing = true, .bdrv_co_create_opts = parallels_co_create_opts, - .bdrv_check = parallels_check, + .bdrv_co_check = parallels_co_check, .create_opts = &parallels_create_opts, }; diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ int qcow2_mark_consistent(BlockDriverState *bs) return 0; } -static int qcow2_check(BlockDriverState *bs, BdrvCheckResult *result, - BdrvCheckMode fix) +static int coroutine_fn qcow2_co_check_locked(BlockDriverState *bs, + BdrvCheckResult *result, + BdrvCheckMode fix) { int ret = qcow2_check_refcounts(bs, result, fix); if (ret < 0) { @@ -XXX,XX +XXX,XX @@ static int qcow2_check(BlockDriverState *bs, BdrvCheckResult *result, return ret; } +static int coroutine_fn qcow2_co_check(BlockDriverState *bs, + BdrvCheckResult *result, + BdrvCheckMode fix) +{ + BDRVQcow2State *s = bs->opaque; + int ret; + + qemu_co_mutex_lock(&s->lock); + ret = qcow2_co_check_locked(bs, result, fix); + qemu_co_mutex_unlock(&s->lock); + return ret; +} + static int validate_table_offset(BlockDriverState *bs, uint64_t offset, uint64_t entries, size_t entry_len) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, (s->incompatible_features & QCOW2_INCOMPAT_DIRTY)) { BdrvCheckResult result = {0}; - ret = qcow2_check(bs, &result, BDRV_FIX_ERRORS | BDRV_FIX_LEAKS); + ret = qcow2_co_check_locked(bs, &result, + BDRV_FIX_ERRORS | BDRV_FIX_LEAKS); if (ret < 0 || result.check_errors) { if (ret >= 0) { ret = -EIO; @@ -XXX,XX +XXX,XX @@ BlockDriver bdrv_qcow2 = { .bdrv_inactivate = qcow2_inactivate, .create_opts = &qcow2_create_opts, - .bdrv_check = qcow2_check, + .bdrv_co_check = qcow2_co_check, .bdrv_amend_options = qcow2_amend_options, .bdrv_detach_aio_context = qcow2_detach_aio_context, diff --git a/block/qed-check.c b/block/qed-check.c index XXXXXXX..XXXXXXX 100644 --- a/block/qed-check.c +++ b/block/qed-check.c @@ -XXX,XX +XXX,XX @@ static void qed_check_mark_clean(BDRVQEDState *s, BdrvCheckResult *result) qed_write_header_sync(s); } +/* Called with table_lock held. */ int qed_check(BDRVQEDState *s, BdrvCheckResult *result, bool fix) { QEDCheck check = { diff --git a/block/qed-table.c b/block/qed-table.c index XXXXXXX..XXXXXXX 100644 --- a/block/qed-table.c +++ b/block/qed-table.c @@ -XXX,XX +XXX,XX @@ #include "qed.h" #include "qemu/bswap.h" -/* Called either from qed_check or with table_lock held. */ +/* Called with table_lock held. */ static int qed_read_table(BDRVQEDState *s, uint64_t offset, QEDTable *table) { QEMUIOVector qiov; @@ -XXX,XX +XXX,XX @@ static int qed_read_table(BDRVQEDState *s, uint64_t offset, QEDTable *table) trace_qed_read_table(s, offset, table); - if (qemu_in_coroutine()) { - qemu_co_mutex_unlock(&s->table_lock); - } + qemu_co_mutex_unlock(&s->table_lock); ret = bdrv_preadv(s->bs->file, offset, &qiov); - if (qemu_in_coroutine()) { - qemu_co_mutex_lock(&s->table_lock); - } + qemu_co_mutex_lock(&s->table_lock); if (ret < 0) { goto out; } @@ -XXX,XX +XXX,XX @@ out: * @n: Number of elements * @flush: Whether or not to sync to disk * - * Called either from qed_check or with table_lock held. + * Called with table_lock held. */ static int qed_write_table(BDRVQEDState *s, uint64_t offset, QEDTable *table, unsigned int index, unsigned int n, bool flush) @@ -XXX,XX +XXX,XX @@ static int qed_write_table(BDRVQEDState *s, uint64_t offset, QEDTable *table, /* Adjust for offset into table */ offset += start * sizeof(uint64_t); - if (qemu_in_coroutine()) { - qemu_co_mutex_unlock(&s->table_lock); - } + qemu_co_mutex_unlock(&s->table_lock); ret = bdrv_pwritev(s->bs->file, offset, &qiov); - if (qemu_in_coroutine()) { - qemu_co_mutex_lock(&s->table_lock); - } + qemu_co_mutex_lock(&s->table_lock); trace_qed_write_table_cb(s, table, flush, ret); if (ret < 0) { goto out; @@ -XXX,XX +XXX,XX @@ int qed_read_l1_table_sync(BDRVQEDState *s) return qed_read_table(s, s->header.l1_table_offset, s->l1_table); } -/* Called either from qed_check or with table_lock held. */ +/* Called with table_lock held. */ int qed_write_l1_table(BDRVQEDState *s, unsigned int index, unsigned int n) { BLKDBG_EVENT(s->bs->file, BLKDBG_L1_UPDATE); @@ -XXX,XX +XXX,XX @@ int qed_write_l1_table_sync(BDRVQEDState *s, unsigned int index, return qed_write_l1_table(s, index, n); } -/* Called either from qed_check or with table_lock held. */ +/* Called with table_lock held. */ int qed_read_l2_table(BDRVQEDState *s, QEDRequest *request, uint64_t offset) { int ret; @@ -XXX,XX +XXX,XX @@ int qed_read_l2_table_sync(BDRVQEDState *s, QEDRequest *request, uint64_t offset return qed_read_l2_table(s, request, offset); } -/* Called either from qed_check or with table_lock held. */ +/* Called with table_lock held. */ int qed_write_l2_table(BDRVQEDState *s, QEDRequest *request, unsigned int index, unsigned int n, bool flush) { diff --git a/block/qed.c b/block/qed.c index XXXXXXX..XXXXXXX 100644 --- a/block/qed.c +++ b/block/qed.c @@ -XXX,XX +XXX,XX @@ static void coroutine_fn bdrv_qed_co_invalidate_cache(BlockDriverState *bs, } } -static int bdrv_qed_check(BlockDriverState *bs, BdrvCheckResult *result, - BdrvCheckMode fix) +static int bdrv_qed_co_check(BlockDriverState *bs, BdrvCheckResult *result, + BdrvCheckMode fix) { BDRVQEDState *s = bs->opaque; + int ret; - return qed_check(s, result, !!fix); + qemu_co_mutex_lock(&s->table_lock); + ret = qed_check(s, result, !!fix); + qemu_co_mutex_unlock(&s->table_lock); + + return ret; } static QemuOptsList qed_create_opts = { @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_qed = { .bdrv_refresh_limits = bdrv_qed_refresh_limits, .bdrv_change_backing_file = bdrv_qed_change_backing_file, .bdrv_co_invalidate_cache = bdrv_qed_co_invalidate_cache, - .bdrv_check = bdrv_qed_check, + .bdrv_co_check = bdrv_qed_co_check, .bdrv_detach_aio_context = bdrv_qed_detach_aio_context, .bdrv_attach_aio_context = bdrv_qed_attach_aio_context, .bdrv_co_drain_begin = bdrv_qed_co_drain_begin, diff --git a/block/vdi.c b/block/vdi.c index XXXXXXX..XXXXXXX 100644 --- a/block/vdi.c +++ b/block/vdi.c @@ -XXX,XX +XXX,XX @@ static void vdi_header_print(VdiHeader *header) } #endif -static int vdi_check(BlockDriverState *bs, BdrvCheckResult *res, - BdrvCheckMode fix) +static int coroutine_fn vdi_co_check(BlockDriverState *bs, BdrvCheckResult *res, + BdrvCheckMode fix) { /* TODO: additional checks possible. */ BDRVVdiState *s = (BDRVVdiState *)bs->opaque; @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_vdi = { .bdrv_get_info = vdi_get_info, .create_opts = &vdi_create_opts, - .bdrv_check = vdi_check, + .bdrv_co_check = vdi_co_check, }; static void bdrv_vdi_init(void) diff --git a/block/vhdx.c b/block/vhdx.c index XXXXXXX..XXXXXXX 100644 --- a/block/vhdx.c +++ b/block/vhdx.c @@ -XXX,XX +XXX,XX @@ exit: * r/w and any log has already been replayed, so there is nothing (currently) * for us to do here */ -static int vhdx_check(BlockDriverState *bs, BdrvCheckResult *result, - BdrvCheckMode fix) +static int coroutine_fn vhdx_co_check(BlockDriverState *bs, + BdrvCheckResult *result, + BdrvCheckMode fix) { BDRVVHDXState *s = bs->opaque; @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_vhdx = { .bdrv_co_writev = vhdx_co_writev, .bdrv_co_create_opts = vhdx_co_create_opts, .bdrv_get_info = vhdx_get_info, - .bdrv_check = vhdx_check, + .bdrv_co_check = vhdx_co_check, .bdrv_has_zero_init = bdrv_has_zero_init_1, .create_opts = &vhdx_create_opts, diff --git a/block/vmdk.c b/block/vmdk.c index XXXXXXX..XXXXXXX 100644 --- a/block/vmdk.c +++ b/block/vmdk.c @@ -XXX,XX +XXX,XX @@ static ImageInfo *vmdk_get_extent_info(VmdkExtent *extent) return info; } -static int vmdk_check(BlockDriverState *bs, BdrvCheckResult *result, - BdrvCheckMode fix) +static int coroutine_fn vmdk_co_check(BlockDriverState *bs, + BdrvCheckResult *result, + BdrvCheckMode fix) { BDRVVmdkState *s = bs->opaque; VmdkExtent *extent = NULL; @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_vmdk = { .instance_size = sizeof(BDRVVmdkState), .bdrv_probe = vmdk_probe, .bdrv_open = vmdk_open, - .bdrv_check = vmdk_check, + .bdrv_co_check = vmdk_co_check, .bdrv_reopen_prepare = vmdk_reopen_prepare, .bdrv_child_perm = bdrv_format_default_perms, .bdrv_co_preadv = vmdk_co_preadv, -- 2.13.6 From: Alberto Garcia <berto@igalia.com> This function checks that the offset and size of a table are valid. While the offset checks are fine, the size check is too generic, since it only verifies that the total size in bytes fits in a 64-bit integer. In practice all tables used in qcow2 have much smaller size limits, so the size needs to be checked again for each table using its actual limit. This patch generalizes this function by allowing the caller to specify the maximum size for that table. In addition to that it allows passing an Error variable. The function is also renamed and made public since we're going to use it in other parts of the code. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.h | 10 +++--- block/qcow2.c | 77 ++++++++++++++++++---------------------------- tests/qemu-iotests/080.out | 16 +++++----- 3 files changed, 43 insertions(+), 60 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ static inline int64_t qcow2_vm_state_offset(BDRVQcow2State *s) return (int64_t)s->l1_vm_state_index << (s->cluster_bits + s->l2_bits); } -static inline uint64_t qcow2_max_refcount_clusters(BDRVQcow2State *s) -{ - return QCOW_MAX_REFTABLE_SIZE >> s->cluster_bits; -} - static inline QCow2ClusterType qcow2_get_cluster_type(uint64_t l2_entry) { if (l2_entry & QCOW_OFLAG_COMPRESSED) { @@ -XXX,XX +XXX,XX @@ void qcow2_signal_corruption(BlockDriverState *bs, bool fatal, int64_t offset, int64_t size, const char *message_format, ...) GCC_FMT_ATTR(5, 6); +int qcow2_validate_table(BlockDriverState *bs, uint64_t offset, + uint64_t entries, size_t entry_len, + int64_t max_size_bytes, const char *table_name, + Error **errp); + /* qcow2-refcount.c functions */ int qcow2_refcount_init(BlockDriverState *bs); void qcow2_refcount_close(BlockDriverState *bs); diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_check(BlockDriverState *bs, return ret; } -static int validate_table_offset(BlockDriverState *bs, uint64_t offset, - uint64_t entries, size_t entry_len) +int qcow2_validate_table(BlockDriverState *bs, uint64_t offset, + uint64_t entries, size_t entry_len, + int64_t max_size_bytes, const char *table_name, + Error **errp) { BDRVQcow2State *s = bs->opaque; - uint64_t size; - /* Use signed INT64_MAX as the maximum even for uint64_t header fields, - * because values will be passed to qemu functions taking int64_t. */ - if (entries > INT64_MAX / entry_len) { - return -EINVAL; - } - - size = entries * entry_len; - - if (INT64_MAX - size < offset) { - return -EINVAL; + if (entries > max_size_bytes / entry_len) { + error_setg(errp, "%s too large", table_name); + return -EFBIG; } - /* Tables must be cluster aligned */ - if (offset_into_cluster(s, offset) != 0) { + /* Use signed INT64_MAX as the maximum even for uint64_t header fields, + * because values will be passed to qemu functions taking int64_t. */ + if ((INT64_MAX - entries * entry_len < offset) || + (offset_into_cluster(s, offset) != 0)) { + error_setg(errp, "%s offset invalid", table_name); return -EINVAL; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, s->refcount_table_size = header.refcount_table_clusters << (s->cluster_bits - 3); - if (header.refcount_table_clusters > qcow2_max_refcount_clusters(s)) { - error_setg(errp, "Reference count table too large"); - ret = -EINVAL; - goto fail; - } - if (header.refcount_table_clusters == 0 && !(flags & BDRV_O_CHECK)) { error_setg(errp, "Image does not contain a reference count table"); ret = -EINVAL; goto fail; } - ret = validate_table_offset(bs, s->refcount_table_offset, - s->refcount_table_size, sizeof(uint64_t)); + ret = qcow2_validate_table(bs, s->refcount_table_offset, + header.refcount_table_clusters, + s->cluster_size, QCOW_MAX_REFTABLE_SIZE, + "Reference count table", errp); if (ret < 0) { - error_setg(errp, "Invalid reference count table offset"); goto fail; } - /* Snapshot table offset/length */ - if (header.nb_snapshots > QCOW_MAX_SNAPSHOTS) { - error_setg(errp, "Too many snapshots"); - ret = -EINVAL; - goto fail; - } - - ret = validate_table_offset(bs, header.snapshots_offset, - header.nb_snapshots, - sizeof(QCowSnapshotHeader)); + /* The total size in bytes of the snapshot table is checked in + * qcow2_read_snapshots() because the size of each snapshot is + * variable and we don't know it yet. + * Here we only check the offset and number of snapshots. */ + ret = qcow2_validate_table(bs, header.snapshots_offset, + header.nb_snapshots, + sizeof(QCowSnapshotHeader), + sizeof(QCowSnapshotHeader) * QCOW_MAX_SNAPSHOTS, + "Snapshot table", errp); if (ret < 0) { - error_setg(errp, "Invalid snapshot table offset"); goto fail; } /* read the level 1 table */ - if (header.l1_size > QCOW_MAX_L1_SIZE / sizeof(uint64_t)) { - error_setg(errp, "Active L1 table too large"); - ret = -EFBIG; + ret = qcow2_validate_table(bs, header.l1_table_offset, + header.l1_size, sizeof(uint64_t), + QCOW_MAX_L1_SIZE, "Active L1 table", errp); + if (ret < 0) { goto fail; } s->l1_size = header.l1_size; + s->l1_table_offset = header.l1_table_offset; l1_vm_state_index = size_to_l1(s, header.size); if (l1_vm_state_index > INT_MAX) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, goto fail; } - ret = validate_table_offset(bs, header.l1_table_offset, - header.l1_size, sizeof(uint64_t)); - if (ret < 0) { - error_setg(errp, "Invalid L1 table offset"); - goto fail; - } - s->l1_table_offset = header.l1_table_offset; - - if (s->l1_size > 0) { s->l1_table = qemu_try_blockalign(bs->file->bs, ROUND_UP(s->l1_size * sizeof(uint64_t), 512)); diff --git a/tests/qemu-iotests/080.out b/tests/qemu-iotests/080.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/080.out +++ b/tests/qemu-iotests/080.out @@ -XXX,XX +XXX,XX @@ can't open device TEST_DIR/t.qcow2: Reference count table too large == Misaligned refcount table == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -can't open device TEST_DIR/t.qcow2: Invalid reference count table offset +can't open device TEST_DIR/t.qcow2: Reference count table offset invalid == Huge refcount offset == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -can't open device TEST_DIR/t.qcow2: Invalid reference count table offset +can't open device TEST_DIR/t.qcow2: Reference count table offset invalid == Invalid snapshot table == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -can't open device TEST_DIR/t.qcow2: Too many snapshots -can't open device TEST_DIR/t.qcow2: Too many snapshots -can't open device TEST_DIR/t.qcow2: Invalid snapshot table offset -can't open device TEST_DIR/t.qcow2: Invalid snapshot table offset +can't open device TEST_DIR/t.qcow2: Snapshot table too large +can't open device TEST_DIR/t.qcow2: Snapshot table too large +can't open device TEST_DIR/t.qcow2: Snapshot table offset invalid +can't open device TEST_DIR/t.qcow2: Snapshot table offset invalid == Hitting snapshot table size limit == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 @@ -XXX,XX +XXX,XX @@ read 512/512 bytes at offset 0 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 can't open device TEST_DIR/t.qcow2: Active L1 table too large can't open device TEST_DIR/t.qcow2: Active L1 table too large -can't open device TEST_DIR/t.qcow2: Invalid L1 table offset -can't open device TEST_DIR/t.qcow2: Invalid L1 table offset +can't open device TEST_DIR/t.qcow2: Active L1 table offset invalid +can't open device TEST_DIR/t.qcow2: Active L1 table offset invalid == Invalid L1 table (with internal snapshot in the image) == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -- 2.13.6 From: Alberto Garcia <berto@igalia.com> This function checks that the size of a snapshot's L1 table is not too large, but it doesn't validate the offset. We now have a function to take care of this, so let's use it. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2-snapshot.c | 8 +++++--- tests/qemu-iotests/080 | 10 +++++++++- tests/qemu-iotests/080.out | 8 +++++++- 3 files changed, 21 insertions(+), 5 deletions(-) diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-snapshot.c +++ b/block/qcow2-snapshot.c @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_load_tmp(BlockDriverState *bs, sn = &s->snapshots[snapshot_index]; /* Allocate and read in the snapshot's L1 table */ - if (sn->l1_size > QCOW_MAX_L1_SIZE / sizeof(uint64_t)) { - error_setg(errp, "Snapshot L1 table too large"); - return -EFBIG; + ret = qcow2_validate_table(bs, sn->l1_table_offset, sn->l1_size, + sizeof(uint64_t), QCOW_MAX_L1_SIZE, + "Snapshot L1 table", errp); + if (ret < 0) { + return ret; } new_l1_bytes = sn->l1_size * sizeof(uint64_t); new_l1_table = qemu_try_blockalign(bs->file->bs, diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/080 +++ b/tests/qemu-iotests/080 @@ -XXX,XX +XXX,XX @@ poke_file "$TEST_IMG" "$offset_l2_table_0" "\x80\x00\x00\xff\xff\xff\x00\x00" { $QEMU_IMG snapshot -c test $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir echo -echo "== Invalid snapshot L1 table ==" +echo "== Invalid snapshot L1 table offset ==" +_make_test_img 64M +{ $QEMU_IO -c "write 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir +{ $QEMU_IMG snapshot -c test $TEST_IMG; } 2>&1 | _filter_testdir +poke_file "$TEST_IMG" "$offset_snap1_l1_offset" "\x00\x00\x00\x00\x00\x40\x02\x00" +{ $QEMU_IMG convert -s test $TEST_IMG $TEST_IMG.snap; } 2>&1 | _filter_testdir + +echo +echo "== Invalid snapshot L1 table size ==" _make_test_img 64M { $QEMU_IO -c "write 0 512" $TEST_IMG; } 2>&1 | _filter_qemu_io | _filter_testdir { $QEMU_IMG snapshot -c test $TEST_IMG; } 2>&1 | _filter_testdir diff --git a/tests/qemu-iotests/080.out b/tests/qemu-iotests/080.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/080.out +++ b/tests/qemu-iotests/080.out @@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 0 qemu-img: Could not create snapshot 'test': -27 (File too large) qemu-img: Could not create snapshot 'test': -11 (Resource temporarily unavailable) -== Invalid snapshot L1 table == +== Invalid snapshot L1 table offset == +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +wrote 512/512 bytes at offset 0 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +qemu-img: Failed to load snapshot: Snapshot L1 table offset invalid + +== Invalid snapshot L1 table size == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 wrote 512/512 bytes at offset 0 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -- 2.13.6 From: Alberto Garcia <berto@igalia.com> This function iterates over all snapshots of a qcow2 file in order to expand all zero clusters, but it does not validate the snapshots' L1 tables first. We now have a function to take care of this, so let's use it. We can also take the opportunity to replace the sector-based bdrv_read() with bdrv_pread(). Cc: Eric Blake <eblake@redhat.com> Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2-cluster.c | 24 +++++++++++++++++------- tests/qemu-iotests/080 | 2 ++ tests/qemu-iotests/080.out | 4 ++++ 3 files changed, 23 insertions(+), 7 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -XXX,XX +XXX,XX @@ #include "qemu/osdep.h" #include <zlib.h> +#include "qapi/error.h" #include "qemu-common.h" #include "block/block_int.h" #include "block/qcow2.h" @@ -XXX,XX +XXX,XX @@ int qcow2_expand_zero_clusters(BlockDriverState *bs, } for (i = 0; i < s->nb_snapshots; i++) { - int l1_sectors = DIV_ROUND_UP(s->snapshots[i].l1_size * - sizeof(uint64_t), BDRV_SECTOR_SIZE); + int l1_size2; + uint64_t *new_l1_table; + Error *local_err = NULL; + + ret = qcow2_validate_table(bs, s->snapshots[i].l1_table_offset, + s->snapshots[i].l1_size, sizeof(uint64_t), + QCOW_MAX_L1_SIZE, "Snapshot L1 table", + &local_err); + if (ret < 0) { + error_report_err(local_err); + goto fail; + } - uint64_t *new_l1_table = - g_try_realloc(l1_table, l1_sectors * BDRV_SECTOR_SIZE); + l1_size2 = s->snapshots[i].l1_size * sizeof(uint64_t); + new_l1_table = g_try_realloc(l1_table, l1_size2); if (!new_l1_table) { ret = -ENOMEM; @@ -XXX,XX +XXX,XX @@ int qcow2_expand_zero_clusters(BlockDriverState *bs, l1_table = new_l1_table; - ret = bdrv_read(bs->file, - s->snapshots[i].l1_table_offset / BDRV_SECTOR_SIZE, - (void *)l1_table, l1_sectors); + ret = bdrv_pread(bs->file, s->snapshots[i].l1_table_offset, + l1_table, l1_size2); if (ret < 0) { goto fail; } diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/080 +++ b/tests/qemu-iotests/080 @@ -XXX,XX +XXX,XX @@ _make_test_img 64M { $QEMU_IMG snapshot -c test $TEST_IMG; } 2>&1 | _filter_testdir poke_file "$TEST_IMG" "$offset_snap1_l1_offset" "\x00\x00\x00\x00\x00\x40\x02\x00" { $QEMU_IMG convert -s test $TEST_IMG $TEST_IMG.snap; } 2>&1 | _filter_testdir +{ $QEMU_IMG amend -o compat=0.10 $TEST_IMG; } 2>&1 | _filter_testdir echo echo "== Invalid snapshot L1 table size ==" @@ -XXX,XX +XXX,XX @@ _make_test_img 64M { $QEMU_IMG snapshot -c test $TEST_IMG; } 2>&1 | _filter_testdir poke_file "$TEST_IMG" "$offset_snap1_l1_size" "\x10\x00\x00\x00" { $QEMU_IMG convert -s test $TEST_IMG $TEST_IMG.snap; } 2>&1 | _filter_testdir +{ $QEMU_IMG amend -o compat=0.10 $TEST_IMG; } 2>&1 | _filter_testdir # success, all done echo "*** done" diff --git a/tests/qemu-iotests/080.out b/tests/qemu-iotests/080.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/080.out +++ b/tests/qemu-iotests/080.out @@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 wrote 512/512 bytes at offset 0 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) qemu-img: Failed to load snapshot: Snapshot L1 table offset invalid +qemu-img: Snapshot L1 table offset invalid +qemu-img: Error while amending options: Invalid argument == Invalid snapshot L1 table size == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 wrote 512/512 bytes at offset 0 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) qemu-img: Failed to load snapshot: Snapshot L1 table too large +qemu-img: Snapshot L1 table too large +qemu-img: Error while amending options: File too large *** done -- 2.13.6 From: Alberto Garcia <berto@igalia.com> The inactive-l2 overlap check iterates uses the L1 tables from all snapshots, but it does not validate them first. We now have a function to take care of this, so let's use it. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2-refcount.c | 10 +++++++++- tests/qemu-iotests/080 | 4 ++++ tests/qemu-iotests/080.out | 4 ++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -XXX,XX +XXX,XX @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, uint64_t l1_ofs = s->snapshots[i].l1_table_offset; uint32_t l1_sz = s->snapshots[i].l1_size; uint64_t l1_sz2 = l1_sz * sizeof(uint64_t); - uint64_t *l1 = g_try_malloc(l1_sz2); + uint64_t *l1; int ret; + ret = qcow2_validate_table(bs, l1_ofs, l1_sz, sizeof(uint64_t), + QCOW_MAX_L1_SIZE, "", NULL); + if (ret < 0) { + return ret; + } + + l1 = g_try_malloc(l1_sz2); + if (l1_sz2 && l1 == NULL) { return -ENOMEM; } diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/080 +++ b/tests/qemu-iotests/080 @@ -XXX,XX +XXX,XX @@ _make_test_img 64M poke_file "$TEST_IMG" "$offset_snap1_l1_offset" "\x00\x00\x00\x00\x00\x40\x02\x00" { $QEMU_IMG convert -s test $TEST_IMG $TEST_IMG.snap; } 2>&1 | _filter_testdir { $QEMU_IMG amend -o compat=0.10 $TEST_IMG; } 2>&1 | _filter_testdir +{ $QEMU_IO -c "open -o overlap-check.inactive-l2=on $TEST_IMG" \ + -c 'write 0 4k'; } 2>&1 | _filter_qemu_io | _filter_testdir echo echo "== Invalid snapshot L1 table size ==" @@ -XXX,XX +XXX,XX @@ _make_test_img 64M poke_file "$TEST_IMG" "$offset_snap1_l1_size" "\x10\x00\x00\x00" { $QEMU_IMG convert -s test $TEST_IMG $TEST_IMG.snap; } 2>&1 | _filter_testdir { $QEMU_IMG amend -o compat=0.10 $TEST_IMG; } 2>&1 | _filter_testdir +{ $QEMU_IO -c "open -o overlap-check.inactive-l2=on $TEST_IMG" \ + -c 'write 0 4k'; } 2>&1 | _filter_qemu_io | _filter_testdir # success, all done echo "*** done" diff --git a/tests/qemu-iotests/080.out b/tests/qemu-iotests/080.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/080.out +++ b/tests/qemu-iotests/080.out @@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 0 qemu-img: Failed to load snapshot: Snapshot L1 table offset invalid qemu-img: Snapshot L1 table offset invalid qemu-img: Error while amending options: Invalid argument +Failed to flush the refcount block cache: Invalid argument +write failed: Invalid argument == Invalid snapshot L1 table size == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 @@ -XXX,XX +XXX,XX @@ wrote 512/512 bytes at offset 0 qemu-img: Failed to load snapshot: Snapshot L1 table too large qemu-img: Snapshot L1 table too large qemu-img: Error while amending options: File too large +Failed to flush the refcount block cache: File too large +write failed: File too large *** done -- 2.13.6 From: Alberto Garcia <berto@igalia.com> This function copies a snapshot's L1 table into the active one without validating it first. We now have a function to take care of this, so let's use it. Cc: Eric Blake <eblake@redhat.com> Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2-snapshot.c | 9 +++++++++ tests/qemu-iotests/080 | 2 ++ tests/qemu-iotests/080.out | 4 ++++ 3 files changed, 15 insertions(+) diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-snapshot.c +++ b/block/qcow2-snapshot.c @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id) { BDRVQcow2State *s = bs->opaque; QCowSnapshot *sn; + Error *local_err = NULL; int i, snapshot_index; int cur_l1_bytes, sn_l1_bytes; int ret; @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id) } sn = &s->snapshots[snapshot_index]; + ret = qcow2_validate_table(bs, sn->l1_table_offset, sn->l1_size, + sizeof(uint64_t), QCOW_MAX_L1_SIZE, + "Snapshot L1 table", &local_err); + if (ret < 0) { + error_report_err(local_err); + goto fail; + } + if (sn->disk_size != bs->total_sectors * BDRV_SECTOR_SIZE) { error_report("qcow2: Loading snapshots with different disk " "size is not implemented"); diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/080 +++ b/tests/qemu-iotests/080 @@ -XXX,XX +XXX,XX @@ poke_file "$TEST_IMG" "$offset_snap1_l1_offset" "\x00\x00\x00\x00\x00\x40\x02\x0 { $QEMU_IMG amend -o compat=0.10 $TEST_IMG; } 2>&1 | _filter_testdir { $QEMU_IO -c "open -o overlap-check.inactive-l2=on $TEST_IMG" \ -c 'write 0 4k'; } 2>&1 | _filter_qemu_io | _filter_testdir +{ $QEMU_IMG snapshot -a test $TEST_IMG; } 2>&1 | _filter_testdir echo echo "== Invalid snapshot L1 table size ==" @@ -XXX,XX +XXX,XX @@ poke_file "$TEST_IMG" "$offset_snap1_l1_size" "\x10\x00\x00\x00" { $QEMU_IMG amend -o compat=0.10 $TEST_IMG; } 2>&1 | _filter_testdir { $QEMU_IO -c "open -o overlap-check.inactive-l2=on $TEST_IMG" \ -c 'write 0 4k'; } 2>&1 | _filter_qemu_io | _filter_testdir +{ $QEMU_IMG snapshot -a test $TEST_IMG; } 2>&1 | _filter_testdir # success, all done echo "*** done" diff --git a/tests/qemu-iotests/080.out b/tests/qemu-iotests/080.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/080.out +++ b/tests/qemu-iotests/080.out @@ -XXX,XX +XXX,XX @@ qemu-img: Snapshot L1 table offset invalid qemu-img: Error while amending options: Invalid argument Failed to flush the refcount block cache: Invalid argument write failed: Invalid argument +qemu-img: Snapshot L1 table offset invalid +qemu-img: Could not apply snapshot 'test': Failed to load snapshot: Invalid argument == Invalid snapshot L1 table size == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 @@ -XXX,XX +XXX,XX @@ qemu-img: Snapshot L1 table too large qemu-img: Error while amending options: File too large Failed to flush the refcount block cache: File too large write failed: File too large +qemu-img: Snapshot L1 table too large +qemu-img: Could not apply snapshot 'test': Failed to load snapshot: File too large *** done -- 2.13.6 From: Alberto Garcia <berto@igalia.com> This function deletes a snapshot from disk, removing its entry from the snapshot table, freeing its L1 table and decreasing the refcounts of all clusters. The L1 table offset and size are however not validated. If we use invalid values in this function we'll probably corrupt the image even more, so we should return an error instead. We now have a function to take care of this, so let's use it. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2-snapshot.c | 7 +++++++ tests/qemu-iotests/080 | 2 ++ tests/qemu-iotests/080.out | 2 ++ 3 files changed, 11 insertions(+) diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-snapshot.c +++ b/block/qcow2-snapshot.c @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_delete(BlockDriverState *bs, } sn = s->snapshots[snapshot_index]; + ret = qcow2_validate_table(bs, sn.l1_table_offset, sn.l1_size, + sizeof(uint64_t), QCOW_MAX_L1_SIZE, + "Snapshot L1 table", errp); + if (ret < 0) { + return ret; + } + /* Remove it from the snapshot list */ memmove(s->snapshots + snapshot_index, s->snapshots + snapshot_index + 1, diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/080 +++ b/tests/qemu-iotests/080 @@ -XXX,XX +XXX,XX @@ poke_file "$TEST_IMG" "$offset_snap1_l1_offset" "\x00\x00\x00\x00\x00\x40\x02\x0 { $QEMU_IO -c "open -o overlap-check.inactive-l2=on $TEST_IMG" \ -c 'write 0 4k'; } 2>&1 | _filter_qemu_io | _filter_testdir { $QEMU_IMG snapshot -a test $TEST_IMG; } 2>&1 | _filter_testdir +{ $QEMU_IMG snapshot -d test $TEST_IMG; } 2>&1 | _filter_testdir echo echo "== Invalid snapshot L1 table size ==" @@ -XXX,XX +XXX,XX @@ poke_file "$TEST_IMG" "$offset_snap1_l1_size" "\x10\x00\x00\x00" { $QEMU_IO -c "open -o overlap-check.inactive-l2=on $TEST_IMG" \ -c 'write 0 4k'; } 2>&1 | _filter_qemu_io | _filter_testdir { $QEMU_IMG snapshot -a test $TEST_IMG; } 2>&1 | _filter_testdir +{ $QEMU_IMG snapshot -d test $TEST_IMG; } 2>&1 | _filter_testdir # success, all done echo "*** done" diff --git a/tests/qemu-iotests/080.out b/tests/qemu-iotests/080.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/080.out +++ b/tests/qemu-iotests/080.out @@ -XXX,XX +XXX,XX @@ Failed to flush the refcount block cache: Invalid argument write failed: Invalid argument qemu-img: Snapshot L1 table offset invalid qemu-img: Could not apply snapshot 'test': Failed to load snapshot: Invalid argument +qemu-img: Could not delete snapshot 'test': Snapshot L1 table offset invalid == Invalid snapshot L1 table size == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 @@ -XXX,XX +XXX,XX @@ Failed to flush the refcount block cache: File too large write failed: File too large qemu-img: Snapshot L1 table too large qemu-img: Could not apply snapshot 'test': Failed to load snapshot: File too large +qemu-img: Could not delete snapshot 'test': Snapshot L1 table too large *** done -- 2.13.6 From: Alberto Garcia <berto@igalia.com> 'qemu-img check' cannot detect if a snapshot's L1 table is corrupted. This patch checks the table's offset and size and reports corruption if the values are not valid. This patch doesn't add code to fix that corruption yet, only to detect and report it. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2-refcount.c | 14 ++++++++++++++ tests/qemu-iotests/080 | 2 ++ tests/qemu-iotests/080.out | 20 ++++++++++++++++++++ 3 files changed, 36 insertions(+) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -XXX,XX +XXX,XX @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, /* snapshots */ for (i = 0; i < s->nb_snapshots; i++) { sn = s->snapshots + i; + if (offset_into_cluster(s, sn->l1_table_offset)) { + fprintf(stderr, "ERROR snapshot %s (%s) l1_offset=%#" PRIx64 ": " + "L1 table is not cluster aligned; snapshot table entry " + "corrupted\n", sn->id_str, sn->name, sn->l1_table_offset); + res->corruptions++; + continue; + } + if (sn->l1_size > QCOW_MAX_L1_SIZE / sizeof(uint64_t)) { + fprintf(stderr, "ERROR snapshot %s (%s) l1_size=%#" PRIx32 ": " + "L1 table is too large; snapshot table entry corrupted\n", + sn->id_str, sn->name, sn->l1_size); + res->corruptions++; + continue; + } ret = check_refcounts_l1(bs, res, refcount_table, nb_clusters, sn->l1_table_offset, sn->l1_size, 0, fix); if (ret < 0) { diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/080 +++ b/tests/qemu-iotests/080 @@ -XXX,XX +XXX,XX @@ poke_file "$TEST_IMG" "$offset_snap1_l1_offset" "\x00\x00\x00\x00\x00\x40\x02\x0 -c 'write 0 4k'; } 2>&1 | _filter_qemu_io | _filter_testdir { $QEMU_IMG snapshot -a test $TEST_IMG; } 2>&1 | _filter_testdir { $QEMU_IMG snapshot -d test $TEST_IMG; } 2>&1 | _filter_testdir +_check_test_img echo echo "== Invalid snapshot L1 table size ==" @@ -XXX,XX +XXX,XX @@ poke_file "$TEST_IMG" "$offset_snap1_l1_size" "\x10\x00\x00\x00" -c 'write 0 4k'; } 2>&1 | _filter_qemu_io | _filter_testdir { $QEMU_IMG snapshot -a test $TEST_IMG; } 2>&1 | _filter_testdir { $QEMU_IMG snapshot -d test $TEST_IMG; } 2>&1 | _filter_testdir +_check_test_img # success, all done echo "*** done" diff --git a/tests/qemu-iotests/080.out b/tests/qemu-iotests/080.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/080.out +++ b/tests/qemu-iotests/080.out @@ -XXX,XX +XXX,XX @@ write failed: Invalid argument qemu-img: Snapshot L1 table offset invalid qemu-img: Could not apply snapshot 'test': Failed to load snapshot: Invalid argument qemu-img: Could not delete snapshot 'test': Snapshot L1 table offset invalid +ERROR snapshot 1 (test) l1_offset=0x400200: L1 table is not cluster aligned; snapshot table entry corrupted +Leaked cluster 4 refcount=2 reference=1 +Leaked cluster 5 refcount=2 reference=1 +Leaked cluster 6 refcount=1 reference=0 + +1 errors were found on the image. +Data may be corrupted, or further writes to the image may corrupt it. + +3 leaked clusters were found on the image. +This means waste of disk space, but no harm to data. == Invalid snapshot L1 table size == Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 @@ -XXX,XX +XXX,XX @@ write failed: File too large qemu-img: Snapshot L1 table too large qemu-img: Could not apply snapshot 'test': Failed to load snapshot: File too large qemu-img: Could not delete snapshot 'test': Snapshot L1 table too large +ERROR snapshot 1 (test) l1_size=0x10000000: L1 table is too large; snapshot table entry corrupted +Leaked cluster 4 refcount=2 reference=1 +Leaked cluster 5 refcount=2 reference=1 +Leaked cluster 6 refcount=1 reference=0 + +1 errors were found on the image. +Data may be corrupted, or further writes to the image may corrupt it. + +3 leaked clusters were found on the image. +This means waste of disk space, but no harm to data. *** done -- 2.13.6 This creates a BlockdevCreateOptions union type that will contain all of the options for image creation. We'll start out with an empty struct type BlockdevCreateNotSupported for all drivers. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ { 'command': 'blockdev-del', 'data': { 'node-name': 'str' } } ## +# @BlockdevCreateNotSupported: +# +# This is used for all drivers that don't support creating images. +# +# Since: 2.12 +## +{ 'struct': 'BlockdevCreateNotSupported', 'data': {}} + +## +# @BlockdevCreateOptions: +# +# Options for creating an image format on a given node. +# +# @driver block driver to create the image format +# +# Since: 2.12 +## +{ 'union': 'BlockdevCreateOptions', + 'base': { + 'driver': 'BlockdevDriver' }, + 'discriminator': 'driver', + 'data': { + 'blkdebug': 'BlockdevCreateNotSupported', + 'blkverify': 'BlockdevCreateNotSupported', + 'bochs': 'BlockdevCreateNotSupported', + 'cloop': 'BlockdevCreateNotSupported', + 'dmg': 'BlockdevCreateNotSupported', + 'file': 'BlockdevCreateNotSupported', + 'ftp': 'BlockdevCreateNotSupported', + 'ftps': 'BlockdevCreateNotSupported', + 'gluster': 'BlockdevCreateNotSupported', + 'host_cdrom': 'BlockdevCreateNotSupported', + 'host_device': 'BlockdevCreateNotSupported', + 'http': 'BlockdevCreateNotSupported', + 'https': 'BlockdevCreateNotSupported', + 'iscsi': 'BlockdevCreateNotSupported', + 'luks': 'BlockdevCreateNotSupported', + 'nbd': 'BlockdevCreateNotSupported', + 'nfs': 'BlockdevCreateNotSupported', + 'null-aio': 'BlockdevCreateNotSupported', + 'null-co': 'BlockdevCreateNotSupported', + 'nvme': 'BlockdevCreateNotSupported', + 'parallels': 'BlockdevCreateNotSupported', + 'qcow2': 'BlockdevCreateNotSupported', + 'qcow': 'BlockdevCreateNotSupported', + 'qed': 'BlockdevCreateNotSupported', + 'quorum': 'BlockdevCreateNotSupported', + 'raw': 'BlockdevCreateNotSupported', + 'rbd': 'BlockdevCreateNotSupported', + 'replication': 'BlockdevCreateNotSupported', + 'sheepdog': 'BlockdevCreateNotSupported', + 'ssh': 'BlockdevCreateNotSupported', + 'throttle': 'BlockdevCreateNotSupported', + 'vdi': 'BlockdevCreateNotSupported', + 'vhdx': 'BlockdevCreateNotSupported', + 'vmdk': 'BlockdevCreateNotSupported', + 'vpc': 'BlockdevCreateNotSupported', + 'vvfat': 'BlockdevCreateNotSupported', + 'vxhs': 'BlockdevCreateNotSupported' + } } + +## # @blockdev-open-tray: # # Opens a block device's tray. If there is a block driver state tree inserted as -- 2.13.6 Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ { 'command': 'blockdev-del', 'data': { 'node-name': 'str' } } ## +# @BlockdevQcow2Version: +# +# @v2: The original QCOW2 format as introduced in qemu 0.10 (version 2) +# @v3: The extended QCOW2 format as introduced in qemu 1.1 (version 3) +# +# Since: 2.12 +## +{ 'enum': 'BlockdevQcow2Version', + 'data': [ 'v2', 'v3' ] } + + +## +# @BlockdevCreateOptionsQcow2: +# +# Driver specific image creation options for qcow2. +# +# @file Node to create the image format on +# @size Size of the virtual disk in bytes +# @version Compatibility level (default: v3) +# @backing-file File name of the backing file if a backing file +# should be used +# @backing-fmt Name of the block driver to use for the backing file +# @encrypt Encryption options if the image should be encrypted +# @cluster-size qcow2 cluster size in bytes (default: 65536) +# @preallocation Preallocation mode for the new image (default: off) +# @lazy-refcounts True if refcounts may be updated lazily (default: off) +# @refcount-bits Width of reference counts in bits (default: 16) +# +# Since: 2.12 +## +{ 'struct': 'BlockdevCreateOptionsQcow2', + 'data': { 'file': 'BlockdevRef', + 'size': 'size', + '*version': 'BlockdevQcow2Version', + '*backing-file': 'str', + '*backing-fmt': 'BlockdevDriver', + '*encrypt': 'QCryptoBlockCreateOptions', + '*cluster-size': 'size', + '*preallocation': 'PreallocMode', + '*lazy-refcounts': 'bool', + '*refcount-bits': 'int' } } + +## # @BlockdevCreateNotSupported: # # This is used for all drivers that don't support creating images. @@ -XXX,XX +XXX,XX @@ 'null-co': 'BlockdevCreateNotSupported', 'nvme': 'BlockdevCreateNotSupported', 'parallels': 'BlockdevCreateNotSupported', - 'qcow2': 'BlockdevCreateNotSupported', + 'qcow2': 'BlockdevCreateOptionsQcow2', 'qcow': 'BlockdevCreateNotSupported', 'qed': 'BlockdevCreateNotSupported', 'quorum': 'BlockdevCreateNotSupported', -- 2.13.6 The functions originally known as qcow2_create() and qcow2_create2() are now called qcow2_co_create_opts() and qcow2_co_create(), which matches the names of the BlockDriver callbacks that they will implement at the end of this patch series. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/qcow2.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static uint64_t qcow2_opt_get_refcount_bits_del(QemuOpts *opts, int version, } static int coroutine_fn -qcow2_co_create2(const char *filename, int64_t total_size, - const char *backing_file, const char *backing_format, - int flags, size_t cluster_size, PreallocMode prealloc, - QemuOpts *opts, int version, int refcount_order, - const char *encryptfmt, Error **errp) +qcow2_co_create(const char *filename, int64_t total_size, + const char *backing_file, const char *backing_format, + int flags, size_t cluster_size, PreallocMode prealloc, + QemuOpts *opts, int version, int refcount_order, + const char *encryptfmt, Error **errp) { QDict *options; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt refcount_order = ctz32(refcount_bits); - ret = qcow2_co_create2(filename, size, backing_file, backing_fmt, flags, - cluster_size, prealloc, opts, version, refcount_order, - encryptfmt, &local_err); + ret = qcow2_co_create(filename, size, backing_file, backing_fmt, flags, + cluster_size, prealloc, opts, version, refcount_order, + encryptfmt, &local_err); error_propagate(errp, local_err); finish: -- 2.13.6 Currently, qcow2_create() only parses the QemuOpts and then calls qcow2_co_create() for the actual image creation, which includes both the creation of the actual file on the file system and writing a valid empty qcow2 image into that file. The plan is that qcow2_co_create() becomes the function that implements the functionality for a future 'blockdev-create' QMP command, which only creates the qcow2 layer on an already opened file node. This is a first step towards that goal: Let's move out anything that deals with the protocol layer from qcow2_co_create() into qcow2_create(). This means that qcow2_co_create() doesn't need a file name any more. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/qcow2.c | 64 +++++++++++++++++++++++++++++++++++------------------------ 1 file changed, 38 insertions(+), 26 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static uint64_t qcow2_opt_get_refcount_bits_del(QemuOpts *opts, int version, } static int coroutine_fn -qcow2_co_create(const char *filename, int64_t total_size, +qcow2_co_create(BlockDriverState *bs, int64_t total_size, const char *backing_file, const char *backing_format, int flags, size_t cluster_size, PreallocMode prealloc, QemuOpts *opts, int version, int refcount_order, @@ -XXX,XX +XXX,XX @@ qcow2_co_create(const char *filename, int64_t total_size, Error *local_err = NULL; int ret; - if (prealloc == PREALLOC_MODE_FULL || prealloc == PREALLOC_MODE_FALLOC) { - int64_t prealloc_size = - qcow2_calc_prealloc_size(total_size, cluster_size, refcount_order); - qemu_opt_set_number(opts, BLOCK_OPT_SIZE, prealloc_size, &error_abort); - qemu_opt_set(opts, BLOCK_OPT_PREALLOC, PreallocMode_str(prealloc), - &error_abort); - } - - ret = bdrv_create_file(filename, opts, &local_err); + blk = blk_new(BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); + ret = blk_insert_bs(blk, bs, errp); if (ret < 0) { - error_propagate(errp, local_err); - return ret; - } - - blk = blk_new_open(filename, NULL, NULL, - BDRV_O_RDWR | BDRV_O_RESIZE | BDRV_O_PROTOCOL, - &local_err); - if (blk == NULL) { - error_propagate(errp, local_err); - return -EIO; + goto out; } - blk_set_allow_write_beyond_eof(blk, true); /* Write the header */ @@ -XXX,XX +XXX,XX @@ qcow2_co_create(const char *filename, int64_t total_size, */ options = qdict_new(); qdict_put_str(options, "driver", "qcow2"); - blk = blk_new_open(filename, NULL, options, + qdict_put_str(options, "file", bs->node_name); + blk = blk_new_open(NULL, NULL, options, BDRV_O_RDWR | BDRV_O_RESIZE | BDRV_O_NO_FLUSH, &local_err); if (blk == NULL) { @@ -XXX,XX +XXX,XX @@ qcow2_co_create(const char *filename, int64_t total_size, */ options = qdict_new(); qdict_put_str(options, "driver", "qcow2"); - blk = blk_new_open(filename, NULL, options, + qdict_put_str(options, "file", bs->node_name); + blk = blk_new_open(NULL, NULL, options, BDRV_O_RDWR | BDRV_O_NO_BACKING | BDRV_O_NO_IO, &local_err); if (blk == NULL) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt uint64_t refcount_bits; int refcount_order; char *encryptfmt = NULL; + BlockDriverState *bs = NULL; Error *local_err = NULL; int ret; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt refcount_order = ctz32(refcount_bits); - ret = qcow2_co_create(filename, size, backing_file, backing_fmt, flags, + /* Create and open the file (protocol layer) */ + if (prealloc == PREALLOC_MODE_FULL || prealloc == PREALLOC_MODE_FALLOC) { + int64_t prealloc_size = + qcow2_calc_prealloc_size(size, cluster_size, refcount_order); + qemu_opt_set_number(opts, BLOCK_OPT_SIZE, prealloc_size, &error_abort); + qemu_opt_set(opts, BLOCK_OPT_PREALLOC, PreallocMode_str(prealloc), + &error_abort); + } + + ret = bdrv_create_file(filename, opts, errp); + if (ret < 0) { + goto finish; + } + + bs = bdrv_open(filename, NULL, NULL, + BDRV_O_RDWR | BDRV_O_RESIZE | BDRV_O_PROTOCOL, errp); + if (bs == NULL) { + ret = -EIO; + goto finish; + } + + /* Create the qcow2 image (format layer) */ + ret = qcow2_co_create(bs, size, backing_file, backing_fmt, flags, cluster_size, prealloc, opts, version, refcount_order, - encryptfmt, &local_err); - error_propagate(errp, local_err); + encryptfmt, errp); + if (ret < 0) { + goto finish; + } finish: + bdrv_unref(bs); + g_free(backing_file); g_free(backing_fmt); g_free(encryptfmt); -- 2.13.6 All of the simple options are now passed to qcow2_co_create() in a BlockdevCreateOptions object. Still missing: node-name and the encryption options. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- block/qcow2.c | 189 ++++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 151 insertions(+), 38 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int64_t qcow2_calc_prealloc_size(int64_t total_size, return meta_size + aligned_total_size; } -static size_t qcow2_opt_get_cluster_size_del(QemuOpts *opts, Error **errp) +static bool validate_cluster_size(size_t cluster_size, Error **errp) { - size_t cluster_size; - int cluster_bits; - - cluster_size = qemu_opt_get_size_del(opts, BLOCK_OPT_CLUSTER_SIZE, - DEFAULT_CLUSTER_SIZE); - cluster_bits = ctz32(cluster_size); + int cluster_bits = ctz32(cluster_size); if (cluster_bits < MIN_CLUSTER_BITS || cluster_bits > MAX_CLUSTER_BITS || (1 << cluster_bits) != cluster_size) { error_setg(errp, "Cluster size must be a power of two between %d and " "%dk", 1 << MIN_CLUSTER_BITS, 1 << (MAX_CLUSTER_BITS - 10)); + return false; + } + return true; +} + +static size_t qcow2_opt_get_cluster_size_del(QemuOpts *opts, Error **errp) +{ + size_t cluster_size; + + cluster_size = qemu_opt_get_size_del(opts, BLOCK_OPT_CLUSTER_SIZE, + DEFAULT_CLUSTER_SIZE); + if (!validate_cluster_size(cluster_size, errp)) { return 0; } return cluster_size; @@ -XXX,XX +XXX,XX @@ static uint64_t qcow2_opt_get_refcount_bits_del(QemuOpts *opts, int version, } static int coroutine_fn -qcow2_co_create(BlockDriverState *bs, int64_t total_size, - const char *backing_file, const char *backing_format, - int flags, size_t cluster_size, PreallocMode prealloc, - QemuOpts *opts, int version, int refcount_order, - const char *encryptfmt, Error **errp) +qcow2_co_create(BlockDriverState *bs, BlockdevCreateOptions *create_options, + QemuOpts *opts, const char *encryptfmt, Error **errp) { + BlockdevCreateOptionsQcow2 *qcow2_opts; QDict *options; /* @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, int64_t total_size, */ BlockBackend *blk; QCowHeader *header; + size_t cluster_size; + int version; + int refcount_order; uint64_t* refcount_table; Error *local_err = NULL; int ret; + /* Validate options and set default values */ + assert(create_options->driver == BLOCKDEV_DRIVER_QCOW2); + qcow2_opts = &create_options->u.qcow2; + + if (!QEMU_IS_ALIGNED(qcow2_opts->size, BDRV_SECTOR_SIZE)) { + error_setg(errp, "Image size must be a multiple of 512 bytes"); + ret = -EINVAL; + goto out; + } + + if (qcow2_opts->has_version) { + switch (qcow2_opts->version) { + case BLOCKDEV_QCOW2_VERSION_V2: + version = 2; + break; + case BLOCKDEV_QCOW2_VERSION_V3: + version = 3; + break; + default: + g_assert_not_reached(); + } + } else { + version = 3; + } + + if (qcow2_opts->has_cluster_size) { + cluster_size = qcow2_opts->cluster_size; + } else { + cluster_size = DEFAULT_CLUSTER_SIZE; + } + + if (!validate_cluster_size(cluster_size, errp)) { + return -EINVAL; + } + + if (!qcow2_opts->has_preallocation) { + qcow2_opts->preallocation = PREALLOC_MODE_OFF; + } + if (qcow2_opts->has_backing_file && + qcow2_opts->preallocation != PREALLOC_MODE_OFF) + { + error_setg(errp, "Backing file and preallocation cannot be used at " + "the same time"); + return -EINVAL; + } + if (qcow2_opts->has_backing_fmt && !qcow2_opts->has_backing_file) { + error_setg(errp, "Backing format cannot be used without backing file"); + return -EINVAL; + } + + if (!qcow2_opts->has_lazy_refcounts) { + qcow2_opts->lazy_refcounts = false; + } + if (version < 3 && qcow2_opts->lazy_refcounts) { + error_setg(errp, "Lazy refcounts only supported with compatibility " + "level 1.1 and above (use compat=1.1 or greater)"); + return -EINVAL; + } + + if (!qcow2_opts->has_refcount_bits) { + qcow2_opts->refcount_bits = 16; + } + if (qcow2_opts->refcount_bits > 64 || + !is_power_of_2(qcow2_opts->refcount_bits)) + { + error_setg(errp, "Refcount width must be a power of two and may not " + "exceed 64 bits"); + return -EINVAL; + } + if (version < 3 && qcow2_opts->refcount_bits != 16) { + error_setg(errp, "Different refcount widths than 16 bits require " + "compatibility level 1.1 or above (use compat=1.1 or " + "greater)"); + return -EINVAL; + } + refcount_order = ctz32(qcow2_opts->refcount_bits); + + + /* Create BlockBackend to write to the image */ blk = blk_new(BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); ret = blk_insert_bs(blk, bs, errp); if (ret < 0) { @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, int64_t total_size, /* We'll update this to correct value later */ header->crypt_method = cpu_to_be32(QCOW_CRYPT_NONE); - if (flags & BLOCK_FLAG_LAZY_REFCOUNTS) { + if (qcow2_opts->lazy_refcounts) { header->compatible_features |= cpu_to_be64(QCOW2_COMPAT_LAZY_REFCOUNTS); } @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, int64_t total_size, } /* Okay, now that we have a valid image, let's give it the right size */ - ret = blk_truncate(blk, total_size, PREALLOC_MODE_OFF, errp); + ret = blk_truncate(blk, qcow2_opts->size, PREALLOC_MODE_OFF, errp); if (ret < 0) { error_prepend(errp, "Could not resize image: "); goto out; } /* Want a backing file? There you go.*/ - if (backing_file) { - ret = bdrv_change_backing_file(blk_bs(blk), backing_file, backing_format); + if (qcow2_opts->has_backing_file) { + const char *backing_format = NULL; + + if (qcow2_opts->has_backing_fmt) { + backing_format = BlockdevDriver_str(qcow2_opts->backing_fmt); + } + + ret = bdrv_change_backing_file(blk_bs(blk), qcow2_opts->backing_file, + backing_format); if (ret < 0) { error_setg_errno(errp, -ret, "Could not assign backing file '%s' " - "with format '%s'", backing_file, backing_format); + "with format '%s'", qcow2_opts->backing_file, + backing_format); goto out; } } @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, int64_t total_size, } /* And if we're supposed to preallocate metadata, do that now */ - if (prealloc != PREALLOC_MODE_OFF) { - ret = preallocate(blk_bs(blk), 0, total_size); + if (qcow2_opts->preallocation != PREALLOC_MODE_OFF) { + ret = preallocate(blk_bs(blk), 0, qcow2_opts->size); if (ret < 0) { error_setg_errno(errp, -ret, "Could not preallocate metadata"); goto out; @@ -XXX,XX +XXX,XX @@ out: static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opts, Error **errp) { + BlockdevCreateOptions create_options; char *backing_file = NULL; char *backing_fmt = NULL; + BlockdevDriver backing_drv; char *buf = NULL; uint64_t size = 0; int flags = 0; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt PreallocMode prealloc; int version; uint64_t refcount_bits; - int refcount_order; char *encryptfmt = NULL; BlockDriverState *bs = NULL; Error *local_err = NULL; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt BDRV_SECTOR_SIZE); backing_file = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE); backing_fmt = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FMT); + backing_drv = qapi_enum_parse(&BlockdevDriver_lookup, backing_fmt, + 0, &local_err); + if (local_err) { + error_propagate(errp, local_err); + ret = -EINVAL; + goto finish; + } encryptfmt = qemu_opt_get_del(opts, BLOCK_OPT_ENCRYPT_FORMAT); if (encryptfmt) { if (qemu_opt_get(opts, BLOCK_OPT_ENCRYPT)) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt flags |= BLOCK_FLAG_LAZY_REFCOUNTS; } - if (backing_file && prealloc != PREALLOC_MODE_OFF) { - error_setg(errp, "Backing file and preallocation cannot be used at " - "the same time"); - ret = -EINVAL; - goto finish; - } - - if (version < 3 && (flags & BLOCK_FLAG_LAZY_REFCOUNTS)) { - error_setg(errp, "Lazy refcounts only supported with compatibility " - "level 1.1 and above (use compat=1.1 or greater)"); - ret = -EINVAL; - goto finish; - } - refcount_bits = qcow2_opt_get_refcount_bits_del(opts, version, &local_err); if (local_err) { error_propagate(errp, local_err); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt goto finish; } - refcount_order = ctz32(refcount_bits); /* Create and open the file (protocol layer) */ if (prealloc == PREALLOC_MODE_FULL || prealloc == PREALLOC_MODE_FALLOC) { + int refcount_order = ctz32(refcount_bits); int64_t prealloc_size = qcow2_calc_prealloc_size(size, cluster_size, refcount_order); qemu_opt_set_number(opts, BLOCK_OPT_SIZE, prealloc_size, &error_abort); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt } /* Create the qcow2 image (format layer) */ - ret = qcow2_co_create(bs, size, backing_file, backing_fmt, flags, - cluster_size, prealloc, opts, version, refcount_order, - encryptfmt, errp); + create_options = (BlockdevCreateOptions) { + .driver = BLOCKDEV_DRIVER_QCOW2, + .u.qcow2 = { + .file = &(BlockdevRef) { + .type = QTYPE_QSTRING, + .u.reference = bs->node_name, + }, + .size = size, + .has_version = true, + .version = version == 2 + ? BLOCKDEV_QCOW2_VERSION_V2 + : BLOCKDEV_QCOW2_VERSION_V3, + .has_backing_file = (backing_file != NULL), + .backing_file = backing_file, + .has_backing_fmt = (backing_fmt != NULL), + .backing_fmt = backing_drv, + .has_cluster_size = true, + .cluster_size = cluster_size, + .has_preallocation = true, + .preallocation = prealloc, + .has_lazy_refcounts = true, + .lazy_refcounts = (flags & BLOCK_FLAG_LAZY_REFCOUNTS), + .has_refcount_bits = true, + .refcount_bits = refcount_bits, + }, + }; + ret = qcow2_co_create(bs, &create_options, opts, encryptfmt, errp); if (ret < 0) { goto finish; } -- 2.13.6 Instead of passing a separate BlockDriverState* into qcow2_co_create(), make use of the BlockdevRef that is included in BlockdevCreateOptions. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- include/block/block.h | 1 + block.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 39 +++++++++++++++++++++++++-------------- 3 files changed, 73 insertions(+), 14 deletions(-) diff --git a/include/block/block.h b/include/block/block.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -XXX,XX +XXX,XX @@ BdrvChild *bdrv_open_child(const char *filename, BlockDriverState* parent, const BdrvChildRole *child_role, bool allow_none, Error **errp); +BlockDriverState *bdrv_open_blockdev_ref(BlockdevRef *ref, Error **errp); void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd, Error **errp); int bdrv_open_backing_file(BlockDriverState *bs, QDict *parent_options, diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ #include "qapi/qmp/qdict.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/qstring.h" +#include "qapi/qobject-output-visitor.h" +#include "qapi/qapi-visit-block-core.h" #include "sysemu/block-backend.h" #include "sysemu/sysemu.h" #include "qemu/notify.h" @@ -XXX,XX +XXX,XX @@ BdrvChild *bdrv_open_child(const char *filename, return c; } +/* TODO Future callers may need to specify parent/child_role in order for + * option inheritance to work. Existing callers use it for the root node. */ +BlockDriverState *bdrv_open_blockdev_ref(BlockdevRef *ref, Error **errp) +{ + BlockDriverState *bs = NULL; + Error *local_err = NULL; + QObject *obj = NULL; + QDict *qdict = NULL; + const char *reference = NULL; + Visitor *v = NULL; + + if (ref->type == QTYPE_QSTRING) { + reference = ref->u.reference; + } else { + BlockdevOptions *options = &ref->u.definition; + assert(ref->type == QTYPE_QDICT); + + v = qobject_output_visitor_new(&obj); + visit_type_BlockdevOptions(v, NULL, &options, &local_err); + if (local_err) { + error_propagate(errp, local_err); + goto fail; + } + visit_complete(v, &obj); + + qdict = qobject_to_qdict(obj); + qdict_flatten(qdict); + + /* bdrv_open_inherit() defaults to the values in bdrv_flags (for + * compatibility with other callers) rather than what we want as the + * real defaults. Apply the defaults here instead. */ + qdict_set_default_str(qdict, BDRV_OPT_CACHE_DIRECT, "off"); + qdict_set_default_str(qdict, BDRV_OPT_CACHE_NO_FLUSH, "off"); + qdict_set_default_str(qdict, BDRV_OPT_READ_ONLY, "off"); + } + + bs = bdrv_open_inherit(NULL, reference, qdict, 0, NULL, NULL, errp); + obj = NULL; + +fail: + qobject_decref(obj); + visit_free(v); + return bs; +} + static BlockDriverState *bdrv_append_temp_snapshot(BlockDriverState *bs, int flags, QDict *snapshot_options, diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static uint64_t qcow2_opt_get_refcount_bits_del(QemuOpts *opts, int version, } static int coroutine_fn -qcow2_co_create(BlockDriverState *bs, BlockdevCreateOptions *create_options, - QemuOpts *opts, const char *encryptfmt, Error **errp) +qcow2_co_create(BlockdevCreateOptions *create_options, QemuOpts *opts, + const char *encryptfmt, Error **errp) { BlockdevCreateOptionsQcow2 *qcow2_opts; QDict *options; @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, BlockdevCreateOptions *create_options, * 2 GB for 64k clusters, and we don't want to have a 2 GB initial file * size for any qcow2 image. */ - BlockBackend *blk; + BlockBackend *blk = NULL; + BlockDriverState *bs = NULL; QCowHeader *header; size_t cluster_size; int version; @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, BlockdevCreateOptions *create_options, Error *local_err = NULL; int ret; - /* Validate options and set default values */ assert(create_options->driver == BLOCKDEV_DRIVER_QCOW2); qcow2_opts = &create_options->u.qcow2; + bs = bdrv_open_blockdev_ref(qcow2_opts->file, errp); + if (bs == NULL) { + return -EIO; + } + + /* Validate options and set default values */ if (!QEMU_IS_ALIGNED(qcow2_opts->size, BDRV_SECTOR_SIZE)) { error_setg(errp, "Image size must be a multiple of 512 bytes"); ret = -EINVAL; @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, BlockdevCreateOptions *create_options, } if (!validate_cluster_size(cluster_size, errp)) { - return -EINVAL; + ret = -EINVAL; + goto out; } if (!qcow2_opts->has_preallocation) { @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, BlockdevCreateOptions *create_options, { error_setg(errp, "Backing file and preallocation cannot be used at " "the same time"); - return -EINVAL; + ret = -EINVAL; + goto out; } if (qcow2_opts->has_backing_fmt && !qcow2_opts->has_backing_file) { error_setg(errp, "Backing format cannot be used without backing file"); - return -EINVAL; + ret = -EINVAL; + goto out; } if (!qcow2_opts->has_lazy_refcounts) { @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, BlockdevCreateOptions *create_options, if (version < 3 && qcow2_opts->lazy_refcounts) { error_setg(errp, "Lazy refcounts only supported with compatibility " "level 1.1 and above (use compat=1.1 or greater)"); - return -EINVAL; + ret = -EINVAL; + goto out; } if (!qcow2_opts->has_refcount_bits) { @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, BlockdevCreateOptions *create_options, { error_setg(errp, "Refcount width must be a power of two and may not " "exceed 64 bits"); - return -EINVAL; + ret = -EINVAL; + goto out; } if (version < 3 && qcow2_opts->refcount_bits != 16) { error_setg(errp, "Different refcount widths than 16 bits require " "compatibility level 1.1 or above (use compat=1.1 or " "greater)"); - return -EINVAL; + ret = -EINVAL; + goto out; } refcount_order = ctz32(qcow2_opts->refcount_bits); @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockDriverState *bs, BlockdevCreateOptions *create_options, ret = 0; out: - if (blk) { - blk_unref(blk); - } + blk_unref(blk); + bdrv_unref(bs); return ret; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt .refcount_bits = refcount_bits, }, }; - ret = qcow2_co_create(bs, &create_options, opts, encryptfmt, errp); + ret = qcow2_co_create(&create_options, opts, encryptfmt, errp); if (ret < 0) { goto finish; } -- 2.13.6 Instead of passing the encryption format name and the QemuOpts down, use the QCryptoBlockCreateOptions contained in BlockdevCreateOptions. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/qcow2.c | 62 +++++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 17 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_crypt_method_from_format(const char *encryptfmt) } } -static int qcow2_set_up_encryption(BlockDriverState *bs, const char *encryptfmt, - QemuOpts *opts, Error **errp) +static QCryptoBlockCreateOptions * +qcow2_parse_encryption(const char *encryptfmt, QemuOpts *opts, Error **errp) { - BDRVQcow2State *s = bs->opaque; QCryptoBlockCreateOptions *cryptoopts = NULL; - QCryptoBlock *crypto = NULL; - int ret = -EINVAL; QDict *options, *encryptopts; int fmt; @@ -XXX,XX +XXX,XX @@ static int qcow2_set_up_encryption(BlockDriverState *bs, const char *encryptfmt, error_setg(errp, "Unknown encryption format '%s'", encryptfmt); break; } - if (!cryptoopts) { - ret = -EINVAL; - goto out; + + QDECREF(encryptopts); + return cryptoopts; +} + +static int qcow2_set_up_encryption(BlockDriverState *bs, + QCryptoBlockCreateOptions *cryptoopts, + Error **errp) +{ + BDRVQcow2State *s = bs->opaque; + QCryptoBlock *crypto = NULL; + int fmt, ret; + + switch (cryptoopts->format) { + case Q_CRYPTO_BLOCK_FORMAT_LUKS: + fmt = QCOW_CRYPT_LUKS; + break; + case Q_CRYPTO_BLOCK_FORMAT_QCOW: + fmt = QCOW_CRYPT_AES; + break; + default: + error_setg(errp, "Crypto format not supported in qcow2"); + return -EINVAL; } + s->crypt_method_header = fmt; crypto = qcrypto_block_create(cryptoopts, "encrypt.", @@ -XXX,XX +XXX,XX @@ static int qcow2_set_up_encryption(BlockDriverState *bs, const char *encryptfmt, qcow2_crypto_hdr_write_func, bs, errp); if (!crypto) { - ret = -EINVAL; - goto out; + return -EINVAL; } ret = qcow2_update_header(bs); @@ -XXX,XX +XXX,XX @@ static int qcow2_set_up_encryption(BlockDriverState *bs, const char *encryptfmt, goto out; } + ret = 0; out: - QDECREF(encryptopts); qcrypto_block_free(crypto); - qapi_free_QCryptoBlockCreateOptions(cryptoopts); return ret; } @@ -XXX,XX +XXX,XX @@ static uint64_t qcow2_opt_get_refcount_bits_del(QemuOpts *opts, int version, } static int coroutine_fn -qcow2_co_create(BlockdevCreateOptions *create_options, QemuOpts *opts, - const char *encryptfmt, Error **errp) +qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) { BlockdevCreateOptionsQcow2 *qcow2_opts; QDict *options; @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, QemuOpts *opts, } /* Want encryption? There you go. */ - if (encryptfmt) { - ret = qcow2_set_up_encryption(blk_bs(blk), encryptfmt, opts, errp); + if (qcow2_opts->has_encrypt) { + ret = qcow2_set_up_encryption(blk_bs(blk), qcow2_opts->encrypt, errp); if (ret < 0) { goto out; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt int version; uint64_t refcount_bits; char *encryptfmt = NULL; + QCryptoBlockCreateOptions *cryptoopts = NULL; BlockDriverState *bs = NULL; Error *local_err = NULL; int ret; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt ret = -EINVAL; goto finish; } + encryptfmt = qemu_opt_get_del(opts, BLOCK_OPT_ENCRYPT_FORMAT); if (encryptfmt) { if (qemu_opt_get(opts, BLOCK_OPT_ENCRYPT)) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt } else if (qemu_opt_get_bool_del(opts, BLOCK_OPT_ENCRYPT, false)) { encryptfmt = g_strdup("aes"); } + if (encryptfmt) { + cryptoopts = qcow2_parse_encryption(encryptfmt, opts, errp); + if (cryptoopts == NULL) { + ret = -EINVAL; + goto finish; + } + } + cluster_size = qcow2_opt_get_cluster_size_del(opts, &local_err); if (local_err) { error_propagate(errp, local_err); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt .backing_file = backing_file, .has_backing_fmt = (backing_fmt != NULL), .backing_fmt = backing_drv, + .has_encrypt = (encryptfmt != NULL), + .encrypt = cryptoopts, .has_cluster_size = true, .cluster_size = cluster_size, .has_preallocation = true, @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt .refcount_bits = refcount_bits, }, }; - ret = qcow2_co_create(&create_options, opts, encryptfmt, errp); + ret = qcow2_co_create(&create_options, errp); if (ret < 0) { goto finish; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt finish: bdrv_unref(bs); + qapi_free_QCryptoBlockCreateOptions(cryptoopts); g_free(backing_file); g_free(backing_fmt); g_free(encryptfmt); -- 2.13.6 Once qcow2_co_create() can be called directly on an already existing node, we must provide the 'full' and 'falloc' preallocation modes outside of creating the image on the protocol layer. Fortunately, we have preallocated truncate now which can provide this functionality. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/qcow2.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } blk_set_allow_write_beyond_eof(blk, true); + /* Clear the protocol layer and preallocate it if necessary */ + ret = blk_truncate(blk, 0, PREALLOC_MODE_OFF, errp); + if (ret < 0) { + goto out; + } + + if (qcow2_opts->preallocation == PREALLOC_MODE_FULL || + qcow2_opts->preallocation == PREALLOC_MODE_FALLOC) + { + int64_t prealloc_size = + qcow2_calc_prealloc_size(qcow2_opts->size, cluster_size, + refcount_order); + + ret = blk_truncate(blk, prealloc_size, qcow2_opts->preallocation, errp); + if (ret < 0) { + goto out; + } + } + /* Write the header */ QEMU_BUILD_BUG_ON((1 << MIN_CLUSTER_BITS) < sizeof(*header)); header = g_malloc0(cluster_size); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt /* Create and open the file (protocol layer) */ - if (prealloc == PREALLOC_MODE_FULL || prealloc == PREALLOC_MODE_FALLOC) { - int refcount_order = ctz32(refcount_bits); - int64_t prealloc_size = - qcow2_calc_prealloc_size(size, cluster_size, refcount_order); - qemu_opt_set_number(opts, BLOCK_OPT_SIZE, prealloc_size, &error_abort); - qemu_opt_set(opts, BLOCK_OPT_PREALLOC, PreallocMode_str(prealloc), - &error_abort); - } - ret = bdrv_create_file(filename, opts, errp); if (ret < 0) { goto finish; -- 2.13.6 This allows, given a QemuOpts for a QemuOptsList that was merged from multiple QemuOptsList, to only consider those options that exist in one specific list. Block drivers need this to separate format-layer create options from protocol-level options. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- include/qemu/option.h | 2 ++ util/qemu-option.c | 42 +++++++++++++++++++++++++++++++++++++----- 2 files changed, 39 insertions(+), 5 deletions(-) diff --git a/include/qemu/option.h b/include/qemu/option.h index XXXXXXX..XXXXXXX 100644 --- a/include/qemu/option.h +++ b/include/qemu/option.h @@ -XXX,XX +XXX,XX @@ void qemu_opts_set_defaults(QemuOptsList *list, const char *params, int permit_abbrev); QemuOpts *qemu_opts_from_qdict(QemuOptsList *list, const QDict *qdict, Error **errp); +QDict *qemu_opts_to_qdict_filtered(QemuOpts *opts, QDict *qdict, + QemuOptsList *list, bool del); QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict); void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp); diff --git a/util/qemu-option.c b/util/qemu-option.c index XXXXXXX..XXXXXXX 100644 --- a/util/qemu-option.c +++ b/util/qemu-option.c @@ -XXX,XX +XXX,XX @@ void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp) } /* - * Convert from QemuOpts to QDict. - * The QDict values are of type QString. + * Convert from QemuOpts to QDict. The QDict values are of type QString. + * + * If @list is given, only add those options to the QDict that are contained in + * the list. If @del is true, any options added to the QDict are removed from + * the QemuOpts, otherwise they remain there. + * + * If two options in @opts have the same name, they are processed in order + * so that the last one wins (consistent with the reverse iteration in + * qemu_opt_find()), but all of them are deleted if @del is true. + * * TODO We'll want to use types appropriate for opt->desc->type, but * this is enough for now. */ -QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict) +QDict *qemu_opts_to_qdict_filtered(QemuOpts *opts, QDict *qdict, + QemuOptsList *list, bool del) { - QemuOpt *opt; + QemuOpt *opt, *next; if (!qdict) { qdict = qdict_new(); @@ -XXX,XX +XXX,XX @@ QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict) if (opts->id) { qdict_put_str(qdict, "id", opts->id); } - QTAILQ_FOREACH(opt, &opts->head, next) { + QTAILQ_FOREACH_SAFE(opt, &opts->head, next, next) { + if (list) { + QemuOptDesc *desc; + bool found = false; + for (desc = list->desc; desc->name; desc++) { + if (!strcmp(desc->name, opt->name)) { + found = true; + break; + } + } + if (!found) { + continue; + } + } qdict_put_str(qdict, opt->name, opt->str); + if (del) { + qemu_opt_del(opt); + } } return qdict; } +/* Copy all options in a QemuOpts to the given QDict. See + * qemu_opts_to_qdict_filtered() for details. */ +QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict) +{ + return qemu_opts_to_qdict_filtered(opts, qdict, NULL, false); +} + /* Validate parsed opts against descriptions where no * descriptions were provided in the QemuOptsList. */ -- 2.13.6 Basic test for merging two QemuOptsLists. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- tests/test-qemu-opts.c | 128 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) diff --git a/tests/test-qemu-opts.c b/tests/test-qemu-opts.c index XXXXXXX..XXXXXXX 100644 --- a/tests/test-qemu-opts.c +++ b/tests/test-qemu-opts.c @@ -XXX,XX +XXX,XX @@ static QemuOptsList opts_list_01 = { { .name = "str1", .type = QEMU_OPT_STRING, + .help = "Help texts are preserved in qemu_opts_append", + .def_value_str = "default", },{ .name = "str2", .type = QEMU_OPT_STRING, @@ -XXX,XX +XXX,XX @@ static QemuOptsList opts_list_01 = { },{ .name = "number1", .type = QEMU_OPT_NUMBER, + .help = "Having help texts only for some options is okay", },{ .name = "number2", .type = QEMU_OPT_NUMBER, @@ -XXX,XX +XXX,XX @@ static void test_opts_parse_size(void) qemu_opts_reset(&opts_list_02); } +static void append_verify_list_01(QemuOptDesc *desc, bool with_overlapping) +{ + int i = 0; + + if (with_overlapping) { + g_assert_cmpstr(desc[i].name, ==, "str1"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_STRING); + g_assert_cmpstr(desc[i].help, ==, + "Help texts are preserved in qemu_opts_append"); + g_assert_cmpstr(desc[i].def_value_str, ==, "default"); + i++; + + g_assert_cmpstr(desc[i].name, ==, "str2"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_STRING); + g_assert_cmpstr(desc[i].help, ==, NULL); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); + i++; + } + + g_assert_cmpstr(desc[i].name, ==, "str3"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_STRING); + g_assert_cmpstr(desc[i].help, ==, NULL); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); + i++; + + g_assert_cmpstr(desc[i].name, ==, "number1"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_NUMBER); + g_assert_cmpstr(desc[i].help, ==, + "Having help texts only for some options is okay"); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); + i++; + + g_assert_cmpstr(desc[i].name, ==, "number2"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_NUMBER); + g_assert_cmpstr(desc[i].help, ==, NULL); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); + i++; + + g_assert_cmpstr(desc[i].name, ==, NULL); +} + +static void append_verify_list_02(QemuOptDesc *desc) +{ + int i = 0; + + g_assert_cmpstr(desc[i].name, ==, "str1"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_STRING); + g_assert_cmpstr(desc[i].help, ==, NULL); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); + i++; + + g_assert_cmpstr(desc[i].name, ==, "str2"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_STRING); + g_assert_cmpstr(desc[i].help, ==, NULL); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); + i++; + + g_assert_cmpstr(desc[i].name, ==, "bool1"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_BOOL); + g_assert_cmpstr(desc[i].help, ==, NULL); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); + i++; + + g_assert_cmpstr(desc[i].name, ==, "bool2"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_BOOL); + g_assert_cmpstr(desc[i].help, ==, NULL); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); + i++; + + g_assert_cmpstr(desc[i].name, ==, "size1"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_SIZE); + g_assert_cmpstr(desc[i].help, ==, NULL); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); + i++; + + g_assert_cmpstr(desc[i].name, ==, "size2"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_SIZE); + g_assert_cmpstr(desc[i].help, ==, NULL); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); + i++; + + g_assert_cmpstr(desc[i].name, ==, "size3"); + g_assert_cmpint(desc[i].type, ==, QEMU_OPT_SIZE); + g_assert_cmpstr(desc[i].help, ==, NULL); + g_assert_cmpstr(desc[i].def_value_str, ==, NULL); +} + +static void test_opts_append_to_null(void) +{ + QemuOptsList *merged; + + merged = qemu_opts_append(NULL, &opts_list_01); + g_assert(merged != &opts_list_01); + + g_assert_cmpstr(merged->name, ==, NULL); + g_assert_cmpstr(merged->implied_opt_name, ==, NULL); + g_assert_false(merged->merge_lists); + + append_verify_list_01(merged->desc, true); + + qemu_opts_free(merged); +} + +static void test_opts_append(void) +{ + QemuOptsList *first, *merged; + + first = qemu_opts_append(NULL, &opts_list_02); + merged = qemu_opts_append(first, &opts_list_01); + g_assert(first != &opts_list_02); + g_assert(merged != &opts_list_01); + + g_assert_cmpstr(merged->name, ==, NULL); + g_assert_cmpstr(merged->implied_opt_name, ==, NULL); + g_assert_false(merged->merge_lists); + + append_verify_list_02(&merged->desc[0]); + append_verify_list_01(&merged->desc[7], false); + + qemu_opts_free(merged); +} + + int main(int argc, char *argv[]) { register_opts(); @@ -XXX,XX +XXX,XX @@ int main(int argc, char *argv[]) g_test_add_func("/qemu-opts/opts_parse/bool", test_opts_parse_bool); g_test_add_func("/qemu-opts/opts_parse/number", test_opts_parse_number); g_test_add_func("/qemu-opts/opts_parse/size", test_opts_parse_size); + g_test_add_func("/qemu-opts/append_to_null", test_opts_append_to_null); + g_test_add_func("/qemu-opts/append", test_opts_append); g_test_run(); return 0; } -- 2.13.6 Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- tests/test-qemu-opts.c | 125 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) diff --git a/tests/test-qemu-opts.c b/tests/test-qemu-opts.c index XXXXXXX..XXXXXXX 100644 --- a/tests/test-qemu-opts.c +++ b/tests/test-qemu-opts.c @@ -XXX,XX +XXX,XX @@ #include "qemu/osdep.h" #include "qemu/cutils.h" #include "qemu/option.h" +#include "qemu/option_int.h" #include "qapi/error.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qstring.h" @@ -XXX,XX +XXX,XX @@ static void test_opts_append(void) qemu_opts_free(merged); } +static void test_opts_to_qdict_basic(void) +{ + QemuOpts *opts; + QDict *dict; + + opts = qemu_opts_parse(&opts_list_01, "str1=foo,str2=,str3=bar,number1=42", + false, &error_abort); + g_assert(opts != NULL); + + dict = qemu_opts_to_qdict(opts, NULL); + g_assert(dict != NULL); + + g_assert_cmpstr(qdict_get_str(dict, "str1"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(dict, "str2"), ==, ""); + g_assert_cmpstr(qdict_get_str(dict, "str3"), ==, "bar"); + g_assert_cmpstr(qdict_get_str(dict, "number1"), ==, "42"); + g_assert_false(qdict_haskey(dict, "number2")); + + QDECREF(dict); + qemu_opts_del(opts); +} + +static void test_opts_to_qdict_filtered(void) +{ + QemuOptsList *first, *merged; + QemuOpts *opts; + QDict *dict; + + first = qemu_opts_append(NULL, &opts_list_02); + merged = qemu_opts_append(first, &opts_list_01); + + opts = qemu_opts_parse(merged, + "str1=foo,str2=,str3=bar,bool1=off,number1=42", + false, &error_abort); + g_assert(opts != NULL); + + /* Convert to QDict without deleting from opts */ + dict = qemu_opts_to_qdict_filtered(opts, NULL, &opts_list_01, false); + g_assert(dict != NULL); + g_assert_cmpstr(qdict_get_str(dict, "str1"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(dict, "str2"), ==, ""); + g_assert_cmpstr(qdict_get_str(dict, "str3"), ==, "bar"); + g_assert_cmpstr(qdict_get_str(dict, "number1"), ==, "42"); + g_assert_false(qdict_haskey(dict, "number2")); + g_assert_false(qdict_haskey(dict, "bool1")); + QDECREF(dict); + + dict = qemu_opts_to_qdict_filtered(opts, NULL, &opts_list_02, false); + g_assert(dict != NULL); + g_assert_cmpstr(qdict_get_str(dict, "str1"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(dict, "str2"), ==, ""); + g_assert_cmpstr(qdict_get_str(dict, "bool1"), ==, "off"); + g_assert_false(qdict_haskey(dict, "str3")); + g_assert_false(qdict_haskey(dict, "number1")); + g_assert_false(qdict_haskey(dict, "number2")); + QDECREF(dict); + + /* Now delete converted options from opts */ + dict = qemu_opts_to_qdict_filtered(opts, NULL, &opts_list_01, true); + g_assert(dict != NULL); + g_assert_cmpstr(qdict_get_str(dict, "str1"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(dict, "str2"), ==, ""); + g_assert_cmpstr(qdict_get_str(dict, "str3"), ==, "bar"); + g_assert_cmpstr(qdict_get_str(dict, "number1"), ==, "42"); + g_assert_false(qdict_haskey(dict, "number2")); + g_assert_false(qdict_haskey(dict, "bool1")); + QDECREF(dict); + + dict = qemu_opts_to_qdict_filtered(opts, NULL, &opts_list_02, true); + g_assert(dict != NULL); + g_assert_cmpstr(qdict_get_str(dict, "bool1"), ==, "off"); + g_assert_false(qdict_haskey(dict, "str1")); + g_assert_false(qdict_haskey(dict, "str2")); + g_assert_false(qdict_haskey(dict, "str3")); + g_assert_false(qdict_haskey(dict, "number1")); + g_assert_false(qdict_haskey(dict, "number2")); + QDECREF(dict); + + g_assert_true(QTAILQ_EMPTY(&opts->head)); + + qemu_opts_del(opts); + qemu_opts_free(merged); +} + +static void test_opts_to_qdict_duplicates(void) +{ + QemuOpts *opts; + QemuOpt *opt; + QDict *dict; + + opts = qemu_opts_parse(&opts_list_03, "foo=a,foo=b", false, &error_abort); + g_assert(opts != NULL); + + /* Verify that opts has two options with the same name */ + opt = QTAILQ_FIRST(&opts->head); + g_assert_cmpstr(opt->name, ==, "foo"); + g_assert_cmpstr(opt->str , ==, "a"); + + opt = QTAILQ_NEXT(opt, next); + g_assert_cmpstr(opt->name, ==, "foo"); + g_assert_cmpstr(opt->str , ==, "b"); + + opt = QTAILQ_NEXT(opt, next); + g_assert(opt == NULL); + + /* In the conversion to QDict, the last one wins */ + dict = qemu_opts_to_qdict(opts, NULL); + g_assert(dict != NULL); + g_assert_cmpstr(qdict_get_str(dict, "foo"), ==, "b"); + QDECREF(dict); + + /* The last one still wins if entries are deleted, and both are deleted */ + dict = qemu_opts_to_qdict_filtered(opts, NULL, NULL, true); + g_assert(dict != NULL); + g_assert_cmpstr(qdict_get_str(dict, "foo"), ==, "b"); + QDECREF(dict); + + g_assert_true(QTAILQ_EMPTY(&opts->head)); + + qemu_opts_del(opts); +} int main(int argc, char *argv[]) { @@ -XXX,XX +XXX,XX @@ int main(int argc, char *argv[]) g_test_add_func("/qemu-opts/opts_parse/size", test_opts_parse_size); g_test_add_func("/qemu-opts/append_to_null", test_opts_append_to_null); g_test_add_func("/qemu-opts/append", test_opts_append); + g_test_add_func("/qemu-opts/to_qdict/basic", test_opts_to_qdict_basic); + g_test_add_func("/qemu-opts/to_qdict/filtered", test_opts_to_qdict_filtered); + g_test_add_func("/qemu-opts/to_qdict/duplicates", test_opts_to_qdict_duplicates); g_test_run(); return 0; } -- 2.13.6 A few block drivers will need to rename .bdrv_create options for their QAPIfication, so let's have a helper function for that. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- include/qapi/qmp/qdict.h | 6 +++ qobject/qdict.c | 34 +++++++++++++ tests/check-qdict.c | 129 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 169 insertions(+) diff --git a/include/qapi/qmp/qdict.h b/include/qapi/qmp/qdict.h index XXXXXXX..XXXXXXX 100644 --- a/include/qapi/qmp/qdict.h +++ b/include/qapi/qmp/qdict.h @@ -XXX,XX +XXX,XX @@ QObject *qdict_crumple(const QDict *src, Error **errp); void qdict_join(QDict *dest, QDict *src, bool overwrite); +typedef struct QDictRenames { + const char *from; + const char *to; +} QDictRenames; +bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp); + #endif /* QDICT_H */ diff --git a/qobject/qdict.c b/qobject/qdict.c index XXXXXXX..XXXXXXX 100644 --- a/qobject/qdict.c +++ b/qobject/qdict.c @@ -XXX,XX +XXX,XX @@ void qdict_join(QDict *dest, QDict *src, bool overwrite) entry = next; } } + +/** + * qdict_rename_keys(): Rename keys in qdict according to the replacements + * specified in the array renames. The array must be terminated by an entry + * with from = NULL. + * + * The renames are performed individually in the order of the array, so entries + * may be renamed multiple times and may or may not conflict depending on the + * order of the renames array. + * + * Returns true for success, false in error cases. + */ +bool qdict_rename_keys(QDict *qdict, const QDictRenames *renames, Error **errp) +{ + QObject *qobj; + + while (renames->from) { + if (qdict_haskey(qdict, renames->from)) { + if (qdict_haskey(qdict, renames->to)) { + error_setg(errp, "'%s' and its alias '%s' can't be used at the " + "same time", renames->to, renames->from); + return false; + } + + qobj = qdict_get(qdict, renames->from); + qobject_incref(qobj); + qdict_put_obj(qdict, renames->to, qobj); + qdict_del(qdict, renames->from); + } + + renames++; + } + return true; +} diff --git a/tests/check-qdict.c b/tests/check-qdict.c index XXXXXXX..XXXXXXX 100644 --- a/tests/check-qdict.c +++ b/tests/check-qdict.c @@ -XXX,XX +XXX,XX @@ static void qdict_crumple_test_empty(void) QDECREF(dst); } +static int qdict_count_entries(QDict *dict) +{ + const QDictEntry *e; + int count = 0; + + for (e = qdict_first(dict); e; e = qdict_next(dict, e)) { + count++; + } + + return count; +} + +static void qdict_rename_keys_test(void) +{ + QDict *dict = qdict_new(); + QDict *copy; + QDictRenames *renames; + Error *local_err = NULL; + + qdict_put_str(dict, "abc", "foo"); + qdict_put_str(dict, "abcdef", "bar"); + qdict_put_int(dict, "number", 42); + qdict_put_bool(dict, "flag", true); + qdict_put_null(dict, "nothing"); + + /* Empty rename list */ + renames = (QDictRenames[]) { + { NULL, "this can be anything" } + }; + copy = qdict_clone_shallow(dict); + qdict_rename_keys(copy, renames, &error_abort); + + g_assert_cmpstr(qdict_get_str(copy, "abc"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(copy, "abcdef"), ==, "bar"); + g_assert_cmpint(qdict_get_int(copy, "number"), ==, 42); + g_assert_cmpint(qdict_get_bool(copy, "flag"), ==, true); + g_assert(qobject_type(qdict_get(copy, "nothing")) == QTYPE_QNULL); + g_assert_cmpint(qdict_count_entries(copy), ==, 5); + + QDECREF(copy); + + /* Simple rename of all entries */ + renames = (QDictRenames[]) { + { "abc", "str1" }, + { "abcdef", "str2" }, + { "number", "int" }, + { "flag", "bool" }, + { "nothing", "null" }, + { NULL , NULL } + }; + copy = qdict_clone_shallow(dict); + qdict_rename_keys(copy, renames, &error_abort); + + g_assert(!qdict_haskey(copy, "abc")); + g_assert(!qdict_haskey(copy, "abcdef")); + g_assert(!qdict_haskey(copy, "number")); + g_assert(!qdict_haskey(copy, "flag")); + g_assert(!qdict_haskey(copy, "nothing")); + + g_assert_cmpstr(qdict_get_str(copy, "str1"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(copy, "str2"), ==, "bar"); + g_assert_cmpint(qdict_get_int(copy, "int"), ==, 42); + g_assert_cmpint(qdict_get_bool(copy, "bool"), ==, true); + g_assert(qobject_type(qdict_get(copy, "null")) == QTYPE_QNULL); + g_assert_cmpint(qdict_count_entries(copy), ==, 5); + + QDECREF(copy); + + /* Renames are processed top to bottom */ + renames = (QDictRenames[]) { + { "abc", "tmp" }, + { "abcdef", "abc" }, + { "number", "abcdef" }, + { "flag", "number" }, + { "nothing", "flag" }, + { "tmp", "nothing" }, + { NULL , NULL } + }; + copy = qdict_clone_shallow(dict); + qdict_rename_keys(copy, renames, &error_abort); + + g_assert_cmpstr(qdict_get_str(copy, "nothing"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(copy, "abc"), ==, "bar"); + g_assert_cmpint(qdict_get_int(copy, "abcdef"), ==, 42); + g_assert_cmpint(qdict_get_bool(copy, "number"), ==, true); + g_assert(qobject_type(qdict_get(copy, "flag")) == QTYPE_QNULL); + g_assert(!qdict_haskey(copy, "tmp")); + g_assert_cmpint(qdict_count_entries(copy), ==, 5); + + QDECREF(copy); + + /* Conflicting rename */ + renames = (QDictRenames[]) { + { "abcdef", "abc" }, + { NULL , NULL } + }; + copy = qdict_clone_shallow(dict); + qdict_rename_keys(copy, renames, &local_err); + + g_assert(local_err != NULL); + error_free(local_err); + local_err = NULL; + + g_assert_cmpstr(qdict_get_str(copy, "abc"), ==, "foo"); + g_assert_cmpstr(qdict_get_str(copy, "abcdef"), ==, "bar"); + g_assert_cmpint(qdict_get_int(copy, "number"), ==, 42); + g_assert_cmpint(qdict_get_bool(copy, "flag"), ==, true); + g_assert(qobject_type(qdict_get(copy, "nothing")) == QTYPE_QNULL); + g_assert_cmpint(qdict_count_entries(copy), ==, 5); + + QDECREF(copy); + + /* Renames in an empty dict */ + renames = (QDictRenames[]) { + { "abcdef", "abc" }, + { NULL , NULL } + }; + + QDECREF(dict); + dict = qdict_new(); + + qdict_rename_keys(dict, renames, &error_abort); + g_assert(qdict_first(dict) == NULL); + + QDECREF(dict); +} + static void qdict_crumple_test_bad_inputs(void) { QDict *src; @@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv) g_test_add_func("/public/crumple/bad_inputs", qdict_crumple_test_bad_inputs); + g_test_add_func("/public/rename_keys", qdict_rename_keys_test); + /* The Big one */ if (g_test_slow()) { g_test_add_func("/stress/test", qdict_stress_test); -- 2.13.6 Instead of manually creating the BlockdevCreateOptions object, use a visitor to parse the given options into the QAPI object. This involves translation from the old command line syntax to the syntax mandated by the QAPI schema. Option names are still checked against qcow2_create_opts, so only the old option names are allowed on the command line, even if they are translated in qcow2_create(). In contrast, new option values are optionally recognised besides the old values: 'compat' accepts 'v2'/'v3' as an alias for '0.10'/'1.1', and 'encrypt.format' accepts 'qcow' as an alias for 'aes' now. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- block/qcow2.c | 218 ++++++++++++++++----------------------------- tests/qemu-iotests/049.out | 8 +- tests/qemu-iotests/112.out | 4 +- 3 files changed, 84 insertions(+), 146 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ #include "qemu/option_int.h" #include "qemu/cutils.h" #include "qemu/bswap.h" -#include "qapi/opts-visitor.h" +#include "qapi/qobject-input-visitor.h" +#include "qapi/qapi-visit-block-core.h" #include "block/crypto.h" /* @@ -XXX,XX +XXX,XX @@ static int qcow2_crypt_method_from_format(const char *encryptfmt) } } -static QCryptoBlockCreateOptions * -qcow2_parse_encryption(const char *encryptfmt, QemuOpts *opts, Error **errp) -{ - QCryptoBlockCreateOptions *cryptoopts = NULL; - QDict *options, *encryptopts; - int fmt; - - options = qemu_opts_to_qdict(opts, NULL); - qdict_extract_subqdict(options, &encryptopts, "encrypt."); - QDECREF(options); - - fmt = qcow2_crypt_method_from_format(encryptfmt); - - switch (fmt) { - case QCOW_CRYPT_LUKS: - cryptoopts = block_crypto_create_opts_init( - Q_CRYPTO_BLOCK_FORMAT_LUKS, encryptopts, errp); - break; - case QCOW_CRYPT_AES: - cryptoopts = block_crypto_create_opts_init( - Q_CRYPTO_BLOCK_FORMAT_QCOW, encryptopts, errp); - break; - default: - error_setg(errp, "Unknown encryption format '%s'", encryptfmt); - break; - } - - QDECREF(encryptopts); - return cryptoopts; -} - static int qcow2_set_up_encryption(BlockDriverState *bs, QCryptoBlockCreateOptions *cryptoopts, Error **errp) @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } if (version < 3 && qcow2_opts->lazy_refcounts) { error_setg(errp, "Lazy refcounts only supported with compatibility " - "level 1.1 and above (use compat=1.1 or greater)"); + "level 1.1 and above (use version=v3 or greater)"); ret = -EINVAL; goto out; } @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } if (version < 3 && qcow2_opts->refcount_bits != 16) { error_setg(errp, "Different refcount widths than 16 bits require " - "compatibility level 1.1 or above (use compat=1.1 or " + "compatibility level 1.1 or above (use version=v3 or " "greater)"); ret = -EINVAL; goto out; @@ -XXX,XX +XXX,XX @@ out: static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opts, Error **errp) { - BlockdevCreateOptions create_options; - char *backing_file = NULL; - char *backing_fmt = NULL; - BlockdevDriver backing_drv; - char *buf = NULL; - uint64_t size = 0; - int flags = 0; - size_t cluster_size = DEFAULT_CLUSTER_SIZE; - PreallocMode prealloc; - int version; - uint64_t refcount_bits; - char *encryptfmt = NULL; - QCryptoBlockCreateOptions *cryptoopts = NULL; + BlockdevCreateOptions *create_options = NULL; + QDict *qdict = NULL; + QObject *qobj; + Visitor *v; BlockDriverState *bs = NULL; Error *local_err = NULL; + const char *val; int ret; - /* Read out options */ - size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), - BDRV_SECTOR_SIZE); - backing_file = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE); - backing_fmt = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FMT); - backing_drv = qapi_enum_parse(&BlockdevDriver_lookup, backing_fmt, - 0, &local_err); - if (local_err) { - error_propagate(errp, local_err); + /* Only the keyval visitor supports the dotted syntax needed for + * encryption, so go through a QDict before getting a QAPI type. Ignore + * options meant for the protocol layer so that the visitor doesn't + * complain. */ + qdict = qemu_opts_to_qdict_filtered(opts, NULL, bdrv_qcow2.create_opts, + true); + + /* Handle encryption options */ + val = qdict_get_try_str(qdict, BLOCK_OPT_ENCRYPT); + if (val && !strcmp(val, "on")) { + qdict_put_str(qdict, BLOCK_OPT_ENCRYPT, "qcow"); + } else if (val && !strcmp(val, "off")) { + qdict_del(qdict, BLOCK_OPT_ENCRYPT); + } + + val = qdict_get_try_str(qdict, BLOCK_OPT_ENCRYPT_FORMAT); + if (val && !strcmp(val, "aes")) { + qdict_put_str(qdict, BLOCK_OPT_ENCRYPT_FORMAT, "qcow"); + } + + /* Convert compat=0.10/1.1 into compat=v2/v3, to be renamed into + * version=v2/v3 below. */ + val = qdict_get_try_str(qdict, BLOCK_OPT_COMPAT_LEVEL); + if (val && !strcmp(val, "0.10")) { + qdict_put_str(qdict, BLOCK_OPT_COMPAT_LEVEL, "v2"); + } else if (val && !strcmp(val, "1.1")) { + qdict_put_str(qdict, BLOCK_OPT_COMPAT_LEVEL, "v3"); + } + + /* Change legacy command line options into QMP ones */ + static const QDictRenames opt_renames[] = { + { BLOCK_OPT_BACKING_FILE, "backing-file" }, + { BLOCK_OPT_BACKING_FMT, "backing-fmt" }, + { BLOCK_OPT_CLUSTER_SIZE, "cluster-size" }, + { BLOCK_OPT_LAZY_REFCOUNTS, "lazy-refcounts" }, + { BLOCK_OPT_REFCOUNT_BITS, "refcount-bits" }, + { BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT }, + { BLOCK_OPT_COMPAT_LEVEL, "version" }, + { NULL, NULL }, + }; + + if (!qdict_rename_keys(qdict, opt_renames, errp)) { ret = -EINVAL; goto finish; } - encryptfmt = qemu_opt_get_del(opts, BLOCK_OPT_ENCRYPT_FORMAT); - if (encryptfmt) { - if (qemu_opt_get(opts, BLOCK_OPT_ENCRYPT)) { - error_setg(errp, "Options " BLOCK_OPT_ENCRYPT " and " - BLOCK_OPT_ENCRYPT_FORMAT " are mutually exclusive"); - ret = -EINVAL; - goto finish; - } - } else if (qemu_opt_get_bool_del(opts, BLOCK_OPT_ENCRYPT, false)) { - encryptfmt = g_strdup("aes"); - } - if (encryptfmt) { - cryptoopts = qcow2_parse_encryption(encryptfmt, opts, errp); - if (cryptoopts == NULL) { - ret = -EINVAL; - goto finish; - } - } - - cluster_size = qcow2_opt_get_cluster_size_del(opts, &local_err); - if (local_err) { - error_propagate(errp, local_err); - ret = -EINVAL; + /* Create and open the file (protocol layer) */ + ret = bdrv_create_file(filename, opts, errp); + if (ret < 0) { goto finish; } - buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC); - prealloc = qapi_enum_parse(&PreallocMode_lookup, buf, - PREALLOC_MODE_OFF, &local_err); - if (local_err) { - error_propagate(errp, local_err); - ret = -EINVAL; + + bs = bdrv_open(filename, NULL, NULL, + BDRV_O_RDWR | BDRV_O_RESIZE | BDRV_O_PROTOCOL, errp); + if (bs == NULL) { + ret = -EIO; goto finish; } - version = qcow2_opt_get_version_del(opts, &local_err); - if (local_err) { - error_propagate(errp, local_err); + /* Set 'driver' and 'node' options */ + qdict_put_str(qdict, "driver", "qcow2"); + qdict_put_str(qdict, "file", bs->node_name); + + /* Now get the QAPI type BlockdevCreateOptions */ + qobj = qdict_crumple(qdict, errp); + QDECREF(qdict); + qdict = qobject_to_qdict(qobj); + if (qdict == NULL) { ret = -EINVAL; goto finish; } - if (qemu_opt_get_bool_del(opts, BLOCK_OPT_LAZY_REFCOUNTS, false)) { - flags |= BLOCK_FLAG_LAZY_REFCOUNTS; - } + v = qobject_input_visitor_new_keyval(QOBJECT(qdict)); + visit_type_BlockdevCreateOptions(v, NULL, &create_options, &local_err); + visit_free(v); - refcount_bits = qcow2_opt_get_refcount_bits_del(opts, version, &local_err); if (local_err) { error_propagate(errp, local_err); ret = -EINVAL; goto finish; } - - /* Create and open the file (protocol layer) */ - ret = bdrv_create_file(filename, opts, errp); - if (ret < 0) { - goto finish; - } - - bs = bdrv_open(filename, NULL, NULL, - BDRV_O_RDWR | BDRV_O_RESIZE | BDRV_O_PROTOCOL, errp); - if (bs == NULL) { - ret = -EIO; - goto finish; - } + /* Silently round up size */ + create_options->u.qcow2.size = ROUND_UP(create_options->u.qcow2.size, + BDRV_SECTOR_SIZE); /* Create the qcow2 image (format layer) */ - create_options = (BlockdevCreateOptions) { - .driver = BLOCKDEV_DRIVER_QCOW2, - .u.qcow2 = { - .file = &(BlockdevRef) { - .type = QTYPE_QSTRING, - .u.reference = bs->node_name, - }, - .size = size, - .has_version = true, - .version = version == 2 - ? BLOCKDEV_QCOW2_VERSION_V2 - : BLOCKDEV_QCOW2_VERSION_V3, - .has_backing_file = (backing_file != NULL), - .backing_file = backing_file, - .has_backing_fmt = (backing_fmt != NULL), - .backing_fmt = backing_drv, - .has_encrypt = (encryptfmt != NULL), - .encrypt = cryptoopts, - .has_cluster_size = true, - .cluster_size = cluster_size, - .has_preallocation = true, - .preallocation = prealloc, - .has_lazy_refcounts = true, - .lazy_refcounts = (flags & BLOCK_FLAG_LAZY_REFCOUNTS), - .has_refcount_bits = true, - .refcount_bits = refcount_bits, - }, - }; - ret = qcow2_co_create(&create_options, errp); + ret = qcow2_co_create(create_options, errp); if (ret < 0) { goto finish; } + ret = 0; finish: + QDECREF(qdict); bdrv_unref(bs); - - qapi_free_QCryptoBlockCreateOptions(cryptoopts); - g_free(backing_file); - g_free(backing_fmt); - g_free(encryptfmt); - g_free(buf); + qapi_free_BlockdevCreateOptions(create_options); return ret; } diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/049.out +++ b/tests/qemu-iotests/049.out @@ -XXX,XX +XXX,XX @@ qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M -qemu-img: TEST_DIR/t.qcow2: Invalid compatibility level: '0.42' +qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42' Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M -qemu-img: TEST_DIR/t.qcow2: Invalid compatibility level: 'foobar' +qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar' Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 == Check preallocation option == @@ -XXX,XX +XXX,XX @@ qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M -qemu-img: TEST_DIR/t.qcow2: invalid parameter value: 1234 +qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234' Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 == Check encryption option == @@ -XXX,XX +XXX,XX @@ qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M -qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use compat=1.1 or greater) +qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater) Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 *** done diff --git a/tests/qemu-iotests/112.out b/tests/qemu-iotests/112.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/112.out +++ b/tests/qemu-iotests/112.out @@ -XXX,XX +XXX,XX @@ refcount bits: 16 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 refcount bits: 16 -qemu-img: TEST_DIR/t.IMGFMT: Different refcount widths than 16 bits require compatibility level 1.1 or above (use or greater) +qemu-img: TEST_DIR/t.IMGFMT: Different refcount widths than 16 bits require compatibility level 1.1 or above (use version=v3 or greater) Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 -qemu-img: TEST_DIR/t.IMGFMT: Different refcount widths than 16 bits require compatibility level 1.1 or above (use or greater) +qemu-img: TEST_DIR/t.IMGFMT: Different refcount widths than 16 bits require compatibility level 1.1 or above (use version=v3 or greater) Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 === Snapshot limit on refcount_bits=1 === -- 2.13.6 We'll use a separate source file for image creation, and we need to check there whether the requested driver is whitelisted. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- include/block/block.h | 1 + block.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/include/block/block.h b/include/block/block.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -XXX,XX +XXX,XX @@ char *bdrv_perm_names(uint64_t perm); void bdrv_init(void); void bdrv_init_with_whitelist(void); bool bdrv_uses_whitelist(void); +int bdrv_is_whitelisted(BlockDriver *drv, bool read_only); BlockDriver *bdrv_find_protocol(const char *filename, bool allow_protocol_prefix, Error **errp); diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ BlockDriver *bdrv_find_format(const char *format_name) return bdrv_do_find_format(format_name); } -static int bdrv_is_whitelisted(BlockDriver *drv, bool read_only) +int bdrv_is_whitelisted(BlockDriver *drv, bool read_only) { static const char *whitelist_rw[] = { CONFIG_BDRV_RW_WHITELIST -- 2.13.6 This adds a synchronous x-blockdev-create QMP command that can create qcow2 images on a given node name. We don't want to block while creating an image, so this is not the final interface in all aspects, but BlockdevCreateOptionsQcow2 and .bdrv_co_create() are what they actually might look like in the end. In any case, this should be good enough to test whether we interpret BlockdevCreateOptions as we should. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- qapi/block-core.json | 12 ++++++++ include/block/block_int.h | 5 +++- block/create.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 1 + block/Makefile.objs | 2 +- 5 files changed, 94 insertions(+), 2 deletions(-) create mode 100644 block/create.c diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ } } ## +# @x-blockdev-create: +# +# Create an image format on a given node. +# TODO Replace with something asynchronous (block job?) +# +# Since: 2.12 +## +{ 'command': 'x-blockdev-create', + 'data': 'BlockdevCreateOptions', + 'boxed': true } + +## # @blockdev-open-tray: # # Opens a block device's tray. If there is a block driver state tree inserted as diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ struct BlockDriver { int (*bdrv_file_open)(BlockDriverState *bs, QDict *options, int flags, Error **errp); void (*bdrv_close)(BlockDriverState *bs); - int coroutine_fn (*bdrv_co_create_opts)(const char *filename, QemuOpts *opts, + int coroutine_fn (*bdrv_co_create)(BlockdevCreateOptions *opts, Error **errp); + int coroutine_fn (*bdrv_co_create_opts)(const char *filename, + QemuOpts *opts, + Error **errp); int (*bdrv_make_empty)(BlockDriverState *bs); void (*bdrv_refresh_filename)(BlockDriverState *bs, QDict *options); diff --git a/block/create.c b/block/create.c new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/block/create.c @@ -XXX,XX +XXX,XX @@ +/* + * Block layer code related to image creation + * + * Copyright (c) 2018 Kevin Wolf <kwolf@redhat.com> + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + * THE SOFTWARE. + */ + +#include "qemu/osdep.h" +#include "block/block_int.h" +#include "qapi/qapi-commands-block-core.h" +#include "qapi/error.h" + +typedef struct BlockdevCreateCo { + BlockDriver *drv; + BlockdevCreateOptions *opts; + int ret; + Error **errp; +} BlockdevCreateCo; + +static void coroutine_fn bdrv_co_create_co_entry(void *opaque) +{ + BlockdevCreateCo *cco = opaque; + cco->ret = cco->drv->bdrv_co_create(cco->opts, cco->errp); +} + +void qmp_x_blockdev_create(BlockdevCreateOptions *options, Error **errp) +{ + const char *fmt = BlockdevDriver_str(options->driver); + BlockDriver *drv = bdrv_find_format(fmt); + Coroutine *co; + BlockdevCreateCo cco; + + /* If the driver is in the schema, we know that it exists. But it may not + * be whitelisted. */ + assert(drv); + if (bdrv_uses_whitelist() && !bdrv_is_whitelisted(drv, false)) { + error_setg(errp, "Driver is not whitelisted"); + return; + } + + /* Call callback if it exists */ + if (!drv->bdrv_co_create) { + error_setg(errp, "Driver does not support blockdev-create"); + return; + } + + cco = (BlockdevCreateCo) { + .drv = drv, + .opts = options, + .ret = -EINPROGRESS, + .errp = errp, + }; + + co = qemu_coroutine_create(bdrv_co_create_co_entry, &cco); + qemu_coroutine_enter(co); + while (cco.ret == -EINPROGRESS) { + aio_poll(qemu_get_aio_context(), true); + } +} diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ BlockDriver bdrv_qcow2 = { .bdrv_join_options = qcow2_join_options, .bdrv_child_perm = bdrv_format_default_perms, .bdrv_co_create_opts = qcow2_co_create_opts, + .bdrv_co_create = qcow2_co_create, .bdrv_has_zero_init = bdrv_has_zero_init_1, .bdrv_co_block_status = qcow2_co_block_status, diff --git a/block/Makefile.objs b/block/Makefile.objs index XXXXXXX..XXXXXXX 100644 --- a/block/Makefile.objs +++ b/block/Makefile.objs @@ -XXX,XX +XXX,XX @@ block-obj-y += block-backend.o snapshot.o qapi.o block-obj-$(CONFIG_WIN32) += file-win32.o win32-aio.o block-obj-$(CONFIG_POSIX) += file-posix.o block-obj-$(CONFIG_LINUX_AIO) += linux-aio.o -block-obj-y += null.o mirror.o commit.o io.o +block-obj-y += null.o mirror.o commit.o io.o create.o block-obj-y += throttle-groups.o block-obj-$(CONFIG_LINUX) += nvme.o -- 2.13.6 This adds the .bdrv_co_create driver callback to file, which enables image creation over QMP. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- qapi/block-core.json | 20 ++++++++++++- block/file-posix.c | 79 +++++++++++++++++++++++++++++++++++++--------------- 2 files changed, 75 insertions(+), 24 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ { 'command': 'blockdev-del', 'data': { 'node-name': 'str' } } ## +# @BlockdevCreateOptionsFile: +# +# Driver specific image creation options for file. +# +# @filename Filename for the new image file +# @size Size of the virtual disk in bytes +# @preallocation Preallocation mode for the new image (default: off) +# @nocow Turn off copy-on-write (valid only on btrfs; default: off) +# +# Since: 2.12 +## +{ 'struct': 'BlockdevCreateOptionsFile', + 'data': { 'filename': 'str', + 'size': 'size', + '*preallocation': 'PreallocMode', + '*nocow': 'bool' } } + +## # @BlockdevQcow2Version: # # @v2: The original QCOW2 format as introduced in qemu 0.10 (version 2) @@ -XXX,XX +XXX,XX @@ 'bochs': 'BlockdevCreateNotSupported', 'cloop': 'BlockdevCreateNotSupported', 'dmg': 'BlockdevCreateNotSupported', - 'file': 'BlockdevCreateNotSupported', + 'file': 'BlockdevCreateOptionsFile', 'ftp': 'BlockdevCreateNotSupported', 'ftps': 'BlockdevCreateNotSupported', 'gluster': 'BlockdevCreateNotSupported', diff --git a/block/file-posix.c b/block/file-posix.c index XXXXXXX..XXXXXXX 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -XXX,XX +XXX,XX @@ static int64_t raw_get_allocated_file_size(BlockDriverState *bs) return (int64_t)st.st_blocks * 512; } -static int coroutine_fn raw_co_create_opts(const char *filename, QemuOpts *opts, - Error **errp) +static int raw_co_create(BlockdevCreateOptions *options, Error **errp) { + BlockdevCreateOptionsFile *file_opts; int fd; int result = 0; - int64_t total_size = 0; - bool nocow = false; - PreallocMode prealloc; - char *buf = NULL; - Error *local_err = NULL; - strstart(filename, "file:", &filename); + /* Validate options and set default values */ + assert(options->driver == BLOCKDEV_DRIVER_FILE); + file_opts = &options->u.file; - /* Read out options */ - total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), - BDRV_SECTOR_SIZE); - nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false); - buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC); - prealloc = qapi_enum_parse(&PreallocMode_lookup, buf, - PREALLOC_MODE_OFF, &local_err); - g_free(buf); - if (local_err) { - error_propagate(errp, local_err); - result = -EINVAL; - goto out; + if (!file_opts->has_nocow) { + file_opts->nocow = false; + } + if (!file_opts->has_preallocation) { + file_opts->preallocation = PREALLOC_MODE_OFF; } - fd = qemu_open(filename, O_RDWR | O_CREAT | O_TRUNC | O_BINARY, + /* Create file */ + fd = qemu_open(file_opts->filename, O_RDWR | O_CREAT | O_TRUNC | O_BINARY, 0644); if (fd < 0) { result = -errno; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn raw_co_create_opts(const char *filename, QemuOpts *opts, goto out; } - if (nocow) { + if (file_opts->nocow) { #ifdef __linux__ /* Set NOCOW flag to solve performance issue on fs like btrfs. * This is an optimisation. The FS_IOC_SETFLAGS ioctl return value @@ -XXX,XX +XXX,XX @@ static int coroutine_fn raw_co_create_opts(const char *filename, QemuOpts *opts, #endif } - result = raw_regular_truncate(fd, total_size, prealloc, errp); + result = raw_regular_truncate(fd, file_opts->size, file_opts->preallocation, + errp); if (result < 0) { goto out_close; } @@ -XXX,XX +XXX,XX @@ out: return result; } +static int coroutine_fn raw_co_create_opts(const char *filename, QemuOpts *opts, + Error **errp) +{ + BlockdevCreateOptions options; + int64_t total_size = 0; + bool nocow = false; + PreallocMode prealloc; + char *buf = NULL; + Error *local_err = NULL; + + /* Skip file: protocol prefix */ + strstart(filename, "file:", &filename); + + /* Read out options */ + total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), + BDRV_SECTOR_SIZE); + nocow = qemu_opt_get_bool(opts, BLOCK_OPT_NOCOW, false); + buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC); + prealloc = qapi_enum_parse(&PreallocMode_lookup, buf, + PREALLOC_MODE_OFF, &local_err); + g_free(buf); + if (local_err) { + error_propagate(errp, local_err); + return -EINVAL; + } + + options = (BlockdevCreateOptions) { + .driver = BLOCKDEV_DRIVER_FILE, + .u.file = { + .filename = (char *) filename, + .size = total_size, + .has_preallocation = true, + .preallocation = prealloc, + .has_nocow = true, + .nocow = nocow, + }, + }; + return raw_co_create(&options, errp); +} + /* * Find allocation range in @bs around offset @start. * May change underlying file descriptor's file offset. @@ -XXX,XX +XXX,XX @@ BlockDriver bdrv_file = { .bdrv_reopen_commit = raw_reopen_commit, .bdrv_reopen_abort = raw_reopen_abort, .bdrv_close = raw_close, + .bdrv_co_create = raw_co_create, .bdrv_co_create_opts = raw_co_create_opts, .bdrv_has_zero_init = bdrv_has_zero_init_1, .bdrv_co_block_status = raw_co_block_status, -- 2.13.6 This adds the .bdrv_co_create driver callback to file-win32, which enables image creation over QMP. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- block/file-win32.c | 47 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 9 deletions(-) diff --git a/block/file-win32.c b/block/file-win32.c index XXXXXXX..XXXXXXX 100644 --- a/block/file-win32.c +++ b/block/file-win32.c @@ -XXX,XX +XXX,XX @@ static int64_t raw_get_allocated_file_size(BlockDriverState *bs) return st.st_size; } -static int coroutine_fn raw_co_create_opts(const char *filename, QemuOpts *opts, - Error **errp) +static int raw_co_create(BlockdevCreateOptions *options, Error **errp) { + BlockdevCreateOptionsFile *file_opts; int fd; - int64_t total_size = 0; - strstart(filename, "file:", &filename); + assert(options->driver == BLOCKDEV_DRIVER_FILE); + file_opts = &options->u.file; - /* Read out options */ - total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), - BDRV_SECTOR_SIZE); + if (file_opts->has_preallocation) { + error_setg(errp, "Preallocation is not supported on Windows"); + return -EINVAL; + } + if (file_opts->has_nocow) { + error_setg(errp, "nocow is not supported on Windows"); + return -EINVAL; + } - fd = qemu_open(filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, + fd = qemu_open(file_opts->filename, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, 0644); if (fd < 0) { error_setg_errno(errp, errno, "Could not create file"); return -EIO; } set_sparse(fd); - ftruncate(fd, total_size); + ftruncate(fd, file_opts->size); qemu_close(fd); + return 0; } +static int coroutine_fn raw_co_create_opts(const char *filename, QemuOpts *opts, + Error **errp) +{ + BlockdevCreateOptions options; + int64_t total_size = 0; + + strstart(filename, "file:", &filename); + + /* Read out options */ + total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), + BDRV_SECTOR_SIZE); + + options = (BlockdevCreateOptions) { + .driver = BLOCKDEV_DRIVER_FILE, + .u.file = { + .filename = (char *) filename, + .size = total_size, + .has_preallocation = false, + .has_nocow = false, + }, + }; + return raw_co_create(&options, errp); +} static QemuOptsList raw_create_opts = { .name = "raw-create-opts", -- 2.13.6 This adds the .bdrv_co_create driver callback to gluster, which enables image creation over QMP. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- qapi/block-core.json | 18 ++++++- block/gluster.c | 135 ++++++++++++++++++++++++++++++++++----------------- 2 files changed, 108 insertions(+), 45 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ '*nocow': 'bool' } } ## +# @BlockdevCreateOptionsGluster: +# +# Driver specific image creation options for gluster. +# +# @location Where to store the new image file +# @size Size of the virtual disk in bytes +# @preallocation Preallocation mode for the new image (default: off) +# +# Since: 2.12 +## +{ 'struct': 'BlockdevCreateOptionsGluster', + 'data': { 'location': 'BlockdevOptionsGluster', + 'size': 'size', + '*preallocation': 'PreallocMode' } } + +## # @BlockdevQcow2Version: # # @v2: The original QCOW2 format as introduced in qemu 0.10 (version 2) @@ -XXX,XX +XXX,XX @@ 'file': 'BlockdevCreateOptionsFile', 'ftp': 'BlockdevCreateNotSupported', 'ftps': 'BlockdevCreateNotSupported', - 'gluster': 'BlockdevCreateNotSupported', + 'gluster': 'BlockdevCreateOptionsGluster', 'host_cdrom': 'BlockdevCreateNotSupported', 'host_device': 'BlockdevCreateNotSupported', 'http': 'BlockdevCreateNotSupported', diff --git a/block/gluster.c b/block/gluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/gluster.c +++ b/block/gluster.c @@ -XXX,XX +XXX,XX @@ out: return -errno; } -static struct glfs *qemu_gluster_init(BlockdevOptionsGluster *gconf, - const char *filename, - QDict *options, Error **errp) +/* Converts options given in @filename and the @options QDict into the QAPI + * object @gconf. */ +static int qemu_gluster_parse(BlockdevOptionsGluster *gconf, + const char *filename, + QDict *options, Error **errp) { int ret; if (filename) { @@ -XXX,XX +XXX,XX @@ static struct glfs *qemu_gluster_init(BlockdevOptionsGluster *gconf, "[host[:port]]volume/path[?socket=...]" "[,file.debug=N]" "[,file.logfile=/path/filename.log]\n"); - errno = -ret; - return NULL; + return ret; } } else { ret = qemu_gluster_parse_json(gconf, options, errp); @@ -XXX,XX +XXX,XX @@ static struct glfs *qemu_gluster_init(BlockdevOptionsGluster *gconf, "file.server.1.transport=unix," "file.server.1.socket=/var/run/glusterd.socket ..." "\n"); - errno = -ret; - return NULL; + return ret; } + } + return 0; +} + +static struct glfs *qemu_gluster_init(BlockdevOptionsGluster *gconf, + const char *filename, + QDict *options, Error **errp) +{ + int ret; + + ret = qemu_gluster_parse(gconf, filename, options, errp); + if (ret < 0) { + errno = -ret; + return NULL; } return qemu_gluster_glfs_init(gconf, errp); @@ -XXX,XX +XXX,XX @@ static int qemu_gluster_do_truncate(struct glfs_fd *fd, int64_t offset, return 0; } +static int qemu_gluster_co_create(BlockdevCreateOptions *options, + Error **errp) +{ + BlockdevCreateOptionsGluster *opts = &options->u.gluster; + struct glfs *glfs; + struct glfs_fd *fd = NULL; + int ret = 0; + + assert(options->driver == BLOCKDEV_DRIVER_GLUSTER); + + glfs = qemu_gluster_glfs_init(opts->location, errp); + if (!glfs) { + ret = -errno; + goto out; + } + + fd = glfs_creat(glfs, opts->location->path, + O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, S_IRUSR | S_IWUSR); + if (!fd) { + ret = -errno; + goto out; + } + + ret = qemu_gluster_do_truncate(fd, opts->size, opts->preallocation, errp); + +out: + if (fd) { + if (glfs_close(fd) != 0 && ret == 0) { + ret = -errno; + } + } + glfs_clear_preopened(glfs); + return ret; +} + static int coroutine_fn qemu_gluster_co_create_opts(const char *filename, QemuOpts *opts, Error **errp) { + BlockdevCreateOptions *options; + BlockdevCreateOptionsGluster *gopts; BlockdevOptionsGluster *gconf; - struct glfs *glfs; - struct glfs_fd *fd = NULL; - int ret = 0; - PreallocMode prealloc; - int64_t total_size = 0; char *tmp = NULL; Error *local_err = NULL; + int ret; + + options = g_new0(BlockdevCreateOptions, 1); + options->driver = BLOCKDEV_DRIVER_GLUSTER; + gopts = &options->u.gluster; gconf = g_new0(BlockdevOptionsGluster, 1); + gopts->location = gconf; + + gopts->size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), + BDRV_SECTOR_SIZE); + + tmp = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC); + gopts->preallocation = qapi_enum_parse(&PreallocMode_lookup, tmp, + PREALLOC_MODE_OFF, &local_err); + g_free(tmp); + if (local_err) { + error_propagate(errp, local_err); + ret = -EINVAL; + goto fail; + } + gconf->debug = qemu_opt_get_number_del(opts, GLUSTER_OPT_DEBUG, GLUSTER_DEBUG_DEFAULT); if (gconf->debug < 0) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qemu_gluster_co_create_opts(const char *filename, } gconf->has_logfile = true; - glfs = qemu_gluster_init(gconf, filename, NULL, errp); - if (!glfs) { - ret = -errno; - goto out; - } - - total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), - BDRV_SECTOR_SIZE); - - tmp = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC); - prealloc = qapi_enum_parse(&PreallocMode_lookup, tmp, PREALLOC_MODE_OFF, - &local_err); - g_free(tmp); - if (local_err) { - error_propagate(errp, local_err); - ret = -EINVAL; - goto out; + ret = qemu_gluster_parse(gconf, filename, NULL, errp); + if (ret < 0) { + goto fail; } - fd = glfs_creat(glfs, gconf->path, - O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, S_IRUSR | S_IWUSR); - if (!fd) { - ret = -errno; - goto out; + ret = qemu_gluster_co_create(options, errp); + if (ret < 0) { + goto fail; } - ret = qemu_gluster_do_truncate(fd, total_size, prealloc, errp); - -out: - if (fd) { - if (glfs_close(fd) != 0 && ret == 0) { - ret = -errno; - } - } - qapi_free_BlockdevOptionsGluster(gconf); - glfs_clear_preopened(glfs); + ret = 0; +fail: + qapi_free_BlockdevCreateOptions(options); return ret; } @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_gluster = { .bdrv_reopen_commit = qemu_gluster_reopen_commit, .bdrv_reopen_abort = qemu_gluster_reopen_abort, .bdrv_close = qemu_gluster_close, + .bdrv_co_create = qemu_gluster_co_create, .bdrv_co_create_opts = qemu_gluster_co_create_opts, .bdrv_getlength = qemu_gluster_getlength, .bdrv_get_allocated_file_size = qemu_gluster_allocated_file_size, @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_gluster_tcp = { .bdrv_reopen_commit = qemu_gluster_reopen_commit, .bdrv_reopen_abort = qemu_gluster_reopen_abort, .bdrv_close = qemu_gluster_close, + .bdrv_co_create = qemu_gluster_co_create, .bdrv_co_create_opts = qemu_gluster_co_create_opts, .bdrv_getlength = qemu_gluster_getlength, .bdrv_get_allocated_file_size = qemu_gluster_allocated_file_size, @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_gluster_unix = { .bdrv_reopen_commit = qemu_gluster_reopen_commit, .bdrv_reopen_abort = qemu_gluster_reopen_abort, .bdrv_close = qemu_gluster_close, + .bdrv_co_create = qemu_gluster_co_create, .bdrv_co_create_opts = qemu_gluster_co_create_opts, .bdrv_getlength = qemu_gluster_getlength, .bdrv_get_allocated_file_size = qemu_gluster_allocated_file_size, @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_gluster_rdma = { .bdrv_reopen_commit = qemu_gluster_reopen_commit, .bdrv_reopen_abort = qemu_gluster_reopen_abort, .bdrv_close = qemu_gluster_close, + .bdrv_co_create = qemu_gluster_co_create, .bdrv_co_create_opts = qemu_gluster_co_create_opts, .bdrv_getlength = qemu_gluster_getlength, .bdrv_get_allocated_file_size = qemu_gluster_allocated_file_size, -- 2.13.6 If we want to include the invalid option name in the error message, we can't free the string earlier than that. Cc: qemu-stable@nongnu.org Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- block/rbd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/block/rbd.c b/block/rbd.c index XXXXXXX..XXXXXXX 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_set_keypairs(rados_t cluster, const char *keypairs_json, key = qstring_get_str(name); ret = rados_conf_set(cluster, key, qstring_get_str(value)); - QDECREF(name); QDECREF(value); if (ret < 0) { error_setg_errno(errp, -ret, "invalid conf option %s", key); + QDECREF(name); ret = -EINVAL; break; } + QDECREF(name); } QDECREF(keypairs); -- 2.13.6 The code to establish an RBD connection is duplicated between open and create. In order to be able to share the code, factor out the code from qemu_rbd_open() as a first step. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/rbd.c | 100 ++++++++++++++++++++++++++++++++++++------------------------ 1 file changed, 60 insertions(+), 40 deletions(-) diff --git a/block/rbd.c b/block/rbd.c index XXXXXXX..XXXXXXX 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -XXX,XX +XXX,XX @@ out: return rados_str; } -static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, - Error **errp) +static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx, + char **s_snap, char **s_image_name, + QDict *options, bool cache, Error **errp) { - BDRVRBDState *s = bs->opaque; - const char *pool, *snap, *conf, *user, *image_name, *keypairs; - const char *secretid, *filename; QemuOpts *opts; - Error *local_err = NULL; char *mon_host = NULL; + const char *pool, *snap, *conf, *user, *image_name, *keypairs; + const char *secretid; + Error *local_err = NULL; int r; - /* If we are given a filename, parse the filename, with precedence given to - * filename encoded options */ - filename = qdict_get_try_str(options, "filename"); - if (filename) { - warn_report("'filename' option specified. " - "This is an unsupported option, and may be deprecated " - "in the future"); - qemu_rbd_parse_filename(filename, options, &local_err); - if (local_err) { - r = -EINVAL; - error_propagate(errp, local_err); - goto exit; - } - } - opts = qemu_opts_create(&runtime_opts, NULL, 0, &error_abort); qemu_opts_absorb_qdict(opts, options, &local_err); if (local_err) { @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, goto failed_opts; } - r = rados_create(&s->cluster, user); + r = rados_create(cluster, user); if (r < 0) { error_setg_errno(errp, -r, "error initializing"); goto failed_opts; } - s->snap = g_strdup(snap); - s->image_name = g_strdup(image_name); + *s_snap = g_strdup(snap); + *s_image_name = g_strdup(image_name); /* try default location when conf=NULL, but ignore failure */ - r = rados_conf_read_file(s->cluster, conf); + r = rados_conf_read_file(*cluster, conf); if (conf && r < 0) { error_setg_errno(errp, -r, "error reading conf file %s", conf); goto failed_shutdown; } - r = qemu_rbd_set_keypairs(s->cluster, keypairs, errp); + r = qemu_rbd_set_keypairs(*cluster, keypairs, errp); if (r < 0) { goto failed_shutdown; } if (mon_host) { - r = rados_conf_set(s->cluster, "mon_host", mon_host); + r = rados_conf_set(*cluster, "mon_host", mon_host); if (r < 0) { goto failed_shutdown; } } - if (qemu_rbd_set_auth(s->cluster, secretid, errp) < 0) { + if (qemu_rbd_set_auth(*cluster, secretid, errp) < 0) { r = -EIO; goto failed_shutdown; } @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, * librbd defaults to no caching. If write through caching cannot * be set up, fall back to no caching. */ - if (flags & BDRV_O_NOCACHE) { - rados_conf_set(s->cluster, "rbd_cache", "false"); + if (cache) { + rados_conf_set(*cluster, "rbd_cache", "true"); } else { - rados_conf_set(s->cluster, "rbd_cache", "true"); + rados_conf_set(*cluster, "rbd_cache", "false"); } - r = rados_connect(s->cluster); + r = rados_connect(*cluster); if (r < 0) { error_setg_errno(errp, -r, "error connecting"); goto failed_shutdown; } - r = rados_ioctx_create(s->cluster, pool, &s->io_ctx); + r = rados_ioctx_create(*cluster, pool, io_ctx); if (r < 0) { error_setg_errno(errp, -r, "error opening pool %s", pool); goto failed_shutdown; } + qemu_opts_del(opts); + return 0; + +failed_shutdown: + rados_shutdown(*cluster); + g_free(*s_snap); + g_free(*s_image_name); +failed_opts: + qemu_opts_del(opts); + g_free(mon_host); + return r; +} + +static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, + Error **errp) +{ + BDRVRBDState *s = bs->opaque; + Error *local_err = NULL; + const char *filename; + int r; + + /* If we are given a filename, parse the filename, with precedence given to + * filename encoded options */ + filename = qdict_get_try_str(options, "filename"); + if (filename) { + warn_report("'filename' option specified. " + "This is an unsupported option, and may be deprecated " + "in the future"); + qemu_rbd_parse_filename(filename, options, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return -EINVAL; + } + } + + r = qemu_rbd_connect(&s->cluster, &s->io_ctx, &s->snap, &s->image_name, + options, !(flags & BDRV_O_NOCACHE), errp); + if (r < 0) { + return r; + } + /* rbd_open is always r/w */ r = rbd_open(s->io_ctx, s->image_name, &s->image, s->snap); if (r < 0) { @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, } } - qemu_opts_del(opts); return 0; failed_open: rados_ioctx_destroy(s->io_ctx); -failed_shutdown: - rados_shutdown(s->cluster); g_free(s->snap); g_free(s->image_name); -failed_opts: - qemu_opts_del(opts); - g_free(mon_host); -exit: + rados_shutdown(s->cluster); return r; } -- 2.13.6 Instead of the QemuOpts in qemu_rbd_connect(), we want to use QAPI objects. As a preparation, fetch those options directly from the QDict that .bdrv_open() supports in the rbd driver and that are not in the schema. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/rbd.c | 55 ++++++++++++++++++++++++------------------------------- 1 file changed, 24 insertions(+), 31 deletions(-) diff --git a/block/rbd.c b/block/rbd.c index XXXXXXX..XXXXXXX 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -XXX,XX +XXX,XX @@ static QemuOptsList runtime_opts = { /* * server.* extracted manually, see qemu_rbd_mon_host() */ - { - .name = "password-secret", - .type = QEMU_OPT_STRING, - .help = "ID of secret providing the password", - }, - - /* - * Keys for qemu_rbd_parse_filename(), not in the QAPI schema - */ - { - /* - * HACK: name starts with '=' so that qemu_opts_parse() - * can't set it - */ - .name = "=keyvalue-pairs", - .type = QEMU_OPT_STRING, - .help = "Legacy rados key/value option parameters", - }, - { - .name = "filename", - .type = QEMU_OPT_STRING, - }, { /* end of list */ } }, }; @@ -XXX,XX +XXX,XX @@ out: static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx, char **s_snap, char **s_image_name, - QDict *options, bool cache, Error **errp) + QDict *options, bool cache, + const char *keypairs, const char *secretid, + Error **errp) { QemuOpts *opts; char *mon_host = NULL; - const char *pool, *snap, *conf, *user, *image_name, *keypairs; - const char *secretid; + const char *pool, *snap, *conf, *user, *image_name; Error *local_err = NULL; int r; @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx, goto failed_opts; } - secretid = qemu_opt_get(opts, "password-secret"); - pool = qemu_opt_get(opts, "pool"); conf = qemu_opt_get(opts, "conf"); snap = qemu_opt_get(opts, "snapshot"); user = qemu_opt_get(opts, "user"); image_name = qemu_opt_get(opts, "image"); - keypairs = qemu_opt_get(opts, "=keyvalue-pairs"); if (!pool || !image_name) { error_setg(errp, "Parameters 'pool' and 'image' are required"); @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, BDRVRBDState *s = bs->opaque; Error *local_err = NULL; const char *filename; + char *keypairs, *secretid; int r; /* If we are given a filename, parse the filename, with precedence given to @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, "This is an unsupported option, and may be deprecated " "in the future"); qemu_rbd_parse_filename(filename, options, &local_err); + qdict_del(options, "filename"); if (local_err) { error_propagate(errp, local_err); return -EINVAL; } } + keypairs = g_strdup(qdict_get_try_str(options, "=keyvalue-pairs")); + if (keypairs) { + qdict_del(options, "=keyvalue-pairs"); + } + + secretid = g_strdup(qdict_get_try_str(options, "password-secret")); + if (secretid) { + qdict_del(options, "password-secret"); + } + r = qemu_rbd_connect(&s->cluster, &s->io_ctx, &s->snap, &s->image_name, - options, !(flags & BDRV_O_NOCACHE), errp); + options, !(flags & BDRV_O_NOCACHE), keypairs, secretid, + errp); if (r < 0) { - return r; + goto out; } /* rbd_open is always r/w */ @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, } } - return 0; + r = 0; + goto out; failed_open: rados_ioctx_destroy(s->io_ctx); g_free(s->snap); g_free(s->image_name); rados_shutdown(s->cluster); +out: + g_free(keypairs); + g_free(secretid); return r; } -- 2.13.6 With the conversion to a QAPI options object, the function is now prepared to be used in a .bdrv_co_create implementation. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/rbd.c | 115 +++++++++++++++++++++++++++++------------------------------- 1 file changed, 55 insertions(+), 60 deletions(-) diff --git a/block/rbd.c b/block/rbd.c index XXXXXXX..XXXXXXX 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -XXX,XX +XXX,XX @@ #include "qapi/qmp/qdict.h" #include "qapi/qmp/qjson.h" #include "qapi/qmp/qlist.h" +#include "qapi/qobject-input-visitor.h" +#include "qapi/qapi-visit-block-core.h" /* * When specifying the image filename use: @@ -XXX,XX +XXX,XX @@ static void qemu_rbd_complete_aio(RADOSCB *rcb) qemu_aio_unref(acb); } -static char *qemu_rbd_mon_host(QDict *options, Error **errp) +static char *qemu_rbd_mon_host(BlockdevOptionsRbd *opts, Error **errp) { - const char **vals = g_new(const char *, qdict_size(options) + 1); - char keybuf[32]; + const char **vals; const char *host, *port; char *rados_str; - int i; - - for (i = 0;; i++) { - sprintf(keybuf, "server.%d.host", i); - host = qdict_get_try_str(options, keybuf); - qdict_del(options, keybuf); - sprintf(keybuf, "server.%d.port", i); - port = qdict_get_try_str(options, keybuf); - qdict_del(options, keybuf); - if (!host && !port) { - break; - } - if (!host) { - error_setg(errp, "Parameter server.%d.host is missing", i); - rados_str = NULL; - goto out; - } + InetSocketAddressBaseList *p; + int i, cnt; + + if (!opts->has_server) { + return NULL; + } + + for (cnt = 0, p = opts->server; p; p = p->next) { + cnt++; + } + + vals = g_new(const char *, cnt + 1); + + for (i = 0, p = opts->server; p; p = p->next, i++) { + host = p->value->host; + port = p->value->port; if (strchr(host, ':')) { - vals[i] = port ? g_strdup_printf("[%s]:%s", host, port) - : g_strdup_printf("[%s]", host); + vals[i] = g_strdup_printf("[%s]:%s", host, port); } else { - vals[i] = port ? g_strdup_printf("%s:%s", host, port) - : g_strdup(host); + vals[i] = g_strdup_printf("%s:%s", host, port); } } vals[i] = NULL; rados_str = i ? g_strjoinv(";", (char **)vals) : NULL; -out: g_strfreev((char **)vals); return rados_str; } static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx, char **s_snap, char **s_image_name, - QDict *options, bool cache, + BlockdevOptionsRbd *opts, bool cache, const char *keypairs, const char *secretid, Error **errp) { - QemuOpts *opts; char *mon_host = NULL; - const char *pool, *snap, *conf, *user, *image_name; Error *local_err = NULL; int r; - opts = qemu_opts_create(&runtime_opts, NULL, 0, &error_abort); - qemu_opts_absorb_qdict(opts, options, &local_err); - if (local_err) { - error_propagate(errp, local_err); - r = -EINVAL; - goto failed_opts; - } - - mon_host = qemu_rbd_mon_host(options, &local_err); + mon_host = qemu_rbd_mon_host(opts, &local_err); if (local_err) { error_propagate(errp, local_err); r = -EINVAL; goto failed_opts; } - pool = qemu_opt_get(opts, "pool"); - conf = qemu_opt_get(opts, "conf"); - snap = qemu_opt_get(opts, "snapshot"); - user = qemu_opt_get(opts, "user"); - image_name = qemu_opt_get(opts, "image"); - - if (!pool || !image_name) { - error_setg(errp, "Parameters 'pool' and 'image' are required"); - r = -EINVAL; - goto failed_opts; - } - - r = rados_create(cluster, user); + r = rados_create(cluster, opts->user); if (r < 0) { error_setg_errno(errp, -r, "error initializing"); goto failed_opts; } - *s_snap = g_strdup(snap); - *s_image_name = g_strdup(image_name); + *s_snap = g_strdup(opts->snapshot); + *s_image_name = g_strdup(opts->image); /* try default location when conf=NULL, but ignore failure */ - r = rados_conf_read_file(*cluster, conf); - if (conf && r < 0) { - error_setg_errno(errp, -r, "error reading conf file %s", conf); + r = rados_conf_read_file(*cluster, opts->conf); + if (opts->has_conf && r < 0) { + error_setg_errno(errp, -r, "error reading conf file %s", opts->conf); goto failed_shutdown; } @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx, goto failed_shutdown; } - r = rados_ioctx_create(*cluster, pool, io_ctx); + r = rados_ioctx_create(*cluster, opts->pool, io_ctx); if (r < 0) { - error_setg_errno(errp, -r, "error opening pool %s", pool); + error_setg_errno(errp, -r, "error opening pool %s", opts->pool); goto failed_shutdown; } - qemu_opts_del(opts); return 0; failed_shutdown: @@ -XXX,XX +XXX,XX @@ failed_shutdown: g_free(*s_snap); g_free(*s_image_name); failed_opts: - qemu_opts_del(opts); g_free(mon_host); return r; } @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, Error **errp) { BDRVRBDState *s = bs->opaque; + BlockdevOptionsRbd *opts = NULL; + Visitor *v; + QObject *crumpled = NULL; Error *local_err = NULL; const char *filename; char *keypairs, *secretid; @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, qdict_del(options, "password-secret"); } + /* Convert the remaining options into a QAPI object */ + crumpled = qdict_crumple(options, errp); + if (crumpled == NULL) { + r = -EINVAL; + goto out; + } + + v = qobject_input_visitor_new_keyval(crumpled); + visit_type_BlockdevOptionsRbd(v, NULL, &opts, &local_err); + visit_free(v); + qobject_decref(crumpled); + + if (local_err) { + error_propagate(errp, local_err); + r = -EINVAL; + goto out; + } + r = qemu_rbd_connect(&s->cluster, &s->io_ctx, &s->snap, &s->image_name, - options, !(flags & BDRV_O_NOCACHE), keypairs, secretid, + opts, !(flags & BDRV_O_NOCACHE), keypairs, secretid, errp); if (r < 0) { goto out; @@ -XXX,XX +XXX,XX @@ failed_open: g_free(s->image_name); rados_shutdown(s->cluster); out: + qapi_free_BlockdevOptionsRbd(opts); g_free(keypairs); g_free(secretid); return r; -- 2.13.6 This adds the .bdrv_co_create driver callback to rbd, which enables image creation over QMP. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 19 ++++++- block/rbd.c | 150 ++++++++++++++++++++++++++++++++++----------------- 2 files changed, 118 insertions(+), 51 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ '*refcount-bits': 'int' } } ## +# @BlockdevCreateOptionsRbd: +# +# Driver specific image creation options for rbd/Ceph. +# +# @location Where to store the new image file. This location cannot +# point to a snapshot. +# @size Size of the virtual disk in bytes +# @cluster-size RBD object size +# +# Since: 2.12 +## +{ 'struct': 'BlockdevCreateOptionsRbd', + 'data': { 'location': 'BlockdevOptionsRbd', + 'size': 'size', + '*cluster-size' : 'size' } } + +## # @BlockdevCreateNotSupported: # # This is used for all drivers that don't support creating images. @@ -XXX,XX +XXX,XX @@ 'qed': 'BlockdevCreateNotSupported', 'quorum': 'BlockdevCreateNotSupported', 'raw': 'BlockdevCreateNotSupported', - 'rbd': 'BlockdevCreateNotSupported', + 'rbd': 'BlockdevCreateOptionsRbd', 'replication': 'BlockdevCreateNotSupported', 'sheepdog': 'BlockdevCreateNotSupported', 'ssh': 'BlockdevCreateNotSupported', diff --git a/block/rbd.c b/block/rbd.c index XXXXXXX..XXXXXXX 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -XXX,XX +XXX,XX @@ static QemuOptsList runtime_opts = { }, }; -static int coroutine_fn qemu_rbd_co_create_opts(const char *filename, - QemuOpts *opts, - Error **errp) +/* FIXME Deprecate and remove keypairs or make it available in QMP. + * password_secret should eventually be configurable in opts->location. Support + * for it in .bdrv_open will make it work here as well. */ +static int qemu_rbd_do_create(BlockdevCreateOptions *options, + const char *keypairs, const char *password_secret, + Error **errp) { - Error *local_err = NULL; - int64_t bytes = 0; - int64_t objsize; - int obj_order = 0; - const char *pool, *image_name, *conf, *user, *keypairs; - const char *secretid; + BlockdevCreateOptionsRbd *opts = &options->u.rbd; rados_t cluster; rados_ioctx_t io_ctx; - QDict *options = NULL; - int ret = 0; + int obj_order = 0; + int ret; + + assert(options->driver == BLOCKDEV_DRIVER_RBD); + if (opts->location->has_snapshot) { + error_setg(errp, "Can't use snapshot name for image creation"); + return -EINVAL; + } - secretid = qemu_opt_get(opts, "password-secret"); + /* TODO Remove the limitation */ + if (opts->location->has_server) { + error_setg(errp, "Can't specify server for image creation"); + return -EINVAL; + } - /* Read out options */ - bytes = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), - BDRV_SECTOR_SIZE); - objsize = qemu_opt_get_size_del(opts, BLOCK_OPT_CLUSTER_SIZE, 0); - if (objsize) { + if (opts->has_cluster_size) { + int64_t objsize = opts->cluster_size; if ((objsize - 1) & objsize) { /* not a power of 2? */ error_setg(errp, "obj size needs to be power of 2"); - ret = -EINVAL; - goto exit; + return -EINVAL; } if (objsize < 4096) { error_setg(errp, "obj size too small"); - ret = -EINVAL; - goto exit; + return -EINVAL; } obj_order = ctz32(objsize); } - options = qdict_new(); - qemu_rbd_parse_filename(filename, options, &local_err); - if (local_err) { - ret = -EINVAL; - error_propagate(errp, local_err); - goto exit; - } - - /* - * Caution: while qdict_get_try_str() is fine, getting non-string - * types would require more care. When @options come from -blockdev - * or blockdev_add, its members are typed according to the QAPI - * schema, but when they come from -drive, they're all QString. - */ - pool = qdict_get_try_str(options, "pool"); - conf = qdict_get_try_str(options, "conf"); - user = qdict_get_try_str(options, "user"); - image_name = qdict_get_try_str(options, "image"); - keypairs = qdict_get_try_str(options, "=keyvalue-pairs"); - - ret = rados_create(&cluster, user); + ret = rados_create(&cluster, opts->location->user); if (ret < 0) { error_setg_errno(errp, -ret, "error initializing"); - goto exit; + return ret; } /* try default location when conf=NULL, but ignore failure */ - ret = rados_conf_read_file(cluster, conf); - if (conf && ret < 0) { - error_setg_errno(errp, -ret, "error reading conf file %s", conf); + ret = rados_conf_read_file(cluster, opts->location->conf); + if (opts->location->conf && ret < 0) { + error_setg_errno(errp, -ret, "error reading conf file %s", + opts->location->conf); ret = -EIO; goto shutdown; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qemu_rbd_co_create_opts(const char *filename, goto shutdown; } - if (qemu_rbd_set_auth(cluster, secretid, errp) < 0) { + if (qemu_rbd_set_auth(cluster, password_secret, errp) < 0) { ret = -EIO; goto shutdown; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qemu_rbd_co_create_opts(const char *filename, goto shutdown; } - ret = rados_ioctx_create(cluster, pool, &io_ctx); + ret = rados_ioctx_create(cluster, opts->location->pool, &io_ctx); if (ret < 0) { - error_setg_errno(errp, -ret, "error opening pool %s", pool); + error_setg_errno(errp, -ret, "error opening pool %s", + opts->location->pool); goto shutdown; } - ret = rbd_create(io_ctx, image_name, bytes, &obj_order); + ret = rbd_create(io_ctx, opts->location->image, opts->size, &obj_order); if (ret < 0) { error_setg_errno(errp, -ret, "error rbd create"); } rados_ioctx_destroy(io_ctx); + ret = 0; shutdown: rados_shutdown(cluster); + return ret; +} + +static int qemu_rbd_co_create(BlockdevCreateOptions *options, Error **errp) +{ + return qemu_rbd_do_create(options, NULL, NULL, errp); +} + +static int coroutine_fn qemu_rbd_co_create_opts(const char *filename, + QemuOpts *opts, + Error **errp) +{ + BlockdevCreateOptions *create_options; + BlockdevCreateOptionsRbd *rbd_opts; + BlockdevOptionsRbd *loc; + Error *local_err = NULL; + const char *keypairs, *password_secret; + QDict *options = NULL; + int ret = 0; + + create_options = g_new0(BlockdevCreateOptions, 1); + create_options->driver = BLOCKDEV_DRIVER_RBD; + rbd_opts = &create_options->u.rbd; + + rbd_opts->location = g_new0(BlockdevOptionsRbd, 1); + + password_secret = qemu_opt_get(opts, "password-secret"); + + /* Read out options */ + rbd_opts->size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), + BDRV_SECTOR_SIZE); + rbd_opts->cluster_size = qemu_opt_get_size_del(opts, + BLOCK_OPT_CLUSTER_SIZE, 0); + rbd_opts->has_cluster_size = (rbd_opts->cluster_size != 0); + + options = qdict_new(); + qemu_rbd_parse_filename(filename, options, &local_err); + if (local_err) { + ret = -EINVAL; + error_propagate(errp, local_err); + goto exit; + } + + /* + * Caution: while qdict_get_try_str() is fine, getting non-string + * types would require more care. When @options come from -blockdev + * or blockdev_add, its members are typed according to the QAPI + * schema, but when they come from -drive, they're all QString. + */ + loc = rbd_opts->location; + loc->pool = g_strdup(qdict_get_try_str(options, "pool")); + loc->conf = g_strdup(qdict_get_try_str(options, "conf")); + loc->has_conf = !!loc->conf; + loc->user = g_strdup(qdict_get_try_str(options, "user")); + loc->has_user = !!loc->user; + loc->image = g_strdup(qdict_get_try_str(options, "image")); + keypairs = qdict_get_try_str(options, "=keyvalue-pairs"); + + ret = qemu_rbd_do_create(create_options, keypairs, password_secret, errp); + if (ret < 0) { + goto exit; + } exit: QDECREF(options); + qapi_free_BlockdevCreateOptions(create_options); return ret; } @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_rbd = { .bdrv_file_open = qemu_rbd_open, .bdrv_close = qemu_rbd_close, .bdrv_reopen_prepare = qemu_rbd_reopen_prepare, + .bdrv_co_create = qemu_rbd_co_create, .bdrv_co_create_opts = qemu_rbd_co_create_opts, .bdrv_has_zero_init = bdrv_has_zero_init_1, .bdrv_get_info = qemu_rbd_getinfo, -- 2.13.6 Now that the options are already available in qemu_rbd_open() and not only parsed in qemu_rbd_connect(), we can assign s->snap and s->image_name there instead of passing the fields by reference to qemu_rbd_connect(). Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/rbd.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/block/rbd.c b/block/rbd.c index XXXXXXX..XXXXXXX 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -XXX,XX +XXX,XX @@ static char *qemu_rbd_mon_host(BlockdevOptionsRbd *opts, Error **errp) } static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx, - char **s_snap, char **s_image_name, BlockdevOptionsRbd *opts, bool cache, const char *keypairs, const char *secretid, Error **errp) @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx, goto failed_opts; } - *s_snap = g_strdup(opts->snapshot); - *s_image_name = g_strdup(opts->image); - /* try default location when conf=NULL, but ignore failure */ r = rados_conf_read_file(*cluster, opts->conf); if (opts->has_conf && r < 0) { @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx, failed_shutdown: rados_shutdown(*cluster); - g_free(*s_snap); - g_free(*s_image_name); failed_opts: g_free(mon_host); return r; @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_open(BlockDriverState *bs, QDict *options, int flags, goto out; } - r = qemu_rbd_connect(&s->cluster, &s->io_ctx, &s->snap, &s->image_name, - opts, !(flags & BDRV_O_NOCACHE), keypairs, secretid, - errp); + r = qemu_rbd_connect(&s->cluster, &s->io_ctx, opts, + !(flags & BDRV_O_NOCACHE), keypairs, secretid, errp); if (r < 0) { goto out; } + s->snap = g_strdup(opts->snapshot); + s->image_name = g_strdup(opts->image); + /* rbd_open is always r/w */ r = rbd_open(s->io_ctx, s->image_name, &s->image, s->snap); if (r < 0) { -- 2.13.6 This is almost exactly the same code. The differences are that qemu_rbd_connect() supports BlockdevOptionsRbd.server and that the cache mode is set explicitly. Supporting 'server' is a welcome new feature for image creation. Caching is disabled by default, so leave it that way. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/rbd.c | 54 ++++++++++-------------------------------------------- 1 file changed, 10 insertions(+), 44 deletions(-) diff --git a/block/rbd.c b/block/rbd.c index XXXXXXX..XXXXXXX 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -XXX,XX +XXX,XX @@ typedef struct BDRVRBDState { char *snap; } BDRVRBDState; +static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx, + BlockdevOptionsRbd *opts, bool cache, + const char *keypairs, const char *secretid, + Error **errp); + static char *qemu_rbd_next_tok(char *src, char delim, char **p) { char *end; @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_do_create(BlockdevCreateOptions *options, return -EINVAL; } - /* TODO Remove the limitation */ - if (opts->location->has_server) { - error_setg(errp, "Can't specify server for image creation"); - return -EINVAL; - } - if (opts->has_cluster_size) { int64_t objsize = opts->cluster_size; if ((objsize - 1) & objsize) { /* not a power of 2? */ @@ -XXX,XX +XXX,XX @@ static int qemu_rbd_do_create(BlockdevCreateOptions *options, obj_order = ctz32(objsize); } - ret = rados_create(&cluster, opts->location->user); + ret = qemu_rbd_connect(&cluster, &io_ctx, opts->location, false, keypairs, + password_secret, errp); if (ret < 0) { - error_setg_errno(errp, -ret, "error initializing"); return ret; } - /* try default location when conf=NULL, but ignore failure */ - ret = rados_conf_read_file(cluster, opts->location->conf); - if (opts->location->conf && ret < 0) { - error_setg_errno(errp, -ret, "error reading conf file %s", - opts->location->conf); - ret = -EIO; - goto shutdown; - } - - ret = qemu_rbd_set_keypairs(cluster, keypairs, errp); - if (ret < 0) { - ret = -EIO; - goto shutdown; - } - - if (qemu_rbd_set_auth(cluster, password_secret, errp) < 0) { - ret = -EIO; - goto shutdown; - } - - ret = rados_connect(cluster); - if (ret < 0) { - error_setg_errno(errp, -ret, "error connecting"); - goto shutdown; - } - - ret = rados_ioctx_create(cluster, opts->location->pool, &io_ctx); - if (ret < 0) { - error_setg_errno(errp, -ret, "error opening pool %s", - opts->location->pool); - goto shutdown; - } - ret = rbd_create(io_ctx, opts->location->image, opts->size, &obj_order); if (ret < 0) { error_setg_errno(errp, -ret, "error rbd create"); + goto out; } - rados_ioctx_destroy(io_ctx); - ret = 0; -shutdown: +out: + rados_ioctx_destroy(io_ctx); rados_shutdown(cluster); return ret; } -- 2.13.6 Using the QAPI visitor to turn all options into QAPI BlockdevOptionsNfs simplifies the code a lot. It will also be useful for implementing the QAPI based .bdrv_co_create callback. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/nfs.c | 176 ++++++++++++++++++------------------------------------------ 1 file changed, 53 insertions(+), 123 deletions(-) diff --git a/block/nfs.c b/block/nfs.c index XXXXXXX..XXXXXXX 100644 --- a/block/nfs.c +++ b/block/nfs.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn nfs_co_flush(BlockDriverState *bs) return task.ret; } -static QemuOptsList runtime_opts = { - .name = "nfs", - .head = QTAILQ_HEAD_INITIALIZER(runtime_opts.head), - .desc = { - { - .name = "path", - .type = QEMU_OPT_STRING, - .help = "Path of the image on the host", - }, - { - .name = "user", - .type = QEMU_OPT_NUMBER, - .help = "UID value to use when talking to the server", - }, - { - .name = "group", - .type = QEMU_OPT_NUMBER, - .help = "GID value to use when talking to the server", - }, - { - .name = "tcp-syn-count", - .type = QEMU_OPT_NUMBER, - .help = "Number of SYNs to send during the session establish", - }, - { - .name = "readahead-size", - .type = QEMU_OPT_NUMBER, - .help = "Set the readahead size in bytes", - }, - { - .name = "page-cache-size", - .type = QEMU_OPT_NUMBER, - .help = "Set the pagecache size in bytes", - }, - { - .name = "debug", - .type = QEMU_OPT_NUMBER, - .help = "Set the NFS debug level (max 2)", - }, - { /* end of list */ } - }, -}; - static void nfs_detach_aio_context(BlockDriverState *bs) { NFSClient *client = bs->opaque; @@ -XXX,XX +XXX,XX @@ static void nfs_file_close(BlockDriverState *bs) nfs_client_close(client); } -static NFSServer *nfs_config(QDict *options, Error **errp) -{ - NFSServer *server = NULL; - QDict *addr = NULL; - QObject *crumpled_addr = NULL; - Visitor *iv = NULL; - Error *local_error = NULL; - - qdict_extract_subqdict(options, &addr, "server."); - if (!qdict_size(addr)) { - error_setg(errp, "NFS server address missing"); - goto out; - } - - crumpled_addr = qdict_crumple(addr, errp); - if (!crumpled_addr) { - goto out; - } - - /* - * Caution: this works only because all scalar members of - * NFSServer are QString in @crumpled_addr. The visitor expects - * @crumpled_addr to be typed according to the QAPI schema. It - * is when @options come from -blockdev or blockdev_add. But when - * they come from -drive, they're all QString. - */ - iv = qobject_input_visitor_new(crumpled_addr); - visit_type_NFSServer(iv, NULL, &server, &local_error); - if (local_error) { - error_propagate(errp, local_error); - goto out; - } - -out: - QDECREF(addr); - qobject_decref(crumpled_addr); - visit_free(iv); - return server; -} - - -static int64_t nfs_client_open(NFSClient *client, QDict *options, +static int64_t nfs_client_open(NFSClient *client, BlockdevOptionsNfs *opts, int flags, int open_flags, Error **errp) { int64_t ret = -EINVAL; - QemuOpts *opts = NULL; - Error *local_err = NULL; struct stat st; char *file = NULL, *strp = NULL; qemu_mutex_init(&client->mutex); - opts = qemu_opts_create(&runtime_opts, NULL, 0, &error_abort); - qemu_opts_absorb_qdict(opts, options, &local_err); - if (local_err) { - error_propagate(errp, local_err); - ret = -EINVAL; - goto fail; - } - client->path = g_strdup(qemu_opt_get(opts, "path")); - if (!client->path) { - ret = -EINVAL; - error_setg(errp, "No path was specified"); - goto fail; - } + client->path = g_strdup(opts->path); strp = strrchr(client->path, '/'); if (strp == NULL) { @@ -XXX,XX +XXX,XX @@ static int64_t nfs_client_open(NFSClient *client, QDict *options, file = g_strdup(strp); *strp = 0; - /* Pop the config into our state object, Exit if invalid */ - client->server = nfs_config(options, errp); - if (!client->server) { - ret = -EINVAL; - goto fail; - } + /* Steal the NFSServer object from opts; set the original pointer to NULL + * to avoid use after free and double free. */ + client->server = opts->server; + opts->server = NULL; client->context = nfs_init_context(); if (client->context == NULL) { @@ -XXX,XX +XXX,XX @@ static int64_t nfs_client_open(NFSClient *client, QDict *options, goto fail; } - if (qemu_opt_get(opts, "user")) { - client->uid = qemu_opt_get_number(opts, "user", 0); + if (opts->has_user) { + client->uid = opts->user; nfs_set_uid(client->context, client->uid); } - if (qemu_opt_get(opts, "group")) { - client->gid = qemu_opt_get_number(opts, "group", 0); + if (opts->has_group) { + client->gid = opts->group; nfs_set_gid(client->context, client->gid); } - if (qemu_opt_get(opts, "tcp-syn-count")) { - client->tcp_syncnt = qemu_opt_get_number(opts, "tcp-syn-count", 0); + if (opts->has_tcp_syn_count) { + client->tcp_syncnt = opts->tcp_syn_count; nfs_set_tcp_syncnt(client->context, client->tcp_syncnt); } #ifdef LIBNFS_FEATURE_READAHEAD - if (qemu_opt_get(opts, "readahead-size")) { + if (opts->has_readahead_size) { if (open_flags & BDRV_O_NOCACHE) { error_setg(errp, "Cannot enable NFS readahead " "if cache.direct = on"); goto fail; } - client->readahead = qemu_opt_get_number(opts, "readahead-size", 0); + client->readahead = opts->readahead_size; if (client->readahead > QEMU_NFS_MAX_READAHEAD_SIZE) { warn_report("Truncating NFS readahead size to %d", QEMU_NFS_MAX_READAHEAD_SIZE); @@ -XXX,XX +XXX,XX @@ static int64_t nfs_client_open(NFSClient *client, QDict *options, #endif #ifdef LIBNFS_FEATURE_PAGECACHE - if (qemu_opt_get(opts, "page-cache-size")) { + if (opts->has_page_cache_size) { if (open_flags & BDRV_O_NOCACHE) { error_setg(errp, "Cannot enable NFS pagecache " "if cache.direct = on"); goto fail; } - client->pagecache = qemu_opt_get_number(opts, "page-cache-size", 0); + client->pagecache = opts->page_cache_size; if (client->pagecache > QEMU_NFS_MAX_PAGECACHE_SIZE) { warn_report("Truncating NFS pagecache size to %d pages", QEMU_NFS_MAX_PAGECACHE_SIZE); @@ -XXX,XX +XXX,XX @@ static int64_t nfs_client_open(NFSClient *client, QDict *options, #endif #ifdef LIBNFS_FEATURE_DEBUG - if (qemu_opt_get(opts, "debug")) { - client->debug = qemu_opt_get_number(opts, "debug", 0); + if (opts->has_debug) { + client->debug = opts->debug; /* limit the maximum debug level to avoid potential flooding * of our log files. */ if (client->debug > QEMU_NFS_MAX_DEBUG_LEVEL) { @@ -XXX,XX +XXX,XX @@ static int64_t nfs_client_open(NFSClient *client, QDict *options, fail: nfs_client_close(client); out: - qemu_opts_del(opts); g_free(file); return ret; } +static int64_t nfs_client_open_qdict(NFSClient *client, QDict *options, + int flags, int open_flags, Error **errp) +{ + BlockdevOptionsNfs *opts = NULL; + QObject *crumpled = NULL; + Visitor *v; + Error *local_err = NULL; + int ret; + + crumpled = qdict_crumple(options, errp); + if (crumpled == NULL) { + return -EINVAL; + } + + v = qobject_input_visitor_new_keyval(crumpled); + visit_type_BlockdevOptionsNfs(v, NULL, &opts, &local_err); + visit_free(v); + + if (local_err) { + error_propagate(errp, local_err); + ret = -EINVAL; + goto fail; + } + + ret = nfs_client_open(client, opts, flags, open_flags, errp); +fail: + qobject_decref(crumpled); + qapi_free_BlockdevOptionsNfs(opts); + return ret; +} + static int nfs_file_open(BlockDriverState *bs, QDict *options, int flags, Error **errp) { NFSClient *client = bs->opaque; @@ -XXX,XX +XXX,XX @@ static int nfs_file_open(BlockDriverState *bs, QDict *options, int flags, client->aio_context = bdrv_get_aio_context(bs); - ret = nfs_client_open(client, options, - (flags & BDRV_O_RDWR) ? O_RDWR : O_RDONLY, - bs->open_flags, errp); + ret = nfs_client_open_qdict(client, options, + (flags & BDRV_O_RDWR) ? O_RDWR : O_RDONLY, + bs->open_flags, errp); if (ret < 0) { return ret; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn nfs_file_co_create_opts(const char *url, QemuOpts *opts, goto out; } - ret = nfs_client_open(client, options, O_CREAT, 0, errp); + ret = nfs_client_open_qdict(client, options, O_CREAT, 0, errp); if (ret < 0) { goto out; } -- 2.13.6 This adds the .bdrv_co_create driver callback to nfs, which enables image creation over QMP. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 16 ++++++++++- block/nfs.c | 76 +++++++++++++++++++++++++++++++++++++++++----------- 2 files changed, 75 insertions(+), 17 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ '*preallocation': 'PreallocMode' } } ## +# @BlockdevCreateOptionsNfs: +# +# Driver specific image creation options for NFS. +# +# @location Where to store the new image file +# @size Size of the virtual disk in bytes +# +# Since: 2.12 +## +{ 'struct': 'BlockdevCreateOptionsNfs', + 'data': { 'location': 'BlockdevOptionsNfs', + 'size': 'size' } } + +## # @BlockdevQcow2Version: # # @v2: The original QCOW2 format as introduced in qemu 0.10 (version 2) @@ -XXX,XX +XXX,XX @@ 'iscsi': 'BlockdevCreateNotSupported', 'luks': 'BlockdevCreateNotSupported', 'nbd': 'BlockdevCreateNotSupported', - 'nfs': 'BlockdevCreateNotSupported', + 'nfs': 'BlockdevCreateOptionsNfs', 'null-aio': 'BlockdevCreateNotSupported', 'null-co': 'BlockdevCreateNotSupported', 'nvme': 'BlockdevCreateNotSupported', diff --git a/block/nfs.c b/block/nfs.c index XXXXXXX..XXXXXXX 100644 --- a/block/nfs.c +++ b/block/nfs.c @@ -XXX,XX +XXX,XX @@ out: return ret; } -static int64_t nfs_client_open_qdict(NFSClient *client, QDict *options, - int flags, int open_flags, Error **errp) +static BlockdevOptionsNfs *nfs_options_qdict_to_qapi(QDict *options, + Error **errp) { BlockdevOptionsNfs *opts = NULL; QObject *crumpled = NULL; Visitor *v; Error *local_err = NULL; - int ret; crumpled = qdict_crumple(options, errp); if (crumpled == NULL) { - return -EINVAL; + return NULL; } v = qobject_input_visitor_new_keyval(crumpled); visit_type_BlockdevOptionsNfs(v, NULL, &opts, &local_err); visit_free(v); + qobject_decref(crumpled); if (local_err) { - error_propagate(errp, local_err); + return NULL; + } + + return opts; +} + +static int64_t nfs_client_open_qdict(NFSClient *client, QDict *options, + int flags, int open_flags, Error **errp) +{ + BlockdevOptionsNfs *opts; + int ret; + + opts = nfs_options_qdict_to_qapi(options, errp); + if (opts == NULL) { ret = -EINVAL; goto fail; } ret = nfs_client_open(client, opts, flags, open_flags, errp); fail: - qobject_decref(crumpled); qapi_free_BlockdevOptionsNfs(opts); return ret; } @@ -XXX,XX +XXX,XX @@ static QemuOptsList nfs_create_opts = { } }; -static int coroutine_fn nfs_file_co_create_opts(const char *url, QemuOpts *opts, - Error **errp) +static int nfs_file_co_create(BlockdevCreateOptions *options, Error **errp) { - int64_t ret, total_size; + BlockdevCreateOptionsNfs *opts = &options->u.nfs; NFSClient *client = g_new0(NFSClient, 1); - QDict *options = NULL; + int ret; + + assert(options->driver == BLOCKDEV_DRIVER_NFS); client->aio_context = qemu_get_aio_context(); + ret = nfs_client_open(client, opts->location, O_CREAT, 0, errp); + if (ret < 0) { + goto out; + } + ret = nfs_ftruncate(client->context, client->fh, opts->size); + nfs_client_close(client); + +out: + g_free(client); + return ret; +} + +static int coroutine_fn nfs_file_co_create_opts(const char *url, QemuOpts *opts, + Error **errp) +{ + BlockdevCreateOptions *create_options; + BlockdevCreateOptionsNfs *nfs_opts; + QDict *options; + int ret; + + create_options = g_new0(BlockdevCreateOptions, 1); + create_options->driver = BLOCKDEV_DRIVER_NFS; + nfs_opts = &create_options->u.nfs; + /* Read out options */ - total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), - BDRV_SECTOR_SIZE); + nfs_opts->size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), + BDRV_SECTOR_SIZE); options = qdict_new(); ret = nfs_parse_uri(url, options, errp); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn nfs_file_co_create_opts(const char *url, QemuOpts *opts, goto out; } - ret = nfs_client_open_qdict(client, options, O_CREAT, 0, errp); + nfs_opts->location = nfs_options_qdict_to_qapi(options, errp); + if (nfs_opts->location == NULL) { + ret = -EINVAL; + goto out; + } + + ret = nfs_file_co_create(create_options, errp); if (ret < 0) { goto out; } - ret = nfs_ftruncate(client->context, client->fh, total_size); - nfs_client_close(client); + + ret = 0; out: QDECREF(options); - g_free(client); + qapi_free_BlockdevCreateOptions(create_options); return ret; } @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_nfs = { .bdrv_file_open = nfs_file_open, .bdrv_close = nfs_file_close, + .bdrv_co_create = nfs_file_co_create, .bdrv_co_create_opts = nfs_file_co_create_opts, .bdrv_reopen_prepare = nfs_reopen_prepare, -- 2.13.6 The "redundancy" option for Sheepdog image creation is currently a string that can encode one or two integers depending on its format, which at the same time implicitly selects a mode. This patch turns it into a QAPI union and converts the string into such a QAPI object before interpreting the values. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 45 +++++++++++++++++++++++++ block/sheepdog.c | 94 +++++++++++++++++++++++++++++++++++++--------------- 2 files changed, 112 insertions(+), 27 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ '*cluster-size' : 'size' } } ## +# @SheepdogRedundancyType: +# +# @full Create a fully replicated vdi with x copies +# @erasure-coded Create an erasure coded vdi with x data strips and +# y parity strips +# +# Since: 2.12 +## +{ 'enum': 'SheepdogRedundancyType', + 'data': [ 'full', 'erasure-coded' ] } + +## +# @SheepdogRedundancyFull: +# +# @copies Number of copies to use (between 1 and 31) +# +# Since: 2.12 +## +{ 'struct': 'SheepdogRedundancyFull', + 'data': { 'copies': 'int' }} + +## +# @SheepdogRedundancyErasureCoded: +# +# @data-strips Number of data strips to use (one of {2,4,8,16}) +# @parity-strips Number of parity strips to use (between 1 and 15) +# +# Since: 2.12 +## +{ 'struct': 'SheepdogRedundancyErasureCoded', + 'data': { 'data-strips': 'int', + 'parity-strips': 'int' }} + +## +# @SheepdogRedundancy: +# +# Since: 2.12 +## +{ 'union': 'SheepdogRedundancy', + 'base': { 'type': 'SheepdogRedundancyType' }, + 'discriminator': 'type', + 'data': { 'full': 'SheepdogRedundancyFull', + 'erasure-coded': 'SheepdogRedundancyErasureCoded' } } + +## # @BlockdevCreateNotSupported: # # This is used for all drivers that don't support creating images. diff --git a/block/sheepdog.c b/block/sheepdog.c index XXXXXXX..XXXXXXX 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -XXX,XX +XXX,XX @@ out_with_err_set: return ret; } +static int parse_redundancy(BDRVSheepdogState *s, SheepdogRedundancy *opt) +{ + struct SheepdogInode *inode = &s->inode; + + switch (opt->type) { + case SHEEPDOG_REDUNDANCY_TYPE_FULL: + if (opt->u.full.copies > SD_MAX_COPIES || opt->u.full.copies < 1) { + return -EINVAL; + } + inode->copy_policy = 0; + inode->nr_copies = opt->u.full.copies; + return 0; + + case SHEEPDOG_REDUNDANCY_TYPE_ERASURE_CODED: + { + int64_t copy = opt->u.erasure_coded.data_strips; + int64_t parity = opt->u.erasure_coded.parity_strips; + + if (copy != 2 && copy != 4 && copy != 8 && copy != 16) { + return -EINVAL; + } + + if (parity >= SD_EC_MAX_STRIP || parity < 1) { + return -EINVAL; + } + + /* + * 4 bits for parity and 4 bits for data. + * We have to compress upper data bits because it can't represent 16 + */ + inode->copy_policy = ((copy / 2) << 4) + parity; + inode->nr_copies = copy + parity; + return 0; + } + + default: + g_assert_not_reached(); + } + + return -EINVAL; +} + /* * Sheepdog support two kinds of redundancy, full replication and erasure * coding. @@ -XXX,XX +XXX,XX @@ out_with_err_set: * # create a erasure coded vdi with x data strips and y parity strips * -o redundancy=x:y (x must be one of {2,4,8,16} and 1 <= y < SD_EC_MAX_STRIP) */ -static int parse_redundancy(BDRVSheepdogState *s, const char *opt) +static int parse_redundancy_str(BDRVSheepdogState *s, const char *opt) { - struct SheepdogInode *inode = &s->inode; + struct SheepdogRedundancy redundancy; const char *n1, *n2; long copy, parity; char p[10]; + int ret; pstrcpy(p, sizeof(p), opt); n1 = strtok(p, ":"); @@ -XXX,XX +XXX,XX @@ static int parse_redundancy(BDRVSheepdogState *s, const char *opt) return -EINVAL; } - copy = strtol(n1, NULL, 10); - /* FIXME fix error checking by switching to qemu_strtol() */ - if (copy > SD_MAX_COPIES || copy < 1) { - return -EINVAL; - } - if (!n2) { - inode->copy_policy = 0; - inode->nr_copies = copy; - return 0; + ret = qemu_strtol(n1, NULL, 10, &copy); + if (ret < 0) { + return ret; } - if (copy != 2 && copy != 4 && copy != 8 && copy != 16) { - return -EINVAL; - } + if (!n2) { + redundancy = (SheepdogRedundancy) { + .type = SHEEPDOG_REDUNDANCY_TYPE_FULL, + .u.full.copies = copy, + }; + } else { + ret = qemu_strtol(n2, NULL, 10, &parity); + if (ret < 0) { + return ret; + } - parity = strtol(n2, NULL, 10); - /* FIXME fix error checking by switching to qemu_strtol() */ - if (parity >= SD_EC_MAX_STRIP || parity < 1) { - return -EINVAL; + redundancy = (SheepdogRedundancy) { + .type = SHEEPDOG_REDUNDANCY_TYPE_ERASURE_CODED, + .u.erasure_coded = { + .data_strips = copy, + .parity_strips = parity, + }, + }; } - /* - * 4 bits for parity and 4 bits for data. - * We have to compress upper data bits because it can't represent 16 - */ - inode->copy_policy = ((copy / 2) << 4) + parity; - inode->nr_copies = copy + parity; - - return 0; + return parse_redundancy(s, &redundancy); } static int parse_block_size_shift(BDRVSheepdogState *s, QemuOpts *opt) @@ -XXX,XX +XXX,XX @@ static int coroutine_fn sd_co_create_opts(const char *filename, QemuOpts *opts, g_free(buf); buf = qemu_opt_get_del(opts, BLOCK_OPT_REDUNDANCY); if (buf) { - ret = parse_redundancy(s, buf); + ret = parse_redundancy_str(s, buf); if (ret < 0) { error_setg(errp, "Invalid redundancy mode: '%s'", buf); goto out; -- 2.13.6 This adds the .bdrv_co_create driver callback to sheepdog, which enables image creation over QMP. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 24 ++++- block/sheepdog.c | 243 +++++++++++++++++++++++++++++++++++---------------- 2 files changed, 192 insertions(+), 75 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ 'erasure-coded': 'SheepdogRedundancyErasureCoded' } } ## +# @BlockdevCreateOptionsSheepdog: +# +# Driver specific image creation options for Sheepdog. +# +# @location Where to store the new image file +# @size Size of the virtual disk in bytes +# @backing-file File name of a base image +# @preallocation Preallocation mode (allowed values: off, full) +# @redundancy Redundancy of the image +# @object-size Object size of the image +# +# Since: 2.12 +## +{ 'struct': 'BlockdevCreateOptionsSheepdog', + 'data': { 'location': 'BlockdevOptionsSheepdog', + 'size': 'size', + '*backing-file': 'str', + '*preallocation': 'PreallocMode', + '*redundancy': 'SheepdogRedundancy', + '*object-size': 'size' } } + +## # @BlockdevCreateNotSupported: # # This is used for all drivers that don't support creating images. @@ -XXX,XX +XXX,XX @@ 'raw': 'BlockdevCreateNotSupported', 'rbd': 'BlockdevCreateOptionsRbd', 'replication': 'BlockdevCreateNotSupported', - 'sheepdog': 'BlockdevCreateNotSupported', + 'sheepdog': 'BlockdevCreateOptionsSheepdog', 'ssh': 'BlockdevCreateNotSupported', 'throttle': 'BlockdevCreateNotSupported', 'vdi': 'BlockdevCreateNotSupported', diff --git a/block/sheepdog.c b/block/sheepdog.c index XXXXXXX..XXXXXXX 100644 --- a/block/sheepdog.c +++ b/block/sheepdog.c @@ -XXX,XX +XXX,XX @@ #include "qemu/osdep.h" #include "qapi/error.h" #include "qapi/qapi-visit-sockets.h" +#include "qapi/qapi-visit-block-core.h" #include "qapi/qmp/qdict.h" #include "qapi/qobject-input-visitor.h" +#include "qapi/qobject-output-visitor.h" #include "qemu/uri.h" #include "qemu/error-report.h" #include "qemu/option.h" @@ -XXX,XX +XXX,XX @@ static void sd_aio_setup(SheepdogAIOCB *acb, BDRVSheepdogState *s, qemu_co_mutex_unlock(&s->queue_lock); } -static SocketAddress *sd_socket_address(const char *path, - const char *host, const char *port) -{ - SocketAddress *addr = g_new0(SocketAddress, 1); - - if (path) { - addr->type = SOCKET_ADDRESS_TYPE_UNIX; - addr->u.q_unix.path = g_strdup(path); - } else { - addr->type = SOCKET_ADDRESS_TYPE_INET; - addr->u.inet.host = g_strdup(host ?: SD_DEFAULT_ADDR); - addr->u.inet.port = g_strdup(port ?: stringify(SD_DEFAULT_PORT)); - } - - return addr; -} - static SocketAddress *sd_server_config(QDict *options, Error **errp) { QDict *server = NULL; @@ -XXX,XX +XXX,XX @@ out_with_err_set: return ret; } +static int sd_create_prealloc(BlockdevOptionsSheepdog *location, int64_t size, + Error **errp) +{ + BlockDriverState *bs; + Visitor *v; + QObject *obj = NULL; + QDict *qdict; + Error *local_err = NULL; + int ret; + + v = qobject_output_visitor_new(&obj); + visit_type_BlockdevOptionsSheepdog(v, NULL, &location, &local_err); + visit_free(v); + + if (local_err) { + error_propagate(errp, local_err); + qobject_decref(obj); + return -EINVAL; + } + + qdict = qobject_to_qdict(obj); + qdict_flatten(qdict); + + qdict_put_str(qdict, "driver", "sheepdog"); + + bs = bdrv_open(NULL, NULL, qdict, BDRV_O_PROTOCOL | BDRV_O_RDWR, errp); + if (bs == NULL) { + ret = -EIO; + goto fail; + } + + ret = sd_prealloc(bs, 0, size, errp); +fail: + bdrv_unref(bs); + QDECREF(qdict); + return ret; +} + static int parse_redundancy(BDRVSheepdogState *s, SheepdogRedundancy *opt) { struct SheepdogInode *inode = &s->inode; @@ -XXX,XX +XXX,XX @@ static int parse_redundancy(BDRVSheepdogState *s, SheepdogRedundancy *opt) * # create a erasure coded vdi with x data strips and y parity strips * -o redundancy=x:y (x must be one of {2,4,8,16} and 1 <= y < SD_EC_MAX_STRIP) */ -static int parse_redundancy_str(BDRVSheepdogState *s, const char *opt) +static SheepdogRedundancy *parse_redundancy_str(const char *opt) { - struct SheepdogRedundancy redundancy; + SheepdogRedundancy *redundancy; const char *n1, *n2; long copy, parity; char p[10]; @@ -XXX,XX +XXX,XX @@ static int parse_redundancy_str(BDRVSheepdogState *s, const char *opt) n2 = strtok(NULL, ":"); if (!n1) { - return -EINVAL; + return NULL; } ret = qemu_strtol(n1, NULL, 10, &copy); if (ret < 0) { - return ret; + return NULL; } + redundancy = g_new0(SheepdogRedundancy, 1); if (!n2) { - redundancy = (SheepdogRedundancy) { + *redundancy = (SheepdogRedundancy) { .type = SHEEPDOG_REDUNDANCY_TYPE_FULL, .u.full.copies = copy, }; } else { ret = qemu_strtol(n2, NULL, 10, &parity); if (ret < 0) { - return ret; + return NULL; } - redundancy = (SheepdogRedundancy) { + *redundancy = (SheepdogRedundancy) { .type = SHEEPDOG_REDUNDANCY_TYPE_ERASURE_CODED, .u.erasure_coded = { .data_strips = copy, @@ -XXX,XX +XXX,XX @@ static int parse_redundancy_str(BDRVSheepdogState *s, const char *opt) }; } - return parse_redundancy(s, &redundancy); + return redundancy; } -static int parse_block_size_shift(BDRVSheepdogState *s, QemuOpts *opt) +static int parse_block_size_shift(BDRVSheepdogState *s, + BlockdevCreateOptionsSheepdog *opts) { struct SheepdogInode *inode = &s->inode; uint64_t object_size; int obj_order; - object_size = qemu_opt_get_size_del(opt, BLOCK_OPT_OBJECT_SIZE, 0); - if (object_size) { + if (opts->has_object_size) { + object_size = opts->object_size; + if ((object_size - 1) & object_size) { /* not a power of 2? */ return -EINVAL; } @@ -XXX,XX +XXX,XX @@ static int parse_block_size_shift(BDRVSheepdogState *s, QemuOpts *opt) return 0; } -static int coroutine_fn sd_co_create_opts(const char *filename, QemuOpts *opts, - Error **errp) +static int sd_co_create(BlockdevCreateOptions *options, Error **errp) { - Error *err = NULL; + BlockdevCreateOptionsSheepdog *opts = &options->u.sheepdog; int ret = 0; uint32_t vid = 0; char *backing_file = NULL; char *buf = NULL; BDRVSheepdogState *s; - SheepdogConfig cfg; uint64_t max_vdi_size; bool prealloc = false; + assert(options->driver == BLOCKDEV_DRIVER_SHEEPDOG); + s = g_new0(BDRVSheepdogState, 1); - if (strstr(filename, "://")) { - sd_parse_uri(&cfg, filename, &err); - } else { - parse_vdiname(&cfg, filename, &err); - } - if (err) { - error_propagate(errp, err); + /* Steal SocketAddress from QAPI, set NULL to prevent double free */ + s->addr = opts->location->server; + opts->location->server = NULL; + + if (strlen(opts->location->vdi) >= sizeof(s->name)) { + error_setg(errp, "'vdi' string too long"); + ret = -EINVAL; goto out; } + pstrcpy(s->name, sizeof(s->name), opts->location->vdi); - buf = cfg.port ? g_strdup_printf("%d", cfg.port) : NULL; - s->addr = sd_socket_address(cfg.path, cfg.host, buf); - g_free(buf); - strcpy(s->name, cfg.vdi); - sd_config_done(&cfg); + s->inode.vdi_size = opts->size; + backing_file = opts->backing_file; - s->inode.vdi_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), - BDRV_SECTOR_SIZE); - backing_file = qemu_opt_get_del(opts, BLOCK_OPT_BACKING_FILE); - buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC); - if (!buf || !strcmp(buf, "off")) { + if (!opts->has_preallocation) { + opts->preallocation = PREALLOC_MODE_OFF; + } + switch (opts->preallocation) { + case PREALLOC_MODE_OFF: prealloc = false; - } else if (!strcmp(buf, "full")) { + break; + case PREALLOC_MODE_FULL: prealloc = true; - } else { - error_setg(errp, "Invalid preallocation mode: '%s'", buf); + break; + default: + error_setg(errp, "Preallocation mode not supported for Sheepdog"); ret = -EINVAL; goto out; } - g_free(buf); - buf = qemu_opt_get_del(opts, BLOCK_OPT_REDUNDANCY); - if (buf) { - ret = parse_redundancy_str(s, buf); + if (opts->has_redundancy) { + ret = parse_redundancy(s, opts->redundancy); if (ret < 0) { - error_setg(errp, "Invalid redundancy mode: '%s'", buf); + error_setg(errp, "Invalid redundancy mode"); goto out; } } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn sd_co_create_opts(const char *filename, QemuOpts *opts, goto out; } - if (backing_file) { + if (opts->has_backing_file) { BlockBackend *blk; BDRVSheepdogState *base; BlockDriver *drv; /* Currently, only Sheepdog backing image is supported. */ - drv = bdrv_find_protocol(backing_file, true, NULL); + drv = bdrv_find_protocol(opts->backing_file, true, NULL); if (!drv || strcmp(drv->protocol_name, "sheepdog") != 0) { error_setg(errp, "backing_file must be a sheepdog image"); ret = -EINVAL; goto out; } - blk = blk_new_open(backing_file, NULL, NULL, + blk = blk_new_open(opts->backing_file, NULL, NULL, BDRV_O_PROTOCOL, errp); if (blk == NULL) { ret = -EIO; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn sd_co_create_opts(const char *filename, QemuOpts *opts, } if (prealloc) { - BlockDriverState *bs; - QDict *opts; - - opts = qdict_new(); - qdict_put_str(opts, "driver", "sheepdog"); - bs = bdrv_open(filename, NULL, opts, BDRV_O_PROTOCOL | BDRV_O_RDWR, - errp); - if (!bs) { - goto out; - } - - ret = sd_prealloc(bs, 0, s->inode.vdi_size, errp); - - bdrv_unref(bs); + ret = sd_create_prealloc(opts->location, opts->size, errp); } out: g_free(backing_file); g_free(buf); + g_free(s->addr); g_free(s); return ret; } +static int coroutine_fn sd_co_create_opts(const char *filename, QemuOpts *opts, + Error **errp) +{ + BlockdevCreateOptions *create_options = NULL; + QDict *qdict, *location_qdict; + QObject *crumpled; + Visitor *v; + const char *redundancy; + Error *local_err = NULL; + int ret; + + redundancy = qemu_opt_get_del(opts, BLOCK_OPT_REDUNDANCY); + + qdict = qemu_opts_to_qdict(opts, NULL); + qdict_put_str(qdict, "driver", "sheepdog"); + + location_qdict = qdict_new(); + qdict_put(qdict, "location", location_qdict); + + sd_parse_filename(filename, location_qdict, &local_err); + if (local_err) { + error_propagate(errp, local_err); + ret = -EINVAL; + goto fail; + } + + qdict_flatten(qdict); + + /* Change legacy command line options into QMP ones */ + static const QDictRenames opt_renames[] = { + { BLOCK_OPT_BACKING_FILE, "backing-file" }, + { BLOCK_OPT_OBJECT_SIZE, "object-size" }, + { NULL, NULL }, + }; + + if (!qdict_rename_keys(qdict, opt_renames, errp)) { + ret = -EINVAL; + goto fail; + } + + /* Get the QAPI object */ + crumpled = qdict_crumple(qdict, errp); + if (crumpled == NULL) { + ret = -EINVAL; + goto fail; + } + + v = qobject_input_visitor_new_keyval(crumpled); + visit_type_BlockdevCreateOptions(v, NULL, &create_options, &local_err); + visit_free(v); + qobject_decref(crumpled); + + if (local_err) { + error_propagate(errp, local_err); + ret = -EINVAL; + goto fail; + } + + assert(create_options->driver == BLOCKDEV_DRIVER_SHEEPDOG); + create_options->u.sheepdog.size = + ROUND_UP(create_options->u.sheepdog.size, BDRV_SECTOR_SIZE); + + if (redundancy) { + create_options->u.sheepdog.has_redundancy = true; + create_options->u.sheepdog.redundancy = + parse_redundancy_str(redundancy); + if (create_options->u.sheepdog.redundancy == NULL) { + error_setg(errp, "Invalid redundancy mode"); + ret = -EINVAL; + goto fail; + } + } + + ret = sd_co_create(create_options, errp); +fail: + qapi_free_BlockdevCreateOptions(create_options); + QDECREF(qdict); + return ret; +} + static void sd_close(BlockDriverState *bs) { Error *local_err = NULL; @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_sheepdog = { .bdrv_reopen_commit = sd_reopen_commit, .bdrv_reopen_abort = sd_reopen_abort, .bdrv_close = sd_close, + .bdrv_co_create = sd_co_create, .bdrv_co_create_opts = sd_co_create_opts, .bdrv_has_zero_init = bdrv_has_zero_init_1, .bdrv_getlength = sd_getlength, @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_sheepdog_tcp = { .bdrv_reopen_commit = sd_reopen_commit, .bdrv_reopen_abort = sd_reopen_abort, .bdrv_close = sd_close, + .bdrv_co_create = sd_co_create, .bdrv_co_create_opts = sd_co_create_opts, .bdrv_has_zero_init = bdrv_has_zero_init_1, .bdrv_getlength = sd_getlength, @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_sheepdog_unix = { .bdrv_reopen_commit = sd_reopen_commit, .bdrv_reopen_abort = sd_reopen_abort, .bdrv_close = sd_close, + .bdrv_co_create = sd_co_create, .bdrv_co_create_opts = sd_co_create_opts, .bdrv_has_zero_init = bdrv_has_zero_init_1, .bdrv_getlength = sd_getlength, -- 2.13.6 Create a BlockdevOptionsSsh object in connect_to_ssh() and take the options from there. 'host_key_check' is still processed separately because it's not in the schema yet. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/ssh.c | 137 +++++++++++++++++++++++++++--------------------------------- 1 file changed, 62 insertions(+), 75 deletions(-) diff --git a/block/ssh.c b/block/ssh.c index XXXXXXX..XXXXXXX 100644 --- a/block/ssh.c +++ b/block/ssh.c @@ -XXX,XX +XXX,XX @@ #include "qemu/sockets.h" #include "qemu/uri.h" #include "qapi/qapi-visit-sockets.h" +#include "qapi/qapi-visit-block-core.h" #include "qapi/qmp/qdict.h" #include "qapi/qmp/qstring.h" #include "qapi/qobject-input-visitor.h" @@ -XXX,XX +XXX,XX @@ static QemuOptsList ssh_runtime_opts = { .type = QEMU_OPT_NUMBER, .help = "Port to connect to", }, - { - .name = "path", - .type = QEMU_OPT_STRING, - .help = "Path of the image on the host", - }, - { - .name = "user", - .type = QEMU_OPT_STRING, - .help = "User as which to connect", - }, - { - .name = "host_key_check", - .type = QEMU_OPT_STRING, - .help = "Defines how and what to check the host key against", - }, { /* end of list */ } }, }; @@ -XXX,XX +XXX,XX @@ static bool ssh_process_legacy_socket_options(QDict *output_opts, return true; } -static InetSocketAddress *ssh_config(QDict *options, Error **errp) +static BlockdevOptionsSsh *ssh_parse_options(QDict *options, Error **errp) { - InetSocketAddress *inet = NULL; - QDict *addr = NULL; - QObject *crumpled_addr = NULL; - Visitor *iv = NULL; - Error *local_error = NULL; - - qdict_extract_subqdict(options, &addr, "server."); - if (!qdict_size(addr)) { - error_setg(errp, "SSH server address missing"); - goto out; + BlockdevOptionsSsh *result = NULL; + QemuOpts *opts = NULL; + Error *local_err = NULL; + QObject *crumpled; + const QDictEntry *e; + Visitor *v; + + /* Translate legacy options */ + opts = qemu_opts_create(&ssh_runtime_opts, NULL, 0, &error_abort); + qemu_opts_absorb_qdict(opts, options, &local_err); + if (local_err) { + error_propagate(errp, local_err); + goto fail; } - crumpled_addr = qdict_crumple(addr, errp); - if (!crumpled_addr) { - goto out; + if (!ssh_process_legacy_socket_options(options, opts, errp)) { + goto fail; + } + + /* Create the QAPI object */ + crumpled = qdict_crumple(options, errp); + if (crumpled == NULL) { + goto fail; } /* @@ -XXX,XX +XXX,XX @@ static InetSocketAddress *ssh_config(QDict *options, Error **errp) * but when they come from -drive, they're all QString. The * visitor expects the former. */ - iv = qobject_input_visitor_new(crumpled_addr); - visit_type_InetSocketAddress(iv, NULL, &inet, &local_error); - if (local_error) { - error_propagate(errp, local_error); - goto out; + v = qobject_input_visitor_new(crumpled); + visit_type_BlockdevOptionsSsh(v, NULL, &result, &local_err); + visit_free(v); + qobject_decref(crumpled); + + if (local_err) { + error_propagate(errp, local_err); + goto fail; } -out: - QDECREF(addr); - qobject_decref(crumpled_addr); - visit_free(iv); - return inet; + /* Remove the processed options from the QDict (the visitor processes + * _all_ options in the QDict) */ + while ((e = qdict_first(options))) { + qdict_del(options, e->key); + } + +fail: + qemu_opts_del(opts); + return result; } static int connect_to_ssh(BDRVSSHState *s, QDict *options, int ssh_flags, int creat_mode, Error **errp) { + BlockdevOptionsSsh *opts; int r, ret; - QemuOpts *opts = NULL; - Error *local_err = NULL; - const char *user, *path, *host_key_check; + const char *user, *host_key_check; long port = 0; - opts = qemu_opts_create(&ssh_runtime_opts, NULL, 0, &error_abort); - qemu_opts_absorb_qdict(opts, options, &local_err); - if (local_err) { - ret = -EINVAL; - error_propagate(errp, local_err); - goto err; - } - - if (!ssh_process_legacy_socket_options(options, opts, errp)) { - ret = -EINVAL; - goto err; + host_key_check = qdict_get_try_str(options, "host_key_check"); + if (!host_key_check) { + host_key_check = "yes"; + } else { + qdict_del(options, "host_key_check"); } - path = qemu_opt_get(opts, "path"); - if (!path) { - ret = -EINVAL; - error_setg(errp, "No path was specified"); - goto err; + opts = ssh_parse_options(options, errp); + if (opts == NULL) { + return -EINVAL; } - user = qemu_opt_get(opts, "user"); - if (!user) { + if (opts->has_user) { + user = opts->user; + } else { user = g_get_user_name(); if (!user) { error_setg_errno(errp, errno, "Can't get user name"); @@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, QDict *options, } } - host_key_check = qemu_opt_get(opts, "host_key_check"); - if (!host_key_check) { - host_key_check = "yes"; - } - /* Pop the config into our state object, Exit if invalid */ - s->inet = ssh_config(options, errp); - if (!s->inet) { - ret = -EINVAL; - goto err; - } + s->inet = opts->server; + opts->server = NULL; if (qemu_strtol(s->inet->port, NULL, 10, &port) < 0) { error_setg(errp, "Use only numeric port value"); @@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, QDict *options, /* Open the remote file. */ DPRINTF("opening file %s flags=0x%x creat_mode=0%o", - path, ssh_flags, creat_mode); - s->sftp_handle = libssh2_sftp_open(s->sftp, path, ssh_flags, creat_mode); + opts->path, ssh_flags, creat_mode); + s->sftp_handle = libssh2_sftp_open(s->sftp, opts->path, ssh_flags, + creat_mode); if (!s->sftp_handle) { - session_error_setg(errp, s, "failed to open remote file '%s'", path); + session_error_setg(errp, s, "failed to open remote file '%s'", + opts->path); ret = -EINVAL; goto err; } - qemu_opts_del(opts); + qapi_free_BlockdevOptionsSsh(opts); r = libssh2_sftp_fstat(s->sftp_handle, &s->attrs); if (r < 0) { @@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, QDict *options, } s->session = NULL; - qemu_opts_del(opts); + qapi_free_BlockdevOptionsSsh(opts); return ret; } -- 2.13.6 This makes the host-key-check option available in blockdev-add. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 63 +++++++++++++++++++++++++++++++++++-- block/ssh.c | 88 +++++++++++++++++++++++++++++++++------------------- 2 files changed, 117 insertions(+), 34 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ '*encrypt': 'BlockdevQcow2Encryption' } } ## +# @SshHostKeyCheckMode: +# +# @none Don't check the host key at all +# @hash Compare the host key with a given hash +# @known_hosts Check the host key against the known_hosts file +# +# Since: 2.12 +## +{ 'enum': 'SshHostKeyCheckMode', + 'data': [ 'none', 'hash', 'known_hosts' ] } + +## +# @SshHostKeyCheckHashType: +# +# @md5 The given hash is an md5 hash +# @sha1 The given hash is an sha1 hash +# +# Since: 2.12 +## +{ 'enum': 'SshHostKeyCheckHashType', + 'data': [ 'md5', 'sha1' ] } + +## +# @SshHostKeyHash: +# +# @type The hash algorithm used for the hash +# @hash The expected hash value +# +# Since: 2.12 +## +{ 'struct': 'SshHostKeyHash', + 'data': { 'type': 'SshHostKeyCheckHashType', + 'hash': 'str' }} + +## +# @SshHostKeyDummy: +# +# For those union branches that don't need additional fields. +# +# Since: 2.12 +## +{ 'struct': 'SshHostKeyDummy', + 'data': {} } + +## +# @SshHostKeyCheck: +# +# Since: 2.12 +## +{ 'union': 'SshHostKeyCheck', + 'base': { 'mode': 'SshHostKeyCheckMode' }, + 'discriminator': 'mode', + 'data': { 'none': 'SshHostKeyDummy', + 'hash': 'SshHostKeyHash', + 'known_hosts': 'SshHostKeyDummy' } } + +## # @BlockdevOptionsSsh: # # @server: host address @@ -XXX,XX +XXX,XX @@ # @user: user as which to connect, defaults to current # local user name # -# TODO: Expose the host_key_check option in QMP +# @host-key-check: Defines how and what to check the host key against +# (default: known_hosts) # # Since: 2.9 ## { 'struct': 'BlockdevOptionsSsh', 'data': { 'server': 'InetSocketAddress', 'path': 'str', - '*user': 'str' } } + '*user': 'str', + '*host-key-check': 'SshHostKeyCheck' } } ## diff --git a/block/ssh.c b/block/ssh.c index XXXXXXX..XXXXXXX 100644 --- a/block/ssh.c +++ b/block/ssh.c @@ -XXX,XX +XXX,XX @@ check_host_key_hash(BDRVSSHState *s, const char *hash, } static int check_host_key(BDRVSSHState *s, const char *host, int port, - const char *host_key_check, Error **errp) + SshHostKeyCheck *hkc, Error **errp) { - /* host_key_check=no */ - if (strcmp(host_key_check, "no") == 0) { - return 0; - } + SshHostKeyCheckMode mode; - /* host_key_check=md5:xx:yy:zz:... */ - if (strncmp(host_key_check, "md5:", 4) == 0) { - return check_host_key_hash(s, &host_key_check[4], - LIBSSH2_HOSTKEY_HASH_MD5, 16, errp); - } - - /* host_key_check=sha1:xx:yy:zz:... */ - if (strncmp(host_key_check, "sha1:", 5) == 0) { - return check_host_key_hash(s, &host_key_check[5], - LIBSSH2_HOSTKEY_HASH_SHA1, 20, errp); + if (hkc) { + mode = hkc->mode; + } else { + mode = SSH_HOST_KEY_CHECK_MODE_KNOWN_HOSTS; } - /* host_key_check=yes */ - if (strcmp(host_key_check, "yes") == 0) { + switch (mode) { + case SSH_HOST_KEY_CHECK_MODE_NONE: + return 0; + case SSH_HOST_KEY_CHECK_MODE_HASH: + if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_MD5) { + return check_host_key_hash(s, hkc->u.hash.hash, + LIBSSH2_HOSTKEY_HASH_MD5, 16, errp); + } else if (hkc->u.hash.type == SSH_HOST_KEY_CHECK_HASH_TYPE_SHA1) { + return check_host_key_hash(s, hkc->u.hash.hash, + LIBSSH2_HOSTKEY_HASH_SHA1, 20, errp); + } + g_assert_not_reached(); + break; + case SSH_HOST_KEY_CHECK_MODE_KNOWN_HOSTS: return check_host_key_knownhosts(s, host, port, errp); + default: + g_assert_not_reached(); } - error_setg(errp, "unknown host_key_check setting (%s)", host_key_check); return -EINVAL; } @@ -XXX,XX +XXX,XX @@ static QemuOptsList ssh_runtime_opts = { .type = QEMU_OPT_NUMBER, .help = "Port to connect to", }, + { + .name = "host_key_check", + .type = QEMU_OPT_STRING, + .help = "Defines how and what to check the host key against", + }, { /* end of list */ } }, }; -static bool ssh_process_legacy_socket_options(QDict *output_opts, - QemuOpts *legacy_opts, - Error **errp) +static bool ssh_process_legacy_options(QDict *output_opts, + QemuOpts *legacy_opts, + Error **errp) { const char *host = qemu_opt_get(legacy_opts, "host"); const char *port = qemu_opt_get(legacy_opts, "port"); + const char *host_key_check = qemu_opt_get(legacy_opts, "host_key_check"); if (!host && port) { error_setg(errp, "port may not be used without host"); @@ -XXX,XX +XXX,XX @@ static bool ssh_process_legacy_socket_options(QDict *output_opts, qdict_put_str(output_opts, "server.port", port ?: stringify(22)); } + if (host_key_check) { + if (strcmp(host_key_check, "no") == 0) { + qdict_put_str(output_opts, "host-key-check.mode", "none"); + } else if (strncmp(host_key_check, "md5:", 4) == 0) { + qdict_put_str(output_opts, "host-key-check.mode", "hash"); + qdict_put_str(output_opts, "host-key-check.type", "md5"); + qdict_put_str(output_opts, "host-key-check.hash", + &host_key_check[4]); + } else if (strncmp(host_key_check, "sha1:", 5) == 0) { + qdict_put_str(output_opts, "host-key-check.mode", "hash"); + qdict_put_str(output_opts, "host-key-check.type", "sha1"); + qdict_put_str(output_opts, "host-key-check.hash", + &host_key_check[5]); + } else if (strcmp(host_key_check, "yes") == 0) { + qdict_put_str(output_opts, "host-key-check.mode", "known_hosts"); + } else { + error_setg(errp, "unknown host_key_check setting (%s)", + host_key_check); + return false; + } + } + return true; } @@ -XXX,XX +XXX,XX @@ static BlockdevOptionsSsh *ssh_parse_options(QDict *options, Error **errp) goto fail; } - if (!ssh_process_legacy_socket_options(options, opts, errp)) { + if (!ssh_process_legacy_options(options, opts, errp)) { goto fail; } @@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, QDict *options, { BlockdevOptionsSsh *opts; int r, ret; - const char *user, *host_key_check; + const char *user; long port = 0; - host_key_check = qdict_get_try_str(options, "host_key_check"); - if (!host_key_check) { - host_key_check = "yes"; - } else { - qdict_del(options, "host_key_check"); - } - opts = ssh_parse_options(options, errp); if (opts == NULL) { return -EINVAL; @@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, QDict *options, } /* Check the remote host's key against known_hosts. */ - ret = check_host_key(s, s->inet->host, port, host_key_check, - errp); + ret = check_host_key(s, s->inet->host, port, opts->host_key_check, errp); if (ret < 0) { goto err; } -- 2.13.6 Move the parsing of the QDict options up to the callers, in preparation for the .bdrv_co_create implementation that directly gets a QAPI type. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- block/ssh.c | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/block/ssh.c b/block/ssh.c index XXXXXXX..XXXXXXX 100644 --- a/block/ssh.c +++ b/block/ssh.c @@ -XXX,XX +XXX,XX @@ fail: return result; } -static int connect_to_ssh(BDRVSSHState *s, QDict *options, +static int connect_to_ssh(BDRVSSHState *s, BlockdevOptionsSsh *opts, int ssh_flags, int creat_mode, Error **errp) { - BlockdevOptionsSsh *opts; int r, ret; const char *user; long port = 0; - opts = ssh_parse_options(options, errp); - if (opts == NULL) { - return -EINVAL; - } - if (opts->has_user) { user = opts->user; } else { @@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, QDict *options, goto err; } - qapi_free_BlockdevOptionsSsh(opts); - r = libssh2_sftp_fstat(s->sftp_handle, &s->attrs); if (r < 0) { sftp_error_setg(errp, s, "failed to read file attributes"); @@ -XXX,XX +XXX,XX @@ static int connect_to_ssh(BDRVSSHState *s, QDict *options, } s->session = NULL; - qapi_free_BlockdevOptionsSsh(opts); - return ret; } @@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags, Error **errp) { BDRVSSHState *s = bs->opaque; + BlockdevOptionsSsh *opts; int ret; int ssh_flags; @@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags, ssh_flags |= LIBSSH2_FXF_WRITE; } + opts = ssh_parse_options(options, errp); + if (opts == NULL) { + return -EINVAL; + } + /* Start up SSH. */ - ret = connect_to_ssh(s, options, ssh_flags, 0, errp); + ret = connect_to_ssh(s, opts, ssh_flags, 0, errp); if (ret < 0) { goto err; } @@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags, /* Go non-blocking. */ libssh2_session_set_blocking(s->session, 0); + qapi_free_BlockdevOptionsSsh(opts); + return 0; err: @@ -XXX,XX +XXX,XX @@ static int ssh_file_open(BlockDriverState *bs, QDict *options, int bdrv_flags, } s->sock = -1; + qapi_free_BlockdevOptionsSsh(opts); + return ret; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_create_opts(const char *filename, QemuOpts *opts, int r, ret; int64_t total_size = 0; QDict *uri_options = NULL; + BlockdevOptionsSsh *ssh_opts = NULL; BDRVSSHState s; ssh_state_init(&s); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_create_opts(const char *filename, QemuOpts *opts, goto out; } - r = connect_to_ssh(&s, uri_options, + ssh_opts = ssh_parse_options(uri_options, errp); + if (ssh_opts == NULL) { + ret = -EINVAL; + goto out; + } + + r = connect_to_ssh(&s, ssh_opts, LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE| LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC, 0644, errp); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn ssh_co_create_opts(const char *filename, QemuOpts *opts, if (uri_options != NULL) { QDECREF(uri_options); } + qapi_free_BlockdevOptionsSsh(ssh_opts); return ret; } -- 2.13.6 This adds the .bdrv_co_create driver callback to ssh, which enables image creation over QMP. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- qapi/block-core.json | 16 +++++++++- block/ssh.c | 83 ++++++++++++++++++++++++++++++---------------------- 2 files changed, 63 insertions(+), 36 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ '*object-size': 'size' } } ## +# @BlockdevCreateOptionsSsh: +# +# Driver specific image creation options for SSH. +# +# @location Where to store the new image file +# @size Size of the virtual disk in bytes +# +# Since: 2.12 +## +{ 'struct': 'BlockdevCreateOptionsSsh', + 'data': { 'location': 'BlockdevOptionsSsh', + 'size': 'size' } } + +## # @BlockdevCreateNotSupported: # # This is used for all drivers that don't support creating images. @@ -XXX,XX +XXX,XX @@ 'rbd': 'BlockdevCreateOptionsRbd', 'replication': 'BlockdevCreateNotSupported', 'sheepdog': 'BlockdevCreateOptionsSheepdog', - 'ssh': 'BlockdevCreateNotSupported', + 'ssh': 'BlockdevCreateOptionsSsh', 'throttle': 'BlockdevCreateNotSupported', 'vdi': 'BlockdevCreateNotSupported', 'vhdx': 'BlockdevCreateNotSupported', diff --git a/block/ssh.c b/block/ssh.c index XXXXXXX..XXXXXXX 100644 --- a/block/ssh.c +++ b/block/ssh.c @@ -XXX,XX +XXX,XX @@ static QemuOptsList ssh_create_opts = { } }; +static int ssh_co_create(BlockdevCreateOptions *options, Error **errp) +{ + BlockdevCreateOptionsSsh *opts = &options->u.ssh; + BDRVSSHState s; + int ret; + + assert(options->driver == BLOCKDEV_DRIVER_SSH); + + ssh_state_init(&s); + + ret = connect_to_ssh(&s, opts->location, + LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE| + LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC, + 0644, errp); + if (ret < 0) { + goto fail; + } + + if (opts->size > 0) { + ret = ssh_grow_file(&s, opts->size, errp); + if (ret < 0) { + goto fail; + } + } + + ret = 0; +fail: + ssh_state_free(&s); + return ret; +} + static int coroutine_fn ssh_co_create_opts(const char *filename, QemuOpts *opts, Error **errp) { - int r, ret; - int64_t total_size = 0; + BlockdevCreateOptions *create_options; + BlockdevCreateOptionsSsh *ssh_opts; + int ret; QDict *uri_options = NULL; - BlockdevOptionsSsh *ssh_opts = NULL; - BDRVSSHState s; - ssh_state_init(&s); + create_options = g_new0(BlockdevCreateOptions, 1); + create_options->driver = BLOCKDEV_DRIVER_SSH; + ssh_opts = &create_options->u.ssh; /* Get desired file size. */ - total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), - BDRV_SECTOR_SIZE); - DPRINTF("total_size=%" PRIi64, total_size); + ssh_opts->size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0), + BDRV_SECTOR_SIZE); + DPRINTF("total_size=%" PRIi64, ssh_opts->size); uri_options = qdict_new(); - r = parse_uri(filename, uri_options, errp); - if (r < 0) { - ret = r; + ret = parse_uri(filename, uri_options, errp); + if (ret < 0) { goto out; } - ssh_opts = ssh_parse_options(uri_options, errp); - if (ssh_opts == NULL) { + ssh_opts->location = ssh_parse_options(uri_options, errp); + if (ssh_opts->location == NULL) { ret = -EINVAL; goto out; } - r = connect_to_ssh(&s, ssh_opts, - LIBSSH2_FXF_READ|LIBSSH2_FXF_WRITE| - LIBSSH2_FXF_CREAT|LIBSSH2_FXF_TRUNC, - 0644, errp); - if (r < 0) { - ret = r; - goto out; - } - - if (total_size > 0) { - ret = ssh_grow_file(&s, total_size, errp); - if (ret < 0) { - goto out; - } - } - - ret = 0; + ret = ssh_co_create(create_options, errp); out: - ssh_state_free(&s); - if (uri_options != NULL) { - QDECREF(uri_options); - } - qapi_free_BlockdevOptionsSsh(ssh_opts); + QDECREF(uri_options); + qapi_free_BlockdevCreateOptions(create_options); return ret; } @@ -XXX,XX +XXX,XX @@ static BlockDriver bdrv_ssh = { .instance_size = sizeof(BDRVSSHState), .bdrv_parse_filename = ssh_parse_filename, .bdrv_file_open = ssh_file_open, + .bdrv_co_create = ssh_co_create, .bdrv_co_create_opts = ssh_co_create_opts, .bdrv_close = ssh_close, .bdrv_has_zero_init = ssh_has_zero_init, -- 2.13.6 If bdrv_truncate() is called, but the requested size is the same as before, don't call posix_fallocate(), which returns -EINVAL for length zero and would therefore make bdrv_truncate() fail. The problem can be triggered by creating a zero-sized raw image with 'falloc' preallocation mode. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- block/file-posix.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/block/file-posix.c b/block/file-posix.c index XXXXXXX..XXXXXXX 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -XXX,XX +XXX,XX @@ static int raw_regular_truncate(int fd, int64_t offset, PreallocMode prealloc, * file systems that do not support fallocate(), trying to check if a * block is allocated before allocating it, so don't do that here. */ - result = -posix_fallocate(fd, current_length, offset - current_length); - if (result != 0) { - /* posix_fallocate() doesn't set errno. */ - error_setg_errno(errp, -result, - "Could not preallocate new data"); + if (offset != current_length) { + result = -posix_fallocate(fd, current_length, offset - current_length); + if (result != 0) { + /* posix_fallocate() doesn't set errno. */ + error_setg_errno(errp, -result, + "Could not preallocate new data"); + } + } else { + result = 0; } goto out; #endif -- 2.13.6 Most callers have their own checks, but something like this should also be checked centrally. As it happens, x-blockdev-create can pass negative image sizes to format drivers (because there is no QAPI type that would reject negative numbers) and triggers the check added by this patch. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- block.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ int bdrv_truncate(BdrvChild *child, int64_t offset, PreallocMode prealloc, error_setg(errp, "No medium inserted"); return -ENOMEDIUM; } + if (offset < 0) { + error_setg(errp, "Image size cannot be negative"); + return -EINVAL; + } + if (!drv->bdrv_truncate) { if (bs->file && drv->is_filter) { return bdrv_truncate(bs->file, offset, prealloc, errp); -- 2.13.6 Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/206 | 436 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/206.out | 209 ++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 646 insertions(+) create mode 100755 tests/qemu-iotests/206 create mode 100644 tests/qemu-iotests/206.out diff --git a/tests/qemu-iotests/206 b/tests/qemu-iotests/206 new file mode 100755 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/206 @@ -XXX,XX +XXX,XX @@ +#!/bin/bash +# +# Test qcow2 and file image creation +# +# Copyright (C) 2018 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator +owner=kwolf@redhat.com + +seq=`basename $0` +echo "QA output created by $seq" + +here=`pwd` +status=1 # failure is the default! + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +function do_run_qemu() +{ + echo Testing: "$@" + $QEMU -nographic -qmp stdio -serial none "$@" + echo +} + +function run_qemu() +{ + do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qmp \ + | _filter_qemu | _filter_imgfmt \ + | _filter_actual_image_size +} + +echo +echo "=== Successful image creation (defaults) ===" +echo + +size=$((128 * 1024 * 1024)) + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "file", + "filename": "$TEST_IMG", + "size": 0 + } +} +{ "execute": "blockdev-add", + "arguments": { + "driver": "file", + "node-name": "imgfile", + "filename": "$TEST_IMG" + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "imgfile", + "size": $size + } +} +{ "execute": "quit" } +EOF + +_img_info --format-specific + +echo +echo "=== Successful image creation (inline blockdev-add, explicit defaults) ===" +echo + +# Choose a different size to show that we got a new image +size=$((64 * 1024 * 1024)) + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "file", + "filename": "$TEST_IMG", + "size": 0, + "preallocation": "off", + "nocow": false + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": { + "driver": "file", + "filename": "$TEST_IMG" + }, + "size": $size, + "version": "v3", + "cluster-size": 65536, + "preallocation": "off", + "lazy-refcounts": false, + "refcount-bits": 16 + } +} +{ "execute": "quit" } +EOF + +_img_info --format-specific + +echo +echo "=== Successful image creation (v3 non-default options) ===" +echo + +# Choose a different size to show that we got a new image +size=$((32 * 1024 * 1024)) + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "file", + "filename": "$TEST_IMG", + "size": 0, + "preallocation": "falloc", + "nocow": true + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": { + "driver": "file", + "filename": "$TEST_IMG" + }, + "size": $size, + "version": "v3", + "cluster-size": 2097152, + "preallocation": "metadata", + "lazy-refcounts": true, + "refcount-bits": 1 + } +} +{ "execute": "quit" } +EOF + +_img_info --format-specific + +echo +echo "=== Successful image creation (v2 non-default options) ===" +echo + +mv $TEST_IMG $TEST_IMG.base + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "file", + "filename": "$TEST_IMG", + "size": 0 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": { + "driver": "file", + "filename": "$TEST_IMG" + }, + "size": $size, + "backing-file": "$TEST_IMG.base", + "backing-fmt": "qcow2", + "version": "v2", + "cluster-size": 512 + } +} +{ "execute": "quit" } +EOF + +_img_info --format-specific + +echo +echo "=== Successful image creation (encrypted) ===" +echo + +run_qemu -object secret,id=keysec0,data="foo" <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": { + "driver": "file", + "filename": "$TEST_IMG" + }, + "size": $size, + "encrypt": { + "format": "luks", + "key-secret": "keysec0", + "cipher-alg": "twofish-128", + "cipher-mode": "ctr", + "ivgen-alg": "plain64", + "ivgen-hash-alg": "md5", + "hash-alg": "sha1", + "iter-time": 10 + } + } +} +{ "execute": "quit" } +EOF + +_img_info --format-specific | _filter_img_info --format-specific + +echo +echo "=== Invalid BlockdevRef ===" +echo + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "this doesn't exist", + "size": $size + } +} +{ "execute": "quit" } +EOF + + +echo +echo "=== Invalid sizes ===" +echo + +# TODO Negative image sizes aren't handled correctly, but this is a problem +# with QAPI's implementation of the 'size' type and affects other commands as +# well. Once this is fixed, we may want to add a test case here. + +# 1. Misaligned image size +# 2. 2^64 - 512 +# 3. 2^63 = 8 EB (qemu-img enforces image sizes less than this) +# 4. 2^63 - 512 (generally valid, but qcow2 can't handle images this size) + +run_qemu -blockdev driver=file,filename="$TEST_IMG",node-name=node0 <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 1234 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 18446744073709551104 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 9223372036854775808 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 9223372036854775296 + } +} +{ "execute": "quit" } +EOF + +echo +echo "=== Invalid version ===" +echo + +run_qemu -blockdev driver=file,filename="$TEST_IMG",node-name=node0 <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "version": "v1" + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "version": "v2", + "lazy-refcounts": true + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "version": "v2", + "refcount-bits": 8 + } +} +{ "execute": "quit" } +EOF + +echo +echo "=== Invalid backing file options ===" +echo + +run_qemu -blockdev driver=file,filename="$TEST_IMG",node-name=node0 <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "backing-file": "/dev/null", + "preallocation": "full" + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "backing-fmt": "$IMGFMT" + } +} +{ "execute": "quit" } +EOF + +echo +echo "=== Invalid cluster size ===" +echo + +run_qemu -blockdev driver=file,filename="$TEST_IMG",node-name=node0 <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "cluster-size": 1234 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "cluster-size": 128 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "cluster-size": 4194304 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "cluster-size": 0 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 281474976710656, + "cluster-size": 512 + } +} +{ "execute": "quit" } +EOF + +echo +echo "=== Invalid refcount width ===" +echo + +run_qemu -blockdev driver=file,filename="$TEST_IMG",node-name=node0 <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "refcount-bits": 128 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "refcount-bits": 0 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "$IMGFMT", + "file": "node0", + "size": 67108864, + "refcount-bits": 7 + } +} +{ "execute": "quit" } +EOF + +# success, all done +echo "*** done" +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/206.out b/tests/qemu-iotests/206.out new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/206.out @@ -XXX,XX +XXX,XX @@ +QA output created by 206 + +=== Successful image creation (defaults) === + +Testing: +QMP_VERSION +{"return": {}} +{"return": {}} +{"return": {}} +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 128M (134217728 bytes) +cluster_size: 65536 +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=== Successful image creation (inline blockdev-add, explicit defaults) === + +Testing: +QMP_VERSION +{"return": {}} +{"return": {}} +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 64M (67108864 bytes) +cluster_size: 65536 +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false + +=== Successful image creation (v3 non-default options) === + +Testing: +QMP_VERSION +{"return": {}} +{"return": {}} +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 32M (33554432 bytes) +cluster_size: 2097152 +Format specific information: + compat: 1.1 + lazy refcounts: true + refcount bits: 1 + corrupt: false + +=== Successful image creation (v2 non-default options) === + +Testing: +QMP_VERSION +{"return": {}} +{"return": {}} +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 32M (33554432 bytes) +cluster_size: 512 +backing file: TEST_DIR/t.IMGFMT.base +backing file format: IMGFMT +Format specific information: + compat: 0.10 + refcount bits: 16 + +=== Successful image creation (encrypted) === + +Testing: -object secret,id=keysec0,data=foo +QMP_VERSION +{"return": {}} +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 32M (33554432 bytes) +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + encrypt: + ivgen alg: plain64 + hash alg: sha1 + cipher alg: twofish-128 + uuid: 00000000-0000-0000-0000-000000000000 + format: luks + cipher mode: ctr + slots: + [0]: + active: true + iters: 1024 + key offset: 4096 + stripes: 4000 + [1]: + active: false + key offset: 69632 + [2]: + active: false + key offset: 135168 + [3]: + active: false + key offset: 200704 + [4]: + active: false + key offset: 266240 + [5]: + active: false + key offset: 331776 + [6]: + active: false + key offset: 397312 + [7]: + active: false + key offset: 462848 + payload offset: 528384 + master key iters: 1024 + corrupt: false + +=== Invalid BlockdevRef === + +Testing: +QMP_VERSION +{"return": {}} +{"error": {"class": "GenericError", "desc": "Cannot find device=this doesn't exist nor node_name=this doesn't exist"}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + + +=== Invalid sizes === + +Testing: -blockdev driver=file,filename=TEST_DIR/t.IMGFMT,node-name=node0 +QMP_VERSION +{"return": {}} +{"error": {"class": "GenericError", "desc": "Image size must be a multiple of 512 bytes"}} +{"error": {"class": "GenericError", "desc": "Could not resize image: Image size cannot be negative"}} +{"error": {"class": "GenericError", "desc": "Could not resize image: Image size cannot be negative"}} +{"error": {"class": "GenericError", "desc": "Could not resize image: Failed to grow the L1 table: File too large"}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + + +=== Invalid version === + +Testing: -blockdev driver=file,filename=TEST_DIR/t.IMGFMT,node-name=node0 +QMP_VERSION +{"return": {}} +{"error": {"class": "GenericError", "desc": "Invalid parameter 'v1'"}} +{"error": {"class": "GenericError", "desc": "Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater)"}} +{"error": {"class": "GenericError", "desc": "Different refcount widths than 16 bits require compatibility level 1.1 or above (use version=v3 or greater)"}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + + +=== Invalid backing file options === + +Testing: -blockdev driver=file,filename=TEST_DIR/t.IMGFMT,node-name=node0 +QMP_VERSION +{"return": {}} +{"error": {"class": "GenericError", "desc": "Backing file and preallocation cannot be used at the same time"}} +{"error": {"class": "GenericError", "desc": "Backing format cannot be used without backing file"}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + + +=== Invalid cluster size === + +Testing: -blockdev driver=file,filename=TEST_DIR/t.IMGFMT,node-name=node0 +QMP_VERSION +{"return": {}} +{"error": {"class": "GenericError", "desc": "Cluster size must be a power of two between 512 and 2048k"}} +{"error": {"class": "GenericError", "desc": "Cluster size must be a power of two between 512 and 2048k"}} +{"error": {"class": "GenericError", "desc": "Cluster size must be a power of two between 512 and 2048k"}} +{"error": {"class": "GenericError", "desc": "Cluster size must be a power of two between 512 and 2048k"}} +{"error": {"class": "GenericError", "desc": "Could not resize image: Failed to grow the L1 table: File too large"}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + + +=== Invalid refcount width === + +Testing: -blockdev driver=file,filename=TEST_DIR/t.IMGFMT,node-name=node0 +QMP_VERSION +{"return": {}} +{"error": {"class": "GenericError", "desc": "Refcount width must be a power of two and may not exceed 64 bits"}} +{"error": {"class": "GenericError", "desc": "Refcount width must be a power of two and may not exceed 64 bits"}} +{"error": {"class": "GenericError", "desc": "Refcount width must be a power of two and may not exceed 64 bits"}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -XXX,XX +XXX,XX @@ 203 rw auto 204 rw auto quick 205 rw auto quick +206 rw auto -- 2.13.6 Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/207 | 261 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/207.out | 75 +++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 337 insertions(+) create mode 100755 tests/qemu-iotests/207 create mode 100644 tests/qemu-iotests/207.out diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207 new file mode 100755 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/207 @@ -XXX,XX +XXX,XX @@ +#!/bin/bash +# +# Test ssh image creation +# +# Copyright (C) 2018 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator +owner=kwolf@redhat.com + +seq=`basename $0` +echo "QA output created by $seq" + +here=`pwd` +status=1 # failure is the default! + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt raw +_supported_proto ssh +_supported_os Linux + +function do_run_qemu() +{ + echo Testing: "$@" + $QEMU -nographic -qmp stdio -serial none "$@" + echo +} + +function run_qemu() +{ + do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qmp \ + | _filter_qemu | _filter_imgfmt \ + | _filter_actual_image_size +} + +echo +echo "=== Successful image creation (defaults) ===" +echo + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "ssh", + "location": { + "path": "$TEST_IMG_FILE", + "server": { + "host": "127.0.0.1", + "port": "22" + } + }, + "size": 4194304 + } +} +{ "execute": "quit" } +EOF + +_img_info | _filter_img_info +echo +TEST_IMG=$TEST_IMG_FILE _img_info | _filter_img_info + +echo +echo "=== Test host-key-check options ===" +echo + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "ssh", + "location": { + "path": "$TEST_IMG_FILE", + "server": { + "host": "127.0.0.1", + "port": "22" + }, + "host-key-check": { + "mode": "none" + } + }, + "size": 8388608 + } +} +{ "execute": "quit" } +EOF + +_img_info | _filter_img_info + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "ssh", + "location": { + "path": "$TEST_IMG_FILE", + "server": { + "host": "127.0.0.1", + "port": "22" + }, + "host-key-check": { + "mode": "known_hosts" + } + }, + "size": 4194304 + } +} +{ "execute": "quit" } +EOF + +_img_info | _filter_img_info + + +key=$(ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | + cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1) + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "ssh", + "location": { + "path": "$TEST_IMG_FILE", + "server": { + "host": "127.0.0.1", + "port": "22" + }, + "host-key-check": { + "mode": "hash", + "type": "md5", + "hash": "wrong" + } + }, + "size": 8388608 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "ssh", + "location": { + "path": "$TEST_IMG_FILE", + "server": { + "host": "127.0.0.1", + "port": "22" + }, + "host-key-check": { + "mode": "hash", + "type": "md5", + "hash": "$key" + } + }, + "size": 8388608 + } +} +{ "execute": "quit" } +EOF + +_img_info | _filter_img_info + + +key=$(ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | + cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1) + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "ssh", + "location": { + "path": "$TEST_IMG_FILE", + "server": { + "host": "127.0.0.1", + "port": "22" + }, + "host-key-check": { + "mode": "hash", + "type": "sha1", + "hash": "wrong" + } + }, + "size": 4194304 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "ssh", + "location": { + "path": "$TEST_IMG_FILE", + "server": { + "host": "127.0.0.1", + "port": "22" + }, + "host-key-check": { + "mode": "hash", + "type": "sha1", + "hash": "$key" + } + }, + "size": 4194304 + } +} +{ "execute": "quit" } +EOF + +_img_info | _filter_img_info + +echo +echo "=== Invalid path and user ===" +echo + +run_qemu <<EOF +{ "execute": "qmp_capabilities" } +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "ssh", + "location": { + "path": "/this/is/not/an/existing/path", + "server": { + "host": "127.0.0.1", + "port": "22" + } + }, + "size": 4194304 + } +} +{ "execute": "x-blockdev-create", + "arguments": { + "driver": "ssh", + "location": { + "path": "$TEST_IMG_FILE", + "user": "invalid user", + "server": { + "host": "127.0.0.1", + "port": "22" + } + }, + "size": 4194304 + } +} +{ "execute": "quit" } +EOF + +# success, all done +echo "*** done" +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/207.out @@ -XXX,XX +XXX,XX @@ +QA output created by 207 + +=== Successful image creation (defaults) === + +Testing: +QMP_VERSION +{"return": {}} +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}} +file format: IMGFMT +virtual size: 4.0M (4194304 bytes) + +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 4.0M (4194304 bytes) + +=== Test host-key-check options === + +Testing: +QMP_VERSION +{"return": {}} +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}} +file format: IMGFMT +virtual size: 8.0M (8388608 bytes) +Testing: +QMP_VERSION +{"return": {}} +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}} +file format: IMGFMT +virtual size: 4.0M (4194304 bytes) +Testing: +QMP_VERSION +{"return": {}} +{"error": {"class": "GenericError", "desc": "remote host key does not match host_key_check 'wrong'"}} +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}} +file format: IMGFMT +virtual size: 8.0M (8388608 bytes) +Testing: +QMP_VERSION +{"return": {}} +{"error": {"class": "GenericError", "desc": "remote host key does not match host_key_check 'wrong'"}} +{"return": {}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_DIR/t.IMGFMT"}} +file format: IMGFMT +virtual size: 4.0M (4194304 bytes) + +=== Invalid path and user === + +Testing: +QMP_VERSION +{"return": {}} +{"error": {"class": "GenericError", "desc": "failed to open remote file '/this/is/not/an/existing/path': Failed opening remote file (libssh2 error code: -31)"}} +{"error": {"class": "GenericError", "desc": "failed to authenticate using publickey authentication and the identities held by your ssh-agent"}} +{"return": {}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "SHUTDOWN", "data": {"guest": false}} + +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -XXX,XX +XXX,XX @@ 204 rw auto quick 205 rw auto quick 206 rw auto +207 rw auto -- 2.13.6 From: Fam Zheng <famz@redhat.com> Signed-off-by: Fam Zheng <famz@redhat.com> Message-id: 20171225025107.23985-1-famz@redhat.com Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/153 | 8 +++++--- tests/qemu-iotests/153.out | 7 ++++--- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/153 b/tests/qemu-iotests/153 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/153 +++ b/tests/qemu-iotests/153 @@ -XXX,XX +XXX,XX @@ _cleanup() { _cleanup_test_img rm -f "${TEST_IMG}.base" + rm -f "${TEST_IMG}.overlay" rm -f "${TEST_IMG}.convert" rm -f "${TEST_IMG}.a" rm -f "${TEST_IMG}.b" @@ -XXX,XX +XXX,XX @@ rm -f "${TEST_IMG}.lnk" &>/dev/null ln -s ${TEST_IMG} "${TEST_IMG}.lnk" || echo "Failed to create link" _run_qemu_with_images "${TEST_IMG}.lnk" "${TEST_IMG}" -echo -echo "== Closing an image should unlock it ==" _launch_qemu _send_qemu_cmd $QEMU_HANDLE \ @@ -XXX,XX +XXX,XX @@ _send_qemu_cmd $QEMU_HANDLE \ _run_cmd $QEMU_IO "${TEST_IMG}" -c 'write 0 512' -echo "Closing drive" +echo "Creating overlay with qemu-img when the guest is running should be allowed" +_run_cmd $QEMU_IMG create -f $IMGFMT -b "${TEST_IMG}" "${TEST_IMG}.overlay" + +echo "== Closing an image should unlock it ==" _send_qemu_cmd $QEMU_HANDLE \ "{ 'execute': 'human-monitor-command', 'arguments': { 'command-line': 'drive_del d0' } }" \ diff --git a/tests/qemu-iotests/153.out b/tests/qemu-iotests/153.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/153.out +++ b/tests/qemu-iotests/153.out @@ -XXX,XX +XXX,XX @@ Is another process using the image? == Symbolic link == QEMU_PROG: -drive if=none,file=TEST_DIR/t.qcow2: Failed to get "write" lock Is another process using the image? - -== Closing an image should unlock it == {"return": {}} Adding drive _qemu_io_wrapper TEST_DIR/t.qcow2 -c write 0 512 can't open device TEST_DIR/t.qcow2: Failed to get "write" lock Is another process using the image? -Closing drive +Creating overlay with qemu-img when the guest is running should be allowed + +_qemu_img_wrapper create -f qcow2 -b TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.overlay +== Closing an image should unlock it == _qemu_io_wrapper TEST_DIR/t.qcow2 -c write 0 512 Adding two and closing one -- 2.13.6 From: Fam Zheng <famz@redhat.com> The AFL image is to exercise the code validating image size, which doesn't work on 32 bit or when out of memory (there is a large allocation before the interesting point). So check that and skip the test, instead of faking the result. Signed-off-by: Fam Zheng <famz@redhat.com> Message-id: 20180301011413.11531-1-famz@redhat.com Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/059 | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/059 +++ b/tests/qemu-iotests/059 @@ -XXX,XX +XXX,XX @@ done echo echo "=== Testing afl image with a very large capacity ===" _use_sample_img afl9.vmdk.bz2 -# The sed makes this test pass on machines with little RAM -# (and also with 32 bit builds) -_img_info | sed -e 's/Cannot allocate memory/Invalid argument/' +_img_info | grep -q 'Cannot allocate memory' && _notrun "Insufficent memory, skipped test" +_img_info _cleanup_test_img # success, all done -- 2.13.6 From: Alberto Garcia <berto@igalia.com> This patch tweaks TestParallelOps in iotest 030 so it allocates data in smaller regions (256KB/512KB instead of 512KB/1MB) and the block-stream job in test_stream_commit() only needs to copy data that is at the very end of the image. This way when the block-stream job is awakened it will finish right away without any chance of being stopped by block_job_sleep_ns(). This triggers the bug that was fixed by 3d5d319e1221082974711af1d09d82f and 1a63a907507fbbcfaee3f622907ec24 and is therefore a more useful test case for parallel block jobs. After this patch the aforementiond bug can also be reproduced with the test_stream_parallel() test case. Since with this change the stream job in test_stream_commit() finishes early, this patch introduces a similar test case where both jobs are slowed down so they can actually run in parallel. Signed-off-by: Alberto Garcia <berto@igalia.com> Cc: John Snow <jsnow@redhat.com> Message-id: 20180306130121.30243-1-berto@igalia.com Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/030 | 52 ++++++++++++++++++++++++++++++++++++++-------- tests/qemu-iotests/030.out | 4 ++-- 2 files changed, 45 insertions(+), 11 deletions(-) diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/030 +++ b/tests/qemu-iotests/030 @@ -XXX,XX +XXX,XX @@ class TestSingleDrive(iotests.QMPTestCase): class TestParallelOps(iotests.QMPTestCase): num_ops = 4 # Number of parallel block-stream operations num_imgs = num_ops * 2 + 1 - image_len = num_ops * 1024 * 1024 + image_len = num_ops * 512 * 1024 imgs = [] def setUp(self): @@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase): '-o', 'backing_file=%s' % self.imgs[i-1], self.imgs[i]) # Put data into the images we are copying data from - for i in range(self.num_imgs / 2): - img_index = i * 2 + 1 - # Alternate between 512k and 1M. + odd_img_indexes = [x for x in reversed(range(self.num_imgs)) if x % 2 == 1] + for i in range(len(odd_img_indexes)): + # Alternate between 256KB and 512KB. # This way jobs will not finish in the same order they were created - num_kb = 512 + 512 * (i % 2) + num_kb = 256 + 256 * (i % 2) qemu_io('-f', iotests.imgfmt, - '-c', 'write -P %d %d %d' % (i, i*1024*1024, num_kb * 1024), - self.imgs[img_index]) + '-c', 'write -P 0xFF %dk %dk' % (i * 512, num_kb), + self.imgs[odd_img_indexes[i]]) # Attach the drive to the VM self.vm = iotests.VM() @@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase): self.wait_until_completed(drive='commit-drive0') # Test a block-stream and a block-commit job in parallel - def test_stream_commit(self): + # Here the stream job is supposed to finish quickly in order to reproduce + # the scenario that triggers the bug fixed in 3d5d319e1221 and 1a63a907507 + def test_stream_commit_1(self): self.assertLessEqual(8, self.num_imgs) self.assert_no_active_block_jobs() # Stream from node0 into node2 - result = self.vm.qmp('block-stream', device='node2', job_id='node2') + result = self.vm.qmp('block-stream', device='node2', base_node='node0', job_id='node2') self.assert_qmp(result, 'return', {}) # Commit from the active layer into node3 @@ -XXX,XX +XXX,XX @@ class TestParallelOps(iotests.QMPTestCase): self.assert_no_active_block_jobs() + # This is similar to test_stream_commit_1 but both jobs are slowed + # down so they can run in parallel for a little while. + def test_stream_commit_2(self): + self.assertLessEqual(8, self.num_imgs) + self.assert_no_active_block_jobs() + + # Stream from node0 into node4 + result = self.vm.qmp('block-stream', device='node4', base_node='node0', job_id='node4', speed=1024*1024) + self.assert_qmp(result, 'return', {}) + + # Commit from the active layer into node5 + result = self.vm.qmp('block-commit', device='drive0', base=self.imgs[5], speed=1024*1024) + self.assert_qmp(result, 'return', {}) + + # Wait for all jobs to be finished. + pending_jobs = ['node4', 'drive0'] + while len(pending_jobs) > 0: + for event in self.vm.get_qmp_events(wait=True): + if event['event'] == 'BLOCK_JOB_COMPLETED': + node_name = self.dictpath(event, 'data/device') + self.assertTrue(node_name in pending_jobs) + self.assert_qmp_absent(event, 'data/error') + pending_jobs.remove(node_name) + if event['event'] == 'BLOCK_JOB_READY': + self.assert_qmp(event, 'data/device', 'drive0') + self.assert_qmp(event, 'data/type', 'commit') + self.assert_qmp_absent(event, 'data/error') + self.assertTrue('drive0' in pending_jobs) + self.vm.qmp('block-job-complete', device='drive0') + + self.assert_no_active_block_jobs() + # Test the base_node parameter def test_stream_base_node_name(self): self.assert_no_active_block_jobs() diff --git a/tests/qemu-iotests/030.out b/tests/qemu-iotests/030.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/030.out +++ b/tests/qemu-iotests/030.out @@ -XXX,XX +XXX,XX @@ -....................... +........................ ---------------------------------------------------------------------- -Ran 23 tests +Ran 24 tests OK -- 2.13.6 From: Stefan Hajnoczi <stefanha@redhat.com> There is a race between the test's 'query-migrate' QMP command after the QMP 'STOP' event and completing the migration: The test case invokes 'query-migrate' upon receiving 'STOP'. At this point the migration thread may still be in the process of completing. Therefore 'query-migrate' can return 'status': 'active' for a brief window of time instead of 'status': 'completed'. This results in qemu-iotests 203 hanging. Solve the race by enabling the 'events' migration capability, which causes QEMU to emit migration-specific QMP events that do not suffer from this race condition. Wait for the QMP 'MIGRATION' event with 'status': 'completed'. Reported-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Message-id: 20180305155926.25858-1-stefanha@redhat.com Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com> --- tests/qemu-iotests/203 | 15 +++++++++++---- tests/qemu-iotests/203.out | 5 +++++ 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/203 b/tests/qemu-iotests/203 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/203 +++ b/tests/qemu-iotests/203 @@ -XXX,XX +XXX,XX @@ with iotests.FilePath('disk0.img') as disk0_img_path, \ node_name='drive1-node', iothread='iothread0', force=True)) + iotests.log('Enabling migration QMP events...') + iotests.log(vm.qmp('migrate-set-capabilities', capabilities=[ + { + 'capability': 'events', + 'state': True + } + ])) + iotests.log('Starting migration...') iotests.log(vm.qmp('migrate', uri='exec:cat >/dev/null')) while True: - vm.get_qmp_event(wait=60.0) - result = vm.qmp('query-migrate') - status = result.get('return', {}).get('status', None) - if status == 'completed': + event = vm.event_wait('MIGRATION') + iotests.log(event, filters=[iotests.filter_qmp_event]) + if event['data']['status'] == 'completed': break diff --git a/tests/qemu-iotests/203.out b/tests/qemu-iotests/203.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/203.out +++ b/tests/qemu-iotests/203.out @@ -XXX,XX +XXX,XX @@ Launching VM... Setting IOThreads... {u'return': {}} {u'return': {}} +Enabling migration QMP events... +{u'return': {}} Starting migration... {u'return': {}} +{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'} +{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'} +{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'} -- 2.13.6