The following changes since commit 07f426c35eddd79388a23d11cb278600d7e3831d: Merge remote-tracking branch 'remotes/rth/tags/pull-tcg-20180926' into staging (2018-09-28 18:56:09 +0100) are available in the git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for you to fetch changes up to dd353157942a59c21da07da5ac8749a871f7c3ed: tests/test-bdrv-drain: Fix too late qemu_event_reset() (2018-10-01 19:13:55 +0200) ---------------------------------------------------------------- Block layer patches: - qcow2 cache option default changes (Linux: 32 MB maximum, limited by whatever cache size can be made use of with the specific image; default cache-clean-interval of 10 minutes) - reopen: Allow specifying unchanged child node references, and changing a few generic options (discard, detect-zeroes) - Fix werror/rerror defaults for -device drive=<node-name> - Test case fixes ---------------------------------------------------------------- Alberto Garcia (9): qemu-io: Fix writethrough check in reopen file-posix: x-check-cache-dropped should default to false on reopen block: Remove child references from bs->{options,explicit_options} block: Don't look for child references in append_open_options() block: Allow child references on reopen block: Forbid trying to change unsupported options during reopen file-posix: Forbid trying to change unsupported options during reopen block: Allow changing 'discard' on reopen block: Allow changing 'detect-zeroes' on reopen Fam Zheng (1): file-posix: Include filename in locking error message Kevin Wolf (3): block-backend: Set werror/rerror defaults in blk_new() test-replication: Lock AioContext around blk_unref() tests/test-bdrv-drain: Fix too late qemu_event_reset() Leonid Bloch (10): qcow2: Options' documentation fixes include: Add a lookup table of sizes qcow2: Make sizes more humanly readable qcow2: Avoid duplication in setting the refcount cache size qcow2: Assign the L2 cache relatively to the image size qcow2: Increase the default upper limit on the L2 cache size qcow2: Resize the cache upon image resizing qcow2: Set the default cache-clean-interval to 10 minutes qcow2: Explicit number replaced by a constant qcow2: Fix cache-clean-interval documentation qapi/block-core.json | 4 +- docs/qcow2-cache.txt | 59 ++++++++++++-------- block/qcow2.h | 19 ++++--- include/block/block.h | 1 + include/qemu/units.h | 55 ++++++++++++++++++ block.c | 135 +++++++++++++++++++++++++++++---------------- block/block-backend.c | 3 + block/file-posix.c | 19 +++++-- block/qcow2.c | 43 +++++++++------ qemu-io-cmds.c | 2 +- tests/test-bdrv-drain.c | 4 +- tests/test-replication.c | 11 ++++ qemu-options.hx | 12 ++-- tests/qemu-iotests/067.out | 1 + tests/qemu-iotests/137 | 8 ++- tests/qemu-iotests/137.out | 4 +- tests/qemu-iotests/153.out | 76 ++++++++++++------------- tests/qemu-iotests/182.out | 2 +- 18 files changed, 307 insertions(+), 151 deletions(-) From: Fam Zheng <famz@redhat.com> Image locking errors happening at device initialization time doesn't say which file cannot be locked, for instance, -device scsi-disk,drive=drive-1: Failed to get shared "write" lock Is another process using the image? could refer to either the overlay image or its backing image. Hoist the error_append_hint to the caller of raw_check_lock_bytes where file name is known, and include it in the error hint. Signed-off-by: Fam Zheng <famz@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/file-posix.c | 10 +++--- tests/qemu-iotests/153.out | 76 +++++++++++++++++++++++----------------------- tests/qemu-iotests/182.out | 2 +- 3 files changed, 45 insertions(+), 43 deletions(-) diff --git a/block/file-posix.c b/block/file-posix.c index XXXXXXX..XXXXXXX 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -XXX,XX +XXX,XX @@ static int raw_check_lock_bytes(int fd, uint64_t perm, uint64_t shared_perm, "Failed to get \"%s\" lock", perm_name); g_free(perm_name); - error_append_hint(errp, - "Is another process using the image?\n"); return ret; } } @@ -XXX,XX +XXX,XX @@ static int raw_check_lock_bytes(int fd, uint64_t perm, uint64_t shared_perm, "Failed to get shared \"%s\" lock", perm_name); g_free(perm_name); - error_append_hint(errp, - "Is another process using the image?\n"); return ret; } } @@ -XXX,XX +XXX,XX @@ static int raw_handle_perm_lock(BlockDriverState *bs, if (!ret) { return 0; } + error_append_hint(errp, + "Is another process using the image [%s]?\n", + bs->filename); } op = RAW_PL_ABORT; /* fall through to unlock bytes. */ @@ -XXX,XX +XXX,XX @@ raw_co_create(BlockdevCreateOptions *options, Error **errp) /* Step two: Check that nobody else has taken conflicting locks */ result = raw_check_lock_bytes(fd, perm, shared, errp); if (result < 0) { + error_append_hint(errp, + "Is another process using the image [%s]?\n", + file_opts->filename); goto out_unlock; } diff --git a/tests/qemu-iotests/153.out b/tests/qemu-iotests/153.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/153.out +++ b/tests/qemu-iotests/153.out @@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 backing_file=TEST_DIR/t == Launching another QEMU, opts: '' == QEMU_PROG: -drive file=TEST_DIR/t.qcow2,if=none,: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? == Launching another QEMU, opts: 'read-only=on' == QEMU_PROG: -drive file=TEST_DIR/t.qcow2,if=none,read-only=on: Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? == Launching another QEMU, opts: 'read-only=on,force-share=on' == @@ -XXX,XX +XXX,XX @@ Is another process using the image? _qemu_io_wrapper -c read 0 512 TEST_DIR/t.qcow2 can't open device TEST_DIR/t.qcow2: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_io_wrapper -r -c read 0 512 TEST_DIR/t.qcow2 can't open device TEST_DIR/t.qcow2: Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_io_wrapper -c open TEST_DIR/t.qcow2 -c read 0 512 can't open device TEST_DIR/t.qcow2: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? no file open, try 'help open' _qemu_io_wrapper -c open -r TEST_DIR/t.qcow2 -c read 0 512 can't open device TEST_DIR/t.qcow2: Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? no file open, try 'help open' _qemu_img_wrapper info TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper check TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper compare TEST_DIR/t.qcow2 TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper map TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper amend -o TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper commit TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper resize TEST_DIR/t.qcow2 32M qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper rebase TEST_DIR/t.qcow2 -b TEST_DIR/t.qcow2.base qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper snapshot -l TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper convert TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.convert qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper dd if=TEST_DIR/t.qcow2 of=TEST_DIR/t.qcow2.convert bs=512 count=1 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper bench -c 1 TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper bench -w -c 1 TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper create -f qcow2 TEST_DIR/t.qcow2 -b TEST_DIR/t.qcow2.base qemu-img: TEST_DIR/t.qcow2: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? file format: IMGFMT == Running utility commands -U == @@ -XXX,XX +XXX,XX @@ Try 'qemu-img --help' for more information _qemu_img_wrapper rebase -U TEST_DIR/t.qcow2 -b TEST_DIR/t.qcow2.base qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper snapshot -l -U TEST_DIR/t.qcow2 @@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432 backing_file=TEST_DIR/t == Launching another QEMU, opts: '' == QEMU_PROG: -drive file=TEST_DIR/t.qcow2,if=none,: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? == Launching another QEMU, opts: 'read-only=on' == @@ -XXX,XX +XXX,XX @@ Is another process using the image? _qemu_io_wrapper -c read 0 512 TEST_DIR/t.qcow2 can't open device TEST_DIR/t.qcow2: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_io_wrapper -r -c read 0 512 TEST_DIR/t.qcow2 _qemu_io_wrapper -c open TEST_DIR/t.qcow2 -c read 0 512 can't open device TEST_DIR/t.qcow2: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? no file open, try 'help open' _qemu_io_wrapper -c open -r TEST_DIR/t.qcow2 -c read 0 512 @@ -XXX,XX +XXX,XX @@ _qemu_img_wrapper map TEST_DIR/t.qcow2 _qemu_img_wrapper amend -o TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper commit TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper resize TEST_DIR/t.qcow2 32M qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper rebase TEST_DIR/t.qcow2 -b TEST_DIR/t.qcow2.base qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper snapshot -l TEST_DIR/t.qcow2 @@ -XXX,XX +XXX,XX @@ _qemu_img_wrapper bench -c 1 TEST_DIR/t.qcow2 _qemu_img_wrapper bench -w -c 1 TEST_DIR/t.qcow2 qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper create -f qcow2 TEST_DIR/t.qcow2 -b TEST_DIR/t.qcow2.base qemu-img: TEST_DIR/t.qcow2: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? file format: IMGFMT == Running utility commands -U == @@ -XXX,XX +XXX,XX @@ Try 'qemu-img --help' for more information _qemu_img_wrapper rebase -U TEST_DIR/t.qcow2 -b TEST_DIR/t.qcow2.base qemu-img: Could not open 'TEST_DIR/t.qcow2': Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? _qemu_img_wrapper snapshot -l -U TEST_DIR/t.qcow2 @@ -XXX,XX +XXX,XX @@ Round done == Two devices with the same image (read-only=off - read-only=off) == QEMU_PROG: -drive if=none,file=TEST_DIR/t.qcow2,read-only=off: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? == Two devices with the same image (read-only=off - read-only=on) == QEMU_PROG: -drive if=none,file=TEST_DIR/t.qcow2,read-only=on: Failed to get shared "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? == Two devices with the same image (read-only=off - read-only=on,force-share=on) == == Two devices with the same image (read-only=on - read-only=off) == QEMU_PROG: -drive if=none,file=TEST_DIR/t.qcow2,read-only=off: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? == Two devices with the same image (read-only=on - read-only=on) == @@ -XXX,XX +XXX,XX @@ Formatting 'TEST_DIR/t.IMGFMT.c', fmt=IMGFMT size=33554432 backing_file=TEST_DIR == Backing image also as an active device == QEMU_PROG: -drive if=none,file=TEST_DIR/t.qcow2: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? == Backing image also as an active device (ro) == == Symbolic link == QEMU_PROG: -drive if=none,file=TEST_DIR/t.qcow2: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? == Active commit to intermediate layer should work when base in use == {"return": {}} @@ -XXX,XX +XXX,XX @@ Adding drive _qemu_io_wrapper TEST_DIR/t.qcow2 -c write 0 512 can't open device TEST_DIR/t.qcow2: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? Creating overlay with qemu-img when the guest is running should be allowed _qemu_img_wrapper create -f qcow2 -b TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.overlay @@ -XXX,XX +XXX,XX @@ _qemu_img_wrapper info TEST_DIR/t.qcow2 _qemu_io_wrapper TEST_DIR/t.qcow2 -c write 0 512 can't open device TEST_DIR/t.qcow2: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? Closing the other _qemu_io_wrapper TEST_DIR/t.qcow2 -c write 0 512 diff --git a/tests/qemu-iotests/182.out b/tests/qemu-iotests/182.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/182.out +++ b/tests/qemu-iotests/182.out @@ -XXX,XX +XXX,XX @@ Starting QEMU Starting a second QEMU using the same image should fail QEMU_PROG: -drive file=TEST_DIR/t.qcow2,if=none,id=drive0,file.locking=on: Failed to get "write" lock -Is another process using the image? +Is another process using the image [TEST_DIR/t.qcow2]? *** done -- 2.13.6 From: Alberto Garcia <berto@igalia.com> "qemu-io reopen" doesn't allow changing the writethrough setting of the cache, but the check is wrong, causing an error even on a simple reopen with the default parameters: $ qemu-img create -f qcow2 hd.qcow2 1M $ qemu-system-x86_64 -monitor stdio -drive if=virtio,file=hd.qcow2 (qemu) qemu-io virtio0 reopen Cannot change cache.writeback: Device attached Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qemu-io-cmds.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-io-cmds.c +++ b/qemu-io-cmds.c @@ -XXX,XX +XXX,XX @@ static int reopen_f(BlockBackend *blk, int argc, char **argv) return -EINVAL; } - if (writethrough != blk_enable_write_cache(blk) && + if (!writethrough != blk_enable_write_cache(blk) && blk_get_attached_dev(blk)) { error_report("Cannot change cache.writeback: Device attached"); -- 2.13.6 From: Alberto Garcia <berto@igalia.com> The default value of x-check-cache-dropped is false. There's no reason to use the previous value as a default in raw_reopen_prepare() because bdrv_reopen_queue_child() already takes care of putting the old options in the BDRVReopenState.options QDict. If x-check-cache-dropped was previously set but is now missing from the reopen QDict then it should be reset to false. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/file-posix.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/file-posix.c b/block/file-posix.c index XXXXXXX..XXXXXXX 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -XXX,XX +XXX,XX @@ static int raw_reopen_prepare(BDRVReopenState *state, } rs->check_cache_dropped = qemu_opt_get_bool(opts, "x-check-cache-dropped", - s->check_cache_dropped); + false); if (s->type == FTYPE_CD) { rs->open_flags |= O_NONBLOCK; -- 2.13.6 From: Alberto Garcia <berto@igalia.com> Block drivers allow opening their children using a reference to an existing BlockDriverState. These references remain stored in the 'options' and 'explicit_options' QDicts, but we don't need to keep them once everything is open. What is more important, these values can become wrong if the children change: $ qemu-img create -f qcow2 hd0.qcow2 10M $ qemu-img create -f qcow2 hd1.qcow2 10M $ qemu-img create -f qcow2 hd2.qcow2 10M $ $QEMU -drive if=none,file=hd0.qcow2,node-name=hd0 \ -drive if=none,file=hd1.qcow2,node-name=hd1,backing=hd0 \ -drive file=hd2.qcow2,node-name=hd2,backing=hd1 After this hd2 has hd1 as its backing file. Now let's remove it using block_stream: (qemu) block_stream hd2 0 hd0.qcow2 Now hd0 is the backing file of hd2, but hd2's options QDicts still contain backing=hd1. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ static BlockDriverState *bdrv_open_inherit(const char *filename, } } - /* Remove all children options from bs->options and bs->explicit_options */ + /* Remove all children options and references + * from bs->options and bs->explicit_options */ QLIST_FOREACH(child, &bs->children, next) { char *child_key_dot; child_key_dot = g_strdup_printf("%s.", child->name); qdict_extract_subqdict(bs->explicit_options, NULL, child_key_dot); qdict_extract_subqdict(bs->options, NULL, child_key_dot); + qdict_del(bs->explicit_options, child->name); + qdict_del(bs->options, child->name); g_free(child_key_dot); } @@ -XXX,XX +XXX,XX @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state) { BlockDriver *drv; BlockDriverState *bs; + BdrvChild *child; bool old_can_write, new_can_write; assert(reopen_state != NULL); @@ -XXX,XX +XXX,XX @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state) bs->open_flags = reopen_state->flags; bs->read_only = !(reopen_state->flags & BDRV_O_RDWR); + /* Remove child references from bs->options and bs->explicit_options. + * Child options were already removed in bdrv_reopen_queue_child() */ + QLIST_FOREACH(child, &bs->children, next) { + qdict_del(bs->explicit_options, child->name); + qdict_del(bs->options, child->name); + } + bdrv_refresh_limits(bs, NULL); bdrv_set_perm(reopen_state->bs, reopen_state->perm, -- 2.13.6 From: Alberto Garcia <berto@igalia.com> In the previous patch we removed child references from bs->options, so there's no need to look for them here anymore. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block.c | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ static bool append_open_options(QDict *d, BlockDriverState *bs) { const QDictEntry *entry; QemuOptDesc *desc; - BdrvChild *child; bool found_any = false; for (entry = qdict_first(bs->options); entry; entry = qdict_next(bs->options, entry)) { - /* Exclude node-name references to children */ - QLIST_FOREACH(child, &bs->children, next) { - if (!strcmp(entry->key, child->name)) { - break; - } - } - if (child) { - continue; - } - - /* And exclude all non-driver-specific options */ + /* Exclude all non-driver-specific options */ for (desc = bdrv_runtime_opts.desc; desc->name; desc++) { if (!strcmp(qdict_entry_key(entry), desc->name)) { break; -- 2.13.6 From: Alberto Garcia <berto@igalia.com> In the previous patches we removed all child references from bs->{options,explicit_options} because keeping them is useless and wrong. Because of this, any attempt to reopen a BlockDriverState using a child reference as one of its options would result in a failure, because bdrv_reopen_prepare() would detect that there's a new option (the child reference) that wasn't present in bs->options. But passing child references on reopen can be useful. It's a way to specify a BDS's child without having to pass recursively all of the child's options, and if the reference points to a different BDS then this can allow us to replace the child. However, replacing the child is something that needs to be implemented case by case and only when it makes sense. For now, this patch allows passing a child reference as long as it points to the current child of the BlockDriverState. It's also important to remember that, as a consequence of the previous patches, this child reference will be removed from bs->{options,explicit_options} after the reopening has been completed. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, QObject *new = entry->value; QObject *old = qdict_get(reopen_state->bs->options, entry->key); + /* Allow child references (child_name=node_name) as long as they + * point to the current child (i.e. everything stays the same). */ + if (qobject_type(new) == QTYPE_QSTRING) { + BdrvChild *child; + QLIST_FOREACH(child, &reopen_state->bs->children, next) { + if (!strcmp(child->name, entry->key)) { + break; + } + } + + if (child) { + const char *str = qobject_get_try_str(new); + if (!strcmp(child->bs->node_name, str)) { + continue; /* Found child with this name, skip option */ + } + } + } + /* * TODO: When using -drive to specify blockdev options, all values * will be strings; however, when using -blockdev, blockdev-add or -- 2.13.6 From: Alberto Garcia <berto@igalia.com> The bdrv_reopen_prepare() function checks all options passed to each BlockDriverState (in the reopen_state->options QDict) and makes all necessary preparations to apply the option changes requested by the user. Options are removed from the QDict as they are processed, so at the end of bdrv_reopen_prepare() only the options that can't be changed are left. Then a loop goes over all remaining options and verifies that the old and new values are identical, returning an error if they're not. The problem is that at the moment there are options that are removed from the QDict although they can't be changed. The consequence of this is any modification to any of those options is silently ignored: (qemu) qemu-io virtio0 "reopen -o discard=on" This happens when all options from bdrv_runtime_opts are removed from the QDict but then only a few of them are processed. Since it's especially important that "node-name" and "driver" are not changed, the code puts them back into the QDict so they are checked at the end of the function. Instead of putting only those two options back into the QDict, this patch puts all unprocessed options using qemu_opts_to_qdict(). update_flags_from_options() also needs to be modified to prevent BDRV_OPT_CACHE_NO_FLUSH, BDRV_OPT_CACHE_DIRECT and BDRV_OPT_READ_ONLY from going back to the QDict. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block.c | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-) diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ static void update_flags_from_options(int *flags, QemuOpts *opts) *flags &= ~BDRV_O_CACHE_MASK; assert(qemu_opt_find(opts, BDRV_OPT_CACHE_NO_FLUSH)); - if (qemu_opt_get_bool(opts, BDRV_OPT_CACHE_NO_FLUSH, false)) { + if (qemu_opt_get_bool_del(opts, BDRV_OPT_CACHE_NO_FLUSH, false)) { *flags |= BDRV_O_NO_FLUSH; } assert(qemu_opt_find(opts, BDRV_OPT_CACHE_DIRECT)); - if (qemu_opt_get_bool(opts, BDRV_OPT_CACHE_DIRECT, false)) { + if (qemu_opt_get_bool_del(opts, BDRV_OPT_CACHE_DIRECT, false)) { *flags |= BDRV_O_NOCACHE; } *flags &= ~BDRV_O_RDWR; assert(qemu_opt_find(opts, BDRV_OPT_READ_ONLY)); - if (!qemu_opt_get_bool(opts, BDRV_OPT_READ_ONLY, false)) { + if (!qemu_opt_get_bool_del(opts, BDRV_OPT_READ_ONLY, false)) { *flags |= BDRV_O_RDWR; } @@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, BlockDriver *drv; QemuOpts *opts; QDict *orig_reopen_opts; - const char *value; bool read_only; assert(reopen_state != NULL); @@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, update_flags_from_options(&reopen_state->flags, opts); - /* node-name and driver must be unchanged. Put them back into the QDict, so - * that they are checked at the end of this function. */ - value = qemu_opt_get(opts, "node-name"); - if (value) { - qdict_put_str(reopen_state->options, "node-name", value); - } - - value = qemu_opt_get(opts, "driver"); - if (value) { - qdict_put_str(reopen_state->options, "driver", value); - } + /* All other options (including node-name and driver) must be unchanged. + * Put them back into the QDict, so that they are checked at the end + * of this function. */ + qemu_opts_to_qdict(opts, reopen_state->options); /* If we are to stay read-only, do not allow permission change * to r/w. Attempting to set to r/w may fail if either BDRV_O_ALLOW_RDWR is -- 2.13.6 From: Alberto Garcia <berto@igalia.com> The file-posix code is used for the "file", "host_device" and "host_cdrom" drivers, and it allows reopening images. However the only option that is actually processed is "x-check-cache-dropped", and changes in all other options (e.g. "filename") are silently ignored: (qemu) qemu-io virtio0 "reopen -o file.filename=no-such-file" While we could allow changing some of the other options, let's keep things as they are for now but return an error if the user tries to change any of them. Signed-off-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/file-posix.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/block/file-posix.c b/block/file-posix.c index XXXXXXX..XXXXXXX 100644 --- a/block/file-posix.c +++ b/block/file-posix.c @@ -XXX,XX +XXX,XX @@ static int raw_reopen_prepare(BDRVReopenState *state, goto out; } - rs->check_cache_dropped = qemu_opt_get_bool(opts, "x-check-cache-dropped", - false); + rs->check_cache_dropped = + qemu_opt_get_bool_del(opts, "x-check-cache-dropped", false); + + /* This driver's reopen function doesn't currently allow changing + * other options, so let's put them back in the original QDict and + * bdrv_reopen_prepare() will detect changes and complain. */ + qemu_opts_to_qdict(opts, state->options); if (s->type == FTYPE_CD) { rs->open_flags |= O_NONBLOCK; -- 2.13.6 From: Alberto Garcia <berto@igalia.com> 'discard' is one of the basic BlockdevOptions available for all drivers, but it's not handled by bdrv_reopen_prepare() so any attempt to change it results in an error: (qemu) qemu-io virtio0 "reopen -o discard=on" Cannot change the option 'discard' Since there's no reason why we shouldn't allow changing it and the implementation is simple let's just do it. Signed-off-by: Alberto Garcia <berto@igalia.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, BlockDriver *drv; QemuOpts *opts; QDict *orig_reopen_opts; + char *discard = NULL; bool read_only; assert(reopen_state != NULL); @@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, update_flags_from_options(&reopen_state->flags, opts); + discard = qemu_opt_get_del(opts, "discard"); + if (discard != NULL) { + if (bdrv_parse_discard_flags(discard, &reopen_state->flags) != 0) { + error_setg(errp, "Invalid discard option"); + ret = -EINVAL; + goto error; + } + } + /* All other options (including node-name and driver) must be unchanged. * Put them back into the QDict, so that they are checked at the end * of this function. */ @@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, error: qemu_opts_del(opts); qobject_unref(orig_reopen_opts); + g_free(discard); return ret; } -- 2.13.6 From: Alberto Garcia <berto@igalia.com> 'detect-zeroes' is one of the basic BlockdevOptions available for all drivers, but it's not handled by bdrv_reopen_prepare(), so any attempt to change it results in an error: (qemu) qemu-io virtio0 "reopen -o detect-zeroes=on" Cannot change the option 'detect-zeroes' Since there's no reason why we shouldn't allow changing it and the implementation is simple let's just do it. Signed-off-by: Alberto Garcia <berto@igalia.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- include/block/block.h | 1 + block.c | 64 ++++++++++++++++++++++++++++++++------------------- 2 files changed, 41 insertions(+), 24 deletions(-) diff --git a/include/block/block.h b/include/block/block.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -XXX,XX +XXX,XX @@ typedef QSIMPLEQ_HEAD(BlockReopenQueue, BlockReopenQueueEntry) BlockReopenQueue; typedef struct BDRVReopenState { BlockDriverState *bs; int flags; + BlockdevDetectZeroesOptions detect_zeroes; uint64_t perm, shared_perm; QDict *options; QDict *explicit_options; diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ static void bdrv_join_options(BlockDriverState *bs, QDict *options, } } +static BlockdevDetectZeroesOptions bdrv_parse_detect_zeroes(QemuOpts *opts, + int open_flags, + Error **errp) +{ + Error *local_err = NULL; + char *value = qemu_opt_get_del(opts, "detect-zeroes"); + BlockdevDetectZeroesOptions detect_zeroes = + qapi_enum_parse(&BlockdevDetectZeroesOptions_lookup, value, + BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF, &local_err); + g_free(value); + if (local_err) { + error_propagate(errp, local_err); + return detect_zeroes; + } + + if (detect_zeroes == BLOCKDEV_DETECT_ZEROES_OPTIONS_UNMAP && + !(open_flags & BDRV_O_UNMAP)) + { + error_setg(errp, "setting detect-zeroes to unmap is not allowed " + "without setting discard operation to unmap"); + } + + return detect_zeroes; +} + /** * Set open flags for a given discard mode * @@ -XXX,XX +XXX,XX @@ static int bdrv_open_common(BlockDriverState *bs, BlockBackend *file, const char *driver_name = NULL; const char *node_name = NULL; const char *discard; - const char *detect_zeroes; QemuOpts *opts; BlockDriver *drv; Error *local_err = NULL; @@ -XXX,XX +XXX,XX @@ static int bdrv_open_common(BlockDriverState *bs, BlockBackend *file, } } - detect_zeroes = qemu_opt_get(opts, "detect-zeroes"); - if (detect_zeroes) { - BlockdevDetectZeroesOptions value = - qapi_enum_parse(&BlockdevDetectZeroesOptions_lookup, - detect_zeroes, - BLOCKDEV_DETECT_ZEROES_OPTIONS_OFF, - &local_err); - if (local_err) { - error_propagate(errp, local_err); - ret = -EINVAL; - goto fail_opts; - } - - if (value == BLOCKDEV_DETECT_ZEROES_OPTIONS_UNMAP && - !(bs->open_flags & BDRV_O_UNMAP)) - { - error_setg(errp, "setting detect-zeroes to unmap is not allowed " - "without setting discard operation to unmap"); - ret = -EINVAL; - goto fail_opts; - } - - bs->detect_zeroes = value; + bs->detect_zeroes = + bdrv_parse_detect_zeroes(opts, bs->open_flags, &local_err); + if (local_err) { + error_propagate(errp, local_err); + ret = -EINVAL; + goto fail_opts; } if (filename != NULL) { @@ -XXX,XX +XXX,XX @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, } } + reopen_state->detect_zeroes = + bdrv_parse_detect_zeroes(opts, reopen_state->flags, &local_err); + if (local_err) { + error_propagate(errp, local_err); + ret = -EINVAL; + goto error; + } + /* All other options (including node-name and driver) must be unchanged. * Put them back into the QDict, so that they are checked at the end * of this function. */ @@ -XXX,XX +XXX,XX @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state) bs->options = reopen_state->options; bs->open_flags = reopen_state->flags; bs->read_only = !(reopen_state->flags & BDRV_O_RDWR); + bs->detect_zeroes = reopen_state->detect_zeroes; /* Remove child references from bs->options and bs->explicit_options. * Child options were already removed in bdrv_reopen_queue_child() */ -- 2.13.6 From: Leonid Bloch <lbloch@janustech.com> Signed-off-by: Leonid Bloch <lbloch@janustech.com> Reviewed-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- docs/qcow2-cache.txt | 21 ++++++++++++++------- qemu-options.hx | 9 ++++++--- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt index XXXXXXX..XXXXXXX 100644 --- a/docs/qcow2-cache.txt +++ b/docs/qcow2-cache.txt @@ -XXX,XX +XXX,XX @@ Choosing the right cache sizes In order to choose the cache sizes we need to know how they relate to the amount of allocated space. -The amount of virtual disk that can be mapped by the L2 and refcount +The part of the virtual disk that can be mapped by the L2 and refcount caches (in bytes) is: disk_size = l2_cache_size * cluster_size / 8 disk_size = refcount_cache_size * cluster_size * 8 / refcount_bits With the default values for cluster_size (64KB) and refcount_bits -(16), that is +(16), this becomes: disk_size = l2_cache_size * 8192 disk_size = refcount_cache_size * 32768 @@ -XXX,XX +XXX,XX @@ need: l2_cache_size = disk_size_GB * 131072 refcount_cache_size = disk_size_GB * 32768 -QEMU has a default L2 cache of 1MB (1048576 bytes) and a refcount -cache of 256KB (262144 bytes), so using the formulas we've just seen -we have +For example, 1MB of L2 cache is needed to cover every 8 GB of the virtual +image size (given that the default cluster size is used): - 1048576 / 131072 = 8 GB of virtual disk covered by that cache - 262144 / 32768 = 8 GB + 8 GB / 8192 = 1 MB + +The refcount cache is 4 times the cluster size by default. With the default +cluster size of 64 KB, it is 256 KB (262144 bytes). This is sufficient for +8 GB of image size: + + 262144 * 32768 = 8 GB How to configure the cache sizes @@ -XXX,XX +XXX,XX @@ There are a few things that need to be taken into account: memory as possible to the L2 cache before increasing the refcount cache size. + - At most two of "l2-cache-size", "refcount-cache-size", and "cache-size" + can be set simultaneously. + Unlike L2 tables, refcount blocks are not used during normal I/O but only during allocations and internal snapshots. In most cases they are accessed sequentially (even during random guest I/O) so increasing the diff --git a/qemu-options.hx b/qemu-options.hx index XXXXXXX..XXXXXXX 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -XXX,XX +XXX,XX @@ image file) @item cache-size The maximum total size of the L2 table and refcount block caches in bytes -(default: 1048576 bytes or 8 clusters, whichever is larger) +(default: the sum of l2-cache-size and refcount-cache-size) @item l2-cache-size The maximum size of the L2 table cache in bytes -(default: 4/5 of the total cache size) +(default: if cache-size is not defined - 1048576 bytes or 8 clusters, whichever +is larger; otherwise, as large as possible or needed within the cache-size, +while permitting the requested or the minimal refcount cache size) @item refcount-cache-size The maximum size of the refcount block cache in bytes -(default: 1/5 of the total cache size) +(default: 4 times the cluster size; or if cache-size is specified, the part of +it which is not used for the L2 cache) @item cache-clean-interval Clean unused entries in the L2 and refcount caches. The interval is in seconds. -- 2.13.6 From: Leonid Bloch <lbloch@janustech.com> Adding a lookup table for the powers of two, with the appropriate size prefixes. This is needed when a size has to be stringified, in which case something like '(1 * KiB)' would become a literal '(1 * (1L << 10))' string. Powers of two are used very often for sizes, so such a table will also make it easier and more intuitive to write them. This table is generatred using the following AWK script: BEGIN { suffix="KMGTPE"; for(i=10; i<64; i++) { val=2**i; s=substr(suffix, int(i/10), 1); n=2**(i%10); pad=21-int(log(n)/log(10)); printf("#define S_%d%siB %*d\n", n, s, pad, val); } } Signed-off-by: Leonid Bloch <lbloch@janustech.com> Reviewed-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- include/qemu/units.h | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/include/qemu/units.h b/include/qemu/units.h index XXXXXXX..XXXXXXX 100644 --- a/include/qemu/units.h +++ b/include/qemu/units.h @@ -XXX,XX +XXX,XX @@ #define PiB (INT64_C(1) << 50) #define EiB (INT64_C(1) << 60) +#define S_1KiB 1024 +#define S_2KiB 2048 +#define S_4KiB 4096 +#define S_8KiB 8192 +#define S_16KiB 16384 +#define S_32KiB 32768 +#define S_64KiB 65536 +#define S_128KiB 131072 +#define S_256KiB 262144 +#define S_512KiB 524288 +#define S_1MiB 1048576 +#define S_2MiB 2097152 +#define S_4MiB 4194304 +#define S_8MiB 8388608 +#define S_16MiB 16777216 +#define S_32MiB 33554432 +#define S_64MiB 67108864 +#define S_128MiB 134217728 +#define S_256MiB 268435456 +#define S_512MiB 536870912 +#define S_1GiB 1073741824 +#define S_2GiB 2147483648 +#define S_4GiB 4294967296 +#define S_8GiB 8589934592 +#define S_16GiB 17179869184 +#define S_32GiB 34359738368 +#define S_64GiB 68719476736 +#define S_128GiB 137438953472 +#define S_256GiB 274877906944 +#define S_512GiB 549755813888 +#define S_1TiB 1099511627776 +#define S_2TiB 2199023255552 +#define S_4TiB 4398046511104 +#define S_8TiB 8796093022208 +#define S_16TiB 17592186044416 +#define S_32TiB 35184372088832 +#define S_64TiB 70368744177664 +#define S_128TiB 140737488355328 +#define S_256TiB 281474976710656 +#define S_512TiB 562949953421312 +#define S_1PiB 1125899906842624 +#define S_2PiB 2251799813685248 +#define S_4PiB 4503599627370496 +#define S_8PiB 9007199254740992 +#define S_16PiB 18014398509481984 +#define S_32PiB 36028797018963968 +#define S_64PiB 72057594037927936 +#define S_128PiB 144115188075855872 +#define S_256PiB 288230376151711744 +#define S_512PiB 576460752303423488 +#define S_1EiB 1152921504606846976 +#define S_2EiB 2305843009213693952 +#define S_4EiB 4611686018427387904 +#define S_8EiB 9223372036854775808 + #endif -- 2.13.6 From: Leonid Bloch <lbloch@janustech.com> Signed-off-by: Leonid Bloch <lbloch@janustech.com> Reviewed-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.h | 9 +++++---- block/qcow2.c | 2 +- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ #include "crypto/block.h" #include "qemu/coroutine.h" +#include "qemu/units.h" //#define DEBUG_ALLOC //#define DEBUG_ALLOC2 @@ -XXX,XX +XXX,XX @@ /* 8 MB refcount table is enough for 2 PB images at 64k cluster size * (128 GB for 512 byte clusters, 2 EB for 2 MB clusters) */ -#define QCOW_MAX_REFTABLE_SIZE 0x800000 +#define QCOW_MAX_REFTABLE_SIZE S_8MiB /* 32 MB L1 table is enough for 2 PB images at 64k cluster size * (128 GB for 512 byte clusters, 2 EB for 2 MB clusters) */ -#define QCOW_MAX_L1_SIZE 0x2000000 +#define QCOW_MAX_L1_SIZE S_32MiB /* Allow for an average of 1k per snapshot table entry, should be plenty of * space for snapshot names and IDs */ @@ -XXX,XX +XXX,XX @@ /* Whichever is more */ #define DEFAULT_L2_CACHE_CLUSTERS 8 /* clusters */ -#define DEFAULT_L2_CACHE_BYTE_SIZE 1048576 /* bytes */ +#define DEFAULT_L2_CACHE_SIZE S_1MiB -#define DEFAULT_CLUSTER_SIZE 65536 +#define DEFAULT_CLUSTER_SIZE S_64KiB #define QCOW2_OPT_LAZY_REFCOUNTS "lazy-refcounts" diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts, } } else { if (!l2_cache_size_set) { - *l2_cache_size = MAX(DEFAULT_L2_CACHE_BYTE_SIZE, + *l2_cache_size = MAX(DEFAULT_L2_CACHE_SIZE, (uint64_t)DEFAULT_L2_CACHE_CLUSTERS * s->cluster_size); } -- 2.13.6 From: Leonid Bloch <lbloch@janustech.com> The refcount cache size does not need to be set to its minimum value in read_cache_sizes(), as it is set to at least its minimum value in qcow2_update_options_prepare(). Signed-off-by: Leonid Bloch <lbloch@janustech.com> Reviewed-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts, (uint64_t)DEFAULT_L2_CACHE_CLUSTERS * s->cluster_size); } - if (!refcount_cache_size_set) { - *refcount_cache_size = min_refcount_cache; - } } + /* l2_cache_size and refcount_cache_size are ensured to have at least + * their minimum values in qcow2_update_options_prepare() */ if (*l2_cache_entry_size < (1 << MIN_CLUSTER_BITS) || *l2_cache_entry_size > s->cluster_size || -- 2.13.6 From: Leonid Bloch <lbloch@janustech.com> Sufficient L2 cache can noticeably improve the performance when using large images with frequent I/O. Previously, unless 'cache-size' was specified and was large enough, the L2 cache was set to a certain size without taking the virtual image size into account. Now, the L2 cache assignment is aware of the virtual size of the image, and will cover the entire image, unless the cache size needed for that is larger than a certain maximum. This maximum is set to 1 MB by default (enough to cover an 8 GB image with the default cluster size) but can be increased or decreased using the 'l2-cache-size' option. This option was previously documented as the *maximum* L2 cache size, and this patch makes it behave as such, instead of as a constant size. Also, the existing option 'cache-size' can limit the sum of both L2 and refcount caches, as previously. Signed-off-by: Leonid Bloch <lbloch@janustech.com> Reviewed-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- docs/qcow2-cache.txt | 15 ++++++++++----- block/qcow2.h | 4 +--- block/qcow2.c | 21 +++++++++------------ qemu-options.hx | 6 +++--- tests/qemu-iotests/137 | 8 +++++++- tests/qemu-iotests/137.out | 4 +++- 6 files changed, 33 insertions(+), 25 deletions(-) diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt index XXXXXXX..XXXXXXX 100644 --- a/docs/qcow2-cache.txt +++ b/docs/qcow2-cache.txt @@ -XXX,XX +XXX,XX @@ There are a few things that need to be taken into account: - Both caches must have a size that is a multiple of the cluster size (or the cache entry size: see "Using smaller cache sizes" below). - - The default L2 cache size is 8 clusters or 1MB (whichever is more), - and the minimum is 2 clusters (or 2 cache entries, see below). + - The maximum L2 cache size is 1 MB by default (enough for full coverage + of 8 GB images, with the default cluster size). This value can be + modified using the "l2-cache-size" option. QEMU will not use more memory + than needed to hold all of the image's L2 tables, regardless of this max. + value. The minimal L2 cache size is 2 clusters (or 2 cache entries, see + below). - The default (and minimum) refcount cache size is 4 clusters. @@ -XXX,XX +XXX,XX @@ Some things to take into account: always uses the cluster size as the entry size. - If the L2 cache is big enough to hold all of the image's L2 tables - (as explained in the "Choosing the right cache sizes" section - earlier in this document) then none of this is necessary and you - can omit the "l2-cache-entry-size" parameter altogether. + (as explained in the "Choosing the right cache sizes" and "How to + configure the cache sizes" sections in this document) then none of + this is necessary and you can omit the "l2-cache-entry-size" + parameter altogether. Reducing the memory usage diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ /* Must be at least 4 to cover all cases of refcount table growth */ #define MIN_REFCOUNT_CACHE_SIZE 4 /* clusters */ -/* Whichever is more */ -#define DEFAULT_L2_CACHE_CLUSTERS 8 /* clusters */ -#define DEFAULT_L2_CACHE_SIZE S_1MiB +#define DEFAULT_L2_CACHE_MAX_SIZE S_1MiB #define DEFAULT_CLUSTER_SIZE S_64KiB diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts, uint64_t *refcount_cache_size, Error **errp) { BDRVQcow2State *s = bs->opaque; - uint64_t combined_cache_size; + uint64_t combined_cache_size, l2_cache_max_setting; bool l2_cache_size_set, refcount_cache_size_set, combined_cache_size_set; int min_refcount_cache = MIN_REFCOUNT_CACHE_SIZE * s->cluster_size; + uint64_t virtual_disk_size = bs->total_sectors * BDRV_SECTOR_SIZE; + uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size / 8); combined_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_CACHE_SIZE); l2_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_L2_CACHE_SIZE); refcount_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_REFCOUNT_CACHE_SIZE); combined_cache_size = qemu_opt_get_size(opts, QCOW2_OPT_CACHE_SIZE, 0); - *l2_cache_size = qemu_opt_get_size(opts, QCOW2_OPT_L2_CACHE_SIZE, 0); + l2_cache_max_setting = qemu_opt_get_size(opts, QCOW2_OPT_L2_CACHE_SIZE, + DEFAULT_L2_CACHE_MAX_SIZE); *refcount_cache_size = qemu_opt_get_size(opts, QCOW2_OPT_REFCOUNT_CACHE_SIZE, 0); *l2_cache_entry_size = qemu_opt_get_size( opts, QCOW2_OPT_L2_CACHE_ENTRY_SIZE, s->cluster_size); + *l2_cache_size = MIN(max_l2_cache, l2_cache_max_setting); + if (combined_cache_size_set) { if (l2_cache_size_set && refcount_cache_size_set) { error_setg(errp, QCOW2_OPT_CACHE_SIZE ", " QCOW2_OPT_L2_CACHE_SIZE " and " QCOW2_OPT_REFCOUNT_CACHE_SIZE " may not be set " "at the same time"); return; - } else if (*l2_cache_size > combined_cache_size) { + } else if (l2_cache_size_set && + (l2_cache_max_setting > combined_cache_size)) { error_setg(errp, QCOW2_OPT_L2_CACHE_SIZE " may not exceed " QCOW2_OPT_CACHE_SIZE); return; @@ -XXX,XX +XXX,XX @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts, } else if (refcount_cache_size_set) { *l2_cache_size = combined_cache_size - *refcount_cache_size; } else { - uint64_t virtual_disk_size = bs->total_sectors * BDRV_SECTOR_SIZE; - uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size / 8); - /* Assign as much memory as possible to the L2 cache, and * use the remainder for the refcount cache */ if (combined_cache_size >= max_l2_cache + min_refcount_cache) { @@ -XXX,XX +XXX,XX @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts, *l2_cache_size = combined_cache_size - *refcount_cache_size; } } - } else { - if (!l2_cache_size_set) { - *l2_cache_size = MAX(DEFAULT_L2_CACHE_SIZE, - (uint64_t)DEFAULT_L2_CACHE_CLUSTERS - * s->cluster_size); - } } /* l2_cache_size and refcount_cache_size are ensured to have at least * their minimum values in qcow2_update_options_prepare() */ diff --git a/qemu-options.hx b/qemu-options.hx index XXXXXXX..XXXXXXX 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -XXX,XX +XXX,XX @@ The maximum total size of the L2 table and refcount block caches in bytes @item l2-cache-size The maximum size of the L2 table cache in bytes -(default: if cache-size is not defined - 1048576 bytes or 8 clusters, whichever -is larger; otherwise, as large as possible or needed within the cache-size, -while permitting the requested or the minimal refcount cache size) +(default: if cache-size is not specified - 1M; otherwise, as large as possible +within the cache-size, while permitting the requested or the minimal refcount +cache size) @item refcount-cache-size The maximum size of the refcount block cache in bytes diff --git a/tests/qemu-iotests/137 b/tests/qemu-iotests/137 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/137 +++ b/tests/qemu-iotests/137 @@ -XXX,XX +XXX,XX @@ $QEMU_IO \ -c "reopen -o cache-size=1M,l2-cache-size=64k,refcount-cache-size=64k" \ -c "reopen -o cache-size=1M,l2-cache-size=2M" \ -c "reopen -o cache-size=1M,refcount-cache-size=2M" \ - -c "reopen -o l2-cache-size=256T" \ -c "reopen -o l2-cache-entry-size=33k" \ -c "reopen -o l2-cache-entry-size=128k" \ -c "reopen -o refcount-cache-size=256T" \ @@ -XXX,XX +XXX,XX @@ $QEMU_IO \ -c "reopen -o cache-clean-interval=-1" \ "$TEST_IMG" | _filter_qemu_io +IMGOPTS="cluster_size=256k" _make_test_img 32P +$QEMU_IO \ + -c "reopen -o l2-cache-entry-size=512,l2-cache-size=1T" \ + "$TEST_IMG" | _filter_qemu_io + +_make_test_img 64M + echo echo === Test transaction semantics === echo diff --git a/tests/qemu-iotests/137.out b/tests/qemu-iotests/137.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/137.out +++ b/tests/qemu-iotests/137.out @@ -XXX,XX +XXX,XX @@ Parameter 'lazy-refcounts' expects 'on' or 'off' cache-size, l2-cache-size and refcount-cache-size may not be set at the same time l2-cache-size may not exceed cache-size refcount-cache-size may not exceed cache-size -L2 cache size too big L2 cache entry size must be a power of two between 512 and the cluster size (65536) L2 cache entry size must be a power of two between 512 and the cluster size (65536) Refcount cache size too big @@ -XXX,XX +XXX,XX @@ Conflicting values for qcow2 options 'overlap-check' ('constant') and 'overlap-c Unsupported value 'blubb' for qcow2 option 'overlap-check'. Allowed are any of the following: none, constant, cached, all Unsupported value 'blubb' for qcow2 option 'overlap-check'. Allowed are any of the following: none, constant, cached, all Cache clean interval too big +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=36028797018963968 +L2 cache size too big +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 === Test transaction semantics === -- 2.13.6 From: Leonid Bloch <lbloch@janustech.com> The upper limit on the L2 cache size is increased from 1 MB to 32 MB on Linux platforms, and to 8 MB on other platforms (this difference is caused by the ability to set intervals for cache cleaning on Linux platforms only). This is done in order to allow default full coverage with the L2 cache for images of up to 256 GB in size (was 8 GB). Note, that only the needed amount to cover the full image is allocated. The value which is changed here is just the upper limit on the L2 cache size, beyond which it will not grow, even if the size of the image will require it to. Signed-off-by: Leonid Bloch <lbloch@janustech.com> Reviewed-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- docs/qcow2-cache.txt | 15 +++++++++------ block/qcow2.h | 6 +++++- qemu-options.hx | 6 +++--- 3 files changed, 17 insertions(+), 10 deletions(-) diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt index XXXXXXX..XXXXXXX 100644 --- a/docs/qcow2-cache.txt +++ b/docs/qcow2-cache.txt @@ -XXX,XX +XXX,XX @@ There are a few things that need to be taken into account: - Both caches must have a size that is a multiple of the cluster size (or the cache entry size: see "Using smaller cache sizes" below). - - The maximum L2 cache size is 1 MB by default (enough for full coverage - of 8 GB images, with the default cluster size). This value can be - modified using the "l2-cache-size" option. QEMU will not use more memory - than needed to hold all of the image's L2 tables, regardless of this max. - value. The minimal L2 cache size is 2 clusters (or 2 cache entries, see - below). + - The maximum L2 cache size is 32 MB by default on Linux platforms (enough + for full coverage of 256 GB images, with the default cluster size). This + value can be modified using the "l2-cache-size" option. QEMU will not use + more memory than needed to hold all of the image's L2 tables, regardless + of this max. value. + On non-Linux platforms the maximal value is smaller by default (8 MB) and + this difference stems from the fact that on Linux the cache can be cleared + periodically if needed, using the "cache-clean-interval" option (see below). + The minimal L2 cache size is 2 clusters (or 2 cache entries, see below). - The default (and minimum) refcount cache size is 4 clusters. diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ /* Must be at least 4 to cover all cases of refcount table growth */ #define MIN_REFCOUNT_CACHE_SIZE 4 /* clusters */ -#define DEFAULT_L2_CACHE_MAX_SIZE S_1MiB +#ifdef CONFIG_LINUX +#define DEFAULT_L2_CACHE_MAX_SIZE S_32MiB +#else +#define DEFAULT_L2_CACHE_MAX_SIZE S_8MiB +#endif #define DEFAULT_CLUSTER_SIZE S_64KiB diff --git a/qemu-options.hx b/qemu-options.hx index XXXXXXX..XXXXXXX 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -XXX,XX +XXX,XX @@ The maximum total size of the L2 table and refcount block caches in bytes @item l2-cache-size The maximum size of the L2 table cache in bytes -(default: if cache-size is not specified - 1M; otherwise, as large as possible -within the cache-size, while permitting the requested or the minimal refcount -cache size) +(default: if cache-size is not specified - 32M on Linux platforms, and 8M on +non-Linux platforms; otherwise, as large as possible within the cache-size, +while permitting the requested or the minimal refcount cache size) @item refcount-cache-size The maximum size of the refcount block cache in bytes -- 2.13.6 From: Leonid Bloch <lbloch@janustech.com> The caches are now recalculated upon image resizing. This is done because the new default behavior of assigning L2 cache relatively to the image size, implies that the cache will be adapted accordingly after an image resize. Signed-off-by: Leonid Bloch <lbloch@janustech.com> Reviewed-by: Alberto Garcia <berto@igalia.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, uint64_t old_length; int64_t new_l1_size; int ret; + QDict *options; if (prealloc != PREALLOC_MODE_OFF && prealloc != PREALLOC_MODE_METADATA && prealloc != PREALLOC_MODE_FALLOC && prealloc != PREALLOC_MODE_FULL) @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, } } + bs->total_sectors = offset / BDRV_SECTOR_SIZE; + /* write updated header.size */ offset = cpu_to_be64(offset); ret = bdrv_pwrite_sync(bs->file, offsetof(QCowHeader, size), @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, } s->l1_vm_state_index = new_l1_size; + + /* Update cache sizes */ + options = qdict_clone_shallow(bs->options); + ret = qcow2_update_options(bs, options, s->flags, errp); + qobject_unref(options); + if (ret < 0) { + goto fail; + } ret = 0; fail: qemu_co_mutex_unlock(&s->lock); -- 2.13.6 From: Leonid Bloch <lbloch@janustech.com> The default cache-clean-interval is set to 10 minutes, in order to lower the overhead of the qcow2 caches (before the default was 0, i.e. disabled). * For non-Linux platforms the default is kept at 0, because cache-clean-interval is not supported there yet. Signed-off-by: Leonid Bloch <lbloch@janustech.com> Reviewed-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qapi/block-core.json | 3 ++- docs/qcow2-cache.txt | 4 ++-- block/qcow2.h | 4 +++- block/qcow2.c | 2 +- qemu-options.hx | 2 +- 5 files changed, 9 insertions(+), 6 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ # # @cache-clean-interval: clean unused entries in the L2 and refcount # caches. The interval is in seconds. The default value -# is 0 and it disables this feature (since 2.5) +# is 600, and 0 disables this feature. (since 2.5) +# # @encrypt: Image decryption options. Mandatory for # encrypted images, except when doing a metadata-only # probe of the image. (since 2.10) diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt index XXXXXXX..XXXXXXX 100644 --- a/docs/qcow2-cache.txt +++ b/docs/qcow2-cache.txt @@ -XXX,XX +XXX,XX @@ This example removes all unused cache entries every 15 minutes: -drive file=hd.qcow2,cache-clean-interval=900 -If unset, the default value for this parameter is 0 and it disables -this feature. +If unset, the default value for this parameter is 600. Setting it to 0 +disables this feature. Note that this functionality currently relies on the MADV_DONTNEED argument for madvise() to actually free the memory. This is a diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ #ifdef CONFIG_LINUX #define DEFAULT_L2_CACHE_MAX_SIZE S_32MiB +#define DEFAULT_CACHE_CLEAN_INTERVAL 600 /* seconds */ #else #define DEFAULT_L2_CACHE_MAX_SIZE S_8MiB +/* Cache clean interval is currently available only on Linux, so must be 0 */ +#define DEFAULT_CACHE_CLEAN_INTERVAL 0 #endif #define DEFAULT_CLUSTER_SIZE S_64KiB - #define QCOW2_OPT_LAZY_REFCOUNTS "lazy-refcounts" #define QCOW2_OPT_DISCARD_REQUEST "pass-discard-request" #define QCOW2_OPT_DISCARD_SNAPSHOT "pass-discard-snapshot" diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_update_options_prepare(BlockDriverState *bs, /* New interval for cache cleanup timer */ r->cache_clean_interval = qemu_opt_get_number(opts, QCOW2_OPT_CACHE_CLEAN_INTERVAL, - s->cache_clean_interval); + DEFAULT_CACHE_CLEAN_INTERVAL); #ifndef CONFIG_LINUX if (r->cache_clean_interval != 0) { error_setg(errp, QCOW2_OPT_CACHE_CLEAN_INTERVAL diff --git a/qemu-options.hx b/qemu-options.hx index XXXXXXX..XXXXXXX 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -XXX,XX +XXX,XX @@ it which is not used for the L2 cache) @item cache-clean-interval Clean unused entries in the L2 and refcount caches. The interval is in seconds. -The default value is 0 and it disables this feature. +The default value is 600. Setting it to 0 disables this feature. @item pass-discard-request Whether discard requests to the qcow2 device should be forwarded to the data -- 2.13.6 From: Leonid Bloch <lbloch@janustech.com> Signed-off-by: Leonid Bloch <lbloch@janustech.com> Reviewed-by: Alberto Garcia <berto@igalia.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, /* 2^(s->refcount_order - 3) is the refcount width in bytes */ s->refcount_block_bits = s->cluster_bits - (s->refcount_order - 3); s->refcount_block_size = 1 << s->refcount_block_bits; - bs->total_sectors = header.size / 512; + bs->total_sectors = header.size / BDRV_SECTOR_SIZE; s->csize_shift = (62 - (s->cluster_bits - 8)); s->csize_mask = (1 << (s->cluster_bits - 8)) - 1; s->cluster_offset_mask = (1LL << s->csize_shift) - 1; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, goto fail; } - old_length = bs->total_sectors * 512; + old_length = bs->total_sectors * BDRV_SECTOR_SIZE; new_l1_size = size_to_l1(s, offset); if (offset < old_length) { -- 2.13.6 Currently, the default values for werror and rerror have to be set explicitly with blk_set_on_error() by the callers of blk_new(). The only caller actually doing this is blockdev_init(), which is called for BlockBackends created using -drive. In particular, anonymous BlockBackends created with -device ...,drive=<node-name> didn't get the correct default set and instead defaulted to the integer value 0 (= BLOCKDEV_ON_ERROR_REPORT). This is the intended default for rerror anyway, but the default for werror should be BLOCKDEV_ON_ERROR_ENOSPC. Set the defaults in blk_new() instead so that they apply no matter what way the BlockBackend was created. Cc: qemu-stable@nongnu.org Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Fam Zheng <famz@redhat.com> --- block/block-backend.c | 3 +++ tests/qemu-iotests/067.out | 1 + 2 files changed, 4 insertions(+) diff --git a/block/block-backend.c b/block/block-backend.c index XXXXXXX..XXXXXXX 100644 --- a/block/block-backend.c +++ b/block/block-backend.c @@ -XXX,XX +XXX,XX @@ BlockBackend *blk_new(uint64_t perm, uint64_t shared_perm) blk->shared_perm = shared_perm; blk_set_enable_write_cache(blk, true); + blk->on_read_error = BLOCKDEV_ON_ERROR_REPORT; + blk->on_write_error = BLOCKDEV_ON_ERROR_ENOSPC; + block_acct_init(&blk->stats); notifier_list_init(&blk->remove_bs_notifiers); diff --git a/tests/qemu-iotests/067.out b/tests/qemu-iotests/067.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/067.out +++ b/tests/qemu-iotests/067.out @@ -XXX,XX +XXX,XX @@ Testing: -device virtio-scsi -device scsi-cd,id=cd0 { "return": [ { + "io-status": "ok", "device": "", "locked": false, "removable": true, -- 2.13.6 From: Leonid Bloch <lbloch@janustech.com> Fixing cache-clean-interval documentation following the recent change to a default of 600 seconds on supported plarforms (only Linux currently). Signed-off-by: Leonid Bloch <lbloch@janustech.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qapi/block-core.json | 3 ++- docs/qcow2-cache.txt | 20 ++++++++++---------- qemu-options.hx | 3 ++- 3 files changed, 14 insertions(+), 12 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ # # @cache-clean-interval: clean unused entries in the L2 and refcount # caches. The interval is in seconds. The default value -# is 600, and 0 disables this feature. (since 2.5) +# is 600 on supporting platforms, and 0 on other +# platforms. 0 disables this feature. (since 2.5) # # @encrypt: Image decryption options. Mandatory for # encrypted images, except when doing a metadata-only diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt index XXXXXXX..XXXXXXX 100644 --- a/docs/qcow2-cache.txt +++ b/docs/qcow2-cache.txt @@ -XXX,XX +XXX,XX @@ Reducing the memory usage It is possible to clean unused cache entries in order to reduce the memory usage during periods of low I/O activity. -The parameter "cache-clean-interval" defines an interval (in seconds). -All cache entries that haven't been accessed during that interval are -removed from memory. +The parameter "cache-clean-interval" defines an interval (in seconds), +after which all the cache entries that haven't been accessed during the +interval are removed from memory. Setting this parameter to 0 disables this +feature. -This example removes all unused cache entries every 15 minutes: +The following example removes all unused cache entries every 15 minutes: -drive file=hd.qcow2,cache-clean-interval=900 -If unset, the default value for this parameter is 600. Setting it to 0 -disables this feature. +If unset, the default value for this parameter is 600 on platforms which +support this functionality, and is 0 (disabled) on other platforms. -Note that this functionality currently relies on the MADV_DONTNEED -argument for madvise() to actually free the memory. This is a -Linux-specific feature, so cache-clean-interval is not supported in -other systems. +This functionality currently relies on the MADV_DONTNEED argument for +madvise() to actually free the memory. This is a Linux-specific feature, +so cache-clean-interval is not supported on other systems. diff --git a/qemu-options.hx b/qemu-options.hx index XXXXXXX..XXXXXXX 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -XXX,XX +XXX,XX @@ it which is not used for the L2 cache) @item cache-clean-interval Clean unused entries in the L2 and refcount caches. The interval is in seconds. -The default value is 600. Setting it to 0 disables this feature. +The default value is 600 on supporting platforms, and 0 on other platforms. +Setting it to 0 disables this feature. @item pass-discard-request Whether discard requests to the qcow2 device should be forwarded to the data -- 2.13.6 Recently, the test case has started failing because some job related functions want to drop the AioContext lock even though it hasn't been taken: (gdb) bt #0 0x00007f51c067c9fb in raise () from /lib64/libc.so.6 #1 0x00007f51c067e77d in abort () from /lib64/libc.so.6 #2 0x0000558c9d5dde7b in error_exit (err=<optimized out>, msg=msg@entry=0x558c9d6fe120 <__func__.18373> "qemu_mutex_unlock_impl") at util/qemu-thread-posix.c:36 #3 0x0000558c9d6b5263 in qemu_mutex_unlock_impl (mutex=mutex@entry=0x558c9f3999a0, file=file@entry=0x558c9d6fd36f "util/async.c", line=line@entry=516) at util/qemu-thread-posix.c:96 #4 0x0000558c9d6b0565 in aio_context_release (ctx=ctx@entry=0x558c9f399940) at util/async.c:516 #5 0x0000558c9d5eb3da in job_completed_txn_abort (job=0x558c9f68e640) at job.c:738 #6 0x0000558c9d5eb227 in job_finish_sync (job=0x558c9f68e640, finish=finish@entry=0x558c9d5eb8d0 <job_cancel_err>, errp=errp@entry=0x0) at job.c:986 #7 0x0000558c9d5eb8ee in job_cancel_sync (job=<optimized out>) at job.c:941 #8 0x0000558c9d64d853 in replication_close (bs=<optimized out>) at block/replication.c:148 #9 0x0000558c9d5e5c9f in bdrv_close (bs=0x558c9f41b020) at block.c:3420 #10 bdrv_delete (bs=0x558c9f41b020) at block.c:3629 #11 bdrv_unref (bs=0x558c9f41b020) at block.c:4685 #12 0x0000558c9d62a3f3 in blk_remove_bs (blk=blk@entry=0x558c9f42a7c0) at block/block-backend.c:783 #13 0x0000558c9d62a667 in blk_delete (blk=0x558c9f42a7c0) at block/block-backend.c:402 #14 blk_unref (blk=0x558c9f42a7c0) at block/block-backend.c:457 #15 0x0000558c9d5dfcea in test_secondary_stop () at tests/test-replication.c:478 #16 0x00007f51c1f13178 in g_test_run_suite_internal () from /lib64/libglib-2.0.so.0 #17 0x00007f51c1f1337b in g_test_run_suite_internal () from /lib64/libglib-2.0.so.0 #18 0x00007f51c1f1337b in g_test_run_suite_internal () from /lib64/libglib-2.0.so.0 #19 0x00007f51c1f13552 in g_test_run_suite () from /lib64/libglib-2.0.so.0 #20 0x00007f51c1f13571 in g_test_run () from /lib64/libglib-2.0.so.0 #21 0x0000558c9d5de31f in main (argc=<optimized out>, argv=<optimized out>) at tests/test-replication.c:581 It is yet unclear whether this should really be considered a bug in the test case or whether blk_unref() should work for callers that haven't taken the AioContext lock, but in order to fix the build tests quickly, just take the AioContext lock around blk_unref(). Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/test-replication.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/tests/test-replication.c b/tests/test-replication.c index XXXXXXX..XXXXXXX 100644 --- a/tests/test-replication.c +++ b/tests/test-replication.c @@ -XXX,XX +XXX,XX @@ static BlockBackend *start_primary(void) static void teardown_primary(void) { BlockBackend *blk; + AioContext *ctx; /* remove P_ID */ blk = blk_by_name(P_ID); assert(blk); + ctx = blk_get_aio_context(blk); + aio_context_acquire(ctx); monitor_remove_blk(blk); blk_unref(blk); + aio_context_release(ctx); } static void test_primary_read(void) @@ -XXX,XX +XXX,XX @@ static void teardown_secondary(void) { /* only need to destroy two BBs */ BlockBackend *blk; + AioContext *ctx; /* remove S_LOCAL_DISK_ID */ blk = blk_by_name(S_LOCAL_DISK_ID); assert(blk); + ctx = blk_get_aio_context(blk); + aio_context_acquire(ctx); monitor_remove_blk(blk); blk_unref(blk); + aio_context_release(ctx); /* remove S_ID */ blk = blk_by_name(S_ID); assert(blk); + ctx = blk_get_aio_context(blk); + aio_context_acquire(ctx); monitor_remove_blk(blk); blk_unref(blk); + aio_context_release(ctx); } static void test_secondary_read(void) -- 2.13.6 qemu_event_reset() must be called before the AIO request in a different iothread is submitted. Otherwise the request could be completed before we do the qemu_event_reset() and the test would hang in qemu_event_wait(). Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Tested-by: Max Reitz <mreitz@redhat.com> --- tests/test-bdrv-drain.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test-bdrv-drain.c b/tests/test-bdrv-drain.c index XXXXXXX..XXXXXXX 100644 --- a/tests/test-bdrv-drain.c +++ b/tests/test-bdrv-drain.c @@ -XXX,XX +XXX,XX @@ static void test_iothread_common(enum drain_type drain_type, int drain_thread) s->bh_indirection_ctx = ctx_b; aio_ret = -EINPROGRESS; + qemu_event_reset(&done_event); + if (drain_thread == 0) { acb = blk_aio_preadv(blk, 0, &qiov, 0, test_iothread_aio_cb, &aio_ret); } else { @@ -XXX,XX +XXX,XX @@ static void test_iothread_common(enum drain_type drain_type, int drain_thread) * but the drain in this thread can continue immediately after * bdrv_dec_in_flight() and aio_ret might be assigned only slightly * later. */ - qemu_event_reset(&done_event); do_drain_begin(drain_type, bs); g_assert_cmpint(bs->in_flight, ==, 0); @@ -XXX,XX +XXX,XX @@ static void test_iothread_common(enum drain_type drain_type, int drain_thread) } break; case 1: - qemu_event_reset(&done_event); aio_bh_schedule_oneshot(ctx_a, test_iothread_drain_entry, &data); qemu_event_wait(&done_event); break; -- 2.13.6 The following changes since commit c4e0780ed1ffd056f205348d387a61b4136a45df: Merge remote-tracking branch 'remotes/vivier2/tags/linux-user-for-4.0-pull-request' into staging (2019-03-07 18:40:43 +0000) are available in the Git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for you to fetch changes up to e88153ea9a40009a8ae7648282c0eac1b7f5494f: qcow2 spec: Describe string header extensions (2019-03-08 12:26:46 +0100) ---------------------------------------------------------------- Block layer patches: - qcow2: Support for external data files - qcow2: Default to 4KB for the qcow2 cache entry size - Apply block driver whitelist for -drive format=help - Several qemu-iotests improvements ---------------------------------------------------------------- Alberto Garcia (1): qcow2: Default to 4KB for the qcow2 cache entry size Andrey Shinkevich (4): iotests: open notrun files in text mode block: iterate_format with account of whitelisting iotests: ask QEMU for supported formats iotests: check whitelisted formats Kevin Wolf (21): qemu-iotests: Test qcow2 preallocation modes qcow2: Simplify preallocation code qcow2: Extend spec for external data files qcow2: Basic definitions for external data files qcow2: Pass bs to qcow2_get_cluster_type() qcow2: Prepare qcow2_get_cluster_type() for external data file qcow2: Prepare count_contiguous_clusters() for external data file qcow2: Don't assume 0 is an invalid cluster offset qcow2: Return 0/-errno in qcow2_alloc_compressed_cluster_offset() qcow2: Prepare qcow2_co_block_status() for data file qcow2: External file I/O qcow2: Return error for snapshot operation with data file qcow2: Support external data file in qemu-img check qcow2: Add basic data-file infrastructure qcow2: Creating images with external data file qcow2: Store data file name in the image qcow2: Implement data-file-raw create option qemu-iotests: Preallocation with external data file qemu-iotests: General tests for qcow2 with external data file qemu-iotests: amend with external data file qcow2 spec: Describe string header extensions Philippe Mathieu-Daudé (6): tests/multiboot: Improve portability by searching bash in the $PATH tests/bios-tables: Improve portability by searching bash in the $PATH qemu-iotests: Improve portability by searching bash in the $PATH qemu-iotests: Ensure GNU sed is used ahci-test: Add dependency to qemu-img tool qemu-iotests: Add dependency to qemu-nbd tool Stefan Hajnoczi (1): iotests: use iotests.VM in 238 qapi/block-core.json | 26 ++- docs/interop/qcow2.txt | 54 +++++- docs/qcow2-cache.txt | 17 +- block/qcow2.h | 66 +++++-- include/block/block.h | 2 +- include/block/block_int.h | 2 + block.c | 23 ++- block/qcow2-bitmap.c | 7 +- block/qcow2-cache.c | 6 +- block/qcow2-cluster.c | 182 +++++++++++------- block/qcow2-refcount.c | 88 ++++++--- block/qcow2-snapshot.c | 22 ++- block/qcow2.c | 326 +++++++++++++++++++++++++++----- blockdev.c | 4 +- qemu-img.c | 2 +- tests/Makefile.include | 4 +- tests/data/acpi/rebuild-expected-aml.sh | 2 +- tests/multiboot/run_test.sh | 2 +- tests/qemu-iotests/001 | 2 +- tests/qemu-iotests/002 | 2 +- tests/qemu-iotests/003 | 2 +- tests/qemu-iotests/004 | 2 +- tests/qemu-iotests/005 | 2 +- tests/qemu-iotests/007 | 2 +- tests/qemu-iotests/008 | 2 +- tests/qemu-iotests/009 | 2 +- tests/qemu-iotests/010 | 2 +- tests/qemu-iotests/011 | 2 +- tests/qemu-iotests/012 | 2 +- tests/qemu-iotests/013 | 2 +- tests/qemu-iotests/014 | 2 +- tests/qemu-iotests/015 | 2 +- tests/qemu-iotests/017 | 2 +- tests/qemu-iotests/018 | 2 +- tests/qemu-iotests/019 | 2 +- tests/qemu-iotests/020 | 2 +- tests/qemu-iotests/021 | 2 +- tests/qemu-iotests/022 | 2 +- tests/qemu-iotests/023 | 2 +- tests/qemu-iotests/024 | 2 +- tests/qemu-iotests/025 | 2 +- tests/qemu-iotests/026 | 2 +- tests/qemu-iotests/027 | 2 +- tests/qemu-iotests/028 | 2 +- tests/qemu-iotests/029 | 2 +- tests/qemu-iotests/031 | 2 +- tests/qemu-iotests/031.out | 8 +- tests/qemu-iotests/032 | 2 +- tests/qemu-iotests/033 | 2 +- tests/qemu-iotests/034 | 2 +- tests/qemu-iotests/035 | 2 +- tests/qemu-iotests/036 | 2 +- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/037 | 2 +- tests/qemu-iotests/038 | 2 +- tests/qemu-iotests/039 | 2 +- tests/qemu-iotests/042 | 2 +- tests/qemu-iotests/043 | 2 +- tests/qemu-iotests/046 | 2 +- tests/qemu-iotests/047 | 2 +- tests/qemu-iotests/048 | 2 +- tests/qemu-iotests/049 | 2 +- tests/qemu-iotests/050 | 2 +- tests/qemu-iotests/051 | 2 +- tests/qemu-iotests/052 | 2 +- tests/qemu-iotests/053 | 2 +- tests/qemu-iotests/054 | 2 +- tests/qemu-iotests/058 | 2 +- tests/qemu-iotests/059 | 2 +- tests/qemu-iotests/060 | 2 +- tests/qemu-iotests/061 | 47 ++++- tests/qemu-iotests/061.out | 103 +++++++++- tests/qemu-iotests/062 | 2 +- tests/qemu-iotests/063 | 2 +- tests/qemu-iotests/064 | 2 +- tests/qemu-iotests/066 | 2 +- tests/qemu-iotests/067 | 2 +- tests/qemu-iotests/068 | 2 +- tests/qemu-iotests/069 | 2 +- tests/qemu-iotests/070 | 2 +- tests/qemu-iotests/071 | 2 +- tests/qemu-iotests/072 | 2 +- tests/qemu-iotests/073 | 2 +- tests/qemu-iotests/074 | 2 +- tests/qemu-iotests/075 | 2 +- tests/qemu-iotests/076 | 2 +- tests/qemu-iotests/077 | 2 +- tests/qemu-iotests/078 | 2 +- tests/qemu-iotests/079 | 2 +- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/081 | 2 +- tests/qemu-iotests/082 | 2 +- tests/qemu-iotests/082.out | 54 ++++++ tests/qemu-iotests/083 | 2 +- tests/qemu-iotests/084 | 2 +- tests/qemu-iotests/085 | 2 +- tests/qemu-iotests/086 | 2 +- tests/qemu-iotests/087 | 2 +- tests/qemu-iotests/088 | 2 +- tests/qemu-iotests/089 | 2 +- tests/qemu-iotests/090 | 2 +- tests/qemu-iotests/091 | 2 +- tests/qemu-iotests/092 | 2 +- tests/qemu-iotests/094 | 2 +- tests/qemu-iotests/095 | 2 +- tests/qemu-iotests/097 | 2 +- tests/qemu-iotests/098 | 2 +- tests/qemu-iotests/099 | 2 +- tests/qemu-iotests/101 | 2 +- tests/qemu-iotests/102 | 2 +- tests/qemu-iotests/103 | 2 +- tests/qemu-iotests/104 | 2 +- tests/qemu-iotests/105 | 2 +- tests/qemu-iotests/106 | 2 +- tests/qemu-iotests/107 | 2 +- tests/qemu-iotests/108 | 2 +- tests/qemu-iotests/109 | 2 +- tests/qemu-iotests/110 | 2 +- tests/qemu-iotests/111 | 2 +- tests/qemu-iotests/112 | 2 +- tests/qemu-iotests/113 | 2 +- tests/qemu-iotests/114 | 2 +- tests/qemu-iotests/115 | 2 +- tests/qemu-iotests/116 | 2 +- tests/qemu-iotests/117 | 2 +- tests/qemu-iotests/119 | 2 +- tests/qemu-iotests/120 | 2 +- tests/qemu-iotests/121 | 2 +- tests/qemu-iotests/122 | 2 +- tests/qemu-iotests/123 | 2 +- tests/qemu-iotests/125 | 2 +- tests/qemu-iotests/126 | 2 +- tests/qemu-iotests/127 | 2 +- tests/qemu-iotests/128 | 2 +- tests/qemu-iotests/130 | 2 +- tests/qemu-iotests/131 | 2 +- tests/qemu-iotests/133 | 2 +- tests/qemu-iotests/134 | 2 +- tests/qemu-iotests/135 | 2 +- tests/qemu-iotests/137 | 2 +- tests/qemu-iotests/138 | 2 +- tests/qemu-iotests/139 | 3 + tests/qemu-iotests/140 | 2 +- tests/qemu-iotests/141 | 2 +- tests/qemu-iotests/142 | 2 +- tests/qemu-iotests/143 | 2 +- tests/qemu-iotests/144 | 2 +- tests/qemu-iotests/145 | 2 +- tests/qemu-iotests/146 | 2 +- tests/qemu-iotests/150 | 2 +- tests/qemu-iotests/153 | 2 +- tests/qemu-iotests/154 | 2 +- tests/qemu-iotests/156 | 2 +- tests/qemu-iotests/157 | 2 +- tests/qemu-iotests/158 | 2 +- tests/qemu-iotests/159 | 2 +- tests/qemu-iotests/160 | 2 +- tests/qemu-iotests/161 | 2 +- tests/qemu-iotests/162 | 2 +- tests/qemu-iotests/170 | 2 +- tests/qemu-iotests/171 | 2 +- tests/qemu-iotests/172 | 2 +- tests/qemu-iotests/173 | 2 +- tests/qemu-iotests/174 | 2 +- tests/qemu-iotests/175 | 2 +- tests/qemu-iotests/176 | 2 +- tests/qemu-iotests/177 | 2 +- tests/qemu-iotests/178 | 2 +- tests/qemu-iotests/179 | 2 +- tests/qemu-iotests/181 | 2 +- tests/qemu-iotests/182 | 2 +- tests/qemu-iotests/183 | 2 +- tests/qemu-iotests/184 | 2 +- tests/qemu-iotests/185 | 2 +- tests/qemu-iotests/186 | 2 +- tests/qemu-iotests/187 | 2 +- tests/qemu-iotests/188 | 2 +- tests/qemu-iotests/189 | 2 +- tests/qemu-iotests/190 | 2 +- tests/qemu-iotests/191 | 2 +- tests/qemu-iotests/192 | 2 +- tests/qemu-iotests/195 | 2 +- tests/qemu-iotests/197 | 2 +- tests/qemu-iotests/198 | 2 +- tests/qemu-iotests/200 | 2 +- tests/qemu-iotests/201 | 2 +- tests/qemu-iotests/204 | 2 +- tests/qemu-iotests/214 | 2 +- tests/qemu-iotests/215 | 2 +- tests/qemu-iotests/217 | 2 +- tests/qemu-iotests/220 | 2 +- tests/qemu-iotests/220.out | 2 +- tests/qemu-iotests/221 | 2 +- tests/qemu-iotests/223 | 2 +- tests/qemu-iotests/225 | 2 +- tests/qemu-iotests/226 | 2 +- tests/qemu-iotests/227 | 2 +- tests/qemu-iotests/229 | 2 +- tests/qemu-iotests/231 | 2 +- tests/qemu-iotests/232 | 2 +- tests/qemu-iotests/233 | 2 +- tests/qemu-iotests/238 | 7 +- tests/qemu-iotests/243 | 85 +++++++++ tests/qemu-iotests/243.out | 58 ++++++ tests/qemu-iotests/244 | 200 ++++++++++++++++++++ tests/qemu-iotests/244.out | 125 ++++++++++++ tests/qemu-iotests/check | 15 +- tests/qemu-iotests/common.config | 2 +- tests/qemu-iotests/common.filter | 38 ++-- tests/qemu-iotests/common.nbd | 2 +- tests/qemu-iotests/common.pattern | 2 +- tests/qemu-iotests/common.qemu | 2 +- tests/qemu-iotests/common.rc | 15 +- tests/qemu-iotests/common.tls | 2 +- tests/qemu-iotests/group | 2 + tests/qemu-iotests/iotests.py | 45 ++++- 216 files changed, 1585 insertions(+), 423 deletions(-) create mode 100755 tests/qemu-iotests/243 create mode 100644 tests/qemu-iotests/243.out create mode 100755 tests/qemu-iotests/244 create mode 100644 tests/qemu-iotests/244.out From: Stefan Hajnoczi <stefanha@redhat.com> Test 238 does not require the kvm accelerator. Using the qtest accelerator allows the test to run in both non-kvm and non-tcg environments. iotests.VM implicitly uses the qtest accelerator and is really the class that this test should be using. Switch to that instead of qemu.QEMUMachine. Suggested-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/qemu-iotests/238 | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/tests/qemu-iotests/238 b/tests/qemu-iotests/238 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/238 +++ b/tests/qemu-iotests/238 @@ -XXX,XX +XXX,XX @@ import os import iotests from iotests import log -sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python')) - -from qemu import QEMUMachine - if iotests.qemu_default_machine == 's390-ccw-virtio': virtio_scsi_device = 'virtio-scsi-ccw' else: virtio_scsi_device = 'virtio-scsi-pci' -vm = QEMUMachine(iotests.qemu_prog) -vm.add_args('-machine', 'accel=kvm') +vm = iotests.VM() vm.launch() log(vm.qmp('blockdev-add', node_name='hd0', driver='null-co')) -- 2.20.1 From: Alberto Garcia <berto@igalia.com> QEMU 2.12 (commit 1221fe6f636754ab5f2c1c87caa77633e9123622) introduced a new setting called l2-cache-entry-size that allows making entries on the qcow2 L2 cache smaller than the cluster size. I have been performing several tests with different cluster and entry sizes and all of them show that reducing the entry size (aka L2 slice) consistently improves I/O performance, notably during random I/O (all tests done with sequential I/O show similar results). This is to be expected because loading and evicting an L2 slice is more expensive the larger the slice is. Here are some numbers on fully populated 40GB qcow2 images. The rightmost column represents the maximum L2 cache size in both cases. Cluster size = 64 KB |-------------+--------------+--------------+--------------| | | 1MB L2 cache | 3MB L2 cache | 5MB L2 cache | |-------------+--------------+--------------+--------------| | 4KB slices | 6545 IOPS | 12045 IOPS | 55680 IOPS | | 16KB slices | 5177 IOPS | 9798 IOPS | 56278 IOPS | | 64KB slices | 2718 IOPS | 5326 IOPS | 57355 IOPS | |-------------+--------------+--------------+--------------| Cluster size = 256 KB |--------------+----------------+--------------+-----------------| | | 512KB L2 cache | 1MB L2 cache | 1280KB L2 cache | |--------------+----------------+--------------+-----------------| | 4KB slices | 8539 IOPS | 21071 IOPS | 55417 IOPS | | 64KB slices | 3598 IOPS | 9772 IOPS | 57687 IOPS | | 256KB slices | 1415 IOPS | 4120 IOPS | 58001 IOPS | |--------------+----------------+--------------+-----------------| As can be seen in the numbers, the only exception to the rule is when the cache is large enough to hold all L2 tables. This is also to be expected because in this case no cache entry is ever evicted so reducing its size doesn't bring any benefit. This patch sets the default L2 cache entry size to 4KB except when the cache is large enough for the whole disk. Signed-off-by: Alberto Garcia <berto@igalia.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- docs/qcow2-cache.txt | 17 +++++++++++------ block/qcow2.c | 12 ++++++++++++ 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt index XXXXXXX..XXXXXXX 100644 --- a/docs/qcow2-cache.txt +++ b/docs/qcow2-cache.txt @@ -XXX,XX +XXX,XX @@ refcount cache is as small as possible unless overridden by the user. Using smaller cache entries --------------------------- -The qcow2 L2 cache stores complete tables by default. This means that -if QEMU needs an entry from an L2 table then the whole table is read -from disk and is kept in the cache. If the cache is full then a -complete table needs to be evicted first. +The qcow2 L2 cache can store complete tables. This means that if QEMU +needs an entry from an L2 table then the whole table is read from disk +and is kept in the cache. If the cache is full then a complete table +needs to be evicted first. This can be inefficient with large cluster sizes since it results in more disk I/O and wastes more cache memory. @@ -XXX,XX +XXX,XX @@ it smaller than the cluster size. This can be configured using the -drive file=hd.qcow2,l2-cache-size=2097152,l2-cache-entry-size=4096 +Since QEMU 4.0 the value of l2-cache-entry-size defaults to 4KB (or +the cluster size if it's smaller). + Some things to take into account: - The L2 cache entry size has the same restrictions as the cluster @@ -XXX,XX +XXX,XX @@ Some things to take into account: - Try different entry sizes to see which one gives faster performance in your case. The block size of the host filesystem is generally a - good default (usually 4096 bytes in the case of ext4). + good default (usually 4096 bytes in the case of ext4, hence the + default). - Only the L2 cache can be configured this way. The refcount cache always uses the cluster size as the entry size. @@ -XXX,XX +XXX,XX @@ Some things to take into account: (as explained in the "Choosing the right cache sizes" and "How to configure the cache sizes" sections in this document) then none of this is necessary and you can omit the "l2-cache-entry-size" - parameter altogether. + parameter altogether. In this case QEMU makes the entry size + equal to the cluster size by default. Reducing the memory usage diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts, BDRVQcow2State *s = bs->opaque; uint64_t combined_cache_size, l2_cache_max_setting; bool l2_cache_size_set, refcount_cache_size_set, combined_cache_size_set; + bool l2_cache_entry_size_set; int min_refcount_cache = MIN_REFCOUNT_CACHE_SIZE * s->cluster_size; uint64_t virtual_disk_size = bs->total_sectors * BDRV_SECTOR_SIZE; uint64_t max_l2_cache = virtual_disk_size / (s->cluster_size / 8); @@ -XXX,XX +XXX,XX @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts, combined_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_CACHE_SIZE); l2_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_L2_CACHE_SIZE); refcount_cache_size_set = qemu_opt_get(opts, QCOW2_OPT_REFCOUNT_CACHE_SIZE); + l2_cache_entry_size_set = qemu_opt_get(opts, QCOW2_OPT_L2_CACHE_ENTRY_SIZE); combined_cache_size = qemu_opt_get_size(opts, QCOW2_OPT_CACHE_SIZE, 0); l2_cache_max_setting = qemu_opt_get_size(opts, QCOW2_OPT_L2_CACHE_SIZE, @@ -XXX,XX +XXX,XX @@ static void read_cache_sizes(BlockDriverState *bs, QemuOpts *opts, } } } + + /* + * If the L2 cache is not enough to cover the whole disk then + * default to 4KB entries. Smaller entries reduce the cost of + * loads and evictions and increase I/O performance. + */ + if (*l2_cache_size < max_l2_cache && !l2_cache_entry_size_set) { + *l2_cache_entry_size = MIN(s->cluster_size, 4096); + } + /* l2_cache_size and refcount_cache_size are ensured to have at least * their minimum values in qcow2_update_options_prepare() */ -- 2.20.1 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Replace the binary mode with the default text one when *.notrun files are opened for skipped tests. That change is made for the compatibility with Python 3 which returns error otherwise. Signed-off-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ def notrun(reason): # Each test in qemu-iotests has a number ("seq") seq = os.path.basename(sys.argv[0]) - open('%s/%s.notrun' % (output_dir, seq), 'wb').write(reason + '\n') + open('%s/%s.notrun' % (output_dir, seq), 'w').write(reason + '\n') print('%s not run: %s' % (seq, reason)) sys.exit(0) -- 2.20.1 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> bdrv_iterate_format (which is currently only used for printing out the formats supported by the block layer) doesn't take format whitelisting into account. This creates a problem for tests: they enumerate supported formats to decide which tests to enable, but then discover that QEMU doesn't let them actually use some of those formats. To avoid that, exclude formats that are not whitelisted from enumeration, if whitelisting is in use. Since we have separate whitelists for r/w and r/o, take this a parameter to bdrv_iterate_format, and print two lists of supported formats (r/w and r/o) in main qemu. Signed-off-by: Roman Kagan <rkagan@virtuozzo.com> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- include/block/block.h | 2 +- block.c | 23 +++++++++++++++++++---- blockdev.c | 4 +++- qemu-img.c | 2 +- 4 files changed, 24 insertions(+), 7 deletions(-) diff --git a/include/block/block.h b/include/block/block.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block.h +++ b/include/block/block.h @@ -XXX,XX +XXX,XX @@ void bdrv_next_cleanup(BdrvNextIterator *it); BlockDriverState *bdrv_next_monitor_owned(BlockDriverState *bs); bool bdrv_is_encrypted(BlockDriverState *bs); void bdrv_iterate_format(void (*it)(void *opaque, const char *name), - void *opaque); + void *opaque, bool read_only); const char *bdrv_get_node_name(const BlockDriverState *bs); const char *bdrv_get_device_name(const BlockDriverState *bs); const char *bdrv_get_device_or_node_name(const BlockDriverState *bs); diff --git a/block.c b/block.c index XXXXXXX..XXXXXXX 100644 --- a/block.c +++ b/block.c @@ -XXX,XX +XXX,XX @@ BlockDriver *bdrv_find_format(const char *format_name) return bdrv_do_find_format(format_name); } -int bdrv_is_whitelisted(BlockDriver *drv, bool read_only) +static int bdrv_format_is_whitelisted(const char *format_name, bool read_only) { static const char *whitelist_rw[] = { CONFIG_BDRV_RW_WHITELIST @@ -XXX,XX +XXX,XX @@ int bdrv_is_whitelisted(BlockDriver *drv, bool read_only) } for (p = whitelist_rw; *p; p++) { - if (!strcmp(drv->format_name, *p)) { + if (!strcmp(format_name, *p)) { return 1; } } if (read_only) { for (p = whitelist_ro; *p; p++) { - if (!strcmp(drv->format_name, *p)) { + if (!strcmp(format_name, *p)) { return 1; } } @@ -XXX,XX +XXX,XX @@ int bdrv_is_whitelisted(BlockDriver *drv, bool read_only) return 0; } +int bdrv_is_whitelisted(BlockDriver *drv, bool read_only) +{ + return bdrv_format_is_whitelisted(drv->format_name, read_only); +} + bool bdrv_uses_whitelist(void) { return use_bdrv_whitelist; @@ -XXX,XX +XXX,XX @@ static int qsort_strcmp(const void *a, const void *b) } void bdrv_iterate_format(void (*it)(void *opaque, const char *name), - void *opaque) + void *opaque, bool read_only) { BlockDriver *drv; int count = 0; @@ -XXX,XX +XXX,XX @@ void bdrv_iterate_format(void (*it)(void *opaque, const char *name), if (drv->format_name) { bool found = false; int i = count; + + if (use_bdrv_whitelist && !bdrv_is_whitelisted(drv, read_only)) { + continue; + } + while (formats && i && !found) { found = !strcmp(formats[--i], drv->format_name); } @@ -XXX,XX +XXX,XX @@ void bdrv_iterate_format(void (*it)(void *opaque, const char *name), bool found = false; int j = count; + if (use_bdrv_whitelist && + !bdrv_format_is_whitelisted(format_name, read_only)) { + continue; + } + while (formats && j && !found) { found = !strcmp(formats[--j], format_name); } diff --git a/blockdev.c b/blockdev.c index XXXXXXX..XXXXXXX 100644 --- a/blockdev.c +++ b/blockdev.c @@ -XXX,XX +XXX,XX @@ static BlockBackend *blockdev_init(const char *file, QDict *bs_opts, if ((buf = qemu_opt_get(opts, "format")) != NULL) { if (is_help_option(buf)) { error_printf("Supported formats:"); - bdrv_iterate_format(bdrv_format_print, NULL); + bdrv_iterate_format(bdrv_format_print, NULL, false); + error_printf("\nSupported formats (read-only):"); + bdrv_iterate_format(bdrv_format_print, NULL, true); error_printf("\n"); goto early_err; } diff --git a/qemu-img.c b/qemu-img.c index XXXXXXX..XXXXXXX 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -XXX,XX +XXX,XX @@ static void QEMU_NORETURN help(void) " 'skip=N' skip N bs-sized blocks at the start of input\n"; printf("%s\nSupported formats:", help_msg); - bdrv_iterate_format(format_print, NULL); + bdrv_iterate_format(format_print, NULL, false); printf("\n\n" QEMU_HELP_BOTTOM "\n"); exit(EXIT_SUCCESS); } -- 2.20.1 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Supported formats listed by 'qemu' may differ from those listed by 'qemu-img' due to whitelists. Some test cases require specific formats that may be used with qemu. They can be inquired directly by running 'qemu -drive format=help'. The response takes whitelists into account. The method supported_formats() serves for that. The method decorator skip_if_unsupported() checks if all requested formats are whitelisted. If not, the test case will be skipped. That has been implemented in the 'check' file in the way similar to the 'test notrun' mechanism. Suggested-by: Roman Kagan <rkagan@virtuozzo.com> Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> Suggested-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/qemu-iotests/check | 13 ++++++++++- tests/qemu-iotests/iotests.py | 43 +++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/check +++ b/tests/qemu-iotests/check @@ -XXX,XX +XXX,XX @@ try=0 n_bad=0 bad="" notrun="" +casenotrun="" interrupt=true # by default don't output timestamps @@ -XXX,XX +XXX,XX @@ END { if (NR > 0) { echo "Not run:$notrun" echo "Not run:$notrun" >>check.log fi + if [ ! -z "$casenotrun" ] + then + echo "Some cases not run in:$casenotrun" + echo "Some cases not run in:$casenotrun" >>check.log + fi if [ ! -z "$n_bad" -a $n_bad != 0 ] then echo "Failures:$bad" @@ -XXX,XX +XXX,XX @@ do printf " " # prettier output with timestamps. fi rm -f core $seq.notrun + rm -f $seq.casenotrun start=$(_wallclock) $timestamp && printf %s " [$(date "+%T")]" @@ -XXX,XX +XXX,XX @@ do fi fi fi - + if [ -f $seq.casenotrun ] + then + cat $seq.casenotrun + casenotrun="$casenotrun $seq" + fi fi # come here for each test, except when $showme is true diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -XXX,XX +XXX,XX @@ def notrun(reason): print('%s not run: %s' % (seq, reason)) sys.exit(0) +def case_notrun(reason): + '''Skip this test case''' + # Each test in qemu-iotests has a number ("seq") + seq = os.path.basename(sys.argv[0]) + + open('%s/%s.casenotrun' % (output_dir, seq), 'a').write( + ' [case not run] ' + reason + '\n') + def verify_image_format(supported_fmts=[], unsupported_fmts=[]): assert not (supported_fmts and unsupported_fmts) @@ -XXX,XX +XXX,XX @@ def verify_quorum(): if not supports_quorum(): notrun('quorum support missing') +def qemu_pipe(*args): + '''Run qemu with an option to print something and exit (e.g. a help option), + and return its output''' + args = [qemu_prog] + qemu_opts + list(args) + subp = subprocess.Popen(args, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + universal_newlines=True) + exitcode = subp.wait() + if exitcode < 0: + sys.stderr.write('qemu received signal %i: %s\n' % (-exitcode, + ' '.join(args))) + return subp.communicate()[0] + +def supported_formats(read_only=False): + '''Set 'read_only' to True to check ro-whitelist + Otherwise, rw-whitelist is checked''' + format_message = qemu_pipe("-drive", "format=help") + line = 1 if read_only else 0 + return format_message.splitlines()[line].split(":")[1].split() + +def skip_if_unsupported(required_formats=[], read_only=False): + '''Skip Test Decorator + Runs the test if all the required formats are whitelisted''' + def skip_test_decorator(func): + def func_wrapper(*args, **kwargs): + usf_list = list(set(required_formats) - + set(supported_formats(read_only))) + if usf_list: + case_notrun('{}: formats {} are not whitelisted'.format( + args[0], usf_list)) + else: + return func(*args, **kwargs) + return func_wrapper + return skip_test_decorator + def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], unsupported_fmts=[]): '''Run tests''' -- 2.20.1 From: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Some test cases require specific formats. The method decorator skip_if_unsupported() checks if requested formats are whitelisted. The test #139 was selected for a sample output, after running $ ./check -qcow2 131-140 137 3s ... 138 0s ... 139 2s ... [case not run] testBlkDebug (__main__.TestBlockdevDel): formats ['blkdebug'] are not whitelisted [case not run] testBlkVerify (__main__.TestBlockdevDel): formats ['blkverify'] are not whitelisted [case not run] testQuorum (__main__.TestBlockdevDel): formats ['quorum'] are not whitelisted 140 0s ... Not run: 131 135 136 Some cases not run in: 139 Passed all 7 tests Signed-off-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/qemu-iotests/139 | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/139 +++ b/tests/qemu-iotests/139 @@ -XXX,XX +XXX,XX @@ class TestBlockdevDel(iotests.QMPTestCase): # FIXME mirror0 disappears, drive-mirror doesn't take a reference #self.delBlockDriverState('mirror0') + @iotests.skip_if_unsupported(['blkdebug']) def testBlkDebug(self): self.addBlkDebug('debug0', 'node0') # 'node0' is used by the blkdebug node @@ -XXX,XX +XXX,XX @@ class TestBlockdevDel(iotests.QMPTestCase): self.delBlockDriverState('debug0') self.checkBlockDriverState('node0', False) + @iotests.skip_if_unsupported(['blkverify']) def testBlkVerify(self): self.addBlkVerify('verify0', 'node0', 'node1') # We cannot remove the children of a blkverify device @@ -XXX,XX +XXX,XX @@ class TestBlockdevDel(iotests.QMPTestCase): self.checkBlockDriverState('node0', False) self.checkBlockDriverState('node1', False) + @iotests.skip_if_unsupported(['quorum']) def testQuorum(self): if not iotests.supports_quorum(): return -- 2.20.1 From: Philippe Mathieu-Daudé <philmd@redhat.com> Bash is not always installed as /bin/bash. In particular on OpenBSD, the package installs it in /usr/local/bin. Use the 'env' shebang to search bash in the $PATH. Reviewed-by: Kamil Rytarowski <n54@gmx.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/multiboot/run_test.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/multiboot/run_test.sh b/tests/multiboot/run_test.sh index XXXXXXX..XXXXXXX 100755 --- a/tests/multiboot/run_test.sh +++ b/tests/multiboot/run_test.sh @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # Copyright (c) 2013 Kevin Wolf <kwolf@redhat.com> # -- 2.20.1 From: Philippe Mathieu-Daudé <philmd@redhat.com> Bash is not always installed as /bin/bash. In particular on OpenBSD, the package installs it in /usr/local/bin. Use the 'env' shebang to search bash in the $PATH. Reviewed-by: Kamil Rytarowski <n54@gmx.com> Reviewed-by: Igor Mammedov <imammedo@redhat.com> Reviewed-by: Michael S. Tsirkin <mst@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Acked-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/data/acpi/rebuild-expected-aml.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/data/acpi/rebuild-expected-aml.sh b/tests/data/acpi/rebuild-expected-aml.sh index XXXXXXX..XXXXXXX 100755 --- a/tests/data/acpi/rebuild-expected-aml.sh +++ b/tests/data/acpi/rebuild-expected-aml.sh @@ -XXX,XX +XXX,XX @@ -#! /bin/bash +#!/usr/bin/env bash # # Rebuild expected AML files for acpi unit-test -- 2.20.1 From: Philippe Mathieu-Daudé <philmd@redhat.com> Bash is not always installed as /bin/bash. In particular on OpenBSD, the package installs it in /usr/local/bin. Use the 'env' shebang to search bash in the $PATH. Patch created mechanically by running: $ git grep -lE '#! ?/bin/bash' -- tests/qemu-iotests \ | while read f; do \ sed -i 's|^#!.\?/bin/bash$|#!/usr/bin/env bash|' $f; \ done Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/qemu-iotests/001 | 2 +- tests/qemu-iotests/002 | 2 +- tests/qemu-iotests/003 | 2 +- tests/qemu-iotests/004 | 2 +- tests/qemu-iotests/005 | 2 +- tests/qemu-iotests/007 | 2 +- tests/qemu-iotests/008 | 2 +- tests/qemu-iotests/009 | 2 +- tests/qemu-iotests/010 | 2 +- tests/qemu-iotests/011 | 2 +- tests/qemu-iotests/012 | 2 +- tests/qemu-iotests/013 | 2 +- tests/qemu-iotests/014 | 2 +- tests/qemu-iotests/015 | 2 +- tests/qemu-iotests/017 | 2 +- tests/qemu-iotests/018 | 2 +- tests/qemu-iotests/019 | 2 +- tests/qemu-iotests/020 | 2 +- tests/qemu-iotests/021 | 2 +- tests/qemu-iotests/022 | 2 +- tests/qemu-iotests/023 | 2 +- tests/qemu-iotests/024 | 2 +- tests/qemu-iotests/025 | 2 +- tests/qemu-iotests/026 | 2 +- tests/qemu-iotests/027 | 2 +- tests/qemu-iotests/028 | 2 +- tests/qemu-iotests/029 | 2 +- tests/qemu-iotests/031 | 2 +- tests/qemu-iotests/032 | 2 +- tests/qemu-iotests/033 | 2 +- tests/qemu-iotests/034 | 2 +- tests/qemu-iotests/035 | 2 +- tests/qemu-iotests/036 | 2 +- tests/qemu-iotests/037 | 2 +- tests/qemu-iotests/038 | 2 +- tests/qemu-iotests/039 | 2 +- tests/qemu-iotests/042 | 2 +- tests/qemu-iotests/043 | 2 +- tests/qemu-iotests/046 | 2 +- tests/qemu-iotests/047 | 2 +- tests/qemu-iotests/048 | 2 +- tests/qemu-iotests/049 | 2 +- tests/qemu-iotests/050 | 2 +- tests/qemu-iotests/051 | 2 +- tests/qemu-iotests/052 | 2 +- tests/qemu-iotests/053 | 2 +- tests/qemu-iotests/054 | 2 +- tests/qemu-iotests/058 | 2 +- tests/qemu-iotests/059 | 2 +- tests/qemu-iotests/060 | 2 +- tests/qemu-iotests/061 | 2 +- tests/qemu-iotests/062 | 2 +- tests/qemu-iotests/063 | 2 +- tests/qemu-iotests/064 | 2 +- tests/qemu-iotests/066 | 2 +- tests/qemu-iotests/067 | 2 +- tests/qemu-iotests/068 | 2 +- tests/qemu-iotests/069 | 2 +- tests/qemu-iotests/070 | 2 +- tests/qemu-iotests/071 | 2 +- tests/qemu-iotests/072 | 2 +- tests/qemu-iotests/073 | 2 +- tests/qemu-iotests/074 | 2 +- tests/qemu-iotests/075 | 2 +- tests/qemu-iotests/076 | 2 +- tests/qemu-iotests/077 | 2 +- tests/qemu-iotests/078 | 2 +- tests/qemu-iotests/079 | 2 +- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/081 | 2 +- tests/qemu-iotests/082 | 2 +- tests/qemu-iotests/083 | 2 +- tests/qemu-iotests/084 | 2 +- tests/qemu-iotests/085 | 2 +- tests/qemu-iotests/086 | 2 +- tests/qemu-iotests/087 | 2 +- tests/qemu-iotests/088 | 2 +- tests/qemu-iotests/089 | 2 +- tests/qemu-iotests/090 | 2 +- tests/qemu-iotests/091 | 2 +- tests/qemu-iotests/092 | 2 +- tests/qemu-iotests/094 | 2 +- tests/qemu-iotests/095 | 2 +- tests/qemu-iotests/097 | 2 +- tests/qemu-iotests/098 | 2 +- tests/qemu-iotests/099 | 2 +- tests/qemu-iotests/101 | 2 +- tests/qemu-iotests/102 | 2 +- tests/qemu-iotests/103 | 2 +- tests/qemu-iotests/104 | 2 +- tests/qemu-iotests/105 | 2 +- tests/qemu-iotests/106 | 2 +- tests/qemu-iotests/107 | 2 +- tests/qemu-iotests/108 | 2 +- tests/qemu-iotests/109 | 2 +- tests/qemu-iotests/110 | 2 +- tests/qemu-iotests/111 | 2 +- tests/qemu-iotests/112 | 2 +- tests/qemu-iotests/113 | 2 +- tests/qemu-iotests/114 | 2 +- tests/qemu-iotests/115 | 2 +- tests/qemu-iotests/116 | 2 +- tests/qemu-iotests/117 | 2 +- tests/qemu-iotests/119 | 2 +- tests/qemu-iotests/120 | 2 +- tests/qemu-iotests/121 | 2 +- tests/qemu-iotests/122 | 2 +- tests/qemu-iotests/123 | 2 +- tests/qemu-iotests/125 | 2 +- tests/qemu-iotests/126 | 2 +- tests/qemu-iotests/127 | 2 +- tests/qemu-iotests/128 | 2 +- tests/qemu-iotests/130 | 2 +- tests/qemu-iotests/131 | 2 +- tests/qemu-iotests/133 | 2 +- tests/qemu-iotests/134 | 2 +- tests/qemu-iotests/135 | 2 +- tests/qemu-iotests/137 | 2 +- tests/qemu-iotests/138 | 2 +- tests/qemu-iotests/140 | 2 +- tests/qemu-iotests/141 | 2 +- tests/qemu-iotests/142 | 2 +- tests/qemu-iotests/143 | 2 +- tests/qemu-iotests/144 | 2 +- tests/qemu-iotests/145 | 2 +- tests/qemu-iotests/146 | 2 +- tests/qemu-iotests/150 | 2 +- tests/qemu-iotests/153 | 2 +- tests/qemu-iotests/154 | 2 +- tests/qemu-iotests/156 | 2 +- tests/qemu-iotests/157 | 2 +- tests/qemu-iotests/158 | 2 +- tests/qemu-iotests/159 | 2 +- tests/qemu-iotests/160 | 2 +- tests/qemu-iotests/161 | 2 +- tests/qemu-iotests/162 | 2 +- tests/qemu-iotests/170 | 2 +- tests/qemu-iotests/171 | 2 +- tests/qemu-iotests/172 | 2 +- tests/qemu-iotests/173 | 2 +- tests/qemu-iotests/174 | 2 +- tests/qemu-iotests/175 | 2 +- tests/qemu-iotests/176 | 2 +- tests/qemu-iotests/177 | 2 +- tests/qemu-iotests/178 | 2 +- tests/qemu-iotests/179 | 2 +- tests/qemu-iotests/181 | 2 +- tests/qemu-iotests/182 | 2 +- tests/qemu-iotests/183 | 2 +- tests/qemu-iotests/184 | 2 +- tests/qemu-iotests/185 | 2 +- tests/qemu-iotests/186 | 2 +- tests/qemu-iotests/187 | 2 +- tests/qemu-iotests/188 | 2 +- tests/qemu-iotests/189 | 2 +- tests/qemu-iotests/190 | 2 +- tests/qemu-iotests/191 | 2 +- tests/qemu-iotests/192 | 2 +- tests/qemu-iotests/195 | 2 +- tests/qemu-iotests/197 | 2 +- tests/qemu-iotests/198 | 2 +- tests/qemu-iotests/200 | 2 +- tests/qemu-iotests/201 | 2 +- tests/qemu-iotests/204 | 2 +- tests/qemu-iotests/214 | 2 +- tests/qemu-iotests/215 | 2 +- tests/qemu-iotests/217 | 2 +- tests/qemu-iotests/220 | 2 +- tests/qemu-iotests/221 | 2 +- tests/qemu-iotests/223 | 2 +- tests/qemu-iotests/225 | 2 +- tests/qemu-iotests/226 | 2 +- tests/qemu-iotests/227 | 2 +- tests/qemu-iotests/229 | 2 +- tests/qemu-iotests/231 | 2 +- tests/qemu-iotests/232 | 2 +- tests/qemu-iotests/233 | 2 +- tests/qemu-iotests/check | 2 +- tests/qemu-iotests/common.config | 2 +- tests/qemu-iotests/common.filter | 2 +- tests/qemu-iotests/common.nbd | 2 +- tests/qemu-iotests/common.pattern | 2 +- tests/qemu-iotests/common.qemu | 2 +- tests/qemu-iotests/common.rc | 2 +- tests/qemu-iotests/common.tls | 2 +- 185 files changed, 185 insertions(+), 185 deletions(-) diff --git a/tests/qemu-iotests/001 b/tests/qemu-iotests/001 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/001 +++ b/tests/qemu-iotests/001 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test simple read/write using plain bdrv_read/bdrv_write # diff --git a/tests/qemu-iotests/002 b/tests/qemu-iotests/002 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/002 +++ b/tests/qemu-iotests/002 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test simple read/write using plain bdrv_pread/bdrv_pwrite # diff --git a/tests/qemu-iotests/003 b/tests/qemu-iotests/003 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/003 +++ b/tests/qemu-iotests/003 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test simple read/write using bdrv_aio_readv/bdrv_aio_writev # diff --git a/tests/qemu-iotests/004 b/tests/qemu-iotests/004 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/004 +++ b/tests/qemu-iotests/004 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Make sure we can't read and write outside of the image size. # diff --git a/tests/qemu-iotests/005 b/tests/qemu-iotests/005 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/005 +++ b/tests/qemu-iotests/005 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Make sure qemu-img can create 5TB images # diff --git a/tests/qemu-iotests/007 b/tests/qemu-iotests/007 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/007 +++ b/tests/qemu-iotests/007 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Check for one possible case of qcow2 refcount corruption. # diff --git a/tests/qemu-iotests/008 b/tests/qemu-iotests/008 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/008 +++ b/tests/qemu-iotests/008 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test simple asynchronous read/write operations. # diff --git a/tests/qemu-iotests/009 b/tests/qemu-iotests/009 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/009 +++ b/tests/qemu-iotests/009 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Nolan I qcow2 corruption - incorrectly reports free clusters # diff --git a/tests/qemu-iotests/010 b/tests/qemu-iotests/010 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/010 +++ b/tests/qemu-iotests/010 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Nolan II qcow2 corruption - wrong used cluster # diff --git a/tests/qemu-iotests/011 b/tests/qemu-iotests/011 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/011 +++ b/tests/qemu-iotests/011 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test for AIO allocation on the same cluster # diff --git a/tests/qemu-iotests/012 b/tests/qemu-iotests/012 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/012 +++ b/tests/qemu-iotests/012 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Make sure we can open read-only images # diff --git a/tests/qemu-iotests/013 b/tests/qemu-iotests/013 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/013 +++ b/tests/qemu-iotests/013 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # qcow2 pattern test, empty and compressed image - 4k cluster patterns # diff --git a/tests/qemu-iotests/014 b/tests/qemu-iotests/014 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/014 +++ b/tests/qemu-iotests/014 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # qcow2 pattern test, complex patterns including compression and snapshots # Using patterns for 4k cluster size. diff --git a/tests/qemu-iotests/015 b/tests/qemu-iotests/015 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/015 +++ b/tests/qemu-iotests/015 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Combined test to grow the refcount table and test snapshots. # diff --git a/tests/qemu-iotests/017 b/tests/qemu-iotests/017 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/017 +++ b/tests/qemu-iotests/017 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Simple backing file reads # diff --git a/tests/qemu-iotests/018 b/tests/qemu-iotests/018 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/018 +++ b/tests/qemu-iotests/018 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Merge backing file into test image when converting the image # diff --git a/tests/qemu-iotests/019 b/tests/qemu-iotests/019 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/019 +++ b/tests/qemu-iotests/019 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # When using a backing file for the output image in qemu-img convert, # the backing file clusters must not copied. The data must still be diff --git a/tests/qemu-iotests/020 b/tests/qemu-iotests/020 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/020 +++ b/tests/qemu-iotests/020 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Commit changes to backing file # diff --git a/tests/qemu-iotests/021 b/tests/qemu-iotests/021 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/021 +++ b/tests/qemu-iotests/021 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test handling of invalid patterns arguments to qemu-io # diff --git a/tests/qemu-iotests/022 b/tests/qemu-iotests/022 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/022 +++ b/tests/qemu-iotests/022 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test bdrv_load/save_vmstate using the usual patterns # diff --git a/tests/qemu-iotests/023 b/tests/qemu-iotests/023 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/023 +++ b/tests/qemu-iotests/023 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # qcow2 pattern test with various cluster sizes # diff --git a/tests/qemu-iotests/024 b/tests/qemu-iotests/024 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/024 +++ b/tests/qemu-iotests/024 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Rebasing COW images # diff --git a/tests/qemu-iotests/025 b/tests/qemu-iotests/025 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/025 +++ b/tests/qemu-iotests/025 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Resizing images # diff --git a/tests/qemu-iotests/026 b/tests/qemu-iotests/026 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/026 +++ b/tests/qemu-iotests/026 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # qcow2 error path testing # diff --git a/tests/qemu-iotests/027 b/tests/qemu-iotests/027 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/027 +++ b/tests/qemu-iotests/027 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test that sub-cluster allocating writes zero the rest of the cluster # diff --git a/tests/qemu-iotests/028 b/tests/qemu-iotests/028 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/028 +++ b/tests/qemu-iotests/028 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test that backing files can be smaller than the image # diff --git a/tests/qemu-iotests/029 b/tests/qemu-iotests/029 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/029 +++ b/tests/qemu-iotests/029 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # qcow2 internal snapshots/VM state tests # diff --git a/tests/qemu-iotests/031 b/tests/qemu-iotests/031 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/031 +++ b/tests/qemu-iotests/031 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test that all qcow2 header extensions survive a header rewrite # diff --git a/tests/qemu-iotests/032 b/tests/qemu-iotests/032 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/032 +++ b/tests/qemu-iotests/032 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test that AIO requests are drained before an image is closed. This used # to segfault because the request coroutine kept running even after the diff --git a/tests/qemu-iotests/033 b/tests/qemu-iotests/033 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/033 +++ b/tests/qemu-iotests/033 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test aligned and misaligned write zeroes operations. # diff --git a/tests/qemu-iotests/034 b/tests/qemu-iotests/034 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/034 +++ b/tests/qemu-iotests/034 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test bdrv_pwrite_zeroes with backing files (see also 154) # diff --git a/tests/qemu-iotests/035 b/tests/qemu-iotests/035 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/035 +++ b/tests/qemu-iotests/035 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Let a few AIO requests run in parallel and have them access different L2 # tables so that the cache has a chance to get used up. diff --git a/tests/qemu-iotests/036 b/tests/qemu-iotests/036 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/036 +++ b/tests/qemu-iotests/036 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qcow2 feature bits # diff --git a/tests/qemu-iotests/037 b/tests/qemu-iotests/037 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/037 +++ b/tests/qemu-iotests/037 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test COW from backing files # diff --git a/tests/qemu-iotests/038 b/tests/qemu-iotests/038 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/038 +++ b/tests/qemu-iotests/038 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test COW from backing files with AIO # diff --git a/tests/qemu-iotests/039 b/tests/qemu-iotests/039 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/039 +++ b/tests/qemu-iotests/039 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qcow2 lazy refcounts # diff --git a/tests/qemu-iotests/042 b/tests/qemu-iotests/042 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/042 +++ b/tests/qemu-iotests/042 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qemu-img operation on zero size images # diff --git a/tests/qemu-iotests/043 b/tests/qemu-iotests/043 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/043 +++ b/tests/qemu-iotests/043 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test that qemu-img info --backing-chain detects infinite loops # diff --git a/tests/qemu-iotests/046 b/tests/qemu-iotests/046 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/046 +++ b/tests/qemu-iotests/046 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test concurrent cluster allocations # diff --git a/tests/qemu-iotests/047 b/tests/qemu-iotests/047 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/047 +++ b/tests/qemu-iotests/047 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Regression test for commit b7ab0fea (which was a corruption fix, # despite the commit message claiming otherwise) diff --git a/tests/qemu-iotests/048 b/tests/qemu-iotests/048 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/048 +++ b/tests/qemu-iotests/048 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash ## ## qemu-img compare test ## diff --git a/tests/qemu-iotests/049 b/tests/qemu-iotests/049 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/049 +++ b/tests/qemu-iotests/049 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Check qemu-img option parsing # diff --git a/tests/qemu-iotests/050 b/tests/qemu-iotests/050 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/050 +++ b/tests/qemu-iotests/050 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qemu-img rebase with zero clusters # diff --git a/tests/qemu-iotests/051 b/tests/qemu-iotests/051 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/051 +++ b/tests/qemu-iotests/051 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test command line configuration of block devices and driver-specific options # diff --git a/tests/qemu-iotests/052 b/tests/qemu-iotests/052 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/052 +++ b/tests/qemu-iotests/052 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test bdrv_read/bdrv_write using BDRV_O_SNAPSHOT # diff --git a/tests/qemu-iotests/053 b/tests/qemu-iotests/053 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/053 +++ b/tests/qemu-iotests/053 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qemu-img convert when image length is not a multiple of cluster size # diff --git a/tests/qemu-iotests/054 b/tests/qemu-iotests/054 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/054 +++ b/tests/qemu-iotests/054 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test huge qcow2 images # diff --git a/tests/qemu-iotests/058 b/tests/qemu-iotests/058 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/058 +++ b/tests/qemu-iotests/058 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test export internal snapshot by qemu-nbd, convert it by qemu-img. # diff --git a/tests/qemu-iotests/059 b/tests/qemu-iotests/059 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/059 +++ b/tests/qemu-iotests/059 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for vmdk # diff --git a/tests/qemu-iotests/060 b/tests/qemu-iotests/060 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/060 +++ b/tests/qemu-iotests/060 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for image corruption (overlapping data structures) in qcow2 # diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/061 +++ b/tests/qemu-iotests/061 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for image option amendment in qcow2. # diff --git a/tests/qemu-iotests/062 b/tests/qemu-iotests/062 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/062 +++ b/tests/qemu-iotests/062 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for snapshotting images with unallocated zero clusters in # qcow2 diff --git a/tests/qemu-iotests/063 b/tests/qemu-iotests/063 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/063 +++ b/tests/qemu-iotests/063 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # test of qemu-img convert -n - convert without creation # diff --git a/tests/qemu-iotests/064 b/tests/qemu-iotests/064 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/064 +++ b/tests/qemu-iotests/064 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test VHDX read/write from a sample image created with Hyper-V # diff --git a/tests/qemu-iotests/066 b/tests/qemu-iotests/066 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/066 +++ b/tests/qemu-iotests/066 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for preallocated zero clusters in qcow2 # diff --git a/tests/qemu-iotests/067 b/tests/qemu-iotests/067 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/067 +++ b/tests/qemu-iotests/067 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test automatic deletion of BDSes created by -drive/drive_add # diff --git a/tests/qemu-iotests/068 b/tests/qemu-iotests/068 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/068 +++ b/tests/qemu-iotests/068 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for loading a saved VM state from a qcow2 image # diff --git a/tests/qemu-iotests/069 b/tests/qemu-iotests/069 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/069 +++ b/tests/qemu-iotests/069 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for deleting a backing file # diff --git a/tests/qemu-iotests/070 b/tests/qemu-iotests/070 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/070 +++ b/tests/qemu-iotests/070 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test VHDX log replay from an image with a journal that needs to be # replayed diff --git a/tests/qemu-iotests/071 b/tests/qemu-iotests/071 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/071 +++ b/tests/qemu-iotests/071 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for the QMP blkdebug and blkverify interfaces # diff --git a/tests/qemu-iotests/072 b/tests/qemu-iotests/072 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/072 +++ b/tests/qemu-iotests/072 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for nested image formats # diff --git a/tests/qemu-iotests/073 b/tests/qemu-iotests/073 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/073 +++ b/tests/qemu-iotests/073 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test count_contiguous_clusters in qcow2 # diff --git a/tests/qemu-iotests/074 b/tests/qemu-iotests/074 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/074 +++ b/tests/qemu-iotests/074 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash ## ## qemu-img compare test (qcow2 only ones) ## diff --git a/tests/qemu-iotests/075 b/tests/qemu-iotests/075 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/075 +++ b/tests/qemu-iotests/075 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # cloop format input validation tests # diff --git a/tests/qemu-iotests/076 b/tests/qemu-iotests/076 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/076 +++ b/tests/qemu-iotests/076 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # parallels format input validation tests # diff --git a/tests/qemu-iotests/077 b/tests/qemu-iotests/077 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/077 +++ b/tests/qemu-iotests/077 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test concurrent pread/pwrite # diff --git a/tests/qemu-iotests/078 b/tests/qemu-iotests/078 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/078 +++ b/tests/qemu-iotests/078 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # bochs format input validation tests # diff --git a/tests/qemu-iotests/079 b/tests/qemu-iotests/079 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/079 +++ b/tests/qemu-iotests/079 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qcow2 preallocation with different cluster_sizes # diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/080 +++ b/tests/qemu-iotests/080 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # qcow2 format input validation tests # diff --git a/tests/qemu-iotests/081 b/tests/qemu-iotests/081 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/081 +++ b/tests/qemu-iotests/081 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test Quorum block driver # diff --git a/tests/qemu-iotests/082 b/tests/qemu-iotests/082 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/082 +++ b/tests/qemu-iotests/082 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qemu-img command line parsing # diff --git a/tests/qemu-iotests/083 b/tests/qemu-iotests/083 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/083 +++ b/tests/qemu-iotests/083 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test NBD client unexpected disconnect # diff --git a/tests/qemu-iotests/084 b/tests/qemu-iotests/084 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/084 +++ b/tests/qemu-iotests/084 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for VDI header corruption; image too large, and too many blocks. # Also simple test for creating dynamic and static VDI images. diff --git a/tests/qemu-iotests/085 b/tests/qemu-iotests/085 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/085 +++ b/tests/qemu-iotests/085 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Live snapshot tests # diff --git a/tests/qemu-iotests/086 b/tests/qemu-iotests/086 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/086 +++ b/tests/qemu-iotests/086 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qemu-img progress output # diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/087 +++ b/tests/qemu-iotests/087 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test unsupported blockdev-add cases # diff --git a/tests/qemu-iotests/088 b/tests/qemu-iotests/088 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/088 +++ b/tests/qemu-iotests/088 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # vpc (VHD) format input validation tests # diff --git a/tests/qemu-iotests/089 b/tests/qemu-iotests/089 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/089 +++ b/tests/qemu-iotests/089 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for support of JSON filenames # diff --git a/tests/qemu-iotests/090 b/tests/qemu-iotests/090 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/090 +++ b/tests/qemu-iotests/090 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test for discarding compressed clusters on qcow2 images # diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/091 +++ b/tests/qemu-iotests/091 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Live migration test # diff --git a/tests/qemu-iotests/092 b/tests/qemu-iotests/092 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/092 +++ b/tests/qemu-iotests/092 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # qcow1 format input validation tests # diff --git a/tests/qemu-iotests/094 b/tests/qemu-iotests/094 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/094 +++ b/tests/qemu-iotests/094 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for drive-mirror to NBD # diff --git a/tests/qemu-iotests/095 b/tests/qemu-iotests/095 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/095 +++ b/tests/qemu-iotests/095 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test for commit of larger active layer # diff --git a/tests/qemu-iotests/097 b/tests/qemu-iotests/097 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/097 +++ b/tests/qemu-iotests/097 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Commit changes into backing chains and empty the top image if the # backing image is not explicitly specified diff --git a/tests/qemu-iotests/098 b/tests/qemu-iotests/098 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/098 +++ b/tests/qemu-iotests/098 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qcow2's bdrv_make_empty for images without internal snapshots # diff --git a/tests/qemu-iotests/099 b/tests/qemu-iotests/099 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/099 +++ b/tests/qemu-iotests/099 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test valid filenames for blkdebug and blkverify representatively for # other protocols (such as NBD) when queried diff --git a/tests/qemu-iotests/101 b/tests/qemu-iotests/101 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/101 +++ b/tests/qemu-iotests/101 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test short file I/O # diff --git a/tests/qemu-iotests/102 b/tests/qemu-iotests/102 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/102 +++ b/tests/qemu-iotests/102 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for qemu-io -c map and qemu-img map # diff --git a/tests/qemu-iotests/103 b/tests/qemu-iotests/103 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/103 +++ b/tests/qemu-iotests/103 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for qcow2 metadata cache size specification # diff --git a/tests/qemu-iotests/104 b/tests/qemu-iotests/104 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/104 +++ b/tests/qemu-iotests/104 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test image creation with aligned and unaligned sizes # diff --git a/tests/qemu-iotests/105 b/tests/qemu-iotests/105 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/105 +++ b/tests/qemu-iotests/105 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Create, read, write big image # diff --git a/tests/qemu-iotests/106 b/tests/qemu-iotests/106 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/106 +++ b/tests/qemu-iotests/106 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test preallocated resize of raw images # diff --git a/tests/qemu-iotests/107 b/tests/qemu-iotests/107 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/107 +++ b/tests/qemu-iotests/107 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Tests updates of the qcow2 L1 table # diff --git a/tests/qemu-iotests/108 b/tests/qemu-iotests/108 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/108 +++ b/tests/qemu-iotests/108 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for repairing qcow2 images which cannot be repaired using # the on-disk refcount structures diff --git a/tests/qemu-iotests/109 b/tests/qemu-iotests/109 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/109 +++ b/tests/qemu-iotests/109 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test writing image headers of other formats into raw images # diff --git a/tests/qemu-iotests/110 b/tests/qemu-iotests/110 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/110 +++ b/tests/qemu-iotests/110 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for relative backing file names in complex BDS trees # diff --git a/tests/qemu-iotests/111 b/tests/qemu-iotests/111 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/111 +++ b/tests/qemu-iotests/111 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for non-existing backing file when creating a qcow2 image # and not specifying the size diff --git a/tests/qemu-iotests/112 b/tests/qemu-iotests/112 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/112 +++ b/tests/qemu-iotests/112 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test cases for different refcount_bits values # diff --git a/tests/qemu-iotests/113 b/tests/qemu-iotests/113 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/113 +++ b/tests/qemu-iotests/113 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for accessing creation options on image formats and # protocols not supporting image creation diff --git a/tests/qemu-iotests/114 b/tests/qemu-iotests/114 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/114 +++ b/tests/qemu-iotests/114 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test invalid backing file format in qcow2 images # diff --git a/tests/qemu-iotests/115 b/tests/qemu-iotests/115 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/115 +++ b/tests/qemu-iotests/115 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for non-self-referential qcow2 refcount blocks # diff --git a/tests/qemu-iotests/116 b/tests/qemu-iotests/116 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/116 +++ b/tests/qemu-iotests/116 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test error code paths for invalid QED images # diff --git a/tests/qemu-iotests/117 b/tests/qemu-iotests/117 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/117 +++ b/tests/qemu-iotests/117 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for shared BDS between backend trees # diff --git a/tests/qemu-iotests/119 b/tests/qemu-iotests/119 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/119 +++ b/tests/qemu-iotests/119 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # NBD test case for overriding BDRV_O_PROTOCOL by explicitly specifying # a driver diff --git a/tests/qemu-iotests/120 b/tests/qemu-iotests/120 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/120 +++ b/tests/qemu-iotests/120 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Non-NBD test cases for overriding BDRV_O_PROTOCOL by explicitly # specifying a driver diff --git a/tests/qemu-iotests/121 b/tests/qemu-iotests/121 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/121 +++ b/tests/qemu-iotests/121 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test cases for qcow2 refcount table growth # diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/122 +++ b/tests/qemu-iotests/122 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test some qemu-img convert cases # diff --git a/tests/qemu-iotests/123 b/tests/qemu-iotests/123 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/123 +++ b/tests/qemu-iotests/123 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for qemu-img convert to NBD # diff --git a/tests/qemu-iotests/125 b/tests/qemu-iotests/125 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/125 +++ b/tests/qemu-iotests/125 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test preallocated growth of qcow2 images # diff --git a/tests/qemu-iotests/126 b/tests/qemu-iotests/126 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/126 +++ b/tests/qemu-iotests/126 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Tests handling of colons in filenames (which may be confused with protocol # prefixes) diff --git a/tests/qemu-iotests/127 b/tests/qemu-iotests/127 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/127 +++ b/tests/qemu-iotests/127 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for mirroring with dataplane # diff --git a/tests/qemu-iotests/128 b/tests/qemu-iotests/128 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/128 +++ b/tests/qemu-iotests/128 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test that opening O_DIRECT succeeds when image file I/O produces EIO # diff --git a/tests/qemu-iotests/130 b/tests/qemu-iotests/130 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/130 +++ b/tests/qemu-iotests/130 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test that temporary backing file overrides (on the command line or in # blockdev-add) don't replace the original path stored in the image during diff --git a/tests/qemu-iotests/131 b/tests/qemu-iotests/131 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/131 +++ b/tests/qemu-iotests/131 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # parallels format validation tests (created by QEMU) # diff --git a/tests/qemu-iotests/133 b/tests/qemu-iotests/133 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/133 +++ b/tests/qemu-iotests/133 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test for reopen # diff --git a/tests/qemu-iotests/134 b/tests/qemu-iotests/134 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/134 +++ b/tests/qemu-iotests/134 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test encrypted read/write using plain bdrv_read/bdrv_write # diff --git a/tests/qemu-iotests/135 b/tests/qemu-iotests/135 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/135 +++ b/tests/qemu-iotests/135 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test VPC open of image with large Max Table Entries value. # diff --git a/tests/qemu-iotests/137 b/tests/qemu-iotests/137 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/137 +++ b/tests/qemu-iotests/137 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qcow2 reopen # diff --git a/tests/qemu-iotests/138 b/tests/qemu-iotests/138 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/138 +++ b/tests/qemu-iotests/138 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # General test case for qcow2's image check # diff --git a/tests/qemu-iotests/140 b/tests/qemu-iotests/140 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/140 +++ b/tests/qemu-iotests/140 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for ejecting a BlockBackend with an NBD server attached to it # diff --git a/tests/qemu-iotests/141 b/tests/qemu-iotests/141 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/141 +++ b/tests/qemu-iotests/141 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for ejecting BDSs with block jobs still running on them # diff --git a/tests/qemu-iotests/142 b/tests/qemu-iotests/142 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/142 +++ b/tests/qemu-iotests/142 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test for configuring cache modes of arbitrary nodes (requires O_DIRECT) # diff --git a/tests/qemu-iotests/143 b/tests/qemu-iotests/143 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/143 +++ b/tests/qemu-iotests/143 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for connecting to a non-existing NBD export name # diff --git a/tests/qemu-iotests/144 b/tests/qemu-iotests/144 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/144 +++ b/tests/qemu-iotests/144 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # Check live snapshot, followed by active commit, and another snapshot. # # This test is to catch the error case of BZ #1300209: diff --git a/tests/qemu-iotests/145 b/tests/qemu-iotests/145 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/145 +++ b/tests/qemu-iotests/145 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test the combination of -incoming and snapshot=on # diff --git a/tests/qemu-iotests/146 b/tests/qemu-iotests/146 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/146 +++ b/tests/qemu-iotests/146 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test VHD image format creator detection and override # diff --git a/tests/qemu-iotests/150 b/tests/qemu-iotests/150 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/150 +++ b/tests/qemu-iotests/150 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test that qemu-img convert -S 0 fully allocates the target image # diff --git a/tests/qemu-iotests/153 b/tests/qemu-iotests/153 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/153 +++ b/tests/qemu-iotests/153 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test image locking # diff --git a/tests/qemu-iotests/154 b/tests/qemu-iotests/154 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/154 +++ b/tests/qemu-iotests/154 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # qcow2 specific bdrv_pwrite_zeroes tests with backing files (complements 034) # diff --git a/tests/qemu-iotests/156 b/tests/qemu-iotests/156 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/156 +++ b/tests/qemu-iotests/156 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Tests oVirt-like storage migration: # - Create snapshot diff --git a/tests/qemu-iotests/157 b/tests/qemu-iotests/157 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/157 +++ b/tests/qemu-iotests/157 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test command line configuration of block devices with qdev # diff --git a/tests/qemu-iotests/158 b/tests/qemu-iotests/158 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/158 +++ b/tests/qemu-iotests/158 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test encrypted read/write using backing files # diff --git a/tests/qemu-iotests/159 b/tests/qemu-iotests/159 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/159 +++ b/tests/qemu-iotests/159 @@ -XXX,XX +XXX,XX @@ -#! /bin/bash +#!/usr/bin/env bash # # qemu-img dd test with different block sizes # diff --git a/tests/qemu-iotests/160 b/tests/qemu-iotests/160 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/160 +++ b/tests/qemu-iotests/160 @@ -XXX,XX +XXX,XX @@ -#! /bin/bash +#!/usr/bin/env bash # # qemu-img dd test for the skip option # diff --git a/tests/qemu-iotests/161 b/tests/qemu-iotests/161 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/161 +++ b/tests/qemu-iotests/161 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test reopening a backing image after block-stream and block-commit # diff --git a/tests/qemu-iotests/162 b/tests/qemu-iotests/162 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/162 +++ b/tests/qemu-iotests/162 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for specifying runtime options of the wrong type to some # block drivers diff --git a/tests/qemu-iotests/170 b/tests/qemu-iotests/170 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/170 +++ b/tests/qemu-iotests/170 @@ -XXX,XX +XXX,XX @@ -#! /bin/bash +#!/usr/bin/env bash # # qemu-img dd test # diff --git a/tests/qemu-iotests/171 b/tests/qemu-iotests/171 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/171 +++ b/tests/qemu-iotests/171 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test 'offset' and 'size' options of the raw driver. Make sure we can't # (or can) read and write outside of the image size. diff --git a/tests/qemu-iotests/172 b/tests/qemu-iotests/172 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/172 +++ b/tests/qemu-iotests/172 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test floppy configuration # diff --git a/tests/qemu-iotests/173 b/tests/qemu-iotests/173 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/173 +++ b/tests/qemu-iotests/173 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test QAPI commands looking up protocol based images with relative # filename backing strings diff --git a/tests/qemu-iotests/174 b/tests/qemu-iotests/174 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/174 +++ b/tests/qemu-iotests/174 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test that qemu-io fail with non-zero exit code # diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/175 +++ b/tests/qemu-iotests/175 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test creating raw image preallocation mode # diff --git a/tests/qemu-iotests/176 b/tests/qemu-iotests/176 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/176 +++ b/tests/qemu-iotests/176 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Commit changes into backing chains and empty the top image if the # backing image is not explicitly specified. diff --git a/tests/qemu-iotests/177 b/tests/qemu-iotests/177 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/177 +++ b/tests/qemu-iotests/177 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test corner cases with unusual block geometries # diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/178 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/178 +++ b/tests/qemu-iotests/178 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # qemu-img measure sub-command tests # diff --git a/tests/qemu-iotests/179 b/tests/qemu-iotests/179 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/179 +++ b/tests/qemu-iotests/179 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for write zeroes with unmap # diff --git a/tests/qemu-iotests/181 b/tests/qemu-iotests/181 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/181 +++ b/tests/qemu-iotests/181 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test postcopy live migration with shared storage # diff --git a/tests/qemu-iotests/182 b/tests/qemu-iotests/182 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/182 +++ b/tests/qemu-iotests/182 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test image locking for POSIX locks # diff --git a/tests/qemu-iotests/183 b/tests/qemu-iotests/183 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/183 +++ b/tests/qemu-iotests/183 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test old-style block migration (migrate -b) # diff --git a/tests/qemu-iotests/184 b/tests/qemu-iotests/184 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/184 +++ b/tests/qemu-iotests/184 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test I/O throttle block filter driver interface # diff --git a/tests/qemu-iotests/185 b/tests/qemu-iotests/185 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/185 +++ b/tests/qemu-iotests/185 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test exiting qemu while jobs are still running # diff --git a/tests/qemu-iotests/186 b/tests/qemu-iotests/186 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/186 +++ b/tests/qemu-iotests/186 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test 'info block' with all kinds of configurations # diff --git a/tests/qemu-iotests/187 b/tests/qemu-iotests/187 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/187 +++ b/tests/qemu-iotests/187 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test switching between read-only and read-write # diff --git a/tests/qemu-iotests/188 b/tests/qemu-iotests/188 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/188 +++ b/tests/qemu-iotests/188 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test encrypted read/write using plain bdrv_read/bdrv_write # diff --git a/tests/qemu-iotests/189 b/tests/qemu-iotests/189 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/189 +++ b/tests/qemu-iotests/189 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test encrypted read/write using backing files # diff --git a/tests/qemu-iotests/190 b/tests/qemu-iotests/190 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/190 +++ b/tests/qemu-iotests/190 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # qemu-img measure sub-command tests on huge qcow2 files # diff --git a/tests/qemu-iotests/191 b/tests/qemu-iotests/191 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/191 +++ b/tests/qemu-iotests/191 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test commit block job where top has two parents # diff --git a/tests/qemu-iotests/192 b/tests/qemu-iotests/192 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/192 +++ b/tests/qemu-iotests/192 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test NBD export with -incoming (non-shared storage migration use case from # libvirt) diff --git a/tests/qemu-iotests/195 b/tests/qemu-iotests/195 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/195 +++ b/tests/qemu-iotests/195 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test change-backing-file command # diff --git a/tests/qemu-iotests/197 b/tests/qemu-iotests/197 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/197 +++ b/tests/qemu-iotests/197 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for copy-on-read into qcow2 # diff --git a/tests/qemu-iotests/198 b/tests/qemu-iotests/198 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/198 +++ b/tests/qemu-iotests/198 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test commit of encrypted qcow2 files # diff --git a/tests/qemu-iotests/200 b/tests/qemu-iotests/200 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/200 +++ b/tests/qemu-iotests/200 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Block job co-routine race condition test. # diff --git a/tests/qemu-iotests/201 b/tests/qemu-iotests/201 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/201 +++ b/tests/qemu-iotests/201 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test savevm and loadvm after live migration with postcopy flag # diff --git a/tests/qemu-iotests/204 b/tests/qemu-iotests/204 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/204 +++ b/tests/qemu-iotests/204 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test corner cases with unusual block geometries # diff --git a/tests/qemu-iotests/214 b/tests/qemu-iotests/214 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/214 +++ b/tests/qemu-iotests/214 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qcow2 image compression # diff --git a/tests/qemu-iotests/215 b/tests/qemu-iotests/215 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/215 +++ b/tests/qemu-iotests/215 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test case for copy-on-read into qcow2, using the COR filter driver # diff --git a/tests/qemu-iotests/217 b/tests/qemu-iotests/217 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/217 +++ b/tests/qemu-iotests/217 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # I/O errors when working with internal qcow2 snapshots, and repairing # the result diff --git a/tests/qemu-iotests/220 b/tests/qemu-iotests/220 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/220 +++ b/tests/qemu-iotests/220 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # max limits on compression in huge qcow2 files # diff --git a/tests/qemu-iotests/221 b/tests/qemu-iotests/221 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/221 +++ b/tests/qemu-iotests/221 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test qemu-img vs. unaligned images # diff --git a/tests/qemu-iotests/223 b/tests/qemu-iotests/223 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/223 +++ b/tests/qemu-iotests/223 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test reading dirty bitmap over NBD # diff --git a/tests/qemu-iotests/225 b/tests/qemu-iotests/225 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/225 +++ b/tests/qemu-iotests/225 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test vmdk backing file correlation # diff --git a/tests/qemu-iotests/226 b/tests/qemu-iotests/226 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/226 +++ b/tests/qemu-iotests/226 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # This test covers expected filetypes for the file, host_cdrom and # host_device drivers. diff --git a/tests/qemu-iotests/227 b/tests/qemu-iotests/227 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/227 +++ b/tests/qemu-iotests/227 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test query-blockstats with different ways to create a BB # diff --git a/tests/qemu-iotests/229 b/tests/qemu-iotests/229 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/229 +++ b/tests/qemu-iotests/229 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test for force canceling a running blockjob that is paused in # an error state. diff --git a/tests/qemu-iotests/231 b/tests/qemu-iotests/231 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/231 +++ b/tests/qemu-iotests/231 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test legacy and modern option parsing for rbd/ceph. This will not # actually connect to a ceph server, but rather looks for the appropriate diff --git a/tests/qemu-iotests/232 b/tests/qemu-iotests/232 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/232 +++ b/tests/qemu-iotests/232 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test for auto-read-only # diff --git a/tests/qemu-iotests/233 b/tests/qemu-iotests/233 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/233 +++ b/tests/qemu-iotests/233 @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Test NBD TLS certificate / authorization integration # diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/check +++ b/tests/qemu-iotests/check @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Copyright (C) 2009 Red Hat, Inc. # Copyright (c) 2000-2002,2006 Silicon Graphics, Inc. All Rights Reserved. diff --git a/tests/qemu-iotests/common.config b/tests/qemu-iotests/common.config index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.config +++ b/tests/qemu-iotests/common.config @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Copyright (C) 2009 Red Hat, Inc. # Copyright (c) 2000-2003,2006 Silicon Graphics, Inc. All Rights Reserved. diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.filter +++ b/tests/qemu-iotests/common.filter @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Copyright (C) 2009 Red Hat, Inc. # Copyright (c) 2000-2001 Silicon Graphics, Inc. All Rights Reserved. diff --git a/tests/qemu-iotests/common.nbd b/tests/qemu-iotests/common.nbd index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.nbd +++ b/tests/qemu-iotests/common.nbd @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # -*- shell-script-mode -*- # # Helpers for NBD server related config diff --git a/tests/qemu-iotests/common.pattern b/tests/qemu-iotests/common.pattern index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.pattern +++ b/tests/qemu-iotests/common.pattern @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Copyright (C) 2009 Red Hat, Inc. # diff --git a/tests/qemu-iotests/common.qemu b/tests/qemu-iotests/common.qemu index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.qemu +++ b/tests/qemu-iotests/common.qemu @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # This allows for launching of multiple QEMU instances, with independent # communication possible to each instance. diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.rc +++ b/tests/qemu-iotests/common.rc @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Copyright (C) 2009 Red Hat, Inc. # Copyright (c) 2000-2006 Silicon Graphics, Inc. All Rights Reserved. diff --git a/tests/qemu-iotests/common.tls b/tests/qemu-iotests/common.tls index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.tls +++ b/tests/qemu-iotests/common.tls @@ -XXX,XX +XXX,XX @@ -#!/bin/bash +#!/usr/bin/env bash # # Helpers for TLS related config # -- 2.20.1 From: Philippe Mathieu-Daudé <philmd@redhat.com> Various sed regexp from common.filter use sed GNU extensions. Instead of spending time to write these regex to be POSIX compliant, verify the GNU sed is available and use it. Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/qemu-iotests/common.filter | 36 ++++++++++++++++---------------- tests/qemu-iotests/common.rc | 13 ++++++++++++ 2 files changed, 31 insertions(+), 18 deletions(-) diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.filter +++ b/tests/qemu-iotests/common.filter @@ -XXX,XX +XXX,XX @@ # _filter_date() { - sed \ + $SED \ -e 's/[A-Z][a-z][a-z] [A-z][a-z][a-z] *[0-9][0-9]* [0-9][0-9]:[0-9][0-9]:[0-9][0-9] [0-9][0-9][0-9][0-9]$/DATE/' } _filter_generated_node_ids() { - sed -re 's/\#block[0-9]{3,}/NODE_NAME/' + $SED -re 's/\#block[0-9]{3,}/NODE_NAME/' } _filter_qom_path() { - sed -e 's#\(Attached to: *\) /.*#\1 PATH#' + $SED -e 's#\(Attached to: *\) /.*#\1 PATH#' } # replace occurrences of the actual TEST_DIR value with TEST_DIR _filter_testdir() { - sed -e "s#$TEST_DIR/#TEST_DIR/#g" + $SED -e "s#$TEST_DIR/#TEST_DIR/#g" } # replace occurrences of the actual IMGFMT value with IMGFMT _filter_imgfmt() { - sed -e "s#$IMGFMT#IMGFMT#g" + $SED -e "s#$IMGFMT#IMGFMT#g" } # Replace error message when the format is not supported and delete # the output lines after the first one _filter_qemu_img_check() { - sed -e '/allocated.*fragmented.*compressed clusters/d' \ + $SED -e '/allocated.*fragmented.*compressed clusters/d' \ -e 's/qemu-img: This image format does not support checks/No errors were found on the image./' \ -e '/Image end offset: [0-9]\+/d' } @@ -XXX,XX +XXX,XX @@ _filter_qemu_img_check() # Removes \r from messages _filter_win32() { - sed -e 's/\r//g' + $SED -e 's/\r//g' } # sanitize qemu-io output _filter_qemu_io() { - _filter_win32 | sed -e "s/[0-9]* ops\; [0-9/:. sec]* ([0-9/.inf]* [EPTGMKiBbytes]*\/sec and [0-9/.inf]* ops\/sec)/X ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/" \ + _filter_win32 | $SED -e "s/[0-9]* ops\; [0-9/:. sec]* ([0-9/.inf]* [EPTGMKiBbytes]*\/sec and [0-9/.inf]* ops\/sec)/X ops\; XX:XX:XX.X (XXX YYY\/sec and XXX ops\/sec)/" \ -e "s/: line [0-9][0-9]*: *[0-9][0-9]*\( Aborted\| Killed\)/:\1/" \ -e "s/qemu-io> //g" } @@ -XXX,XX +XXX,XX @@ _filter_qemu_io() # replace occurrences of QEMU_PROG with "qemu" _filter_qemu() { - sed -e "s#\\(^\\|(qemu) \\)$(basename $QEMU_PROG):#\1QEMU_PROG:#" \ + $SED -e "s#\\(^\\|(qemu) \\)$(basename $QEMU_PROG):#\1QEMU_PROG:#" \ -e 's#^QEMU [0-9]\+\.[0-9]\+\.[0-9]\+ monitor#QEMU X.Y.Z monitor#' \ -e $'s#\r##' # QEMU monitor uses \r\n line endings } @@ -XXX,XX +XXX,XX @@ _filter_qemu() _filter_qmp() { _filter_win32 | \ - sed -e 's#\("\(micro\)\?seconds": \)[0-9]\+#\1 TIMESTAMP#g' \ + $SED -e 's#\("\(micro\)\?seconds": \)[0-9]\+#\1 TIMESTAMP#g' \ -e 's#^{"QMP":.*}$#QMP_VERSION#' \ -e '/^ "QMP": {\s*$/, /^ }\s*$/ c\' \ -e ' QMP_VERSION' @@ -XXX,XX +XXX,XX @@ _filter_qmp() # readline makes HMP command strings so long that git complains _filter_hmp() { - sed -e $'s/^\\((qemu) \\)\\?.*\e\\[D/\\1/g' \ + $SED -e $'s/^\\((qemu) \\)\\?.*\e\\[D/\\1/g' \ -e $'s/\e\\[K//g' } # replace block job offset _filter_block_job_offset() { - sed -e 's/, "offset": [0-9]\+,/, "offset": OFFSET,/' + $SED -e 's/, "offset": [0-9]\+,/, "offset": OFFSET,/' } # replace block job len _filter_block_job_len() { - sed -e 's/, "len": [0-9]\+,/, "len": LEN,/g' + $SED -e 's/, "len": [0-9]\+,/, "len": LEN,/g' } # replace actual image size (depends on the host filesystem) _filter_actual_image_size() { - sed -s 's/\("actual-size":\s*\)[0-9]\+/\1SIZE/g' + $SED -s 's/\("actual-size":\s*\)[0-9]\+/\1SIZE/g' } # replace driver-specific options in the "Formatting..." line _filter_img_create() { - sed -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \ + $SED -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \ -e "s#$IMGPROTO:$TEST_DIR#TEST_DIR#g" \ -e "s#$TEST_DIR#TEST_DIR#g" \ -e "s#$IMGFMT#IMGFMT#g" \ @@ -XXX,XX +XXX,XX @@ _filter_img_info() discard=0 regex_json_spec_start='^ *"format-specific": \{' - sed -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \ + $SED -e "s#$REMOTE_TEST_DIR#TEST_DIR#g" \ -e "s#$IMGPROTO:$TEST_DIR#TEST_DIR#g" \ -e "s#$TEST_DIR#TEST_DIR#g" \ -e "s#$IMGFMT#IMGFMT#g" \ @@ -XXX,XX +XXX,XX @@ _filter_img_info() # human and json output _filter_qemu_img_map() { - sed -e 's/\([0-9a-fx]* *[0-9a-fx]* *\)[0-9a-fx]* */\1/g' \ + $SED -e 's/\([0-9a-fx]* *[0-9a-fx]* *\)[0-9a-fx]* */\1/g' \ -e 's/"offset": [0-9]\+/"offset": OFFSET/g' \ -e 's/Mapped to *//' | _filter_testdir | _filter_imgfmt } @@ -XXX,XX +XXX,XX @@ _filter_nbd() # receive callbacks sometimes, making them unreliable. # # Filter out the TCP port number since this changes between runs. - sed -e '/nbd\/.*\.c:/d' \ + $SED -e '/nbd\/.*\.c:/d' \ -e 's#127\.0\.0\.1:[0-9]*#127.0.0.1:PORT#g' \ -e "s#?socket=$TEST_DIR#?socket=TEST_DIR#g" \ -e 's#\(foo\|PORT/\?\|.sock\): Failed to .*$#\1#' diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/common.rc +++ b/tests/qemu-iotests/common.rc @@ -XXX,XX +XXX,XX @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. # +SED= +for sed in sed gsed; do + ($sed --version | grep 'GNU sed') > /dev/null 2>&1 + if [ "$?" -eq 0 ]; then + SED=$sed + break + fi +done +if [ -z "$SED" ]; then + echo "$0: GNU sed not found" + exit 1 +fi + dd() { if [ "$HOSTOS" == "Linux" ] -- 2.20.1 Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> --- tests/qemu-iotests/243 | 63 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/243.out | 26 ++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 90 insertions(+) create mode 100755 tests/qemu-iotests/243 create mode 100644 tests/qemu-iotests/243.out diff --git a/tests/qemu-iotests/243 b/tests/qemu-iotests/243 new file mode 100755 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/243 @@ -XXX,XX +XXX,XX @@ +#!/bin/bash +# +# Test qcow2 preallocation +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator +owner=kwolf@redhat.com + +seq=$(basename $0) +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +for mode in off metadata falloc full; do + + echo + echo "=== preallocation=$mode ===" + echo + + IMGOPTS="preallocation=$mode" _make_test_img 64M + + printf "File size: " + du -b $TEST_IMG | cut -f1 + + # Can't use precise numbers here because they differ between filesystems + printf "Disk usage: " + [ $(du -B1 $TEST_IMG | cut -f1) -lt 1048576 ] && echo "low" || echo "high" + +done + +# success, all done +echo "*** done" +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/243.out b/tests/qemu-iotests/243.out new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/243.out @@ -XXX,XX +XXX,XX @@ +QA output created by 243 + +=== preallocation=off === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=off +File size: 196616 +Disk usage: low + +=== preallocation=metadata === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=metadata +File size: 67436544 +Disk usage: low + +=== preallocation=falloc === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=falloc +File size: 67436544 +Disk usage: high + +=== preallocation=full === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=full +File size: 67436544 +Disk usage: high +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -XXX,XX +XXX,XX @@ 239 rw auto quick 240 auto quick 242 rw auto quick +243 rw auto quick -- 2.20.1 Image creation already involves a bdrv_co_truncate() call, which allows to specify a preallocation mode. Just pass the right mode there and remove the code that is made redundant by this. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.c | 28 +--------------------------- 1 file changed, 1 insertion(+), 27 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) goto out; } - if (qcow2_opts->preallocation == PREALLOC_MODE_FULL || - qcow2_opts->preallocation == PREALLOC_MODE_FALLOC) - { - int64_t prealloc_size = - qcow2_calc_prealloc_size(qcow2_opts->size, cluster_size, - refcount_order); - - ret = blk_truncate(blk, prealloc_size, qcow2_opts->preallocation, errp); - if (ret < 0) { - goto out; - } - } - /* Write the header */ QEMU_BUILD_BUG_ON((1 << MIN_CLUSTER_BITS) < sizeof(*header)); header = g_malloc0(cluster_size); @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } /* Okay, now that we have a valid image, let's give it the right size */ - ret = blk_truncate(blk, qcow2_opts->size, PREALLOC_MODE_OFF, errp); + ret = blk_truncate(blk, qcow2_opts->size, qcow2_opts->preallocation, errp); if (ret < 0) { error_prepend(errp, "Could not resize image: "); goto out; @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } } - /* And if we're supposed to preallocate metadata, do that now */ - if (qcow2_opts->preallocation != PREALLOC_MODE_OFF) { - BDRVQcow2State *s = blk_bs(blk)->opaque; - qemu_co_mutex_lock(&s->lock); - ret = preallocate_co(blk_bs(blk), 0, qcow2_opts->size); - qemu_co_mutex_unlock(&s->lock); - - if (ret < 0) { - error_setg_errno(errp, -ret, "Could not preallocate metadata"); - goto out; - } - } - blk_unref(blk); blk = NULL; -- 2.20.1 This adds external data file to the qcow2 spec as a new incompatible feature. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- docs/interop/qcow2.txt | 42 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 38 insertions(+), 4 deletions(-) diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index XXXXXXX..XXXXXXX 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -XXX,XX +XXX,XX @@ in the description of a field. be written to (unless for regaining consistency). - Bits 2-63: Reserved (set to 0) + Bit 2: External data file bit. If this bit is set, an + external data file is used. Guest clusters are + then stored in the external data file. For such + images, clusters in the external data file are + not refcounted. The offset field in the + Standard Cluster Descriptor must match the + guest offset and neither compressed clusters + nor internal snapshots are supported. + + An External Data File Name header extension may + be present if this bit is set. + + Bits 3-63: Reserved (set to 0) 80 - 87: compatible_features Bitmask of compatible features. An implementation can @@ -XXX,XX +XXX,XX @@ in the description of a field. bit is unset, the bitmaps extension data must be considered inconsistent. - Bits 1-63: Reserved (set to 0) + Bit 1: If this bit is set, the external data file can + be read as a consistent standalone raw image + without looking at the qcow2 metadata. + + Setting this bit has a performance impact for + some operations on the image (e.g. writing + zeros requires writing to the data file instead + of only setting the zero flag in the L2 table + entry) and conflicts with backing files. + + This bit may only be set if the External Data + File bit (incompatible feature bit 1) is also + set. + + Bits 2-63: Reserved (set to 0) 96 - 99: refcount_order Describes the width of a reference count block entry (width @@ -XXX,XX +XXX,XX @@ be stored. Each extension has a structure like the following: 0x6803f857 - Feature name table 0x23852875 - Bitmaps extension 0x0537be77 - Full disk encryption header pointer + 0x44415441 - External data file name other - Unknown header extension, can be safely ignored @@ -XXX,XX +XXX,XX @@ L2 table entry: This information is only accurate in L2 tables that are reachable from the active L1 table. + With external data files, all guest clusters have an + implicit refcount of 1 (because of the fixed host = guest + mapping for guest cluster offsets), so this bit should be 1 + for all allocated clusters. + Standard Cluster Descriptor: Bit 0: If set to 1, the cluster reads as all zeros. The host @@ -XXX,XX +XXX,XX @@ Standard Cluster Descriptor: 1 - 8: Reserved (set to 0) 9 - 55: Bits 9-55 of host cluster offset. Must be aligned to a - cluster boundary. If the offset is 0, the cluster is - unallocated. + cluster boundary. If the offset is 0 and bit 63 is clear, + the cluster is unallocated. The offset may only be 0 with + bit 63 set (indicating a host cluster offset of 0) when an + external data file is used. 56 - 61: Reserved (set to 0) -- 2.20.1 This adds basic constants, struct fields and helper function for external data file support to the implementation. QCOW2_INCOMPAT_MASK and QCOW2_AUTOCLEAR_MASK are not updated yet so that opening images with an external data file still fails (we don't handle them correctly yet). Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.h | 32 ++++++++++++++++++++++---------- block/qcow2.c | 9 +++++++++ tests/qemu-iotests/031.out | 8 ++++---- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 14 +++++++------- 5 files changed, 44 insertions(+), 23 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ enum { /* Incompatible feature bits */ enum { - QCOW2_INCOMPAT_DIRTY_BITNR = 0, - QCOW2_INCOMPAT_CORRUPT_BITNR = 1, - QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR, - QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR, - - QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY - | QCOW2_INCOMPAT_CORRUPT, + QCOW2_INCOMPAT_DIRTY_BITNR = 0, + QCOW2_INCOMPAT_CORRUPT_BITNR = 1, + QCOW2_INCOMPAT_DATA_FILE_BITNR = 2, + QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR, + QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR, + QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR, + + QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY + | QCOW2_INCOMPAT_CORRUPT, }; /* Compatible feature bits */ @@ -XXX,XX +XXX,XX @@ enum { /* Autoclear feature bits */ enum { - QCOW2_AUTOCLEAR_BITMAPS_BITNR = 0, - QCOW2_AUTOCLEAR_BITMAPS = 1 << QCOW2_AUTOCLEAR_BITMAPS_BITNR, + QCOW2_AUTOCLEAR_BITMAPS_BITNR = 0, + QCOW2_AUTOCLEAR_DATA_FILE_RAW_BITNR = 1, + QCOW2_AUTOCLEAR_BITMAPS = 1 << QCOW2_AUTOCLEAR_BITMAPS_BITNR, + QCOW2_AUTOCLEAR_DATA_FILE_RAW = 1 << QCOW2_AUTOCLEAR_DATA_FILE_RAW_BITNR, - QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS, + QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS, }; enum qcow2_discard_type { @@ -XXX,XX +XXX,XX @@ typedef struct BDRVQcow2State { CoQueue compress_wait_queue; int nb_compress_threads; + + BdrvChild *data_file; } BDRVQcow2State; typedef struct Qcow2COWRegion { @@ -XXX,XX +XXX,XX @@ typedef enum QCow2MetadataOverlap { #define REFT_OFFSET_MASK 0xfffffffffffffe00ULL +static inline bool has_data_file(BlockDriverState *bs) +{ + BDRVQcow2State *s = bs->opaque; + return (s->data_file != bs->file); +} + static inline int64_t start_of_cluster(BDRVQcow2State *s, int64_t offset) { return offset & ~(s->cluster_size - 1); diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ typedef struct { #define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857 #define QCOW2_EXT_MAGIC_CRYPTO_HEADER 0x0537be77 #define QCOW2_EXT_MAGIC_BITMAPS 0x23852875 +#define QCOW2_EXT_MAGIC_DATA_FILE 0x44415441 static int coroutine_fn qcow2_co_preadv_compressed(BlockDriverState *bs, @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, goto fail; } + /* TODO Open external data file */ + s->data_file = bs->file; + /* qcow2_read_extension may have set up the crypto context * if the crypt method needs a header region, some methods * don't need header extensions, so must check here @@ -XXX,XX +XXX,XX @@ int qcow2_update_header(BlockDriverState *bs) .bit = QCOW2_INCOMPAT_CORRUPT_BITNR, .name = "corrupt bit", }, + { + .type = QCOW2_FEAT_TYPE_INCOMPATIBLE, + .bit = QCOW2_INCOMPAT_DATA_FILE_BITNR, + .name = "external data file", + }, { .type = QCOW2_FEAT_TYPE_COMPATIBLE, .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/031.out +++ b/tests/qemu-iotests/031.out @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> Header extension: @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> Header extension: @@ -XXX,XX +XXX,XX @@ No errors were found on the image. magic 0x514649fb version 3 -backing_file_offset 0x148 +backing_file_offset 0x178 backing_file_size 0x17 cluster_bits 16 size 67108864 @@ -XXX,XX +XXX,XX @@ data 'host_device' Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> Header extension: diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/036.out +++ b/tests/qemu-iotests/036.out @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> *** done diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/061.out +++ b/tests/qemu-iotests/061.out @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> magic 0x514649fb @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> magic 0x514649fb @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> ERROR cluster 5 refcount=0 reference=1 @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> magic 0x514649fb @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> read 65536/65536 bytes at offset 44040192 @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> ERROR cluster 5 refcount=0 reference=1 @@ -XXX,XX +XXX,XX @@ header_length 104 Header extension: magic 0x6803f857 -length 144 +length 192 data <binary> read 131072/131072 bytes at offset 0 -- 2.20.1 Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.h | 3 ++- block/qcow2-cluster.c | 37 +++++++++++++++++++------------------ block/qcow2-refcount.c | 10 +++++----- 3 files changed, 26 insertions(+), 24 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ static inline int64_t qcow2_vm_state_offset(BDRVQcow2State *s) return (int64_t)s->l1_vm_state_index << (s->cluster_bits + s->l2_bits); } -static inline QCow2ClusterType qcow2_get_cluster_type(uint64_t l2_entry) +static inline QCow2ClusterType qcow2_get_cluster_type(BlockDriverState *bs, + uint64_t l2_entry) { if (l2_entry & QCOW_OFLAG_COMPRESSED) { return QCOW2_CLUSTER_COMPRESSED; diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -XXX,XX +XXX,XX @@ fail: * as contiguous. (This allows it, for example, to stop at the first compressed * cluster which may require a different handling) */ -static int count_contiguous_clusters(int nb_clusters, int cluster_size, - uint64_t *l2_slice, uint64_t stop_flags) +static int count_contiguous_clusters(BlockDriverState *bs, int nb_clusters, + int cluster_size, uint64_t *l2_slice, uint64_t stop_flags) { int i; QCow2ClusterType first_cluster_type; @@ -XXX,XX +XXX,XX @@ static int count_contiguous_clusters(int nb_clusters, int cluster_size, } /* must be allocated */ - first_cluster_type = qcow2_get_cluster_type(first_entry); + first_cluster_type = qcow2_get_cluster_type(bs, first_entry); assert(first_cluster_type == QCOW2_CLUSTER_NORMAL || first_cluster_type == QCOW2_CLUSTER_ZERO_ALLOC); @@ -XXX,XX +XXX,XX @@ static int count_contiguous_clusters(int nb_clusters, int cluster_size, * Checks how many consecutive unallocated clusters in a given L2 * slice have the same cluster type. */ -static int count_contiguous_clusters_unallocated(int nb_clusters, +static int count_contiguous_clusters_unallocated(BlockDriverState *bs, + int nb_clusters, uint64_t *l2_slice, QCow2ClusterType wanted_type) { @@ -XXX,XX +XXX,XX @@ static int count_contiguous_clusters_unallocated(int nb_clusters, wanted_type == QCOW2_CLUSTER_UNALLOCATED); for (i = 0; i < nb_clusters; i++) { uint64_t entry = be64_to_cpu(l2_slice[i]); - QCow2ClusterType type = qcow2_get_cluster_type(entry); + QCow2ClusterType type = qcow2_get_cluster_type(bs, entry); if (type != wanted_type) { break; @@ -XXX,XX +XXX,XX @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset, * true */ assert(nb_clusters <= INT_MAX); - type = qcow2_get_cluster_type(*cluster_offset); + type = qcow2_get_cluster_type(bs, *cluster_offset); if (s->qcow_version < 3 && (type == QCOW2_CLUSTER_ZERO_PLAIN || type == QCOW2_CLUSTER_ZERO_ALLOC)) { qcow2_signal_corruption(bs, true, -1, -1, "Zero cluster entry found" @@ -XXX,XX +XXX,XX @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset, case QCOW2_CLUSTER_ZERO_PLAIN: case QCOW2_CLUSTER_UNALLOCATED: /* how many empty clusters ? */ - c = count_contiguous_clusters_unallocated(nb_clusters, + c = count_contiguous_clusters_unallocated(bs, nb_clusters, &l2_slice[l2_index], type); *cluster_offset = 0; break; case QCOW2_CLUSTER_ZERO_ALLOC: case QCOW2_CLUSTER_NORMAL: /* how many allocated clusters ? */ - c = count_contiguous_clusters(nb_clusters, s->cluster_size, + c = count_contiguous_clusters(bs, nb_clusters, s->cluster_size, &l2_slice[l2_index], QCOW_OFLAG_ZERO); *cluster_offset &= L2E_OFFSET_MASK; if (offset_into_cluster(s, *cluster_offset)) { @@ -XXX,XX +XXX,XX @@ void qcow2_alloc_cluster_abort(BlockDriverState *bs, QCowL2Meta *m) * write, but require COW to be performed (this includes yet unallocated space, * which must copy from the backing file) */ -static int count_cow_clusters(BDRVQcow2State *s, int nb_clusters, +static int count_cow_clusters(BlockDriverState *bs, int nb_clusters, uint64_t *l2_slice, int l2_index) { int i; for (i = 0; i < nb_clusters; i++) { uint64_t l2_entry = be64_to_cpu(l2_slice[l2_index + i]); - QCow2ClusterType cluster_type = qcow2_get_cluster_type(l2_entry); + QCow2ClusterType cluster_type = qcow2_get_cluster_type(bs, l2_entry); switch(cluster_type) { case QCOW2_CLUSTER_NORMAL: @@ -XXX,XX +XXX,XX @@ static int handle_copied(BlockDriverState *bs, uint64_t guest_offset, cluster_offset = be64_to_cpu(l2_slice[l2_index]); /* Check how many clusters are already allocated and don't need COW */ - if (qcow2_get_cluster_type(cluster_offset) == QCOW2_CLUSTER_NORMAL + if (qcow2_get_cluster_type(bs, cluster_offset) == QCOW2_CLUSTER_NORMAL && (cluster_offset & QCOW_OFLAG_COPIED)) { /* If a specific host_offset is required, check it */ @@ -XXX,XX +XXX,XX @@ static int handle_copied(BlockDriverState *bs, uint64_t guest_offset, /* We keep all QCOW_OFLAG_COPIED clusters */ keep_clusters = - count_contiguous_clusters(nb_clusters, s->cluster_size, + count_contiguous_clusters(bs, nb_clusters, s->cluster_size, &l2_slice[l2_index], QCOW_OFLAG_COPIED | QCOW_OFLAG_ZERO); assert(keep_clusters <= nb_clusters); @@ -XXX,XX +XXX,XX @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset, if (entry & QCOW_OFLAG_COMPRESSED) { nb_clusters = 1; } else { - nb_clusters = count_cow_clusters(s, nb_clusters, l2_slice, l2_index); + nb_clusters = count_cow_clusters(bs, nb_clusters, l2_slice, l2_index); } /* This function is only called when there were no non-COW clusters, so if @@ -XXX,XX +XXX,XX @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset, * wrong with our code. */ assert(nb_clusters > 0); - if (qcow2_get_cluster_type(entry) == QCOW2_CLUSTER_ZERO_ALLOC && + if (qcow2_get_cluster_type(bs, entry) == QCOW2_CLUSTER_ZERO_ALLOC && (entry & QCOW_OFLAG_COPIED) && (!*host_offset || start_of_cluster(s, *host_offset) == (entry & L2E_OFFSET_MASK))) @@ -XXX,XX +XXX,XX @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset, * would be fine, too, but count_cow_clusters() above has limited * nb_clusters already to a range of COW clusters */ preallocated_nb_clusters = - count_contiguous_clusters(nb_clusters, s->cluster_size, + count_contiguous_clusters(bs, nb_clusters, s->cluster_size, &l2_slice[l2_index], QCOW_OFLAG_COPIED); assert(preallocated_nb_clusters > 0); @@ -XXX,XX +XXX,XX @@ static int discard_in_l2_slice(BlockDriverState *bs, uint64_t offset, * If full_discard is true, the sector should not read back as zeroes, * but rather fall through to the backing file. */ - switch (qcow2_get_cluster_type(old_l2_entry)) { + switch (qcow2_get_cluster_type(bs, old_l2_entry)) { case QCOW2_CLUSTER_UNALLOCATED: if (full_discard || !bs->backing) { continue; @@ -XXX,XX +XXX,XX @@ static int zero_in_l2_slice(BlockDriverState *bs, uint64_t offset, * Minimize L2 changes if the cluster already reads back as * zeroes with correct allocation. */ - cluster_type = qcow2_get_cluster_type(old_offset); + cluster_type = qcow2_get_cluster_type(bs, old_offset); if (cluster_type == QCOW2_CLUSTER_ZERO_PLAIN || (cluster_type == QCOW2_CLUSTER_ZERO_ALLOC && !unmap)) { continue; @@ -XXX,XX +XXX,XX @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, uint64_t l2_entry = be64_to_cpu(l2_slice[j]); int64_t offset = l2_entry & L2E_OFFSET_MASK; QCow2ClusterType cluster_type = - qcow2_get_cluster_type(l2_entry); + qcow2_get_cluster_type(bs, l2_entry); if (cluster_type != QCOW2_CLUSTER_ZERO_PLAIN && cluster_type != QCOW2_CLUSTER_ZERO_ALLOC) { diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -XXX,XX +XXX,XX @@ void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry, { BDRVQcow2State *s = bs->opaque; - switch (qcow2_get_cluster_type(l2_entry)) { + switch (qcow2_get_cluster_type(bs, l2_entry)) { case QCOW2_CLUSTER_COMPRESSED: { int nb_csectors; @@ -XXX,XX +XXX,XX @@ int qcow2_update_snapshot_refcount(BlockDriverState *bs, entry &= ~QCOW_OFLAG_COPIED; offset = entry & L2E_OFFSET_MASK; - switch (qcow2_get_cluster_type(entry)) { + switch (qcow2_get_cluster_type(bs, entry)) { case QCOW2_CLUSTER_COMPRESSED: nb_csectors = ((entry >> s->csize_shift) & s->csize_mask) + 1; @@ -XXX,XX +XXX,XX @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, for(i = 0; i < s->l2_size; i++) { l2_entry = be64_to_cpu(l2_table[i]); - switch (qcow2_get_cluster_type(l2_entry)) { + switch (qcow2_get_cluster_type(bs, l2_entry)) { case QCOW2_CLUSTER_COMPRESSED: /* Compressed clusters don't have QCOW_OFLAG_COPIED */ if (l2_entry & QCOW_OFLAG_COPIED) { @@ -XXX,XX +XXX,XX @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, /* Correct offsets are cluster aligned */ if (offset_into_cluster(s, offset)) { - if (qcow2_get_cluster_type(l2_entry) == + if (qcow2_get_cluster_type(bs, l2_entry) == QCOW2_CLUSTER_ZERO_ALLOC) { fprintf(stderr, "%s offset=%" PRIx64 ": Preallocated zero " @@ -XXX,XX +XXX,XX @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res, for (j = 0; j < s->l2_size; j++) { uint64_t l2_entry = be64_to_cpu(l2_table[j]); uint64_t data_offset = l2_entry & L2E_OFFSET_MASK; - QCow2ClusterType cluster_type = qcow2_get_cluster_type(l2_entry); + QCow2ClusterType cluster_type = qcow2_get_cluster_type(bs, l2_entry); if (cluster_type == QCOW2_CLUSTER_NORMAL || cluster_type == QCOW2_CLUSTER_ZERO_ALLOC) { -- 2.20.1 Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.h | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ static inline QCow2ClusterType qcow2_get_cluster_type(BlockDriverState *bs, } return QCOW2_CLUSTER_ZERO_PLAIN; } else if (!(l2_entry & L2E_OFFSET_MASK)) { - return QCOW2_CLUSTER_UNALLOCATED; + /* Offset 0 generally means unallocated, but it is ambiguous with + * external data files because 0 is a valid offset there. However, all + * clusters in external data files always have refcount 1, so we can + * rely on QCOW_OFLAG_COPIED to disambiguate. */ + if (has_data_file(bs) && (l2_entry & QCOW_OFLAG_COPIED)) { + return QCOW2_CLUSTER_NORMAL; + } else { + return QCOW2_CLUSTER_UNALLOCATED; + } } else { return QCOW2_CLUSTER_NORMAL; } -- 2.20.1 Offset 0 can be valid for normal (allocated) clusters now, so use qcow2_get_cluster_type() instead. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2-cluster.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -XXX,XX +XXX,XX @@ static int count_contiguous_clusters(BlockDriverState *bs, int nb_clusters, uint64_t first_entry = be64_to_cpu(l2_slice[0]); uint64_t offset = first_entry & mask; - if (!offset) { + first_cluster_type = qcow2_get_cluster_type(bs, first_entry); + if (first_cluster_type == QCOW2_CLUSTER_UNALLOCATED) { return 0; } /* must be allocated */ - first_cluster_type = qcow2_get_cluster_type(bs, first_entry); assert(first_cluster_type == QCOW2_CLUSTER_NORMAL || first_cluster_type == QCOW2_CLUSTER_ZERO_ALLOC); -- 2.20.1 The cluster allocation code uses 0 as an invalid offset that is used in case of errors or as "offset not yet determined". With external data files, a host cluster offset of 0 becomes valid, though. Define a constant INV_OFFSET (which is not cluster aligned and will therefore never be a valid offset) that can be used for such purposes. This removes the additional host_offset == 0 check that commit ff52aab2df5 introduced; the confusion between an invalid offset and (erroneous) allocation at offset 0 is removed with this change. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.h | 2 ++ block/qcow2-cluster.c | 59 ++++++++++++++++++++----------------------- 2 files changed, 29 insertions(+), 32 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ typedef enum QCow2MetadataOverlap { #define REFT_OFFSET_MASK 0xfffffffffffffe00ULL +#define INV_OFFSET (-1ULL) + static inline bool has_data_file(BlockDriverState *bs) { BDRVQcow2State *s = bs->opaque; diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -XXX,XX +XXX,XX @@ static int handle_dependencies(BlockDriverState *bs, uint64_t guest_offset, /* * Checks how many already allocated clusters that don't require a copy on - * write there are at the given guest_offset (up to *bytes). If - * *host_offset is not zero, only physically contiguous clusters beginning at - * this host offset are counted. + * write there are at the given guest_offset (up to *bytes). If *host_offset is + * not INV_OFFSET, only physically contiguous clusters beginning at this host + * offset are counted. * * Note that guest_offset may not be cluster aligned. In this case, the * returned *host_offset points to exact byte referenced by guest_offset and @@ -XXX,XX +XXX,XX @@ static int handle_copied(BlockDriverState *bs, uint64_t guest_offset, trace_qcow2_handle_copied(qemu_coroutine_self(), guest_offset, *host_offset, *bytes); - assert(*host_offset == 0 || offset_into_cluster(s, guest_offset) - == offset_into_cluster(s, *host_offset)); + assert(*host_offset == INV_OFFSET || offset_into_cluster(s, guest_offset) + == offset_into_cluster(s, *host_offset)); /* * Calculate the number of clusters to look for. We stop at L2 slice @@ -XXX,XX +XXX,XX @@ static int handle_copied(BlockDriverState *bs, uint64_t guest_offset, goto out; } - if (*host_offset != 0 && !offset_matches) { + if (*host_offset != INV_OFFSET && !offset_matches) { *bytes = 0; ret = 0; goto out; @@ -XXX,XX +XXX,XX @@ out: * contain the number of clusters that have been allocated and are contiguous * in the image file. * - * If *host_offset is non-zero, it specifies the offset in the image file at - * which the new clusters must start. *nb_clusters can be 0 on return in this - * case if the cluster at host_offset is already in use. If *host_offset is - * zero, the clusters can be allocated anywhere in the image file. + * If *host_offset is not INV_OFFSET, it specifies the offset in the image file + * at which the new clusters must start. *nb_clusters can be 0 on return in + * this case if the cluster at host_offset is already in use. If *host_offset + * is INV_OFFSET, the clusters can be allocated anywhere in the image file. * * *host_offset is updated to contain the offset into the image file at which * the first allocated cluster starts. @@ -XXX,XX +XXX,XX @@ static int do_alloc_cluster_offset(BlockDriverState *bs, uint64_t guest_offset, /* Allocate new clusters */ trace_qcow2_cluster_alloc_phys(qemu_coroutine_self()); - if (*host_offset == 0) { + if (*host_offset == INV_OFFSET) { int64_t cluster_offset = qcow2_alloc_clusters(bs, *nb_clusters * s->cluster_size); if (cluster_offset < 0) { @@ -XXX,XX +XXX,XX @@ static int do_alloc_cluster_offset(BlockDriverState *bs, uint64_t guest_offset, /* * Allocates new clusters for an area that either is yet unallocated or needs a - * copy on write. If *host_offset is non-zero, clusters are only allocated if - * the new allocation can match the specified host offset. + * copy on write. If *host_offset is not INV_OFFSET, clusters are only + * allocated if the new allocation can match the specified host offset. * * Note that guest_offset may not be cluster aligned. In this case, the * returned *host_offset points to exact byte referenced by guest_offset and @@ -XXX,XX +XXX,XX @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset, int ret; bool keep_old_clusters = false; - uint64_t alloc_cluster_offset = 0; + uint64_t alloc_cluster_offset = INV_OFFSET; trace_qcow2_handle_alloc(qemu_coroutine_self(), guest_offset, *host_offset, *bytes); @@ -XXX,XX +XXX,XX @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset, if (qcow2_get_cluster_type(bs, entry) == QCOW2_CLUSTER_ZERO_ALLOC && (entry & QCOW_OFLAG_COPIED) && - (!*host_offset || + (*host_offset == INV_OFFSET || start_of_cluster(s, *host_offset) == (entry & L2E_OFFSET_MASK))) { int preallocated_nb_clusters; @@ -XXX,XX +XXX,XX @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset, qcow2_cache_put(s->l2_table_cache, (void **) &l2_slice); - if (!alloc_cluster_offset) { + if (alloc_cluster_offset == INV_OFFSET) { /* Allocate, if necessary at a given offset in the image file */ - alloc_cluster_offset = start_of_cluster(s, *host_offset); + alloc_cluster_offset = *host_offset == INV_OFFSET ? INV_OFFSET : + start_of_cluster(s, *host_offset); ret = do_alloc_cluster_offset(bs, guest_offset, &alloc_cluster_offset, &nb_clusters); if (ret < 0) { @@ -XXX,XX +XXX,XX @@ static int handle_alloc(BlockDriverState *bs, uint64_t guest_offset, return 0; } - /* !*host_offset would overwrite the image header and is reserved for - * "no host offset preferred". If 0 was a valid host offset, it'd - * trigger the following overlap check; do that now to avoid having an - * invalid value in *host_offset. */ - if (!alloc_cluster_offset) { - ret = qcow2_pre_write_overlap_check(bs, 0, alloc_cluster_offset, - nb_clusters * s->cluster_size); - assert(ret < 0); - goto fail; - } + assert(alloc_cluster_offset != INV_OFFSET); } /* @@ -XXX,XX +XXX,XX @@ int qcow2_alloc_cluster_offset(BlockDriverState *bs, uint64_t offset, again: start = offset; remaining = *bytes; - cluster_offset = 0; - *host_offset = 0; + cluster_offset = INV_OFFSET; + *host_offset = INV_OFFSET; cur_bytes = 0; *m = NULL; while (true) { - if (!*host_offset) { + if (*host_offset == INV_OFFSET && cluster_offset != INV_OFFSET) { *host_offset = start_of_cluster(s, cluster_offset); } @@ -XXX,XX +XXX,XX @@ again: start += cur_bytes; remaining -= cur_bytes; - cluster_offset += cur_bytes; + + if (cluster_offset != INV_OFFSET) { + cluster_offset += cur_bytes; + } if (remaining == 0) { break; @@ -XXX,XX +XXX,XX @@ again: *bytes -= remaining; assert(*bytes > 0); - assert(*host_offset != 0); + assert(*host_offset != INV_OFFSET); return 0; } -- 2.20.1 qcow2_alloc_compressed_cluster_offset() used to return the cluster offset for success and 0 for error. This doesn't only conflict with 0 as a valid host offset, but also loses the error code. Similar to the change made to qcow2_alloc_cluster_offset() for uncompressed clusters in commit 148da7ea9d6, make the function return 0/-errno and return the allocated cluster offset in a by-reference parameter. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.h | 7 ++++--- block/qcow2-cluster.c | 28 +++++++++++++--------------- block/qcow2.c | 19 ++++++++----------- tests/qemu-iotests/220.out | 2 +- 4 files changed, 26 insertions(+), 30 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset, int qcow2_alloc_cluster_offset(BlockDriverState *bs, uint64_t offset, unsigned int *bytes, uint64_t *host_offset, QCowL2Meta **m); -uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs, - uint64_t offset, - int compressed_size); +int qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs, + uint64_t offset, + int compressed_size, + uint64_t *host_offset); int qcow2_alloc_cluster_link_l2(BlockDriverState *bs, QCowL2Meta *m); void qcow2_alloc_cluster_abort(BlockDriverState *bs, QCowL2Meta *m); diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -XXX,XX +XXX,XX @@ static int get_cluster_table(BlockDriverState *bs, uint64_t offset, /* * alloc_compressed_cluster_offset * - * For a given offset of the disk image, return cluster offset in - * qcow2 file. - * - * If the offset is not found, allocate a new compressed cluster. - * - * Return the cluster offset if successful, - * Return 0, otherwise. + * For a given offset on the virtual disk, allocate a new compressed cluster + * and put the host offset of the cluster into *host_offset. If a cluster is + * already allocated at the offset, return an error. * + * Return 0 on success and -errno in error cases */ - -uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs, - uint64_t offset, - int compressed_size) +int qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs, + uint64_t offset, + int compressed_size, + uint64_t *host_offset) { BDRVQcow2State *s = bs->opaque; int l2_index, ret; @@ -XXX,XX +XXX,XX @@ uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs, ret = get_cluster_table(bs, offset, &l2_slice, &l2_index); if (ret < 0) { - return 0; + return ret; } /* Compression can't overwrite anything. Fail if the cluster was already @@ -XXX,XX +XXX,XX @@ uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs, cluster_offset = be64_to_cpu(l2_slice[l2_index]); if (cluster_offset & L2E_OFFSET_MASK) { qcow2_cache_put(s->l2_table_cache, (void **) &l2_slice); - return 0; + return -EIO; } cluster_offset = qcow2_alloc_bytes(bs, compressed_size); if (cluster_offset < 0) { qcow2_cache_put(s->l2_table_cache, (void **) &l2_slice); - return 0; + return cluster_offset; } nb_csectors = ((cluster_offset + compressed_size - 1) >> 9) - @@ -XXX,XX +XXX,XX @@ uint64_t qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs, l2_slice[l2_index] = cpu_to_be64(cluster_offset); qcow2_cache_put(s->l2_table_cache, (void **) &l2_slice); - return cluster_offset; + *host_offset = cluster_offset & s->cluster_offset_mask; + return 0; } static int perform_cow(BlockDriverState *bs, QCowL2Meta *m) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset, int ret; size_t out_len; uint8_t *buf, *out_buf; - int64_t cluster_offset; + uint64_t cluster_offset; if (bytes == 0) { /* align end of file to a sector boundary to ease reading with sector based I/Os */ - cluster_offset = bdrv_getlength(bs->file->bs); - if (cluster_offset < 0) { - return cluster_offset; + int64_t len = bdrv_getlength(bs->file->bs); + if (len < 0) { + return len; } - return bdrv_co_truncate(bs->file, cluster_offset, PREALLOC_MODE_OFF, - NULL); + return bdrv_co_truncate(bs->file, len, PREALLOC_MODE_OFF, NULL); } if (offset_into_cluster(s, offset)) { @@ -XXX,XX +XXX,XX @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset, } qemu_co_mutex_lock(&s->lock); - cluster_offset = - qcow2_alloc_compressed_cluster_offset(bs, offset, out_len); - if (!cluster_offset) { + ret = qcow2_alloc_compressed_cluster_offset(bs, offset, out_len, + &cluster_offset); + if (ret < 0) { qemu_co_mutex_unlock(&s->lock); - ret = -EIO; goto fail; } - cluster_offset &= s->cluster_offset_mask; ret = qcow2_pre_write_overlap_check(bs, 0, cluster_offset, out_len); qemu_co_mutex_unlock(&s->lock); diff --git a/tests/qemu-iotests/220.out b/tests/qemu-iotests/220.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/220.out +++ b/tests/qemu-iotests/220.out @@ -XXX,XX +XXX,XX @@ wrote 2097152/2097152 bytes at offset 37748736 No errors were found on the image. image size 39845888 == Trying to write compressed cluster == -write failed: Input/output error +write failed: File too large image size 562949957615616 == Writing normal cluster == wrote 2097152/2097152 bytes at offset 0 -- 2.20.1 Offset 0 cannot be assumed to mean an unallocated cluster any more. Instead, the cluster type needs to be checked. *file must refer to the data file instead of the image file if a valid offset is returned from qcow2_co_block_status(). Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_block_status(BlockDriverState *bs, *pnum = bytes; - if (cluster_offset != 0 && ret != QCOW2_CLUSTER_COMPRESSED && + if ((ret == QCOW2_CLUSTER_NORMAL || ret == QCOW2_CLUSTER_ZERO_ALLOC) && !s->crypto) { index_in_cluster = offset & (s->cluster_size - 1); *map = cluster_offset | index_in_cluster; - *file = bs->file->bs; + *file = s->data_file->bs; status |= BDRV_BLOCK_OFFSET_VALID; } if (ret == QCOW2_CLUSTER_ZERO_PLAIN || ret == QCOW2_CLUSTER_ZERO_ALLOC) { -- 2.20.1 This changes the qcow2 implementation to direct all guest data I/O to s->data_file rather than bs->file, while metadata I/O still uses bs->file. At the moment, this is still always the same, but soon we'll add options to set s->data_file to an external data file. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2.h | 2 +- block/qcow2-bitmap.c | 7 +++--- block/qcow2-cache.c | 6 ++--- block/qcow2-cluster.c | 46 +++++++++++++++++++++++++++++++------ block/qcow2-refcount.c | 39 +++++++++++++++++++++++-------- block/qcow2-snapshot.c | 7 +++--- block/qcow2.c | 52 +++++++++++++++++++++++++++++++++--------- 7 files changed, 122 insertions(+), 37 deletions(-) diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ void qcow2_process_discards(BlockDriverState *bs, int ret); int qcow2_check_metadata_overlap(BlockDriverState *bs, int ign, int64_t offset, int64_t size); int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, - int64_t size); + int64_t size, bool data_file); int qcow2_inc_refcounts_imrt(BlockDriverState *bs, BdrvCheckResult *res, void **refcount_table, int64_t *refcount_table_size, diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -XXX,XX +XXX,XX @@ static int bitmap_list_store(BlockDriverState *bs, Qcow2BitmapList *bm_list, * directory in-place (actually, turn-off the extension), which is checked * in qcow2_check_metadata_overlap() */ ret = qcow2_pre_write_overlap_check( - bs, in_place ? QCOW2_OL_BITMAP_DIRECTORY : 0, dir_offset, dir_size); + bs, in_place ? QCOW2_OL_BITMAP_DIRECTORY : 0, dir_offset, dir_size, + false); if (ret < 0) { goto fail; } @@ -XXX,XX +XXX,XX @@ static uint64_t *store_bitmap_data(BlockDriverState *bs, memset(buf + write_size, 0, s->cluster_size - write_size); } - ret = qcow2_pre_write_overlap_check(bs, 0, off, s->cluster_size); + ret = qcow2_pre_write_overlap_check(bs, 0, off, s->cluster_size, false); if (ret < 0) { error_setg_errno(errp, -ret, "Qcow2 overlap check failed"); goto fail; @@ -XXX,XX +XXX,XX @@ static int store_bitmap(BlockDriverState *bs, Qcow2Bitmap *bm, Error **errp) } ret = qcow2_pre_write_overlap_check(bs, 0, tb_offset, - tb_size * sizeof(tb[0])); + tb_size * sizeof(tb[0]), false); if (ret < 0) { error_setg_errno(errp, -ret, "Qcow2 overlap check failed"); goto fail; diff --git a/block/qcow2-cache.c b/block/qcow2-cache.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cache.c +++ b/block/qcow2-cache.c @@ -XXX,XX +XXX,XX @@ static int qcow2_cache_entry_flush(BlockDriverState *bs, Qcow2Cache *c, int i) if (c == s->refcount_block_cache) { ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_REFCOUNT_BLOCK, - c->entries[i].offset, c->table_size); + c->entries[i].offset, c->table_size, false); } else if (c == s->l2_table_cache) { ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_ACTIVE_L2, - c->entries[i].offset, c->table_size); + c->entries[i].offset, c->table_size, false); } else { ret = qcow2_pre_write_overlap_check(bs, 0, - c->entries[i].offset, c->table_size); + c->entries[i].offset, c->table_size, false); } if (ret < 0) { diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -XXX,XX +XXX,XX @@ int qcow2_grow_l1_table(BlockDriverState *bs, uint64_t min_size, /* the L1 position has not yet been updated, so these clusters must * indeed be completely free */ ret = qcow2_pre_write_overlap_check(bs, 0, new_l1_table_offset, - new_l1_size2); + new_l1_size2, false); if (ret < 0) { goto fail; } @@ -XXX,XX +XXX,XX @@ int qcow2_write_l1_entry(BlockDriverState *bs, int l1_index) } ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_ACTIVE_L1, - s->l1_table_offset + 8 * l1_start_index, sizeof(buf)); + s->l1_table_offset + 8 * l1_start_index, sizeof(buf), false); if (ret < 0) { return ret; } @@ -XXX,XX +XXX,XX @@ static int coroutine_fn do_perform_cow_write(BlockDriverState *bs, unsigned offset_in_cluster, QEMUIOVector *qiov) { + BDRVQcow2State *s = bs->opaque; int ret; if (qiov->size == 0) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn do_perform_cow_write(BlockDriverState *bs, } ret = qcow2_pre_write_overlap_check(bs, 0, - cluster_offset + offset_in_cluster, qiov->size); + cluster_offset + offset_in_cluster, qiov->size, true); if (ret < 0) { return ret; } BLKDBG_EVENT(bs->file, BLKDBG_COW_WRITE); - ret = bdrv_co_pwritev(bs->file, cluster_offset + offset_in_cluster, + ret = bdrv_co_pwritev(s->data_file, cluster_offset + offset_in_cluster, qiov->size, qiov, 0); if (ret < 0) { return ret; @@ -XXX,XX +XXX,XX @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset, } switch (type) { case QCOW2_CLUSTER_COMPRESSED: + if (has_data_file(bs)) { + qcow2_signal_corruption(bs, true, -1, -1, "Compressed cluster " + "entry found in image with external data " + "file (L2 offset: %#" PRIx64 ", L2 index: " + "%#x)", l2_offset, l2_index); + ret = -EIO; + goto fail; + } /* Compressed clusters can only be processed one by one */ c = 1; *cluster_offset &= L2E_COMPRESSED_OFFSET_SIZE_MASK; @@ -XXX,XX +XXX,XX @@ int qcow2_get_cluster_offset(BlockDriverState *bs, uint64_t offset, ret = -EIO; goto fail; } + if (has_data_file(bs) && *cluster_offset != offset - offset_in_cluster) + { + qcow2_signal_corruption(bs, true, -1, -1, + "External data file host cluster offset %#" + PRIx64 " does not match guest cluster " + "offset: %#" PRIx64 + ", L2 index: %#x)", *cluster_offset, + offset - offset_in_cluster, l2_index); + ret = -EIO; + goto fail; + } break; default: abort(); @@ -XXX,XX +XXX,XX @@ int qcow2_alloc_compressed_cluster_offset(BlockDriverState *bs, int64_t cluster_offset; int nb_csectors; + if (has_data_file(bs)) { + return 0; + } + ret = get_cluster_table(bs, offset, &l2_slice, &l2_index); if (ret < 0) { return ret; @@ -XXX,XX +XXX,XX @@ static int do_alloc_cluster_offset(BlockDriverState *bs, uint64_t guest_offset, trace_qcow2_do_alloc_clusters_offset(qemu_coroutine_self(), guest_offset, *host_offset, *nb_clusters); + if (has_data_file(bs)) { + assert(*host_offset == INV_OFFSET || + *host_offset == start_of_cluster(s, guest_offset)); + *host_offset = start_of_cluster(s, guest_offset); + return 0; + } + /* Allocate new clusters */ trace_qcow2_cluster_alloc_phys(qemu_coroutine_self()); if (*host_offset == INV_OFFSET) { @@ -XXX,XX +XXX,XX @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, } ret = qcow2_pre_write_overlap_check(bs, 0, offset, - s->cluster_size); + s->cluster_size, true); if (ret < 0) { if (cluster_type == QCOW2_CLUSTER_ZERO_PLAIN) { qcow2_free_clusters(bs, offset, s->cluster_size, @@ -XXX,XX +XXX,XX @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, goto fail; } - ret = bdrv_pwrite_zeroes(bs->file, offset, s->cluster_size, 0); + ret = bdrv_pwrite_zeroes(s->data_file, offset, + s->cluster_size, 0); if (ret < 0) { if (cluster_type == QCOW2_CLUSTER_ZERO_PLAIN) { qcow2_free_clusters(bs, offset, s->cluster_size, @@ -XXX,XX +XXX,XX @@ static int expand_zero_clusters_in_l1(BlockDriverState *bs, uint64_t *l1_table, if (l2_dirty) { ret = qcow2_pre_write_overlap_check( bs, QCOW2_OL_INACTIVE_L2 | QCOW2_OL_ACTIVE_L2, - slice_offset, slice_size2); + slice_offset, slice_size2, false); if (ret < 0) { goto fail; } diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -XXX,XX +XXX,XX @@ void qcow2_free_any_clusters(BlockDriverState *bs, uint64_t l2_entry, int nb_clusters, enum qcow2_discard_type type) { BDRVQcow2State *s = bs->opaque; + QCow2ClusterType ctype = qcow2_get_cluster_type(bs, l2_entry); - switch (qcow2_get_cluster_type(bs, l2_entry)) { + if (has_data_file(bs)) { + if (s->discard_passthrough[type] && + (ctype == QCOW2_CLUSTER_NORMAL || + ctype == QCOW2_CLUSTER_ZERO_ALLOC)) + { + bdrv_pdiscard(s->data_file, l2_entry & L2E_OFFSET_MASK, + nb_clusters << s->cluster_bits); + } + return; + } + + switch (ctype) { case QCOW2_CLUSTER_COMPRESSED: { int nb_csectors; @@ -XXX,XX +XXX,XX @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, l2_table[i] = cpu_to_be64(l2_entry); ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_ACTIVE_L2 | QCOW2_OL_INACTIVE_L2, - l2e_offset, sizeof(uint64_t)); + l2e_offset, sizeof(uint64_t), false); if (ret < 0) { fprintf(stderr, "ERROR: Overlap check failed\n"); res->check_errors++; @@ -XXX,XX +XXX,XX @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res, if (l2_dirty) { ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_ACTIVE_L2, - l2_offset, s->cluster_size); + l2_offset, s->cluster_size, + false); if (ret < 0) { fprintf(stderr, "ERROR: Could not write L2 table; metadata " "overlap check failed: %s\n", strerror(-ret)); @@ -XXX,XX +XXX,XX @@ write_refblocks: } ret = qcow2_pre_write_overlap_check(bs, 0, refblock_offset, - s->cluster_size); + s->cluster_size, false); if (ret < 0) { fprintf(stderr, "ERROR writing refblock: %s\n", strerror(-ret)); goto fail; @@ -XXX,XX +XXX,XX @@ write_refblocks: } ret = qcow2_pre_write_overlap_check(bs, 0, reftable_offset, - reftable_size * sizeof(uint64_t)); + reftable_size * sizeof(uint64_t), + false); if (ret < 0) { fprintf(stderr, "ERROR writing reftable: %s\n", strerror(-ret)); goto fail; @@ -XXX,XX +XXX,XX @@ QEMU_BUILD_BUG_ON(QCOW2_OL_MAX_BITNR != ARRAY_SIZE(metadata_ol_names)); * overlaps; or a negative value (-errno) on error. */ int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t offset, - int64_t size) + int64_t size, bool data_file) { - int ret = qcow2_check_metadata_overlap(bs, ign, offset, size); + int ret; + + if (data_file && has_data_file(bs)) { + return 0; + } + ret = qcow2_check_metadata_overlap(bs, ign, offset, size); if (ret < 0) { return ret; } else if (ret > 0) { @@ -XXX,XX +XXX,XX @@ static int flush_refblock(BlockDriverState *bs, uint64_t **reftable, if (reftable_index < *reftable_size && (*reftable)[reftable_index]) { offset = (*reftable)[reftable_index]; - ret = qcow2_pre_write_overlap_check(bs, 0, offset, s->cluster_size); + ret = qcow2_pre_write_overlap_check(bs, 0, offset, s->cluster_size, + false); if (ret < 0) { error_setg_errno(errp, -ret, "Overlap check failed"); return ret; @@ -XXX,XX +XXX,XX @@ int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order, /* Write the new reftable */ ret = qcow2_pre_write_overlap_check(bs, 0, new_reftable_offset, - new_reftable_size * sizeof(uint64_t)); + new_reftable_size * sizeof(uint64_t), + false); if (ret < 0) { error_setg_errno(errp, -ret, "Overlap check failed"); goto done; diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-snapshot.c +++ b/block/qcow2-snapshot.c @@ -XXX,XX +XXX,XX @@ static int qcow2_write_snapshots(BlockDriverState *bs) /* The snapshot list position has not yet been updated, so these clusters * must indeed be completely free */ - ret = qcow2_pre_write_overlap_check(bs, 0, offset, snapshots_size); + ret = qcow2_pre_write_overlap_check(bs, 0, offset, snapshots_size, false); if (ret < 0) { goto fail; } @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info) } ret = qcow2_pre_write_overlap_check(bs, 0, sn->l1_table_offset, - s->l1_size * sizeof(uint64_t)); + s->l1_size * sizeof(uint64_t), false); if (ret < 0) { goto fail; } @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id) } ret = qcow2_pre_write_overlap_check(bs, QCOW2_OL_ACTIVE_L1, - s->l1_table_offset, cur_l1_bytes); + s->l1_table_offset, cur_l1_bytes, + false); if (ret < 0) { goto fail; } diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static ssize_t qcow2_crypto_hdr_init_func(QCryptoBlock *block, size_t headerlen, /* Zero fill remaining space in cluster so it has predictable * content in case of future spec changes */ clusterlen = size_to_clusters(s, headerlen) * s->cluster_size; - assert(qcow2_pre_write_overlap_check(bs, 0, ret, clusterlen) == 0); + assert(qcow2_pre_write_overlap_check(bs, 0, ret, clusterlen, false) == 0); ret = bdrv_pwrite_zeroes(bs->file, ret + headerlen, clusterlen - headerlen, 0); @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_preadv(BlockDriverState *bs, uint64_t offset, */ if (!cluster_data) { cluster_data = - qemu_try_blockalign(bs->file->bs, + qemu_try_blockalign(s->data_file->bs, QCOW_MAX_CRYPT_CLUSTERS * s->cluster_size); if (cluster_data == NULL) { @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_preadv(BlockDriverState *bs, uint64_t offset, BLKDBG_EVENT(bs->file, BLKDBG_READ_AIO); qemu_co_mutex_unlock(&s->lock); - ret = bdrv_co_preadv(bs->file, + ret = bdrv_co_preadv(s->data_file, cluster_offset + offset_in_cluster, cur_bytes, &hd_qiov, 0); qemu_co_mutex_lock(&s->lock); @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t offset, } ret = qcow2_pre_write_overlap_check(bs, 0, - cluster_offset + offset_in_cluster, cur_bytes); + cluster_offset + offset_in_cluster, cur_bytes, true); if (ret < 0) { goto fail; } @@ -XXX,XX +XXX,XX @@ static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t offset, BLKDBG_EVENT(bs->file, BLKDBG_WRITE_AIO); trace_qcow2_writev_data(qemu_coroutine_self(), cluster_offset + offset_in_cluster); - ret = bdrv_co_pwritev(bs->file, + ret = bdrv_co_pwritev(s->data_file, cluster_offset + offset_in_cluster, cur_bytes, &hd_qiov, 0); qemu_co_mutex_lock(&s->lock); @@ -XXX,XX +XXX,XX @@ qcow2_co_copy_range_from(BlockDriverState *bs, goto out; case QCOW2_CLUSTER_NORMAL: - child = bs->file; + child = s->data_file; copy_offset += offset_into_cluster(s, src_offset); if ((copy_offset & 511) != 0) { ret = -EIO; @@ -XXX,XX +XXX,XX @@ qcow2_co_copy_range_to(BlockDriverState *bs, assert((cluster_offset & 511) == 0); ret = qcow2_pre_write_overlap_check(bs, 0, - cluster_offset + offset_in_cluster, cur_bytes); + cluster_offset + offset_in_cluster, cur_bytes, true); if (ret < 0) { goto fail; } qemu_co_mutex_unlock(&s->lock); ret = bdrv_co_copy_range_to(src, src_offset, - bs->file, + s->data_file, cluster_offset + offset_in_cluster, cur_bytes, read_flags, write_flags); qemu_co_mutex_lock(&s->lock); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, int64_t old_file_size, new_file_size; uint64_t nb_new_data_clusters, nb_new_l2_tables; + /* With a data file, preallocation means just allocating the metadata + * and forwarding the truncate request to the data file */ + if (has_data_file(bs)) { + ret = preallocate_co(bs, old_length, offset); + if (ret < 0) { + error_setg_errno(errp, -ret, "Preallocation failed"); + goto fail; + } + break; + } + old_file_size = bdrv_getlength(bs->file->bs); if (old_file_size < 0) { error_setg_errno(errp, -old_file_size, @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_truncate(BlockDriverState *bs, int64_t offset, bs->total_sectors = offset / BDRV_SECTOR_SIZE; + if (has_data_file(bs)) { + if (prealloc == PREALLOC_MODE_METADATA) { + prealloc = PREALLOC_MODE_OFF; + } + ret = bdrv_co_truncate(s->data_file, offset, prealloc, errp); + if (ret < 0) { + goto fail; + } + } + /* write updated header.size */ offset = cpu_to_be64(offset); ret = bdrv_pwrite_sync(bs->file, offsetof(QCowHeader, size), @@ -XXX,XX +XXX,XX @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset, uint8_t *buf, *out_buf; uint64_t cluster_offset; + if (has_data_file(bs)) { + return -ENOTSUP; + } + if (bytes == 0) { /* align end of file to a sector boundary to ease reading with sector based I/Os */ @@ -XXX,XX +XXX,XX @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset, goto fail; } - ret = qcow2_pre_write_overlap_check(bs, 0, cluster_offset, out_len); + ret = qcow2_pre_write_overlap_check(bs, 0, cluster_offset, out_len, true); qemu_co_mutex_unlock(&s->lock); if (ret < 0) { goto fail; @@ -XXX,XX +XXX,XX @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset, qemu_iovec_init_buf(&hd_qiov, out_buf, out_len); - BLKDBG_EVENT(bs->file, BLKDBG_WRITE_COMPRESSED); - ret = bdrv_co_pwritev(bs->file, cluster_offset, out_len, &hd_qiov, 0); + BLKDBG_EVENT(s->data_file, BLKDBG_WRITE_COMPRESSED); + ret = bdrv_co_pwritev(s->data_file, cluster_offset, out_len, &hd_qiov, 0); if (ret < 0) { goto fail; } @@ -XXX,XX +XXX,XX @@ static int qcow2_downgrade(BlockDriverState *bs, int target_version, return -ENOTSUP; } + if (has_data_file(bs)) { + error_setg(errp, "Cannot downgrade an image with a data file"); + return -ENOTSUP; + } + /* clear incompatible features */ if (s->incompatible_features & QCOW2_INCOMPAT_DIRTY) { ret = qcow2_mark_clean(bs); -- 2.20.1 Internal snapshots and an external data file are incompatible because snapshots require refcounting and non-linear mapping. Return an error for all of the snapshot operations if an external data file is in use. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2-snapshot.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-snapshot.c +++ b/block/qcow2-snapshot.c @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info) return -EFBIG; } + if (has_data_file(bs)) { + return -ENOTSUP; + } + memset(sn, 0, sizeof(*sn)); /* Generate an ID */ @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id) int ret; uint64_t *sn_l1_table = NULL; + if (has_data_file(bs)) { + return -ENOTSUP; + } + /* Search the snapshot */ snapshot_index = find_snapshot_by_id_or_name(bs, snapshot_id); if (snapshot_index < 0) { @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_delete(BlockDriverState *bs, QCowSnapshot sn; int snapshot_index, ret; + if (has_data_file(bs)) { + return -ENOTSUP; + } + /* Search the snapshot */ snapshot_index = find_snapshot_by_id_and_name(bs, snapshot_id, name); if (snapshot_index < 0) { @@ -XXX,XX +XXX,XX @@ int qcow2_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_tab) QCowSnapshot *sn; int i; + if (has_data_file(bs)) { + return -ENOTSUP; + } if (!s->nb_snapshots) { *psn_tab = NULL; return s->nb_snapshots; -- 2.20.1 For external data files, data clusters must be excluded from the refcount calculations. Instead, an implicit refcount of 1 is assumed for the COPIED flag. Compressed clusters and internal snapshots are incompatible with external data files, so print an error if they are in use for images with an external data file. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- block/qcow2-refcount.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-) diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-refcount.c +++ b/block/qcow2-refcount.c @@ -XXX,XX +XXX,XX @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, res->corruptions++; } + if (has_data_file(bs)) { + fprintf(stderr, "ERROR compressed cluster %d with data file, " + "entry=0x%" PRIx64 "\n", i, l2_entry); + res->corruptions++; + break; + } + /* Mark cluster as used */ nb_csectors = ((l2_entry >> s->csize_shift) & s->csize_mask) + 1; @@ -XXX,XX +XXX,XX @@ static int check_refcounts_l2(BlockDriverState *bs, BdrvCheckResult *res, } /* Mark cluster as used */ - ret = qcow2_inc_refcounts_imrt(bs, res, - refcount_table, refcount_table_size, - offset, s->cluster_size); - if (ret < 0) { - goto fail; + if (!has_data_file(bs)) { + ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table, + refcount_table_size, + offset, s->cluster_size); + if (ret < 0) { + goto fail; + } } break; } @@ -XXX,XX +XXX,XX @@ static int check_oflag_copied(BlockDriverState *bs, BdrvCheckResult *res, if (cluster_type == QCOW2_CLUSTER_NORMAL || cluster_type == QCOW2_CLUSTER_ZERO_ALLOC) { - ret = qcow2_get_refcount(bs, - data_offset >> s->cluster_bits, - &refcount); - if (ret < 0) { - /* don't print message nor increment check_errors */ - continue; + if (has_data_file(bs)) { + refcount = 1; + } else { + ret = qcow2_get_refcount(bs, + data_offset >> s->cluster_bits, + &refcount); + if (ret < 0) { + /* don't print message nor increment check_errors */ + continue; + } } if ((refcount == 1) != ((l2_entry & QCOW_OFLAG_COPIED) != 0)) { fprintf(stderr, "%s OFLAG_COPIED data cluster: " @@ -XXX,XX +XXX,XX @@ static int calculate_refcounts(BlockDriverState *bs, BdrvCheckResult *res, } /* snapshots */ + if (has_data_file(bs) && s->nb_snapshots) { + fprintf(stderr, "ERROR %d snapshots in image with data file\n", + s->nb_snapshots); + res->corruptions++; + } + for (i = 0; i < s->nb_snapshots; i++) { sn = s->snapshots + i; if (offset_into_cluster(s, sn->l1_table_offset)) { -- 2.20.1 This adds a .bdrv_open option to specify the external data file node. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qapi/block-core.json | 7 ++++++- block/qcow2.h | 4 +++- block/qcow2.c | 34 ++++++++++++++++++++++++++++++++-- 3 files changed, 41 insertions(+), 4 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ # encrypted images, except when doing a metadata-only # probe of the image. (since 2.10) # +# @data-file: reference to or definition of the external data file. +# This may only be specified for images that require an +# external data file. (since 4.0) +# # Since: 2.9 ## { 'struct': 'BlockdevOptionsQcow2', @@ -XXX,XX +XXX,XX @@ '*l2-cache-entry-size': 'int', '*refcount-cache-size': 'int', '*cache-clean-interval': 'int', - '*encrypt': 'BlockdevQcow2Encryption' } } + '*encrypt': 'BlockdevQcow2Encryption', + '*data-file': 'BlockdevRef' } } ## # @SshHostKeyCheckMode: diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ #define DEFAULT_CLUSTER_SIZE 65536 +#define QCOW2_OPT_DATA_FILE "data-file" #define QCOW2_OPT_LAZY_REFCOUNTS "lazy-refcounts" #define QCOW2_OPT_DISCARD_REQUEST "pass-discard-request" #define QCOW2_OPT_DISCARD_SNAPSHOT "pass-discard-snapshot" @@ -XXX,XX +XXX,XX @@ enum { QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR, QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY - | QCOW2_INCOMPAT_CORRUPT, + | QCOW2_INCOMPAT_CORRUPT + | QCOW2_INCOMPAT_DATA_FILE, }; /* Compatible feature bits */ diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, goto fail; } - /* TODO Open external data file */ - s->data_file = bs->file; + /* Open external data file */ + s->data_file = bdrv_open_child(NULL, options, "data-file", bs, &child_file, + true, &local_err); + if (local_err) { + error_propagate(errp, local_err); + ret = -EINVAL; + goto fail; + } + + if (s->incompatible_features & QCOW2_INCOMPAT_DATA_FILE) { + if (!s->data_file) { + error_setg(errp, "'data-file' is required for this image"); + ret = -EINVAL; + goto fail; + } + } else { + if (s->data_file) { + error_setg(errp, "'data-file' can only be set for images with an " + "external data file"); + ret = -EINVAL; + goto fail; + } else { + s->data_file = bs->file; + } + } /* qcow2_read_extension may have set up the crypto context * if the crypt method needs a header region, some methods @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, return ret; fail: + if (has_data_file(bs)) { + bdrv_unref_child(bs, s->data_file); + } g_free(s->unknown_header_fields); cleanup_unknown_header_ext(bs); qcow2_free_snapshots(bs); @@ -XXX,XX +XXX,XX @@ static void qcow2_close(BlockDriverState *bs) g_free(s->image_backing_file); g_free(s->image_backing_format); + if (has_data_file(bs)) { + bdrv_unref_child(bs, s->data_file); + } + qcow2_refcount_close(bs); qcow2_free_snapshots(bs); } -- 2.20.1 This adds a .bdrv_create option to use an external data file. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qapi/block-core.json | 4 ++++ include/block/block_int.h | 1 + block/qcow2.c | 26 ++++++++++++++++++++++++++ 3 files changed, 31 insertions(+) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ # Driver specific image creation options for qcow2. # # @file Node to create the image format on +# @data-file Node to use as an external data file in which all guest +# data is stored so that only metadata remains in the qcow2 +# file (since: 4.0) # @size Size of the virtual disk in bytes # @version Compatibility level (default: v3) # @backing-file File name of the backing file if a backing file @@ -XXX,XX +XXX,XX @@ ## { 'struct': 'BlockdevCreateOptionsQcow2', 'data': { 'file': 'BlockdevRef', + '*data-file': 'BlockdevRef', 'size': 'size', '*version': 'BlockdevQcow2Version', '*backing-file': 'str', diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ #define BLOCK_OPT_NOCOW "nocow" #define BLOCK_OPT_OBJECT_SIZE "object_size" #define BLOCK_OPT_REFCOUNT_BITS "refcount_bits" +#define BLOCK_OPT_DATA_FILE "data_file" #define BLOCK_PROBE_BUF_SIZE 512 diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) */ BlockBackend *blk = NULL; BlockDriverState *bs = NULL; + BlockDriverState *data_bs = NULL; QCowHeader *header; size_t cluster_size; int version; @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } refcount_order = ctz32(qcow2_opts->refcount_bits); + if (qcow2_opts->data_file) { + if (version < 3) { + error_setg(errp, "External data files are only supported with " + "compatibility level 1.1 and above (use version=v3 or " + "greater)"); + ret = -EINVAL; + goto out; + } + data_bs = bdrv_open_blockdev_ref(qcow2_opts->data_file, errp); + if (bs == NULL) { + ret = -EIO; + goto out; + } + } /* Create BlockBackend to write to the image */ blk = blk_new(BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL); @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) header->compatible_features |= cpu_to_be64(QCOW2_COMPAT_LAZY_REFCOUNTS); } + if (data_bs) { + header->incompatible_features |= + cpu_to_be64(QCOW2_INCOMPAT_DATA_FILE); + } ret = blk_pwrite(blk, 0, header, cluster_size, 0); g_free(header); @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) options = qdict_new(); qdict_put_str(options, "driver", "qcow2"); qdict_put_str(options, "file", bs->node_name); + if (data_bs) { + qdict_put_str(options, "data-file", data_bs->node_name); + } blk = blk_new_open(NULL, NULL, options, BDRV_O_RDWR | BDRV_O_RESIZE | BDRV_O_NO_FLUSH, &local_err); @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) options = qdict_new(); qdict_put_str(options, "driver", "qcow2"); qdict_put_str(options, "file", bs->node_name); + if (data_bs) { + qdict_put_str(options, "data-file", data_bs->node_name); + } blk = blk_new_open(NULL, NULL, options, BDRV_O_RDWR | BDRV_O_NO_BACKING | BDRV_O_NO_IO, &local_err); @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) out: blk_unref(blk); bdrv_unref(bs); + bdrv_unref(data_bs); return ret; } -- 2.20.1 Rather than requiring that the external data file node is passed explicitly when creating the qcow2 node, store the filename in the designated header extension during .bdrv_create and read it from there as a default during .bdrv_open. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qapi/block-core.json | 8 +++- block/qcow2.h | 1 + block/qcow2.c | 94 +++++++++++++++++++++++++++++++++++++- tests/qemu-iotests/082.out | 27 +++++++++++ 4 files changed, 128 insertions(+), 2 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ # # @compat: compatibility level # +# @data-file: the filename of the external data file that is stored in the +# image and used as a default for opening the image (since: 4.0) +# # @lazy-refcounts: on or off; only valid for compat >= 1.1 # # @corrupt: true if the image has been marked corrupt; only valid for @@ -XXX,XX +XXX,XX @@ { 'struct': 'ImageInfoSpecificQCow2', 'data': { 'compat': 'str', + '*data-file': 'str', '*lazy-refcounts': 'bool', '*corrupt': 'bool', 'refcount-bits': 'int', @@ -XXX,XX +XXX,XX @@ # # @data-file: reference to or definition of the external data file. # This may only be specified for images that require an -# external data file. (since 4.0) +# external data file. If it is not specified for such +# an image, the data file name is loaded from the image +# file. (since 4.0) # # Since: 2.9 ## diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ typedef struct BDRVQcow2State { * override) */ char *image_backing_file; char *image_backing_format; + char *image_data_file; CoQueue compress_wait_queue; int nb_compress_threads; diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset, #endif break; + case QCOW2_EXT_MAGIC_DATA_FILE: + { + s->image_data_file = g_malloc0(ext.len + 1); + ret = bdrv_pread(bs->file, offset, s->image_data_file, ext.len); + if (ret < 0) { + error_setg_errno(errp, -ret, + "ERROR: Could not read data file name"); + return ret; + } +#ifdef DEBUG_EXT + printf("Qcow2: Got external data file %s\n", s->image_data_file); +#endif + break; + } + default: /* unknown magic - save it in case we need to rewrite the header */ /* If you add a new feature, make sure to also update the fast @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, } if (s->incompatible_features & QCOW2_INCOMPAT_DATA_FILE) { + if (!s->data_file && s->image_data_file) { + s->data_file = bdrv_open_child(s->image_data_file, options, + "data-file", bs, &child_file, + false, errp); + if (!s->data_file) { + ret = -EINVAL; + goto fail; + } + } if (!s->data_file) { error_setg(errp, "'data-file' is required for this image"); ret = -EINVAL; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, return ret; fail: + g_free(s->image_data_file); if (has_data_file(bs)) { bdrv_unref_child(bs, s->data_file); } @@ -XXX,XX +XXX,XX @@ static void qcow2_close(BlockDriverState *bs) g_free(s->unknown_header_fields); cleanup_unknown_header_ext(bs); + g_free(s->image_data_file); g_free(s->image_backing_file); g_free(s->image_backing_format); @@ -XXX,XX +XXX,XX @@ int qcow2_update_header(BlockDriverState *bs) buflen -= ret; } + /* External data file header extension */ + if (has_data_file(bs) && s->image_data_file) { + ret = header_ext_add(buf, QCOW2_EXT_MAGIC_DATA_FILE, + s->image_data_file, strlen(s->image_data_file), + buflen); + if (ret < 0) { + goto fail; + } + + buf += ret; + buflen -= ret; + } + /* Full disk encryption header pointer extension */ if (s->crypto_header.offset != 0) { s->crypto_header.offset = cpu_to_be64(s->crypto_header.offset); @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) abort(); } + /* Set the external data file if necessary */ + if (data_bs) { + BDRVQcow2State *s = blk_bs(blk)->opaque; + s->image_data_file = g_strdup(data_bs->filename); + } + /* Create a full header (including things like feature table) */ ret = qcow2_update_header(blk_bs(blk)); if (ret < 0) { @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt QDict *qdict; Visitor *v; BlockDriverState *bs = NULL; + BlockDriverState *data_bs = NULL; Error *local_err = NULL; const char *val; int ret; @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt goto finish; } + /* Create and open an external data file (protocol layer) */ + val = qdict_get_try_str(qdict, BLOCK_OPT_DATA_FILE); + if (val) { + ret = bdrv_create_file(val, opts, errp); + if (ret < 0) { + goto finish; + } + + data_bs = bdrv_open(val, NULL, NULL, + BDRV_O_RDWR | BDRV_O_RESIZE | BDRV_O_PROTOCOL, + errp); + if (data_bs == NULL) { + ret = -EIO; + goto finish; + } + + qdict_del(qdict, BLOCK_OPT_DATA_FILE); + qdict_put_str(qdict, "data-file", data_bs->node_name); + } + /* Set 'driver' and 'node' options */ qdict_put_str(qdict, "driver", "qcow2"); qdict_put_str(qdict, "file", bs->node_name); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt finish: qobject_unref(qdict); bdrv_unref(bs); + bdrv_unref(data_bs); qapi_free_BlockdevCreateOptions(create_options); return ret; } @@ -XXX,XX +XXX,XX @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs, .refcount_bits = s->refcount_bits, .has_bitmaps = !!bitmaps, .bitmaps = bitmaps, + .has_data_file = !!s->image_data_file, + .data_file = g_strdup(s->image_data_file), }; } else { /* if this assertion fails, this probably means a new version was @@ -XXX,XX +XXX,XX @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, BDRVQcow2State *s = bs->opaque; int old_version = s->qcow_version, new_version = old_version; uint64_t new_size = 0; - const char *backing_file = NULL, *backing_format = NULL; + const char *backing_file = NULL, *backing_format = NULL, *data_file = NULL; bool lazy_refcounts = s->use_lazy_refcounts; const char *compat = NULL; uint64_t cluster_size = s->cluster_size; @@ -XXX,XX +XXX,XX @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, "may not exceed 64 bits"); return -EINVAL; } + } else if (!strcmp(desc->name, BLOCK_OPT_DATA_FILE)) { + data_file = qemu_opt_get(opts, BLOCK_OPT_DATA_FILE); + if (data_file && !has_data_file(bs)) { + error_setg(errp, "data-file can only be set for images that " + "use an external data file"); + return -EINVAL; + } } else { /* if this point is reached, this probably means a new option was * added without having it covered here */ @@ -XXX,XX +XXX,XX @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, } } + if (data_file) { + g_free(s->image_data_file); + s->image_data_file = *data_file ? g_strdup(data_file) : NULL; + } + + ret = qcow2_update_header(bs); + if (ret < 0) { + error_setg_errno(errp, -ret, "Failed to update the image header"); + return ret; + } + if (backing_file || backing_format) { ret = qcow2_change_backing_file(bs, backing_file ?: s->image_backing_file, @@ -XXX,XX +XXX,XX @@ static QemuOptsList qcow2_create_opts = { .type = QEMU_OPT_STRING, .help = "Image format of the base image" }, + { + .name = BLOCK_OPT_DATA_FILE, + .type = QEMU_OPT_STRING, + .help = "File name of an external data file" + }, { .name = BLOCK_OPT_ENCRYPT, .type = QEMU_OPT_BOOL, diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/082.out +++ b/tests/qemu-iotests/082.out @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': backing_fmt=<str> - Image format of the base image cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) + data_file=<str> - File name of an external data file encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' -- 2.20.1 Provide an option to force QEMU to always keep the external data file consistent as a standalone read-only raw image. At the moment, this means making sure that write_zeroes requests are forwarded to the data file instead of just updating the metadata, and checking that no backing file is used. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- qapi/block-core.json | 9 ++++++ block/qcow2.h | 9 +++++- include/block/block_int.h | 1 + block/qcow2-cluster.c | 10 +++++++ block/qcow2.c | 56 ++++++++++++++++++++++++++++++++++++-- tests/qemu-iotests/082.out | 27 ++++++++++++++++++ 6 files changed, 109 insertions(+), 3 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index XXXXXXX..XXXXXXX 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -XXX,XX +XXX,XX @@ # @data-file: the filename of the external data file that is stored in the # image and used as a default for opening the image (since: 4.0) # +# @data-file-raw: True if the external data file must stay valid as a +# standalone (read-only) raw image without looking at qcow2 +# metadata (since: 4.0) +# # @lazy-refcounts: on or off; only valid for compat >= 1.1 # # @corrupt: true if the image has been marked corrupt; only valid for @@ -XXX,XX +XXX,XX @@ 'data': { 'compat': 'str', '*data-file': 'str', + '*data-file-raw': 'bool', '*lazy-refcounts': 'bool', '*corrupt': 'bool', 'refcount-bits': 'int', @@ -XXX,XX +XXX,XX @@ # @data-file Node to use as an external data file in which all guest # data is stored so that only metadata remains in the qcow2 # file (since: 4.0) +# @data-file-raw True if the external data file must stay valid as a +# standalone (read-only) raw image without looking at qcow2 +# metadata (default: false; since: 4.0) # @size Size of the virtual disk in bytes # @version Compatibility level (default: v3) # @backing-file File name of the backing file if a backing file @@ -XXX,XX +XXX,XX @@ { 'struct': 'BlockdevCreateOptionsQcow2', 'data': { 'file': 'BlockdevRef', '*data-file': 'BlockdevRef', + '*data-file-raw': 'bool', 'size': 'size', '*version': 'BlockdevQcow2Version', '*backing-file': 'str', diff --git a/block/qcow2.h b/block/qcow2.h index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -XXX,XX +XXX,XX @@ enum { QCOW2_AUTOCLEAR_BITMAPS = 1 << QCOW2_AUTOCLEAR_BITMAPS_BITNR, QCOW2_AUTOCLEAR_DATA_FILE_RAW = 1 << QCOW2_AUTOCLEAR_DATA_FILE_RAW_BITNR, - QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS, + QCOW2_AUTOCLEAR_MASK = QCOW2_AUTOCLEAR_BITMAPS + | QCOW2_AUTOCLEAR_DATA_FILE_RAW, }; enum qcow2_discard_type { @@ -XXX,XX +XXX,XX @@ static inline bool has_data_file(BlockDriverState *bs) return (s->data_file != bs->file); } +static inline bool data_file_is_raw(BlockDriverState *bs) +{ + BDRVQcow2State *s = bs->opaque; + return !!(s->autoclear_features & QCOW2_AUTOCLEAR_DATA_FILE_RAW); +} + static inline int64_t start_of_cluster(BDRVQcow2State *s, int64_t offset) { return offset & ~(s->cluster_size - 1); diff --git a/include/block/block_int.h b/include/block/block_int.h index XXXXXXX..XXXXXXX 100644 --- a/include/block/block_int.h +++ b/include/block/block_int.h @@ -XXX,XX +XXX,XX @@ #define BLOCK_OPT_OBJECT_SIZE "object_size" #define BLOCK_OPT_REFCOUNT_BITS "refcount_bits" #define BLOCK_OPT_DATA_FILE "data_file" +#define BLOCK_OPT_DATA_FILE_RAW "data_file_raw" #define BLOCK_PROBE_BUF_SIZE 512 diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2-cluster.c +++ b/block/qcow2-cluster.c @@ -XXX,XX +XXX,XX @@ int qcow2_cluster_zeroize(BlockDriverState *bs, uint64_t offset, int64_t cleared; int ret; + /* If we have to stay in sync with an external data file, zero out + * s->data_file first. */ + if (data_file_is_raw(bs)) { + assert(has_data_file(bs)); + ret = bdrv_co_pwrite_zeroes(s->data_file, offset, bytes, flags); + if (ret < 0) { + return ret; + } + } + /* Caller must pass aligned values, except at image end */ assert(QEMU_IS_ALIGNED(offset, s->cluster_size)); assert(QEMU_IS_ALIGNED(end_offset, s->cluster_size) || diff --git a/block/qcow2.c b/block/qcow2.c index XXXXXXX..XXXXXXX 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, "external data file"); ret = -EINVAL; goto fail; - } else { - s->data_file = bs->file; + } + + s->data_file = bs->file; + + if (data_file_is_raw(bs)) { + error_setg(errp, "data-file-raw requires a data file"); + ret = -EINVAL; + goto fail; } } @@ -XXX,XX +XXX,XX @@ static int qcow2_change_backing_file(BlockDriverState *bs, { BDRVQcow2State *s = bs->opaque; + /* Adding a backing file means that the external data file alone won't be + * enough to make sense of the content */ + if (backing_file && data_file_is_raw(bs)) { + return -EINVAL; + } + if (backing_file && strlen(backing_file) > 1023) { return -EINVAL; } @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } refcount_order = ctz32(qcow2_opts->refcount_bits); + if (qcow2_opts->data_file_raw && !qcow2_opts->data_file) { + error_setg(errp, "data-file-raw requires data-file"); + ret = -EINVAL; + goto out; + } + if (qcow2_opts->data_file_raw && qcow2_opts->has_backing_file) { + error_setg(errp, "Backing file and data-file-raw cannot be used at " + "the same time"); + ret = -EINVAL; + goto out; + } + if (qcow2_opts->data_file) { if (version < 3) { error_setg(errp, "External data files are only supported with " @@ -XXX,XX +XXX,XX @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) header->incompatible_features |= cpu_to_be64(QCOW2_INCOMPAT_DATA_FILE); } + if (qcow2_opts->data_file_raw) { + header->autoclear_features |= + cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW); + } ret = blk_pwrite(blk, 0, header, cluster_size, 0); g_free(header); @@ -XXX,XX +XXX,XX @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt { BLOCK_OPT_REFCOUNT_BITS, "refcount-bits" }, { BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT }, { BLOCK_OPT_COMPAT_LEVEL, "version" }, + { BLOCK_OPT_DATA_FILE_RAW, "data-file-raw" }, { NULL, NULL }, }; @@ -XXX,XX +XXX,XX @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs, .bitmaps = bitmaps, .has_data_file = !!s->image_data_file, .data_file = g_strdup(s->image_data_file), + .has_data_file_raw = has_data_file(bs), + .data_file_raw = data_file_is_raw(bs), }; } else { /* if this assertion fails, this probably means a new version was @@ -XXX,XX +XXX,XX @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, uint64_t new_size = 0; const char *backing_file = NULL, *backing_format = NULL, *data_file = NULL; bool lazy_refcounts = s->use_lazy_refcounts; + bool data_file_raw = data_file_is_raw(bs); const char *compat = NULL; uint64_t cluster_size = s->cluster_size; bool encrypt; @@ -XXX,XX +XXX,XX @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, "use an external data file"); return -EINVAL; } + } else if (!strcmp(desc->name, BLOCK_OPT_DATA_FILE_RAW)) { + data_file_raw = qemu_opt_get_bool(opts, BLOCK_OPT_DATA_FILE_RAW, + data_file_raw); + if (data_file_raw && !data_file_is_raw(bs)) { + error_setg(errp, "data-file-raw cannot be set on existing " + "images"); + return -EINVAL; + } } else { /* if this point is reached, this probably means a new option was * added without having it covered here */ @@ -XXX,XX +XXX,XX @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, } } + /* data-file-raw blocks backing files, so clear it first if requested */ + if (data_file_raw) { + s->autoclear_features |= QCOW2_AUTOCLEAR_DATA_FILE_RAW; + } else { + s->autoclear_features &= ~QCOW2_AUTOCLEAR_DATA_FILE_RAW; + } + if (data_file) { g_free(s->image_data_file); s->image_data_file = *data_file ? g_strdup(data_file) : NULL; @@ -XXX,XX +XXX,XX @@ static QemuOptsList qcow2_create_opts = { .type = QEMU_OPT_STRING, .help = "File name of an external data file" }, + { + .name = BLOCK_OPT_DATA_FILE_RAW, + .type = QEMU_OPT_BOOL, + .help = "The external data file must stay valid as a raw image" + }, { .name = BLOCK_OPT_ENCRYPT, .type = QEMU_OPT_BOOL, diff --git a/tests/qemu-iotests/082.out b/tests/qemu-iotests/082.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/082.out +++ b/tests/qemu-iotests/082.out @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Supported options: cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' @@ -XXX,XX +XXX,XX @@ Creation options for 'qcow2': cluster_size=<size> - qcow2 cluster size compat=<str> - Compatibility level (0.10 or 1.1) data_file=<str> - File name of an external data file + data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image encrypt.cipher-alg=<str> - Name of encryption cipher algorithm encrypt.cipher-mode=<str> - Name of encryption cipher mode encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks' -- 2.20.1 Test that preallocating metadata results in a somewhat larger qcow2 file, but preallocating data only affects the disk usage of the data file and the qcow2 file stays small. Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/qemu-iotests/243 | 22 ++++++++++++++++++++++ tests/qemu-iotests/243.out | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/tests/qemu-iotests/243 b/tests/qemu-iotests/243 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/243 +++ b/tests/qemu-iotests/243 @@ -XXX,XX +XXX,XX @@ status=1 # failure is the default! _cleanup() { _cleanup_test_img + rm -f $TEST_IMG.data } trap "_cleanup; exit \$status" 0 1 2 3 15 @@ -XXX,XX +XXX,XX @@ for mode in off metadata falloc full; do done +for mode in off metadata falloc full; do + + echo + echo "=== External data file: preallocation=$mode ===" + echo + + IMGOPTS="data_file=$TEST_IMG.data,preallocation=$mode" _make_test_img 64M + + echo -n "qcow2 file size: " + du -b $TEST_IMG | cut -f1 + echo -n "data file size: " + du -b $TEST_IMG.data | cut -f1 + + # Can't use precise numbers here because they differ between filesystems + echo -n "qcow2 disk usage: " + [ $(du -B1 $TEST_IMG | cut -f1) -lt 1048576 ] && echo "low" || echo "high" + echo -n "data disk usage: " + [ $(du -B1 $TEST_IMG.data | cut -f1) -lt 1048576 ] && echo "low" || echo "high" + +done + # success, all done echo "*** done" rm -f $seq.full diff --git a/tests/qemu-iotests/243.out b/tests/qemu-iotests/243.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/243.out +++ b/tests/qemu-iotests/243.out @@ -XXX,XX +XXX,XX @@ Disk usage: high Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 preallocation=full File size: 67436544 Disk usage: high + +=== External data file: preallocation=off === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=off +qcow2 file size: 196616 +data file size: 67108864 +qcow2 disk usage: low +data disk usage: low + +=== External data file: preallocation=metadata === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=metadata +qcow2 file size: 327680 +data file size: 67108864 +qcow2 disk usage: low +data disk usage: low + +=== External data file: preallocation=falloc === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=falloc +qcow2 file size: 327680 +data file size: 67108864 +qcow2 disk usage: low +data disk usage: high + +=== External data file: preallocation=full === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data preallocation=full +qcow2 file size: 327680 +data file size: 67108864 +qcow2 disk usage: low +data disk usage: high *** done -- 2.20.1 Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/qemu-iotests/244 | 200 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/244.out | 125 +++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 326 insertions(+) create mode 100755 tests/qemu-iotests/244 create mode 100644 tests/qemu-iotests/244.out diff --git a/tests/qemu-iotests/244 b/tests/qemu-iotests/244 new file mode 100755 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/244 @@ -XXX,XX +XXX,XX @@ +#!/bin/bash +# +# Test qcow2 with external data files +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator +owner=kwolf@redhat.com + +seq=$(basename $0) +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img + rm -f $TEST_IMG.data + rm -f $TEST_IMG.src +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +echo +echo "=== Create and open image with external data file ===" +echo + +echo "With data file name in the image:" +IMGOPTS="data_file=$TEST_IMG.data" _make_test_img 64M +_check_test_img + +$QEMU_IO -c "open $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "open -odata-file.filename=$TEST_IMG.data $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "open -odata-file.filename=inexistent $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir + +echo +echo "Data file required, but without data file name in the image:" +$QEMU_IMG amend -odata_file= $TEST_IMG + +$QEMU_IO -c "open $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "open -odata-file.filename=$TEST_IMG.data $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "open -odata-file.filename=inexistent $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir + +echo +echo "Setting data-file for an image with internal data:" +_make_test_img 64M + +$QEMU_IO -c "open -odata-file.filename=$TEST_IMG.data $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "open -odata-file.filename=inexistent $TEST_IMG" -c "read -P 0 0 64k" 2>&1 | _filter_qemu_io | _filter_testdir + +echo +echo "=== Conflicting features ===" +echo + +echo "Convert to compressed target with data file:" +TEST_IMG="$TEST_IMG.src" _make_test_img 64M + +$QEMU_IO -c 'write -P 0x11 0 1M' \ + -f $IMGFMT "$TEST_IMG.src" | + _filter_qemu_io + +$QEMU_IMG convert -f $IMGFMT -O $IMGFMT -c -odata_file="$TEST_IMG.data" \ + "$TEST_IMG.src" "$TEST_IMG" + +echo +echo "Convert uncompressed, then write compressed data manually:" +$QEMU_IMG convert -f $IMGFMT -O $IMGFMT -odata_file="$TEST_IMG.data" \ + "$TEST_IMG.src" "$TEST_IMG" +$QEMU_IMG compare "$TEST_IMG.src" "$TEST_IMG" + +$QEMU_IO -c 'write -c -P 0x22 0 1M' \ + -f $IMGFMT "$TEST_IMG" | + _filter_qemu_io +_check_test_img + +echo +echo "Take an internal snapshot:" + +$QEMU_IMG snapshot -c test "$TEST_IMG" +_check_test_img + +echo +echo "=== Standalone image with external data file (efficient) ===" +echo + +IMGOPTS="data_file=$TEST_IMG.data" _make_test_img 64M + +echo -n "qcow2 file size before I/O: " +du -b $TEST_IMG | cut -f1 + +# Create image with the following layout +# 0-1 MB: Unallocated +# 1-2 MB: Written (pattern 0x11) +# 2-3 MB: Discarded +# 3-4 MB: Zero write over discarded space +# 4-5 MB: Zero write over written space +# 5-6 MB: Zero write over unallocated space + +echo +$QEMU_IO -c 'write -P 0x11 1M 4M' \ + -c 'discard 2M 2M' \ + -c 'write -z 3M 3M' \ + -f $IMGFMT "$TEST_IMG" | + _filter_qemu_io +_check_test_img + +echo +$QEMU_IMG map --output=json "$TEST_IMG" + +echo +$QEMU_IO -c 'read -P 0 0 1M' \ + -c 'read -P 0x11 1M 1M' \ + -c 'read -P 0 2M 4M' \ + -f $IMGFMT "$TEST_IMG" | + _filter_qemu_io + +# Zero clusters are only marked as such in the qcow2 metadata, but contain +# stale data in the external data file +echo +$QEMU_IO -c 'read -P 0 0 1M' \ + -c 'read -P 0x11 1M 1M' \ + -c 'read -P 0 2M 2M' \ + -c 'read -P 0x11 4M 1M' \ + -c 'read -P 0 5M 1M' \ + -f raw "$TEST_IMG.data" | + _filter_qemu_io + + +echo -n "qcow2 file size after I/O: " +du -b $TEST_IMG | cut -f1 + +echo +echo "=== Standalone image with external data file (valid raw) ===" +echo + +IMGOPTS="data_file=$TEST_IMG.data,data_file_raw=on" _make_test_img 64M + +echo -n "qcow2 file size before I/O: " +du -b $TEST_IMG | cut -f1 + +echo +$QEMU_IO -c 'write -P 0x11 1M 4M' \ + -c 'discard 2M 2M' \ + -c 'write -z 3M 3M' \ + -f $IMGFMT "$TEST_IMG" | + _filter_qemu_io +_check_test_img + +echo +$QEMU_IMG map --output=json "$TEST_IMG" + +echo +$QEMU_IO -c 'read -P 0 0 1M' \ + -c 'read -P 0x11 1M 1M' \ + -c 'read -P 0 2M 4M' \ + -f $IMGFMT "$TEST_IMG" | + _filter_qemu_io + +echo +$QEMU_IMG compare "$TEST_IMG" "$TEST_IMG.data" + +echo -n "qcow2 file size after I/O: " +du -b $TEST_IMG | cut -f1 + +echo +echo "=== bdrv_co_block_status test for file and offset=0 ===" +echo + +IMGOPTS="data_file=$TEST_IMG.data" _make_test_img 64M + +$QEMU_IO -c 'write -P 0x11 0 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c 'read -P 0x11 0 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG map --output=human "$TEST_IMG" | _filter_testdir +$QEMU_IMG map --output=json "$TEST_IMG" + +# success, all done +echo "*** done" +rm -f $seq.full +status=0 diff --git a/tests/qemu-iotests/244.out b/tests/qemu-iotests/244.out new file mode 100644 index XXXXXXX..XXXXXXX --- /dev/null +++ b/tests/qemu-iotests/244.out @@ -XXX,XX +XXX,XX @@ +QA output created by 244 + +=== Create and open image with external data file === + +With data file name in the image: +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data +No errors were found on the image. +read 65536/65536 bytes at offset 0 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 65536/65536 bytes at offset 0 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +can't open device TEST_DIR/t.qcow2: Could not open 'inexistent': No such file or directory +no file open, try 'help open' + +Data file required, but without data file name in the image: +can't open device TEST_DIR/t.qcow2: 'data-file' is required for this image +no file open, try 'help open' +read 65536/65536 bytes at offset 0 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +can't open device TEST_DIR/t.qcow2: Could not open 'inexistent': No such file or directory +no file open, try 'help open' + +Setting data-file for an image with internal data: +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +can't open device TEST_DIR/t.qcow2: 'data-file' can only be set for images with an external data file +no file open, try 'help open' +can't open device TEST_DIR/t.qcow2: Could not open 'inexistent': No such file or directory +no file open, try 'help open' + +=== Conflicting features === + +Convert to compressed target with data file: +Formatting 'TEST_DIR/t.IMGFMT.src', fmt=IMGFMT size=67108864 +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +qemu-img: error while writing sector 0: Operation not supported + +Convert uncompressed, then write compressed data manually: +Images are identical. +write failed: Operation not supported +No errors were found on the image. + +Take an internal snapshot: +qemu-img: Could not create snapshot 'test': -95 (Operation not supported) +No errors were found on the image. + +=== Standalone image with external data file (efficient) === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data +qcow2 file size before I/O: 196616 + +wrote 4194304/4194304 bytes at offset 1048576 +4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +discard 2097152/2097152 bytes at offset 2097152 +2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 3145728/3145728 bytes at offset 3145728 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +No errors were found on the image. + +[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false}, +{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576}, +{ "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false}, +{ "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304}, +{ "start": 5242880, "length": 61865984, "depth": 0, "zero": true, "data": false}] + +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 4194304/4194304 bytes at offset 2097152 +4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 2097152/2097152 bytes at offset 2097152 +2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 1048576/1048576 bytes at offset 4194304 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 1048576/1048576 bytes at offset 5242880 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +qcow2 file size after I/O: 327680 + +=== Standalone image with external data file (valid raw) === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data data_file_raw=on +qcow2 file size before I/O: 196616 + +wrote 4194304/4194304 bytes at offset 1048576 +4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +discard 2097152/2097152 bytes at offset 2097152 +2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 3145728/3145728 bytes at offset 3145728 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +No errors were found on the image. + +[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false}, +{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 1048576}, +{ "start": 2097152, "length": 2097152, "depth": 0, "zero": true, "data": false}, +{ "start": 4194304, "length": 1048576, "depth": 0, "zero": true, "data": false, "offset": 4194304}, +{ "start": 5242880, "length": 61865984, "depth": 0, "zero": true, "data": false}] + +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 4194304/4194304 bytes at offset 2097152 +4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +Images are identical. +qcow2 file size after I/O: 327680 + +=== bdrv_co_block_status test for file and offset=0 === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Offset Length Mapped to File +0 0x100000 0 TEST_DIR/t.qcow2.data +[{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true, "offset": 0}, +{ "start": 1048576, "length": 66060288, "depth": 0, "zero": true, "data": false}] +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -XXX,XX +XXX,XX @@ 240 auto quick 242 rw auto quick 243 rw auto quick +244 rw auto quick -- 2.20.1 Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/qemu-iotests/061 | 45 ++++++++++++++++++- tests/qemu-iotests/061.out | 89 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 133 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/061 b/tests/qemu-iotests/061 index XXXXXXX..XXXXXXX 100755 --- a/tests/qemu-iotests/061 +++ b/tests/qemu-iotests/061 @@ -XXX,XX +XXX,XX @@ status=1 # failure is the default! _cleanup() { - _cleanup_test_img + _cleanup_test_img + rm -f $TEST_IMG.data } trap "_cleanup; exit \$status" 0 1 2 3 15 @@ -XXX,XX +XXX,XX @@ $QEMU_IMG snapshot -c foo "$TEST_IMG" $QEMU_IMG amend -p -o "compat=0.10" "$TEST_IMG" _check_test_img +echo +echo "=== Testing version downgrade with external data file ===" +echo +IMGOPTS="compat=1.1,data_file=$TEST_IMG.data" _make_test_img 64M +$QEMU_IMG amend -o "compat=0.10" "$TEST_IMG" +_img_info --format-specific +_check_test_img + +echo +echo "=== Try changing the external data file ===" +echo +IMGOPTS="compat=1.1" _make_test_img 64M +$QEMU_IMG amend -o "data_file=foo" "$TEST_IMG" + +echo +IMGOPTS="compat=1.1,data_file=$TEST_IMG.data" _make_test_img 64M +$QEMU_IMG amend -o "data_file=foo" "$TEST_IMG" +_img_info --format-specific +TEST_IMG="data-file.filename=$TEST_IMG.data,file.filename=$TEST_IMG" _img_info --format-specific --image-opts + +echo +$QEMU_IMG amend -o "data_file=" --image-opts "data-file.filename=$TEST_IMG.data,file.filename=$TEST_IMG" +_img_info --format-specific +TEST_IMG="data-file.filename=$TEST_IMG.data,file.filename=$TEST_IMG" _img_info --format-specific --image-opts + +echo +echo "=== Clearing and setting data-file-raw ===" +echo +IMGOPTS="compat=1.1,data_file=$TEST_IMG.data,data_file_raw=on" _make_test_img 64M +$QEMU_IMG amend -o "data_file_raw=on" "$TEST_IMG" +_img_info --format-specific +_check_test_img + +$QEMU_IMG amend -o "data_file_raw=off" "$TEST_IMG" +_img_info --format-specific +_check_test_img + +$QEMU_IMG amend -o "data_file_raw=on" "$TEST_IMG" +_img_info --format-specific +_check_test_img + + # success, all done echo "*** done" rm -f $seq.full diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out index XXXXXXX..XXXXXXX 100644 --- a/tests/qemu-iotests/061.out +++ b/tests/qemu-iotests/061.out @@ -XXX,XX +XXX,XX @@ wrote 65536/65536 bytes at offset 3221225472 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) (0.00/100%) (6.25/100%) (12.50/100%) (18.75/100%) (25.00/100%) (31.25/100%) (37.50/100%) (43.75/100%) (50.00/100%) (56.25/100%) (62.50/100%) (68.75/100%) (75.00/100%) (81.25/100%) (87.50/100%) (93.75/100%) (100.00/100%) (100.00/100%) No errors were found on the image. + +=== Testing version downgrade with external data file === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data +qemu-img: Cannot downgrade an image with a data file +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 64M (67108864 bytes) +cluster_size: 65536 +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + data file: TEST_DIR/t.IMGFMT.data + data file raw: false + corrupt: false +No errors were found on the image. + +=== Try changing the external data file === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +qemu-img: data-file can only be set for images that use an external data file + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data +qemu-img: Could not open 'TEST_DIR/t.IMGFMT': Could not open 'foo': No such file or directory +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 64M (67108864 bytes) +cluster_size: 65536 +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + data file: foo + data file raw: false + corrupt: false + +qemu-img: Could not open 'TEST_DIR/t.IMGFMT': 'data-file' is required for this image +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 64M (67108864 bytes) +cluster_size: 65536 +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + data file raw: false + corrupt: false + +=== Clearing and setting data-file-raw === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 data_file=TEST_DIR/t.IMGFMT.data data_file_raw=on +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 64M (67108864 bytes) +cluster_size: 65536 +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + data file: TEST_DIR/t.IMGFMT.data + data file raw: true + corrupt: false +No errors were found on the image. +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 64M (67108864 bytes) +cluster_size: 65536 +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + data file: TEST_DIR/t.IMGFMT.data + data file raw: false + corrupt: false +No errors were found on the image. +qemu-img: data-file-raw cannot be set on existing images +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 64M (67108864 bytes) +cluster_size: 65536 +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + data file: TEST_DIR/t.IMGFMT.data + data file raw: false + corrupt: false +No errors were found on the image. *** done -- 2.20.1 From: Philippe Mathieu-Daudé <philmd@redhat.com> Since the ahci-test uses qemu-img, add a dependency to build it before using it. This fixes: $ gmake check-qtest V=1 QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/ahci-test Failed to execute child process "/tmp/qemu-test.19tMRF/qemu-img" (No such file or directory) ERROR:tests/libqos/libqos.c:192:mkimg: assertion failed: (ret && !err) Reviewed-by: John Snow <jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/Makefile.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/Makefile.include b/tests/Makefile.include index XXXXXXX..XXXXXXX 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -XXX,XX +XXX,XX @@ tests/prom-env-test$(EXESUF): tests/prom-env-test.o $(libqos-obj-y) tests/rtas-test$(EXESUF): tests/rtas-test.o $(libqos-spapr-obj-y) tests/fdc-test$(EXESUF): tests/fdc-test.o tests/ide-test$(EXESUF): tests/ide-test.o $(libqos-pc-obj-y) -tests/ahci-test$(EXESUF): tests/ahci-test.o $(libqos-pc-obj-y) +tests/ahci-test$(EXESUF): tests/ahci-test.o $(libqos-pc-obj-y) qemu-img$(EXESUF) tests/ipmi-kcs-test$(EXESUF): tests/ipmi-kcs-test.o tests/ipmi-bt-test$(EXESUF): tests/ipmi-bt-test.o tests/hd-geo-test$(EXESUF): tests/hd-geo-test.o -- 2.20.1 From: Philippe Mathieu-Daudé <philmd@redhat.com> Since a9660664fde, some iotests use qemu-nbd. Add a dependency to build it before using it. This fixes: $ make check-block GEN qemu-img-cmds.h CC qemu-img.o LINK qemu-img CC qemu-io.o LINK qemu-io CC tests/qemu-iotests/socket_scm_helper.o LINK tests/qemu-iotests/socket_scm_helper tests/qemu-iotests-quick.sh check: qemu-nbd not found make: *** [tests/Makefile.include:1059: check-tests/qemu-iotests-quick.sh] Error 1 Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> --- tests/Makefile.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/Makefile.include b/tests/Makefile.include index XXXXXXX..XXXXXXX 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -XXX,XX +XXX,XX @@ clean-tcg: $(CLEAN_TCG_TARGET_RULES) QEMU_IOTESTS_HELPERS-$(call land,$(CONFIG_SOFTMMU),$(CONFIG_LINUX)) = tests/qemu-iotests/socket_scm_helper$(EXESUF) .PHONY: check-tests/qemu-iotests-quick.sh -check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) qemu-io$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) +check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) qemu-io$(EXESUF) qemu-nbd$(EXESUF) $(QEMU_IOTESTS_HELPERS-y) $< .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y)) -- 2.20.1 Be more specific about the string representation in header extensions. Suggested-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> --- docs/interop/qcow2.txt | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index XXXXXXX..XXXXXXX 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -XXX,XX +XXX,XX @@ be stored. Each extension has a structure like the following: Byte 0 - 3: Header extension type: 0x00000000 - End of the header extension area - 0xE2792ACA - Backing file format name + 0xE2792ACA - Backing file format name string 0x6803f857 - Feature name table 0x23852875 - Bitmaps extension 0x0537be77 - Full disk encryption header pointer - 0x44415441 - External data file name + 0x44415441 - External data file name string other - Unknown header extension, can be safely ignored @@ -XXX,XX +XXX,XX @@ data of compatible features that it doesn't support. Compatible features that need space for additional data can use a header extension. +== String header extensions == + +Some header extensions (such as the backing file format name and the external +data file name) are just a single string. In this case, the header extension +length is the string length and the string is not '\0' terminated. (The header +extension padding can make it look like a string is '\0' terminated, but +neither is padding always necessary nor is there a guarantee that zero bytes +are used for padding.) + + == Feature name table == The feature name table is an optional header extension that contains the name -- 2.20.1