The following changes since commit 0b6206b9c6825619cd721085fe082d7a0abc9af4:

Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210914-4' into staging (2021-09-15 13:27:49 +0100)

https://github.com/XanClic/qemu.git tags/pull-block-2021-09-15

for you to fetch changes up to 1899bf47375ad40555dcdff12ba49b4b8b82df38:

qemu-img: Add -F shorthand to convert (2021-09-15 18:42:38 +0200)

- Block-status cache for data regions

- qcow2 optimization (when using subclusters)

- iotests delinting, and let 297 (lint checker) cover named iotests

- qcow2 check improvements

- Added -F (target backing file format) option to qemu-img convert

- Mirror job fix

- Fix for when a migration is initiated while a backup job runs

- Fix for uncached qemu-img convert to a volume with 4k sectors (for an

unaligned image)

- Minor gluster driver fix

Eric Blake (1):

qemu-img: Add -F shorthand to convert

Hanna Reitz (15):

gluster: Align block-status tail

block: Drop BDS comment regarding bdrv_append()

block: block-status cache for data regions

block: Clarify that @bytes is no limit on *pnum

block/file-posix: Do not force-cap *pnum

block/gluster: Do not force-cap *pnum

block/iscsi: Do not force-cap *pnum

iotests: Fix unspecified-encoding pylint warnings

iotests: Fix use-{list,dict}-literal warnings

iotests/297: Drop 169 and 199 from the skip list

migrate-bitmaps-postcopy-test: Fix pylint warnings

migrate-bitmaps-test: Fix pylint warnings

mirror-top-perms: Fix AbnormalShutdown path

iotests/297: Cover tests/

qemu-img: Allow target be aligned to sector size

Stefano Garzarella (1):

block/mirror: fix NULL pointer dereference in

mirror_wait_on_conflicts()

Vladimir Sementsov-Ogievskiy (15):

tests: add migrate-during-backup

block: bdrv_inactivate_recurse(): check for permissions and fix crash

simplebench: add img_bench_templater.py

qcow2: refactor handle_dependencies() loop body

qcow2: handle_dependencies(): relax conflict detection

qcow2-refcount: improve style of check_refcounts_l2()

qcow2: compressed read: simplify cluster descriptor passing

qcow2: introduce qcow2_parse_compressed_l2_entry() helper

qcow2-refcount: introduce fix_l2_entry_by_zero()

qcow2-refcount: fix_l2_entry_by_zero(): also zero L2 entry bitmap

qcow2-refcount: check_refcounts_l2(): check l2_bitmap

qcow2-refcount: check_refcounts_l2(): check reserved bits

qcow2-refcount: improve style of check_refcounts_l1()

qcow2-refcount: check_refcounts_l1(): check reserved bits

qcow2-refcount: check_refblocks(): add separate message for reserved

docs/tools/qemu-img.rst | 4 +-

block/qcow2.h | 7 +-

include/block/block_int.h | 61 +++-

block.c | 88 +++++

block/file-posix.c | 7 +-

block/gluster.c | 23 +-

block/io.c | 68 +++-

block/iscsi.c | 3 -

block/mirror.c | 25 +-

block/qcow2-cluster.c | 78 +++--

block/qcow2-refcount.c | 326 ++++++++++++------

block/qcow2.c | 13 +-

qemu-img.c | 18 +-

qemu-img-cmds.hx | 2 +-

scripts/simplebench/img_bench_templater.py | 95 +++++

scripts/simplebench/table_templater.py | 62 ++++

tests/qemu-iotests/122 | 2 +-

tests/qemu-iotests/271 | 5 +-

tests/qemu-iotests/271.out | 4 +-

tests/qemu-iotests/297 | 9 +-

tests/qemu-iotests/iotests.py | 12 +-

.../tests/migrate-bitmaps-postcopy-test | 13 +-

tests/qemu-iotests/tests/migrate-bitmaps-test | 43 ++-

.../qemu-iotests/tests/migrate-during-backup | 97 ++++++

.../tests/migrate-during-backup.out | 5 +

tests/qemu-iotests/tests/mirror-top-perms | 2 +-

26 files changed, 855 insertions(+), 217 deletions(-)

create mode 100755 scripts/simplebench/img_bench_templater.py

create mode 100644 scripts/simplebench/table_templater.py

create mode 100755 tests/qemu-iotests/tests/migrate-during-backup

create mode 100644 tests/qemu-iotests/tests/migrate-during-backup.out

2.31.1

From: Eric Blake <eblake@redhat.com>

Although we have long supported 'qemu-img convert -o

backing_file=foo,backing_fmt=bar', the fact that we have a shortcut -B

for backing_file but none for backing_fmt has made it more likely that

users accidentally run into:

qemu-img: warning: Deprecated use of backing file without explicit backing format

when using -B instead of -o. For similarity with other qemu-img

commands, such as create and compare, add '-F $fmt' as the shorthand

for '-o backing_fmt=$fmt'. Update iotest 122 for coverage of both

spellings.

Signed-off-by: Eric Blake <eblake@redhat.com>

Message-Id: <20210913131735.1948339-1-eblake@redhat.com>

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>

Signed-off-by: Hanna Reitz <hreitz@redhat.com>

---

docs/tools/qemu-img.rst | 4 ++--

qemu-img.c | 10 +++++++---

qemu-img-cmds.hx | 2 +-

tests/qemu-iotests/122 | 2 +-

4 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst

--- a/docs/tools/qemu-img.rst

+++ b/docs/tools/qemu-img.rst

@@ -XXX,XX +XXX,XX @@ Command description:

4

Error on reading data

-.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps [--skip-broken-bitmaps]] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-r RATE_LIMIT] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME

+.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps [--skip-broken-bitmaps]] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE [-F backing_fmt]] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-r RATE_LIMIT] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME

Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM*

to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can

@@ -XXX,XX +XXX,XX @@ Command description:

You can use the *BACKING_FILE* option to force the output image to be

created as a copy on write image of the specified base image; the

*BACKING_FILE* should have the same content as the input's base image,

- however the path, image format, etc may differ.

+ however the path, image format (as given by *BACKING_FMT*), etc may differ.

If a relative path name is given, the backing file is looked up relative to

the directory containing *OUTPUT_FILENAME*.

diff --git a/qemu-img.c b/qemu-img.c

index XXXXXXX..XXXXXXX 100644

--- a/qemu-img.c

+++ b/qemu-img.c

@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)

int c, bs_i, flags, src_flags = BDRV_O_NO_SHARE;

const char *fmt = NULL, *out_fmt = NULL, *cache = "unsafe",

*src_cache = BDRV_DEFAULT_CACHE, *out_baseimg = NULL,

- *out_filename, *out_baseimg_param, *snapshot_name = NULL;

+ *out_filename, *out_baseimg_param, *snapshot_name = NULL,

+ *backing_fmt = NULL;

BlockDriver *drv = NULL, *proto_drv = NULL;

BlockDriverInfo bdi;

BlockDriverState *out_bs;

@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)

{"skip-broken-bitmaps", no_argument, 0, OPTION_SKIP_BROKEN},

{0, 0, 0, 0}

};

- c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WUr:",

+ c = getopt_long(argc, argv, ":hf:O:B:CcF:o:l:S:pt:T:qnm:WUr:",

long_options, NULL);

if (c == -1) {

break;

@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)

case 'c':

s.compressed = true;

break;

+ case 'F':

+ backing_fmt = optarg;

+ break;

case 'o':

if (accumulate_options(&options, optarg) < 0) {

goto fail_getopt;

@@ -XXX,XX +XXX,XX @@ static int img_convert(int argc, char **argv)

qemu_opt_set_number(opts, BLOCK_OPT_SIZE,

s.total_sectors * BDRV_SECTOR_SIZE, &error_abort);

- ret = add_old_style_options(out_fmt, opts, out_baseimg, NULL);

+ ret = add_old_style_options(out_fmt, opts, out_baseimg, backing_fmt);

if (ret < 0) {

goto out;

}

diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx

index XXXXXXX..XXXXXXX 100644

--- a/qemu-img-cmds.hx

+++ b/qemu-img-cmds.hx

@@ -XXX,XX +XXX,XX @@ SRST

ERST

DEF("convert", img_convert,

- "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-r rate_limit] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")

+ "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file [-F backing_fmt]] [-o options] [-l snapshot_param] [-S sparse_size] [-r rate_limit] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")

SRST

.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-r RATE_LIMIT] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME

ERST

diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/122

+++ b/tests/qemu-iotests/122

@@ -XXX,XX +XXX,XX @@ echo

_make_test_img -b "$TEST_IMG".base -F $IMGFMT

$QEMU_IO -c "write -P 0 0 3M" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir

-$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base -o backing_fmt=$IMGFMT \

+$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base -F $IMGFMT \

"$TEST_IMG" "$TEST_IMG".orig

$QEMU_IO -c "read -P 0 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir

$QEMU_IMG convert -O $IMGFMT -c -B "$TEST_IMG".base -o backing_fmt=$IMGFMT \

2.31.1

Split checking for reserved bits out of aligned offset check.

Reviewed-by: Eric Blake <eblake@redhat.com>

Tested-by: Kirill Tkhai <ktkhai@virtuozzo.com>

Reviewed-by: Hanna Reitz <hreitz@redhat.com>

Message-Id: <20210914122454.141075-11-vsementsov@virtuozzo.com>

Signed-off-by: Hanna Reitz <hreitz@redhat.com>

block/qcow2.h | 1 +

block/qcow2-refcount.c | 10 +++++++++-

2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/block/qcow2.h b/block/qcow2.h

--- a/block/qcow2.h

+++ b/block/qcow2.h

@@ -XXX,XX +XXX,XX @@ typedef enum QCow2MetadataOverlap {

#define L2E_STD_RESERVED_MASK 0x3f000000000001feULL

#define REFT_OFFSET_MASK 0xfffffffffffffe00ULL

+#define REFT_RESERVED_MASK 0x1ffULL

#define INV_OFFSET (-1ULL)

diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c

index XXXXXXX..XXXXXXX 100644

--- a/block/qcow2-refcount.c

+++ b/block/qcow2-refcount.c

@@ -XXX,XX +XXX,XX @@ static int check_refblocks(BlockDriverState *bs, BdrvCheckResult *res,

for(i = 0; i < s->refcount_table_size; i++) {

uint64_t offset, cluster;

- offset = s->refcount_table[i];

+ offset = s->refcount_table[i] & REFT_OFFSET_MASK;

cluster = offset >> s->cluster_bits;

+ if (s->refcount_table[i] & REFT_RESERVED_MASK) {

+ fprintf(stderr, "ERROR refcount table entry %" PRId64 " has "

+ "reserved bits set\n", i);

+ res->corruptions++;

+ *rebuild = true;

+ continue;

+ }

+

/* Refcount blocks are cluster aligned */

if (offset_into_cluster(s, offset)) {

fprintf(stderr, "ERROR refcount block %" PRId64 " is not "

2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Tested-by: Kirill Tkhai <ktkhai@virtuozzo.com>

Reviewed-by: Hanna Reitz <hreitz@redhat.com>

Message-Id: <20210914122454.141075-10-vsementsov@virtuozzo.com>

Signed-off-by: Hanna Reitz <hreitz@redhat.com>

block/qcow2.h | 1 +

block/qcow2-refcount.c | 6 ++++++

2 files changed, 7 insertions(+)

diff --git a/block/qcow2.h b/block/qcow2.h

--- a/block/qcow2.h

+++ b/block/qcow2.h

@@ -XXX,XX +XXX,XX @@ typedef enum QCow2MetadataOverlap {

(QCOW2_OL_CACHED | QCOW2_OL_INACTIVE_L2)

#define L1E_OFFSET_MASK 0x00fffffffffffe00ULL

+#define L1E_RESERVED_MASK 0x7f000000000001ffULL

#define L2E_OFFSET_MASK 0x00fffffffffffe00ULL

#define L2E_STD_RESERVED_MASK 0x3f000000000001feULL

diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c

index XXXXXXX..XXXXXXX 100644

--- a/block/qcow2-refcount.c

+++ b/block/qcow2-refcount.c

@@ -XXX,XX +XXX,XX @@ static int check_refcounts_l1(BlockDriverState *bs,

continue;

}

+ if (l1_table[i] & L1E_RESERVED_MASK) {

+ fprintf(stderr, "ERROR found L1 entry with reserved bits set: "

+ "%" PRIx64 "\n", l1_table[i]);

+ res->corruptions++;

+ }

l2_offset = l1_table[i] & L1E_OFFSET_MASK;

/* Mark L2 table as used */

2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

- use g_autofree for l1_table

- better name for size in bytes variable

- reduce code blocks nesting

- whitespaces, braces, newlines

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Reviewed-by: Hanna Reitz <hreitz@redhat.com>

Message-Id: <20210914122454.141075-9-vsementsov@virtuozzo.com>

Signed-off-by: Hanna Reitz <hreitz@redhat.com>

block/qcow2-refcount.c | 98 +++++++++++++++++++++---------------------

1 file changed, 50 insertions(+), 48 deletions(-)

diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c

--- a/block/qcow2-refcount.c

+++ b/block/qcow2-refcount.c

@@ -XXX,XX +XXX,XX @@ static int check_refcounts_l1(BlockDriverState *bs,

int flags, BdrvCheckMode fix, bool active)

{

BDRVQcow2State *s = bs->opaque;

- uint64_t *l1_table = NULL, l2_offset, l1_size2;

+ size_t l1_size_bytes = l1_size * L1E_SIZE;

+ g_autofree uint64_t *l1_table = NULL;

+ uint64_t l2_offset;

int i, ret;

- l1_size2 = l1_size * L1E_SIZE;

+ if (!l1_size) {

+ return 0;

+ }

/* Mark L1 table as used */

ret = qcow2_inc_refcounts_imrt(bs, res, refcount_table, refcount_table_size,

- l1_table_offset, l1_size2);

+ l1_table_offset, l1_size_bytes);

if (ret < 0) {

- goto fail;

+ return ret;

+ }

+

+ l1_table = g_try_malloc(l1_size_bytes);

+ if (l1_table == NULL) {

+ res->check_errors++;

+ return -ENOMEM;

/* Read L1 table entries from disk */

- if (l1_size2 > 0) {

- l1_table = g_try_malloc(l1_size2);

- if (l1_table == NULL) {

- ret = -ENOMEM;

- res->check_errors++;

- goto fail;

- }

- ret = bdrv_pread(bs->file, l1_table_offset, l1_table, l1_size2);

- if (ret < 0) {

- fprintf(stderr, "ERROR: I/O error in check_refcounts_l1\n");

- res->check_errors++;

- goto fail;

- }

- for(i = 0;i < l1_size; i++)

- be64_to_cpus(&l1_table[i]);

+ ret = bdrv_pread(bs->file, l1_table_offset, l1_table, l1_size_bytes);

+ if (ret < 0) {

+ fprintf(stderr, "ERROR: I/O error in check_refcounts_l1\n");

+ res->check_errors++;

+ return ret;

+ }

+

+ for (i = 0; i < l1_size; i++) {

+ be64_to_cpus(&l1_table[i]);

2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

There is no conflict and no dependency if we have parallel writes to

different subclusters of one cluster when the cluster itself is already

allocated. So, relax extra dependency.

Measure performance:

First, prepare build/qemu-img-old and build/qemu-img-new images.

cd scripts/simplebench

./img_bench_templater.py

Paste the following to stdin of running script:

qemu_img=../../build/qemu-img-{old|new}

$qemu_img create -f qcow2 -o extended_l2=on /ssd/x.qcow2 1G

$qemu_img bench -c 100000 -d 8 [-s 2K|-s 2K -o 512|-s $((1024*2+512))] \

-w -t none -n /ssd/x.qcow2

The result:

All results are in seconds

------------------ --------- ---------

old new

-s 2K 6.7 ± 15% 6.2 ± 12%

-7%

-s 2K -o 512 13 ± 3% 11 ± 5%

-16%

-s $((1024*2+512)) 9.5 ± 4% 8.4

-12%

------------------ --------- ---------

So small writes are more independent now and that helps to keep deeper

io queue which improves performance.

271 iotest output becomes racy for three allocation in one cluster.

Second and third writes may finish in different order. Second and

third requests don't depend on each other any more. Still they both

depend on first request anyway. Filter out second and third write

offsets to cover both possible outputs.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Message-Id: <20210824101517.59802-4-vsementsov@virtuozzo.com>

Reviewed-by: Eric Blake <eblake@redhat.com>

Reviewed-by: Hanna Reitz <hreitz@redhat.com>

[hreitz: s/ an / and /]

Signed-off-by: Hanna Reitz <hreitz@redhat.com>

---

block/qcow2-cluster.c | 11 +++++++++++

tests/qemu-iotests/271 | 5 ++++-

tests/qemu-iotests/271.out | 4 ++--

3 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c

--- a/block/qcow2-cluster.c

+++ b/block/qcow2-cluster.c

@@ -XXX,XX +XXX,XX @@ static int handle_dependencies(BlockDriverState *bs, uint64_t guest_offset,

continue;

}

+ if (old_alloc->keep_old_clusters &&

+ (end <= l2meta_cow_start(old_alloc) ||

+ start >= l2meta_cow_end(old_alloc)))

+ {

+ /*

+ * Clusters intersect but COW areas don't. And cluster itself is

+ * already allocated. So, there is no actual conflict.

+ */

+ continue;

+ }

+

/* Conflict */

if (start < old_start) {

diff --git a/tests/qemu-iotests/271 b/tests/qemu-iotests/271

index XXXXXXX..XXXXXXX 100755

--- a/tests/qemu-iotests/271

+++ b/tests/qemu-iotests/271

@@ -XXX,XX +XXX,XX @@ EOF

}

_make_test_img -o extended_l2=on 1M

-_concurrent_io | $QEMU_IO | _filter_qemu_io

+# Second and third writes in _concurrent_io() are independent and may finish in

+# different order. So, filter offset out to match both possible variants.

+_concurrent_io | $QEMU_IO | _filter_qemu_io | \

+ $SED -e 's/\(20480\|40960\)/OFFSET/'

_concurrent_verify | $QEMU_IO | _filter_qemu_io

# success, all done

diff --git a/tests/qemu-iotests/271.out b/tests/qemu-iotests/271.out

--- a/tests/qemu-iotests/271.out

+++ b/tests/qemu-iotests/271.out

@@ -XXX,XX +XXX,XX @@ blkdebug: Suspended request 'A'

blkdebug: Resuming request 'A'

wrote 2048/2048 bytes at offset 30720

2 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

-wrote 2048/2048 bytes at offset 20480

+wrote 2048/2048 bytes at offset OFFSET

2 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

-wrote 2048/2048 bytes at offset 40960

+wrote 2048/2048 bytes at offset OFFSET

2 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

2.31.1

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Split fix_l2_entry_by_zero() out of check_refcounts_l2() to be

reused in further patch.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Reviewed-by: Eric Blake <eblake@redhat.com>

Reviewed-by: Hanna Reitz <hreitz@redhat.com>

Message-Id: <20210914122454.141075-5-vsementsov@virtuozzo.com>

Signed-off-by: Hanna Reitz <hreitz@redhat.com>