From nobody Wed Oct 29 17:11:31 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1525194004444377.81062353450784; Tue, 1 May 2018 10:00:04 -0700 (PDT) Received: from localhost ([::1]:45328 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fDYdH-0002et-Jr for importer@patchew.org; Tue, 01 May 2018 13:00:03 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59568) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fDYbO-0001Os-6K for qemu-devel@nongnu.org; Tue, 01 May 2018 12:58:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fDYbN-0005sS-3o for qemu-devel@nongnu.org; Tue, 01 May 2018 12:58:06 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:57504 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fDYbI-0005pZ-DO; Tue, 01 May 2018 12:58:00 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0EAF5406C78D; Tue, 1 May 2018 16:57:58 +0000 (UTC) Received: from localhost (ovpn-204-22.brq.redhat.com [10.40.204.22]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AB47D111AF28; Tue, 1 May 2018 16:57:57 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 1 May 2018 18:57:49 +0200 Message-Id: <20180501165750.19242-2-mreitz@redhat.com> In-Reply-To: <20180501165750.19242-1-mreitz@redhat.com> References: <20180501165750.19242-1-mreitz@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Tue, 01 May 2018 16:57:58 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Tue, 01 May 2018 16:57:58 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'mreitz@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 1/2] qemu-img: Special post-backing convert handling X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Currently, qemu-img convert writes zeroes when it reads zeroes. Sometimes it does not because the target is initialized to zeroes anyway, so we do not need to overwrite (and thus potentially allocate) it. This is never the case for targets with backing files, though. But even them may have an area that is initialized to zeroes, and that is the area past the end of the backing file (if that is shorter than the overlay). So if the target format's unallocated blocks are zero and there is a gap between the target's backing file's end and the target's end, we do not have to explicitly write zeroes there. Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=3D1527898 Signed-off-by: Max Reitz Reviewed-by: Eric Blake --- qemu-img.c | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/qemu-img.c b/qemu-img.c index 855fa52514..5365afbd13 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -1542,7 +1542,9 @@ typedef struct ImgConvertState { BlockBackend *target; bool has_zero_init; bool compressed; + bool unallocated_blocks_are_zero; bool target_has_backing; + int64_t target_backing_sectors; /* negative if unknown */ bool wr_in_order; int min_sparse; size_t cluster_sectors; @@ -1571,12 +1573,23 @@ static int convert_iteration_sectors(ImgConvertStat= e *s, int64_t sector_num) { int64_t src_cur_offset; int ret, n, src_cur; + bool post_backing_zero =3D false; =20 convert_select_part(s, sector_num, &src_cur, &src_cur_offset); =20 assert(s->total_sectors > sector_num); n =3D MIN(s->total_sectors - sector_num, BDRV_REQUEST_MAX_SECTORS); =20 + if (s->target_backing_sectors >=3D 0) { + if (sector_num >=3D s->target_backing_sectors) { + post_backing_zero =3D s->unallocated_blocks_are_zero; + } else if (sector_num + n > s->target_backing_sectors) { + /* Split requests around target_backing_sectors (because + * starting from there, zeros are handled differently) */ + n =3D s->target_backing_sectors - sector_num; + } + } + if (s->sector_next_status <=3D sector_num) { int64_t count =3D n * BDRV_SECTOR_SIZE; =20 @@ -1598,7 +1611,7 @@ static int convert_iteration_sectors(ImgConvertState = *s, int64_t sector_num) n =3D DIV_ROUND_UP(count, BDRV_SECTOR_SIZE); =20 if (ret & BDRV_BLOCK_ZERO) { - s->status =3D BLK_ZERO; + s->status =3D post_backing_zero ? BLK_BACKING_FILE : BLK_ZERO; } else if (ret & BDRV_BLOCK_DATA) { s->status =3D BLK_DATA; } else { @@ -2319,6 +2332,16 @@ static int img_convert(int argc, char **argv) } } =20 + if (s.target_has_backing) { + /* Errors are treated as "backing length unknown" (which means + * s.target_backing_sectors has to be negative, which it will + * be automatically). The backing file length is used only + * for optimizations, so such a case is not fatal. */ + s.target_backing_sectors =3D bdrv_nb_sectors(out_bs->backing->bs); + } else { + s.target_backing_sectors =3D -1; + } + ret =3D bdrv_get_info(out_bs, &bdi); if (ret < 0) { if (s.compressed) { @@ -2328,6 +2351,7 @@ static int img_convert(int argc, char **argv) } else { s.compressed =3D s.compressed || bdi.needs_compressed_writes; s.cluster_sectors =3D bdi.cluster_size / BDRV_SECTOR_SIZE; + s.unallocated_blocks_are_zero =3D bdi.unallocated_blocks_are_zero; } =20 ret =3D convert_do_copy(&s); --=20 2.14.3 From nobody Wed Oct 29 17:11:31 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1525194136270380.8629757832018; Tue, 1 May 2018 10:02:16 -0700 (PDT) Received: from localhost ([::1]:45351 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fDYfG-0004Ob-PC for importer@patchew.org; Tue, 01 May 2018 13:02:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59566) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fDYbO-0001Oq-3n for qemu-devel@nongnu.org; Tue, 01 May 2018 12:58:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fDYbN-0005sL-0w for qemu-devel@nongnu.org; Tue, 01 May 2018 12:58:06 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51382 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fDYbI-0005ph-Gn; Tue, 01 May 2018 12:58:00 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E5A6F81A88A7; Tue, 1 May 2018 16:57:59 +0000 (UTC) Received: from localhost (ovpn-204-22.brq.redhat.com [10.40.204.22]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 931C0111AF28; Tue, 1 May 2018 16:57:59 +0000 (UTC) From: Max Reitz To: qemu-block@nongnu.org Date: Tue, 1 May 2018 18:57:50 +0200 Message-Id: <20180501165750.19242-3-mreitz@redhat.com> In-Reply-To: <20180501165750.19242-1-mreitz@redhat.com> References: <20180501165750.19242-1-mreitz@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Tue, 01 May 2018 16:57:59 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Tue, 01 May 2018 16:57:59 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'mreitz@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH 2/2] iotests: Test post-backing convert target behavior X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , qemu-devel@nongnu.org, Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This adds a test case to 122 for what happens when you convert to a target with a backing file that is shorter than the target, and the image format does not support efficient zero writes (as is the case with qcow2 v2). Signed-off-by: Max Reitz Reviewed-by: Eric Blake --- tests/qemu-iotests/122 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 18 ++++++++++++++++++ 2 files changed, 60 insertions(+) diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122 index 45b359c2ba..d8c8ad722d 100755 --- a/tests/qemu-iotests/122 +++ b/tests/qemu-iotests/122 @@ -76,6 +76,48 @@ $QEMU_IMG convert -O $IMGFMT -c -B "$TEST_IMG".base "$TE= ST_IMG" "$TEST_IMG".orig $QEMU_IO -c "read -P 0 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _fi= lter_testdir =20 =20 +echo +echo "=3D=3D=3D Converting to an overlay larger than its backing file =3D= =3D=3D" +echo + +TEST_IMG=3D"$TEST_IMG".base _make_test_img 256M +# Needs to be at least how much an L2 table covers +# (64 kB/entry * 64 kB / 8 B/entry =3D 512 MB) +# That way, qcow2 will yield at least two status request responses. +# With just a single response, it would always say "Allocated in the +# backing file", so the optimization qemu-img convert tries to do is +# done automatically. Once it has to be queried twice, however (and +# one of the queries is completely after the end of the backing file), +# the block layer will automatically add a ZERO flag that qemu-img +# convert used to follow up with a zero write to the target. +# We do not want such a zero write, however, because we are past the +# end of the backing file on the target as well, so we do not need to +# write anything there. +_make_test_img -b "$TEST_IMG".base 768M + +# Use compat=3D0.10 as the output so there is no zero cluster support +$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base -o compat=3D0.10 \ + "$TEST_IMG" "$TEST_IMG".orig +# See that nothing has been allocated past 64M +$QEMU_IMG map "$TEST_IMG".orig | _filter_qemu_img_map + +echo + +# Just before the end of the backing file +$QEMU_IO -c 'write -P 0x11 255M 1M' "$TEST_IMG".base 2>&1 | _filter_qemu_io +# Somewhere in the second L2 table +$QEMU_IO -c 'write -P 0x22 600M 1M' "$TEST_IMG" 2>&1 | _filter_qemu_io + +$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base -o compat=3D0.10 \ + "$TEST_IMG" "$TEST_IMG".orig + +$QEMU_IMG map "$TEST_IMG".orig | _filter_qemu_img_map +$QEMU_IO -c 'read -P 0x11 255M 1M' \ + -c 'read -P 0x22 600M 1M' \ + "$TEST_IMG".orig \ + | _filter_qemu_io + + echo echo "=3D=3D=3D Concatenate multiple source images =3D=3D=3D" echo diff --git a/tests/qemu-iotests/122.out b/tests/qemu-iotests/122.out index 47d8656db8..6c7ee1da6c 100644 --- a/tests/qemu-iotests/122.out +++ b/tests/qemu-iotests/122.out @@ -28,6 +28,24 @@ read 3145728/3145728 bytes at offset 0 read 3145728/3145728 bytes at offset 0 3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) =20 +=3D=3D=3D Converting to an overlay larger than its backing file =3D=3D=3D + +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=3DIMGFMT size=3D268435456 +Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D805306368 backing_file= =3DTEST_DIR/t.IMGFMT.base +Offset Length File + +wrote 1048576/1048576 bytes at offset 267386880 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 1048576/1048576 bytes at offset 629145600 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Offset Length File +0xff00000 0x100000 TEST_DIR/t.IMGFMT.base +0x25800000 0x100000 TEST_DIR/t.IMGFMT.orig +read 1048576/1048576 bytes at offset 267386880 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 1048576/1048576 bytes at offset 629145600 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + =3D=3D=3D Concatenate multiple source images =3D=3D=3D =20 Formatting 'TEST_DIR/t.IMGFMT.1', fmt=3DIMGFMT size=3D4194304 --=20 2.14.3