From nobody Sat May 4 16:31:08 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1685647798; cv=none; d=zohomail.com; s=zohoarc; b=SFPLDZQi3Sq5jKMNrCnvcHWP1ljQJwCb7tJfifd1pX3aZRNUNzB1g+cfiycm9UwpuAkIzZf0hjDp8FJvsyUJ8sDzXgAZiF5r1ahiWAAJUG2BHWQYs5HNKVJMrpFY16Ry9+28YO2Vb4igQhs36vAwguAnHl/DgmWrUIYPakgNFFw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1685647798; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=1+iZO2eqe7bJJVKqWLuAtOwCBi7Y53t/y79s+w6HweI=; b=JaLC4fKivCzjTKd8n5JHr0YOY6MqNJFw+lqNsxTXcQ8dQMLYF7uaI/QiyIxovIi/ibbp2cl10vkGpGM2NKy/j3guXAsRBqDT/8mmtaeZAmuKFWGlOMmzMvZKMxVU5xtbjJCOJGqoyXJKQa4VN3wx5atXF3nvS4Fg8HqOB34ixv4= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1685647798623785.7924655890241; Thu, 1 Jun 2023 12:29:58 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q4nyn-0006oG-L0; Thu, 01 Jun 2023 15:29:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyZ-0006js-17; Thu, 01 Jun 2023 15:28:48 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyV-0005mC-S3; Thu, 01 Jun 2023 15:28:46 -0400 Received: from dev005.ch-qa.vzint.dev ([172.29.1.10]) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1q4nyC-00DLDg-0X; Thu, 01 Jun 2023 21:28:36 +0200 To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, kwolf@redhat.com, hreitz@redhat.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 1/6] qemu-img: rebase: stop when reaching EOF of old backing file Date: Thu, 1 Jun 2023 22:28:31 +0300 Message-Id: <20230601192836.598602-2-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> References: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Andrey Drobyshev From: Andrey Drobyshev via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1685647799235100001 Content-Type: text/plain; charset="utf-8" In case when we're rebasing within one backing chain, and when target image is larger than old backing file, bdrv_is_allocated_above() ends up setting *pnum =3D 0. As a result, target offset isn't getting incremented, and we get stuck in an infinite for loop. Let's detect this case and proceed further down the loop body, as the offsets beyond the old backing size need to be explicitly zeroed. Signed-off-by: Andrey Drobyshev Reviewed-by: Denis V. Lunev Reviewed-by: Hanna Czenczek --- qemu-img.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/qemu-img.c b/qemu-img.c index 27f48051b0..78433f3746 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -3801,6 +3801,8 @@ static int img_rebase(int argc, char **argv) } =20 if (prefix_chain_bs) { + uint64_t bytes =3D n; + /* * If cluster wasn't changed since prefix_chain, we don't = need * to take action @@ -3813,9 +3815,18 @@ static int img_rebase(int argc, char **argv) strerror(-ret)); goto out; } - if (!ret) { + if (!ret && n) { continue; } + if (!n) { + /* + * If we've reached EOF of the old backing, it means t= hat + * offsets beyond the old backing size were read as ze= roes. + * Now we will need to explicitly zero the cluster in + * order to preserve that state after the rebase. + */ + n =3D bytes; + } } =20 /* --=20 2.31.1 From nobody Sat May 4 16:31:08 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1685647799; cv=none; d=zohomail.com; s=zohoarc; b=RgbT/B+lh09ZqZTwWOTdIesz6e8w7B/3TWN7AZnCPEbNcwYyEcULe+csZUsgxaKMH1CfC0hb2rR5UIjsuCFso7RdUz10j5Ab+YhWhIBW228nrDzFnACTgR2QgdJ3upgb8FswnqzPVNIPv4gDkBq68D0ipj6kN84+9tbcZfdcPE0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1685647799; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=3MdWU0qOQXTnWlxZdFDE6S+cyhsD+nTJcQ7CUCxF0Yw=; b=m5SDUPgYGYgM3AkYle8GHWRBVnnrTnc9tK7fnQsFtf6WXXESArtrUldVTXpyM1dnl9taPi9jL9WkK/ybFCkDTYAakFDpLIueONWQtN6/VQn6R3XHIOkzjLMqFKB1dWIlx3/UBKPXW/J/0iVN9FopF9V6DcVcpUIktMiuwDSE0Vs= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1685647799910360.48104540769384; Thu, 1 Jun 2023 12:29:59 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q4nya-0006jt-Ll; Thu, 01 Jun 2023 15:28:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyX-0006jJ-FJ; Thu, 01 Jun 2023 15:28:45 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyV-0005mD-OB; Thu, 01 Jun 2023 15:28:45 -0400 Received: from dev005.ch-qa.vzint.dev ([172.29.1.10]) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1q4nyC-00DLDg-0h; Thu, 01 Jun 2023 21:28:36 +0200 To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, kwolf@redhat.com, hreitz@redhat.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 2/6] qemu-iotests: 024: add rebasing test case for overlay_size > backing_size Date: Thu, 1 Jun 2023 22:28:32 +0300 Message-Id: <20230601192836.598602-3-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> References: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Andrey Drobyshev From: Andrey Drobyshev via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1685647801800100023 Content-Type: text/plain; charset="utf-8" Before previous commit, rebase was getting infitely stuck in case of rebasing within the same backing chain and when overlay_size > backing_size. Let's add this case to the rebasing test 024 to make sure it doesn't break again. Signed-off-by: Andrey Drobyshev Reviewed-by: Denis V. Lunev Reviewed-by: Hanna Czenczek --- tests/qemu-iotests/024 | 57 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/024.out | 30 ++++++++++++++++++++ 2 files changed, 87 insertions(+) diff --git a/tests/qemu-iotests/024 b/tests/qemu-iotests/024 index 25a564a150..98a7c8fd65 100755 --- a/tests/qemu-iotests/024 +++ b/tests/qemu-iotests/024 @@ -199,6 +199,63 @@ echo # $BASE_OLD and $BASE_NEW) $QEMU_IMG map "$OVERLAY" | _filter_qemu_img_map =20 +# Check that rebase within the chain is working when +# overlay_size > old_backing_size +# +# base_new <-- base_old <-- overlay +# +# Backing (new): 11 11 11 11 11 +# Backing (old): 22 22 22 22 +# Overlay: -- -- -- -- -- +# +# As a result, overlay should contain data identical to base_old, with the +# last cluster remaining unallocated. + +echo +echo "=3D=3D=3D Test rebase within one backing chain =3D=3D=3D" +echo + +echo "Creating backing chain" +echo + +TEST_IMG=3D$BASE_NEW _make_test_img $(( CLUSTER_SIZE * 5 )) +TEST_IMG=3D$BASE_OLD _make_test_img -b "$BASE_NEW" -F $IMGFMT \ + $(( CLUSTER_SIZE * 4 )) +TEST_IMG=3D$OVERLAY _make_test_img -b "$BASE_OLD" -F $IMGFMT \ + $(( CLUSTER_SIZE * 5 )) + +echo +echo "Fill backing files with data" +echo + +$QEMU_IO "$BASE_NEW" -c "write -P 0x11 0 $(( CLUSTER_SIZE * 5 ))" \ + | _filter_qemu_io +$QEMU_IO "$BASE_OLD" -c "write -P 0x22 0 $(( CLUSTER_SIZE * 4 ))" \ + | _filter_qemu_io + +echo +echo "Check the last cluster is zeroed in overlay before the rebase" +echo +$QEMU_IO "$OVERLAY" -c "read -P 0x00 $(( CLUSTER_SIZE * 4 )) $CLUSTER_SIZE= " \ + | _filter_qemu_io + +echo +echo "Rebase onto another image in the same chain" +echo + +$QEMU_IMG rebase -b "$BASE_NEW" -F $IMGFMT "$OVERLAY" + +echo "Verify that data is read the same before and after rebase" +echo + +# Verify the first 4 clusters are still read the same as in the old base +$QEMU_IO "$OVERLAY" -c "read -P 0x22 0 $(( CLUSTER_SIZE * 4 ))" \ + | _filter_qemu_io +# Verify the last cluster still reads as zeroes +$QEMU_IO "$OVERLAY" -c "read -P 0x00 $(( CLUSTER_SIZE * 4 )) $CLUSTER_SIZE= " \ + | _filter_qemu_io + +echo =20 # success, all done echo "*** done" diff --git a/tests/qemu-iotests/024.out b/tests/qemu-iotests/024.out index 973a5a3711..245fe8b1d1 100644 --- a/tests/qemu-iotests/024.out +++ b/tests/qemu-iotests/024.out @@ -171,4 +171,34 @@ read 65536/65536 bytes at offset 196608 Offset Length File 0 0x30000 TEST_DIR/subdir/t.IMGFMT 0x30000 0x10000 TEST_DIR/subdir/t.IMGFMT.base_new + +=3D=3D=3D Test rebase within one backing chain =3D=3D=3D + +Creating backing chain + +Formatting 'TEST_DIR/subdir/t.IMGFMT.base_new', fmt=3DIMGFMT size=3D327680 +Formatting 'TEST_DIR/subdir/t.IMGFMT.base_old', fmt=3DIMGFMT size=3D262144= backing_file=3DTEST_DIR/subdir/t.IMGFMT.base_new backing_fmt=3DIMGFMT +Formatting 'TEST_DIR/subdir/t.IMGFMT', fmt=3DIMGFMT size=3D327680 backing_= file=3DTEST_DIR/subdir/t.IMGFMT.base_old backing_fmt=3DIMGFMT + +Fill backing files with data + +wrote 327680/327680 bytes at offset 0 +320 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 262144/262144 bytes at offset 0 +256 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +Check the last cluster is zeroed in overlay before the rebase + +read 65536/65536 bytes at offset 262144 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +Rebase onto another image in the same chain + +Verify that data is read the same before and after rebase + +read 262144/262144 bytes at offset 0 +256 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 65536/65536 bytes at offset 262144 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + *** done --=20 2.31.1 From nobody Sat May 4 16:31:08 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1685647798; cv=none; d=zohomail.com; s=zohoarc; b=JwxXMccaMDDMmtut0+zUVqlP8Nim/R8Qk1Qe0Dpd+81/ti3vwiGVvNvnYnUKGkbny81EI4CLfAn3sKayBJRrn790BdjdMWEGm6cgw0XWGQ8NsnjjOUeL/DqbhhFEWVOFyQffGj49jCC4ACtbtmM5lK0qWySitvnvkPuAxv0q36Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1685647798; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=9anG3Rb3/R7wVK/QtKsv8I34QsnvkVUghxhHuuy2fPA=; b=Egg7PQnGkA06cwiAFfr7tuislRVl7b6goVv+a4zsRkxskkiO6V/3Z4mbRwDQrL1GeL7vr9785LI+S1h5mQCbqCIkxYz+QWb+vjFxV3zfNZHoltyHLOdK8CSDVkU8DVFpliXeArGcH/+eGPBAV0HzeAUGfuLjWKt6qQl+vWwj4yA= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1685647798376572.053147435482; Thu, 1 Jun 2023 12:29:58 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q4nyh-0006mQ-9k; Thu, 01 Jun 2023 15:28:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyY-0006jh-NT; Thu, 01 Jun 2023 15:28:46 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyV-0005mB-RE; Thu, 01 Jun 2023 15:28:46 -0400 Received: from dev005.ch-qa.vzint.dev ([172.29.1.10]) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1q4nyC-00DLDg-0s; Thu, 01 Jun 2023 21:28:36 +0200 To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, kwolf@redhat.com, hreitz@redhat.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 3/6] qemu-img: rebase: use backing files' BlockBackend for buffer alignment Date: Thu, 1 Jun 2023 22:28:33 +0300 Message-Id: <20230601192836.598602-4-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> References: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Andrey Drobyshev From: Andrey Drobyshev via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1685647799235100002 Content-Type: text/plain; charset="utf-8" Since commit bb1c05973cf ("qemu-img: Use qemu_blockalign"), buffers for the data read from the old and new backing files are aligned using BlockDriverState (or BlockBackend later on) referring to the target image. However, this isn't quite right, because target image is only being written to and has nothing to do with those buffers. Let's fix that. Signed-off-by: Andrey Drobyshev Reviewed-by: Denis V. Lunev --- qemu-img.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index 78433f3746..60f4c06487 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -3746,8 +3746,8 @@ static int img_rebase(int argc, char **argv) int64_t n; float local_progress =3D 0; =20 - buf_old =3D blk_blockalign(blk, IO_BUF_SIZE); - buf_new =3D blk_blockalign(blk, IO_BUF_SIZE); + buf_old =3D blk_blockalign(blk_old_backing, IO_BUF_SIZE); + buf_new =3D blk_blockalign(blk_new_backing, IO_BUF_SIZE); =20 size =3D blk_getlength(blk); if (size < 0) { --=20 2.31.1 From nobody Sat May 4 16:31:08 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1685647798; cv=none; d=zohomail.com; s=zohoarc; b=cRvCeJ8hOaqe/XFSxCsA28+FXfwcU9Ptn4SBtsHqt5mxSFCAKm+TzT22Cx08sLZ+8fZj6EtPrmmguruMV/uR27eqqewkFr7KGlgNgDVt7spSFJosr4BoqyVFzp4A0WAVehGYQz94ou8rS6lAfzBJoiRmK1hsef8GlWJfwmGxMlU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1685647798; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=U4PgCD9g+22hR7XPQiy42CR2JCtKgrziH/qy8q4uDlw=; b=DiODCh2ciJ2XdiNsnurxWI/C1HO0TwPu5TyLPu/hr+6L3zLPqXRxTjGeqkQk1D1rI+VztjQEdFeJ4vYGgwzhvCqC4jyd6O11uQecGcr36idQjI+6WuScftDsSR0rV1y2FGT0D5gjvjFlxFemYf7ActeDKWzT0ScIuvjDzMZVPdU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1685647798628244.95902560416448; Thu, 1 Jun 2023 12:29:58 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q4nyk-0006nI-1d; Thu, 01 Jun 2023 15:28:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyZ-0006ju-6c; Thu, 01 Jun 2023 15:28:48 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyV-0005mE-TJ; Thu, 01 Jun 2023 15:28:46 -0400 Received: from dev005.ch-qa.vzint.dev ([172.29.1.10]) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1q4nyC-00DLDg-12; Thu, 01 Jun 2023 21:28:36 +0200 To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, kwolf@redhat.com, hreitz@redhat.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 4/6] qemu-img: rebase: avoid unnecessary COW operations Date: Thu, 1 Jun 2023 22:28:34 +0300 Message-Id: <20230601192836.598602-5-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> References: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Andrey Drobyshev From: Andrey Drobyshev via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1685647799247100004 Content-Type: text/plain; charset="utf-8" When rebasing an image from one backing file to another, we need to compare data from old and new backings. If the diff between that data happens to be unaligned to the target cluster size, we might end up doing partial writes, which would lead to copy-on-write and additional IO. Consider the following simple case (virtual_size =3D=3D cluster_size =3D=3D= 64K): base <-- inc1 <-- inc2 qemu-io -c "write -P 0xaa 0 32K" base.qcow2 qemu-io -c "write -P 0xcc 32K 32K" base.qcow2 qemu-io -c "write -P 0xbb 0 32K" inc1.qcow2 qemu-io -c "write -P 0xcc 32K 32K" inc1.qcow2 qemu-img rebase -f qcow2 -b base.qcow2 -F qcow2 inc2.qcow2 While doing rebase, we'll write a half of the cluster to inc2, and block layer will have to read the 2nd half of the same cluster from the base image inc1 while doing this write operation, although the whole cluster is already read earlier to perform data comparison. In order to avoid these unnecessary IO cycles, let's make sure every write request is aligned to the overlay cluster size. Signed-off-by: Andrey Drobyshev Reviewed-by: Denis V. Lunev --- qemu-img.c | 72 +++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 52 insertions(+), 20 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index 60f4c06487..9a469cd609 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -3513,6 +3513,7 @@ static int img_rebase(int argc, char **argv) uint8_t *buf_new =3D NULL; BlockDriverState *bs =3D NULL, *prefix_chain_bs =3D NULL; BlockDriverState *unfiltered_bs; + BlockDriverInfo bdi =3D {0}; char *filename; const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg; int c, flags, src_flags, ret; @@ -3646,6 +3647,15 @@ static int img_rebase(int argc, char **argv) } } =20 + /* We need overlay cluster size to make sure write requests are aligne= d */ + ret =3D bdrv_get_info(unfiltered_bs, &bdi); + if (ret < 0) { + error_report("could not get block driver info"); + goto out; + } else if (bdi.cluster_size =3D=3D 0) { + bdi.cluster_size =3D 1; + } + /* For safe rebasing we need to compare old and new backing file */ if (!unsafe) { QDict *options =3D NULL; @@ -3744,6 +3754,7 @@ static int img_rebase(int argc, char **argv) int64_t new_backing_size =3D 0; uint64_t offset; int64_t n; + int64_t n_old =3D 0, n_new =3D 0; float local_progress =3D 0; =20 buf_old =3D blk_blockalign(blk_old_backing, IO_BUF_SIZE); @@ -3784,7 +3795,7 @@ static int img_rebase(int argc, char **argv) } =20 for (offset =3D 0; offset < size; offset +=3D n) { - bool buf_old_is_zero =3D false; + bool old_backing_eof =3D false; =20 /* How many bytes can we handle with the next read? */ n =3D MIN(IO_BUF_SIZE, size - offset); @@ -3829,33 +3840,38 @@ static int img_rebase(int argc, char **argv) } } =20 + /* At this point n must be aligned to the target cluster size.= */ + if (offset + n < size) { + assert(n % bdi.cluster_size =3D=3D 0); + } + + /* + * Much like the with the target image, we'll try to read as m= uch + * of the old and new backings as we can. + */ + n_old =3D MIN(n, MAX(0, old_backing_size - (int64_t) offset)); + if (blk_new_backing) { + n_new =3D MIN(n, MAX(0, new_backing_size - (int64_t) offse= t)); + } + /* * Read old and new backing file and take into consideration t= hat * backing files may be smaller than the COW image. */ - if (offset >=3D old_backing_size) { - memset(buf_old, 0, n); - buf_old_is_zero =3D true; + memset(buf_old + n_old, 0, n - n_old); + if (!n_old) { + old_backing_eof =3D true; } else { - if (offset + n > old_backing_size) { - n =3D old_backing_size - offset; - } - - ret =3D blk_pread(blk_old_backing, offset, n, buf_old, 0); + ret =3D blk_pread(blk_old_backing, offset, n_old, buf_old,= 0); if (ret < 0) { error_report("error while reading from old backing fil= e"); goto out; } } =20 - if (offset >=3D new_backing_size || !blk_new_backing) { - memset(buf_new, 0, n); - } else { - if (offset + n > new_backing_size) { - n =3D new_backing_size - offset; - } - - ret =3D blk_pread(blk_new_backing, offset, n, buf_new, 0); + memset(buf_new + n_new, 0, n - n_new); + if (blk_new_backing && n_new) { + ret =3D blk_pread(blk_new_backing, offset, n_new, buf_new,= 0); if (ret < 0) { error_report("error while reading from new backing fil= e"); goto out; @@ -3867,15 +3883,28 @@ static int img_rebase(int argc, char **argv) =20 while (written < n) { int64_t pnum; + int64_t start, end; =20 if (compare_buffers(buf_old + written, buf_new + written, n - written, &pnum)) { - if (buf_old_is_zero) { + if (old_backing_eof) { ret =3D blk_pwrite_zeroes(blk, offset + written, p= num, 0); } else { - ret =3D blk_pwrite(blk, offset + written, pnum, - buf_old + written, 0); + /* + * If we've got to this point, it means the cluster + * we're dealing with is unallocated, and any part= ial + * write will cause COW. To avoid that, we make s= ure + * request is aligned to cluster size. + */ + start =3D QEMU_ALIGN_DOWN(offset + written, + bdi.cluster_size); + end =3D QEMU_ALIGN_UP(offset + written + pnum, + bdi.cluster_size); + end =3D end > size ? size : end; + ret =3D blk_pwrite(blk, start, end - start, + buf_old + (start - offset), 0); + pnum =3D end - (offset + written); } if (ret < 0) { error_report("Error while writing to COW image: %s= ", @@ -3885,6 +3914,9 @@ static int img_rebase(int argc, char **argv) } =20 written +=3D pnum; + if (offset + written >=3D old_backing_size) { + old_backing_eof =3D true; + } } qemu_progress_print(local_progress, 100); } --=20 2.31.1 From nobody Sat May 4 16:31:08 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1685647798; cv=none; d=zohomail.com; s=zohoarc; b=Leou2O9u1+Yvg24PSQv8G2ULdzlUxiEy6/1utTasoPvihKfnb/0Ly2tz5qqX4Rs8vgcCRqxMSMvsbqc9OnKyUrMZWBhJUNNBe4U0s4RD8mHjp+CCza4a/QU14PxM8I0QLJ/juVSd0nFBybMQDzBkggAEJMS+zZC9SzGvtZBbUco= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1685647798; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=9irQOhY+9I+q+o9vE6Usvv430vg5tTFhk81hoF/oeJg=; b=X/A7c73SMEXqzEu0hxUqElC0EYHCkfmonTZCBuQwSCoggO6HE/4tWr3tfx66RI5Lk3keqbHc+KlEI0HttWQeFo7cfbVwUO30U0o/IVx09u9TzntIanvsAtIyluEynrCBsUfIBGlCP5SkFT+0KyOWybf9h8B5VwbZMg4uT4qP5ZI= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1685647798377722.0513475651431; Thu, 1 Jun 2023 12:29:58 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q4nyk-0006nd-IS; Thu, 01 Jun 2023 15:28:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyZ-0006jv-CQ; Thu, 01 Jun 2023 15:28:48 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyV-0005m9-Uv; Thu, 01 Jun 2023 15:28:47 -0400 Received: from dev005.ch-qa.vzint.dev ([172.29.1.10]) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1q4nyC-00DLDg-1D; Thu, 01 Jun 2023 21:28:36 +0200 To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, kwolf@redhat.com, hreitz@redhat.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 5/6] qemu-img: add compression option to rebase subcommand Date: Thu, 1 Jun 2023 22:28:35 +0300 Message-Id: <20230601192836.598602-6-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> References: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Andrey Drobyshev From: Andrey Drobyshev via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1685647799257100005 Content-Type: text/plain; charset="utf-8" If we rebase an image whose backing file has compressed clusters, we might end up wasting disk space since the copied clusters are now uncompressed. In order to have better control over this, let's add "--compress" option to the "qemu-img rebase" command. Note that this option affects only the clusters which are actually being copied from the original backing file. The clusters which were uncompressed in the target image will remain so. Signed-off-by: Andrey Drobyshev Reviewed-by: Denis V. Lunev Reviewed-by: Hanna Czenczek --- docs/tools/qemu-img.rst | 6 ++++-- qemu-img-cmds.hx | 4 ++-- qemu-img.c | 19 +++++++++++++++++-- 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst index 15aeddc6d8..973a912dec 100644 --- a/docs/tools/qemu-img.rst +++ b/docs/tools/qemu-img.rst @@ -663,7 +663,7 @@ Command description: =20 List, apply, create or delete snapshots in image *FILENAME*. =20 -.. option:: rebase [--object OBJECTDEF] [--image-opts] [-U] [-q] [-f FMT] = [-t CACHE] [-T SRC_CACHE] [-p] [-u] -b BACKING_FILE [-F BACKING_FMT] FILENA= ME +.. option:: rebase [--object OBJECTDEF] [--image-opts] [-U] [-q] [-f FMT] = [-t CACHE] [-T SRC_CACHE] [-p] [-u] [-c] -b BACKING_FILE [-F BACKING_FMT] F= ILENAME =20 Changes the backing file of an image. Only the formats ``qcow2`` and ``qed`` support changing the backing file. @@ -690,7 +690,9 @@ Command description: =20 In order to achieve this, any clusters that differ between *BACKING_FILE* and the old backing file of *FILENAME* are merged - into *FILENAME* before actually changing the backing file. + into *FILENAME* before actually changing the backing file. With ``-c`` + option specified, the clusters which are being merged (but not the + entire *FILENAME* image) are written in the compressed mode. =20 Note that the safe mode is an expensive operation, comparable to converting an image. It only works if the old backing file still diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx index 1b1dab5b17..068692d13e 100644 --- a/qemu-img-cmds.hx +++ b/qemu-img-cmds.hx @@ -88,9 +88,9 @@ SRST ERST =20 DEF("rebase", img_rebase, - "rebase [--object objectdef] [--image-opts] [-U] [-q] [-f fmt] [-t cac= he] [-T src_cache] [-p] [-u] -b backing_file [-F backing_fmt] filename") + "rebase [--object objectdef] [--image-opts] [-U] [-q] [-f fmt] [-t cac= he] [-T src_cache] [-p] [-u] [-c] -b backing_file [-F backing_fmt] filename= ") SRST -.. option:: rebase [--object OBJECTDEF] [--image-opts] [-U] [-q] [-f FMT] = [-t CACHE] [-T SRC_CACHE] [-p] [-u] -b BACKING_FILE [-F BACKING_FMT] FILENA= ME +.. option:: rebase [--object OBJECTDEF] [--image-opts] [-U] [-q] [-f FMT] = [-t CACHE] [-T SRC_CACHE] [-p] [-u] [-c] -b BACKING_FILE [-F BACKING_FMT] F= ILENAME ERST =20 DEF("resize", img_resize, diff --git a/qemu-img.c b/qemu-img.c index 9a469cd609..108da27b23 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -3517,11 +3517,13 @@ static int img_rebase(int argc, char **argv) char *filename; const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg; int c, flags, src_flags, ret; + BdrvRequestFlags write_flags =3D 0; bool writethrough, src_writethrough; int unsafe =3D 0; bool force_share =3D false; int progress =3D 0; bool quiet =3D false; + bool compress =3D false; Error *local_err =3D NULL; bool image_opts =3D false; =20 @@ -3537,9 +3539,10 @@ static int img_rebase(int argc, char **argv) {"object", required_argument, 0, OPTION_OBJECT}, {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS}, {"force-share", no_argument, 0, 'U'}, + {"compress", no_argument, 0, 'c'}, {0, 0, 0, 0} }; - c =3D getopt_long(argc, argv, ":hf:F:b:upt:T:qU", + c =3D getopt_long(argc, argv, ":hf:F:b:upt:T:qUc", long_options, NULL); if (c =3D=3D -1) { break; @@ -3587,6 +3590,9 @@ static int img_rebase(int argc, char **argv) case 'U': force_share =3D true; break; + case 'c': + compress =3D true; + break; } } =20 @@ -3639,6 +3645,14 @@ static int img_rebase(int argc, char **argv) =20 unfiltered_bs =3D bdrv_skip_filters(bs); =20 + if (compress && !block_driver_can_compress(unfiltered_bs->drv)) { + error_report("Compression not supported for this file format"); + ret =3D -1; + goto out; + } else if (compress) { + write_flags |=3D BDRV_REQ_WRITE_COMPRESSED; + } + if (out_basefmt !=3D NULL) { if (bdrv_find_format(out_basefmt) =3D=3D NULL) { error_report("Invalid format name: '%s'", out_basefmt); @@ -3903,7 +3917,8 @@ static int img_rebase(int argc, char **argv) bdi.cluster_size); end =3D end > size ? size : end; ret =3D blk_pwrite(blk, start, end - start, - buf_old + (start - offset), 0); + buf_old + (start - offset), + write_flags); pnum =3D end - (offset + written); } if (ret < 0) { --=20 2.31.1 From nobody Sat May 4 16:31:08 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1685647799; cv=none; d=zohomail.com; s=zohoarc; b=ZdaWou2XoPKjRSc6aCJKcpcpV5JeMbZhCsuwa0meKwMdrj4cCbbVBbcchkz6usMNeYMKieNVOFDU4unOFOeSP3CE3oGhYUuk0eSg1HEcdjmX8fYuJMcw3Swl33R3js9FKQgZlC2/JW1xwex6WdjjbgWzHlUpC3e+heA+Lt297zo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1685647799; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=0lUK960nwT2Wy78rSEe7YdpLntqtsFNZA+ny0KnU3XA=; b=WQ97MDcGc29q50ucHi3gkYepiNc8S3kWz5qROK0kBwpfncI8ZvFVmVE1Q0QnhRFD79AQR6mcQMSvpaY+hjkWf7ptVvSO3U3K/iy2p/TIXfMmbL3Rz4o+oKbzkHSJHntKlh1U7LGFUoNk6JlqviPo+f213VX7kj6hHxfZA41Q+Vs= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1685647799726971.146398841891; Thu, 1 Jun 2023 12:29:59 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1q4nya-0006jx-L1; Thu, 01 Jun 2023 15:28:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyX-0006jU-UL; Thu, 01 Jun 2023 15:28:45 -0400 Received: from relay.virtuozzo.com ([130.117.225.111]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1q4nyV-0005mA-Bd; Thu, 01 Jun 2023 15:28:45 -0400 Received: from dev005.ch-qa.vzint.dev ([172.29.1.10]) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1q4nyC-00DLDg-1O; Thu, 01 Jun 2023 21:28:37 +0200 To: qemu-block@nongnu.org Cc: qemu-devel@nongnu.org, kwolf@redhat.com, hreitz@redhat.com, andrey.drobyshev@virtuozzo.com, den@virtuozzo.com Subject: [PATCH 6/6] iotests: add test 314 for "qemu-img rebase" with compression Date: Thu, 1 Jun 2023 22:28:36 +0300 Message-Id: <20230601192836.598602-7-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> References: <20230601192836.598602-1-andrey.drobyshev@virtuozzo.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=130.117.225.111; envelope-from=andrey.drobyshev@virtuozzo.com; helo=relay.virtuozzo.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Andrey Drobyshev From: Andrey Drobyshev via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1685647801079100021 Content-Type: text/plain; charset="utf-8" The test cases considered so far: 1. Check that compression mode isn't compatible with "-f raw" (raw format doesn't support compression). 2. Check that rebasing an image onto no backing file preserves the data and writes the copied clusters actually compressed. 3. Same as 2, but with a raw backing file (i.e. the clusters copied from the backing are originally uncompressed -- we check they end up compressed after being merged). 4. Remove a single delta from a backing chain, perform the same checks as in 2. 5. Check that even when backing and overlay are initially uncompressed, copied clusters end up compressed when rebase with compression is performed. Signed-off-by: Andrey Drobyshev Reviewed-by: Denis V. Lunev Reviewed-by: Hanna Czenczek --- tests/qemu-iotests/314 | 165 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/314.out | 75 +++++++++++++++++ 2 files changed, 240 insertions(+) create mode 100755 tests/qemu-iotests/314 create mode 100644 tests/qemu-iotests/314.out diff --git a/tests/qemu-iotests/314 b/tests/qemu-iotests/314 new file mode 100755 index 0000000000..96d7b4d258 --- /dev/null +++ b/tests/qemu-iotests/314 @@ -0,0 +1,165 @@ +#!/usr/bin/env bash +# group: rw backing auto quick +# +# Test qemu-img rebase with compression +# +# Copyright (c) 2023 Virtuozzo International GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +# creator +owner=3Dandrey.drobyshev@virtuozzo.com + +seq=3D`basename $0` +echo "QA output created by $seq" + +status=3D1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img + _rm_test_img "$TEST_IMG.base" + _rm_test_img "$TEST_IMG.itmd" +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +# Want the size divisible by 2 and 3 +size=3D$(( 48 * 1024 * 1024 )) +half_size=3D$(( size / 2 )) +third_size=3D$(( size / 3 )) + +# 1. "qemu-img rebase -c" should refuse working with any format which does= n't +# support compression. We only check "-f raw" here. +echo +echo "=3D=3D=3D Testing compressed rebase format compatibility =3D=3D=3D" +echo + +$QEMU_IMG create -f raw "$TEST_IMG" "$size" | _filter_img_create +$QEMU_IMG rebase -c -f raw -b "" "$TEST_IMG" + +# 2. Write the 1st half of $size to backing file (compressed), 2nd half --= to +# the top image (also compressed). Rebase the top image onto no backing f= ile, +# with compression (i.e. "qemu-img -c -b ''"). Check that the resulting i= mage +# has the written data preserved, and "qemu-img check" reports 100% cluste= rs +# as compressed. +echo +echo "=3D=3D=3D Testing rebase with compression onto no backing file =3D= =3D=3D" +echo + +TEST_IMG=3D"$TEST_IMG.base" _make_test_img $size +_make_test_img -b "$TEST_IMG.base" -F $IMGFMT $size + +$QEMU_IO -c "write -c -P 0xaa 0 $half_size" "$TEST_IMG.base" | _filter_qem= u_io +$QEMU_IO -c "write -c -P 0xbb $half_size $half_size" "$TEST_IMG" \ + | _filter_qemu_io + +$QEMU_IMG rebase -c -f $IMGFMT -b "" "$TEST_IMG" + +$QEMU_IO -c "read -P 0xaa 0 $half_size" "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -P 0xbb $half_size $half_size" "$TEST_IMG" | _filter_qem= u_io + +$QEMU_IMG check "$TEST_IMG" | _filter_testdir + +# 3. Same as the previous one, but with raw backing file (hence write to +# the backing is uncompressed). +echo +echo "=3D=3D=3D Testing rebase with compression with raw backing file =3D= =3D=3D" +echo + +$QEMU_IMG create -f raw "$TEST_IMG.base" "$half_size" | _filter_img_create +_make_test_img -b "$TEST_IMG.base" -F raw $size + +$QEMU_IO -f raw -c "write -P 0xaa 0 $half_size" "$TEST_IMG.base" \ + | _filter_qemu_io +$QEMU_IO -c "write -c -P 0xbb $half_size $half_size" \ + "$TEST_IMG" | _filter_qemu_io + +$QEMU_IMG rebase -c -f $IMGFMT -b "" "$TEST_IMG" + +$QEMU_IO -c "read -P 0xaa 0 $half_size" "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -P 0xbb $half_size $half_size" "$TEST_IMG" | _filter_qem= u_io + +$QEMU_IMG check "$TEST_IMG" | _filter_testdir + +# 4. Create a backing chain base<--itmd<--img, filling 1st, 2nd and 3rd +# thirds of them, respectively (with compression). Rebase img onto base, +# effectively deleting itmd from the chain, and check that written data is +# preserved in the resulting image. Also check that "qemu-img check" repo= rts +# 100% clusters as compressed. +echo +echo "=3D=3D=3D Testing compressed rebase removing single delta from the c= hain =3D=3D=3D" +echo + +TEST_IMG=3D"$TEST_IMG.base" _make_test_img $size +TEST_IMG=3D"$TEST_IMG.itmd" _make_test_img -b "$TEST_IMG.base" -F $IMGFMT = $size +_make_test_img -b "$TEST_IMG.itmd" -F $IMGFMT $size + +$QEMU_IO -c "write -c -P 0xaa 0 $third_size" \ + "$TEST_IMG.base" | _filter_qemu_io +$QEMU_IO -c "write -c -P 0xbb $third_size $third_size" \ + "$TEST_IMG.itmd" | _filter_qemu_io +$QEMU_IO -c "write -c -P 0xcc $((third_size * 2 )) $third_size" \ + "$TEST_IMG" | _filter_qemu_io + +$QEMU_IMG rebase -c -f $IMGFMT -b "$TEST_IMG.base" -F $IMGFMT "$TEST_IMG" + +$QEMU_IO -c "read -P 0xaa 0 $third_size" "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -P 0xbb $third_size $third_size" \ + "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -P 0xcc $(( third_size * 2 )) $third_size" \ + "$TEST_IMG" | _filter_qemu_io + +$QEMU_IMG check "$TEST_IMG" | _filter_testdir + +# 5. Create one-cluster backing and overlay images, and fill only the first +# (half - 1) bytes of the backing with data (uncompressed). Rebase the +# overlay onto no backing file with compression. Check that data is still +# read correctly, and that cluster is now really compressed ("qemu-img che= ck" +# reports 100% clusters as compressed. +echo +echo "=3D=3D=3D Testing compressed rebase with unaligned unmerged data =3D= =3D=3D" +echo + +CLUSTER_SIZE=3D65536 + +TEST_IMG=3D"$TEST_IMG.base" _make_test_img $CLUSTER_SIZE +_make_test_img -b "$TEST_IMG.base" -F $IMGFMT $CLUSTER_SIZE + +$QEMU_IO -c "write -P 0xaa 0 $(( CLUSTER_SIZE / 2 - 1 ))" $TEST_IMG.base \ + | _filter_qemu_io + +$QEMU_IMG rebase -c -f $IMGFMT -b "" "$TEST_IMG" + +$QEMU_IO -c "read -P 0xaa 0 $(( CLUSTER_SIZE / 2 - 1 ))" "$TEST_IMG" \ + | _filter_qemu_io +$QEMU_IO -c \ + "read -P 0x00 $(( CLUSTER_SIZE / 2 - 1 )) $(( CLUSTER_SIZE / 2 + 1 ))"= \ + "$TEST_IMG" | _filter_qemu_io + +$QEMU_IMG check "$TEST_IMG" | _filter_testdir + +# success, all done +echo +echo '*** done' +rm -f $seq.full +status=3D0 diff --git a/tests/qemu-iotests/314.out b/tests/qemu-iotests/314.out new file mode 100644 index 0000000000..ac9337a543 --- /dev/null +++ b/tests/qemu-iotests/314.out @@ -0,0 +1,75 @@ +QA output created by 314 + +=3D=3D=3D Testing compressed rebase format compatibility =3D=3D=3D + +Formatting 'TEST_DIR/t.IMGFMT', fmt=3Draw size=3D50331648 +qemu-img: Compression not supported for this file format + +=3D=3D=3D Testing rebase with compression onto no backing file =3D=3D=3D + +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=3DIMGFMT size=3D50331648 +Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D50331648 backing_file= =3DTEST_DIR/t.IMGFMT.base backing_fmt=3DIMGFMT +wrote 25165824/25165824 bytes at offset 0 +24 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 25165824/25165824 bytes at offset 25165824 +24 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 25165824/25165824 bytes at offset 0 +24 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 25165824/25165824 bytes at offset 25165824 +24 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +No errors were found on the image. +768/768 =3D 100.00% allocated, 100.00% fragmented, 100.00% compressed clus= ters +Image end offset: 458752 + +=3D=3D=3D Testing rebase with compression with raw backing file =3D=3D=3D + +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=3Draw size=3D25165824 +Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D50331648 backing_file= =3DTEST_DIR/t.IMGFMT.base backing_fmt=3Draw +wrote 25165824/25165824 bytes at offset 0 +24 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 25165824/25165824 bytes at offset 25165824 +24 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 25165824/25165824 bytes at offset 0 +24 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 25165824/25165824 bytes at offset 25165824 +24 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +No errors were found on the image. +768/768 =3D 100.00% allocated, 100.00% fragmented, 100.00% compressed clus= ters +Image end offset: 458752 + +=3D=3D=3D Testing compressed rebase removing single delta from the chain = =3D=3D=3D + +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=3DIMGFMT size=3D50331648 +Formatting 'TEST_DIR/t.IMGFMT.itmd', fmt=3DIMGFMT size=3D50331648 backing_= file=3DTEST_DIR/t.IMGFMT.base backing_fmt=3DIMGFMT +Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D50331648 backing_file= =3DTEST_DIR/t.IMGFMT.itmd backing_fmt=3DIMGFMT +wrote 16777216/16777216 bytes at offset 0 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16777216/16777216 bytes at offset 16777216 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 16777216/16777216 bytes at offset 33554432 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 16777216/16777216 bytes at offset 0 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 16777216/16777216 bytes at offset 16777216 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 16777216/16777216 bytes at offset 33554432 +16 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +No errors were found on the image. +512/768 =3D 66.67% allocated, 100.00% fragmented, 100.00% compressed clust= ers +Image end offset: 458752 + +=3D=3D=3D Testing compressed rebase with unaligned unmerged data =3D=3D=3D + +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=3DIMGFMT size=3D65536 +Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D65536 backing_file=3DT= EST_DIR/t.IMGFMT.base backing_fmt=3DIMGFMT +wrote 32767/32767 bytes at offset 0 +31.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 32767/32767 bytes at offset 0 +31.999 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 32769/32769 bytes at offset 32767 +32.001 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +No errors were found on the image. +1/1 =3D 100.00% allocated, 100.00% fragmented, 100.00% compressed clusters +Image end offset: 393216 + +*** done --=20 2.31.1