From nobody Sat May 18 21:00:27 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 162689693878697.63722609291028; Wed, 21 Jul 2021 12:48:58 -0700 (PDT) Received: from localhost ([::1]:35758 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6IDB-0001Au-Cs for importer@patchew.org; Wed, 21 Jul 2021 15:48:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34658) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6IBw-0007pS-Mj for qemu-devel@nongnu.org; Wed, 21 Jul 2021 15:47:40 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:28751) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6IBu-0005Vz-0W for qemu-devel@nongnu.org; Wed, 21 Jul 2021 15:47:40 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-440-KOpm-mqCNVKydgGw60PwOA-1; Wed, 21 Jul 2021 15:47:34 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A5AE910074A0; Wed, 21 Jul 2021 19:47:33 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-216.phx2.redhat.com [10.3.112.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id E237369CB6; Wed, 21 Jul 2021 19:47:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626896856; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ShHzm8W9B0322gtrUqASV/e1SWapqxmeqYr4Ku/w3Ag=; b=cBEDsy7yxaPjhbPf3OEVhFtKsY1k2cuhqRIQ6zeXcid8kKkOdlvRYPXfWjC6+zjaVEIyA0 MjBiHGq/UMKxoxohjh8q9wyry+/bp3tEflP/53McMc8/kuANyGphcG01Vk5LoReJW/sNWf I4uFjKQC4NM8eM7lKCZV8UaJh993shw= X-MC-Unique: KOpm-mqCNVKydgGw60PwOA-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 1/3] iotests: Improve and rename test 291 to qemu-img-bitmap Date: Wed, 21 Jul 2021 14:47:27 -0500 Message-Id: <20210721194729.648763-2-eblake@redhat.com> In-Reply-To: <20210721194729.648763-1-eblake@redhat.com> References: <20210721194729.648763-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eblake@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -23 X-Spam_score: -2.4 X-Spam_bar: -- X-Spam_report: (-2.4 / 5.0 requ) DKIMWL_WL_HIGH=-1.459, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Dirty Bitmaps" , Max Reitz , Nir Soffer , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1626896944619100001 Content-Type: text/plain; charset="utf-8" Enhance the test to demonstrate existing less-than-stellar behavior of qemu-img with a qcow2 image containing an inconsistent bitmap: we don't diagnose the problem until after copying the entire image (a potentially long time), and when we do diagnose the failure, we still end up leaving an empty bitmap in the destination. This mess will be cleaned up in the next patch. While at it, rename the test now that we support useful iotest names, and fix a missing newline in the error message thus exposed. Signed-off-by: Eric Blake Message-Id: <20210709153951.2801666-2-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Nir Soffer --- block/dirty-bitmap.c | 2 +- .../{291 =3D> tests/qemu-img-bitmaps} | 21 +++++++- .../{291.out =3D> tests/qemu-img-bitmaps.out} | 49 ++++++++++++++++++- 3 files changed, 69 insertions(+), 3 deletions(-) rename tests/qemu-iotests/{291 =3D> tests/qemu-img-bitmaps} (87%) rename tests/qemu-iotests/{291.out =3D> tests/qemu-img-bitmaps.out} (76%) diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 68d295d6e3ed..0ef46163e3ea 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -193,7 +193,7 @@ int bdrv_dirty_bitmap_check(const BdrvDirtyBitmap *bitm= ap, uint32_t flags, error_setg(errp, "Bitmap '%s' is inconsistent and cannot be used", bitmap->name); error_append_hint(errp, "Try block-dirty-bitmap-remove to delete" - " this bitmap from disk"); + " this bitmap from disk\n"); return -1; } diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/tests/qemu-img-bit= maps similarity index 87% rename from tests/qemu-iotests/291 rename to tests/qemu-iotests/tests/qemu-img-bitmaps index 20efb080a6c0..409c4497a303 100755 --- a/tests/qemu-iotests/291 +++ b/tests/qemu-iotests/tests/qemu-img-bitmaps @@ -3,7 +3,7 @@ # # Test qemu-img bitmap handling # -# Copyright (C) 2018-2020 Red Hat, Inc. +# Copyright (C) 2018-2021 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -27,11 +27,13 @@ status=3D1 # failure is the default! _cleanup() { _cleanup_test_img + _rm_test_img "$TEST_IMG.copy" nbd_server_stop } trap "_cleanup; exit \$status" 0 1 2 3 15 # get standard environment, filters and checks +cd .. . ./common.rc . ./common.filter . ./common.nbd @@ -129,6 +131,23 @@ $QEMU_IMG map --output=3Djson --image-opts \ nbd_server_stop +echo +echo "=3D=3D=3D Check handling of inconsistent bitmap =3D=3D=3D" +echo + +# Prepare image with corrupted bitmap +$QEMU_IO -c abort "$TEST_IMG" 2>/dev/null +$QEMU_IMG bitmap --add "$TEST_IMG" b4 +$QEMU_IMG bitmap --remove "$TEST_IMG" b1 +_img_info --format-specific | _filter_irrelevant_img_info +echo +$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG" "$TEST_IMG.copy" && + echo "unexpected success" +# Bug - even though we failed at conversion, we left a file around with +# a bitmap marked as not corrupt +TEST_IMG=3D$TEST_IMG.copy _img_info --format-specific \ + | _filter_irrelevant_img_info + # success, all done echo '*** done' rm -f $seq.full diff --git a/tests/qemu-iotests/291.out b/tests/qemu-iotests/tests/qemu-img= -bitmaps.out similarity index 76% rename from tests/qemu-iotests/291.out rename to tests/qemu-iotests/tests/qemu-img-bitmaps.out index 018d6b103f87..6824d4112893 100644 --- a/tests/qemu-iotests/291.out +++ b/tests/qemu-iotests/tests/qemu-img-bitmaps.out @@ -1,4 +1,4 @@ -QA output created by 291 +QA output created by qemu-img-bitmaps =3D=3D=3D Initial image setup =3D=3D=3D @@ -115,4 +115,51 @@ Format specific information: [{ "start": 0, "length": 2097152, "depth": 0, "present": true, "zero": fal= se, "data": true, "offset": OFFSET}, { "start": 2097152, "length": 1048576, "depth": 0, "present": false, "zero= ": false, "data": false}, { "start": 3145728, "length": 7340032, "depth": 0, "present": true, "zero"= : false, "data": true, "offset": OFFSET}] + +=3D=3D=3D Check handling of inconsistent bitmap =3D=3D=3D + +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 10 MiB (10485760 bytes) +cluster_size: 65536 +backing file: TEST_DIR/t.IMGFMT.base +backing file format: IMGFMT +Format specific information: + bitmaps: + [0]: + flags: + [0]: in-use + [1]: auto + name: b2 + granularity: 65536 + [1]: + flags: + [0]: in-use + name: b0 + granularity: 65536 + [2]: + flags: + [0]: auto + name: b4 + granularity: 65536 + corrupt: false + +qemu-img: Failed to populate bitmap b0: Bitmap 'b0' is inconsistent and ca= nnot be used +Try block-dirty-bitmap-remove to delete this bitmap from disk +image: TEST_DIR/t.IMGFMT.copy +file format: IMGFMT +virtual size: 10 MiB (10485760 bytes) +cluster_size: 65536 +Format specific information: + bitmaps: + [0]: + flags: + name: b0 + granularity: 65536 + [1]: + flags: + [0]: auto + name: b4 + granularity: 65536 + corrupt: false *** done --=20 2.31.1 From nobody Sat May 18 21:00:27 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1626897058218681.8482199410597; Wed, 21 Jul 2021 12:50:58 -0700 (PDT) Received: from localhost ([::1]:39908 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6IF7-0003zg-7C for importer@patchew.org; Wed, 21 Jul 2021 15:50:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34680) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6IBy-0007sa-3F for qemu-devel@nongnu.org; Wed, 21 Jul 2021 15:47:42 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:51387) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6IBt-0005W5-Tn for qemu-devel@nongnu.org; Wed, 21 Jul 2021 15:47:41 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-588-4vWEpxpWP8KD5PxdRDjCEg-1; Wed, 21 Jul 2021 15:47:35 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5DF321084F53; Wed, 21 Jul 2021 19:47:34 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-216.phx2.redhat.com [10.3.112.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id DB43D5B826; Wed, 21 Jul 2021 19:47:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626896857; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xgHU0baUWYZ4QaIYYCK4Rz7QcI7QRxus2fnGSVZziqA=; b=hVs8w+qtbsirn+p9yyX4r52rDte23AiFAk4Mp1FaBOd/wfWx12X74ZxgvpQQar1LsmCf7j suBglOWOY+n5PcCu4E7ELr+uYr1TKe9o3pv8aQWejWgn+u0wsp/XKqNhOXtTuWqdUNpk6+ TZc3KMGYBBvOXowUG8OzuYHH0fFFz6g= X-MC-Unique: 4vWEpxpWP8KD5PxdRDjCEg-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 2/3] qemu-img: Fail fast on convert --bitmaps with inconsistent bitmap Date: Wed, 21 Jul 2021 14:47:28 -0500 Message-Id: <20210721194729.648763-3-eblake@redhat.com> In-Reply-To: <20210721194729.648763-1-eblake@redhat.com> References: <20210721194729.648763-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eblake@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.459, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nir Soffer , Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1626897059513100001 Content-Type: text/plain; charset="utf-8" Waiting until the end of the convert operation (a potentially time-consuming task) to finally detect that we can't copy a bitmap is bad, comparing to failing fast up front. Furthermore, this prevents us from leaving a file behind with a bitmap that is not marked as inconsistent even though it does not have sane contents. This fixes the problems exposed in the previous patch to the iotest: it adds a fast failure up front, and even if we don't fail early, it ensures that any bitmap we add but do not properly populate is removed again rather than left behind incomplete. Signed-off-by: Eric Blake Message-Id: <20210709153951.2801666-3-eblake@redhat.com> [eblake: add a hint to the warning message, simplify name computation] Reviewed-by: Nir Soffer Reviewed-by: Vladimir Sementsov-Ogievskiy --- qemu-img.c | 29 +++++++++++++++++-- tests/qemu-iotests/tests/qemu-img-bitmaps | 3 +- tests/qemu-iotests/tests/qemu-img-bitmaps.out | 21 ++------------ 3 files changed, 30 insertions(+), 23 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index 797742a44331..c5496e82e025 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -2101,6 +2101,29 @@ static int convert_do_copy(ImgConvertState *s) return s->ret; } +/* Check that bitmaps can be copied, or output an error */ +static int convert_check_bitmaps(BlockDriverState *src) +{ + BdrvDirtyBitmap *bm; + + if (!bdrv_supports_persistent_dirty_bitmap(src)) { + error_report("Source lacks bitmap support"); + return -1; + } + FOR_EACH_DIRTY_BITMAP(src, bm) { + if (!bdrv_dirty_bitmap_get_persistence(bm)) { + continue; + } + if (bdrv_dirty_bitmap_inconsistent(bm)) { + error_report("Cannot copy inconsistent bitmap '%s'", + bdrv_dirty_bitmap_name(bm)); + error_printf("Try 'qemu-img bitmap --remove' to delete it\n"); + return -1; + } + } + return 0; +} + static int convert_copy_bitmaps(BlockDriverState *src, BlockDriverState *d= st) { BdrvDirtyBitmap *bm; @@ -2127,6 +2150,7 @@ static int convert_copy_bitmaps(BlockDriverState *src= , BlockDriverState *dst) &err); if (err) { error_reportf_err(err, "Failed to populate bitmap %s: ", name); + qmp_block_dirty_bitmap_remove(dst->node_name, name, NULL); return -1; } } @@ -2554,9 +2578,8 @@ static int img_convert(int argc, char **argv) ret =3D -1; goto out; } - if (!bdrv_supports_persistent_dirty_bitmap(blk_bs(s.src[0]))) { - error_report("Source lacks bitmap support"); - ret =3D -1; + ret =3D convert_check_bitmaps(blk_bs(s.src[0])); + if (ret < 0) { goto out; } } diff --git a/tests/qemu-iotests/tests/qemu-img-bitmaps b/tests/qemu-iotests= /tests/qemu-img-bitmaps index 409c4497a303..09c3d395d1eb 100755 --- a/tests/qemu-iotests/tests/qemu-img-bitmaps +++ b/tests/qemu-iotests/tests/qemu-img-bitmaps @@ -140,11 +140,10 @@ $QEMU_IO -c abort "$TEST_IMG" 2>/dev/null $QEMU_IMG bitmap --add "$TEST_IMG" b4 $QEMU_IMG bitmap --remove "$TEST_IMG" b1 _img_info --format-specific | _filter_irrelevant_img_info +# Proof that we fail fast if bitmaps can't be copied echo $QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG" "$TEST_IMG.copy" && echo "unexpected success" -# Bug - even though we failed at conversion, we left a file around with -# a bitmap marked as not corrupt TEST_IMG=3D$TEST_IMG.copy _img_info --format-specific \ | _filter_irrelevant_img_info diff --git a/tests/qemu-iotests/tests/qemu-img-bitmaps.out b/tests/qemu-iot= ests/tests/qemu-img-bitmaps.out index 6824d4112893..d99c279d0c63 100644 --- a/tests/qemu-iotests/tests/qemu-img-bitmaps.out +++ b/tests/qemu-iotests/tests/qemu-img-bitmaps.out @@ -144,22 +144,7 @@ Format specific information: granularity: 65536 corrupt: false -qemu-img: Failed to populate bitmap b0: Bitmap 'b0' is inconsistent and ca= nnot be used -Try block-dirty-bitmap-remove to delete this bitmap from disk -image: TEST_DIR/t.IMGFMT.copy -file format: IMGFMT -virtual size: 10 MiB (10485760 bytes) -cluster_size: 65536 -Format specific information: - bitmaps: - [0]: - flags: - name: b0 - granularity: 65536 - [1]: - flags: - [0]: auto - name: b4 - granularity: 65536 - corrupt: false +qemu-img: Cannot copy inconsistent bitmap 'b0' +Try 'qemu-img bitmap --remove' to delete it +qemu-img: Could not open 'TEST_DIR/t.IMGFMT.copy': Could not open 'TEST_DI= R/t.IMGFMT.copy': No such file or directory *** done --=20 2.31.1 From nobody Sat May 18 21:00:27 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1626897061086495.9616851787513; Wed, 21 Jul 2021 12:51:01 -0700 (PDT) Received: from localhost ([::1]:40084 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m6IF9-000476-Pv for importer@patchew.org; Wed, 21 Jul 2021 15:50:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34676) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6IBx-0007qD-Bf for qemu-devel@nongnu.org; Wed, 21 Jul 2021 15:47:41 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22556) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m6IBu-0005Wc-Vc for qemu-devel@nongnu.org; Wed, 21 Jul 2021 15:47:41 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-333-TOFz3-53MP2XtGN5qjkpmw-1; Wed, 21 Jul 2021 15:47:36 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1DC5610074AD; Wed, 21 Jul 2021 19:47:35 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-216.phx2.redhat.com [10.3.112.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id A71C45B826; Wed, 21 Jul 2021 19:47:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1626896858; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m0O4EeuVB/jxo4OlDa6gIe+VeXSD0qINDj7XDl9FSkk=; b=h8KG1KbKrtdWY/kg+ZFXqYBm9gOrYxbpA9tFhqXfuLg67COC/FTMWtRUl1Vv/LPE6lCn84 SEDSsMkcY4TdTgGblcOG2DB3zOaPev8CXoXgYOJ2HyHW7vKLoJdPS2GyYH9jPjIn60LJcY krxpFniSGWwb211jLK/gCaRpiUWVfNc= X-MC-Unique: TOFz3-53MP2XtGN5qjkpmw-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 3/3] qemu-img: Add --skip-broken-bitmaps for 'convert --bitmaps' Date: Wed, 21 Jul 2021 14:47:29 -0500 Message-Id: <20210721194729.648763-4-eblake@redhat.com> In-Reply-To: <20210721194729.648763-1-eblake@redhat.com> References: <20210721194729.648763-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eblake@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.459, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1626897062081100001 Content-Type: text/plain; charset="utf-8" The point of 'qemu-img convert --bitmaps' is to be a convenience for actions that are already possible through a string of smaller 'qemu-img bitmap' sub-commands. One situation not accounted for already is that if a source image contains an inconsistent bitmap (for example, because a qemu process died abruptly before flushing bitmap state), the user MUST delete those inconsistent bitmaps before anything else useful can be done with the image. We don't want to delete inconsistent bitmaps by default: although a corrupt bitmap is only a loss of optimization rather than a corruption of user-visible data, it is still nice to require the user to opt in to the fact that they are aware of the loss of the bitmap. Still, requiring the user to check 'qemu-img info' to see whether bitmaps are consistent, then use 'qemu-img bitmap --remove' to remove offenders, all before using 'qemu-img convert', is a lot more work than just adding a knob 'qemu-img convert --bitmaps --skip-broken-bitmaps' which opts in to skipping the broken bitmaps. After testing the new option, also demonstrate the way to manually fix things (either deleting bad bitmaps, or re-creating them as empty) so that it is possible to convert without the option. Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=3D1946084 Signed-off-by: Eric Blake Message-Id: <20210709153951.2801666-4-eblake@redhat.com> [eblake: warning message tweak, test enhancements] Reviewed-by: Vladimir Sementsov-Ogievskiy --- docs/tools/qemu-img.rst | 8 ++++- qemu-img.c | 29 +++++++++++---- tests/qemu-iotests/tests/qemu-img-bitmaps | 16 ++++++++- tests/qemu-iotests/tests/qemu-img-bitmaps.out | 35 ++++++++++++++++++- 4 files changed, 79 insertions(+), 9 deletions(-) diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst index 1d8470eada0e..b7d602a28804 100644 --- a/docs/tools/qemu-img.rst +++ b/docs/tools/qemu-img.rst @@ -414,7 +414,7 @@ Command description: 4 Error on reading data -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-op= ts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [= -t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l= SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-r RATE_LIMIT] [-m NUM_COROUTINES] [-W] = FILENAME [FILENAME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-op= ts] [--target-is-zero] [--bitmaps [--skip-broken-bitmaps]] [-U] [-C] [-c] [= -p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKIN= G_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-r RATE_LIMIT] [= -m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM* to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can @@ -456,6 +456,12 @@ Command description: *NUM_COROUTINES* specifies how many coroutines work in parallel during the convert process (defaults to 8). + Use of ``--bitmaps`` requests that any persistent bitmaps present in + the original are also copied to the destination. If any bitmap is + inconsistent in the source, the conversion will fail unless + ``--skip-broken-bitmaps`` is also specified to copy only the + consistent bitmaps. + .. option:: create [--object OBJECTDEF] [-q] [-f FMT] [-b BACKING_FILE] [-= F BACKING_FMT] [-u] [-o OPTIONS] FILENAME [SIZE] Create the new disk image *FILENAME* of size *SIZE* and format diff --git a/qemu-img.c b/qemu-img.c index c5496e82e025..908fd0cce5b4 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -82,6 +82,7 @@ enum { OPTION_MERGE =3D 274, OPTION_BITMAPS =3D 275, OPTION_FORCE =3D 276, + OPTION_SKIP_BROKEN =3D 277, }; typedef enum OutputFormat { @@ -2102,7 +2103,7 @@ static int convert_do_copy(ImgConvertState *s) } /* Check that bitmaps can be copied, or output an error */ -static int convert_check_bitmaps(BlockDriverState *src) +static int convert_check_bitmaps(BlockDriverState *src, bool skip_broken) { BdrvDirtyBitmap *bm; @@ -2114,17 +2115,19 @@ static int convert_check_bitmaps(BlockDriverState *= src) if (!bdrv_dirty_bitmap_get_persistence(bm)) { continue; } - if (bdrv_dirty_bitmap_inconsistent(bm)) { + if (!skip_broken && bdrv_dirty_bitmap_inconsistent(bm)) { error_report("Cannot copy inconsistent bitmap '%s'", bdrv_dirty_bitmap_name(bm)); - error_printf("Try 'qemu-img bitmap --remove' to delete it\n"); + error_printf("Try --skip-broken-bitmaps, or " + "use 'qemu-img bitmap --remove' to delete it\n"); return -1; } } return 0; } -static int convert_copy_bitmaps(BlockDriverState *src, BlockDriverState *d= st) +static int convert_copy_bitmaps(BlockDriverState *src, BlockDriverState *d= st, + bool skip_broken) { BdrvDirtyBitmap *bm; Error *err =3D NULL; @@ -2136,6 +2139,10 @@ static int convert_copy_bitmaps(BlockDriverState *sr= c, BlockDriverState *dst) continue; } name =3D bdrv_dirty_bitmap_name(bm); + if (skip_broken && bdrv_dirty_bitmap_inconsistent(bm)) { + warn_report("Skipping inconsistent bitmap '%s'", name); + continue; + } qmp_block_dirty_bitmap_add(dst->node_name, name, true, bdrv_dirty_bitmap_granularity(bm), true, true, @@ -2191,6 +2198,7 @@ static int img_convert(int argc, char **argv) bool force_share =3D false; bool explict_min_sparse =3D false; bool bitmaps =3D false; + bool skip_broken =3D false; int64_t rate_limit =3D 0; ImgConvertState s =3D (ImgConvertState) { @@ -2212,6 +2220,7 @@ static int img_convert(int argc, char **argv) {"salvage", no_argument, 0, OPTION_SALVAGE}, {"target-is-zero", no_argument, 0, OPTION_TARGET_IS_ZERO}, {"bitmaps", no_argument, 0, OPTION_BITMAPS}, + {"skip-broken-bitmaps", no_argument, 0, OPTION_SKIP_BROKEN}, {0, 0, 0, 0} }; c =3D getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WUr:", @@ -2340,6 +2349,9 @@ static int img_convert(int argc, char **argv) case OPTION_BITMAPS: bitmaps =3D true; break; + case OPTION_SKIP_BROKEN: + skip_broken =3D true; + break; } } @@ -2347,6 +2359,11 @@ static int img_convert(int argc, char **argv) out_fmt =3D "raw"; } + if (skip_broken && !bitmaps) { + error_report("Use of --skip-broken-bitmaps requires --bitmaps"); + goto fail_getopt; + } + if (s.compressed && s.copy_range) { error_report("Cannot enable copy offloading when -c is used"); goto fail_getopt; @@ -2578,7 +2595,7 @@ static int img_convert(int argc, char **argv) ret =3D -1; goto out; } - ret =3D convert_check_bitmaps(blk_bs(s.src[0])); + ret =3D convert_check_bitmaps(blk_bs(s.src[0]), skip_broken); if (ret < 0) { goto out; } @@ -2703,7 +2720,7 @@ static int img_convert(int argc, char **argv) /* Now copy the bitmaps */ if (bitmaps && ret =3D=3D 0) { - ret =3D convert_copy_bitmaps(blk_bs(s.src[0]), out_bs); + ret =3D convert_copy_bitmaps(blk_bs(s.src[0]), out_bs, skip_broken= ); } out: diff --git a/tests/qemu-iotests/tests/qemu-img-bitmaps b/tests/qemu-iotests= /tests/qemu-img-bitmaps index 09c3d395d1eb..7a3fe8c3d37a 100755 --- a/tests/qemu-iotests/tests/qemu-img-bitmaps +++ b/tests/qemu-iotests/tests/qemu-img-bitmaps @@ -144,7 +144,21 @@ _img_info --format-specific | _filter_irrelevant_img_i= nfo echo $QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG" "$TEST_IMG.copy" && echo "unexpected success" -TEST_IMG=3D$TEST_IMG.copy _img_info --format-specific \ +TEST_IMG=3D"$TEST_IMG.copy" _img_info --format-specific \ + | _filter_irrelevant_img_info +# Skipping the broken bitmaps works,... +echo +$QEMU_IMG convert --bitmaps --skip-broken-bitmaps \ + -O qcow2 "$TEST_IMG" "$TEST_IMG.copy" +TEST_IMG=3D"$TEST_IMG.copy" _img_info --format-specific \ + | _filter_irrelevant_img_info +# ...as does removing them +echo +_rm_test_img "$TEST_IMG.copy" +$QEMU_IMG bitmap --remove "$TEST_IMG" b0 +$QEMU_IMG bitmap --remove --add "$TEST_IMG" b2 +$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG" "$TEST_IMG.copy" +TEST_IMG=3D"$TEST_IMG.copy" _img_info --format-specific \ | _filter_irrelevant_img_info # success, all done diff --git a/tests/qemu-iotests/tests/qemu-img-bitmaps.out b/tests/qemu-iot= ests/tests/qemu-img-bitmaps.out index d99c279d0c63..e851f0320ecb 100644 --- a/tests/qemu-iotests/tests/qemu-img-bitmaps.out +++ b/tests/qemu-iotests/tests/qemu-img-bitmaps.out @@ -145,6 +145,39 @@ Format specific information: corrupt: false qemu-img: Cannot copy inconsistent bitmap 'b0' -Try 'qemu-img bitmap --remove' to delete it +Try --skip-broken-bitmaps, or use 'qemu-img bitmap --remove' to delete it qemu-img: Could not open 'TEST_DIR/t.IMGFMT.copy': Could not open 'TEST_DI= R/t.IMGFMT.copy': No such file or directory + +qemu-img: warning: Skipping inconsistent bitmap 'b0' +qemu-img: warning: Skipping inconsistent bitmap 'b2' +image: TEST_DIR/t.IMGFMT.copy +file format: IMGFMT +virtual size: 10 MiB (10485760 bytes) +cluster_size: 65536 +Format specific information: + bitmaps: + [0]: + flags: + [0]: auto + name: b4 + granularity: 65536 + corrupt: false + +image: TEST_DIR/t.IMGFMT.copy +file format: IMGFMT +virtual size: 10 MiB (10485760 bytes) +cluster_size: 65536 +Format specific information: + bitmaps: + [0]: + flags: + [0]: auto + name: b4 + granularity: 65536 + [1]: + flags: + [0]: auto + name: b2 + granularity: 65536 + corrupt: false *** done --=20 2.31.1