From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511420; cv=none; d=zohomail.com; s=zohoarc; b=NkdSJ9qnsbf/DDycxfYMUBWqWHz+j0CcG8zhZFg7ur4119vrTXPwS1gdI9Dt0FoYL+ukcijqeveRe/swueXPamwUZBYRTptBzWOeAcEaoygcz2pIYnf2x67vLRJXG5T/6gY/h9TDkopwoLnBHfhuasGpr1lt5HvLNH8Uh4+GB2g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511420; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=K9IEHoDlJS+FjDytuoMfolDjl7ZsoI2ZIWaE7cuA0UY=; b=l0fgbsVkxfQrTTSW+2nZAGX4KyZRZWrZAi3roZNwp1W73bnPswK/G4lg01QqPOg//ox0mtRF+Zevca3bpTB/xS4xFL9Q6HJoMtvJLhtBybgmnt8DA0Ucp38Z9bSOsQ+1nG67u9OL6RWgTGKlI+ovXQlHVQdLVH+Sxf1kjvycJc4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1590511420140643.6385027980944; Tue, 26 May 2020 09:43:40 -0700 (PDT) Received: from localhost ([::1]:48224 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdcfy-0006HT-MB for importer@patchew.org; Tue, 26 May 2020 12:43:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41288) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdceq-0004YQ-W7 for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:29 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:60817 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdcep-0002qL-4n for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:28 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-413-OxiSAE20Mq-sBsu8QGaqeQ-1; Tue, 26 May 2020 12:42:24 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0EA1E460; Tue, 26 May 2020 16:42:23 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5204D86C7E; Tue, 26 May 2020 16:42:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511346; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K9IEHoDlJS+FjDytuoMfolDjl7ZsoI2ZIWaE7cuA0UY=; b=JW7DB0/iOimD1qDOmepgbR7eCkv6cazfJ6XpDbDFwvvrKtZ4NXhYV3dVVvl3Kb9iJkFyhm zkUppxqAgKSpY1Em46cY8GgXCOL21Vm/Aef+b7xJWxKjw+v2dJHPlUiv7Uzrg+DExiWQBC jhlCWukBef82oaena+BY0emWj7wXIdU= X-MC-Unique: OxiSAE20Mq-sBsu8QGaqeQ-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 01/11] migration: refactor init_dirty_bitmap_migration Date: Tue, 26 May 2020 11:42:01 -0500 Message-Id: <20200526164211.1569366-2-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=207.211.31.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 01:51:57 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Vladimir Sementsov-Ogievskiy , "open list:Block I/O path" , Juan Quintela , "Dr. David Alan Gilbert" , Stefan Hajnoczi , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Split out handling one bs, it is needed for the following commit, which will handle BlockBackends separately. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Message-Id: <20200521220648.3255-2-vsementsov@virtuozzo.com> [eblake: shorter subject line] Signed-off-by: Eric Blake --- migration/block-dirty-bitmap.c | 89 +++++++++++++++++++--------------- 1 file changed, 49 insertions(+), 40 deletions(-) diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c index 7eafface6149..7e9371808653 100644 --- a/migration/block-dirty-bitmap.c +++ b/migration/block-dirty-bitmap.c @@ -267,13 +267,58 @@ static void dirty_bitmap_mig_cleanup(void) } } +/* Called with iothread lock taken. */ +static int add_bitmaps_to_list(BlockDriverState *bs, const char *bs_name) +{ + BdrvDirtyBitmap *bitmap; + DirtyBitmapMigBitmapState *dbms; + Error *local_err =3D NULL; + + FOR_EACH_DIRTY_BITMAP(bs, bitmap) { + if (!bdrv_dirty_bitmap_name(bitmap)) { + continue; + } + + if (!bs_name || strcmp(bs_name, "") =3D=3D 0) { + error_report("Found bitmap '%s' in unnamed node %p. It can't " + "be migrated", bdrv_dirty_bitmap_name(bitmap), bs= ); + return -1; + } + + if (bdrv_dirty_bitmap_check(bitmap, BDRV_BITMAP_DEFAULT, &local_er= r)) { + error_report_err(local_err); + return -1; + } + + bdrv_ref(bs); + bdrv_dirty_bitmap_set_busy(bitmap, true); + + dbms =3D g_new0(DirtyBitmapMigBitmapState, 1); + dbms->bs =3D bs; + dbms->node_name =3D bs_name; + dbms->bitmap =3D bitmap; + dbms->total_sectors =3D bdrv_nb_sectors(bs); + dbms->sectors_per_chunk =3D CHUNK_SIZE * 8 * + bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS; + if (bdrv_dirty_bitmap_enabled(bitmap)) { + dbms->flags |=3D DIRTY_BITMAP_MIG_START_FLAG_ENABLED; + } + if (bdrv_dirty_bitmap_get_persistence(bitmap)) { + dbms->flags |=3D DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT; + } + + QSIMPLEQ_INSERT_TAIL(&dirty_bitmap_mig_state.dbms_list, + dbms, entry); + } + + return 0; +} + /* Called with iothread lock taken. */ static int init_dirty_bitmap_migration(void) { BlockDriverState *bs; - BdrvDirtyBitmap *bitmap; DirtyBitmapMigBitmapState *dbms; - Error *local_err =3D NULL; dirty_bitmap_mig_state.bulk_completed =3D false; dirty_bitmap_mig_state.prev_bs =3D NULL; @@ -281,44 +326,8 @@ static int init_dirty_bitmap_migration(void) dirty_bitmap_mig_state.no_bitmaps =3D false; for (bs =3D bdrv_next_all_states(NULL); bs; bs =3D bdrv_next_all_state= s(bs)) { - const char *name =3D bdrv_get_device_or_node_name(bs); - - FOR_EACH_DIRTY_BITMAP(bs, bitmap) { - if (!bdrv_dirty_bitmap_name(bitmap)) { - continue; - } - - if (!name || strcmp(name, "") =3D=3D 0) { - error_report("Found bitmap '%s' in unnamed node %p. It can= 't " - "be migrated", bdrv_dirty_bitmap_name(bitmap)= , bs); - goto fail; - } - - if (bdrv_dirty_bitmap_check(bitmap, BDRV_BITMAP_DEFAULT, - &local_err)) { - error_report_err(local_err); - goto fail; - } - - bdrv_ref(bs); - bdrv_dirty_bitmap_set_busy(bitmap, true); - - dbms =3D g_new0(DirtyBitmapMigBitmapState, 1); - dbms->bs =3D bs; - dbms->node_name =3D name; - dbms->bitmap =3D bitmap; - dbms->total_sectors =3D bdrv_nb_sectors(bs); - dbms->sectors_per_chunk =3D CHUNK_SIZE * 8 * - bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS; - if (bdrv_dirty_bitmap_enabled(bitmap)) { - dbms->flags |=3D DIRTY_BITMAP_MIG_START_FLAG_ENABLED; - } - if (bdrv_dirty_bitmap_get_persistence(bitmap)) { - dbms->flags |=3D DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT; - } - - QSIMPLEQ_INSERT_TAIL(&dirty_bitmap_mig_state.dbms_list, - dbms, entry); + if (add_bitmaps_to_list(bs, bdrv_get_device_or_node_name(bs))) { + goto fail; } } --=20 2.26.2 From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511488; cv=none; d=zohomail.com; s=zohoarc; b=MJ4Kucw/3yAz+rKfYRHX/OHQXUNIf/lfTt84YsDgq6TvmGBruKkmO/U5g/eSqL7HF2Q7X9pMum6kepur2Sek5likaeuaiUWsIfFfCwymzPjOVqGFFzDrcwONJzToWj1DUthxUG8sGPN0bDLf94go6k6Oc4OcU8oOMskKy8BgYLg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511488; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=xydBB4a77F6ev/Rdwtmv5eQnd75LN+ldAUqy0nGnk+w=; b=S0Bz2zo/dofmLf6rQd7XcYfgwYROBQzMEnsWUW9W9RvJcQ0dl1rxuwmU78yUqY2gSWIPh2IpX3Boyko28rFLFL2MujsZtsvG5V0u9GIKrKP8UIVTCa1H9qXeBo5h8NdIVKYzipdWX9NnBfeYTOqdDtLQaMk8F4HK6Dikn9tY6BY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1590511488365693.1855330344649; Tue, 26 May 2020 09:44:48 -0700 (PDT) Received: from localhost ([::1]:51524 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdch4-0007cu-Ra for importer@patchew.org; Tue, 26 May 2020 12:44:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41360) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdcet-0004dt-Pe for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:31 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:47877 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdces-0002sA-KN for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:31 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-62-ycy2himkOxCLRcyCiHDnKw-1; Tue, 26 May 2020 12:42:27 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0DC1883DB38; Tue, 26 May 2020 16:42:26 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3F2419007C; Tue, 26 May 2020 16:42:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511349; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xydBB4a77F6ev/Rdwtmv5eQnd75LN+ldAUqy0nGnk+w=; b=YM3sf1SXjuj3PY3CJNjj8LVF0R95rviPnDrugEsseKURh1SI/+7KlJZrt09HRMjJ5zBJE2 Srd31l3qI/VlQJg8kCWfKsYnAvRpIjCwgnWceEip6hCgWODHRbKfMSJ+ydYKrLwlG6c/iA aKcHGlWgx0GR/0z8kS/55zZg38Ywrrk= X-MC-Unique: ycy2himkOxCLRcyCiHDnKw-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 02/11] block/dirty-bitmap: add bdrv_has_named_bitmaps helper Date: Tue, 26 May 2020 11:42:02 -0500 Message-Id: <20200526164211.1569366-3-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.61; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 01:14:47 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Dirty Bitmaps" , Max Reitz , Andrey Shinkevich , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy To be used for bitmap migration in further commit. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Eric Blake Message-Id: <20200521220648.3255-3-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- include/block/dirty-bitmap.h | 1 + block/dirty-bitmap.c | 13 +++++++++++++ 2 files changed, 14 insertions(+) diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h index 5a8d52e4deaf..36e8da4fc2e3 100644 --- a/include/block/dirty-bitmap.h +++ b/include/block/dirty-bitmap.h @@ -95,6 +95,7 @@ int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap); void bdrv_dirty_bitmap_truncate(BlockDriverState *bs, int64_t bytes); bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap); bool bdrv_has_readonly_bitmaps(BlockDriverState *bs); +bool bdrv_has_named_bitmaps(BlockDriverState *bs); bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap); bool bdrv_dirty_bitmap_get_persistence(BdrvDirtyBitmap *bitmap); bool bdrv_dirty_bitmap_inconsistent(const BdrvDirtyBitmap *bitmap); diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index f9bfc77985e8..c01319b188c3 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -818,6 +818,19 @@ bool bdrv_has_readonly_bitmaps(BlockDriverState *bs) return false; } +bool bdrv_has_named_bitmaps(BlockDriverState *bs) +{ + BdrvDirtyBitmap *bm; + + QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) { + if (bdrv_dirty_bitmap_name(bm)) { + return true; + } + } + + return false; +} + /* Called with BQL taken. */ void bdrv_dirty_bitmap_set_persistence(BdrvDirtyBitmap *bitmap, bool persi= stent) { --=20 2.26.2 From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511451; cv=none; d=zohomail.com; s=zohoarc; b=J45G91VxdY5v3VH10kTEUaKCjXrF41wUN6R6StfvFzA2JSEA/GGeB8/5mcDjNhiFrZS9CkWtsFuBzC7VGU/btakohDD4NDWqDSHIC5p99HGqEKnKCfvnLRD5tYZRnvs5/QOUvlEZuHHxO2Q0AMQNNn0/9LeeuwqTu5fje3Nitz8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511451; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=FzLG/YoBJu2Cdm4qXbVPDSGQ6Rw+an87q7CnjlpFAPk=; b=dGo4YiPodRuzfIYEWn/0SkxCIYLiDHpzKk2VzjagyEzBcWigaaxdz3W6RM4gAk0lidrQq0Qaqe0C6LjmDjp9UPec9XYeC0yUd+E5nllCf/AbyrkMnsy2bE1KvjIS7dM5+ulTn+z9Y+dQzaiOip/dUiJMi7FCZJPo6jEeO0vbSMM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1590511451814984.5745201901472; Tue, 26 May 2020 09:44:11 -0700 (PDT) Received: from localhost ([::1]:49598 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdcgU-0006ou-Fa for importer@patchew.org; Tue, 26 May 2020 12:44:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41452) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdcew-0004kI-Qx for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:34 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:33760 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdcev-0002ul-19 for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:34 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-86-DZtiH-AKMsKnmMzHnhp4VA-1; Tue, 26 May 2020 12:42:28 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DC47C1800D42; Tue, 26 May 2020 16:42:26 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3BA736062D; Tue, 26 May 2020 16:42:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FzLG/YoBJu2Cdm4qXbVPDSGQ6Rw+an87q7CnjlpFAPk=; b=NEXKtTrSap16Y4qQPqIntYgWVqFkfOnTP/qYd03rCCCOZB3jfmHHbm5MCTzcPX0HPBeEob EK63frpAG2kVZrfLr1ROeWD1L8hCwpW8AFmMcpclPvyMv1vP16nk/j91lXmanS5jG14FuP l+wNyAxfiBGjtu4/YRkmEvMVRivkk+E= X-MC-Unique: DZtiH-AKMsKnmMzHnhp4VA-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 03/11] migration: fix bitmaps pre-blockdev migration with mirror job Date: Tue, 26 May 2020 11:42:03 -0500 Message-Id: <20200526164211.1569366-4-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.61; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 01:14:47 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Vladimir Sementsov-Ogievskiy , "open list:Block I/O path" , Juan Quintela , "Dr. David Alan Gilbert" , Stefan Hajnoczi , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Important thing for bitmap migration is to select destination block node to obtain the migrated bitmap. Prepatch, on source we use bdrv_get_device_or_node_name() to identify the node, and on target we do bdrv_lookup_bs. bdrv_get_device_or_node_name() returns blk name only for direct children of blk. So, bitmaps of direct children of blks are migrated by blk name and others - by node name. Old libvirt is unprepared to bitmap migration by node-name, node-names are mostly auto-generated. So actually only migration by blk name works for it. Newer libvirt will use new interface (which will be added soon) to specify node-mapping for bitmaps migration explicitly. Still, let's improve the current behavior a bit. Now, consider classic libvirt migrations assisted by mirror block job: mirror block job inserts filter, so our source is not a direct child of blk, and bitmaps are migrated by node-names. And this just doesn't work with auto-generated node names. Let's fix it by using blk-name even if some implicit filters are inserted. Note2: we, of course, can't skip filters and use blk name to migrate bitmaps in filtered node by blk name for this blk if these filters have named bitmaps which should be migrated. Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=3D1652424 Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200521220648.3255-4-vsementsov@virtuozzo.com> Reviewed-by: Eric Blake [eblake: comment typo fix, shorter subject line] Signed-off-by: Eric Blake --- migration/block-dirty-bitmap.c | 45 +++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c index 7e9371808653..69ddf289ddb9 100644 --- a/migration/block-dirty-bitmap.c +++ b/migration/block-dirty-bitmap.c @@ -319,14 +319,54 @@ static int init_dirty_bitmap_migration(void) { BlockDriverState *bs; DirtyBitmapMigBitmapState *dbms; + GHashTable *handled_by_blk =3D g_hash_table_new(NULL, NULL); + BlockBackend *blk; dirty_bitmap_mig_state.bulk_completed =3D false; dirty_bitmap_mig_state.prev_bs =3D NULL; dirty_bitmap_mig_state.prev_bitmap =3D NULL; dirty_bitmap_mig_state.no_bitmaps =3D false; + /* + * Use blockdevice name for direct (or filtered) children of named blo= ck + * backends. + */ + for (blk =3D blk_next(NULL); blk; blk =3D blk_next(blk)) { + const char *name =3D blk_name(blk); + + if (!name || strcmp(name, "") =3D=3D 0) { + continue; + } + + bs =3D blk_bs(blk); + + /* Skip filters without bitmaps */ + while (bs && bs->drv && bs->drv->is_filter && + !bdrv_has_named_bitmaps(bs)) + { + if (bs->backing) { + bs =3D bs->backing->bs; + } else if (bs->file) { + bs =3D bs->file->bs; + } else { + bs =3D NULL; + } + } + + if (bs && bs->drv && !bs->drv->is_filter) { + if (add_bitmaps_to_list(bs, name)) { + goto fail; + } + g_hash_table_add(handled_by_blk, bs); + } + } + for (bs =3D bdrv_next_all_states(NULL); bs; bs =3D bdrv_next_all_state= s(bs)) { - if (add_bitmaps_to_list(bs, bdrv_get_device_or_node_name(bs))) { + if (g_hash_table_contains(handled_by_blk, bs)) { + continue; + } + + if (add_bitmaps_to_list(bs, bdrv_get_node_name(bs))) { goto fail; } } @@ -340,9 +380,12 @@ static int init_dirty_bitmap_migration(void) dirty_bitmap_mig_state.no_bitmaps =3D true; } + g_hash_table_destroy(handled_by_blk); + return 0; fail: + g_hash_table_destroy(handled_by_blk); dirty_bitmap_mig_cleanup(); return -1; --=20 2.26.2 From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511667; cv=none; d=zohomail.com; s=zohoarc; b=CY0WFknZuSmNLrM2PtQNLfQApC7SHvhv+risk91tkVViiPSNysL6cxNY2ODlBVM/7NWT8tqe5ilHbsAC4a16memTabMzal8Y2jhnAKN7YAXNofdIE4B62bZSXMmz3CTEuY1o+MURWgTsIvOAiQjwS6CV9wms95AGfvPpH8KIO2g= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511667; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=6tpp61MDA8VB4l9tCN713iGLe0RwF9nSSBZ98FZeJlg=; b=Mz3XnmLbj47nbyboC+mbTjU0liIyYQCGlnpTEYoc9l5KSlF7ISf7i5BCdiU7yM8CshKbeSidvGNWJhVmAsejouE1sDmodDIHNjZ9IxnO+/5Plz/4Emrtd4TtuqkevXTICJPBA9+b5t5HJDU5SvkK094albOM1+20U718NqlzXV4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1590511667086627.0302297154835; Tue, 26 May 2020 09:47:47 -0700 (PDT) Received: from localhost ([::1]:60810 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdcjx-00032D-Pb for importer@patchew.org; Tue, 26 May 2020 12:47:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41486) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdcex-0004lk-Gy for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:35 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:41245 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdcev-0002vD-K2 for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:35 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-142-IiGtZ8A4MquxAkewbgW7iA-1; Tue, 26 May 2020 12:42:28 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8949F18FE861; Tue, 26 May 2020 16:42:27 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 179737D975; Tue, 26 May 2020 16:42:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6tpp61MDA8VB4l9tCN713iGLe0RwF9nSSBZ98FZeJlg=; b=ZsmT2QpJYmy/dguCNGkoxEgqwW6NHupYwauEalvdUSGDHlPcRaWCx9+z+79MgY9JSgtytu +MWAqjHak5Oui4rVP2eIHFj+ChCcf80OOMTaIIp/imYKSfM6PF7sd55wZx+KKYSPh3vjMe Jt0wNkTbPlHLH+xEihNG7oI7ynuoxO8= X-MC-Unique: IiGtZ8A4MquxAkewbgW7iA-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 04/11] iotests: 194: test also migration of dirty bitmap Date: Tue, 26 May 2020 11:42:04 -0500 Message-Id: <20200526164211.1569366-5-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 01:19:28 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Test that dirty bitmap migration works when we deal with mirror. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Message-Id: <20200521220648.3255-5-vsementsov@virtuozzo.com> Signed-off-by: Eric Blake --- tests/qemu-iotests/194 | 14 ++++++++++---- tests/qemu-iotests/194.out | 6 ++++++ 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/194 b/tests/qemu-iotests/194 index 8b1f720af4d8..3fad7c6c1ab6 100755 --- a/tests/qemu-iotests/194 +++ b/tests/qemu-iotests/194 @@ -42,6 +42,8 @@ with iotests.FilePath('source.img') as source_img_path, \ .add_incoming('unix:{0}'.format(migration_sock_path)) .launch()) + source_vm.qmp_log('block-dirty-bitmap-add', node=3D'drive0', name=3D'b= itmap0') + iotests.log('Launching NBD server on destination...') iotests.log(dest_vm.qmp('nbd-server-start', addr=3D{'type': 'unix', 'd= ata': {'path': nbd_sock_path}})) iotests.log(dest_vm.qmp('nbd-server-add', device=3D'drive0', writable= =3DTrue)) @@ -61,12 +63,14 @@ with iotests.FilePath('source.img') as source_img_path,= \ filters=3D[iotests.filter_qmp_event]) iotests.log('Starting migration...') - source_vm.qmp('migrate-set-capabilities', - capabilities=3D[{'capability': 'events', 'state': True}]) - dest_vm.qmp('migrate-set-capabilities', - capabilities=3D[{'capability': 'events', 'state': True}]) + capabilities =3D [{'capability': 'events', 'state': True}, + {'capability': 'dirty-bitmaps', 'state': True}] + source_vm.qmp('migrate-set-capabilities', capabilities=3Dcapabilities) + dest_vm.qmp('migrate-set-capabilities', capabilities=3Dcapabilities) iotests.log(source_vm.qmp('migrate', uri=3D'unix:{0}'.format(migration= _sock_path))) + source_vm.qmp_log('migrate-start-postcopy') + while True: event1 =3D source_vm.event_wait('MIGRATION') iotests.log(event1, filters=3D[iotests.filter_qmp_event]) @@ -82,3 +86,5 @@ with iotests.FilePath('source.img') as source_img_path, \ iotests.log('Stopping the NBD server on destination...') iotests.log(dest_vm.qmp('nbd-server-stop')) break + + iotests.log(source_vm.qmp('query-block')['return'][0]['dirty-bitmaps']) diff --git a/tests/qemu-iotests/194.out b/tests/qemu-iotests/194.out index 71857853fb4d..dd60dcc14f11 100644 --- a/tests/qemu-iotests/194.out +++ b/tests/qemu-iotests/194.out @@ -1,4 +1,6 @@ Launching VMs... +{"execute": "block-dirty-bitmap-add", "arguments": {"name": "bitmap0", "no= de": "drive0"}} +{"return": {}} Launching NBD server on destination... {"return": {}} {"return": {}} @@ -8,11 +10,15 @@ Waiting for `drive-mirror` to complete... {"data": {"device": "mirror-job0", "len": 1073741824, "offset": 1073741824= , "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_READY", "timestamp": {= "microseconds": "USECS", "seconds": "SECS"}} Starting migration... {"return": {}} +{"execute": "migrate-start-postcopy", "arguments": {}} +{"return": {}} {"data": {"status": "setup"}, "event": "MIGRATION", "timestamp": {"microse= conds": "USECS", "seconds": "SECS"}} {"data": {"status": "active"}, "event": "MIGRATION", "timestamp": {"micros= econds": "USECS", "seconds": "SECS"}} +{"data": {"status": "postcopy-active"}, "event": "MIGRATION", "timestamp":= {"microseconds": "USECS", "seconds": "SECS"}} {"data": {"status": "completed"}, "event": "MIGRATION", "timestamp": {"mic= roseconds": "USECS", "seconds": "SECS"}} Gracefully ending the `drive-mirror` job on source... {"return": {}} {"data": {"device": "mirror-job0", "len": 1073741824, "offset": 1073741824= , "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_COMPLETED", "timestamp= ": {"microseconds": "USECS", "seconds": "SECS"}} Stopping the NBD server on destination... {"return": {}} +[{"busy": false, "count": 0, "granularity": 65536, "name": "bitmap0", "per= sistent": false, "recording": true, "status": "active"}] --=20 2.26.2 From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511741; cv=none; d=zohomail.com; s=zohoarc; b=jp07BMMKMOOsqbILmRA/Yv4FOzGbfvGzIXtoMamSyalQZJpTMkEkV0N4QZwiPJDe+iskfQcogM4Amk6ZyUIO8Nb+wp6p7BRsRl1ZPCgRi4pV++vmiiXC6AxJo0eBlSYYEqszR+2IAXoIsPyJsf+/HQKQp32LSige4YE9i/ri1Cg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511741; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=sWK+K0D5a5hQeUb6kjfcddcUKtLePVH6IDx2s5CFCV8=; b=Jaq1Ir54uWB2H5mroIxL4r1Mr5KkQpnZXn8YzK42gYpi1YHsMdcFbh4YACxrvt7mSkWQMcBkMPg22cMJcOaFvozAW0dAm0qX4wVMr7uyKR0TfApfgd4wVzOsVEA/0Vq9ZuYK7Eol4AVTLZVASyWuDj5dIpA3FzKFKdqA8beIWUM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1590511741338487.4594990771815; Tue, 26 May 2020 09:49:01 -0700 (PDT) Received: from localhost ([::1]:37550 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdcl9-00055p-G2 for importer@patchew.org; Tue, 26 May 2020 12:48:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41432) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdcew-0004ii-6w for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:34 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:31800 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdceu-0002uf-Po for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:33 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-305-rXhhrHoXONqmIK_fdEmdjA-1; Tue, 26 May 2020 12:42:29 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6A6821009441; Tue, 26 May 2020 16:42:28 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id B1E76648DB; Tue, 26 May 2020 16:42:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sWK+K0D5a5hQeUb6kjfcddcUKtLePVH6IDx2s5CFCV8=; b=iR+lJ7sHM5dUIZaekWMX4CpcUOF5FZE5fwxtTWWpeZNbcXbeNIQLKDqmHYnyvd+4+bo65W dn8N3UhL4Ac8Wf3g5DaePMutUdh9/MrVw1cOHiYAvapXwvMIBEIWnK36/5guf5ev36pXGN NTJYF2DQvkF9yymESzCWfb8MD8jrpUM= X-MC-Unique: rXhhrHoXONqmIK_fdEmdjA-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 05/11] migration: add_bitmaps_to_list: check disk name once Date: Tue, 26 May 2020 11:42:05 -0500 Message-Id: <20200526164211.1569366-6-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=207.211.31.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 01:51:57 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Vladimir Sementsov-Ogievskiy , "open list:Block I/O path" , Juan Quintela , "Dr. David Alan Gilbert" , Stefan Hajnoczi , Andrey Shinkevich , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Eric Blake Message-Id: <20200521220648.3255-6-vsementsov@virtuozzo.com> [eblake: shorter subject line] Signed-off-by: Eric Blake --- migration/block-dirty-bitmap.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c index 69ddf289ddb9..7be1644b330a 100644 --- a/migration/block-dirty-bitmap.c +++ b/migration/block-dirty-bitmap.c @@ -274,17 +274,22 @@ static int add_bitmaps_to_list(BlockDriverState *bs, = const char *bs_name) DirtyBitmapMigBitmapState *dbms; Error *local_err =3D NULL; + bitmap =3D bdrv_dirty_bitmap_first(bs); + if (!bitmap) { + return 0; + } + + if (!bs_name || strcmp(bs_name, "") =3D=3D 0) { + error_report("Bitmap '%s' in unnamed node can't be migrated", + bdrv_dirty_bitmap_name(bitmap)); + return -1; + } + FOR_EACH_DIRTY_BITMAP(bs, bitmap) { if (!bdrv_dirty_bitmap_name(bitmap)) { continue; } - if (!bs_name || strcmp(bs_name, "") =3D=3D 0) { - error_report("Found bitmap '%s' in unnamed node %p. It can't " - "be migrated", bdrv_dirty_bitmap_name(bitmap), bs= ); - return -1; - } - if (bdrv_dirty_bitmap_check(bitmap, BDRV_BITMAP_DEFAULT, &local_er= r)) { error_report_err(local_err); return -1; --=20 2.26.2 From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511865; cv=none; d=zohomail.com; s=zohoarc; b=mDH/gX9OsDdl/BTNnhfo90X9TqoD218zDqObms7KMrmLOvVXs/r/wUKxnfGVCwH5m3gBLxy/CSk+NGXJ35DQwDagbmsPXY5OnP6pMPP59mQErR3Vi4bxFhkuldhJ68BimCDQB1htEs3WaC9owcfV7EHwcPn3cGazOx4g/i5evbo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511865; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=AumXH01F5gamBw2muKK3Go3fv2xAW1yMOQS1sMJTIHE=; b=Wqc9MD1ZkxQMLb2Ud7oROz6SRTa6IEI8+fsRFXJpnNWh6TZae2/v7OTT7thLbTbg4rFE+BctYLja44wHui2M+R7/0nLcKyNYmpX+jNi+V9IE+S3NtC3wtjy+x2wEfE+Dmsa+ah4RyzunnNZpeMcGuBsauui+EvaoJYWt7M4Rf5Y= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1590511865936401.1582223712554; Tue, 26 May 2020 09:51:05 -0700 (PDT) Received: from localhost ([::1]:44798 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdcnA-0008CJ-GH for importer@patchew.org; Tue, 26 May 2020 12:51:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41536) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdcez-0004qF-9a for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:37 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:35763 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdcey-0002yH-98 for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:36 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-122-BR0aEYayOxWxOXbLk-fPIA-1; Tue, 26 May 2020 12:42:33 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CF84E107ACF9; Tue, 26 May 2020 16:42:31 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 98ACD6E50D; Tue, 26 May 2020 16:42:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511355; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AumXH01F5gamBw2muKK3Go3fv2xAW1yMOQS1sMJTIHE=; b=V1WHWaCQPuAXKzqfwaQe+Dz4aaQP/qRgLhoM+B3R9gWj5h8sfD6dRbkTzh/TUXrL5tBxAD 34sTL+OckvSMMZI/GXtX7bEgmfYuanbLecDv7UH03wWBEx4j5eM5ddAzFGLvoOA2XxUaq6 XV0EFs26Cpo946oToEHLYciWEdEVTWo= X-MC-Unique: BR0aEYayOxWxOXbLk-fPIA-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 06/11] migration: forbid bitmap migration by generated node-name Date: Tue, 26 May 2020 11:42:06 -0500 Message-Id: <20200526164211.1569366-7-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.61; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 01:14:47 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Vladimir Sementsov-Ogievskiy , "open list:Block I/O path" , Juan Quintela , "Dr. David Alan Gilbert" , Stefan Hajnoczi , Andrey Shinkevich , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" From: Vladimir Sementsov-Ogievskiy It actually never worked with libvirt, as auto-generated names are different on source and destination. It's unsafe and useless to migrate by auto-generated node-names, so let's forbid it. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Eric Blake Message-Id: <20200521220648.3255-7-vsementsov@virtuozzo.com> [eblake: shorter subject line] Signed-off-by: Eric Blake --- migration/block-dirty-bitmap.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c index 7be1644b330a..47bc0f650c1e 100644 --- a/migration/block-dirty-bitmap.c +++ b/migration/block-dirty-bitmap.c @@ -285,6 +285,13 @@ static int add_bitmaps_to_list(BlockDriverState *bs, c= onst char *bs_name) return -1; } + if (bs_name[0] =3D=3D '#') { + error_report("Bitmap '%s' in a node with auto-generated " + "name '%s' can't be migrated", + bdrv_dirty_bitmap_name(bitmap), bs_name); + return -1; + } + FOR_EACH_DIRTY_BITMAP(bs, bitmap) { if (!bdrv_dirty_bitmap_name(bitmap)) { continue; --=20 2.26.2 From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511581; cv=none; d=zohomail.com; s=zohoarc; b=j3keNdzNlyCIUxTXKJ9c3hyatQyOwSYhznguyPKbi6QUxhmGy6GXJ66qVQG0R6s75uFcerZm0y2qX8P6rN0pnfcMiHWRRdCSC/O8Wc/ROb+Uo5u0ils7oZpzuO37FtzDZtSd8m6p5pYziOWWj9xMWF4OMC1L2JOCRhkwve8920w= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511581; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=cIjBlyXJ7X2S4+j+wnaT22TXvDB3n856fH07taNl2bM=; b=anoR/40gL1F5YAaTUTTADJQXtWBQYbxRNtl7C79KDTbPAm+WW2S3fZMCas+2MRE8jwzyU6jUUUeWwhpAFTh0JM5FzwpsmjYoiXt5Ohw6Ayo7gV7ZgAMt963PJJsFRrapORjgyMQPBU0gwIeBJVCL0TViZwQ8PxlnL0hRGE1JbiU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 159051158178748.45511980588242; Tue, 26 May 2020 09:46:21 -0700 (PDT) Received: from localhost ([::1]:56984 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdcia-0001Rj-HI for importer@patchew.org; Tue, 26 May 2020 12:46:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41698) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdcf5-00056I-QY for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:43 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:43941 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdcf3-00033K-LM for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:42 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-217-UP9TZw8yOR-X9kQMlX2XVw-1; Tue, 26 May 2020 12:42:36 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 03F9118FE860; Tue, 26 May 2020 16:42:35 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0A1C56E50D; Tue, 26 May 2020 16:42:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cIjBlyXJ7X2S4+j+wnaT22TXvDB3n856fH07taNl2bM=; b=e1iv3ZIe87kake/2mju0NGe+gBdz88SpWdVmDlAVmQGPi2aDehbjOqKh4p2pKJFQrWL/8R VKBcUB+yJKr4nNgDeAbPmHdQqSkcyaLe6PYu0l5uR4omulRQZoFWlTQA78+QyWhdwljv4p i9Dh1a+4Icp8nbVSRjDftHliuB8wHG0= X-MC-Unique: UP9TZw8yOR-X9kQMlX2XVw-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 07/11] iotests: Fix test 178 Date: Tue, 26 May 2020 11:42:07 -0500 Message-Id: <20200526164211.1569366-8-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=207.211.31.120; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 01:51:57 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" A recent change to qemu-img changed expected error message output, but 178 takes long enough to execute that it does not get run by 'make check' or './check -g quick'. Fixes: 43d589b074 Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200521192137.1120211-2-eblake@redhat.com> --- tests/qemu-iotests/178.out.qcow2 | 2 +- tests/qemu-iotests/178.out.raw | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.= qcow2 index f59bf4b2fbc4..4b69524c80ee 100644 --- a/tests/qemu-iotests/178.out.qcow2 +++ b/tests/qemu-iotests/178.out.qcow2 @@ -13,7 +13,7 @@ qemu-img: Invalid option list: , qemu-img: Invalid parameter 'snapshot.foo' qemu-img: Failed in parsing snapshot param 'snapshot.foo' qemu-img: --output must be used with human or json as argument. -qemu-img: Image size must be less than 8 EiB! +qemu-img: Invalid image size specified. Must be between 0 and 922337203685= 4775807. qemu-img: Unknown file format 'foo' =3D=3D Size calculation for a new file (human) =3D=3D diff --git a/tests/qemu-iotests/178.out.raw b/tests/qemu-iotests/178.out.raw index 404ca908d8c2..20e17da115cb 100644 --- a/tests/qemu-iotests/178.out.raw +++ b/tests/qemu-iotests/178.out.raw @@ -13,7 +13,7 @@ qemu-img: Invalid option list: , qemu-img: Invalid parameter 'snapshot.foo' qemu-img: Failed in parsing snapshot param 'snapshot.foo' qemu-img: --output must be used with human or json as argument. -qemu-img: Image size must be less than 8 EiB! +qemu-img: Invalid image size specified. Must be between 0 and 922337203685= 4775807. qemu-img: Unknown file format 'foo' =3D=3D Size calculation for a new file (human) =3D=3D --=20 2.26.2 From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511985; cv=none; d=zohomail.com; s=zohoarc; b=UJghQQrGOrvO9XQbi33Dbp4Jeb4KOOeH4hKen6SP7Fiwkx55/rqC/Z0yLNlHhTyxrkLizItQ1+wpk2yDbjxkshQclbmHlZudh2TrXGjrS99JgDCzLksMm+Sbn8UuR/79pHQhccdFaSRfe+2hx1eEPoKkVdYC9WyU1KZGqs8wqBU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511985; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=FnjlsxU8EC6R5JGu2gmb+BO+ZL3niFJnWJRxBYNIWA0=; b=niNDj8pAJ3gIXYrM75vyJ59OhTAm1Zgxd7OanngDrVI/wVi5A3+uTQEJVGAeR71No0GDuvPU4zkMQvNABOfpR6uXAYMOVl0rjOLXflPDrEVGlzz35WnA0GdeGg84gA+LDL/AiooV6ybTCzG6+sz7yK8P/vNKT+jrlEgrztuOZhc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1590511985779834.6313334360118; Tue, 26 May 2020 09:53:05 -0700 (PDT) Received: from localhost ([::1]:53912 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdcp6-0003Vw-Co for importer@patchew.org; Tue, 26 May 2020 12:53:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41712) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdcf6-00057r-7m for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:44 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:40615) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdcf3-00032v-BL for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:43 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-17-Z9cituhvOguVYLiXDjxTeg-1; Tue, 26 May 2020 12:42:37 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 05FE9461; Tue, 26 May 2020 16:42:36 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4B7C2648DB; Tue, 26 May 2020 16:42:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FnjlsxU8EC6R5JGu2gmb+BO+ZL3niFJnWJRxBYNIWA0=; b=Co+v4ATaNV7UBZmHLPsrTioCZGX/BwRz6FmmEULPhS73xjlh70hPkFjaANjKNibxYZPpCN sKFcl/ALDcUzsE1jwY0XffDJ5xWRnpb/D7BGr1Ms61xzXLyBrJ7ThJ/O3puQ01ENXkj4Sw Ym0miovnvdAA3mXdYfDOscaVoKjMfCU= X-MC-Unique: Z9cituhvOguVYLiXDjxTeg-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 08/11] qcow2: Expose bitmaps' size during measure Date: Tue, 26 May 2020 11:42:08 -0500 Message-Id: <20200526164211.1569366-9-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=207.211.31.81; envelope-from=eblake@redhat.com; helo=us-smtp-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 10:22:33 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Markus Armbruster , Max Reitz , Nir Soffer , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps. Update iotest 178 and 190 to updated output, as well as new coverage in 190 demonstrating non-zero values made possible with the recently-added qemu-img bitmap command (see 3b51ab4b). The new 'bitmaps size:' field is displayed automatically as part of 'qemu-img measure' any time it is present in QMP (that is, any time both the source image being measured and destination format support bitmaps, even if the measurement is 0 because there are no bitmaps present). If the field is absent, it means that no bitmaps can be copied (source, destination, or both lack bitmaps, including when measuring based on size rather than on a source image). This behavior is compatible with an upcoming patch adding 'qemu-img convert --bitmaps': that command will fail in the same situations where this patch omits the field. The addition of a new field demonstrates why we should always zero-initialize qapi C structs; while the qcow2 driver still fully populates all fields, the raw and crypto drivers had to be tweaked to avoid uninitialized data. Consideration was also given towards having a 'qemu-img measure --bitmaps' which errors out when bitmaps are not possible, and otherwise sums the bitmaps into the existing allocation totals rather than displaying as a separate field, as a potential convenience factor. But this was ultimately decided to be more complexity than necessary when the QMP interface was sufficient enough with bitmaps remaining a separate field. See also: https://bugzilla.redhat.com/1779904 Reported-by: Nir Soffer Signed-off-by: Eric Blake Message-Id: <20200521192137.1120211-3-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- docs/tools/qemu-img.rst | 7 +++++ qapi/block-core.json | 16 ++++++++---- block/qcow2.h | 2 ++ block/crypto.c | 2 +- block/qcow2-bitmap.c | 36 +++++++++++++++++++++++++ block/qcow2.c | 14 +++++++--- block/raw-format.c | 2 +- qemu-img.c | 3 +++ tests/qemu-iotests/178.out.qcow2 | 16 ++++++++++++ tests/qemu-iotests/190 | 45 ++++++++++++++++++++++++++++++-- tests/qemu-iotests/190.out | 27 ++++++++++++++++++- 11 files changed, 157 insertions(+), 13 deletions(-) diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst index 38d464ea3f23..320cb52b9f61 100644 --- a/docs/tools/qemu-img.rst +++ b/docs/tools/qemu-img.rst @@ -616,6 +616,7 @@ Command description: required size: 524288 fully allocated size: 1074069504 + bitmaps size: 0 The ``required size`` is the file size of the new image. It may be smal= ler than the virtual disk size if the image format supports compact represen= tation. @@ -625,6 +626,12 @@ Command description: occupy with the exception of internal snapshots, dirty bitmaps, vmstate = data, and other advanced image format features. + The ``bitmaps size`` is the additional size required in order to + copy bitmaps from a source image in addition to the guest-visible + data; the line is omitted if either source or destination lacks + bitmap support, or 0 if bitmaps are supported but there is nothing + to copy. + .. option:: snapshot [--object OBJECTDEF] [--image-opts] [-U] [-q] [-l | -= a SNAPSHOT | -c SNAPSHOT | -d SNAPSHOT] FILENAME List, apply, create or delete snapshots in image *FILENAME*. diff --git a/qapi/block-core.json b/qapi/block-core.json index 6fbacddab2cc..0e1c6a59f228 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -636,18 +636,24 @@ # efficiently so file size may be smaller than virtual disk size. # # The values are upper bounds that are guaranteed to fit the new image fil= e. -# Subsequent modification, such as internal snapshot or bitmap creation, m= ay -# require additional space and is not covered here. +# Subsequent modification, such as internal snapshot or further bitmap +# creation, may require additional space and is not covered here. # -# @required: Size required for a new image file, in bytes. +# @required: Size required for a new image file, in bytes, when copying ju= st +# allocated guest-visible contents. # # @fully-allocated: Image file size, in bytes, once data has been written -# to all sectors. +# to all sectors, when copying just guest-visible conten= ts. +# +# @bitmaps: Additional size required if all the top-level bitmap metadata +# in the source image were to be copied to the destination, +# present only when source and destination both support +# persistent bitmaps. (since 5.1) # # Since: 2.10 ## { 'struct': 'BlockMeasureInfo', - 'data': {'required': 'int', 'fully-allocated': 'int'} } + 'data': {'required': 'int', 'fully-allocated': 'int', '*bitmaps': 'int'}= } ## # @query-block: diff --git a/block/qcow2.h b/block/qcow2.h index 402e8acb1cb7..7ce2c23bdb7a 100644 --- a/block/qcow2.h +++ b/block/qcow2.h @@ -783,6 +783,8 @@ int qcow2_co_remove_persistent_dirty_bitmap(BlockDriver= State *bs, const char *name, Error **errp); bool qcow2_supports_persistent_dirty_bitmap(BlockDriverState *bs); +uint64_t qcow2_get_persistent_dirty_bitmap_size(BlockDriverState *bs, + uint32_t cluster_size); ssize_t coroutine_fn qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, diff --git a/block/crypto.c b/block/crypto.c index b216e12c3154..973b57b3eb74 100644 --- a/block/crypto.c +++ b/block/crypto.c @@ -552,7 +552,7 @@ static BlockMeasureInfo *block_crypto_measure(QemuOpts = *opts, * Unallocated blocks are still encrypted so allocation status makes no * difference to the file size. */ - info =3D g_new(BlockMeasureInfo, 1); + info =3D g_new0(BlockMeasureInfo, 1); info->fully_allocated =3D luks_payload_size + size; info->required =3D luks_payload_size + size; return info; diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c index 1cf6d2ab77a3..7bf12502da8c 100644 --- a/block/qcow2-bitmap.c +++ b/block/qcow2-bitmap.c @@ -1755,3 +1755,39 @@ bool qcow2_supports_persistent_dirty_bitmap(BlockDri= verState *bs) return s->qcow_version >=3D 3; } + +/* + * Compute the space required for bitmaps in @bs. + * + * The computation is based as if copying to a new image with the + * given @cluster_size, which may differ from the cluster size in @bs. + */ +uint64_t qcow2_get_persistent_dirty_bitmap_size(BlockDriverState *bs, + uint32_t cluster_size) +{ + uint64_t bitmaps_size =3D 0; + BdrvDirtyBitmap *bm; + size_t bitmap_dir_size =3D 0; + + FOR_EACH_DIRTY_BITMAP(bs, bm) { + if (bdrv_dirty_bitmap_get_persistence(bm)) { + const char *name =3D bdrv_dirty_bitmap_name(bm); + uint32_t granularity =3D bdrv_dirty_bitmap_granularity(bm); + uint64_t bmbytes =3D + get_bitmap_bytes_needed(bdrv_dirty_bitmap_size(bm), + granularity); + uint64_t bmclusters =3D DIV_ROUND_UP(bmbytes, cluster_size); + + /* Assume the entire bitmap is allocated */ + bitmaps_size +=3D bmclusters * cluster_size; + /* Also reserve space for the bitmap table entries */ + bitmaps_size +=3D ROUND_UP(bmclusters * sizeof(uint64_t), + cluster_size); + /* And space for contribution to bitmap directory size */ + bitmap_dir_size +=3D calc_dir_entry_size(strlen(name), 0); + } + } + bitmaps_size +=3D ROUND_UP(bitmap_dir_size, cluster_size); + + return bitmaps_size; +} diff --git a/block/qcow2.c b/block/qcow2.c index dfab8d2f6cd8..0cd2e6757e8c 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -4953,16 +4953,24 @@ static BlockMeasureInfo *qcow2_measure(QemuOpts *op= ts, BlockDriverState *in_bs, required =3D virtual_size; } - info =3D g_new(BlockMeasureInfo, 1); + info =3D g_new0(BlockMeasureInfo, 1); info->fully_allocated =3D qcow2_calc_prealloc_size(virtual_size, cluster_size, ctz32(refcount_bits)) + luks_payload_size; - /* Remove data clusters that are not required. This overestimates the + /* + * Remove data clusters that are not required. This overestimates the * required size because metadata needed for the fully allocated file = is - * still counted. + * still counted. Show bitmaps only if both source and destination + * would support them. */ info->required =3D info->fully_allocated - virtual_size + required; + info->has_bitmaps =3D version >=3D 3 && in_bs && + bdrv_supports_persistent_dirty_bitmap(in_bs); + if (info->has_bitmaps) { + info->bitmaps =3D qcow2_get_persistent_dirty_bitmap_size(in_bs, + cluster_siz= e); + } return info; err: diff --git a/block/raw-format.c b/block/raw-format.c index 018441bddf27..233d019ca338 100644 --- a/block/raw-format.c +++ b/block/raw-format.c @@ -359,7 +359,7 @@ static BlockMeasureInfo *raw_measure(QemuOpts *opts, Bl= ockDriverState *in_bs, BDRV_SECTOR_SIZE); } - info =3D g_new(BlockMeasureInfo, 1); + info =3D g_new0(BlockMeasureInfo, 1); info->required =3D required; /* Unallocated sectors count towards the file size in raw images */ diff --git a/qemu-img.c b/qemu-img.c index 2d30682f129f..b2311bd3f6b4 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -5302,6 +5302,9 @@ static int img_measure(int argc, char **argv) if (output_format =3D=3D OFORMAT_HUMAN) { printf("required size: %" PRIu64 "\n", info->required); printf("fully allocated size: %" PRIu64 "\n", info->fully_allocate= d); + if (info->has_bitmaps) { + printf("bitmaps size: %" PRIu64 "\n", info->bitmaps); + } } else { dump_json_block_measure_info(info); } diff --git a/tests/qemu-iotests/178.out.qcow2 b/tests/qemu-iotests/178.out.= qcow2 index 4b69524c80ee..c7997760fd6f 100644 --- a/tests/qemu-iotests/178.out.qcow2 +++ b/tests/qemu-iotests/178.out.qcow2 @@ -37,6 +37,7 @@ qemu-img: The image size is too large (try using a larger= cluster size) Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D0 required size: 196608 fully allocated size: 196608 +bitmaps size: 0 converted image file size in bytes: 196608 @@ -45,6 +46,7 @@ converted image file size in bytes: 196608 Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D1073741824 required size: 393216 fully allocated size: 1074135040 +bitmaps size: 0 wrote 512/512 bytes at offset 512 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) wrote 65536/65536 bytes at offset 65536 @@ -53,6 +55,7 @@ wrote 64512/64512 bytes at offset 134217728 63 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) required size: 589824 fully allocated size: 1074135040 +bitmaps size: 0 converted image file size in bytes: 524288 @@ -60,6 +63,7 @@ converted image file size in bytes: 524288 required size: 524288 fully allocated size: 1074135040 +bitmaps size: 0 converted image file size in bytes: 458752 @@ -67,16 +71,19 @@ converted image file size in bytes: 458752 required size: 1074135040 fully allocated size: 1074135040 +bitmaps size: 0 =3D=3D qcow2 input image and LUKS encryption =3D=3D required size: 2686976 fully allocated size: 1076232192 +bitmaps size: 0 =3D=3D qcow2 input image and preallocation (human) =3D=3D required size: 1074135040 fully allocated size: 1074135040 +bitmaps size: 0 converted image file size in bytes: 1074135040 @@ -87,6 +94,7 @@ wrote 8388608/8388608 bytes at offset 0 8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) required size: 8716288 fully allocated size: 8716288 +bitmaps size: 0 converted image file size in bytes: 8716288 @@ -173,6 +181,7 @@ qemu-img: The image size is too large (try using a larg= er cluster size) Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D0 { + "bitmaps": 0, "required": 196608, "fully-allocated": 196608 } @@ -183,6 +192,7 @@ converted image file size in bytes: 196608 Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D1073741824 { + "bitmaps": 0, "required": 393216, "fully-allocated": 1074135040 } @@ -193,6 +203,7 @@ wrote 65536/65536 bytes at offset 65536 wrote 64512/64512 bytes at offset 134217728 63 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) { + "bitmaps": 0, "required": 589824, "fully-allocated": 1074135040 } @@ -202,6 +213,7 @@ converted image file size in bytes: 524288 =3D=3D qcow2 input image with internal snapshot (json) =3D=3D { + "bitmaps": 0, "required": 524288, "fully-allocated": 1074135040 } @@ -211,6 +223,7 @@ converted image file size in bytes: 458752 =3D=3D qcow2 input image and a backing file (json) =3D=3D { + "bitmaps": 0, "required": 1074135040, "fully-allocated": 1074135040 } @@ -218,6 +231,7 @@ converted image file size in bytes: 458752 =3D=3D qcow2 input image and LUKS encryption =3D=3D { + "bitmaps": 0, "required": 2686976, "fully-allocated": 1076232192 } @@ -225,6 +239,7 @@ converted image file size in bytes: 458752 =3D=3D qcow2 input image and preallocation (json) =3D=3D { + "bitmaps": 0, "required": 1074135040, "fully-allocated": 1074135040 } @@ -237,6 +252,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D838= 8608 wrote 8388608/8388608 bytes at offset 0 8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) { + "bitmaps": 0, "required": 8716288, "fully-allocated": 8716288 } diff --git a/tests/qemu-iotests/190 b/tests/qemu-iotests/190 index 6d41650438e1..0c37862c6709 100755 --- a/tests/qemu-iotests/190 +++ b/tests/qemu-iotests/190 @@ -2,7 +2,7 @@ # # qemu-img measure sub-command tests on huge qcow2 files # -# Copyright (C) 2017 Red Hat, Inc. +# Copyright (C) 2017-2020 Red Hat, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -42,7 +42,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15 _supported_fmt qcow2 _supported_proto file -echo "=3D=3D Huge file =3D=3D" +echo "=3D=3D Huge file without bitmaps =3D=3D" echo _make_test_img -o 'cluster_size=3D2M' 2T @@ -51,6 +51,47 @@ $QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" $QEMU_IMG measure -O qcow2 -o cluster_size=3D64k -f qcow2 "$TEST_IMG" $QEMU_IMG measure -O qcow2 -o cluster_size=3D2M -f qcow2 "$TEST_IMG" +echo +echo "=3D=3D Huge file with bitmaps =3D=3D" +echo + +$QEMU_IMG bitmap --add --granularity 512 -f qcow2 "$TEST_IMG" b1 +$QEMU_IMG bitmap --add -g 2M -f qcow2 "$TEST_IMG" b2 + +# No bitmap without a source +$QEMU_IMG measure -O qcow2 --size 10M +# No bitmap output, since raw does not support it +$QEMU_IMG measure -O raw -f qcow2 "$TEST_IMG" +# No bitmap output, since no bitmaps on raw source +$QEMU_IMG measure -O qcow2 -f raw "$TEST_IMG" +# No bitmap output, since v2 does not support it +$QEMU_IMG measure -O qcow2 -o compat=3D0.10 -f qcow2 "$TEST_IMG" + +# Compute expected output: bitmap clusters + bitmap tables + bitmaps direc= tory +echo +val2T=3D$((2*1024*1024*1024*1024)) +cluster=3D$((64*1024)) +b1clusters=3D$(( (val2T/512/8 + cluster - 1) / cluster )) +b2clusters=3D$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster )) +echo expected bitmap $((b1clusters * cluster + + (b1clusters * 8 + cluster - 1) / cluster * cluster= + + b2clusters * cluster + + (b2clusters * 8 + cluster - 1) / cluster * cluster= + + cluster)) +$QEMU_IMG measure -O qcow2 -o cluster_size=3D64k -f qcow2 "$TEST_IMG" + +# Compute expected output: bitmap clusters + bitmap tables + bitmaps direc= tory +echo +cluster=3D$((2*1024*1024)) +b1clusters=3D$(( (val2T/512/8 + cluster - 1) / cluster )) +b2clusters=3D$(( (val2T/2/1024/1024/8 + cluster - 1) / cluster )) +echo expected bitmap $((b1clusters * cluster + + (b1clusters * 8 + cluster - 1) / cluster * cluster= + + b2clusters * cluster + + (b2clusters * 8 + cluster - 1) / cluster * cluster= + + cluster)) +$QEMU_IMG measure --output=3Djson -O qcow2 -o cluster_size=3D2M -f qcow2 "= $TEST_IMG" + # success, all done echo "*** done" rm -f $seq.full diff --git a/tests/qemu-iotests/190.out b/tests/qemu-iotests/190.out index d001942002db..e9c5a2e1e481 100644 --- a/tests/qemu-iotests/190.out +++ b/tests/qemu-iotests/190.out @@ -1,11 +1,36 @@ QA output created by 190 -=3D=3D Huge file =3D=3D +=3D=3D Huge file without bitmaps =3D=3D Formatting 'TEST_DIR/t.IMGFMT', fmt=3DIMGFMT size=3D2199023255552 required size: 2199023255552 fully allocated size: 2199023255552 required size: 335806464 fully allocated size: 2199359062016 +bitmaps size: 0 required size: 18874368 fully allocated size: 2199042129920 +bitmaps size: 0 + +=3D=3D Huge file with bitmaps =3D=3D + +required size: 327680 +fully allocated size: 10813440 +required size: 2199023255552 +fully allocated size: 2199023255552 +required size: 7012352 +fully allocated size: 17170432 +required size: 335806464 +fully allocated size: 2199359062016 + +expected bitmap 537198592 +required size: 335806464 +fully allocated size: 2199359062016 +bitmaps size: 537198592 + +expected bitmap 545259520 +{ + "bitmaps": 545259520, + "required": 18874368, + "fully-allocated": 2199042129920 +} *** done --=20 2.26.2 From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511621; cv=none; d=zohomail.com; s=zohoarc; b=Fuyq0zOCJDgDijLHxLzObnjEF2tqVRMVapLxUKafrVJSde17NHqnf+0gXHTyfGqEB840ObEmufHz2a8CaLP7V/3CWGbaGNPXqonL0A7jtUr9wPVkbkCDQaNONaf9ZCp7kLl1xErL8V+mkA1/jtNq0qbyT0JdmUFVcPmXCuyuyUg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511621; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=GahAtAd+aAFjS7BoLY7dnLsO6OwTXstkpAbCyXSvYig=; b=b6iwUQr3ta8UYecmacF4oTi2iJhP7hQHjy31VFQPab4fouUpXrqQfWUEoAK11FRjtlXfgWTyDdAwMsN/wHZBpSvHvLx3ZzwiXScltlVNaV7qIZuUzHqEnamDy6xwzGnM6jTPh6XlTlLJYzZsZhWitIkDmZsKy/Pd4eSUeBI3Oj0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1590511621253383.89806016316606; Tue, 26 May 2020 09:47:01 -0700 (PDT) Received: from localhost ([::1]:59282 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdcjD-0002PZ-Vx for importer@patchew.org; Tue, 26 May 2020 12:47:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41770) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdcf8-0005Ch-4U for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:46 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:34762 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdcf6-00034q-3R for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:45 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-261-iGQ9eDhSMSSVUENXVePW7Q-1; Tue, 26 May 2020 12:42:37 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A42E5835B50; Tue, 26 May 2020 16:42:36 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3578E7D975; Tue, 26 May 2020 16:42:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511362; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GahAtAd+aAFjS7BoLY7dnLsO6OwTXstkpAbCyXSvYig=; b=HA3zvfIysqvw49if3/JRIKte1niZtZYioLrWzOza7VCNedCkm+KYntIAbFi0BUSOLPZPDw ftgBNN5x1ntfTNkyuE6lfA35wJZvKUEquf2EDnxIQB2TWCPFmDFosxVE6Z2q1iFCTzoGsm jXkyyyqQwM7uVxqfCccTKBAPutQaIHI= X-MC-Unique: iGQ9eDhSMSSVUENXVePW7Q-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 09/11] qemu-img: Factor out code for merging bitmaps Date: Tue, 26 May 2020 11:42:09 -0500 Message-Id: <20200526164211.1569366-10-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=207.211.31.81; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 10:22:33 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" The next patch will add another client that wants to merge dirty bitmaps; it will be easier to refactor the code to construct the QAPI struct correctly into a helper function. Signed-off-by: Eric Blake Message-Id: <20200521192137.1120211-4-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- qemu-img.c | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/qemu-img.c b/qemu-img.c index b2311bd3f6b4..0778d8f56614 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -1638,6 +1638,24 @@ out4: return ret; } +/* Convenience wrapper around qmp_block_dirty_bitmap_merge */ +static void do_dirty_bitmap_merge(const char *dst_node, const char *dst_na= me, + const char *src_node, const char *src_na= me, + Error **errp) +{ + BlockDirtyBitmapMergeSource *merge_src; + BlockDirtyBitmapMergeSourceList *list; + + merge_src =3D g_new0(BlockDirtyBitmapMergeSource, 1); + merge_src->type =3D QTYPE_QDICT; + merge_src->u.external.node =3D g_strdup(src_node); + merge_src->u.external.name =3D g_strdup(src_name); + list =3D g_new0(BlockDirtyBitmapMergeSourceList, 1); + list->value =3D merge_src; + qmp_block_dirty_bitmap_merge(dst_node, dst_name, list, errp); + qapi_free_BlockDirtyBitmapMergeSourceList(list); +} + enum ImgConvertBlockStatus { BLK_DATA, BLK_ZERO, @@ -4714,21 +4732,11 @@ static int img_bitmap(int argc, char **argv) qmp_block_dirty_bitmap_disable(bs->node_name, bitmap, &err); op =3D "disable"; break; - case BITMAP_MERGE: { - BlockDirtyBitmapMergeSource *merge_src; - BlockDirtyBitmapMergeSourceList *list; - - merge_src =3D g_new0(BlockDirtyBitmapMergeSource, 1); - merge_src->type =3D QTYPE_QDICT; - merge_src->u.external.node =3D g_strdup(src_bs->node_name); - merge_src->u.external.name =3D g_strdup(act->src); - list =3D g_new0(BlockDirtyBitmapMergeSourceList, 1); - list->value =3D merge_src; - qmp_block_dirty_bitmap_merge(bs->node_name, bitmap, list, &err= ); - qapi_free_BlockDirtyBitmapMergeSourceList(list); + case BITMAP_MERGE: + do_dirty_bitmap_merge(bs->node_name, bitmap, src_bs->node_name, + act->src, &err); op =3D "merge"; break; - } default: g_assert_not_reached(); } --=20 2.26.2 From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511740; cv=none; d=zohomail.com; s=zohoarc; b=AepJHLRH6VAs/nzkapyjncY+DH7bgeYisjT/V+jvixWuFkpYvH67EuZ3tMlRBRdYpi316J3z22qybI+aasr4ZS2fd2VusUpsAR0Jd+jSGGLY8eUjqpKAUnrYMtN+sBpi5xR3TaI/I2K3JwiXodNH5wQ7PEX6qtYzGnOSozIOwJs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511740; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=dZ06/EF32pP44r+61nn9lQ7F45rEKu7NonHz3UMpGZ4=; b=GSmqIeHrH/AeYNuGgU0PY89GKN2P8C4AVe8jZ6nU69RitiOCjLk1PagAyGYlfULNMqDYrpf4SIFHa93svelUNa53fJSdPH+KATJCtBVNSavFb5zyOMPA8zv2brvanqVS1CdFDxSNh6/fT/2fPfy32eFys0x1KPbhybDlk9Czz5M= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1590511740674675.6057676815116; Tue, 26 May 2020 09:49:00 -0700 (PDT) Received: from localhost ([::1]:37446 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdcl9-00053A-2S for importer@patchew.org; Tue, 26 May 2020 12:48:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41774) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdcf8-0005Cr-9j for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:46 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:54658 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdcf5-000345-8P for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:45 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-446-gdc1dTdeNSGW5lMcnW40eQ-1; Tue, 26 May 2020 12:42:38 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5034018FE872; Tue, 26 May 2020 16:42:37 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id D708D7D975; Tue, 26 May 2020 16:42:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511361; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dZ06/EF32pP44r+61nn9lQ7F45rEKu7NonHz3UMpGZ4=; b=FkLJHR+8lhmuqqyMTM6sGgIuVmFvrufeq9jPC27OQhi/7mqOh9MJz4mpJVna+R7wSRv1yM 2uLgpOB2r9X6GU3VTOSKMAAjfBcewo5CMtVktb7RQQ50+NYdEeW+mEf47qUwr2OLghf/WO uMgwJkibSBRWpUpqG7kYvgAnDxRiD9Q= X-MC-Unique: gdc1dTdeNSGW5lMcnW40eQ-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 10/11] qemu-img: Add convert --bitmaps option Date: Tue, 26 May 2020 11:42:10 -0500 Message-Id: <20200526164211.1569366-11-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.61; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 01:14:47 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Make it easier to copy all the persistent bitmaps of (the top layer of) a source image along with its guest-visible contents, by adding a boolean flag for use with qemu-img convert. This is basically shorthand, as the same effect could be accomplished with a series of 'qemu-img bitmap --add' and 'qemu-img bitmap --merge -b source' commands, or by their corresponding QMP commands. Note that this command will fail in the same scenarios where 'qemu-img measure' omits a 'bitmaps size:' line, namely, when either the source or the destination lacks persistent bitmap support altogether. See also https://bugzilla.redhat.com/show_bug.cgi?id=3D1779893 While touching this, clean up a couple coding issues spotted in the same function: an extra blank line, and merging back-to-back 'if (!skip_create)' blocks. Signed-off-by: Eric Blake Message-Id: <20200521192137.1120211-5-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy --- docs/tools/qemu-img.rst | 6 +++- qemu-img.c | 70 +++++++++++++++++++++++++++++++++++++++-- qemu-img-cmds.hx | 4 +-- 3 files changed, 74 insertions(+), 6 deletions(-) diff --git a/docs/tools/qemu-img.rst b/docs/tools/qemu-img.rst index 320cb52b9f61..69cd9a30373a 100644 --- a/docs/tools/qemu-img.rst +++ b/docs/tools/qemu-img.rst @@ -162,6 +162,10 @@ Parameters to convert subcommand: .. program:: qemu-img-convert +.. option:: --bitmaps + + Additionally copy all persistent bitmaps from the top layer of the source + .. option:: -n Skip the creation of the target volume @@ -397,7 +401,7 @@ Command description: 4 Error on reading data -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-op= ts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-= T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PA= RAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] O= UTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-op= ts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [= -t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l= SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENA= ME2 [...]] OUTPUT_FILENAME Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM* to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can diff --git a/qemu-img.c b/qemu-img.c index 0778d8f56614..d7e846e60742 100644 --- a/qemu-img.c +++ b/qemu-img.c @@ -78,6 +78,7 @@ enum { OPTION_ENABLE =3D 272, OPTION_DISABLE =3D 273, OPTION_MERGE =3D 274, + OPTION_BITMAPS =3D 275, }; typedef enum OutputFormat { @@ -191,6 +192,7 @@ static void QEMU_NORETURN help(void) " hiding corruption that has already occurred.\n" "\n" "Parameters to convert subcommand:\n" + " '--bitmaps' copies all top-level persistent bitmaps to desti= nation\n" " '-m' specifies how many coroutines work in parallel during t= he convert\n" " process (defaults to 8)\n" " '-W' allow to write to the target out of order rather than s= equential\n" @@ -2139,6 +2141,39 @@ static int convert_do_copy(ImgConvertState *s) return s->ret; } +static int convert_copy_bitmaps(BlockDriverState *src, BlockDriverState *d= st) +{ + BdrvDirtyBitmap *bm; + Error *err =3D NULL; + + FOR_EACH_DIRTY_BITMAP(src, bm) { + const char *name; + + if (!bdrv_dirty_bitmap_get_persistence(bm)) { + continue; + } + name =3D bdrv_dirty_bitmap_name(bm); + qmp_block_dirty_bitmap_add(dst->node_name, name, + true, bdrv_dirty_bitmap_granularity(bm), + true, true, + true, !bdrv_dirty_bitmap_enabled(bm), + &err); + if (err) { + error_reportf_err(err, "Failed to create bitmap %s: ", name); + return -1; + } + + do_dirty_bitmap_merge(dst->node_name, name, src->node_name, name, + &err); + if (err) { + error_reportf_err(err, "Failed to populate bitmap %s: ", name); + return -1; + } + } + + return 0; +} + #define MAX_BUF_SECTORS 32768 static int img_convert(int argc, char **argv) @@ -2160,6 +2195,7 @@ static int img_convert(int argc, char **argv) int64_t ret =3D -EINVAL; bool force_share =3D false; bool explict_min_sparse =3D false; + bool bitmaps =3D false; ImgConvertState s =3D (ImgConvertState) { /* Need at least 4k of zeros for sparse detection */ @@ -2179,6 +2215,7 @@ static int img_convert(int argc, char **argv) {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS= }, {"salvage", no_argument, 0, OPTION_SALVAGE}, {"target-is-zero", no_argument, 0, OPTION_TARGET_IS_ZERO}, + {"bitmaps", no_argument, 0, OPTION_BITMAPS}, {0, 0, 0, 0} }; c =3D getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU", @@ -2304,6 +2341,9 @@ static int img_convert(int argc, char **argv) */ s.has_zero_init =3D true; break; + case OPTION_BITMAPS: + bitmaps =3D true; + break; } } @@ -2365,7 +2405,6 @@ static int img_convert(int argc, char **argv) goto fail_getopt; } - /* ret is still -EINVAL until here */ ret =3D bdrv_parse_cache_mode(src_cache, &src_flags, &src_writethrough= ); if (ret < 0) { @@ -2525,6 +2564,20 @@ static int img_convert(int argc, char **argv) } } + /* Determine if bitmaps need copying */ + if (bitmaps) { + if (s.src_num > 1) { + error_report("Copying bitmaps only possible with single source= "); + ret =3D -1; + goto out; + } + if (!bdrv_supports_persistent_dirty_bitmap(blk_bs(s.src[0]))) { + error_report("Source lacks bitmap support"); + ret =3D -1; + goto out; + } + } + /* * The later open call will need any decryption secrets, and * bdrv_create() will purge "opts", so extract them now before @@ -2533,9 +2586,7 @@ static int img_convert(int argc, char **argv) if (!skip_create) { open_opts =3D qdict_new(); qemu_opt_foreach(opts, img_add_key_secrets, open_opts, &error_abor= t); - } - if (!skip_create) { /* Create the new image */ ret =3D bdrv_create(drv, out_filename, opts, &local_err); if (ret < 0) { @@ -2573,6 +2624,13 @@ static int img_convert(int argc, char **argv) } out_bs =3D blk_bs(s.target); + if (bitmaps && !bdrv_supports_persistent_dirty_bitmap(out_bs)) { + error_report("Format driver '%s' does not support bitmaps", + out_bs->drv->format_name); + ret =3D -1; + goto out; + } + if (s.compressed && !block_driver_can_compress(out_bs->drv)) { error_report("Compression not supported for this file format"); ret =3D -1; @@ -2632,6 +2690,12 @@ static int img_convert(int argc, char **argv) } ret =3D convert_do_copy(&s); + + /* Now copy the bitmaps */ + if (bitmaps && ret =3D=3D 0) { + ret =3D convert_copy_bitmaps(blk_bs(s.src[0]), out_bs); + } + out: if (!ret) { qemu_progress_print(100, 0); diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx index a87d3cb264ce..10b910b67cf8 100644 --- a/qemu-img-cmds.hx +++ b/qemu-img-cmds.hx @@ -46,9 +46,9 @@ SRST ERST DEF("convert", img_convert, - "convert [--object objectdef] [--image-opts] [--target-image-opts] [--= target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_c= ache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-= S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [..= .]] output_filename") + "convert [--object objectdef] [--image-opts] [--target-image-opts] [--= target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cach= e] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapsh= ot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [f= ilename2 [...]] output_filename") SRST -.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-op= ts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-= T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PA= RAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENA= ME2 [...]] OUTPUT_FILENAME +.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-op= ts] [--target-is-zero] [--bitmaps] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [= -t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l= SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILE= NAME [FILENAME2 [...]] OUTPUT_FILENAME ERST DEF("create", img_create, --=20 2.26.2 From nobody Sun Oct 5 17:23:00 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1590511860; cv=none; d=zohomail.com; s=zohoarc; b=OgvT0SaM+0d+jPrcMoATcutzdcTlFuOKPV8UOgwzeQ/lmL9jy/l1x5/Mp02daTccVAqvjpsHDzFw1PFE6iGJGRSY61STLPhPJeVA8uoWupkdR/T1RWQzgrXbhFBpQswYZdR1zgaqHT1E1qjzTyECnaPTrAuHAbHT4J7uGMVkAws= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1590511860; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=5+8N0laAh6JXO7upcW6kp1tPX0pIL9vlOO6vRS04kbQ=; b=A21J+/VI7d9cppcIFsbyPrdSfmjJRxty+leqpqTCfdtqA0L59to1Jju0gkj+V9Cb7vX8ryBCdTv4dl4kv/vPbjczIrDdsrEywbmgSZ6Q3H5qMyOVr+vsBRj6VrXfET5r80OAMqsP4Nk3E0ESjm/y4ur+Vh/5F3zf9uCaSjwmsXg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1590511860464388.00010312574807; Tue, 26 May 2020 09:51:00 -0700 (PDT) Received: from localhost ([::1]:44390 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jdcn5-00081E-0o for importer@patchew.org; Tue, 26 May 2020 12:50:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:41950) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jdcfF-0005Sr-7m for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:53 -0400 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:23308 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1jdcfA-0003CW-NZ for qemu-devel@nongnu.org; Tue, 26 May 2020 12:42:52 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-382-7Wmq4wLNPSWhD0T3SAe_fw-1; Tue, 26 May 2020 12:42:41 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 703911009443; Tue, 26 May 2020 16:42:40 +0000 (UTC) Received: from blue.redhat.com (ovpn-112-88.phx2.redhat.com [10.3.112.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 830A07D975; Tue, 26 May 2020 16:42:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1590511367; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5+8N0laAh6JXO7upcW6kp1tPX0pIL9vlOO6vRS04kbQ=; b=HlY7h6u1oFiJJ3myC8Pj35UJI0TKYqR0GRMMgUs1nz+6AC8owg8SrvTFo1QsUJcnOmldHx Yd8CHinmmLzRiBe1ekk5pRL6V/9teJB32sn/oT9e34oaPLRjmYzmdWCQCf1EiWwpTc010J /6PskXtUr+QZPoVtDBgHeymK+MRyRq4= X-MC-Unique: 7Wmq4wLNPSWhD0T3SAe_fw-1 From: Eric Blake To: qemu-devel@nongnu.org Subject: [PULL 11/11] iotests: Add test 291 to for qemu-img bitmap coverage Date: Tue, 26 May 2020 11:42:11 -0500 Message-Id: <20200526164211.1569366-12-eblake@redhat.com> In-Reply-To: <20200526164211.1569366-1-eblake@redhat.com> References: <20200526164211.1569366-1-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=205.139.110.61; envelope-from=eblake@redhat.com; helo=us-smtp-delivery-1.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/26 01:14:47 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , Vladimir Sementsov-Ogievskiy , "open list:Block layer core" , Max Reitz Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Add a new test covering the 'qemu-img bitmap' subcommand, as well as 'qemu-img convert --bitmaps', both added in recent patches. Signed-off-by: Eric Blake Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20200521192137.1120211-6-eblake@redhat.com> --- tests/qemu-iotests/291 | 112 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/291.out | 80 ++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 193 insertions(+) create mode 100755 tests/qemu-iotests/291 create mode 100644 tests/qemu-iotests/291.out diff --git a/tests/qemu-iotests/291 b/tests/qemu-iotests/291 new file mode 100755 index 000000000000..3ca83b9cd1f7 --- /dev/null +++ b/tests/qemu-iotests/291 @@ -0,0 +1,112 @@ +#!/usr/bin/env bash +# +# Test qemu-img bitmap handling +# +# Copyright (C) 2018-2020 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +seq=3D"$(basename $0)" +echo "QA output created by $seq" + +status=3D1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img + nbd_server_stop +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter +. ./common.nbd + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux +_require_command QEMU_NBD + +echo +echo "=3D=3D=3D Initial image setup =3D=3D=3D" +echo + +# Create backing image with one bitmap +TEST_IMG=3D"$TEST_IMG.base" _make_test_img 10M +$QEMU_IMG bitmap --add -f $IMGFMT "$TEST_IMG.base" b0 +$QEMU_IO -c 'w 3M 1M' -f $IMGFMT "$TEST_IMG.base" | _filter_qemu_io + +# Create initial image and populate two bitmaps: one active, one inactive. +ORIG_IMG=3D$TEST_IMG +TEST_IMG=3D$TEST_IMG.orig +_make_test_img -b "$ORIG_IMG.base" -F $IMGFMT 10M +$QEMU_IO -c 'w 0 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG bitmap --add -g 512k -f $IMGFMT "$TEST_IMG" b1 +$QEMU_IMG bitmap --add --disable -f $IMGFMT "$TEST_IMG" b2 +$QEMU_IO -c 'w 3M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG bitmap --clear -f $IMGFMT "$TEST_IMG" b1 +$QEMU_IO -c 'w 1M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io +$QEMU_IMG bitmap --disable -f $IMGFMT "$TEST_IMG" b1 +$QEMU_IMG bitmap --enable -f $IMGFMT "$TEST_IMG" b2 +$QEMU_IO -c 'w 2M 1M' -f $IMGFMT "$TEST_IMG" | _filter_qemu_io + +echo +echo "=3D=3D=3D Bitmap preservation not possible to non-qcow2 =3D=3D=3D" +echo + +TEST_IMG=3D$ORIG_IMG +$QEMU_IMG convert --bitmaps -O raw "$TEST_IMG.orig" "$TEST_IMG" && + echo "unexpected success" + +echo +echo "=3D=3D=3D Convert with bitmap preservation =3D=3D=3D" +echo + +# Only bitmaps from the active layer are copied +$QEMU_IMG convert --bitmaps -O qcow2 "$TEST_IMG.orig" "$TEST_IMG" +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific +# But we can also merge in bitmaps from other layers. This test is a bit +# contrived to cover more code paths, in reality, you could merge directly +# into b0 without going through tmp +$QEMU_IMG bitmap --add --disable -f $IMGFMT "$TEST_IMG" b0 +$QEMU_IMG bitmap --add --merge b0 -b "$TEST_IMG.base" -F $IMGFMT \ + -f $IMGFMT "$TEST_IMG" tmp +$QEMU_IMG bitmap --merge tmp -f $IMGFMT "$TEST_IMG" b0 +$QEMU_IMG bitmap --remove --image-opts \ + driver=3D$IMGFMT,file.driver=3Dfile,file.filename=3D"$TEST_IMG" tmp +$QEMU_IMG info "$TEST_IMG" | _filter_img_info --format-specific + +echo +echo "=3D=3D=3D Check bitmap contents =3D=3D=3D" +echo + +# x-dirty-bitmap is a hack for reading bitmaps; it abuses block status to +# report "data":false for portions of the bitmap which are set +IMG=3D"driver=3Dnbd,server.type=3Dunix,server.path=3D$nbd_unix_socket" +nbd_server_start_unix_socket -r -f qcow2 -B b0 "$TEST_IMG" +$QEMU_IMG map --output=3Djson --image-opts \ + "$IMG,x-dirty-bitmap=3Dqemu:dirty-bitmap:b0" | _filter_qemu_img_map +nbd_server_start_unix_socket -r -f qcow2 -B b1 "$TEST_IMG" +$QEMU_IMG map --output=3Djson --image-opts \ + "$IMG,x-dirty-bitmap=3Dqemu:dirty-bitmap:b1" | _filter_qemu_img_map +nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG" +$QEMU_IMG map --output=3Djson --image-opts \ + "$IMG,x-dirty-bitmap=3Dqemu:dirty-bitmap:b2" | _filter_qemu_img_map + +# success, all done +echo '*** done' +rm -f $seq.full +status=3D0 diff --git a/tests/qemu-iotests/291.out b/tests/qemu-iotests/291.out new file mode 100644 index 000000000000..8c62017567e9 --- /dev/null +++ b/tests/qemu-iotests/291.out @@ -0,0 +1,80 @@ +QA output created by 291 + +=3D=3D=3D Initial image setup =3D=3D=3D + +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=3DIMGFMT size=3D10485760 +wrote 1048576/1048576 bytes at offset 3145728 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=3DIMGFMT size=3D10485760 backing_= file=3DTEST_DIR/t.IMGFMT.base backing_fmt=3DIMGFMT +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 1048576/1048576 bytes at offset 3145728 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 1048576/1048576 bytes at offset 2097152 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=3D=3D=3D Bitmap preservation not possible to non-qcow2 =3D=3D=3D + +qemu-img: Format driver 'raw' does not support bitmaps + +=3D=3D=3D Convert with bitmap preservation =3D=3D=3D + +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 10 MiB (10485760 bytes) +disk size: 4.39 MiB +Format specific information: + compat: 1.1 + compression type: zlib + lazy refcounts: false + bitmaps: + [0]: + flags: + name: b1 + granularity: 524288 + [1]: + flags: + [0]: auto + name: b2 + granularity: 65536 + refcount bits: 16 + corrupt: false +image: TEST_DIR/t.IMGFMT +file format: IMGFMT +virtual size: 10 MiB (10485760 bytes) +disk size: 4.48 MiB +Format specific information: + compat: 1.1 + compression type: zlib + lazy refcounts: false + bitmaps: + [0]: + flags: + name: b1 + granularity: 524288 + [1]: + flags: + [0]: auto + name: b2 + granularity: 65536 + [2]: + flags: + name: b0 + granularity: 65536 + refcount bits: 16 + corrupt: false + +=3D=3D=3D Check bitmap contents =3D=3D=3D + +[{ "start": 0, "length": 3145728, "depth": 0, "zero": false, "data": true,= "offset": OFFSET}, +{ "start": 3145728, "length": 1048576, "depth": 0, "zero": false, "data": = false}, +{ "start": 4194304, "length": 6291456, "depth": 0, "zero": false, "data": = true, "offset": OFFSET}] +[{ "start": 0, "length": 1048576, "depth": 0, "zero": false, "data": true,= "offset": OFFSET}, +{ "start": 1048576, "length": 1048576, "depth": 0, "zero": false, "data": = false}, +{ "start": 2097152, "length": 8388608, "depth": 0, "zero": false, "data": = true, "offset": OFFSET}] +[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true,= "offset": OFFSET}, +{ "start": 2097152, "length": 1048576, "depth": 0, "zero": false, "data": = false}, +{ "start": 3145728, "length": 7340032, "depth": 0, "zero": false, "data": = true, "offset": OFFSET}] +*** done diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 445c26f8d2e2..d886fa0cb355 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -299,5 +299,6 @@ 288 quick 289 rw quick 290 rw auto quick +291 rw quick 292 rw auto quick 297 meta --=20 2.26.2