From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545385442962488.5605405987883; Fri, 21 Dec 2018 01:44:02 -0800 (PST) Received: from localhost ([::1]:44598 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHLd-0006hV-9h for importer@patchew.org; Fri, 21 Dec 2018 04:44:01 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48893) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDi-0008Mq-BQ for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDh-00074G-Ex for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:50 -0500 Received: from mx1.redhat.com ([209.132.183.28]:33998) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDb-0006uC-UT; Fri, 21 Dec 2018 04:35:44 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B16EF7E437; Fri, 21 Dec 2018 09:35:35 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2402571D46; Fri, 21 Dec 2018 09:35:34 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:19 -0500 Message-Id: <20181221093529.23855-2-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Fri, 21 Dec 2018 09:35:35 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 01/11] blockdev: abort transactions in reverse order X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Presently, we abort transactions in the same order they were processed in. Bitmap commands, though, attempt to restore backup data structures on abort. That's not valid, they need to be aborted in reverse chronological order. Replace the QSIMPLEQ data structure with a QTAILQ one, so we can iterate in reverse for the abort phase of the transaction. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- blockdev.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/blockdev.c b/blockdev.c index a6f71f9d83..43e4c22da5 100644 --- a/blockdev.c +++ b/blockdev.c @@ -1339,7 +1339,7 @@ struct BlkActionState { const BlkActionOps *ops; JobTxn *block_job_txn; TransactionProperties *txn_props; - QSIMPLEQ_ENTRY(BlkActionState) entry; + QTAILQ_ENTRY(BlkActionState) entry; }; =20 /* internal snapshot private data */ @@ -2266,8 +2266,8 @@ void qmp_transaction(TransactionActionList *dev_list, BlkActionState *state, *next; Error *local_err =3D NULL; =20 - QSIMPLEQ_HEAD(snap_bdrv_states, BlkActionState) snap_bdrv_states; - QSIMPLEQ_INIT(&snap_bdrv_states); + QTAILQ_HEAD(snap_bdrv_states, BlkActionState) snap_bdrv_states; + QTAILQ_INIT(&snap_bdrv_states); =20 /* Does this transaction get canceled as a group on failure? * If not, we don't really need to make a JobTxn. @@ -2298,7 +2298,7 @@ void qmp_transaction(TransactionActionList *dev_list, state->action =3D dev_info; state->block_job_txn =3D block_job_txn; state->txn_props =3D props; - QSIMPLEQ_INSERT_TAIL(&snap_bdrv_states, state, entry); + QTAILQ_INSERT_TAIL(&snap_bdrv_states, state, entry); =20 state->ops->prepare(state, &local_err); if (local_err) { @@ -2307,7 +2307,7 @@ void qmp_transaction(TransactionActionList *dev_list, } } =20 - QSIMPLEQ_FOREACH(state, &snap_bdrv_states, entry) { + QTAILQ_FOREACH(state, &snap_bdrv_states, entry) { if (state->ops->commit) { state->ops->commit(state); } @@ -2318,13 +2318,13 @@ void qmp_transaction(TransactionActionList *dev_lis= t, =20 delete_and_fail: /* failure, and it is all-or-none; roll back all operations */ - QSIMPLEQ_FOREACH(state, &snap_bdrv_states, entry) { + QTAILQ_FOREACH_REVERSE(state, &snap_bdrv_states, snap_bdrv_states, ent= ry) { if (state->ops->abort) { state->ops->abort(state); } } exit: - QSIMPLEQ_FOREACH_SAFE(state, &snap_bdrv_states, entry, next) { + QTAILQ_FOREACH_SAFE(state, &snap_bdrv_states, entry, next) { if (state->ops->clean) { state->ops->clean(state); } --=20 2.17.2 From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545385286602225.3078285982989; Fri, 21 Dec 2018 01:41:26 -0800 (PST) Received: from localhost ([::1]:44584 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHJ7-0004Ii-77 for importer@patchew.org; Fri, 21 Dec 2018 04:41:25 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48888) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDi-0008Mo-9y for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDh-00074N-Gb for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:50 -0500 Received: from mx1.redhat.com ([209.132.183.28]:33664) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDd-0006vv-Uw; Fri, 21 Dec 2018 04:35:46 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 20506796E8; Fri, 21 Dec 2018 09:35:37 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id DB3634AC; Fri, 21 Dec 2018 09:35:35 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:20 -0500 Message-Id: <20181221093529.23855-3-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Fri, 21 Dec 2018 09:35:37 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 02/11] block/dirty-bitmap: remove assertion from restore X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" When making a backup of a dirty bitmap (for transactions), we want to restore that backup whether or not the bitmap is enabled. It is perfectly valid to write into bitmaps that are disabled. It is only illegitimate for the guest to have done so. Remove this assertion. Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: John Snow --- block/dirty-bitmap.c | 1 - 1 file changed, 1 deletion(-) diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c index 89fd1d7f8b..6b688394e4 100644 --- a/block/dirty-bitmap.c +++ b/block/dirty-bitmap.c @@ -625,7 +625,6 @@ void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap, H= Bitmap **out) void bdrv_restore_dirty_bitmap(BdrvDirtyBitmap *bitmap, HBitmap *backup) { HBitmap *tmp =3D bitmap->bitmap; - assert(bdrv_dirty_bitmap_enabled(bitmap)); assert(!bdrv_dirty_bitmap_readonly(bitmap)); bitmap->bitmap =3D backup; hbitmap_free(tmp); --=20 2.17.2 From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545385457937192.25302245010766; Fri, 21 Dec 2018 01:44:17 -0800 (PST) Received: from localhost ([::1]:44599 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHLs-000717-F8 for importer@patchew.org; Fri, 21 Dec 2018 04:44:16 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48853) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDh-0008MT-Or for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDg-00073Z-Rz for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:49 -0500 Received: from mx1.redhat.com ([209.132.183.28]:47528) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDX-0006xM-Uj; Fri, 21 Dec 2018 04:35:41 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BB7102F39; Fri, 21 Dec 2018 09:35:38 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4B6D24AC; Fri, 21 Dec 2018 09:35:37 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:21 -0500 Message-Id: <20181221093529.23855-4-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Fri, 21 Dec 2018 09:35:38 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 03/11] blockdev: n-ary bitmap merge X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Especially outside of transactions, it is helpful to provide all-or-nothing semantics for bitmap merges. This facilitates the coalescing of multiple bitmaps into a single target for the "checkpoint" interpretation when assembling bitmaps that represent arbitrary points in time from component bitmaps. This is an incompatible change from the preliminary version of the API. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- blockdev.c | 75 ++++++++++++++++++++++++++++++-------------- qapi/block-core.json | 22 ++++++------- 2 files changed, 62 insertions(+), 35 deletions(-) diff --git a/blockdev.c b/blockdev.c index 43e4c22da5..6031c94121 100644 --- a/blockdev.c +++ b/blockdev.c @@ -2119,33 +2119,28 @@ static void block_dirty_bitmap_disable_abort(BlkAct= ionState *common) } } =20 +static BdrvDirtyBitmap *do_block_dirty_bitmap_merge(const char *node, + const char *target, + strList *bitmaps, + HBitmap **backup, + Error **errp); + static void block_dirty_bitmap_merge_prepare(BlkActionState *common, Error **errp) { BlockDirtyBitmapMerge *action; BlockDirtyBitmapState *state =3D DO_UPCAST(BlockDirtyBitmapState, common, common); - BdrvDirtyBitmap *merge_source; =20 if (action_check_completion_mode(common, errp) < 0) { return; } =20 action =3D common->action->u.x_block_dirty_bitmap_merge.data; - state->bitmap =3D block_dirty_bitmap_lookup(action->node, - action->dst_name, - &state->bs, - errp); - if (!state->bitmap) { - return; - } =20 - merge_source =3D bdrv_find_dirty_bitmap(state->bs, action->src_name); - if (!merge_source) { - return; - } - - bdrv_merge_dirty_bitmap(state->bitmap, merge_source, &state->backup, e= rrp); + state->bitmap =3D do_block_dirty_bitmap_merge(action->node, action->ta= rget, + action->bitmaps, &state->b= ackup, + errp); } =20 static void abort_prepare(BlkActionState *common, Error **errp) @@ -2977,24 +2972,56 @@ void qmp_x_block_dirty_bitmap_disable(const char *n= ode, const char *name, bdrv_disable_dirty_bitmap(bitmap); } =20 -void qmp_x_block_dirty_bitmap_merge(const char *node, const char *dst_name, - const char *src_name, Error **errp) +static BdrvDirtyBitmap *do_block_dirty_bitmap_merge(const char *node, + const char *target, + strList *bitmaps, + HBitmap **backup, + Error **errp) { BlockDriverState *bs; - BdrvDirtyBitmap *dst, *src; + BdrvDirtyBitmap *dst, *src, *anon; + strList *lst; + Error *local_err =3D NULL; =20 - dst =3D block_dirty_bitmap_lookup(node, dst_name, &bs, errp); + dst =3D block_dirty_bitmap_lookup(node, target, &bs, errp); if (!dst) { - return; + return NULL; } =20 - src =3D bdrv_find_dirty_bitmap(bs, src_name); - if (!src) { - error_setg(errp, "Dirty bitmap '%s' not found", src_name); - return; + anon =3D bdrv_create_dirty_bitmap(bs, bdrv_dirty_bitmap_granularity(ds= t), + NULL, errp); + if (!anon) { + return NULL; } =20 - bdrv_merge_dirty_bitmap(dst, src, NULL, errp); + for (lst =3D bitmaps; lst; lst =3D lst->next) { + src =3D bdrv_find_dirty_bitmap(bs, lst->value); + if (!src) { + error_setg(errp, "Dirty bitmap '%s' not found", lst->value); + dst =3D NULL; + goto out; + } + + bdrv_merge_dirty_bitmap(anon, src, NULL, &local_err); + if (local_err) { + error_propagate(errp, local_err); + dst =3D NULL; + goto out; + } + } + + /* Merge into dst; dst is unchanged on failure. */ + bdrv_merge_dirty_bitmap(dst, anon, backup, errp); + + out: + bdrv_release_dirty_bitmap(bs, anon); + return dst; +} + +void qmp_x_block_dirty_bitmap_merge(const char *node, const char *target, + strList *bitmaps, Error **errp) +{ + do_block_dirty_bitmap_merge(node, target, bitmaps, NULL, errp); } =20 BlockDirtyBitmapSha256 *qmp_x_debug_block_dirty_bitmap_sha256(const char *= node, diff --git a/qapi/block-core.json b/qapi/block-core.json index 762000f31f..a153ea4420 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -1821,14 +1821,14 @@ # # @node: name of device/node which the bitmap is tracking # -# @dst_name: name of the destination dirty bitmap +# @target: name of the destination dirty bitmap # -# @src_name: name of the source dirty bitmap +# @bitmaps: name(s) of the source dirty bitmap(s) # # Since: 3.0 ## { 'struct': 'BlockDirtyBitmapMerge', - 'data': { 'node': 'str', 'dst_name': 'str', 'src_name': 'str' } } + 'data': { 'node': 'str', 'target': 'str', 'bitmaps': ['str'] } } =20 ## # @block-dirty-bitmap-add: @@ -1943,23 +1943,23 @@ ## # @x-block-dirty-bitmap-merge: # -# FIXME: Rename @src_name and @dst_name to src-name and dst-name. -# -# Merge @src_name dirty bitmap to @dst_name dirty bitmap. @src_name dirty -# bitmap is unchanged. On error, @dst_name is unchanged. +# Merge dirty bitmaps listed in @bitmaps to the @target dirty bitmap. +# The @bitmaps dirty bitmaps are unchanged. +# On error, @target is unchanged. # # Returns: nothing on success # If @node is not a valid block device, DeviceNotFound -# If @dst_name or @src_name is not found, GenericError -# If bitmaps has different sizes or granularities, GenericError +# If any bitmap in @bitmaps or @target is not found, GenericError +# If any of the bitmaps have different sizes or granularities, +# GenericError # # Since: 3.0 # # Example: # # -> { "execute": "x-block-dirty-bitmap-merge", -# "arguments": { "node": "drive0", "dst_name": "bitmap0", -# "src_name": "bitmap1" } } +# "arguments": { "node": "drive0", "target": "bitmap0", +# "bitmaps": ["bitmap1"] } } # <- { "return": {} } # ## --=20 2.17.2 From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545385305425350.08744522691563; Fri, 21 Dec 2018 01:41:45 -0800 (PST) Received: from localhost ([::1]:44586 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHJQ-0004YU-0w for importer@patchew.org; Fri, 21 Dec 2018 04:41:44 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48913) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDi-0008NQ-P0 for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDh-00074c-Mv for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:50 -0500 Received: from mx1.redhat.com ([209.132.183.28]:47536) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDa-0006yL-PU; Fri, 21 Dec 2018 04:35:43 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7F56D613C; Fri, 21 Dec 2018 09:35:40 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4177617130; Fri, 21 Dec 2018 09:35:39 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:22 -0500 Message-Id: <20181221093529.23855-5-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Fri, 21 Dec 2018 09:35:40 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 04/11] block: remove 'x' prefix from experimental bitmap APIs X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The 'x' prefix was added because I was uncertain of the direction we'd take for the libvirt API. With the general approach solidified, I feel comfortable committing to this API for 4.0. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- blockdev.c | 22 +++++++++++----------- qapi/block-core.json | 34 +++++++++++++++++----------------- qapi/transaction.json | 12 ++++++------ tests/qemu-iotests/223 | 4 ++-- 4 files changed, 36 insertions(+), 36 deletions(-) diff --git a/blockdev.c b/blockdev.c index 6031c94121..8e37dd659e 100644 --- a/blockdev.c +++ b/blockdev.c @@ -1963,7 +1963,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionS= tate *common, action->has_granularity, action->granularit= y, action->has_persistent, action->persistent, action->has_autoload, action->autoload, - action->has_x_disabled, action->x_disabled, + action->has_disabled, action->disabled, &local_err); =20 if (!local_err) { @@ -2048,7 +2048,7 @@ static void block_dirty_bitmap_enable_prepare(BlkActi= onState *common, return; } =20 - action =3D common->action->u.x_block_dirty_bitmap_enable.data; + action =3D common->action->u.block_dirty_bitmap_enable.data; state->bitmap =3D block_dirty_bitmap_lookup(action->node, action->name, NULL, @@ -2089,7 +2089,7 @@ static void block_dirty_bitmap_disable_prepare(BlkAct= ionState *common, return; } =20 - action =3D common->action->u.x_block_dirty_bitmap_disable.data; + action =3D common->action->u.block_dirty_bitmap_disable.data; state->bitmap =3D block_dirty_bitmap_lookup(action->node, action->name, NULL, @@ -2136,7 +2136,7 @@ static void block_dirty_bitmap_merge_prepare(BlkActio= nState *common, return; } =20 - action =3D common->action->u.x_block_dirty_bitmap_merge.data; + action =3D common->action->u.block_dirty_bitmap_merge.data; =20 state->bitmap =3D do_block_dirty_bitmap_merge(action->node, action->ta= rget, action->bitmaps, &state->b= ackup, @@ -2204,17 +2204,17 @@ static const BlkActionOps actions[] =3D { .commit =3D block_dirty_bitmap_free_backup, .abort =3D block_dirty_bitmap_restore, }, - [TRANSACTION_ACTION_KIND_X_BLOCK_DIRTY_BITMAP_ENABLE] =3D { + [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_ENABLE] =3D { .instance_size =3D sizeof(BlockDirtyBitmapState), .prepare =3D block_dirty_bitmap_enable_prepare, .abort =3D block_dirty_bitmap_enable_abort, }, - [TRANSACTION_ACTION_KIND_X_BLOCK_DIRTY_BITMAP_DISABLE] =3D { + [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_DISABLE] =3D { .instance_size =3D sizeof(BlockDirtyBitmapState), .prepare =3D block_dirty_bitmap_disable_prepare, .abort =3D block_dirty_bitmap_disable_abort, }, - [TRANSACTION_ACTION_KIND_X_BLOCK_DIRTY_BITMAP_MERGE] =3D { + [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_MERGE] =3D { .instance_size =3D sizeof(BlockDirtyBitmapState), .prepare =3D block_dirty_bitmap_merge_prepare, .commit =3D block_dirty_bitmap_free_backup, @@ -2930,7 +2930,7 @@ void qmp_block_dirty_bitmap_clear(const char *node, c= onst char *name, bdrv_clear_dirty_bitmap(bitmap, NULL); } =20 -void qmp_x_block_dirty_bitmap_enable(const char *node, const char *name, +void qmp_block_dirty_bitmap_enable(const char *node, const char *name, Error **errp) { BlockDriverState *bs; @@ -2951,7 +2951,7 @@ void qmp_x_block_dirty_bitmap_enable(const char *node= , const char *name, bdrv_enable_dirty_bitmap(bitmap); } =20 -void qmp_x_block_dirty_bitmap_disable(const char *node, const char *name, +void qmp_block_dirty_bitmap_disable(const char *node, const char *name, Error **errp) { BlockDriverState *bs; @@ -3018,8 +3018,8 @@ static BdrvDirtyBitmap *do_block_dirty_bitmap_merge(c= onst char *node, return dst; } =20 -void qmp_x_block_dirty_bitmap_merge(const char *node, const char *target, - strList *bitmaps, Error **errp) +void qmp_block_dirty_bitmap_merge(const char *node, const char *target, + strList *bitmaps, Error **errp) { do_block_dirty_bitmap_merge(node, target, bitmaps, NULL, errp); } diff --git a/qapi/block-core.json b/qapi/block-core.json index a153ea4420..91685be6c2 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -1806,15 +1806,15 @@ # Currently, all dirty tracking bitmaps are loaded from Qcow2 on # open. # -# @x-disabled: the bitmap is created in the disabled state, which means th= at -# it will not track drive changes. The bitmap may be enabled = with -# x-block-dirty-bitmap-enable. Default is false. (Since: 3.0) +# @disabled: the bitmap is created in the disabled state, which means that +# it will not track drive changes. The bitmap may be enabled wi= th +# block-dirty-bitmap-enable. Default is false. (Since: 4.0) # # Since: 2.4 ## { 'struct': 'BlockDirtyBitmapAdd', 'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32', - '*persistent': 'bool', '*autoload': 'bool', '*x-disabled': 'bo= ol' } } + '*persistent': 'bool', '*autoload': 'bool', '*disabled': 'bool= ' } } =20 ## # @BlockDirtyBitmapMerge: @@ -1825,7 +1825,7 @@ # # @bitmaps: name(s) of the source dirty bitmap(s) # -# Since: 3.0 +# Since: 4.0 ## { 'struct': 'BlockDirtyBitmapMerge', 'data': { 'node': 'str', 'target': 'str', 'bitmaps': ['str'] } } @@ -1899,7 +1899,7 @@ 'data': 'BlockDirtyBitmap' } =20 ## -# @x-block-dirty-bitmap-enable: +# @block-dirty-bitmap-enable: # # Enables a dirty bitmap so that it will begin tracking disk changes. # @@ -1907,20 +1907,20 @@ # If @node is not a valid block device, DeviceNotFound # If @name is not found, GenericError with an explanation # -# Since: 3.0 +# Since: 4.0 # # Example: # -# -> { "execute": "x-block-dirty-bitmap-enable", +# -> { "execute": "block-dirty-bitmap-enable", # "arguments": { "node": "drive0", "name": "bitmap0" } } # <- { "return": {} } # ## - { 'command': 'x-block-dirty-bitmap-enable', + { 'command': 'block-dirty-bitmap-enable', 'data': 'BlockDirtyBitmap' } =20 ## -# @x-block-dirty-bitmap-disable: +# @block-dirty-bitmap-disable: # # Disables a dirty bitmap so that it will stop tracking disk changes. # @@ -1928,20 +1928,20 @@ # If @node is not a valid block device, DeviceNotFound # If @name is not found, GenericError with an explanation # -# Since: 3.0 +# Since: 4.0 # # Example: # -# -> { "execute": "x-block-dirty-bitmap-disable", +# -> { "execute": "block-dirty-bitmap-disable", # "arguments": { "node": "drive0", "name": "bitmap0" } } # <- { "return": {} } # ## - { 'command': 'x-block-dirty-bitmap-disable', + { 'command': 'block-dirty-bitmap-disable', 'data': 'BlockDirtyBitmap' } =20 ## -# @x-block-dirty-bitmap-merge: +# @block-dirty-bitmap-merge: # # Merge dirty bitmaps listed in @bitmaps to the @target dirty bitmap. # The @bitmaps dirty bitmaps are unchanged. @@ -1953,17 +1953,17 @@ # If any of the bitmaps have different sizes or granularities, # GenericError # -# Since: 3.0 +# Since: 4.0 # # Example: # -# -> { "execute": "x-block-dirty-bitmap-merge", +# -> { "execute": "block-dirty-bitmap-merge", # "arguments": { "node": "drive0", "target": "bitmap0", # "bitmaps": ["bitmap1"] } } # <- { "return": {} } # ## - { 'command': 'x-block-dirty-bitmap-merge', + { 'command': 'block-dirty-bitmap-merge', 'data': 'BlockDirtyBitmapMerge' } =20 ## diff --git a/qapi/transaction.json b/qapi/transaction.json index 5875cdb16c..95edb78227 100644 --- a/qapi/transaction.json +++ b/qapi/transaction.json @@ -46,9 +46,9 @@ # - @abort: since 1.6 # - @block-dirty-bitmap-add: since 2.5 # - @block-dirty-bitmap-clear: since 2.5 -# - @x-block-dirty-bitmap-enable: since 3.0 -# - @x-block-dirty-bitmap-disable: since 3.0 -# - @x-block-dirty-bitmap-merge: since 3.1 +# - @block-dirty-bitmap-enable: since 4.0 +# - @block-dirty-bitmap-disable: since 4.0 +# - @block-dirty-bitmap-merge: since 4.0 # - @blockdev-backup: since 2.3 # - @blockdev-snapshot: since 2.5 # - @blockdev-snapshot-internal-sync: since 1.7 @@ -62,9 +62,9 @@ 'abort': 'Abort', 'block-dirty-bitmap-add': 'BlockDirtyBitmapAdd', 'block-dirty-bitmap-clear': 'BlockDirtyBitmap', - 'x-block-dirty-bitmap-enable': 'BlockDirtyBitmap', - 'x-block-dirty-bitmap-disable': 'BlockDirtyBitmap', - 'x-block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge', + 'block-dirty-bitmap-enable': 'BlockDirtyBitmap', + 'block-dirty-bitmap-disable': 'BlockDirtyBitmap', + 'block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge', 'blockdev-backup': 'BlockdevBackup', 'blockdev-snapshot': 'BlockdevSnapshot', 'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal', diff --git a/tests/qemu-iotests/223 b/tests/qemu-iotests/223 index 397b865d34..5513dc6215 100755 --- a/tests/qemu-iotests/223 +++ b/tests/qemu-iotests/223 @@ -112,9 +112,9 @@ _send_qemu_cmd $QEMU_HANDLE '{"execute":"qmp_capabiliti= es"}' "return" _send_qemu_cmd $QEMU_HANDLE '{"execute":"blockdev-add", "arguments":{"driver":"qcow2", "node-name":"n", "file":{"driver":"file", "filename":"'"$TEST_IMG"'"}}}' "return" -_send_qemu_cmd $QEMU_HANDLE '{"execute":"x-block-dirty-bitmap-disable", +_send_qemu_cmd $QEMU_HANDLE '{"execute":"block-dirty-bitmap-disable", "arguments":{"node":"n", "name":"b"}}' "return" -_send_qemu_cmd $QEMU_HANDLE '{"execute":"x-block-dirty-bitmap-disable", +_send_qemu_cmd $QEMU_HANDLE '{"execute":"block-dirty-bitmap-disable", "arguments":{"node":"n", "name":"b2"}}' "return" _send_qemu_cmd $QEMU_HANDLE '{"execute":"nbd-server-start", "arguments":{"addr":{"type":"unix", --=20 2.17.2 From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545385263780455.5698406076435; Fri, 21 Dec 2018 01:41:03 -0800 (PST) Received: from localhost ([::1]:44583 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHIk-0003yH-3n for importer@patchew.org; Fri, 21 Dec 2018 04:41:02 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48887) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDi-0008Mn-9w for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDh-000748-E4 for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:50 -0500 Received: from mx1.redhat.com ([209.132.183.28]:40830) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDc-0006zQ-2B; Fri, 21 Dec 2018 04:35:44 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 217BE41A55; Fri, 21 Dec 2018 09:35:43 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id AD8B24AC; Fri, 21 Dec 2018 09:35:40 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:23 -0500 Message-Id: <20181221093529.23855-6-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Fri, 21 Dec 2018 09:35:43 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 05/11] iotests.py: don't abort if IMGKEYSECRET is undefined X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Instead of using os.environ[], use .get with a default of empty string to match the setup in check to allow us to import the iotests module (for debugging, say) without needing a crafted environment just to import the module. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index d537538ba0..a34e66813a 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -63,7 +63,7 @@ socket_scm_helper =3D os.environ.get('SOCKET_SCM_HELPER',= 'socket_scm_helper') debug =3D False =20 luks_default_secret_object =3D 'secret,id=3Dkeysec0,data=3D' + \ - os.environ['IMGKEYSECRET'] + os.environ.get('IMGKEYSECRET', '') luks_default_key_secret_opt =3D 'key-secret=3Dkeysec0' =20 =20 --=20 2.17.2 From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545385116588687.7773667288012; Fri, 21 Dec 2018 01:38:36 -0800 (PST) Received: from localhost ([::1]:44567 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHGI-0001mS-6U for importer@patchew.org; Fri, 21 Dec 2018 04:38:30 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48880) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDi-0008Mj-7O for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDh-000743-Cn for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:50 -0500 Received: from mx1.redhat.com ([209.132.183.28]:51389) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDe-00070e-3p; Fri, 21 Dec 2018 04:35:46 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C9226169742; Fri, 21 Dec 2018 09:35:44 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4BC4168D45; Fri, 21 Dec 2018 09:35:43 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:24 -0500 Message-Id: <20181221093529.23855-7-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Fri, 21 Dec 2018 09:35:44 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 06/11] iotests: add filter_generated_node_ids X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" To mimic the common filter of the same name, but for the python tests. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index a34e66813a..9595429fea 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -239,6 +239,9 @@ def filter_testfiles(msg): prefix =3D os.path.join(test_dir, "%s-" % (os.getpid())) return msg.replace(prefix, 'TEST_DIR/PID-') =20 +def filter_generated_node_ids(msg): + return re.sub("#block[0-9]+", "NODE_NAME", msg) + def filter_img_info(output, filename): lines =3D [] for line in output.split('\n'): --=20 2.17.2 From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545385127449641.2553741426699; Fri, 21 Dec 2018 01:38:47 -0800 (PST) Received: from localhost ([::1]:44569 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHGY-00024J-3z for importer@patchew.org; Fri, 21 Dec 2018 04:38:46 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48914) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDi-0008NR-P5 for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDh-00074w-Q5 for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:50 -0500 Received: from mx1.redhat.com ([209.132.183.28]:47584) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDe-00071Y-Vi; Fri, 21 Dec 2018 04:35:47 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2F1A52F39; Fri, 21 Dec 2018 09:35:46 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id E95D04AC; Fri, 21 Dec 2018 09:35:44 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:25 -0500 Message-Id: <20181221093529.23855-8-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Fri, 21 Dec 2018 09:35:46 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 07/11] iotests: add qmp recursive sorting function X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Python before 3.6 does not sort dictionaries (including kwargs). Therefore, printing QMP objects involves sorting the keys to have a predictable ordering in the iotests output. This means that iotests output will sometimes show arguments in an order not specified by the test author. Presently, we accomplish this by using json.dumps' sort_keys argument, where we only serialize the arguments dictionary, but not the command. However, if we want to pretty-print QMP objects being sent to the QEMU process, we need to build the entire command before logging it. Ordinarily, this would then involve "arguments" being sorted above "execute", which would necessitate a rather ugly and harder-to-read change to many iotests outputs. To facilitate pretty-printing AND maintaining predictable output AND having "arguments" sort after "execute", add a custom sort function that takes a dictionary and recursively builds an OrderedDict that maintains the specific key order we wish to see in iotests output. The qmp_log function uses this to build a QMP object that keeps "execute" above "arguments", but sorts all keys and keys in any subdicts in "arguments" lexicographically to maintain consistent iotests output, with no incompatible changes to any current test. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 9595429fea..565eebb1ab 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -30,6 +30,7 @@ import signal import logging import atexit import io +from collections import OrderedDict =20 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scrip= ts')) import qtest @@ -75,6 +76,16 @@ def qemu_img(*args): sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode,= ' '.join(qemu_img_args + list(args)))) return exitcode =20 +def ordered_kwargs(kwargs): + # kwargs prior to 3.6 are not ordered, so: + od =3D OrderedDict() + for k, v in sorted(kwargs.items()): + if isinstance(v, dict): + od[k] =3D ordered_kwargs(v) + else: + od[k] =3D v + return od + def qemu_img_create(*args): args =3D list(args) =20 @@ -257,8 +268,10 @@ def filter_img_info(output, filename): def log(msg, filters=3D[]): for flt in filters: msg =3D flt(msg) - if type(msg) is dict or type(msg) is list: - print(json.dumps(msg, sort_keys=3DTrue)) + if isinstance(msg, dict) or isinstance(msg, list): + # Don't sort if it's already sorted + do_sort =3D not isinstance(msg, OrderedDict) + print(json.dumps(msg, sort_keys=3Ddo_sort)) else: print(msg) =20 @@ -448,8 +461,11 @@ class VM(qtest.QEMUQtestMachine): return result =20 def qmp_log(self, cmd, filters=3D[filter_testfiles], **kwargs): - logmsg =3D '{"execute": "%s", "arguments": %s}' % \ - (cmd, json.dumps(kwargs, sort_keys=3DTrue)) + full_cmd =3D OrderedDict(( + ("execute", cmd), + ("arguments", ordered_kwargs(kwargs)) + )) + logmsg =3D json.dumps(full_cmd) log(logmsg, filters) result =3D self.qmp(cmd, **kwargs) log(json.dumps(result, sort_keys=3DTrue), filters) --=20 2.17.2 From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545385125504487.81224353076414; Fri, 21 Dec 2018 01:38:45 -0800 (PST) Received: from localhost ([::1]:44568 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHGV-00021u-1W for importer@patchew.org; Fri, 21 Dec 2018 04:38:43 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48955) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDk-0008PY-FC for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDj-00076u-P0 for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:52 -0500 Received: from mx1.redhat.com ([209.132.183.28]:60660) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDh-00073p-IB; Fri, 21 Dec 2018 04:35:49 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C74A719CBD7; Fri, 21 Dec 2018 09:35:48 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id 59D544AC; Fri, 21 Dec 2018 09:35:46 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:26 -0500 Message-Id: <20181221093529.23855-9-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Fri, 21 Dec 2018 09:35:48 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 08/11] iotests: remove default filters from qmp_log X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Several places in iotests deal with serializing objects into JSON strings, but to add pretty-printing it seems desirable to localize all of those cases. log() seems like a good candidate for that centralized behavior. log() can already serialize json objects, but when it does so, it assumes filters=3D[] operates on QMP objects, not strings. qmp_log currently operates by dumping outgoing and incoming QMP objects into strings and filtering them assuming that filters=3D[] are string filters. To have qmp_log use log's serialization, qmp_log will need to accept only qmp filters, not text filters. However, only a single caller of qmp_log actually requires any filters at all. I remove the default filter and add it explicitly to the caller in preparation for refactoring qmp_log to use rich filters instead. test 206 is amended to name the filter explicitly and the default is removed. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/206 | 8 ++++++-- tests/qemu-iotests/iotests.py | 2 +- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/206 b/tests/qemu-iotests/206 index 128c334c7c..e92550fa59 100755 --- a/tests/qemu-iotests/206 +++ b/tests/qemu-iotests/206 @@ -26,7 +26,9 @@ from iotests import imgfmt iotests.verify_image_format(supported_fmts=3D['qcow2']) =20 def blockdev_create(vm, options): - result =3D vm.qmp_log('blockdev-create', job_id=3D'job0', options=3Dop= tions) + result =3D vm.qmp_log('blockdev-create', + filters=3D[iotests.filter_testfiles], + job_id=3D'job0', options=3Doptions) =20 if 'return' in result: assert result['return'] =3D=3D {} @@ -52,7 +54,9 @@ with iotests.FilePath('t.qcow2') as disk_path, \ 'filename': disk_path, 'size': 0 }) =20 - vm.qmp_log('blockdev-add', driver=3D'file', filename=3Ddisk_path, + vm.qmp_log('blockdev-add', + filters=3D[iotests.filter_testfiles], + driver=3D'file', filename=3Ddisk_path, node_name=3D'imgfile') =20 blockdev_create(vm, { 'driver': imgfmt, diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 565eebb1ab..57fe20db45 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -460,7 +460,7 @@ class VM(qtest.QEMUQtestMachine): result.append(filter_qmp_event(ev)) return result =20 - def qmp_log(self, cmd, filters=3D[filter_testfiles], **kwargs): + def qmp_log(self, cmd, filters=3D[], **kwargs): full_cmd =3D OrderedDict(( ("execute", cmd), ("arguments", ordered_kwargs(kwargs)) --=20 2.17.2 From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545385433927989.318502893825; Fri, 21 Dec 2018 01:43:53 -0800 (PST) Received: from localhost ([::1]:44597 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHLP-0006V8-Ul for importer@patchew.org; Fri, 21 Dec 2018 04:43:48 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49042) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDq-00005i-KA for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDp-0007C2-R8 for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:35:58 -0500 Received: from mx1.redhat.com ([209.132.183.28]:49656) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDl-000786-Gp; Fri, 21 Dec 2018 04:35:53 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D286CC049D7C; Fri, 21 Dec 2018 09:35:52 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id F3E294AC; Fri, 21 Dec 2018 09:35:48 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:27 -0500 Message-Id: <20181221093529.23855-10-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Fri, 21 Dec 2018 09:35:52 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 09/11] iotests: change qmp_log filters to expect QMP objects only X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" As laid out in the previous commit's message: ``` Several places in iotests deal with serializing objects into JSON strings, but to add pretty-printing it seems desirable to localize all of those cases. log() seems like a good candidate for that centralized behavior. log() can already serialize json objects, but when it does so, it assumes filters=3D[] operates on QMP objects, not strings. qmp_log currently operates by dumping outgoing and incoming QMP objects into strings and filtering them assuming that filters=3D[] are string filters. ``` Therefore: Change qmp_log to treat filters as if they're always qmp object filters, then change the logging call to rely on log()'s ability to serialize QMP objects, so we're not duplicating that effort. Add a qmp version of filter_testfiles and adjust the only caller using it for qmp_log to use the qmp version. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/206 | 4 ++-- tests/qemu-iotests/iotests.py | 28 +++++++++++++++++++++++++--- 2 files changed, 27 insertions(+), 5 deletions(-) diff --git a/tests/qemu-iotests/206 b/tests/qemu-iotests/206 index e92550fa59..5bb738bf23 100755 --- a/tests/qemu-iotests/206 +++ b/tests/qemu-iotests/206 @@ -27,7 +27,7 @@ iotests.verify_image_format(supported_fmts=3D['qcow2']) =20 def blockdev_create(vm, options): result =3D vm.qmp_log('blockdev-create', - filters=3D[iotests.filter_testfiles], + filters=3D[iotests.filter_qmp_testfiles], job_id=3D'job0', options=3Doptions) =20 if 'return' in result: @@ -55,7 +55,7 @@ with iotests.FilePath('t.qcow2') as disk_path, \ 'size': 0 }) =20 vm.qmp_log('blockdev-add', - filters=3D[iotests.filter_testfiles], + filters=3D[iotests.filter_qmp_testfiles], driver=3D'file', filename=3Ddisk_path, node_name=3D'imgfile') =20 diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 57fe20db45..a96a7010d4 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -246,10 +246,33 @@ def filter_qmp_event(event): event['timestamp']['microseconds'] =3D 'USECS' return event =20 +def filter_qmp(qmsg, filter_fn): + '''Given a string filter, filter a QMP object's values. + filter_fn takes a (key, value) pair.''' + # Iterate through either lists or dicts; + if isinstance(qmsg, list): + items =3D enumerate(qmsg) + else: + items =3D qmsg.items() + + for k, v in items: + if isinstance(v, list) or isinstance(v, dict): + qmsg[k] =3D filter_qmp(v, filter_fn) + else: + qmsg[k] =3D filter_fn(k, v) + return qmsg + def filter_testfiles(msg): prefix =3D os.path.join(test_dir, "%s-" % (os.getpid())) return msg.replace(prefix, 'TEST_DIR/PID-') =20 +def filter_qmp_testfiles(qmsg): + def _filter(key, value): + if key =3D=3D 'filename' or key =3D=3D 'backing-file': + return filter_testfiles(value) + return value + return filter_qmp(qmsg, _filter) + def filter_generated_node_ids(msg): return re.sub("#block[0-9]+", "NODE_NAME", msg) =20 @@ -465,10 +488,9 @@ class VM(qtest.QEMUQtestMachine): ("execute", cmd), ("arguments", ordered_kwargs(kwargs)) )) - logmsg =3D json.dumps(full_cmd) - log(logmsg, filters) + log(full_cmd, filters) result =3D self.qmp(cmd, **kwargs) - log(json.dumps(result, sort_keys=3DTrue), filters) + log(result, filters) return result =20 def run_job(self, job, auto_finalize=3DTrue, auto_dismiss=3DFalse): --=20 2.17.2 From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545385302605293.77825180260095; Fri, 21 Dec 2018 01:41:42 -0800 (PST) Received: from localhost ([::1]:44585 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHJN-0004Wc-49 for importer@patchew.org; Fri, 21 Dec 2018 04:41:41 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49068) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDs-000079-BT for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:36:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDr-0007E6-KJ for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:36:00 -0500 Received: from mx1.redhat.com ([209.132.183.28]:33770) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDn-00079F-Dp; Fri, 21 Dec 2018 04:35:55 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BBF308AE76; Fri, 21 Dec 2018 09:35:54 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id 37B7069313; Fri, 21 Dec 2018 09:35:52 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:28 -0500 Message-Id: <20181221093529.23855-11-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Fri, 21 Dec 2018 09:35:54 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 10/11] iotests: implement pretty-print for log and qmp_log X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" If iotests have lines exceeding >998 characters long, git doesn't want to send it plaintext to the list. We can solve this by allowing the iotests to use pretty printed QMP output that we can match against instead. As a bonus, it's much nicer for human eyes too. Signed-off-by: John Snow Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index a96a7010d4..cbedfaf1df 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -288,13 +288,18 @@ def filter_img_info(output, filename): lines.append(line) return '\n'.join(lines) =20 -def log(msg, filters=3D[]): +def log(msg, filters=3D[], indent=3DNone): + '''Logs either a string message or a JSON serializable message (like Q= MP). + If indent is provided, JSON serializable messages are pretty-printed.'= '' for flt in filters: msg =3D flt(msg) if isinstance(msg, dict) or isinstance(msg, list): + # Python < 3.4 needs to know not to add whitespace when pretty-pri= nting: + separators =3D (', ', ': ') if indent is None else (',', ': ') # Don't sort if it's already sorted do_sort =3D not isinstance(msg, OrderedDict) - print(json.dumps(msg, sort_keys=3Ddo_sort)) + print(json.dumps(msg, sort_keys=3Ddo_sort, + indent=3Dindent, separators=3Dseparators)) else: print(msg) =20 @@ -483,14 +488,14 @@ class VM(qtest.QEMUQtestMachine): result.append(filter_qmp_event(ev)) return result =20 - def qmp_log(self, cmd, filters=3D[], **kwargs): + def qmp_log(self, cmd, filters=3D[], indent=3DNone, **kwargs): full_cmd =3D OrderedDict(( ("execute", cmd), ("arguments", ordered_kwargs(kwargs)) )) - log(full_cmd, filters) + log(full_cmd, filters, indent=3Dindent) result =3D self.qmp(cmd, **kwargs) - log(result, filters) + log(result, filters, indent=3Dindent) return result =20 def run_job(self, job, auto_finalize=3DTrue, auto_dismiss=3DFalse): --=20 2.17.2 From nobody Fri May 17 05:54:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 15453855726761001.5967757954585; Fri, 21 Dec 2018 01:46:12 -0800 (PST) Received: from localhost ([::1]:44617 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHNj-0000TQ-AB for importer@patchew.org; Fri, 21 Dec 2018 04:46:11 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49093) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaHDt-00008Q-LZ for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:36:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gaHDs-0007Er-CZ for qemu-devel@nongnu.org; Fri, 21 Dec 2018 04:36:01 -0500 Received: from mx1.redhat.com ([209.132.183.28]:49318) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gaHDp-0007BB-Ip; Fri, 21 Dec 2018 04:35:57 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CC5C27654F; Fri, 21 Dec 2018 09:35:56 +0000 (UTC) Received: from probe.redhat.com (ovpn-121-225.rdu2.redhat.com [10.10.121.225]) by smtp.corp.redhat.com (Postfix) with ESMTP id E8C1A4AC; Fri, 21 Dec 2018 09:35:54 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 21 Dec 2018 04:35:29 -0500 Message-Id: <20181221093529.23855-12-jsnow@redhat.com> In-Reply-To: <20181221093529.23855-1-jsnow@redhat.com> References: <20181221093529.23855-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Fri, 21 Dec 2018 09:35:56 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v6 11/11] iotests: add iotest 236 for testing bitmap merge X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" New interface, new smoke test. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- tests/qemu-iotests/236 | 161 +++++++++++++++++ tests/qemu-iotests/236.out | 351 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 513 insertions(+) create mode 100755 tests/qemu-iotests/236 create mode 100644 tests/qemu-iotests/236.out diff --git a/tests/qemu-iotests/236 b/tests/qemu-iotests/236 new file mode 100755 index 0000000000..79a6381f8e --- /dev/null +++ b/tests/qemu-iotests/236 @@ -0,0 +1,161 @@ +#!/usr/bin/env python +# +# Test bitmap merges. +# +# Copyright (c) 2018 John Snow for Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +# owner=3Djsnow@redhat.com + +import iotests +from iotests import log + +iotests.verify_image_format(supported_fmts=3D['generic']) +size =3D 64 * 1024 * 1024 +granularity =3D 64 * 1024 + +patterns =3D [("0x5d", "0", "64k"), + ("0xd5", "1M", "64k"), + ("0xdc", "32M", "64k"), + ("0xcd", "0x3ff0000", "64k")] # 64M - 64K + +overwrite =3D [("0xab", "0", "64k"), # Full overwrite + ("0xad", "0x00f8000", "64k"), # Partial-left (1M-32K) + ("0x1d", "0x2008000", "64k"), # Partial-right (32M+32K) + ("0xea", "0x3fe0000", "64k")] # Adjacent-left (64M - 128K) + +def query_bitmaps(vm): + res =3D vm.qmp("query-block") + return { "bitmaps": { device['device']: device.get('dirty-bitmaps', []= ) for + device in res['return'] } } + +with iotests.FilePath('img') as img_path, \ + iotests.VM() as vm: + + log('--- Preparing image & VM ---\n') + iotests.qemu_img_create('-f', iotests.imgfmt, img_path, str(size)) + vm.add_drive(img_path) + vm.launch() + + log('\n--- Adding preliminary bitmaps A & B ---\n') + vm.qmp_log("block-dirty-bitmap-add", node=3D"drive0", + name=3D"bitmapA", granularity=3Dgranularity) + vm.qmp_log("block-dirty-bitmap-add", node=3D"drive0", + name=3D"bitmapB", granularity=3Dgranularity) + + # Dirties 4 clusters. count=3D262144 + log('\n--- Emulating writes ---\n') + for p in patterns: + cmd =3D "write -P%s %s %s" % p + log(cmd) + log(vm.hmp_qemu_io("drive0", cmd)) + + log(query_bitmaps(vm), indent=3D2) + + log('\n--- Submitting & Aborting Transaction ---\n') + vm.qmp_log("transaction", indent=3D2, actions=3D[ + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapB" }}, + { "type": "block-dirty-bitmap-add", + "data": { "node": "drive0", "name": "bitmapC", + "granularity": granularity }}, + { "type": "block-dirty-bitmap-clear", + "data": { "node": "drive0", "name": "bitmapA" }}, + { "type": "abort", "data": {}} + ]) + log(query_bitmaps(vm), indent=3D2) + + log('\n--- Disabling B & Adding C ---\n') + vm.qmp_log("transaction", indent=3D2, actions=3D[ + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapB" }}, + { "type": "block-dirty-bitmap-add", + "data": { "node": "drive0", "name": "bitmapC", + "granularity": granularity }}, + # Purely extraneous, but test that it works: + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapC" }}, + { "type": "block-dirty-bitmap-enable", + "data": { "node": "drive0", "name": "bitmapC" }}, + ]) + + log('\n--- Emulating further writes ---\n') + # Dirties 6 clusters, 3 of which are new in contrast to "A". + # A =3D 64 * 1024 * (4 + 3) =3D 458752 + # C =3D 64 * 1024 * 6 =3D 393216 + for p in overwrite: + cmd =3D "write -P%s %s %s" % p + log(cmd) + log(vm.hmp_qemu_io("drive0", cmd)) + + log('\n--- Disabling A & C ---\n') + vm.qmp_log("transaction", indent=3D2, actions=3D[ + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapA" }}, + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapC" }} + ]) + + # A: 7 clusters + # B: 4 clusters + # C: 6 clusters + log(query_bitmaps(vm), indent=3D2) + + log('\n--- Submitting & Aborting Merge Transaction ---\n') + vm.qmp_log("transaction", indent=3D2, actions=3D[ + { "type": "block-dirty-bitmap-add", + "data": { "node": "drive0", "name": "bitmapD", + "disabled": True, "granularity": granularity }}, + { "type": "block-dirty-bitmap-merge", + "data": { "node": "drive0", "target": "bitmapD", + "bitmaps": ["bitmapB", "bitmapC"] }}, + { "type": "abort", "data": {}} + ]) + log(query_bitmaps(vm), indent=3D2) + + log('\n--- Creating D as a merge of B & C ---\n') + # Good hygiene: create a disabled bitmap as a merge target. + vm.qmp_log("transaction", indent=3D2, actions=3D[ + { "type": "block-dirty-bitmap-add", + "data": { "node": "drive0", "name": "bitmapD", + "disabled": True, "granularity": granularity }}, + { "type": "block-dirty-bitmap-merge", + "data": { "node": "drive0", "target": "bitmapD", + "bitmaps": ["bitmapB", "bitmapC"] }} + ]) + + # A and D should now both have 7 clusters apiece. + # B and C remain unchanged with 4 and 6 respectively. + log(query_bitmaps(vm), indent=3D2) + + # A and D should be equivalent. + # Some formats round the size of the disk, so don't print the checksum= s. + check_a =3D vm.qmp('x-debug-block-dirty-bitmap-sha256', + node=3D"drive0", name=3D"bitmapA")['return']['sha256'] + check_d =3D vm.qmp('x-debug-block-dirty-bitmap-sha256', + node=3D"drive0", name=3D"bitmapD")['return']['sha256'] + assert(check_a =3D=3D check_d) + + log('\n--- Removing bitmaps A, B, C, and D ---\n') + vm.qmp_log("block-dirty-bitmap-remove", node=3D"drive0", name=3D"bitma= pA") + vm.qmp_log("block-dirty-bitmap-remove", node=3D"drive0", name=3D"bitma= pB") + vm.qmp_log("block-dirty-bitmap-remove", node=3D"drive0", name=3D"bitma= pC") + vm.qmp_log("block-dirty-bitmap-remove", node=3D"drive0", name=3D"bitma= pD") + + log('\n--- Final Query ---\n') + log(query_bitmaps(vm), indent=3D2) + + log('\n--- Done ---\n') + vm.shutdown() diff --git a/tests/qemu-iotests/236.out b/tests/qemu-iotests/236.out new file mode 100644 index 0000000000..1934035795 --- /dev/null +++ b/tests/qemu-iotests/236.out @@ -0,0 +1,351 @@ +--- Preparing image & VM --- + + +--- Adding preliminary bitmaps A & B --- + +{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, = "name": "bitmapA", "node": "drive0"}} +{"return": {}} +{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, = "name": "bitmapB", "node": "drive0"}} +{"return": {}} + +--- Emulating writes --- + +write -P0x5d 0 64k +{"return": ""} +write -P0xd5 1M 64k +{"return": ""} +write -P0xdc 32M 64k +{"return": ""} +write -P0xcd 0x3ff0000 64k +{"return": ""} +{ + "bitmaps": { + "drive0": [ + { + "count": 262144, + "granularity": 65536, + "name": "bitmapB", + "status": "active" + }, + { + "count": 262144, + "granularity": 65536, + "name": "bitmapA", + "status": "active" + } + ] + } +} + +--- Submitting Bad Transaction --- + +{ + "execute": "transaction", + "arguments": { + "actions": [ + { + "data": { + "node": "drive0", + "name": "bitmapB" + }, + "type": "block-dirty-bitmap-disable" + }, + { + "data": { + "node": "drive0", + "name": "bitmapC", + "granularity": 65536 + }, + "type": "block-dirty-bitmap-add" + }, + { + "data": { + "node": "drive0", + "name": "bitmapA" + }, + "type": "block-dirty-bitmap-clear" + }, + { + "data": {}, + "type": "abort" + } + ] + } +} +{ + "error": { + "class": "GenericError", + "desc": "Transaction aborted using Abort action" + } +} +{ + "bitmaps": { + "drive0": [ + { + "count": 262144, + "granularity": 65536, + "name": "bitmapB", + "status": "active" + }, + { + "count": 262144, + "granularity": 65536, + "name": "bitmapA", + "status": "active" + } + ] + } +} + +--- Disabling B & Adding C --- + +{ + "execute": "transaction", + "arguments": { + "actions": [ + { + "data": { + "node": "drive0", + "name": "bitmapB" + }, + "type": "block-dirty-bitmap-disable" + }, + { + "data": { + "node": "drive0", + "name": "bitmapC", + "granularity": 65536 + }, + "type": "block-dirty-bitmap-add" + }, + { + "data": { + "node": "drive0", + "name": "bitmapC" + }, + "type": "block-dirty-bitmap-disable" + }, + { + "data": { + "node": "drive0", + "name": "bitmapC" + }, + "type": "block-dirty-bitmap-enable" + } + ] + } +} +{ + "return": {} +} + +--- Emulating further writes --- + +write -P0xab 0 64k +{"return": ""} +write -P0xad 0x00f8000 64k +{"return": ""} +write -P0x1d 0x2008000 64k +{"return": ""} +write -P0xea 0x3fe0000 64k +{"return": ""} + +--- Disabling A & C --- + +{ + "execute": "transaction", + "arguments": { + "actions": [ + { + "data": { + "node": "drive0", + "name": "bitmapA" + }, + "type": "block-dirty-bitmap-disable" + }, + { + "data": { + "node": "drive0", + "name": "bitmapC" + }, + "type": "block-dirty-bitmap-disable" + } + ] + } +} +{ + "return": {} +} +{ + "bitmaps": { + "drive0": [ + { + "count": 393216, + "granularity": 65536, + "name": "bitmapC", + "status": "disabled" + }, + { + "count": 262144, + "granularity": 65536, + "name": "bitmapB", + "status": "disabled" + }, + { + "count": 458752, + "granularity": 65536, + "name": "bitmapA", + "status": "disabled" + } + ] + } +} + +--- Submitting Bad Merge --- + +{ + "execute": "transaction", + "arguments": { + "actions": [ + { + "data": { + "node": "drive0", + "disabled": true, + "name": "bitmapD", + "granularity": 65536 + }, + "type": "block-dirty-bitmap-add" + }, + { + "data": { + "node": "drive0", + "target": "bitmapD", + "bitmaps": [ + "bitmapB", + "bitmapC" + ] + }, + "type": "block-dirty-bitmap-merge" + }, + { + "data": {}, + "type": "abort" + } + ] + } +} +{ + "error": { + "class": "GenericError", + "desc": "Transaction aborted using Abort action" + } +} +{ + "bitmaps": { + "drive0": [ + { + "count": 393216, + "granularity": 65536, + "name": "bitmapC", + "status": "disabled" + }, + { + "count": 262144, + "granularity": 65536, + "name": "bitmapB", + "status": "disabled" + }, + { + "count": 458752, + "granularity": 65536, + "name": "bitmapA", + "status": "disabled" + } + ] + } +} + +--- Creating D as a merge of B & C --- + +{ + "execute": "transaction", + "arguments": { + "actions": [ + { + "data": { + "node": "drive0", + "disabled": true, + "name": "bitmapD", + "granularity": 65536 + }, + "type": "block-dirty-bitmap-add" + }, + { + "data": { + "node": "drive0", + "target": "bitmapD", + "bitmaps": [ + "bitmapB", + "bitmapC" + ] + }, + "type": "block-dirty-bitmap-merge" + } + ] + } +} +{ + "return": {} +} +{ + "bitmaps": { + "drive0": [ + { + "count": 458752, + "granularity": 65536, + "name": "bitmapD", + "status": "disabled" + }, + { + "count": 393216, + "granularity": 65536, + "name": "bitmapC", + "status": "disabled" + }, + { + "count": 262144, + "granularity": 65536, + "name": "bitmapB", + "status": "disabled" + }, + { + "count": 458752, + "granularity": 65536, + "name": "bitmapA", + "status": "disabled" + } + ] + } +} + +--- Removing bitmaps A, B, C, and D --- + +{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmapA", = "node": "drive0"}} +{"return": {}} +{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmapB", = "node": "drive0"}} +{"return": {}} +{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmapC", = "node": "drive0"}} +{"return": {}} +{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmapD", = "node": "drive0"}} +{"return": {}} + +--- Final Query --- + +{ + "bitmaps": { + "drive0": [] + } +} + +--- Done --- + diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 61a6d98ebd..f6b245917a 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -233,3 +233,4 @@ 233 auto quick 234 auto quick migration 235 auto quick +236 auto quick --=20 2.17.2