From nobody Fri Nov 7 00:39:19 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545184506415484.5127477030994; Tue, 18 Dec 2018 17:55:06 -0800 (PST) Received: from localhost ([::1]:57228 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZR4d-0001aL-CF for importer@patchew.org; Tue, 18 Dec 2018 20:54:59 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43265) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZR2W-000061-4q for qemu-devel@nongnu.org; Tue, 18 Dec 2018 20:52:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gZR2V-0004SG-0t for qemu-devel@nongnu.org; Tue, 18 Dec 2018 20:52:48 -0500 Received: from mx1.redhat.com ([209.132.183.28]:38352) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gZR2S-0004Qd-BQ; Tue, 18 Dec 2018 20:52:44 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1092DD3C3F; Wed, 19 Dec 2018 01:52:43 +0000 (UTC) Received: from probe.redhat.com (ovpn-124-24.rdu2.redhat.com [10.10.124.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 76BAE67CEF; Wed, 19 Dec 2018 01:52:37 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Dec 2018 20:52:26 -0500 Message-Id: <20181219015230.18652-2-jsnow@redhat.com> In-Reply-To: <20181219015230.18652-1-jsnow@redhat.com> References: <20181219015230.18652-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Wed, 19 Dec 2018 01:52:43 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v4 1/5] iotests: add qmp recursive sorting function X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Python before 3.6 does not sort kwargs by default. If we want to print out pretty-printed QMP objects while preserving the "exec" > "arguments" ordering, we need a custom sort. We can accomplish this by sorting **kwargs into an OrderedDict, which does preserve addition order. --- tests/qemu-iotests/iotests.py | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 9595429fea..9aec03c7a3 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -30,6 +30,7 @@ import signal import logging import atexit import io +from collections import OrderedDict =20 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scrip= ts')) import qtest @@ -75,6 +76,16 @@ def qemu_img(*args): sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode,= ' '.join(qemu_img_args + list(args)))) return exitcode =20 +def ordered_kwargs(kwargs): + # kwargs prior to 3.6 are not ordered, so: + od =3D OrderedDict() + for k in sorted(kwargs.keys()): + if isinstance(kwargs[k], dict): + od[k] =3D ordered_kwargs(kwargs[k]) + else: + od[k] =3D kwargs[k] + return od + def qemu_img_create(*args): args =3D list(args) =20 @@ -257,8 +268,9 @@ def filter_img_info(output, filename): def log(msg, filters=3D[]): for flt in filters: msg =3D flt(msg) - if type(msg) is dict or type(msg) is list: - print(json.dumps(msg, sort_keys=3DTrue)) + if isinstance(msg, dict) or isinstance(msg, list): + sort_keys =3D not isinstance(msg, OrderedDict) + print(json.dumps(msg, sort_keys=3Dsort_keys)) else: print(msg) =20 @@ -448,8 +460,9 @@ class VM(qtest.QEMUQtestMachine): return result =20 def qmp_log(self, cmd, filters=3D[filter_testfiles], **kwargs): - logmsg =3D '{"execute": "%s", "arguments": %s}' % \ - (cmd, json.dumps(kwargs, sort_keys=3DTrue)) + full_cmd =3D OrderedDict({"execute": cmd, + "arguments": ordered_kwargs(kwargs)}) + logmsg =3D json.dumps(full_cmd) log(logmsg, filters) result =3D self.qmp(cmd, **kwargs) log(json.dumps(result, sort_keys=3DTrue), filters) --=20 2.17.2 From nobody Fri Nov 7 00:39:19 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545184617388214.8600071256676; Tue, 18 Dec 2018 17:56:57 -0800 (PST) Received: from localhost ([::1]:57248 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZR6W-0003Pp-6z for importer@patchew.org; Tue, 18 Dec 2018 20:56:56 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43286) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZR2X-00007E-CE for qemu-devel@nongnu.org; Tue, 18 Dec 2018 20:52:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gZR2W-0004TQ-Ei for qemu-devel@nongnu.org; Tue, 18 Dec 2018 20:52:49 -0500 Received: from mx1.redhat.com ([209.132.183.28]:42866) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gZR2U-0004RL-54; Tue, 18 Dec 2018 20:52:46 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1D45137E87; Wed, 19 Dec 2018 01:52:45 +0000 (UTC) Received: from probe.redhat.com (ovpn-124-24.rdu2.redhat.com [10.10.124.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3F8D665978; Wed, 19 Dec 2018 01:52:43 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Dec 2018 20:52:27 -0500 Message-Id: <20181219015230.18652-3-jsnow@redhat.com> In-Reply-To: <20181219015230.18652-1-jsnow@redhat.com> References: <20181219015230.18652-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Wed, 19 Dec 2018 01:52:45 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v4 2/5] iotests: remove default filters from qmp_log X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Only test 206 uses it, so remove it. Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/206 | 8 ++++++-- tests/qemu-iotests/iotests.py | 2 +- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/206 b/tests/qemu-iotests/206 index 128c334c7c..e92550fa59 100755 --- a/tests/qemu-iotests/206 +++ b/tests/qemu-iotests/206 @@ -26,7 +26,9 @@ from iotests import imgfmt iotests.verify_image_format(supported_fmts=3D['qcow2']) =20 def blockdev_create(vm, options): - result =3D vm.qmp_log('blockdev-create', job_id=3D'job0', options=3Dop= tions) + result =3D vm.qmp_log('blockdev-create', + filters=3D[iotests.filter_testfiles], + job_id=3D'job0', options=3Doptions) =20 if 'return' in result: assert result['return'] =3D=3D {} @@ -52,7 +54,9 @@ with iotests.FilePath('t.qcow2') as disk_path, \ 'filename': disk_path, 'size': 0 }) =20 - vm.qmp_log('blockdev-add', driver=3D'file', filename=3Ddisk_path, + vm.qmp_log('blockdev-add', + filters=3D[iotests.filter_testfiles], + driver=3D'file', filename=3Ddisk_path, node_name=3D'imgfile') =20 blockdev_create(vm, { 'driver': imgfmt, diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 9aec03c7a3..55fb60e039 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -459,7 +459,7 @@ class VM(qtest.QEMUQtestMachine): result.append(filter_qmp_event(ev)) return result =20 - def qmp_log(self, cmd, filters=3D[filter_testfiles], **kwargs): + def qmp_log(self, cmd, filters=3D[], **kwargs): full_cmd =3D OrderedDict({"execute": cmd, "arguments": ordered_kwargs(kwargs)}) logmsg =3D json.dumps(full_cmd) --=20 2.17.2 From nobody Fri Nov 7 00:39:19 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545184695109914.7282992342854; Tue, 18 Dec 2018 17:58:15 -0800 (PST) Received: from localhost ([::1]:57257 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZR7l-0004Kv-SB for importer@patchew.org; Tue, 18 Dec 2018 20:58:13 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43329) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZR2c-0000BT-Ef for qemu-devel@nongnu.org; Tue, 18 Dec 2018 20:52:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gZR2a-0004WT-GN for qemu-devel@nongnu.org; Tue, 18 Dec 2018 20:52:54 -0500 Received: from mx1.redhat.com ([209.132.183.28]:46876) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gZR2V-0004SB-If; Tue, 18 Dec 2018 20:52:47 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C3E1F81DED; Wed, 19 Dec 2018 01:52:46 +0000 (UTC) Received: from probe.redhat.com (ovpn-124-24.rdu2.redhat.com [10.10.124.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4B76268875; Wed, 19 Dec 2018 01:52:45 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Dec 2018 20:52:28 -0500 Message-Id: <20181219015230.18652-4-jsnow@redhat.com> In-Reply-To: <20181219015230.18652-1-jsnow@redhat.com> References: <20181219015230.18652-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Wed, 19 Dec 2018 01:52:46 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v4 3/5] iotests: change qmp_log filters to expect QMP objects only X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" log() treats filters as if they can always filter its primary argument. qmp_log treats filters as if they're always text. Change qmp_log to treat filters as if they're always qmp object filters, then change the logging call to rely on log()'s ability to serialize QMP objects, so we're not duplicating that effort. Because kwargs have been sorted already, the order is preserved. Edit the only caller who uses filters on qmp_log to use a qmp version, also added in this patch. --- tests/qemu-iotests/206 | 4 ++-- tests/qemu-iotests/iotests.py | 24 +++++++++++++++++++++--- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/tests/qemu-iotests/206 b/tests/qemu-iotests/206 index e92550fa59..5bb738bf23 100755 --- a/tests/qemu-iotests/206 +++ b/tests/qemu-iotests/206 @@ -27,7 +27,7 @@ iotests.verify_image_format(supported_fmts=3D['qcow2']) =20 def blockdev_create(vm, options): result =3D vm.qmp_log('blockdev-create', - filters=3D[iotests.filter_testfiles], + filters=3D[iotests.filter_qmp_testfiles], job_id=3D'job0', options=3Doptions) =20 if 'return' in result: @@ -55,7 +55,7 @@ with iotests.FilePath('t.qcow2') as disk_path, \ 'size': 0 }) =20 vm.qmp_log('blockdev-add', - filters=3D[iotests.filter_testfiles], + filters=3D[iotests.filter_qmp_testfiles], driver=3D'file', filename=3Ddisk_path, node_name=3D'imgfile') =20 diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 55fb60e039..812302538d 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -246,10 +246,29 @@ def filter_qmp_event(event): event['timestamp']['microseconds'] =3D 'USECS' return event =20 +def filter_qmp(qmsg, filter_fn): + '''Given a string filter, filter a QMP object's values. + filter_fn takes a (key, value) pair.''' + for key in qmsg: + if isinstance(qmsg[key], list): + qmsg[key] =3D [filter_qmp(atom, filter_fn) for atom in qmsg[ke= y]] + elif isinstance(qmsg[key], dict): + qmsg[key] =3D filter_qmp(qmsg[key], filter_fn) + else: + qmsg[key] =3D filter_fn(key, qmsg[key]) + return qmsg + def filter_testfiles(msg): prefix =3D os.path.join(test_dir, "%s-" % (os.getpid())) return msg.replace(prefix, 'TEST_DIR/PID-') =20 +def filter_qmp_testfiles(qmsg): + def _filter(key, value): + if key =3D=3D 'filename' or key =3D=3D 'backing-file': + return filter_testfiles(value) + return value + return filter_qmp(qmsg, _filter) + def filter_generated_node_ids(msg): return re.sub("#block[0-9]+", "NODE_NAME", msg) =20 @@ -462,10 +481,9 @@ class VM(qtest.QEMUQtestMachine): def qmp_log(self, cmd, filters=3D[], **kwargs): full_cmd =3D OrderedDict({"execute": cmd, "arguments": ordered_kwargs(kwargs)}) - logmsg =3D json.dumps(full_cmd) - log(logmsg, filters) + log(full_cmd, filters) result =3D self.qmp(cmd, **kwargs) - log(json.dumps(result, sort_keys=3DTrue), filters) + log(result, filters) return result =20 def run_job(self, job, auto_finalize=3DTrue, auto_dismiss=3DFalse): --=20 2.17.2 From nobody Fri Nov 7 00:39:19 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545184513134409.88125620943515; Tue, 18 Dec 2018 17:55:13 -0800 (PST) Received: from localhost ([::1]:57230 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZR4l-0001hv-97 for importer@patchew.org; Tue, 18 Dec 2018 20:55:07 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43352) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZR2d-0000E2-Cm for qemu-devel@nongnu.org; Tue, 18 Dec 2018 20:52:56 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gZR2c-0004YS-Fr for qemu-devel@nongnu.org; Tue, 18 Dec 2018 20:52:55 -0500 Received: from mx1.redhat.com ([209.132.183.28]:47940) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gZR2X-0004TY-9U; Tue, 18 Dec 2018 20:52:49 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7F588C050DEE; Wed, 19 Dec 2018 01:52:48 +0000 (UTC) Received: from probe.redhat.com (ovpn-124-24.rdu2.redhat.com [10.10.124.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0BD5D65978; Wed, 19 Dec 2018 01:52:46 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Dec 2018 20:52:29 -0500 Message-Id: <20181219015230.18652-5-jsnow@redhat.com> In-Reply-To: <20181219015230.18652-1-jsnow@redhat.com> References: <20181219015230.18652-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.32]); Wed, 19 Dec 2018 01:52:48 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v4 4/5] iotests: implement pretty-print for log and qmp_log X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" If iotests have lines exceeding >998 characters long, git doesn't want to send it plaintext to the list. We can solve this by allowing the iotests to use pretty printed QMP output that we can match against instead. As a bonus, it's much nicer for human eyes too. --- tests/qemu-iotests/iotests.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 812302538d..d3c57e266f 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -284,12 +284,18 @@ def filter_img_info(output, filename): lines.append(line) return '\n'.join(lines) =20 -def log(msg, filters=3D[]): +def log(msg, filters=3D[], indent=3DNone): + '''Logs either a string message or a JSON serializable message (like Q= MP). + If indent is provided, JSON serializable messages are pretty-printed.'= '' for flt in filters: msg =3D flt(msg) if isinstance(msg, dict) or isinstance(msg, list): - sort_keys =3D not isinstance(msg, OrderedDict) - print(json.dumps(msg, sort_keys=3Dsort_keys)) + # Python < 3.4 needs to know not to add whitespace when pretty-pri= nting: + separators =3D (', ', ': ') if indent is None else (',', ': ') + # Don't sort if it's already sorted + do_sort =3D not isinstance(msg, OrderedDict) + print(json.dumps(msg, sort_keys=3Ddo_sort, + indent=3Dindent, separators=3Dseparators)) else: print(msg) =20 @@ -478,12 +484,12 @@ class VM(qtest.QEMUQtestMachine): result.append(filter_qmp_event(ev)) return result =20 - def qmp_log(self, cmd, filters=3D[], **kwargs): + def qmp_log(self, cmd, filters=3D[], indent=3DNone, **kwargs): full_cmd =3D OrderedDict({"execute": cmd, "arguments": ordered_kwargs(kwargs)}) - log(full_cmd, filters) + log(full_cmd, filters, indent=3Dindent) result =3D self.qmp(cmd, **kwargs) - log(result, filters) + log(result, filters, indent=3Dindent) return result =20 def run_job(self, job, auto_finalize=3DTrue, auto_dismiss=3DFalse): --=20 2.17.2 From nobody Fri Nov 7 00:39:19 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1545184514515860.4961640588316; Tue, 18 Dec 2018 17:55:14 -0800 (PST) Received: from localhost ([::1]:57231 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZR4q-0001oH-QW for importer@patchew.org; Tue, 18 Dec 2018 20:55:12 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43373) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gZR2f-0000I5-Hb for qemu-devel@nongnu.org; Tue, 18 Dec 2018 20:52:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gZR2d-0004Zm-UR for qemu-devel@nongnu.org; Tue, 18 Dec 2018 20:52:57 -0500 Received: from mx1.redhat.com ([209.132.183.28]:50992) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gZR2Z-0004VL-AP; Tue, 18 Dec 2018 20:52:51 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A40DE811C0; Wed, 19 Dec 2018 01:52:50 +0000 (UTC) Received: from probe.redhat.com (ovpn-124-24.rdu2.redhat.com [10.10.124.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id BC86365978; Wed, 19 Dec 2018 01:52:48 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Tue, 18 Dec 2018 20:52:30 -0500 Message-Id: <20181219015230.18652-6-jsnow@redhat.com> In-Reply-To: <20181219015230.18652-1-jsnow@redhat.com> References: <20181219015230.18652-1-jsnow@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Wed, 19 Dec 2018 01:52:50 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v4 5/5] iotests: add iotest 236 for testing bitmap merge X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kevin Wolf , vsementsov@virtuozzo.com, Markus Armbruster , Max Reitz , John Snow Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" New interface, new smoke test. --- tests/qemu-iotests/236 | 131 ++++++++++++++++++++++++ tests/qemu-iotests/236.out | 198 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 330 insertions(+) create mode 100755 tests/qemu-iotests/236 create mode 100644 tests/qemu-iotests/236.out diff --git a/tests/qemu-iotests/236 b/tests/qemu-iotests/236 new file mode 100755 index 0000000000..bf8d6882d0 --- /dev/null +++ b/tests/qemu-iotests/236 @@ -0,0 +1,131 @@ +#!/usr/bin/env python +# +# Test bitmap merges. +# +# Copyright (c) 2018 John Snow for Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +# owner=3Djsnow@redhat.com + +import iotests +from iotests import log + +iotests.verify_image_format(supported_fmts=3D['generic']) +size =3D 64 * 1024 * 1024 +granularity =3D 64 * 1024 + +patterns =3D [("0x5d", "0", "64k"), + ("0xd5", "1M", "64k"), + ("0xdc", "32M", "64k"), + ("0xcd", "0x3ff0000", "64k")] # 64M - 64K + +overwrite =3D [("0xab", "0", "64k"), # Full overwrite + ("0xad", "0x00f8000", "64k"), # Partial-left (1M-32K) + ("0x1d", "0x2008000", "64k"), # Partial-right (32M+32K) + ("0xea", "0x3fe0000", "64k")] # Adjacent-left (64M - 128K) + +def query_bitmaps(vm): + res =3D vm.qmp("query-block") + return {device['device']: device['dirty-bitmaps'] for + device in res['return']} + +with iotests.FilePath('img') as img_path, \ + iotests.VM() as vm: + + log('--- Preparing image & VM ---\n') + iotests.qemu_img_create('-f', iotests.imgfmt, img_path, str(size)) + vm.add_drive(img_path) + vm.launch() + + log('--- Adding preliminary bitmaps A & B ---\n') + vm.qmp_log("block-dirty-bitmap-add", node=3D"drive0", + name=3D"bitmapA", granularity=3Dgranularity) + vm.qmp_log("block-dirty-bitmap-add", node=3D"drive0", + name=3D"bitmapB", granularity=3Dgranularity) + + # Dirties 4 clusters. count=3D262144 + log('') + log('--- Emulating writes ---\n') + for p in patterns: + cmd =3D "write -P%s %s %s" % p + log(cmd) + log(vm.hmp_qemu_io("drive0", cmd)) + + log(query_bitmaps(vm), indent=3D2) + + log('') + log('--- Disabling B & Adding C ---\n') + vm.qmp_log("transaction", indent=3D2, actions=3D[ + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapB" }}, + { "type": "block-dirty-bitmap-add", + "data": { "node": "drive0", "name": "bitmapC", + "granularity": granularity }}, + # Purely extraneous, but test that it works: + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapC" }}, + { "type": "block-dirty-bitmap-enable", + "data": { "node": "drive0", "name": "bitmapC" }}, + ]) + + log('') + log('--- Emulating further writes ---\n') + # Dirties 6 clusters, 3 of which are new in contrast to "A". + # A =3D 64 * 1024 * (4 + 3) =3D 458752 + # C =3D 64 * 1024 * 6 =3D 393216 + for p in overwrite: + cmd =3D "write -P%s %s %s" % p + log(cmd) + log(vm.hmp_qemu_io("drive0", cmd)) + + log('') + log('--- Disabling A & C ---\n') + vm.qmp_log("transaction", indent=3D2, actions=3D[ + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapA" }}, + { "type": "block-dirty-bitmap-disable", + "data": { "node": "drive0", "name": "bitmapC" }} + ]) + + # A: 7 clusters + # B: 4 clusters + # C: 6 clusters + log(query_bitmaps(vm), indent=3D2) + + log('') + log('--- Creating D as a merge of B & C ---\n') + # Good hygiene: create a disabled bitmap as a merge target. + vm.qmp_log("transaction", indent=3D2, actions=3D[ + { "type": "block-dirty-bitmap-add", + "data": { "node": "drive0", "name": "bitmapD", + "disabled": True, "granularity": granularity }}, + { "type": "block-dirty-bitmap-merge", + "data": { "node": "drive0", "target": "bitmapD", + "bitmaps": ["bitmapB", "bitmapC"] }} + ]) + + # A and D should now both have 7 clusters apiece. + # B and C remain unchanged with 4 and 6 respectively. + log(query_bitmaps(vm), indent=3D2) + + # A and D should be equivalent. + # Some formats round the size of the disk, so don't print the checksum= s. + check_a =3D vm.qmp('x-debug-block-dirty-bitmap-sha256', + node=3D"drive0", name=3D"bitmapA")['return']['sha256'] + check_b =3D vm.qmp('x-debug-block-dirty-bitmap-sha256', + node=3D"drive0", name=3D"bitmapD")['return']['sha256'] + assert(check_a =3D=3D check_b) + + vm.shutdown() diff --git a/tests/qemu-iotests/236.out b/tests/qemu-iotests/236.out new file mode 100644 index 0000000000..b5f8300816 --- /dev/null +++ b/tests/qemu-iotests/236.out @@ -0,0 +1,198 @@ +--- Preparing image & VM --- + +--- Adding preliminary bitmaps A & B --- + +{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, = "name": "bitmapA", "node": "drive0"}} +{"return": {}} +{"execute": "block-dirty-bitmap-add", "arguments": {"granularity": 65536, = "name": "bitmapB", "node": "drive0"}} +{"return": {}} + +--- Emulating writes --- + +write -P0x5d 0 64k +{"return": ""} +write -P0xd5 1M 64k +{"return": ""} +write -P0xdc 32M 64k +{"return": ""} +write -P0xcd 0x3ff0000 64k +{"return": ""} +{ + "drive0": [ + { + "count": 262144, + "granularity": 65536, + "name": "bitmapB", + "status": "active" + }, + { + "count": 262144, + "granularity": 65536, + "name": "bitmapA", + "status": "active" + } + ] +} + +--- Disabling B & Adding C --- + +{ + "execute": "transaction", + "arguments": { + "actions": [ + { + "data": { + "node": "drive0", + "name": "bitmapB" + }, + "type": "block-dirty-bitmap-disable" + }, + { + "data": { + "node": "drive0", + "name": "bitmapC", + "granularity": 65536 + }, + "type": "block-dirty-bitmap-add" + }, + { + "data": { + "node": "drive0", + "name": "bitmapC" + }, + "type": "block-dirty-bitmap-disable" + }, + { + "data": { + "node": "drive0", + "name": "bitmapC" + }, + "type": "block-dirty-bitmap-enable" + } + ] + } +} +{ + "return": {} +} + +--- Emulating further writes --- + +write -P0xab 0 64k +{"return": ""} +write -P0xad 0x00f8000 64k +{"return": ""} +write -P0x1d 0x2008000 64k +{"return": ""} +write -P0xea 0x3fe0000 64k +{"return": ""} + +--- Disabling A & C --- + +{ + "execute": "transaction", + "arguments": { + "actions": [ + { + "data": { + "node": "drive0", + "name": "bitmapA" + }, + "type": "block-dirty-bitmap-disable" + }, + { + "data": { + "node": "drive0", + "name": "bitmapC" + }, + "type": "block-dirty-bitmap-disable" + } + ] + } +} +{ + "return": {} +} +{ + "drive0": [ + { + "count": 393216, + "granularity": 65536, + "name": "bitmapC", + "status": "disabled" + }, + { + "count": 262144, + "granularity": 65536, + "name": "bitmapB", + "status": "disabled" + }, + { + "count": 458752, + "granularity": 65536, + "name": "bitmapA", + "status": "disabled" + } + ] +} + +--- Creating D as a merge of B & C --- + +{ + "execute": "transaction", + "arguments": { + "actions": [ + { + "data": { + "node": "drive0", + "disabled": true, + "name": "bitmapD", + "granularity": 65536 + }, + "type": "block-dirty-bitmap-add" + }, + { + "data": { + "node": "drive0", + "target": "bitmapD", + "bitmaps": [ + "bitmapB", + "bitmapC" + ] + }, + "type": "block-dirty-bitmap-merge" + } + ] + } +} +{ + "return": {} +} +{ + "drive0": [ + { + "count": 458752, + "granularity": 65536, + "name": "bitmapD", + "status": "disabled" + }, + { + "count": 393216, + "granularity": 65536, + "name": "bitmapC", + "status": "disabled" + }, + { + "count": 262144, + "granularity": 65536, + "name": "bitmapB", + "status": "disabled" + }, + { + "count": 458752, + "granularity": 65536, + "name": "bitmapA", + "status": "disabled" + } + ] +} diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 61a6d98ebd..a61f9e83d6 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -233,3 +233,4 @@ 233 auto quick 234 auto quick migration 235 auto quick +236 auto quick \ No newline at end of file --=20 2.17.2