From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596119195; cv=none; d=zohomail.com; s=zohoarc; b=PEOXdi1hVkpCzwBiCYly9iVKFHylKQ1kEAl7H6PS2a92yUNJy1SOupa0jGVvCaBwsIfopWAJ/jkXVC5OgnQTQbEYqJU9J4egm63MEs3EUASfMx3NPm24sPX5UmalJ3KVMTyJX32XAStLrjYFZzlm6rWlvZfZgYDCqNxnWFwefQM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596119195; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=/6BrGuZIR2BfeEV+FCEQhk7JNbm37AsanTXL8G4B3Ko=; b=Fk2VBRp2ijbOVjP1QkB1WUCpdyXFXJ926uEfI1GBfXrlY/1ZadI8cu8rlOUUc0TaZyLYOu0BlX1f3Z7w/FlwtwcUCbvNGJ+GS43p7ZnOv69Lj1Y3tKZFowcTjKNmlRHgo6+0D25bWJ40ISgcyGRe94+wS1RKKchQnkBsXeVC3uk= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1596119195537750.3845158018273; Thu, 30 Jul 2020 07:26:35 -0700 (PDT) Received: from localhost ([::1]:47600 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19Vy-0008CW-BW for importer@patchew.org; Thu, 30 Jul 2020 10:26:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54298) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19LQ-0005e4-UH; Thu, 30 Jul 2020 10:15:40 -0400 Received: from relay.sw.ru ([185.231.240.75]:44440 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19LO-00052e-SS; Thu, 30 Jul 2020 10:15:40 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Km-0004Cz-LN; Thu, 30 Jul 2020 17:15:00 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 01/11] iotests: add test for QCOW2 header dump Date: Thu, 30 Jul 2020 17:15:02 +0300 Message-Id: <1596118512-424960-2-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The simple script creates a QCOW2 image and fills it with some data. Two bitmaps are created as well. Then the script reads the image header with extensions from the disk by running the script qcow2.py and dumps the information to the output. Other entities, such as snapshots, may be added to the test later. Suggested-by: Eric Blake Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/303 | 59 ++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/303.out | 64 ++++++++++++++++++++++++++++++++++++++++++= ++++ tests/qemu-iotests/group | 1 + 3 files changed, 124 insertions(+) create mode 100755 tests/qemu-iotests/303 create mode 100644 tests/qemu-iotests/303.out diff --git a/tests/qemu-iotests/303 b/tests/qemu-iotests/303 new file mode 100755 index 0000000..3c7a611 --- /dev/null +++ b/tests/qemu-iotests/303 @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +# +# Test for dumping of qcow2 image metadata +# +# Copyright (c) 2020 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# + +import iotests +import subprocess +from iotests import qemu_img_create, qemu_io, file_path, log, filter_qemu_= io + +iotests.script_initialize(supported_fmts=3D['qcow2']) + +disk =3D file_path('disk') +chunk =3D 1024 * 1024 + + +def create_bitmap(bitmap_number, disabled): + granularity =3D 1 << (14 + bitmap_number) + bitmap_name =3D 'bitmap-' + str(bitmap_number) + vm =3D iotests.VM().add_drive(disk) + vm.launch() + vm.qmp_log('block-dirty-bitmap-add', node=3D'drive0', name=3Dbitmap_na= me, + granularity=3Dgranularity, persistent=3DTrue, disabled=3Ddi= sabled) + vm.shutdown() + + +def write_to_disk(offset, size): + write =3D f'write {offset} {size}' + log(qemu_io('-c', write, disk), filters=3D[filter_qemu_io]) + + +def add_bitmap(num, begin, end, disabled): + log(f'Add bitmap {num}') + create_bitmap(num, disabled) + for i in range(begin, end): + write_to_disk((i-1) * chunk, chunk) + log('') + + +qemu_img_create('-f', iotests.imgfmt, disk, '10M') + +add_bitmap(1, 1, 7, False) +add_bitmap(2, 7, 9, True) +dump =3D ['qcow2.py', f'{disk}', 'dump-header'] +subprocess.run(dump) diff --git a/tests/qemu-iotests/303.out b/tests/qemu-iotests/303.out new file mode 100644 index 0000000..8b11169 --- /dev/null +++ b/tests/qemu-iotests/303.out @@ -0,0 +1,64 @@ +Add bitmap 1 +{"execute": "block-dirty-bitmap-add", "arguments": {"disabled": false, "gr= anularity": 32768, "name": "bitmap-1", "node": "drive0", "persistent": true= }} +{"return": {}} +wrote 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +wrote 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +wrote 1048576/1048576 bytes at offset 2097152 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +wrote 1048576/1048576 bytes at offset 3145728 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +wrote 1048576/1048576 bytes at offset 4194304 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +wrote 1048576/1048576 bytes at offset 5242880 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + + +Add bitmap 2 +{"execute": "block-dirty-bitmap-add", "arguments": {"disabled": true, "gra= nularity": 65536, "name": "bitmap-2", "node": "drive0", "persistent": true}} +{"return": {}} +wrote 1048576/1048576 bytes at offset 6291456 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +wrote 1048576/1048576 bytes at offset 7340032 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + + +magic 0x514649fb +version 3 +backing_file_offset 0x0 +backing_file_size 0x0 +cluster_bits 16 +size 10485760 +crypt_method 0 +l1_size 1 +l1_table_offset 0x30000 +refcount_table_offset 0x10000 +refcount_table_clusters 1 +nb_snapshots 0 +snapshot_offset 0x0 +incompatible_features [] +compatible_features [] +autoclear_features [0] +refcount_order 4 +header_length 112 + +Header extension: +magic 0x6803f857 (Feature table) +length 336 +data + +Header extension: +magic 0x23852875 (Bitmaps) +length 24 +nb_bitmaps 2 +reserved32 0 +bitmap_directory_size 0x40 +bitmap_directory_offset 0x9d0000 + diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group index 025ed52..7e12e1f 100644 --- a/tests/qemu-iotests/group +++ b/tests/qemu-iotests/group @@ -309,3 +309,4 @@ 299 auto quick 301 backing quick 302 quick +303 rw quick --=20 1.8.3.1 From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596118758; cv=none; d=zohomail.com; s=zohoarc; b=erEnmqOA0VIeZ7G3k/p8FT/MV3NhH56j6cI9pHrujwNdAwpowa4mXNFKJ8p7cPu0jQFRdD45UHnoM0gOAOnMiWmT/pcatJfTRH/zmwHTjIVX1Jg1QwFSDzog/qNMAdX6HLu8SoG1AUEBf3NnfPrLQdMpJkTqHLJsqtpxHCTasX8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596118758; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=jTaQe+3C4V+UcRj4btutD2ZIpAv4td/tmxwTbSW57kA=; b=XeLaRj5+vf4UhQtezhWev+tqXVMF3uX7wuR9u3LurqpaBotajUrusr8tul8JVIQBmNMfU7dQ0+NgZBj/SIhqIlkfnqwBY5yiNAEDs2l4rl+UBLV9ME5fXo34kU/L73/62nduybox0ZqWzht4i72r8kxr6vPPAUkEnw6edQ7Rqu8= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 159611875880128.942788708740864; Thu, 30 Jul 2020 07:19:18 -0700 (PDT) Received: from localhost ([::1]:45148 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19Ov-0003qT-Ii for importer@patchew.org; Thu, 30 Jul 2020 10:19:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54152) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19LA-00051R-IZ; Thu, 30 Jul 2020 10:15:24 -0400 Received: from relay.sw.ru ([185.231.240.75]:44444 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L4-00052f-Nf; Thu, 30 Jul 2020 10:15:24 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Km-0004Cz-Ma; Thu, 30 Jul 2020 17:15:00 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 02/11] qcow2_format.py: make printable data an extension class member Date: Thu, 30 Jul 2020 17:15:03 +0300 Message-Id: <1596118512-424960-3-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Let us differ binary data type from string one for the extension data variable and keep the string as the QcowHeaderExtension class member. Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index cc432e7..2f3681b 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -165,6 +165,13 @@ class QcowHeaderExtension(Qcow2Struct): self.data =3D fd.read(padded) assert self.data is not None =20 + data_str =3D self.data[:self.length] + if all(c in string.printable.encode('ascii') for c in data_str): + data_str =3D f"'{ data_str.decode('ascii') }'" + else: + data_str =3D '' + self.data_str =3D data_str + if self.magic =3D=3D QCOW2_EXT_MAGIC_BITMAPS: self.obj =3D Qcow2BitmapExt(data=3Dself.data) else: @@ -174,12 +181,7 @@ class QcowHeaderExtension(Qcow2Struct): super().dump() =20 if self.obj is None: - data =3D self.data[:self.length] - if all(c in string.printable.encode('ascii') for c in data): - data =3D f"'{ data.decode('ascii') }'" - else: - data =3D '' - print(f'{"data":<25} {data}') + print(f'{"data":<25} {self.data_str}') else: self.obj.dump() =20 --=20 1.8.3.1 From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596119116; cv=none; d=zohomail.com; s=zohoarc; b=DUKUflxY6Q+Acb3rd7/EvbCPPvHAZuYBBauN+uNgHtxXyuEcf+NB/uhjtCwhdU9Y/Ou+3XJlHQrOwRyBV16YXke7Dz3+TyYIKYG/8dmwc5IXG0KuoQwTnXBt+AV6OXqTD29jsjH26NYl6MoJLiDAV1rWyBqeR+yDj3VLa4VEwDY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596119116; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=po6iy39KpJCpsbML0K1ktaQnSiNcC7KW/I5eyNQir1Y=; b=cy9hD/c+XFBVrf3nIv6wEPUhubm9nkRlMMUnn/Wq2ETQHGHd9NO470KGYS2z3kboLeSTzOrotkgB8JNhSUJAdUCWjWp5vgxH9bbk6twkx6Hlgxn5u2AdiT0PIWopymjzYeeJ9j9qZkWw1nqcDI60xXQyuFDAysRwP39MszZjK/M= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1596119116955477.11165139802733; Thu, 30 Jul 2020 07:25:16 -0700 (PDT) Received: from localhost ([::1]:43160 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19Uh-0006En-IG for importer@patchew.org; Thu, 30 Jul 2020 10:25:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54118) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L9-0004yF-3U; Thu, 30 Jul 2020 10:15:23 -0400 Received: from relay.sw.ru ([185.231.240.75]:44441 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L4-00052g-9y; Thu, 30 Jul 2020 10:15:22 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Km-0004Cz-Nl; Thu, 30 Jul 2020 17:15:00 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 03/11] qcow2_format.py: change Qcow2BitmapExt initialization method Date: Thu, 30 Jul 2020 17:15:04 +0300 Message-Id: <1596118512-424960-4-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" There are two ways to initialize a class derived from Qcow2Struct: 1. Pass a block of binary data to the constructor. 2. Pass the file descriptor to allow reading the file from constructor. Let's change the Qcow2BitmapExt initialization method from 1 to 2 to support a scattered reading in the initialization chain. The implementation comes with the patch that follows. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 36 ++++++++++++++++++++++------------= -- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index 2f3681b..d4a9974 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -113,6 +113,11 @@ class Qcow2BitmapExt(Qcow2Struct): ('u64', '{:#x}', 'bitmap_directory_offset') ) =20 + def __init__(self, fd): + super().__init__(fd=3Dfd) + tail =3D struct.calcsize(self.fmt) % 8 + if tail: + fd.seek(8 - tail, 1) =20 QCOW2_EXT_MAGIC_BITMAPS =3D 0x23852875 =20 @@ -161,21 +166,24 @@ class QcowHeaderExtension(Qcow2Struct): else: assert all(v is None for v in (magic, length, data)) super().__init__(fd=3Dfd) - padded =3D (self.length + 7) & ~7 - self.data =3D fd.read(padded) - assert self.data is not None - - data_str =3D self.data[:self.length] - if all(c in string.printable.encode('ascii') for c in data_str): - data_str =3D f"'{ data_str.decode('ascii') }'" - else: - data_str =3D '' - self.data_str =3D data_str + if self.magic =3D=3D QCOW2_EXT_MAGIC_BITMAPS: + self.obj =3D Qcow2BitmapExt(fd=3Dfd) + self.data =3D None + else: + padded =3D (self.length + 7) & ~7 + self.data =3D fd.read(padded) + assert self.data is not None + self.obj =3D None + + if self.data is not None: + data_str =3D self.data[:self.length] + if all(c in string.printable.encode( + 'ascii') for c in data_str): + data_str =3D f"'{ data_str.decode('ascii') }'" + else: + data_str =3D '' + self.data_str =3D data_str =20 - if self.magic =3D=3D QCOW2_EXT_MAGIC_BITMAPS: - self.obj =3D Qcow2BitmapExt(data=3Dself.data) - else: - self.obj =3D None =20 def dump(self): super().dump() --=20 1.8.3.1 From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596118853; cv=none; d=zohomail.com; s=zohoarc; b=JWh1BW8J3h138deDv2TUSohYEuxoRgKTJED97d83tBjjM7jfMq3wc0LJlGIN8r3kK7IHPdFXXgQ5kJK0J0XT/yXuWmPNOhY/iRuOJcYzYK6kxPvegrFqG2pds0sizyW022p+arKM9Yyv7LP+GRuvCmGAm8etS9A4Xm5hqdqFzH4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596118853; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=aWjclfz1ZcI/TI94jF7yVLgfxuyJWL9apZHcugkLCsQ=; b=Ctgr6boUmFcZP6dADPxQhunie52rTdY5l0mamC32z+BEhZ4N2nWGxeip90dmFmIpNQQLGwRBq9bewmiThVefvPb1v/z6V7sBkLeglcaKVWs4X9b6RRP49m3d8O3lTTT8XSpShlYLi0LQuR9fCXnn8dV89hqWC85MMllH6h6gIQM= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1596118853041409.16343093223236; Thu, 30 Jul 2020 07:20:53 -0700 (PDT) Received: from localhost ([::1]:53530 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19QR-0007JB-O3 for importer@patchew.org; Thu, 30 Jul 2020 10:20:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54194) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19LD-00057M-0m; Thu, 30 Jul 2020 10:15:27 -0400 Received: from relay.sw.ru ([185.231.240.75]:44446 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L4-00052d-Mw; Thu, 30 Jul 2020 10:15:26 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Km-0004Cz-Ou; Thu, 30 Jul 2020 17:15:00 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 04/11] qcow2_format.py: dump bitmap flags in human readable way. Date: Thu, 30 Jul 2020 17:15:05 +0300 Message-Id: <1596118512-424960-5-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Introduce the class BitmapFlags that parses a bitmap flags mask. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index d4a9974..b447344 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -40,6 +40,22 @@ class Flags64(Qcow2Field): return str(bits) =20 =20 +class BitmapFlags(Qcow2Field): + + flags =3D { + 0x1: 'in-use', + 0x2: 'auto' + } + + def __str__(self): + bits =3D [] + for bit in range(64): + flag =3D self.value & (1 << bit) + if flag: + bits.append(self.flags.get(flag, f'bit-{bit}')) + return f'{self.value:#x} ({bits})' + + class Enum(Qcow2Field): =20 def __str__(self): --=20 1.8.3.1 From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596118660; cv=none; d=zohomail.com; s=zohoarc; b=nt6yQt84QK/IdkH6aEKXbgcEqAMI4fp/nBujAWKrnff69o0lOkAzfdQgEbReEIJbCSEoon7k1Dy40laX3rMJz73nebUhYz/cMyQbLLaBz/DuQ3qQmbS48OqpojpzPvMmFVW4xRzdnqaNp+eKjE9W+9v6EBLbvNv2nsO0fiTosqw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596118660; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=Y7n4mTuhuh87G1LbtL+dq3cX+y35RlC3wVC8tquxC4o=; b=MJhiW0mPTTXVMJNcB1ck80EEy9HPVb5CfiihBXGPfRKBHnItDBJ8ibu83L/8KsvcsJuZBXItBa3sY3y8DxI/G4x485nDyeYEA6yKRlWa/yo8TF1KJ3iLrlKhLADoRttKmJBclJaqLIrioq0IhLlDLqheg1sZUPtKzRVLCUN1+iY= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1596118660432597.6743565502528; Thu, 30 Jul 2020 07:17:40 -0700 (PDT) Received: from localhost ([::1]:36940 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19NK-0000Ut-UQ for importer@patchew.org; Thu, 30 Jul 2020 10:17:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54130) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L9-0004zN-KA; Thu, 30 Jul 2020 10:15:23 -0400 Received: from relay.sw.ru ([185.231.240.75]:44448 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L4-00052h-Lg; Thu, 30 Jul 2020 10:15:23 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Km-0004Cz-Q6; Thu, 30 Jul 2020 17:15:00 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 05/11] qcow2_format.py: Dump bitmap directory information Date: Thu, 30 Jul 2020 17:15:06 +0300 Message-Id: <1596118512-424960-6-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Read and dump entries from the bitmap directory of QCOW2 image. Header extension: magic 0x23852875 (Bitmaps) ... Bitmap name bitmap-1 bitmap_table_offset 0xf0000 bitmap_table_size 1 flags 0x2 (['auto']) type 1 granularity_bits 16 name_size 8 extra_data_size 0 Suggested-by: Kevin Wolf Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/303.out | 18 +++++++++++++++ tests/qemu-iotests/qcow2_format.py | 47 ++++++++++++++++++++++++++++++++++= ++++ 2 files changed, 65 insertions(+) diff --git a/tests/qemu-iotests/303.out b/tests/qemu-iotests/303.out index 8b11169..dc3739b 100644 --- a/tests/qemu-iotests/303.out +++ b/tests/qemu-iotests/303.out @@ -62,3 +62,21 @@ reserved32 0 bitmap_directory_size 0x40 bitmap_directory_offset 0x9d0000 =20 +Bitmap name bitmap-1 +bitmap_table_offset 0x9b0000 +bitmap_table_size 1 +flags 0x2 (['auto']) +type 1 +granularity_bits 15 +name_size 8 +extra_data_size 0 + +Bitmap name bitmap-2 +bitmap_table_offset 0x9c0000 +bitmap_table_size 1 +flags 0x0 ([]) +type 1 +granularity_bits 16 +name_size 8 +extra_data_size 0 + diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index b447344..05a8aa9 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -134,6 +134,53 @@ class Qcow2BitmapExt(Qcow2Struct): tail =3D struct.calcsize(self.fmt) % 8 if tail: fd.seek(8 - tail, 1) + position =3D fd.tell() + self.read_bitmap_directory(fd) + fd.seek(position) + + def read_bitmap_directory(self, fd): + fd.seek(self.bitmap_directory_offset) + self.bitmap_directory =3D \ + [Qcow2BitmapDirEntry(fd) for _ in range(self.nb_bitmaps)] + + def dump(self): + super().dump() + for entry in self.bitmap_directory: + print() + entry.dump() + + +class Qcow2BitmapDirEntry(Qcow2Struct): + + fields =3D ( + ('u64', '{:#x}', 'bitmap_table_offset'), + ('u32', '{}', 'bitmap_table_size'), + ('u32', BitmapFlags, 'flags'), + ('u8', '{}', 'type'), + ('u8', '{}', 'granularity_bits'), + ('u16', '{}', 'name_size'), + ('u32', '{}', 'extra_data_size') + ) + + def __init__(self, fd): + super().__init__(fd=3Dfd) + # Seek relative to the current position in the file + fd.seek(self.extra_data_size, 1) + bitmap_name =3D fd.read(self.name_size) + self.name =3D bitmap_name.decode('ascii') + # Move position to the end of the entry in the directory + entry_raw_size =3D self.bitmap_dir_entry_raw_size() + padding =3D ((entry_raw_size + 7) & ~7) - entry_raw_size + fd.seek(padding, 1) + + def bitmap_dir_entry_raw_size(self): + return struct.calcsize(self.fmt) + self.name_size + \ + self.extra_data_size + + def dump(self): + print(f'{"Bitmap name":<25} {self.name}') + super(Qcow2BitmapDirEntry, self).dump() + =20 QCOW2_EXT_MAGIC_BITMAPS =3D 0x23852875 =20 --=20 1.8.3.1 From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596118931; cv=none; d=zohomail.com; s=zohoarc; b=h8PAEqm+/bn6j+J3HYnaofd/T2BTkgV2Nfv82qngIBssajIBAnwNxHhl+IWFgxfbX74ChMW1UwyXU9sKPW+3XHPmBerfqRh/Im9hIwYnzf//AdLqOkEy6TsQ63NodH79X0DxsfMbEKFkPfkgea7iBGaG634FprqfX8VwA7mIQPA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596118931; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=+g9p4ALMkB7yCtrpy2vMkuG50zHcXTT9tVR5uufFT50=; b=FtESO6QDGxiSgub66Vwr4zIQa3Thl/EoCVc1sFRC4dkd0IdTYnjdt+1BSLWCqDSpbfYd0hWZ01XdctonhPY0yo64uOwvvMOZsLuWlr8R8X0ykbjqk7STavw9swvRrhRfeErCOaTJeSQ/QX5Mx9HHQSmaz4lpCjd6WZ06clVkfnk= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1596118931667471.0375935259275; Thu, 30 Jul 2020 07:22:11 -0700 (PDT) Received: from localhost ([::1]:60144 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19Ri-0001YJ-Bl for importer@patchew.org; Thu, 30 Jul 2020 10:22:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54054) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L7-0004sJ-5U; Thu, 30 Jul 2020 10:15:21 -0400 Received: from relay.sw.ru ([185.231.240.75]:44430 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L4-00052Z-8v; Thu, 30 Jul 2020 10:15:20 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Km-0004Cz-RM; Thu, 30 Jul 2020 17:15:00 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 06/11] qcow2_format.py: pass cluster size to substructures Date: Thu, 30 Jul 2020 17:15:07 +0300 Message-Id: <1596118512-424960-7-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The cluster size of an image is the QcowHeader class member and may be obtained by dependent extension structures such as Qcow2BitmapExt for further bitmap table details print. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index 05a8aa9..ca0d350 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -129,19 +129,21 @@ class Qcow2BitmapExt(Qcow2Struct): ('u64', '{:#x}', 'bitmap_directory_offset') ) =20 - def __init__(self, fd): + def __init__(self, fd, cluster_size): super().__init__(fd=3Dfd) tail =3D struct.calcsize(self.fmt) % 8 if tail: fd.seek(8 - tail, 1) position =3D fd.tell() + self.cluster_size =3D cluster_size self.read_bitmap_directory(fd) fd.seek(position) =20 def read_bitmap_directory(self, fd): fd.seek(self.bitmap_directory_offset) self.bitmap_directory =3D \ - [Qcow2BitmapDirEntry(fd) for _ in range(self.nb_bitmaps)] + [Qcow2BitmapDirEntry(fd, cluster_size=3Dself.cluster_size) + for _ in range(self.nb_bitmaps)] =20 def dump(self): super().dump() @@ -162,8 +164,9 @@ class Qcow2BitmapDirEntry(Qcow2Struct): ('u32', '{}', 'extra_data_size') ) =20 - def __init__(self, fd): + def __init__(self, fd, cluster_size): super().__init__(fd=3Dfd) + self.cluster_size =3D cluster_size # Seek relative to the current position in the file fd.seek(self.extra_data_size, 1) bitmap_name =3D fd.read(self.name_size) @@ -203,11 +206,13 @@ class QcowHeaderExtension(Qcow2Struct): # then padding to next multiply of 8 ) =20 - def __init__(self, magic=3DNone, length=3DNone, data=3DNone, fd=3DNone= ): + def __init__(self, magic=3DNone, length=3DNone, data=3DNone, fd=3DNone, + cluster_size=3DNone): """ Support both loading from fd and creation from user data. For fd-based creation current position in a file will be used to r= ead the data. + The cluster_size value may be obtained by dependent structures. =20 This should be somehow refactored and functionality should be move= d to superclass (to allow creation of any qcow2 struct), but then, fiel= ds @@ -230,7 +235,7 @@ class QcowHeaderExtension(Qcow2Struct): assert all(v is None for v in (magic, length, data)) super().__init__(fd=3Dfd) if self.magic =3D=3D QCOW2_EXT_MAGIC_BITMAPS: - self.obj =3D Qcow2BitmapExt(fd=3Dfd) + self.obj =3D Qcow2BitmapExt(fd=3Dfd, cluster_size=3Dcluste= r_size) self.data =3D None else: padded =3D (self.length + 7) & ~7 @@ -319,7 +324,7 @@ class QcowHeader(Qcow2Struct): end =3D self.cluster_size =20 while fd.tell() < end: - ext =3D QcowHeaderExtension(fd=3Dfd) + ext =3D QcowHeaderExtension(fd=3Dfd, cluster_size=3Dself.clust= er_size) if ext.magic =3D=3D 0: break else: --=20 1.8.3.1 From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596119027; cv=none; d=zohomail.com; s=zohoarc; b=Wo700b/IW8kWe42xhm81gwmfY8xpApSSFvZoh6ZiHmMGKyTtxgkHGS8/NIe8LQyUxjx9/8g3RVpEW429fgCrgNxUe3S4t1eM6jUT8asIRT4Eur6phqvEIBI5AcDoqALnkzr7ea5KAVMYj4Am5BDXcvhHyC6asaTKA9n2cvtR/R4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596119027; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=6qsDMjp3uMyJAZQeTwA5EI8vptwP0wSWEKWX3PiNb1s=; b=jp/7wB7igR0719GkduQNeS0/v/vw+pL72FxieXvUc74mXkNqHx0MQ30TFdNJ3/CETxErp5lJVaHBW/XpQSEeZUF6DCMnI3yEMd4O4QKUUeCx/beUpx3MHulioMxaaIDdesDMnRk0/XJW3+pAF4/6kBDW1pPczhP8ix9avM4pOYg= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 159611902767223.76207076175092; Thu, 30 Jul 2020 07:23:47 -0700 (PDT) Received: from localhost ([::1]:37976 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19TG-00043m-D5 for importer@patchew.org; Thu, 30 Jul 2020 10:23:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54102) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L8-0004vv-8N; Thu, 30 Jul 2020 10:15:22 -0400 Received: from relay.sw.ru ([185.231.240.75]:44434 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L4-00052b-Ep; Thu, 30 Jul 2020 10:15:21 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Km-0004Cz-Sb; Thu, 30 Jul 2020 17:15:00 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 07/11] qcow2_format.py: Dump bitmap table serialized entries Date: Thu, 30 Jul 2020 17:15:08 +0300 Message-Id: <1596118512-424960-8-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add bitmap table information to the QCOW2 metadata dump. Bitmap name bitmap-1 ... Bitmap table type size offset 0 serialized 65536 10092544 1 all-zeroes 65536 0 2 all-zeroes 65536 0 Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/303.out | 4 ++++ tests/qemu-iotests/qcow2_format.py | 47 ++++++++++++++++++++++++++++++++++= ++++ 2 files changed, 51 insertions(+) diff --git a/tests/qemu-iotests/303.out b/tests/qemu-iotests/303.out index dc3739b..d581fb4 100644 --- a/tests/qemu-iotests/303.out +++ b/tests/qemu-iotests/303.out @@ -70,6 +70,8 @@ type 1 granularity_bits 15 name_size 8 extra_data_size 0 +Bitmap table type size offset +0 serialized 65536 10092544 =20 Bitmap name bitmap-2 bitmap_table_offset 0x9c0000 @@ -79,4 +81,6 @@ type 1 granularity_bits 16 name_size 8 extra_data_size 0 +Bitmap table type size offset +0 all-zeroes 65536 0 =20 diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index ca0d350..1f033d4 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -175,6 +175,10 @@ class Qcow2BitmapDirEntry(Qcow2Struct): entry_raw_size =3D self.bitmap_dir_entry_raw_size() padding =3D ((entry_raw_size + 7) & ~7) - entry_raw_size fd.seek(padding, 1) + self.bitmap_table =3D Qcow2BitmapTable(fd=3Dfd, + offset=3Dself.bitmap_table_of= fset, + nb_entries=3Dself.bitmap_tabl= e_size, + cluster_size=3Dself.cluster_s= ize) =20 def bitmap_dir_entry_raw_size(self): return struct.calcsize(self.fmt) + self.name_size + \ @@ -183,6 +187,49 @@ class Qcow2BitmapDirEntry(Qcow2Struct): def dump(self): print(f'{"Bitmap name":<25} {self.name}') super(Qcow2BitmapDirEntry, self).dump() + self.bitmap_table.dump() + + +class Qcow2BitmapTableEntry(Qcow2Struct): + + fields =3D ( + ('u64', '{}', 'entry'), + ) + + BME_TABLE_ENTRY_RESERVED_MASK =3D 0xff000000000001fe + BME_TABLE_ENTRY_OFFSET_MASK =3D 0x00fffffffffffe00 + BME_TABLE_ENTRY_FLAG_ALL_ONES =3D 1 + + def __init__(self, fd): + super().__init__(fd=3Dfd) + self.reserved =3D self.entry & self.BME_TABLE_ENTRY_RESERVED_MASK + self.offset =3D self.entry & self.BME_TABLE_ENTRY_OFFSET_MASK + if self.offset: + if self.entry & self.BME_TABLE_ENTRY_FLAG_ALL_ONES: + self.type =3D 'invalid' + else: + self.type =3D 'serialized' + elif self.entry & self.BME_TABLE_ENTRY_FLAG_ALL_ONES: + self.type =3D 'all-ones' + else: + self.type =3D 'all-zeroes' + + +class Qcow2BitmapTable: + + def __init__(self, fd, offset, nb_entries, cluster_size): + self.cluster_size =3D cluster_size + position =3D fd.tell() + fd.seek(offset) + self.entries =3D [Qcow2BitmapTableEntry(fd) for _ in range(nb_entr= ies)] + fd.seek(position) + + def dump(self): + size =3D self.cluster_size + bitmap_table =3D enumerate(self.entries) + print(f'{"Bitmap table":<14} {"type":<15} {"size":<12} {"offset"}') + for i, entry in bitmap_table: + print(f'{i:<14} {entry.type:<15} {size:<12} {entry.offset}') =20 =20 QCOW2_EXT_MAGIC_BITMAPS =3D 0x23852875 --=20 1.8.3.1 From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596118842; cv=none; d=zohomail.com; s=zohoarc; b=GCeilyoz8P1URON9zl6spe4evLDS+qkJVOsqc5zzBqiA1+m+mya5i5eNa+Fj8xSHubn4uauF0bXbmghh3NjY+jVmieFNR7wr0aryhKtpGQFmj9QF4iJ3KZWeqq1WNDM3aeUtUrPRmmkmRyVV0vlGqX0EHfyHEr3pm2zGyk4jKVE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596118842; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=H6mFs7TwSLaiEo8ARypAjOSHX0ej+gakH2riarxFYR4=; b=PWKWueO3KgOGn3LKfsZ2GF1X4fsa8xIx2Z3YRSiN5+AfFcOAvFsDr5Woeykm916P9Z0V0K9dWT+HnRMF+Imxi/ZUvE3/iPkuKeasDRbXMfL/b1OhntYMNhQVP3P9WWCdCEMmHUloF2Po5MmjTzz+EQuiI9L8RfB1k4aljzdBpOs= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1596118842967216.78802718754127; Thu, 30 Jul 2020 07:20:42 -0700 (PDT) Received: from localhost ([::1]:52524 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19QH-0006pP-8w for importer@patchew.org; Thu, 30 Jul 2020 10:20:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54098) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L8-0004vh-39; Thu, 30 Jul 2020 10:15:22 -0400 Received: from relay.sw.ru ([185.231.240.75]:44424 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L4-00052X-9e; Thu, 30 Jul 2020 10:15:21 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Km-0004Cz-Tp; Thu, 30 Jul 2020 17:15:00 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 08/11] qcow2.py: Introduce '-j' key to dump in JSON format Date: Thu, 30 Jul 2020 17:15:09 +0300 Message-Id: <1596118512-424960-9-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add the command key to the qcow2.py arguments list to dump QCOW2 metadata in JSON format. Here is the suggested way to do that. The implementation of the dump in JSON format is in the patch that follows. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2.py | 18 ++++++++++++++---- tests/qemu-iotests/qcow2_format.py | 4 ++-- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py index 0910e6a..77ca59c 100755 --- a/tests/qemu-iotests/qcow2.py +++ b/tests/qemu-iotests/qcow2.py @@ -26,16 +26,19 @@ from qcow2_format import ( ) =20 =20 +is_json =3D False + + def cmd_dump_header(fd): h =3D QcowHeader(fd) - h.dump() + h.dump(is_json) print() - h.dump_extensions() + h.dump_extensions(is_json) =20 =20 def cmd_dump_header_exts(fd): h =3D QcowHeader(fd) - h.dump_extensions() + h.dump_extensions(is_json) =20 =20 def cmd_set_header(fd, name, value): @@ -151,11 +154,14 @@ def main(filename, cmd, args): =20 =20 def usage(): - print("Usage: %s [, ...]" % sys.argv[0]) + print("Usage: %s [, ...] [, ...]" % sys.argv[0]) print("") print("Supported commands:") for name, handler, num_args, desc in cmds: print(" %-20s - %s" % (name, desc)) + print("") + print("Supported keys:") + print(" %-20s - %s" % ('-j', 'Dump in JSON format')) =20 =20 if __name__ =3D=3D '__main__': @@ -163,4 +169,8 @@ if __name__ =3D=3D '__main__': usage() sys.exit(1) =20 + is_json =3D '-j' in sys.argv + if is_json: + sys.argv.remove('-j') + main(sys.argv[1], sys.argv[2], sys.argv[3:]) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index 1f033d4..2000de3 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -109,7 +109,7 @@ class Qcow2Struct(metaclass=3DQcow2StructMeta): self.__dict__ =3D dict((field[2], values[i]) for i, field in enumerate(self.fields)) =20 - def dump(self): + def dump(self, is_json=3DFalse): for f in self.fields: value =3D self.__dict__[f[2]] if isinstance(f[1], str): @@ -405,7 +405,7 @@ class QcowHeader(Qcow2Struct): buf =3D buf[0:header_bytes-1] fd.write(buf) =20 - def dump_extensions(self): + def dump_extensions(self, is_json=3DFalse): for ex in self.extensions: print('Header extension:') ex.dump() --=20 1.8.3.1 From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596118755; cv=none; d=zohomail.com; s=zohoarc; b=eyEOUK1mx2kCiUyMrZimcOIZvMjEULfj6xHfx7Tq0OQeD0kao9VKwF9E7W9VqqM8CMOqOMgdF4Qsw9m8mbN4gJpYQqgkI6ByAUsIfV+yW5jnMDUs51FDYMh4Nms8z+BUdd8Hn8Wmo6jjEYa7b7c8NV7dXlWZxoDd397F2YSCQUM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596118755; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=tgG0GOM47c2E1q4yQs73CATmfdGJgE74N1za4f8Bgws=; b=R82pyld17VVIashPaeYmfV2UPNqm2Q4vtdYkzlt6SueSxJhjAO23R1E24cVsO7kEgf5o+Pa9URncZBB9VOGkhjanjnB2H3EGQy1dK0P+LsqDM8IUrVFAWBvuN3Q/ijHtTwokQYCFjBQD6jBdxJdcKu44JrhDOUBVhkiXHF+ooSk= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 159611875530331.355280985725813; Thu, 30 Jul 2020 07:19:15 -0700 (PDT) Received: from localhost ([::1]:44824 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19Os-0003hu-0x for importer@patchew.org; Thu, 30 Jul 2020 10:19:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54156) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19LA-00052I-Vb; Thu, 30 Jul 2020 10:15:24 -0400 Received: from relay.sw.ru ([185.231.240.75]:44426 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L4-00052Y-On; Thu, 30 Jul 2020 10:15:24 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Km-0004Cz-V2; Thu, 30 Jul 2020 17:15:01 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 09/11] qcow2_format.py: collect fields to dump in JSON format Date: Thu, 30 Jul 2020 17:15:10 +0300 Message-Id: <1596118512-424960-10-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" As __dict__ is being extended with class members we do not want to print, add the to_dict() method to classes that returns a dictionary with desired fields and their values. Extend it in subclass when necessary to print the final dictionary in the JSON output which follows. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index 2000de3..a4114cb 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -119,6 +119,9 @@ class Qcow2Struct(metaclass=3DQcow2StructMeta): =20 print('{:<25} {}'.format(f[2], value_str)) =20 + def to_dict(self): + return dict((f[2], self.__dict__[f[2]]) for f in self.fields) + =20 class Qcow2BitmapExt(Qcow2Struct): =20 @@ -151,6 +154,11 @@ class Qcow2BitmapExt(Qcow2Struct): print() entry.dump() =20 + def to_dict(self): + fields_dict =3D super().to_dict() + fields_dict['bitmap_directory'] =3D self.bitmap_directory + return fields_dict + =20 class Qcow2BitmapDirEntry(Qcow2Struct): =20 @@ -189,6 +197,14 @@ class Qcow2BitmapDirEntry(Qcow2Struct): super(Qcow2BitmapDirEntry, self).dump() self.bitmap_table.dump() =20 + def to_dict(self): + fields_dict =3D super().to_dict() + fields_dict['bitmap_table'] =3D self.bitmap_table.entries + bmp_name =3D dict(name=3Dself.name) + # Put the name ahead of the dict + bme_dict =3D {**bmp_name, **fields_dict} + return bme_dict + =20 class Qcow2BitmapTableEntry(Qcow2Struct): =20 @@ -214,6 +230,9 @@ class Qcow2BitmapTableEntry(Qcow2Struct): else: self.type =3D 'all-zeroes' =20 + def to_dict(self): + return dict(type=3Dself.type, offset=3Dself.offset, reserved=3Dsel= f.reserved) + =20 class Qcow2BitmapTable: =20 @@ -246,6 +265,9 @@ class QcowHeaderExtension(Qcow2Struct): 0x44415441: 'Data file' } =20 + def to_dict(self): + return self.mapping.get(self.value, "") + fields =3D ( ('u32', Magic, 'magic'), ('u32', '{}', 'length') @@ -308,6 +330,18 @@ class QcowHeaderExtension(Qcow2Struct): else: self.obj.dump() =20 + def to_dict(self): + fields_dict =3D super().to_dict() + ext_name =3D dict(name=3Dself.Magic(self.magic)) + # Put the name ahead of the dict + he_dict =3D {**ext_name, **fields_dict} + if self.obj is not None: + he_dict['data'] =3D self.obj + else: + he_dict['data_str'] =3D self.data_str + + return he_dict + @classmethod def create(cls, magic, data): return QcowHeaderExtension(magic, len(data), data) --=20 1.8.3.1 From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596118926; cv=none; d=zohomail.com; s=zohoarc; b=ZuVZNqNbZQu59e1mKaH/ArQxY81qaVMsEH01Eul92uL3RNGisw5aLGke1LJFJOmmrP3vTOg/qTa8HGUBegBQOx6QqkPUF3Z8F3Thiwlgo7HvOLbXIsneyfTkLngKgUG+486hbguq7QvzdELcLM60ERK6p+1UlojNqzqAxem05Vc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596118926; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=0a/9nfwr0QuaI715VQGur9bhtq+AOl222r7Ewz8Kgn4=; b=Fk+kDpx3A24tTz5LQOe2cT951TJKocdDFVP89O4/uIKQHIxV5bwILSG0QuVD9rhpL66XY50Ryf7AjrGHMlXVMfcq9SvnoLNtv2lfDlgdRWy3c1m0nDj/oQii1mmatI5h7ykFnkE3hWenlSQ+/sdfGm1az3FXHPPuIWpYAldvB1c= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1596118926987989.4725922812344; Thu, 30 Jul 2020 07:22:06 -0700 (PDT) Received: from localhost ([::1]:59896 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19Rd-0001SF-MM for importer@patchew.org; Thu, 30 Jul 2020 10:22:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54124) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L9-0004yp-CI; Thu, 30 Jul 2020 10:15:23 -0400 Received: from relay.sw.ru ([185.231.240.75]:44428 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L4-00052a-FK; Thu, 30 Jul 2020 10:15:22 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Kn-0004Cz-1M; Thu, 30 Jul 2020 17:15:01 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 10/11] qcow2_format.py: support dumping metadata in JSON format Date: Thu, 30 Jul 2020 17:15:11 +0300 Message-Id: <1596118512-424960-11-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Implementation of dumping QCOW2 image metadata. The sample output: { "Header_extensions": [ { "name": "Feature table", "magic": 1745090647, "length": 192, "data_str": "" }, { "name": "Bitmaps", "magic": 595929205, "length": 24, "data": { "nb_bitmaps": 2, "reserved32": 0, "bitmap_directory_size": 64, "bitmap_directory_offset": 1048576, "bitmap_directory": [ { "name": "bitmap-1", "bitmap_table_offset": 589824, "bitmap_table_size": 1, "flags": 2, "type": 1, "granularity_bits": 15, "name_size": 8, "extra_data_size": 0, "bitmap_table": [ { "type": "serialized", "offset": 655360 }, ... Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index a4114cb..7487720 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -19,6 +19,15 @@ =20 import struct import string +import json + + +class ComplexEncoder(json.JSONEncoder): + def default(self, obj): + if hasattr(obj, 'to_dict'): + return obj.to_dict() + else: + return json.JSONEncoder.default(self, obj) =20 =20 class Qcow2Field: @@ -110,6 +119,10 @@ class Qcow2Struct(metaclass=3DQcow2StructMeta): for i, field in enumerate(self.fields)) =20 def dump(self, is_json=3DFalse): + if is_json: + print(json.dumps(self.to_dict(), indent=3D4, cls=3DComplexEnco= der)) + return + for f in self.fields: value =3D self.__dict__[f[2]] if isinstance(f[1], str): @@ -440,6 +453,10 @@ class QcowHeader(Qcow2Struct): fd.write(buf) =20 def dump_extensions(self, is_json=3DFalse): + if is_json: + print(json.dumps(self.extensions, indent=3D4, cls=3DComplexEnc= oder)) + return + for ex in self.extensions: print('Header extension:') ex.dump() --=20 1.8.3.1 From nobody Thu Apr 25 02:19:49 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1596119024; cv=none; d=zohomail.com; s=zohoarc; b=CSIcjS4x21FNLdGEKKw6jI41cBBLGwAvxPeBQ3flveLSb8c3vVeWyo/yJf2xGdy/VeOv+fcorDCJHTUgOK19ZlFUrtWYa3PJsFv72ZpWNffKJ/zjO/nJSX6X8UOzFa5YvHqDeqYHJNSzXvHkmCsLjS6+a2zMer+B3DvqLzeBOsI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596119024; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=YdmMxvdLYRaoc6YWRF6JViy6GNF8Nq/VO+sFW0IbPj8=; b=QTXNE18c81gOHDGzhVf7o/grq7sEg88is00jpxR6INB5dMYX72ca1wRHRuWdwkQFtIr2CAuSAsQPlhXzWFMZ8e+k+OqwReWQz84pK5ri857Cg3jPNB6z+g/0JpeLRTndhpU0mjEIeqo5gJgxaTTrMxNpTK5rRKD1faYitHQh8WQ= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1596119024358975.3987213074697; Thu, 30 Jul 2020 07:23:44 -0700 (PDT) Received: from localhost ([::1]:37558 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1k19TD-0003tJ-1R for importer@patchew.org; Thu, 30 Jul 2020 10:23:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54146) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19LA-000510-C5; Thu, 30 Jul 2020 10:15:24 -0400 Received: from relay.sw.ru ([185.231.240.75]:44454 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1k19L4-00052i-NA; Thu, 30 Jul 2020 10:15:23 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1k19Kn-0004Cz-2f; Thu, 30 Jul 2020 17:15:01 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v12 11/11] iotests: dump QCOW2 header in JSON in #303 Date: Thu, 30 Jul 2020 17:15:12 +0300 Message-Id: <1596118512-424960-12-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1596118512-424960-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/07/30 10:15:15 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Extend the test case #303 by dumping QCOW2 image metadata in JSON format. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/303 | 3 ++ tests/qemu-iotests/303.out | 76 ++++++++++++++++++++++++++++++++++++++++++= ++++ 2 files changed, 79 insertions(+) diff --git a/tests/qemu-iotests/303 b/tests/qemu-iotests/303 index 3c7a611..6821bd3 100755 --- a/tests/qemu-iotests/303 +++ b/tests/qemu-iotests/303 @@ -57,3 +57,6 @@ add_bitmap(1, 1, 7, False) add_bitmap(2, 7, 9, True) dump =3D ['qcow2.py', f'{disk}', 'dump-header'] subprocess.run(dump) +# Dump the metadata in JSON format +dump.append('-j') +subprocess.run(dump) diff --git a/tests/qemu-iotests/303.out b/tests/qemu-iotests/303.out index d581fb4..ead3b63 100644 --- a/tests/qemu-iotests/303.out +++ b/tests/qemu-iotests/303.out @@ -84,3 +84,79 @@ extra_data_size 0 Bitmap table type size offset 0 all-zeroes 65536 0 =20 +{ + "magic": 1363560955, + "version": 3, + "backing_file_offset": 0, + "backing_file_size": 0, + "cluster_bits": 16, + "size": 10485760, + "crypt_method": 0, + "l1_size": 1, + "l1_table_offset": 196608, + "refcount_table_offset": 65536, + "refcount_table_clusters": 1, + "nb_snapshots": 0, + "snapshot_offset": 0, + "incompatible_features": 0, + "compatible_features": 0, + "autoclear_features": 1, + "refcount_order": 4, + "header_length": 112 +} + +[ + { + "name": "Feature table", + "magic": 1745090647, + "length": 336, + "data_str": "" + }, + { + "name": "Bitmaps", + "magic": 595929205, + "length": 24, + "data": { + "nb_bitmaps": 2, + "reserved32": 0, + "bitmap_directory_size": 64, + "bitmap_directory_offset": 10289152, + "bitmap_directory": [ + { + "name": "bitmap-1", + "bitmap_table_offset": 10158080, + "bitmap_table_size": 1, + "flags": 2, + "type": 1, + "granularity_bits": 15, + "name_size": 8, + "extra_data_size": 0, + "bitmap_table": [ + { + "type": "serialized", + "offset": 10092544, + "reserved": 0 + } + ] + }, + { + "name": "bitmap-2", + "bitmap_table_offset": 10223616, + "bitmap_table_size": 1, + "flags": 0, + "type": 1, + "granularity_bits": 16, + "name_size": 8, + "extra_data_size": 0, + "bitmap_table": [ + { + "type": "all-zeroes", + "offset": 0, + "reserved": 0 + } + ] + } + ] + } + } +] --=20 1.8.3.1