From nobody Sun May 19 09:42:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1591910758; cv=none; d=zohomail.com; s=zohoarc; b=NhQBrAJ+LJUDhypQVyW0ky+GehqNWV0EFbNgns0UKwGgBKxgQtteOWshebJbPIpZGDVBvgkLugEA7ea8c/iEgwlUa4hUmcTq630p1KmdRg3P9YKTU6weCKH4OnhxZOGg1EzKRI68iC6N1FvXiLLfgofDme+JqLobDsORiOIPqCc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1591910758; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=5j52zIyJ7h91qu9hmAm+RftLtQQI+uwoY4LI941x/l0=; b=mOscZa+aYQFdosNpekSH+nT3hiffAkCL9ij/LasR1SpG6SJ5aQaQQPhF8enmtP9jrUZlpiBtLSwjS2FgWZi2hGzz/GEvmPIca4Vf3yk73Sd1zlKz1XDZsbHDKlXK10eeePd8Pf/+MjEtLp5yWm6PNT9878qAbYIqeoH9vy+OBYs= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1591910758813411.4567071945693; Thu, 11 Jun 2020 14:25:58 -0700 (PDT) Received: from localhost ([::1]:51726 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jjUhx-00053P-JS for importer@patchew.org; Thu, 11 Jun 2020 17:25:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46764) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbl-00038n-E4; Thu, 11 Jun 2020 17:19:33 -0400 Received: from relay.sw.ru ([185.231.240.75]:40356 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbf-00057o-7V; Thu, 11 Jun 2020 17:19:33 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jjUbW-0001xE-Nh; Fri, 12 Jun 2020 00:19:18 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v6 1/8] qcow2: Fix capitalization of header extension constant. Date: Fri, 12 Jun 2020 00:19:13 +0300 Message-Id: <1591910360-867499-2-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/11 17:19:22 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Make the capitalization of the hexadecimal numbers consistent for the QCOW2 header extension constants in docs/interop/qcow2.txt. Suggested-by: Eric Blake Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake --- block/qcow2.c | 2 +- docs/interop/qcow2.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 0cd2e67..80dfe5f 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -66,7 +66,7 @@ typedef struct { } QEMU_PACKED QCowExtension; =20 #define QCOW2_EXT_MAGIC_END 0 -#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA +#define QCOW2_EXT_MAGIC_BACKING_FORMAT 0xe2792aca #define QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857 #define QCOW2_EXT_MAGIC_CRYPTO_HEADER 0x0537be77 #define QCOW2_EXT_MAGIC_BITMAPS 0x23852875 diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index cb72346..f072e27 100644 --- a/docs/interop/qcow2.txt +++ b/docs/interop/qcow2.txt @@ -231,7 +231,7 @@ be stored. Each extension has a structure like the foll= owing: =20 Byte 0 - 3: Header extension type: 0x00000000 - End of the header extension area - 0xE2792ACA - Backing file format name string + 0xe2792aca - Backing file format name string 0x6803f857 - Feature name table 0x23852875 - Bitmaps extension 0x0537be77 - Full disk encryption header pointer --=20 1.8.3.1 From nobody Sun May 19 09:42:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1591910516; cv=none; d=zohomail.com; s=zohoarc; b=T/Dn0zJJUnf0XKP9d/Im9tPgWROWxzLzlpjaBmEXNWh01VouiuOH6riPCahPz+qZPod2jB/PnGKRgDVKrk5FNDJJYXbFmq0z+aaigAZQwL4Q4AjiYfRKsge1CojZGcVfRFUUY/uxayo+Z80UpTW0TBJfptLPULSoRixBewGfaeA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1591910516; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=j98/2L8Eg2YeBaa7IbMCoTSo3hCWgqXe9buMwSKjwT0=; b=ba/34E/YELpfVilE9A8feXBZ4otTqS0Dsp4lGKdJekI+ue4vt62ZEuns9iKmqSCUvs6+KfniHg1rsubQFijOb7RdoCmjDPvWSNKy92resgk33sWyYcHGkI9TF83d09T0NbrzrUKkxoQjPzR2scowbonB2444B2LjRJYweGWtKFE= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1591910516525355.76995250497816; Thu, 11 Jun 2020 14:21:56 -0700 (PDT) Received: from localhost ([::1]:35554 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jjUe2-00056I-FY for importer@patchew.org; Thu, 11 Jun 2020 17:21:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46642) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbi-00034M-63; Thu, 11 Jun 2020 17:19:30 -0400 Received: from relay.sw.ru ([185.231.240.75]:40350 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbe-00057p-LZ; Thu, 11 Jun 2020 17:19:29 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jjUbW-0001xE-Oo; Fri, 12 Jun 2020 00:19:18 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v6 2/8] qcow2_format.py: make printable data an extension class member Date: Fri, 12 Jun 2020 00:19:14 +0300 Message-Id: <1591910360-867499-3-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/11 17:19:22 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Let us differ binary data type from string one for the extension data variable and keep the string as the QcowHeaderExtension class member. Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index 0f65fd1..d4f0000 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -164,6 +164,13 @@ class QcowHeaderExtension(Qcow2Struct): self.data =3D fd.read(padded) assert self.data is not None =20 + data_str =3D self.data[:self.length] + if all(c in string.printable.encode('ascii') for c in data_str): + data_str =3D f"'{ data_str.decode('ascii') }'" + else: + data_str =3D '' + self.data_str =3D data_str + if self.magic =3D=3D QCOW2_EXT_MAGIC_BITMAPS: self.obj =3D Qcow2BitmapExt(data=3Dself.data) else: @@ -173,12 +180,7 @@ class QcowHeaderExtension(Qcow2Struct): super().dump() =20 if self.obj is None: - data =3D self.data[:self.length] - if all(c in string.printable.encode('ascii') for c in data): - data =3D f"'{ data.decode('ascii') }'" - else: - data =3D '' - print(f'{"data":<25} {data}') + print(f'{"data":<25} {self.data_str}') else: self.obj.dump() =20 --=20 1.8.3.1 From nobody Sun May 19 09:42:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1591910700; cv=none; d=zohomail.com; s=zohoarc; b=fCdMeX+VRn9hWgObx2G1ayatdtlLTN3gtycPvxV9TdNRof79R7+nlm4l02cKIJbd83EsQsuVBszLCgKlAzJZgndM4jzS8URwdKRNFqDbaoQ90+cBc8H2Cr/iNkCilpqU4Ruj4l0VN/fCWPMvg35IX5zNmW9F/U0zEkfbmO/fVgI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1591910700; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=Sjw7PhxfjetVrYvr/X3l3P4P/DBWt5PuTwqCyGqyjWA=; b=MIiPk64TLV7QcjQzwibddSErM+Rtf1nWAMeOb3Prcu8GWFIlCt4ynpglerl3vhrXUhhp/eY6pY4g3a62w6blfZF7Bq9Zf9Zbxm3lAgXMEf02rKIjWnjnO9zXmcEd/DkgiECb+Slp3xX2gK/0IsJ59blehRj/bJ+zBMghBvYTbvw= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1591910700915433.5033802016045; Thu, 11 Jun 2020 14:25:00 -0700 (PDT) Received: from localhost ([::1]:48006 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jjUh1-0002kY-Au for importer@patchew.org; Thu, 11 Jun 2020 17:24:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46648) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbi-00034W-HL; Thu, 11 Jun 2020 17:19:30 -0400 Received: from relay.sw.ru ([185.231.240.75]:40360 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbf-00057s-2N; Thu, 11 Jun 2020 17:19:30 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jjUbW-0001xE-Pv; Fri, 12 Jun 2020 00:19:18 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v6 3/8] qcow2_format.py: Dump bitmap directory info Date: Fri, 12 Jun 2020 00:19:15 +0300 Message-Id: <1591910360-867499-4-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/11 17:19:22 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Read and dump entries from the bitmap directory of QCOW2 image with the script qcow2.py. Header extension: magic 0x23852875 (Bitmaps) ... Bitmap name bitmap-1 flag auto table size 8 (bytes) bitmap_table_offset 0x90000 bitmap_table_size 1 flags 0 type 1 granularity_bits 16 name_size 8 extra_data_size 0 Suggested-by: Kevin Wolf Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/qcow2_format.py | 75 ++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 75 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index d4f0000..a7868a7 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -103,6 +103,10 @@ class Qcow2Struct(metaclass=3DQcow2StructMeta): print('{:<25} {}'.format(f[2], value_str)) =20 =20 +# seek relative to the current position in the file +FROM_CURRENT =3D 1 + + class Qcow2BitmapExt(Qcow2Struct): =20 fields =3D ( @@ -112,6 +116,73 @@ class Qcow2BitmapExt(Qcow2Struct): ('u64', '{:#x}', 'bitmap_directory_offset') ) =20 + def read_bitmap_directory(self, fd): + self.bitmaps =3D [] + fd.seek(self.bitmap_directory_offset) + buf_size =3D struct.calcsize(Qcow2BitmapDirEntry.fmt) + + for n in range(self.nb_bitmaps): + buf =3D fd.read(buf_size) + dir_entry =3D Qcow2BitmapDirEntry(data=3Dbuf) + fd.seek(dir_entry.extra_data_size, FROM_CURRENT) + bitmap_name =3D fd.read(dir_entry.name_size) + dir_entry.name =3D bitmap_name.decode('ascii') + self.bitmaps.append(dir_entry) + entry_raw_size =3D dir_entry.bitmap_dir_entry_raw_size() + shift =3D ((entry_raw_size + 7) & ~7) - entry_raw_size + fd.seek(shift, FROM_CURRENT) + + def load(self, fd): + self.read_bitmap_directory(fd) + + def dump(self): + super().dump() + for bm in self.bitmaps: + bm.dump_bitmap_dir_entry() + + +BME_FLAG_IN_USE =3D 1 << 0 +BME_FLAG_AUTO =3D 1 << 1 + + +class Qcow2BitmapDirEntry(Qcow2Struct): + + name =3D '' + + fields =3D ( + ('u64', '{:#x}', 'bitmap_table_offset'), + ('u32', '{}', 'bitmap_table_size'), + ('u32', '{}', 'flags'), + ('u8', '{}', 'type'), + ('u8', '{}', 'granularity_bits'), + ('u16', '{}', 'name_size'), + ('u32', '{}', 'extra_data_size') + ) + + def __init__(self, data): + super().__init__(data=3Ddata) + + self.bitmap_table_bytes =3D self.bitmap_table_size \ + * struct.calcsize('Q') + + self.bitmap_flags =3D [] + if (self.flags & BME_FLAG_IN_USE): + self.bitmap_flags.append("in-use") + if (self.flags & BME_FLAG_AUTO): + self.bitmap_flags.append("auto") + + def bitmap_dir_entry_raw_size(self): + return struct.calcsize(self.fmt) + self.name_size + \ + self.extra_data_size + + def dump_bitmap_dir_entry(self): + print() + print(f'{"Bitmap name":<25} {self.name}') + for fl in self.bitmap_flags: + print(f'{"flag":<25} {fl}') + print(f'{"table size ":<25} {self.bitmap_table_bytes} {"(bytes)"}') + super().dump() + =20 QCOW2_EXT_MAGIC_BITMAPS =3D 0x23852875 =20 @@ -253,6 +324,10 @@ class QcowHeader(Qcow2Struct): else: self.extensions.append(ext) =20 + for ext in self.extensions: + if ext.obj is not None: + ext.obj.load(fd) + def update_extensions(self, fd): =20 fd.seek(self.header_length) --=20 1.8.3.1 From nobody Sun May 19 09:42:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1591910526; cv=none; d=zohomail.com; s=zohoarc; b=QUDCBtV2pAf/oxotDvXleYidLZ3yADq9rE47mc1yrkKWf7yyy4sAn7rJKU6Ar5aH6dgRTxxMYymBnwIRkI5m0eui88ODrOgG3av9q4FF5s+5M1guDodCr4aafJlqqynG/V72ekezp51YwitFtEr6x0fKWtOLm9LnMS8GxuILIi8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1591910526; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=FEBKY7EeEyOpnM8wj/Neg1tYvdKqK94JHcLWSHVy1hs=; b=apyGhIzIZ4ebclq9JwjqoWZDobgzasaqZgoBdmt+1jxVycFc5ILY5atdWTIBjedyR9xiW+JMXJwJeilmOxhQQR/z7EC/2pMYQ1CAh+sWOSfWTG5DcbE154nuNlB2IgvvhGbaqHf4XwA2KNohl2Ut44ZstpvHu5abQHHPuVU6FfU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1591910526335221.4428459309338; Thu, 11 Jun 2020 14:22:06 -0700 (PDT) Received: from localhost ([::1]:36444 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jjUeC-0005Sh-Tk for importer@patchew.org; Thu, 11 Jun 2020 17:22:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46656) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbi-00034c-Mb; Thu, 11 Jun 2020 17:19:30 -0400 Received: from relay.sw.ru ([185.231.240.75]:40362 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbf-00057r-5p; Thu, 11 Jun 2020 17:19:30 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jjUbW-0001xE-Qz; Fri, 12 Jun 2020 00:19:18 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v6 4/8] qcow2_format.py: pass cluster size to substructures Date: Fri, 12 Jun 2020 00:19:16 +0300 Message-Id: <1591910360-867499-5-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/11 17:19:22 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" The cluster size of an image is the QcowHeader class member and may be obtained by dependent extension structures such as Qcow2BitmapExt for further bitmap table details print. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index a7868a7..6589f68 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -116,6 +116,10 @@ class Qcow2BitmapExt(Qcow2Struct): ('u64', '{:#x}', 'bitmap_directory_offset') ) =20 + def __init__(self, data, cluster_size): + super().__init__(data=3Ddata) + self.cluster_size =3D cluster_size + def read_bitmap_directory(self, fd): self.bitmaps =3D [] fd.seek(self.bitmap_directory_offset) @@ -123,7 +127,8 @@ class Qcow2BitmapExt(Qcow2Struct): =20 for n in range(self.nb_bitmaps): buf =3D fd.read(buf_size) - dir_entry =3D Qcow2BitmapDirEntry(data=3Dbuf) + dir_entry =3D Qcow2BitmapDirEntry(data=3Dbuf, + cluster_size=3Dself.cluster_si= ze) fd.seek(dir_entry.extra_data_size, FROM_CURRENT) bitmap_name =3D fd.read(dir_entry.name_size) dir_entry.name =3D bitmap_name.decode('ascii') @@ -159,8 +164,9 @@ class Qcow2BitmapDirEntry(Qcow2Struct): ('u32', '{}', 'extra_data_size') ) =20 - def __init__(self, data): + def __init__(self, data, cluster_size): super().__init__(data=3Ddata) + self.cluster_size =3D cluster_size =20 self.bitmap_table_bytes =3D self.bitmap_table_size \ * struct.calcsize('Q') @@ -205,11 +211,13 @@ class QcowHeaderExtension(Qcow2Struct): # then padding to next multiply of 8 ) =20 - def __init__(self, magic=3DNone, length=3DNone, data=3DNone, fd=3DNone= ): + def __init__(self, magic=3DNone, length=3DNone, data=3DNone, fd=3DNone, + cluster_size=3DNone): """ Support both loading from fd and creation from user data. For fd-based creation current position in a file will be used to r= ead the data. + The cluster_size value may be obtained by dependent structures. =20 This should be somehow refactored and functionality should be move= d to superclass (to allow creation of any qcow2 struct), but then, fiel= ds @@ -243,7 +251,8 @@ class QcowHeaderExtension(Qcow2Struct): self.data_str =3D data_str =20 if self.magic =3D=3D QCOW2_EXT_MAGIC_BITMAPS: - self.obj =3D Qcow2BitmapExt(data=3Dself.data) + self.obj =3D Qcow2BitmapExt(data=3Dself.data, + cluster_size=3Dcluster_size) else: self.obj =3D None =20 @@ -318,7 +327,7 @@ class QcowHeader(Qcow2Struct): end =3D self.cluster_size =20 while fd.tell() < end: - ext =3D QcowHeaderExtension(fd=3Dfd) + ext =3D QcowHeaderExtension(fd=3Dfd, cluster_size=3Dself.clust= er_size) if ext.magic =3D=3D 0: break else: --=20 1.8.3.1 From nobody Sun May 19 09:42:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1591910662; cv=none; d=zohomail.com; s=zohoarc; b=ED1TahZcfJ5mkqd/2VHR0obcoQzkOk8QaupZRCXZoxGfHKq6rgjQgjqmm6jZiPimgyZIhAKJaodN3JdXY5x2PfSe6VeAwBFklzDJjEP3/ducggmA03iFuof/tNZN+bolXXF7+8QSiSnKrxCR8GSrmUFbHysU8wosS3dHjEBdeEM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1591910662; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=lvcDp/KSuPWe1WNsF5Hc842GL8n9Dw6tMWXpDHsE6i4=; b=i9SMGok4donU80M478dBJoi/kdDKgPtfqCUTc1yuaDQ1Hjivt+4loeaX9bqQ3SKqjDQt7pLpUn1fKGxXxavHpOJs9D+krANqqyxFMy2dhBVrxnFHjNMB2zjEpdO9cdFwFAFMLjRbq5PQa1k+V4MhgkDgzZSyBYp9hWUTsI80Vv0= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1591910662970561.3597299770915; Thu, 11 Jun 2020 14:24:22 -0700 (PDT) Received: from localhost ([::1]:45260 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jjUgP-0001Dp-Jh for importer@patchew.org; Thu, 11 Jun 2020 17:24:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46708) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbj-000363-S3; Thu, 11 Jun 2020 17:19:31 -0400 Received: from relay.sw.ru ([185.231.240.75]:40358 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbf-00057n-7e; Thu, 11 Jun 2020 17:19:31 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jjUbW-0001xE-S3; Fri, 12 Jun 2020 00:19:18 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v6 5/8] qcow2_format.py: Dump bitmap table serialized entries Date: Fri, 12 Jun 2020 00:19:17 +0300 Message-Id: <1591910360-867499-6-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/11 17:19:22 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add bitmap table information to the QCOW2 metadata dump. Bitmap name bitmap-1 ... Bitmap table type offset size 0 serialized 0xa0000 65536 1 all-zeroes 0x0 65536 2 all-zeroes 0x0 65536 3 all-zeroes 0x0 65536 4 all-zeroes 0x0 65536 5 all-zeroes 0x0 65536 6 all-zeroes 0x0 65536 7 all-zeroes 0x0 65536 Signed-off-by: Andrey Shinkevich Reviewed-by: Eric Blake --- tests/qemu-iotests/qcow2_format.py | 42 ++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 42 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index 6589f68..2e7c058 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -137,6 +137,9 @@ class Qcow2BitmapExt(Qcow2Struct): shift =3D ((entry_raw_size + 7) & ~7) - entry_raw_size fd.seek(shift, FROM_CURRENT) =20 + for bm in self.bitmaps: + bm.read_bitmap_table(fd) + def load(self, fd): self.read_bitmap_directory(fd) =20 @@ -181,6 +184,12 @@ class Qcow2BitmapDirEntry(Qcow2Struct): return struct.calcsize(self.fmt) + self.name_size + \ self.extra_data_size =20 + def read_bitmap_table(self, fd): + fd.seek(self.bitmap_table_offset) + table_size =3D self.bitmap_table_bytes * struct.calcsize('Q') + table =3D [e[0] for e in struct.iter_unpack('>Q', fd.read(table_si= ze))] + self.bitmap_table =3D Qcow2BitmapTable(table) + def dump_bitmap_dir_entry(self): print() print(f'{"Bitmap name":<25} {self.name}') @@ -188,6 +197,39 @@ class Qcow2BitmapDirEntry(Qcow2Struct): print(f'{"flag":<25} {fl}') print(f'{"table size ":<25} {self.bitmap_table_bytes} {"(bytes)"}') super().dump() + self.bitmap_table.print_bitmap_table(self.cluster_size) + + +class Qcow2BitmapTableEntry: + + BME_TABLE_ENTRY_OFFSET_MASK =3D 0x00fffffffffffe00 + BME_TABLE_ENTRY_FLAG_ALL_ONES =3D 1 + bmte_type =3D ['all-zeroes', 'all-ones', 'serialized'] + + def __init__(self, entry): + self.offset =3D entry & self.BME_TABLE_ENTRY_OFFSET_MASK + if self.offset: + self.type =3D 'serialized' + elif entry & self.BME_TABLE_ENTRY_FLAG_ALL_ONES: + self.type =3D 'all-ones' + else: + self.type =3D 'all-zeroes' + + +class Qcow2BitmapTable: + + def __init__(self, raw_table): + self.entries =3D [] + for entry in raw_table: + self.entries.append(Qcow2BitmapTableEntry(entry)) + + def print_bitmap_table(self, cluster_size): + bitmap_table =3D enumerate(self.entries) + print("Bitmap table\ttype\t\toffset\t\tsize") + for i, entry in bitmap_table: + print("\t%-4d\t%s\t%#x\t\t%d" % (i, entry.type, entry.offset, + cluster_size)) + print("") =20 =20 QCOW2_EXT_MAGIC_BITMAPS =3D 0x23852875 --=20 1.8.3.1 From nobody Sun May 19 09:42:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1591910528; cv=none; d=zohomail.com; s=zohoarc; b=gWww7/EHsXm1GY9BR15BXkP15r5U1aPKbSsaVcpYECVJwUyXwYrkMenoIY58z1KP8SQCv+/aPsQo/ykwX/CzBTB65tJCjIPHRdH7cSAObMGsrI6VEPGEUA7PvFnSGRY1vXk6QVEblG20LEguWGv7+tAWRIkzQO9pDF2kM9Flfek= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1591910528; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=3y6q51PjJ39u6ZuGHzdq3vE7IwwJarG2kkz95e5WQFY=; b=DOmlaPHz+upxVILJIyzMHWXjAYh/5Y+L4HoexAYWgI1TW+FwEJ8BGDnhFCN0sl2UrS1sbYLipp5fXJRYqYSK/3wUzy/e860+1V80B1nKP/nGRrzNBJl7Myc0nD2dtgIiljZ6+3PPpFTlAW275IHQuLFBqYg4ny1TNjmR21aTSRM= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1591910528701351.66004944977306; Thu, 11 Jun 2020 14:22:08 -0700 (PDT) Received: from localhost ([::1]:36568 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jjUeF-0005VV-3O for importer@patchew.org; Thu, 11 Jun 2020 17:22:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46724) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbk-00036q-5m; Thu, 11 Jun 2020 17:19:32 -0400 Received: from relay.sw.ru ([185.231.240.75]:40352 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbf-00057m-8w; Thu, 11 Jun 2020 17:19:31 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jjUbW-0001xE-TB; Fri, 12 Jun 2020 00:19:18 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v6 6/8] qcow2.py: Introduce '-j' key to dump in JSON format Date: Fri, 12 Jun 2020 00:19:18 +0300 Message-Id: <1591910360-867499-7-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/11 17:19:22 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Add the command key to the qcow2.py arguments list to dump QCOW2 metadata in JSON format. Here is the suggested way to do that. The implementation of the dump in JSON format is in the patch that follows. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2.py | 20 +++++++++++++++----- tests/qemu-iotests/qcow2_format.py | 18 +++++++++--------- 2 files changed, 24 insertions(+), 14 deletions(-) diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py index 8c187e9..b08d8fc 100755 --- a/tests/qemu-iotests/qcow2.py +++ b/tests/qemu-iotests/qcow2.py @@ -24,16 +24,19 @@ from qcow2_format import ( ) =20 =20 +dump_json =3D False + + def cmd_dump_header(fd): h =3D QcowHeader(fd) - h.dump() + h.dump(dump_json) print() - h.dump_extensions() + h.dump_extensions(dump_json) =20 =20 def cmd_dump_header_exts(fd): h =3D QcowHeader(fd) - h.dump_extensions() + h.dump_extensions(dump_json) =20 =20 def cmd_set_header(fd, name, value): @@ -132,6 +135,11 @@ cmds =3D [ =20 =20 def main(filename, cmd, args): + global dump_json + dump_json =3D '-j' in sys.argv + if dump_json: + sys.argv.remove('-j') + args.remove('-j') fd =3D open(filename, "r+b") try: for name, handler, num_args, desc in cmds: @@ -149,12 +157,14 @@ def main(filename, cmd, args): =20 =20 def usage(): - print("Usage: %s [, ...]" % sys.argv[0]) + print("Usage: %s [, ...] [, ...]" % sys.argv[0]) print("") print("Supported commands:") for name, handler, num_args, desc in cmds: print(" %-20s - %s" % (name, desc)) - + print("") + print("Supported keys:") + print(" %-20s - %s" % ('-j', 'Dump in JSON format')) =20 if __name__ =3D=3D '__main__': if len(sys.argv) < 3: diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index 2e7c058..95db7a9 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -92,7 +92,7 @@ class Qcow2Struct(metaclass=3DQcow2StructMeta): self.__dict__ =3D dict((field[2], values[i]) for i, field in enumerate(self.fields)) =20 - def dump(self): + def dump(self, dump_json=3DNone): for f in self.fields: value =3D self.__dict__[f[2]] if isinstance(f[1], str): @@ -143,10 +143,10 @@ class Qcow2BitmapExt(Qcow2Struct): def load(self, fd): self.read_bitmap_directory(fd) =20 - def dump(self): - super().dump() + def dump(self, dump_json=3DNone): + super().dump(dump_json) for bm in self.bitmaps: - bm.dump_bitmap_dir_entry() + bm.dump_bitmap_dir_entry(dump_json) =20 =20 BME_FLAG_IN_USE =3D 1 << 0 @@ -190,7 +190,7 @@ class Qcow2BitmapDirEntry(Qcow2Struct): table =3D [e[0] for e in struct.iter_unpack('>Q', fd.read(table_si= ze))] self.bitmap_table =3D Qcow2BitmapTable(table) =20 - def dump_bitmap_dir_entry(self): + def dump_bitmap_dir_entry(self, dump_json=3DNone): print() print(f'{"Bitmap name":<25} {self.name}') for fl in self.bitmap_flags: @@ -298,13 +298,13 @@ class QcowHeaderExtension(Qcow2Struct): else: self.obj =3D None =20 - def dump(self): + def dump(self, dump_json=3DNone): super().dump() =20 if self.obj is None: print(f'{"data":<25} {self.data_str}') else: - self.obj.dump() + self.obj.dump(dump_json) =20 @classmethod def create(cls, magic, data): @@ -407,8 +407,8 @@ class QcowHeader(Qcow2Struct): buf =3D buf[0:header_bytes-1] fd.write(buf) =20 - def dump_extensions(self): + def dump_extensions(self, dump_json=3DNone): for ex in self.extensions: print('Header extension:') - ex.dump() + ex.dump(dump_json) print() --=20 1.8.3.1 From nobody Sun May 19 09:42:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1591910581; cv=none; d=zohomail.com; s=zohoarc; b=lZGkVJLa/LzN6hRvXT+6ydq7RDL6GGHE6HyDaGyMigq5ZGl+a7d4BgM/DlsxRcLntk1Me7Z77pcqti4ILIEa0zIpYVUMOa2s92SWEHTujce7rME36z48Or2FizszOCzArRA+vMzalDSzKJqiK4qJrdVVS/aJFhwWMMNLEzzZ8Rk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1591910581; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=Kl6dl+5P+xjuOmIj8A+sIGlGi3Rm6LJHeW4pQ0fiNjY=; b=Ag9mCixl0reUmx4Z77icbZxkxUZbiuCfiqJHiaSqPhhK1yRPx+IRyQL2WCCkAEHGuGBebxUeE1cVl+ZuWZcSfqO1R3lfgnUolVnVyjpVOA02mnfP0J5mCaHVoAPPAtezS5WQCBMc/cjUht+JJaKEF3igPI4VPfgXPUv2QRdxhd4= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1591910581073961.4554822436229; Thu, 11 Jun 2020 14:23:01 -0700 (PDT) Received: from localhost ([::1]:40058 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jjUf5-000710-PR for importer@patchew.org; Thu, 11 Jun 2020 17:22:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46600) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbh-00033a-RD; Thu, 11 Jun 2020 17:19:29 -0400 Received: from relay.sw.ru ([185.231.240.75]:40354 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbe-00057q-P2; Thu, 11 Jun 2020 17:19:28 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jjUbW-0001xE-UE; Fri, 12 Jun 2020 00:19:18 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v6 7/8] qcow2_format.py: collect fields to dump in JSON format Date: Fri, 12 Jun 2020 00:19:19 +0300 Message-Id: <1591910360-867499-8-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/11 17:19:22 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" As __dict__ is being extended with class members we do not want to print in JSON format dump, make a light copy of the initial __dict__ and extend the copy by adding lists we have to print in the JSON output. Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index 95db7a9..bb2f13b 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -92,6 +92,8 @@ class Qcow2Struct(metaclass=3DQcow2StructMeta): self.__dict__ =3D dict((field[2], values[i]) for i, field in enumerate(self.fields)) =20 + self.fields_dict =3D self.__dict__.copy() + def dump(self, dump_json=3DNone): for f in self.fields: value =3D self.__dict__[f[2]] @@ -102,7 +104,6 @@ class Qcow2Struct(metaclass=3DQcow2StructMeta): =20 print('{:<25} {}'.format(f[2], value_str)) =20 - # seek relative to the current position in the file FROM_CURRENT =3D 1 =20 @@ -142,6 +143,7 @@ class Qcow2BitmapExt(Qcow2Struct): =20 def load(self, fd): self.read_bitmap_directory(fd) + self.fields_dict.update(entries=3Dself.bitmaps) =20 def dump(self, dump_json=3DNone): super().dump(dump_json) @@ -189,6 +191,7 @@ class Qcow2BitmapDirEntry(Qcow2Struct): table_size =3D self.bitmap_table_bytes * struct.calcsize('Q') table =3D [e[0] for e in struct.iter_unpack('>Q', fd.read(table_si= ze))] self.bitmap_table =3D Qcow2BitmapTable(table) + self.fields_dict.update(bitmap_table=3Dself.bitmap_table) =20 def dump_bitmap_dir_entry(self, dump_json=3DNone): print() --=20 1.8.3.1 From nobody Sun May 19 09:42:12 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1591910650; cv=none; d=zohomail.com; s=zohoarc; b=l2jb8rZOuK9rcKgq+z0OB/oAbIit0PqUS5fdDGvkM2Omf76GneGEAReBMGWPMj/BNT3wfRm5eut36x/dPunT7z0Gepdxg/Ybiy2t77YWzEmyEUcKizFIiwekT0hNIWOYgbo986Gz+R1r1CTSVABfYv40kq0p61t5u9iwVStnnW0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1591910650; h=Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=s85O3zgbSZev3Jz8G32rwtkLWm8rP8Ef5L6ytnvN39Q=; b=XXaCVUSlEX79S5URAo9mBO1JHIPvFqolDQLXbDh4XNRV/QIULJS5mn7eH/+PhLK9bI4+KUDIzQkLYGQ0DTXMr3pz0W5N3hzuHyRmdk1ijVegySI8bD2IGkPEvv5aggDW5PL4uuWA8jRRvSnXDDA6Zorb7T5iy7j/MLtbe1KmbCk= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1591910650926276.77024284336403; Thu, 11 Jun 2020 14:24:10 -0700 (PDT) Received: from localhost ([::1]:44384 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jjUgD-0000kL-Ig for importer@patchew.org; Thu, 11 Jun 2020 17:24:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46736) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbk-00037T-GR; Thu, 11 Jun 2020 17:19:32 -0400 Received: from relay.sw.ru ([185.231.240.75]:40346 helo=relay3.sw.ru) by eggs.gnu.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjUbf-00057l-6h; Thu, 11 Jun 2020 17:19:32 -0400 Received: from [172.16.25.136] (helo=localhost.sw.ru) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jjUbW-0001xE-VJ; Fri, 12 Jun 2020 00:19:18 +0300 From: Andrey Shinkevich To: qemu-block@nongnu.org Subject: [PATCH v6 8/8] qcow2_format.py: support dumping metadata in JSON format Date: Fri, 12 Jun 2020 00:19:20 +0300 Message-Id: <1591910360-867499-9-git-send-email-andrey.shinkevich@virtuozzo.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> References: <1591910360-867499-1-git-send-email-andrey.shinkevich@virtuozzo.com> Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.231.240.75; envelope-from=andrey.shinkevich@virtuozzo.com; helo=relay3.sw.ru X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/11 17:19:22 X-ACL-Warn: Detected OS = Linux 3.11 and newer X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, qemu-devel@nongnu.org, mreitz@redhat.com, andrey.shinkevich@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Implementation of dumping QCOW2 image metadata. The sample output: { "Header_extensions": [ { "name": "Feature table", "magic": 1745090647, "length": 192, "data_str": "" }, { "name": "Bitmaps", "magic": 595929205, "length": 24, "data": { "nb_bitmaps": 2, "reserved32": 0, "bitmap_directory_size": 64, "bitmap_directory_offset": 1048576, "entries": [ { "name": "bitmap-1", "bitmap_table_offset": 589824, "bitmap_table_size": 1, "flags": [ "auto" ], "type": 1, "granularity_bits": 16, "name_size": 8, "extra_data_size": 0, "bitmap_table": { "table_entries": [ { "type": "serialized", "offset": 655360 }, ... Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich --- tests/qemu-iotests/qcow2_format.py | 60 ++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 60 insertions(+) diff --git a/tests/qemu-iotests/qcow2_format.py b/tests/qemu-iotests/qcow2_= format.py index bb2f13b..d87f8e0 100644 --- a/tests/qemu-iotests/qcow2_format.py +++ b/tests/qemu-iotests/qcow2_format.py @@ -18,6 +18,15 @@ =20 import struct import string +import json + + +class ComplexEncoder(json.JSONEncoder): + def default(self, obj): + if hasattr(obj, 'get_fields_dict'): + return obj.get_fields_dict() + else: + return json.JSONEncoder.default(self, obj) =20 =20 class Qcow2Field: @@ -95,6 +104,11 @@ class Qcow2Struct(metaclass=3DQcow2StructMeta): self.fields_dict =3D self.__dict__.copy() =20 def dump(self, dump_json=3DNone): + if dump_json: + print(json.dumps(self.get_fields_dict(), indent=3D4, + cls=3DComplexEncoder)) + return + for f in self.fields: value =3D self.__dict__[f[2]] if isinstance(f[1], str): @@ -150,6 +164,9 @@ class Qcow2BitmapExt(Qcow2Struct): for bm in self.bitmaps: bm.dump_bitmap_dir_entry(dump_json) =20 + def get_fields_dict(self): + return self.fields_dict + =20 BME_FLAG_IN_USE =3D 1 << 0 BME_FLAG_AUTO =3D 1 << 1 @@ -202,6 +219,12 @@ class Qcow2BitmapDirEntry(Qcow2Struct): super().dump() self.bitmap_table.print_bitmap_table(self.cluster_size) =20 + def get_fields_dict(self): + bmp_name =3D dict(name=3Dself.name) + bme_dict =3D {**bmp_name, **self.fields_dict} + bme_dict['flags'] =3D self.bitmap_flags + return bme_dict + =20 class Qcow2BitmapTableEntry: =20 @@ -218,6 +241,9 @@ class Qcow2BitmapTableEntry: else: self.type =3D 'all-zeroes' =20 + def get_fields_dict(self): + return dict(type=3Dself.type, offset=3Dself.offset) + =20 class Qcow2BitmapTable: =20 @@ -234,6 +260,18 @@ class Qcow2BitmapTable: cluster_size)) print("") =20 + def get_fields_dict(self): + return dict(table_entries=3Dself.entries) + + +class Qcow2HeaderExtensionsDoc: + + def __init__(self, extensions): + self.extensions =3D extensions + + def get_fields_dict(self): + return dict(Header_extensions=3Dself.extensions) + =20 QCOW2_EXT_MAGIC_BITMAPS =3D 0x23852875 =20 @@ -249,6 +287,9 @@ class QcowHeaderExtension(Qcow2Struct): 0x44415441: 'Data file' } =20 + def get_fields_dict(self): + return self.mapping.get(self.value, "") + fields =3D ( ('u32', Magic, 'magic'), ('u32', '{}', 'length') @@ -309,6 +350,16 @@ class QcowHeaderExtension(Qcow2Struct): else: self.obj.dump(dump_json) =20 + def get_fields_dict(self): + ext_name =3D dict(name=3Dself.Magic(self.magic)) + he_dict =3D {**ext_name, **self.fields_dict} + if self.obj is not None: + he_dict.update(data=3Dself.obj) + else: + he_dict.update(data_str=3Dself.data_str) + + return he_dict + @classmethod def create(cls, magic, data): return QcowHeaderExtension(magic, len(data), data) @@ -411,7 +462,16 @@ class QcowHeader(Qcow2Struct): fd.write(buf) =20 def dump_extensions(self, dump_json=3DNone): + if dump_json: + ext_doc =3D Qcow2HeaderExtensionsDoc(self.extensions) + print(json.dumps(ext_doc.get_fields_dict(), indent=3D4, + cls=3DComplexEncoder)) + return + for ex in self.extensions: print('Header extension:') ex.dump(dump_json) print() + + def get_fields_dict(self): + return self.fields_dict --=20 1.8.3.1