[PATCH v3 0/3] Dump QCOW2 metadata

Andrey Shinkevich posted 3 patches 29 weeks ago
Test docker-mingw@fedora passed
Test checkpatch passed
Test docker-quick@centos7 passed
Test FreeBSD passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/1578990137-308222-1-git-send-email-andrey.shinkevich@virtuozzo.com
Maintainers: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>, Max Reitz <mreitz@redhat.com>, John Snow <jsnow@redhat.com>, Kevin Wolf <kwolf@redhat.com>, Markus Armbruster <armbru@redhat.com>, Eric Blake <eblake@redhat.com>
block/qcow2-bitmap.c   |  54 ++++++++++++-
block/qcow2-refcount.c |  84 ++++++++++++++++----
block/qcow2.c          |  30 +++++++
block/qcow2.h          |   6 +-
include/block/block.h  |   3 +-
qapi/block-core.json   | 209 ++++++++++++++++++++++++++++++++++++++++++++++++-
qemu-img.c             |  50 +++++++++---
qemu-img.texi          |   7 +-
8 files changed, 408 insertions(+), 35 deletions(-)

[PATCH v3 0/3] Dump QCOW2 metadata

Posted by Andrey Shinkevich 29 weeks ago
The information about QCOW2 metadata allocations in an image ELF-file is
helpful for finding issues with the image data integrity.

v3: Descriptions of the new key option were added. The names of identifiers
    were amended. The QEMU-IMG key options were put in the sorted order in
    the code. All suggested by Eric. Discussed in the email thread with ID
    <1577447039-400109-1-git-send-email-andrey.shinkevich@virtuozzo.com>

Snapshots dump example:

$ sudo ./qemu-img check /.../.../harddisk.hdd -M --output=json
{
    "image-end-offset": 24820842496,
    "total-clusters": 153600,
    "check-errors": 0,
    "viscera": {
        "refcount-table": {
            "location": {
                "offset": 3845128192,
                "size": 1048576
            }
        },
        "active-l1": {
            "name": "L1 active table",
            "location": {
                "offset": 4194304,
                "size": 16
            },
            "l2-list": [
                {
                    "offset": 619708416,
                    "size": 1048576
                },
                {
                    "offset": 1156579328,
                    "size": 1048576
                }
            ]
        },
        "qcow2-header": {
            "location": {
                "offset": 0,
                "size": 1048576
            },
            "version": 3
        },
        "snapshot-table": {
            "location": {
                "offset": 648019968,
                "size": 191
            },
            "l1-list": [
                {
                    "name": "{3036f6c5-3a1f-44cb-af1f-653cc87fba04}",
                    "location": {
                        "offset": 14680064,
                        "size": 16
                    },
                    "l2-list": [
                        {
                            "offset": 3957325824,
                            "size": 1048576
                        },
                        {
                            "offset": 7025459200,
                            "size": 1048576
                        }
                    ]
                },
                {
                    "name": "{0aa1a7d6-16ee-4b44-a515-b5ecc571c959}",
                    "location": {
                        "offset": 638582784,
                        "size": 16
                    },
                    "l2-list": [
                        {
                            "offset": 3957325824,
                            "size": 1048576
                        },
                        {
                            "offset": 7025459200,
                            "size": 1048576
                        }
                    ]
                }
            ]
        }
    },
    "allocated-clusters": 22485,
    "filename": "/.../.../harddisk.hdd",
    "format": "qcow2",
    "fragmented-clusters": 3549
}

Bitmaps dump example:

$ ./qemu-img check /home/disk -M --output=json
{
    "image-end-offset": 1441792,
    "total-clusters": 16,
    "check-errors": 0,
    "viscera": {
        "refcount-table": {
            "location": {
                "offset": 65536,
                "size": 65536
            }
        },
        "active-l1": {
            "name": "L1 active table",
            "location": {
                "offset": 196608,
                "size": 8
            },
            "l2-list": [
                {
                    "offset": 262144,
                    "size": 65536
                }
            ]
        },
        "bitmaps": {
            "bitmap-dir": {
                "location": {
                    "offset": 1048576,
                    "size": 64
                },
                "dir-entries": [
                    {
                        "bitmap-table": {
                            "location": {
                                "offset": 589824,
                                "size": 8
                            },
                            "table-entries": [
                                {
                                    "type": "all-zeros"
                                }
                            ]
                        },
                        "bitmap-name": "bitmap-1"
                    },
                    {
                        "bitmap-table": {
                            "location": {
                                "offset": 983040,
                                "size": 8
                            },
                            "table-entries": [
                                {
                                    "cluster": {
                                        "offset": 655360,
                                        "size": 65536
                                    },
                                    "type": "serialized"
                                }
                            ]
                        },
                        "bitmap-name": "bitmap-2"
                    }
                ]
            },
            "nb-bitmaps": 2
        },
        "qcow2-header": {
            "location": {
                "offset": 0,
                "size": 65536
            },
            "version": 3
        }
    },
    "allocated-clusters": 12,
    "filename": "/home/disk",
    "format": "qcow2",
    "fragmented-clusters": 2
}

Andrey Shinkevich (3):
  qcow2: introduce Qcow2Metadata structure
  qemu-img: sort key options alphabetically
  qcow2: dump QCOW2 metadata

 block/qcow2-bitmap.c   |  54 ++++++++++++-
 block/qcow2-refcount.c |  84 ++++++++++++++++----
 block/qcow2.c          |  30 +++++++
 block/qcow2.h          |   6 +-
 include/block/block.h  |   3 +-
 qapi/block-core.json   | 209 ++++++++++++++++++++++++++++++++++++++++++++++++-
 qemu-img.c             |  50 +++++++++---
 qemu-img.texi          |   7 +-
 8 files changed, 408 insertions(+), 35 deletions(-)

-- 
1.8.3.1


Re: [PATCH v3 0/3] Dump QCOW2 metadata

Posted by Max Reitz 24 weeks ago
On 14.01.20 09:22, Andrey Shinkevich wrote:
> The information about QCOW2 metadata allocations in an image ELF-file is
> helpful for finding issues with the image data integrity.

Sorry that I’m replying only so late – but I don’t know why we need this
in qemu, and this cover letter doesn’t provide a justification.  I mean,
it isn’t too complex (from the diffstat), but wouldn’t it be better to
just have a script for this?

I suppose one reason to put it in qemu/qemu-img is that a script
wouldn’t be packaged by distributions.  So if a user has a corrupted
image, with this series we could tell them to run qemu-img check -M and
put the output somewhere.  With a script, we’d first have to tell them
to download the script.  But then again, downloading a script (that
should be part of the qemu repository) doesn’t seem too much trouble to
me either.

So I’m curious as to whether you had a specific reason in mind when you
decided to implement this as part of qemu itself?

(I suppose the additional complexity is fully limited to the check
infrastructure, so it wouldn’t interfere with the rest of the qcow2
driver.  Hm.  Fair enough.)

Max

Re: [PATCH v3 0/3] Dump QCOW2 metadata

Posted by Kevin Wolf 24 weeks ago
Am 20.02.2020 um 12:58 hat Max Reitz geschrieben:
> On 14.01.20 09:22, Andrey Shinkevich wrote:
> > The information about QCOW2 metadata allocations in an image ELF-file is
> > helpful for finding issues with the image data integrity.
> 
> Sorry that I’m replying only so late – but I don’t know why we need this
> in qemu, and this cover letter doesn’t provide a justification.  I mean,
> it isn’t too complex (from the diffstat), but wouldn’t it be better to
> just have a script for this?

Specifically, we could extend tests/qemu-iotests/qcow2.py. This seems to
be debugging output that would be in line with what the script is
already used for.

Kevin

Re: [PATCH v3 0/3] Dump QCOW2 metadata

Posted by Eric Blake 24 weeks ago
On 2/20/20 6:28 AM, Kevin Wolf wrote:
> Am 20.02.2020 um 12:58 hat Max Reitz geschrieben:
>> On 14.01.20 09:22, Andrey Shinkevich wrote:
>>> The information about QCOW2 metadata allocations in an image ELF-file is
>>> helpful for finding issues with the image data integrity.
>>
>> Sorry that I’m replying only so late – but I don’t know why we need this
>> in qemu, and this cover letter doesn’t provide a justification.  I mean,
>> it isn’t too complex (from the diffstat), but wouldn’t it be better to
>> just have a script for this?
> 
> Specifically, we could extend tests/qemu-iotests/qcow2.py. This seems to
> be debugging output that would be in line with what the script is
> already used for.

I also just discovered GNU poke, http://jemarch.net/poke, which is an 
arbitrary binary-format editor with a fairly good example of how it can 
be used to inspect ELF files.  I'm wondering if it would be easier to 
write a pickle describing the qcow2 format that would make it easier to 
do interactive browsing/editing of a qcow2 file, at the expense of 
having to depend on poke (which has not yet hit the 1.0 release and is 
not yet bundled for Fedora).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


Re: [PATCH v3 0/3] Dump QCOW2 metadata

Posted by Vladimir Sementsov-Ogievskiy 24 weeks ago
20.02.2020 14:58, Max Reitz wrote:
> On 14.01.20 09:22, Andrey Shinkevich wrote:
>> The information about QCOW2 metadata allocations in an image ELF-file is
>> helpful for finding issues with the image data integrity.
> 
> Sorry that I’m replying only so late – but I don’t know why we need this
> in qemu, and this cover letter doesn’t provide a justification.  I mean,
> it isn’t too complex (from the diffstat), but wouldn’t it be better to
> just have a script for this?
> 
> I suppose one reason to put it in qemu/qemu-img is that a script
> wouldn’t be packaged by distributions.  So if a user has a corrupted
> image, with this series we could tell them to run qemu-img check -M and
> put the output somewhere.  With a script, we’d first have to tell them
> to download the script.  But then again, downloading a script (that
> should be part of the qemu repository) doesn’t seem too much trouble to
> me either.
> 
> So I’m curious as to whether you had a specific reason in mind when you
> decided to implement this as part of qemu itself?
> 
> (I suppose the additional complexity is fully limited to the check
> infrastructure, so it wouldn’t interfere with the rest of the qcow2
> driver.  Hm.  Fair enough.)
> 

Just not to parse qcow2 from scratch. Qemu already can read qcow2, and
it looks through all its structures during check, why not
to add an ability to represent these structures as a qobject?


-- 
Best regards,
Vladimir

Re: [PATCH v3 0/3] Dump QCOW2 metadata

Posted by Max Reitz 23 weeks ago
On 20.02.20 13:10, Vladimir Sementsov-Ogievskiy wrote:
> 20.02.2020 14:58, Max Reitz wrote:
>> On 14.01.20 09:22, Andrey Shinkevich wrote:
>>> The information about QCOW2 metadata allocations in an image ELF-file is
>>> helpful for finding issues with the image data integrity.
>>
>> Sorry that I’m replying only so late – but I don’t know why we need this
>> in qemu, and this cover letter doesn’t provide a justification.  I mean,
>> it isn’t too complex (from the diffstat), but wouldn’t it be better to
>> just have a script for this?
>>
>> I suppose one reason to put it in qemu/qemu-img is that a script
>> wouldn’t be packaged by distributions.  So if a user has a corrupted
>> image, with this series we could tell them to run qemu-img check -M and
>> put the output somewhere.  With a script, we’d first have to tell them
>> to download the script.  But then again, downloading a script (that
>> should be part of the qemu repository) doesn’t seem too much trouble to
>> me either.
>>
>> So I’m curious as to whether you had a specific reason in mind when you
>> decided to implement this as part of qemu itself?
>>
>> (I suppose the additional complexity is fully limited to the check
>> infrastructure, so it wouldn’t interfere with the rest of the qcow2
>> driver.  Hm.  Fair enough.)
>>
> 
> Just not to parse qcow2 from scratch. Qemu already can read qcow2, and
> it looks through all its structures during check, why not
> to add an ability to represent these structures as a qobject?

Because it’d be code in qemu (i.e., a liability) that users are pretty
much never going to use.

Max