Hi,

This is a new (very small) block job that writes a pattern into a
bitmap. The only pattern implemented is the top allocation information.

This can be used to "recover" an incremental bitmap chain if an external
snapshot was taken without creating a new bitmap first: any writes made
to the image will be reflected by the allocation status and can be
written back into a bitmap.

This is useful for e.g. libvirt managing backup chains if a user creates
an external snapshot outside of libvirt.

v2:
 - Addressed some, but not all feedback
 - Rebased on latest 'job-runner' series; but it's not clear if it
   should be kept.
 - This version doesn't address all of the feedback from v1,
   but I am posting it to the list as an RFC.

John Snow (5):
  block: add bitmap-populate job
  blockdev: combine DriveBackupState and BlockdevBackupState
  qmp: expose block-dirty-bitmap-populate
  iotests: move bitmap helpers into their own file
  iotests: add 287 for block-dirty-bitmap-populate

 qapi/block-core.json          |   66 +
 qapi/job.json                 |    2 +-
 qapi/transaction.json         |    2 +
 include/block/block_int.h     |   21 +
 block/bitmap-alloc.c          |  207 ++
 blockdev.c                    |  104 +-
 blockjob.c                    |    3 +-
 block/Makefile.objs           |    1 +
 tests/qemu-iotests/257        |  110 +-
 tests/qemu-iotests/287        |  242 ++
 tests/qemu-iotests/287.out    | 4544 +++++++++++++++++++++++++++++++++
 tests/qemu-iotests/bitmaps.py |  131 +
 tests/qemu-iotests/group      |    1 +
 13 files changed, 5305 insertions(+), 129 deletions(-)
 create mode 100644 block/bitmap-alloc.c
 create mode 100755 tests/qemu-iotests/287
 create mode 100644 tests/qemu-iotests/287.out
 create mode 100644 tests/qemu-iotests/bitmaps.py

-- 
2.21.1

On 5/13/20 10:49 PM, John Snow wrote:
> Hi,
> 
> This is a new (very small) block job that writes a pattern into a
> bitmap. The only pattern implemented is the top allocation information.
> 
> This can be used to "recover" an incremental bitmap chain if an external
> snapshot was taken without creating a new bitmap first: any writes made
> to the image will be reflected by the allocation status and can be
> written back into a bitmap.
> 
> This is useful for e.g. libvirt managing backup chains if a user creates
> an external snapshot outside of libvirt.
> 
> v2:
>   - Addressed some, but not all feedback
>   - Rebased on latest 'job-runner' series; but it's not clear if it
>     should be kept.

Message-id for that series? I'm not finding a message with a subject 
containing a literal 'job-runner', but am not sure which subject to look 
for instead.

I also couldn't find an obvious tag or branch at 
https://github.com/jnsnow/qemu/branches where you might have stashed 
this including prerequisites.

>   - This version doesn't address all of the feedback from v1,
>     but I am posting it to the list as an RFC.

I'm happy to try and take over these patches to prepare a v3, but only 
if I can get them to build by finding the prerequisites :)

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

On Wed, May 13, 2020 at 23:49:17 -0400, John Snow wrote:
> Hi,
> 
> This is a new (very small) block job that writes a pattern into a
> bitmap. The only pattern implemented is the top allocation information.
> 
> This can be used to "recover" an incremental bitmap chain if an external
> snapshot was taken without creating a new bitmap first: any writes made
> to the image will be reflected by the allocation status and can be
> written back into a bitmap.
> 
> This is useful for e.g. libvirt managing backup chains if a user creates
> an external snapshot outside of libvirt.

I've dusted-off my patches for using this blockjob for this very
specific case and it works for me.

Tested-by: Peter Krempa <pkrempa@redhat.com>

For now I'll continue the integration with other blockjobs where we
merge bitmaps.

On Mon, May 18, 2020 at 16:52:45 +0200, Peter Krempa wrote:
> On Wed, May 13, 2020 at 23:49:17 -0400, John Snow wrote:
> > Hi,
> > 
> > This is a new (very small) block job that writes a pattern into a
> > bitmap. The only pattern implemented is the top allocation information.
> > 
> > This can be used to "recover" an incremental bitmap chain if an external
> > snapshot was taken without creating a new bitmap first: any writes made
> > to the image will be reflected by the allocation status and can be
> > written back into a bitmap.
> > 
> > This is useful for e.g. libvirt managing backup chains if a user creates
> > an external snapshot outside of libvirt.
> 
> I've dusted-off my patches for using this blockjob for this very
> specific case and it works for me.
> 
> Tested-by: Peter Krempa <pkrempa@redhat.com>
> 
> For now I'll continue the integration with other blockjobs where we
> merge bitmaps.

I've posted the libvirt patches which make use of this blockjob as RFC
on the libvir-list:

https://www.redhat.com/archives/libvir-list/2020-June/msg00292.html

I also have a branch with the patchset rebased to master (except for one
of the test commits IIRC) linked from the cover-letter.