[Qemu-devel] [PATCH v10 00/20] make dirty-bitmap byte-based

Eric Blake posted 20 patches 6 years, 6 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20170925145526.32690-1-eblake@redhat.com
Test checkpatch passed
Test docker passed
Test s390x passed
include/block/block_int.h    |   2 +-
include/block/dirty-bitmap.h |  43 ++++++--------
include/qemu/hbitmap.h       |   8 +--
block/io.c                   |   6 +-
block.c                      |  18 ++++--
block/backup.c               |   7 +--
block/dirty-bitmap.c         | 134 ++++++++++++++-----------------------------
block/mirror.c               |  79 +++++++++++--------------
block/qcow2-bitmap.c         |  62 +++++++++++---------
block/qcow2.c                |  22 ++++---
migration/block.c            |  12 ++--
tests/test-hbitmap.c         |  10 ++--
util/hbitmap.c               |   8 +--
tests/qemu-iotests/165       |   2 +-
14 files changed, 174 insertions(+), 239 deletions(-)
[Qemu-devel] [PATCH v10 00/20] make dirty-bitmap byte-based
Posted by Eric Blake 6 years, 6 months ago
There are patches floating around to add NBD_CMD_BLOCK_STATUS,
but NBD wants to report status on byte granularity (even if the
reporting will probably be naturally aligned to sectors or even
much higher levels).  I've therefore started the task of
converting our block status code to report at a byte granularity
rather than sectors.

Now that 2.11 is open, I'm rebasing/reposting the remaining patches.

The overall conversion currently looks like:
part 1: bdrv_is_allocated (merged in 2.10, commit 51b0a488)
part 2: dirty-bitmap (this series, v9 was here [1])
part 3: bdrv_get_block_status (v4 is posted [2] and is mostly reviewed)
part 4: .bdrv_co_block_status (v3 is posted [3], but needs review)

Available as a tag at:
git fetch git://repo.or.cz/qemu/ericb.git nbd-byte-dirty-v10

[1] https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg05387.html
[2] https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg03543.html
[3] https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg03812.html

Since v9:
- another try at patch 5 [John]
- add R-b where appropriate

001/20:[----] [--] 'block: Make bdrv_img_create() size selection easier to read'
002/20:[----] [--] 'hbitmap: Rename serialization_granularity to serialization_align'
003/20:[----] [--] 'qcow2: Ensure bitmap serialization is aligned'
004/20:[----] [--] 'dirty-bitmap: Drop unused functions'
005/20:[0003] [FC] 'dirty-bitmap: Avoid size query failure during truncate'
006/20:[----] [--] 'dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes'
007/20:[----] [--] 'dirty-bitmap: Track bitmap size by bytes'
008/20:[----] [--] 'dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes'
009/20:[----] [--] 'qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based'
010/20:[----] [--] 'dirty-bitmap: Set iterator start by offset, not sector'
011/20:[----] [--] 'dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset'
012/20:[----] [--] 'dirty-bitmap: Change bdrv_get_dirty_count() to report bytes'
013/20:[----] [--] 'dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes'
014/20:[----] [--] 'dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes'
015/20:[----] [--] 'mirror: Switch mirror_dirty_init() to byte-based iteration'
016/20:[----] [--] 'qcow2: Switch qcow2_measure() to byte-based iteration'
017/20:[----] [--] 'qcow2: Switch load_bitmap_data() to byte-based iteration'
018/20:[----] [--] 'qcow2: Switch store_bitmap_data() to byte-based iteration'
019/20:[----] [--] 'dirty-bitmap: Switch bdrv_set_dirty() to bytes'
020/20:[----] [--] 'dirty-bitmap: Convert internal hbitmap size/granularity'

Eric Blake (20):
  block: Make bdrv_img_create() size selection easier to read
  hbitmap: Rename serialization_granularity to serialization_align
  qcow2: Ensure bitmap serialization is aligned
  dirty-bitmap: Drop unused functions
  dirty-bitmap: Avoid size query failure during truncate
  dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes
  dirty-bitmap: Track bitmap size by bytes
  dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes
  qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based
  dirty-bitmap: Set iterator start by offset, not sector
  dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset
  dirty-bitmap: Change bdrv_get_dirty_count() to report bytes
  dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes
  dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes
  mirror: Switch mirror_dirty_init() to byte-based iteration
  qcow2: Switch qcow2_measure() to byte-based iteration
  qcow2: Switch load_bitmap_data() to byte-based iteration
  qcow2: Switch store_bitmap_data() to byte-based iteration
  dirty-bitmap: Switch bdrv_set_dirty() to bytes
  dirty-bitmap: Convert internal hbitmap size/granularity

 include/block/block_int.h    |   2 +-
 include/block/dirty-bitmap.h |  43 ++++++--------
 include/qemu/hbitmap.h       |   8 +--
 block/io.c                   |   6 +-
 block.c                      |  18 ++++--
 block/backup.c               |   7 +--
 block/dirty-bitmap.c         | 134 ++++++++++++++-----------------------------
 block/mirror.c               |  79 +++++++++++--------------
 block/qcow2-bitmap.c         |  62 +++++++++++---------
 block/qcow2.c                |  22 ++++---
 migration/block.c            |  12 ++--
 tests/test-hbitmap.c         |  10 ++--
 util/hbitmap.c               |   8 +--
 tests/qemu-iotests/165       |   2 +-
 14 files changed, 174 insertions(+), 239 deletions(-)

-- 
2.13.5


Re: [Qemu-devel] [PATCH v10 00/20] make dirty-bitmap byte-based
Posted by John Snow 6 years, 6 months ago

On 09/25/2017 10:55 AM, Eric Blake wrote:
> There are patches floating around to add NBD_CMD_BLOCK_STATUS,
> but NBD wants to report status on byte granularity (even if the
> reporting will probably be naturally aligned to sectors or even
> much higher levels).  I've therefore started the task of
> converting our block status code to report at a byte granularity
> rather than sectors.
> 
> Now that 2.11 is open, I'm rebasing/reposting the remaining patches.
> 
> The overall conversion currently looks like:
> part 1: bdrv_is_allocated (merged in 2.10, commit 51b0a488)
> part 2: dirty-bitmap (this series, v9 was here [1])
> part 3: bdrv_get_block_status (v4 is posted [2] and is mostly reviewed)
> part 4: .bdrv_co_block_status (v3 is posted [3], but needs review)
> 
> Available as a tag at:
> git fetch git://repo.or.cz/qemu/ericb.git nbd-byte-dirty-v10
> 
> [1] https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg05387.html
> [2] https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg03543.html
> [3] https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg03812.html
> 
> Since v9:
> - another try at patch 5 [John]
> - add R-b where appropriate
> 
> 001/20:[----] [--] 'block: Make bdrv_img_create() size selection easier to read'
> 002/20:[----] [--] 'hbitmap: Rename serialization_granularity to serialization_align'
> 003/20:[----] [--] 'qcow2: Ensure bitmap serialization is aligned'
> 004/20:[----] [--] 'dirty-bitmap: Drop unused functions'
> 005/20:[0003] [FC] 'dirty-bitmap: Avoid size query failure during truncate'
> 006/20:[----] [--] 'dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes'
> 007/20:[----] [--] 'dirty-bitmap: Track bitmap size by bytes'
> 008/20:[----] [--] 'dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes'
> 009/20:[----] [--] 'qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based'
> 010/20:[----] [--] 'dirty-bitmap: Set iterator start by offset, not sector'
> 011/20:[----] [--] 'dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset'
> 012/20:[----] [--] 'dirty-bitmap: Change bdrv_get_dirty_count() to report bytes'
> 013/20:[----] [--] 'dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes'
> 014/20:[----] [--] 'dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes'
> 015/20:[----] [--] 'mirror: Switch mirror_dirty_init() to byte-based iteration'
> 016/20:[----] [--] 'qcow2: Switch qcow2_measure() to byte-based iteration'
> 017/20:[----] [--] 'qcow2: Switch load_bitmap_data() to byte-based iteration'
> 018/20:[----] [--] 'qcow2: Switch store_bitmap_data() to byte-based iteration'
> 019/20:[----] [--] 'dirty-bitmap: Switch bdrv_set_dirty() to bytes'
> 020/20:[----] [--] 'dirty-bitmap: Convert internal hbitmap size/granularity'
> 
> Eric Blake (20):
>   block: Make bdrv_img_create() size selection easier to read
>   hbitmap: Rename serialization_granularity to serialization_align
>   qcow2: Ensure bitmap serialization is aligned
>   dirty-bitmap: Drop unused functions
>   dirty-bitmap: Avoid size query failure during truncate
>   dirty-bitmap: Change bdrv_dirty_bitmap_size() to report bytes
>   dirty-bitmap: Track bitmap size by bytes
>   dirty-bitmap: Change bdrv_dirty_bitmap_*serialize*() to take bytes
>   qcow2: Switch sectors_covered_by_bitmap_cluster() to byte-based
>   dirty-bitmap: Set iterator start by offset, not sector
>   dirty-bitmap: Change bdrv_dirty_iter_next() to report byte offset
>   dirty-bitmap: Change bdrv_get_dirty_count() to report bytes
>   dirty-bitmap: Change bdrv_get_dirty_locked() to take bytes
>   dirty-bitmap: Change bdrv_[re]set_dirty_bitmap() to use bytes
>   mirror: Switch mirror_dirty_init() to byte-based iteration
>   qcow2: Switch qcow2_measure() to byte-based iteration
>   qcow2: Switch load_bitmap_data() to byte-based iteration
>   qcow2: Switch store_bitmap_data() to byte-based iteration
>   dirty-bitmap: Switch bdrv_set_dirty() to bytes
>   dirty-bitmap: Convert internal hbitmap size/granularity
> 
>  include/block/block_int.h    |   2 +-
>  include/block/dirty-bitmap.h |  43 ++++++--------
>  include/qemu/hbitmap.h       |   8 +--
>  block/io.c                   |   6 +-
>  block.c                      |  18 ++++--
>  block/backup.c               |   7 +--
>  block/dirty-bitmap.c         | 134 ++++++++++++++-----------------------------
>  block/mirror.c               |  79 +++++++++++--------------
>  block/qcow2-bitmap.c         |  62 +++++++++++---------
>  block/qcow2.c                |  22 ++++---
>  migration/block.c            |  12 ++--
>  tests/test-hbitmap.c         |  10 ++--
>  util/hbitmap.c               |   8 +--
>  tests/qemu-iotests/165       |   2 +-
>  14 files changed, 174 insertions(+), 239 deletions(-)
> 

Tested-by: John Snow <jsnow@redhat.com>

(Just in case I had to merge it, I ran tests.)

Re: [Qemu-devel] [PATCH v10 00/20] make dirty-bitmap byte-based
Posted by Kevin Wolf 6 years, 6 months ago
Am 25.09.2017 um 16:55 hat Eric Blake geschrieben:
> There are patches floating around to add NBD_CMD_BLOCK_STATUS,
> but NBD wants to report status on byte granularity (even if the
> reporting will probably be naturally aligned to sectors or even
> much higher levels).  I've therefore started the task of
> converting our block status code to report at a byte granularity
> rather than sectors.
> 
> Now that 2.11 is open, I'm rebasing/reposting the remaining patches.
> 
> The overall conversion currently looks like:
> part 1: bdrv_is_allocated (merged in 2.10, commit 51b0a488)
> part 2: dirty-bitmap (this series, v9 was here [1])
> part 3: bdrv_get_block_status (v4 is posted [2] and is mostly reviewed)
> part 4: .bdrv_co_block_status (v3 is posted [3], but needs review)
> 
> Available as a tag at:
> git fetch git://repo.or.cz/qemu/ericb.git nbd-byte-dirty-v10

Thanks, applied to the block branch.

Kevin