[PATCH v4 00/10] Further bitmaps improvements

Vladimir Sementsov-Ogievskiy posted 10 patches 4 years, 2 months ago
Test docker-quick@centos7 passed
Test FreeBSD passed
Test docker-mingw@fedora passed
Test checkpatch passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20200205112041.6003-1-vsementsov@virtuozzo.com
Maintainers: Kevin Wolf <kwolf@redhat.com>, John Snow <jsnow@redhat.com>, Eric Blake <eblake@redhat.com>, Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>, Max Reitz <mreitz@redhat.com>
include/block/dirty-bitmap.h |   9 +-
include/qemu/hbitmap.h       |  97 +++--------
block/dirty-bitmap.c         |  16 +-
block/qcow2-bitmap.c         |  15 +-
nbd/server.c                 | 251 ++++++++++++++--------------
tests/test-hbitmap.c         | 314 +++++++++++++----------------------
util/hbitmap.c               | 134 +++++++++------
7 files changed, 375 insertions(+), 461 deletions(-)
[PATCH v4 00/10] Further bitmaps improvements
Posted by Vladimir Sementsov-Ogievskiy 4 years, 2 months ago
Hi!

The main feature here is improvement of _next_dirty_area API, which I'm
going to use then for backup / block-copy.

Somehow, I thought that it was merged, but seems I even forgot to send
v4.

v4:
01-04: add Max's r-b
05: switch test_hbitmap_next_zero_check_range args to int64_t too
06: fix s/UINT64_MAX/INT64_MAX/ in comment to hbitmap_next_dirty
    s/firt_dirty_off/first_dirty_off/
    Context changed due to 05 change, but I keep Max's r-b
07: simplify parameter check in hbitmap_next_dirty_area
    drop initialization in hbitmap_sparse_merge
    add Max's r-b
08: commit message tweak
    refactor converted flag to separated converted_to_be and can_add
    do not convert to be automatically in nbd_extent_array_add
    check uint32 overflow in nbd_extent_array_add
10: drop extra check from store_bitmap_data, add Max's r-b

Vladimir Sementsov-Ogievskiy (10):
  hbitmap: assert that we don't create bitmap larger than INT64_MAX
  hbitmap: move hbitmap_iter_next_word to hbitmap.c
  hbitmap: unpublish hbitmap_iter_skip_words
  hbitmap: drop meta bitmaps as they are unused
  block/dirty-bitmap: switch _next_dirty_area and _next_zero to int64_t
  block/dirty-bitmap: add _next_dirty API
  block/dirty-bitmap: improve _next_dirty_area API
  nbd/server: introduce NBDExtentArray
  nbd/server: use bdrv_dirty_bitmap_next_dirty_area
  block/qcow2-bitmap: use bdrv_dirty_bitmap_next_dirty

 include/block/dirty-bitmap.h |   9 +-
 include/qemu/hbitmap.h       |  97 +++--------
 block/dirty-bitmap.c         |  16 +-
 block/qcow2-bitmap.c         |  15 +-
 nbd/server.c                 | 251 ++++++++++++++--------------
 tests/test-hbitmap.c         | 314 +++++++++++++----------------------
 util/hbitmap.c               | 134 +++++++++------
 7 files changed, 375 insertions(+), 461 deletions(-)

-- 
2.21.0


Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by Vladimir Sementsov-Ogievskiy 4 years, 2 months ago
gently ping.. almost all patches has r-b marks (except for 5 and 8)

05.02.2020 14:20, Vladimir Sementsov-Ogievskiy wrote:
> Hi!
> 
> The main feature here is improvement of _next_dirty_area API, which I'm
> going to use then for backup / block-copy.
> 
> Somehow, I thought that it was merged, but seems I even forgot to send
> v4.
> 
> v4:
> 01-04: add Max's r-b
> 05: switch test_hbitmap_next_zero_check_range args to int64_t too
> 06: fix s/UINT64_MAX/INT64_MAX/ in comment to hbitmap_next_dirty
>      s/firt_dirty_off/first_dirty_off/
>      Context changed due to 05 change, but I keep Max's r-b
> 07: simplify parameter check in hbitmap_next_dirty_area
>      drop initialization in hbitmap_sparse_merge
>      add Max's r-b
> 08: commit message tweak
>      refactor converted flag to separated converted_to_be and can_add
>      do not convert to be automatically in nbd_extent_array_add
>      check uint32 overflow in nbd_extent_array_add
> 10: drop extra check from store_bitmap_data, add Max's r-b
> 
> Vladimir Sementsov-Ogievskiy (10):
>    hbitmap: assert that we don't create bitmap larger than INT64_MAX
>    hbitmap: move hbitmap_iter_next_word to hbitmap.c
>    hbitmap: unpublish hbitmap_iter_skip_words
>    hbitmap: drop meta bitmaps as they are unused
>    block/dirty-bitmap: switch _next_dirty_area and _next_zero to int64_t
>    block/dirty-bitmap: add _next_dirty API
>    block/dirty-bitmap: improve _next_dirty_area API
>    nbd/server: introduce NBDExtentArray
>    nbd/server: use bdrv_dirty_bitmap_next_dirty_area
>    block/qcow2-bitmap: use bdrv_dirty_bitmap_next_dirty
> 
>   include/block/dirty-bitmap.h |   9 +-
>   include/qemu/hbitmap.h       |  97 +++--------
>   block/dirty-bitmap.c         |  16 +-
>   block/qcow2-bitmap.c         |  15 +-
>   nbd/server.c                 | 251 ++++++++++++++--------------
>   tests/test-hbitmap.c         | 314 +++++++++++++----------------------
>   util/hbitmap.c               | 134 +++++++++------
>   7 files changed, 375 insertions(+), 461 deletions(-)
> 


-- 
Best regards,
Vladimir

Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by Max Reitz 4 years, 2 months ago
On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote:
> Hi!
> 
> The main feature here is improvement of _next_dirty_area API, which I'm
> going to use then for backup / block-copy.
> 
> Somehow, I thought that it was merged, but seems I even forgot to send
> v4.

The changes from v3 look good to me, but I’d prefer a review from Eric
on patch 8.

Max

Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by Vladimir Sementsov-Ogievskiy 4 years, 1 month ago
26.02.2020 16:13, Max Reitz wrote:
> On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote:
>> Hi!
>>
>> The main feature here is improvement of _next_dirty_area API, which I'm
>> going to use then for backup / block-copy.
>>
>> Somehow, I thought that it was merged, but seems I even forgot to send
>> v4.
> 
> The changes from v3 look good to me, but I’d prefer a review from Eric
> on patch 8.
> 

Hi!

Could you take it now, or do you prefer me to resend?


-- 
Best regards,
Vladimir

Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by Max Reitz 4 years, 1 month ago
On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote:
> 26.02.2020 16:13, Max Reitz wrote:
>> On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote:
>>> Hi!
>>>
>>> The main feature here is improvement of _next_dirty_area API, which I'm
>>> going to use then for backup / block-copy.
>>>
>>> Somehow, I thought that it was merged, but seems I even forgot to send
>>> v4.
>>
>> The changes from v3 look good to me, but I’d prefer a review from Eric
>> on patch 8.
>>
> 
> Hi!
> 
> Could you take it now, or do you prefer me to resend?

I understand that you agreed to drop the comment above
bd_extent_array_convert_to_be(), then do the
“s/further call/so further calls/” replacement, and finally replace the
whole four lines Eric has quoted by “(this ensures that after a failure,
no further extents can accidentally change the bounds of the last extent
in the array)”?

Max

Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by Vladimir Sementsov-Ogievskiy 4 years, 1 month ago
10.03.2020 20:17, Max Reitz wrote:
> On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote:
>> 26.02.2020 16:13, Max Reitz wrote:
>>> On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote:
>>>> Hi!
>>>>
>>>> The main feature here is improvement of _next_dirty_area API, which I'm
>>>> going to use then for backup / block-copy.
>>>>
>>>> Somehow, I thought that it was merged, but seems I even forgot to send
>>>> v4.
>>>
>>> The changes from v3 look good to me, but I’d prefer a review from Eric
>>> on patch 8.
>>>
>>
>> Hi!
>>
>> Could you take it now, or do you prefer me to resend?
> 
> I understand that you agreed to drop the comment above
> bd_extent_array_convert_to_be(), then do the
> “s/further call/so further calls/” replacement, and finally replace the
> whole four lines Eric has quoted by “(this ensures that after a failure,
> no further extents can accidentally change the bounds of the last extent
> in the array)”?
> 

Yes, all true.


-- 
Best regards,
Vladimir

Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by Max Reitz 4 years, 1 month ago
On 11.03.20 07:17, Vladimir Sementsov-Ogievskiy wrote:
> 10.03.2020 20:17, Max Reitz wrote:
>> On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote:
>>> 26.02.2020 16:13, Max Reitz wrote:
>>>> On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote:
>>>>> Hi!
>>>>>
>>>>> The main feature here is improvement of _next_dirty_area API, which
>>>>> I'm
>>>>> going to use then for backup / block-copy.
>>>>>
>>>>> Somehow, I thought that it was merged, but seems I even forgot to send
>>>>> v4.
>>>>
>>>> The changes from v3 look good to me, but I’d prefer a review from Eric
>>>> on patch 8.
>>>>
>>>
>>> Hi!
>>>
>>> Could you take it now, or do you prefer me to resend?
>>
>> I understand that you agreed to drop the comment above
>> bd_extent_array_convert_to_be(), then do the
>> “s/further call/so further calls/” replacement, and finally replace the
>> whole four lines Eric has quoted by “(this ensures that after a failure,
>> no further extents can accidentally change the bounds of the last extent
>> in the array)”?
>>
> 
> Yes, all true.

Hm, I could take it then, but on second thought, John is the maintainer
for 8/10 patches, and Eric is for the other two...  So I’m not sure
whether I’m even the right person to do so.

Max

Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by Vladimir Sementsov-Ogievskiy 4 years, 1 month ago
11.03.2020 12:55, Max Reitz wrote:
> On 11.03.20 07:17, Vladimir Sementsov-Ogievskiy wrote:
>> 10.03.2020 20:17, Max Reitz wrote:
>>> On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote:
>>>> 26.02.2020 16:13, Max Reitz wrote:
>>>>> On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote:
>>>>>> Hi!
>>>>>>
>>>>>> The main feature here is improvement of _next_dirty_area API, which
>>>>>> I'm
>>>>>> going to use then for backup / block-copy.
>>>>>>
>>>>>> Somehow, I thought that it was merged, but seems I even forgot to send
>>>>>> v4.
>>>>>
>>>>> The changes from v3 look good to me, but I’d prefer a review from Eric
>>>>> on patch 8.
>>>>>
>>>>
>>>> Hi!
>>>>
>>>> Could you take it now, or do you prefer me to resend?jjjjj
>>>
>>> I understand that you agreed to drop the comment above
>>> bd_extent_array_convert_to_be(), then do the
>>> “s/further call/so further calls/” replacement, and finally replace the
>>> whole four lines Eric has quoted by “(this ensures that after a failure,
>>> no further extents can accidentally change the bounds of the last extent
>>> in the array)”?
>>>
>>
>> Yes, all true.
> 
> Hm, I could take it then, but on second thought, John is the maintainer
> for 8/10 patches, and Eric is for the other two...  So I’m not sure
> whether I’m even the right person to do so.
> 

Hmm, true. Let's wait for John?


-- 
Best regards,
Vladimir

Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by John Snow 4 years, 1 month ago

On 3/11/20 9:58 AM, Vladimir Sementsov-Ogievskiy wrote:
> 11.03.2020 12:55, Max Reitz wrote:
>> On 11.03.20 07:17, Vladimir Sementsov-Ogievskiy wrote:
>>> 10.03.2020 20:17, Max Reitz wrote:
>>>> On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote:
>>>>> 26.02.2020 16:13, Max Reitz wrote:
>>>>>> On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>> Hi!
>>>>>>>
>>>>>>> The main feature here is improvement of _next_dirty_area API, which
>>>>>>> I'm
>>>>>>> going to use then for backup / block-copy.
>>>>>>>
>>>>>>> Somehow, I thought that it was merged, but seems I even forgot to
>>>>>>> send
>>>>>>> v4.
>>>>>>
>>>>>> The changes from v3 look good to me, but I’d prefer a review from
>>>>>> Eric
>>>>>> on patch 8.
>>>>>>
>>>>>
>>>>> Hi!
>>>>>
>>>>> Could you take it now, or do you prefer me to resend?jjjjj
>>>>
>>>> I understand that you agreed to drop the comment above
>>>> bd_extent_array_convert_to_be(), then do the
>>>> “s/further call/so further calls/” replacement, and finally replace the
>>>> whole four lines Eric has quoted by “(this ensures that after a
>>>> failure,
>>>> no further extents can accidentally change the bounds of the last
>>>> extent
>>>> in the array)”?
>>>>
>>>
>>> Yes, all true.
>>
>> Hm, I could take it then, but on second thought, John is the maintainer
>> for 8/10 patches, and Eric is for the other two...  So I’m not sure
>> whether I’m even the right person to do so.
>>
> 
> Hmm, true. Let's wait for John?
> 
> 

I am *VERY* behind on my email, and this patch series is sitting in my
to-review folder. However, if it's ready to go and reviewed, I'm willing
to merge it, test it, and give it a quick look-over and get you on your way.

--js


Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by Vladimir Sementsov-Ogievskiy 4 years, 1 month ago
11.03.2020 20:03, John Snow wrote:
> 
> 
> On 3/11/20 9:58 AM, Vladimir Sementsov-Ogievskiy wrote:
>> 11.03.2020 12:55, Max Reitz wrote:
>>> On 11.03.20 07:17, Vladimir Sementsov-Ogievskiy wrote:
>>>> 10.03.2020 20:17, Max Reitz wrote:
>>>>> On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote:
>>>>>> 26.02.2020 16:13, Max Reitz wrote:
>>>>>>> On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> The main feature here is improvement of _next_dirty_area API, which
>>>>>>>> I'm
>>>>>>>> going to use then for backup / block-copy.
>>>>>>>>
>>>>>>>> Somehow, I thought that it was merged, but seems I even forgot to
>>>>>>>> send
>>>>>>>> v4.
>>>>>>>
>>>>>>> The changes from v3 look good to me, but I’d prefer a review from
>>>>>>> Eric
>>>>>>> on patch 8.
>>>>>>>
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> Could you take it now, or do you prefer me to resend?jjjjj
>>>>>
>>>>> I understand that you agreed to drop the comment above
>>>>> bd_extent_array_convert_to_be(), then do the
>>>>> “s/further call/so further calls/” replacement, and finally replace the
>>>>> whole four lines Eric has quoted by “(this ensures that after a
>>>>> failure,
>>>>> no further extents can accidentally change the bounds of the last
>>>>> extent
>>>>> in the array)”?
>>>>>
>>>>
>>>> Yes, all true.
>>>
>>> Hm, I could take it then, but on second thought, John is the maintainer
>>> for 8/10 patches, and Eric is for the other two...  So I’m not sure
>>> whether I’m even the right person to do so.
>>>
>>
>> Hmm, true. Let's wait for John?
>>
>>
> 
> I am *VERY* behind on my email, and this patch series is sitting in my
> to-review folder. However, if it's ready to go and reviewed, I'm willing
> to merge it, test it, and give it a quick look-over and get you on your way.
> 

It would be great, if it is convenient for you. Thanks!
All patches are reviewed now by Max or Eric, so, I'd be very glad if this get in 5.0.



-- 
Best regards,
Vladimir

Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by John Snow 4 years, 1 month ago

On 3/12/20 1:59 AM, Vladimir Sementsov-Ogievskiy wrote:
> 
> It would be great, if it is convenient for you. Thanks!
> All patches are reviewed now by Max or Eric, so, I'd be very glad if
> this get in 5.0.

Testing and staging right now.

--js


Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by John Snow 4 years, 1 month ago

On 3/12/20 1:59 AM, Vladimir Sementsov-Ogievskiy wrote:
> 11.03.2020 20:03, John Snow wrote:
>>
>>
>> On 3/11/20 9:58 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> 11.03.2020 12:55, Max Reitz wrote:
>>>> On 11.03.20 07:17, Vladimir Sementsov-Ogievskiy wrote:
>>>>> 10.03.2020 20:17, Max Reitz wrote:
>>>>>> On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>> 26.02.2020 16:13, Max Reitz wrote:
>>>>>>>> On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> The main feature here is improvement of _next_dirty_area API,
>>>>>>>>> which
>>>>>>>>> I'm
>>>>>>>>> going to use then for backup / block-copy.
>>>>>>>>>
>>>>>>>>> Somehow, I thought that it was merged, but seems I even forgot to
>>>>>>>>> send
>>>>>>>>> v4.
>>>>>>>>
>>>>>>>> The changes from v3 look good to me, but I’d prefer a review from
>>>>>>>> Eric
>>>>>>>> on patch 8.
>>>>>>>>
>>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> Could you take it now, or do you prefer me to resend?jjjjj
>>>>>>
>>>>>> I understand that you agreed to drop the comment above
>>>>>> bd_extent_array_convert_to_be(), then do the
>>>>>> “s/further call/so further calls/” replacement, and finally
>>>>>> replace the
>>>>>> whole four lines Eric has quoted by “(this ensures that after a
>>>>>> failure,
>>>>>> no further extents can accidentally change the bounds of the last
>>>>>> extent
>>>>>> in the array)”?
>>>>>>
>>>>>
>>>>> Yes, all true.
>>>>
>>>> Hm, I could take it then, but on second thought, John is the maintainer
>>>> for 8/10 patches, and Eric is for the other two...  So I’m not sure
>>>> whether I’m even the right person to do so.
>>>>
>>>
>>> Hmm, true. Let's wait for John?
>>>
>>>
>>
>> I am *VERY* behind on my email, and this patch series is sitting in my
>> to-review folder. However, if it's ready to go and reviewed, I'm willing
>> to merge it, test it, and give it a quick look-over and get you on
>> your way.
>>
> 
> It would be great, if it is convenient for you. Thanks!
> All patches are reviewed now by Max or Eric, so, I'd be very glad if
> this get in 5.0.
> 
> 
> 

Thanks, applied to my bitmaps tree:

https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git

--js


Re: [PATCH v4 00/10] Further bitmaps improvements
Posted by Vladimir Sementsov-Ogievskiy 4 years, 1 month ago
12.03.2020 23:41, John Snow wrote:
> 
> 
> On 3/12/20 1:59 AM, Vladimir Sementsov-Ogievskiy wrote:
>> 11.03.2020 20:03, John Snow wrote:
>>>
>>>
>>> On 3/11/20 9:58 AM, Vladimir Sementsov-Ogievskiy wrote:
>>>> 11.03.2020 12:55, Max Reitz wrote:
>>>>> On 11.03.20 07:17, Vladimir Sementsov-Ogievskiy wrote:
>>>>>> 10.03.2020 20:17, Max Reitz wrote:
>>>>>>> On 06.03.20 08:45, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>>> 26.02.2020 16:13, Max Reitz wrote:
>>>>>>>>> On 05.02.20 12:20, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>>>>> Hi!
>>>>>>>>>>
>>>>>>>>>> The main feature here is improvement of _next_dirty_area API,
>>>>>>>>>> which
>>>>>>>>>> I'm
>>>>>>>>>> going to use then for backup / block-copy.
>>>>>>>>>>
>>>>>>>>>> Somehow, I thought that it was merged, but seems I even forgot to
>>>>>>>>>> send
>>>>>>>>>> v4.
>>>>>>>>>
>>>>>>>>> The changes from v3 look good to me, but I’d prefer a review from
>>>>>>>>> Eric
>>>>>>>>> on patch 8.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> Could you take it now, or do you prefer me to resend?jjjjj
>>>>>>>
>>>>>>> I understand that you agreed to drop the comment above
>>>>>>> bd_extent_array_convert_to_be(), then do the
>>>>>>> “s/further call/so further calls/” replacement, and finally
>>>>>>> replace the
>>>>>>> whole four lines Eric has quoted by “(this ensures that after a
>>>>>>> failure,
>>>>>>> no further extents can accidentally change the bounds of the last
>>>>>>> extent
>>>>>>> in the array)”?
>>>>>>>
>>>>>>
>>>>>> Yes, all true.
>>>>>
>>>>> Hm, I could take it then, but on second thought, John is the maintainer
>>>>> for 8/10 patches, and Eric is for the other two...  So I’m not sure
>>>>> whether I’m even the right person to do so.
>>>>>
>>>>
>>>> Hmm, true. Let's wait for John?
>>>>
>>>>
>>>
>>> I am *VERY* behind on my email, and this patch series is sitting in my
>>> to-review folder. However, if it's ready to go and reviewed, I'm willing
>>> to merge it, test it, and give it a quick look-over and get you on
>>> your way.
>>>
>>
>> It would be great, if it is convenient for you. Thanks!
>> All patches are reviewed now by Max or Eric, so, I'd be very glad if
>> this get in 5.0.
>>
>>
>>
> 
> Thanks, applied to my bitmaps tree:
> 
> https://github.com/jnsnow/qemu/commits/bitmaps
> https://github.com/jnsnow/qemu.git
> 

Thank you!


-- 
Best regards,
Vladimir