[PATCH v5 0/9] qcow2-bitmaps: rewrite reopening logic

Vladimir Sementsov-Ogievskiy posted 9 patches 4 years, 7 months ago
Test checkpatch passed
Test docker-mingw@fedora failed
Test docker-quick@centos7 failed
Test docker-clang@ubuntu failed
Test asan failed
Test FreeBSD failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20190927122355.7344-1-vsementsov@virtuozzo.com
Maintainers: John Snow <jsnow@redhat.com>, Kevin Wolf <kwolf@redhat.com>, Max Reitz <mreitz@redhat.com>, Fam Zheng <fam@euphon.net>
block/qcow2.h                |   5 +-
include/block/block.h        |   2 +-
include/block/block_int.h    |   6 --
include/block/dirty-bitmap.h |   1 -
block.c                      |  53 +++++------
block/dirty-bitmap.c         |  12 ---
block/qcow2-bitmap.c         | 164 ++++++++++++++++++++++-------------
block/qcow2.c                |  17 +++-
tests/qemu-iotests/165       |  57 +++++++++++-
tests/qemu-iotests/165.out   |   4 +-
tests/qemu-iotests/260       |  89 +++++++++++++++++++
tests/qemu-iotests/260.out   |  52 +++++++++++
tests/qemu-iotests/group     |   1 +
13 files changed, 343 insertions(+), 120 deletions(-)
create mode 100755 tests/qemu-iotests/260
create mode 100644 tests/qemu-iotests/260.out
[PATCH v5 0/9] qcow2-bitmaps: rewrite reopening logic
Posted by Vladimir Sementsov-Ogievskiy 4 years, 7 months ago
Hi all!

Bitmaps reopening is buggy, reopening-rw just not working at all and
reopening-ro may lead to producing broken incremental
backup if we do temporary snapshot in a meantime.

v5:
01: - add Max's r-b
    - fix s/QSIMPLE_INIT/QTAILQ_INIT/ in a comment
02: - add Max's and John's a-b
03: - improve test to check bitmap hashes in more safe way
07: - drop wrong statement from commit message
    - log events by hand
08: - drop 'the' from comment
    - add Corruption in case of existent IN_USE on RO->RW reopen
09: - add Max's a-b and John's r-b

v4: Drop complicated solution around reopening logic [Kevin], fix
    the existing bug in a simplest way

Vladimir Sementsov-Ogievskiy (9):
  block: switch reopen queue from QSIMPLEQ to QTAILQ
  block: reverse order for reopen commits
  iotests: add test-case to 165 to test reopening qcow2 bitmaps to RW
  block/qcow2-bitmap: get rid of bdrv_has_changed_persistent_bitmaps
  block/qcow2-bitmap: drop qcow2_reopen_bitmaps_rw_hint()
  block/qcow2-bitmap: do not remove bitmaps on reopen-ro
  iotests: add test 260 to check bitmap life after snapshot + commit
  block/qcow2-bitmap: fix and improve qcow2_reopen_bitmaps_rw
  qcow2-bitmap: move bitmap reopen-rw code to qcow2_reopen_commit

 block/qcow2.h                |   5 +-
 include/block/block.h        |   2 +-
 include/block/block_int.h    |   6 --
 include/block/dirty-bitmap.h |   1 -
 block.c                      |  53 +++++------
 block/dirty-bitmap.c         |  12 ---
 block/qcow2-bitmap.c         | 164 ++++++++++++++++++++++-------------
 block/qcow2.c                |  17 +++-
 tests/qemu-iotests/165       |  57 +++++++++++-
 tests/qemu-iotests/165.out   |   4 +-
 tests/qemu-iotests/260       |  89 +++++++++++++++++++
 tests/qemu-iotests/260.out   |  52 +++++++++++
 tests/qemu-iotests/group     |   1 +
 13 files changed, 343 insertions(+), 120 deletions(-)
 create mode 100755 tests/qemu-iotests/260
 create mode 100644 tests/qemu-iotests/260.out

-- 
2.21.0


Re: [PATCH v5 0/9] qcow2-bitmaps: rewrite reopening logic
Posted by John Snow 4 years, 7 months ago

On 9/27/19 8:23 AM, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> Bitmaps reopening is buggy, reopening-rw just not working at all and
> reopening-ro may lead to producing broken incremental
> backup if we do temporary snapshot in a meantime.
> 
> v5:
> 01: - add Max's r-b
>     - fix s/QSIMPLE_INIT/QTAILQ_INIT/ in a comment
> 02: - add Max's and John's a-b
> 03: - improve test to check bitmap hashes in more safe way
> 07: - drop wrong statement from commit message
>     - log events by hand
> 08: - drop 'the' from comment
>     - add Corruption in case of existent IN_USE on RO->RW reopen
> 09: - add Max's a-b and John's r-b
> 
> v4: Drop complicated solution around reopening logic [Kevin], fix
>     the existing bug in a simplest way
> 
> Vladimir Sementsov-Ogievskiy (9):
>   block: switch reopen queue from QSIMPLEQ to QTAILQ
>   block: reverse order for reopen commits
>   iotests: add test-case to 165 to test reopening qcow2 bitmaps to RW
>   block/qcow2-bitmap: get rid of bdrv_has_changed_persistent_bitmaps
>   block/qcow2-bitmap: drop qcow2_reopen_bitmaps_rw_hint()
>   block/qcow2-bitmap: do not remove bitmaps on reopen-ro
>   iotests: add test 260 to check bitmap life after snapshot + commit
>   block/qcow2-bitmap: fix and improve qcow2_reopen_bitmaps_rw
>   qcow2-bitmap: move bitmap reopen-rw code to qcow2_reopen_commit
> 
>  block/qcow2.h                |   5 +-
>  include/block/block.h        |   2 +-
>  include/block/block_int.h    |   6 --
>  include/block/dirty-bitmap.h |   1 -
>  block.c                      |  53 +++++------
>  block/dirty-bitmap.c         |  12 ---
>  block/qcow2-bitmap.c         | 164 ++++++++++++++++++++++-------------
>  block/qcow2.c                |  17 +++-
>  tests/qemu-iotests/165       |  57 +++++++++++-
>  tests/qemu-iotests/165.out   |   4 +-
>  tests/qemu-iotests/260       |  89 +++++++++++++++++++
>  tests/qemu-iotests/260.out   |  52 +++++++++++
>  tests/qemu-iotests/group     |   1 +
>  13 files changed, 343 insertions(+), 120 deletions(-)
>  create mode 100755 tests/qemu-iotests/260
>  create mode 100644 tests/qemu-iotests/260.out
> 

With the understanding that there are caveats that make this unsafe, but
less unsafe and is a strict improvement:

Reviewed-by: John Snow <jsnow@redhat.com>

Re: [PATCH v5 0/9] qcow2-bitmaps: rewrite reopening logic
Posted by John Snow 4 years, 7 months ago

On 9/27/19 8:23 AM, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> Bitmaps reopening is buggy, reopening-rw just not working at all and
> reopening-ro may lead to producing broken incremental
> backup if we do temporary snapshot in a meantime.
> 
> v5:
> 01: - add Max's r-b
>     - fix s/QSIMPLE_INIT/QTAILQ_INIT/ in a comment
> 02: - add Max's and John's a-b
> 03: - improve test to check bitmap hashes in more safe way
> 07: - drop wrong statement from commit message
>     - log events by hand
> 08: - drop 'the' from comment
>     - add Corruption in case of existent IN_USE on RO->RW reopen
> 09: - add Max's a-b and John's r-b
> 
> v4: Drop complicated solution around reopening logic [Kevin], fix
>     the existing bug in a simplest way
> 
> Vladimir Sementsov-Ogievskiy (9):
>   block: switch reopen queue from QSIMPLEQ to QTAILQ
>   block: reverse order for reopen commits
>   iotests: add test-case to 165 to test reopening qcow2 bitmaps to RW
>   block/qcow2-bitmap: get rid of bdrv_has_changed_persistent_bitmaps
>   block/qcow2-bitmap: drop qcow2_reopen_bitmaps_rw_hint()
>   block/qcow2-bitmap: do not remove bitmaps on reopen-ro
>   iotests: add test 260 to check bitmap life after snapshot + commit
>   block/qcow2-bitmap: fix and improve qcow2_reopen_bitmaps_rw
>   qcow2-bitmap: move bitmap reopen-rw code to qcow2_reopen_commit
> 
>  block/qcow2.h                |   5 +-
>  include/block/block.h        |   2 +-
>  include/block/block_int.h    |   6 --
>  include/block/dirty-bitmap.h |   1 -
>  block.c                      |  53 +++++------
>  block/dirty-bitmap.c         |  12 ---
>  block/qcow2-bitmap.c         | 164 ++++++++++++++++++++++-------------
>  block/qcow2.c                |  17 +++-
>  tests/qemu-iotests/165       |  57 +++++++++++-
>  tests/qemu-iotests/165.out   |   4 +-
>  tests/qemu-iotests/260       |  89 +++++++++++++++++++
>  tests/qemu-iotests/260.out   |  52 +++++++++++
>  tests/qemu-iotests/group     |   1 +
>  13 files changed, 343 insertions(+), 120 deletions(-)
>  create mode 100755 tests/qemu-iotests/260
>  create mode 100644 tests/qemu-iotests/260.out
> 

Provisionally staged, pending feedback from Kevin.

Some minor rebase conflicts against the current bitmaps branch:

012/17:[----] [-C] 'block/qcow2-bitmap: get rid of
bdrv_has_changed_persistent_bitmaps'
014/17:[0004] [FC] 'block/qcow2-bitmap: do not remove bitmaps on reopen-ro'
017/17:[----] [-C] 'qcow2-bitmap: move bitmap reopen-rw code to
qcow2_reopen_commit'


12: just context against changed _first and _next prototypes.
14: the signature of bdrv_release_dirty_bitmap has changed, which incurs
a change in the refactor.
17: bdrv_can_store... has changed to bdrv_co_can_store...
    Same for bdrv_remove_persistent.


Thanks, applied to my bitmaps tree:

https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git

--js

Re: [PATCH v5 0/9] qcow2-bitmaps: rewrite reopening logic
Posted by Vladimir Sementsov-Ogievskiy 4 years, 7 months ago
28.09.2019 2:36, John Snow wrote:
> 
> 
> On 9/27/19 8:23 AM, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all!
>>
>> Bitmaps reopening is buggy, reopening-rw just not working at all and
>> reopening-ro may lead to producing broken incremental
>> backup if we do temporary snapshot in a meantime.
>>
>> v5:
>> 01: - add Max's r-b
>>      - fix s/QSIMPLE_INIT/QTAILQ_INIT/ in a comment
>> 02: - add Max's and John's a-b
>> 03: - improve test to check bitmap hashes in more safe way
>> 07: - drop wrong statement from commit message
>>      - log events by hand
>> 08: - drop 'the' from comment
>>      - add Corruption in case of existent IN_USE on RO->RW reopen
>> 09: - add Max's a-b and John's r-b
>>
>> v4: Drop complicated solution around reopening logic [Kevin], fix
>>      the existing bug in a simplest way
>>
>> Vladimir Sementsov-Ogievskiy (9):
>>    block: switch reopen queue from QSIMPLEQ to QTAILQ
>>    block: reverse order for reopen commits
>>    iotests: add test-case to 165 to test reopening qcow2 bitmaps to RW
>>    block/qcow2-bitmap: get rid of bdrv_has_changed_persistent_bitmaps
>>    block/qcow2-bitmap: drop qcow2_reopen_bitmaps_rw_hint()
>>    block/qcow2-bitmap: do not remove bitmaps on reopen-ro
>>    iotests: add test 260 to check bitmap life after snapshot + commit
>>    block/qcow2-bitmap: fix and improve qcow2_reopen_bitmaps_rw
>>    qcow2-bitmap: move bitmap reopen-rw code to qcow2_reopen_commit
>>
>>   block/qcow2.h                |   5 +-
>>   include/block/block.h        |   2 +-
>>   include/block/block_int.h    |   6 --
>>   include/block/dirty-bitmap.h |   1 -
>>   block.c                      |  53 +++++------
>>   block/dirty-bitmap.c         |  12 ---
>>   block/qcow2-bitmap.c         | 164 ++++++++++++++++++++++-------------
>>   block/qcow2.c                |  17 +++-
>>   tests/qemu-iotests/165       |  57 +++++++++++-
>>   tests/qemu-iotests/165.out   |   4 +-
>>   tests/qemu-iotests/260       |  89 +++++++++++++++++++
>>   tests/qemu-iotests/260.out   |  52 +++++++++++
>>   tests/qemu-iotests/group     |   1 +
>>   13 files changed, 343 insertions(+), 120 deletions(-)
>>   create mode 100755 tests/qemu-iotests/260
>>   create mode 100644 tests/qemu-iotests/260.out
>>
> 
> Provisionally staged, pending feedback from Kevin.
> 
> Some minor rebase conflicts against the current bitmaps branch:
> 
> 012/17:[----] [-C] 'block/qcow2-bitmap: get rid of
> bdrv_has_changed_persistent_bitmaps'
> 014/17:[0004] [FC] 'block/qcow2-bitmap: do not remove bitmaps on reopen-ro'
> 017/17:[----] [-C] 'qcow2-bitmap: move bitmap reopen-rw code to
> qcow2_reopen_commit'
> 
> 
> 12: just context against changed _first and _next prototypes.
> 14: the signature of bdrv_release_dirty_bitmap has changed, which incurs
> a change in the refactor.
> 17: bdrv_can_store... has changed to bdrv_co_can_store...
>      Same for bdrv_remove_persistent.
> 
> 
> Thanks, applied to my bitmaps tree:
> 
> https://github.com/jnsnow/qemu/commits/bitmaps
> https://github.com/jnsnow/qemu.git
> 


Thanks a lot!


-- 
Best regards,
Vladimir