[Qemu-devel] [PATCH v4 00/10] qcow2-bitmaps: rewrite reopening logic

Vladimir Sementsov-Ogievskiy posted 10 patches 4 years, 9 months ago
Test asan passed
Test FreeBSD passed
Test checkpatch passed
Test docker-mingw@fedora passed
Test docker-clang@ubuntu passed
Test s390x passed
Failed in applying to current master (apply log)
There is a newer version of this series
block/qcow2.h                 |   5 +-
include/block/block.h         |   2 +-
include/block/block_int.h     |   6 --
include/block/dirty-bitmap.h  |   1 -
block.c                       |  51 +++++-------
block/dirty-bitmap.c          |  12 ---
block/qcow2-bitmap.c          | 143 ++++++++++++++++++++--------------
block/qcow2.c                 |  17 +++-
tests/qemu-iotests/165        |  46 ++++++++++-
tests/qemu-iotests/165.out    |   4 +-
tests/qemu-iotests/260        |  87 +++++++++++++++++++++
tests/qemu-iotests/260.out    |  52 +++++++++++++
tests/qemu-iotests/group      |   1 +
tests/qemu-iotests/iotests.py |  10 +++
14 files changed, 318 insertions(+), 119 deletions(-)
create mode 100755 tests/qemu-iotests/260
create mode 100644 tests/qemu-iotests/260.out
[Qemu-devel] [PATCH v4 00/10] qcow2-bitmaps: rewrite reopening logic
Posted by Vladimir Sementsov-Ogievskiy 4 years, 9 months ago
Hi all!

Bitmaps reopening is buggy, reopening-rw just not working at all and
reopening-ro may lead to producing broken incremental
backup if we do temporary snapshot in a meantime.

v4: Drop complicated solution around reopening logic [Kevin], fix
    the existing bug in a simplest way

Structure:

02: fix reopen to RW
03: test reopen to RW

07: fix reopen to RO
08: test reopen to RO

Others are less significant improvements and refactoring

Changelog:

01-03: new patches, to fix reopening bitmaps to RW and personal test for
       this bug
08: merged test from v3, it covers both bugs (reopen to RW and reopen to RO)
10: instead of moving bitmap-reopening to prepare(as in 09 in v3) we now keep it
    in commit, but in right place
others: unchanged

v3:
02: John's events_wait already merged in, so my 02 from v2 is not needed.
    Instead, add two simple logging wrappers here
03: rebase on 02 - use new wrappers, move to 260
05: add John's r-b
06: improve function docs [John], add John's r-b

v2:
01: new
02-03: test: splat into two patches, some wording
       improvements and event_wait improved
04: add John's r-b
05: new
06-09: fixes: changed, splat, use patch 01

Vladimir Sementsov-Ogievskiy (10):
  block: switch reopen queue from QSIMPLEQ to QTAILQ
  block: reverse order for reopen commits
  iotests: add test-case to 165 to test reopening qcow2 bitmaps to RW
  iotests.py: add event_wait_log and events_wait_log helpers
  block/qcow2-bitmap: get rid of bdrv_has_changed_persistent_bitmaps
  block/qcow2-bitmap: drop qcow2_reopen_bitmaps_rw_hint()
  block/qcow2-bitmap: do not remove bitmaps on reopen-ro
  iotests: add test 260 to check bitmap life after snapshot + commit
  block/qcow2-bitmap: fix and improve qcow2_reopen_bitmaps_rw
  qcow2-bitmap: move bitmap reopen-rw code to qcow2_reopen_commit

 block/qcow2.h                 |   5 +-
 include/block/block.h         |   2 +-
 include/block/block_int.h     |   6 --
 include/block/dirty-bitmap.h  |   1 -
 block.c                       |  51 +++++-------
 block/dirty-bitmap.c          |  12 ---
 block/qcow2-bitmap.c          | 143 ++++++++++++++++++++--------------
 block/qcow2.c                 |  17 +++-
 tests/qemu-iotests/165        |  46 ++++++++++-
 tests/qemu-iotests/165.out    |   4 +-
 tests/qemu-iotests/260        |  87 +++++++++++++++++++++
 tests/qemu-iotests/260.out    |  52 +++++++++++++
 tests/qemu-iotests/group      |   1 +
 tests/qemu-iotests/iotests.py |  10 +++
 14 files changed, 318 insertions(+), 119 deletions(-)
 create mode 100755 tests/qemu-iotests/260
 create mode 100644 tests/qemu-iotests/260.out

-- 
2.18.0


Re: [Qemu-devel] [PATCH v4 00/10] qcow2-bitmaps: rewrite reopening logic
Posted by Vladimir Sementsov-Ogievskiy 4 years, 8 months ago
ping

07.08.2019 17:12, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> Bitmaps reopening is buggy, reopening-rw just not working at all and
> reopening-ro may lead to producing broken incremental
> backup if we do temporary snapshot in a meantime.
> 
> v4: Drop complicated solution around reopening logic [Kevin], fix
>      the existing bug in a simplest way
> 
> Structure:
> 
> 02: fix reopen to RW
> 03: test reopen to RW
> 
> 07: fix reopen to RO
> 08: test reopen to RO
> 
> Others are less significant improvements and refactoring
> 
> Changelog:
> 
> 01-03: new patches, to fix reopening bitmaps to RW and personal test for
>         this bug
> 08: merged test from v3, it covers both bugs (reopen to RW and reopen to RO)
> 10: instead of moving bitmap-reopening to prepare(as in 09 in v3) we now keep it
>      in commit, but in right place
> others: unchanged
> 
> v3:
> 02: John's events_wait already merged in, so my 02 from v2 is not needed.
>      Instead, add two simple logging wrappers here
> 03: rebase on 02 - use new wrappers, move to 260
> 05: add John's r-b
> 06: improve function docs [John], add John's r-b
> 
> v2:
> 01: new
> 02-03: test: splat into two patches, some wording
>         improvements and event_wait improved
> 04: add John's r-b
> 05: new
> 06-09: fixes: changed, splat, use patch 01
> 
> Vladimir Sementsov-Ogievskiy (10):
>    block: switch reopen queue from QSIMPLEQ to QTAILQ
>    block: reverse order for reopen commits
>    iotests: add test-case to 165 to test reopening qcow2 bitmaps to RW
>    iotests.py: add event_wait_log and events_wait_log helpers
>    block/qcow2-bitmap: get rid of bdrv_has_changed_persistent_bitmaps
>    block/qcow2-bitmap: drop qcow2_reopen_bitmaps_rw_hint()
>    block/qcow2-bitmap: do not remove bitmaps on reopen-ro
>    iotests: add test 260 to check bitmap life after snapshot + commit
>    block/qcow2-bitmap: fix and improve qcow2_reopen_bitmaps_rw
>    qcow2-bitmap: move bitmap reopen-rw code to qcow2_reopen_commit
> 
>   block/qcow2.h                 |   5 +-
>   include/block/block.h         |   2 +-
>   include/block/block_int.h     |   6 --
>   include/block/dirty-bitmap.h  |   1 -
>   block.c                       |  51 +++++-------
>   block/dirty-bitmap.c          |  12 ---
>   block/qcow2-bitmap.c          | 143 ++++++++++++++++++++--------------
>   block/qcow2.c                 |  17 +++-
>   tests/qemu-iotests/165        |  46 ++++++++++-
>   tests/qemu-iotests/165.out    |   4 +-
>   tests/qemu-iotests/260        |  87 +++++++++++++++++++++
>   tests/qemu-iotests/260.out    |  52 +++++++++++++
>   tests/qemu-iotests/group      |   1 +
>   tests/qemu-iotests/iotests.py |  10 +++
>   14 files changed, 318 insertions(+), 119 deletions(-)
>   create mode 100755 tests/qemu-iotests/260
>   create mode 100644 tests/qemu-iotests/260.out
> 


-- 
Best regards,
Vladimir
bugfix ping Re: [PATCH v4 00/10] qcow2-bitmaps: rewrite reopening logic
Posted by Vladimir Sementsov-Ogievskiy 4 years, 8 months ago
ping
05.09.2019 12:54, Vladimir Sementsov-Ogievskiy wrote:
> ping
> 
> 07.08.2019 17:12, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all!
>>
>> Bitmaps reopening is buggy, reopening-rw just not working at all and
>> reopening-ro may lead to producing broken incremental
>> backup if we do temporary snapshot in a meantime.
>>
>> v4: Drop complicated solution around reopening logic [Kevin], fix
>>      the existing bug in a simplest way
>>
>> Structure:
>>
>> 02: fix reopen to RW
>> 03: test reopen to RW
>>
>> 07: fix reopen to RO
>> 08: test reopen to RO
>>
>> Others are less significant improvements and refactoring
>>
>> Changelog:
>>
>> 01-03: new patches, to fix reopening bitmaps to RW and personal test for
>>         this bug
>> 08: merged test from v3, it covers both bugs (reopen to RW and reopen to RO)
>> 10: instead of moving bitmap-reopening to prepare(as in 09 in v3) we now keep it
>>      in commit, but in right place
>> others: unchanged
>>
>> v3:
>> 02: John's events_wait already merged in, so my 02 from v2 is not needed.
>>      Instead, add two simple logging wrappers here
>> 03: rebase on 02 - use new wrappers, move to 260
>> 05: add John's r-b
>> 06: improve function docs [John], add John's r-b
>>
>> v2:
>> 01: new
>> 02-03: test: splat into two patches, some wording
>>         improvements and event_wait improved
>> 04: add John's r-b
>> 05: new
>> 06-09: fixes: changed, splat, use patch 01
>>
>> Vladimir Sementsov-Ogievskiy (10):
>>    block: switch reopen queue from QSIMPLEQ to QTAILQ
>>    block: reverse order for reopen commits
>>    iotests: add test-case to 165 to test reopening qcow2 bitmaps to RW
>>    iotests.py: add event_wait_log and events_wait_log helpers
>>    block/qcow2-bitmap: get rid of bdrv_has_changed_persistent_bitmaps
>>    block/qcow2-bitmap: drop qcow2_reopen_bitmaps_rw_hint()
>>    block/qcow2-bitmap: do not remove bitmaps on reopen-ro
>>    iotests: add test 260 to check bitmap life after snapshot + commit
>>    block/qcow2-bitmap: fix and improve qcow2_reopen_bitmaps_rw
>>    qcow2-bitmap: move bitmap reopen-rw code to qcow2_reopen_commit
>>
>>   block/qcow2.h                 |   5 +-
>>   include/block/block.h         |   2 +-
>>   include/block/block_int.h     |   6 --
>>   include/block/dirty-bitmap.h  |   1 -
>>   block.c                       |  51 +++++-------
>>   block/dirty-bitmap.c          |  12 ---
>>   block/qcow2-bitmap.c          | 143 ++++++++++++++++++++--------------
>>   block/qcow2.c                 |  17 +++-
>>   tests/qemu-iotests/165        |  46 ++++++++++-
>>   tests/qemu-iotests/165.out    |   4 +-
>>   tests/qemu-iotests/260        |  87 +++++++++++++++++++++
>>   tests/qemu-iotests/260.out    |  52 +++++++++++++
>>   tests/qemu-iotests/group      |   1 +
>>   tests/qemu-iotests/iotests.py |  10 +++
>>   14 files changed, 318 insertions(+), 119 deletions(-)
>>   create mode 100755 tests/qemu-iotests/260
>>   create mode 100644 tests/qemu-iotests/260.out
>>
> 
> 


-- 
Best regards,
Vladimir
Re: [Qemu-devel] [PATCH v4 00/10] qcow2-bitmaps: rewrite reopening logic
Posted by John Snow 4 years, 7 months ago

On 8/7/19 10:12 AM, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> Bitmaps reopening is buggy, reopening-rw just not working at all and
> reopening-ro may lead to producing broken incremental
> backup if we do temporary snapshot in a meantime.
> 
> v4: Drop complicated solution around reopening logic [Kevin], fix
>     the existing bug in a simplest way
> 

Overall, seems good. I want Max to take a look at 10/10, and there are
some minor rebase conflicts. I hope to get this staged next week if at
all possible.

Thanks,
--js

> Structure:
> 
> 02: fix reopen to RW
> 03: test reopen to RW
> 
> 07: fix reopen to RO
> 08: test reopen to RO
> 
> Others are less significant improvements and refactoring
> 
> Changelog:
> 
> 01-03: new patches, to fix reopening bitmaps to RW and personal test for
>        this bug
> 08: merged test from v3, it covers both bugs (reopen to RW and reopen to RO)
> 10: instead of moving bitmap-reopening to prepare(as in 09 in v3) we now keep it
>     in commit, but in right place
> others: unchanged
> 
> v3:
> 02: John's events_wait already merged in, so my 02 from v2 is not needed.
>     Instead, add two simple logging wrappers here
> 03: rebase on 02 - use new wrappers, move to 260
> 05: add John's r-b
> 06: improve function docs [John], add John's r-b
> 
> v2:
> 01: new
> 02-03: test: splat into two patches, some wording
>        improvements and event_wait improved
> 04: add John's r-b
> 05: new
> 06-09: fixes: changed, splat, use patch 01
> 
> Vladimir Sementsov-Ogievskiy (10):
>   block: switch reopen queue from QSIMPLEQ to QTAILQ
>   block: reverse order for reopen commits
>   iotests: add test-case to 165 to test reopening qcow2 bitmaps to RW
>   iotests.py: add event_wait_log and events_wait_log helpers
>   block/qcow2-bitmap: get rid of bdrv_has_changed_persistent_bitmaps
>   block/qcow2-bitmap: drop qcow2_reopen_bitmaps_rw_hint()
>   block/qcow2-bitmap: do not remove bitmaps on reopen-ro
>   iotests: add test 260 to check bitmap life after snapshot + commit
>   block/qcow2-bitmap: fix and improve qcow2_reopen_bitmaps_rw
>   qcow2-bitmap: move bitmap reopen-rw code to qcow2_reopen_commit
> 
>  block/qcow2.h                 |   5 +-
>  include/block/block.h         |   2 +-
>  include/block/block_int.h     |   6 --
>  include/block/dirty-bitmap.h  |   1 -
>  block.c                       |  51 +++++-------
>  block/dirty-bitmap.c          |  12 ---
>  block/qcow2-bitmap.c          | 143 ++++++++++++++++++++--------------
>  block/qcow2.c                 |  17 +++-
>  tests/qemu-iotests/165        |  46 ++++++++++-
>  tests/qemu-iotests/165.out    |   4 +-
>  tests/qemu-iotests/260        |  87 +++++++++++++++++++++
>  tests/qemu-iotests/260.out    |  52 +++++++++++++
>  tests/qemu-iotests/group      |   1 +
>  tests/qemu-iotests/iotests.py |  10 +++
>  14 files changed, 318 insertions(+), 119 deletions(-)
>  create mode 100755 tests/qemu-iotests/260
>  create mode 100644 tests/qemu-iotests/260.out
> 

Re: [Qemu-devel] [PATCH v4 00/10] qcow2-bitmaps: rewrite reopening logic
Posted by Vladimir Sementsov-Ogievskiy 4 years, 7 months ago
27.09.2019 2:25, John Snow wrote:
> 
> On 8/7/19 10:12 AM, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all!
>>
>> Bitmaps reopening is buggy, reopening-rw just not working at all and
>> reopening-ro may lead to producing broken incremental
>> backup if we do temporary snapshot in a meantime.
>>
>> v4: Drop complicated solution around reopening logic [Kevin], fix
>>      the existing bug in a simplest way
>>
> Overall, seems good. I want Max to take a look at 10/10, and there are
> some minor rebase conflicts. I hope to get this staged next week if at
> all possible.
> 
> Thanks,
> --js
> 

Thank you for reviewing!

-- 
Best regards,
Vladimir