[PATCH v2 00/22] Fix error handling during bitmap postcopy

Vladimir Sementsov-Ogievskiy posted 22 patches 4 years, 2 months ago
Test FreeBSD passed
Test docker-quick@centos7 passed
Test docker-mingw@fedora passed
Test checkpatch failed
Failed in applying to current master (apply log)
There is a newer version of this series
migration/migration.h          |   3 +-
migration/block-dirty-bitmap.c | 444 +++++++++++++++++++++------------
migration/migration.c          |  15 +-
migration/savevm.c             |  37 ++-
python/qemu/machine.py         |  12 +-
tests/qemu-iotests/199         | 244 ++++++++++++++----
tests/qemu-iotests/199.out     |   4 +-
7 files changed, 529 insertions(+), 230 deletions(-)
[PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Vladimir Sementsov-Ogievskiy 4 years, 2 months ago
Original idea of bitmaps postcopy migration is that bitmaps are non
critical data, and their loss is not serious problem. So, using postcopy
method on any failure we should just drop unfinished bitmaps and
continue guest execution.

However, it doesn't work so. It crashes, fails, it goes to
postcopy-recovery feature. It does anything except for behavior we want.
These series fixes at least some problems with error handling during
bitmaps migration postcopy.

v1 was "[PATCH 0/7] Fix crashes on early shutdown during bitmaps postcopy"

v2:

Most of patches are new or changed a lot.
Only patches 06,07 mostly unchanged, just rebased on refactorings.

Vladimir Sementsov-Ogievskiy (22):
  migration/block-dirty-bitmap: fix dirty_bitmap_mig_before_vm_start
  migration/block-dirty-bitmap: rename state structure types
  migration/block-dirty-bitmap: rename dirty_bitmap_mig_cleanup
  migration/block-dirty-bitmap: move mutex init to dirty_bitmap_mig_init
  migration/block-dirty-bitmap: refactor state global variables
  migration/block-dirty-bitmap: rename finish_lock to just lock
  migration/block-dirty-bitmap: simplify dirty_bitmap_load_complete
  migration/block-dirty-bitmap: keep bitmap state for all bitmaps
  migration/block-dirty-bitmap: relax error handling in incoming part
  migration/block-dirty-bitmap: cancel migration on shutdown
  migration/savevm: don't worry if bitmap migration postcopy failed
  qemu-iotests/199: fix style
  qemu-iotests/199: drop extra constraints
  qemu-iotests/199: better catch postcopy time
  qemu-iotests/199: improve performance: set bitmap by discard
  qemu-iotests/199: change discard patterns
  qemu-iotests/199: increase postcopy period
  python/qemu/machine: add kill() method
  qemu-iotests/199: prepare for new test-cases addition
  qemu-iotests/199: check persistent bitmaps
  qemu-iotests/199: add early shutdown case to bitmaps postcopy
  qemu-iotests/199: add source-killed case to bitmaps postcopy

Cc: John Snow <jsnow@redhat.com>
Cc: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
Cc: Fam Zheng <fam@euphon.net>
Cc: Juan Quintela <quintela@redhat.com>
Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: Eduardo Habkost <ehabkost@redhat.com>
Cc: Cleber Rosa <crosa@redhat.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Cc: Max Reitz <mreitz@redhat.com>
Cc: qemu-block@nongnu.org
Cc: qemu-devel@nongnu.org
Cc: qemu-stable@nongnu.org # for patch 01

 migration/migration.h          |   3 +-
 migration/block-dirty-bitmap.c | 444 +++++++++++++++++++++------------
 migration/migration.c          |  15 +-
 migration/savevm.c             |  37 ++-
 python/qemu/machine.py         |  12 +-
 tests/qemu-iotests/199         | 244 ++++++++++++++----
 tests/qemu-iotests/199.out     |   4 +-
 7 files changed, 529 insertions(+), 230 deletions(-)

-- 
2.21.0


Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by no-reply@patchew.org 4 years, 2 months ago
Patchew URL: https://patchew.org/QEMU/20200217150246.29180-1-vsementsov@virtuozzo.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Message-id: 20200217150246.29180-1-vsementsov@virtuozzo.com
Type: series

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
fatal: git fetch_pack: expected ACK/NAK, got 'ERR upload-pack: not our ref 247b588c357694c896d056836da2341d75451c4f'
fatal: The remote end hung up unexpectedly
error: Could not fetch 3c8cf5a9c21ff8782164d1def7f44bd888713384
Traceback (most recent call last):
  File "patchew-tester/src/patchew-cli", line 521, in test_one
    git_clone_repo(clone, r["repo"], r["head"], logf, True)
  File "patchew-tester/src/patchew-cli", line 48, in git_clone_repo
    stdout=logf, stderr=logf)
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'remote', 'add', '-f', '--mirror=fetch', '3c8cf5a9c21ff8782164d1def7f44bd888713384', 'https://github.com/patchew-project/qemu']' returned non-zero exit status 1.



The full log is available at
http://patchew.org/logs/20200217150246.29180-1-vsementsov@virtuozzo.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Andrey Shinkevich 4 years, 2 months ago
qemu-iotests:$ ./check -qcow2
PASSED
(except always failed 261 and 272)

Andrey

On 17/02/2020 18:02, Vladimir Sementsov-Ogievskiy wrote:
> Original idea of bitmaps postcopy migration is that bitmaps are non
> critical data, and their loss is not serious problem. So, using postcopy
> method on any failure we should just drop unfinished bitmaps and
> continue guest execution.
> 
> However, it doesn't work so. It crashes, fails, it goes to
> postcopy-recovery feature. It does anything except for behavior we want.
> These series fixes at least some problems with error handling during
> bitmaps migration postcopy.
> 
> v1 was "[PATCH 0/7] Fix crashes on early shutdown during bitmaps postcopy"
> 
> v2:
> 
> Most of patches are new or changed a lot.
> Only patches 06,07 mostly unchanged, just rebased on refactorings.
> 
> Vladimir Sementsov-Ogievskiy (22):
>    migration/block-dirty-bitmap: fix dirty_bitmap_mig_before_vm_start
>    migration/block-dirty-bitmap: rename state structure types
>    migration/block-dirty-bitmap: rename dirty_bitmap_mig_cleanup
>    migration/block-dirty-bitmap: move mutex init to dirty_bitmap_mig_init
>    migration/block-dirty-bitmap: refactor state global variables
>    migration/block-dirty-bitmap: rename finish_lock to just lock
>    migration/block-dirty-bitmap: simplify dirty_bitmap_load_complete
>    migration/block-dirty-bitmap: keep bitmap state for all bitmaps
>    migration/block-dirty-bitmap: relax error handling in incoming part
>    migration/block-dirty-bitmap: cancel migration on shutdown
>    migration/savevm: don't worry if bitmap migration postcopy failed
>    qemu-iotests/199: fix style
>    qemu-iotests/199: drop extra constraints
>    qemu-iotests/199: better catch postcopy time
>    qemu-iotests/199: improve performance: set bitmap by discard
>    qemu-iotests/199: change discard patterns
>    qemu-iotests/199: increase postcopy period
>    python/qemu/machine: add kill() method
>    qemu-iotests/199: prepare for new test-cases addition
>    qemu-iotests/199: check persistent bitmaps
>    qemu-iotests/199: add early shutdown case to bitmaps postcopy
>    qemu-iotests/199: add source-killed case to bitmaps postcopy
> 
> Cc: John Snow <jsnow@redhat.com>
> Cc: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> Cc: Fam Zheng <fam@euphon.net>
> Cc: Juan Quintela <quintela@redhat.com>
> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> Cc: Eduardo Habkost <ehabkost@redhat.com>
> Cc: Cleber Rosa <crosa@redhat.com>
> Cc: Kevin Wolf <kwolf@redhat.com>
> Cc: Max Reitz <mreitz@redhat.com>
> Cc: qemu-block@nongnu.org
> Cc: qemu-devel@nongnu.org
> Cc: qemu-stable@nongnu.org # for patch 01
> 
>   migration/migration.h          |   3 +-
>   migration/block-dirty-bitmap.c | 444 +++++++++++++++++++++------------
>   migration/migration.c          |  15 +-
>   migration/savevm.c             |  37 ++-
>   python/qemu/machine.py         |  12 +-
>   tests/qemu-iotests/199         | 244 ++++++++++++++----
>   tests/qemu-iotests/199.out     |   4 +-
>   7 files changed, 529 insertions(+), 230 deletions(-)
> 

-- 
With the best regards,
Andrey Shinkevich

Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Eric Blake 4 years, 2 months ago
On 2/18/20 2:02 PM, Andrey Shinkevich wrote:
> qemu-iotests:$ ./check -qcow2
> PASSED
> (except always failed 261 and 272)

Have you reported those failures on the threads that introduced those tests?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Andrey Shinkevich 4 years, 2 months ago

On 18/02/2020 23:57, Eric Blake wrote:
> On 2/18/20 2:02 PM, Andrey Shinkevich wrote:
>> qemu-iotests:$ ./check -qcow2
>> PASSED
>> (except always failed 261 and 272)
> 
> Have you reported those failures on the threads that introduced those 
> tests?
> 

Not yet unfortunately. I have not investigated the case.
"$ ./check -qcow2 261" dumps

+od: unrecognized option '--endian=big'
+Try 'od --help' for more information.
+od: invalid -N argument '--endian=big'
+qemu-img: Could not open 'TEST_DIR/t.IMGFMT': IMGFMT header exceeds 
cluster size

and "$ ./check -qcow2 272" dumps

+od: unrecognized option '--endian=big'
+Try 'od --help' for more information.
+od: invalid -N argument '--endian=big'
+qemu-io: can't open device .../qemu/tests/qemu-iotests/scratch/t.qcow2: 
Image is not in qcow2 format

-- 
With the best regards,
Andrey Shinkevich

Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Eric Blake 4 years, 2 months ago
On 2/19/20 7:25 AM, Andrey Shinkevich wrote:
> 
> 
> On 18/02/2020 23:57, Eric Blake wrote:
>> On 2/18/20 2:02 PM, Andrey Shinkevich wrote:
>>> qemu-iotests:$ ./check -qcow2
>>> PASSED
>>> (except always failed 261 and 272)
>>
>> Have you reported those failures on the threads that introduced those 
>> tests?
>>
> 
> Not yet unfortunately. I have not investigated the case.
> "$ ./check -qcow2 261" dumps
> 
> +od: unrecognized option '--endian=big'
> +Try 'od --help' for more information.
> +od: invalid -N argument '--endian=big'
> +qemu-img: Could not open 'TEST_DIR/t.IMGFMT': IMGFMT header exceeds 
> cluster size

Which version of od are you using?  I do recall wondering whether 
reliance on the GNU coreutils extension --endian=big was going to cause 
problems later - well, here we are, it's later :)

https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg06781.html

> 
> and "$ ./check -qcow2 272" dumps
> 
> +od: unrecognized option '--endian=big'
> +Try 'od --help' for more information.
> +od: invalid -N argument '--endian=big'

Yay, same problem for both tests.  Fix common.rc once, and both tests 
should start working for you.

> +qemu-io: can't open device .../qemu/tests/qemu-iotests/scratch/t.qcow2: 
> Image is not in qcow2 format
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Andrey Shinkevich 4 years, 2 months ago

On 19/02/2020 16:36, Eric Blake wrote:
> On 2/19/20 7:25 AM, Andrey Shinkevich wrote:
>>
>>
>> On 18/02/2020 23:57, Eric Blake wrote:
>>> On 2/18/20 2:02 PM, Andrey Shinkevich wrote:
>>>> qemu-iotests:$ ./check -qcow2
>>>> PASSED
>>>> (except always failed 261 and 272)
>>>
>>> Have you reported those failures on the threads that introduced those 
>>> tests?
>>>
>>
>> Not yet unfortunately. I have not investigated the case.
>> "$ ./check -qcow2 261" dumps
>>
>> +od: unrecognized option '--endian=big'
>> +Try 'od --help' for more information.
>> +od: invalid -N argument '--endian=big'
>> +qemu-img: Could not open 'TEST_DIR/t.IMGFMT': IMGFMT header exceeds 
>> cluster size
> 
> Which version of od are you using?  I do recall wondering whether 

$ od --version
od (GNU coreutils) 8.22
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later...

> reliance on the GNU coreutils extension --endian=big was going to cause 
> problems later - well, here we are, it's later :)
> 
> https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg06781.html
> 
>>
>> and "$ ./check -qcow2 272" dumps
>>
>> +od: unrecognized option '--endian=big'
>> +Try 'od --help' for more information.
>> +od: invalid -N argument '--endian=big'
> 
> Yay, same problem for both tests.  Fix common.rc once, and both tests 
> should start working for you.

Thank you Eric! I want to sort it out later...

> 
>> +qemu-io: can't open device 
>> .../qemu/tests/qemu-iotests/scratch/t.qcow2: Image is not in qcow2 format
>>
> 

-- 
With the best regards,
Andrey Shinkevich


Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Eric Blake 4 years, 2 months ago
On 2/19/20 7:52 AM, Andrey Shinkevich wrote:

>>> +od: unrecognized option '--endian=big'
>>> +Try 'od --help' for more information.
>>> +od: invalid -N argument '--endian=big'
>>
>> Yay, same problem for both tests.  Fix common.rc once, and both tests 
>> should start working for you.
> 
> Thank you Eric! I want to sort it out later...

Patch proposed:
https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg05188.html


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Andrey Shinkevich 4 years, 2 months ago

On 19/02/2020 17:58, Eric Blake wrote:
> On 2/19/20 7:52 AM, Andrey Shinkevich wrote:
> 
>>>> +od: unrecognized option '--endian=big'
>>>> +Try 'od --help' for more information.
>>>> +od: invalid -N argument '--endian=big'
>>>
>>> Yay, same problem for both tests.  Fix common.rc once, and both tests 
>>> should start working for you.
>>
>> Thank you Eric! I want to sort it out later...
> 
> Patch proposed:
> https://lists.gnu.org/archive/html/qemu-devel/2020-02/msg05188.html
> 
> 

Thank you Eric, I appreciate.
-- 
With the best regards,
Andrey Shinkevich


Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Eric Blake 4 years, 2 months ago
On 2/19/20 7:36 AM, Eric Blake wrote:

>> +od: unrecognized option '--endian=big'
>> +Try 'od --help' for more information.
>> +od: invalid -N argument '--endian=big'
>> +qemu-img: Could not open 'TEST_DIR/t.IMGFMT': IMGFMT header exceeds 
>> cluster size
> 
> Which version of od are you using?  I do recall wondering whether 
> reliance on the GNU coreutils extension --endian=big was going to cause 
> problems later - well, here we are, it's later :)
> 
> https://lists.gnu.org/archive/html/qemu-devel/2019-07/msg06781.html

coreutils documents that od --endian was added in 8.23, released in 
2014-07-18.  Per https://wiki.qemu.org/Supported_Build_Platforms, we 
still have support for RHEL 7 through 2022, and it was first released 
2014-06-09 (all other supported distros have newer releases, but I 
didn't check what coreutils version they included, or even if the BSD 
builds which don't use coreutils would also be impacted by this 
problem).  Still, I'd like to know your specific setup, and why the CI 
tools have not flagged it.

But even one counterexample within the bounds of our supported distro 
page is a good argument that use of od --endian is not yet portable. 
Or, if your setup is not on the supported page, it becomes a question of 
whether it should be added or whether you should upgrade to something 
that is supported.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Vladimir Sementsov-Ogievskiy 4 years, 1 month ago
Ping!

It's a fix, but not a degradation and I'm afraid too big for 5.0.

Still, I think I should ping it anyway. John, I'm afraid, that this all is for your branch :)


17.02.2020 18:02, Vladimir Sementsov-Ogievskiy wrote:
> Original idea of bitmaps postcopy migration is that bitmaps are non
> critical data, and their loss is not serious problem. So, using postcopy
> method on any failure we should just drop unfinished bitmaps and
> continue guest execution.
> 
> However, it doesn't work so. It crashes, fails, it goes to
> postcopy-recovery feature. It does anything except for behavior we want.
> These series fixes at least some problems with error handling during
> bitmaps migration postcopy.
> 
> v1 was "[PATCH 0/7] Fix crashes on early shutdown during bitmaps postcopy"
> 
> v2:
> 
> Most of patches are new or changed a lot.
> Only patches 06,07 mostly unchanged, just rebased on refactorings.
> 
> Vladimir Sementsov-Ogievskiy (22):
>    migration/block-dirty-bitmap: fix dirty_bitmap_mig_before_vm_start
>    migration/block-dirty-bitmap: rename state structure types
>    migration/block-dirty-bitmap: rename dirty_bitmap_mig_cleanup
>    migration/block-dirty-bitmap: move mutex init to dirty_bitmap_mig_init
>    migration/block-dirty-bitmap: refactor state global variables
>    migration/block-dirty-bitmap: rename finish_lock to just lock
>    migration/block-dirty-bitmap: simplify dirty_bitmap_load_complete
>    migration/block-dirty-bitmap: keep bitmap state for all bitmaps
>    migration/block-dirty-bitmap: relax error handling in incoming part
>    migration/block-dirty-bitmap: cancel migration on shutdown
>    migration/savevm: don't worry if bitmap migration postcopy failed
>    qemu-iotests/199: fix style
>    qemu-iotests/199: drop extra constraints
>    qemu-iotests/199: better catch postcopy time
>    qemu-iotests/199: improve performance: set bitmap by discard
>    qemu-iotests/199: change discard patterns
>    qemu-iotests/199: increase postcopy period
>    python/qemu/machine: add kill() method
>    qemu-iotests/199: prepare for new test-cases addition
>    qemu-iotests/199: check persistent bitmaps
>    qemu-iotests/199: add early shutdown case to bitmaps postcopy
>    qemu-iotests/199: add source-killed case to bitmaps postcopy
> 
> Cc: John Snow <jsnow@redhat.com>
> Cc: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> Cc: Fam Zheng <fam@euphon.net>
> Cc: Juan Quintela <quintela@redhat.com>
> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
> Cc: Eduardo Habkost <ehabkost@redhat.com>
> Cc: Cleber Rosa <crosa@redhat.com>
> Cc: Kevin Wolf <kwolf@redhat.com>
> Cc: Max Reitz <mreitz@redhat.com>
> Cc: qemu-block@nongnu.org
> Cc: qemu-devel@nongnu.org
> Cc: qemu-stable@nongnu.org # for patch 01
> 
>   migration/migration.h          |   3 +-
>   migration/block-dirty-bitmap.c | 444 +++++++++++++++++++++------------
>   migration/migration.c          |  15 +-
>   migration/savevm.c             |  37 ++-
>   python/qemu/machine.py         |  12 +-
>   tests/qemu-iotests/199         | 244 ++++++++++++++----
>   tests/qemu-iotests/199.out     |   4 +-
>   7 files changed, 529 insertions(+), 230 deletions(-)
> 


-- 
Best regards,
Vladimir

Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Eric Blake 3 years, 11 months ago
On 4/2/20 2:42 AM, Vladimir Sementsov-Ogievskiy wrote:
> Ping!
> 
> It's a fix, but not a degradation and I'm afraid too big for 5.0.
> 
> Still, I think I should ping it anyway. John, I'm afraid, that this all 
> is for your branch :)

Just noticing this thread, now that we've shuffled bitmaps maintainers. 
Is there anything here that we still need to include in 5.1?

> 
> 
> 17.02.2020 18:02, Vladimir Sementsov-Ogievskiy wrote:
>> Original idea of bitmaps postcopy migration is that bitmaps are non
>> critical data, and their loss is not serious problem. So, using postcopy
>> method on any failure we should just drop unfinished bitmaps and
>> continue guest execution.
>>
>> However, it doesn't work so. It crashes, fails, it goes to
>> postcopy-recovery feature. It does anything except for behavior we want.
>> These series fixes at least some problems with error handling during
>> bitmaps migration postcopy.
>>
>> v1 was "[PATCH 0/7] Fix crashes on early shutdown during bitmaps 
>> postcopy"
>>
>> v2:
>>
>> Most of patches are new or changed a lot.
>> Only patches 06,07 mostly unchanged, just rebased on refactorings.
>>
>> Vladimir Sementsov-Ogievskiy (22):
>>    migration/block-dirty-bitmap: fix dirty_bitmap_mig_before_vm_start
>>    migration/block-dirty-bitmap: rename state structure types
>>    migration/block-dirty-bitmap: rename dirty_bitmap_mig_cleanup
>>    migration/block-dirty-bitmap: move mutex init to dirty_bitmap_mig_init
>>    migration/block-dirty-bitmap: refactor state global variables
>>    migration/block-dirty-bitmap: rename finish_lock to just lock
>>    migration/block-dirty-bitmap: simplify dirty_bitmap_load_complete
>>    migration/block-dirty-bitmap: keep bitmap state for all bitmaps
>>    migration/block-dirty-bitmap: relax error handling in incoming part
>>    migration/block-dirty-bitmap: cancel migration on shutdown
>>    migration/savevm: don't worry if bitmap migration postcopy failed
>>    qemu-iotests/199: fix style
>>    qemu-iotests/199: drop extra constraints
>>    qemu-iotests/199: better catch postcopy time
>>    qemu-iotests/199: improve performance: set bitmap by discard
>>    qemu-iotests/199: change discard patterns
>>    qemu-iotests/199: increase postcopy period
>>    python/qemu/machine: add kill() method
>>    qemu-iotests/199: prepare for new test-cases addition
>>    qemu-iotests/199: check persistent bitmaps
>>    qemu-iotests/199: add early shutdown case to bitmaps postcopy
>>    qemu-iotests/199: add source-killed case to bitmaps postcopy
>>
>> Cc: John Snow <jsnow@redhat.com>
>> Cc: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> Cc: Stefan Hajnoczi <stefanha@redhat.com>
>> Cc: Fam Zheng <fam@euphon.net>
>> Cc: Juan Quintela <quintela@redhat.com>
>> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>> Cc: Eduardo Habkost <ehabkost@redhat.com>
>> Cc: Cleber Rosa <crosa@redhat.com>
>> Cc: Kevin Wolf <kwolf@redhat.com>
>> Cc: Max Reitz <mreitz@redhat.com>
>> Cc: qemu-block@nongnu.org
>> Cc: qemu-devel@nongnu.org
>> Cc: qemu-stable@nongnu.org # for patch 01
>>
>>   migration/migration.h          |   3 +-
>>   migration/block-dirty-bitmap.c | 444 +++++++++++++++++++++------------
>>   migration/migration.c          |  15 +-
>>   migration/savevm.c             |  37 ++-
>>   python/qemu/machine.py         |  12 +-
>>   tests/qemu-iotests/199         | 244 ++++++++++++++----
>>   tests/qemu-iotests/199.out     |   4 +-
>>   7 files changed, 529 insertions(+), 230 deletions(-)
>>
> 
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org


Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Vladimir Sementsov-Ogievskiy 3 years, 11 months ago
29.05.2020 14:58, Eric Blake wrote:
> On 4/2/20 2:42 AM, Vladimir Sementsov-Ogievskiy wrote:
>> Ping!
>>
>> It's a fix, but not a degradation and I'm afraid too big for 5.0.
>>
>> Still, I think I should ping it anyway. John, I'm afraid, that this all is for your branch :)
> 
> Just noticing this thread, now that we've shuffled bitmaps maintainers. Is there anything here that we still need to include in 5.1?

Yes, we need the whole series.

> 
>>
>>
>> 17.02.2020 18:02, Vladimir Sementsov-Ogievskiy wrote:
>>> Original idea of bitmaps postcopy migration is that bitmaps are non
>>> critical data, and their loss is not serious problem. So, using postcopy
>>> method on any failure we should just drop unfinished bitmaps and
>>> continue guest execution.
>>>
>>> However, it doesn't work so. It crashes, fails, it goes to
>>> postcopy-recovery feature. It does anything except for behavior we want.
>>> These series fixes at least some problems with error handling during
>>> bitmaps migration postcopy.
>>>
>>> v1 was "[PATCH 0/7] Fix crashes on early shutdown during bitmaps postcopy"
>>>
>>> v2:
>>>
>>> Most of patches are new or changed a lot.
>>> Only patches 06,07 mostly unchanged, just rebased on refactorings.
>>>
>>> Vladimir Sementsov-Ogievskiy (22):
>>>    migration/block-dirty-bitmap: fix dirty_bitmap_mig_before_vm_start
>>>    migration/block-dirty-bitmap: rename state structure types
>>>    migration/block-dirty-bitmap: rename dirty_bitmap_mig_cleanup
>>>    migration/block-dirty-bitmap: move mutex init to dirty_bitmap_mig_init
>>>    migration/block-dirty-bitmap: refactor state global variables
>>>    migration/block-dirty-bitmap: rename finish_lock to just lock
>>>    migration/block-dirty-bitmap: simplify dirty_bitmap_load_complete
>>>    migration/block-dirty-bitmap: keep bitmap state for all bitmaps
>>>    migration/block-dirty-bitmap: relax error handling in incoming part
>>>    migration/block-dirty-bitmap: cancel migration on shutdown
>>>    migration/savevm: don't worry if bitmap migration postcopy failed
>>>    qemu-iotests/199: fix style
>>>    qemu-iotests/199: drop extra constraints
>>>    qemu-iotests/199: better catch postcopy time
>>>    qemu-iotests/199: improve performance: set bitmap by discard
>>>    qemu-iotests/199: change discard patterns
>>>    qemu-iotests/199: increase postcopy period
>>>    python/qemu/machine: add kill() method
>>>    qemu-iotests/199: prepare for new test-cases addition
>>>    qemu-iotests/199: check persistent bitmaps
>>>    qemu-iotests/199: add early shutdown case to bitmaps postcopy
>>>    qemu-iotests/199: add source-killed case to bitmaps postcopy
>>>
>>> Cc: John Snow <jsnow@redhat.com>
>>> Cc: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>> Cc: Stefan Hajnoczi <stefanha@redhat.com>
>>> Cc: Fam Zheng <fam@euphon.net>
>>> Cc: Juan Quintela <quintela@redhat.com>
>>> Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
>>> Cc: Eduardo Habkost <ehabkost@redhat.com>
>>> Cc: Cleber Rosa <crosa@redhat.com>
>>> Cc: Kevin Wolf <kwolf@redhat.com>
>>> Cc: Max Reitz <mreitz@redhat.com>
>>> Cc: qemu-block@nongnu.org
>>> Cc: qemu-devel@nongnu.org
>>> Cc: qemu-stable@nongnu.org # for patch 01
>>>
>>>   migration/migration.h          |   3 +-
>>>   migration/block-dirty-bitmap.c | 444 +++++++++++++++++++++------------
>>>   migration/migration.c          |  15 +-
>>>   migration/savevm.c             |  37 ++-
>>>   python/qemu/machine.py         |  12 +-
>>>   tests/qemu-iotests/199         | 244 ++++++++++++++----
>>>   tests/qemu-iotests/199.out     |   4 +-
>>>   7 files changed, 529 insertions(+), 230 deletions(-)
>>>
>>
>>
> 


-- 
Best regards,
Vladimir

Re: [PATCH v2 00/22] Fix error handling during bitmap postcopy
Posted by Eric Blake 3 years, 9 months ago
On 5/29/20 7:16 AM, Vladimir Sementsov-Ogievskiy wrote:
> 29.05.2020 14:58, Eric Blake wrote:
>> On 4/2/20 2:42 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> Ping!
>>>
>>> It's a fix, but not a degradation and I'm afraid too big for 5.0.
>>>
>>> Still, I think I should ping it anyway. John, I'm afraid, that this 
>>> all is for your branch :)
>>
>> Just noticing this thread, now that we've shuffled bitmaps 
>> maintainers. Is there anything here that we still need to include in 5.1?
> 
> Yes, we need the whole series.

I'm starting to go through it now, to see what is still worth getting in 
to 5.1-rc2, but no promises as it is a long series and I don't want to 
introduce last-minute regressions (the fact that this missed 5.0 says 
that 5.1 will be no worse than 5.0 if we don't get this in until 5.2).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org