[PULL 00/30] Migration pull patches (take 4)

Juan Quintela posted 30 patches 1 week ago
Test docker-mingw@fedora passed
Test checkpatch passed
Test docker-quick@centos7 passed
Test FreeBSD passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20200114125254.4515-1-quintela@redhat.com
Maintainers: Paolo Bonzini <pbonzini@redhat.com>, Corey Minyard <cminyard@mvista.com>, Peter Maydell <peter.maydell@linaro.org>, Andrzej Zaborowski <balrogg@gmail.com>, "Dr. David Alan Gilbert" <dgilbert@redhat.com>, Stefan Berger <stefanb@linux.ibm.com>, David Gibson <david@gibson.dropbear.id.au>, Thomas Huth <thuth@redhat.com>, Laurent Vivier <lvivier@redhat.com>, Jason Wang <jasowang@redhat.com>, "Marc-André Lureau" <marcandre.lureau@redhat.com>, Eduardo Habkost <ehabkost@redhat.com>, Richard Henderson <rth@twiddle.net>, Juan Quintela <quintela@redhat.com>, Stefan Weil <sw@weilnetz.de>, "Daniel P. Berrangé" <berrange@redhat.com>, Marcel Apfelbaum <marcel.apfelbaum@gmail.com>, "Michael S. Tsirkin" <mst@redhat.com>
backends/dbus-vmstate.c      |   3 +-
exec.c                       |   4 +-
hw/arm/stellaris.c           |   2 +-
hw/core/qdev.c               |   3 +-
hw/display/ads7846.c         |   2 +-
hw/i2c/core.c                |   2 +-
hw/input/stellaris_input.c   |   3 +-
hw/intc/apic_common.c        |   7 +-
hw/misc/max111x.c            |   3 +-
hw/net/eepro100.c            |   3 +-
hw/pci/pci.c                 |   2 +-
hw/ppc/spapr.c               |   2 +-
hw/timer/arm_timer.c         |   2 +-
hw/tpm/tpm_emulator.c        |   3 +-
include/migration/register.h |   2 +-
include/migration/vmstate.h  |  25 ++++-
include/qemu/queue.h         |  39 ++++++++
migration/migration.c        |  72 +++++++-------
migration/migration.h        |   1 +
migration/ram.c              | 185 ++++++++++++++++++++++++++---------
migration/savevm.c           |  61 ++++++++----
migration/trace-events       |   9 +-
migration/vmstate-types.c    |  70 +++++++++++++
stubs/vmstate.c              |   2 +-
tests/qtest/migration-test.c |  97 +++++++++++++++++-
tests/test-vmstate.c         | 170 ++++++++++++++++++++++++++++++++
vl.c                         |  10 +-
27 files changed, 654 insertions(+), 130 deletions(-)

[PULL 00/30] Migration pull patches (take 4)

Posted by Juan Quintela 1 week ago
The following changes since commit 3c8a6575985b1652b45bfa670b5e1907d642cfa0:

  Merge remote-tracking branch 'remotes/kraxel/tags/usb-20200113-pull-request' into staging (2020-01-13 14:19:57 +0000)

are available in the Git repository at:

  https://github.com/juanquintela/qemu.git tags/migration-pull-pull-request

for you to fetch changes up to b039b02c25d1725cf0296721d98e35e024468b20:

  migration: Support QLIST migration (2020-01-14 13:45:29 +0100)

----------------------------------------------------------------
Migration pull request (take 4)

- both patches for x32 archs in
- reorder the pages to make git bisect happy

Thanks a lot to Daniel for helping with the bugs.  Twice.

----------------------------------------------------------------

Alexey Romko (1):
  Bug #1829242 correction.

Daniel Henrique Barboza (1):
  ram.c: remove unneeded labels

Dr. David Alan Gilbert (1):
  migration: Rate limit inside host pages

Eric Auger (1):
  migration: Support QLIST migration

Fangrui Song (1):
  migration: Fix incorrect integer->float conversion caught by clang

Jiahui Cen (2):
  migration/multifd: fix nullptr access in terminating multifd threads
  migration/multifd: fix destroyed mutex access in terminating multifd
    threads

Juan Quintela (5):
  multifd: Initialize local variable
  multifd: Allocate uint64_t instead of ram_addr_t
  migration-test: Add migration multifd test
  migration: Make sure that we don't call write() in case of error
  migration-test: introduce functions to handle string parameters

Laurent Vivier (2):
  migration-test: ppc64: fix FORTH test program
  runstate: ignore finishmigrate -> prelaunch transition

Marc-André Lureau (1):
  misc: use QEMU_IS_ALIGNED

Peter Xu (3):
  migration: Define VMSTATE_INSTANCE_ID_ANY
  migration: Change SaveStateEntry.instance_id into uint32_t
  apic: Use 32bit APIC ID for migration instance ID

Scott Cheloha (2):
  migration: add savevm_state_handler_remove()
  migration: savevm_state_handler_insert: constant-time element
    insertion

Wei Yang (8):
  migration/postcopy: reduce memset when it is zero page and
    matches_target_page_size
  migration/postcopy: wait for decompress thread in precopy
  migration/postcopy: count target page number to decide the
    place_needed
  migration/postcopy: set all_zero to true on the first target page
  migration/postcopy: enable random order target page arrival
  migration/postcopy: enable compress during postcopy
  migration/multifd: clean pages after filling packet
  migration/multifd: not use multifd during postcopy

Yury Kotov (2):
  migration: Fix the re-run check of the migrate-incoming command
  migration/ram: Yield periodically to the main loop

 backends/dbus-vmstate.c      |   3 +-
 exec.c                       |   4 +-
 hw/arm/stellaris.c           |   2 +-
 hw/core/qdev.c               |   3 +-
 hw/display/ads7846.c         |   2 +-
 hw/i2c/core.c                |   2 +-
 hw/input/stellaris_input.c   |   3 +-
 hw/intc/apic_common.c        |   7 +-
 hw/misc/max111x.c            |   3 +-
 hw/net/eepro100.c            |   3 +-
 hw/pci/pci.c                 |   2 +-
 hw/ppc/spapr.c               |   2 +-
 hw/timer/arm_timer.c         |   2 +-
 hw/tpm/tpm_emulator.c        |   3 +-
 include/migration/register.h |   2 +-
 include/migration/vmstate.h  |  25 ++++-
 include/qemu/queue.h         |  39 ++++++++
 migration/migration.c        |  72 +++++++-------
 migration/migration.h        |   1 +
 migration/ram.c              | 185 ++++++++++++++++++++++++++---------
 migration/savevm.c           |  61 ++++++++----
 migration/trace-events       |   9 +-
 migration/vmstate-types.c    |  70 +++++++++++++
 stubs/vmstate.c              |   2 +-
 tests/qtest/migration-test.c |  97 +++++++++++++++++-
 tests/test-vmstate.c         | 170 ++++++++++++++++++++++++++++++++
 vl.c                         |  10 +-
 27 files changed, 654 insertions(+), 130 deletions(-)

-- 
2.24.1


Re: [PULL 00/30] Migration pull patches (take 4)

Posted by Peter Maydell 1 week ago
On Tue, 14 Jan 2020 at 12:53, Juan Quintela <quintela@redhat.com> wrote:
>
> The following changes since commit 3c8a6575985b1652b45bfa670b5e1907d642cfa0:
>
>   Merge remote-tracking branch 'remotes/kraxel/tags/usb-20200113-pull-request' into staging (2020-01-13 14:19:57 +0000)
>
> are available in the Git repository at:
>
>   https://github.com/juanquintela/qemu.git tags/migration-pull-pull-request
>
> for you to fetch changes up to b039b02c25d1725cf0296721d98e35e024468b20:
>
>   migration: Support QLIST migration (2020-01-14 13:45:29 +0100)
>
> ----------------------------------------------------------------
> Migration pull request (take 4)
>
> - both patches for x32 archs in
> - reorder the pages to make git bisect happy
>
> Thanks a lot to Daniel for helping with the bugs.  Twice.

tests/qtest/migration-test seems to hang in the aarch32 environment:

peter.m+  3380  0.1  0.0  13484 12848 ?        S    11:10   0:03
                   \_ make -C build/all-a32 check V=1 GCC_COLORS= -j9
peter.m+  5471  0.0  0.0   1460   388 ?        S    11:11   0:00
                       \_ /bin/sh -c
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RAND
peter.m+  5472  0.3  0.0  12984  2228 ?        Sl   11:11   0:09
                       |   \_ tests/qtest/migration-test -m=quick -k
--tap
peter.m+ 17798 99.8  0.9 646064 150512 ?       Sl   11:13  43:04
                       |   |   \_ aarch64-softmmu/qemu-system-aarch64
-qtest unix:
peter.m+ 17814  0.0  0.0      0     0 ?        Z    11:13   0:00
                       |   |   \_ [qemu-system-aar] <defunct>
peter.m+  5473  0.0  0.0   8176  6652 ?        S    11:11   0:00
                       |   \_ perl ./scripts/tap-driver.pl
--test-name=migration-t
peter.m+ 17200  0.0  0.0   1460   404 ?        S    11:13   0:00
                       \_ /bin/sh -c
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RAND
peter.m+ 17201  0.3  0.0  13080  2340 ?        Sl   11:13   0:08
                       |   \_ tests/qtest/migration-test -m=quick -k
--tap
peter.m+ 24275  100  0.8 549340 144160 ?       Sl   11:15  41:32
                       |   |   \_ i386-softmmu/qemu-system-i386 -qtest
unix:/tmp/q
peter.m+ 24308  0.0  0.0      0     0 ?        Z    11:15   0:00
                       |   |   \_ [qemu-system-i38] <defunct>
peter.m+ 17202  0.0  0.0   8172  6772 ?        S    11:13   0:00
                       |   \_ perl ./scripts/tap-driver.pl
--test-name=migration-t
peter.m+ 25399  0.0  0.0   1460   384 ?        S    11:15   0:00
                       \_ /bin/sh -c
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RAND
peter.m+ 25400  0.3  0.0  13092  2280 ?        Sl   11:15   0:08
                           \_ tests/qtest/migration-test -m=quick -k
--tap
peter.m+  2755  100  0.8 557600 145696 ?       Sl   11:17  39:20
                           |   \_ x86_64-softmmu/qemu-system-x86_64
-qtest unix:/t
peter.m+  2763  0.0  0.0      0     0 ?        Z    11:17   0:00
                           |   \_ [qemu-system-x86] <defunct>
peter.m+ 25401  0.0  0.0   8172  6692 ?        S    11:15   0:00
                           \_ perl ./scripts/tap-driver.pl
--test-name=migration-t

(processes started 45 mins ago, unreaped processes in zombie state.)

It looks like the multifd/tcp test fails, but doesn't manage to
actually turn the failure into the test case exiting:

/i386/migration/multifd/tcp: qemu-system-i386: -accel kvm: invalid
accelerator kvm
qemu-system-i386: falling back to tcg
qemu-system-i386: -accel kvm: invalid accelerator kvm
qemu-system-i386: falling back to tcg
qemu-system-i386: multifd_send_sync_main: multifd_send_pages fail
qemu-system-i386: failed to save SaveStateEntry with id(name): 3(ram)
qemu-system-i386: Unable to write to socket: Broken pipe
qemu-system-i386: Unknown combination of migration flags: 0
qemu-system-i386: error while loading state section id 3(ram)
qemu-system-i386: load of migration failed: Invalid argument
[hangs here]

I think you need to find a system which has 32-bit ram_addr_t
and test this, because this is about the fourth time round
for this patchset failing on this configuration.

thanks
-- PMM

Re: [PULL 00/30] Migration pull patches (take 4)

Posted by Juan Quintela 1 week ago
Peter Maydell <peter.maydell@linaro.org> wrote:
> On Tue, 14 Jan 2020 at 12:53, Juan Quintela <quintela@redhat.com> wrote:
>>
>> The following changes since commit 3c8a6575985b1652b45bfa670b5e1907d642cfa0:
>>
>>   Merge remote-tracking branch
>> 'remotes/kraxel/tags/usb-20200113-pull-request' into staging
>> (2020-01-13 14:19:57 +0000)
>>
>> are available in the Git repository at:
>>
>>   https://github.com/juanquintela/qemu.git tags/migration-pull-pull-request
>>
>> for you to fetch changes up to b039b02c25d1725cf0296721d98e35e024468b20:
>>
>>   migration: Support QLIST migration (2020-01-14 13:45:29 +0100)
>>
>> ----------------------------------------------------------------
>> Migration pull request (take 4)
>>
>> - both patches for x32 archs in
>> - reorder the pages to make git bisect happy
>>
>> Thanks a lot to Daniel for helping with the bugs.  Twice.
> (processes started 45 mins ago, unreaped processes in zombie state.)
>
> It looks like the multifd/tcp test fails, but doesn't manage to
> actually turn the failure into the test case exiting:
>
> /i386/migration/multifd/tcp: qemu-system-i386: -accel kvm: invalid
> accelerator kvm
> qemu-system-i386: falling back to tcg
> qemu-system-i386: -accel kvm: invalid accelerator kvm
> qemu-system-i386: falling back to tcg
> qemu-system-i386: multifd_send_sync_main: multifd_send_pages fail
> qemu-system-i386: failed to save SaveStateEntry with id(name): 3(ram)
> qemu-system-i386: Unable to write to socket: Broken pipe
> qemu-system-i386: Unknown combination of migration flags: 0
> qemu-system-i386: error while loading state section id 3(ram)
> qemu-system-i386: load of migration failed: Invalid argument
> [hangs here]
>
> I think you need to find a system which has 32-bit ram_addr_t
> and test this, because this is about the fourth time round
> for this patchset failing on this configuration.

That is arm32 bits at the moment, right?

Later, Juan.


Re: [PULL 00/30] Migration pull patches (take 4)

Posted by Juan Quintela 1 week ago
Juan Quintela <quintela@redhat.com> wrote:
> Peter Maydell <peter.maydell@linaro.org> wrote:
>> On Tue, 14 Jan 2020 at 12:53, Juan Quintela <quintela@redhat.com> wrote:
>>
>> I think you need to find a system which has 32-bit ram_addr_t
>> and test this, because this is about the fourth time round
>> for this patchset failing on this configuration.
>
> That is arm32 bits at the moment, right?

Not, i686 with XEN disabled appears to make the trick.

Guest what I had enabled?

Trying to see if I can reproduce it with it.

Sorry again, Juan.