[PULL 00/34] Migration patches for 2024-09-04

Fabiano Rosas posted 34 patches 2 months, 2 weeks ago
migration/file.c                     |   3 +-
migration/file.h                     |   2 +-
migration/meson.build                |   1 +
migration/multifd-nocomp.c           | 389 +++++++++++++++++++
migration/multifd-qpl.c              |  79 +---
migration/multifd-uadk.c             | 104 ++---
migration/multifd-zero-page.c        |  13 +-
migration/multifd-zlib.c             |  99 ++---
migration/multifd-zstd.c             |  98 +----
migration/multifd.c                  | 559 +++++----------------------
migration/multifd.h                  | 152 ++++++--
migration/ram.c                      |  10 +-
migration/savevm.c                   |  10 +-
migration/trace-events               |   9 +-
target/ppc/machine.c                 |   2 +-
tests/qtest/libqtest.c               |  17 +-
tests/qtest/libqtest.h               |   2 -
tests/qtest/migration-helpers.c      |  20 +-
tests/qtest/migration-test.c         | 114 +-----
tests/unit/crypto-tls-x509-helpers.c |  13 +-
tests/unit/crypto-tls-x509-helpers.h |   6 +
21 files changed, 772 insertions(+), 930 deletions(-)
create mode 100644 migration/multifd-nocomp.c
[PULL 00/34] Migration patches for 2024-09-04
Posted by Fabiano Rosas 2 months, 2 weeks ago
The following changes since commit e638d685ec2a0700fb9529cbd1b2823ac4120c53:

  Open 9.2 development tree (2024-09-03 09:18:43 -0700)

are available in the Git repository at:

  https://gitlab.com/farosas/qemu.git tags/migration-20240904-pull-request

for you to fetch changes up to d41c9896f49076d1eaaa32214bd2296bd36d866c:

  tests/qtest/migration: Add a check for the availability of the "pc" machine (2024-09-03 16:24:37 -0300)

----------------------------------------------------------------
Migration pull request

- Steve's cleanup of unused variable
- Peter Maydell's fixes for several leaks in migration-test
- Fabiano's flexibilization of multifd data structures for device
  state migration
- Arman Nabiev's fix for ppc e500 migration
- Thomas' fix for migration-test vs. --without-default-devices

----------------------------------------------------------------

Arman Nabiev (1):
  target/ppc: Fix migration of CPUs with TLB_EMB TLB type

Fabiano Rosas (22):
  tests/qtest/migration: Remove vmstate-static-checker test
  migration/multifd: Reduce access to p->pages
  migration/multifd: Inline page_size and page_count
  migration/multifd: Remove pages->allocated
  migration/multifd: Pass in MultiFDPages_t to file_write_ramblock_iov
  migration/multifd: Introduce MultiFDSendData
  migration/multifd: Make MultiFDPages_t:offset a flexible array member
  migration/multifd: Replace p->pages with an union pointer
  migration/multifd: Move pages accounting into
    multifd_send_zero_page_detect()
  migration/multifd: Remove total pages tracing
  migration/multifd: Isolate ram pages packet data
  migration/multifd: Don't send ram data during SYNC
  migration/multifd: Replace multifd_send_state->pages with client data
  migration/multifd: Allow multifd sync without flush
  migration/multifd: Standardize on multifd ops names
  migration/multifd: Register nocomp ops dynamically
  migration/multifd: Move nocomp code into multifd-nocomp.c
  migration/multifd: Make MultiFDMethods const
  migration/multifd: Stop changing the packet on recv side
  migration/multifd: Fix p->iov leak in multifd-uadk.c
  migration/multifd: Add a couple of asserts for p->iov
  migration/multifd: Add documentation for multifd methods

Peter Maydell (9):
  tests/qtest/migration-test: Fix bootfile cleanup handling
  tests/qtest/migration-test: Don't leak resp in
    multifd_mapped_ram_fdset_end()
  tests/qtest/migration-test: Fix leaks in calc_dirtyrate_ready()
  tests/qtest/migration-helpers: Fix migrate_get_socket_address() leak
  tests/qtest/migration-test: Free QCRyptoTLSTestCertReq objects
  tests/unit/crypto-tls-x509-helpers: deinit privkey in test_tls_cleanup
  tests/qtest/migration-helpers: Don't dup argument to qdict_put_str()
  tests/qtest/migration-test: Don't strdup in get_dirty_rate()
  tests/qtest/migration-test: Don't leak QTestState in
    test_multifd_tcp_cancel()

Steve Sistare (1):
  migration: delete unused parameter mis

Thomas Huth (1):
  tests/qtest/migration: Add a check for the availability of the "pc"
    machine

 migration/file.c                     |   3 +-
 migration/file.h                     |   2 +-
 migration/meson.build                |   1 +
 migration/multifd-nocomp.c           | 389 +++++++++++++++++++
 migration/multifd-qpl.c              |  79 +---
 migration/multifd-uadk.c             | 104 ++---
 migration/multifd-zero-page.c        |  13 +-
 migration/multifd-zlib.c             |  99 ++---
 migration/multifd-zstd.c             |  98 +----
 migration/multifd.c                  | 559 +++++----------------------
 migration/multifd.h                  | 152 ++++++--
 migration/ram.c                      |  10 +-
 migration/savevm.c                   |  10 +-
 migration/trace-events               |   9 +-
 target/ppc/machine.c                 |   2 +-
 tests/qtest/libqtest.c               |  17 +-
 tests/qtest/libqtest.h               |   2 -
 tests/qtest/migration-helpers.c      |  20 +-
 tests/qtest/migration-test.c         | 114 +-----
 tests/unit/crypto-tls-x509-helpers.c |  13 +-
 tests/unit/crypto-tls-x509-helpers.h |   6 +
 21 files changed, 772 insertions(+), 930 deletions(-)
 create mode 100644 migration/multifd-nocomp.c

-- 
2.35.3
Re: [PULL 00/34] Migration patches for 2024-09-04
Posted by Peter Maydell 2 months, 2 weeks ago
On Wed, 4 Sept 2024 at 13:49, Fabiano Rosas <farosas@suse.de> wrote:
>
> The following changes since commit e638d685ec2a0700fb9529cbd1b2823ac4120c53:
>
>   Open 9.2 development tree (2024-09-03 09:18:43 -0700)
>
> are available in the Git repository at:
>
>   https://gitlab.com/farosas/qemu.git tags/migration-20240904-pull-request
>
> for you to fetch changes up to d41c9896f49076d1eaaa32214bd2296bd36d866c:
>
>   tests/qtest/migration: Add a check for the availability of the "pc" machine (2024-09-03 16:24:37 -0300)
>
> ----------------------------------------------------------------
> Migration pull request
>
> - Steve's cleanup of unused variable
> - Peter Maydell's fixes for several leaks in migration-test
> - Fabiano's flexibilization of multifd data structures for device
>   state migration
> - Arman Nabiev's fix for ppc e500 migration
> - Thomas' fix for migration-test vs. --without-default-devices

Hi. This generates a bunch of new warning messages when running
"make check":

105/845 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test
           OK             256.17s   51
 subtests passed
――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
warning: fd: migration to a file is deprecated. Use file: instead.
warning: fd: migration to a file is deprecated. Use file: instead.
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

Can you investigate or suppress these, please?

I also see a complaint from the migration-compat-x86_64 job:
https://gitlab.com/qemu-project/qemu/-/jobs/7752621835

Traceback (most recent call last):
File "/builds/qemu-project/qemu/build-previous/scripts/vmstate-static-checker.py",
line 438, in <module>
sys.exit(main())
^^^^^^
File "/builds/qemu-project/qemu/build-previous/scripts/vmstate-static-checker.py",
line 395, in main
dest_data = json.load(args.dest)
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/json/__init__.py", line 293, in load
return loads(fp.read(),
^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
^^^^^^^^^^^^^^^^^^^^^^
json.decoder.JSONDecodeError: Unterminated string starting at: line
5085 column 7 (char 131064)
# Failed to run vmstate-static-checker.py
not ok 3 /x86_64/migration/vmstate-checker-script
Bail out!

I think this is probably a pre-existing failure, as
I also saw it on the previous pullreq:
https://gitlab.com/qemu-project/qemu/-/jobs/7751785881

But since this is a migration pullreq, could you have a look?

thanks
-- PMM
Re: [PULL 00/34] Migration patches for 2024-09-04
Posted by Fabiano Rosas 2 months, 2 weeks ago
Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 4 Sept 2024 at 13:49, Fabiano Rosas <farosas@suse.de> wrote:
>>
>> The following changes since commit e638d685ec2a0700fb9529cbd1b2823ac4120c53:
>>
>>   Open 9.2 development tree (2024-09-03 09:18:43 -0700)
>>
>> are available in the Git repository at:
>>
>>   https://gitlab.com/farosas/qemu.git tags/migration-20240904-pull-request
>>
>> for you to fetch changes up to d41c9896f49076d1eaaa32214bd2296bd36d866c:
>>
>>   tests/qtest/migration: Add a check for the availability of the "pc" machine (2024-09-03 16:24:37 -0300)
>>
>> ----------------------------------------------------------------
>> Migration pull request
>>
>> - Steve's cleanup of unused variable
>> - Peter Maydell's fixes for several leaks in migration-test
>> - Fabiano's flexibilization of multifd data structures for device
>>   state migration
>> - Arman Nabiev's fix for ppc e500 migration
>> - Thomas' fix for migration-test vs. --without-default-devices
>
> Hi. This generates a bunch of new warning messages when running
> "make check":
>
> 105/845 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test
>            OK             256.17s   51
>  subtests passed
> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> warning: fd: migration to a file is deprecated. Use file: instead.
> warning: fd: migration to a file is deprecated. Use file: instead.
> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
>
> Can you investigate or suppress these, please?

We did deprecate the feature. Not sure if qtest has a proper way of
silencing these. I'll take a look.

>
> I also see a complaint from the migration-compat-x86_64 job:
> https://gitlab.com/qemu-project/qemu/-/jobs/7752621835
>
> Traceback (most recent call last):
> File "/builds/qemu-project/qemu/build-previous/scripts/vmstate-static-checker.py",
> line 438, in <module>
> sys.exit(main())
> ^^^^^^
> File "/builds/qemu-project/qemu/build-previous/scripts/vmstate-static-checker.py",
> line 395, in main
> dest_data = json.load(args.dest)
> ^^^^^^^^^^^^^^^^^^^^
> File "/usr/lib64/python3.11/json/__init__.py", line 293, in load
> return loads(fp.read(),
> ^^^^^^^^^^^^^^^^
> File "/usr/lib64/python3.11/json/__init__.py", line 346, in loads
> return _default_decoder.decode(s)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/usr/lib64/python3.11/json/decoder.py", line 337, in decode
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/usr/lib64/python3.11/json/decoder.py", line 353, in raw_decode
> obj, end = self.scan_once(s, idx)
> ^^^^^^^^^^^^^^^^^^^^^^
> json.decoder.JSONDecodeError: Unterminated string starting at: line
> 5085 column 7 (char 131064)
> # Failed to run vmstate-static-checker.py
> not ok 3 /x86_64/migration/vmstate-checker-script
> Bail out!

This is a test that was committed by mistake. I removed it in this PR,
but the migration-compat job uses the previous QEMU version of the code,
so the test won't go away until the next release.

This test should not have been picked up as part of the migration-compat
job because we don't set the PYTHON variable there. The test has
something like:

const char *python = g_getenv("PYTHON");
if (!python) {
    g_test_skip("PYTHON variable not set");
    return;
}

In my fork the CI is green:
https://gitlab.com/farosas/qemu/-/pipelines/1438640697

I'll probably have to unset PYTHON for that job.

>
> I think this is probably a pre-existing failure, as
> I also saw it on the previous pullreq:
> https://gitlab.com/qemu-project/qemu/-/jobs/7751785881
>
> But since this is a migration pullreq, could you have a look?

Yes, the problem is not with this pull request. We'd be better off
merging this because it removes the faulty test.

>
> thanks
> -- PMM
Re: [PULL 00/34] Migration patches for 2024-09-04
Posted by Peter Maydell 2 months, 2 weeks ago
On Wed, 4 Sept 2024 at 13:49, Fabiano Rosas <farosas@suse.de> wrote:
>
> The following changes since commit e638d685ec2a0700fb9529cbd1b2823ac4120c53:
>
>   Open 9.2 development tree (2024-09-03 09:18:43 -0700)
>
> are available in the Git repository at:
>
>   https://gitlab.com/farosas/qemu.git tags/migration-20240904-pull-request
>
> for you to fetch changes up to d41c9896f49076d1eaaa32214bd2296bd36d866c:
>
>   tests/qtest/migration: Add a check for the availability of the "pc" machine (2024-09-03 16:24:37 -0300)
>
> ----------------------------------------------------------------
> Migration pull request
>
> - Steve's cleanup of unused variable
> - Peter Maydell's fixes for several leaks in migration-test
> - Fabiano's flexibilization of multifd data structures for device
>   state migration
> - Arman Nabiev's fix for ppc e500 migration
> - Thomas' fix for migration-test vs. --without-default-devices
>


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/9.2
for any user-visible changes.

-- PMM