[PATCH v12 00/10] migration: Modify 'migrate' and 'migrate-incoming' QAPI commands for migration

Het Gala posted 10 patches 7 months, 1 week ago
Failed in applying to current master (apply log)
Maintainers: Juan Quintela <quintela@redhat.com>, Peter Xu <peterx@redhat.com>, Fabiano Rosas <farosas@suse.de>, Leonardo Bras <leobras@redhat.com>, Li Zhijian <lizhijian@fujitsu.com>, Eric Blake <eblake@redhat.com>, Markus Armbruster <armbru@redhat.com>, Thomas Huth <thuth@redhat.com>, Laurent Vivier <lvivier@redhat.com>, Paolo Bonzini <pbonzini@redhat.com>
There is a newer version of this series
migration/exec.c               |  74 +++++++++----
migration/exec.h               |   8 +-
migration/migration-hmp-cmds.c |  17 ++-
migration/migration.c          | 189 +++++++++++++++++++++++++++------
migration/migration.h          |   3 +-
migration/rdma.c               |  34 +++---
migration/rdma.h               |   6 +-
migration/socket.c             |  39 ++-----
migration/socket.h             |   7 +-
qapi/migration.json            | 150 +++++++++++++++++++++++++-
softmmu/vl.c                   |   2 +-
tests/qtest/migration-test.c   |   7 +-
12 files changed, 412 insertions(+), 124 deletions(-)
[PATCH v12 00/10] migration: Modify 'migrate' and 'migrate-incoming' QAPI commands for migration
Posted by Het Gala 7 months, 1 week ago
This is v12 patchset of modified 'migrate' and 'migrate-incoming' QAPI design
for upstream review.

Would like to thank all the maintainers that actively participated in the v11
patchset discussion and gave insightful suggestions to improve the patches.

Link to previous upstream community patchset links:
v1: https://lists.gnu.org/archive/html/qemu-devel/2022-12/msg04339.html
v2: https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg02106.html
v3: https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg02473.html
v4: https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg03064.html
v5: https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg04845.html
v6: https://lists.gnu.org/archive/html/qemu-devel/2023-06/msg01251.html
v7: https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg02027.html
v8: https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg02770.html
v9: https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg04216.html
v10: https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05022.html
v11: https://lists.gnu.org/archive/html/qemu-devel/2023-10/msg00740.html

v11 -> v12 changelog:
-------------------
- Resolved double-freeing when using g_autoptr in structures that are
  nested into another.
- Overwriting of pointers to an existing allocated memory is solved at
  various places.
- Use of g_autoptr caused make check errors in non-error path while going
  out of scope inside function. Added g_steal_pointer() in the non-error
  paths wherever required.

Abstract:
---------

Current QAPI 'migrate' command design (for initiating a migration
stream) contains information regarding different migrate transport mechanism
(tcp / unix / exec), dest-host IP address, and binding port number in form of
a string. Thus the design does seem to have some design issues. Some of the
issues, stated below are:

1. Use of string URIs is a data encoding scheme within a data encoding scheme.
   QEMU code should directly be able to work with the results from QAPI,
   without resorting to do a second level of parsing (eg. socket_parse()).
2. For features / parameters related to migration, the migration tunables needs
   to be defined and updated upfront. For example, 'migrate-set-capability'
   and 'migrate-set-parameter' is required to enable multifd capability and
   multifd-number of channels respectively. Instead, 'Multifd-channels' can
   directly be represented as a single additional parameter to 'migrate'
   QAPI. 'migrate-set-capability' and 'migrate-set-parameter' commands could
   be used for runtime tunables that need setting after migration has already
   started.

The current patchset focuses on solving the first problem of multi-level
encoding of URIs. The patch defines 'migrate' command as a QAPI discriminated
union for the various transport backends (like socket, exec and rdma), and on
basis of transport backends, different migration parameters are defined.

(uri) string -->  (channel) Channel-type
                            Transport-type
                            Migration parameters based on transport type
------------------------------------------------------------------------------

Het Gala (10):
  migration: New QAPI type 'MigrateAddress'
  migration: convert migration 'uri' into 'MigrateAddress'
  migration: convert socket backend to accept MigrateAddress
  migration: convert rdma backend to accept MigrateAddress
  migration: convert exec backend to accept MigrateAddress.
  migration: New migrate and migrate-incoming argument 'channels'
  migration: modify migration_channels_and_uri_compatible() for new QAPI
    syntax
  migration: Implement MigrateChannelList to qmp migration flow.
  migration: Implement MigrateChannelList to hmp migration flow.
  migration: modify test_multifd_tcp_none() to use new QAPI syntax.

 migration/exec.c               |  74 +++++++++----
 migration/exec.h               |   8 +-
 migration/migration-hmp-cmds.c |  17 ++-
 migration/migration.c          | 189 +++++++++++++++++++++++++++------
 migration/migration.h          |   3 +-
 migration/rdma.c               |  34 +++---
 migration/rdma.h               |   6 +-
 migration/socket.c             |  39 ++-----
 migration/socket.h             |   7 +-
 qapi/migration.json            | 150 +++++++++++++++++++++++++-
 softmmu/vl.c                   |   2 +-
 tests/qtest/migration-test.c   |   7 +-
 12 files changed, 412 insertions(+), 124 deletions(-)

-- 
2.22.3
Re: [PATCH v12 00/10] migration: Modify 'migrate' and 'migrate-incoming' QAPI commands for migration
Posted by Fabiano Rosas 7 months, 1 week ago
Het Gala <het.gala@nutanix.com> writes:

> This is v12 patchset of modified 'migrate' and 'migrate-incoming' QAPI design
> for upstream review.
>
> Would like to thank all the maintainers that actively participated in the v11
> patchset discussion and gave insightful suggestions to improve the patches.
>
> Link to previous upstream community patchset links:
> v1: https://lists.gnu.org/archive/html/qemu-devel/2022-12/msg04339.html
> v2: https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg02106.html
> v3: https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg02473.html
> v4: https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg03064.html
> v5: https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg04845.html
> v6: https://lists.gnu.org/archive/html/qemu-devel/2023-06/msg01251.html
> v7: https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg02027.html
> v8: https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg02770.html
> v9: https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg04216.html
> v10: https://lists.gnu.org/archive/html/qemu-devel/2023-07/msg05022.html
> v11: https://lists.gnu.org/archive/html/qemu-devel/2023-10/msg00740.html
>
> v11 -> v12 changelog:
> -------------------
> - Resolved double-freeing when using g_autoptr in structures that are
>   nested into another.
> - Overwriting of pointers to an existing allocated memory is solved at
>   various places.
> - Use of g_autoptr caused make check errors in non-error path while going
>   out of scope inside function. Added g_steal_pointer() in the non-error
>   paths wherever required.

Please run make check before sending:
▶ 242/355 qcow2 181  FAIL

You can also push your code to your gitlab and run a pipeline with the
branch.
Re: [PATCH v12 00/10] migration: Modify 'migrate' and 'migrate-incoming' QAPI commands for migration
Posted by Het Gala 7 months, 1 week ago
On 10/10/2023 2:34 AM, Fabiano Rosas wrote:
> Het Gala<het.gala@nutanix.com>  writes:
>
>> This is v12 patchset of modified 'migrate' and 'migrate-incoming' QAPI design
>> for upstream review.
>>
>> Would like to thank all the maintainers that actively participated in the v11
>> patchset discussion and gave insightful suggestions to improve the patches.
>>
>> Link to previous upstream community patchset links:
>> v1:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2022-2D12_msg04339.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=OXhp-cq93AZ1ZRIwKL5wXhx5-8ei7RfBdmmbU9KNDfg&e=  
>> v2:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2023-2D02_msg02106.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=r7SYaB2fxLcEP2DiHslsbEyaY7ZPrXxageSBRtRZ7TA&e=  
>> v3:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2023-2D02_msg02473.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=fnGhJw56ypUavnslnUL6JeK4OLi7xwh7TGsafaSSZvw&e=  
>> v4:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2023-2D05_msg03064.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=SA4q18GEE4q3Eh7sy5nhQ8OZO5KM8kfapiBkSPZYDxE&e=  
>> v5:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2023-2D05_msg04845.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=QRW_aAGHmTBajBnu1a4jcxQFZ1lf1N3RCyLgOt82Ji4&e=  
>> v6:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2023-2D06_msg01251.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=7Dmgm47UdQ0h0Y9-XffsUW_ZdeQ-eCCVzhUkigTMKbc&e=  
>> v7:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2023-2D07_msg02027.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=8a3tAfIJ-6st1tlYkbjsRWEv-JvEFxokXzanf0WCqzw&e=  
>> v8:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2023-2D07_msg02770.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=4q_F05ZPdhWsPJ0fa850gHS90AsVX7MbsaIHi-3OsMI&e=  
>> v9:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2023-2D07_msg04216.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=1wNhJSfSvAoadG06F2JKFHZ2mA4QWSgqvYpt1zRX9qw&e=  
>> v10:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2023-2D07_msg05022.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=guEm3FuFn7jutT4ZB4RlgwttD4IMSBJy1MNh2zp3tYY&e=  
>> v11:https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gnu.org_archive_html_qemu-2Ddevel_2023-2D10_msg00740.html&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=-qwZZzrw4EKSsq0BK7MBd3wW1WEpXmJeng3ZUT5uBCg&m=s4FkORPOMqBTxCrkaYO6frUTsbcGM6c3K4FCdGaNAKeCIZ2GS9VaQxJGytQv0mGv&s=W7rbQhebtuWtT2ydMuG21OOkbqlh9KxMi1ZM5yZP6Ig&e=  
>>
>> v11 -> v12 changelog:
>> -------------------
>> - Resolved double-freeing when using g_autoptr in structures that are
>>    nested into another.
>> - Overwriting of pointers to an existing allocated memory is solved at
>>    various places.
>> - Use of g_autoptr caused make check errors in non-error path while going
>>    out of scope inside function. Added g_steal_pointer() in the non-error
>>    paths wherever required.
> Please run make check before sending:
> ▶ 242/355 qcow2 181  FAIL
>
> You can also push your code to your gitlab and run a pipeline with the
> branch.

I tested on my setup, and it passed all the checks.

[root@06f1b38a5d6a build]# make check
/rpmbuild/SOURCES/qemu/build/pyvenv/bin/meson test  --no-rebuild -t 0  
--num-processes 1 --print-errorlogs
   1/348 qemu:qtest+qtest-x86_64 / 
qtest-x86_64/qom-test                        OK 14.00s   8 subtests passed
   2/348 qemu:qtest+qtest-x86_64 / 
qtest-x86_64/migration-test                  OK 82.28s   13 subtests passed
   3/348 qemu:qtest+qtest-x86_64 / 
qtest-x86_64/bios-tables-test                OK 79.38s   51 subtests passed
   4/348 qemu:qtest+qtest-x86_64 / 
qtest-x86_64/test-hmp                        OK 5.67s   9 subtests passed
   5/348 qemu:qtest+qtest-x86_64 / 
qtest-x86_64/ahci-test                       OK 21.16s   74 subtests passed
   6/348 qemu:qtest+qtest-x86_64 / 
qtest-x86_64/qos-test                        OK 34.48s   119 subtests passed
   7/348 qemu:unit / check-block-qdict OK              0.01s   10 
subtests passed
   8/348 qemu:unit / check-qdict OK              0.01s   15 subtests passed
   9/348 qemu:unit / check-qnum OK              0.01s   8 subtests passed
  10/348 qemu:unit / check-qstring OK              0.01s   4 subtests passed
  11/348 qemu:unit / check-qlist OK              0.01s   4 subtests passed
[...]
340/348 qemu:softfloat+softfloat-compare / 
fp-test-lt_quiet                    OK              0.10s
341/348 qemu:softfloat+softfloat-ops / 
fp-test-add                             OK              1.61s
342/348 qemu:softfloat+softfloat-ops / 
fp-test-sub                             OK              1.55s
343/348 qemu:softfloat+softfloat-ops / 
fp-test-mul                             OK              7.81s
344/348 qemu:softfloat+softfloat-ops / 
fp-test-div                             OK              6.84s
345/348 qemu:softfloat+softfloat-ops / 
fp-test-rem                             OK              4.24s
346/348 qemu:softfloat+softfloat-ops / 
fp-test-sqrt                            OK              0.10s
347/348 qemu:softfloat+softfloat-ops / 
fp-test-log2                            OK              0.25s
348/348 qemu:qapi-schema+qapi-frontend / QAPI schema regression 
tests          OK              0.20s

Ok:                 339
Expected Fail:      0
Fail:               0
Unexpected Pass:    0
Skipped:            9
Timeout:            0

I am not sure if something is different in the configuration. I have 
never run a pipeline on gitlab though.  Can you point me out to the 
documentation of gitlab once again

Regards,
Het Gala