Series comparison

-[PATCH v3 0/4] Allow to enable multifd and postcopy migration together
+[PATCH v6 0/4] Allow to enable multifd and postcopy migration together
 From: Prasad Pandit <pjp@fedoraproject.org>
 Hello,
+* This series (v6) shuts down Multifd threads before starting Postcopy
+  migration. It helps to avoid an issue of multifd pages arriving late
+  at the destination during Postcopy phase and corrupting the vCPU
+  state. It also reorders the qtest patches and does some refactoring
+  changes as suggested in previous review.
+===
+/67 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test                 OK             161.35s   73 subtests passed
+===
+v5: https://lore.kernel.org/qemu-devel/20250205122712.229151-1-ppandit@redhat.com/T/#t
+* This series (v5) consolidates migration capabilities setting in one
+  'set_migration_capabilities()' function, thus simplifying test sources.
+  It passes all migration tests.
+===
+/66 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test                 OK             143.66s   71 subtests passed
+===
+v4: https://lore.kernel.org/qemu-devel/20250127120823.144949-1-ppandit@redhat.com/T/#t
+* This series (v4) adds more 'multifd+postcopy' qtests which test
+  Precopy migration with 'postcopy-ram' attribute set. And run
+  Postcopy migrations with 'multifd' channels enabled.
+===
+$ ../qtest/migration-test --tap -k -r '/x86_64/migration/multifd+postcopy' | grep -i 'slow test'
+# slow test /x86_64/migration/multifd+postcopy/plain executed in 1.29 secs
+# slow test /x86_64/migration/multifd+postcopy/recovery/tls/psk executed in 2.48 secs
+# slow test /x86_64/migration/multifd+postcopy/preempt/plain executed in 1.49 secs
+# slow test /x86_64/migration/multifd+postcopy/preempt/recovery/tls/psk executed in 2.52 secs
+# slow test /x86_64/migration/multifd+postcopy/tcp/tls/psk/match executed in 3.62 secs
+# slow test /x86_64/migration/multifd+postcopy/tcp/plain/zstd executed in 1.34 secs
+# slow test /x86_64/migration/multifd+postcopy/tcp/plain/cancel executed in 2.24 secs
+...
+/66 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test                 OK             148.41s   71 subtests passed
+===
+v3: https://lore.kernel.org/qemu-devel/20250121131032.1611245-1-ppandit@redhat.com/T/#t
+* This series (v3) passes all existing 'tests/qtest/migration/*' tests
+  and adds a new one to enable multifd channels with postcopy migration.
+v2: https://lore.kernel.org/qemu-devel/20241129122256.96778-1-ppandit@redhat.com/T/#u
+* This series (v2) further refactors the 'ram_save_target_page'
+  function to make it independent of the multifd & postcopy change.
+v1: https://lore.kernel.org/qemu-devel/20241126115748.118683-1-ppandit@redhat.com/T/#u
+* This series removes magic value (4-bytes) introduced in the
+  previous series for the Postcopy channel.
+v0: https://lore.kernel.org/qemu-devel/20241029150908.1136894-1-ppandit@redhat.com/T/#u
 * Currently Multifd and Postcopy migration can not be used together.
   QEMU shows "Postcopy is not yet compatible with multifd" message.
   When migrating guests with large (100's GB) RAM, Multifd threads
   help to accelerate migration, but inability to use it with the
 ...
   migration together. Precopy and Multifd threads work during
   the initial guest (RAM) transfer. When migration moves to the
   Postcopy phase, Multifd threads are restrained and the Postcopy
   threads start to request pages from the source side.
-* This series removes magic value (4-bytes) introduced in the
+* This series introduces magic value (4-bytes) to be sent on the
-  previous series for the Postcopy channel. And refactoring of
+  Postcopy channel. It helps to differentiate channels and properly
-  the 'ram_save_target_page' function is made independent of
+  setup incoming connections on the destination side.
   the multifd & postcopy change.
 * This series passes all existing 'tests/qtest/migration/*' test
   cases and adds a new one to enable multifd channels with postcopy
   migration.
   v2: https://lore.kernel.org/qemu-devel/20241129122256.96778-1-ppandit@redhat.com/T/#u
   v1: https://lore.kernel.org/qemu-devel/20241126115748.118683-1-ppandit@redhat.com/T/#u
   v0: https://lore.kernel.org/qemu-devel/20241029150908.1136894-1-ppandit@redhat.com/T/#u
 Thank you.
 ---
 Prasad Pandit (4):
   migration/multifd: move macros to multifd header
-  migration: refactor ram_save_target_page functions
   migration: enable multifd and postcopy together
-  tests/qtest/migration: add postcopy test with multifd
+  tests/qtest/migration: consolidate set capabilities
   tests/qtest/migration: add postcopy tests with multifd
- migration/migration.c                  | 106 +++++++++++++++----------
+ migration/migration.c                     | 107 ++++++++++++----------
- migration/multifd-nocomp.c             |   3 +-
+ migration/multifd-nocomp.c                |   3 +-
- migration/multifd.c                    |   5 --
+ migration/multifd.c                       |   9 +-
- migration/multifd.h                    |   5 ++
+ migration/multifd.h                       |   5 +
- migration/options.c                    |   5 --
+ migration/options.c                       |   5 -
- migration/ram.c                        |  69 +++++-----------
+ migration/ram.c                           |   7 +-
- tests/qtest/migration/framework.c      |   8 ++
+ tests/qtest/migration/compression-tests.c |  23 ++++-
- tests/qtest/migration/framework.h      |   3 +
+ tests/qtest/migration/cpr-tests.c         |   4 +-
- tests/qtest/migration/postcopy-tests.c |  10 +++
+ tests/qtest/migration/file-tests.c        |  44 +++------
-files changed, 112 insertions(+), 102 deletions(-)
+ tests/qtest/migration/framework.c         |  58 +++++++++---
  tests/qtest/migration/framework.h         |   4 +-
  tests/qtest/migration/postcopy-tests.c    |  27 +++++-
  tests/qtest/migration/precopy-tests.c     |  38 +++++---
  tests/qtest/migration/tls-tests.c         |  51 ++++++++++-
 files changed, 247 insertions(+), 138 deletions(-)
 --
 .48.1

-[PATCH v3 1/4] migration/multifd: move macros to multifd header
+[PATCH v6 1/4] migration/multifd: move macros to multifd header
 From: Prasad Pandit <pjp@fedoraproject.org>
 Move MULTIFD_ macros to the header file so that
 they are accessible from other source files.
+Reviewed-by: Fabiano Rosas <farosas@suse.de>
 Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
 ---
  migration/multifd.c | 5 -----
  migration/multifd.h | 5 +++++
 files changed, 5 insertions(+), 5 deletions(-)
-v2:
+v5: no change
- - https://lore.kernel.org/qemu-devel/20241129122256.96778-1-ppandit@redhat.com/
+- https://lore.kernel.org/qemu-devel/20250205122712.229151-1-ppandit@redhat.com/T/#t
 diff --git a/migration/multifd.c b/migration/multifd.c
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/multifd.c
 +++ b/migration/multifd.c
 ...
 +#define MULTIFD_VERSION 1
 +
  /* Multifd Compression flags */
  #define MULTIFD_FLAG_SYNC (1 << 0)
 --
 .48.1

-[PATCH v3 3/4] migration: enable multifd and postcopy together
+[PATCH v6 2/4] migration: enable multifd and postcopy together
 ...
 channels.
 The Precopy and Multifd threads work during the
 initial guest RAM transfer. When migration moves
 to the Postcopy phase, the multifd threads are
-restrained and Postcopy threads on the destination
+shutdown and Postcopy threads on the destination
 request/pull data from the source side.
 Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
 ---
- migration/migration.c      | 106 +++++++++++++++++++++++--------------
+ migration/migration.c      | 107 ++++++++++++++++++++-----------------
  migration/multifd-nocomp.c |   3 +-
+ migration/multifd.c        |   4 +-
  migration/options.c        |   5 --
- migration/ram.c            |   4 +-
+ migration/ram.c            |   7 ++-
-files changed, 70 insertions(+), 48 deletions(-)
+files changed, 64 insertions(+), 62 deletions(-)
-v2: Minor changes in migration_ioc_process_incoming() to pass test cases
+v6:
- - https://lore.kernel.org/qemu-devel/20241129122256.96778-1-ppandit@redhat.com/
+- Shutdown multifd threads before postcopy_start()
 - Reorder tests/qtest/migration/ patches
 - Some refactoring of functions
 v5:
 - https://lore.kernel.org/qemu-devel/20250205122712.229151-1-ppandit@redhat.com/T/#t
 diff --git a/migration/migration.c b/migration/migration.c
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/migration.c
 +++ b/migration/migration.c
 @@ -XXX,XX +XXX,XX @@ enum mig_rp_message_type {
      MIG_RP_MSG_MAX
  };
 +/* Migration channel types */
-+enum { CH_DEFAULT, CH_MULTIFD, CH_POSTCOPY };
++enum { CH_MAIN, CH_MULTIFD, CH_POSTCOPY };
 +
  /* When we add fault tolerance, we could have several
     migrations at once.  For now we don't need to add
     dynamic creation of migration */
 @@ -XXX,XX +XXX,XX @@ void migration_fd_process_incoming(QEMUFile *f)
- /*
+     migration_incoming_process();
-  * Returns true when we want to start a new incoming migration process,
+ }
-  * false otherwise.
-+ *
+-/*
-+ * All the required channels must be in place before a new incoming
+- * Returns true when we want to start a new incoming migration process,
-+ * migration process starts.
+- * false otherwise.
-+ *  - Multifd enabled:
+- */
 + *    The main channel and the multifd channels are required.
 + *  - Multifd/Postcopy disabled:
 + *    The main channel is required.
 + *  - Postcopy enabled:
 + *    We don't want to start a new incoming migration when
 + *    the postcopy channel is created. Because it is created
 + *    towards the end of the precopy migration.
 + *
   */
 -static bool migration_should_start_incoming(bool main_channel)
-+static bool migration_should_start_incoming(uint8_t channel)
++static bool migration_has_main_and_multifd_channels(void)
  {
 -    /* Multifd doesn't start unless all channels are established */
 -    if (migrate_multifd()) {
 -        return migration_has_all_channels();
--    }
++    MigrationIncomingState *mis = migration_incoming_get_current();
-+    bool ret = false;
++    if (!mis->from_src_file) {
-+
++        /* main channel not established */
-+    if (channel != CH_POSTCOPY) {
++        return false;
-+        MigrationIncomingState *mis = migration_incoming_get_current();
+     }
 +        ret = mis->from_src_file ? true : false;
 -    /* Preempt channel only starts when the main channel is created */
 -    if (migrate_postcopy_preempt()) {
 -        return main_channel;
-+        if (ret && migrate_multifd()) {
++    if (migrate_multifd() && !multifd_recv_all_channels_created()) {
-+            ret = multifd_recv_all_channels_created();
++        return false;
-+        }
+     }
      }
 -    /*
 -     * For all the rest types of migration, we should only reach here when
 -     * it's the main channel that's being created, and we should always
 -     * proceed with this channel.
 -     */
 -    assert(main_channel);
--    return true;
++    /* main channel and all multifd channels are established */
-+    return ret;
+     return true;
  }
- void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
 @@ -XXX,XX +XXX,XX @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
      MigrationIncomingState *mis = migration_incoming_get_current();
      Error *local_err = NULL;
      QEMUFile *f;
 -    bool default_channel = true;
      uint32_t channel_magic = 0;
-+    uint8_t channel = CH_DEFAULT;
++    uint8_t channel = CH_MAIN;
      int ret = 0;
 -    if (migrate_multifd() && !migrate_mapped_ram() &&
 -        !migrate_postcopy_ram() &&
 -        qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_READ_MSG_PEEK)) {
-+    if (!migration_should_start_incoming(channel)) {
++    if (!migration_has_main_and_multifd_channels()) {
 +        if (qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_READ_MSG_PEEK)) {
          /*
           * With multiple channels, it is possible that we receive channels
           * out of order on destination side, causing incorrect mapping of
 @@ -XXX,XX +XXX,XX @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
 ...
 +            }
 -        if (ret != 0) {
 -            return;
 +            if (channel_magic == cpu_to_be32(QEMU_VM_FILE_MAGIC)) {
-+                channel = CH_DEFAULT;
++                channel = CH_MAIN;
 +            } else if (channel_magic == cpu_to_be32(MULTIFD_MAGIC)) {
 +                channel = CH_MULTIFD;
 +            } else if (!mis->from_src_file
 +                        && mis->state == MIGRATION_STATUS_POSTCOPY_PAUSED) {
 +                /* reconnect default channel for postcopy recovery */
-+                channel = CH_DEFAULT;
++                channel = CH_MAIN;
 +            } else {
-+                error_report("%s: could not identify channel, unknown magic: %u",
++                error_report("%s: unknown channel magic: %u",
 +                                __func__, channel_magic);
 +                return;
 +            }
 +        } else if (mis->from_src_file
-+            && (!strcmp(ioc->name, "migration-tls-incoming")
++                && (!strcmp(ioc->name, "migration-tls-incoming")
 +                || !strcmp(ioc->name, "migration-file-incoming"))) {
 +            channel = CH_MULTIFD;
          }
 -
 -        default_channel = (channel_magic == cpu_to_be32(QEMU_VM_FILE_MAGIC));
 -    } else {
 -        default_channel = !mis->from_src_file;
-+    } else if (mis->from_src_file) { // && migrate_postcopy_preempt()
++    } else if (mis->from_src_file) {
 +        channel = CH_POSTCOPY;
      }
      if (multifd_recv_setup(errp) != 0) {
          return;
      }
 -    if (default_channel) {
-+    if (channel == CH_DEFAULT) {
++    if (channel == CH_MAIN) {
          f = qemu_file_new_input(ioc);
          migration_incoming_setup(f);
 -    } else {
 +    } else if (channel == CH_MULTIFD) {
          /* Multiple connections */
 ...
 +        f = qemu_file_new_input(ioc);
 +        postcopy_preempt_new_channel(mis, f);
      }
 -    if (migration_should_start_incoming(default_channel)) {
-+    if (migration_should_start_incoming(channel)) {
++    if (channel != CH_POSTCOPY && migration_has_main_and_multifd_channels()) {
          /* If it's a recovery, we're done */
          if (postcopy_try_recover()) {
              return;
 @@ -XXX,XX +XXX,XX @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
   */
  bool migration_has_all_channels(void)
  {
-+    bool ret = false;
++    if (!migration_has_main_and_multifd_channels()) {
 +        return false;
 +    }
 +
      MigrationIncomingState *mis = migration_incoming_get_current();
+-
-     if (!mis->from_src_file) {
+-    if (!mis->from_src_file) {
--        return false;
++    if (migrate_postcopy_preempt() && !mis->postcopy_qemufile_dst) {
-+        return ret;
+         return false;
      }
-     if (migrate_multifd()) {
+-    if (migrate_multifd()) {
 -        return multifd_recv_all_channels_created();
-+        ret = multifd_recv_all_channels_created();
+-    }
-     }
+-
 -    if (migrate_postcopy_preempt()) {
 -        return mis->postcopy_qemufile_dst != NULL;
-+    if (ret && migrate_postcopy_preempt()) {
+-    }
-+        ret = mis->postcopy_qemufile_dst != NULL;
+-
-     }
+     return true;
 -    return true;
 +    return ret;
  }
- int migrate_send_rp_switchover_ack(MigrationIncomingState *mis)
+@@ -XXX,XX +XXX,XX @@ static void migrate_fd_cleanup(MigrationState *s)
      assert(!migration_is_active());
 +    file_cleanup_outgoing_migration();
 +    socket_cleanup_outgoing_migration();
      if (s->state == MIGRATION_STATUS_CANCELLING) {
          migrate_set_state(&s->state, MIGRATION_STATUS_CANCELLING,
                            MIGRATION_STATUS_CANCELLED);
@@ -XXX,XX +XXX,XX @@ static MigIterateState migration_iteration_run(MigrationState *s)
      }
      /* Still a significant amount to transfer */
 -    if (!in_postcopy && must_precopy <= s->threshold_size && can_switchover &&
 -        qatomic_read(&s->start_postcopy)) {
 +    if (!in_postcopy && must_precopy <= s->threshold_size
 +        && can_switchover && qatomic_read(&s->start_postcopy)) {
 +        if (migrate_multifd()) {
 +            multifd_send_shutdown();
 +        }
          if (postcopy_start(s, &local_err)) {
              migrate_set_error(s, local_err);
              error_report_err(local_err);
 diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/multifd-nocomp.c
 +++ b/migration/multifd-nocomp.c
 @@ -XXX,XX +XXX,XX @@
 ...
 -    if (!migrate_multifd()) {
 +    if (!migrate_multifd() || migration_in_postcopy()) {
          return 0;
      }
+diff --git a/migration/multifd.c b/migration/multifd.c
+index XXXXXXX..XXXXXXX 100644
+--- a/migration/multifd.c
++++ b/migration/multifd.c
+@@ -XXX,XX +XXX,XX @@ static bool multifd_send_cleanup_channel(MultiFDSendParams *p, Error **errp)
+ static void multifd_send_cleanup_state(void)
+ {
+-    file_cleanup_outgoing_migration();
+-    socket_cleanup_outgoing_migration();
+     qemu_sem_destroy(&multifd_send_state->channels_created);
+     qemu_sem_destroy(&multifd_send_state->channels_ready);
+     g_free(multifd_send_state->params);
+@@ -XXX,XX +XXX,XX @@ void multifd_send_shutdown(void)
+ {
+     int i;
+-    if (!migrate_multifd()) {
++    if (!migrate_multifd() || !multifd_send_state) {
+         return;
+     }
 diff --git a/migration/options.c b/migration/options.c
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/options.c
 +++ b/migration/options.c
 ...
 @@ -XXX,XX +XXX,XX @@ static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss)
          }
      }
 -    if (migrate_multifd()) {
+-        RAMBlock *block = pss->block;
+-        return ram_save_multifd_page(block, offset);
 +    if (migrate_multifd() && !migration_in_postcopy()) {
-         RAMBlock *block = pss->block;
++        return ram_save_multifd_page(pss->block, offset);
-         return ram_save_multifd_page(block, offset);
+     }
-     }
---
+     if (control_save_page(pss, offset, &res)) {
 --
 .48.1

-[PATCH v3 2/4] migration: refactor ram_save_target_page functions
+[PATCH v6 3/4] tests/qtest/migration: consolidate set capabilities
 From: Prasad Pandit <pjp@fedoraproject.org>
-Refactor ram_save_target_page legacy and multifd
+Migration capabilities are set in multiple '.start_hook'
-functions into one. Other than simplifying it,
+functions for various tests. Instead, consolidate setting
-it frees 'migration_ops' object from usage, so it
+capabilities in 'set_migration_capabilities()' function
-is expunged.
+which is called from various 'test_*_common()' functions.
 While simplifying the capabilities setting, it helps
 to declutter the qtest sources.
+Suggested-by: Fabiano Rosas <farosas@suse.de>
 Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
 ---
- migration/ram.c | 67 +++++++++++++------------------------------------
+ tests/qtest/migration/compression-tests.c | 10 ++--
-file changed, 17 insertions(+), 50 deletions(-)
+ tests/qtest/migration/cpr-tests.c         |  4 +-
  tests/qtest/migration/file-tests.c        | 44 +++++-------------
  tests/qtest/migration/framework.c         | 56 +++++++++++++++++------
  tests/qtest/migration/framework.h         |  4 +-
  tests/qtest/migration/postcopy-tests.c    |  4 +-
  tests/qtest/migration/precopy-tests.c     | 19 +++-----
  tests/qtest/migration/tls-tests.c         | 11 ++++-
 files changed, 80 insertions(+), 72 deletions(-)
-v2:
+v6:
- - https://lore.kernel.org/qemu-devel/20241129122256.96778-1-ppandit@redhat.com/
+- Reorder, make this the first qtest patch in this series.
-diff --git a/migration/ram.c b/migration/ram.c
+v5:
-index XXXXXXX..XXXXXXX 100644
+- https://lore.kernel.org/qemu-devel/20250205122712.229151-1-ppandit@redhat.com/T/#t
---- a/migration/ram.c
-+++ b/migration/ram.c
+diff --git a/tests/qtest/migration/compression-tests.c b/tests/qtest/migration/compression-tests.c
-@@ -XXX,XX +XXX,XX @@ void ram_transferred_add(uint64_t bytes)
+index XXXXXXX..XXXXXXX 100644
-     }
+--- a/tests/qtest/migration/compression-tests.c
- }
++++ b/tests/qtest/migration/compression-tests.c
+@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_zstd(void)
--struct MigrationOps {
+ {
--    int (*ram_save_target_page)(RAMState *rs, PageSearchStatus *pss);
+     MigrateCommon args = {
--};
+         .listen_uri = "defer",
--typedef struct MigrationOps MigrationOps;
++        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
--
+         .start_hook = migrate_hook_start_precopy_tcp_multifd_zstd,
--MigrationOps *migration_ops;
+     };
--
+     test_precopy_common(&args);
- static int ram_save_host_page_urgent(PageSearchStatus *pss);
+@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_qatzip(void)
+ {
- /* NOTE: page is the PFN not real ram_addr_t. */
+     MigrateCommon args = {
-@@ -XXX,XX +XXX,XX @@ int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len,
+         .listen_uri = "defer",
- }
++        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
+         .start_hook = migrate_hook_start_precopy_tcp_multifd_qatzip,
- /**
+     };
-- * ram_save_target_page_legacy: save one target page
+     test_precopy_common(&args);
-- *
+@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_qpl(void)
-- * Returns the number of pages written
+ {
-+ * ram_save_target_page: save one target page to the precopy thread
+     MigrateCommon args = {
-+ * OR to multifd workers.
+         .listen_uri = "defer",
-  *
++        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
-  * @rs: current RAM state
+         .start_hook = migrate_hook_start_precopy_tcp_multifd_qpl,
-  * @pss: data about the page we want to send
+     };
-  */
+     test_precopy_common(&args);
--static int ram_save_target_page_legacy(RAMState *rs, PageSearchStatus *pss)
+@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_uadk(void)
-+static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss)
+ {
- {
+     MigrateCommon args = {
-     ram_addr_t offset = ((ram_addr_t)pss->page) << TARGET_PAGE_BITS;
+         .listen_uri = "defer",
-     int res;
++        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
+         .start_hook = migrate_hook_start_precopy_tcp_multifd_uadk,
-+    if (!migrate_multifd()
+     };
-+        || migrate_zero_page_detection() == ZERO_PAGE_DETECTION_LEGACY) {
+     test_precopy_common(&args);
-+        if (save_zero_page(rs, pss, offset)) {
+@@ -XXX,XX +XXX,XX @@ migrate_hook_start_xbzrle(QTestState *from,
-+            return 1;
+                           QTestState *to)
  {
      migrate_set_parameter_int(from, "xbzrle-cache-size", 33554432);
 -
 -    migrate_set_capability(from, "xbzrle", true);
 -    migrate_set_capability(to, "xbzrle", true);
 -
      return NULL;
  }
@@ -XXX,XX +XXX,XX @@ static void test_precopy_unix_xbzrle(void)
          .listen_uri = uri,
          .start_hook = migrate_hook_start_xbzrle,
          .iterations = 2,
 +        .caps[MIGRATION_CAPABILITY_XBZRLE] = true,
          /*
           * XBZRLE needs pages to be modified when doing the 2nd+ round
           * iteration to have real data pushed to the stream.
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_zlib(void)
  {
      MigrateCommon args = {
          .listen_uri = "defer",
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
          .start_hook = migrate_hook_start_precopy_tcp_multifd_zlib,
      };
      test_precopy_common(&args);
 diff --git a/tests/qtest/migration/cpr-tests.c b/tests/qtest/migration/cpr-tests.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/cpr-tests.c
 +++ b/tests/qtest/migration/cpr-tests.c
@@ -XXX,XX +XXX,XX @@ static void *migrate_hook_start_mode_reboot(QTestState *from, QTestState *to)
      migrate_set_parameter_str(from, "mode", "cpr-reboot");
      migrate_set_parameter_str(to, "mode", "cpr-reboot");
 -    migrate_set_capability(from, "x-ignore-shared", true);
 -    migrate_set_capability(to, "x-ignore-shared", true);
 -
      return NULL;
  }
@@ -XXX,XX +XXX,XX @@ static void test_mode_reboot(void)
          .connect_uri = uri,
          .listen_uri = "defer",
          .start_hook = migrate_hook_start_mode_reboot,
 +        .caps[MIGRATION_CAPABILITY_X_IGNORE_SHARED] = true,
      };
      test_file_common(&args, true);
 diff --git a/tests/qtest/migration/file-tests.c b/tests/qtest/migration/file-tests.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/file-tests.c
 +++ b/tests/qtest/migration/file-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_precopy_file_offset_bad(void)
      test_file_common(&args, false);
  }
 -static void *migrate_hook_start_mapped_ram(QTestState *from,
 -                                           QTestState *to)
 -{
 -    migrate_set_capability(from, "mapped-ram", true);
 -    migrate_set_capability(to, "mapped-ram", true);
 -
 -    return NULL;
 -}
 -
  static void test_precopy_file_mapped_ram_live(void)
  {
      g_autofree char *uri = g_strdup_printf("file:%s/%s", tmpfs,
@@ -XXX,XX +XXX,XX @@ static void test_precopy_file_mapped_ram_live(void)
      MigrateCommon args = {
          .connect_uri = uri,
          .listen_uri = "defer",
 -        .start_hook = migrate_hook_start_mapped_ram,
 +        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
      };
      test_file_common(&args, false);
@@ -XXX,XX +XXX,XX @@ static void test_precopy_file_mapped_ram(void)
      MigrateCommon args = {
          .connect_uri = uri,
          .listen_uri = "defer",
 -        .start_hook = migrate_hook_start_mapped_ram,
 +        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
      };
      test_file_common(&args, true);
  }
 -static void *migrate_hook_start_multifd_mapped_ram(QTestState *from,
 -                                                   QTestState *to)
 -{
 -    migrate_hook_start_mapped_ram(from, to);
 -
 -    migrate_set_parameter_int(from, "multifd-channels", 4);
 -    migrate_set_parameter_int(to, "multifd-channels", 4);
 -
 -    migrate_set_capability(from, "multifd", true);
 -    migrate_set_capability(to, "multifd", true);
 -
 -    return NULL;
 -}
 -
  static void test_multifd_file_mapped_ram_live(void)
  {
      g_autofree char *uri = g_strdup_printf("file:%s/%s", tmpfs,
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram_live(void)
      MigrateCommon args = {
          .connect_uri = uri,
          .listen_uri = "defer",
 -        .start_hook = migrate_hook_start_multifd_mapped_ram,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
 +        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
      };
      test_file_common(&args, false);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram(void)
      MigrateCommon args = {
          .connect_uri = uri,
          .listen_uri = "defer",
 -        .start_hook = migrate_hook_start_multifd_mapped_ram,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
 +        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
      };
      test_file_common(&args, true);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram(void)
  static void *migrate_hook_start_multifd_mapped_ram_dio(QTestState *from,
                                                         QTestState *to)
  {
 -    migrate_hook_start_multifd_mapped_ram(from, to);
 -
      migrate_set_parameter_bool(from, "direct-io", true);
      migrate_set_parameter_bool(to, "direct-io", true);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram_dio(void)
          .connect_uri = uri,
          .listen_uri = "defer",
          .start_hook = migrate_hook_start_multifd_mapped_ram_dio,
 +        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
      };
      if (!probe_o_direct_support(tmpfs)) {
@@ -XXX,XX +XXX,XX @@ static void *migrate_hook_start_multifd_mapped_ram_fdset_dio(QTestState *from,
      fdset_add_fds(from, file, O_WRONLY, 2, true);
      fdset_add_fds(to, file, O_RDONLY, 2, true);
 -    migrate_hook_start_multifd_mapped_ram(from, to);
      migrate_set_parameter_bool(from, "direct-io", true);
      migrate_set_parameter_bool(to, "direct-io", true);
@@ -XXX,XX +XXX,XX @@ static void *migrate_hook_start_multifd_mapped_ram_fdset(QTestState *from,
      fdset_add_fds(from, file, O_WRONLY, 2, false);
      fdset_add_fds(to, file, O_RDONLY, 2, false);
 -    migrate_hook_start_multifd_mapped_ram(from, to);
 -
      return NULL;
  }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram_fdset(void)
          .listen_uri = "defer",
          .start_hook = migrate_hook_start_multifd_mapped_ram_fdset,
          .end_hook = migrate_hook_end_multifd_mapped_ram_fdset,
 +        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
      };
      test_file_common(&args, true);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram_fdset_dio(void)
          .listen_uri = "defer",
          .start_hook = migrate_hook_start_multifd_mapped_ram_fdset_dio,
          .end_hook = migrate_hook_end_multifd_mapped_ram_fdset,
 +        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
      };
      if (!probe_o_direct_support(tmpfs)) {
 diff --git a/tests/qtest/migration/framework.c b/tests/qtest/migration/framework.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/framework.c
 +++ b/tests/qtest/migration/framework.c
@@ -XXX,XX +XXX,XX @@ static QList *migrate_start_get_qmp_capabilities(const MigrateStart *args)
      return capabilities;
  }
 +static void set_migration_capabilities(QTestState *from,
 +                                        QTestState *to, MigrateCommon *args)
 +{
 +    /*
 +     * MigrationCapability_lookup and MIGRATION_CAPABILITY_ constants
 +     * are from qapi-types-migration.h.
 +     */
 +    for (uint8_t i = 0; i < MIGRATION_CAPABILITY__MAX; i++)
 +    {
 +        if (!args->caps[i]) {
 +            continue;
 +        }
 +        if (from) {
 +            migrate_set_capability(from,
 +                            MigrationCapability_lookup.array[i], true);
 +        }
 +        if (to) {
 +            migrate_set_capability(to,
 +                            MigrationCapability_lookup.array[i], true);
 +        }
 +    }
 +
-+    if (migrate_multifd()) {
++    return;
-+        RAMBlock *block = pss->block;
++}
-+        return ram_save_multifd_page(block, offset);
++
  int migrate_start(QTestState **from, QTestState **to, const char *uri,
                    MigrateStart *args)
  {
@@ -XXX,XX +XXX,XX @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
          args->postcopy_data = args->start_hook(from, to);
      }
 -    migrate_set_capability(from, "postcopy-ram", true);
 -    migrate_set_capability(to, "postcopy-ram", true);
 -    migrate_set_capability(to, "postcopy-blocktime", true);
 -
 -    if (args->postcopy_preempt) {
 -        migrate_set_capability(from, "postcopy-preempt", true);
 -        migrate_set_capability(to, "postcopy-preempt", true);
 -    }
 +    /* set postcopy capabilities */
 +    args->caps[MIGRATION_CAPABILITY_POSTCOPY_BLOCKTIME] = true;
 +    args->caps[MIGRATION_CAPABILITY_POSTCOPY_RAM] = true;
 +    set_migration_capabilities(from, to, args);
      migrate_ensure_non_converge(from);
 -
      migrate_prepare_for_dirty_mem(from);
      qtest_qmp_assert_success(to, "{ 'execute': 'migrate-incoming',"
                               "  'arguments': { "
@@ -XXX,XX +XXX,XX @@ void test_precopy_common(MigrateCommon *args)
          return;
      }
 +    set_migration_capabilities(from, to, args);
 +    if (args->caps[MIGRATION_CAPABILITY_MULTIFD]) {
 +        migrate_set_parameter_int(from, "multifd-channels", 16);
 +        migrate_set_parameter_int(to, "multifd-channels", 16);
 +    }
 +
-     if (control_save_page(pss, offset, &res)) {
+     if (args->start_hook) {
-         return res;
+         data_hook = args->start_hook(from, to);
      }
+@@ -XXX,XX +XXX,XX @@ void test_file_common(MigrateCommon *args, bool stop_src)
--    if (save_zero_page(rs, pss, offset)) {
+      */
--        return 1;
+     g_assert_false(args->live);
--    }
--
++    set_migration_capabilities(from, to, args);
-     return ram_save_page(rs, pss);
++    if (args->caps[MIGRATION_CAPABILITY_MULTIFD]) {
- }
++        migrate_set_parameter_int(from, "multifd-channels", 4);
++        migrate_set_parameter_int(to, "multifd-channels", 4);
--/**
++    }
-- * ram_save_target_page_multifd: send one target page to multifd workers
++
-- *
+     if (g_strrstr(args->connect_uri, "offset=")) {
-- * Returns 1 if the page was queued, -1 otherwise.
+         check_offset = true;
-- *
+         /*
-- * @rs: current RAM state
+@@ -XXX,XX +XXX,XX @@ void *migrate_hook_start_precopy_tcp_multifd_common(QTestState *from,
-- * @pss: data about the page we want to send
+                                                     QTestState *to,
-- */
+                                                     const char *method)
--static int ram_save_target_page_multifd(RAMState *rs, PageSearchStatus *pss)
+ {
 -    migrate_set_parameter_int(from, "multifd-channels", 16);
 -    migrate_set_parameter_int(to, "multifd-channels", 16);
 -
      migrate_set_parameter_str(from, "multifd-compression", method);
      migrate_set_parameter_str(to, "multifd-compression", method);
 -    migrate_set_capability(from, "multifd", true);
 -    migrate_set_capability(to, "multifd", true);
 -
      /* Start incoming migration from the 1st socket */
      migrate_incoming_qmp(to, "tcp:127.0.0.1:0", NULL, "{}");
 diff --git a/tests/qtest/migration/framework.h b/tests/qtest/migration/framework.h
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/framework.h
 +++ b/tests/qtest/migration/framework.h
@@ -XXX,XX +XXX,XX @@
  #define TEST_FRAMEWORK_H
  #include "libqtest.h"
 +#include <qapi/qapi-types-migration.h>
  #define FILE_TEST_FILENAME "migfile"
  #define FILE_TEST_OFFSET 0x1000
@@ -XXX,XX +XXX,XX @@ typedef struct {
      /* Postcopy specific fields */
      void *postcopy_data;
 -    bool postcopy_preempt;
      PostcopyRecoveryFailStage postcopy_recovery_fail_stage;
 +
 +    bool caps[MIGRATION_CAPABILITY__MAX];
  } MigrateCommon;
  void wait_for_serial(const char *side);
 diff --git a/tests/qtest/migration/postcopy-tests.c b/tests/qtest/migration/postcopy-tests.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/postcopy-tests.c
 +++ b/tests/qtest/migration/postcopy-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_postcopy_suspend(void)
  static void test_postcopy_preempt(void)
  {
      MigrateCommon args = {
 -        .postcopy_preempt = true,
 +        .caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT] = true,
      };
      test_postcopy_common(&args);
@@ -XXX,XX +XXX,XX @@ static void test_postcopy_recovery_fail_reconnect(void)
  static void test_postcopy_preempt_recovery(void)
  {
      MigrateCommon args = {
 -        .postcopy_preempt = true,
 +        .caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT] = true,
      };
      test_postcopy_recovery_common(&args);
 diff --git a/tests/qtest/migration/precopy-tests.c b/tests/qtest/migration/precopy-tests.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/precopy-tests.c
 +++ b/tests/qtest/migration/precopy-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_precopy_tcp_plain(void)
      test_precopy_common(&args);
  }
 -static void *migrate_hook_start_switchover_ack(QTestState *from, QTestState *to)
 -{
--    RAMBlock *block = pss->block;
+-
--    ram_addr_t offset = ((ram_addr_t)pss->page) << TARGET_PAGE_BITS;
+-    migrate_set_capability(from, "return-path", true);
--
+-    migrate_set_capability(to, "return-path", true);
--    /*
+-
--     * While using multifd live migration, we still need to handle zero
+-    migrate_set_capability(from, "switchover-ack", true);
--     * page checking on the migration main thread.
+-    migrate_set_capability(to, "switchover-ack", true);
--     */
+-
--    if (migrate_zero_page_detection() == ZERO_PAGE_DETECTION_LEGACY) {
+-    return NULL;
 -        if (save_zero_page(rs, pss, offset)) {
 -            return 1;
 -        }
 -    }
 -
 -    return ram_save_multifd_page(block, offset);
 -}
 -
- /* Should be called before sending a host page */
+ static void test_precopy_tcp_switchover_ack(void)
- static void pss_host_page_prepare(PageSearchStatus *pss)
+ {
- {
+     MigrateCommon args = {
-@@ -XXX,XX +XXX,XX @@ static int ram_save_host_page_urgent(PageSearchStatus *pss)
+         .listen_uri = "tcp:127.0.0.1:0",
+-        .start_hook = migrate_hook_start_switchover_ack,
-         if (page_dirty) {
++        .caps[MIGRATION_CAPABILITY_RETURN_PATH] = true,
-             /* Be strict to return code; it must be 1, or what else? */
++        .caps[MIGRATION_CAPABILITY_SWITCHOVER_ACK] = true,
--            if (migration_ops->ram_save_target_page(rs, pss) != 1) {
+         /*
-+            if (ram_save_target_page(rs, pss) != 1) {
+          * Source VM must be running in order to consider the switchover ACK
-                 error_report_once("%s: ram_save_target_page failed", __func__);
+          * when deciding to do switchover or not.
-                 ret = -1;
+@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_uri_none(void)
-                 goto out;
+     MigrateCommon args = {
-@@ -XXX,XX +XXX,XX @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss)
+         .listen_uri = "defer",
-             if (preempt_active) {
+         .start_hook = migrate_hook_start_precopy_tcp_multifd,
-                 qemu_mutex_unlock(&rs->bitmap_mutex);
++        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
-             }
+         /*
--            tmppages = migration_ops->ram_save_target_page(rs, pss);
+          * Multifd is more complicated than most of the features, it
-+            tmppages = ram_save_target_page(rs, pss);
+          * directly takes guest page buffers when sending, make sure
-             if (tmppages >= 0) {
+@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_zero_page_legacy(void)
-                 pages += tmppages;
+     MigrateCommon args = {
-                 /*
+         .listen_uri = "defer",
-@@ -XXX,XX +XXX,XX @@ static void ram_save_cleanup(void *opaque)
+         .start_hook = migrate_hook_start_precopy_tcp_multifd_zero_page_legacy,
-     xbzrle_cleanup();
++        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
-     multifd_ram_save_cleanup();
+         /*
-     ram_state_cleanup(rsp);
+          * Multifd is more complicated than most of the features, it
--    g_free(migration_ops);
+          * directly takes guest page buffers when sending, make sure
--    migration_ops = NULL;
+@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_no_zero_page(void)
- }
+     MigrateCommon args = {
+         .listen_uri = "defer",
- static void ram_state_reset(RAMState *rs)
+         .start_hook = migrate_hook_start_precopy_tcp_multifd_no_zero_page,
-@@ -XXX,XX +XXX,XX @@ static int ram_save_setup(QEMUFile *f, void *opaque, Error **errp)
++        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
-         return ret;
+         /*
-     }
+          * Multifd is more complicated than most of the features, it
+          * directly takes guest page buffers when sending, make sure
--    migration_ops = g_malloc0(sizeof(MigrationOps));
+@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_channels_none(void)
--
+         .listen_uri = "defer",
-     if (migrate_multifd()) {
+         .start_hook = migrate_hook_start_precopy_tcp_multifd,
-         multifd_ram_save_setup();
+         .live = true,
--        migration_ops->ram_save_target_page = ram_save_target_page_multifd;
++        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
--    } else {
+         .connect_channels = ("[ { 'channel-type': 'main',"
--        migration_ops->ram_save_target_page = ram_save_target_page_legacy;
+                              "    'addr': { 'transport': 'socket',"
-     }
+                              "              'type': 'inet',"
+diff --git a/tests/qtest/migration/tls-tests.c b/tests/qtest/migration/tls-tests.c
-     /*
+index XXXXXXX..XXXXXXX 100644
---
+--- a/tests/qtest/migration/tls-tests.c
 +++ b/tests/qtest/migration/tls-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_postcopy_tls_psk(void)
  static void test_postcopy_preempt_tls_psk(void)
  {
      MigrateCommon args = {
 -        .postcopy_preempt = true,
          .start_hook = migrate_hook_start_tls_psk_match,
          .end_hook = migrate_hook_end_tls_psk,
 +        .caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT] = true,
      };
      test_postcopy_common(&args);
@@ -XXX,XX +XXX,XX @@ static void test_postcopy_recovery_tls_psk(void)
  static void test_postcopy_preempt_all(void)
  {
      MigrateCommon args = {
 -        .postcopy_preempt = true,
          .start_hook = migrate_hook_start_tls_psk_match,
          .end_hook = migrate_hook_end_tls_psk,
 +        .caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT] = true,
      };
      test_postcopy_recovery_common(&args);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_psk_match(void)
          .listen_uri = "defer",
          .start_hook = migrate_hook_start_multifd_tcp_tls_psk_match,
          .end_hook = migrate_hook_end_tls_psk,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
      };
      test_precopy_common(&args);
  }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_psk_mismatch(void)
          .start_hook = migrate_hook_start_multifd_tcp_tls_psk_mismatch,
          .end_hook = migrate_hook_end_tls_psk,
          .result = MIG_TEST_FAIL,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
      };
      test_precopy_common(&args);
  }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_default_host(void)
          .listen_uri = "defer",
          .start_hook = migrate_hook_start_multifd_tls_x509_default_host,
          .end_hook = migrate_hook_end_tls_x509,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
      };
      test_precopy_common(&args);
  }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_override_host(void)
          .listen_uri = "defer",
          .start_hook = migrate_hook_start_multifd_tls_x509_override_host,
          .end_hook = migrate_hook_end_tls_x509,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
      };
      test_precopy_common(&args);
  }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_mismatch_host(void)
          .start_hook = migrate_hook_start_multifd_tls_x509_mismatch_host,
          .end_hook = migrate_hook_end_tls_x509,
          .result = MIG_TEST_FAIL,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
      };
      test_precopy_common(&args);
  }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_allow_anon_client(void)
          .listen_uri = "defer",
          .start_hook = migrate_hook_start_multifd_tls_x509_allow_anon_client,
          .end_hook = migrate_hook_end_tls_x509,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
      };
      test_precopy_common(&args);
  }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_reject_anon_client(void)
          .start_hook = migrate_hook_start_multifd_tls_x509_reject_anon_client,
          .end_hook = migrate_hook_end_tls_x509,
          .result = MIG_TEST_FAIL,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
      };
      test_precopy_common(&args);
  }
 --
 .48.1

-[PATCH v3 4/4] tests/qtest/migration: add postcopy test with multifd
+[PATCH v6 4/4] tests/qtest/migration: add postcopy tests with multifd
 From: Prasad Pandit <pjp@fedoraproject.org>
-Add a new postcopy test 'migration/postcopy/multifd'
+Add new qtests to run postcopy migration with multifd
-to run postcopy migration with multifd channels enabled.
+channels enabled.
 Add a boolean field 'multifd' to the MigrateCommon struct.
 It helps to enable multifd channels.
 Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
 ---
- tests/qtest/migration/framework.c      |  8 ++++++++
+ tests/qtest/migration/compression-tests.c | 13 ++++++++
- tests/qtest/migration/framework.h      |  3 +++
+ tests/qtest/migration/framework.c         |  4 +++
- tests/qtest/migration/postcopy-tests.c | 10 ++++++++++
+ tests/qtest/migration/postcopy-tests.c    | 23 +++++++++++++
-files changed, 21 insertions(+)
+ tests/qtest/migration/precopy-tests.c     | 19 +++++++++++
+ tests/qtest/migration/tls-tests.c         | 40 +++++++++++++++++++++++
 files changed, 99 insertions(+)
 v6:
 - Reorder, make this the second patch in this series.
 v5:
 - https://lore.kernel.org/qemu-devel/20250205122712.229151-1-ppandit@redhat.com/T/#t
 diff --git a/tests/qtest/migration/compression-tests.c b/tests/qtest/migration/compression-tests.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/compression-tests.c
 +++ b/tests/qtest/migration/compression-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_zstd(void)
      };
      test_precopy_common(&args);
  }
 +
 +static void test_multifd_postcopy_tcp_zstd(void)
 +{
 +    MigrateCommon args = {
 +        .listen_uri = "defer",
 +        .caps[MIGRATION_CAPABILITY_POSTCOPY_RAM] = true,
 +        .start_hook = migrate_hook_start_precopy_tcp_multifd_zstd,
 +    };
 +
 +    test_precopy_common(&args);
 +}
  #endif /* CONFIG_ZSTD */
  #ifdef CONFIG_QATZIP
@@ -XXX,XX +XXX,XX @@ void migration_test_add_compression(MigrationTestEnv *env)
  #ifdef CONFIG_ZSTD
      migration_test_add("/migration/multifd/tcp/plain/zstd",
                         test_multifd_tcp_zstd);
 +    migration_test_add("/migration/multifd+postcopy/tcp/plain/zstd",
 +                       test_multifd_postcopy_tcp_zstd);
  #endif
  #ifdef CONFIG_QATZIP
 diff --git a/tests/qtest/migration/framework.c b/tests/qtest/migration/framework.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/framework.c
 +++ b/tests/qtest/migration/framework.c
 @@ -XXX,XX +XXX,XX @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
-         migrate_set_capability(to, "postcopy-preempt", true);
+     args->caps[MIGRATION_CAPABILITY_POSTCOPY_BLOCKTIME] = true;
-     }
+     args->caps[MIGRATION_CAPABILITY_POSTCOPY_RAM] = true;
+     set_migration_capabilities(from, to, args);
-+    if (args->multifd) {
++    if (args->caps[MIGRATION_CAPABILITY_MULTIFD]) {
 +        migrate_set_capability(from, "multifd", true);
 +        migrate_set_capability(to, "multifd", true);
 +
 +        migrate_set_parameter_int(from, "multifd-channels", 8);
 +        migrate_set_parameter_int(to, "multifd-channels", 8);
 +    }
-+
      migrate_ensure_non_converge(from);
      migrate_prepare_for_dirty_mem(from);
-diff --git a/tests/qtest/migration/framework.h b/tests/qtest/migration/framework.h
-index XXXXXXX..XXXXXXX 100644
---- a/tests/qtest/migration/framework.h
-+++ b/tests/qtest/migration/framework.h
-@@ -XXX,XX +XXX,XX @@ typedef struct {
-      */
-     bool live;
-+    /* set multifd on */
-+    bool multifd;
-+
-     /* Postcopy specific fields */
-     void *postcopy_data;
-     bool postcopy_preempt;
 diff --git a/tests/qtest/migration/postcopy-tests.c b/tests/qtest/migration/postcopy-tests.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/postcopy-tests.c
 +++ b/tests/qtest/migration/postcopy-tests.c
-@@ -XXX,XX +XXX,XX @@ static void test_postcopy(void)
+@@ -XXX,XX +XXX,XX @@ static void migration_test_add_postcopy_smoke(MigrationTestEnv *env)
-     test_postcopy_common(&args);
+     }
  }
-+static void test_postcopy_multifd(void)
++static void test_multifd_postcopy(void)
 +{
 +    MigrateCommon args = {
-+        .multifd = true,
++        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
 +    };
 +
 +    test_postcopy_common(&args);
 +}
 +
- static void test_postcopy_suspend(void)
++static void test_multifd_postcopy_preempt(void)
- {
++{
-     MigrateCommon args = {
++    MigrateCommon args = {
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
 +        .caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT] = true,
 +    };
 +
 +    test_postcopy_common(&args);
 +}
 +
  void migration_test_add_postcopy(MigrationTestEnv *env)
  {
      migration_test_add_postcopy_smoke(env);
 @@ -XXX,XX +XXX,XX @@ void migration_test_add_postcopy(MigrationTestEnv *env)
- {
+             "/migration/postcopy/recovery/double-failures/reconnect",
-     if (env->has_uffd) {
+             test_postcopy_recovery_fail_reconnect);
-         migration_test_add("/migration/postcopy/plain", test_postcopy);
-+        migration_test_add("/migration/postcopy/multifd", test_postcopy_multifd);
++        migration_test_add("/migration/multifd+postcopy/plain",
-         migration_test_add("/migration/postcopy/recovery/plain",
++                           test_multifd_postcopy);
-                            test_postcopy_recovery);
++        migration_test_add("/migration/multifd+postcopy/preempt/plain",
-         migration_test_add("/migration/postcopy/preempt/plain",
++                           test_multifd_postcopy_preempt);
          if (env->is_x86) {
              migration_test_add("/migration/postcopy/suspend",
                                 test_postcopy_suspend);
 diff --git a/tests/qtest/migration/precopy-tests.c b/tests/qtest/migration/precopy-tests.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/precopy-tests.c
 +++ b/tests/qtest/migration/precopy-tests.c
@@ -XXX,XX +XXX,XX @@
  #define DIRTYLIMIT_TOLERANCE_RANGE  25  /* MB/s */
  static char *tmpfs;
 +static bool postcopy_ram = false;
  static void test_precopy_unix_plain(void)
  {
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_cancel(void)
      migrate_ensure_non_converge(from);
      migrate_prepare_for_dirty_mem(from);
 +    if (postcopy_ram) {
 +        migrate_set_capability(from, "postcopy-ram", true);
 +        migrate_set_capability(to, "postcopy-ram", true);
 +    }
 +
      migrate_set_parameter_int(from, "multifd-channels", 16);
      migrate_set_parameter_int(to, "multifd-channels", 16);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_cancel(void)
          return;
      }
 +    if (postcopy_ram) {
 +        migrate_set_capability(to2, "postcopy-ram", true);
 +    }
 +
      migrate_set_parameter_int(to2, "multifd-channels", 16);
      migrate_set_capability(to2, "multifd", true);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_cancel(void)
      migrate_end(from, to2, true);
  }
 +static void test_multifd_postcopy_tcp_cancel(void)
 +{
 +    postcopy_ram = true;
 +    test_multifd_tcp_cancel();
 +    postcopy_ram = false;
 +}
 +
  static void calc_dirty_rate(QTestState *who, uint64_t calc_time)
  {
      qtest_qmp_assert_success(who,
@@ -XXX,XX +XXX,XX @@ void migration_test_add_precopy(MigrationTestEnv *env)
                         test_multifd_tcp_zero_page_legacy);
      migration_test_add("/migration/multifd/tcp/plain/zero-page/none",
                         test_multifd_tcp_no_zero_page);
 +    migration_test_add("migration/multifd+postcopy/tcp/plain/cancel",
 +                       test_multifd_postcopy_tcp_cancel);
      if (g_str_equal(env->arch, "x86_64")
          && env->has_kvm && env->has_dirty_ring) {
 diff --git a/tests/qtest/migration/tls-tests.c b/tests/qtest/migration/tls-tests.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration/tls-tests.c
 +++ b/tests/qtest/migration/tls-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_postcopy_recovery_tls_psk(void)
      test_postcopy_recovery_common(&args);
  }
 +static void test_multifd_postcopy_recovery_tls_psk(void)
 +{
 +    MigrateCommon args = {
 +        .start_hook = migrate_hook_start_tls_psk_match,
 +        .end_hook = migrate_hook_end_tls_psk,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
 +    };
 +
 +    test_postcopy_recovery_common(&args);
 +}
 +
  /* This contains preempt+recovery+tls test altogether */
  static void test_postcopy_preempt_all(void)
  {
@@ -XXX,XX +XXX,XX @@ static void test_postcopy_preempt_all(void)
      test_postcopy_recovery_common(&args);
  }
 +static void test_multifd_postcopy_preempt_recovery_tls_psk(void)
 +{
 +    MigrateCommon args = {
 +        .start_hook = migrate_hook_start_tls_psk_match,
 +        .end_hook = migrate_hook_end_tls_psk,
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
 +    };
 +
 +    test_postcopy_recovery_common(&args);
 +}
 +
  static void test_precopy_unix_tls_psk(void)
  {
      g_autofree char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_psk_mismatch(void)
      test_precopy_common(&args);
  }
 +static void test_multifd_postcopy_tcp_tls_psk_match(void)
 +{
 +    MigrateCommon args = {
 +        .listen_uri = "defer",
 +        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
 +        .start_hook = migrate_hook_start_multifd_tcp_tls_psk_match,
 +        .end_hook = migrate_hook_end_tls_psk,
 +    };
 +
 +    test_precopy_common(&args);
 +}
 +
  #ifdef CONFIG_TASN1
  static void test_multifd_tcp_tls_x509_default_host(void)
  {
@@ -XXX,XX +XXX,XX @@ void migration_test_add_tls(MigrationTestEnv *env)
                             test_postcopy_preempt_tls_psk);
          migration_test_add("/migration/postcopy/preempt/recovery/tls/psk",
                             test_postcopy_preempt_all);
 +        migration_test_add("/migration/multifd+postcopy/recovery/tls/psk",
 +                           test_multifd_postcopy_recovery_tls_psk);
 +        migration_test_add("/migration/multifd+postcopy/preempt/recovery/tls/psk",
 +                           test_multifd_postcopy_preempt_recovery_tls_psk);
      }
  #ifdef CONFIG_TASN1
      migration_test_add("/migration/precopy/unix/tls/x509/default-host",
@@ -XXX,XX +XXX,XX @@ void migration_test_add_tls(MigrationTestEnv *env)
                         test_multifd_tcp_tls_psk_match);
      migration_test_add("/migration/multifd/tcp/tls/psk/mismatch",
                         test_multifd_tcp_tls_psk_mismatch);
 +    migration_test_add("/migration/multifd+postcopy/tcp/tls/psk/match",
 +                       test_multifd_postcopy_tcp_tls_psk_match);
  #ifdef CONFIG_TASN1
      migration_test_add("/migration/multifd/tcp/tls/x509/default-host",
                         test_multifd_tcp_tls_x509_default_host);
 --
 .48.1

From: Prasad Pandit <pjp@fedoraproject.org>

Hello,

* Currently Multifd and Postcopy migration can not be used together.
  QEMU shows "Postcopy is not yet compatible with multifd" message.

When migrating guests with large (100's GB) RAM, Multifd threads
  help to accelerate migration, but inability to use it with the
  Postcopy mode delays guest start up on the destination side.

* This patch series allows to enable both Multifd and Postcopy
  migration together. Precopy and Multifd threads work during
  the initial guest (RAM) transfer. When migration moves to the
  Postcopy phase, Multifd threads are restrained and the Postcopy
  threads start to request pages from the source side.

* This series removes magic value (4-bytes) introduced in the
  previous series for the Postcopy channel. And refactoring of
  the 'ram_save_target_page' function is made independent of
  the multifd & postcopy change.

* This series passes all existing 'tests/qtest/migration/*' test
  cases and adds a new one to enable multifd channels with postcopy
  migration.

v2: https://lore.kernel.org/qemu-devel/20241129122256.96778-1-ppandit@redhat.com/T/#u
  v1: https://lore.kernel.org/qemu-devel/20241126115748.118683-1-ppandit@redhat.com/T/#u
  v0: https://lore.kernel.org/qemu-devel/20241029150908.1136894-1-ppandit@redhat.com/T/#u

Thank you.
---
Prasad Pandit (4):
  migration/multifd: move macros to multifd header
  migration: refactor ram_save_target_page functions
  migration: enable multifd and postcopy together
  tests/qtest/migration: add postcopy test with multifd

--
2.48.1

From: Prasad Pandit <pjp@fedoraproject.org>

Move MULTIFD_ macros to the header file so that
they are accessible from other source files.

Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
---
 migration/multifd.c | 5 -----
 migration/multifd.h | 5 +++++
 2 files changed, 5 insertions(+), 5 deletions(-)

v2:
 - https://lore.kernel.org/qemu-devel/20241129122256.96778-1-ppandit@redhat.com/

diff --git a/migration/multifd.c b/migration/multifd.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -XXX,XX +XXX,XX @@
 #include "io/channel-socket.h"
 #include "yank_functions.h"
 
-/* Multiple fd's */
-
-#define MULTIFD_MAGIC 0x11223344U
-#define MULTIFD_VERSION 1
-
 typedef struct {
     uint32_t magic;
     uint32_t version;
diff --git a/migration/multifd.h b/migration/multifd.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -XXX,XX +XXX,XX @@ bool multifd_queue_page(RAMBlock *block, ram_addr_t offset);
 bool multifd_recv(void);
 MultiFDRecvData *multifd_get_recv_data(void);
 
+/* Multiple fd's */
+
+#define MULTIFD_MAGIC 0x11223344U
+#define MULTIFD_VERSION 1
+
 /* Multifd Compression flags */
 #define MULTIFD_FLAG_SYNC (1 << 0)
 
-- 
2.48.1

From: Prasad Pandit <pjp@fedoraproject.org>

Refactor ram_save_target_page legacy and multifd
functions into one. Other than simplifying it,
it frees 'migration_ops' object from usage, so it
is expunged.

Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
---
 migration/ram.c | 67 +++++++++++++------------------------------------
 1 file changed, 17 insertions(+), 50 deletions(-)

v2:
 - https://lore.kernel.org/qemu-devel/20241129122256.96778-1-ppandit@redhat.com/

diff --git a/migration/ram.c b/migration/ram.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -XXX,XX +XXX,XX @@ void ram_transferred_add(uint64_t bytes)
     }
 }
 
-struct MigrationOps {
-    int (*ram_save_target_page)(RAMState *rs, PageSearchStatus *pss);
-};
-typedef struct MigrationOps MigrationOps;
-
-MigrationOps *migration_ops;
-
 static int ram_save_host_page_urgent(PageSearchStatus *pss);
 
 /* NOTE: page is the PFN not real ram_addr_t. */
@@ -XXX,XX +XXX,XX @@ int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len,
 }
 
 /**
- * ram_save_target_page_legacy: save one target page
- *
- * Returns the number of pages written
+ * ram_save_target_page: save one target page to the precopy thread
+ * OR to multifd workers.
  *
  * @rs: current RAM state
  * @pss: data about the page we want to send
  */
-static int ram_save_target_page_legacy(RAMState *rs, PageSearchStatus *pss)
+static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss)
 {
     ram_addr_t offset = ((ram_addr_t)pss->page) << TARGET_PAGE_BITS;
     int res;
 
+    if (!migrate_multifd()
+        || migrate_zero_page_detection() == ZERO_PAGE_DETECTION_LEGACY) {
+        if (save_zero_page(rs, pss, offset)) {
+            return 1;
+        }
+    }
+
+    if (migrate_multifd()) {
+        RAMBlock *block = pss->block;
+        return ram_save_multifd_page(block, offset);
+    }
+
     if (control_save_page(pss, offset, &res)) {
         return res;
     }
 
-    if (save_zero_page(rs, pss, offset)) {
-        return 1;
-    }
-
     return ram_save_page(rs, pss);
 }
 
-/**
- * ram_save_target_page_multifd: send one target page to multifd workers
- *
- * Returns 1 if the page was queued, -1 otherwise.
- *
- * @rs: current RAM state
- * @pss: data about the page we want to send
- */
-static int ram_save_target_page_multifd(RAMState *rs, PageSearchStatus *pss)
-{
-    RAMBlock *block = pss->block;
-    ram_addr_t offset = ((ram_addr_t)pss->page) << TARGET_PAGE_BITS;
-
-    /*
-     * While using multifd live migration, we still need to handle zero
-     * page checking on the migration main thread.
-     */
-    if (migrate_zero_page_detection() == ZERO_PAGE_DETECTION_LEGACY) {
-        if (save_zero_page(rs, pss, offset)) {
-            return 1;
-        }
-    }
-
-    return ram_save_multifd_page(block, offset);
-}
-
 /* Should be called before sending a host page */
 static void pss_host_page_prepare(PageSearchStatus *pss)
 {
@@ -XXX,XX +XXX,XX @@ static int ram_save_host_page_urgent(PageSearchStatus *pss)
 
         if (page_dirty) {
             /* Be strict to return code; it must be 1, or what else? */
-            if (migration_ops->ram_save_target_page(rs, pss) != 1) {
+            if (ram_save_target_page(rs, pss) != 1) {
                 error_report_once("%s: ram_save_target_page failed", __func__);
                 ret = -1;
                 goto out;
@@ -XXX,XX +XXX,XX @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss)
             if (preempt_active) {
                 qemu_mutex_unlock(&rs->bitmap_mutex);
             }
-            tmppages = migration_ops->ram_save_target_page(rs, pss);
+            tmppages = ram_save_target_page(rs, pss);
             if (tmppages >= 0) {
                 pages += tmppages;
                 /*
@@ -XXX,XX +XXX,XX @@ static void ram_save_cleanup(void *opaque)
     xbzrle_cleanup();
     multifd_ram_save_cleanup();
     ram_state_cleanup(rsp);
-    g_free(migration_ops);
-    migration_ops = NULL;
 }
 
 static void ram_state_reset(RAMState *rs)
@@ -XXX,XX +XXX,XX @@ static int ram_save_setup(QEMUFile *f, void *opaque, Error **errp)
         return ret;
     }
 
-    migration_ops = g_malloc0(sizeof(MigrationOps));
-
     if (migrate_multifd()) {
         multifd_ram_save_setup();
-        migration_ops->ram_save_target_page = ram_save_target_page_multifd;
-    } else {
-        migration_ops->ram_save_target_page = ram_save_target_page_legacy;
     }
 
     /*
-- 
2.48.1

From: Prasad Pandit <pjp@fedoraproject.org>

Enable Multifd and Postcopy migration together.
The migration_ioc_process_incoming() routine
checks magic value sent on each channel and
helps to properly setup multifd and postcopy
channels.

The Precopy and Multifd threads work during the
initial guest RAM transfer. When migration moves
to the Postcopy phase, the multifd threads are
restrained and Postcopy threads on the destination
request/pull data from the source side.

Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
---
 migration/migration.c      | 106 +++++++++++++++++++++++--------------
 migration/multifd-nocomp.c |   3 +-
 migration/options.c        |   5 --
 migration/ram.c            |   4 +-
 4 files changed, 70 insertions(+), 48 deletions(-)

v2: Minor changes in migration_ioc_process_incoming() to pass test cases
 - https://lore.kernel.org/qemu-devel/20241129122256.96778-1-ppandit@redhat.com/

diff --git a/migration/migration.c b/migration/migration.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -XXX,XX +XXX,XX @@ enum mig_rp_message_type {
     MIG_RP_MSG_MAX
 };
 
+/* Migration channel types */
+enum { CH_DEFAULT, CH_MULTIFD, CH_POSTCOPY };
+
 /* When we add fault tolerance, we could have several
    migrations at once.  For now we don't need to add
    dynamic creation of migration */
@@ -XXX,XX +XXX,XX @@ void migration_fd_process_incoming(QEMUFile *f)
 /*
  * Returns true when we want to start a new incoming migration process,
  * false otherwise.
+ *
+ * All the required channels must be in place before a new incoming
+ * migration process starts.
+ *  - Multifd enabled:
+ *    The main channel and the multifd channels are required.
+ *  - Multifd/Postcopy disabled:
+ *    The main channel is required.
+ *  - Postcopy enabled:
+ *    We don't want to start a new incoming migration when
+ *    the postcopy channel is created. Because it is created
+ *    towards the end of the precopy migration.
+ *
  */
-static bool migration_should_start_incoming(bool main_channel)
+static bool migration_should_start_incoming(uint8_t channel)
 {
-    /* Multifd doesn't start unless all channels are established */
-    if (migrate_multifd()) {
-        return migration_has_all_channels();
-    }
+    bool ret = false;
+
+    if (channel != CH_POSTCOPY) {
+        MigrationIncomingState *mis = migration_incoming_get_current();
+        ret = mis->from_src_file ? true : false;
 
-    /* Preempt channel only starts when the main channel is created */
-    if (migrate_postcopy_preempt()) {
-        return main_channel;
+        if (ret && migrate_multifd()) {
+            ret = multifd_recv_all_channels_created();
+        }
     }
 
-    /*
-     * For all the rest types of migration, we should only reach here when
-     * it's the main channel that's being created, and we should always
-     * proceed with this channel.
-     */
-    assert(main_channel);
-    return true;
+    return ret;
 }
 
 void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
@@ -XXX,XX +XXX,XX @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
     MigrationIncomingState *mis = migration_incoming_get_current();
     Error *local_err = NULL;
     QEMUFile *f;
-    bool default_channel = true;
     uint32_t channel_magic = 0;
+    uint8_t channel = CH_DEFAULT;
     int ret = 0;
 
-    if (migrate_multifd() && !migrate_mapped_ram() &&
-        !migrate_postcopy_ram() &&
-        qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_READ_MSG_PEEK)) {
+    if (!migration_should_start_incoming(channel)) {
+        if (qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_READ_MSG_PEEK)) {
         /*
          * With multiple channels, it is possible that we receive channels
          * out of order on destination side, causing incorrect mapping of
@@ -XXX,XX +XXX,XX @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
          * tls handshake while initializing main channel so with tls this
          * issue is not possible.
          */
-        ret = migration_channel_read_peek(ioc, (void *)&channel_magic,
-                                          sizeof(channel_magic), errp);
+            ret = migration_channel_read_peek(ioc, (void *)&channel_magic,
+                                              sizeof(channel_magic), errp);
+            if (ret != 0) {
+                return;
+            }
 
-        if (ret != 0) {
-            return;
+            if (channel_magic == cpu_to_be32(QEMU_VM_FILE_MAGIC)) {
+                channel = CH_DEFAULT;
+            } else if (channel_magic == cpu_to_be32(MULTIFD_MAGIC)) {
+                channel = CH_MULTIFD;
+            } else if (!mis->from_src_file
+                        && mis->state == MIGRATION_STATUS_POSTCOPY_PAUSED) {
+                /* reconnect default channel for postcopy recovery */
+                channel = CH_DEFAULT;
+            } else {
+                error_report("%s: could not identify channel, unknown magic: %u",
+                                __func__, channel_magic);
+                return;
+            }
+        } else if (mis->from_src_file
+            && (!strcmp(ioc->name, "migration-tls-incoming")
+                || !strcmp(ioc->name, "migration-file-incoming"))) {
+            channel = CH_MULTIFD;
         }
-
-        default_channel = (channel_magic == cpu_to_be32(QEMU_VM_FILE_MAGIC));
-    } else {
-        default_channel = !mis->from_src_file;
+    } else if (mis->from_src_file) { // && migrate_postcopy_preempt()
+        channel = CH_POSTCOPY;
     }
 
     if (multifd_recv_setup(errp) != 0) {
         return;
     }
 
-    if (default_channel) {
+    if (channel == CH_DEFAULT) {
         f = qemu_file_new_input(ioc);
         migration_incoming_setup(f);
-    } else {
+    } else if (channel == CH_MULTIFD) {
         /* Multiple connections */
-        assert(migration_needs_multiple_sockets());
         if (migrate_multifd()) {
             multifd_recv_new_channel(ioc, &local_err);
-        } else {
-            assert(migrate_postcopy_preempt());
-            f = qemu_file_new_input(ioc);
-            postcopy_preempt_new_channel(mis, f);
         }
         if (local_err) {
             error_propagate(errp, local_err);
             return;
         }
+    } else if (channel == CH_POSTCOPY) {
+        assert(migrate_postcopy_preempt());
+        assert(!mis->postcopy_qemufile_dst);
+        f = qemu_file_new_input(ioc);
+        postcopy_preempt_new_channel(mis, f);
     }
 
-    if (migration_should_start_incoming(default_channel)) {
+    if (migration_should_start_incoming(channel)) {
         /* If it's a recovery, we're done */
         if (postcopy_try_recover()) {
             return;
@@ -XXX,XX +XXX,XX @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
  */
 bool migration_has_all_channels(void)
 {
+    bool ret = false;
     MigrationIncomingState *mis = migration_incoming_get_current();
 
     if (!mis->from_src_file) {
-        return false;
+        return ret;
     }
 
     if (migrate_multifd()) {
-        return multifd_recv_all_channels_created();
+        ret = multifd_recv_all_channels_created();
     }
 
-    if (migrate_postcopy_preempt()) {
-        return mis->postcopy_qemufile_dst != NULL;
+    if (ret && migrate_postcopy_preempt()) {
+        ret = mis->postcopy_qemufile_dst != NULL;
     }
 
-    return true;
+    return ret;
 }
 
 int migrate_send_rp_switchover_ack(MigrationIncomingState *mis)
diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd-nocomp.c
+++ b/migration/multifd-nocomp.c
@@ -XXX,XX +XXX,XX @@
 #include "file.h"
 #include "multifd.h"
 #include "options.h"
+#include "migration.h"
 #include "qapi/error.h"
 #include "qemu/cutils.h"
 #include "qemu/error-report.h"
@@ -XXX,XX +XXX,XX @@ int multifd_ram_flush_and_sync(QEMUFile *f)
     MultiFDSyncReq req;
     int ret;
 
-    if (!migrate_multifd()) {
+    if (!migrate_multifd() || migration_in_postcopy()) {
         return 0;
     }
 
diff --git a/migration/options.c b/migration/options.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -XXX,XX +XXX,XX @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp)
             error_setg(errp, "Postcopy is not compatible with ignore-shared");
             return false;
         }
-
-        if (new_caps[MIGRATION_CAPABILITY_MULTIFD]) {
-            error_setg(errp, "Postcopy is not yet compatible with multifd");
-            return false;
-        }
     }
 
     if (new_caps[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT]) {
diff --git a/migration/ram.c b/migration/ram.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -XXX,XX +XXX,XX @@ static int find_dirty_block(RAMState *rs, PageSearchStatus *pss)
         pss->page = 0;
         pss->block = QLIST_NEXT_RCU(pss->block, next);
         if (!pss->block) {
-            if (multifd_ram_sync_per_round()) {
+            if (multifd_ram_sync_per_round() && !migration_in_postcopy()) {
                 QEMUFile *f = rs->pss[RAM_CHANNEL_PRECOPY].pss_channel;
                 int ret = multifd_ram_flush_and_sync(f);
                 if (ret < 0) {
@@ -XXX,XX +XXX,XX @@ static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss)
         }
     }
 
-    if (migrate_multifd()) {
+    if (migrate_multifd() && !migration_in_postcopy()) {
         RAMBlock *block = pss->block;
         return ram_save_multifd_page(block, offset);
     }
-- 
2.48.1

From: Prasad Pandit <pjp@fedoraproject.org>

Add a new postcopy test 'migration/postcopy/multifd'
to run postcopy migration with multifd channels enabled.
Add a boolean field 'multifd' to the MigrateCommon struct.
It helps to enable multifd channels.

Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
---
 tests/qtest/migration/framework.c      |  8 ++++++++
 tests/qtest/migration/framework.h      |  3 +++
 tests/qtest/migration/postcopy-tests.c | 10 ++++++++++
 3 files changed, 21 insertions(+)

diff --git a/tests/qtest/migration/framework.c b/tests/qtest/migration/framework.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/framework.c
+++ b/tests/qtest/migration/framework.c
@@ -XXX,XX +XXX,XX @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
         migrate_set_capability(to, "postcopy-preempt", true);
     }
 
+    if (args->multifd) {
+        migrate_set_capability(from, "multifd", true);
+        migrate_set_capability(to, "multifd", true);
+
+        migrate_set_parameter_int(from, "multifd-channels", 8);
+        migrate_set_parameter_int(to, "multifd-channels", 8);
+    }
+
     migrate_ensure_non_converge(from);
 
     migrate_prepare_for_dirty_mem(from);
diff --git a/tests/qtest/migration/framework.h b/tests/qtest/migration/framework.h
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/framework.h
+++ b/tests/qtest/migration/framework.h
@@ -XXX,XX +XXX,XX @@ typedef struct {
      */
     bool live;
 
+    /* set multifd on */
+    bool multifd;
+
     /* Postcopy specific fields */
     void *postcopy_data;
     bool postcopy_preempt;
diff --git a/tests/qtest/migration/postcopy-tests.c b/tests/qtest/migration/postcopy-tests.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/postcopy-tests.c
+++ b/tests/qtest/migration/postcopy-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_postcopy(void)
     test_postcopy_common(&args);
 }
 
+static void test_postcopy_multifd(void)
+{
+    MigrateCommon args = {
+        .multifd = true,
+    };
+
+    test_postcopy_common(&args);
+}
+
 static void test_postcopy_suspend(void)
 {
     MigrateCommon args = {
@@ -XXX,XX +XXX,XX @@ void migration_test_add_postcopy(MigrationTestEnv *env)
 {
     if (env->has_uffd) {
         migration_test_add("/migration/postcopy/plain", test_postcopy);
+        migration_test_add("/migration/postcopy/multifd", test_postcopy_multifd);
         migration_test_add("/migration/postcopy/recovery/plain",
                            test_postcopy_recovery);
         migration_test_add("/migration/postcopy/preempt/plain",
--
2.48.1

From: Prasad Pandit <pjp@fedoraproject.org>

Hello,

* This series (v6) shuts down Multifd threads before starting Postcopy
  migration. It helps to avoid an issue of multifd pages arriving late
  at the destination during Postcopy phase and corrupting the vCPU
  state. It also reorders the qtest patches and does some refactoring
  changes as suggested in previous review.
===
67/67 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test                 OK             161.35s   73 subtests passed
===

v5: https://lore.kernel.org/qemu-devel/20250205122712.229151-1-ppandit@redhat.com/T/#t
* This series (v5) consolidates migration capabilities setting in one
  'set_migration_capabilities()' function, thus simplifying test sources.
  It passes all migration tests.
===
66/66 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test                 OK             143.66s   71 subtests passed
===

v4: https://lore.kernel.org/qemu-devel/20250127120823.144949-1-ppandit@redhat.com/T/#t
* This series (v4) adds more 'multifd+postcopy' qtests which test
  Precopy migration with 'postcopy-ram' attribute set. And run
  Postcopy migrations with 'multifd' channels enabled.
===
$ ../qtest/migration-test --tap -k -r '/x86_64/migration/multifd+postcopy' | grep -i 'slow test'
# slow test /x86_64/migration/multifd+postcopy/plain executed in 1.29 secs
# slow test /x86_64/migration/multifd+postcopy/recovery/tls/psk executed in 2.48 secs
# slow test /x86_64/migration/multifd+postcopy/preempt/plain executed in 1.49 secs
# slow test /x86_64/migration/multifd+postcopy/preempt/recovery/tls/psk executed in 2.52 secs
# slow test /x86_64/migration/multifd+postcopy/tcp/tls/psk/match executed in 3.62 secs
# slow test /x86_64/migration/multifd+postcopy/tcp/plain/zstd executed in 1.34 secs
# slow test /x86_64/migration/multifd+postcopy/tcp/plain/cancel executed in 2.24 secs
...
66/66 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test                 OK             148.41s   71 subtests passed
===

v3: https://lore.kernel.org/qemu-devel/20250121131032.1611245-1-ppandit@redhat.com/T/#t
* This series (v3) passes all existing 'tests/qtest/migration/*' tests
  and adds a new one to enable multifd channels with postcopy migration.

v2: https://lore.kernel.org/qemu-devel/20241129122256.96778-1-ppandit@redhat.com/T/#u
* This series (v2) further refactors the 'ram_save_target_page'
  function to make it independent of the multifd & postcopy change.

v1: https://lore.kernel.org/qemu-devel/20241126115748.118683-1-ppandit@redhat.com/T/#u
* This series removes magic value (4-bytes) introduced in the
  previous series for the Postcopy channel.

v0: https://lore.kernel.org/qemu-devel/20241029150908.1136894-1-ppandit@redhat.com/T/#u
* Currently Multifd and Postcopy migration can not be used together.
  QEMU shows "Postcopy is not yet compatible with multifd" message.

When migrating guests with large (100's GB) RAM, Multifd threads
  help to accelerate migration, but inability to use it with the
  Postcopy mode delays guest start up on the destination side.

* This series introduces magic value (4-bytes) to be sent on the
  Postcopy channel. It helps to differentiate channels and properly
  setup incoming connections on the destination side.

Thank you.
---
Prasad Pandit (4):
  migration/multifd: move macros to multifd header
  migration: enable multifd and postcopy together
  tests/qtest/migration: consolidate set capabilities
  tests/qtest/migration: add postcopy tests with multifd

--
2.48.1

From: Prasad Pandit <pjp@fedoraproject.org>

Move MULTIFD_ macros to the header file so that
they are accessible from other source files.

Reviewed-by: Fabiano Rosas <farosas@suse.de>
Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
---
 migration/multifd.c | 5 -----
 migration/multifd.h | 5 +++++
 2 files changed, 5 insertions(+), 5 deletions(-)

v5: no change
- https://lore.kernel.org/qemu-devel/20250205122712.229151-1-ppandit@redhat.com/T/#t

From: Prasad Pandit <pjp@fedoraproject.org>

Enable Multifd and Postcopy migration together.
The migration_ioc_process_incoming() routine
checks magic value sent on each channel and
helps to properly setup multifd and postcopy
channels.

The Precopy and Multifd threads work during the
initial guest RAM transfer. When migration moves
to the Postcopy phase, the multifd threads are
shutdown and Postcopy threads on the destination
request/pull data from the source side.

Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
---
 migration/migration.c      | 107 ++++++++++++++++++++-----------------
 migration/multifd-nocomp.c |   3 +-
 migration/multifd.c        |   4 +-
 migration/options.c        |   5 --
 migration/ram.c            |   7 ++-
 5 files changed, 64 insertions(+), 62 deletions(-)

v6:
- Shutdown multifd threads before postcopy_start()
- Reorder tests/qtest/migration/ patches
- Some refactoring of functions

v5:
- https://lore.kernel.org/qemu-devel/20250205122712.229151-1-ppandit@redhat.com/T/#t

diff --git a/migration/migration.c b/migration/migration.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -XXX,XX +XXX,XX @@ enum mig_rp_message_type {
     MIG_RP_MSG_MAX
 };
 
+/* Migration channel types */
+enum { CH_MAIN, CH_MULTIFD, CH_POSTCOPY };
+
 /* When we add fault tolerance, we could have several
    migrations at once.  For now we don't need to add
    dynamic creation of migration */
@@ -XXX,XX +XXX,XX @@ void migration_fd_process_incoming(QEMUFile *f)
     migration_incoming_process();
 }
 
-/*
- * Returns true when we want to start a new incoming migration process,
- * false otherwise.
- */
-static bool migration_should_start_incoming(bool main_channel)
+static bool migration_has_main_and_multifd_channels(void)
 {
-    /* Multifd doesn't start unless all channels are established */
-    if (migrate_multifd()) {
-        return migration_has_all_channels();
+    MigrationIncomingState *mis = migration_incoming_get_current();
+    if (!mis->from_src_file) {
+        /* main channel not established */
+        return false;
     }
 
-    /* Preempt channel only starts when the main channel is created */
-    if (migrate_postcopy_preempt()) {
-        return main_channel;
+    if (migrate_multifd() && !multifd_recv_all_channels_created()) {
+        return false;
     }
 
-    /*
-     * For all the rest types of migration, we should only reach here when
-     * it's the main channel that's being created, and we should always
-     * proceed with this channel.
-     */
-    assert(main_channel);
+    /* main channel and all multifd channels are established */
     return true;
 }
 
@@ -XXX,XX +XXX,XX @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
     MigrationIncomingState *mis = migration_incoming_get_current();
     Error *local_err = NULL;
     QEMUFile *f;
-    bool default_channel = true;
     uint32_t channel_magic = 0;
+    uint8_t channel = CH_MAIN;
     int ret = 0;
 
-    if (migrate_multifd() && !migrate_mapped_ram() &&
-        !migrate_postcopy_ram() &&
-        qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_READ_MSG_PEEK)) {
+    if (!migration_has_main_and_multifd_channels()) {
+        if (qio_channel_has_feature(ioc, QIO_CHANNEL_FEATURE_READ_MSG_PEEK)) {
         /*
          * With multiple channels, it is possible that we receive channels
          * out of order on destination side, causing incorrect mapping of
@@ -XXX,XX +XXX,XX @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
          * tls handshake while initializing main channel so with tls this
          * issue is not possible.
          */
-        ret = migration_channel_read_peek(ioc, (void *)&channel_magic,
-                                          sizeof(channel_magic), errp);
+            ret = migration_channel_read_peek(ioc, (void *)&channel_magic,
+                                              sizeof(channel_magic), errp);
+            if (ret != 0) {
+                return;
+            }
 
-        if (ret != 0) {
-            return;
+            if (channel_magic == cpu_to_be32(QEMU_VM_FILE_MAGIC)) {
+                channel = CH_MAIN;
+            } else if (channel_magic == cpu_to_be32(MULTIFD_MAGIC)) {
+                channel = CH_MULTIFD;
+            } else if (!mis->from_src_file
+                        && mis->state == MIGRATION_STATUS_POSTCOPY_PAUSED) {
+                /* reconnect default channel for postcopy recovery */
+                channel = CH_MAIN;
+            } else {
+                error_report("%s: unknown channel magic: %u",
+                                __func__, channel_magic);
+                return;
+            }
+        } else if (mis->from_src_file
+                && (!strcmp(ioc->name, "migration-tls-incoming")
+                || !strcmp(ioc->name, "migration-file-incoming"))) {
+            channel = CH_MULTIFD;
         }
-
-        default_channel = (channel_magic == cpu_to_be32(QEMU_VM_FILE_MAGIC));
-    } else {
-        default_channel = !mis->from_src_file;
+    } else if (mis->from_src_file) {
+        channel = CH_POSTCOPY;
     }
 
     if (multifd_recv_setup(errp) != 0) {
         return;
     }
 
-    if (default_channel) {
+    if (channel == CH_MAIN) {
         f = qemu_file_new_input(ioc);
         migration_incoming_setup(f);
-    } else {
+    } else if (channel == CH_MULTIFD) {
         /* Multiple connections */
-        assert(migration_needs_multiple_sockets());
         if (migrate_multifd()) {
             multifd_recv_new_channel(ioc, &local_err);
-        } else {
-            assert(migrate_postcopy_preempt());
-            f = qemu_file_new_input(ioc);
-            postcopy_preempt_new_channel(mis, f);
         }
         if (local_err) {
             error_propagate(errp, local_err);
             return;
         }
+    } else if (channel == CH_POSTCOPY) {
+        assert(migrate_postcopy_preempt());
+        assert(!mis->postcopy_qemufile_dst);
+        f = qemu_file_new_input(ioc);
+        postcopy_preempt_new_channel(mis, f);
     }
 
-    if (migration_should_start_incoming(default_channel)) {
+    if (channel != CH_POSTCOPY && migration_has_main_and_multifd_channels()) {
         /* If it's a recovery, we're done */
         if (postcopy_try_recover()) {
             return;
@@ -XXX,XX +XXX,XX @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp)
  */
 bool migration_has_all_channels(void)
 {
+    if (!migration_has_main_and_multifd_channels()) {
+        return false;
+    }
+
     MigrationIncomingState *mis = migration_incoming_get_current();
-
-    if (!mis->from_src_file) {
+    if (migrate_postcopy_preempt() && !mis->postcopy_qemufile_dst) {
         return false;
     }
 
-    if (migrate_multifd()) {
-        return multifd_recv_all_channels_created();
-    }
-
-    if (migrate_postcopy_preempt()) {
-        return mis->postcopy_qemufile_dst != NULL;
-    }
-
     return true;
 }
 
@@ -XXX,XX +XXX,XX @@ static void migrate_fd_cleanup(MigrationState *s)
 
     assert(!migration_is_active());
 
+    file_cleanup_outgoing_migration();
+    socket_cleanup_outgoing_migration();
     if (s->state == MIGRATION_STATUS_CANCELLING) {
         migrate_set_state(&s->state, MIGRATION_STATUS_CANCELLING,
                           MIGRATION_STATUS_CANCELLED);
@@ -XXX,XX +XXX,XX @@ static MigIterateState migration_iteration_run(MigrationState *s)
     }
 
     /* Still a significant amount to transfer */
-    if (!in_postcopy && must_precopy <= s->threshold_size && can_switchover &&
-        qatomic_read(&s->start_postcopy)) {
+    if (!in_postcopy && must_precopy <= s->threshold_size
+        && can_switchover && qatomic_read(&s->start_postcopy)) {
+        if (migrate_multifd()) {
+            multifd_send_shutdown();
+        }
         if (postcopy_start(s, &local_err)) {
             migrate_set_error(s, local_err);
             error_report_err(local_err);
diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd-nocomp.c
+++ b/migration/multifd-nocomp.c
@@ -XXX,XX +XXX,XX @@
 #include "file.h"
 #include "multifd.h"
 #include "options.h"
+#include "migration.h"
 #include "qapi/error.h"
 #include "qemu/cutils.h"
 #include "qemu/error-report.h"
@@ -XXX,XX +XXX,XX @@ int multifd_ram_flush_and_sync(QEMUFile *f)
     MultiFDSyncReq req;
     int ret;
 
-    if (!migrate_multifd()) {
+    if (!migrate_multifd() || migration_in_postcopy()) {
         return 0;
     }
 
diff --git a/migration/multifd.c b/migration/multifd.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -XXX,XX +XXX,XX @@ static bool multifd_send_cleanup_channel(MultiFDSendParams *p, Error **errp)
 
 static void multifd_send_cleanup_state(void)
 {
-    file_cleanup_outgoing_migration();
-    socket_cleanup_outgoing_migration();
     qemu_sem_destroy(&multifd_send_state->channels_created);
     qemu_sem_destroy(&multifd_send_state->channels_ready);
     g_free(multifd_send_state->params);
@@ -XXX,XX +XXX,XX @@ void multifd_send_shutdown(void)
 {
     int i;
 
-    if (!migrate_multifd()) {
+    if (!migrate_multifd() || !multifd_send_state) {
         return;
     }
 
diff --git a/migration/options.c b/migration/options.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -XXX,XX +XXX,XX @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp)
             error_setg(errp, "Postcopy is not compatible with ignore-shared");
             return false;
         }
-
-        if (new_caps[MIGRATION_CAPABILITY_MULTIFD]) {
-            error_setg(errp, "Postcopy is not yet compatible with multifd");
-            return false;
-        }
     }
 
     if (new_caps[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT]) {
diff --git a/migration/ram.c b/migration/ram.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -XXX,XX +XXX,XX @@ static int find_dirty_block(RAMState *rs, PageSearchStatus *pss)
         pss->page = 0;
         pss->block = QLIST_NEXT_RCU(pss->block, next);
         if (!pss->block) {
-            if (multifd_ram_sync_per_round()) {
+            if (multifd_ram_sync_per_round() && !migration_in_postcopy()) {
                 QEMUFile *f = rs->pss[RAM_CHANNEL_PRECOPY].pss_channel;
                 int ret = multifd_ram_flush_and_sync(f);
                 if (ret < 0) {
@@ -XXX,XX +XXX,XX @@ static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss)
         }
     }
 
-    if (migrate_multifd()) {
-        RAMBlock *block = pss->block;
-        return ram_save_multifd_page(block, offset);
+    if (migrate_multifd() && !migration_in_postcopy()) {
+        return ram_save_multifd_page(pss->block, offset);
     }
 
     if (control_save_page(pss, offset, &res)) {
--
2.48.1

From: Prasad Pandit <pjp@fedoraproject.org>

Migration capabilities are set in multiple '.start_hook'
functions for various tests. Instead, consolidate setting
capabilities in 'set_migration_capabilities()' function
which is called from various 'test_*_common()' functions.
While simplifying the capabilities setting, it helps
to declutter the qtest sources.

Suggested-by: Fabiano Rosas <farosas@suse.de>
Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
---
 tests/qtest/migration/compression-tests.c | 10 ++--
 tests/qtest/migration/cpr-tests.c         |  4 +-
 tests/qtest/migration/file-tests.c        | 44 +++++-------------
 tests/qtest/migration/framework.c         | 56 +++++++++++++++++------
 tests/qtest/migration/framework.h         |  4 +-
 tests/qtest/migration/postcopy-tests.c    |  4 +-
 tests/qtest/migration/precopy-tests.c     | 19 +++-----
 tests/qtest/migration/tls-tests.c         | 11 ++++-
 8 files changed, 80 insertions(+), 72 deletions(-)

v6:
- Reorder, make this the first qtest patch in this series.

v5:
- https://lore.kernel.org/qemu-devel/20250205122712.229151-1-ppandit@redhat.com/T/#t

diff --git a/tests/qtest/migration/compression-tests.c b/tests/qtest/migration/compression-tests.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/compression-tests.c
+++ b/tests/qtest/migration/compression-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_zstd(void)
 {
     MigrateCommon args = {
         .listen_uri = "defer",
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
         .start_hook = migrate_hook_start_precopy_tcp_multifd_zstd,
     };
     test_precopy_common(&args);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_qatzip(void)
 {
     MigrateCommon args = {
         .listen_uri = "defer",
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
         .start_hook = migrate_hook_start_precopy_tcp_multifd_qatzip,
     };
     test_precopy_common(&args);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_qpl(void)
 {
     MigrateCommon args = {
         .listen_uri = "defer",
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
         .start_hook = migrate_hook_start_precopy_tcp_multifd_qpl,
     };
     test_precopy_common(&args);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_uadk(void)
 {
     MigrateCommon args = {
         .listen_uri = "defer",
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
         .start_hook = migrate_hook_start_precopy_tcp_multifd_uadk,
     };
     test_precopy_common(&args);
@@ -XXX,XX +XXX,XX @@ migrate_hook_start_xbzrle(QTestState *from,
                           QTestState *to)
 {
     migrate_set_parameter_int(from, "xbzrle-cache-size", 33554432);
-
-    migrate_set_capability(from, "xbzrle", true);
-    migrate_set_capability(to, "xbzrle", true);
-
     return NULL;
 }
 
@@ -XXX,XX +XXX,XX @@ static void test_precopy_unix_xbzrle(void)
         .listen_uri = uri,
         .start_hook = migrate_hook_start_xbzrle,
         .iterations = 2,
+        .caps[MIGRATION_CAPABILITY_XBZRLE] = true,
         /*
          * XBZRLE needs pages to be modified when doing the 2nd+ round
          * iteration to have real data pushed to the stream.
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_zlib(void)
 {
     MigrateCommon args = {
         .listen_uri = "defer",
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
         .start_hook = migrate_hook_start_precopy_tcp_multifd_zlib,
     };
     test_precopy_common(&args);
diff --git a/tests/qtest/migration/cpr-tests.c b/tests/qtest/migration/cpr-tests.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/cpr-tests.c
+++ b/tests/qtest/migration/cpr-tests.c
@@ -XXX,XX +XXX,XX @@ static void *migrate_hook_start_mode_reboot(QTestState *from, QTestState *to)
     migrate_set_parameter_str(from, "mode", "cpr-reboot");
     migrate_set_parameter_str(to, "mode", "cpr-reboot");
 
-    migrate_set_capability(from, "x-ignore-shared", true);
-    migrate_set_capability(to, "x-ignore-shared", true);
-
     return NULL;
 }
 
@@ -XXX,XX +XXX,XX @@ static void test_mode_reboot(void)
         .connect_uri = uri,
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_mode_reboot,
+        .caps[MIGRATION_CAPABILITY_X_IGNORE_SHARED] = true,
     };
 
     test_file_common(&args, true);
diff --git a/tests/qtest/migration/file-tests.c b/tests/qtest/migration/file-tests.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/file-tests.c
+++ b/tests/qtest/migration/file-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_precopy_file_offset_bad(void)
     test_file_common(&args, false);
 }
 
-static void *migrate_hook_start_mapped_ram(QTestState *from,
-                                           QTestState *to)
-{
-    migrate_set_capability(from, "mapped-ram", true);
-    migrate_set_capability(to, "mapped-ram", true);
-
-    return NULL;
-}
-
 static void test_precopy_file_mapped_ram_live(void)
 {
     g_autofree char *uri = g_strdup_printf("file:%s/%s", tmpfs,
@@ -XXX,XX +XXX,XX @@ static void test_precopy_file_mapped_ram_live(void)
     MigrateCommon args = {
         .connect_uri = uri,
         .listen_uri = "defer",
-        .start_hook = migrate_hook_start_mapped_ram,
+        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
     };
 
     test_file_common(&args, false);
@@ -XXX,XX +XXX,XX @@ static void test_precopy_file_mapped_ram(void)
     MigrateCommon args = {
         .connect_uri = uri,
         .listen_uri = "defer",
-        .start_hook = migrate_hook_start_mapped_ram,
+        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
     };
 
     test_file_common(&args, true);
 }
 
-static void *migrate_hook_start_multifd_mapped_ram(QTestState *from,
-                                                   QTestState *to)
-{
-    migrate_hook_start_mapped_ram(from, to);
-
-    migrate_set_parameter_int(from, "multifd-channels", 4);
-    migrate_set_parameter_int(to, "multifd-channels", 4);
-
-    migrate_set_capability(from, "multifd", true);
-    migrate_set_capability(to, "multifd", true);
-
-    return NULL;
-}
-
 static void test_multifd_file_mapped_ram_live(void)
 {
     g_autofree char *uri = g_strdup_printf("file:%s/%s", tmpfs,
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram_live(void)
     MigrateCommon args = {
         .connect_uri = uri,
         .listen_uri = "defer",
-        .start_hook = migrate_hook_start_multifd_mapped_ram,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
+        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
     };
 
     test_file_common(&args, false);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram(void)
     MigrateCommon args = {
         .connect_uri = uri,
         .listen_uri = "defer",
-        .start_hook = migrate_hook_start_multifd_mapped_ram,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
+        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
     };
 
     test_file_common(&args, true);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram(void)
 static void *migrate_hook_start_multifd_mapped_ram_dio(QTestState *from,
                                                        QTestState *to)
 {
-    migrate_hook_start_multifd_mapped_ram(from, to);
-
     migrate_set_parameter_bool(from, "direct-io", true);
     migrate_set_parameter_bool(to, "direct-io", true);
 
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram_dio(void)
         .connect_uri = uri,
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_multifd_mapped_ram_dio,
+        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
     };
 
     if (!probe_o_direct_support(tmpfs)) {
@@ -XXX,XX +XXX,XX @@ static void *migrate_hook_start_multifd_mapped_ram_fdset_dio(QTestState *from,
     fdset_add_fds(from, file, O_WRONLY, 2, true);
     fdset_add_fds(to, file, O_RDONLY, 2, true);
 
-    migrate_hook_start_multifd_mapped_ram(from, to);
     migrate_set_parameter_bool(from, "direct-io", true);
     migrate_set_parameter_bool(to, "direct-io", true);
 
@@ -XXX,XX +XXX,XX @@ static void *migrate_hook_start_multifd_mapped_ram_fdset(QTestState *from,
     fdset_add_fds(from, file, O_WRONLY, 2, false);
     fdset_add_fds(to, file, O_RDONLY, 2, false);
 
-    migrate_hook_start_multifd_mapped_ram(from, to);
-
     return NULL;
 }
 
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram_fdset(void)
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_multifd_mapped_ram_fdset,
         .end_hook = migrate_hook_end_multifd_mapped_ram_fdset,
+        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
     };
 
     test_file_common(&args, true);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_file_mapped_ram_fdset_dio(void)
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_multifd_mapped_ram_fdset_dio,
         .end_hook = migrate_hook_end_multifd_mapped_ram_fdset,
+        .caps[MIGRATION_CAPABILITY_MAPPED_RAM] = true,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
     };
 
     if (!probe_o_direct_support(tmpfs)) {
diff --git a/tests/qtest/migration/framework.c b/tests/qtest/migration/framework.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/framework.c
+++ b/tests/qtest/migration/framework.c
@@ -XXX,XX +XXX,XX @@ static QList *migrate_start_get_qmp_capabilities(const MigrateStart *args)
     return capabilities;
 }
 
+static void set_migration_capabilities(QTestState *from,
+                                        QTestState *to, MigrateCommon *args)
+{
+    /*
+     * MigrationCapability_lookup and MIGRATION_CAPABILITY_ constants
+     * are from qapi-types-migration.h.
+     */
+    for (uint8_t i = 0; i < MIGRATION_CAPABILITY__MAX; i++)
+    {
+        if (!args->caps[i]) {
+            continue;
+        }
+        if (from) {
+            migrate_set_capability(from,
+                            MigrationCapability_lookup.array[i], true);
+        }
+        if (to) {
+            migrate_set_capability(to,
+                            MigrationCapability_lookup.array[i], true);
+        }
+    }
+
+    return;
+}
+
 int migrate_start(QTestState **from, QTestState **to, const char *uri,
                   MigrateStart *args)
 {
@@ -XXX,XX +XXX,XX @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
         args->postcopy_data = args->start_hook(from, to);
     }
 
-    migrate_set_capability(from, "postcopy-ram", true);
-    migrate_set_capability(to, "postcopy-ram", true);
-    migrate_set_capability(to, "postcopy-blocktime", true);
-
-    if (args->postcopy_preempt) {
-        migrate_set_capability(from, "postcopy-preempt", true);
-        migrate_set_capability(to, "postcopy-preempt", true);
-    }
+    /* set postcopy capabilities */
+    args->caps[MIGRATION_CAPABILITY_POSTCOPY_BLOCKTIME] = true;
+    args->caps[MIGRATION_CAPABILITY_POSTCOPY_RAM] = true;
+    set_migration_capabilities(from, to, args);
 
     migrate_ensure_non_converge(from);
-
     migrate_prepare_for_dirty_mem(from);
     qtest_qmp_assert_success(to, "{ 'execute': 'migrate-incoming',"
                              "  'arguments': { "
@@ -XXX,XX +XXX,XX @@ void test_precopy_common(MigrateCommon *args)
         return;
     }
 
+    set_migration_capabilities(from, to, args);
+    if (args->caps[MIGRATION_CAPABILITY_MULTIFD]) {
+        migrate_set_parameter_int(from, "multifd-channels", 16);
+        migrate_set_parameter_int(to, "multifd-channels", 16);
+    }
+
     if (args->start_hook) {
         data_hook = args->start_hook(from, to);
     }
@@ -XXX,XX +XXX,XX @@ void test_file_common(MigrateCommon *args, bool stop_src)
      */
     g_assert_false(args->live);
 
+    set_migration_capabilities(from, to, args);
+    if (args->caps[MIGRATION_CAPABILITY_MULTIFD]) {
+        migrate_set_parameter_int(from, "multifd-channels", 4);
+        migrate_set_parameter_int(to, "multifd-channels", 4);
+    }
+
     if (g_strrstr(args->connect_uri, "offset=")) {
         check_offset = true;
         /*
@@ -XXX,XX +XXX,XX @@ void *migrate_hook_start_precopy_tcp_multifd_common(QTestState *from,
                                                     QTestState *to,
                                                     const char *method)
 {
-    migrate_set_parameter_int(from, "multifd-channels", 16);
-    migrate_set_parameter_int(to, "multifd-channels", 16);
-
     migrate_set_parameter_str(from, "multifd-compression", method);
     migrate_set_parameter_str(to, "multifd-compression", method);
 
-    migrate_set_capability(from, "multifd", true);
-    migrate_set_capability(to, "multifd", true);
-
     /* Start incoming migration from the 1st socket */
     migrate_incoming_qmp(to, "tcp:127.0.0.1:0", NULL, "{}");
 
diff --git a/tests/qtest/migration/framework.h b/tests/qtest/migration/framework.h
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/framework.h
+++ b/tests/qtest/migration/framework.h
@@ -XXX,XX +XXX,XX @@
 #define TEST_FRAMEWORK_H
 
 #include "libqtest.h"
+#include <qapi/qapi-types-migration.h>
 
 #define FILE_TEST_FILENAME "migfile"
 #define FILE_TEST_OFFSET 0x1000
@@ -XXX,XX +XXX,XX @@ typedef struct {
 
     /* Postcopy specific fields */
     void *postcopy_data;
-    bool postcopy_preempt;
     PostcopyRecoveryFailStage postcopy_recovery_fail_stage;
+
+    bool caps[MIGRATION_CAPABILITY__MAX];
 } MigrateCommon;
 
 void wait_for_serial(const char *side);
diff --git a/tests/qtest/migration/postcopy-tests.c b/tests/qtest/migration/postcopy-tests.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/postcopy-tests.c
+++ b/tests/qtest/migration/postcopy-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_postcopy_suspend(void)
 static void test_postcopy_preempt(void)
 {
     MigrateCommon args = {
-        .postcopy_preempt = true,
+        .caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT] = true,
     };
 
     test_postcopy_common(&args);
@@ -XXX,XX +XXX,XX @@ static void test_postcopy_recovery_fail_reconnect(void)
 static void test_postcopy_preempt_recovery(void)
 {
     MigrateCommon args = {
-        .postcopy_preempt = true,
+        .caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT] = true,
     };
 
     test_postcopy_recovery_common(&args);
diff --git a/tests/qtest/migration/precopy-tests.c b/tests/qtest/migration/precopy-tests.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/precopy-tests.c
+++ b/tests/qtest/migration/precopy-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_precopy_tcp_plain(void)
     test_precopy_common(&args);
 }
 
-static void *migrate_hook_start_switchover_ack(QTestState *from, QTestState *to)
-{
-
-    migrate_set_capability(from, "return-path", true);
-    migrate_set_capability(to, "return-path", true);
-
-    migrate_set_capability(from, "switchover-ack", true);
-    migrate_set_capability(to, "switchover-ack", true);
-
-    return NULL;
-}
-
 static void test_precopy_tcp_switchover_ack(void)
 {
     MigrateCommon args = {
         .listen_uri = "tcp:127.0.0.1:0",
-        .start_hook = migrate_hook_start_switchover_ack,
+        .caps[MIGRATION_CAPABILITY_RETURN_PATH] = true,
+        .caps[MIGRATION_CAPABILITY_SWITCHOVER_ACK] = true,
         /*
          * Source VM must be running in order to consider the switchover ACK
          * when deciding to do switchover or not.
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_uri_none(void)
     MigrateCommon args = {
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_precopy_tcp_multifd,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
         /*
          * Multifd is more complicated than most of the features, it
          * directly takes guest page buffers when sending, make sure
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_zero_page_legacy(void)
     MigrateCommon args = {
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_precopy_tcp_multifd_zero_page_legacy,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
         /*
          * Multifd is more complicated than most of the features, it
          * directly takes guest page buffers when sending, make sure
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_no_zero_page(void)
     MigrateCommon args = {
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_precopy_tcp_multifd_no_zero_page,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
         /*
          * Multifd is more complicated than most of the features, it
          * directly takes guest page buffers when sending, make sure
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_channels_none(void)
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_precopy_tcp_multifd,
         .live = true,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
         .connect_channels = ("[ { 'channel-type': 'main',"
                              "    'addr': { 'transport': 'socket',"
                              "              'type': 'inet',"
diff --git a/tests/qtest/migration/tls-tests.c b/tests/qtest/migration/tls-tests.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/tls-tests.c
+++ b/tests/qtest/migration/tls-tests.c
@@ -XXX,XX +XXX,XX @@ static void test_postcopy_tls_psk(void)
 static void test_postcopy_preempt_tls_psk(void)
 {
     MigrateCommon args = {
-        .postcopy_preempt = true,
         .start_hook = migrate_hook_start_tls_psk_match,
         .end_hook = migrate_hook_end_tls_psk,
+        .caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT] = true,
     };
 
     test_postcopy_common(&args);
@@ -XXX,XX +XXX,XX @@ static void test_postcopy_recovery_tls_psk(void)
 static void test_postcopy_preempt_all(void)
 {
     MigrateCommon args = {
-        .postcopy_preempt = true,
         .start_hook = migrate_hook_start_tls_psk_match,
         .end_hook = migrate_hook_end_tls_psk,
+        .caps[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT] = true,
     };
 
     test_postcopy_recovery_common(&args);
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_psk_match(void)
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_multifd_tcp_tls_psk_match,
         .end_hook = migrate_hook_end_tls_psk,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
     };
     test_precopy_common(&args);
 }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_psk_mismatch(void)
         .start_hook = migrate_hook_start_multifd_tcp_tls_psk_mismatch,
         .end_hook = migrate_hook_end_tls_psk,
         .result = MIG_TEST_FAIL,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
     };
     test_precopy_common(&args);
 }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_default_host(void)
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_multifd_tls_x509_default_host,
         .end_hook = migrate_hook_end_tls_x509,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
     };
     test_precopy_common(&args);
 }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_override_host(void)
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_multifd_tls_x509_override_host,
         .end_hook = migrate_hook_end_tls_x509,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
     };
     test_precopy_common(&args);
 }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_mismatch_host(void)
         .start_hook = migrate_hook_start_multifd_tls_x509_mismatch_host,
         .end_hook = migrate_hook_end_tls_x509,
         .result = MIG_TEST_FAIL,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
     };
     test_precopy_common(&args);
 }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_allow_anon_client(void)
         .listen_uri = "defer",
         .start_hook = migrate_hook_start_multifd_tls_x509_allow_anon_client,
         .end_hook = migrate_hook_end_tls_x509,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
     };
     test_precopy_common(&args);
 }
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_reject_anon_client(void)
         .start_hook = migrate_hook_start_multifd_tls_x509_reject_anon_client,
         .end_hook = migrate_hook_end_tls_x509,
         .result = MIG_TEST_FAIL,
+        .caps[MIGRATION_CAPABILITY_MULTIFD] = true,
     };
     test_precopy_common(&args);
 }
--
2.48.1

From: Prasad Pandit <pjp@fedoraproject.org>

Add new qtests to run postcopy migration with multifd
channels enabled.

Signed-off-by: Prasad Pandit <pjp@fedoraproject.org>
---
 tests/qtest/migration/compression-tests.c | 13 ++++++++
 tests/qtest/migration/framework.c         |  4 +++
 tests/qtest/migration/postcopy-tests.c    | 23 +++++++++++++
 tests/qtest/migration/precopy-tests.c     | 19 +++++++++++
 tests/qtest/migration/tls-tests.c         | 40 +++++++++++++++++++++++
 5 files changed, 99 insertions(+)

v6:
- Reorder, make this the second patch in this series.

v5:
- https://lore.kernel.org/qemu-devel/20250205122712.229151-1-ppandit@redhat.com/T/#t