Series comparison

-[PATCH v2 00/20] Use Intel DSA accelerator to offload zero page checking in multifd live migration.
+[PATCH v9 00/12] Use Intel DSA accelerator to offload zero page checking in multifd live migration.
+v9
+* Rebase on top of aa3a285b5bc56a4208b3b57d4a55291e9c260107;
+* Optimize the error handling in multifd_send_setup();
+* Use the correct way for skipping unit test;
+v8
+* Rebase on top of 1cf9bc6eba7506ab6d9de635f224259225f63466;
+* Fixed the hmp parsing crash in migrate_set_parameter;
+* Addressed initialization/cleanup comments from v7;
+v7
+* Rebase on top of f0a5a31c33a8109061c2493e475c8a2f4d022432;
+* Fix a bug that will crash QEMU when DSA initialization failed;
+* Use a more generalized accel-path to support other accelerators;
+* Remove multifd-packet-size in the parameter list;
+v6
+* Rebase on top of 838fc0a8769d7cc6edfe50451ba4e3368395f5c1;
+* Refactor code to have clean history on all commits;
+* Add comments on DSA specific defines about how the value is picked;
+* Address all comments from v5 reviews about api defines, questions, etc.;
+v5
+* Rebase on top of 39a032cea23e522268519d89bb738974bc43b6f6.
+* Rename struct definitions with typedef and CamelCase names;
+* Add build and runtime checks about DSA accelerator;
+* Address all comments from v4 reviews about typos, licenses, comments,
+error reporting, etc.
+v4
+* Rebase on top of 85b597413d4370cb168f711192eaef2eb70535ac.
+* A separate "multifd zero page checking" patchset was split from this
+patchset's v3 and got merged into master. v4 re-applied the rest of all
+commits on top of that patchset, re-factored and re-tested.
+https://lore.kernel.org/all/20240311180015.3359271-1-hao.xiang@linux.dev/
+* There are some feedback from v3 I likely overlooked.
+v3
+* Rebase on top of 7425b6277f12e82952cede1f531bfc689bf77fb1.
+* Fix error/warning from checkpatch.pl
+* Fix use-after-free bug when multifd-dsa-accel option is not set.
+* Handle error from dsa_init and correctly propogate the error.
+* Remove unnecessary call to dsa_stop.
+* Detect availability of DSA feature at compile time.
+* Implement a generic batch_task structure and a DSA specific one dsa_batch_task.
+* Remove all exit() calls and propagate errors correctly.
+* Use bytes instead of page count to configure multifd-packet-size option.
 v2
 * Rebase on top of 3e01f1147a16ca566694b97eafc941d62fa1e8d8.
 * Leave Juan's changes in their original form instead of squashing them.
 * Add a new commit to refactor the multifd_send_thread function to prepare for introducing the DSA offload functionality.
 * Use page count to configure multifd-packet-size option.
 ...
 CPU to DSA accelerator hardware. This patchset implements a solution to offload
 QEMU's zero page checking from CPU to DSA accelerator hardware. We gain
 two benefits from this change:
 . Reduces CPU usage in multifd live migration workflow across all use
 cases.
 . Reduces migration total time in some use cases.
 * Design:
 These are the logical steps to perform DSA offloading:
 . Configure DSA accelerators and create user space openable DSA work
 ...
 CPU MHz:             2538.624
 CPU max MHz:         3800.0000
 CPU min MHz:         800.0000
 We perform multifd live migration with below setup:
 . VM has 100GB memory.
 . Use the new migration option multifd-set-normal-page-ratio to control the total
 size of the payload sent over the network.
 . Use 8 multifd channels.
 . Use tcp for live migration.
 . Use CPU to perform zero page checking as the baseline.
 ...
 that is 23% total CPU usage savings.
     Latency
     |---------------|---------------|---------------|---------------|---------------|---------------|
     |        |total time    |down time    |throughput    |transferred-ram|total-ram    |
     |---------------|---------------|---------------|---------------|---------------|---------------|
     |Baseline    |10343 ms    |161 ms        |41007.00 mbps    |51583797 kb    |102400520 kb    |
     |---------------|---------------|---------------|---------------|-------------------------------|
     |DSA offload    |9535 ms    |135 ms        |46554.40 mbps    |53947545 kb    |102400520 kb    |
     |---------------|---------------|---------------|---------------|---------------|---------------|
 Total time is 8% faster and down time is 16% faster.
 B) Scenario 2: 100% (100GB) zero pages on an 100GB vm.
 ...
 msec. That is 22% CPU savings.
     Latency
     |--------------------------------------------------------------------------------------------|
     |        |total time    |down time    |throughput    |transferred-ram|total-ram   |
     |---------------|---------------|---------------|---------------|---------------|------------|
     |Baseline    |4867 ms    |20 ms        |1.51 mbps    |565 kb        |102400520 kb|
     |---------------|---------------|---------------|---------------|----------------------------|
     |DSA offload    |3888 ms    |18 ms        |1.89 mbps    |565 kb        |102400520 kb|
     |---------------|---------------|---------------|---------------|---------------|------------|
 Total time 20% faster and down time 10% faster.
 * Testing:
 . Added unit tests for cover the added code path in dsa.c
 . Added integration tests to cover multifd live migration using DSA
 offloading.
-* Patchset
+Hao Xiang (10):
 Apply this patchset on top of commit
 f78ea7ddb0e18766ece9fdfe02061744a7afc41b
 Hao Xiang (16):
   meson: Introduce new instruction set enqcmd to the build system.
-  util/dsa: Add dependency idxd.
   util/dsa: Implement DSA device start and stop logic.
   util/dsa: Implement DSA task enqueue and dequeue.
   util/dsa: Implement DSA task asynchronous completion thread model.
   util/dsa: Implement zero page checking in DSA task.
   util/dsa: Implement DSA task asynchronous submission and wait for
     completion.
   migration/multifd: Add new migration option for multifd DSA
     offloading.
-  migration/multifd: Prepare to introduce DSA acceleration on the
-    multifd path.
   migration/multifd: Enable DSA offloading in multifd sender path.
-  migration/multifd: Add test hook to set normal page ratio.
-  migration/multifd: Enable set normal page ratio test hook in multifd.
-  migration/multifd: Add migration option set packet size.
-  migration/multifd: Enable set packet size migration option.
   util/dsa: Add unit test coverage for Intel DSA task submission and
     completion.
   migration/multifd: Add integration tests for multifd with Intel DSA
     offloading.
-Juan Quintela (4):
+Yichen Wang (1):
-  multifd: Add capability to enable/disable zero_page
+  util/dsa: Add idxd into linux header copy list.
-  multifd: Support for zero pages transmission
-  multifd: Zero pages transmission
+Yuan Liu (1):
-  So we use multifd to transmit zero pages.
+  migration/doc: Add DSA zero page detection doc
- include/qemu/dsa.h             |  119 ++++
+ .../migration/dsa-zero-page-detection.rst     |  290 +++++
- linux-headers/linux/idxd.h     |  356 ++++++++++
+ docs/devel/migration/features.rst             |    1 +
- meson.build                    |    2 +
+ hmp-commands.hx                               |    2 +-
- meson_options.txt              |    2 +
+ include/qemu/dsa.h                            |  190 +++
- migration/migration-hmp-cmds.c |   22 +
+ meson.build                                   |   14 +
- migration/multifd-zlib.c       |    8 +-
+ meson_options.txt                             |    2 +
- migration/multifd-zstd.c       |    8 +-
+ migration/migration-hmp-cmds.c                |   20 +-
- migration/multifd.c            |  203 +++++-
+ migration/multifd-zero-page.c                 |  149 ++-
- migration/multifd.h            |   28 +-
+ migration/multifd.c                           |   23 +-
- migration/options.c            |  107 +++
+ migration/multifd.h                           |    6 +
- migration/options.h            |    4 +
+ migration/options.c                           |   43 +
- migration/ram.c                |   45 +-
+ migration/options.h                           |    2 +
- migration/trace-events         |    8 +-
+ qapi/migration.json                           |   32 +-
- qapi/migration.json            |   53 +-
+ scripts/meson-buildoptions.sh                 |    3 +
- scripts/meson-buildoptions.sh  |    3 +
+ scripts/update-linux-headers.sh               |    2 +-
- tests/qtest/migration-test.c   |   77 ++-
+ tests/qtest/meson.build                       |   10 +-
- tests/unit/meson.build         |    6 +
+ tests/qtest/migration-test.c                  |    3 +
- tests/unit/test-dsa.c          |  466 +++++++++++++
+ tests/qtest/migration/dsa-tests.c             |   59 +
- util/dsa.c                     | 1132 ++++++++++++++++++++++++++++++++
+ tests/qtest/migration/framework.h             |    1 +
- util/meson.build               |    1 +
+ tests/unit/meson.build                        |    6 +
-files changed, 2612 insertions(+), 38 deletions(-)
+ tests/unit/test-dsa.c                         |  504 ++++++++
  util/dsa.c                                    | 1112 +++++++++++++++++
  util/meson.build                              |    3 +
 files changed, 2453 insertions(+), 24 deletions(-)
  create mode 100644 docs/devel/migration/dsa-zero-page-detection.rst
  create mode 100644 include/qemu/dsa.h
- create mode 100644 linux-headers/linux/idxd.h
+ create mode 100644 tests/qtest/migration/dsa-tests.c
  create mode 100644 tests/unit/test-dsa.c
  create mode 100644 util/dsa.c
 --
-.30.2
+Yichen Wang

-[PATCH v2 01/20] multifd: Add capability to enable/disable zero_page
+Deleted patch
-From: Juan Quintela <quintela@redhat.com>
-We have to enable it by default until we introduce the new code.
-Signed-off-by: Juan Quintela <quintela@redhat.com>
----
- migration/options.c | 13 +++++++++++++
- migration/options.h |  1 +
- qapi/migration.json |  8 +++++++-
-files changed, 21 insertions(+), 1 deletion(-)
-diff --git a/migration/options.c b/migration/options.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/options.c
-+++ b/migration/options.c
-@@ -XXX,XX +XXX,XX @@ Property migration_properties[] = {
-     DEFINE_PROP_MIG_CAP("x-switchover-ack",
-                         MIGRATION_CAPABILITY_SWITCHOVER_ACK),
-     DEFINE_PROP_MIG_CAP("x-dirty-limit", MIGRATION_CAPABILITY_DIRTY_LIMIT),
-+    DEFINE_PROP_MIG_CAP("main-zero-page",
-+            MIGRATION_CAPABILITY_MAIN_ZERO_PAGE),
-     DEFINE_PROP_END_OF_LIST(),
- };
-@@ -XXX,XX +XXX,XX @@ bool migrate_multifd(void)
-     return s->capabilities[MIGRATION_CAPABILITY_MULTIFD];
- }
-+bool migrate_use_main_zero_page(void)
-+{
-+    //MigrationState *s;
-+
-+    //s = migrate_get_current();
-+
-+    // We will enable this when we add the right code.
-+    // return s->enabled_capabilities[MIGRATION_CAPABILITY_MAIN_ZERO_PAGE];
-+    return true;
-+}
-+
- bool migrate_pause_before_switchover(void)
- {
-     MigrationState *s = migrate_get_current();
-diff --git a/migration/options.h b/migration/options.h
-index XXXXXXX..XXXXXXX 100644
---- a/migration/options.h
-+++ b/migration/options.h
-@@ -XXX,XX +XXX,XX @@ int migrate_multifd_channels(void);
- MultiFDCompression migrate_multifd_compression(void);
- int migrate_multifd_zlib_level(void);
- int migrate_multifd_zstd_level(void);
-+bool migrate_use_main_zero_page(void);
- uint8_t migrate_throttle_trigger_threshold(void);
- const char *migrate_tls_authz(void);
- const char *migrate_tls_creds(void);
-diff --git a/qapi/migration.json b/qapi/migration.json
-index XXXXXXX..XXXXXXX 100644
---- a/qapi/migration.json
-+++ b/qapi/migration.json
-@@ -XXX,XX +XXX,XX @@
- #     and can result in more stable read performance.  Requires KVM
- #     with accelerator property "dirty-ring-size" set.  (Since 8.1)
- #
-+#
-+# @main-zero-page: If enabled, the detection of zero pages will be
-+#                  done on the main thread.  Otherwise it is done on
-+#                  the multifd threads.
-+#                  (since 8.2)
-+#
- # Features:
- #
- # @deprecated: Member @block is deprecated.  Use blockdev-mirror with
-@@ -XXX,XX +XXX,XX @@
-            { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
-            'validate-uuid', 'background-snapshot',
-            'zero-copy-send', 'postcopy-preempt', 'switchover-ack',
--           'dirty-limit'] }
-+           'dirty-limit', 'main-zero-page'] }
- ##
- # @MigrationCapabilityStatus:
---
-.30.2

-[PATCH v2 02/20] multifd: Support for zero pages transmission
+Deleted patch
-From: Juan Quintela <quintela@redhat.com>
-This patch adds counters and similar. Logic will be added on the
-following patch.
-Signed-off-by: Juan Quintela <quintela@redhat.com>
----
- migration/multifd.c    | 37 ++++++++++++++++++++++++++++++-------
- migration/multifd.h    | 17 ++++++++++++++++-
- migration/trace-events |  8 ++++----
-files changed, 50 insertions(+), 12 deletions(-)
-diff --git a/migration/multifd.c b/migration/multifd.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/multifd.c
-+++ b/migration/multifd.c
-@@ -XXX,XX +XXX,XX @@ static void multifd_send_fill_packet(MultiFDSendParams *p)
-     packet->normal_pages = cpu_to_be32(p->normal_num);
-     packet->next_packet_size = cpu_to_be32(p->next_packet_size);
-     packet->packet_num = cpu_to_be64(p->packet_num);
-+    packet->zero_pages = cpu_to_be32(p->zero_num);
-     if (p->pages->block) {
-         strncpy(packet->ramblock, p->pages->block->idstr, 256);
-@@ -XXX,XX +XXX,XX @@ static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp)
-     p->next_packet_size = be32_to_cpu(packet->next_packet_size);
-     p->packet_num = be64_to_cpu(packet->packet_num);
--    if (p->normal_num == 0) {
-+    p->zero_num = be32_to_cpu(packet->zero_pages);
-+    if (p->zero_num > packet->pages_alloc - p->normal_num) {
-+        error_setg(errp, "multifd: received packet "
-+                   "with %u zero pages and expected maximum pages are %u",
-+                   p->zero_num, packet->pages_alloc - p->normal_num) ;
-+        return -1;
-+    }
-+
-+    if (p->normal_num == 0 && p->zero_num == 0) {
-         return 0;
-     }
-@@ -XXX,XX +XXX,XX @@ static int multifd_send_pages(QEMUFile *f)
-     p->packet_num = multifd_send_state->packet_num++;
-     multifd_send_state->pages = p->pages;
-     p->pages = pages;
-+
-     qemu_mutex_unlock(&p->mutex);
-     qemu_sem_post(&p->sem);
-@@ -XXX,XX +XXX,XX @@ void multifd_save_cleanup(void)
-         p->iov = NULL;
-         g_free(p->normal);
-         p->normal = NULL;
-+        g_free(p->zero);
-+        p->zero = NULL;
-         multifd_send_state->ops->send_cleanup(p, &local_err);
-         if (local_err) {
-             migrate_set_error(migrate_get_current(), local_err);
-@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
-             uint64_t packet_num = p->packet_num;
-             uint32_t flags;
-             p->normal_num = 0;
-+            p->zero_num = 0;
-             if (use_zero_copy_send) {
-                 p->iovs_num = 0;
-@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
-             p->flags = 0;
-             p->num_packets++;
-             p->total_normal_pages += p->normal_num;
-+            p->total_zero_pages += p->zero_num;
-             p->pages->num = 0;
-             p->pages->block = NULL;
-             qemu_mutex_unlock(&p->mutex);
--            trace_multifd_send(p->id, packet_num, p->normal_num, flags,
--                               p->next_packet_size);
-+            trace_multifd_send(p->id, packet_num, p->normal_num, p->zero_num,
-+                               flags, p->next_packet_size);
-             if (use_zero_copy_send) {
-                 /* Send header first, without zerocopy */
-@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
-             stat64_add(&mig_stats.multifd_bytes,
-                        p->next_packet_size + p->packet_len);
-+            stat64_add(&mig_stats.normal_pages, p->normal_num);
-+            stat64_add(&mig_stats.zero_pages, p->zero_num);
-             p->next_packet_size = 0;
-             qemu_mutex_lock(&p->mutex);
-             p->pending_job--;
-@@ -XXX,XX +XXX,XX @@ out:
-     rcu_unregister_thread();
-     migration_threads_remove(thread);
--    trace_multifd_send_thread_end(p->id, p->num_packets, p->total_normal_pages);
-+    trace_multifd_send_thread_end(p->id, p->num_packets, p->total_normal_pages,
-+                                  p->total_zero_pages);
-     return NULL;
- }
-@@ -XXX,XX +XXX,XX @@ int multifd_save_setup(Error **errp)
-         p->normal = g_new0(ram_addr_t, page_count);
-         p->page_size = qemu_target_page_size();
-         p->page_count = page_count;
-+        p->zero = g_new0(ram_addr_t, page_count);
-         if (migrate_zero_copy_send()) {
-             p->write_flags = QIO_CHANNEL_WRITE_FLAG_ZERO_COPY;
-@@ -XXX,XX +XXX,XX @@ void multifd_load_cleanup(void)
-         p->iov = NULL;
-         g_free(p->normal);
-         p->normal = NULL;
-+        g_free(p->zero);
-+        p->zero = NULL;
-         multifd_recv_state->ops->recv_cleanup(p);
-     }
-     qemu_sem_destroy(&multifd_recv_state->sem_sync);
-@@ -XXX,XX +XXX,XX @@ static void *multifd_recv_thread(void *opaque)
-         flags = p->flags;
-         /* recv methods don't know how to handle the SYNC flag */
-         p->flags &= ~MULTIFD_FLAG_SYNC;
--        trace_multifd_recv(p->id, p->packet_num, p->normal_num, flags,
--                           p->next_packet_size);
-+        trace_multifd_recv(p->id, p->packet_num, p->normal_num, p->zero_num,
-+                           flags, p->next_packet_size);
-         p->num_packets++;
-         p->total_normal_pages += p->normal_num;
-+        p->total_zero_pages += p->zero_num;
-         qemu_mutex_unlock(&p->mutex);
-         if (p->normal_num) {
-@@ -XXX,XX +XXX,XX @@ static void *multifd_recv_thread(void *opaque)
-     qemu_mutex_unlock(&p->mutex);
-     rcu_unregister_thread();
--    trace_multifd_recv_thread_end(p->id, p->num_packets, p->total_normal_pages);
-+    trace_multifd_recv_thread_end(p->id, p->num_packets, p->total_normal_pages,
-+                                  p->total_zero_pages);
-     return NULL;
- }
-@@ -XXX,XX +XXX,XX @@ int multifd_load_setup(Error **errp)
-         p->normal = g_new0(ram_addr_t, page_count);
-         p->page_count = page_count;
-         p->page_size = qemu_target_page_size();
-+        p->zero = g_new0(ram_addr_t, page_count);
-     }
-     for (i = 0; i < thread_count; i++) {
-diff --git a/migration/multifd.h b/migration/multifd.h
-index XXXXXXX..XXXXXXX 100644
---- a/migration/multifd.h
-+++ b/migration/multifd.h
-@@ -XXX,XX +XXX,XX @@ typedef struct {
-     /* size of the next packet that contains pages */
-     uint32_t next_packet_size;
-     uint64_t packet_num;
--    uint64_t unused[4];    /* Reserved for future use */
-+    /* zero pages */
-+    uint32_t zero_pages;
-+    uint32_t unused32[1];    /* Reserved for future use */
-+    uint64_t unused64[3];    /* Reserved for future use */
-     char ramblock[256];
-     uint64_t offset[];
- } __attribute__((packed)) MultiFDPacket_t;
-@@ -XXX,XX +XXX,XX @@ typedef struct {
-     uint64_t num_packets;
-     /* non zero pages sent through this channel */
-     uint64_t total_normal_pages;
-+    /* zero pages sent through this channel */
-+    uint64_t total_zero_pages;
-     /* buffers to send */
-     struct iovec *iov;
-     /* number of iovs used */
-@@ -XXX,XX +XXX,XX @@ typedef struct {
-     ram_addr_t *normal;
-     /* num of non zero pages */
-     uint32_t normal_num;
-+    /* Pages that are zero */
-+    ram_addr_t *zero;
-+    /* num of zero pages */
-+    uint32_t zero_num;
-     /* used for compression methods */
-     void *data;
- }  MultiFDSendParams;
-@@ -XXX,XX +XXX,XX @@ typedef struct {
-     uint8_t *host;
-     /* non zero pages recv through this channel */
-     uint64_t total_normal_pages;
-+    /* zero pages recv through this channel */
-+    uint64_t total_zero_pages;
-     /* buffers to recv */
-     struct iovec *iov;
-     /* Pages that are not zero */
-     ram_addr_t *normal;
-     /* num of non zero pages */
-     uint32_t normal_num;
-+    /* Pages that are zero */
-+    ram_addr_t *zero;
-+    /* num of zero pages */
-+    uint32_t zero_num;
-     /* used for de-compression methods */
-     void *data;
- } MultiFDRecvParams;
-diff --git a/migration/trace-events b/migration/trace-events
-index XXXXXXX..XXXXXXX 100644
---- a/migration/trace-events
-+++ b/migration/trace-events
-@@ -XXX,XX +XXX,XX @@ postcopy_preempt_reset_channel(void) ""
- # multifd.c
- multifd_new_send_channel_async(uint8_t id) "channel %u"
- multifd_new_send_channel_async_error(uint8_t id, void *err) "channel=%u err=%p"
--multifd_recv(uint8_t id, uint64_t packet_num, uint32_t used, uint32_t flags, uint32_t next_packet_size) "channel %u packet_num %" PRIu64 " pages %u flags 0x%x next packet size %u"
-+multifd_recv(uint8_t id, uint64_t packet_num, uint32_t normal, uint32_t zero, uint32_t flags, uint32_t next_packet_size) "channel %u packet_num %" PRIu64 " normal pages %u zero pages %u flags 0x%x next packet size %u"
- multifd_recv_new_channel(uint8_t id) "channel %u"
- multifd_recv_sync_main(long packet_num) "packet num %ld"
- multifd_recv_sync_main_signal(uint8_t id) "channel %u"
- multifd_recv_sync_main_wait(uint8_t id) "channel %u"
- multifd_recv_terminate_threads(bool error) "error %d"
--multifd_recv_thread_end(uint8_t id, uint64_t packets, uint64_t pages) "channel %u packets %" PRIu64 " pages %" PRIu64
-+multifd_recv_thread_end(uint8_t id, uint64_t packets, uint64_t normal_pages, uint64_t zero_pages) "channel %u packets %" PRIu64 " normal pages %" PRIu64 " zero pages %" PRIu64
- multifd_recv_thread_start(uint8_t id) "%u"
--multifd_send(uint8_t id, uint64_t packet_num, uint32_t normal, uint32_t flags, uint32_t next_packet_size) "channel %u packet_num %" PRIu64 " normal pages %u flags 0x%x next packet size %u"
-+multifd_send(uint8_t id, uint64_t packet_num, uint32_t normalpages, uint32_t zero_pages, uint32_t flags, uint32_t next_packet_size) "channel %u packet_num %" PRIu64 " normal pages %u zero pages %u flags 0x%x next packet size %u"
- multifd_send_error(uint8_t id) "channel %u"
- multifd_send_sync_main(long packet_num) "packet num %ld"
- multifd_send_sync_main_signal(uint8_t id) "channel %u"
- multifd_send_sync_main_wait(uint8_t id) "channel %u"
- multifd_send_terminate_threads(bool error) "error %d"
--multifd_send_thread_end(uint8_t id, uint64_t packets, uint64_t normal_pages) "channel %u packets %" PRIu64 " normal pages %"  PRIu64
-+multifd_send_thread_end(uint8_t id, uint64_t packets, uint64_t normal_pages, uint64_t zero_pages) "channel %u packets %" PRIu64 " normal pages %"  PRIu64 " zero pages %"  PRIu64
- multifd_send_thread_start(uint8_t id) "%u"
- multifd_tls_outgoing_handshake_start(void *ioc, void *tioc, const char *hostname) "ioc=%p tioc=%p hostname=%s"
- multifd_tls_outgoing_handshake_error(void *ioc, const char *err) "ioc=%p err=%s"
---
-.30.2

-[PATCH v2 03/20] multifd: Zero pages transmission
+Deleted patch
-From: Juan Quintela <quintela@redhat.com>
-This implements the zero page dection and handling.
-Signed-off-by: Juan Quintela <quintela@redhat.com>
----
- migration/multifd.c | 41 +++++++++++++++++++++++++++++++++++++++--
- migration/multifd.h |  5 +++++
-files changed, 44 insertions(+), 2 deletions(-)
-diff --git a/migration/multifd.c b/migration/multifd.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/multifd.c
-+++ b/migration/multifd.c
-@@ -XXX,XX +XXX,XX @@
-  */
- #include "qemu/osdep.h"
-+#include "qemu/cutils.h"
- #include "qemu/rcu.h"
- #include "exec/target_page.h"
- #include "sysemu/sysemu.h"
-@@ -XXX,XX +XXX,XX @@ static void multifd_send_fill_packet(MultiFDSendParams *p)
-         packet->offset[i] = cpu_to_be64(temp);
-     }
-+    for (i = 0; i < p->zero_num; i++) {
-+        /* there are architectures where ram_addr_t is 32 bit */
-+        uint64_t temp = p->zero[i];
-+
-+        packet->offset[p->normal_num + i] = cpu_to_be64(temp);
-+    }
- }
- static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp)
-@@ -XXX,XX +XXX,XX @@ static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp)
-         p->normal[i] = offset;
-     }
-+    for (i = 0; i < p->zero_num; i++) {
-+        uint64_t offset = be64_to_cpu(packet->offset[p->normal_num + i]);
-+
-+        if (offset > (p->block->used_length - p->page_size)) {
-+            error_setg(errp, "multifd: offset too long %" PRIu64
-+                       " (max " RAM_ADDR_FMT ")",
-+                       offset, p->block->used_length);
-+            return -1;
-+        }
-+        p->zero[i] = offset;
-+    }
-+
-     return 0;
- }
-@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
-     MultiFDSendParams *p = opaque;
-     MigrationThread *thread = NULL;
-     Error *local_err = NULL;
-+    /* qemu older than 8.2 don't understand zero page on multifd channel */
-+    bool use_zero_page = !migrate_use_main_zero_page();
-     int ret = 0;
-     bool use_zero_copy_send = migrate_zero_copy_send();
-@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
-         qemu_mutex_lock(&p->mutex);
-         if (p->pending_job) {
-+            RAMBlock *rb = p->pages->block;
-             uint64_t packet_num = p->packet_num;
-             uint32_t flags;
-             p->normal_num = 0;
-@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
-             }
-             for (int i = 0; i < p->pages->num; i++) {
--                p->normal[p->normal_num] = p->pages->offset[i];
--                p->normal_num++;
-+                uint64_t offset = p->pages->offset[i];
-+                if (use_zero_page &&
-+                    buffer_is_zero(rb->host + offset, p->page_size)) {
-+                    p->zero[p->zero_num] = offset;
-+                    p->zero_num++;
-+                    ram_release_page(rb->idstr, offset);
-+                } else {
-+                    p->normal[p->normal_num] = offset;
-+                    p->normal_num++;
-+                }
-             }
-             if (p->normal_num) {
-@@ -XXX,XX +XXX,XX @@ static void *multifd_recv_thread(void *opaque)
-             }
-         }
-+        for (int i = 0; i < p->zero_num; i++) {
-+            void *page = p->host + p->zero[i];
-+            if (!buffer_is_zero(page, p->page_size)) {
-+                memset(page, 0, p->page_size);
-+            }
-+        }
-+
-         if (flags & MULTIFD_FLAG_SYNC) {
-             qemu_sem_post(&multifd_recv_state->sem_sync);
-             qemu_sem_wait(&p->sem_sync);
-diff --git a/migration/multifd.h b/migration/multifd.h
-index XXXXXXX..XXXXXXX 100644
---- a/migration/multifd.h
-+++ b/migration/multifd.h
-@@ -XXX,XX +XXX,XX @@ typedef struct {
-     uint32_t unused32[1];    /* Reserved for future use */
-     uint64_t unused64[3];    /* Reserved for future use */
-     char ramblock[256];
-+    /*
-+     * This array contains the pointers to:
-+     *  - normal pages (initial normal_pages entries)
-+     *  - zero pages (following zero_pages entries)
-+     */
-     uint64_t offset[];
- } __attribute__((packed)) MultiFDPacket_t;
---
-.30.2

-[PATCH v2 04/20] So we use multifd to transmit zero pages.
+Deleted patch
-From: Juan Quintela <quintela@redhat.com>
-Signed-off-by: Juan Quintela <quintela@redhat.com>
-Reviewed-by: Leonardo Bras <leobras@redhat.com>
----
- migration/multifd.c |  7 ++++---
- migration/options.c | 13 +++++++------
- migration/ram.c     | 45 ++++++++++++++++++++++++++++++++++++++-------
- qapi/migration.json |  1 -
-files changed, 49 insertions(+), 17 deletions(-)
-diff --git a/migration/multifd.c b/migration/multifd.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/multifd.c
-+++ b/migration/multifd.c
-@@ -XXX,XX +XXX,XX @@
- #include "qemu/osdep.h"
- #include "qemu/cutils.h"
- #include "qemu/rcu.h"
-+#include "qemu/cutils.h"
- #include "exec/target_page.h"
- #include "sysemu/sysemu.h"
- #include "exec/ramblock.h"
-@@ -XXX,XX +XXX,XX @@ static int multifd_send_pages(QEMUFile *f)
-     p->packet_num = multifd_send_state->packet_num++;
-     multifd_send_state->pages = p->pages;
-     p->pages = pages;
--
-     qemu_mutex_unlock(&p->mutex);
-     qemu_sem_post(&p->sem);
-@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
-     MigrationThread *thread = NULL;
-     Error *local_err = NULL;
-     /* qemu older than 8.2 don't understand zero page on multifd channel */
--    bool use_zero_page = !migrate_use_main_zero_page();
-+    bool use_multifd_zero_page = !migrate_use_main_zero_page();
-     int ret = 0;
-     bool use_zero_copy_send = migrate_zero_copy_send();
-@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
-             RAMBlock *rb = p->pages->block;
-             uint64_t packet_num = p->packet_num;
-             uint32_t flags;
-+
-             p->normal_num = 0;
-             p->zero_num = 0;
-@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
-             for (int i = 0; i < p->pages->num; i++) {
-                 uint64_t offset = p->pages->offset[i];
--                if (use_zero_page &&
-+                if (use_multifd_zero_page &&
-                     buffer_is_zero(rb->host + offset, p->page_size)) {
-                     p->zero[p->zero_num] = offset;
-                     p->zero_num++;
-diff --git a/migration/options.c b/migration/options.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/options.c
-+++ b/migration/options.c
-@@ -XXX,XX +XXX,XX @@ Property migration_properties[] = {
-     DEFINE_PROP_MIG_CAP("x-block", MIGRATION_CAPABILITY_BLOCK),
-     DEFINE_PROP_MIG_CAP("x-return-path", MIGRATION_CAPABILITY_RETURN_PATH),
-     DEFINE_PROP_MIG_CAP("x-multifd", MIGRATION_CAPABILITY_MULTIFD),
-+    DEFINE_PROP_MIG_CAP("x-main-zero-page", MIGRATION_CAPABILITY_MAIN_ZERO_PAGE),
-     DEFINE_PROP_MIG_CAP("x-background-snapshot",
-             MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT),
- #ifdef CONFIG_LINUX
-@@ -XXX,XX +XXX,XX @@ bool migrate_multifd(void)
- bool migrate_use_main_zero_page(void)
- {
--    //MigrationState *s;
--
--    //s = migrate_get_current();
-+    MigrationState *s = migrate_get_current();
--    // We will enable this when we add the right code.
--    // return s->enabled_capabilities[MIGRATION_CAPABILITY_MAIN_ZERO_PAGE];
--    return true;
-+    return s->capabilities[MIGRATION_CAPABILITY_MAIN_ZERO_PAGE];
- }
- bool migrate_pause_before_switchover(void)
-@@ -XXX,XX +XXX,XX @@ INITIALIZE_MIGRATE_CAPS_SET(check_caps_background_snapshot,
-     MIGRATION_CAPABILITY_LATE_BLOCK_ACTIVATE,
-     MIGRATION_CAPABILITY_RETURN_PATH,
-     MIGRATION_CAPABILITY_MULTIFD,
-+    MIGRATION_CAPABILITY_MAIN_ZERO_PAGE,
-     MIGRATION_CAPABILITY_PAUSE_BEFORE_SWITCHOVER,
-     MIGRATION_CAPABILITY_AUTO_CONVERGE,
-     MIGRATION_CAPABILITY_RELEASE_RAM,
-@@ -XXX,XX +XXX,XX @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp)
-             error_setg(errp, "Postcopy is not yet compatible with multifd");
-             return false;
-         }
-+        if (new_caps[MIGRATION_CAPABILITY_MAIN_ZERO_PAGE]) {
-+            error_setg(errp, "Postcopy is not yet compatible with main zero copy");
-+        }
-     }
-     if (new_caps[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT]) {
-diff --git a/migration/ram.c b/migration/ram.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/ram.c
-+++ b/migration/ram.c
-@@ -XXX,XX +XXX,XX @@ static int ram_save_target_page_legacy(RAMState *rs, PageSearchStatus *pss)
-     if (save_zero_page(rs, pss, offset)) {
-         return 1;
-     }
--
-     /*
--     * Do not use multifd in postcopy as one whole host page should be
--     * placed.  Meanwhile postcopy requires atomic update of pages, so even
--     * if host page size == guest page size the dest guest during run may
--     * still see partially copied pages which is data corruption.
-+     * Do not use multifd for:
-+     * 1. Compression as the first page in the new block should be posted out
-+     *    before sending the compressed page
-+     * 2. In postcopy as one whole host page should be placed
-      */
--    if (migrate_multifd() && !migration_in_postcopy()) {
-+    if (!migrate_compress() && migrate_multifd() && !migration_in_postcopy()) {
-+        return ram_save_multifd_page(pss->pss_channel, block, offset);
-+    }
-+
-+    return ram_save_page(rs, pss);
-+}
-+
-+/**
-+ * ram_save_target_page_multifd: save one target page
-+ *
-+ * Returns the number of pages written
-+ *
-+ * @rs: current RAM state
-+ * @pss: data about the page we want to send
-+ */
-+static int ram_save_target_page_multifd(RAMState *rs, PageSearchStatus *pss)
-+{
-+    RAMBlock *block = pss->block;
-+    ram_addr_t offset = ((ram_addr_t)pss->page) << TARGET_PAGE_BITS;
-+    int res;
-+
-+    if (!migration_in_postcopy()) {
-         return ram_save_multifd_page(pss->pss_channel, block, offset);
-     }
-+    res = save_zero_page(rs, pss, offset);
-+    if (res > 0) {
-+        return res;
-+    }
-+
-     return ram_save_page(rs, pss);
- }
-@@ -XXX,XX +XXX,XX @@ static int ram_save_setup(QEMUFile *f, void *opaque)
-     }
-     migration_ops = g_malloc0(sizeof(MigrationOps));
--    migration_ops->ram_save_target_page = ram_save_target_page_legacy;
-+
-+    if (migrate_multifd() && !migrate_use_main_zero_page()) {
-+        migration_ops->ram_save_target_page = ram_save_target_page_multifd;
-+    } else {
-+        migration_ops->ram_save_target_page = ram_save_target_page_legacy;
-+    }
-     qemu_mutex_unlock_iothread();
-+
-     ret = multifd_send_sync_main(f);
-     qemu_mutex_lock_iothread();
-     if (ret < 0) {
-diff --git a/qapi/migration.json b/qapi/migration.json
-index XXXXXXX..XXXXXXX 100644
---- a/qapi/migration.json
-+++ b/qapi/migration.json
-@@ -XXX,XX +XXX,XX @@
- #     and can result in more stable read performance.  Requires KVM
- #     with accelerator property "dirty-ring-size" set.  (Since 8.1)
- #
--#
- # @main-zero-page: If enabled, the detection of zero pages will be
- #                  done on the main thread.  Otherwise it is done on
- #                  the multifd threads.
---
-.30.2

-[PATCH v2 05/20] meson: Introduce new instruction set enqcmd to the build system.
+[PATCH v9 01/12] meson: Introduce new instruction set enqcmd to the build system.
+From: Hao Xiang <hao.xiang@linux.dev>
 Enable instruction set enqcmd in build.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
 Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
 Reviewed-by: Fabiano Rosas <farosas@suse.de>
 ---
- meson.build                   | 2 ++
+ meson.build                   | 14 ++++++++++++++
- meson_options.txt             | 2 ++
+ meson_options.txt             |  2 ++
- scripts/meson-buildoptions.sh | 3 +++
+ scripts/meson-buildoptions.sh |  3 +++
-files changed, 7 insertions(+)
+files changed, 19 insertions(+)
 diff --git a/meson.build b/meson.build
 index XXXXXXX..XXXXXXX 100644
 --- a/meson.build
 +++ b/meson.build
 @@ -XXX,XX +XXX,XX @@ config_host_data.set('CONFIG_AVX512BW_OPT', get_option('avx512bw') \
      int main(int argc, char *argv[]) { return bar(argv[0]); }
    '''), error_message: 'AVX512BW not available').allowed())
-+config_host_data.set('CONFIG_DSA_OPT', get_option('enqcmd'))
++config_host_data.set('CONFIG_DSA_OPT', get_option('enqcmd') \
 +  .require(have_cpuid_h, error_message: 'cpuid.h not available, cannot enable ENQCMD') \
 +  .require(cc.links('''
 +    #include <stdint.h>
 +    #include <cpuid.h>
 +    #include <immintrin.h>
 +    static int __attribute__((target("enqcmd"))) bar(void *a) {
 +      uint64_t dst[8] = { 0 };
 +      uint64_t src[8] = { 0 };
 +      return _enqcmd(dst, src);
 +    }
 +    int main(int argc, char *argv[]) { return bar(argv[argc - 1]); }
 +  '''), error_message: 'ENQCMD not available').allowed())
 +
  # For both AArch64 and AArch32, detect if builtins are available.
  config_host_data.set('CONFIG_ARM_AES_BUILTIN', cc.compiles('''
      #include <arm_neon.h>
 diff --git a/meson_options.txt b/meson_options.txt
 index XXXXXXX..XXXXXXX 100644
 --- a/meson_options.txt
 +++ b/meson_options.txt
-@@ -XXX,XX +XXX,XX @@ option('avx512f', type: 'feature', value: 'disabled',
+@@ -XXX,XX +XXX,XX @@ option('avx2', type: 'feature', value: 'auto',
-        description: 'AVX512F optimizations')
+        description: 'AVX2 optimizations')
  option('avx512bw', type: 'feature', value: 'auto',
         description: 'AVX512BW optimizations')
-+option('enqcmd', type: 'boolean', value: false,
++option('enqcmd', type: 'feature', value: 'disabled',
-+       description: 'MENQCMD optimizations')
++       description: 'ENQCMD optimizations')
  option('keyring', type: 'feature', value: 'auto',
         description: 'Linux keyring support')
  option('libkeyutils', type: 'feature', value: 'auto',
 diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
 index XXXXXXX..XXXXXXX 100644
 --- a/scripts/meson-buildoptions.sh
 +++ b/scripts/meson-buildoptions.sh
 @@ -XXX,XX +XXX,XX @@ meson_options_help() {
+   printf "%s\n" '  auth-pam        PAM access control'
    printf "%s\n" '  avx2            AVX2 optimizations'
    printf "%s\n" '  avx512bw        AVX512BW optimizations'
-   printf "%s\n" '  avx512f         AVX512F optimizations'
 +  printf "%s\n" '  enqcmd          ENQCMD optimizations'
    printf "%s\n" '  blkio           libblkio block device driver'
    printf "%s\n" '  bochs           bochs image format support'
    printf "%s\n" '  bpf             eBPF support'
 @@ -XXX,XX +XXX,XX @@ _meson_option_parse() {
+     --disable-avx2) printf "%s" -Davx2=disabled ;;
+     --enable-avx512bw) printf "%s" -Davx512bw=enabled ;;
      --disable-avx512bw) printf "%s" -Davx512bw=disabled ;;
-     --enable-avx512f) printf "%s" -Davx512f=enabled ;;
++    --enable-enqcmd) printf "%s" -Denqcmd=enabled ;;
-     --disable-avx512f) printf "%s" -Davx512f=disabled ;;
++    --disable-enqcmd) printf "%s" -Denqcmd=disabled ;;
 +    --enable-enqcmd) printf "%s" -Denqcmd=true ;;
 +    --disable-enqcmd) printf "%s" -Denqcmd=false ;;
      --enable-gcov) printf "%s" -Db_coverage=true ;;
      --disable-gcov) printf "%s" -Db_coverage=false ;;
      --enable-lto) printf "%s" -Db_lto=true ;;
 --
-.30.2
+Yichen Wang

-[PATCH v2 13/20] migration/multifd: Prepare to introduce DSA acceleration on the multifd path.
+[PATCH v9 02/12] util/dsa: Add idxd into linux header copy list.
-. Refactor multifd_send_thread function.
+Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
-. Implement buffer_is_zero_use_cpu to handle CPU based zero page
+Reviewed-by: Fabiano Rosas <farosas@suse.de>
-checking.
+---
-. Introduce the batch task structure in MultiFDSendParams.
+ scripts/update-linux-headers.sh | 2 +-
 file changed, 1 insertion(+), 1 deletion(-)
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+diff --git a/scripts/update-linux-headers.sh b/scripts/update-linux-headers.sh
----
+index XXXXXXX..XXXXXXX 100755
- migration/multifd.c | 82 ++++++++++++++++++++++++++++++++++++---------
+--- a/scripts/update-linux-headers.sh
- migration/multifd.h |  3 ++
++++ b/scripts/update-linux-headers.sh
-files changed, 70 insertions(+), 15 deletions(-)
+@@ -XXX,XX +XXX,XX @@ rm -rf "$output/linux-headers/linux"
+ mkdir -p "$output/linux-headers/linux"
-diff --git a/migration/multifd.c b/migration/multifd.c
+ for header in const.h stddef.h kvm.h vfio.h vfio_ccw.h vfio_zdev.h vhost.h \
-index XXXXXXX..XXXXXXX 100644
+               psci.h psp-sev.h userfaultfd.h memfd.h mman.h nvme_ioctl.h \
---- a/migration/multifd.c
+-              vduse.h iommufd.h bits.h; do
-+++ b/migration/multifd.c
++              vduse.h iommufd.h bits.h idxd.h; do
-@@ -XXX,XX +XXX,XX @@
+     cp "$hdrdir/include/linux/$header" "$output/linux-headers/linux"
- #include "qemu/cutils.h"
+ done
  #include "qemu/rcu.h"
  #include "qemu/cutils.h"
 +#include "qemu/dsa.h"
 +#include "qemu/memalign.h"
  #include "exec/target_page.h"
  #include "sysemu/sysemu.h"
  #include "exec/ramblock.h"
@@ -XXX,XX +XXX,XX @@ void multifd_save_cleanup(void)
          p->name = NULL;
          multifd_pages_clear(p->pages);
          p->pages = NULL;
 +        g_free(p->addr);
 +        p->addr = NULL;
 +        buffer_zero_batch_task_destroy(p->batch_task);
 +        qemu_vfree(p->batch_task);
 +        p->batch_task = NULL;
          p->packet_len = 0;
          g_free(p->packet);
          p->packet = NULL;
@@ -XXX,XX +XXX,XX @@ int multifd_send_sync_main(QEMUFile *f)
      return 0;
  }
 +static void set_page(MultiFDSendParams *p, bool zero_page, uint64_t offset)
 +{
 +    RAMBlock *rb = p->pages->block;
 +    if (zero_page) {
 +        p->zero[p->zero_num] = offset;
 +        p->zero_num++;
 +        ram_release_page(rb->idstr, offset);
 +    } else {
 +        p->normal[p->normal_num] = offset;
 +        p->normal_num++;
 +    }
 +}
 +
 +static void buffer_is_zero_use_cpu(MultiFDSendParams *p)
 +{
 +    const void **buf = (const void **)p->addr;
 +    assert(!migrate_use_main_zero_page());
 +
 +    for (int i = 0; i < p->pages->num; i++) {
 +        p->batch_task->results[i] = buffer_is_zero(buf[i], p->page_size);
 +    }
 +}
 +
 +static void set_normal_pages(MultiFDSendParams *p)
 +{
 +    for (int i = 0; i < p->pages->num; i++) {
 +        p->batch_task->results[i] = false;
 +    }
 +}
 +
 +static void multifd_zero_page_check(MultiFDSendParams *p)
 +{
 +    /* older qemu don't understand zero page on multifd channel */
 +    bool use_multifd_zero_page = !migrate_use_main_zero_page();
 +
 +    RAMBlock *rb = p->pages->block;
 +
 +    for (int i = 0; i < p->pages->num; i++) {
 +        p->addr[i] = (ram_addr_t)(rb->host + p->pages->offset[i]);
 +    }
 +
 +    if (use_multifd_zero_page) {
 +        buffer_is_zero_use_cpu(p);
 +    } else {
 +        // No zero page checking. All pages are normal pages.
 +        set_normal_pages(p);
 +    }
 +
 +    for (int i = 0; i < p->pages->num; i++) {
 +        uint64_t offset = p->pages->offset[i];
 +        bool zero_page = p->batch_task->results[i];
 +        set_page(p, zero_page, offset);
 +    }
 +}
 +
  static void *multifd_send_thread(void *opaque)
  {
      MultiFDSendParams *p = opaque;
      MigrationThread *thread = NULL;
      Error *local_err = NULL;
 -    /* qemu older than 8.2 don't understand zero page on multifd channel */
 -    bool use_multifd_zero_page = !migrate_use_main_zero_page();
      int ret = 0;
      bool use_zero_copy_send = migrate_zero_copy_send();
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
          qemu_mutex_lock(&p->mutex);
          if (p->pending_job) {
 -            RAMBlock *rb = p->pages->block;
              uint64_t packet_num = p->packet_num;
              uint32_t flags;
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
                  p->iovs_num = 1;
              }
 -            for (int i = 0; i < p->pages->num; i++) {
 -                uint64_t offset = p->pages->offset[i];
 -                if (use_multifd_zero_page &&
 -                    buffer_is_zero(rb->host + offset, p->page_size)) {
 -                    p->zero[p->zero_num] = offset;
 -                    p->zero_num++;
 -                    ram_release_page(rb->idstr, offset);
 -                } else {
 -                    p->normal[p->normal_num] = offset;
 -                    p->normal_num++;
 -                }
 -            }
 +            multifd_zero_page_check(p);
              if (p->normal_num) {
                  ret = multifd_send_state->ops->send_prepare(p, &local_err);
@@ -XXX,XX +XXX,XX @@ int multifd_save_setup(Error **errp)
          p->pending_job = 0;
          p->id = i;
          p->pages = multifd_pages_init(page_count);
 +        p->addr = g_new0(ram_addr_t, page_count);
 +        p->batch_task =
 +            (struct buffer_zero_batch_task *)qemu_memalign(64, sizeof(*p->batch_task));
 +        buffer_zero_batch_task_init(p->batch_task, page_count);
          p->packet_len = sizeof(MultiFDPacket_t)
                        + sizeof(uint64_t) * page_count;
          p->packet = g_malloc0(p->packet_len);
 diff --git a/migration/multifd.h b/migration/multifd.h
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/multifd.h
 +++ b/migration/multifd.h
@@ -XXX,XX +XXX,XX @@ typedef struct {
       * pending_job != 0 -> multifd_channel can use it.
       */
      MultiFDPages_t *pages;
 +    /* Address of each pages in pages */
 +    ram_addr_t *addr;
 +    struct buffer_zero_batch_task *batch_task;
      /* thread local variables. No locking required */
 --
-.30.2
+Yichen Wang

-[PATCH v2 07/20] util/dsa: Implement DSA device start and stop logic.
+[PATCH v9 03/12] util/dsa: Implement DSA device start and stop logic.
+From: Hao Xiang <hao.xiang@linux.dev>
 * DSA device open and close.
 * DSA group contains multiple DSA devices.
 * DSA group configure/start/stop/clean.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
 Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
+Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
+Reviewed-by: Fabiano Rosas <farosas@suse.de>
 ---
- include/qemu/dsa.h |  49 +++++++
+ include/qemu/dsa.h |  99 ++++++++++++++++
- util/dsa.c         | 338 +++++++++++++++++++++++++++++++++++++++++++++
+ util/dsa.c         | 280 +++++++++++++++++++++++++++++++++++++++++++++
- util/meson.build   |   1 +
+ util/meson.build   |   3 +
-files changed, 388 insertions(+)
+files changed, 382 insertions(+)
  create mode 100644 include/qemu/dsa.h
  create mode 100644 util/dsa.c
 diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/include/qemu/dsa.h
 @@ -XXX,XX +XXX,XX @@
++/*
++ * Interface for using Intel Data Streaming Accelerator to offload certain
++ * background operations.
++ *
++ * Copyright (C) Bytedance Ltd.
++ *
++ * Authors:
++ *  Hao Xiang <hao.xiang@bytedance.com>
++ *  Yichen Wang <yichen.wang@bytedance.com>
++ *
++ * This work is licensed under the terms of the GNU GPL, version 2 or later.
++ * See the COPYING file in the top-level directory.
++ */
++
 +#ifndef QEMU_DSA_H
 +#define QEMU_DSA_H
 +
++#include "qapi/error.h"
 +#include "qemu/thread.h"
 +#include "qemu/queue.h"
 +
 +#ifdef CONFIG_DSA_OPT
 +
 +#pragma GCC push_options
 +#pragma GCC target("enqcmd")
 +
 +#include <linux/idxd.h>
 +#include "x86intrin.h"
 +
++typedef struct {
++    void *work_queue;
++} QemuDsaDevice;
++
++typedef QSIMPLEQ_HEAD(QemuDsaTaskQueue, QemuDsaBatchTask) QemuDsaTaskQueue;
++
++typedef struct {
++    QemuDsaDevice *dsa_devices;
++    int num_dsa_devices;
++    /* The index of the next DSA device to be used. */
++    uint32_t device_allocator_index;
++    bool running;
++    QemuMutex task_queue_lock;
++    QemuCond task_queue_cond;
++    QemuDsaTaskQueue task_queue;
++} QemuDsaDeviceGroup;
++
++/**
++ * @brief Initializes DSA devices.
++ *
++ * @param dsa_parameter A list of DSA device path from migration parameter.
++ *
++ * @return int Zero if successful, otherwise non zero.
++ */
++int qemu_dsa_init(const strList *dsa_parameter, Error **errp);
++
++/**
++ * @brief Start logic to enable using DSA.
++ */
++void qemu_dsa_start(void);
++
++/**
++ * @brief Stop the device group and the completion thread.
++ */
++void qemu_dsa_stop(void);
++
++/**
++ * @brief Clean up system resources created for DSA offloading.
++ */
++void qemu_dsa_cleanup(void);
++
++/**
++ * @brief Check if DSA is running.
++ *
++ * @return True if DSA is running, otherwise false.
++ */
++bool qemu_dsa_is_running(void);
++
++#else
++
++static inline bool qemu_dsa_is_running(void)
++{
++    return false;
++}
++
++static inline int qemu_dsa_init(const strList *dsa_parameter, Error **errp)
++{
++    error_setg(errp, "DSA accelerator is not enabled.");
++    return -1;
++}
++
++static inline void qemu_dsa_start(void) {}
++
++static inline void qemu_dsa_stop(void) {}
++
++static inline void qemu_dsa_cleanup(void) {}
++
 +#endif
 +
-+/**
-+ * @brief Initializes DSA devices.
-+ *
-+ * @param dsa_parameter A list of DSA device path from migration parameter.
-+ * @return int Zero if successful, otherwise non zero.
-+ */
-+int dsa_init(const char *dsa_parameter);
-+
-+/**
-+ * @brief Start logic to enable using DSA.
-+ */
-+void dsa_start(void);
-+
-+/**
-+ * @brief Stop logic to clean up DSA by halting the device group and cleaning up
-+ * the completion thread.
-+ */
-+void dsa_stop(void);
-+
-+/**
-+ * @brief Clean up system resources created for DSA offloading.
-+ *        This function is called during QEMU process teardown.
-+ */
-+void dsa_cleanup(void);
-+
-+/**
-+ * @brief Check if DSA is running.
-+ *
-+ * @return True if DSA is running, otherwise false.
-+ */
-+bool dsa_is_running(void);
-+
 +#endif
-\ No newline at end of file
 diff --git a/util/dsa.c b/util/dsa.c
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/util/dsa.c
 @@ -XXX,XX +XXX,XX @@
 +/*
 + * Use Intel Data Streaming Accelerator to offload certain background
 + * operations.
 + *
-+ * Copyright (c) 2023 Hao Xiang <hao.xiang@bytedance.com>
++ * Copyright (C) Bytedance Ltd.
-+ *                    Bryan Zhang <bryan.zhang@bytedance.com>
++ *
-+ *
++ * Authors:
-+ * Permission is hereby granted, free of charge, to any person obtaining a copy
++ *  Hao Xiang <hao.xiang@bytedance.com>
-+ * of this software and associated documentation files (the "Software"), to deal
++ *  Bryan Zhang <bryan.zhang@bytedance.com>
-+ * in the Software without restriction, including without limitation the rights
++ *  Yichen Wang <yichen.wang@bytedance.com>
-+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
++ *
-+ * copies of the Software, and to permit persons to whom the Software is
++ * This work is licensed under the terms of the GNU GPL, version 2 or later.
-+ * furnished to do so, subject to the following conditions:
++ * See the COPYING file in the top-level directory.
 + *
 + * The above copyright notice and this permission notice shall be included in
 + * all copies or substantial portions of the Software.
 + *
 + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
 + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 + * THE SOFTWARE.
 + */
 +
 +#include "qemu/osdep.h"
++#include "qapi/error.h"
 +#include "qemu/queue.h"
 +#include "qemu/memalign.h"
 +#include "qemu/lockable.h"
 +#include "qemu/cutils.h"
 +#include "qemu/dsa.h"
 +#include "qemu/bswap.h"
 +#include "qemu/error-report.h"
 +#include "qemu/rcu.h"
 +
-+#ifdef CONFIG_DSA_OPT
-+
 +#pragma GCC push_options
 +#pragma GCC target("enqcmd")
 +
 +#include <linux/idxd.h>
 +#include "x86intrin.h"
 +
-+#define DSA_WQ_SIZE 4096
++#define DSA_WQ_PORTAL_SIZE 4096
 +#define MAX_DSA_DEVICES 16
 +
-+typedef QSIMPLEQ_HEAD(dsa_task_queue, buffer_zero_batch_task) dsa_task_queue;
++uint32_t max_retry_count;
-+
++static QemuDsaDeviceGroup dsa_group;
 +struct dsa_device {
 +    void *work_queue;
 +};
 +
 +struct dsa_device_group {
 +    struct dsa_device *dsa_devices;
 +    int num_dsa_devices;
 +    uint32_t index;
 +    bool running;
 +    QemuMutex task_queue_lock;
 +    QemuCond task_queue_cond;
 +    dsa_task_queue task_queue;
 +};
 +
 +uint64_t max_retry_count;
 +static struct dsa_device_group dsa_group;
 +
 +
 +/**
 + * @brief This function opens a DSA device's work queue and
 + *        maps the DSA device memory into the current process.
 + *
 + * @param dsa_wq_path A pointer to the DSA device work queue's file path.
-+ * @return A pointer to the mapped memory.
++ * @return A pointer to the mapped memory, or MAP_FAILED on failure.
 + */
 +static void *
 +map_dsa_device(const char *dsa_wq_path)
 +{
 +    void *dsa_device;
 +    int fd;
 +
 +    fd = open(dsa_wq_path, O_RDWR);
 +    if (fd < 0) {
-+        fprintf(stderr, "open %s failed with errno = %d.\n",
++        error_report("Open %s failed with errno = %d.",
 +                dsa_wq_path, errno);
 +        return MAP_FAILED;
 +    }
-+    dsa_device = mmap(NULL, DSA_WQ_SIZE, PROT_WRITE,
++    dsa_device = mmap(NULL, DSA_WQ_PORTAL_SIZE, PROT_WRITE,
 +                      MAP_SHARED | MAP_POPULATE, fd, 0);
 +    close(fd);
 +    if (dsa_device == MAP_FAILED) {
-+        fprintf(stderr, "mmap failed with errno = %d.\n", errno);
++        error_report("mmap failed with errno = %d.", errno);
 +        return MAP_FAILED;
 +    }
 +    return dsa_device;
 +}
 +
 +/**
 + * @brief Initializes a DSA device structure.
 + *
 + * @param instance A pointer to the DSA device.
-+ * @param work_queue  A pointer to the DSA work queue.
++ * @param work_queue A pointer to the DSA work queue.
 + */
 +static void
-+dsa_device_init(struct dsa_device *instance,
++dsa_device_init(QemuDsaDevice *instance,
 +                void *dsa_work_queue)
 +{
 +    instance->work_queue = dsa_work_queue;
 +}
 +
 +/**
 + * @brief Cleans up a DSA device structure.
 + *
 + * @param instance A pointer to the DSA device to cleanup.
 + */
 +static void
-+dsa_device_cleanup(struct dsa_device *instance)
++dsa_device_cleanup(QemuDsaDevice *instance)
 +{
 +    if (instance->work_queue != MAP_FAILED) {
-+        munmap(instance->work_queue, DSA_WQ_SIZE);
++        munmap(instance->work_queue, DSA_WQ_PORTAL_SIZE);
 +    }
 +}
 +
 +/**
 + * @brief Initializes a DSA device group.
 + *
 + * @param group A pointer to the DSA device group.
-+ * @param num_dsa_devices The number of DSA devices this group will have.
++ * @param dsa_parameter A list of DSA device path from are separated by space
 + * character migration parameter. Multiple DSA device path.
 + *
 + * @return Zero if successful, non-zero otherwise.
 + */
 +static int
-+dsa_device_group_init(struct dsa_device_group *group,
++dsa_device_group_init(QemuDsaDeviceGroup *group,
-+                      const char *dsa_parameter)
++                      const strList *dsa_parameter,
-+{
++                      Error **errp)
-+    if (dsa_parameter == NULL || strlen(dsa_parameter) == 0) {
++{
-+        return 0;
++    if (dsa_parameter == NULL) {
 +        error_setg(errp, "dsa device path is not supplied.");
 +        return -1;
 +    }
 +
 +    int ret = 0;
-+    char *local_dsa_parameter = g_strdup(dsa_parameter);
 +    const char *dsa_path[MAX_DSA_DEVICES];
 +    int num_dsa_devices = 0;
-+    char delim[2] = " ";
++
-+
++    while (dsa_parameter) {
-+    char *current_dsa_path = strtok(local_dsa_parameter, delim);
++        dsa_path[num_dsa_devices++] = dsa_parameter->value;
 +
 +    while (current_dsa_path != NULL) {
 +        dsa_path[num_dsa_devices++] = current_dsa_path;
 +        if (num_dsa_devices == MAX_DSA_DEVICES) {
 +            break;
 +        }
-+        current_dsa_path = strtok(NULL, delim);
++        dsa_parameter = dsa_parameter->next;
 +    }
 +
 +    group->dsa_devices =
-+        malloc(sizeof(struct dsa_device) * num_dsa_devices);
++        g_new0(QemuDsaDevice, num_dsa_devices);
 +    group->num_dsa_devices = num_dsa_devices;
-+    group->index = 0;
++    group->device_allocator_index = 0;
 +
 +    group->running = false;
 +    qemu_mutex_init(&group->task_queue_lock);
 +    qemu_cond_init(&group->task_queue_cond);
 +    QSIMPLEQ_INIT(&group->task_queue);
 +
 +    void *dsa_wq = MAP_FAILED;
 +    for (int i = 0; i < num_dsa_devices; i++) {
 +        dsa_wq = map_dsa_device(dsa_path[i]);
-+        if (dsa_wq == MAP_FAILED) {
++        if (dsa_wq == MAP_FAILED && ret != -1) {
-+            fprintf(stderr, "map_dsa_device failed MAP_FAILED, "
++            error_setg(errp, "map_dsa_device failed MAP_FAILED.");
 +                    "using simulation.\n");
 +            ret = -1;
-+            goto exit;
 +        }
-+        dsa_device_init(&dsa_group.dsa_devices[i], dsa_wq);
++        dsa_device_init(&group->dsa_devices[i], dsa_wq);
 +    }
 +
 +exit:
 +    g_free(local_dsa_parameter);
 +    return ret;
 +}
 +
 +/**
 + * @brief Starts a DSA device group.
 + *
 + * @param group A pointer to the DSA device group.
-+ * @param dsa_path An array of DSA device path.
++ */
-+ * @param num_dsa_devices The number of DSA devices in the device group.
++static void
-+ */
++dsa_device_group_start(QemuDsaDeviceGroup *group)
 +static void
 +dsa_device_group_start(struct dsa_device_group *group)
 +{
 +    group->running = true;
 +}
 +
 +/**
 + * @brief Stops a DSA device group.
 + *
 + * @param group A pointer to the DSA device group.
 + */
 +__attribute__((unused))
 +static void
-+dsa_device_group_stop(struct dsa_device_group *group)
++dsa_device_group_stop(QemuDsaDeviceGroup *group)
 +{
 +    group->running = false;
 +}
 +
 +/**
 + * @brief Cleans up a DSA device group.
 + *
 + * @param group A pointer to the DSA device group.
 + */
 +static void
-+dsa_device_group_cleanup(struct dsa_device_group *group)
++dsa_device_group_cleanup(QemuDsaDeviceGroup *group)
 +{
 +    if (!group->dsa_devices) {
 +        return;
 +    }
 +    for (int i = 0; i < group->num_dsa_devices; i++) {
 +        dsa_device_cleanup(&group->dsa_devices[i]);
 +    }
-+    free(group->dsa_devices);
++    g_free(group->dsa_devices);
 +    group->dsa_devices = NULL;
 +
 +    qemu_mutex_destroy(&group->task_queue_lock);
 +    qemu_cond_destroy(&group->task_queue_cond);
 +}
 +
 +/**
 + * @brief Returns the next available DSA device in the group.
 + *
 + * @param group A pointer to the DSA device group.
 + *
-+ * @return struct dsa_device* A pointer to the next available DSA device
++ * @return struct QemuDsaDevice* A pointer to the next available DSA device
 + *         in the group.
 + */
 +__attribute__((unused))
-+static struct dsa_device *
++static QemuDsaDevice *
-+dsa_device_group_get_next_device(struct dsa_device_group *group)
++dsa_device_group_get_next_device(QemuDsaDeviceGroup *group)
 +{
 +    if (group->num_dsa_devices == 0) {
 +        return NULL;
 +    }
-+    uint32_t current = qatomic_fetch_inc(&group->index);
++    uint32_t current = qatomic_fetch_inc(&group->device_allocator_index);
 +    current %= group->num_dsa_devices;
 +    return &group->dsa_devices[current];
 +}
 +
 +/**
 + * @brief Check if DSA is running.
 + *
 + * @return True if DSA is running, otherwise false.
 + */
-+bool dsa_is_running(void)
++bool qemu_dsa_is_running(void)
 +{
 +    return false;
 +}
 +
 +static void
 +dsa_globals_init(void)
 +{
-+    max_retry_count = UINT64_MAX;
++    max_retry_count = UINT32_MAX;
 +}
 +
 +/**
 + * @brief Initializes DSA devices.
 + *
 + * @param dsa_parameter A list of DSA device path from migration parameter.
++ *
 + * @return int Zero if successful, otherwise non zero.
 + */
-+int dsa_init(const char *dsa_parameter)
++int qemu_dsa_init(const strList *dsa_parameter, Error **errp)
 +{
 +    dsa_globals_init();
 +
-+    return dsa_device_group_init(&dsa_group, dsa_parameter);
++    return dsa_device_group_init(&dsa_group, dsa_parameter, errp);
 +}
 +
 +/**
 + * @brief Start logic to enable using DSA.
 + *
 + */
-+void dsa_start(void)
++void qemu_dsa_start(void)
 +{
 +    if (dsa_group.num_dsa_devices == 0) {
 +        return;
 +    }
 +    if (dsa_group.running) {
 +        return;
 +    }
 +    dsa_device_group_start(&dsa_group);
 +}
 +
 +/**
-+ * @brief Stop logic to clean up DSA by halting the device group and cleaning up
++ * @brief Stop the device group and the completion thread.
-+ * the completion thread.
++ *
-+ *
++ */
-+ */
++void qemu_dsa_stop(void)
-+void dsa_stop(void)
++{
-+{
++    QemuDsaDeviceGroup *group = &dsa_group;
 +    struct dsa_device_group *group = &dsa_group;
 +
 +    if (!group->running) {
 +        return;
 +    }
 +}
 +
 +/**
 + * @brief Clean up system resources created for DSA offloading.
-+ *        This function is called during QEMU process teardown.
++ *
-+ *
++ */
-+ */
++void qemu_dsa_cleanup(void)
-+void dsa_cleanup(void)
++{
-+{
++    qemu_dsa_stop();
 +    dsa_stop();
 +    dsa_device_group_cleanup(&dsa_group);
 +}
-+
-+#else
-+
-+bool dsa_is_running(void)
-+{
-+    return false;
-+}
-+
-+int dsa_init(const char *dsa_parameter)
-+{
-+    fprintf(stderr, "Intel Data Streaming Accelerator is not supported "
-+                    "on this platform.\n");
-+    return -1;
-+}
-+
-+void dsa_start(void) {}
-+
-+void dsa_stop(void) {}
-+
-+void dsa_cleanup(void) {}
-+
-+#endif
 +
 diff --git a/util/meson.build b/util/meson.build
 index XXXXXXX..XXXXXXX 100644
 --- a/util/meson.build
 +++ b/util/meson.build
-@@ -XXX,XX +XXX,XX @@ if have_block or have_ga
+@@ -XXX,XX +XXX,XX @@ if cpu == 'aarch64'
- endif
+   util_ss.add(files('cpuinfo-aarch64.c'))
- if have_block
+ elif cpu in ['x86', 'x86_64']
-   util_ss.add(files('aio-wait.c'))
+   util_ss.add(files('cpuinfo-i386.c'))
-+  util_ss.add(files('dsa.c'))
++  if config_host_data.get('CONFIG_DSA_OPT')
-   util_ss.add(files('buffer.c'))
++    util_ss.add(files('dsa.c'))
-   util_ss.add(files('bufferiszero.c'))
++  endif
-   util_ss.add(files('hbitmap.c'))
+ elif cpu == 'loongarch64'
    util_ss.add(files('cpuinfo-loongarch.c'))
  elif cpu in ['ppc', 'ppc64']
 --
-.30.2
+Yichen Wang

-[PATCH v2 08/20] util/dsa: Implement DSA task enqueue and dequeue.
+[PATCH v9 04/12] util/dsa: Implement DSA task enqueue and dequeue.
+From: Hao Xiang <hao.xiang@linux.dev>
 * Use a safe thread queue for DSA task enqueue/dequeue.
 * Implement DSA task submission.
 * Implement DSA batch task submission.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
 Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
 Reviewed-by: Fabiano Rosas <farosas@suse.de>
 ---
- include/qemu/dsa.h |  35 ++++++++
+ include/qemu/dsa.h |  29 +++++++
- util/dsa.c         | 196 +++++++++++++++++++++++++++++++++++++++++++++
+ util/dsa.c         | 186 ++++++++++++++++++++++++++++++++++++++++++++-
-files changed, 231 insertions(+)
+files changed, 214 insertions(+), 1 deletion(-)
 diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
 index XXXXXXX..XXXXXXX 100644
 --- a/include/qemu/dsa.h
 +++ b/include/qemu/dsa.h
 @@ -XXX,XX +XXX,XX @@
  #include <linux/idxd.h>
  #include "x86intrin.h"
-+enum dsa_task_type {
++typedef enum QemuDsaTaskType {
-+    DSA_TASK = 0,
++    QEMU_DSA_TASK = 0,
-+    DSA_BATCH_TASK
++    QEMU_DSA_BATCH_TASK
-+};
++} QemuDsaTaskType;
 +
-+enum dsa_task_status {
++typedef enum QemuDsaTaskStatus {
-+    DSA_TASK_READY = 0,
++    QEMU_DSA_TASK_READY = 0,
-+    DSA_TASK_PROCESSING,
++    QEMU_DSA_TASK_PROCESSING,
-+    DSA_TASK_COMPLETION
++    QEMU_DSA_TASK_COMPLETION
-+};
++} QemuDsaTaskStatus;
 +
-+typedef void (*buffer_zero_dsa_completion_fn)(void *);
+ typedef struct {
-+
+     void *work_queue;
-+typedef struct buffer_zero_batch_task {
+ } QemuDsaDevice;
@@ -XXX,XX +XXX,XX @@ typedef struct {
      QemuDsaTaskQueue task_queue;
  } QemuDsaDeviceGroup;
 +typedef void (*qemu_dsa_completion_fn)(void *);
 +
 +typedef struct QemuDsaBatchTask {
 +    struct dsa_hw_desc batch_descriptor;
 +    struct dsa_hw_desc *descriptors;
 +    struct dsa_completion_record batch_completion __attribute__((aligned(32)));
 +    struct dsa_completion_record *completions;
-+    struct dsa_device_group *group;
++    QemuDsaDeviceGroup *group;
-+    struct dsa_device *device;
++    QemuDsaDevice *device;
-+    buffer_zero_dsa_completion_fn completion_callback;
++    qemu_dsa_completion_fn completion_callback;
 +    QemuSemaphore sem_task_complete;
-+    enum dsa_task_type task_type;
++    QemuDsaTaskType task_type;
-+    enum dsa_task_status status;
++    QemuDsaTaskStatus status;
 +    bool *results;
 +    int batch_size;
-+    QSIMPLEQ_ENTRY(buffer_zero_batch_task) entry;
++    QSIMPLEQ_ENTRY(QemuDsaBatchTask) entry;
-+} buffer_zero_batch_task;
++} QemuDsaBatchTask;
 +
-+#else
++
-+
+ /**
-+struct buffer_zero_batch_task {
+  * @brief Initializes DSA devices.
-+    bool *results;
+  *
 +};
 +
  #endif
  /**
 diff --git a/util/dsa.c b/util/dsa.c
 index XXXXXXX..XXXXXXX 100644
 --- a/util/dsa.c
 +++ b/util/dsa.c
-@@ -XXX,XX +XXX,XX @@ dsa_device_group_get_next_device(struct dsa_device_group *group)
+@@ -XXX,XX +XXX,XX @@
  #include "x86intrin.h"
  #define DSA_WQ_PORTAL_SIZE 4096
 +#define DSA_WQ_DEPTH 128
  #define MAX_DSA_DEVICES 16
  uint32_t max_retry_count;
@@ -XXX,XX +XXX,XX @@ dsa_device_group_get_next_device(QemuDsaDeviceGroup *group)
      return &group->dsa_devices[current];
  }
 +/**
 + * @brief Empties out the DSA task queue.
 + *
 + * @param group A pointer to the DSA device group.
 + */
 +static void
-+dsa_empty_task_queue(struct dsa_device_group *group)
++dsa_empty_task_queue(QemuDsaDeviceGroup *group)
 +{
 +    qemu_mutex_lock(&group->task_queue_lock);
-+    dsa_task_queue *task_queue = &group->task_queue;
++    QemuDsaTaskQueue *task_queue = &group->task_queue;
 +    while (!QSIMPLEQ_EMPTY(task_queue)) {
 +        QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
 +    }
 +    qemu_mutex_unlock(&group->task_queue_lock);
 +}
 +
 +/**
 + * @brief Adds a task to the DSA task queue.
 + *
 + * @param group A pointer to the DSA device group.
-+ * @param context A pointer to the DSA task to enqueue.
++ * @param task A pointer to the DSA task to enqueue.
 + *
 + * @return int Zero if successful, otherwise a proper error code.
 + */
 +static int
-+dsa_task_enqueue(struct dsa_device_group *group,
++dsa_task_enqueue(QemuDsaDeviceGroup *group,
-+                 struct buffer_zero_batch_task *task)
++                 QemuDsaBatchTask *task)
 +{
 +    dsa_task_queue *task_queue = &group->task_queue;
 +    QemuMutex *task_queue_lock = &group->task_queue_lock;
 +    QemuCond *task_queue_cond = &group->task_queue_cond;
 +
 +    bool notify = false;
 +
-+    qemu_mutex_lock(task_queue_lock);
++    qemu_mutex_lock(&group->task_queue_lock);
 +
 +    if (!group->running) {
-+        fprintf(stderr, "DSA: Tried to queue task to stopped device queue\n");
++        error_report("DSA: Tried to queue task to stopped device queue.");
-+        qemu_mutex_unlock(task_queue_lock);
++        qemu_mutex_unlock(&group->task_queue_lock);
 +        return -1;
 +    }
 +
-+    // The queue is empty. This enqueue operation is a 0->1 transition.
++    /* The queue is empty. This enqueue operation is a 0->1 transition. */
-+    if (QSIMPLEQ_EMPTY(task_queue))
++    if (QSIMPLEQ_EMPTY(&group->task_queue)) {
 +        notify = true;
-+
++    }
-+    QSIMPLEQ_INSERT_TAIL(task_queue, task, entry);
++
-+
++    QSIMPLEQ_INSERT_TAIL(&group->task_queue, task, entry);
-+    // We need to notify the waiter for 0->1 transitions.
++
-+    if (notify)
++    /* We need to notify the waiter for 0->1 transitions. */
-+        qemu_cond_signal(task_queue_cond);
++    if (notify) {
-+
++        qemu_cond_signal(&group->task_queue_cond);
-+    qemu_mutex_unlock(task_queue_lock);
++    }
 +
 +    qemu_mutex_unlock(&group->task_queue_lock);
 +
 +    return 0;
 +}
 +
 +/**
 + * @brief Takes a DSA task out of the task queue.
 + *
 + * @param group A pointer to the DSA device group.
-+ * @return buffer_zero_batch_task* The DSA task being dequeued.
++ * @return QemuDsaBatchTask* The DSA task being dequeued.
 + */
 +__attribute__((unused))
-+static struct buffer_zero_batch_task *
++static QemuDsaBatchTask *
-+dsa_task_dequeue(struct dsa_device_group *group)
++dsa_task_dequeue(QemuDsaDeviceGroup *group)
 +{
-+    struct buffer_zero_batch_task *task = NULL;
++    QemuDsaBatchTask *task = NULL;
-+    dsa_task_queue *task_queue = &group->task_queue;
++
-+    QemuMutex *task_queue_lock = &group->task_queue_lock;
++    qemu_mutex_lock(&group->task_queue_lock);
 +    QemuCond *task_queue_cond = &group->task_queue_cond;
 +
 +    qemu_mutex_lock(task_queue_lock);
 +
 +    while (true) {
-+        if (!group->running)
++        if (!group->running) {
 +            goto exit;
-+        task = QSIMPLEQ_FIRST(task_queue);
++        }
 +        task = QSIMPLEQ_FIRST(&group->task_queue);
 +        if (task != NULL) {
 +            break;
 +        }
-+        qemu_cond_wait(task_queue_cond, task_queue_lock);
++        qemu_cond_wait(&group->task_queue_cond, &group->task_queue_lock);
 +    }
 +
-+    QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
++    QSIMPLEQ_REMOVE_HEAD(&group->task_queue, entry);
 +
 +exit:
-+    qemu_mutex_unlock(task_queue_lock);
++    qemu_mutex_unlock(&group->task_queue_lock);
 +    return task;
 +}
 +
 +/**
 + * @brief Submits a DSA work item to the device work queue.
 ...
 + * @return Zero if successful, non-zero otherwise.
 + */
 +static int
 +submit_wi_int(void *wq, struct dsa_hw_desc *descriptor)
 +{
-+    uint64_t retry = 0;
++    uint32_t retry = 0;
 +
 +    _mm_sfence();
 +
 +    while (true) {
 +        if (_enqcmd(wq, descriptor) == 0) {
 +            break;
 +        }
 +        retry++;
 +        if (retry > max_retry_count) {
-+            fprintf(stderr, "Submit work retry %lu times.\n", retry);
++            error_report("Submit work retry %u times.", retry);
-+            exit(1);
++            return -1;
 +        }
 +    }
 +
 +    return 0;
-+}
-+
-+/**
-+ * @brief Synchronously submits a DSA work item to the
-+ *        device work queue.
-+ *
-+ * @param wq A pointer to the DSA worjk queue's device memory.
-+ * @param descriptor A pointer to the DSA work item descriptor.
-+ *
-+ * @return int Zero if successful, non-zero otherwise.
-+ */
-+__attribute__((unused))
-+static int
-+submit_wi(void *wq, struct dsa_hw_desc *descriptor)
-+{
-+    return submit_wi_int(wq, descriptor);
 +}
 +
 +/**
 + * @brief Asynchronously submits a DSA work item to the
 + *        device work queue.
 + *
-+ * @param task A pointer to the buffer zero task.
++ * @param task A pointer to the task.
 + *
 + * @return int Zero if successful, non-zero otherwise.
 + */
 +__attribute__((unused))
 +static int
-+submit_wi_async(struct buffer_zero_batch_task *task)
++submit_wi_async(QemuDsaBatchTask *task)
 +{
-+    struct dsa_device_group *device_group = task->group;
++    QemuDsaDeviceGroup *device_group = task->group;
-+    struct dsa_device *device_instance = task->device;
++    QemuDsaDevice *device_instance = task->device;
 +    int ret;
 +
-+    assert(task->task_type == DSA_TASK);
++    assert(task->task_type == QEMU_DSA_TASK);
 +
-+    task->status = DSA_TASK_PROCESSING;
++    task->status = QEMU_DSA_TASK_PROCESSING;
 +
 +    ret = submit_wi_int(device_instance->work_queue,
 +                        &task->descriptors[0]);
-+    if (ret != 0)
++    if (ret != 0) {
 +        return ret;
++    }
 +
 +    return dsa_task_enqueue(device_group, task);
 +}
 +
 +/**
 + * @brief Asynchronously submits a DSA batch work item to the
 + *        device work queue.
 + *
-+ * @param batch_task A pointer to the batch buffer zero task.
++ * @param batch_task A pointer to the batch task.
 + *
 + * @return int Zero if successful, non-zero otherwise.
 + */
 +__attribute__((unused))
 +static int
-+submit_batch_wi_async(struct buffer_zero_batch_task *batch_task)
++submit_batch_wi_async(QemuDsaBatchTask *batch_task)
 +{
-+    struct dsa_device_group *device_group = batch_task->group;
++    QemuDsaDeviceGroup *device_group = batch_task->group;
-+    struct dsa_device *device_instance = batch_task->device;
++    QemuDsaDevice *device_instance = batch_task->device;
 +    int ret;
 +
-+    assert(batch_task->task_type == DSA_BATCH_TASK);
++    assert(batch_task->task_type == QEMU_DSA_BATCH_TASK);
 +    assert(batch_task->batch_descriptor.desc_count <= batch_task->batch_size);
-+    assert(batch_task->status == DSA_TASK_READY);
++    assert(batch_task->status == QEMU_DSA_TASK_READY);
 +
-+    batch_task->status = DSA_TASK_PROCESSING;
++    batch_task->status = QEMU_DSA_TASK_PROCESSING;
 +
 +    ret = submit_wi_int(device_instance->work_queue,
 +                        &batch_task->batch_descriptor);
-+    if (ret != 0)
++    if (ret != 0) {
 +        return ret;
++    }
 +
 +    return dsa_task_enqueue(device_group, batch_task);
 +}
 +
  /**
   * @brief Check if DSA is running.
   *
-@@ -XXX,XX +XXX,XX @@ void dsa_stop(void)
+@@ -XXX,XX +XXX,XX @@ bool qemu_dsa_is_running(void)
  static void
  dsa_globals_init(void)
  {
 -    max_retry_count = UINT32_MAX;
 +    /*
 +     * This value follows a reference example by Intel. The POLL_RETRY_MAX is
 +     * defined to 10000, so here we used the max WQ depth * 100 for the the max
 +     * polling retry count.
 +     */
 +    max_retry_count = DSA_WQ_DEPTH * 100;
  }
  /**
@@ -XXX,XX +XXX,XX @@ void qemu_dsa_stop(void)
      if (!group->running) {
          return;
      }
 +
 +    dsa_empty_task_queue(group);
  }
  /**
 --
-.30.2
+Yichen Wang

-[PATCH v2 09/20] util/dsa: Implement DSA task asynchronous completion thread model.
+[PATCH v9 05/12] util/dsa: Implement DSA task asynchronous completion thread model.
+From: Hao Xiang <hao.xiang@linux.dev>
 * Create a dedicated thread for DSA task completion.
 * DSA completion thread runs a loop and poll for completed tasks.
 * Start and stop DSA completion thread during DSA device start stop.
 User space application can directly submit task to Intel DSA
 accelerator by writing to DSA's device memory (mapped in user space).
 Once a task is submitted, the device starts processing it and write
 the completion status back to the task. A user space application can
 poll the task's completion status to check for completion. This change
 uses a dedicated thread to perform DSA task completion checking.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
 Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
 Reviewed-by: Fabiano Rosas <farosas@suse.de>
 ---
- util/dsa.c | 243 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
+ include/qemu/dsa.h |   1 +
-file changed, 242 insertions(+), 1 deletion(-)
+ util/dsa.c         | 272 ++++++++++++++++++++++++++++++++++++++++++++-
 files changed, 272 insertions(+), 1 deletion(-)
+diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
+index XXXXXXX..XXXXXXX 100644
+--- a/include/qemu/dsa.h
++++ b/include/qemu/dsa.h
+@@ -XXX,XX +XXX,XX @@ typedef struct QemuDsaBatchTask {
+     QemuDsaTaskType task_type;
+     QemuDsaTaskStatus status;
+     int batch_size;
++    bool *results;
+     QSIMPLEQ_ENTRY(QemuDsaBatchTask) entry;
+ } QemuDsaBatchTask;
 diff --git a/util/dsa.c b/util/dsa.c
 index XXXXXXX..XXXXXXX 100644
 --- a/util/dsa.c
 +++ b/util/dsa.c
 @@ -XXX,XX +XXX,XX @@
+ #define DSA_WQ_PORTAL_SIZE 4096
- #define DSA_WQ_SIZE 4096
+ #define DSA_WQ_DEPTH 128
  #define MAX_DSA_DEVICES 16
-+#define DSA_COMPLETION_THREAD "dsa_completion"
++#define DSA_COMPLETION_THREAD "qemu_dsa_completion"
++
- typedef QSIMPLEQ_HEAD(dsa_task_queue, buffer_zero_batch_task) dsa_task_queue;
++typedef struct {
@@ -XXX,XX +XXX,XX @@ struct dsa_device_group {
      dsa_task_queue task_queue;
  };
 +struct dsa_completion_thread {
 +    bool stopping;
 +    bool running;
 +    QemuThread thread;
 +    int thread_id;
 +    QemuSemaphore sem_init_done;
-+    struct dsa_device_group *group;
++    QemuDsaDeviceGroup *group;
-+};
++} QemuDsaCompletionThread;
-+
- uint64_t max_retry_count;
+ uint32_t max_retry_count;
- static struct dsa_device_group dsa_group;
+ static QemuDsaDeviceGroup dsa_group;
-+static struct dsa_completion_thread completion_thread;
++static QemuDsaCompletionThread completion_thread;
  /**
-@@ -XXX,XX +XXX,XX @@ submit_batch_wi_async(struct buffer_zero_batch_task *batch_task)
+@@ -XXX,XX +XXX,XX @@ submit_batch_wi_async(QemuDsaBatchTask *batch_task)
      return dsa_task_enqueue(device_group, batch_task);
  }
 +/**
 + * @brief Poll for the DSA work item completion.
 ...
 +{
 +    uint8_t status;
 +    uint64_t retry = 0;
 +
 +    while (true) {
-+        // The DSA operation completes successfully or fails.
++        /* The DSA operation completes successfully or fails. */
 +        status = completion->status;
 +        if (status == DSA_COMP_SUCCESS ||
 +            status == DSA_COMP_PAGE_FAULT_NOBOF ||
 +            status == DSA_COMP_BATCH_PAGE_FAULT ||
 +            status == DSA_COMP_BATCH_FAIL) {
 +            break;
 +        } else if (status != DSA_COMP_NONE) {
-+            /* TODO: Error handling here on unexpected failure. */
++            error_report("DSA opcode %d failed with status = %d.",
 +            fprintf(stderr, "DSA opcode %d failed with status = %d.\n",
 +                    opcode, status);
-+            exit(1);
++            return 1;
 +        }
 +        retry++;
 +        if (retry > max_retry_count) {
-+            fprintf(stderr, "Wait for completion retry %lu times.\n", retry);
++            error_report("DSA wait for completion retry %lu times.", retry);
-+            exit(1);
++            return 1;
 +        }
 +        _mm_pause();
 +    }
 +
 +    return 0;
 +}
 +
 +/**
 + * @brief Complete a single DSA task in the batch task.
 + *
 + * @param task A pointer to the batch task structure.
-+ */
++ *
-+static void
++ * @return Zero if successful, otherwise non-zero.
-+poll_task_completion(struct buffer_zero_batch_task *task)
++ */
-+{
++static int
-+    assert(task->task_type == DSA_TASK);
++poll_task_completion(QemuDsaBatchTask *task)
 +{
 +    assert(task->task_type == QEMU_DSA_TASK);
 +
 +    struct dsa_completion_record *completion = &task->completions[0];
 +    uint8_t status;
-+
++    int ret;
-+    poll_completion(completion, task->descriptors[0].opcode);
++
 +    ret = poll_completion(completion, task->descriptors[0].opcode);
 +    if (ret != 0) {
 +        goto exit;
 +    }
 +
 +    status = completion->status;
 +    if (status == DSA_COMP_SUCCESS) {
 +        task->results[0] = (completion->result == 0);
-+        return;
++        goto exit;
 +    }
 +
 +    assert(status == DSA_COMP_PAGE_FAULT_NOBOF);
++
++exit:
++    return ret;
 +}
 +
 +/**
 + * @brief Poll a batch task status until it completes. If DSA task doesn't
 + *        complete properly, use CPU to complete the task.
 + *
 + * @param batch_task A pointer to the DSA batch task.
-+ */
++ *
-+static void
++ * @return Zero if successful, otherwise non-zero.
-+poll_batch_task_completion(struct buffer_zero_batch_task *batch_task)
++ */
-+{
++static int
-+    struct dsa_completion_record *batch_completion = &batch_task->batch_completion;
++poll_batch_task_completion(QemuDsaBatchTask *batch_task)
 +{
 +    struct dsa_completion_record *batch_completion =
 +        &batch_task->batch_completion;
 +    struct dsa_completion_record *completion;
 +    uint8_t batch_status;
 +    uint8_t status;
 +    bool *results = batch_task->results;
 +    uint32_t count = batch_task->batch_descriptor.desc_count;
-+
++    int ret;
-+    poll_completion(batch_completion,
++
-+                    batch_task->batch_descriptor.opcode);
++    ret = poll_completion(batch_completion,
 +                          batch_task->batch_descriptor.opcode);
 +    if (ret != 0) {
 +        goto exit;
 +    }
 +
 +    batch_status = batch_completion->status;
 +
 +    if (batch_status == DSA_COMP_SUCCESS) {
 +        if (batch_completion->bytes_completed == count) {
-+            // Let's skip checking for each descriptors' completion status
++            /*
-+            // if the batch descriptor says all succedded.
++             * Let's skip checking for each descriptors' completion status
 +             * if the batch descriptor says all succedded.
 +             */
 +            for (int i = 0; i < count; i++) {
 +                assert(batch_task->completions[i].status == DSA_COMP_SUCCESS);
 +                results[i] = (batch_task->completions[i].result == 0);
 +            }
-+            return;
++            goto exit;
 +        }
 +    } else {
 +        assert(batch_status == DSA_COMP_BATCH_FAIL ||
 +            batch_status == DSA_COMP_BATCH_PAGE_FAULT);
 +    }
 ...
 +            results[i] = (completion->result == 0);
 +            continue;
 +        }
 +
 +        if (status != DSA_COMP_PAGE_FAULT_NOBOF) {
-+            fprintf(stderr,
++            error_report("Unexpected DSA completion status = %u.", status);
-+                    "Unexpected completion status = %u.\n", status);
++            ret = 1;
-+            assert(false);
++            goto exit;
 +        }
 +    }
 +
 +exit:
 +    return ret;
 +}
 +
 +/**
 + * @brief Handles an asynchronous DSA batch task completion.
 + *
 + * @param task A pointer to the batch buffer zero task structure.
 + */
 +static void
-+dsa_batch_task_complete(struct buffer_zero_batch_task *batch_task)
++dsa_batch_task_complete(QemuDsaBatchTask *batch_task)
 +{
-+    batch_task->status = DSA_TASK_COMPLETION;
++    batch_task->status = QEMU_DSA_TASK_COMPLETION;
 +    batch_task->completion_callback(batch_task);
 +}
 +
 +/**
 + * @brief The function entry point called by a dedicated DSA
 ...
 + * @return void* Not used.
 + */
 +static void *
 +dsa_completion_loop(void *opaque)
 +{
-+    struct dsa_completion_thread *thread_context =
++    QemuDsaCompletionThread *thread_context =
-+        (struct dsa_completion_thread *)opaque;
++        (QemuDsaCompletionThread *)opaque;
-+    struct buffer_zero_batch_task *batch_task;
++    QemuDsaBatchTask *batch_task;
-+    struct dsa_device_group *group = thread_context->group;
++    QemuDsaDeviceGroup *group = thread_context->group;
 +    int ret;
 +
 +    rcu_register_thread();
 +
 +    thread_context->thread_id = qemu_get_thread_id();
 +    qemu_sem_post(&thread_context->sem_init_done);
 ...
 +        assert(batch_task != NULL || !group->running);
 +        if (!group->running) {
 +            assert(!thread_context->running);
 +            break;
 +        }
-+        if (batch_task->task_type == DSA_TASK) {
++        if (batch_task->task_type == QEMU_DSA_TASK) {
-+            poll_task_completion(batch_task);
++            ret = poll_task_completion(batch_task);
 +        } else {
-+            assert(batch_task->task_type == DSA_BATCH_TASK);
++            assert(batch_task->task_type == QEMU_DSA_BATCH_TASK);
-+            poll_batch_task_completion(batch_task);
++            ret = poll_batch_task_completion(batch_task);
 +        }
 +
 +        if (ret != 0) {
 +            goto exit;
 +        }
 +
 +        dsa_batch_task_complete(batch_task);
 +    }
 +
++exit:
++    if (ret != 0) {
++        error_report("DSA completion thread exited due to internal error.");
++    }
 +    rcu_unregister_thread();
 +    return NULL;
 +}
 +
 +/**
 ...
 + * @param completion_thread A pointer to the completion thread context.
 + * @param group A pointer to the DSA device group.
 + */
 +static void
 +dsa_completion_thread_init(
-+    struct dsa_completion_thread *completion_thread,
++    QemuDsaCompletionThread *completion_thread,
-+    struct dsa_device_group *group)
++    QemuDsaDeviceGroup *group)
 +{
 +    completion_thread->stopping = false;
 +    completion_thread->running = true;
 +    completion_thread->thread_id = -1;
 +    qemu_sem_init(&completion_thread->sem_init_done, 0);
 ...
 +                       dsa_completion_loop,
 +                       completion_thread,
 +                       QEMU_THREAD_JOINABLE);
 +
 +    /* Wait for initialization to complete */
-+    while (completion_thread->thread_id == -1) {
++    qemu_sem_wait(&completion_thread->sem_init_done);
 +        qemu_sem_wait(&completion_thread->sem_init_done);
 +    }
 +}
 +
 +/**
 + * @brief Stops the completion thread (and implicitly, the device group).
 + *
 + * @param opaque A pointer to the completion thread.
 + */
 +static void dsa_completion_thread_stop(void *opaque)
 +{
-+    struct dsa_completion_thread *thread_context =
++    QemuDsaCompletionThread *thread_context =
-+        (struct dsa_completion_thread *)opaque;
++        (QemuDsaCompletionThread *)opaque;
 +
-+    struct dsa_device_group *group = thread_context->group;
++    QemuDsaDeviceGroup *group = thread_context->group;
 +
 +    qemu_mutex_lock(&group->task_queue_lock);
 +
 +    thread_context->stopping = true;
 +    thread_context->running = false;
 +
++    /* Prevent the compiler from setting group->running first. */
++    barrier();
 +    dsa_device_group_stop(group);
 +
 +    qemu_cond_signal(&group->task_queue_cond);
 +    qemu_mutex_unlock(&group->task_queue_lock);
 +
 ...
 +}
 +
  /**
   * @brief Check if DSA is running.
   *
-@@ -XXX,XX +XXX,XX @@ submit_batch_wi_async(struct buffer_zero_batch_task *batch_task)
+@@ -XXX,XX +XXX,XX @@ submit_batch_wi_async(QemuDsaBatchTask *batch_task)
   */
- bool dsa_is_running(void)
+ bool qemu_dsa_is_running(void)
  {
 -    return false;
 +    return completion_thread.running;
  }
  static void
-@@ -XXX,XX +XXX,XX @@ void dsa_start(void)
+@@ -XXX,XX +XXX,XX @@ void qemu_dsa_start(void)
          return;
      }
      dsa_device_group_start(&dsa_group);
 +    dsa_completion_thread_init(&completion_thread, &dsa_group);
  }
  /**
-@@ -XXX,XX +XXX,XX @@ void dsa_stop(void)
+@@ -XXX,XX +XXX,XX @@ void qemu_dsa_stop(void)
          return;
      }
 +    dsa_completion_thread_stop(&completion_thread);
      dsa_empty_task_queue(group);
  }
 --
-.30.2
+Yichen Wang

-[PATCH v2 10/20] util/dsa: Implement zero page checking in DSA task.
+[PATCH v9 06/12] util/dsa: Implement zero page checking in DSA task.
+From: Hao Xiang <hao.xiang@linux.dev>
 Create DSA task with operation code DSA_OPCODE_COMPVAL.
 Here we create two types of DSA tasks, a single DSA task and
 a batch DSA task. Batch DSA task reduces task submission overhead
 and hence should be the default option. However, due to the way DSA
 hardware works, a DSA batch task must contain at least two individual
 tasks. There are times we need to submit a single task and hence a
 single DSA task submission is also required.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
 Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
+Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
+Reviewed-by: Fabiano Rosas <farosas@suse.de>
 ---
- include/qemu/dsa.h |  16 +++
+ include/qemu/dsa.h |  36 ++++++-
- util/dsa.c         | 252 +++++++++++++++++++++++++++++++++++++++++----
+ util/dsa.c         | 240 ++++++++++++++++++++++++++++++++++++++++++++-
-files changed, 247 insertions(+), 21 deletions(-)
+files changed, 270 insertions(+), 6 deletions(-)
 diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
 index XXXXXXX..XXXXXXX 100644
 --- a/include/qemu/dsa.h
 +++ b/include/qemu/dsa.h
-@@ -XXX,XX +XXX,XX @@ struct buffer_zero_batch_task {
+@@ -XXX,XX +XXX,XX @@
+ #define QEMU_DSA_H
- #endif
+ #include "qapi/error.h"
-+/**
++#include "exec/cpu-common.h"
-+ * @brief Initializes a buffer zero batch task.
+ #include "qemu/thread.h"
-+ *
+ #include "qemu/queue.h"
-+ * @param task A pointer to the batch task to initialize.
-+ * @param batch_size The number of DSA tasks in the batch.
+@@ -XXX,XX +XXX,XX @@ typedef struct QemuDsaBatchTask {
-+ */
+     QemuDsaTaskStatus status;
-+void buffer_zero_batch_task_init(struct buffer_zero_batch_task *task,
+     int batch_size;
-+                                 int batch_size);
+     bool *results;
-+
++    /* Address of each pages in pages */
-+/**
++    ram_addr_t *addr;
-+ * @brief Performs the proper cleanup on a DSA batch task.
+     QSIMPLEQ_ENTRY(QemuDsaBatchTask) entry;
-+ *
+ } QemuDsaBatchTask;
-+ * @param task A pointer to the batch task to cleanup.
-+ */
+-
 +void buffer_zero_batch_task_destroy(struct buffer_zero_batch_task *task);
 +
  /**
   * @brief Initializes DSA devices.
   *
+@@ -XXX,XX +XXX,XX @@ void qemu_dsa_cleanup(void);
+  */
+ bool qemu_dsa_is_running(void);
++/**
++ * @brief Initializes a buffer zero DSA batch task.
++ *
++ * @param batch_size The number of zero page checking tasks in the batch.
++ * @return A pointer to the zero page checking tasks initialized.
++ */
++QemuDsaBatchTask *
++buffer_zero_batch_task_init(int batch_size);
++
++/**
++ * @brief Performs the proper cleanup on a DSA batch task.
++ *
++ * @param task A pointer to the batch task to cleanup.
++ */
++void buffer_zero_batch_task_destroy(QemuDsaBatchTask *task);
++
+ #else
++typedef struct QemuDsaBatchTask {} QemuDsaBatchTask;
++
+ static inline bool qemu_dsa_is_running(void)
+ {
+     return false;
+@@ -XXX,XX +XXX,XX @@ static inline void qemu_dsa_stop(void) {}
+ static inline void qemu_dsa_cleanup(void) {}
++static inline QemuDsaBatchTask *buffer_zero_batch_task_init(int batch_size)
++{
++    return NULL;
++}
++
++static inline void buffer_zero_batch_task_destroy(QemuDsaBatchTask *task) {}
++
++static inline int
++buffer_is_zero_dsa_batch_sync(QemuDsaBatchTask *batch_task,
++                              const void **buf, size_t count, size_t len)
++{
++    return -1;
++}
++
+ #endif
+ #endif
 diff --git a/util/dsa.c b/util/dsa.c
 index XXXXXXX..XXXXXXX 100644
 --- a/util/dsa.c
 +++ b/util/dsa.c
-@@ -XXX,XX +XXX,XX @@ uint64_t max_retry_count;
+@@ -XXX,XX +XXX,XX @@ uint32_t max_retry_count;
- static struct dsa_device_group dsa_group;
+ static QemuDsaDeviceGroup dsa_group;
- static struct dsa_completion_thread completion_thread;
+ static QemuDsaCompletionThread completion_thread;
 +static void buffer_zero_dsa_completion(void *context);
  /**
   * @brief This function opens a DSA device's work queue and
-@@ -XXX,XX +XXX,XX @@ dsa_device_group_start(struct dsa_device_group *group)
+@@ -XXX,XX +XXX,XX @@ dsa_device_group_start(QemuDsaDeviceGroup *group)
   *
   * @param group A pointer to the DSA device group.
   */
 -__attribute__((unused))
  static void
- dsa_device_group_stop(struct dsa_device_group *group)
+ dsa_device_group_stop(QemuDsaDeviceGroup *group)
  {
-@@ -XXX,XX +XXX,XX @@ dsa_device_group_cleanup(struct dsa_device_group *group)
+@@ -XXX,XX +XXX,XX @@ dsa_device_group_cleanup(QemuDsaDeviceGroup *group)
-  * @return struct dsa_device* A pointer to the next available DSA device
+  * @return struct QemuDsaDevice* A pointer to the next available DSA device
   *         in the group.
   */
 -__attribute__((unused))
- static struct dsa_device *
+ static QemuDsaDevice *
- dsa_device_group_get_next_device(struct dsa_device_group *group)
+ dsa_device_group_get_next_device(QemuDsaDeviceGroup *group)
  {
-@@ -XXX,XX +XXX,XX @@ dsa_task_enqueue(struct dsa_device_group *group,
+@@ -XXX,XX +XXX,XX @@ dsa_task_enqueue(QemuDsaDeviceGroup *group,
   * @param group A pointer to the DSA device group.
-  * @return buffer_zero_batch_task* The DSA task being dequeued.
+  * @return QemuDsaBatchTask* The DSA task being dequeued.
   */
 -__attribute__((unused))
- static struct buffer_zero_batch_task *
+ static QemuDsaBatchTask *
- dsa_task_dequeue(struct dsa_device_group *group)
+ dsa_task_dequeue(QemuDsaDeviceGroup *group)
  {
 @@ -XXX,XX +XXX,XX @@ submit_wi_int(void *wq, struct dsa_hw_desc *descriptor)
-     return 0;
- }
--/**
-- * @brief Synchronously submits a DSA work item to the
-- *        device work queue.
-- *
-- * @param wq A pointer to the DSA worjk queue's device memory.
-- * @param descriptor A pointer to the DSA work item descriptor.
-- *
-- * @return int Zero if successful, non-zero otherwise.
-- */
--__attribute__((unused))
--static int
--submit_wi(void *wq, struct dsa_hw_desc *descriptor)
--{
--    return submit_wi_int(wq, descriptor);
--}
--
- /**
-  * @brief Asynchronously submits a DSA work item to the
-  *        device work queue.
-@@ -XXX,XX +XXX,XX @@ submit_wi(void *wq, struct dsa_hw_desc *descriptor)
   *
   * @return int Zero if successful, non-zero otherwise.
   */
 -__attribute__((unused))
  static int
- submit_wi_async(struct buffer_zero_batch_task *task)
+ submit_wi_async(QemuDsaBatchTask *task)
  {
-@@ -XXX,XX +XXX,XX @@ submit_wi_async(struct buffer_zero_batch_task *task)
+@@ -XXX,XX +XXX,XX @@ submit_wi_async(QemuDsaBatchTask *task)
   *
   * @return int Zero if successful, non-zero otherwise.
   */
 -__attribute__((unused))
  static int
- submit_batch_wi_async(struct buffer_zero_batch_task *batch_task)
+ submit_batch_wi_async(QemuDsaBatchTask *batch_task)
  {
-@@ -XXX,XX +XXX,XX @@ static void dsa_completion_thread_stop(void *opaque)
+@@ -XXX,XX +XXX,XX @@ void qemu_dsa_cleanup(void)
-     qemu_sem_destroy(&thread_context->sem_init_done);
+     dsa_device_group_cleanup(&dsa_group);
  }
++
++/* Buffer zero comparison DSA task implementations */
++/* =============================================== */
++
++/**
++ * @brief Sets a buffer zero comparison DSA task.
++ *
++ * @param descriptor A pointer to the DSA task descriptor.
++ * @param buf A pointer to the memory buffer.
++ * @param len The length of the buffer.
++ */
++static void
++buffer_zero_task_set_int(struct dsa_hw_desc *descriptor,
++                         const void *buf,
++                         size_t len)
++{
++    struct dsa_completion_record *completion =
++        (struct dsa_completion_record *)descriptor->completion_addr;
++
++    descriptor->xfer_size = len;
++    descriptor->src_addr = (uintptr_t)buf;
++    completion->status = 0;
++    completion->result = 0;
++}
++
++/**
++ * @brief Resets a buffer zero comparison DSA batch task.
++ *
++ * @param task A pointer to the DSA batch task.
++ */
++static void
++buffer_zero_task_reset(QemuDsaBatchTask *task)
++{
++    task->completions[0].status = DSA_COMP_NONE;
++    task->task_type = QEMU_DSA_TASK;
++    task->status = QEMU_DSA_TASK_READY;
++}
++
++/**
++ * @brief Resets a buffer zero comparison DSA batch task.
++ *
++ * @param task A pointer to the batch task.
++ * @param count The number of DSA tasks this batch task will contain.
++ */
++static void
++buffer_zero_batch_task_reset(QemuDsaBatchTask *task, size_t count)
++{
++    task->batch_completion.status = DSA_COMP_NONE;
++    task->batch_descriptor.desc_count = count;
++    task->task_type = QEMU_DSA_BATCH_TASK;
++    task->status = QEMU_DSA_TASK_READY;
++}
++
++/**
++ * @brief Sets a buffer zero comparison DSA task.
++ *
++ * @param task A pointer to the DSA task.
++ * @param buf A pointer to the memory buffer.
++ * @param len The buffer length.
++ */
++static void
++buffer_zero_task_set(QemuDsaBatchTask *task,
++                     const void *buf,
++                     size_t len)
++{
++    buffer_zero_task_reset(task);
++    buffer_zero_task_set_int(&task->descriptors[0], buf, len);
++}
++
++/**
++ * @brief Sets a buffer zero comparison batch task.
++ *
++ * @param batch_task A pointer to the batch task.
++ * @param buf An array of memory buffers.
++ * @param count The number of buffers in the array.
++ * @param len The length of the buffers.
++ */
++static void
++buffer_zero_batch_task_set(QemuDsaBatchTask *batch_task,
++                           const void **buf, size_t count, size_t len)
++{
++    assert(count > 0);
++    assert(count <= batch_task->batch_size);
++
++    buffer_zero_batch_task_reset(batch_task, count);
++    for (int i = 0; i < count; i++) {
++        buffer_zero_task_set_int(&batch_task->descriptors[i], buf[i], len);
++    }
++}
++
++/**
++ * @brief Asychronously perform a buffer zero DSA operation.
++ *
++ * @param task A pointer to the batch task structure.
++ * @param buf A pointer to the memory buffer.
++ * @param len The length of the memory buffer.
++ *
++ * @return int Zero if successful, otherwise an appropriate error code.
++ */
++__attribute__((unused))
++static int
++buffer_zero_dsa_async(QemuDsaBatchTask *task,
++                      const void *buf, size_t len)
++{
++    buffer_zero_task_set(task, buf, len);
++
++    return submit_wi_async(task);
++}
++
++/**
++ * @brief Sends a memory comparison batch task to a DSA device and wait
++ *        for completion.
++ *
++ * @param batch_task The batch task to be submitted to DSA device.
++ * @param buf An array of memory buffers to check for zero.
++ * @param count The number of buffers.
++ * @param len The buffer length.
++ */
++__attribute__((unused))
++static int
++buffer_zero_dsa_batch_async(QemuDsaBatchTask *batch_task,
++                            const void **buf, size_t count, size_t len)
++{
++    assert(count <= batch_task->batch_size);
++    buffer_zero_batch_task_set(batch_task, buf, count, len);
++
++    return submit_batch_wi_async(batch_task);
++}
++
++/**
++ * @brief The completion callback function for buffer zero
++ *        comparison DSA task completion.
++ *
++ * @param context A pointer to the callback context.
++ */
++static void
++buffer_zero_dsa_completion(void *context)
++{
++    assert(context != NULL);
++
++    QemuDsaBatchTask *task = (QemuDsaBatchTask *)context;
++    qemu_sem_post(&task->sem_task_complete);
++}
++
++/**
++ * @brief Wait for the asynchronous DSA task to complete.
++ *
++ * @param batch_task A pointer to the buffer zero comparison batch task.
++ */
++__attribute__((unused))
++static void
++buffer_zero_dsa_wait(QemuDsaBatchTask *batch_task)
++{
++    qemu_sem_wait(&batch_task->sem_task_complete);
++}
++
 +/**
 + * @brief Initializes a buffer zero comparison DSA task.
 + *
 + * @param descriptor A pointer to the DSA task descriptor.
 + * @param completion A pointer to the DSA task completion record.
 ...
 +    descriptor->comp_pattern = (uint64_t)0;
 +    descriptor->completion_addr = (uint64_t)completion;
 +}
 +
 +/**
-+ * @brief Initializes a buffer zero batch task.
++ * @brief Initializes a buffer zero DSA batch task.
 + *
-+ * @param task A pointer to the batch task to initialize.
++ * @param batch_size The number of zero page checking tasks in the batch.
-+ * @param batch_size The number of DSA tasks in the batch.
++ * @return A pointer to the zero page checking tasks initialized.
 + */
-+void
++QemuDsaBatchTask *
-+buffer_zero_batch_task_init(struct buffer_zero_batch_task *task,
++buffer_zero_batch_task_init(int batch_size)
-+                            int batch_size)
++{
-+{
++    QemuDsaBatchTask *task = qemu_memalign(64, sizeof(QemuDsaBatchTask));
 +    int descriptors_size = sizeof(*task->descriptors) * batch_size;
++
 +    memset(task, 0, sizeof(*task));
-+
++    task->addr = g_new0(ram_addr_t, batch_size);
 +    task->results = g_new0(bool, batch_size);
 +    task->batch_size = batch_size;
 +    task->descriptors =
 +        (struct dsa_hw_desc *)qemu_memalign(64, descriptors_size);
 +    memset(task->descriptors, 0, descriptors_size);
 +    task->completions = (struct dsa_completion_record *)qemu_memalign(
 +        32, sizeof(*task->completions) * batch_size);
-+    task->results = g_new0(bool, batch_size);
-+    task->batch_size = batch_size;
 +
 +    task->batch_completion.status = DSA_COMP_NONE;
 +    task->batch_descriptor.completion_addr = (uint64_t)&task->batch_completion;
-+    // TODO: Ensure that we never send a batch with count <= 1
++    /* TODO: Ensure that we never send a batch with count <= 1 */
 +    task->batch_descriptor.desc_count = 0;
 +    task->batch_descriptor.opcode = DSA_OPCODE_BATCH;
 +    task->batch_descriptor.flags = IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CRAV;
 +    task->batch_descriptor.desc_list_addr = (uintptr_t)task->descriptors;
-+    task->status = DSA_TASK_READY;
++    task->status = QEMU_DSA_TASK_READY;
 +    task->group = &dsa_group;
 +    task->device = dsa_device_group_get_next_device(&dsa_group);
 +
 +    for (int i = 0; i < task->batch_size; i++) {
 +        buffer_zero_task_init_int(&task->descriptors[i],
 +                                  &task->completions[i]);
 +    }
 +
 +    qemu_sem_init(&task->sem_task_complete, 0);
 +    task->completion_callback = buffer_zero_dsa_completion;
++
++    return task;
 +}
 +
 +/**
 + * @brief Performs the proper cleanup on a DSA batch task.
 + *
 + * @param task A pointer to the batch task to cleanup.
 + */
 +void
-+buffer_zero_batch_task_destroy(struct buffer_zero_batch_task *task)
++buffer_zero_batch_task_destroy(QemuDsaBatchTask *task)
 +{
-+    qemu_vfree(task->descriptors);
++    if (task) {
-+    qemu_vfree(task->completions);
++        g_free(task->addr);
-+    g_free(task->results);
++        g_free(task->results);
-+
++        qemu_vfree(task->descriptors);
-+    qemu_sem_destroy(&task->sem_task_complete);
++        qemu_vfree(task->completions);
-+}
++        task->results = NULL;
-+
++        qemu_sem_destroy(&task->sem_task_complete);
-+/**
++        qemu_vfree(task);
 + * @brief Resets a buffer zero comparison DSA batch task.
 + *
 + * @param task A pointer to the batch task.
 + * @param count The number of DSA tasks this batch task will contain.
 + */
 +static void
 +buffer_zero_batch_task_reset(struct buffer_zero_batch_task *task, size_t count)
 +{
 +    task->batch_completion.status = DSA_COMP_NONE;
 +    task->batch_descriptor.desc_count = count;
 +    task->task_type = DSA_BATCH_TASK;
 +    task->status = DSA_TASK_READY;
 +}
 +
 +/**
 + * @brief Sets a buffer zero comparison DSA task.
 + *
 + * @param descriptor A pointer to the DSA task descriptor.
 + * @param buf A pointer to the memory buffer.
 + * @param len The length of the buffer.
 + */
 +static void
 +buffer_zero_task_set_int(struct dsa_hw_desc *descriptor,
 +                         const void *buf,
 +                         size_t len)
 +{
 +    struct dsa_completion_record *completion =
 +        (struct dsa_completion_record *)descriptor->completion_addr;
 +
 +    descriptor->xfer_size = len;
 +    descriptor->src_addr = (uintptr_t)buf;
 +    completion->status = 0;
 +    completion->result = 0;
 +}
 +
 +/**
 + * @brief Resets a buffer zero comparison DSA batch task.
 + *
 + * @param task A pointer to the DSA batch task.
 + */
 +static void
 +buffer_zero_task_reset(struct buffer_zero_batch_task *task)
 +{
 +    task->completions[0].status = DSA_COMP_NONE;
 +    task->task_type = DSA_TASK;
 +    task->status = DSA_TASK_READY;
 +}
 +
 +/**
 + * @brief Sets a buffer zero comparison DSA task.
 + *
 + * @param task A pointer to the DSA task.
 + * @param buf A pointer to the memory buffer.
 + * @param len The buffer length.
 + */
 +static void
 +buffer_zero_task_set(struct buffer_zero_batch_task *task,
 +                     const void *buf,
 +                     size_t len)
 +{
 +    buffer_zero_task_reset(task);
 +    buffer_zero_task_set_int(&task->descriptors[0], buf, len);
 +}
 +
 +/**
 + * @brief Sets a buffer zero comparison batch task.
 + *
 + * @param batch_task A pointer to the batch task.
 + * @param buf An array of memory buffers.
 + * @param count The number of buffers in the array.
 + * @param len The length of the buffers.
 + */
 +static void
 +buffer_zero_batch_task_set(struct buffer_zero_batch_task *batch_task,
 +                           const void **buf, size_t count, size_t len)
 +{
 +    assert(count > 0);
 +    assert(count <= batch_task->batch_size);
 +
 +    buffer_zero_batch_task_reset(batch_task, count);
 +    for (int i = 0; i < count; i++) {
 +        buffer_zero_task_set_int(&batch_task->descriptors[i], buf[i], len);
 +    }
 +}
-+
-+/**
-+ * @brief Asychronously perform a buffer zero DSA operation.
-+ *
-+ * @param task A pointer to the batch task structure.
-+ * @param buf A pointer to the memory buffer.
-+ * @param len The length of the memory buffer.
-+ *
-+ * @return int Zero if successful, otherwise an appropriate error code.
-+ */
-+__attribute__((unused))
-+static int
-+buffer_zero_dsa_async(struct buffer_zero_batch_task *task,
-+                      const void *buf, size_t len)
-+{
-+    buffer_zero_task_set(task, buf, len);
-+
-+    return submit_wi_async(task);
-+}
-+
-+/**
-+ * @brief Sends a memory comparison batch task to a DSA device and wait
-+ *        for completion.
-+ *
-+ * @param batch_task The batch task to be submitted to DSA device.
-+ * @param buf An array of memory buffers to check for zero.
-+ * @param count The number of buffers.
-+ * @param len The buffer length.
-+ */
-+__attribute__((unused))
-+static int
-+buffer_zero_dsa_batch_async(struct buffer_zero_batch_task *batch_task,
-+                            const void **buf, size_t count, size_t len)
-+{
-+    assert(count <= batch_task->batch_size);
-+    buffer_zero_batch_task_set(batch_task, buf, count, len);
-+
-+    return submit_batch_wi_async(batch_task);
-+}
-+
-+/**
-+ * @brief The completion callback function for buffer zero
-+ *        comparison DSA task completion.
-+ *
-+ * @param context A pointer to the callback context.
-+ */
-+static void
-+buffer_zero_dsa_completion(void *context)
-+{
-+    assert(context != NULL);
-+
-+    struct buffer_zero_batch_task *task =
-+        (struct buffer_zero_batch_task *)context;
-+    qemu_sem_post(&task->sem_task_complete);
-+}
-+
-+/**
-+ * @brief Wait for the asynchronous DSA task to complete.
-+ *
-+ * @param batch_task A pointer to the buffer zero comparison batch task.
-+ */
-+__attribute__((unused))
-+static void
-+buffer_zero_dsa_wait(struct buffer_zero_batch_task *batch_task)
-+{
-+    qemu_sem_wait(&batch_task->sem_task_complete);
-+}
-+
- /**
-  * @brief Check if DSA is running.
-  *
-@@ -XXX,XX +XXX,XX @@ void dsa_cleanup(void)
- #else
-+void buffer_zero_batch_task_init(struct buffer_zero_batch_task *task,
-+                                 int batch_size) {}
-+
-+void buffer_zero_batch_task_destroy(struct buffer_zero_batch_task *task) {}
-+
- bool dsa_is_running(void)
- {
-     return false;
 --
-.30.2
+Yichen Wang

-[PATCH v2 11/20] util/dsa: Implement DSA task asynchronous submission and wait for completion.
+[PATCH v9 07/12] util/dsa: Implement DSA task asynchronous submission and wait for completion.
+From: Hao Xiang <hao.xiang@linux.dev>
 * Add a DSA task completion callback.
 * DSA completion thread will call the tasks's completion callback
 on every task/batch task completion.
 * DSA submission path to wait for completion.
 * Implement CPU fallback if DSA is not able to complete the task.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
 Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
+Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
+Reviewed-by: Fabiano Rosas <farosas@suse.de>
 ---
  include/qemu/dsa.h |  14 +++++
- util/dsa.c         | 153 ++++++++++++++++++++++++++++++++++++++++++++-
+ util/dsa.c         | 125 +++++++++++++++++++++++++++++++++++++++++++--
-files changed, 164 insertions(+), 3 deletions(-)
+files changed, 135 insertions(+), 4 deletions(-)
 diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
 index XXXXXXX..XXXXXXX 100644
 --- a/include/qemu/dsa.h
 +++ b/include/qemu/dsa.h
-@@ -XXX,XX +XXX,XX @@ void buffer_zero_batch_task_init(struct buffer_zero_batch_task *task,
+@@ -XXX,XX +XXX,XX @@ buffer_zero_batch_task_init(int batch_size);
   */
- void buffer_zero_batch_task_destroy(struct buffer_zero_batch_task *task);
+ void buffer_zero_batch_task_destroy(QemuDsaBatchTask *task);
 +/**
-+ * @brief Performs buffer zero comparison on a DSA batch task asynchronously.
++ * @brief Performs buffer zero comparison on a DSA batch task synchronously.
 + *
 + * @param batch_task A pointer to the batch task.
 + * @param buf An array of memory buffers.
 + * @param count The number of buffers in the array.
 + * @param len The buffer length.
 + *
 + * @return Zero if successful, otherwise non-zero.
 + */
 +int
-+buffer_is_zero_dsa_batch_async(struct buffer_zero_batch_task *batch_task,
++buffer_is_zero_dsa_batch_sync(QemuDsaBatchTask *batch_task,
-+                               const void **buf, size_t count, size_t len);
++                              const void **buf, size_t count, size_t len);
 +
- /**
+ #else
-  * @brief Initializes DSA devices.
-  *
+ typedef struct QemuDsaBatchTask {} QemuDsaBatchTask;
 diff --git a/util/dsa.c b/util/dsa.c
 index XXXXXXX..XXXXXXX 100644
 --- a/util/dsa.c
 +++ b/util/dsa.c
 @@ -XXX,XX +XXX,XX @@ poll_completion(struct dsa_completion_record *completion,
      return 0;
  }
 +/**
-+ * @brief Use CPU to complete a single zero page checking task.
++ * @brief Helper function to use CPU to complete a single
-+ *
++ *        zero page checking task.
-+ * @param task A pointer to the task.
++ *
 + * @param completion A pointer to a DSA task completion record.
 + * @param descriptor A pointer to a DSA task descriptor.
 + * @param result A pointer to the result of a zero page checking.
 + */
 +static void
-+task_cpu_fallback(struct buffer_zero_batch_task *task)
++task_cpu_fallback_int(struct dsa_completion_record *completion,
 +                      struct dsa_hw_desc *descriptor, bool *result)
 +{
-+    assert(task->task_type == DSA_TASK);
-+
-+    struct dsa_completion_record *completion = &task->completions[0];
 +    const uint8_t *buf;
 +    size_t len;
 +
 +    if (completion->status == DSA_COMP_SUCCESS) {
 +        return;
 ...
 +     * DSA was able to partially complete the operation. Check the
 +     * result. If we already know this is not a zero page, we can
 +     * return now.
 +     */
 +    if (completion->bytes_completed != 0 && completion->result != 0) {
-+        task->results[0] = false;
++        *result = false;
 +        return;
 +    }
 +
 +    /* Let's fallback to use CPU to complete it. */
-+    buf = (const uint8_t *)task->descriptors[0].src_addr;
++    buf = (const uint8_t *)descriptor->src_addr;
-+    len = task->descriptors[0].xfer_size;
++    len = descriptor->xfer_size;
-+    task->results[0] = buffer_is_zero(buf + completion->bytes_completed,
++    *result = buffer_is_zero(buf + completion->bytes_completed,
-+                                      len - completion->bytes_completed);
++                             len - completion->bytes_completed);
 +}
 +
  /**
   * @brief Complete a single DSA task in the batch task.
   *
-@@ -XXX,XX +XXX,XX @@ poll_batch_task_completion(struct buffer_zero_batch_task *batch_task)
+@@ -XXX,XX +XXX,XX @@ dsa_completion_loop(void *opaque)
-     }
+         (QemuDsaCompletionThread *)opaque;
- }
+     QemuDsaBatchTask *batch_task;
+     QemuDsaDeviceGroup *group = thread_context->group;
-+/**
+-    int ret;
-+ * @brief Use CPU to complete the zero page checking batch task.
++    int ret = 0;
-+ *
-+ * @param batch_task A pointer to the batch task.
+     rcu_register_thread();
-+ */
-+static void
+@@ -XXX,XX +XXX,XX @@ buffer_zero_batch_task_set(QemuDsaBatchTask *batch_task,
 +batch_task_cpu_fallback(struct buffer_zero_batch_task *batch_task)
 +{
 +    assert(batch_task->task_type == DSA_BATCH_TASK);
 +
 +    struct dsa_completion_record *batch_completion =
 +        &batch_task->batch_completion;
 +    struct dsa_completion_record *completion;
 +    uint8_t status;
 +    const uint8_t *buf;
 +    size_t len;
 +    bool *results = batch_task->results;
 +    uint32_t count = batch_task->batch_descriptor.desc_count;
 +
 +    // DSA is able to complete the entire batch task.
 +    if (batch_completion->status == DSA_COMP_SUCCESS) {
 +        assert(count == batch_completion->bytes_completed);
 +        return;
 +    }
 +
 +    /*
 +     * DSA encounters some error and is not able to complete
 +     * the entire batch task. Use CPU fallback.
 +     */
 +    for (int i = 0; i < count; i++) {
 +        completion = &batch_task->completions[i];
 +        status = completion->status;
 +        if (status == DSA_COMP_SUCCESS) {
 +            continue;
 +        }
 +        assert(status == DSA_COMP_PAGE_FAULT_NOBOF);
 +
 +        /*
 +         * DSA was able to partially complete the operation. Check the
 +         * result. If we already know this is not a zero page, we can
 +         * return now.
 +         */
 +        if (completion->bytes_completed != 0 && completion->result != 0) {
 +            results[i] = false;
 +            continue;
 +        }
 +
 +        /* Let's fallback to use CPU to complete it. */
 +        buf = (uint8_t *)batch_task->descriptors[i].src_addr;
 +        len = batch_task->descriptors[i].xfer_size;
 +        results[i] =
 +            buffer_is_zero(buf + completion->bytes_completed,
 +                           len - completion->bytes_completed);
 +    }
 +}
 +
  /**
   * @brief Handles an asynchronous DSA batch task completion.
   *
@@ -XXX,XX +XXX,XX @@ buffer_zero_batch_task_set(struct buffer_zero_batch_task *batch_task,
   *
   * @return int Zero if successful, otherwise an appropriate error code.
   */
 -__attribute__((unused))
  static int
- buffer_zero_dsa_async(struct buffer_zero_batch_task *task,
+ buffer_zero_dsa_async(QemuDsaBatchTask *task,
                        const void *buf, size_t len)
-@@ -XXX,XX +XXX,XX @@ buffer_zero_dsa_async(struct buffer_zero_batch_task *task,
+@@ -XXX,XX +XXX,XX @@ buffer_zero_dsa_async(QemuDsaBatchTask *task,
   * @param count The number of buffers.
   * @param len The buffer length.
   */
 -__attribute__((unused))
  static int
- buffer_zero_dsa_batch_async(struct buffer_zero_batch_task *batch_task,
+ buffer_zero_dsa_batch_async(QemuDsaBatchTask *batch_task,
                              const void **buf, size_t count, size_t len)
 @@ -XXX,XX +XXX,XX @@ buffer_zero_dsa_completion(void *context)
   *
   * @param batch_task A pointer to the buffer zero comparison batch task.
   */
 -__attribute__((unused))
  static void
- buffer_zero_dsa_wait(struct buffer_zero_batch_task *batch_task)
+ buffer_zero_dsa_wait(QemuDsaBatchTask *batch_task)
  {
      qemu_sem_wait(&batch_task->sem_task_complete);
  }
 +/**
 + * @brief Use CPU to complete the zero page checking task if DSA
 + *        is not able to complete it.
 + *
 + * @param batch_task A pointer to the batch task.
 + */
 +static void
-+buffer_zero_cpu_fallback(struct buffer_zero_batch_task *batch_task)
++buffer_zero_cpu_fallback(QemuDsaBatchTask *batch_task)
 +{
-+    if (batch_task->task_type == DSA_TASK) {
++    if (batch_task->task_type == QEMU_DSA_TASK) {
-+        task_cpu_fallback(batch_task);
++        if (batch_task->completions[0].status == DSA_COMP_SUCCESS) {
-+    } else {
++            return;
-+        assert(batch_task->task_type == DSA_BATCH_TASK);
++        }
-+        batch_task_cpu_fallback(batch_task);
++        task_cpu_fallback_int(&batch_task->completions[0],
 +                              &batch_task->descriptors[0],
 +                              &batch_task->results[0]);
 +    } else if (batch_task->task_type == QEMU_DSA_BATCH_TASK) {
 +        struct dsa_completion_record *batch_completion =
 +            &batch_task->batch_completion;
 +        struct dsa_completion_record *completion;
 +        uint8_t status;
 +        bool *results = batch_task->results;
 +        uint32_t count = batch_task->batch_descriptor.desc_count;
 +
 +        /* DSA is able to complete the entire batch task. */
 +        if (batch_completion->status == DSA_COMP_SUCCESS) {
 +            assert(count == batch_completion->bytes_completed);
 +            return;
 +        }
 +
 +        /*
 +         * DSA encounters some error and is not able to complete
 +         * the entire batch task. Use CPU fallback.
 +         */
 +        for (int i = 0; i < count; i++) {
 +
 +            completion = &batch_task->completions[i];
 +            status = completion->status;
 +
 +            assert(status == DSA_COMP_SUCCESS ||
 +                status == DSA_COMP_PAGE_FAULT_NOBOF);
 +
 +            task_cpu_fallback_int(completion,
 +                                  &batch_task->descriptors[i],
 +                                  &results[i]);
 +        }
 +    }
 +}
 +
  /**
-  * @brief Check if DSA is running.
+  * @brief Initializes a buffer zero comparison DSA task.
   *
-@@ -XXX,XX +XXX,XX @@ void dsa_cleanup(void)
+@@ -XXX,XX +XXX,XX @@ buffer_zero_batch_task_destroy(QemuDsaBatchTask *task)
-     dsa_device_group_cleanup(&dsa_group);
+         qemu_vfree(task);
      }
  }
++
 +/**
-+ * @brief Performs buffer zero comparison on a DSA batch task asynchronously.
++ * @brief Performs buffer zero comparison on a DSA batch task synchronously.
 + *
 + * @param batch_task A pointer to the batch task.
 + * @param buf An array of memory buffers.
 + * @param count The number of buffers in the array.
 + * @param len The buffer length.
 + *
 + * @return Zero if successful, otherwise non-zero.
 + */
 +int
-+buffer_is_zero_dsa_batch_async(struct buffer_zero_batch_task *batch_task,
++buffer_is_zero_dsa_batch_sync(QemuDsaBatchTask *batch_task,
-+                               const void **buf, size_t count, size_t len)
++                              const void **buf, size_t count, size_t len)
 +{
-+    if (count <= 0 || count > batch_task->batch_size) {
-+        return -1;
-+    }
-+
 +    assert(batch_task != NULL);
 +    assert(len != 0);
 +    assert(buf != NULL);
 +
++    if (count <= 0 || count > batch_task->batch_size) {
++        return -1;
++    }
++
 +    if (count == 1) {
-+        // DSA doesn't take batch operation with only 1 task.
++        /* DSA doesn't take batch operation with only 1 task. */
 +        buffer_zero_dsa_async(batch_task, buf[0], len);
 +    } else {
 +        buffer_zero_dsa_batch_async(batch_task, buf, count, len);
 +    }
 +
 +    buffer_zero_dsa_wait(batch_task);
 +    buffer_zero_cpu_fallback(batch_task);
 +
 +    return 0;
 +}
-+
- #else
- void buffer_zero_batch_task_init(struct buffer_zero_batch_task *task,
-@@ -XXX,XX +XXX,XX @@ void dsa_stop(void) {}
- void dsa_cleanup(void) {}
-+int
-+buffer_is_zero_dsa_batch_async(struct buffer_zero_batch_task *batch_task,
-+                               const void **buf, size_t count, size_t len)
-+{
-+    exit(1);
-+}
-+
- #endif
 --
-.30.2
+Yichen Wang

-[PATCH v2 12/20] migration/multifd: Add new migration option for multifd DSA offloading.
+[PATCH v9 08/12] migration/multifd: Add new migration option for multifd DSA offloading.
+From: Hao Xiang <hao.xiang@linux.dev>
 Intel DSA offloading is an optional feature that turns on if
 proper hardware and software stack is available. To turn on
-DSA offloading in multifd live migration:
+DSA offloading in multifd live migration by setting:
-multifd-dsa-accel="[dsa_dev_path1] ] [dsa_dev_path2] ... [dsa_dev_pathX]"
+zero-page-detection=dsa-accel
 accel-path="dsa:<dsa_dev_path1> dsa:[dsa_dev_path2] ..."
 This feature is turned off by default.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
 Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
 Acked-by: Dr. David Alan Gilbert <dave@treblig.org>
 ---
- migration/migration-hmp-cmds.c |  8 ++++++++
+ hmp-commands.hx                |  2 +-
- migration/options.c            | 28 ++++++++++++++++++++++++++++
+ include/qemu/dsa.h             | 13 +++++++++++++
  migration/migration-hmp-cmds.c | 20 +++++++++++++++++++-
  migration/options.c            | 30 ++++++++++++++++++++++++++++++
  migration/options.h            |  1 +
- qapi/migration.json            | 17 ++++++++++++++---
+ qapi/migration.json            | 32 ++++++++++++++++++++++++++++----
- scripts/meson-buildoptions.sh  |  6 +++---
+ util/dsa.c                     | 31 +++++++++++++++++++++++++++++++
-files changed, 54 insertions(+), 6 deletions(-)
+files changed, 123 insertions(+), 6 deletions(-)
 diff --git a/hmp-commands.hx b/hmp-commands.hx
 index XXXXXXX..XXXXXXX 100644
 --- a/hmp-commands.hx
 +++ b/hmp-commands.hx
@@ -XXX,XX +XXX,XX @@ ERST
      {
          .name       = "migrate_set_parameter",
 -        .args_type  = "parameter:s,value:s",
 +        .args_type  = "parameter:s,value:S",
          .params     = "parameter value",
          .help       = "Set the parameter for migration",
          .cmd        = hmp_migrate_set_parameter,
 diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
 index XXXXXXX..XXXXXXX 100644
 --- a/include/qemu/dsa.h
 +++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@ void qemu_dsa_stop(void);
   */
  void qemu_dsa_cleanup(void);
 +/**
 + * @brief Check if DSA is supported.
 + *
 + * @return True if DSA is supported, otherwise false.
 + */
 +bool qemu_dsa_is_supported(void);
 +
  /**
   * @brief Check if DSA is running.
   *
@@ -XXX,XX +XXX,XX @@ buffer_is_zero_dsa_batch_sync(QemuDsaBatchTask *batch_task,
  typedef struct QemuDsaBatchTask {} QemuDsaBatchTask;
 +static inline bool qemu_dsa_is_supported(void)
 +{
 +    return false;
 +}
 +
 +
  static inline bool qemu_dsa_is_running(void)
  {
      return false;
 diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/migration-hmp-cmds.c
 +++ b/migration/migration-hmp-cmds.c
 @@ -XXX,XX +XXX,XX @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
          monitor_printf(mon, "%s: '%s'\n",
              MigrationParameter_str(MIGRATION_PARAMETER_TLS_AUTHZ),
              params->tls_authz);
-+        monitor_printf(mon, "%s: %s\n",
+-
-+            MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_DSA_ACCEL),
++        if (params->has_accel_path) {
-+            params->multifd_dsa_accel);
++            strList *accel_path = params->accel_path;
++            monitor_printf(mon, "%s:",
 +                MigrationParameter_str(MIGRATION_PARAMETER_ACCEL_PATH));
 +            while (accel_path) {
 +                monitor_printf(mon, " '%s'", accel_path->value);
 +                accel_path = accel_path->next;
 +            }
 +            monitor_printf(mon, "\n");
 +        }
          if (params->has_block_bitmap_mapping) {
              const BitmapMigrationNodeAliasList *bmnal;
 @@ -XXX,XX +XXX,XX @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
-         p->has_block_incremental = true;
+         p->has_x_checkpoint_delay = true;
-         visit_type_bool(v, param, &p->block_incremental, &err);
+         visit_type_uint32(v, param, &p->x_checkpoint_delay, &err);
          break;
-+    case MIGRATION_PARAMETER_MULTIFD_DSA_ACCEL:
++    case MIGRATION_PARAMETER_ACCEL_PATH:
-+        p->multifd_dsa_accel = g_new0(StrOrNull, 1);
++        p->has_accel_path = true;
-+        p->multifd_dsa_accel->type = QTYPE_QSTRING;
++        char **strv = g_strsplit(valuestr ? : "", " ", -1);
-+        visit_type_str(v, param, &p->multifd_dsa_accel->u.s, &err);
++        strList **tail = &p->accel_path;
 +        for (int i = 0; strv[i]; i++) {
 +            QAPI_LIST_APPEND(tail, g_strdup(strv[i]));
 +        }
 +        g_strfreev(strv);
 +        break;
      case MIGRATION_PARAMETER_MULTIFD_CHANNELS:
          p->has_multifd_channels = true;
          visit_type_uint8(v, param, &p->multifd_channels, &err);
 diff --git a/migration/options.c b/migration/options.c
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/options.c
 +++ b/migration/options.c
-@@ -XXX,XX +XXX,XX @@ Property migration_properties[] = {
+@@ -XXX,XX +XXX,XX @@
-     DEFINE_PROP_MIG_MODE("mode", MigrationState,
-                       parameters.mode,
+ #include "qemu/osdep.h"
-                       MIG_MODE_NORMAL),
+ #include "qemu/error-report.h"
-+    DEFINE_PROP_STRING("multifd-dsa-accel", MigrationState,
++#include "qemu/dsa.h"
-+                       parameters.multifd_dsa_accel),
+ #include "exec/target_page.h"
+ #include "qapi/clone-visitor.h"
-     /* Migration capabilities */
+ #include "qapi/error.h"
      DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
 @@ -XXX,XX +XXX,XX @@ const char *migrate_tls_creds(void)
      return s->parameters.tls_creds;
  }
-+const char *migrate_multifd_dsa_accel(void)
++const strList *migrate_accel_path(void)
 +{
 +    MigrationState *s = migrate_get_current();
 +
-+    return s->parameters.multifd_dsa_accel;
++    return s->parameters.accel_path;
 +}
 +
  const char *migrate_tls_hostname(void)
  {
      MigrationState *s = migrate_get_current();
 @@ -XXX,XX +XXX,XX @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
-     params->vcpu_dirty_limit = s->parameters.vcpu_dirty_limit;
+     params->zero_page_detection = s->parameters.zero_page_detection;
-     params->has_mode = true;
+     params->has_direct_io = true;
-     params->mode = s->parameters.mode;
+     params->direct_io = s->parameters.direct_io;
-+    params->multifd_dsa_accel = s->parameters.multifd_dsa_accel;
++    params->has_accel_path = true;
 +    params->accel_path = QAPI_CLONE(strList, s->parameters.accel_path);
      return params;
  }
 @@ -XXX,XX +XXX,XX @@ void migrate_params_init(MigrationParameters *params)
  {
      params->tls_hostname = g_strdup("");
      params->tls_creds = g_strdup("");
-+    params->multifd_dsa_accel = g_strdup("");
++    params->accel_path = NULL;
      /* Set has_* up only for parameter checks */
-     params->has_compress_level = true;
+     params->has_throttle_trigger_threshold = true;
@@ -XXX,XX +XXX,XX @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
          return false;
      }
 +    if (params->has_zero_page_detection &&
 +        params->zero_page_detection == ZERO_PAGE_DETECTION_DSA_ACCEL) {
 +        if (!qemu_dsa_is_supported()) {
 +            error_setg(errp, "DSA acceleration is not supported.");
 +            return false;
 +        }
 +    }
 +
      return true;
  }
 @@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrateSetParameters *params,
-     if (params->has_mode) {
+     if (params->has_direct_io) {
-         dest->mode = params->mode;
+         dest->direct_io = params->direct_io;
      }
 +
-+    if (params->multifd_dsa_accel) {
++    if (params->has_accel_path) {
-+        assert(params->multifd_dsa_accel->type == QTYPE_QSTRING);
++        dest->has_accel_path = true;
-+        dest->multifd_dsa_accel = params->multifd_dsa_accel->u.s;
++        dest->accel_path = params->accel_path;
 +    }
  }
  static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
 @@ -XXX,XX +XXX,XX @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
-     if (params->has_mode) {
+     if (params->has_direct_io) {
-         s->parameters.mode = params->mode;
+         s->parameters.direct_io = params->direct_io;
      }
-+
++    if (params->has_accel_path) {
-+    if (params->multifd_dsa_accel) {
++        qapi_free_strList(s->parameters.accel_path);
-+        g_free(s->parameters.multifd_dsa_accel);
++        s->parameters.has_accel_path = true;
-+        assert(params->multifd_dsa_accel->type == QTYPE_QSTRING);
++        s->parameters.accel_path =
-+        s->parameters.multifd_dsa_accel = g_strdup(params->multifd_dsa_accel->u.s);
++            QAPI_CLONE(strList, params->accel_path);
 +    }
  }
  void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
-@@ -XXX,XX +XXX,XX @@ void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
-         params->tls_authz->type = QTYPE_QSTRING;
-         params->tls_authz->u.s = strdup("");
-     }
-+    if (params->multifd_dsa_accel
-+        && params->multifd_dsa_accel->type == QTYPE_QNULL) {
-+        qobject_unref(params->multifd_dsa_accel->u.n);
-+        params->multifd_dsa_accel->type = QTYPE_QSTRING;
-+        params->multifd_dsa_accel->u.s = strdup("");
-+    }
-     migrate_params_test_apply(params, &tmp);
 diff --git a/migration/options.h b/migration/options.h
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/options.h
 +++ b/migration/options.h
-@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_authz(void);
+@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_creds(void);
  const char *migrate_tls_creds(void);
  const char *migrate_tls_hostname(void);
  uint64_t migrate_xbzrle_cache_size(void);
-+const char *migrate_multifd_dsa_accel(void);
+ ZeroPageDetection migrate_zero_page_detection(void);
++const strList *migrate_accel_path(void);
- /* parameters setters */
  /* parameters helpers */
 diff --git a/qapi/migration.json b/qapi/migration.json
 index XXXXXXX..XXXXXXX 100644
 --- a/qapi/migration.json
 +++ b/qapi/migration.json
 @@ -XXX,XX +XXX,XX @@
- # @mode: Migration mode. See description in @MigMode. Default is 'normal'.
+ #     multifd migration is enabled, else in the main migration thread
- #        (Since 8.2)
+ #     as for @legacy.
  #
-+# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
++# @dsa-accel: Perform zero page checking with the DSA accelerator
-+#                     certain memory operations. (since 8.2)
++#     offloading in multifd sender thread if multifd migration is
 +#     enabled, else in the main migration thread as for @legacy.
 +#
- # Features:
+ # Since: 9.0
  ##
  { 'enum': 'ZeroPageDetection',
 -  'data': [ 'none', 'legacy', 'multifd' ] }
 +  'data': [ 'none', 'legacy', 'multifd', 'dsa-accel' ] }
  ##
  # @BitmapMigrationBitmapAliasTransform:
@@ -XXX,XX +XXX,XX @@
  #     See description in @ZeroPageDetection.  Default is 'multifd'.
  #     (since 9.0)
  #
- # @deprecated: Member @block-incremental is deprecated.  Use
++# @accel-path: If enabled, specify the accelerator paths that to be
 +#     used in QEMU. For example, enable DSA accelerator for zero page
 +#     detection offloading by setting the @zero-page-detection to
 +#     dsa-accel, and defines the accel-path to "dsa:<dsa_device path>".
 +#     This parameter is default to an empty list.  (Since 10.0)
 +#
  # @direct-io: Open migration files with O_DIRECT when possible.  This
  #     only has effect if the @mapped-ram capability is enabled.
  #     (Since 9.1)
 @@ -XXX,XX +XXX,XX @@
             'cpu-throttle-initial', 'cpu-throttle-increment',
             'cpu-throttle-tailslow',
             'tls-creds', 'tls-hostname', 'tls-authz', 'max-bandwidth',
 -           'avail-switchover-bandwidth', 'downtime-limit',
-+           'avail-switchover-bandwidth', 'downtime-limit', 'multifd-dsa-accel',
++           'avail-switchover-bandwidth', 'downtime-limit', 'accel-path',
             { 'name': 'x-checkpoint-delay', 'features': [ 'unstable' ] },
-            { 'name': 'block-incremental', 'features': [ 'deprecated' ] },
             'multifd-channels',
-@@ -XXX,XX +XXX,XX @@
+            'xbzrle-cache-size', 'max-postcopy-bandwidth',
- # @mode: Migration mode. See description in @MigMode. Default is 'normal'.
+@@ -XXX,XX +XXX,XX @@
- #        (Since 8.2)
+ #     See description in @ZeroPageDetection.  Default is 'multifd'.
  #     (since 9.0)
  #
-+# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
++# @accel-path: If enabled, specify the accelerator paths that to be
-+#                     certain memory operations. (since 8.2)
++#     used in QEMU. For example, enable DSA accelerator for zero page
 +#     detection offloading by setting the @zero-page-detection to
 +#     dsa-accel, and defines the accel-path to "dsa:<dsa_device path>".
 +#     This parameter is default to an empty list.  (Since 10.0)
 +#
- # Features:
+ # @direct-io: Open migration files with O_DIRECT when possible.  This
- #
+ #     only has effect if the @mapped-ram capability is enabled.
- # @deprecated: Member @block-incremental is deprecated.  Use
+ #     (Since 9.1)
 @@ -XXX,XX +XXX,XX @@
              '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
                                              'features': [ 'unstable' ] },
              '*vcpu-dirty-limit': 'uint64',
--            '*mode': 'MigMode'} }
+             '*mode': 'MigMode',
-+            '*mode': 'MigMode',
+             '*zero-page-detection': 'ZeroPageDetection',
-+            '*multifd-dsa-accel': 'StrOrNull'} }
+-            '*direct-io': 'bool' } }
 +            '*direct-io': 'bool',
 +            '*accel-path': [ 'str' ] } }
  ##
  # @migrate-set-parameters:
 @@ -XXX,XX +XXX,XX @@
- # @mode: Migration mode. See description in @MigMode. Default is 'normal'.
+ #     See description in @ZeroPageDetection.  Default is 'multifd'.
- #        (Since 8.2)
+ #     (since 9.0)
  #
-+# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
++# @accel-path: If enabled, specify the accelerator paths that to be
-+#                     certain memory operations. (since 8.2)
++#     used in QEMU. For example, enable DSA accelerator for zero page
 +#     detection offloading by setting the @zero-page-detection to
 +#     dsa-accel, and defines the accel-path to "dsa:<dsa_device path>".
 +#     This parameter is default to an empty list.  (Since 10.0)
 +#
- # Features:
+ # @direct-io: Open migration files with O_DIRECT when possible.  This
- #
+ #     only has effect if the @mapped-ram capability is enabled.
- # @deprecated: Member @block-incremental is deprecated.  Use
+ #     (Since 9.1)
 @@ -XXX,XX +XXX,XX @@
              '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
                                              'features': [ 'unstable' ] },
              '*vcpu-dirty-limit': 'uint64',
--            '*mode': 'MigMode'} }
+             '*mode': 'MigMode',
-+            '*mode': 'MigMode',
+             '*zero-page-detection': 'ZeroPageDetection',
-+            '*multifd-dsa-accel': 'str'} }
+-            '*direct-io': 'bool' } }
 +            '*direct-io': 'bool',
 +            '*accel-path': [ 'str' ] } }
  ##
  # @query-migrate-parameters:
-diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
+diff --git a/util/dsa.c b/util/dsa.c
 index XXXXXXX..XXXXXXX 100644
---- a/scripts/meson-buildoptions.sh
+--- a/util/dsa.c
-+++ b/scripts/meson-buildoptions.sh
++++ b/util/dsa.c
-@@ -XXX,XX +XXX,XX @@ meson_options_help() {
+@@ -XXX,XX +XXX,XX @@
-   printf "%s\n" '  --enable-debug-stack-usage'
+ #include "qemu/bswap.h"
-   printf "%s\n" '                           measure coroutine stack usage'
+ #include "qemu/error-report.h"
-   printf "%s\n" '  --enable-debug-tcg       TCG debugging'
+ #include "qemu/rcu.h"
-+  printf "%s\n" '  --enable-enqcmd          MENQCMD optimizations'
++#include <cpuid.h>
-   printf "%s\n" '  --enable-fdt[=CHOICE]    Whether and how to find the libfdt library'
-   printf "%s\n" '                           (choices: auto/disabled/enabled/internal/system)'
+ #pragma GCC push_options
-   printf "%s\n" '  --enable-fuzzing         build fuzzing targets'
+ #pragma GCC target("enqcmd")
-@@ -XXX,XX +XXX,XX @@ meson_options_help() {
+@@ -XXX,XX +XXX,XX @@ static void dsa_completion_thread_stop(void *opaque)
-   printf "%s\n" '  avx2            AVX2 optimizations'
+     qemu_sem_destroy(&thread_context->sem_init_done);
-   printf "%s\n" '  avx512bw        AVX512BW optimizations'
+ }
-   printf "%s\n" '  avx512f         AVX512F optimizations'
--  printf "%s\n" '  enqcmd          ENQCMD optimizations'
++/**
-   printf "%s\n" '  blkio           libblkio block device driver'
++ * @brief Check if DSA is supported.
-   printf "%s\n" '  bochs           bochs image format support'
++ *
-   printf "%s\n" '  bpf             eBPF support'
++ * @return True if DSA is supported, otherwise false.
-@@ -XXX,XX +XXX,XX @@ _meson_option_parse() {
++ */
-     --disable-avx512bw) printf "%s" -Davx512bw=disabled ;;
++bool qemu_dsa_is_supported(void)
-     --enable-avx512f) printf "%s" -Davx512f=enabled ;;
++{
-     --disable-avx512f) printf "%s" -Davx512f=disabled ;;
++    /*
--    --enable-enqcmd) printf "%s" -Denqcmd=true ;;
++     * movdir64b is indicated by bit 28 of ecx in CPUID leaf 7, subleaf 0.
--    --disable-enqcmd) printf "%s" -Denqcmd=false ;;
++     * enqcmd is indicated by bit 29 of ecx in CPUID leaf 7, subleaf 0.
-     --enable-gcov) printf "%s" -Db_coverage=true ;;
++     * Doc: https://cdrdv2-public.intel.com/819680/architecture-instruction-\
-     --disable-gcov) printf "%s" -Db_coverage=false ;;
++     *      set-extensions-programming-reference.pdf
-     --enable-lto) printf "%s" -Db_lto=true ;;
++     */
-@@ -XXX,XX +XXX,XX @@ _meson_option_parse() {
++    uint32_t eax, ebx, ecx, edx;
-     --disable-docs) printf "%s" -Ddocs=disabled ;;
++    bool movedirb_enabled;
-     --enable-dsound) printf "%s" -Ddsound=enabled ;;
++    bool enqcmd_enabled;
-     --disable-dsound) printf "%s" -Ddsound=disabled ;;
++
-+    --enable-enqcmd) printf "%s" -Denqcmd=true ;;
++    __get_cpuid_count(7, 0, &eax, &ebx, &ecx, &edx);
-+    --disable-enqcmd) printf "%s" -Denqcmd=false ;;
++    movedirb_enabled = (ecx >> 28) & 0x1;
-     --enable-fdt) printf "%s" -Dfdt=enabled ;;
++    if (!movedirb_enabled) {
-     --disable-fdt) printf "%s" -Dfdt=disabled ;;
++        return false;
-     --enable-fdt=*) quote_sh "-Dfdt=$2" ;;
++    }
 +    enqcmd_enabled = (ecx >> 29) & 0x1;
 +    if (!enqcmd_enabled) {
 +        return false;
 +    }
 +
 +    return true;
 +}
 +
  /**
   * @brief Check if DSA is running.
   *
 --
-.30.2
+Yichen Wang

-[PATCH v2 14/20] migration/multifd: Enable DSA offloading in multifd sender path.
+[PATCH v9 09/12] migration/multifd: Enable DSA offloading in multifd sender path.
+From: Hao Xiang <hao.xiang@linux.dev>
 Multifd sender path gets an array of pages queued by the migration
 thread. It performs zero page checking on every page in the array.
 The pages are classfied as either a zero page or a normal page. This
 change uses Intel DSA to offload the zero page checking from CPU to
 the DSA accelerator. The sender thread submits a batch of pages to DSA
 hardware and waits for the DSA completion thread to signal for work
 completion.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
 Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
 ---
- migration/multifd.c | 33 ++++++++++++++++++++++++++++++++-
+ migration/multifd-zero-page.c | 149 ++++++++++++++++++++++++++++++----
-file changed, 32 insertions(+), 1 deletion(-)
+ migration/multifd.c           |  23 +++++-
  migration/multifd.h           |   6 ++
  migration/options.c           |  13 +++
  migration/options.h           |   1 +
 files changed, 176 insertions(+), 16 deletions(-)
+diff --git a/migration/multifd-zero-page.c b/migration/multifd-zero-page.c
+index XXXXXXX..XXXXXXX 100644
+--- a/migration/multifd-zero-page.c
++++ b/migration/multifd-zero-page.c
+@@ -XXX,XX +XXX,XX @@
+ static bool multifd_zero_page_enabled(void)
+ {
+-    return migrate_zero_page_detection() == ZERO_PAGE_DETECTION_MULTIFD;
++    ZeroPageDetection curMethod = migrate_zero_page_detection();
++    return (curMethod == ZERO_PAGE_DETECTION_MULTIFD ||
++            curMethod == ZERO_PAGE_DETECTION_DSA_ACCEL);
+ }
+ static void swap_page_offset(ram_addr_t *pages_offset, int a, int b)
+@@ -XXX,XX +XXX,XX @@ static void swap_page_offset(ram_addr_t *pages_offset, int a, int b)
+     pages_offset[b] = temp;
+ }
++#ifdef CONFIG_DSA_OPT
++
++static void swap_result(bool *results, int a, int b)
++{
++    bool temp;
++
++    if (a == b) {
++        return;
++    }
++
++    temp = results[a];
++    results[a] = results[b];
++    results[b] = temp;
++}
++
+ /**
+- * multifd_send_zero_page_detect: Perform zero page detection on all pages.
++ * zero_page_detect_dsa: Perform zero page detection using
++ * Intel Data Streaming Accelerator (DSA).
+  *
+- * Sorts normal pages before zero pages in p->pages->offset and updates
+- * p->pages->normal_num.
++ * Sorts normal pages before zero pages in pages->offset and updates
++ * pages->normal_num.
+  *
+  * @param p A pointer to the send params.
+  */
+-void multifd_send_zero_page_detect(MultiFDSendParams *p)
++static void zero_page_detect_dsa(MultiFDSendParams *p)
+ {
+     MultiFDPages_t *pages = &p->data->u.ram;
+     RAMBlock *rb = pages->block;
+-    int i = 0;
+-    int j = pages->num - 1;
++    bool *results = p->dsa_batch_task->results;
+-    if (!multifd_zero_page_enabled()) {
+-        pages->normal_num = pages->num;
+-        goto out;
++    for (int i = 0; i < pages->num; i++) {
++        p->dsa_batch_task->addr[i] =
++            (ram_addr_t)(rb->host + pages->offset[i]);
+     }
++    buffer_is_zero_dsa_batch_sync(p->dsa_batch_task,
++                                  (const void **)p->dsa_batch_task->addr,
++                                  pages->num,
++                                  multifd_ram_page_size());
++
++    int i = 0;
++    int j = pages->num - 1;
++
+     /*
+      * Sort the page offset array by moving all normal pages to
+      * the left and all zero pages to the right of the array.
+@@ -XXX,XX +XXX,XX @@ void multifd_send_zero_page_detect(MultiFDSendParams *p)
+     while (i <= j) {
+         uint64_t offset = pages->offset[i];
+-        if (!buffer_is_zero(rb->host + offset, multifd_ram_page_size())) {
++        if (!results[i]) {
+             i++;
+             continue;
+         }
++        swap_result(results, i, j);
+         swap_page_offset(pages->offset, i, j);
+         ram_release_page(rb->idstr, offset);
+         j--;
+     }
+     pages->normal_num = i;
++}
+-out:
+-    stat64_add(&mig_stats.normal_pages, pages->normal_num);
+-    stat64_add(&mig_stats.zero_pages, pages->num - pages->normal_num);
++int multifd_dsa_setup(MigrationState *s, Error *local_err)
++{
++    g_autofree strList *dsa_parameter = g_malloc0(sizeof(strList));
++    migrate_dsa_accel_path(&dsa_parameter);
++    if (qemu_dsa_init(dsa_parameter, &local_err)) {
++        migrate_set_error(s, local_err);
++        return -1;
++    } else {
++        qemu_dsa_start();
++    }
++
++    return 0;
++}
++
++void multifd_dsa_cleanup(void)
++{
++    qemu_dsa_cleanup();
++}
++
++#else
++
++static void zero_page_detect_dsa(MultiFDSendParams *p)
++{
++    g_assert_not_reached();
+ }
++int multifd_dsa_setup(MigrationState *s, Error *local_err)
++{
++    g_assert_not_reached();
++    return -1;
++}
++
++void multifd_dsa_cleanup(void)
++{
++    return ;
++}
++
++#endif
++
+ void multifd_recv_zero_page_process(MultiFDRecvParams *p)
+ {
+     for (int i = 0; i < p->zero_num; i++) {
+@@ -XXX,XX +XXX,XX @@ void multifd_recv_zero_page_process(MultiFDRecvParams *p)
+         }
+     }
+ }
++
++/**
++ * zero_page_detect_cpu: Perform zero page detection using CPU.
++ *
++ * Sorts normal pages before zero pages in p->pages->offset and updates
++ * p->pages->normal_num.
++ *
++ * @param p A pointer to the send params.
++ */
++static void zero_page_detect_cpu(MultiFDSendParams *p)
++{
++    MultiFDPages_t *pages = &p->data->u.ram;
++    RAMBlock *rb = pages->block;
++    int i = 0;
++    int j = pages->num - 1;
++
++    /*
++     * Sort the page offset array by moving all normal pages to
++     * the left and all zero pages to the right of the array.
++     */
++    while (i <= j) {
++        uint64_t offset = pages->offset[i];
++
++        if (!buffer_is_zero(rb->host + offset, multifd_ram_page_size())) {
++            i++;
++            continue;
++        }
++
++        swap_page_offset(pages->offset, i, j);
++        ram_release_page(rb->idstr, offset);
++        j--;
++    }
++
++    pages->normal_num = i;
++}
++
++/**
++ * multifd_send_zero_page_detect: Perform zero page detection on all pages.
++ *
++ * @param p A pointer to the send params.
++ */
++void multifd_send_zero_page_detect(MultiFDSendParams *p)
++{
++    MultiFDPages_t *pages = &p->data->u.ram;
++
++    if (!multifd_zero_page_enabled()) {
++        pages->normal_num = pages->num;
++        goto out;
++    }
++
++    if (qemu_dsa_is_running()) {
++        zero_page_detect_dsa(p);
++    } else {
++        zero_page_detect_cpu(p);
++    }
++
++out:
++    stat64_add(&mig_stats.normal_pages, pages->normal_num);
++    stat64_add(&mig_stats.zero_pages, pages->num - pages->normal_num);
++}
 diff --git a/migration/multifd.c b/migration/multifd.c
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/multifd.c
 +++ b/migration/multifd.c
-@@ -XXX,XX +XXX,XX @@ void multifd_save_cleanup(void)
+@@ -XXX,XX +XXX,XX @@
-             qemu_thread_join(&p->thread);
+ #include "qemu/osdep.h"
-         }
+ #include "qemu/cutils.h"
-     }
+ #include "qemu/rcu.h"
-+    dsa_stop();
++#include "qemu/dsa.h"
-+    dsa_cleanup();
+ #include "exec/target_page.h"
  #include "system/system.h"
  #include "exec/ramblock.h"
@@ -XXX,XX +XXX,XX @@ static bool multifd_send_cleanup_channel(MultiFDSendParams *p, Error **errp)
      p->name = NULL;
      g_free(p->data);
      p->data = NULL;
 +    buffer_zero_batch_task_destroy(p->dsa_batch_task);
 +    p->dsa_batch_task = NULL;
      p->packet_len = 0;
      g_free(p->packet);
      p->packet = NULL;
@@ -XXX,XX +XXX,XX @@ void multifd_send_shutdown(void)
      multifd_send_terminate_threads();
 +    multifd_dsa_cleanup();
 +
      for (i = 0; i < migrate_multifd_channels(); i++) {
          MultiFDSendParams *p = &multifd_send_state->params[i];
          Error *local_err = NULL;
-@@ -XXX,XX +XXX,XX @@ static void buffer_is_zero_use_cpu(MultiFDSendParams *p)
+@@ -XXX,XX +XXX,XX @@ bool multifd_send_setup(void)
- {
+     uint32_t page_count = multifd_ram_page_count();
-     const void **buf = (const void **)p->addr;
+     bool use_packets = multifd_use_packets();
      assert(!migrate_use_main_zero_page());
 +    assert(!dsa_is_running());
      for (int i = 0; i < p->pages->num; i++) {
          p->batch_task->results[i] = buffer_is_zero(buf[i], p->page_size);
@@ -XXX,XX +XXX,XX @@ static void buffer_is_zero_use_cpu(MultiFDSendParams *p)
  static void set_normal_pages(MultiFDSendParams *p)
  {
 +    assert(migrate_use_main_zero_page());
 +
      for (int i = 0; i < p->pages->num; i++) {
          p->batch_task->results[i] = false;
      }
  }
 +static void buffer_is_zero_use_dsa(MultiFDSendParams *p)
 +{
 +    assert(!migrate_use_main_zero_page());
 +    assert(dsa_is_running());
 +
 +    buffer_is_zero_dsa_batch_async(p->batch_task,
 +                                   (const void **)p->addr,
 +                                   p->pages->num,
 +                                   p->page_size);
 +}
 +
  static void multifd_zero_page_check(MultiFDSendParams *p)
  {
      /* older qemu don't understand zero page on multifd channel */
      bool use_multifd_zero_page = !migrate_use_main_zero_page();
 +    bool use_multifd_dsa_accel = dsa_is_running();
      RAMBlock *rb = p->pages->block;
@@ -XXX,XX +XXX,XX @@ static void multifd_zero_page_check(MultiFDSendParams *p)
          p->addr[i] = (ram_addr_t)(rb->host + p->pages->offset[i]);
      }
 -    if (use_multifd_zero_page) {
 +    if (use_multifd_dsa_accel && use_multifd_zero_page) {
 +        buffer_is_zero_use_dsa(p);
 +    } else if (use_multifd_zero_page) {
          buffer_is_zero_use_cpu(p);
      } else {
          // No zero page checking. All pages are normal pages.
@@ -XXX,XX +XXX,XX @@ int multifd_save_setup(Error **errp)
      int thread_count;
      uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
      uint8_t i;
-+    const char *dsa_parameter = migrate_multifd_dsa_accel();
++    Error *local_err = NULL;
      if (!migrate_multifd()) {
-         return 0;
+         return true;
-     }
+@@ -XXX,XX +XXX,XX @@ bool multifd_send_setup(void)
+     qatomic_set(&multifd_send_state->exiting, 0);
-+    dsa_init(dsa_parameter);
+     multifd_send_state->ops = multifd_ops[migrate_multifd_compression()];
-+    dsa_start();
-+
++    if (ret) {
-     thread_count = migrate_multifd_channels();
++        goto err;
-     multifd_send_state = g_malloc0(sizeof(*multifd_send_state));
++    }
-     multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
++
-@@ -XXX,XX +XXX,XX @@ int multifd_save_setup(Error **errp)
+     for (i = 0; i < thread_count; i++) {
-             return ret;
+         MultiFDSendParams *p = &multifd_send_state->params[i];
 -        Error *local_err = NULL;
          qemu_sem_init(&p->sem, 0);
          qemu_sem_init(&p->sem_sync, 0);
@@ -XXX,XX +XXX,XX @@ bool multifd_send_setup(void)
          goto err;
      }
 +    if (s && ret == 0 &&
 +        s->parameters.zero_page_detection == ZERO_PAGE_DETECTION_DSA_ACCEL) {
 +        ret = multifd_dsa_setup(s, local_err);
 +    }
 +
 +    if (ret) {
 +        goto err;
 +    }
 +
      for (i = 0; i < thread_count; i++) {
          MultiFDSendParams *p = &multifd_send_state->params[i];
 -        Error *local_err = NULL;
 +        p->dsa_batch_task = buffer_zero_batch_task_init(page_count);
          ret = multifd_send_state->ops->send_setup(p, &local_err);
          if (ret) {
              migrate_set_error(s, local_err);
@@ -XXX,XX +XXX,XX @@ void multifd_recv_cleanup(void)
              qemu_thread_join(&p->thread);
          }
      }
-+
++    multifd_dsa_cleanup();
      return 0;
  }
@@ -XXX,XX +XXX,XX @@ void multifd_load_cleanup(void)
          qemu_thread_join(&p->thread);
      }
 +    dsa_stop();
 +    dsa_cleanup();
      for (i = 0; i < migrate_multifd_channels(); i++) {
-         MultiFDRecvParams *p = &multifd_recv_state->params[i];
+         multifd_recv_cleanup_channel(&multifd_recv_state->params[i]);
+     }
-@@ -XXX,XX +XXX,XX @@ int multifd_load_setup(Error **errp)
+diff --git a/migration/multifd.h b/migration/multifd.h
-     int thread_count;
+index XXXXXXX..XXXXXXX 100644
-     uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
+--- a/migration/multifd.h
-     uint8_t i;
++++ b/migration/multifd.h
-+    const char *dsa_parameter = migrate_multifd_dsa_accel();
+@@ -XXX,XX +XXX,XX @@
-     /*
+ #include "exec/target_page.h"
-      * Return successfully if multiFD recv state is already initialised
+ #include "ram.h"
-@@ -XXX,XX +XXX,XX @@ int multifd_load_setup(Error **errp)
++#include "qemu/dsa.h"
-         return 0;
-     }
+ typedef struct MultiFDRecvData MultiFDRecvData;
+ typedef struct MultiFDSendData MultiFDSendData;
-+    dsa_init(dsa_parameter);
+@@ -XXX,XX +XXX,XX @@ typedef struct {
-+    dsa_start();
+     bool pending_sync;
-+
+     MultiFDSendData *data;
-     thread_count = migrate_multifd_channels();
-     multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state));
++    /* Zero page checking batch task */
-     multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count);
++    QemuDsaBatchTask *dsa_batch_task;
-@@ -XXX,XX +XXX,XX @@ int multifd_load_setup(Error **errp)
++
-             return ret;
+     /* thread local variables. No locking required */
-         }
-     }
+     /* pointer to the packet */
-+
+@@ -XXX,XX +XXX,XX @@ void multifd_send_fill_packet(MultiFDSendParams *p);
-     return 0;
+ bool multifd_send_prepare_common(MultiFDSendParams *p);
- }
+ void multifd_send_zero_page_detect(MultiFDSendParams *p);
  void multifd_recv_zero_page_process(MultiFDRecvParams *p);
 +int multifd_dsa_setup(MigrationState *s, Error *local_err);
 +void multifd_dsa_cleanup(void);
  static inline void multifd_send_prepare_header(MultiFDSendParams *p)
  {
 diff --git a/migration/options.c b/migration/options.c
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/options.c
 +++ b/migration/options.c
@@ -XXX,XX +XXX,XX @@ const strList *migrate_accel_path(void)
      return s->parameters.accel_path;
  }
 +void migrate_dsa_accel_path(strList **dsa_accel_path)
 +{
 +    MigrationState *s = migrate_get_current();
 +    strList *accel_path = s->parameters.accel_path;
 +    strList **tail = dsa_accel_path;
 +    while (accel_path) {
 +        if (strncmp(accel_path->value, "dsa:", 4) == 0) {
 +            QAPI_LIST_APPEND(tail, &accel_path->value[4]);
 +        }
 +        accel_path = accel_path->next;
 +    }
 +}
 +
  const char *migrate_tls_hostname(void)
  {
      MigrationState *s = migrate_get_current();
 diff --git a/migration/options.h b/migration/options.h
 index XXXXXXX..XXXXXXX 100644
 --- a/migration/options.h
 +++ b/migration/options.h
@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_hostname(void);
  uint64_t migrate_xbzrle_cache_size(void);
  ZeroPageDetection migrate_zero_page_detection(void);
  const strList *migrate_accel_path(void);
 +void migrate_dsa_accel_path(strList **dsa_accel_path);
  /* parameters helpers */
 --
-.30.2
+Yichen Wang

-[PATCH v2 19/20] util/dsa: Add unit test coverage for Intel DSA task submission and completion.
+[PATCH v9 10/12] util/dsa: Add unit test coverage for Intel DSA task submission and completion.
+From: Hao Xiang <hao.xiang@linux.dev>
 * Test DSA start and stop path.
 * Test DSA configure and cleanup path.
 * Test DSA task submission and completion path.
 Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
 Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
 ---
  tests/unit/meson.build |   6 +
- tests/unit/test-dsa.c  | 466 +++++++++++++++++++++++++++++++++++++++++
+ tests/unit/test-dsa.c  | 504 +++++++++++++++++++++++++++++++++++++++++
-files changed, 472 insertions(+)
+files changed, 510 insertions(+)
  create mode 100644 tests/unit/test-dsa.c
 diff --git a/tests/unit/meson.build b/tests/unit/meson.build
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/unit/meson.build
 +++ b/tests/unit/meson.build
 @@ -XXX,XX +XXX,XX @@ tests = {
-   'test-virtio-dmabuf': [meson.project_source_root() / 'hw/display/virtio-dmabuf.c'],
+   'test-fifo': [],
  }
 +if config_host_data.get('CONFIG_DSA_OPT')
 +  tests += {
 +    'test-dsa': [],
 ...
 +++ b/tests/unit/test-dsa.c
 @@ -XXX,XX +XXX,XX @@
 +/*
 + * Test DSA functions.
 + *
-+ * Copyright (c) 2023 Hao Xiang <hao.xiang@bytedance.com>
++ * Copyright (C) Bytedance Ltd.
 + * Copyright (c) 2023 Bryan Zhang <bryan.zhang@bytedance.com>
 + *
-+ * This library is free software; you can redistribute it and/or
++ * Authors:
-+ * modify it under the terms of the GNU Lesser General Public
++ *  Hao Xiang <hao.xiang@bytedance.com>
-+ * License as published by the Free Software Foundation; either
++ *  Bryan Zhang <bryan.zhang@bytedance.com>
-+ * version 2.1 of the License, or (at your option) any later version.
++ *  Yichen Wang <yichen.wang@bytedance.com>
 + *
-+ * This library is distributed in the hope that it will be useful,
++ * This work is licensed under the terms of the GNU GPL, version 2 or later.
-+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
++ * See the COPYING file in the top-level directory.
 + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 + * Lesser General Public License for more details.
 + *
 + * You should have received a copy of the GNU Lesser General Public
 + * License along with this library; if not, see <http://www.gnu.org/licenses/>.
 + */
++
 +#include "qemu/osdep.h"
 +#include "qemu/host-utils.h"
 +
 +#include "qemu/cutils.h"
 +#include "qemu/memalign.h"
 +#include "qemu/dsa.h"
 +
-+// TODO Make these not-hardcoded.
++/*
-+static const char *path1 = "/dev/dsa/wq4.0";
++ * TODO Communicate that DSA must be configured to support this batch size.
-+static const char *path2 = "/dev/dsa/wq4.0 /dev/dsa/wq4.1";
++ * TODO Alternatively, poke the DSA device to figure out batch size.
-+static const int num_devices = 2;
++ */
-+
++#define batch_size 128
-+static struct buffer_zero_batch_task batch_task __attribute__((aligned(64)));
++#define page_size 4096
 +
-+// TODO Communicate that DSA must be configured to support this batch size.
++#define oversized_batch_size (batch_size + 1)
-+// TODO Alternatively, poke the DSA device to figure out batch size.
++#define num_devices 2
-+static int batch_size = 128;
++#define max_buffer_size (64 * 1024)
-+static int page_size = 4096;
++
-+
++/* TODO Make these not-hardcoded. */
-+// A helper for running a single task and checking for correctness.
++static const strList path1[] = {
 +    {.value = (char *)"/dev/dsa/wq4.0", .next = NULL}
 +};
 +static const strList path2[] = {
 +    {.value = (char *)"/dev/dsa/wq4.0", .next = (strList*)&path2[1]},
 +    {.value = (char *)"/dev/dsa/wq4.1", .next = NULL}
 +};
 +
 +static Error **errp;
 +
 +static QemuDsaBatchTask *task;
 +
 +/* A helper for running a single task and checking for correctness. */
 +static void do_single_task(void)
 +{
-+    buffer_zero_batch_task_init(&batch_task, batch_size);
++    task = buffer_zero_batch_task_init(batch_size);
 +    char buf[page_size];
-+    char* ptr = buf;
++    char *ptr = buf;
 +
-+    buffer_is_zero_dsa_batch_async(&batch_task,
++    buffer_is_zero_dsa_batch_sync(task,
-+                                   (const void**) &ptr,
++                                  (const void **)&ptr,
-+                                   1,
++                                  1,
-+                                   page_size);
++                                  page_size);
-+    g_assert(batch_task.results[0] == buffer_is_zero(buf, page_size));
++    g_assert(task->results[0] == buffer_is_zero(buf, page_size));
 +
 +    buffer_zero_batch_task_destroy(task);
 +}
 +
 +static void test_single_zero(void)
 +{
-+    g_assert(!dsa_init(path1));
++    g_assert(!qemu_dsa_init(path1, errp));
-+    dsa_start();
++    qemu_dsa_start();
 +
-+    buffer_zero_batch_task_init(&batch_task, batch_size);
++    task = buffer_zero_batch_task_init(batch_size);
 +
 +    char buf[page_size];
-+    char* ptr = buf;
++    char *ptr = buf;
 +
 +    memset(buf, 0x0, page_size);
-+    buffer_is_zero_dsa_batch_async(&batch_task,
++    buffer_is_zero_dsa_batch_sync(task,
-+                                   (const void**) &ptr,
++                                  (const void **)&ptr,
-+                                   1, page_size);
++                                  1, page_size);
-+    g_assert(batch_task.results[0]);
++    g_assert(task->results[0]);
 +
-+    dsa_cleanup();
++    buffer_zero_batch_task_destroy(task);
 +
 +    qemu_dsa_cleanup();
 +}
 +
 +static void test_single_zero_async(void)
 +{
 +    test_single_zero();
 +}
 +
 +static void test_single_nonzero(void)
 +{
-+    g_assert(!dsa_init(path1));
++    g_assert(!qemu_dsa_init(path1, errp));
-+    dsa_start();
++    qemu_dsa_start();
 +
-+    buffer_zero_batch_task_init(&batch_task, batch_size);
++    task = buffer_zero_batch_task_init(batch_size);
 +
 +    char buf[page_size];
-+    char* ptr = buf;
++    char *ptr = buf;
 +
 +    memset(buf, 0x1, page_size);
-+    buffer_is_zero_dsa_batch_async(&batch_task,
++    buffer_is_zero_dsa_batch_sync(task,
-+                                   (const void**) &ptr,
++                                  (const void **)&ptr,
-+                                   1, page_size);
++                                  1, page_size);
-+    g_assert(!batch_task.results[0]);
++    g_assert(!task->results[0]);
 +
-+    dsa_cleanup();
++    buffer_zero_batch_task_destroy(task);
 +
 +    qemu_dsa_cleanup();
 +}
 +
 +static void test_single_nonzero_async(void)
 +{
 +    test_single_nonzero();
 +}
 +
-+// count == 0 should return quickly without calling into DSA.
++/* count == 0 should return quickly without calling into DSA. */
 +static void test_zero_count_async(void)
 +{
 +    char buf[page_size];
-+    buffer_is_zero_dsa_batch_async(&batch_task,
++    buffer_is_zero_dsa_batch_sync(task,
-+                             (const void **) &buf,
++                                  (const void **)&buf,
-+                             0,
++                                  0,
-+                             page_size);
++                                  page_size);
 +}
 +
 +static void test_null_task_async(void)
 +{
 +    if (g_test_subprocess()) {
-+        g_assert(!dsa_init(path1));
++        g_assert(!qemu_dsa_init(path1, errp));
 +
 +        char buf[page_size * batch_size];
 +        char *addrs[batch_size];
 +        for (int i = 0; i < batch_size; i++) {
 +            addrs[i] = buf + (page_size * i);
 +        }
 +
-+        buffer_is_zero_dsa_batch_async(NULL, (const void**) addrs, batch_size,
++        buffer_is_zero_dsa_batch_sync(NULL, (const void **)addrs,
-+                                 page_size);
++                                      batch_size,
 +                                      page_size);
 +    } else {
 +        g_test_trap_subprocess(NULL, 0, 0);
 +        g_test_trap_assert_failed();
 +    }
 +}
 +
 +static void test_oversized_batch(void)
 +{
-+    g_assert(!dsa_init(path1));
++    g_assert(!qemu_dsa_init(path1, errp));
-+    dsa_start();
++    qemu_dsa_start();
 +
-+    buffer_zero_batch_task_init(&batch_task, batch_size);
++    task = buffer_zero_batch_task_init(batch_size);
 +
 +    int oversized_batch_size = batch_size + 1;
 +    char buf[page_size * oversized_batch_size];
 +    char *addrs[batch_size];
 +    for (int i = 0; i < oversized_batch_size; i++) {
 +        addrs[i] = buf + (page_size * i);
 +    }
 +
-+    int ret = buffer_is_zero_dsa_batch_async(&batch_task,
++    int ret = buffer_is_zero_dsa_batch_sync(task,
-+                                            (const void**) addrs,
++                                            (const void **)addrs,
 +                                            oversized_batch_size,
 +                                            page_size);
 +    g_assert(ret != 0);
 +
-+    dsa_cleanup();
++    buffer_zero_batch_task_destroy(task);
 +
 +    qemu_dsa_cleanup();
 +}
 +
 +static void test_oversized_batch_async(void)
 +{
 +    test_oversized_batch();
 +}
 +
 +static void test_zero_len_async(void)
 +{
 +    if (g_test_subprocess()) {
-+        g_assert(!dsa_init(path1));
++        g_assert(!qemu_dsa_init(path1, errp));
 +
-+        buffer_zero_batch_task_init(&batch_task, batch_size);
++        task = buffer_zero_batch_task_init(batch_size);
 +
 +        char buf[page_size];
 +
-+        buffer_is_zero_dsa_batch_async(&batch_task,
++        buffer_is_zero_dsa_batch_sync(task,
-+                                       (const void**) &buf,
++                                      (const void **)&buf,
-+                                       1,
++                                      1,
-+                                       0);
++                                      0);
 +
 +        buffer_zero_batch_task_destroy(task);
 +    } else {
 +        g_test_trap_subprocess(NULL, 0, 0);
 +        g_test_trap_assert_failed();
 +    }
 +}
 +
 +static void test_null_buf_async(void)
 +{
 +    if (g_test_subprocess()) {
-+        g_assert(!dsa_init(path1));
++        g_assert(!qemu_dsa_init(path1, errp));
 +
-+        buffer_zero_batch_task_init(&batch_task, batch_size);
++        task = buffer_zero_batch_task_init(batch_size);
 +
-+        buffer_is_zero_dsa_batch_async(&batch_task, NULL, 1, page_size);
++        buffer_is_zero_dsa_batch_sync(task, NULL, 1, page_size);
 +
 +        buffer_zero_batch_task_destroy(task);
 +    } else {
 +        g_test_trap_subprocess(NULL, 0, 0);
 +        g_test_trap_assert_failed();
 +    }
 +}
 +
 +static void test_batch(void)
 +{
-+    g_assert(!dsa_init(path1));
++    g_assert(!qemu_dsa_init(path1, errp));
-+    dsa_start();
++    qemu_dsa_start();
 +
-+    buffer_zero_batch_task_init(&batch_task, batch_size);
++    task = buffer_zero_batch_task_init(batch_size);
 +
 +    char buf[page_size * batch_size];
 +    char *addrs[batch_size];
 +    for (int i = 0; i < batch_size; i++) {
 +        addrs[i] = buf + (page_size * i);
 +    }
 +
-+    // Using whatever is on the stack is somewhat random.
++    /*
-+    // Manually set some pages to zero and some to nonzero.
++     * Using whatever is on the stack is somewhat random.
 +     * Manually set some pages to zero and some to nonzero.
 +     */
 +    memset(buf + 0, 0, page_size * 10);
 +    memset(buf + (10 * page_size), 0xff, page_size * 10);
 +
-+    buffer_is_zero_dsa_batch_async(&batch_task,
++    buffer_is_zero_dsa_batch_sync(task,
-+                                   (const void**) addrs,
++                                  (const void **)addrs,
-+                                   batch_size,
++                                  batch_size,
-+                                   page_size);
++                                  page_size);
 +
 +    bool is_zero;
 +    for (int i = 0; i < batch_size; i++) {
-+        is_zero = buffer_is_zero((const void*) &buf[page_size * i], page_size);
++        is_zero = buffer_is_zero((const void *)&buf[page_size * i], page_size);
-+        g_assert(batch_task.results[i] == is_zero);
++        g_assert(task->results[i] == is_zero);
 +    }
-+    dsa_cleanup();
++
 +    buffer_zero_batch_task_destroy(task);
 +
 +    qemu_dsa_cleanup();
 +}
 +
 +static void test_batch_async(void)
 +{
 +    test_batch();
 +}
 +
 +static void test_page_fault(void)
 +{
-+    g_assert(!dsa_init(path1));
++    g_assert(!qemu_dsa_init(path1, errp));
-+    dsa_start();
++    qemu_dsa_start();
 +
-+    char* buf[2];
++    char *buf[2];
 +    int prot = PROT_READ | PROT_WRITE;
 +    int flags = MAP_SHARED | MAP_ANON;
-+    buf[0] = (char*) mmap(NULL, page_size * batch_size, prot, flags, -1, 0);
++    buf[0] = (char *)mmap(NULL, page_size * batch_size, prot, flags, -1, 0);
 +    assert(buf[0] != MAP_FAILED);
-+    buf[1] = (char*) malloc(page_size * batch_size);
++    buf[1] = (char *)malloc(page_size * batch_size);
 +    assert(buf[1] != NULL);
 +
 +    for (int j = 0; j < 2; j++) {
-+        buffer_zero_batch_task_init(&batch_task, batch_size);
++        task = buffer_zero_batch_task_init(batch_size);
 +
 +        char *addrs[batch_size];
 +        for (int i = 0; i < batch_size; i++) {
 +            addrs[i] = buf[j] + (page_size * i);
 +        }
 +
-+        buffer_is_zero_dsa_batch_async(&batch_task,
++        buffer_is_zero_dsa_batch_sync(task,
-+                                       (const void**) addrs,
++                                      (const void **)addrs,
-+                                       batch_size,
++                                      batch_size,
-+                                       page_size);
++                                      page_size);
 +
 +        bool is_zero;
 +        for (int i = 0; i < batch_size; i++) {
-+            is_zero = buffer_is_zero((const void*) &buf[j][page_size * i], page_size);
++            is_zero = buffer_is_zero((const void *)&buf[j][page_size * i],
-+            g_assert(batch_task.results[i] == is_zero);
++                                      page_size);
-+        }
++            g_assert(task->results[i] == is_zero);
 +        }
 +        buffer_zero_batch_task_destroy(task);
 +    }
 +
 +    assert(!munmap(buf[0], page_size * batch_size));
 +    free(buf[1]);
-+    dsa_cleanup();
++    qemu_dsa_cleanup();
 +}
 +
 +static void test_various_buffer_sizes(void)
 +{
-+    g_assert(!dsa_init(path1));
++    g_assert(!qemu_dsa_init(path1, errp));
-+    dsa_start();
++    qemu_dsa_start();
 +
-+    int len = 1 << 4;
++    char *buf = malloc(max_buffer_size * batch_size);
-+    for (int count = 12; count > 0; count--, len <<= 1) {
++    char *addrs[batch_size];
-+        buffer_zero_batch_task_init(&batch_task, batch_size);
++
-+
++    for (int len = 16; len <= max_buffer_size; len *= 2) {
-+        char buf[len * batch_size];
++        task = buffer_zero_batch_task_init(batch_size);
-+        char *addrs[batch_size];
++
 +        for (int i = 0; i < batch_size; i++) {
 +            addrs[i] = buf + (len * i);
 +        }
 +
-+        buffer_is_zero_dsa_batch_async(&batch_task,
++        buffer_is_zero_dsa_batch_sync(task,
-+                                       (const void**) addrs,
++                                      (const void **)addrs,
-+                                       batch_size,
++                                      batch_size,
-+                                       len);
++                                      len);
 +
 +        bool is_zero;
 +        for (int j = 0; j < batch_size; j++) {
-+            is_zero = buffer_is_zero((const void*) &buf[len * j], len);
++            is_zero = buffer_is_zero((const void *)&buf[len * j], len);
-+            g_assert(batch_task.results[j] == is_zero);
++            g_assert(task->results[j] == is_zero);
 +        }
-+    }
++
-+
++        buffer_zero_batch_task_destroy(task);
-+    dsa_cleanup();
++    }
 +
 +    free(buf);
 +
 +    qemu_dsa_cleanup();
 +}
 +
 +static void test_various_buffer_sizes_async(void)
 +{
 +    test_various_buffer_sizes();
 +}
 +
 +static void test_double_start_stop(void)
 +{
-+    g_assert(!dsa_init(path1));
++    g_assert(!qemu_dsa_init(path1, errp));
-+    // Double start
++    /* Double start */
-+    dsa_start();
++    qemu_dsa_start();
-+    dsa_start();
++    qemu_dsa_start();
-+    g_assert(dsa_is_running());
++    g_assert(qemu_dsa_is_running());
 +    do_single_task();
 +
-+    // Double stop
++    /* Double stop */
-+    dsa_stop();
++    qemu_dsa_stop();
-+    g_assert(!dsa_is_running());
++    g_assert(!qemu_dsa_is_running());
-+    dsa_stop();
++    qemu_dsa_stop();
-+    g_assert(!dsa_is_running());
++    g_assert(!qemu_dsa_is_running());
 +
-+    // Restart
++    /* Restart */
-+    dsa_start();
++    qemu_dsa_start();
-+    g_assert(dsa_is_running());
++    g_assert(qemu_dsa_is_running());
 +    do_single_task();
-+    dsa_cleanup();
++    qemu_dsa_cleanup();
 +}
 +
 +static void test_is_running(void)
 +{
-+    g_assert(!dsa_init(path1));
++    g_assert(!qemu_dsa_init(path1, errp));
 +
-+    g_assert(!dsa_is_running());
++    g_assert(!qemu_dsa_is_running());
-+    dsa_start();
++    qemu_dsa_start();
-+    g_assert(dsa_is_running());
++    g_assert(qemu_dsa_is_running());
-+    dsa_stop();
++    qemu_dsa_stop();
-+    g_assert(!dsa_is_running());
++    g_assert(!qemu_dsa_is_running());
-+    dsa_cleanup();
++    qemu_dsa_cleanup();
 +}
 +
 +static void test_multiple_engines(void)
 +{
-+    g_assert(!dsa_init(path2));
++    g_assert(!qemu_dsa_init(path2, errp));
-+    dsa_start();
++    qemu_dsa_start();
 +
-+    struct buffer_zero_batch_task tasks[num_devices]
++    QemuDsaBatchTask *tasks[num_devices];
 +        __attribute__((aligned(64)));
 +    char bufs[num_devices][page_size * batch_size];
 +    char *addrs[num_devices][batch_size];
 +
-+    // This is a somewhat implementation-specific way of testing that the tasks
++    /*
-+    // have unique engines assigned to them.
++     *  This is a somewhat implementation-specific way
-+    buffer_zero_batch_task_init(&tasks[0], batch_size);
++     *  of testing that the tasks have unique engines
-+    buffer_zero_batch_task_init(&tasks[1], batch_size);
++     *  assigned to them.
-+    g_assert(tasks[0].device != tasks[1].device);
++     */
 +    tasks[0] = buffer_zero_batch_task_init(batch_size);
 +    tasks[1] = buffer_zero_batch_task_init(batch_size);
 +    g_assert(tasks[0]->device != tasks[1]->device);
 +
 +    for (int i = 0; i < num_devices; i++) {
 +        for (int j = 0; j < batch_size; j++) {
 +            addrs[i][j] = bufs[i] + (page_size * j);
 +        }
 +
-+        buffer_is_zero_dsa_batch_async(&tasks[i],
++        buffer_is_zero_dsa_batch_sync(tasks[i],
-+                                       (const void**) addrs[i],
++                                      (const void **)addrs[i],
-+                                       batch_size, page_size);
++                                      batch_size, page_size);
 +
 +        bool is_zero;
 +        for (int j = 0; j < batch_size; j++) {
-+            is_zero = buffer_is_zero((const void*) &bufs[i][page_size * j],
++            is_zero = buffer_is_zero((const void *)&bufs[i][page_size * j],
 +                                     page_size);
-+            g_assert(tasks[i].results[j] == is_zero);
++            g_assert(tasks[i]->results[j] == is_zero);
 +        }
 +    }
 +
-+    dsa_cleanup();
++    buffer_zero_batch_task_destroy(tasks[0]);
 +    buffer_zero_batch_task_destroy(tasks[1]);
 +
 +    qemu_dsa_cleanup();
 +}
 +
 +static void test_configure_dsa_twice(void)
 +{
-+    g_assert(!dsa_init(path2));
++    g_assert(!qemu_dsa_init(path2, errp));
-+    g_assert(!dsa_init(path2));
++    g_assert(!qemu_dsa_init(path2, errp));
-+    dsa_start();
++    qemu_dsa_start();
 +    do_single_task();
-+    dsa_cleanup();
++    qemu_dsa_cleanup();
 +}
 +
 +static void test_configure_dsa_bad_path(void)
 +{
-+    const char* bad_path = "/not/a/real/path";
++    const strList *bad_path = &(strList) {
-+    g_assert(dsa_init(bad_path));
++        .value = (char *)"/not/a/real/path", .next = NULL
 +    };
 +    g_assert(qemu_dsa_init(bad_path, errp));
 +}
 +
 +static void test_cleanup_before_configure(void)
 +{
-+    dsa_cleanup();
++    qemu_dsa_cleanup();
-+    g_assert(!dsa_init(path2));
++    g_assert(!qemu_dsa_init(path2, errp));
 +}
 +
 +static void test_configure_dsa_num_devices(void)
 +{
-+    g_assert(!dsa_init(path1));
++    g_assert(!qemu_dsa_init(path1, errp));
-+    dsa_start();
++    qemu_dsa_start();
 +
 +    do_single_task();
-+    dsa_stop();
++    qemu_dsa_stop();
-+    dsa_cleanup();
++    qemu_dsa_cleanup();
 +}
 +
 +static void test_cleanup_twice(void)
 +{
-+    g_assert(!dsa_init(path2));
++    g_assert(!qemu_dsa_init(path2, errp));
-+    dsa_cleanup();
++    qemu_dsa_cleanup();
-+    dsa_cleanup();
++    qemu_dsa_cleanup();
 +
-+    g_assert(!dsa_init(path2));
++    g_assert(!qemu_dsa_init(path2, errp));
-+    dsa_start();
++    qemu_dsa_start();
 +    do_single_task();
-+    dsa_cleanup();
++    qemu_dsa_cleanup();
 +}
 +
 +static int check_test_setup(void)
 +{
-+    const char *path[2] = {path1, path2};
++    const strList *path[2] = {path1, path2};
-+    for (int i = 0; i < sizeof(path) / sizeof(char *); i++) {
++    for (int i = 0; i < sizeof(path) / sizeof(strList *); i++) {
-+        if (!dsa_init(path[i])) {
++        if (qemu_dsa_init(path[i], errp)) {
 +            return -1;
 +        }
-+        dsa_cleanup();
++        qemu_dsa_cleanup();
 +    }
 +    return 0;
 +}
 +
 +int main(int argc, char **argv)
 ...
 +        /*
 +         * This test requires extra setup. The current
 +         * setup is not correct. Just skip this test
 +         * for now.
 +         */
-+        exit(0);
++        g_test_skip("DSA hardware is not configured properly.");
 +        return g_test_run();
 +    }
 +
 +    if (num_devices > 1) {
 +        g_test_add_func("/dsa/multiple_engines", test_multiple_engines);
 +    }
 ...
 +    g_test_add_func("/dsa/cleanup_twice", test_cleanup_twice);
 +
 +    return g_test_run();
 +}
 --
-.30.2
+Yichen Wang

-[PATCH v2 20/20] migration/multifd: Add integration tests for multifd with Intel DSA offloading.
+[PATCH v9 11/12] migration/multifd: Add integration tests for multifd with Intel DSA offloading.
+From: Hao Xiang <hao.xiang@linux.dev>
 * Add test case to start and complete multifd live migration with DSA
 offloading enabled.
 * Add test case to start and cancel multifd live migration with DSA
 offloading enabled.
 Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
 Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
 Reviewed-by: Fabiano Rosas <farosas@suse.de>
 ---
- tests/qtest/migration-test.c | 77 +++++++++++++++++++++++++++++++++++-
+ tests/qtest/meson.build           | 10 +++++-
-file changed, 76 insertions(+), 1 deletion(-)
+ tests/qtest/migration-test.c      |  3 ++
  tests/qtest/migration/dsa-tests.c | 59 +++++++++++++++++++++++++++++++
  tests/qtest/migration/framework.h |  1 +
 files changed, 72 insertions(+), 1 deletion(-)
  create mode 100644 tests/qtest/migration/dsa-tests.c
+diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
+index XXXXXXX..XXXXXXX 100644
+--- a/tests/qtest/meson.build
++++ b/tests/qtest/meson.build
+@@ -XXX,XX +XXX,XX @@ if gnutls.found()
+   endif
+ endif
++migration_dsa_files = []
++if config_host_data.get('CONFIG_DSA_OPT')
++  migration_dsa_files = [files(
++    'migration/dsa-tests.c',
++  )]
++endif
++
+ qtests = {
+   'bios-tables-test': [io, 'boot-sector.c', 'acpi-utils.c', 'tpm-emu.c'],
+   'cdrom-test': files('boot-sector.c'),
+@@ -XXX,XX +XXX,XX @@ qtests = {
+                              'migration/migration-util.c') + dbus_vmstate1,
+   'erst-test': files('erst-test.c'),
+   'ivshmem-test': [rt, '../../contrib/ivshmem-server/ivshmem-server.c'],
+-  'migration-test': migration_files + migration_tls_files,
++  'migration-test': migration_files + migration_tls_files + \
++                    migration_dsa_files,
+   'pxe-test': files('boot-sector.c'),
+   'pnv-xive2-test': files('pnv-xive2-common.c', 'pnv-xive2-flush-sync.c'),
+   'qos-test': [chardev, io, qos_test_ss.apply({}).sources()],
 diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
 index XXXXXXX..XXXXXXX 100644
 --- a/tests/qtest/migration-test.c
 +++ b/tests/qtest/migration-test.c
-@@ -XXX,XX +XXX,XX @@ typedef struct {
+@@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv)
-     const char *opts_target;
+     migration_test_add_precopy(env);
- } MigrateStart;
+     migration_test_add_cpr(env);
+     migration_test_add_misc(env);
 +#ifdef CONFIG_DSA_OPT
 +    migration_test_add_dsa(env);
 +#endif
      ret = g_test_run();
 diff --git a/tests/qtest/migration/dsa-tests.c b/tests/qtest/migration/dsa-tests.c
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
 +++ b/tests/qtest/migration/dsa-tests.c
@@ -XXX,XX +XXX,XX @@
 +/*
 + * QTest testcases for DSA accelerator
 + *
 + * Copyright (C) Bytedance Ltd.
 + *   based on the vhost-user-test.c that is:
 + *      Copyright (c) 2014 Virtual Open Systems Sarl.
 + *
 + * This work is licensed under the terms of the GNU GPL, version 2 or later.
 + * See the COPYING file in the top-level directory.
 + */
 +
 +#include "qemu/osdep.h"
 +#include "libqtest.h"
 +#include "migration/framework.h"
 +#include "migration/migration-qmp.h"
 +#include "migration/migration-util.h"
 +
 +/*
 + * It requires separate steps to configure and enable DSA device.
 + * This test assumes that the configuration is done already.
 + */
-+static const char* dsa_dev_path = "/dev/dsa/wq4.0";
++static const char *dsa_dev_path_p = "['dsa:/dev/dsa/wq4.0']";
-+
++static const char *dsa_dev_path = "/dev/dsa/wq4.0";
  /*
   * A hook that runs after the src and dst QEMUs have been
   * created, but before the migration is started. This can
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_reject_anon_client(void)
   *
   *  And see that it works
   */
 -static void test_multifd_tcp_cancel(void)
 +static void test_multifd_tcp_cancel_common(bool use_dsa)
  {
      MigrateStart args = {
          .hide_stderr = true,
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_cancel(void)
      migrate_set_capability(from, "multifd", true);
      migrate_set_capability(to, "multifd", true);
 +    if (use_dsa) {
 +        migrate_set_parameter_str(from, "multifd-dsa-accel", dsa_dev_path);
 +    }
 +
      /* Start incoming migration from the 1st socket */
      migrate_incoming_qmp(to, "tcp:127.0.0.1:0", "{}");
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_cancel(void)
      test_migrate_end(from, to2, true);
  }
 +/*
 + * This test does:
 + *  source               target
 + *                       migrate_incoming
 + *     migrate
 + *     migrate_cancel
 + *                       launch another target
 + *     migrate
 + *
 + *  And see that it works
 + */
 +static void test_multifd_tcp_cancel(void)
 +{
 +    test_multifd_tcp_cancel_common(false);
 +}
 +
 +#ifdef CONFIG_DSA_OPT
 +
 +static void *test_migrate_precopy_tcp_multifd_start_dsa(QTestState *from,
 +                                                        QTestState *to)
 +{
 +    migrate_set_parameter_str(from, "multifd-dsa-accel", dsa_dev_path);
 +    return test_migrate_precopy_tcp_multifd_start_common(from, to, "none");
 +}
 +
 +static void test_multifd_tcp_none_dsa(void)
 +{
 +    MigrateCommon args = {
 +        .listen_uri = "defer",
 +        .start_hook = test_migrate_precopy_tcp_multifd_start_dsa,
 +    };
 +
 +    test_precopy_common(&args);
 +}
 +
 +static void test_multifd_tcp_cancel_dsa(void)
 +{
 +    test_multifd_tcp_cancel_common(true);
 +}
 +
 +#endif
 +
  static void calc_dirty_rate(QTestState *who, uint64_t calc_time)
  {
      qtest_qmp_assert_success(who,
@@ -XXX,XX +XXX,XX @@ static bool kvm_dirty_ring_supported(void)
  #endif
  }
 +#ifdef CONFIG_DSA_OPT
 +static int test_dsa_setup(void)
 +{
 +    int fd;
 +    fd = open(dsa_dev_path, O_RDWR);
 +    if (fd < 0) {
 +        return -1;
 +    }
 +    close(fd);
 +    return 0;
 +}
-+#endif
 +
- int main(int argc, char **argv)
++static void *test_migrate_precopy_tcp_multifd_start_dsa(QTestState *from,
- {
++                                                        QTestState *to)
-     bool has_kvm, has_tcg;
++{
-@@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv)
++    migrate_set_parameter_str(from, "zero-page-detection", "dsa-accel");
-     }
++    migrate_set_parameter_str(from, "accel-path", dsa_dev_path_p);
-     qtest_add_func("/migration/multifd/tcp/plain/none",
++    return migrate_hook_start_precopy_tcp_multifd_common(from, to, "none");
-                    test_multifd_tcp_none);
++}
 +
-+#ifdef CONFIG_DSA_OPT
++static void test_multifd_tcp_zero_page_dsa(void)
-+    if (g_str_equal(arch, "x86_64") && test_dsa_setup() == 0) {
++{
-+        qtest_add_func("/migration/multifd/tcp/plain/none/dsa",
++    MigrateCommon args = {
-+                       test_multifd_tcp_none_dsa);
++        .listen_uri = "defer",
-+        qtest_add_func("/migration/multifd/tcp/plain/cancel/dsa",
++        .start_hook = test_migrate_precopy_tcp_multifd_start_dsa,
-+                       test_multifd_tcp_cancel_dsa);
++    };
 +
 +    test_precopy_common(&args);
 +}
 +
 +void migration_test_add_dsa(MigrationTestEnv *env)
 +{
 +    if (test_dsa_setup() == 0) {
 +        migration_test_add("/migration/multifd/tcp/plain/zero-page/dsa",
 +                       test_multifd_tcp_zero_page_dsa);
 +    }
-+#endif
++}
-+
+diff --git a/tests/qtest/migration/framework.h b/tests/qtest/migration/framework.h
-     /*
+index XXXXXXX..XXXXXXX 100644
-      * This test is flaky and sometimes fails in CI and otherwise:
+--- a/tests/qtest/migration/framework.h
-      * don't run unless user opts in via environment variable.
++++ b/tests/qtest/migration/framework.h
@@ -XXX,XX +XXX,XX @@ void migration_test_add_file(MigrationTestEnv *env);
  void migration_test_add_precopy(MigrationTestEnv *env);
  void migration_test_add_cpr(MigrationTestEnv *env);
  void migration_test_add_misc(MigrationTestEnv *env);
 +void migration_test_add_dsa(MigrationTestEnv *env);
  #endif /* TEST_FRAMEWORK_H */
 --
-.30.2
+Yichen Wang

-[PATCH v2 06/20] util/dsa: Add dependency idxd.
+[PATCH v9 12/12] migration/doc: Add DSA zero page detection doc
-Idxd is the device driver for DSA (Intel Data Streaming
+From: Yuan Liu <yuan1.liu@intel.com>
 Accelerator). The driver is fully functioning since Linux
 kernel 5.19. This change adds the driver's header file used
 for userspace development.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
+Signed-off-by: Yuan Liu <yuan1.liu@intel.com>
 Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
 Reviewed-by: Fabiano Rosas <farosas@suse.de>
 ---
- linux-headers/linux/idxd.h | 356 +++++++++++++++++++++++++++++++++++++
+ .../migration/dsa-zero-page-detection.rst     | 290 ++++++++++++++++++
-file changed, 356 insertions(+)
+ docs/devel/migration/features.rst             |   1 +
- create mode 100644 linux-headers/linux/idxd.h
+files changed, 291 insertions(+)
  create mode 100644 docs/devel/migration/dsa-zero-page-detection.rst
-diff --git a/linux-headers/linux/idxd.h b/linux-headers/linux/idxd.h
+diff --git a/docs/devel/migration/dsa-zero-page-detection.rst b/docs/devel/migration/dsa-zero-page-detection.rst
 new file mode 100644
 index XXXXXXX..XXXXXXX
 --- /dev/null
-+++ b/linux-headers/linux/idxd.h
++++ b/docs/devel/migration/dsa-zero-page-detection.rst
 @@ -XXX,XX +XXX,XX @@
-+/* SPDX-License-Identifier: LGPL-2.1 WITH Linux-syscall-note */
++=============================
-+/* Copyright(c) 2019 Intel Corporation. All rights rsvd. */
++DSA-Based Zero Page Detection
-+#ifndef _USR_IDXD_H_
++=============================
-+#define _USR_IDXD_H_
++Intel Data Streaming Accelerator(``DSA``) is introduced in Intel's 4th
-+
++generation Xeon server, aka Sapphire Rapids(``SPR``). One of the things
-+#ifdef __KERNEL__
++DSA can do is to offload memory comparison workload from CPU to DSA accelerator
-+#include <linux/types.h>
++hardware.
-+#else
++
-+#include <stdint.h>
++The main advantages of using DSA to accelerate zero pages detection include
-+#endif
++
-+
++1. Reduces CPU usage in multifd live migration workflow across all use cases.
-+/* Driver command error status */
++
-+enum idxd_scmd_stat {
++2. Reduces migration total time in some use cases.
-+    IDXD_SCMD_DEV_ENABLED = 0x80000010,
++
-+    IDXD_SCMD_DEV_NOT_ENABLED = 0x80000020,
++
-+    IDXD_SCMD_WQ_ENABLED = 0x80000021,
++DSA-Based Zero Page Detection Introduction
-+    IDXD_SCMD_DEV_DMA_ERR = 0x80020000,
++==========================================
-+    IDXD_SCMD_WQ_NO_GRP = 0x80030000,
++
-+    IDXD_SCMD_WQ_NO_NAME = 0x80040000,
++::
-+    IDXD_SCMD_WQ_NO_SVM = 0x80050000,
++
-+    IDXD_SCMD_WQ_NO_THRESH = 0x80060000,
++
-+    IDXD_SCMD_WQ_PORTAL_ERR = 0x80070000,
++  +----------------+       +------------------+
-+    IDXD_SCMD_WQ_RES_ALLOC_ERR = 0x80080000,
++  | MultiFD Thread |       |accel-config tool |
-+    IDXD_SCMD_PERCPU_ERR = 0x80090000,
++  +-+--------+-----+       +--------+---------+
-+    IDXD_SCMD_DMA_CHAN_ERR = 0x800a0000,
++    |        |                      |
-+    IDXD_SCMD_CDEV_ERR = 0x800b0000,
++    |        |  Open DSA            | Setup DSA
-+    IDXD_SCMD_WQ_NO_SWQ_SUPPORT = 0x800c0000,
++    |        |  Work Queues         | Resources
-+    IDXD_SCMD_WQ_NONE_CONFIGURED = 0x800d0000,
++    |        |       +-----+-----+  |
-+    IDXD_SCMD_WQ_NO_SIZE = 0x800e0000,
++    |        +------>|idxd driver|<-+
-+    IDXD_SCMD_WQ_NO_PRIV = 0x800f0000,
++    |                +-----+-----+
-+    IDXD_SCMD_WQ_IRQ_ERR = 0x80100000,
++    |                      |
-+    IDXD_SCMD_WQ_USER_NO_IOMMU = 0x80110000,
++    |                      |
-+};
++    |                +-----+-----+
-+
++    +----------------+DSA Devices|
-+#define IDXD_SCMD_SOFTERR_MASK    0x80000000
++      Submit jobs    +-----------+
-+#define IDXD_SCMD_SOFTERR_SHIFT    16
++      via enqcmd
 +
-+/* Descriptor flags */
++
-+#define IDXD_OP_FLAG_FENCE    0x0001
++DSA Introduction
-+#define IDXD_OP_FLAG_BOF    0x0002
++----------------
-+#define IDXD_OP_FLAG_CRAV    0x0004
++Intel Data Streaming Accelerator (DSA) is a high-performance data copy and
-+#define IDXD_OP_FLAG_RCR    0x0008
++transformation accelerator that is integrated in Intel Xeon processors,
-+#define IDXD_OP_FLAG_RCI    0x0010
++targeted for optimizing streaming data movement and transformation operations
-+#define IDXD_OP_FLAG_CRSTS    0x0020
++common with applications for high-performance storage, networking, persistent
-+#define IDXD_OP_FLAG_CR        0x0080
++memory, and various data processing applications.
-+#define IDXD_OP_FLAG_CC        0x0100
++
-+#define IDXD_OP_FLAG_ADDR1_TCS    0x0200
++For more ``DSA`` introduction, please refer to `DSA Introduction
-+#define IDXD_OP_FLAG_ADDR2_TCS    0x0400
++<https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/data-streaming-accelerator.html>`_
-+#define IDXD_OP_FLAG_ADDR3_TCS    0x0800
++
-+#define IDXD_OP_FLAG_CR_TCS    0x1000
++For ``DSA`` specification, please refer to `DSA Specification
-+#define IDXD_OP_FLAG_STORD    0x2000
++<https://cdrdv2-public.intel.com/671116/341204-intel-data-streaming-accelerator-spec.pdf>`_
-+#define IDXD_OP_FLAG_DRDBK    0x4000
++
-+#define IDXD_OP_FLAG_DSTS    0x8000
++For ``DSA`` user guide, please refer to `DSA User Guide
-+
++<https://www.intel.com/content/www/us/en/content-details/759709/intel-data-streaming-accelerator-user-guide.html>`_
-+/* IAX */
++
-+#define IDXD_OP_FLAG_RD_SRC2_AECS    0x010000
++DSA Device Management
-+#define IDXD_OP_FLAG_RD_SRC2_2ND    0x020000
++---------------------
-+#define IDXD_OP_FLAG_WR_SRC2_AECS_COMP    0x040000
++
-+#define IDXD_OP_FLAG_WR_SRC2_AECS_OVFL    0x080000
++The number of ``DSA`` devices will vary depending on the Xeon product model.
-+#define IDXD_OP_FLAG_SRC2_STS        0x100000
++On a ``SPR`` server, there can be a maximum of 8 ``DSA`` devices, with up to
-+#define IDXD_OP_FLAG_CRC_RFC3720    0x200000
++4 devices per socket.
 +
-+/* Opcode */
++By default, all ``DSA`` devices are disabled and need to be configured and
-+enum dsa_opcode {
++enabled by users manually.
-+    DSA_OPCODE_NOOP = 0,
++
-+    DSA_OPCODE_BATCH,
++Check the number of devices through the following command
-+    DSA_OPCODE_DRAIN,
++
-+    DSA_OPCODE_MEMMOVE,
++.. code-block:: shell
-+    DSA_OPCODE_MEMFILL,
++
-+    DSA_OPCODE_COMPARE,
++  #lspci -d 8086:0b25
-+    DSA_OPCODE_COMPVAL,
++  6a:01.0 System peripheral: Intel Corporation Device 0b25
-+    DSA_OPCODE_CR_DELTA,
++  6f:01.0 System peripheral: Intel Corporation Device 0b25
-+    DSA_OPCODE_AP_DELTA,
++  74:01.0 System peripheral: Intel Corporation Device 0b25
-+    DSA_OPCODE_DUALCAST,
++  79:01.0 System peripheral: Intel Corporation Device 0b25
-+    DSA_OPCODE_CRCGEN = 0x10,
++  e7:01.0 System peripheral: Intel Corporation Device 0b25
-+    DSA_OPCODE_COPY_CRC,
++  ec:01.0 System peripheral: Intel Corporation Device 0b25
-+    DSA_OPCODE_DIF_CHECK,
++  f1:01.0 System peripheral: Intel Corporation Device 0b25
-+    DSA_OPCODE_DIF_INS,
++  f6:01.0 System peripheral: Intel Corporation Device 0b25
-+    DSA_OPCODE_DIF_STRP,
++
-+    DSA_OPCODE_DIF_UPDT,
++
-+    DSA_OPCODE_CFLUSH = 0x20,
++DSA Device Configuration And Enabling
-+};
++-------------------------------------
 +
-+enum iax_opcode {
++The ``accel-config`` tool is used to enable ``DSA`` devices and configure
-+    IAX_OPCODE_NOOP = 0,
++``DSA`` hardware resources(work queues and engines). One ``DSA`` device
-+    IAX_OPCODE_DRAIN = 2,
++has 8 work queues and 4 processing engines, multiple engines can be assigned
-+    IAX_OPCODE_MEMMOVE,
++to a work queue via ``group`` attribute.
-+    IAX_OPCODE_DECOMPRESS = 0x42,
++
-+    IAX_OPCODE_COMPRESS,
++For ``accel-config`` installation, please refer to `accel-config installation
-+    IAX_OPCODE_CRC64,
++<https://github.com/intel/idxd-config>`_
-+    IAX_OPCODE_ZERO_DECOMP_32 = 0x48,
++
-+    IAX_OPCODE_ZERO_DECOMP_16,
++One example of configuring and enabling an ``DSA`` device.
-+    IAX_OPCODE_ZERO_COMP_32 = 0x4c,
++
-+    IAX_OPCODE_ZERO_COMP_16,
++.. code-block:: shell
-+    IAX_OPCODE_SCAN = 0x50,
++
-+    IAX_OPCODE_SET_MEMBER,
++  #accel-config config-engine dsa0/engine0.0 -g 0
-+    IAX_OPCODE_EXTRACT,
++  #accel-config config-engine dsa0/engine0.1 -g 0
-+    IAX_OPCODE_SELECT,
++  #accel-config config-engine dsa0/engine0.2 -g 0
-+    IAX_OPCODE_RLE_BURST,
++  #accel-config config-engine dsa0/engine0.3 -g 0
-+    IAX_OPCODE_FIND_UNIQUE,
++  #accel-config config-wq dsa0/wq0.0 -g 0 -s 128 -p 10 -b 1 -t 128 -m shared -y user -n app1 -d user
-+    IAX_OPCODE_EXPAND,
++  #accel-config enable-device dsa0
-+};
++  #accel-config enable-wq dsa0/wq0.0
 +
-+/* Completion record status */
++- The ``DSA`` device index is 0, use ``ls -lh /sys/bus/dsa/devices/dsa*``
-+enum dsa_completion_status {
++  command to query the ``DSA`` device index.
-+    DSA_COMP_NONE = 0,
++
-+    DSA_COMP_SUCCESS,
++- 4 engines and 1 work queue are configured in group 0, so that all zero-page
-+    DSA_COMP_SUCCESS_PRED,
++  detection jobs submitted to this work queue can be processed by all engines
-+    DSA_COMP_PAGE_FAULT_NOBOF,
++  simultaneously.
-+    DSA_COMP_PAGE_FAULT_IR,
++
-+    DSA_COMP_BATCH_FAIL,
++- Set work queue attributes including the work mode, work queue size and so on.
-+    DSA_COMP_BATCH_PAGE_FAULT,
++
-+    DSA_COMP_DR_OFFSET_NOINC,
++- Enable the ``dsa0`` device and work queue ``dsa0/wq0.0``
-+    DSA_COMP_DR_OFFSET_ERANGE,
++
-+    DSA_COMP_DIF_ERR,
++.. note::
-+    DSA_COMP_BAD_OPCODE = 0x10,
++
-+    DSA_COMP_INVALID_FLAGS,
++   1. ``DSA`` device driver is Intel Data Accelerator Driver (idxd), it is
-+    DSA_COMP_NOZERO_RESERVE,
++      recommended that the minimum version of Linux kernel is 5.18.
-+    DSA_COMP_XFER_ERANGE,
++
-+    DSA_COMP_DESC_CNT_ERANGE,
++   2. Only ``DSA`` shared work queue mode is supported, it needs to add
-+    DSA_COMP_DR_ERANGE,
++      ``"intel_iommu=on,sm_on"`` parameter to kernel command line.
-+    DSA_COMP_OVERLAP_BUFFERS,
++
-+    DSA_COMP_DCAST_ERR,
++For more detailed configuration, please refer to `DSA Configuration Samples
-+    DSA_COMP_DESCLIST_ALIGN,
++<https://github.com/intel/idxd-config/tree/stable/Documentation/accfg>`_
-+    DSA_COMP_INT_HANDLE_INVAL,
++
-+    DSA_COMP_CRA_XLAT,
++
-+    DSA_COMP_CRA_ALIGN,
++Performances
-+    DSA_COMP_ADDR_ALIGN,
++============
-+    DSA_COMP_PRIV_BAD,
++We use two Intel 4th generation Xeon servers for testing.
-+    DSA_COMP_TRAFFIC_CLASS_CONF,
++
-+    DSA_COMP_PFAULT_RDBA,
++::
-+    DSA_COMP_HW_ERR1,
++
-+    DSA_COMP_HW_ERR_DRB,
++    Architecture:        x86_64
-+    DSA_COMP_TRANSLATION_FAIL,
++    CPU(s):              192
-+};
++    Thread(s) per core:  2
-+
++    Core(s) per socket:  48
-+enum iax_completion_status {
++    Socket(s):           2
-+    IAX_COMP_NONE = 0,
++    NUMA node(s):        2
-+    IAX_COMP_SUCCESS,
++    Vendor ID:           GenuineIntel
-+    IAX_COMP_PAGE_FAULT_IR = 0x04,
++    CPU family:          6
-+    IAX_COMP_ANALYTICS_ERROR = 0x0a,
++    Model:               143
-+    IAX_COMP_OUTBUF_OVERFLOW,
++    Model name:          Intel(R) Xeon(R) Platinum 8457C
-+    IAX_COMP_BAD_OPCODE = 0x10,
++    Stepping:            8
-+    IAX_COMP_INVALID_FLAGS,
++    CPU MHz:             2538.624
-+    IAX_COMP_NOZERO_RESERVE,
++    CPU max MHz:         3800.0000
-+    IAX_COMP_INVALID_SIZE,
++    CPU min MHz:         800.0000
-+    IAX_COMP_OVERLAP_BUFFERS = 0x16,
++
-+    IAX_COMP_INT_HANDLE_INVAL = 0x19,
++We perform multifd live migration with below setup:
-+    IAX_COMP_CRA_XLAT,
++
-+    IAX_COMP_CRA_ALIGN,
++1. VM has 100GB memory.
-+    IAX_COMP_ADDR_ALIGN,
++
-+    IAX_COMP_PRIV_BAD,
++2. Use the new migration option multifd-set-normal-page-ratio to control the
-+    IAX_COMP_TRAFFIC_CLASS_CONF,
++   total size of the payload sent over the network.
-+    IAX_COMP_PFAULT_RDBA,
++
-+    IAX_COMP_HW_ERR1,
++3. Use 8 multifd channels.
-+    IAX_COMP_HW_ERR_DRB,
++
-+    IAX_COMP_TRANSLATION_FAIL,
++4. Use tcp for live migration.
-+    IAX_COMP_PRS_TIMEOUT,
++
-+    IAX_COMP_WATCHDOG,
++5. Use CPU to perform zero page checking as the baseline.
-+    IAX_COMP_INVALID_COMP_FLAG = 0x30,
++
-+    IAX_COMP_INVALID_FILTER_FLAG,
++6. Use one DSA device to offload zero page checking to compare with the baseline.
-+    IAX_COMP_INVALID_INPUT_SIZE,
++
-+    IAX_COMP_INVALID_NUM_ELEMS,
++7. Use "perf sched record" and "perf sched timehist" to analyze CPU usage.
-+    IAX_COMP_INVALID_SRC1_WIDTH,
++
-+    IAX_COMP_INVALID_INVERT_OUT,
++
-+};
++A) Scenario 1: 50% (50GB) normal pages on an 100GB vm
-+
++-----------------------------------------------------
-+#define DSA_COMP_STATUS_MASK        0x7f
++
-+#define DSA_COMP_STATUS_WRITE        0x80
++::
 +
-+struct dsa_hw_desc {
++    CPU usage
-+    uint32_t    pasid:20;
++
-+    uint32_t    rsvd:11;
++    |---------------|---------------|---------------|---------------|
-+    uint32_t    priv:1;
++    |        |comm        |runtime(msec)    |totaltime(msec)|
-+    uint32_t    flags:24;
++    |---------------|---------------|---------------|---------------|
-+    uint32_t    opcode:8;
++    |Baseline    |live_migration    |5657.58    |        |
-+    uint64_t    completion_addr;
++    |        |multifdsend_0    |3931.563    |        |
-+    union {
++    |        |multifdsend_1    |4405.273    |        |
-+        uint64_t    src_addr;
++    |        |multifdsend_2    |3941.968    |        |
-+        uint64_t    rdback_addr;
++    |        |multifdsend_3    |5032.975    |        |
-+        uint64_t    pattern;
++    |        |multifdsend_4    |4533.865    |        |
-+        uint64_t    desc_list_addr;
++    |        |multifdsend_5    |4530.461    |        |
-+    };
++    |        |multifdsend_6    |5171.916    |        |
-+    union {
++    |        |multifdsend_7    |4722.769    |41922        |
-+        uint64_t    dst_addr;
++    |---------------|---------------|---------------|---------------|
-+        uint64_t    rdback_addr2;
++    |DSA        |live_migration    |6129.168    |        |
-+        uint64_t    src2_addr;
++    |        |multifdsend_0    |2954.717    |        |
-+        uint64_t    comp_pattern;
++    |        |multifdsend_1    |2766.359    |        |
-+    };
++    |        |multifdsend_2    |2853.519    |        |
-+    union {
++    |        |multifdsend_3    |2740.717    |        |
-+        uint32_t    xfer_size;
++    |        |multifdsend_4    |2824.169    |        |
-+        uint32_t    desc_count;
++    |        |multifdsend_5    |2966.908    |        |
-+    };
++    |        |multifdsend_6    |2611.137    |        |
-+    uint16_t    int_handle;
++    |        |multifdsend_7    |3114.732    |        |
-+    uint16_t    rsvd1;
++    |        |dsa_completion    |3612.564    |32568        |
-+    union {
++    |---------------|---------------|---------------|---------------|
-+        uint8_t        expected_res;
++
-+        /* create delta record */
++Baseline total runtime is calculated by adding up all multifdsend_X
-+        struct {
++and live_migration threads runtime. DSA offloading total runtime is
-+            uint64_t    delta_addr;
++calculated by adding up all multifdsend_X, live_migration and
-+            uint32_t    max_delta_size;
++dsa_completion threads runtime. 41922 msec VS 32568 msec runtime and
-+            uint32_t     delt_rsvd;
++that is 23% total CPU usage savings.
-+            uint8_t     expected_res_mask;
++
-+        };
++::
-+        uint32_t    delta_rec_size;
++
-+        uint64_t    dest2;
++    Latency
-+        /* CRC */
++    |---------------|---------------|---------------|---------------|---------------|---------------|
-+        struct {
++    |        |total time    |down time    |throughput    |transferred-ram|total-ram    |
-+            uint32_t    crc_seed;
++    |---------------|---------------|---------------|---------------|---------------|---------------|
-+            uint32_t    crc_rsvd;
++    |Baseline    |10343 ms    |161 ms        |41007.00 mbps    |51583797 kb    |102400520 kb    |
-+            uint64_t    seed_addr;
++    |---------------|---------------|---------------|---------------|-------------------------------|
-+        };
++    |DSA offload    |9535 ms    |135 ms        |46554.40 mbps    |53947545 kb    |102400520 kb    |
-+        /* DIF check or strip */
++    |---------------|---------------|---------------|---------------|---------------|---------------|
-+        struct {
++
-+            uint8_t        src_dif_flags;
++Total time is 8% faster and down time is 16% faster.
-+            uint8_t        dif_chk_res;
++
-+            uint8_t        dif_chk_flags;
++
-+            uint8_t        dif_chk_res2[5];
++B) Scenario 2: 100% (100GB) zero pages on an 100GB vm
-+            uint32_t    chk_ref_tag_seed;
++-----------------------------------------------------
-+            uint16_t    chk_app_tag_mask;
++
-+            uint16_t    chk_app_tag_seed;
++::
-+        };
++
-+        /* DIF insert */
++    CPU usage
-+        struct {
++    |---------------|---------------|---------------|---------------|
-+            uint8_t        dif_ins_res;
++    |        |comm        |runtime(msec)    |totaltime(msec)|
-+            uint8_t        dest_dif_flag;
++    |---------------|---------------|---------------|---------------|
-+            uint8_t        dif_ins_flags;
++    |Baseline    |live_migration    |4860.718    |        |
-+            uint8_t        dif_ins_res2[13];
++    |         |multifdsend_0    |748.875    |        |
-+            uint32_t    ins_ref_tag_seed;
++    |        |multifdsend_1    |898.498    |        |
-+            uint16_t    ins_app_tag_mask;
++    |        |multifdsend_2    |787.456    |        |
-+            uint16_t    ins_app_tag_seed;
++    |        |multifdsend_3    |764.537    |        |
-+        };
++    |        |multifdsend_4    |785.687    |        |
-+        /* DIF update */
++    |        |multifdsend_5    |756.941    |        |
-+        struct {
++    |        |multifdsend_6    |774.084    |        |
-+            uint8_t        src_upd_flags;
++    |        |multifdsend_7    |782.900    |11154        |
-+            uint8_t        upd_dest_flags;
++    |---------------|---------------|-------------------------------|
-+            uint8_t        dif_upd_flags;
++    |DSA offloading    |live_migration    |3846.976    |        |
-+            uint8_t        dif_upd_res[5];
++    |        |multifdsend_0    |191.880    |        |
-+            uint32_t    src_ref_tag_seed;
++    |        |multifdsend_1    |166.331    |        |
-+            uint16_t    src_app_tag_mask;
++    |        |multifdsend_2    |168.528    |        |
-+            uint16_t    src_app_tag_seed;
++    |        |multifdsend_3    |197.831    |        |
-+            uint32_t    dest_ref_tag_seed;
++    |        |multifdsend_4    |169.580    |        |
-+            uint16_t    dest_app_tag_mask;
++    |        |multifdsend_5    |167.984    |        |
-+            uint16_t    dest_app_tag_seed;
++    |        |multifdsend_6    |198.042    |        |
-+        };
++    |        |multifdsend_7    |170.624    |        |
-+
++    |        |dsa_completion    |3428.669    |8700        |
-+        uint8_t        op_specific[24];
++    |---------------|---------------|---------------|---------------|
-+    };
++
-+} __attribute__((packed));
++Baseline total runtime is 11154 msec and DSA offloading total runtime is
-+
++8700 msec. That is 22% CPU savings.
-+struct iax_hw_desc {
++
-+    uint32_t        pasid:20;
++::
-+    uint32_t        rsvd:11;
++
-+    uint32_t        priv:1;
++    Latency
-+    uint32_t        flags:24;
++    |--------------------------------------------------------------------------------------------|
-+    uint32_t        opcode:8;
++    |        |total time    |down time    |throughput    |transferred-ram|total-ram   |
-+    uint64_t        completion_addr;
++    |---------------|---------------|---------------|---------------|---------------|------------|
-+    uint64_t        src1_addr;
++    |Baseline    |4867 ms    |20 ms        |1.51 mbps    |565 kb        |102400520 kb|
-+    uint64_t        dst_addr;
++    |---------------|---------------|---------------|---------------|----------------------------|
-+    uint32_t        src1_size;
++    |DSA offload    |3888 ms    |18 ms        |1.89 mbps    |565 kb        |102400520 kb|
-+    uint16_t        int_handle;
++    |---------------|---------------|---------------|---------------|---------------|------------|
-+    union {
++
-+        uint16_t        compr_flags;
++Total time 20% faster and down time 10% faster.
-+        uint16_t        decompr_flags;
++
-+    };
++
-+    uint64_t        src2_addr;
++How To Use DSA In Migration
-+    uint32_t        max_dst_size;
++===========================
-+    uint32_t        src2_size;
++
-+    uint32_t    filter_flags;
++The migration parameter ``accel-path`` is used to specify the resource
-+    uint32_t    num_inputs;
++allocation for DSA. After the user configures
-+} __attribute__((packed));
++``zero-page-detection=dsa-accel``, one or more DSA work queues need to be
-+
++specified for migration.
-+struct dsa_raw_desc {
++
-+    uint64_t    field[8];
++The following example shows two DSA work queues for zero page detection
-+} __attribute__((packed));
++
-+
++.. code-block:: shell
-+/*
++
-+ * The status field will be modified by hardware, therefore it should be
++   migrate_set_parameter zero-page-detection=dsa-accel
-+ * volatile and prevent the compiler from optimize the read.
++   migrate_set_parameter accel-path=dsa:/dev/dsa/wq0.0 dsa:/dev/dsa/wq1.0
-+ */
++
-+struct dsa_completion_record {
++.. note::
-+    volatile uint8_t    status;
++
-+    union {
++  Accessing DSA resources requires ``sudo`` command or ``root`` privileges
-+        uint8_t        result;
++  by default. Administrators can modify the DSA device node ownership
-+        uint8_t        dif_status;
++  so that QEMU can use DSA with specified user permissions.
-+    };
++
-+    uint16_t        rsvd;
++  For example:
-+    uint32_t        bytes_completed;
++
-+    uint64_t        fault_addr;
++  #chown -R qemu /dev/dsa
-+    union {
++
-+        /* common record */
+diff --git a/docs/devel/migration/features.rst b/docs/devel/migration/features.rst
-+        struct {
+index XXXXXXX..XXXXXXX 100644
-+            uint32_t    invalid_flags:24;
+--- a/docs/devel/migration/features.rst
-+            uint32_t    rsvd2:8;
++++ b/docs/devel/migration/features.rst
-+        };
+@@ -XXX,XX +XXX,XX @@ Migration has plenty of features to support different use cases.
-+
+    qpl-compression
-+        uint32_t    delta_rec_size;
+    uadk-compression
-+        uint64_t    crc_val;
+    qatzip-compression
-+
++   dsa-zero-page-detection
 +        /* DIF check & strip */
 +        struct {
 +            uint32_t    dif_chk_ref_tag;
 +            uint16_t    dif_chk_app_tag_mask;
 +            uint16_t    dif_chk_app_tag;
 +        };
 +
 +        /* DIF insert */
 +        struct {
 +            uint64_t    dif_ins_res;
 +            uint32_t    dif_ins_ref_tag;
 +            uint16_t    dif_ins_app_tag_mask;
 +            uint16_t    dif_ins_app_tag;
 +        };
 +
 +        /* DIF update */
 +        struct {
 +            uint32_t    dif_upd_src_ref_tag;
 +            uint16_t    dif_upd_src_app_tag_mask;
 +            uint16_t    dif_upd_src_app_tag;
 +            uint32_t    dif_upd_dest_ref_tag;
 +            uint16_t    dif_upd_dest_app_tag_mask;
 +            uint16_t    dif_upd_dest_app_tag;
 +        };
 +
 +        uint8_t        op_specific[16];
 +    };
 +} __attribute__((packed));
 +
 +struct dsa_raw_completion_record {
 +    uint64_t    field[4];
 +} __attribute__((packed));
 +
 +struct iax_completion_record {
 +    volatile uint8_t        status;
 +    uint8_t                 error_code;
 +    uint16_t                rsvd;
 +    uint32_t                bytes_completed;
 +    uint64_t                fault_addr;
 +    uint32_t                invalid_flags;
 +    uint32_t                rsvd2;
 +    uint32_t                output_size;
 +    uint8_t                 output_bits;
 +    uint8_t                 rsvd3;
 +    uint16_t                xor_csum;
 +    uint32_t                crc;
 +    uint32_t                min;
 +    uint32_t                max;
 +    uint32_t                sum;
 +    uint64_t                rsvd4[2];
 +} __attribute__((packed));
 +
 +struct iax_raw_completion_record {
 +    uint64_t    field[8];
 +} __attribute__((packed));
 +
 +#endif
 --
-.30.2
+Yichen Wang

-[PATCH v2 15/20] migration/multifd: Add test hook to set normal page ratio.
+Deleted patch
-Multifd sender thread performs zero page checking. If a page is
-a zero page, only the page's metadata is sent to the receiver.
-If a page is a normal page, the entire page's content is sent to
-the receiver. This change adds a test hook to set the normal page
-ratio. A zero page will be forced to be sent as a normal page. This
-is useful for live migration performance analysis and optimization.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
----
- migration/options.c | 31 +++++++++++++++++++++++++++++++
- migration/options.h |  1 +
- qapi/migration.json | 18 +++++++++++++++---
-files changed, 47 insertions(+), 3 deletions(-)
-diff --git a/migration/options.c b/migration/options.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/options.c
-+++ b/migration/options.c
-@@ -XXX,XX +XXX,XX @@
- #define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS    5
- #define DEFAULT_MIGRATE_ANNOUNCE_STEP    100
-+/*
-+ * Parameter for multifd normal page test hook.
-+ */
-+#define DEFAULT_MIGRATE_MULTIFD_NORMAL_PAGE_RATIO 101
-+
- #define DEFINE_PROP_MIG_CAP(name, x)             \
-     DEFINE_PROP_BOOL(name, MigrationState, capabilities[x], false)
-@@ -XXX,XX +XXX,XX @@ Property migration_properties[] = {
-                       MIG_MODE_NORMAL),
-     DEFINE_PROP_STRING("multifd-dsa-accel", MigrationState,
-                        parameters.multifd_dsa_accel),
-+    DEFINE_PROP_UINT8("multifd-normal-page-ratio", MigrationState,
-+                      parameters.multifd_normal_page_ratio,
-+                      DEFAULT_MIGRATE_MULTIFD_NORMAL_PAGE_RATIO),
-     /* Migration capabilities */
-     DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
-@@ -XXX,XX +XXX,XX @@ int migrate_multifd_channels(void)
-     return s->parameters.multifd_channels;
- }
-+uint8_t migrate_multifd_normal_page_ratio(void)
-+{
-+    MigrationState *s = migrate_get_current();
-+    return s->parameters.multifd_normal_page_ratio;
-+}
-+
- MultiFDCompression migrate_multifd_compression(void)
- {
-     MigrationState *s = migrate_get_current();
-@@ -XXX,XX +XXX,XX @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
-         return false;
-     }
-+    if (params->has_multifd_normal_page_ratio &&
-+        params->multifd_normal_page_ratio > 100) {
-+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
-+                   "multifd_normal_page_ratio",
-+                   "a value between 0 and 100");
-+        return false;
-+    }
-+
-     return true;
- }
-@@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrateSetParameters *params,
-         assert(params->multifd_dsa_accel->type == QTYPE_QSTRING);
-         dest->multifd_dsa_accel = params->multifd_dsa_accel->u.s;
-     }
-+
-+    if (params->has_multifd_normal_page_ratio) {
-+        dest->has_multifd_normal_page_ratio = true;
-+        dest->multifd_normal_page_ratio = params->multifd_normal_page_ratio;
-+    }
- }
- static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
-@@ -XXX,XX +XXX,XX @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
-         assert(params->multifd_dsa_accel->type == QTYPE_QSTRING);
-         s->parameters.multifd_dsa_accel = g_strdup(params->multifd_dsa_accel->u.s);
-     }
-+
-+    if (params->has_multifd_normal_page_ratio) {
-+        s->parameters.multifd_normal_page_ratio = params->multifd_normal_page_ratio;
-+    }
- }
- void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
-diff --git a/migration/options.h b/migration/options.h
-index XXXXXXX..XXXXXXX 100644
---- a/migration/options.h
-+++ b/migration/options.h
-@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_creds(void);
- const char *migrate_tls_hostname(void);
- uint64_t migrate_xbzrle_cache_size(void);
- const char *migrate_multifd_dsa_accel(void);
-+uint8_t migrate_multifd_normal_page_ratio(void);
- /* parameters setters */
-diff --git a/qapi/migration.json b/qapi/migration.json
-index XXXXXXX..XXXXXXX 100644
---- a/qapi/migration.json
-+++ b/qapi/migration.json
-@@ -XXX,XX +XXX,XX @@
- # @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
- #                     certain memory operations. (since 8.2)
- #
-+# @multifd-normal-page-ratio: Test hook setting the normal page ratio.
-+#     (Since 8.2)
-+#
- # Features:
- #
- # @deprecated: Member @block-incremental is deprecated.  Use
-@@ -XXX,XX +XXX,XX @@
-            'block-bitmap-mapping',
-            { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] },
-            'vcpu-dirty-limit',
--           'mode'] }
-+           'mode',
-+           'multifd-normal-page-ratio'] }
- ##
- # @MigrateSetParameters:
-@@ -XXX,XX +XXX,XX @@
- # @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
- #                     certain memory operations. (since 8.2)
- #
-+# @multifd-normal-page-ratio: Test hook setting the normal page ratio.
-+#     (Since 8.2)
-+#
- # Features:
- #
- # @deprecated: Member @block-incremental is deprecated.  Use
-@@ -XXX,XX +XXX,XX @@
-                                             'features': [ 'unstable' ] },
-             '*vcpu-dirty-limit': 'uint64',
-             '*mode': 'MigMode',
--            '*multifd-dsa-accel': 'StrOrNull'} }
-+            '*multifd-dsa-accel': 'StrOrNull',
-+            '*multifd-normal-page-ratio': 'uint8'} }
- ##
- # @migrate-set-parameters:
-@@ -XXX,XX +XXX,XX @@
- # @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
- #                     certain memory operations. (since 8.2)
- #
-+# @multifd-normal-page-ratio: Test hook setting the normal page ratio.
-+#     (Since 8.2)
-+#
- # Features:
- #
- # @deprecated: Member @block-incremental is deprecated.  Use
-@@ -XXX,XX +XXX,XX @@
-                                             'features': [ 'unstable' ] },
-             '*vcpu-dirty-limit': 'uint64',
-             '*mode': 'MigMode',
--            '*multifd-dsa-accel': 'str'} }
-+            '*multifd-dsa-accel': 'str',
-+            '*multifd-normal-page-ratio': 'uint8'} }
- ##
- # @query-migrate-parameters:
---
-.30.2

-[PATCH v2 16/20] migration/multifd: Enable set normal page ratio test hook in multifd.
+Deleted patch
-Test hook is disabled by default. To set it, a normal page ratio
-between 0 and 100 are valid. If the ratio is set to 50, it means
-at least 50% of all pages are sent as normal pages.
-Set the option:
-migrate_set_parameter multifd-normal-page-ratio 60
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
----
- include/qemu/dsa.h             |  7 ++++++-
- migration/migration-hmp-cmds.c |  7 +++++++
- migration/multifd.c            | 33 +++++++++++++++++++++++++++++++++
-files changed, 46 insertions(+), 1 deletion(-)
-diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
-index XXXXXXX..XXXXXXX 100644
---- a/include/qemu/dsa.h
-+++ b/include/qemu/dsa.h
-@@ -XXX,XX +XXX,XX @@ typedef struct buffer_zero_batch_task {
-     enum dsa_task_type task_type;
-     enum dsa_task_status status;
-     bool *results;
--    int batch_size;
-+    uint32_t batch_size;
-+    // Set normal page ratio test hook.
-+    uint32_t normal_page_index;
-+    uint32_t normal_page_counter;
-     QSIMPLEQ_ENTRY(buffer_zero_batch_task) entry;
- } buffer_zero_batch_task;
-@@ -XXX,XX +XXX,XX @@ typedef struct buffer_zero_batch_task {
- struct buffer_zero_batch_task {
-     bool *results;
-+    uint32_t normal_page_index;
-+    uint32_t normal_page_counter;
- };
- #endif
-diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/migration-hmp-cmds.c
-+++ b/migration/migration-hmp-cmds.c
-@@ -XXX,XX +XXX,XX @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
-         monitor_printf(mon, "%s: %s\n",
-             MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_DSA_ACCEL),
-             params->multifd_dsa_accel);
-+        monitor_printf(mon, "%s: %u\n",
-+            MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_NORMAL_PAGE_RATIO),
-+            params->multifd_normal_page_ratio);
-         if (params->has_block_bitmap_mapping) {
-             const BitmapMigrationNodeAliasList *bmnal;
-@@ -XXX,XX +XXX,XX @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
-         error_setg(&err, "The block-bitmap-mapping parameter can only be set "
-                    "through QMP");
-         break;
-+    case MIGRATION_PARAMETER_MULTIFD_NORMAL_PAGE_RATIO:
-+        p->has_multifd_normal_page_ratio = true;
-+        visit_type_uint8(v, param, &p->multifd_normal_page_ratio, &err);
-+        break;
-     case MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD:
-         p->has_x_vcpu_dirty_limit_period = true;
-         visit_type_size(v, param, &p->x_vcpu_dirty_limit_period, &err);
-diff --git a/migration/multifd.c b/migration/multifd.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/multifd.c
-+++ b/migration/multifd.c
-@@ -XXX,XX +XXX,XX @@ int multifd_send_sync_main(QEMUFile *f)
-     return 0;
- }
-+static void multifd_normal_page_test_hook(MultiFDSendParams *p)
-+{
-+    /*
-+     * The value is between 0 to 100. If the value is 10, it means at
-+     * least 10% of the pages are normal page. A zero page can be made
-+     * a normal page but not the other way around.
-+     */
-+    uint8_t multifd_normal_page_ratio =
-+        migrate_multifd_normal_page_ratio();
-+    struct buffer_zero_batch_task *batch_task = p->batch_task;
-+
-+    // Set normal page test hook is disabled.
-+    if (multifd_normal_page_ratio > 100) {
-+        return;
-+    }
-+
-+    for (int i = 0; i < p->pages->num; i++) {
-+        if (batch_task->normal_page_counter < multifd_normal_page_ratio) {
-+            // Turn a zero page into a normal page.
-+            batch_task->results[i] = false;
-+        }
-+        batch_task->normal_page_index++;
-+        batch_task->normal_page_counter++;
-+
-+        if (batch_task->normal_page_index >= 100) {
-+            batch_task->normal_page_index = 0;
-+            batch_task->normal_page_counter = 0;
-+        }
-+    }
-+}
-+
- static void set_page(MultiFDSendParams *p, bool zero_page, uint64_t offset)
- {
-     RAMBlock *rb = p->pages->block;
-@@ -XXX,XX +XXX,XX @@ static void multifd_zero_page_check(MultiFDSendParams *p)
-         set_normal_pages(p);
-     }
-+    multifd_normal_page_test_hook(p);
-+
-     for (int i = 0; i < p->pages->num; i++) {
-         uint64_t offset = p->pages->offset[i];
-         bool zero_page = p->batch_task->results[i];
---
-.30.2

-[PATCH v2 17/20] migration/multifd: Add migration option set packet size.
+Deleted patch
-The current multifd packet size is 128 * 4kb. This change adds
-an option to set the packet size. Both sender and receiver needs
-to set the same packet size for things to work.
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
----
- migration/options.c | 34 ++++++++++++++++++++++++++++++++++
- migration/options.h |  1 +
- qapi/migration.json | 21 ++++++++++++++++++---
-files changed, 53 insertions(+), 3 deletions(-)
-diff --git a/migration/options.c b/migration/options.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/options.c
-+++ b/migration/options.c
-@@ -XXX,XX +XXX,XX @@
-  */
- #define DEFAULT_MIGRATE_MULTIFD_NORMAL_PAGE_RATIO 101
-+/*
-+ * Parameter for multifd packet size.
-+ */
-+#define DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE 128
-+#define MAX_MIGRATE_MULTIFD_PACKET_SIZE 1024
-+
- #define DEFINE_PROP_MIG_CAP(name, x)             \
-     DEFINE_PROP_BOOL(name, MigrationState, capabilities[x], false)
-@@ -XXX,XX +XXX,XX @@ Property migration_properties[] = {
-     DEFINE_PROP_UINT8("multifd-normal-page-ratio", MigrationState,
-                       parameters.multifd_normal_page_ratio,
-                       DEFAULT_MIGRATE_MULTIFD_NORMAL_PAGE_RATIO),
-+    DEFINE_PROP_SIZE("multifd-packet-size", MigrationState,
-+                     parameters.multifd_packet_size,
-+                     DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE),
-     /* Migration capabilities */
-     DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
-@@ -XXX,XX +XXX,XX @@ uint8_t migrate_multifd_normal_page_ratio(void)
-     return s->parameters.multifd_normal_page_ratio;
- }
-+uint64_t migrate_multifd_packet_size(void)
-+{
-+    MigrationState *s = migrate_get_current();
-+
-+    return s->parameters.multifd_packet_size;
-+}
-+
- MultiFDCompression migrate_multifd_compression(void)
- {
-     MigrationState *s = migrate_get_current();
-@@ -XXX,XX +XXX,XX @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
-     params->x_checkpoint_delay = s->parameters.x_checkpoint_delay;
-     params->has_block_incremental = true;
-     params->block_incremental = s->parameters.block_incremental;
-+    params->has_multifd_packet_size = true;
-+    params->multifd_packet_size = s->parameters.multifd_packet_size;
-     params->has_multifd_channels = true;
-     params->multifd_channels = s->parameters.multifd_channels;
-     params->has_multifd_compression = true;
-@@ -XXX,XX +XXX,XX @@ void migrate_params_init(MigrationParameters *params)
-     params->has_downtime_limit = true;
-     params->has_x_checkpoint_delay = true;
-     params->has_block_incremental = true;
-+    params->has_multifd_packet_size = true;
-     params->has_multifd_channels = true;
-     params->has_multifd_compression = true;
-     params->has_multifd_zlib_level = true;
-@@ -XXX,XX +XXX,XX @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
-     /* x_checkpoint_delay is now always positive */
-+    if (params->has_multifd_packet_size &&
-+        ((params->multifd_packet_size < DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE) ||
-+            (params->multifd_packet_size >  MAX_MIGRATE_MULTIFD_PACKET_SIZE))) {
-+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
-+                    "multifd_packet_size",
-+                    "a value between 128 and 1024");
-+        return false;
-+    }
-+
-     if (params->has_multifd_channels && (params->multifd_channels < 1)) {
-         error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
-                    "multifd_channels",
-@@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrateSetParameters *params,
-     if (params->has_block_incremental) {
-         dest->block_incremental = params->block_incremental;
-     }
-+    if (params->has_multifd_packet_size) {
-+        dest->multifd_packet_size = params->multifd_packet_size;
-+    }
-     if (params->has_multifd_channels) {
-         dest->multifd_channels = params->multifd_channels;
-     }
-@@ -XXX,XX +XXX,XX @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
-                     " use blockdev-mirror with NBD instead");
-         s->parameters.block_incremental = params->block_incremental;
-     }
-+    if (params->has_multifd_packet_size) {
-+        s->parameters.multifd_packet_size = params->multifd_packet_size;
-+    }
-     if (params->has_multifd_channels) {
-         s->parameters.multifd_channels = params->multifd_channels;
-     }
-diff --git a/migration/options.h b/migration/options.h
-index XXXXXXX..XXXXXXX 100644
---- a/migration/options.h
-+++ b/migration/options.h
-@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_hostname(void);
- uint64_t migrate_xbzrle_cache_size(void);
- const char *migrate_multifd_dsa_accel(void);
- uint8_t migrate_multifd_normal_page_ratio(void);
-+uint64_t migrate_multifd_packet_size(void);
- /* parameters setters */
-diff --git a/qapi/migration.json b/qapi/migration.json
-index XXXXXXX..XXXXXXX 100644
---- a/qapi/migration.json
-+++ b/qapi/migration.json
-@@ -XXX,XX +XXX,XX @@
- # @multifd-normal-page-ratio: Test hook setting the normal page ratio.
- #     (Since 8.2)
- #
-+# @multifd-packet-size: Packet size used to migrate data. This value
-+#     indicates the number of pages in a packet. The default value
-+#     is 128 and max value is 1024. (Since 8.2)
-+#
- # Features:
- #
- # @deprecated: Member @block-incremental is deprecated.  Use
-@@ -XXX,XX +XXX,XX @@
-            { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] },
-            'vcpu-dirty-limit',
-            'mode',
--           'multifd-normal-page-ratio'] }
-+           'multifd-normal-page-ratio',
-+           'multifd-packet-size'] }
- ##
- # @MigrateSetParameters:
-@@ -XXX,XX +XXX,XX @@
- # @multifd-normal-page-ratio: Test hook setting the normal page ratio.
- #     (Since 8.2)
- #
-+# @multifd-packet-size: Packet size used to migrate data. This value
-+#     indicates the number of pages in a packet. The default value
-+#     is 128 and max value is 1024. (Since 8.2)
-+#
- # Features:
- #
- # @deprecated: Member @block-incremental is deprecated.  Use
-@@ -XXX,XX +XXX,XX @@
-             '*vcpu-dirty-limit': 'uint64',
-             '*mode': 'MigMode',
-             '*multifd-dsa-accel': 'StrOrNull',
--            '*multifd-normal-page-ratio': 'uint8'} }
-+            '*multifd-normal-page-ratio': 'uint8',
-+            '*multifd-packet-size' : 'uint64'} }
- ##
- # @migrate-set-parameters:
-@@ -XXX,XX +XXX,XX @@
- # @multifd-normal-page-ratio: Test hook setting the normal page ratio.
- #     (Since 8.2)
- #
-+# @multifd-packet-size: Packet size used to migrate data. This value
-+#     indicates the number of pages in a packet. The default value
-+#     is 128 and max value is 1024. (Since 8.2)
-+#
- # Features:
- #
- # @deprecated: Member @block-incremental is deprecated.  Use
-@@ -XXX,XX +XXX,XX @@
-             '*vcpu-dirty-limit': 'uint64',
-             '*mode': 'MigMode',
-             '*multifd-dsa-accel': 'str',
--            '*multifd-normal-page-ratio': 'uint8'} }
-+            '*multifd-normal-page-ratio': 'uint8',
-+            '*multifd-packet-size': 'uint64'} }
- ##
- # @query-migrate-parameters:
---
-.30.2

-[PATCH v2 18/20] migration/multifd: Enable set packet size migration option.
+Deleted patch
-During live migration, if the latency between sender and receiver
-is high but bandwidth is high (a long and fat pipe), using a bigger
-packet size can help reduce migration total time. In addition, Intel
-DSA offloading performs better with a large batch task. Providing an
-option to set the packet size is useful for performance tuning.
-Set the option:
-migrate_set_parameter multifd-packet-size 512
-Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
----
- migration/migration-hmp-cmds.c | 7 +++++++
- migration/multifd-zlib.c       | 8 ++++++--
- migration/multifd-zstd.c       | 8 ++++++--
- migration/multifd.c            | 4 ++--
- migration/multifd.h            | 3 ---
-files changed, 21 insertions(+), 9 deletions(-)
-diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/migration-hmp-cmds.c
-+++ b/migration/migration-hmp-cmds.c
-@@ -XXX,XX +XXX,XX @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
-         monitor_printf(mon, "%s: %s\n",
-             MigrationParameter_str(MIGRATION_PARAMETER_BLOCK_INCREMENTAL),
-             params->block_incremental ? "on" : "off");
-+        monitor_printf(mon, "%s: %" PRIu64 "\n",
-+            MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_PACKET_SIZE),
-+            params->multifd_packet_size);
-         monitor_printf(mon, "%s: %u\n",
-             MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_CHANNELS),
-             params->multifd_channels);
-@@ -XXX,XX +XXX,XX @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
-         p->multifd_dsa_accel->type = QTYPE_QSTRING;
-         visit_type_str(v, param, &p->multifd_dsa_accel->u.s, &err);
-         break;
-+    case MIGRATION_PARAMETER_MULTIFD_PACKET_SIZE:
-+        p->has_multifd_packet_size = true;
-+        visit_type_size(v, param, &p->multifd_packet_size, &err);
-+        break;
-     case MIGRATION_PARAMETER_MULTIFD_CHANNELS:
-         p->has_multifd_channels = true;
-         visit_type_uint8(v, param, &p->multifd_channels, &err);
-diff --git a/migration/multifd-zlib.c b/migration/multifd-zlib.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/multifd-zlib.c
-+++ b/migration/multifd-zlib.c
-@@ -XXX,XX +XXX,XX @@ static int zlib_send_setup(MultiFDSendParams *p, Error **errp)
-     struct zlib_data *z = g_new0(struct zlib_data, 1);
-     z_stream *zs = &z->zs;
-     const char *err_msg;
-+    uint64_t multifd_packet_size =
-+        migrate_multifd_packet_size() * qemu_target_page_size();
-     zs->zalloc = Z_NULL;
-     zs->zfree = Z_NULL;
-@@ -XXX,XX +XXX,XX @@ static int zlib_send_setup(MultiFDSendParams *p, Error **errp)
-         goto err_free_z;
-     }
-     /* This is the maximum size of the compressed buffer */
--    z->zbuff_len = compressBound(MULTIFD_PACKET_SIZE);
-+    z->zbuff_len = compressBound(multifd_packet_size);
-     z->zbuff = g_try_malloc(z->zbuff_len);
-     if (!z->zbuff) {
-         err_msg = "out of memory for zbuff";
-@@ -XXX,XX +XXX,XX @@ static int zlib_send_prepare(MultiFDSendParams *p, Error **errp)
-  */
- static int zlib_recv_setup(MultiFDRecvParams *p, Error **errp)
- {
-+    uint64_t multifd_packet_size =
-+        migrate_multifd_packet_size() * qemu_target_page_size();
-     struct zlib_data *z = g_new0(struct zlib_data, 1);
-     z_stream *zs = &z->zs;
-@@ -XXX,XX +XXX,XX @@ static int zlib_recv_setup(MultiFDRecvParams *p, Error **errp)
-         return -1;
-     }
-     /* To be safe, we reserve twice the size of the packet */
--    z->zbuff_len = MULTIFD_PACKET_SIZE * 2;
-+    z->zbuff_len = multifd_packet_size * 2;
-     z->zbuff = g_try_malloc(z->zbuff_len);
-     if (!z->zbuff) {
-         inflateEnd(zs);
-diff --git a/migration/multifd-zstd.c b/migration/multifd-zstd.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/multifd-zstd.c
-+++ b/migration/multifd-zstd.c
-@@ -XXX,XX +XXX,XX @@ struct zstd_data {
-  */
- static int zstd_send_setup(MultiFDSendParams *p, Error **errp)
- {
-+    uint64_t multifd_packet_size =
-+        migrate_multifd_packet_size() * qemu_target_page_size();
-     struct zstd_data *z = g_new0(struct zstd_data, 1);
-     int res;
-@@ -XXX,XX +XXX,XX @@ static int zstd_send_setup(MultiFDSendParams *p, Error **errp)
-         return -1;
-     }
-     /* This is the maximum size of the compressed buffer */
--    z->zbuff_len = ZSTD_compressBound(MULTIFD_PACKET_SIZE);
-+    z->zbuff_len = ZSTD_compressBound(multifd_packet_size);
-     z->zbuff = g_try_malloc(z->zbuff_len);
-     if (!z->zbuff) {
-         ZSTD_freeCStream(z->zcs);
-@@ -XXX,XX +XXX,XX @@ static int zstd_send_prepare(MultiFDSendParams *p, Error **errp)
-  */
- static int zstd_recv_setup(MultiFDRecvParams *p, Error **errp)
- {
-+    uint64_t multifd_packet_size =
-+        migrate_multifd_packet_size() * qemu_target_page_size();
-     struct zstd_data *z = g_new0(struct zstd_data, 1);
-     int ret;
-@@ -XXX,XX +XXX,XX @@ static int zstd_recv_setup(MultiFDRecvParams *p, Error **errp)
-     }
-     /* To be safe, we reserve twice the size of the packet */
--    z->zbuff_len = MULTIFD_PACKET_SIZE * 2;
-+    z->zbuff_len = multifd_packet_size * 2;
-     z->zbuff = g_try_malloc(z->zbuff_len);
-     if (!z->zbuff) {
-         ZSTD_freeDStream(z->zds);
-diff --git a/migration/multifd.c b/migration/multifd.c
-index XXXXXXX..XXXXXXX 100644
---- a/migration/multifd.c
-+++ b/migration/multifd.c
-@@ -XXX,XX +XXX,XX @@ static void multifd_new_send_channel_create(gpointer opaque)
- int multifd_save_setup(Error **errp)
- {
-     int thread_count;
--    uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
-+    uint32_t page_count = migrate_multifd_packet_size();
-     uint8_t i;
-     const char *dsa_parameter = migrate_multifd_dsa_accel();
-@@ -XXX,XX +XXX,XX @@ static void *multifd_recv_thread(void *opaque)
- int multifd_load_setup(Error **errp)
- {
-     int thread_count;
--    uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
-+    uint32_t page_count = migrate_multifd_packet_size();
-     uint8_t i;
-     const char *dsa_parameter = migrate_multifd_dsa_accel();
-diff --git a/migration/multifd.h b/migration/multifd.h
-index XXXXXXX..XXXXXXX 100644
---- a/migration/multifd.h
-+++ b/migration/multifd.h
-@@ -XXX,XX +XXX,XX @@ int multifd_queue_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset);
- #define MULTIFD_FLAG_ZLIB (1 << 1)
- #define MULTIFD_FLAG_ZSTD (2 << 1)
--/* This value needs to be a multiple of qemu_target_page_size() */
--#define MULTIFD_PACKET_SIZE (512 * 1024)
--
- typedef struct {
-     uint32_t magic;
-     uint32_t version;
---
-.30.2

v2
* Rebase on top of 3e01f1147a16ca566694b97eafc941d62fa1e8d8.
* Leave Juan's changes in their original form instead of squashing them.
* Add a new commit to refactor the multifd_send_thread function to prepare for introducing the DSA offload functionality.
* Use page count to configure multifd-packet-size option.
* Don't use the FLAKY flag in DSA tests.
* Test if DSA integration test is setup correctly and skip the test if
* not.
* Fixed broken link in the previous patch cover.

* Background:

I posted an RFC about DSA offloading in QEMU:
https://patchew.org/QEMU/20230529182001.2232069-1-hao.xiang@bytedance.com/

This patchset implements the DSA offloading on zero page checking in
multifd live migration code path.

* Overview:

Intel Data Streaming Accelerator(DSA) is introduced in Intel's 4th generation
Xeon server, aka Sapphire Rapids.
https://cdrdv2-public.intel.com/671116/341204-intel-data-streaming-accelerator-spec.pdf
https://www.intel.com/content/www/us/en/content-details/759709/intel-data-streaming-accelerator-user-guide.html
One of the things DSA can do is to offload memory comparison workload from
CPU to DSA accelerator hardware. This patchset implements a solution to offload
QEMU's zero page checking from CPU to DSA accelerator hardware. We gain
two benefits from this change:
1. Reduces CPU usage in multifd live migration workflow across all use
cases.
2. Reduces migration total time in some use cases.

* Design:

These are the logical steps to perform DSA offloading:
1. Configure DSA accelerators and create user space openable DSA work
queues via the idxd driver.
2. Map DSA's work queue into a user space address space.
3. Fill an in-memory task descriptor to describe the memory operation.
4. Use dedicated CPU instruction _enqcmd to queue a task descriptor to
the work queue.
5. Pull the task descriptor's completion status field until the task
completes.
6. Check return status.

The memory operation is now totally done by the accelerator hardware but
the new workflow introduces overheads. The overhead is the extra cost CPU
prepares and submits the task descriptors and the extra cost CPU pulls for
completion. The design is around minimizing these two overheads.

1. In order to reduce the overhead on task preparation and submission,
we use batch descriptors. A batch descriptor will contain N individual
zero page checking tasks where the default N is 128 (default packet size
/ page size) and we can increase N by setting the packet size via a new
migration option.
2. The multifd sender threads prepares and submits batch tasks to DSA
hardware and it waits on a synchronization object for task completion.
Whenever a DSA task is submitted, the task structure is added to a
thread safe queue. It's safe to have multiple multifd sender threads to
submit tasks concurrently.
3. Multiple DSA hardware devices can be used. During multifd initialization,
every sender thread will be assigned a DSA device to work with. We
use a round-robin scheme to evenly distribute the work across all used
DSA devices.
4. Use a dedicated thread dsa_completion to perform busy pulling for all
DSA task completions. The thread keeps dequeuing DSA tasks from the
thread safe queue. The thread blocks when there is no outstanding DSA
task. When pulling for completion of a DSA task, the thread uses CPU
instruction _mm_pause between the iterations of a busy loop to save some
CPU power as well as optimizing core resources for the other hypercore.
5. DSA accelerator can encounter errors. The most popular error is a
page fault. We have tested using devices to handle page faults but
performance is bad. Right now, if DSA hits a page fault, we fallback to
use CPU to complete the rest of the work. The CPU fallback is done in
the multifd sender thread.
6. Added a new migration option multifd-dsa-accel to set the DSA device
path. If set, the multifd workflow will leverage the DSA devices for
offloading.
7. Added a new migration option multifd-normal-page-ratio to make
multifd live migration easier to test. Setting a normal page ratio will
make live migration recognize a zero page as a normal page and send
the entire payload over the network. If we want to send a large network
payload and analyze throughput, this option is useful.
8. Added a new migration option multifd-packet-size. This can increase
the number of pages being zero page checked and sent over the network.
The extra synchronization between the sender threads and the dsa
completion thread is an overhead. Using a large packet size can reduce
that overhead.

* Performance:

We use two Intel 4th generation Xeon servers for testing.

Architecture:        x86_64
CPU(s):              192
Thread(s) per core:  2
Core(s) per socket:  48
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               143
Model name:          Intel(R) Xeon(R) Platinum 8457C
Stepping:            8
CPU MHz:             2538.624
CPU max MHz:         3800.0000
CPU min MHz:         800.0000

We perform multifd live migration with below setup:
1. VM has 100GB memory. 
2. Use the new migration option multifd-set-normal-page-ratio to control the total
size of the payload sent over the network.
3. Use 8 multifd channels.
4. Use tcp for live migration.
4. Use CPU to perform zero page checking as the baseline.
5. Use one DSA device to offload zero page checking to compare with the baseline.
6. Use "perf sched record" and "perf sched timehist" to analyze CPU usage.

A) Scenario 1: 50% (50GB) normal pages on an 100GB vm.

CPU usage

|---------------|---------------|---------------|---------------|
	|		|comm		|runtime(msec)	|totaltime(msec)|
	|---------------|---------------|---------------|---------------|
	|Baseline	|live_migration	|5657.58	|		|
	|		|multifdsend_0	|3931.563	|		|
	|		|multifdsend_1	|4405.273	|		|
	|		|multifdsend_2	|3941.968	|		|
	|		|multifdsend_3	|5032.975	|		|
	|		|multifdsend_4	|4533.865	|		|
	|		|multifdsend_5	|4530.461	|		|
	|		|multifdsend_6	|5171.916	|		|
	|		|multifdsend_7	|4722.769	|41922		|
	|---------------|---------------|---------------|---------------|
	|DSA		|live_migration	|6129.168	|		|
	|		|multifdsend_0	|2954.717	|		|
	|		|multifdsend_1	|2766.359	|		|
	|		|multifdsend_2	|2853.519	|		|
	|		|multifdsend_3	|2740.717	|		|
	|		|multifdsend_4	|2824.169	|		|
	|		|multifdsend_5	|2966.908	|		|
	|		|multifdsend_6	|2611.137	|		|
	|		|multifdsend_7	|3114.732	|		|
	|		|dsa_completion	|3612.564	|32568		|
	|---------------|---------------|---------------|---------------|

Baseline total runtime is calculated by adding up all multifdsend_X
and live_migration threads runtime. DSA offloading total runtime is
calculated by adding up all multifdsend_X, live_migration and
dsa_completion threads runtime. 41922 msec VS 32568 msec runtime and
that is 23% total CPU usage savings.

Latency
	|---------------|---------------|---------------|---------------|---------------|---------------|
	|		|total time	|down time	|throughput	|transferred-ram|total-ram	|
	|---------------|---------------|---------------|---------------|---------------|---------------|	
	|Baseline	|10343 ms	|161 ms		|41007.00 mbps	|51583797 kb	|102400520 kb	|
	|---------------|---------------|---------------|---------------|-------------------------------|
	|DSA offload	|9535 ms	|135 ms		|46554.40 mbps	|53947545 kb	|102400520 kb	|	
	|---------------|---------------|---------------|---------------|---------------|---------------|

Total time is 8% faster and down time is 16% faster.

B) Scenario 2: 100% (100GB) zero pages on an 100GB vm.

CPU usage
	|---------------|---------------|---------------|---------------|
	|		|comm		|runtime(msec)	|totaltime(msec)|
	|---------------|---------------|---------------|---------------|
	|Baseline	|live_migration	|4860.718	|		|
	|	 	|multifdsend_0	|748.875	|		|
	|		|multifdsend_1	|898.498	|		|
	|		|multifdsend_2	|787.456	|		|
	|		|multifdsend_3	|764.537	|		|
	|		|multifdsend_4	|785.687	|		|
	|		|multifdsend_5	|756.941	|		|
	|		|multifdsend_6	|774.084	|		|
	|		|multifdsend_7	|782.900	|11154		|
	|---------------|---------------|-------------------------------|
	|DSA offloading	|live_migration	|3846.976	|		|
	|		|multifdsend_0	|191.880	|		|
	|		|multifdsend_1	|166.331	|		|
	|		|multifdsend_2	|168.528	|		|
	|		|multifdsend_3	|197.831	|		|
	|		|multifdsend_4	|169.580	|		|
	|		|multifdsend_5	|167.984	|		|
	|		|multifdsend_6	|198.042	|		|
	|		|multifdsend_7	|170.624	|		|
	|		|dsa_completion	|3428.669	|8700		|
	|---------------|---------------|---------------|---------------|

Baseline total runtime is 11154 msec and DSA offloading total runtime is
8700 msec. That is 22% CPU savings.

Latency
	|--------------------------------------------------------------------------------------------|
	|		|total time	|down time	|throughput	|transferred-ram|total-ram   |
	|---------------|---------------|---------------|---------------|---------------|------------|	
	|Baseline	|4867 ms	|20 ms		|1.51 mbps	|565 kb		|102400520 kb|
	|---------------|---------------|---------------|---------------|----------------------------|
	|DSA offload	|3888 ms	|18 ms		|1.89 mbps	|565 kb		|102400520 kb|	
	|---------------|---------------|---------------|---------------|---------------|------------|

Total time 20% faster and down time 10% faster.

* Testing:

1. Added unit tests for cover the added code path in dsa.c
2. Added integration tests to cover multifd live migration using DSA
offloading.

* Patchset

Apply this patchset on top of commit
f78ea7ddb0e18766ece9fdfe02061744a7afc41b

Hao Xiang (16):
  meson: Introduce new instruction set enqcmd to the build system.
  util/dsa: Add dependency idxd.
  util/dsa: Implement DSA device start and stop logic.
  util/dsa: Implement DSA task enqueue and dequeue.
  util/dsa: Implement DSA task asynchronous completion thread model.
  util/dsa: Implement zero page checking in DSA task.
  util/dsa: Implement DSA task asynchronous submission and wait for
    completion.
  migration/multifd: Add new migration option for multifd DSA
    offloading.
  migration/multifd: Prepare to introduce DSA acceleration on the
    multifd path.
  migration/multifd: Enable DSA offloading in multifd sender path.
  migration/multifd: Add test hook to set normal page ratio.
  migration/multifd: Enable set normal page ratio test hook in multifd.
  migration/multifd: Add migration option set packet size.
  migration/multifd: Enable set packet size migration option.
  util/dsa: Add unit test coverage for Intel DSA task submission and
    completion.
  migration/multifd: Add integration tests for multifd with Intel DSA
    offloading.

Juan Quintela (4):
  multifd: Add capability to enable/disable zero_page
  multifd: Support for zero pages transmission
  multifd: Zero pages transmission
  So we use multifd to transmit zero pages.

-- 
2.30.2

From: Juan Quintela <quintela@redhat.com>

We have to enable it by default until we introduce the new code.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/options.c | 13 +++++++++++++
 migration/options.h |  1 +
 qapi/migration.json |  8 +++++++-
 3 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/migration/options.c b/migration/options.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -XXX,XX +XXX,XX @@ Property migration_properties[] = {
     DEFINE_PROP_MIG_CAP("x-switchover-ack",
                         MIGRATION_CAPABILITY_SWITCHOVER_ACK),
     DEFINE_PROP_MIG_CAP("x-dirty-limit", MIGRATION_CAPABILITY_DIRTY_LIMIT),
+    DEFINE_PROP_MIG_CAP("main-zero-page",
+            MIGRATION_CAPABILITY_MAIN_ZERO_PAGE),
     DEFINE_PROP_END_OF_LIST(),
 };
 
@@ -XXX,XX +XXX,XX @@ bool migrate_multifd(void)
     return s->capabilities[MIGRATION_CAPABILITY_MULTIFD];
 }
 
+bool migrate_use_main_zero_page(void)
+{
+    //MigrationState *s;
+
+    //s = migrate_get_current();
+
+    // We will enable this when we add the right code.
+    // return s->enabled_capabilities[MIGRATION_CAPABILITY_MAIN_ZERO_PAGE];
+    return true;
+}
+
 bool migrate_pause_before_switchover(void)
 {
     MigrationState *s = migrate_get_current();
diff --git a/migration/options.h b/migration/options.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.h
+++ b/migration/options.h
@@ -XXX,XX +XXX,XX @@ int migrate_multifd_channels(void);
 MultiFDCompression migrate_multifd_compression(void);
 int migrate_multifd_zlib_level(void);
 int migrate_multifd_zstd_level(void);
+bool migrate_use_main_zero_page(void);
 uint8_t migrate_throttle_trigger_threshold(void);
 const char *migrate_tls_authz(void);
 const char *migrate_tls_creds(void);
diff --git a/qapi/migration.json b/qapi/migration.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -XXX,XX +XXX,XX @@
 #     and can result in more stable read performance.  Requires KVM
 #     with accelerator property "dirty-ring-size" set.  (Since 8.1)
 #
+#
+# @main-zero-page: If enabled, the detection of zero pages will be
+#                  done on the main thread.  Otherwise it is done on
+#                  the multifd threads.
+#                  (since 8.2)
+#
 # Features:
 #
 # @deprecated: Member @block is deprecated.  Use blockdev-mirror with
@@ -XXX,XX +XXX,XX @@
            { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
            'validate-uuid', 'background-snapshot',
            'zero-copy-send', 'postcopy-preempt', 'switchover-ack',
-           'dirty-limit'] }
+           'dirty-limit', 'main-zero-page'] }
 
 ##
 # @MigrationCapabilityStatus:
-- 
2.30.2

From: Juan Quintela <quintela@redhat.com>

This patch adds counters and similar. Logic will be added on the
following patch.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/multifd.c    | 37 ++++++++++++++++++++++++++++++-------
 migration/multifd.h    | 17 ++++++++++++++++-
 migration/trace-events |  8 ++++----
 3 files changed, 50 insertions(+), 12 deletions(-)

diff --git a/migration/multifd.c b/migration/multifd.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -XXX,XX +XXX,XX @@ static void multifd_send_fill_packet(MultiFDSendParams *p)
     packet->normal_pages = cpu_to_be32(p->normal_num);
     packet->next_packet_size = cpu_to_be32(p->next_packet_size);
     packet->packet_num = cpu_to_be64(p->packet_num);
+    packet->zero_pages = cpu_to_be32(p->zero_num);
 
     if (p->pages->block) {
         strncpy(packet->ramblock, p->pages->block->idstr, 256);
@@ -XXX,XX +XXX,XX @@ static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp)
     p->next_packet_size = be32_to_cpu(packet->next_packet_size);
     p->packet_num = be64_to_cpu(packet->packet_num);
 
-    if (p->normal_num == 0) {
+    p->zero_num = be32_to_cpu(packet->zero_pages);
+    if (p->zero_num > packet->pages_alloc - p->normal_num) {
+        error_setg(errp, "multifd: received packet "
+                   "with %u zero pages and expected maximum pages are %u",
+                   p->zero_num, packet->pages_alloc - p->normal_num) ;
+        return -1;
+    }
+
+    if (p->normal_num == 0 && p->zero_num == 0) {
         return 0;
     }
 
@@ -XXX,XX +XXX,XX @@ static int multifd_send_pages(QEMUFile *f)
     p->packet_num = multifd_send_state->packet_num++;
     multifd_send_state->pages = p->pages;
     p->pages = pages;
+
     qemu_mutex_unlock(&p->mutex);
     qemu_sem_post(&p->sem);
 
@@ -XXX,XX +XXX,XX @@ void multifd_save_cleanup(void)
         p->iov = NULL;
         g_free(p->normal);
         p->normal = NULL;
+        g_free(p->zero);
+        p->zero = NULL;
         multifd_send_state->ops->send_cleanup(p, &local_err);
         if (local_err) {
             migrate_set_error(migrate_get_current(), local_err);
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
             uint64_t packet_num = p->packet_num;
             uint32_t flags;
             p->normal_num = 0;
+            p->zero_num = 0;
 
             if (use_zero_copy_send) {
                 p->iovs_num = 0;
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
             p->flags = 0;
             p->num_packets++;
             p->total_normal_pages += p->normal_num;
+            p->total_zero_pages += p->zero_num;
             p->pages->num = 0;
             p->pages->block = NULL;
             qemu_mutex_unlock(&p->mutex);
 
-            trace_multifd_send(p->id, packet_num, p->normal_num, flags,
-                               p->next_packet_size);
+            trace_multifd_send(p->id, packet_num, p->normal_num, p->zero_num,
+                               flags, p->next_packet_size);
 
             if (use_zero_copy_send) {
                 /* Send header first, without zerocopy */
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
 
             stat64_add(&mig_stats.multifd_bytes,
                        p->next_packet_size + p->packet_len);
+            stat64_add(&mig_stats.normal_pages, p->normal_num);
+            stat64_add(&mig_stats.zero_pages, p->zero_num);
             p->next_packet_size = 0;
             qemu_mutex_lock(&p->mutex);
             p->pending_job--;
@@ -XXX,XX +XXX,XX @@ out:
 
     rcu_unregister_thread();
     migration_threads_remove(thread);
-    trace_multifd_send_thread_end(p->id, p->num_packets, p->total_normal_pages);
+    trace_multifd_send_thread_end(p->id, p->num_packets, p->total_normal_pages,
+                                  p->total_zero_pages);
 
     return NULL;
 }
@@ -XXX,XX +XXX,XX @@ int multifd_save_setup(Error **errp)
         p->normal = g_new0(ram_addr_t, page_count);
         p->page_size = qemu_target_page_size();
         p->page_count = page_count;
+        p->zero = g_new0(ram_addr_t, page_count);
 
         if (migrate_zero_copy_send()) {
             p->write_flags = QIO_CHANNEL_WRITE_FLAG_ZERO_COPY;
@@ -XXX,XX +XXX,XX @@ void multifd_load_cleanup(void)
         p->iov = NULL;
         g_free(p->normal);
         p->normal = NULL;
+        g_free(p->zero);
+        p->zero = NULL;
         multifd_recv_state->ops->recv_cleanup(p);
     }
     qemu_sem_destroy(&multifd_recv_state->sem_sync);
@@ -XXX,XX +XXX,XX @@ static void *multifd_recv_thread(void *opaque)
         flags = p->flags;
         /* recv methods don't know how to handle the SYNC flag */
         p->flags &= ~MULTIFD_FLAG_SYNC;
-        trace_multifd_recv(p->id, p->packet_num, p->normal_num, flags,
-                           p->next_packet_size);
+        trace_multifd_recv(p->id, p->packet_num, p->normal_num, p->zero_num,
+                           flags, p->next_packet_size);
         p->num_packets++;
         p->total_normal_pages += p->normal_num;
+        p->total_zero_pages += p->zero_num;
         qemu_mutex_unlock(&p->mutex);
 
         if (p->normal_num) {
@@ -XXX,XX +XXX,XX @@ static void *multifd_recv_thread(void *opaque)
     qemu_mutex_unlock(&p->mutex);
 
     rcu_unregister_thread();
-    trace_multifd_recv_thread_end(p->id, p->num_packets, p->total_normal_pages);
+    trace_multifd_recv_thread_end(p->id, p->num_packets, p->total_normal_pages,
+                                  p->total_zero_pages);
 
     return NULL;
 }
@@ -XXX,XX +XXX,XX @@ int multifd_load_setup(Error **errp)
         p->normal = g_new0(ram_addr_t, page_count);
         p->page_count = page_count;
         p->page_size = qemu_target_page_size();
+        p->zero = g_new0(ram_addr_t, page_count);
     }
 
     for (i = 0; i < thread_count; i++) {
diff --git a/migration/multifd.h b/migration/multifd.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -XXX,XX +XXX,XX @@ typedef struct {
     /* size of the next packet that contains pages */
     uint32_t next_packet_size;
     uint64_t packet_num;
-    uint64_t unused[4];    /* Reserved for future use */
+    /* zero pages */
+    uint32_t zero_pages;
+    uint32_t unused32[1];    /* Reserved for future use */
+    uint64_t unused64[3];    /* Reserved for future use */
     char ramblock[256];
     uint64_t offset[];
 } __attribute__((packed)) MultiFDPacket_t;
@@ -XXX,XX +XXX,XX @@ typedef struct {
     uint64_t num_packets;
     /* non zero pages sent through this channel */
     uint64_t total_normal_pages;
+    /* zero pages sent through this channel */
+    uint64_t total_zero_pages;
     /* buffers to send */
     struct iovec *iov;
     /* number of iovs used */
@@ -XXX,XX +XXX,XX @@ typedef struct {
     ram_addr_t *normal;
     /* num of non zero pages */
     uint32_t normal_num;
+    /* Pages that are zero */
+    ram_addr_t *zero;
+    /* num of zero pages */
+    uint32_t zero_num;
     /* used for compression methods */
     void *data;
 }  MultiFDSendParams;
@@ -XXX,XX +XXX,XX @@ typedef struct {
     uint8_t *host;
     /* non zero pages recv through this channel */
     uint64_t total_normal_pages;
+    /* zero pages recv through this channel */
+    uint64_t total_zero_pages;
     /* buffers to recv */
     struct iovec *iov;
     /* Pages that are not zero */
     ram_addr_t *normal;
     /* num of non zero pages */
     uint32_t normal_num;
+    /* Pages that are zero */
+    ram_addr_t *zero;
+    /* num of zero pages */
+    uint32_t zero_num;
     /* used for de-compression methods */
     void *data;
 } MultiFDRecvParams;
diff --git a/migration/trace-events b/migration/trace-events
index XXXXXXX..XXXXXXX 100644
--- a/migration/trace-events
+++ b/migration/trace-events
@@ -XXX,XX +XXX,XX @@ postcopy_preempt_reset_channel(void) ""
 # multifd.c
 multifd_new_send_channel_async(uint8_t id) "channel %u"
 multifd_new_send_channel_async_error(uint8_t id, void *err) "channel=%u err=%p"
-multifd_recv(uint8_t id, uint64_t packet_num, uint32_t used, uint32_t flags, uint32_t next_packet_size) "channel %u packet_num %" PRIu64 " pages %u flags 0x%x next packet size %u"
+multifd_recv(uint8_t id, uint64_t packet_num, uint32_t normal, uint32_t zero, uint32_t flags, uint32_t next_packet_size) "channel %u packet_num %" PRIu64 " normal pages %u zero pages %u flags 0x%x next packet size %u"
 multifd_recv_new_channel(uint8_t id) "channel %u"
 multifd_recv_sync_main(long packet_num) "packet num %ld"
 multifd_recv_sync_main_signal(uint8_t id) "channel %u"
 multifd_recv_sync_main_wait(uint8_t id) "channel %u"
 multifd_recv_terminate_threads(bool error) "error %d"
-multifd_recv_thread_end(uint8_t id, uint64_t packets, uint64_t pages) "channel %u packets %" PRIu64 " pages %" PRIu64
+multifd_recv_thread_end(uint8_t id, uint64_t packets, uint64_t normal_pages, uint64_t zero_pages) "channel %u packets %" PRIu64 " normal pages %" PRIu64 " zero pages %" PRIu64
 multifd_recv_thread_start(uint8_t id) "%u"
-multifd_send(uint8_t id, uint64_t packet_num, uint32_t normal, uint32_t flags, uint32_t next_packet_size) "channel %u packet_num %" PRIu64 " normal pages %u flags 0x%x next packet size %u"
+multifd_send(uint8_t id, uint64_t packet_num, uint32_t normalpages, uint32_t zero_pages, uint32_t flags, uint32_t next_packet_size) "channel %u packet_num %" PRIu64 " normal pages %u zero pages %u flags 0x%x next packet size %u"
 multifd_send_error(uint8_t id) "channel %u"
 multifd_send_sync_main(long packet_num) "packet num %ld"
 multifd_send_sync_main_signal(uint8_t id) "channel %u"
 multifd_send_sync_main_wait(uint8_t id) "channel %u"
 multifd_send_terminate_threads(bool error) "error %d"
-multifd_send_thread_end(uint8_t id, uint64_t packets, uint64_t normal_pages) "channel %u packets %" PRIu64 " normal pages %"  PRIu64
+multifd_send_thread_end(uint8_t id, uint64_t packets, uint64_t normal_pages, uint64_t zero_pages) "channel %u packets %" PRIu64 " normal pages %"  PRIu64 " zero pages %"  PRIu64
 multifd_send_thread_start(uint8_t id) "%u"
 multifd_tls_outgoing_handshake_start(void *ioc, void *tioc, const char *hostname) "ioc=%p tioc=%p hostname=%s"
 multifd_tls_outgoing_handshake_error(void *ioc, const char *err) "ioc=%p err=%s"
-- 
2.30.2

From: Juan Quintela <quintela@redhat.com>

This implements the zero page dection and handling.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/multifd.c | 41 +++++++++++++++++++++++++++++++++++++++--
 migration/multifd.h |  5 +++++
 2 files changed, 44 insertions(+), 2 deletions(-)

diff --git a/migration/multifd.c b/migration/multifd.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -XXX,XX +XXX,XX @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/cutils.h"
 #include "qemu/rcu.h"
 #include "exec/target_page.h"
 #include "sysemu/sysemu.h"
@@ -XXX,XX +XXX,XX @@ static void multifd_send_fill_packet(MultiFDSendParams *p)
 
         packet->offset[i] = cpu_to_be64(temp);
     }
+    for (i = 0; i < p->zero_num; i++) {
+        /* there are architectures where ram_addr_t is 32 bit */
+        uint64_t temp = p->zero[i];
+
+        packet->offset[p->normal_num + i] = cpu_to_be64(temp);
+    }
 }
 
 static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp)
@@ -XXX,XX +XXX,XX @@ static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp)
         p->normal[i] = offset;
     }
 
+    for (i = 0; i < p->zero_num; i++) {
+        uint64_t offset = be64_to_cpu(packet->offset[p->normal_num + i]);
+
+        if (offset > (p->block->used_length - p->page_size)) {
+            error_setg(errp, "multifd: offset too long %" PRIu64
+                       " (max " RAM_ADDR_FMT ")",
+                       offset, p->block->used_length);
+            return -1;
+        }
+        p->zero[i] = offset;
+    }
+
     return 0;
 }
 
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
     MultiFDSendParams *p = opaque;
     MigrationThread *thread = NULL;
     Error *local_err = NULL;
+    /* qemu older than 8.2 don't understand zero page on multifd channel */
+    bool use_zero_page = !migrate_use_main_zero_page();
     int ret = 0;
     bool use_zero_copy_send = migrate_zero_copy_send();
 
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
         qemu_mutex_lock(&p->mutex);
 
         if (p->pending_job) {
+            RAMBlock *rb = p->pages->block;
             uint64_t packet_num = p->packet_num;
             uint32_t flags;
             p->normal_num = 0;
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
             }
 
             for (int i = 0; i < p->pages->num; i++) {
-                p->normal[p->normal_num] = p->pages->offset[i];
-                p->normal_num++;
+                uint64_t offset = p->pages->offset[i];
+                if (use_zero_page &&
+                    buffer_is_zero(rb->host + offset, p->page_size)) {
+                    p->zero[p->zero_num] = offset;
+                    p->zero_num++;
+                    ram_release_page(rb->idstr, offset);
+                } else {
+                    p->normal[p->normal_num] = offset;
+                    p->normal_num++;
+                }
             }
 
             if (p->normal_num) {
@@ -XXX,XX +XXX,XX @@ static void *multifd_recv_thread(void *opaque)
             }
         }
 
+        for (int i = 0; i < p->zero_num; i++) {
+            void *page = p->host + p->zero[i];
+            if (!buffer_is_zero(page, p->page_size)) {
+                memset(page, 0, p->page_size);
+            }
+        }
+
         if (flags & MULTIFD_FLAG_SYNC) {
             qemu_sem_post(&multifd_recv_state->sem_sync);
             qemu_sem_wait(&p->sem_sync);
diff --git a/migration/multifd.h b/migration/multifd.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -XXX,XX +XXX,XX @@ typedef struct {
     uint32_t unused32[1];    /* Reserved for future use */
     uint64_t unused64[3];    /* Reserved for future use */
     char ramblock[256];
+    /*
+     * This array contains the pointers to:
+     *  - normal pages (initial normal_pages entries)
+     *  - zero pages (following zero_pages entries)
+     */
     uint64_t offset[];
 } __attribute__((packed)) MultiFDPacket_t;
 
-- 
2.30.2

From: Juan Quintela <quintela@redhat.com>

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Leonardo Bras <leobras@redhat.com>
---
 migration/multifd.c |  7 ++++---
 migration/options.c | 13 +++++++------
 migration/ram.c     | 45 ++++++++++++++++++++++++++++++++++++++-------
 qapi/migration.json |  1 -
 4 files changed, 49 insertions(+), 17 deletions(-)

diff --git a/migration/multifd.c b/migration/multifd.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -XXX,XX +XXX,XX @@
 #include "qemu/osdep.h"
 #include "qemu/cutils.h"
 #include "qemu/rcu.h"
+#include "qemu/cutils.h"
 #include "exec/target_page.h"
 #include "sysemu/sysemu.h"
 #include "exec/ramblock.h"
@@ -XXX,XX +XXX,XX @@ static int multifd_send_pages(QEMUFile *f)
     p->packet_num = multifd_send_state->packet_num++;
     multifd_send_state->pages = p->pages;
     p->pages = pages;
-
     qemu_mutex_unlock(&p->mutex);
     qemu_sem_post(&p->sem);
 
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
     MigrationThread *thread = NULL;
     Error *local_err = NULL;
     /* qemu older than 8.2 don't understand zero page on multifd channel */
-    bool use_zero_page = !migrate_use_main_zero_page();
+    bool use_multifd_zero_page = !migrate_use_main_zero_page();
     int ret = 0;
     bool use_zero_copy_send = migrate_zero_copy_send();
 
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
             RAMBlock *rb = p->pages->block;
             uint64_t packet_num = p->packet_num;
             uint32_t flags;
+
             p->normal_num = 0;
             p->zero_num = 0;
 
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
 
             for (int i = 0; i < p->pages->num; i++) {
                 uint64_t offset = p->pages->offset[i];
-                if (use_zero_page &&
+                if (use_multifd_zero_page &&
                     buffer_is_zero(rb->host + offset, p->page_size)) {
                     p->zero[p->zero_num] = offset;
                     p->zero_num++;
diff --git a/migration/options.c b/migration/options.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -XXX,XX +XXX,XX @@ Property migration_properties[] = {
     DEFINE_PROP_MIG_CAP("x-block", MIGRATION_CAPABILITY_BLOCK),
     DEFINE_PROP_MIG_CAP("x-return-path", MIGRATION_CAPABILITY_RETURN_PATH),
     DEFINE_PROP_MIG_CAP("x-multifd", MIGRATION_CAPABILITY_MULTIFD),
+    DEFINE_PROP_MIG_CAP("x-main-zero-page", MIGRATION_CAPABILITY_MAIN_ZERO_PAGE),
     DEFINE_PROP_MIG_CAP("x-background-snapshot",
             MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT),
 #ifdef CONFIG_LINUX
@@ -XXX,XX +XXX,XX @@ bool migrate_multifd(void)
 
 bool migrate_use_main_zero_page(void)
 {
-    //MigrationState *s;
-
-    //s = migrate_get_current();
+    MigrationState *s = migrate_get_current();
 
-    // We will enable this when we add the right code.
-    // return s->enabled_capabilities[MIGRATION_CAPABILITY_MAIN_ZERO_PAGE];
-    return true;
+    return s->capabilities[MIGRATION_CAPABILITY_MAIN_ZERO_PAGE];
 }
 
 bool migrate_pause_before_switchover(void)
@@ -XXX,XX +XXX,XX @@ INITIALIZE_MIGRATE_CAPS_SET(check_caps_background_snapshot,
     MIGRATION_CAPABILITY_LATE_BLOCK_ACTIVATE,
     MIGRATION_CAPABILITY_RETURN_PATH,
     MIGRATION_CAPABILITY_MULTIFD,
+    MIGRATION_CAPABILITY_MAIN_ZERO_PAGE,
     MIGRATION_CAPABILITY_PAUSE_BEFORE_SWITCHOVER,
     MIGRATION_CAPABILITY_AUTO_CONVERGE,
     MIGRATION_CAPABILITY_RELEASE_RAM,
@@ -XXX,XX +XXX,XX @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp)
             error_setg(errp, "Postcopy is not yet compatible with multifd");
             return false;
         }
+        if (new_caps[MIGRATION_CAPABILITY_MAIN_ZERO_PAGE]) {
+            error_setg(errp, "Postcopy is not yet compatible with main zero copy");
+        }
     }
 
     if (new_caps[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT]) {
diff --git a/migration/ram.c b/migration/ram.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -XXX,XX +XXX,XX @@ static int ram_save_target_page_legacy(RAMState *rs, PageSearchStatus *pss)
     if (save_zero_page(rs, pss, offset)) {
         return 1;
     }
-
     /*
-     * Do not use multifd in postcopy as one whole host page should be
-     * placed.  Meanwhile postcopy requires atomic update of pages, so even
-     * if host page size == guest page size the dest guest during run may
-     * still see partially copied pages which is data corruption.
+     * Do not use multifd for:
+     * 1. Compression as the first page in the new block should be posted out
+     *    before sending the compressed page
+     * 2. In postcopy as one whole host page should be placed
      */
-    if (migrate_multifd() && !migration_in_postcopy()) {
+    if (!migrate_compress() && migrate_multifd() && !migration_in_postcopy()) {
+        return ram_save_multifd_page(pss->pss_channel, block, offset);
+    }
+
+    return ram_save_page(rs, pss);
+}
+
+/**
+ * ram_save_target_page_multifd: save one target page
+ *
+ * Returns the number of pages written
+ *
+ * @rs: current RAM state
+ * @pss: data about the page we want to send
+ */
+static int ram_save_target_page_multifd(RAMState *rs, PageSearchStatus *pss)
+{
+    RAMBlock *block = pss->block;
+    ram_addr_t offset = ((ram_addr_t)pss->page) << TARGET_PAGE_BITS;
+    int res;
+
+    if (!migration_in_postcopy()) {
         return ram_save_multifd_page(pss->pss_channel, block, offset);
     }
 
+    res = save_zero_page(rs, pss, offset);
+    if (res > 0) {
+        return res;
+    }
+
     return ram_save_page(rs, pss);
 }
 
@@ -XXX,XX +XXX,XX @@ static int ram_save_setup(QEMUFile *f, void *opaque)
     }
 
     migration_ops = g_malloc0(sizeof(MigrationOps));
-    migration_ops->ram_save_target_page = ram_save_target_page_legacy;
+
+    if (migrate_multifd() && !migrate_use_main_zero_page()) {
+        migration_ops->ram_save_target_page = ram_save_target_page_multifd;
+    } else {
+        migration_ops->ram_save_target_page = ram_save_target_page_legacy;
+    }
 
     qemu_mutex_unlock_iothread();
+
     ret = multifd_send_sync_main(f);
     qemu_mutex_lock_iothread();
     if (ret < 0) {
diff --git a/qapi/migration.json b/qapi/migration.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -XXX,XX +XXX,XX @@
 #     and can result in more stable read performance.  Requires KVM
 #     with accelerator property "dirty-ring-size" set.  (Since 8.1)
 #
-#
 # @main-zero-page: If enabled, the detection of zero pages will be
 #                  done on the main thread.  Otherwise it is done on
 #                  the multifd threads.
-- 
2.30.2

Enable instruction set enqcmd in build.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 meson.build                   | 2 ++
 meson_options.txt             | 2 ++
 scripts/meson-buildoptions.sh | 3 +++
 3 files changed, 7 insertions(+)

diff --git a/meson.build b/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/meson.build
+++ b/meson.build
@@ -XXX,XX +XXX,XX @@ config_host_data.set('CONFIG_AVX512BW_OPT', get_option('avx512bw') \
     int main(int argc, char *argv[]) { return bar(argv[0]); }
   '''), error_message: 'AVX512BW not available').allowed())
 
+config_host_data.set('CONFIG_DSA_OPT', get_option('enqcmd'))
+
 # For both AArch64 and AArch32, detect if builtins are available.
 config_host_data.set('CONFIG_ARM_AES_BUILTIN', cc.compiles('''
     #include <arm_neon.h>
diff --git a/meson_options.txt b/meson_options.txt
index XXXXXXX..XXXXXXX 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -XXX,XX +XXX,XX @@ option('avx512f', type: 'feature', value: 'disabled',
        description: 'AVX512F optimizations')
 option('avx512bw', type: 'feature', value: 'auto',
        description: 'AVX512BW optimizations')
+option('enqcmd', type: 'boolean', value: false,
+       description: 'MENQCMD optimizations')
 option('keyring', type: 'feature', value: 'auto',
        description: 'Linux keyring support')
 option('libkeyutils', type: 'feature', value: 'auto',
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index XXXXXXX..XXXXXXX 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -XXX,XX +XXX,XX @@ meson_options_help() {
   printf "%s\n" '  avx2            AVX2 optimizations'
   printf "%s\n" '  avx512bw        AVX512BW optimizations'
   printf "%s\n" '  avx512f         AVX512F optimizations'
+  printf "%s\n" '  enqcmd          ENQCMD optimizations'
   printf "%s\n" '  blkio           libblkio block device driver'
   printf "%s\n" '  bochs           bochs image format support'
   printf "%s\n" '  bpf             eBPF support'
@@ -XXX,XX +XXX,XX @@ _meson_option_parse() {
     --disable-avx512bw) printf "%s" -Davx512bw=disabled ;;
     --enable-avx512f) printf "%s" -Davx512f=enabled ;;
     --disable-avx512f) printf "%s" -Davx512f=disabled ;;
+    --enable-enqcmd) printf "%s" -Denqcmd=true ;;
+    --disable-enqcmd) printf "%s" -Denqcmd=false ;;
     --enable-gcov) printf "%s" -Db_coverage=true ;;
     --disable-gcov) printf "%s" -Db_coverage=false ;;
     --enable-lto) printf "%s" -Db_lto=true ;;
-- 
2.30.2

Idxd is the device driver for DSA (Intel Data Streaming
Accelerator). The driver is fully functioning since Linux
kernel 5.19. This change adds the driver's header file used
for userspace development.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 linux-headers/linux/idxd.h | 356 +++++++++++++++++++++++++++++++++++++
 1 file changed, 356 insertions(+)
 create mode 100644 linux-headers/linux/idxd.h

diff --git a/linux-headers/linux/idxd.h b/linux-headers/linux/idxd.h
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/linux-headers/linux/idxd.h
@@ -XXX,XX +XXX,XX @@
+/* SPDX-License-Identifier: LGPL-2.1 WITH Linux-syscall-note */
+/* Copyright(c) 2019 Intel Corporation. All rights rsvd. */
+#ifndef _USR_IDXD_H_
+#define _USR_IDXD_H_
+
+#ifdef __KERNEL__
+#include <linux/types.h>
+#else
+#include <stdint.h>
+#endif
+
+/* Driver command error status */
+enum idxd_scmd_stat {
+	IDXD_SCMD_DEV_ENABLED = 0x80000010,
+	IDXD_SCMD_DEV_NOT_ENABLED = 0x80000020,
+	IDXD_SCMD_WQ_ENABLED = 0x80000021,
+	IDXD_SCMD_DEV_DMA_ERR = 0x80020000,
+	IDXD_SCMD_WQ_NO_GRP = 0x80030000,
+	IDXD_SCMD_WQ_NO_NAME = 0x80040000,
+	IDXD_SCMD_WQ_NO_SVM = 0x80050000,
+	IDXD_SCMD_WQ_NO_THRESH = 0x80060000,
+	IDXD_SCMD_WQ_PORTAL_ERR = 0x80070000,
+	IDXD_SCMD_WQ_RES_ALLOC_ERR = 0x80080000,
+	IDXD_SCMD_PERCPU_ERR = 0x80090000,
+	IDXD_SCMD_DMA_CHAN_ERR = 0x800a0000,
+	IDXD_SCMD_CDEV_ERR = 0x800b0000,
+	IDXD_SCMD_WQ_NO_SWQ_SUPPORT = 0x800c0000,
+	IDXD_SCMD_WQ_NONE_CONFIGURED = 0x800d0000,
+	IDXD_SCMD_WQ_NO_SIZE = 0x800e0000,
+	IDXD_SCMD_WQ_NO_PRIV = 0x800f0000,
+	IDXD_SCMD_WQ_IRQ_ERR = 0x80100000,
+	IDXD_SCMD_WQ_USER_NO_IOMMU = 0x80110000,
+};
+
+#define IDXD_SCMD_SOFTERR_MASK	0x80000000
+#define IDXD_SCMD_SOFTERR_SHIFT	16
+
+/* Descriptor flags */
+#define IDXD_OP_FLAG_FENCE	0x0001
+#define IDXD_OP_FLAG_BOF	0x0002
+#define IDXD_OP_FLAG_CRAV	0x0004
+#define IDXD_OP_FLAG_RCR	0x0008
+#define IDXD_OP_FLAG_RCI	0x0010
+#define IDXD_OP_FLAG_CRSTS	0x0020
+#define IDXD_OP_FLAG_CR		0x0080
+#define IDXD_OP_FLAG_CC		0x0100
+#define IDXD_OP_FLAG_ADDR1_TCS	0x0200
+#define IDXD_OP_FLAG_ADDR2_TCS	0x0400
+#define IDXD_OP_FLAG_ADDR3_TCS	0x0800
+#define IDXD_OP_FLAG_CR_TCS	0x1000
+#define IDXD_OP_FLAG_STORD	0x2000
+#define IDXD_OP_FLAG_DRDBK	0x4000
+#define IDXD_OP_FLAG_DSTS	0x8000
+
+/* IAX */
+#define IDXD_OP_FLAG_RD_SRC2_AECS	0x010000
+#define IDXD_OP_FLAG_RD_SRC2_2ND	0x020000
+#define IDXD_OP_FLAG_WR_SRC2_AECS_COMP	0x040000
+#define IDXD_OP_FLAG_WR_SRC2_AECS_OVFL	0x080000
+#define IDXD_OP_FLAG_SRC2_STS		0x100000
+#define IDXD_OP_FLAG_CRC_RFC3720	0x200000
+
+/* Opcode */
+enum dsa_opcode {
+	DSA_OPCODE_NOOP = 0,
+	DSA_OPCODE_BATCH,
+	DSA_OPCODE_DRAIN,
+	DSA_OPCODE_MEMMOVE,
+	DSA_OPCODE_MEMFILL,
+	DSA_OPCODE_COMPARE,
+	DSA_OPCODE_COMPVAL,
+	DSA_OPCODE_CR_DELTA,
+	DSA_OPCODE_AP_DELTA,
+	DSA_OPCODE_DUALCAST,
+	DSA_OPCODE_CRCGEN = 0x10,
+	DSA_OPCODE_COPY_CRC,
+	DSA_OPCODE_DIF_CHECK,
+	DSA_OPCODE_DIF_INS,
+	DSA_OPCODE_DIF_STRP,
+	DSA_OPCODE_DIF_UPDT,
+	DSA_OPCODE_CFLUSH = 0x20,
+};
+
+enum iax_opcode {
+	IAX_OPCODE_NOOP = 0,
+	IAX_OPCODE_DRAIN = 2,
+	IAX_OPCODE_MEMMOVE,
+	IAX_OPCODE_DECOMPRESS = 0x42,
+	IAX_OPCODE_COMPRESS,
+	IAX_OPCODE_CRC64,
+	IAX_OPCODE_ZERO_DECOMP_32 = 0x48,
+	IAX_OPCODE_ZERO_DECOMP_16,
+	IAX_OPCODE_ZERO_COMP_32 = 0x4c,
+	IAX_OPCODE_ZERO_COMP_16,
+	IAX_OPCODE_SCAN = 0x50,
+	IAX_OPCODE_SET_MEMBER,
+	IAX_OPCODE_EXTRACT,
+	IAX_OPCODE_SELECT,
+	IAX_OPCODE_RLE_BURST,
+	IAX_OPCODE_FIND_UNIQUE,
+	IAX_OPCODE_EXPAND,
+};
+
+/* Completion record status */
+enum dsa_completion_status {
+	DSA_COMP_NONE = 0,
+	DSA_COMP_SUCCESS,
+	DSA_COMP_SUCCESS_PRED,
+	DSA_COMP_PAGE_FAULT_NOBOF,
+	DSA_COMP_PAGE_FAULT_IR,
+	DSA_COMP_BATCH_FAIL,
+	DSA_COMP_BATCH_PAGE_FAULT,
+	DSA_COMP_DR_OFFSET_NOINC,
+	DSA_COMP_DR_OFFSET_ERANGE,
+	DSA_COMP_DIF_ERR,
+	DSA_COMP_BAD_OPCODE = 0x10,
+	DSA_COMP_INVALID_FLAGS,
+	DSA_COMP_NOZERO_RESERVE,
+	DSA_COMP_XFER_ERANGE,
+	DSA_COMP_DESC_CNT_ERANGE,
+	DSA_COMP_DR_ERANGE,
+	DSA_COMP_OVERLAP_BUFFERS,
+	DSA_COMP_DCAST_ERR,
+	DSA_COMP_DESCLIST_ALIGN,
+	DSA_COMP_INT_HANDLE_INVAL,
+	DSA_COMP_CRA_XLAT,
+	DSA_COMP_CRA_ALIGN,
+	DSA_COMP_ADDR_ALIGN,
+	DSA_COMP_PRIV_BAD,
+	DSA_COMP_TRAFFIC_CLASS_CONF,
+	DSA_COMP_PFAULT_RDBA,
+	DSA_COMP_HW_ERR1,
+	DSA_COMP_HW_ERR_DRB,
+	DSA_COMP_TRANSLATION_FAIL,
+};
+
+enum iax_completion_status {
+	IAX_COMP_NONE = 0,
+	IAX_COMP_SUCCESS,
+	IAX_COMP_PAGE_FAULT_IR = 0x04,
+	IAX_COMP_ANALYTICS_ERROR = 0x0a,
+	IAX_COMP_OUTBUF_OVERFLOW,
+	IAX_COMP_BAD_OPCODE = 0x10,
+	IAX_COMP_INVALID_FLAGS,
+	IAX_COMP_NOZERO_RESERVE,
+	IAX_COMP_INVALID_SIZE,
+	IAX_COMP_OVERLAP_BUFFERS = 0x16,
+	IAX_COMP_INT_HANDLE_INVAL = 0x19,
+	IAX_COMP_CRA_XLAT,
+	IAX_COMP_CRA_ALIGN,
+	IAX_COMP_ADDR_ALIGN,
+	IAX_COMP_PRIV_BAD,
+	IAX_COMP_TRAFFIC_CLASS_CONF,
+	IAX_COMP_PFAULT_RDBA,
+	IAX_COMP_HW_ERR1,
+	IAX_COMP_HW_ERR_DRB,
+	IAX_COMP_TRANSLATION_FAIL,
+	IAX_COMP_PRS_TIMEOUT,
+	IAX_COMP_WATCHDOG,
+	IAX_COMP_INVALID_COMP_FLAG = 0x30,
+	IAX_COMP_INVALID_FILTER_FLAG,
+	IAX_COMP_INVALID_INPUT_SIZE,
+	IAX_COMP_INVALID_NUM_ELEMS,
+	IAX_COMP_INVALID_SRC1_WIDTH,
+	IAX_COMP_INVALID_INVERT_OUT,
+};
+
+#define DSA_COMP_STATUS_MASK		0x7f
+#define DSA_COMP_STATUS_WRITE		0x80
+
+struct dsa_hw_desc {
+	uint32_t	pasid:20;
+	uint32_t	rsvd:11;
+	uint32_t	priv:1;
+	uint32_t	flags:24;
+	uint32_t	opcode:8;
+	uint64_t	completion_addr;
+	union {
+		uint64_t	src_addr;
+		uint64_t	rdback_addr;
+		uint64_t	pattern;
+		uint64_t	desc_list_addr;
+	};
+	union {
+		uint64_t	dst_addr;
+		uint64_t	rdback_addr2;
+		uint64_t	src2_addr;
+		uint64_t	comp_pattern;
+	};
+	union {
+		uint32_t	xfer_size;
+		uint32_t	desc_count;
+	};
+	uint16_t	int_handle;
+	uint16_t	rsvd1;
+	union {
+		uint8_t		expected_res;
+		/* create delta record */
+		struct {
+			uint64_t	delta_addr;
+			uint32_t	max_delta_size;
+			uint32_t 	delt_rsvd;
+			uint8_t 	expected_res_mask;
+		};
+		uint32_t	delta_rec_size;
+		uint64_t	dest2;
+		/* CRC */
+		struct {
+			uint32_t	crc_seed;
+			uint32_t	crc_rsvd;
+			uint64_t	seed_addr;
+		};
+		/* DIF check or strip */
+		struct {
+			uint8_t		src_dif_flags;
+			uint8_t		dif_chk_res;
+			uint8_t		dif_chk_flags;
+			uint8_t		dif_chk_res2[5];
+			uint32_t	chk_ref_tag_seed;
+			uint16_t	chk_app_tag_mask;
+			uint16_t	chk_app_tag_seed;
+		};
+		/* DIF insert */
+		struct {
+			uint8_t		dif_ins_res;
+			uint8_t		dest_dif_flag;
+			uint8_t		dif_ins_flags;
+			uint8_t		dif_ins_res2[13];
+			uint32_t	ins_ref_tag_seed;
+			uint16_t	ins_app_tag_mask;
+			uint16_t	ins_app_tag_seed;
+		};
+		/* DIF update */
+		struct {
+			uint8_t		src_upd_flags;
+			uint8_t		upd_dest_flags;
+			uint8_t		dif_upd_flags;
+			uint8_t		dif_upd_res[5];
+			uint32_t	src_ref_tag_seed;
+			uint16_t	src_app_tag_mask;
+			uint16_t	src_app_tag_seed;
+			uint32_t	dest_ref_tag_seed;
+			uint16_t	dest_app_tag_mask;
+			uint16_t	dest_app_tag_seed;
+		};
+
+		uint8_t		op_specific[24];
+	};
+} __attribute__((packed));
+
+struct iax_hw_desc {
+	uint32_t        pasid:20;
+	uint32_t        rsvd:11;
+	uint32_t        priv:1;
+	uint32_t        flags:24;
+	uint32_t        opcode:8;
+	uint64_t        completion_addr;
+	uint64_t        src1_addr;
+	uint64_t        dst_addr;
+	uint32_t        src1_size;
+	uint16_t        int_handle;
+	union {
+		uint16_t        compr_flags;
+		uint16_t        decompr_flags;
+	};
+	uint64_t        src2_addr;
+	uint32_t        max_dst_size;
+	uint32_t        src2_size;
+	uint32_t	filter_flags;
+	uint32_t	num_inputs;
+} __attribute__((packed));
+
+struct dsa_raw_desc {
+	uint64_t	field[8];
+} __attribute__((packed));
+
+/*
+ * The status field will be modified by hardware, therefore it should be
+ * volatile and prevent the compiler from optimize the read.
+ */
+struct dsa_completion_record {
+	volatile uint8_t	status;
+	union {
+		uint8_t		result;
+		uint8_t		dif_status;
+	};
+	uint16_t		rsvd;
+	uint32_t		bytes_completed;
+	uint64_t		fault_addr;
+	union {
+		/* common record */
+		struct {
+			uint32_t	invalid_flags:24;
+			uint32_t	rsvd2:8;
+		};
+
+		uint32_t	delta_rec_size;
+		uint64_t	crc_val;
+
+		/* DIF check & strip */
+		struct {
+			uint32_t	dif_chk_ref_tag;
+			uint16_t	dif_chk_app_tag_mask;
+			uint16_t	dif_chk_app_tag;
+		};
+
+		/* DIF insert */
+		struct {
+			uint64_t	dif_ins_res;
+			uint32_t	dif_ins_ref_tag;
+			uint16_t	dif_ins_app_tag_mask;
+			uint16_t	dif_ins_app_tag;
+		};
+
+		/* DIF update */
+		struct {
+			uint32_t	dif_upd_src_ref_tag;
+			uint16_t	dif_upd_src_app_tag_mask;
+			uint16_t	dif_upd_src_app_tag;
+			uint32_t	dif_upd_dest_ref_tag;
+			uint16_t	dif_upd_dest_app_tag_mask;
+			uint16_t	dif_upd_dest_app_tag;
+		};
+
+		uint8_t		op_specific[16];
+	};
+} __attribute__((packed));
+
+struct dsa_raw_completion_record {
+	uint64_t	field[4];
+} __attribute__((packed));
+
+struct iax_completion_record {
+	volatile uint8_t        status;
+	uint8_t                 error_code;
+	uint16_t                rsvd;
+	uint32_t                bytes_completed;
+	uint64_t                fault_addr;
+	uint32_t                invalid_flags;
+	uint32_t                rsvd2;
+	uint32_t                output_size;
+	uint8_t                 output_bits;
+	uint8_t                 rsvd3;
+	uint16_t                xor_csum;
+	uint32_t                crc;
+	uint32_t                min;
+	uint32_t                max;
+	uint32_t                sum;
+	uint64_t                rsvd4[2];
+} __attribute__((packed));
+
+struct iax_raw_completion_record {
+	uint64_t	field[8];
+} __attribute__((packed));
+
+#endif
-- 
2.30.2

* DSA device open and close.
* DSA group contains multiple DSA devices.
* DSA group configure/start/stop/clean.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
---
 include/qemu/dsa.h |  49 +++++++
 util/dsa.c         | 338 +++++++++++++++++++++++++++++++++++++++++++++
 util/meson.build   |   1 +
 3 files changed, 388 insertions(+)
 create mode 100644 include/qemu/dsa.h
 create mode 100644 util/dsa.c

diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@
+#ifndef QEMU_DSA_H
+#define QEMU_DSA_H
+
+#include "qemu/thread.h"
+#include "qemu/queue.h"
+
+#ifdef CONFIG_DSA_OPT
+
+#pragma GCC push_options
+#pragma GCC target("enqcmd")
+
+#include <linux/idxd.h>
+#include "x86intrin.h"
+
+#endif
+
+/**
+ * @brief Initializes DSA devices.
+ *
+ * @param dsa_parameter A list of DSA device path from migration parameter.
+ * @return int Zero if successful, otherwise non zero.
+ */
+int dsa_init(const char *dsa_parameter);
+
+/**
+ * @brief Start logic to enable using DSA.
+ */
+void dsa_start(void);
+
+/**
+ * @brief Stop logic to clean up DSA by halting the device group and cleaning up
+ * the completion thread.
+ */
+void dsa_stop(void);
+
+/**
+ * @brief Clean up system resources created for DSA offloading.
+ *        This function is called during QEMU process teardown.
+ */
+void dsa_cleanup(void);
+
+/**
+ * @brief Check if DSA is running.
+ *
+ * @return True if DSA is running, otherwise false.
+ */
+bool dsa_is_running(void);
+
+#endif
\ No newline at end of file
diff --git a/util/dsa.c b/util/dsa.c
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@
+/*
+ * Use Intel Data Streaming Accelerator to offload certain background
+ * operations.
+ *
+ * Copyright (c) 2023 Hao Xiang <hao.xiang@bytedance.com>
+ *                    Bryan Zhang <bryan.zhang@bytedance.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/queue.h"
+#include "qemu/memalign.h"
+#include "qemu/lockable.h"
+#include "qemu/cutils.h"
+#include "qemu/dsa.h"
+#include "qemu/bswap.h"
+#include "qemu/error-report.h"
+#include "qemu/rcu.h"
+
+#ifdef CONFIG_DSA_OPT
+
+#pragma GCC push_options
+#pragma GCC target("enqcmd")
+
+#include <linux/idxd.h>
+#include "x86intrin.h"
+
+#define DSA_WQ_SIZE 4096
+#define MAX_DSA_DEVICES 16
+
+typedef QSIMPLEQ_HEAD(dsa_task_queue, buffer_zero_batch_task) dsa_task_queue;
+
+struct dsa_device {
+    void *work_queue;
+};
+
+struct dsa_device_group {
+    struct dsa_device *dsa_devices;
+    int num_dsa_devices;
+    uint32_t index;
+    bool running;
+    QemuMutex task_queue_lock;
+    QemuCond task_queue_cond;
+    dsa_task_queue task_queue;
+};
+
+uint64_t max_retry_count;
+static struct dsa_device_group dsa_group;
+
+
+/**
+ * @brief This function opens a DSA device's work queue and
+ *        maps the DSA device memory into the current process.
+ *
+ * @param dsa_wq_path A pointer to the DSA device work queue's file path.
+ * @return A pointer to the mapped memory.
+ */
+static void *
+map_dsa_device(const char *dsa_wq_path)
+{
+    void *dsa_device;
+    int fd;
+
+    fd = open(dsa_wq_path, O_RDWR);
+    if (fd < 0) {
+        fprintf(stderr, "open %s failed with errno = %d.\n",
+                dsa_wq_path, errno);
+        return MAP_FAILED;
+    }
+    dsa_device = mmap(NULL, DSA_WQ_SIZE, PROT_WRITE,
+                      MAP_SHARED | MAP_POPULATE, fd, 0);
+    close(fd);
+    if (dsa_device == MAP_FAILED) {
+        fprintf(stderr, "mmap failed with errno = %d.\n", errno);
+        return MAP_FAILED;
+    }
+    return dsa_device;
+}
+
+/**
+ * @brief Initializes a DSA device structure.
+ *
+ * @param instance A pointer to the DSA device.
+ * @param work_queue  A pointer to the DSA work queue.
+ */
+static void
+dsa_device_init(struct dsa_device *instance,
+                void *dsa_work_queue)
+{
+    instance->work_queue = dsa_work_queue;
+}
+
+/**
+ * @brief Cleans up a DSA device structure.
+ *
+ * @param instance A pointer to the DSA device to cleanup.
+ */
+static void
+dsa_device_cleanup(struct dsa_device *instance)
+{
+    if (instance->work_queue != MAP_FAILED) {
+        munmap(instance->work_queue, DSA_WQ_SIZE);
+    }
+}
+
+/**
+ * @brief Initializes a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ * @param num_dsa_devices The number of DSA devices this group will have.
+ *
+ * @return Zero if successful, non-zero otherwise.
+ */
+static int
+dsa_device_group_init(struct dsa_device_group *group,
+                      const char *dsa_parameter)
+{
+    if (dsa_parameter == NULL || strlen(dsa_parameter) == 0) {
+        return 0;
+    }
+
+    int ret = 0;
+    char *local_dsa_parameter = g_strdup(dsa_parameter);
+    const char *dsa_path[MAX_DSA_DEVICES];
+    int num_dsa_devices = 0;
+    char delim[2] = " ";
+
+    char *current_dsa_path = strtok(local_dsa_parameter, delim);
+
+    while (current_dsa_path != NULL) {
+        dsa_path[num_dsa_devices++] = current_dsa_path;
+        if (num_dsa_devices == MAX_DSA_DEVICES) {
+            break;
+        }
+        current_dsa_path = strtok(NULL, delim);
+    }
+
+    group->dsa_devices =
+        malloc(sizeof(struct dsa_device) * num_dsa_devices);
+    group->num_dsa_devices = num_dsa_devices;
+    group->index = 0;
+
+    group->running = false;
+    qemu_mutex_init(&group->task_queue_lock);
+    qemu_cond_init(&group->task_queue_cond);
+    QSIMPLEQ_INIT(&group->task_queue);
+
+    void *dsa_wq = MAP_FAILED;
+    for (int i = 0; i < num_dsa_devices; i++) {
+        dsa_wq = map_dsa_device(dsa_path[i]);
+        if (dsa_wq == MAP_FAILED) {
+            fprintf(stderr, "map_dsa_device failed MAP_FAILED, "
+                    "using simulation.\n");
+            ret = -1;
+            goto exit;
+        }
+        dsa_device_init(&dsa_group.dsa_devices[i], dsa_wq);
+    }
+
+exit:
+    g_free(local_dsa_parameter);
+    return ret;
+}
+
+/**
+ * @brief Starts a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ * @param dsa_path An array of DSA device path.
+ * @param num_dsa_devices The number of DSA devices in the device group.
+ */
+static void
+dsa_device_group_start(struct dsa_device_group *group)
+{
+    group->running = true;
+}
+
+/**
+ * @brief Stops a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+__attribute__((unused))
+static void
+dsa_device_group_stop(struct dsa_device_group *group)
+{
+    group->running = false;
+}
+
+/**
+ * @brief Cleans up a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_device_group_cleanup(struct dsa_device_group *group)
+{
+    if (!group->dsa_devices) {
+        return;
+    }
+    for (int i = 0; i < group->num_dsa_devices; i++) {
+        dsa_device_cleanup(&group->dsa_devices[i]);
+    }
+    free(group->dsa_devices);
+    group->dsa_devices = NULL;
+
+    qemu_mutex_destroy(&group->task_queue_lock);
+    qemu_cond_destroy(&group->task_queue_cond);
+}
+
+/**
+ * @brief Returns the next available DSA device in the group.
+ *
+ * @param group A pointer to the DSA device group.
+ *
+ * @return struct dsa_device* A pointer to the next available DSA device
+ *         in the group.
+ */
+__attribute__((unused))
+static struct dsa_device *
+dsa_device_group_get_next_device(struct dsa_device_group *group)
+{
+    if (group->num_dsa_devices == 0) {
+        return NULL;
+    }
+    uint32_t current = qatomic_fetch_inc(&group->index);
+    current %= group->num_dsa_devices;
+    return &group->dsa_devices[current];
+}
+
+/**
+ * @brief Check if DSA is running.
+ *
+ * @return True if DSA is running, otherwise false.
+ */
+bool dsa_is_running(void)
+{
+    return false;
+}
+
+static void
+dsa_globals_init(void)
+{
+    max_retry_count = UINT64_MAX;
+}
+
+/**
+ * @brief Initializes DSA devices.
+ *
+ * @param dsa_parameter A list of DSA device path from migration parameter.
+ * @return int Zero if successful, otherwise non zero.
+ */
+int dsa_init(const char *dsa_parameter)
+{
+    dsa_globals_init();
+
+    return dsa_device_group_init(&dsa_group, dsa_parameter);
+}
+
+/**
+ * @brief Start logic to enable using DSA.
+ *
+ */
+void dsa_start(void)
+{
+    if (dsa_group.num_dsa_devices == 0) {
+        return;
+    }
+    if (dsa_group.running) {
+        return;
+    }
+    dsa_device_group_start(&dsa_group);
+}
+
+/**
+ * @brief Stop logic to clean up DSA by halting the device group and cleaning up
+ * the completion thread.
+ *
+ */
+void dsa_stop(void)
+{
+    struct dsa_device_group *group = &dsa_group;
+
+    if (!group->running) {
+        return;
+    }
+}
+
+/**
+ * @brief Clean up system resources created for DSA offloading.
+ *        This function is called during QEMU process teardown.
+ *
+ */
+void dsa_cleanup(void)
+{
+    dsa_stop();
+    dsa_device_group_cleanup(&dsa_group);
+}
+
+#else
+
+bool dsa_is_running(void)
+{
+    return false;
+}
+
+int dsa_init(const char *dsa_parameter)
+{
+    fprintf(stderr, "Intel Data Streaming Accelerator is not supported "
+                    "on this platform.\n");
+    return -1;
+}
+
+void dsa_start(void) {}
+
+void dsa_stop(void) {}
+
+void dsa_cleanup(void) {}
+
+#endif
+
diff --git a/util/meson.build b/util/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/util/meson.build
+++ b/util/meson.build
@@ -XXX,XX +XXX,XX @@ if have_block or have_ga
 endif
 if have_block
   util_ss.add(files('aio-wait.c'))
+  util_ss.add(files('dsa.c'))
   util_ss.add(files('buffer.c'))
   util_ss.add(files('bufferiszero.c'))
   util_ss.add(files('hbitmap.c'))
-- 
2.30.2

* Use a safe thread queue for DSA task enqueue/dequeue.
* Implement DSA task submission.
* Implement DSA batch task submission.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 include/qemu/dsa.h |  35 ++++++++
 util/dsa.c         | 196 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 231 insertions(+)

diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@
 #include <linux/idxd.h>
 #include "x86intrin.h"
 
+enum dsa_task_type {
+    DSA_TASK = 0,
+    DSA_BATCH_TASK
+};
+
+enum dsa_task_status {
+    DSA_TASK_READY = 0,
+    DSA_TASK_PROCESSING,
+    DSA_TASK_COMPLETION
+};
+
+typedef void (*buffer_zero_dsa_completion_fn)(void *);
+
+typedef struct buffer_zero_batch_task {
+    struct dsa_hw_desc batch_descriptor;
+    struct dsa_hw_desc *descriptors;
+    struct dsa_completion_record batch_completion __attribute__((aligned(32)));
+    struct dsa_completion_record *completions;
+    struct dsa_device_group *group;
+    struct dsa_device *device;
+    buffer_zero_dsa_completion_fn completion_callback;
+    QemuSemaphore sem_task_complete;
+    enum dsa_task_type task_type;
+    enum dsa_task_status status;
+    bool *results;
+    int batch_size;
+    QSIMPLEQ_ENTRY(buffer_zero_batch_task) entry;
+} buffer_zero_batch_task;
+
+#else
+
+struct buffer_zero_batch_task {
+    bool *results;
+};
+
 #endif
 
 /**
diff --git a/util/dsa.c b/util/dsa.c
index XXXXXXX..XXXXXXX 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@ dsa_device_group_get_next_device(struct dsa_device_group *group)
     return &group->dsa_devices[current];
 }
 
+/**
+ * @brief Empties out the DSA task queue.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_empty_task_queue(struct dsa_device_group *group)
+{
+    qemu_mutex_lock(&group->task_queue_lock);
+    dsa_task_queue *task_queue = &group->task_queue;
+    while (!QSIMPLEQ_EMPTY(task_queue)) {
+        QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
+    }
+    qemu_mutex_unlock(&group->task_queue_lock);
+}
+
+/**
+ * @brief Adds a task to the DSA task queue.
+ *
+ * @param group A pointer to the DSA device group.
+ * @param context A pointer to the DSA task to enqueue.
+ *
+ * @return int Zero if successful, otherwise a proper error code.
+ */
+static int
+dsa_task_enqueue(struct dsa_device_group *group,
+                 struct buffer_zero_batch_task *task)
+{
+    dsa_task_queue *task_queue = &group->task_queue;
+    QemuMutex *task_queue_lock = &group->task_queue_lock;
+    QemuCond *task_queue_cond = &group->task_queue_cond;
+
+    bool notify = false;
+
+    qemu_mutex_lock(task_queue_lock);
+
+    if (!group->running) {
+        fprintf(stderr, "DSA: Tried to queue task to stopped device queue\n");
+        qemu_mutex_unlock(task_queue_lock);
+        return -1;
+    }
+
+    // The queue is empty. This enqueue operation is a 0->1 transition.
+    if (QSIMPLEQ_EMPTY(task_queue))
+        notify = true;
+
+    QSIMPLEQ_INSERT_TAIL(task_queue, task, entry);
+
+    // We need to notify the waiter for 0->1 transitions.
+    if (notify)
+        qemu_cond_signal(task_queue_cond);
+
+    qemu_mutex_unlock(task_queue_lock);
+
+    return 0;
+}
+
+/**
+ * @brief Takes a DSA task out of the task queue.
+ *
+ * @param group A pointer to the DSA device group.
+ * @return buffer_zero_batch_task* The DSA task being dequeued.
+ */
+__attribute__((unused))
+static struct buffer_zero_batch_task *
+dsa_task_dequeue(struct dsa_device_group *group)
+{
+    struct buffer_zero_batch_task *task = NULL;
+    dsa_task_queue *task_queue = &group->task_queue;
+    QemuMutex *task_queue_lock = &group->task_queue_lock;
+    QemuCond *task_queue_cond = &group->task_queue_cond;
+
+    qemu_mutex_lock(task_queue_lock);
+
+    while (true) {
+        if (!group->running)
+            goto exit;
+        task = QSIMPLEQ_FIRST(task_queue);
+        if (task != NULL) {
+            break;
+        }
+        qemu_cond_wait(task_queue_cond, task_queue_lock);
+    }
+
+    QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
+
+exit:
+    qemu_mutex_unlock(task_queue_lock);
+    return task;
+}
+
+/**
+ * @brief Submits a DSA work item to the device work queue.
+ *
+ * @param wq A pointer to the DSA work queue's device memory.
+ * @param descriptor A pointer to the DSA work item descriptor.
+ *
+ * @return Zero if successful, non-zero otherwise.
+ */
+static int
+submit_wi_int(void *wq, struct dsa_hw_desc *descriptor)
+{
+    uint64_t retry = 0;
+
+    _mm_sfence();
+
+    while (true) {
+        if (_enqcmd(wq, descriptor) == 0) {
+            break;
+        }
+        retry++;
+        if (retry > max_retry_count) {
+            fprintf(stderr, "Submit work retry %lu times.\n", retry);
+            exit(1);
+        }
+    }
+
+    return 0;
+}
+
+/**
+ * @brief Synchronously submits a DSA work item to the
+ *        device work queue.
+ *
+ * @param wq A pointer to the DSA worjk queue's device memory.
+ * @param descriptor A pointer to the DSA work item descriptor.
+ *
+ * @return int Zero if successful, non-zero otherwise.
+ */
+__attribute__((unused))
+static int
+submit_wi(void *wq, struct dsa_hw_desc *descriptor)
+{
+    return submit_wi_int(wq, descriptor);
+}
+
+/**
+ * @brief Asynchronously submits a DSA work item to the
+ *        device work queue.
+ *
+ * @param task A pointer to the buffer zero task.
+ *
+ * @return int Zero if successful, non-zero otherwise.
+ */
+__attribute__((unused))
+static int
+submit_wi_async(struct buffer_zero_batch_task *task)
+{
+    struct dsa_device_group *device_group = task->group;
+    struct dsa_device *device_instance = task->device;
+    int ret;
+
+    assert(task->task_type == DSA_TASK);
+
+    task->status = DSA_TASK_PROCESSING;
+
+    ret = submit_wi_int(device_instance->work_queue,
+                        &task->descriptors[0]);
+    if (ret != 0)
+        return ret;
+
+    return dsa_task_enqueue(device_group, task);
+}
+
+/**
+ * @brief Asynchronously submits a DSA batch work item to the
+ *        device work queue.
+ *
+ * @param batch_task A pointer to the batch buffer zero task.
+ *
+ * @return int Zero if successful, non-zero otherwise.
+ */
+__attribute__((unused))
+static int
+submit_batch_wi_async(struct buffer_zero_batch_task *batch_task)
+{
+    struct dsa_device_group *device_group = batch_task->group;
+    struct dsa_device *device_instance = batch_task->device;
+    int ret;
+
+    assert(batch_task->task_type == DSA_BATCH_TASK);
+    assert(batch_task->batch_descriptor.desc_count <= batch_task->batch_size);
+    assert(batch_task->status == DSA_TASK_READY);
+
+    batch_task->status = DSA_TASK_PROCESSING;
+
+    ret = submit_wi_int(device_instance->work_queue,
+                        &batch_task->batch_descriptor);
+    if (ret != 0)
+        return ret;
+
+    return dsa_task_enqueue(device_group, batch_task);
+}
+
 /**
  * @brief Check if DSA is running.
  *
@@ -XXX,XX +XXX,XX @@ void dsa_stop(void)
     if (!group->running) {
         return;
     }
+
+    dsa_empty_task_queue(group);
 }
 
 /**
-- 
2.30.2

* Create a dedicated thread for DSA task completion.
* DSA completion thread runs a loop and poll for completed tasks.
* Start and stop DSA completion thread during DSA device start stop.

User space application can directly submit task to Intel DSA
accelerator by writing to DSA's device memory (mapped in user space).
Once a task is submitted, the device starts processing it and write
the completion status back to the task. A user space application can
poll the task's completion status to check for completion. This change
uses a dedicated thread to perform DSA task completion checking.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 util/dsa.c | 243 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 242 insertions(+), 1 deletion(-)

diff --git a/util/dsa.c b/util/dsa.c
index XXXXXXX..XXXXXXX 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@
 
 #define DSA_WQ_SIZE 4096
 #define MAX_DSA_DEVICES 16
+#define DSA_COMPLETION_THREAD "dsa_completion"
 
 typedef QSIMPLEQ_HEAD(dsa_task_queue, buffer_zero_batch_task) dsa_task_queue;
 
@@ -XXX,XX +XXX,XX @@ struct dsa_device_group {
     dsa_task_queue task_queue;
 };
 
+struct dsa_completion_thread {
+    bool stopping;
+    bool running;
+    QemuThread thread;
+    int thread_id;
+    QemuSemaphore sem_init_done;
+    struct dsa_device_group *group;
+};
+
 uint64_t max_retry_count;
 static struct dsa_device_group dsa_group;
+static struct dsa_completion_thread completion_thread;
 
 
 /**
@@ -XXX,XX +XXX,XX @@ submit_batch_wi_async(struct buffer_zero_batch_task *batch_task)
     return dsa_task_enqueue(device_group, batch_task);
 }
 
+/**
+ * @brief Poll for the DSA work item completion.
+ *
+ * @param completion A pointer to the DSA work item completion record.
+ * @param opcode The DSA opcode.
+ *
+ * @return Zero if successful, non-zero otherwise.
+ */
+static int
+poll_completion(struct dsa_completion_record *completion,
+                enum dsa_opcode opcode)
+{
+    uint8_t status;
+    uint64_t retry = 0;
+
+    while (true) {
+        // The DSA operation completes successfully or fails.
+        status = completion->status;
+        if (status == DSA_COMP_SUCCESS ||
+            status == DSA_COMP_PAGE_FAULT_NOBOF ||
+            status == DSA_COMP_BATCH_PAGE_FAULT ||
+            status == DSA_COMP_BATCH_FAIL) {
+            break;
+        } else if (status != DSA_COMP_NONE) {
+            /* TODO: Error handling here on unexpected failure. */
+            fprintf(stderr, "DSA opcode %d failed with status = %d.\n",
+                    opcode, status);
+            exit(1);
+        }
+        retry++;
+        if (retry > max_retry_count) {
+            fprintf(stderr, "Wait for completion retry %lu times.\n", retry);
+            exit(1);
+        }
+        _mm_pause();
+    }
+
+    return 0;
+}
+
+/**
+ * @brief Complete a single DSA task in the batch task.
+ *
+ * @param task A pointer to the batch task structure.
+ */
+static void
+poll_task_completion(struct buffer_zero_batch_task *task)
+{
+    assert(task->task_type == DSA_TASK);
+
+    struct dsa_completion_record *completion = &task->completions[0];
+    uint8_t status;
+
+    poll_completion(completion, task->descriptors[0].opcode);
+
+    status = completion->status;
+    if (status == DSA_COMP_SUCCESS) {
+        task->results[0] = (completion->result == 0);
+        return;
+    }
+
+    assert(status == DSA_COMP_PAGE_FAULT_NOBOF);
+}
+
+/**
+ * @brief Poll a batch task status until it completes. If DSA task doesn't
+ *        complete properly, use CPU to complete the task.
+ *
+ * @param batch_task A pointer to the DSA batch task.
+ */
+static void
+poll_batch_task_completion(struct buffer_zero_batch_task *batch_task)
+{
+    struct dsa_completion_record *batch_completion = &batch_task->batch_completion;
+    struct dsa_completion_record *completion;
+    uint8_t batch_status;
+    uint8_t status;
+    bool *results = batch_task->results;
+    uint32_t count = batch_task->batch_descriptor.desc_count;
+
+    poll_completion(batch_completion,
+                    batch_task->batch_descriptor.opcode);
+
+    batch_status = batch_completion->status;
+
+    if (batch_status == DSA_COMP_SUCCESS) {
+        if (batch_completion->bytes_completed == count) {
+            // Let's skip checking for each descriptors' completion status
+            // if the batch descriptor says all succedded.
+            for (int i = 0; i < count; i++) {
+                assert(batch_task->completions[i].status == DSA_COMP_SUCCESS);
+                results[i] = (batch_task->completions[i].result == 0);
+            }
+            return;
+        }
+    } else {
+        assert(batch_status == DSA_COMP_BATCH_FAIL ||
+            batch_status == DSA_COMP_BATCH_PAGE_FAULT);
+    }
+
+    for (int i = 0; i < count; i++) {
+
+        completion = &batch_task->completions[i];
+        status = completion->status;
+
+        if (status == DSA_COMP_SUCCESS) {
+            results[i] = (completion->result == 0);
+            continue;
+        }
+
+        if (status != DSA_COMP_PAGE_FAULT_NOBOF) {
+            fprintf(stderr,
+                    "Unexpected completion status = %u.\n", status);
+            assert(false);
+        }
+    }
+}
+
+/**
+ * @brief Handles an asynchronous DSA batch task completion.
+ *
+ * @param task A pointer to the batch buffer zero task structure.
+ */
+static void
+dsa_batch_task_complete(struct buffer_zero_batch_task *batch_task)
+{
+    batch_task->status = DSA_TASK_COMPLETION;
+    batch_task->completion_callback(batch_task);
+}
+
+/**
+ * @brief The function entry point called by a dedicated DSA
+ *        work item completion thread.
+ *
+ * @param opaque A pointer to the thread context.
+ *
+ * @return void* Not used.
+ */
+static void *
+dsa_completion_loop(void *opaque)
+{
+    struct dsa_completion_thread *thread_context =
+        (struct dsa_completion_thread *)opaque;
+    struct buffer_zero_batch_task *batch_task;
+    struct dsa_device_group *group = thread_context->group;
+
+    rcu_register_thread();
+
+    thread_context->thread_id = qemu_get_thread_id();
+    qemu_sem_post(&thread_context->sem_init_done);
+
+    while (thread_context->running) {
+        batch_task = dsa_task_dequeue(group);
+        assert(batch_task != NULL || !group->running);
+        if (!group->running) {
+            assert(!thread_context->running);
+            break;
+        }
+        if (batch_task->task_type == DSA_TASK) {
+            poll_task_completion(batch_task);
+        } else {
+            assert(batch_task->task_type == DSA_BATCH_TASK);
+            poll_batch_task_completion(batch_task);
+        }
+
+        dsa_batch_task_complete(batch_task);
+    }
+
+    rcu_unregister_thread();
+    return NULL;
+}
+
+/**
+ * @brief Initializes a DSA completion thread.
+ *
+ * @param completion_thread A pointer to the completion thread context.
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_completion_thread_init(
+    struct dsa_completion_thread *completion_thread,
+    struct dsa_device_group *group)
+{
+    completion_thread->stopping = false;
+    completion_thread->running = true;
+    completion_thread->thread_id = -1;
+    qemu_sem_init(&completion_thread->sem_init_done, 0);
+    completion_thread->group = group;
+
+    qemu_thread_create(&completion_thread->thread,
+                       DSA_COMPLETION_THREAD,
+                       dsa_completion_loop,
+                       completion_thread,
+                       QEMU_THREAD_JOINABLE);
+
+    /* Wait for initialization to complete */
+    while (completion_thread->thread_id == -1) {
+        qemu_sem_wait(&completion_thread->sem_init_done);
+    }
+}
+
+/**
+ * @brief Stops the completion thread (and implicitly, the device group).
+ *
+ * @param opaque A pointer to the completion thread.
+ */
+static void dsa_completion_thread_stop(void *opaque)
+{
+    struct dsa_completion_thread *thread_context =
+        (struct dsa_completion_thread *)opaque;
+
+    struct dsa_device_group *group = thread_context->group;
+
+    qemu_mutex_lock(&group->task_queue_lock);
+
+    thread_context->stopping = true;
+    thread_context->running = false;
+
+    dsa_device_group_stop(group);
+
+    qemu_cond_signal(&group->task_queue_cond);
+    qemu_mutex_unlock(&group->task_queue_lock);
+
+    qemu_thread_join(&thread_context->thread);
+
+    qemu_sem_destroy(&thread_context->sem_init_done);
+}
+
 /**
  * @brief Check if DSA is running.
  *
@@ -XXX,XX +XXX,XX @@ submit_batch_wi_async(struct buffer_zero_batch_task *batch_task)
  */
 bool dsa_is_running(void)
 {
-    return false;
+    return completion_thread.running;
 }
 
 static void
@@ -XXX,XX +XXX,XX @@ void dsa_start(void)
         return;
     }
     dsa_device_group_start(&dsa_group);
+    dsa_completion_thread_init(&completion_thread, &dsa_group);
 }
 
 /**
@@ -XXX,XX +XXX,XX @@ void dsa_stop(void)
         return;
     }
 
+    dsa_completion_thread_stop(&completion_thread);
     dsa_empty_task_queue(group);
 }
 
-- 
2.30.2

Create DSA task with operation code DSA_OPCODE_COMPVAL.
Here we create two types of DSA tasks, a single DSA task and
a batch DSA task. Batch DSA task reduces task submission overhead
and hence should be the default option. However, due to the way DSA
hardware works, a DSA batch task must contain at least two individual
tasks. There are times we need to submit a single task and hence a
single DSA task submission is also required.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
---
 include/qemu/dsa.h |  16 +++
 util/dsa.c         | 252 +++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 247 insertions(+), 21 deletions(-)

diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@ struct buffer_zero_batch_task {
 
 #endif
 
+/**
+ * @brief Initializes a buffer zero batch task.
+ *
+ * @param task A pointer to the batch task to initialize.
+ * @param batch_size The number of DSA tasks in the batch.
+ */
+void buffer_zero_batch_task_init(struct buffer_zero_batch_task *task,
+                                 int batch_size);
+
+/**
+ * @brief Performs the proper cleanup on a DSA batch task.
+ *
+ * @param task A pointer to the batch task to cleanup.
+ */
+void buffer_zero_batch_task_destroy(struct buffer_zero_batch_task *task);
+
 /**
  * @brief Initializes DSA devices.
  *
diff --git a/util/dsa.c b/util/dsa.c
index XXXXXXX..XXXXXXX 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@ uint64_t max_retry_count;
 static struct dsa_device_group dsa_group;
 static struct dsa_completion_thread completion_thread;
 
+static void buffer_zero_dsa_completion(void *context);
 
 /**
  * @brief This function opens a DSA device's work queue and
@@ -XXX,XX +XXX,XX @@ dsa_device_group_start(struct dsa_device_group *group)
  *
  * @param group A pointer to the DSA device group.
  */
-__attribute__((unused))
 static void
 dsa_device_group_stop(struct dsa_device_group *group)
 {
@@ -XXX,XX +XXX,XX @@ dsa_device_group_cleanup(struct dsa_device_group *group)
  * @return struct dsa_device* A pointer to the next available DSA device
  *         in the group.
  */
-__attribute__((unused))
 static struct dsa_device *
 dsa_device_group_get_next_device(struct dsa_device_group *group)
 {
@@ -XXX,XX +XXX,XX @@ dsa_task_enqueue(struct dsa_device_group *group,
  * @param group A pointer to the DSA device group.
  * @return buffer_zero_batch_task* The DSA task being dequeued.
  */
-__attribute__((unused))
 static struct buffer_zero_batch_task *
 dsa_task_dequeue(struct dsa_device_group *group)
 {
@@ -XXX,XX +XXX,XX @@ submit_wi_int(void *wq, struct dsa_hw_desc *descriptor)
     return 0;
 }
 
-/**
- * @brief Synchronously submits a DSA work item to the
- *        device work queue.
- *
- * @param wq A pointer to the DSA worjk queue's device memory.
- * @param descriptor A pointer to the DSA work item descriptor.
- *
- * @return int Zero if successful, non-zero otherwise.
- */
-__attribute__((unused))
-static int
-submit_wi(void *wq, struct dsa_hw_desc *descriptor)
-{
-    return submit_wi_int(wq, descriptor);
-}
-
 /**
  * @brief Asynchronously submits a DSA work item to the
  *        device work queue.
@@ -XXX,XX +XXX,XX @@ submit_wi(void *wq, struct dsa_hw_desc *descriptor)
  *
  * @return int Zero if successful, non-zero otherwise.
  */
-__attribute__((unused))
 static int
 submit_wi_async(struct buffer_zero_batch_task *task)
 {
@@ -XXX,XX +XXX,XX @@ submit_wi_async(struct buffer_zero_batch_task *task)
  *
  * @return int Zero if successful, non-zero otherwise.
  */
-__attribute__((unused))
 static int
 submit_batch_wi_async(struct buffer_zero_batch_task *batch_task)
 {
@@ -XXX,XX +XXX,XX @@ static void dsa_completion_thread_stop(void *opaque)
     qemu_sem_destroy(&thread_context->sem_init_done);
 }
 
+/**
+ * @brief Initializes a buffer zero comparison DSA task.
+ *
+ * @param descriptor A pointer to the DSA task descriptor.
+ * @param completion A pointer to the DSA task completion record.
+ */
+static void
+buffer_zero_task_init_int(struct dsa_hw_desc *descriptor,
+                          struct dsa_completion_record *completion)
+{
+    descriptor->opcode = DSA_OPCODE_COMPVAL;
+    descriptor->flags = IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CRAV;
+    descriptor->comp_pattern = (uint64_t)0;
+    descriptor->completion_addr = (uint64_t)completion;
+}
+
+/**
+ * @brief Initializes a buffer zero batch task.
+ *
+ * @param task A pointer to the batch task to initialize.
+ * @param batch_size The number of DSA tasks in the batch.
+ */
+void
+buffer_zero_batch_task_init(struct buffer_zero_batch_task *task,
+                            int batch_size)
+{
+    int descriptors_size = sizeof(*task->descriptors) * batch_size;
+    memset(task, 0, sizeof(*task));
+
+    task->descriptors =
+        (struct dsa_hw_desc *)qemu_memalign(64, descriptors_size);
+    memset(task->descriptors, 0, descriptors_size);
+    task->completions = (struct dsa_completion_record *)qemu_memalign(
+        32, sizeof(*task->completions) * batch_size);
+    task->results = g_new0(bool, batch_size);
+    task->batch_size = batch_size;
+
+    task->batch_completion.status = DSA_COMP_NONE;
+    task->batch_descriptor.completion_addr = (uint64_t)&task->batch_completion;
+    // TODO: Ensure that we never send a batch with count <= 1
+    task->batch_descriptor.desc_count = 0;
+    task->batch_descriptor.opcode = DSA_OPCODE_BATCH;
+    task->batch_descriptor.flags = IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CRAV;
+    task->batch_descriptor.desc_list_addr = (uintptr_t)task->descriptors;
+    task->status = DSA_TASK_READY;
+    task->group = &dsa_group;
+    task->device = dsa_device_group_get_next_device(&dsa_group);
+
+    for (int i = 0; i < task->batch_size; i++) {
+        buffer_zero_task_init_int(&task->descriptors[i],
+                                  &task->completions[i]);
+    }
+
+    qemu_sem_init(&task->sem_task_complete, 0);
+    task->completion_callback = buffer_zero_dsa_completion;
+}
+
+/**
+ * @brief Performs the proper cleanup on a DSA batch task.
+ *
+ * @param task A pointer to the batch task to cleanup.
+ */
+void
+buffer_zero_batch_task_destroy(struct buffer_zero_batch_task *task)
+{
+    qemu_vfree(task->descriptors);
+    qemu_vfree(task->completions);
+    g_free(task->results);
+
+    qemu_sem_destroy(&task->sem_task_complete);
+}
+
+/**
+ * @brief Resets a buffer zero comparison DSA batch task.
+ *
+ * @param task A pointer to the batch task.
+ * @param count The number of DSA tasks this batch task will contain.
+ */
+static void
+buffer_zero_batch_task_reset(struct buffer_zero_batch_task *task, size_t count)
+{
+    task->batch_completion.status = DSA_COMP_NONE;
+    task->batch_descriptor.desc_count = count;
+    task->task_type = DSA_BATCH_TASK;
+    task->status = DSA_TASK_READY;
+}
+
+/**
+ * @brief Sets a buffer zero comparison DSA task.
+ *
+ * @param descriptor A pointer to the DSA task descriptor.
+ * @param buf A pointer to the memory buffer.
+ * @param len The length of the buffer.
+ */
+static void
+buffer_zero_task_set_int(struct dsa_hw_desc *descriptor,
+                         const void *buf,
+                         size_t len)
+{
+    struct dsa_completion_record *completion =
+        (struct dsa_completion_record *)descriptor->completion_addr;
+
+    descriptor->xfer_size = len;
+    descriptor->src_addr = (uintptr_t)buf;
+    completion->status = 0;
+    completion->result = 0;
+}
+
+/**
+ * @brief Resets a buffer zero comparison DSA batch task.
+ *
+ * @param task A pointer to the DSA batch task.
+ */
+static void
+buffer_zero_task_reset(struct buffer_zero_batch_task *task)
+{
+    task->completions[0].status = DSA_COMP_NONE;
+    task->task_type = DSA_TASK;
+    task->status = DSA_TASK_READY;
+}
+
+/**
+ * @brief Sets a buffer zero comparison DSA task.
+ *
+ * @param task A pointer to the DSA task.
+ * @param buf A pointer to the memory buffer.
+ * @param len The buffer length.
+ */
+static void
+buffer_zero_task_set(struct buffer_zero_batch_task *task,
+                     const void *buf,
+                     size_t len)
+{
+    buffer_zero_task_reset(task);
+    buffer_zero_task_set_int(&task->descriptors[0], buf, len);
+}
+
+/**
+ * @brief Sets a buffer zero comparison batch task.
+ *
+ * @param batch_task A pointer to the batch task.
+ * @param buf An array of memory buffers.
+ * @param count The number of buffers in the array.
+ * @param len The length of the buffers.
+ */
+static void
+buffer_zero_batch_task_set(struct buffer_zero_batch_task *batch_task,
+                           const void **buf, size_t count, size_t len)
+{
+    assert(count > 0);
+    assert(count <= batch_task->batch_size);
+
+    buffer_zero_batch_task_reset(batch_task, count);
+    for (int i = 0; i < count; i++) {
+        buffer_zero_task_set_int(&batch_task->descriptors[i], buf[i], len);
+    }
+}
+
+/**
+ * @brief Asychronously perform a buffer zero DSA operation.
+ *
+ * @param task A pointer to the batch task structure.
+ * @param buf A pointer to the memory buffer.
+ * @param len The length of the memory buffer.
+ *
+ * @return int Zero if successful, otherwise an appropriate error code.
+ */
+__attribute__((unused))
+static int
+buffer_zero_dsa_async(struct buffer_zero_batch_task *task,
+                      const void *buf, size_t len)
+{
+    buffer_zero_task_set(task, buf, len);
+
+    return submit_wi_async(task);
+}
+
+/**
+ * @brief Sends a memory comparison batch task to a DSA device and wait
+ *        for completion.
+ *
+ * @param batch_task The batch task to be submitted to DSA device.
+ * @param buf An array of memory buffers to check for zero.
+ * @param count The number of buffers.
+ * @param len The buffer length.
+ */
+__attribute__((unused))
+static int
+buffer_zero_dsa_batch_async(struct buffer_zero_batch_task *batch_task,
+                            const void **buf, size_t count, size_t len)
+{
+    assert(count <= batch_task->batch_size);
+    buffer_zero_batch_task_set(batch_task, buf, count, len);
+
+    return submit_batch_wi_async(batch_task);
+}
+
+/**
+ * @brief The completion callback function for buffer zero
+ *        comparison DSA task completion.
+ *
+ * @param context A pointer to the callback context.
+ */
+static void
+buffer_zero_dsa_completion(void *context)
+{
+    assert(context != NULL);
+
+    struct buffer_zero_batch_task *task =
+        (struct buffer_zero_batch_task *)context;
+    qemu_sem_post(&task->sem_task_complete);
+}
+
+/**
+ * @brief Wait for the asynchronous DSA task to complete.
+ *
+ * @param batch_task A pointer to the buffer zero comparison batch task.
+ */
+__attribute__((unused))
+static void
+buffer_zero_dsa_wait(struct buffer_zero_batch_task *batch_task)
+{
+    qemu_sem_wait(&batch_task->sem_task_complete);
+}
+
 /**
  * @brief Check if DSA is running.
  *
@@ -XXX,XX +XXX,XX @@ void dsa_cleanup(void)
 
 #else
 
+void buffer_zero_batch_task_init(struct buffer_zero_batch_task *task,
+                                 int batch_size) {}
+
+void buffer_zero_batch_task_destroy(struct buffer_zero_batch_task *task) {}
+
 bool dsa_is_running(void)
 {
     return false;
-- 
2.30.2

* Add a DSA task completion callback.
* DSA completion thread will call the tasks's completion callback
on every task/batch task completion.
* DSA submission path to wait for completion.
* Implement CPU fallback if DSA is not able to complete the task.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
---
 include/qemu/dsa.h |  14 +++++
 util/dsa.c         | 153 ++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 164 insertions(+), 3 deletions(-)

diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@ void buffer_zero_batch_task_init(struct buffer_zero_batch_task *task,
  */
 void buffer_zero_batch_task_destroy(struct buffer_zero_batch_task *task);
 
+/**
+ * @brief Performs buffer zero comparison on a DSA batch task asynchronously.
+ *
+ * @param batch_task A pointer to the batch task.
+ * @param buf An array of memory buffers.
+ * @param count The number of buffers in the array.
+ * @param len The buffer length.
+ *
+ * @return Zero if successful, otherwise non-zero.
+ */
+int
+buffer_is_zero_dsa_batch_async(struct buffer_zero_batch_task *batch_task,
+                               const void **buf, size_t count, size_t len);
+
 /**
  * @brief Initializes DSA devices.
  *
diff --git a/util/dsa.c b/util/dsa.c
index XXXXXXX..XXXXXXX 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@ poll_completion(struct dsa_completion_record *completion,
     return 0;
 }
 
+/**
+ * @brief Use CPU to complete a single zero page checking task.
+ *
+ * @param task A pointer to the task.
+ */
+static void
+task_cpu_fallback(struct buffer_zero_batch_task *task)
+{
+    assert(task->task_type == DSA_TASK);
+
+    struct dsa_completion_record *completion = &task->completions[0];
+    const uint8_t *buf;
+    size_t len;
+
+    if (completion->status == DSA_COMP_SUCCESS) {
+        return;
+    }
+
+    /*
+     * DSA was able to partially complete the operation. Check the
+     * result. If we already know this is not a zero page, we can
+     * return now.
+     */
+    if (completion->bytes_completed != 0 && completion->result != 0) {
+        task->results[0] = false;
+        return;
+    }
+
+    /* Let's fallback to use CPU to complete it. */
+    buf = (const uint8_t *)task->descriptors[0].src_addr;
+    len = task->descriptors[0].xfer_size;
+    task->results[0] = buffer_is_zero(buf + completion->bytes_completed,
+                                      len - completion->bytes_completed);
+}
+
 /**
  * @brief Complete a single DSA task in the batch task.
  *
@@ -XXX,XX +XXX,XX @@ poll_batch_task_completion(struct buffer_zero_batch_task *batch_task)
     }
 }
 
+/**
+ * @brief Use CPU to complete the zero page checking batch task.
+ *
+ * @param batch_task A pointer to the batch task.
+ */
+static void
+batch_task_cpu_fallback(struct buffer_zero_batch_task *batch_task)
+{
+    assert(batch_task->task_type == DSA_BATCH_TASK);
+
+    struct dsa_completion_record *batch_completion =
+        &batch_task->batch_completion;
+    struct dsa_completion_record *completion;
+    uint8_t status;
+    const uint8_t *buf;
+    size_t len;
+    bool *results = batch_task->results;
+    uint32_t count = batch_task->batch_descriptor.desc_count;
+
+    // DSA is able to complete the entire batch task.
+    if (batch_completion->status == DSA_COMP_SUCCESS) {
+        assert(count == batch_completion->bytes_completed);
+        return;
+    }
+
+    /*
+     * DSA encounters some error and is not able to complete
+     * the entire batch task. Use CPU fallback.
+     */
+    for (int i = 0; i < count; i++) {
+        completion = &batch_task->completions[i];
+        status = completion->status;
+        if (status == DSA_COMP_SUCCESS) {
+            continue;
+        }
+        assert(status == DSA_COMP_PAGE_FAULT_NOBOF);
+
+        /*
+         * DSA was able to partially complete the operation. Check the
+         * result. If we already know this is not a zero page, we can
+         * return now.
+         */
+        if (completion->bytes_completed != 0 && completion->result != 0) {
+            results[i] = false;
+            continue;
+        }
+
+        /* Let's fallback to use CPU to complete it. */
+        buf = (uint8_t *)batch_task->descriptors[i].src_addr;
+        len = batch_task->descriptors[i].xfer_size;
+        results[i] =
+            buffer_is_zero(buf + completion->bytes_completed,
+                           len - completion->bytes_completed);
+    }
+}
+
 /**
  * @brief Handles an asynchronous DSA batch task completion.
  *
@@ -XXX,XX +XXX,XX @@ buffer_zero_batch_task_set(struct buffer_zero_batch_task *batch_task,
  *
  * @return int Zero if successful, otherwise an appropriate error code.
  */
-__attribute__((unused))
 static int
 buffer_zero_dsa_async(struct buffer_zero_batch_task *task,
                       const void *buf, size_t len)
@@ -XXX,XX +XXX,XX @@ buffer_zero_dsa_async(struct buffer_zero_batch_task *task,
  * @param count The number of buffers.
  * @param len The buffer length.
  */
-__attribute__((unused))
 static int
 buffer_zero_dsa_batch_async(struct buffer_zero_batch_task *batch_task,
                             const void **buf, size_t count, size_t len)
@@ -XXX,XX +XXX,XX @@ buffer_zero_dsa_completion(void *context)
  *
  * @param batch_task A pointer to the buffer zero comparison batch task.
  */
-__attribute__((unused))
 static void
 buffer_zero_dsa_wait(struct buffer_zero_batch_task *batch_task)
 {
     qemu_sem_wait(&batch_task->sem_task_complete);
 }
 
+/**
+ * @brief Use CPU to complete the zero page checking task if DSA
+ *        is not able to complete it.
+ *
+ * @param batch_task A pointer to the batch task.
+ */
+static void
+buffer_zero_cpu_fallback(struct buffer_zero_batch_task *batch_task)
+{
+    if (batch_task->task_type == DSA_TASK) {
+        task_cpu_fallback(batch_task);
+    } else {
+        assert(batch_task->task_type == DSA_BATCH_TASK);
+        batch_task_cpu_fallback(batch_task);
+    }
+}
+
 /**
  * @brief Check if DSA is running.
  *
@@ -XXX,XX +XXX,XX @@ void dsa_cleanup(void)
     dsa_device_group_cleanup(&dsa_group);
 }
 
+/**
+ * @brief Performs buffer zero comparison on a DSA batch task asynchronously.
+ *
+ * @param batch_task A pointer to the batch task.
+ * @param buf An array of memory buffers.
+ * @param count The number of buffers in the array.
+ * @param len The buffer length.
+ *
+ * @return Zero if successful, otherwise non-zero.
+ */
+int
+buffer_is_zero_dsa_batch_async(struct buffer_zero_batch_task *batch_task,
+                               const void **buf, size_t count, size_t len)
+{
+    if (count <= 0 || count > batch_task->batch_size) {
+        return -1;
+    }
+
+    assert(batch_task != NULL);
+    assert(len != 0);
+    assert(buf != NULL);
+
+    if (count == 1) {
+        // DSA doesn't take batch operation with only 1 task.
+        buffer_zero_dsa_async(batch_task, buf[0], len);
+    } else {
+        buffer_zero_dsa_batch_async(batch_task, buf, count, len);
+    }
+
+    buffer_zero_dsa_wait(batch_task);
+    buffer_zero_cpu_fallback(batch_task);
+
+    return 0;
+}
+
 #else
 
 void buffer_zero_batch_task_init(struct buffer_zero_batch_task *task,
@@ -XXX,XX +XXX,XX @@ void dsa_stop(void) {}
 
 void dsa_cleanup(void) {}
 
+int
+buffer_is_zero_dsa_batch_async(struct buffer_zero_batch_task *batch_task,
+                               const void **buf, size_t count, size_t len)
+{
+    exit(1);
+}
+
 #endif
 
-- 
2.30.2

Intel DSA offloading is an optional feature that turns on if
proper hardware and software stack is available. To turn on
DSA offloading in multifd live migration:

multifd-dsa-accel="[dsa_dev_path1] ] [dsa_dev_path2] ... [dsa_dev_pathX]"

This feature is turned off by default.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 migration/migration-hmp-cmds.c |  8 ++++++++
 migration/options.c            | 28 ++++++++++++++++++++++++++++
 migration/options.h            |  1 +
 qapi/migration.json            | 17 ++++++++++++++---
 scripts/meson-buildoptions.sh  |  6 +++---
 5 files changed, 54 insertions(+), 6 deletions(-)

diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/migration-hmp-cmds.c
+++ b/migration/migration-hmp-cmds.c
@@ -XXX,XX +XXX,XX @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "%s: '%s'\n",
             MigrationParameter_str(MIGRATION_PARAMETER_TLS_AUTHZ),
             params->tls_authz);
+        monitor_printf(mon, "%s: %s\n",
+            MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_DSA_ACCEL),
+            params->multifd_dsa_accel);
 
         if (params->has_block_bitmap_mapping) {
             const BitmapMigrationNodeAliasList *bmnal;
@@ -XXX,XX +XXX,XX @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         p->has_block_incremental = true;
         visit_type_bool(v, param, &p->block_incremental, &err);
         break;
+    case MIGRATION_PARAMETER_MULTIFD_DSA_ACCEL:
+        p->multifd_dsa_accel = g_new0(StrOrNull, 1);
+        p->multifd_dsa_accel->type = QTYPE_QSTRING;
+        visit_type_str(v, param, &p->multifd_dsa_accel->u.s, &err);
+        break;
     case MIGRATION_PARAMETER_MULTIFD_CHANNELS:
         p->has_multifd_channels = true;
         visit_type_uint8(v, param, &p->multifd_channels, &err);
diff --git a/migration/options.c b/migration/options.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -XXX,XX +XXX,XX @@ Property migration_properties[] = {
     DEFINE_PROP_MIG_MODE("mode", MigrationState,
                       parameters.mode,
                       MIG_MODE_NORMAL),
+    DEFINE_PROP_STRING("multifd-dsa-accel", MigrationState,
+                       parameters.multifd_dsa_accel),
 
     /* Migration capabilities */
     DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_creds(void)
     return s->parameters.tls_creds;
 }
 
+const char *migrate_multifd_dsa_accel(void)
+{
+    MigrationState *s = migrate_get_current();
+
+    return s->parameters.multifd_dsa_accel;
+}
+
 const char *migrate_tls_hostname(void)
 {
     MigrationState *s = migrate_get_current();
@@ -XXX,XX +XXX,XX @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
     params->vcpu_dirty_limit = s->parameters.vcpu_dirty_limit;
     params->has_mode = true;
     params->mode = s->parameters.mode;
+    params->multifd_dsa_accel = s->parameters.multifd_dsa_accel;
 
     return params;
 }
@@ -XXX,XX +XXX,XX @@ void migrate_params_init(MigrationParameters *params)
 {
     params->tls_hostname = g_strdup("");
     params->tls_creds = g_strdup("");
+    params->multifd_dsa_accel = g_strdup("");
 
     /* Set has_* up only for parameter checks */
     params->has_compress_level = true;
@@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrateSetParameters *params,
     if (params->has_mode) {
         dest->mode = params->mode;
     }
+
+    if (params->multifd_dsa_accel) {
+        assert(params->multifd_dsa_accel->type == QTYPE_QSTRING);
+        dest->multifd_dsa_accel = params->multifd_dsa_accel->u.s;
+    }
 }
 
 static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
@@ -XXX,XX +XXX,XX @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
     if (params->has_mode) {
         s->parameters.mode = params->mode;
     }
+
+    if (params->multifd_dsa_accel) {
+        g_free(s->parameters.multifd_dsa_accel);
+        assert(params->multifd_dsa_accel->type == QTYPE_QSTRING);
+        s->parameters.multifd_dsa_accel = g_strdup(params->multifd_dsa_accel->u.s);
+    }
 }
 
 void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
@@ -XXX,XX +XXX,XX @@ void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
         params->tls_authz->type = QTYPE_QSTRING;
         params->tls_authz->u.s = strdup("");
     }
+    if (params->multifd_dsa_accel
+        && params->multifd_dsa_accel->type == QTYPE_QNULL) {
+        qobject_unref(params->multifd_dsa_accel->u.n);
+        params->multifd_dsa_accel->type = QTYPE_QSTRING;
+        params->multifd_dsa_accel->u.s = strdup("");
+    }
 
     migrate_params_test_apply(params, &tmp);
 
diff --git a/migration/options.h b/migration/options.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.h
+++ b/migration/options.h
@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_authz(void);
 const char *migrate_tls_creds(void);
 const char *migrate_tls_hostname(void);
 uint64_t migrate_xbzrle_cache_size(void);
+const char *migrate_multifd_dsa_accel(void);
 
 /* parameters setters */
 
diff --git a/qapi/migration.json b/qapi/migration.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -XXX,XX +XXX,XX @@
 # @mode: Migration mode. See description in @MigMode. Default is 'normal'.
 #        (Since 8.2)
 #
+# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
+#                     certain memory operations. (since 8.2)
+#
 # Features:
 #
 # @deprecated: Member @block-incremental is deprecated.  Use
@@ -XXX,XX +XXX,XX @@
            'cpu-throttle-initial', 'cpu-throttle-increment',
            'cpu-throttle-tailslow',
            'tls-creds', 'tls-hostname', 'tls-authz', 'max-bandwidth',
-           'avail-switchover-bandwidth', 'downtime-limit',
+           'avail-switchover-bandwidth', 'downtime-limit', 'multifd-dsa-accel',
            { 'name': 'x-checkpoint-delay', 'features': [ 'unstable' ] },
            { 'name': 'block-incremental', 'features': [ 'deprecated' ] },
            'multifd-channels',
@@ -XXX,XX +XXX,XX @@
 # @mode: Migration mode. See description in @MigMode. Default is 'normal'.
 #        (Since 8.2)
 #
+# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
+#                     certain memory operations. (since 8.2)
+#
 # Features:
 #
 # @deprecated: Member @block-incremental is deprecated.  Use
@@ -XXX,XX +XXX,XX @@
             '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
                                             'features': [ 'unstable' ] },
             '*vcpu-dirty-limit': 'uint64',
-            '*mode': 'MigMode'} }
+            '*mode': 'MigMode',
+            '*multifd-dsa-accel': 'StrOrNull'} }
 
 ##
 # @migrate-set-parameters:
@@ -XXX,XX +XXX,XX @@
 # @mode: Migration mode. See description in @MigMode. Default is 'normal'.
 #        (Since 8.2)
 #
+# @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
+#                     certain memory operations. (since 8.2)
+#
 # Features:
 #
 # @deprecated: Member @block-incremental is deprecated.  Use
@@ -XXX,XX +XXX,XX @@
             '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
                                             'features': [ 'unstable' ] },
             '*vcpu-dirty-limit': 'uint64',
-            '*mode': 'MigMode'} }
+            '*mode': 'MigMode',
+            '*multifd-dsa-accel': 'str'} }
 
 ##
 # @query-migrate-parameters:
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index XXXXXXX..XXXXXXX 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -XXX,XX +XXX,XX @@ meson_options_help() {
   printf "%s\n" '  --enable-debug-stack-usage'
   printf "%s\n" '                           measure coroutine stack usage'
   printf "%s\n" '  --enable-debug-tcg       TCG debugging'
+  printf "%s\n" '  --enable-enqcmd          MENQCMD optimizations'
   printf "%s\n" '  --enable-fdt[=CHOICE]    Whether and how to find the libfdt library'
   printf "%s\n" '                           (choices: auto/disabled/enabled/internal/system)'
   printf "%s\n" '  --enable-fuzzing         build fuzzing targets'
@@ -XXX,XX +XXX,XX @@ meson_options_help() {
   printf "%s\n" '  avx2            AVX2 optimizations'
   printf "%s\n" '  avx512bw        AVX512BW optimizations'
   printf "%s\n" '  avx512f         AVX512F optimizations'
-  printf "%s\n" '  enqcmd          ENQCMD optimizations'
   printf "%s\n" '  blkio           libblkio block device driver'
   printf "%s\n" '  bochs           bochs image format support'
   printf "%s\n" '  bpf             eBPF support'
@@ -XXX,XX +XXX,XX @@ _meson_option_parse() {
     --disable-avx512bw) printf "%s" -Davx512bw=disabled ;;
     --enable-avx512f) printf "%s" -Davx512f=enabled ;;
     --disable-avx512f) printf "%s" -Davx512f=disabled ;;
-    --enable-enqcmd) printf "%s" -Denqcmd=true ;;
-    --disable-enqcmd) printf "%s" -Denqcmd=false ;;
     --enable-gcov) printf "%s" -Db_coverage=true ;;
     --disable-gcov) printf "%s" -Db_coverage=false ;;
     --enable-lto) printf "%s" -Db_lto=true ;;
@@ -XXX,XX +XXX,XX @@ _meson_option_parse() {
     --disable-docs) printf "%s" -Ddocs=disabled ;;
     --enable-dsound) printf "%s" -Ddsound=enabled ;;
     --disable-dsound) printf "%s" -Ddsound=disabled ;;
+    --enable-enqcmd) printf "%s" -Denqcmd=true ;;
+    --disable-enqcmd) printf "%s" -Denqcmd=false ;;
     --enable-fdt) printf "%s" -Dfdt=enabled ;;
     --disable-fdt) printf "%s" -Dfdt=disabled ;;
     --enable-fdt=*) quote_sh "-Dfdt=$2" ;;
-- 
2.30.2

1. Refactor multifd_send_thread function.
2. Implement buffer_is_zero_use_cpu to handle CPU based zero page
checking.
3. Introduce the batch task structure in MultiFDSendParams.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 migration/multifd.c | 82 ++++++++++++++++++++++++++++++++++++---------
 migration/multifd.h |  3 ++
 2 files changed, 70 insertions(+), 15 deletions(-)

diff --git a/migration/multifd.c b/migration/multifd.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -XXX,XX +XXX,XX @@
 #include "qemu/cutils.h"
 #include "qemu/rcu.h"
 #include "qemu/cutils.h"
+#include "qemu/dsa.h"
+#include "qemu/memalign.h"
 #include "exec/target_page.h"
 #include "sysemu/sysemu.h"
 #include "exec/ramblock.h"
@@ -XXX,XX +XXX,XX @@ void multifd_save_cleanup(void)
         p->name = NULL;
         multifd_pages_clear(p->pages);
         p->pages = NULL;
+        g_free(p->addr);
+        p->addr = NULL;
+        buffer_zero_batch_task_destroy(p->batch_task);
+        qemu_vfree(p->batch_task);
+        p->batch_task = NULL;
         p->packet_len = 0;
         g_free(p->packet);
         p->packet = NULL;
@@ -XXX,XX +XXX,XX @@ int multifd_send_sync_main(QEMUFile *f)
     return 0;
 }
 
+static void set_page(MultiFDSendParams *p, bool zero_page, uint64_t offset)
+{
+    RAMBlock *rb = p->pages->block;
+    if (zero_page) {
+        p->zero[p->zero_num] = offset;
+        p->zero_num++;
+        ram_release_page(rb->idstr, offset);
+    } else {
+        p->normal[p->normal_num] = offset;
+        p->normal_num++;
+    }
+}
+
+static void buffer_is_zero_use_cpu(MultiFDSendParams *p)
+{
+    const void **buf = (const void **)p->addr;
+    assert(!migrate_use_main_zero_page());
+
+    for (int i = 0; i < p->pages->num; i++) {
+        p->batch_task->results[i] = buffer_is_zero(buf[i], p->page_size);
+    }
+}
+
+static void set_normal_pages(MultiFDSendParams *p)
+{
+    for (int i = 0; i < p->pages->num; i++) {
+        p->batch_task->results[i] = false;
+    }
+}
+
+static void multifd_zero_page_check(MultiFDSendParams *p)
+{
+    /* older qemu don't understand zero page on multifd channel */
+    bool use_multifd_zero_page = !migrate_use_main_zero_page();
+
+    RAMBlock *rb = p->pages->block;
+
+    for (int i = 0; i < p->pages->num; i++) {
+        p->addr[i] = (ram_addr_t)(rb->host + p->pages->offset[i]);
+    }
+
+    if (use_multifd_zero_page) {
+        buffer_is_zero_use_cpu(p);
+    } else {
+        // No zero page checking. All pages are normal pages.
+        set_normal_pages(p);
+    }
+
+    for (int i = 0; i < p->pages->num; i++) {
+        uint64_t offset = p->pages->offset[i];
+        bool zero_page = p->batch_task->results[i];
+        set_page(p, zero_page, offset);
+    }
+}
+
 static void *multifd_send_thread(void *opaque)
 {
     MultiFDSendParams *p = opaque;
     MigrationThread *thread = NULL;
     Error *local_err = NULL;
-    /* qemu older than 8.2 don't understand zero page on multifd channel */
-    bool use_multifd_zero_page = !migrate_use_main_zero_page();
     int ret = 0;
     bool use_zero_copy_send = migrate_zero_copy_send();
 
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
         qemu_mutex_lock(&p->mutex);
 
         if (p->pending_job) {
-            RAMBlock *rb = p->pages->block;
             uint64_t packet_num = p->packet_num;
             uint32_t flags;
 
@@ -XXX,XX +XXX,XX @@ static void *multifd_send_thread(void *opaque)
                 p->iovs_num = 1;
             }
 
-            for (int i = 0; i < p->pages->num; i++) {
-                uint64_t offset = p->pages->offset[i];
-                if (use_multifd_zero_page &&
-                    buffer_is_zero(rb->host + offset, p->page_size)) {
-                    p->zero[p->zero_num] = offset;
-                    p->zero_num++;
-                    ram_release_page(rb->idstr, offset);
-                } else {
-                    p->normal[p->normal_num] = offset;
-                    p->normal_num++;
-                }
-            }
+            multifd_zero_page_check(p);
 
             if (p->normal_num) {
                 ret = multifd_send_state->ops->send_prepare(p, &local_err);
@@ -XXX,XX +XXX,XX @@ int multifd_save_setup(Error **errp)
         p->pending_job = 0;
         p->id = i;
         p->pages = multifd_pages_init(page_count);
+        p->addr = g_new0(ram_addr_t, page_count);
+        p->batch_task =
+            (struct buffer_zero_batch_task *)qemu_memalign(64, sizeof(*p->batch_task));
+        buffer_zero_batch_task_init(p->batch_task, page_count);
         p->packet_len = sizeof(MultiFDPacket_t)
                       + sizeof(uint64_t) * page_count;
         p->packet = g_malloc0(p->packet_len);
diff --git a/migration/multifd.h b/migration/multifd.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -XXX,XX +XXX,XX @@ typedef struct {
      * pending_job != 0 -> multifd_channel can use it.
      */
     MultiFDPages_t *pages;
+    /* Address of each pages in pages */
+    ram_addr_t *addr;
+    struct buffer_zero_batch_task *batch_task;
 
     /* thread local variables. No locking required */
 
-- 
2.30.2

Multifd sender path gets an array of pages queued by the migration
thread. It performs zero page checking on every page in the array.
The pages are classfied as either a zero page or a normal page. This
change uses Intel DSA to offload the zero page checking from CPU to
the DSA accelerator. The sender thread submits a batch of pages to DSA
hardware and waits for the DSA completion thread to signal for work
completion.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 migration/multifd.c | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/migration/multifd.c b/migration/multifd.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -XXX,XX +XXX,XX @@ void multifd_save_cleanup(void)
             qemu_thread_join(&p->thread);
         }
     }
+    dsa_stop();
+    dsa_cleanup();
     for (i = 0; i < migrate_multifd_channels(); i++) {
         MultiFDSendParams *p = &multifd_send_state->params[i];
         Error *local_err = NULL;
@@ -XXX,XX +XXX,XX @@ static void buffer_is_zero_use_cpu(MultiFDSendParams *p)
 {
     const void **buf = (const void **)p->addr;
     assert(!migrate_use_main_zero_page());
+    assert(!dsa_is_running());
 
     for (int i = 0; i < p->pages->num; i++) {
         p->batch_task->results[i] = buffer_is_zero(buf[i], p->page_size);
@@ -XXX,XX +XXX,XX @@ static void buffer_is_zero_use_cpu(MultiFDSendParams *p)
 
 static void set_normal_pages(MultiFDSendParams *p)
 {
+    assert(migrate_use_main_zero_page());
+
     for (int i = 0; i < p->pages->num; i++) {
         p->batch_task->results[i] = false;
     }
 }
 
+static void buffer_is_zero_use_dsa(MultiFDSendParams *p)
+{
+    assert(!migrate_use_main_zero_page());
+    assert(dsa_is_running());
+
+    buffer_is_zero_dsa_batch_async(p->batch_task,
+                                   (const void **)p->addr,
+                                   p->pages->num,
+                                   p->page_size);
+}
+
 static void multifd_zero_page_check(MultiFDSendParams *p)
 {
     /* older qemu don't understand zero page on multifd channel */
     bool use_multifd_zero_page = !migrate_use_main_zero_page();
+    bool use_multifd_dsa_accel = dsa_is_running();
 
     RAMBlock *rb = p->pages->block;
 
@@ -XXX,XX +XXX,XX @@ static void multifd_zero_page_check(MultiFDSendParams *p)
         p->addr[i] = (ram_addr_t)(rb->host + p->pages->offset[i]);
     }
 
-    if (use_multifd_zero_page) {
+    if (use_multifd_dsa_accel && use_multifd_zero_page) {
+        buffer_is_zero_use_dsa(p);
+    } else if (use_multifd_zero_page) {
         buffer_is_zero_use_cpu(p);
     } else {
         // No zero page checking. All pages are normal pages.
@@ -XXX,XX +XXX,XX @@ int multifd_save_setup(Error **errp)
     int thread_count;
     uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
     uint8_t i;
+    const char *dsa_parameter = migrate_multifd_dsa_accel();
 
     if (!migrate_multifd()) {
         return 0;
     }
 
+    dsa_init(dsa_parameter);
+    dsa_start();
+
     thread_count = migrate_multifd_channels();
     multifd_send_state = g_malloc0(sizeof(*multifd_send_state));
     multifd_send_state->params = g_new0(MultiFDSendParams, thread_count);
@@ -XXX,XX +XXX,XX @@ int multifd_save_setup(Error **errp)
             return ret;
         }
     }
+
     return 0;
 }
 
@@ -XXX,XX +XXX,XX @@ void multifd_load_cleanup(void)
 
         qemu_thread_join(&p->thread);
     }
+    dsa_stop();
+    dsa_cleanup();
     for (i = 0; i < migrate_multifd_channels(); i++) {
         MultiFDRecvParams *p = &multifd_recv_state->params[i];
 
@@ -XXX,XX +XXX,XX @@ int multifd_load_setup(Error **errp)
     int thread_count;
     uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size();
     uint8_t i;
+    const char *dsa_parameter = migrate_multifd_dsa_accel();
 
     /*
      * Return successfully if multiFD recv state is already initialised
@@ -XXX,XX +XXX,XX @@ int multifd_load_setup(Error **errp)
         return 0;
     }
 
+    dsa_init(dsa_parameter);
+    dsa_start();
+
     thread_count = migrate_multifd_channels();
     multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state));
     multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count);
@@ -XXX,XX +XXX,XX @@ int multifd_load_setup(Error **errp)
             return ret;
         }
     }
+
     return 0;
 }
 
-- 
2.30.2

Multifd sender thread performs zero page checking. If a page is
a zero page, only the page's metadata is sent to the receiver.
If a page is a normal page, the entire page's content is sent to
the receiver. This change adds a test hook to set the normal page
ratio. A zero page will be forced to be sent as a normal page. This
is useful for live migration performance analysis and optimization.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 migration/options.c | 31 +++++++++++++++++++++++++++++++
 migration/options.h |  1 +
 qapi/migration.json | 18 +++++++++++++++---
 3 files changed, 47 insertions(+), 3 deletions(-)

diff --git a/migration/options.c b/migration/options.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -XXX,XX +XXX,XX @@
 #define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS    5
 #define DEFAULT_MIGRATE_ANNOUNCE_STEP    100
 
+/*
+ * Parameter for multifd normal page test hook.
+ */
+#define DEFAULT_MIGRATE_MULTIFD_NORMAL_PAGE_RATIO 101
+
 #define DEFINE_PROP_MIG_CAP(name, x)             \
     DEFINE_PROP_BOOL(name, MigrationState, capabilities[x], false)
 
@@ -XXX,XX +XXX,XX @@ Property migration_properties[] = {
                       MIG_MODE_NORMAL),
     DEFINE_PROP_STRING("multifd-dsa-accel", MigrationState,
                        parameters.multifd_dsa_accel),
+    DEFINE_PROP_UINT8("multifd-normal-page-ratio", MigrationState,
+                      parameters.multifd_normal_page_ratio,
+                      DEFAULT_MIGRATE_MULTIFD_NORMAL_PAGE_RATIO),
 
     /* Migration capabilities */
     DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
@@ -XXX,XX +XXX,XX @@ int migrate_multifd_channels(void)
     return s->parameters.multifd_channels;
 }
 
+uint8_t migrate_multifd_normal_page_ratio(void)
+{
+    MigrationState *s = migrate_get_current();
+    return s->parameters.multifd_normal_page_ratio;
+}
+
 MultiFDCompression migrate_multifd_compression(void)
 {
     MigrationState *s = migrate_get_current();
@@ -XXX,XX +XXX,XX @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
         return false;
     }
 
+    if (params->has_multifd_normal_page_ratio &&
+        params->multifd_normal_page_ratio > 100) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
+                   "multifd_normal_page_ratio",
+                   "a value between 0 and 100");
+        return false;
+    }
+
     return true;
 }
 
@@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrateSetParameters *params,
         assert(params->multifd_dsa_accel->type == QTYPE_QSTRING);
         dest->multifd_dsa_accel = params->multifd_dsa_accel->u.s;
     }
+
+    if (params->has_multifd_normal_page_ratio) {
+        dest->has_multifd_normal_page_ratio = true;
+        dest->multifd_normal_page_ratio = params->multifd_normal_page_ratio;
+    }
 }
 
 static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
@@ -XXX,XX +XXX,XX @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
         assert(params->multifd_dsa_accel->type == QTYPE_QSTRING);
         s->parameters.multifd_dsa_accel = g_strdup(params->multifd_dsa_accel->u.s);
     }
+
+    if (params->has_multifd_normal_page_ratio) {
+        s->parameters.multifd_normal_page_ratio = params->multifd_normal_page_ratio;
+    }
 }
 
 void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
diff --git a/migration/options.h b/migration/options.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.h
+++ b/migration/options.h
@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_creds(void);
 const char *migrate_tls_hostname(void);
 uint64_t migrate_xbzrle_cache_size(void);
 const char *migrate_multifd_dsa_accel(void);
+uint8_t migrate_multifd_normal_page_ratio(void);
 
 /* parameters setters */
 
diff --git a/qapi/migration.json b/qapi/migration.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -XXX,XX +XXX,XX @@
 # @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
 #                     certain memory operations. (since 8.2)
 #
+# @multifd-normal-page-ratio: Test hook setting the normal page ratio.
+#     (Since 8.2)
+#
 # Features:
 #
 # @deprecated: Member @block-incremental is deprecated.  Use
@@ -XXX,XX +XXX,XX @@
            'block-bitmap-mapping',
            { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] },
            'vcpu-dirty-limit',
-           'mode'] }
+           'mode',
+           'multifd-normal-page-ratio'] }
 
 ##
 # @MigrateSetParameters:
@@ -XXX,XX +XXX,XX @@
 # @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
 #                     certain memory operations. (since 8.2)
 #
+# @multifd-normal-page-ratio: Test hook setting the normal page ratio.
+#     (Since 8.2)
+#
 # Features:
 #
 # @deprecated: Member @block-incremental is deprecated.  Use
@@ -XXX,XX +XXX,XX @@
                                             'features': [ 'unstable' ] },
             '*vcpu-dirty-limit': 'uint64',
             '*mode': 'MigMode',
-            '*multifd-dsa-accel': 'StrOrNull'} }
+            '*multifd-dsa-accel': 'StrOrNull',
+            '*multifd-normal-page-ratio': 'uint8'} }
 
 ##
 # @migrate-set-parameters:
@@ -XXX,XX +XXX,XX @@
 # @multifd-dsa-accel: If enabled, use DSA accelerator offloading for
 #                     certain memory operations. (since 8.2)
 #
+# @multifd-normal-page-ratio: Test hook setting the normal page ratio.
+#     (Since 8.2)
+#
 # Features:
 #
 # @deprecated: Member @block-incremental is deprecated.  Use
@@ -XXX,XX +XXX,XX @@
                                             'features': [ 'unstable' ] },
             '*vcpu-dirty-limit': 'uint64',
             '*mode': 'MigMode',
-            '*multifd-dsa-accel': 'str'} }
+            '*multifd-dsa-accel': 'str',
+            '*multifd-normal-page-ratio': 'uint8'} }
 
 ##
 # @query-migrate-parameters:
-- 
2.30.2

Test hook is disabled by default. To set it, a normal page ratio
between 0 and 100 are valid. If the ratio is set to 50, it means
at least 50% of all pages are sent as normal pages.

Set the option:
migrate_set_parameter multifd-normal-page-ratio 60

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 include/qemu/dsa.h             |  7 ++++++-
 migration/migration-hmp-cmds.c |  7 +++++++
 migration/multifd.c            | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 46 insertions(+), 1 deletion(-)

diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@ typedef struct buffer_zero_batch_task {
     enum dsa_task_type task_type;
     enum dsa_task_status status;
     bool *results;
-    int batch_size;
+    uint32_t batch_size;
+    // Set normal page ratio test hook.
+    uint32_t normal_page_index;
+    uint32_t normal_page_counter;
     QSIMPLEQ_ENTRY(buffer_zero_batch_task) entry;
 } buffer_zero_batch_task;
 
@@ -XXX,XX +XXX,XX @@ typedef struct buffer_zero_batch_task {
 
 struct buffer_zero_batch_task {
     bool *results;
+    uint32_t normal_page_index;
+    uint32_t normal_page_counter;
 };
 
 #endif
diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/migration-hmp-cmds.c
+++ b/migration/migration-hmp-cmds.c
@@ -XXX,XX +XXX,XX @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "%s: %s\n",
             MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_DSA_ACCEL),
             params->multifd_dsa_accel);
+        monitor_printf(mon, "%s: %u\n",
+            MigrationParameter_str(MIGRATION_PARAMETER_MULTIFD_NORMAL_PAGE_RATIO),
+            params->multifd_normal_page_ratio);
 
         if (params->has_block_bitmap_mapping) {
             const BitmapMigrationNodeAliasList *bmnal;
@@ -XXX,XX +XXX,XX @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         error_setg(&err, "The block-bitmap-mapping parameter can only be set "
                    "through QMP");
         break;
+    case MIGRATION_PARAMETER_MULTIFD_NORMAL_PAGE_RATIO:
+        p->has_multifd_normal_page_ratio = true;
+        visit_type_uint8(v, param, &p->multifd_normal_page_ratio, &err);
+        break;
     case MIGRATION_PARAMETER_X_VCPU_DIRTY_LIMIT_PERIOD:
         p->has_x_vcpu_dirty_limit_period = true;
         visit_type_size(v, param, &p->x_vcpu_dirty_limit_period, &err);
diff --git a/migration/multifd.c b/migration/multifd.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -XXX,XX +XXX,XX @@ int multifd_send_sync_main(QEMUFile *f)
     return 0;
 }
 
+static void multifd_normal_page_test_hook(MultiFDSendParams *p)
+{
+    /*
+     * The value is between 0 to 100. If the value is 10, it means at
+     * least 10% of the pages are normal page. A zero page can be made
+     * a normal page but not the other way around.
+     */
+    uint8_t multifd_normal_page_ratio =
+        migrate_multifd_normal_page_ratio();
+    struct buffer_zero_batch_task *batch_task = p->batch_task;
+
+    // Set normal page test hook is disabled.
+    if (multifd_normal_page_ratio > 100) {
+        return;
+    }
+
+    for (int i = 0; i < p->pages->num; i++) {
+        if (batch_task->normal_page_counter < multifd_normal_page_ratio) {
+            // Turn a zero page into a normal page.
+            batch_task->results[i] = false;
+        }
+        batch_task->normal_page_index++;
+        batch_task->normal_page_counter++;
+
+        if (batch_task->normal_page_index >= 100) {
+            batch_task->normal_page_index = 0;
+            batch_task->normal_page_counter = 0;
+        }
+    }
+}
+
 static void set_page(MultiFDSendParams *p, bool zero_page, uint64_t offset)
 {
     RAMBlock *rb = p->pages->block;
@@ -XXX,XX +XXX,XX @@ static void multifd_zero_page_check(MultiFDSendParams *p)
         set_normal_pages(p);
     }
 
+    multifd_normal_page_test_hook(p);
+
     for (int i = 0; i < p->pages->num; i++) {
         uint64_t offset = p->pages->offset[i];
         bool zero_page = p->batch_task->results[i];
-- 
2.30.2

The current multifd packet size is 128 * 4kb. This change adds
an option to set the packet size. Both sender and receiver needs
to set the same packet size for things to work.

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 migration/options.c | 34 ++++++++++++++++++++++++++++++++++
 migration/options.h |  1 +
 qapi/migration.json | 21 ++++++++++++++++++---
 3 files changed, 53 insertions(+), 3 deletions(-)

diff --git a/migration/options.c b/migration/options.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -XXX,XX +XXX,XX @@
  */
 #define DEFAULT_MIGRATE_MULTIFD_NORMAL_PAGE_RATIO 101
 
+/*
+ * Parameter for multifd packet size.
+ */
+#define DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE 128
+#define MAX_MIGRATE_MULTIFD_PACKET_SIZE 1024
+
 #define DEFINE_PROP_MIG_CAP(name, x)             \
     DEFINE_PROP_BOOL(name, MigrationState, capabilities[x], false)
 
@@ -XXX,XX +XXX,XX @@ Property migration_properties[] = {
     DEFINE_PROP_UINT8("multifd-normal-page-ratio", MigrationState,
                       parameters.multifd_normal_page_ratio,
                       DEFAULT_MIGRATE_MULTIFD_NORMAL_PAGE_RATIO),
+    DEFINE_PROP_SIZE("multifd-packet-size", MigrationState,
+                     parameters.multifd_packet_size,
+                     DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE),
 
     /* Migration capabilities */
     DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE),
@@ -XXX,XX +XXX,XX @@ uint8_t migrate_multifd_normal_page_ratio(void)
     return s->parameters.multifd_normal_page_ratio;
 }
 
+uint64_t migrate_multifd_packet_size(void)
+{
+    MigrationState *s = migrate_get_current();
+
+    return s->parameters.multifd_packet_size;
+}
+
 MultiFDCompression migrate_multifd_compression(void)
 {
     MigrationState *s = migrate_get_current();
@@ -XXX,XX +XXX,XX @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
     params->x_checkpoint_delay = s->parameters.x_checkpoint_delay;
     params->has_block_incremental = true;
     params->block_incremental = s->parameters.block_incremental;
+    params->has_multifd_packet_size = true;
+    params->multifd_packet_size = s->parameters.multifd_packet_size;
     params->has_multifd_channels = true;
     params->multifd_channels = s->parameters.multifd_channels;
     params->has_multifd_compression = true;
@@ -XXX,XX +XXX,XX @@ void migrate_params_init(MigrationParameters *params)
     params->has_downtime_limit = true;
     params->has_x_checkpoint_delay = true;
     params->has_block_incremental = true;
+    params->has_multifd_packet_size = true;
     params->has_multifd_channels = true;
     params->has_multifd_compression = true;
     params->has_multifd_zlib_level = true;
@@ -XXX,XX +XXX,XX @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
 
     /* x_checkpoint_delay is now always positive */
 
+    if (params->has_multifd_packet_size &&
+        ((params->multifd_packet_size < DEFAULT_MIGRATE_MULTIFD_PACKET_SIZE) ||
+            (params->multifd_packet_size >  MAX_MIGRATE_MULTIFD_PACKET_SIZE))) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
+                    "multifd_packet_size",
+                    "a value between 128 and 1024");
+        return false;
+    }
+
     if (params->has_multifd_channels && (params->multifd_channels < 1)) {
         error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
                    "multifd_channels",
@@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrateSetParameters *params,
     if (params->has_block_incremental) {
         dest->block_incremental = params->block_incremental;
     }
+    if (params->has_multifd_packet_size) {
+        dest->multifd_packet_size = params->multifd_packet_size;
+    }
     if (params->has_multifd_channels) {
         dest->multifd_channels = params->multifd_channels;
     }
@@ -XXX,XX +XXX,XX @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
                     " use blockdev-mirror with NBD instead");
         s->parameters.block_incremental = params->block_incremental;
     }
+    if (params->has_multifd_packet_size) {
+        s->parameters.multifd_packet_size = params->multifd_packet_size;
+    }
     if (params->has_multifd_channels) {
         s->parameters.multifd_channels = params->multifd_channels;
     }
diff --git a/migration/options.h b/migration/options.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.h
+++ b/migration/options.h
@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_hostname(void);
 uint64_t migrate_xbzrle_cache_size(void);
 const char *migrate_multifd_dsa_accel(void);
 uint8_t migrate_multifd_normal_page_ratio(void);
+uint64_t migrate_multifd_packet_size(void);
 
 /* parameters setters */
 
diff --git a/qapi/migration.json b/qapi/migration.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -XXX,XX +XXX,XX @@
 # @multifd-normal-page-ratio: Test hook setting the normal page ratio.
 #     (Since 8.2)
 #
+# @multifd-packet-size: Packet size used to migrate data. This value
+#     indicates the number of pages in a packet. The default value
+#     is 128 and max value is 1024. (Since 8.2)
+#
 # Features:
 #
 # @deprecated: Member @block-incremental is deprecated.  Use
@@ -XXX,XX +XXX,XX @@
            { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] },
            'vcpu-dirty-limit',
            'mode',
-           'multifd-normal-page-ratio'] }
+           'multifd-normal-page-ratio',
+           'multifd-packet-size'] }
 
 ##
 # @MigrateSetParameters:
@@ -XXX,XX +XXX,XX @@
 # @multifd-normal-page-ratio: Test hook setting the normal page ratio.
 #     (Since 8.2)
 #
+# @multifd-packet-size: Packet size used to migrate data. This value
+#     indicates the number of pages in a packet. The default value
+#     is 128 and max value is 1024. (Since 8.2)
+#
 # Features:
 #
 # @deprecated: Member @block-incremental is deprecated.  Use
@@ -XXX,XX +XXX,XX @@
             '*vcpu-dirty-limit': 'uint64',
             '*mode': 'MigMode',
             '*multifd-dsa-accel': 'StrOrNull',
-            '*multifd-normal-page-ratio': 'uint8'} }
+            '*multifd-normal-page-ratio': 'uint8',
+            '*multifd-packet-size' : 'uint64'} }
 
 ##
 # @migrate-set-parameters:
@@ -XXX,XX +XXX,XX @@
 # @multifd-normal-page-ratio: Test hook setting the normal page ratio.
 #     (Since 8.2)
 #
+# @multifd-packet-size: Packet size used to migrate data. This value
+#     indicates the number of pages in a packet. The default value
+#     is 128 and max value is 1024. (Since 8.2)
+#
 # Features:
 #
 # @deprecated: Member @block-incremental is deprecated.  Use
@@ -XXX,XX +XXX,XX @@
             '*vcpu-dirty-limit': 'uint64',
             '*mode': 'MigMode',
             '*multifd-dsa-accel': 'str',
-            '*multifd-normal-page-ratio': 'uint8'} }
+            '*multifd-normal-page-ratio': 'uint8',
+            '*multifd-packet-size': 'uint64'} }
 
 ##
 # @query-migrate-parameters:
-- 
2.30.2

During live migration, if the latency between sender and receiver
is high but bandwidth is high (a long and fat pipe), using a bigger
packet size can help reduce migration total time. In addition, Intel
DSA offloading performs better with a large batch task. Providing an
option to set the packet size is useful for performance tuning.

Set the option:
migrate_set_parameter multifd-packet-size 512

Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 migration/migration-hmp-cmds.c | 7 +++++++
 migration/multifd-zlib.c       | 8 ++++++--
 migration/multifd-zstd.c       | 8 ++++++--
 migration/multifd.c            | 4 ++--
 migration/multifd.h            | 3 ---
 5 files changed, 21 insertions(+), 9 deletions(-)

* Test DSA start and stop path.
* Test DSA configure and cleanup path.
* Test DSA task submission and completion path.

Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 tests/unit/meson.build |   6 +
 tests/unit/test-dsa.c  | 466 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 472 insertions(+)
 create mode 100644 tests/unit/test-dsa.c

diff --git a/tests/unit/meson.build b/tests/unit/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/tests/unit/meson.build
+++ b/tests/unit/meson.build
@@ -XXX,XX +XXX,XX @@ tests = {
   'test-virtio-dmabuf': [meson.project_source_root() / 'hw/display/virtio-dmabuf.c'],
 }
 
+if config_host_data.get('CONFIG_DSA_OPT')
+  tests += {
+    'test-dsa': [],
+  }
+endif
+
 if have_system or have_tools
   tests += {
     'test-qmp-event': [testqapi],
diff --git a/tests/unit/test-dsa.c b/tests/unit/test-dsa.c
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/unit/test-dsa.c
@@ -XXX,XX +XXX,XX @@
+/*
+ * Test DSA functions.
+ *
+ * Copyright (c) 2023 Hao Xiang <hao.xiang@bytedance.com>
+ * Copyright (c) 2023 Bryan Zhang <bryan.zhang@bytedance.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>.
+ */
+#include "qemu/osdep.h"
+#include "qemu/host-utils.h"
+
+#include "qemu/cutils.h"
+#include "qemu/memalign.h"
+#include "qemu/dsa.h"
+
+// TODO Make these not-hardcoded.
+static const char *path1 = "/dev/dsa/wq4.0";
+static const char *path2 = "/dev/dsa/wq4.0 /dev/dsa/wq4.1";
+static const int num_devices = 2;
+
+static struct buffer_zero_batch_task batch_task __attribute__((aligned(64)));
+
+// TODO Communicate that DSA must be configured to support this batch size.
+// TODO Alternatively, poke the DSA device to figure out batch size.
+static int batch_size = 128;
+static int page_size = 4096;
+
+// A helper for running a single task and checking for correctness.
+static void do_single_task(void)
+{
+    buffer_zero_batch_task_init(&batch_task, batch_size);
+    char buf[page_size];
+    char* ptr = buf;
+
+    buffer_is_zero_dsa_batch_async(&batch_task,
+                                   (const void**) &ptr,
+                                   1,
+                                   page_size);
+    g_assert(batch_task.results[0] == buffer_is_zero(buf, page_size));
+}
+
+static void test_single_zero(void)
+{
+    g_assert(!dsa_init(path1));
+    dsa_start();
+
+    buffer_zero_batch_task_init(&batch_task, batch_size);
+
+    char buf[page_size];
+    char* ptr = buf;
+
+    memset(buf, 0x0, page_size);
+    buffer_is_zero_dsa_batch_async(&batch_task,
+                                   (const void**) &ptr,
+                                   1, page_size);
+    g_assert(batch_task.results[0]);
+
+    dsa_cleanup();
+}
+
+static void test_single_zero_async(void)
+{
+    test_single_zero();
+}
+
+static void test_single_nonzero(void)
+{
+    g_assert(!dsa_init(path1));
+    dsa_start();
+
+    buffer_zero_batch_task_init(&batch_task, batch_size);
+
+    char buf[page_size];
+    char* ptr = buf;
+
+    memset(buf, 0x1, page_size);
+    buffer_is_zero_dsa_batch_async(&batch_task,
+                                   (const void**) &ptr,
+                                   1, page_size);
+    g_assert(!batch_task.results[0]);
+
+    dsa_cleanup();
+}
+
+static void test_single_nonzero_async(void)
+{
+    test_single_nonzero();
+}
+
+// count == 0 should return quickly without calling into DSA.
+static void test_zero_count_async(void)
+{
+    char buf[page_size];
+    buffer_is_zero_dsa_batch_async(&batch_task,
+                             (const void **) &buf,
+                             0,
+                             page_size);
+}
+
+static void test_null_task_async(void)
+{
+    if (g_test_subprocess()) {
+        g_assert(!dsa_init(path1));
+
+        char buf[page_size * batch_size];
+        char *addrs[batch_size];
+        for (int i = 0; i < batch_size; i++) {
+            addrs[i] = buf + (page_size * i);
+        }
+
+        buffer_is_zero_dsa_batch_async(NULL, (const void**) addrs, batch_size,
+                                 page_size);
+    } else {
+        g_test_trap_subprocess(NULL, 0, 0);
+        g_test_trap_assert_failed();
+    }
+}
+
+static void test_oversized_batch(void)
+{
+    g_assert(!dsa_init(path1));
+    dsa_start();
+
+    buffer_zero_batch_task_init(&batch_task, batch_size);
+
+    int oversized_batch_size = batch_size + 1;
+    char buf[page_size * oversized_batch_size];
+    char *addrs[batch_size];
+    for (int i = 0; i < oversized_batch_size; i++) {
+        addrs[i] = buf + (page_size * i);
+    }
+
+    int ret = buffer_is_zero_dsa_batch_async(&batch_task,
+                                            (const void**) addrs,
+                                            oversized_batch_size,
+                                            page_size);
+    g_assert(ret != 0);
+
+    dsa_cleanup();
+}
+
+static void test_oversized_batch_async(void)
+{
+    test_oversized_batch();
+}
+
+static void test_zero_len_async(void)
+{
+    if (g_test_subprocess()) {
+        g_assert(!dsa_init(path1));
+
+        buffer_zero_batch_task_init(&batch_task, batch_size);
+
+        char buf[page_size];
+
+        buffer_is_zero_dsa_batch_async(&batch_task,
+                                       (const void**) &buf,
+                                       1,
+                                       0);
+    } else {
+        g_test_trap_subprocess(NULL, 0, 0);
+        g_test_trap_assert_failed();
+    }
+}
+
+static void test_null_buf_async(void)
+{
+    if (g_test_subprocess()) {
+        g_assert(!dsa_init(path1));
+
+        buffer_zero_batch_task_init(&batch_task, batch_size);
+
+        buffer_is_zero_dsa_batch_async(&batch_task, NULL, 1, page_size);
+    } else {
+        g_test_trap_subprocess(NULL, 0, 0);
+        g_test_trap_assert_failed();
+    }
+}
+
+static void test_batch(void)
+{
+    g_assert(!dsa_init(path1));
+    dsa_start();
+
+    buffer_zero_batch_task_init(&batch_task, batch_size);
+
+    char buf[page_size * batch_size];
+    char *addrs[batch_size];
+    for (int i = 0; i < batch_size; i++) {
+        addrs[i] = buf + (page_size * i);
+    }
+
+    // Using whatever is on the stack is somewhat random.
+    // Manually set some pages to zero and some to nonzero.
+    memset(buf + 0, 0, page_size * 10);
+    memset(buf + (10 * page_size), 0xff, page_size * 10);
+
+    buffer_is_zero_dsa_batch_async(&batch_task,
+                                   (const void**) addrs,
+                                   batch_size,
+                                   page_size);
+
+    bool is_zero;
+    for (int i = 0; i < batch_size; i++) {
+        is_zero = buffer_is_zero((const void*) &buf[page_size * i], page_size);
+        g_assert(batch_task.results[i] == is_zero);
+    }
+    dsa_cleanup();
+}
+
+static void test_batch_async(void)
+{
+    test_batch();
+}
+
+static void test_page_fault(void)
+{
+    g_assert(!dsa_init(path1));
+    dsa_start();
+
+    char* buf[2];
+    int prot = PROT_READ | PROT_WRITE;
+    int flags = MAP_SHARED | MAP_ANON;
+    buf[0] = (char*) mmap(NULL, page_size * batch_size, prot, flags, -1, 0);
+    assert(buf[0] != MAP_FAILED);
+    buf[1] = (char*) malloc(page_size * batch_size);
+    assert(buf[1] != NULL);
+
+    for (int j = 0; j < 2; j++) {
+        buffer_zero_batch_task_init(&batch_task, batch_size);
+
+        char *addrs[batch_size];
+        for (int i = 0; i < batch_size; i++) {
+            addrs[i] = buf[j] + (page_size * i);
+        }
+
+        buffer_is_zero_dsa_batch_async(&batch_task,
+                                       (const void**) addrs,
+                                       batch_size,
+                                       page_size);
+
+        bool is_zero;
+        for (int i = 0; i < batch_size; i++) {
+            is_zero = buffer_is_zero((const void*) &buf[j][page_size * i], page_size);
+            g_assert(batch_task.results[i] == is_zero);
+        }
+    }
+
+    assert(!munmap(buf[0], page_size * batch_size));
+    free(buf[1]);
+    dsa_cleanup();
+}
+
+static void test_various_buffer_sizes(void)
+{
+    g_assert(!dsa_init(path1));
+    dsa_start();
+
+    int len = 1 << 4;
+    for (int count = 12; count > 0; count--, len <<= 1) {
+        buffer_zero_batch_task_init(&batch_task, batch_size);
+
+        char buf[len * batch_size];
+        char *addrs[batch_size];
+        for (int i = 0; i < batch_size; i++) {
+            addrs[i] = buf + (len * i);
+        }
+
+        buffer_is_zero_dsa_batch_async(&batch_task,
+                                       (const void**) addrs,
+                                       batch_size,
+                                       len);
+
+        bool is_zero;
+        for (int j = 0; j < batch_size; j++) {
+            is_zero = buffer_is_zero((const void*) &buf[len * j], len);
+            g_assert(batch_task.results[j] == is_zero);
+        }
+    }
+
+    dsa_cleanup();
+}
+
+static void test_various_buffer_sizes_async(void)
+{
+    test_various_buffer_sizes();
+}
+
+static void test_double_start_stop(void)
+{
+    g_assert(!dsa_init(path1));
+    // Double start
+    dsa_start();
+    dsa_start();
+    g_assert(dsa_is_running());
+    do_single_task();
+
+    // Double stop
+    dsa_stop();
+    g_assert(!dsa_is_running());
+    dsa_stop();
+    g_assert(!dsa_is_running());
+
+    // Restart
+    dsa_start();
+    g_assert(dsa_is_running());
+    do_single_task();
+    dsa_cleanup();
+}
+
+static void test_is_running(void)
+{
+    g_assert(!dsa_init(path1));
+
+    g_assert(!dsa_is_running());
+    dsa_start();
+    g_assert(dsa_is_running());
+    dsa_stop();
+    g_assert(!dsa_is_running());
+    dsa_cleanup();
+}
+
+static void test_multiple_engines(void)
+{
+    g_assert(!dsa_init(path2));
+    dsa_start();
+
+    struct buffer_zero_batch_task tasks[num_devices]
+        __attribute__((aligned(64)));
+    char bufs[num_devices][page_size * batch_size];
+    char *addrs[num_devices][batch_size];
+
+    // This is a somewhat implementation-specific way of testing that the tasks
+    // have unique engines assigned to them.
+    buffer_zero_batch_task_init(&tasks[0], batch_size);
+    buffer_zero_batch_task_init(&tasks[1], batch_size);
+    g_assert(tasks[0].device != tasks[1].device);
+
+    for (int i = 0; i < num_devices; i++) {
+        for (int j = 0; j < batch_size; j++) {
+            addrs[i][j] = bufs[i] + (page_size * j);
+        }
+
+        buffer_is_zero_dsa_batch_async(&tasks[i],
+                                       (const void**) addrs[i],
+                                       batch_size, page_size);
+
+        bool is_zero;
+        for (int j = 0; j < batch_size; j++) {
+            is_zero = buffer_is_zero((const void*) &bufs[i][page_size * j],
+                                     page_size);
+            g_assert(tasks[i].results[j] == is_zero);
+        }
+    }
+
+    dsa_cleanup();
+}
+
+static void test_configure_dsa_twice(void)
+{
+    g_assert(!dsa_init(path2));
+    g_assert(!dsa_init(path2));
+    dsa_start();
+    do_single_task();
+    dsa_cleanup();
+}
+
+static void test_configure_dsa_bad_path(void)
+{
+    const char* bad_path = "/not/a/real/path";
+    g_assert(dsa_init(bad_path));
+}
+
+static void test_cleanup_before_configure(void)
+{
+    dsa_cleanup();
+    g_assert(!dsa_init(path2));
+}
+
+static void test_configure_dsa_num_devices(void)
+{
+    g_assert(!dsa_init(path1));
+    dsa_start();
+
+    do_single_task();
+    dsa_stop();
+    dsa_cleanup();
+}
+
+static void test_cleanup_twice(void)
+{
+    g_assert(!dsa_init(path2));
+    dsa_cleanup();
+    dsa_cleanup();
+
+    g_assert(!dsa_init(path2));
+    dsa_start();
+    do_single_task();
+    dsa_cleanup();
+}
+
+static int check_test_setup(void)
+{
+    const char *path[2] = {path1, path2};
+    for (int i = 0; i < sizeof(path) / sizeof(char *); i++) {
+        if (!dsa_init(path[i])) {
+            return -1;
+        }
+        dsa_cleanup();
+    }
+    return 0;
+}
+
+int main(int argc, char **argv)
+{
+    g_test_init(&argc, &argv, NULL);
+
+    if (check_test_setup() != 0) {
+        /*
+         * This test requires extra setup. The current
+         * setup is not correct. Just skip this test
+         * for now.
+         */
+        exit(0);
+    }
+
+    if (num_devices > 1) {
+        g_test_add_func("/dsa/multiple_engines", test_multiple_engines);
+    }
+
+    g_test_add_func("/dsa/async/batch", test_batch_async);
+    g_test_add_func("/dsa/async/various_buffer_sizes",
+                    test_various_buffer_sizes_async);
+    g_test_add_func("/dsa/async/null_buf", test_null_buf_async);
+    g_test_add_func("/dsa/async/zero_len", test_zero_len_async);
+    g_test_add_func("/dsa/async/oversized_batch", test_oversized_batch_async);
+    g_test_add_func("/dsa/async/zero_count", test_zero_count_async);
+    g_test_add_func("/dsa/async/single_zero", test_single_zero_async);
+    g_test_add_func("/dsa/async/single_nonzero", test_single_nonzero_async);
+    g_test_add_func("/dsa/async/null_task", test_null_task_async);
+    g_test_add_func("/dsa/async/page_fault", test_page_fault);
+
+    g_test_add_func("/dsa/double_start_stop", test_double_start_stop);
+    g_test_add_func("/dsa/is_running", test_is_running);
+
+    g_test_add_func("/dsa/configure_dsa_twice", test_configure_dsa_twice);
+    g_test_add_func("/dsa/configure_dsa_bad_path", test_configure_dsa_bad_path);
+    g_test_add_func("/dsa/cleanup_before_configure",
+                    test_cleanup_before_configure);
+    g_test_add_func("/dsa/configure_dsa_num_devices",
+                    test_configure_dsa_num_devices);
+    g_test_add_func("/dsa/cleanup_twice", test_cleanup_twice);
+
+    return g_test_run();
+}
-- 
2.30.2

* Add test case to start and complete multifd live migration with DSA
offloading enabled.
* Add test case to start and cancel multifd live migration with DSA
offloading enabled.

Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@bytedance.com>
---
 tests/qtest/migration-test.c | 77 +++++++++++++++++++++++++++++++++++-
 1 file changed, 76 insertions(+), 1 deletion(-)

diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration-test.c
+++ b/tests/qtest/migration-test.c
@@ -XXX,XX +XXX,XX @@ typedef struct {
     const char *opts_target;
 } MigrateStart;
 
+/*
+ * It requires separate steps to configure and enable DSA device.
+ * This test assumes that the configuration is done already.
+ */
+static const char* dsa_dev_path = "/dev/dsa/wq4.0";
+
 /*
  * A hook that runs after the src and dst QEMUs have been
  * created, but before the migration is started. This can
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_tls_x509_reject_anon_client(void)
  *
  *  And see that it works
  */
-static void test_multifd_tcp_cancel(void)
+static void test_multifd_tcp_cancel_common(bool use_dsa)
 {
     MigrateStart args = {
         .hide_stderr = true,
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_cancel(void)
     migrate_set_capability(from, "multifd", true);
     migrate_set_capability(to, "multifd", true);
 
+    if (use_dsa) {
+        migrate_set_parameter_str(from, "multifd-dsa-accel", dsa_dev_path);
+    }
+
     /* Start incoming migration from the 1st socket */
     migrate_incoming_qmp(to, "tcp:127.0.0.1:0", "{}");
 
@@ -XXX,XX +XXX,XX @@ static void test_multifd_tcp_cancel(void)
     test_migrate_end(from, to2, true);
 }
 
+/*
+ * This test does:
+ *  source               target
+ *                       migrate_incoming
+ *     migrate
+ *     migrate_cancel
+ *                       launch another target
+ *     migrate
+ *
+ *  And see that it works
+ */
+static void test_multifd_tcp_cancel(void)
+{
+    test_multifd_tcp_cancel_common(false);
+}
+
+#ifdef CONFIG_DSA_OPT
+
+static void *test_migrate_precopy_tcp_multifd_start_dsa(QTestState *from,
+                                                        QTestState *to)
+{
+    migrate_set_parameter_str(from, "multifd-dsa-accel", dsa_dev_path);
+    return test_migrate_precopy_tcp_multifd_start_common(from, to, "none");
+}
+
+static void test_multifd_tcp_none_dsa(void)
+{
+    MigrateCommon args = {
+        .listen_uri = "defer",
+        .start_hook = test_migrate_precopy_tcp_multifd_start_dsa,
+    };
+
+    test_precopy_common(&args);
+}
+
+static void test_multifd_tcp_cancel_dsa(void)
+{
+    test_multifd_tcp_cancel_common(true);
+}
+
+#endif
+
 static void calc_dirty_rate(QTestState *who, uint64_t calc_time)
 {
     qtest_qmp_assert_success(who,
@@ -XXX,XX +XXX,XX @@ static bool kvm_dirty_ring_supported(void)
 #endif
 }
 
+#ifdef CONFIG_DSA_OPT
+static int test_dsa_setup(void)
+{
+    int fd;
+    fd = open(dsa_dev_path, O_RDWR);
+    if (fd < 0) {
+        return -1;
+    }
+    close(fd);
+    return 0;
+}
+#endif
+
 int main(int argc, char **argv)
 {
     bool has_kvm, has_tcg;
@@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv)
     }
     qtest_add_func("/migration/multifd/tcp/plain/none",
                    test_multifd_tcp_none);
+
+#ifdef CONFIG_DSA_OPT
+    if (g_str_equal(arch, "x86_64") && test_dsa_setup() == 0) {
+        qtest_add_func("/migration/multifd/tcp/plain/none/dsa",
+                       test_multifd_tcp_none_dsa);
+        qtest_add_func("/migration/multifd/tcp/plain/cancel/dsa",
+                       test_multifd_tcp_cancel_dsa);
+    }
+#endif
+
     /*
      * This test is flaky and sometimes fails in CI and otherwise:
      * don't run unless user opts in via environment variable.
-- 
2.30.2

v9
* Rebase on top of aa3a285b5bc56a4208b3b57d4a55291e9c260107;
* Optimize the error handling in multifd_send_setup();
* Use the correct way for skipping unit test;

v8
* Rebase on top of 1cf9bc6eba7506ab6d9de635f224259225f63466;
* Fixed the hmp parsing crash in migrate_set_parameter;
* Addressed initialization/cleanup comments from v7;

v7
* Rebase on top of f0a5a31c33a8109061c2493e475c8a2f4d022432;
* Fix a bug that will crash QEMU when DSA initialization failed;
* Use a more generalized accel-path to support other accelerators;
* Remove multifd-packet-size in the parameter list;

v6
* Rebase on top of 838fc0a8769d7cc6edfe50451ba4e3368395f5c1;
* Refactor code to have clean history on all commits;
* Add comments on DSA specific defines about how the value is picked;
* Address all comments from v5 reviews about api defines, questions, etc.;

v5
* Rebase on top of 39a032cea23e522268519d89bb738974bc43b6f6.
* Rename struct definitions with typedef and CamelCase names;
* Add build and runtime checks about DSA accelerator;
* Address all comments from v4 reviews about typos, licenses, comments,
error reporting, etc.

v4
* Rebase on top of 85b597413d4370cb168f711192eaef2eb70535ac.
* A separate "multifd zero page checking" patchset was split from this
patchset's v3 and got merged into master. v4 re-applied the rest of all
commits on top of that patchset, re-factored and re-tested.
https://lore.kernel.org/all/20240311180015.3359271-1-hao.xiang@linux.dev/
* There are some feedback from v3 I likely overlooked.

v3
* Rebase on top of 7425b6277f12e82952cede1f531bfc689bf77fb1.
* Fix error/warning from checkpatch.pl
* Fix use-after-free bug when multifd-dsa-accel option is not set.
* Handle error from dsa_init and correctly propogate the error.
* Remove unnecessary call to dsa_stop.
* Detect availability of DSA feature at compile time.
* Implement a generic batch_task structure and a DSA specific one dsa_batch_task.
* Remove all exit() calls and propagate errors correctly.
* Use bytes instead of page count to configure multifd-packet-size option.

* Background:

I posted an RFC about DSA offloading in QEMU:
https://patchew.org/QEMU/20230529182001.2232069-1-hao.xiang@bytedance.com/

This patchset implements the DSA offloading on zero page checking in
multifd live migration code path.

* Overview:

* Design:

* Performance:

We use two Intel 4th generation Xeon servers for testing.

We perform multifd live migration with below setup:
1. VM has 100GB memory.
2. Use the new migration option multifd-set-normal-page-ratio to control the total
size of the payload sent over the network.
3. Use 8 multifd channels.
4. Use tcp for live migration.
4. Use CPU to perform zero page checking as the baseline.
5. Use one DSA device to offload zero page checking to compare with the baseline.
6. Use "perf sched record" and "perf sched timehist" to analyze CPU usage.

A) Scenario 1: 50% (50GB) normal pages on an 100GB vm.

CPU usage

Latency
	|---------------|---------------|---------------|---------------|---------------|---------------|
	|		|total time	|down time	|throughput	|transferred-ram|total-ram	|
	|---------------|---------------|---------------|---------------|---------------|---------------|
	|Baseline	|10343 ms	|161 ms		|41007.00 mbps	|51583797 kb	|102400520 kb	|
	|---------------|---------------|---------------|---------------|-------------------------------|
	|DSA offload	|9535 ms	|135 ms		|46554.40 mbps	|53947545 kb	|102400520 kb	|
	|---------------|---------------|---------------|---------------|---------------|---------------|

Total time is 8% faster and down time is 16% faster.

B) Scenario 2: 100% (100GB) zero pages on an 100GB vm.

Baseline total runtime is 11154 msec and DSA offloading total runtime is
8700 msec. That is 22% CPU savings.

Latency
	|--------------------------------------------------------------------------------------------|
	|		|total time	|down time	|throughput	|transferred-ram|total-ram   |
	|---------------|---------------|---------------|---------------|---------------|------------|
	|Baseline	|4867 ms	|20 ms		|1.51 mbps	|565 kb		|102400520 kb|
	|---------------|---------------|---------------|---------------|----------------------------|
	|DSA offload	|3888 ms	|18 ms		|1.89 mbps	|565 kb		|102400520 kb|
	|---------------|---------------|---------------|---------------|---------------|------------|

Total time 20% faster and down time 10% faster.

* Testing:

1. Added unit tests for cover the added code path in dsa.c
2. Added integration tests to cover multifd live migration using DSA
offloading.

Hao Xiang (10):
  meson: Introduce new instruction set enqcmd to the build system.
  util/dsa: Implement DSA device start and stop logic.
  util/dsa: Implement DSA task enqueue and dequeue.
  util/dsa: Implement DSA task asynchronous completion thread model.
  util/dsa: Implement zero page checking in DSA task.
  util/dsa: Implement DSA task asynchronous submission and wait for
    completion.
  migration/multifd: Add new migration option for multifd DSA
    offloading.
  migration/multifd: Enable DSA offloading in multifd sender path.
  util/dsa: Add unit test coverage for Intel DSA task submission and
    completion.
  migration/multifd: Add integration tests for multifd with Intel DSA
    offloading.

Yichen Wang (1):
  util/dsa: Add idxd into linux header copy list.

Yuan Liu (1):
  migration/doc: Add DSA zero page detection doc

.../migration/dsa-zero-page-detection.rst     |  290 +++++
 docs/devel/migration/features.rst             |    1 +
 hmp-commands.hx                               |    2 +-
 include/qemu/dsa.h                            |  190 +++
 meson.build                                   |   14 +
 meson_options.txt                             |    2 +
 migration/migration-hmp-cmds.c                |   20 +-
 migration/multifd-zero-page.c                 |  149 ++-
 migration/multifd.c                           |   23 +-
 migration/multifd.h                           |    6 +
 migration/options.c                           |   43 +
 migration/options.h                           |    2 +
 qapi/migration.json                           |   32 +-
 scripts/meson-buildoptions.sh                 |    3 +
 scripts/update-linux-headers.sh               |    2 +-
 tests/qtest/meson.build                       |   10 +-
 tests/qtest/migration-test.c                  |    3 +
 tests/qtest/migration/dsa-tests.c             |   59 +
 tests/qtest/migration/framework.h             |    1 +
 tests/unit/meson.build                        |    6 +
 tests/unit/test-dsa.c                         |  504 ++++++++
 util/dsa.c                                    | 1112 +++++++++++++++++
 util/meson.build                              |    3 +
 23 files changed, 2453 insertions(+), 24 deletions(-)
 create mode 100644 docs/devel/migration/dsa-zero-page-detection.rst
 create mode 100644 include/qemu/dsa.h
 create mode 100644 tests/qtest/migration/dsa-tests.c
 create mode 100644 tests/unit/test-dsa.c
 create mode 100644 util/dsa.c

-- 
Yichen Wang

From: Hao Xiang <hao.xiang@linux.dev>

Enable instruction set enqcmd in build.

Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
---
 meson.build                   | 14 ++++++++++++++
 meson_options.txt             |  2 ++
 scripts/meson-buildoptions.sh |  3 +++
 3 files changed, 19 insertions(+)

diff --git a/meson.build b/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/meson.build
+++ b/meson.build
@@ -XXX,XX +XXX,XX @@ config_host_data.set('CONFIG_AVX512BW_OPT', get_option('avx512bw') \
     int main(int argc, char *argv[]) { return bar(argv[0]); }
   '''), error_message: 'AVX512BW not available').allowed())
 
+config_host_data.set('CONFIG_DSA_OPT', get_option('enqcmd') \
+  .require(have_cpuid_h, error_message: 'cpuid.h not available, cannot enable ENQCMD') \
+  .require(cc.links('''
+    #include <stdint.h>
+    #include <cpuid.h>
+    #include <immintrin.h>
+    static int __attribute__((target("enqcmd"))) bar(void *a) {
+      uint64_t dst[8] = { 0 };
+      uint64_t src[8] = { 0 };
+      return _enqcmd(dst, src);
+    }
+    int main(int argc, char *argv[]) { return bar(argv[argc - 1]); }
+  '''), error_message: 'ENQCMD not available').allowed())
+
 # For both AArch64 and AArch32, detect if builtins are available.
 config_host_data.set('CONFIG_ARM_AES_BUILTIN', cc.compiles('''
     #include <arm_neon.h>
diff --git a/meson_options.txt b/meson_options.txt
index XXXXXXX..XXXXXXX 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -XXX,XX +XXX,XX @@ option('avx2', type: 'feature', value: 'auto',
        description: 'AVX2 optimizations')
 option('avx512bw', type: 'feature', value: 'auto',
        description: 'AVX512BW optimizations')
+option('enqcmd', type: 'feature', value: 'disabled',
+       description: 'ENQCMD optimizations')
 option('keyring', type: 'feature', value: 'auto',
        description: 'Linux keyring support')
 option('libkeyutils', type: 'feature', value: 'auto',
diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh
index XXXXXXX..XXXXXXX 100644
--- a/scripts/meson-buildoptions.sh
+++ b/scripts/meson-buildoptions.sh
@@ -XXX,XX +XXX,XX @@ meson_options_help() {
   printf "%s\n" '  auth-pam        PAM access control'
   printf "%s\n" '  avx2            AVX2 optimizations'
   printf "%s\n" '  avx512bw        AVX512BW optimizations'
+  printf "%s\n" '  enqcmd          ENQCMD optimizations'
   printf "%s\n" '  blkio           libblkio block device driver'
   printf "%s\n" '  bochs           bochs image format support'
   printf "%s\n" '  bpf             eBPF support'
@@ -XXX,XX +XXX,XX @@ _meson_option_parse() {
     --disable-avx2) printf "%s" -Davx2=disabled ;;
     --enable-avx512bw) printf "%s" -Davx512bw=enabled ;;
     --disable-avx512bw) printf "%s" -Davx512bw=disabled ;;
+    --enable-enqcmd) printf "%s" -Denqcmd=enabled ;;
+    --disable-enqcmd) printf "%s" -Denqcmd=disabled ;;
     --enable-gcov) printf "%s" -Db_coverage=true ;;
     --disable-gcov) printf "%s" -Db_coverage=false ;;
     --enable-lto) printf "%s" -Db_lto=true ;;
-- 
Yichen Wang

From: Hao Xiang <hao.xiang@linux.dev>

* DSA device open and close.
* DSA group contains multiple DSA devices.
* DSA group configure/start/stop/clean.

Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
---
 include/qemu/dsa.h |  99 ++++++++++++++++
 util/dsa.c         | 280 +++++++++++++++++++++++++++++++++++++++++++++
 util/meson.build   |   3 +
 3 files changed, 382 insertions(+)
 create mode 100644 include/qemu/dsa.h
 create mode 100644 util/dsa.c

diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@
+/*
+ * Interface for using Intel Data Streaming Accelerator to offload certain
+ * background operations.
+ *
+ * Copyright (C) Bytedance Ltd.
+ *
+ * Authors:
+ *  Hao Xiang <hao.xiang@bytedance.com>
+ *  Yichen Wang <yichen.wang@bytedance.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_DSA_H
+#define QEMU_DSA_H
+
+#include "qapi/error.h"
+#include "qemu/thread.h"
+#include "qemu/queue.h"
+
+#ifdef CONFIG_DSA_OPT
+
+#pragma GCC push_options
+#pragma GCC target("enqcmd")
+
+#include <linux/idxd.h>
+#include "x86intrin.h"
+
+typedef struct {
+    void *work_queue;
+} QemuDsaDevice;
+
+typedef QSIMPLEQ_HEAD(QemuDsaTaskQueue, QemuDsaBatchTask) QemuDsaTaskQueue;
+
+typedef struct {
+    QemuDsaDevice *dsa_devices;
+    int num_dsa_devices;
+    /* The index of the next DSA device to be used. */
+    uint32_t device_allocator_index;
+    bool running;
+    QemuMutex task_queue_lock;
+    QemuCond task_queue_cond;
+    QemuDsaTaskQueue task_queue;
+} QemuDsaDeviceGroup;
+
+/**
+ * @brief Initializes DSA devices.
+ *
+ * @param dsa_parameter A list of DSA device path from migration parameter.
+ *
+ * @return int Zero if successful, otherwise non zero.
+ */
+int qemu_dsa_init(const strList *dsa_parameter, Error **errp);
+
+/**
+ * @brief Start logic to enable using DSA.
+ */
+void qemu_dsa_start(void);
+
+/**
+ * @brief Stop the device group and the completion thread.
+ */
+void qemu_dsa_stop(void);
+
+/**
+ * @brief Clean up system resources created for DSA offloading.
+ */
+void qemu_dsa_cleanup(void);
+
+/**
+ * @brief Check if DSA is running.
+ *
+ * @return True if DSA is running, otherwise false.
+ */
+bool qemu_dsa_is_running(void);
+
+#else
+
+static inline bool qemu_dsa_is_running(void)
+{
+    return false;
+}
+
+static inline int qemu_dsa_init(const strList *dsa_parameter, Error **errp)
+{
+    error_setg(errp, "DSA accelerator is not enabled.");
+    return -1;
+}
+
+static inline void qemu_dsa_start(void) {}
+
+static inline void qemu_dsa_stop(void) {}
+
+static inline void qemu_dsa_cleanup(void) {}
+
+#endif
+
+#endif
diff --git a/util/dsa.c b/util/dsa.c
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@
+/*
+ * Use Intel Data Streaming Accelerator to offload certain background
+ * operations.
+ *
+ * Copyright (C) Bytedance Ltd.
+ *
+ * Authors:
+ *  Hao Xiang <hao.xiang@bytedance.com>
+ *  Bryan Zhang <bryan.zhang@bytedance.com>
+ *  Yichen Wang <yichen.wang@bytedance.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/queue.h"
+#include "qemu/memalign.h"
+#include "qemu/lockable.h"
+#include "qemu/cutils.h"
+#include "qemu/dsa.h"
+#include "qemu/bswap.h"
+#include "qemu/error-report.h"
+#include "qemu/rcu.h"
+
+#pragma GCC push_options
+#pragma GCC target("enqcmd")
+
+#include <linux/idxd.h>
+#include "x86intrin.h"
+
+#define DSA_WQ_PORTAL_SIZE 4096
+#define MAX_DSA_DEVICES 16
+
+uint32_t max_retry_count;
+static QemuDsaDeviceGroup dsa_group;
+
+
+/**
+ * @brief This function opens a DSA device's work queue and
+ *        maps the DSA device memory into the current process.
+ *
+ * @param dsa_wq_path A pointer to the DSA device work queue's file path.
+ * @return A pointer to the mapped memory, or MAP_FAILED on failure.
+ */
+static void *
+map_dsa_device(const char *dsa_wq_path)
+{
+    void *dsa_device;
+    int fd;
+
+    fd = open(dsa_wq_path, O_RDWR);
+    if (fd < 0) {
+        error_report("Open %s failed with errno = %d.",
+                dsa_wq_path, errno);
+        return MAP_FAILED;
+    }
+    dsa_device = mmap(NULL, DSA_WQ_PORTAL_SIZE, PROT_WRITE,
+                      MAP_SHARED | MAP_POPULATE, fd, 0);
+    close(fd);
+    if (dsa_device == MAP_FAILED) {
+        error_report("mmap failed with errno = %d.", errno);
+        return MAP_FAILED;
+    }
+    return dsa_device;
+}
+
+/**
+ * @brief Initializes a DSA device structure.
+ *
+ * @param instance A pointer to the DSA device.
+ * @param work_queue A pointer to the DSA work queue.
+ */
+static void
+dsa_device_init(QemuDsaDevice *instance,
+                void *dsa_work_queue)
+{
+    instance->work_queue = dsa_work_queue;
+}
+
+/**
+ * @brief Cleans up a DSA device structure.
+ *
+ * @param instance A pointer to the DSA device to cleanup.
+ */
+static void
+dsa_device_cleanup(QemuDsaDevice *instance)
+{
+    if (instance->work_queue != MAP_FAILED) {
+        munmap(instance->work_queue, DSA_WQ_PORTAL_SIZE);
+    }
+}
+
+/**
+ * @brief Initializes a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ * @param dsa_parameter A list of DSA device path from are separated by space
+ * character migration parameter. Multiple DSA device path.
+ *
+ * @return Zero if successful, non-zero otherwise.
+ */
+static int
+dsa_device_group_init(QemuDsaDeviceGroup *group,
+                      const strList *dsa_parameter,
+                      Error **errp)
+{
+    if (dsa_parameter == NULL) {
+        error_setg(errp, "dsa device path is not supplied.");
+        return -1;
+    }
+
+    int ret = 0;
+    const char *dsa_path[MAX_DSA_DEVICES];
+    int num_dsa_devices = 0;
+
+    while (dsa_parameter) {
+        dsa_path[num_dsa_devices++] = dsa_parameter->value;
+        if (num_dsa_devices == MAX_DSA_DEVICES) {
+            break;
+        }
+        dsa_parameter = dsa_parameter->next;
+    }
+
+    group->dsa_devices =
+        g_new0(QemuDsaDevice, num_dsa_devices);
+    group->num_dsa_devices = num_dsa_devices;
+    group->device_allocator_index = 0;
+
+    group->running = false;
+    qemu_mutex_init(&group->task_queue_lock);
+    qemu_cond_init(&group->task_queue_cond);
+    QSIMPLEQ_INIT(&group->task_queue);
+
+    void *dsa_wq = MAP_FAILED;
+    for (int i = 0; i < num_dsa_devices; i++) {
+        dsa_wq = map_dsa_device(dsa_path[i]);
+        if (dsa_wq == MAP_FAILED && ret != -1) {
+            error_setg(errp, "map_dsa_device failed MAP_FAILED.");
+            ret = -1;
+        }
+        dsa_device_init(&group->dsa_devices[i], dsa_wq);
+    }
+
+    return ret;
+}
+
+/**
+ * @brief Starts a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_device_group_start(QemuDsaDeviceGroup *group)
+{
+    group->running = true;
+}
+
+/**
+ * @brief Stops a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+__attribute__((unused))
+static void
+dsa_device_group_stop(QemuDsaDeviceGroup *group)
+{
+    group->running = false;
+}
+
+/**
+ * @brief Cleans up a DSA device group.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_device_group_cleanup(QemuDsaDeviceGroup *group)
+{
+    if (!group->dsa_devices) {
+        return;
+    }
+    for (int i = 0; i < group->num_dsa_devices; i++) {
+        dsa_device_cleanup(&group->dsa_devices[i]);
+    }
+    g_free(group->dsa_devices);
+    group->dsa_devices = NULL;
+
+    qemu_mutex_destroy(&group->task_queue_lock);
+    qemu_cond_destroy(&group->task_queue_cond);
+}
+
+/**
+ * @brief Returns the next available DSA device in the group.
+ *
+ * @param group A pointer to the DSA device group.
+ *
+ * @return struct QemuDsaDevice* A pointer to the next available DSA device
+ *         in the group.
+ */
+__attribute__((unused))
+static QemuDsaDevice *
+dsa_device_group_get_next_device(QemuDsaDeviceGroup *group)
+{
+    if (group->num_dsa_devices == 0) {
+        return NULL;
+    }
+    uint32_t current = qatomic_fetch_inc(&group->device_allocator_index);
+    current %= group->num_dsa_devices;
+    return &group->dsa_devices[current];
+}
+
+/**
+ * @brief Check if DSA is running.
+ *
+ * @return True if DSA is running, otherwise false.
+ */
+bool qemu_dsa_is_running(void)
+{
+    return false;
+}
+
+static void
+dsa_globals_init(void)
+{
+    max_retry_count = UINT32_MAX;
+}
+
+/**
+ * @brief Initializes DSA devices.
+ *
+ * @param dsa_parameter A list of DSA device path from migration parameter.
+ *
+ * @return int Zero if successful, otherwise non zero.
+ */
+int qemu_dsa_init(const strList *dsa_parameter, Error **errp)
+{
+    dsa_globals_init();
+
+    return dsa_device_group_init(&dsa_group, dsa_parameter, errp);
+}
+
+/**
+ * @brief Start logic to enable using DSA.
+ *
+ */
+void qemu_dsa_start(void)
+{
+    if (dsa_group.num_dsa_devices == 0) {
+        return;
+    }
+    if (dsa_group.running) {
+        return;
+    }
+    dsa_device_group_start(&dsa_group);
+}
+
+/**
+ * @brief Stop the device group and the completion thread.
+ *
+ */
+void qemu_dsa_stop(void)
+{
+    QemuDsaDeviceGroup *group = &dsa_group;
+
+    if (!group->running) {
+        return;
+    }
+}
+
+/**
+ * @brief Clean up system resources created for DSA offloading.
+ *
+ */
+void qemu_dsa_cleanup(void)
+{
+    qemu_dsa_stop();
+    dsa_device_group_cleanup(&dsa_group);
+}
+
diff --git a/util/meson.build b/util/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/util/meson.build
+++ b/util/meson.build
@@ -XXX,XX +XXX,XX @@ if cpu == 'aarch64'
   util_ss.add(files('cpuinfo-aarch64.c'))
 elif cpu in ['x86', 'x86_64']
   util_ss.add(files('cpuinfo-i386.c'))
+  if config_host_data.get('CONFIG_DSA_OPT')
+    util_ss.add(files('dsa.c'))
+  endif
 elif cpu == 'loongarch64'
   util_ss.add(files('cpuinfo-loongarch.c'))
 elif cpu in ['ppc', 'ppc64']
-- 
Yichen Wang

From: Hao Xiang <hao.xiang@linux.dev>

* Use a safe thread queue for DSA task enqueue/dequeue.
* Implement DSA task submission.
* Implement DSA batch task submission.

Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
---
 include/qemu/dsa.h |  29 +++++++
 util/dsa.c         | 186 ++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 214 insertions(+), 1 deletion(-)

diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@
 #include <linux/idxd.h>
 #include "x86intrin.h"
 
+typedef enum QemuDsaTaskType {
+    QEMU_DSA_TASK = 0,
+    QEMU_DSA_BATCH_TASK
+} QemuDsaTaskType;
+
+typedef enum QemuDsaTaskStatus {
+    QEMU_DSA_TASK_READY = 0,
+    QEMU_DSA_TASK_PROCESSING,
+    QEMU_DSA_TASK_COMPLETION
+} QemuDsaTaskStatus;
+
 typedef struct {
     void *work_queue;
 } QemuDsaDevice;
@@ -XXX,XX +XXX,XX @@ typedef struct {
     QemuDsaTaskQueue task_queue;
 } QemuDsaDeviceGroup;
 
+typedef void (*qemu_dsa_completion_fn)(void *);
+
+typedef struct QemuDsaBatchTask {
+    struct dsa_hw_desc batch_descriptor;
+    struct dsa_hw_desc *descriptors;
+    struct dsa_completion_record batch_completion __attribute__((aligned(32)));
+    struct dsa_completion_record *completions;
+    QemuDsaDeviceGroup *group;
+    QemuDsaDevice *device;
+    qemu_dsa_completion_fn completion_callback;
+    QemuSemaphore sem_task_complete;
+    QemuDsaTaskType task_type;
+    QemuDsaTaskStatus status;
+    int batch_size;
+    QSIMPLEQ_ENTRY(QemuDsaBatchTask) entry;
+} QemuDsaBatchTask;
+
+
 /**
  * @brief Initializes DSA devices.
  *
diff --git a/util/dsa.c b/util/dsa.c
index XXXXXXX..XXXXXXX 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@
 #include "x86intrin.h"
 
 #define DSA_WQ_PORTAL_SIZE 4096
+#define DSA_WQ_DEPTH 128
 #define MAX_DSA_DEVICES 16
 
 uint32_t max_retry_count;
@@ -XXX,XX +XXX,XX @@ dsa_device_group_get_next_device(QemuDsaDeviceGroup *group)
     return &group->dsa_devices[current];
 }
 
+/**
+ * @brief Empties out the DSA task queue.
+ *
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_empty_task_queue(QemuDsaDeviceGroup *group)
+{
+    qemu_mutex_lock(&group->task_queue_lock);
+    QemuDsaTaskQueue *task_queue = &group->task_queue;
+    while (!QSIMPLEQ_EMPTY(task_queue)) {
+        QSIMPLEQ_REMOVE_HEAD(task_queue, entry);
+    }
+    qemu_mutex_unlock(&group->task_queue_lock);
+}
+
+/**
+ * @brief Adds a task to the DSA task queue.
+ *
+ * @param group A pointer to the DSA device group.
+ * @param task A pointer to the DSA task to enqueue.
+ *
+ * @return int Zero if successful, otherwise a proper error code.
+ */
+static int
+dsa_task_enqueue(QemuDsaDeviceGroup *group,
+                 QemuDsaBatchTask *task)
+{
+    bool notify = false;
+
+    qemu_mutex_lock(&group->task_queue_lock);
+
+    if (!group->running) {
+        error_report("DSA: Tried to queue task to stopped device queue.");
+        qemu_mutex_unlock(&group->task_queue_lock);
+        return -1;
+    }
+
+    /* The queue is empty. This enqueue operation is a 0->1 transition. */
+    if (QSIMPLEQ_EMPTY(&group->task_queue)) {
+        notify = true;
+    }
+
+    QSIMPLEQ_INSERT_TAIL(&group->task_queue, task, entry);
+
+    /* We need to notify the waiter for 0->1 transitions. */
+    if (notify) {
+        qemu_cond_signal(&group->task_queue_cond);
+    }
+
+    qemu_mutex_unlock(&group->task_queue_lock);
+
+    return 0;
+}
+
+/**
+ * @brief Takes a DSA task out of the task queue.
+ *
+ * @param group A pointer to the DSA device group.
+ * @return QemuDsaBatchTask* The DSA task being dequeued.
+ */
+__attribute__((unused))
+static QemuDsaBatchTask *
+dsa_task_dequeue(QemuDsaDeviceGroup *group)
+{
+    QemuDsaBatchTask *task = NULL;
+
+    qemu_mutex_lock(&group->task_queue_lock);
+
+    while (true) {
+        if (!group->running) {
+            goto exit;
+        }
+        task = QSIMPLEQ_FIRST(&group->task_queue);
+        if (task != NULL) {
+            break;
+        }
+        qemu_cond_wait(&group->task_queue_cond, &group->task_queue_lock);
+    }
+
+    QSIMPLEQ_REMOVE_HEAD(&group->task_queue, entry);
+
+exit:
+    qemu_mutex_unlock(&group->task_queue_lock);
+    return task;
+}
+
+/**
+ * @brief Submits a DSA work item to the device work queue.
+ *
+ * @param wq A pointer to the DSA work queue's device memory.
+ * @param descriptor A pointer to the DSA work item descriptor.
+ *
+ * @return Zero if successful, non-zero otherwise.
+ */
+static int
+submit_wi_int(void *wq, struct dsa_hw_desc *descriptor)
+{
+    uint32_t retry = 0;
+
+    _mm_sfence();
+
+    while (true) {
+        if (_enqcmd(wq, descriptor) == 0) {
+            break;
+        }
+        retry++;
+        if (retry > max_retry_count) {
+            error_report("Submit work retry %u times.", retry);
+            return -1;
+        }
+    }
+
+    return 0;
+}
+
+/**
+ * @brief Asynchronously submits a DSA work item to the
+ *        device work queue.
+ *
+ * @param task A pointer to the task.
+ *
+ * @return int Zero if successful, non-zero otherwise.
+ */
+__attribute__((unused))
+static int
+submit_wi_async(QemuDsaBatchTask *task)
+{
+    QemuDsaDeviceGroup *device_group = task->group;
+    QemuDsaDevice *device_instance = task->device;
+    int ret;
+
+    assert(task->task_type == QEMU_DSA_TASK);
+
+    task->status = QEMU_DSA_TASK_PROCESSING;
+
+    ret = submit_wi_int(device_instance->work_queue,
+                        &task->descriptors[0]);
+    if (ret != 0) {
+        return ret;
+    }
+
+    return dsa_task_enqueue(device_group, task);
+}
+
+/**
+ * @brief Asynchronously submits a DSA batch work item to the
+ *        device work queue.
+ *
+ * @param batch_task A pointer to the batch task.
+ *
+ * @return int Zero if successful, non-zero otherwise.
+ */
+__attribute__((unused))
+static int
+submit_batch_wi_async(QemuDsaBatchTask *batch_task)
+{
+    QemuDsaDeviceGroup *device_group = batch_task->group;
+    QemuDsaDevice *device_instance = batch_task->device;
+    int ret;
+
+    assert(batch_task->task_type == QEMU_DSA_BATCH_TASK);
+    assert(batch_task->batch_descriptor.desc_count <= batch_task->batch_size);
+    assert(batch_task->status == QEMU_DSA_TASK_READY);
+
+    batch_task->status = QEMU_DSA_TASK_PROCESSING;
+
+    ret = submit_wi_int(device_instance->work_queue,
+                        &batch_task->batch_descriptor);
+    if (ret != 0) {
+        return ret;
+    }
+
+    return dsa_task_enqueue(device_group, batch_task);
+}
+
 /**
  * @brief Check if DSA is running.
  *
@@ -XXX,XX +XXX,XX @@ bool qemu_dsa_is_running(void)
 static void
 dsa_globals_init(void)
 {
-    max_retry_count = UINT32_MAX;
+    /*
+     * This value follows a reference example by Intel. The POLL_RETRY_MAX is
+     * defined to 10000, so here we used the max WQ depth * 100 for the the max
+     * polling retry count.
+     */
+    max_retry_count = DSA_WQ_DEPTH * 100;
 }
 
 /**
@@ -XXX,XX +XXX,XX @@ void qemu_dsa_stop(void)
     if (!group->running) {
         return;
     }
+
+    dsa_empty_task_queue(group);
 }
 
 /**
-- 
Yichen Wang

From: Hao Xiang <hao.xiang@linux.dev>

* Create a dedicated thread for DSA task completion.
* DSA completion thread runs a loop and poll for completed tasks.
* Start and stop DSA completion thread during DSA device start stop.

Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
---
 include/qemu/dsa.h |   1 +
 util/dsa.c         | 272 ++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 272 insertions(+), 1 deletion(-)

diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@ typedef struct QemuDsaBatchTask {
     QemuDsaTaskType task_type;
     QemuDsaTaskStatus status;
     int batch_size;
+    bool *results;
     QSIMPLEQ_ENTRY(QemuDsaBatchTask) entry;
 } QemuDsaBatchTask;
 
diff --git a/util/dsa.c b/util/dsa.c
index XXXXXXX..XXXXXXX 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@
 #define DSA_WQ_PORTAL_SIZE 4096
 #define DSA_WQ_DEPTH 128
 #define MAX_DSA_DEVICES 16
+#define DSA_COMPLETION_THREAD "qemu_dsa_completion"
+
+typedef struct {
+    bool stopping;
+    bool running;
+    QemuThread thread;
+    int thread_id;
+    QemuSemaphore sem_init_done;
+    QemuDsaDeviceGroup *group;
+} QemuDsaCompletionThread;
 
 uint32_t max_retry_count;
 static QemuDsaDeviceGroup dsa_group;
+static QemuDsaCompletionThread completion_thread;
 
 
 /**
@@ -XXX,XX +XXX,XX @@ submit_batch_wi_async(QemuDsaBatchTask *batch_task)
     return dsa_task_enqueue(device_group, batch_task);
 }
 
+/**
+ * @brief Poll for the DSA work item completion.
+ *
+ * @param completion A pointer to the DSA work item completion record.
+ * @param opcode The DSA opcode.
+ *
+ * @return Zero if successful, non-zero otherwise.
+ */
+static int
+poll_completion(struct dsa_completion_record *completion,
+                enum dsa_opcode opcode)
+{
+    uint8_t status;
+    uint64_t retry = 0;
+
+    while (true) {
+        /* The DSA operation completes successfully or fails. */
+        status = completion->status;
+        if (status == DSA_COMP_SUCCESS ||
+            status == DSA_COMP_PAGE_FAULT_NOBOF ||
+            status == DSA_COMP_BATCH_PAGE_FAULT ||
+            status == DSA_COMP_BATCH_FAIL) {
+            break;
+        } else if (status != DSA_COMP_NONE) {
+            error_report("DSA opcode %d failed with status = %d.",
+                    opcode, status);
+            return 1;
+        }
+        retry++;
+        if (retry > max_retry_count) {
+            error_report("DSA wait for completion retry %lu times.", retry);
+            return 1;
+        }
+        _mm_pause();
+    }
+
+    return 0;
+}
+
+/**
+ * @brief Complete a single DSA task in the batch task.
+ *
+ * @param task A pointer to the batch task structure.
+ *
+ * @return Zero if successful, otherwise non-zero.
+ */
+static int
+poll_task_completion(QemuDsaBatchTask *task)
+{
+    assert(task->task_type == QEMU_DSA_TASK);
+
+    struct dsa_completion_record *completion = &task->completions[0];
+    uint8_t status;
+    int ret;
+
+    ret = poll_completion(completion, task->descriptors[0].opcode);
+    if (ret != 0) {
+        goto exit;
+    }
+
+    status = completion->status;
+    if (status == DSA_COMP_SUCCESS) {
+        task->results[0] = (completion->result == 0);
+        goto exit;
+    }
+
+    assert(status == DSA_COMP_PAGE_FAULT_NOBOF);
+
+exit:
+    return ret;
+}
+
+/**
+ * @brief Poll a batch task status until it completes. If DSA task doesn't
+ *        complete properly, use CPU to complete the task.
+ *
+ * @param batch_task A pointer to the DSA batch task.
+ *
+ * @return Zero if successful, otherwise non-zero.
+ */
+static int
+poll_batch_task_completion(QemuDsaBatchTask *batch_task)
+{
+    struct dsa_completion_record *batch_completion =
+        &batch_task->batch_completion;
+    struct dsa_completion_record *completion;
+    uint8_t batch_status;
+    uint8_t status;
+    bool *results = batch_task->results;
+    uint32_t count = batch_task->batch_descriptor.desc_count;
+    int ret;
+
+    ret = poll_completion(batch_completion,
+                          batch_task->batch_descriptor.opcode);
+    if (ret != 0) {
+        goto exit;
+    }
+
+    batch_status = batch_completion->status;
+
+    if (batch_status == DSA_COMP_SUCCESS) {
+        if (batch_completion->bytes_completed == count) {
+            /*
+             * Let's skip checking for each descriptors' completion status
+             * if the batch descriptor says all succedded.
+             */
+            for (int i = 0; i < count; i++) {
+                assert(batch_task->completions[i].status == DSA_COMP_SUCCESS);
+                results[i] = (batch_task->completions[i].result == 0);
+            }
+            goto exit;
+        }
+    } else {
+        assert(batch_status == DSA_COMP_BATCH_FAIL ||
+            batch_status == DSA_COMP_BATCH_PAGE_FAULT);
+    }
+
+    for (int i = 0; i < count; i++) {
+
+        completion = &batch_task->completions[i];
+        status = completion->status;
+
+        if (status == DSA_COMP_SUCCESS) {
+            results[i] = (completion->result == 0);
+            continue;
+        }
+
+        if (status != DSA_COMP_PAGE_FAULT_NOBOF) {
+            error_report("Unexpected DSA completion status = %u.", status);
+            ret = 1;
+            goto exit;
+        }
+    }
+
+exit:
+    return ret;
+}
+
+/**
+ * @brief Handles an asynchronous DSA batch task completion.
+ *
+ * @param task A pointer to the batch buffer zero task structure.
+ */
+static void
+dsa_batch_task_complete(QemuDsaBatchTask *batch_task)
+{
+    batch_task->status = QEMU_DSA_TASK_COMPLETION;
+    batch_task->completion_callback(batch_task);
+}
+
+/**
+ * @brief The function entry point called by a dedicated DSA
+ *        work item completion thread.
+ *
+ * @param opaque A pointer to the thread context.
+ *
+ * @return void* Not used.
+ */
+static void *
+dsa_completion_loop(void *opaque)
+{
+    QemuDsaCompletionThread *thread_context =
+        (QemuDsaCompletionThread *)opaque;
+    QemuDsaBatchTask *batch_task;
+    QemuDsaDeviceGroup *group = thread_context->group;
+    int ret;
+
+    rcu_register_thread();
+
+    thread_context->thread_id = qemu_get_thread_id();
+    qemu_sem_post(&thread_context->sem_init_done);
+
+    while (thread_context->running) {
+        batch_task = dsa_task_dequeue(group);
+        assert(batch_task != NULL || !group->running);
+        if (!group->running) {
+            assert(!thread_context->running);
+            break;
+        }
+        if (batch_task->task_type == QEMU_DSA_TASK) {
+            ret = poll_task_completion(batch_task);
+        } else {
+            assert(batch_task->task_type == QEMU_DSA_BATCH_TASK);
+            ret = poll_batch_task_completion(batch_task);
+        }
+
+        if (ret != 0) {
+            goto exit;
+        }
+
+        dsa_batch_task_complete(batch_task);
+    }
+
+exit:
+    if (ret != 0) {
+        error_report("DSA completion thread exited due to internal error.");
+    }
+    rcu_unregister_thread();
+    return NULL;
+}
+
+/**
+ * @brief Initializes a DSA completion thread.
+ *
+ * @param completion_thread A pointer to the completion thread context.
+ * @param group A pointer to the DSA device group.
+ */
+static void
+dsa_completion_thread_init(
+    QemuDsaCompletionThread *completion_thread,
+    QemuDsaDeviceGroup *group)
+{
+    completion_thread->stopping = false;
+    completion_thread->running = true;
+    completion_thread->thread_id = -1;
+    qemu_sem_init(&completion_thread->sem_init_done, 0);
+    completion_thread->group = group;
+
+    qemu_thread_create(&completion_thread->thread,
+                       DSA_COMPLETION_THREAD,
+                       dsa_completion_loop,
+                       completion_thread,
+                       QEMU_THREAD_JOINABLE);
+
+    /* Wait for initialization to complete */
+    qemu_sem_wait(&completion_thread->sem_init_done);
+}
+
+/**
+ * @brief Stops the completion thread (and implicitly, the device group).
+ *
+ * @param opaque A pointer to the completion thread.
+ */
+static void dsa_completion_thread_stop(void *opaque)
+{
+    QemuDsaCompletionThread *thread_context =
+        (QemuDsaCompletionThread *)opaque;
+
+    QemuDsaDeviceGroup *group = thread_context->group;
+
+    qemu_mutex_lock(&group->task_queue_lock);
+
+    thread_context->stopping = true;
+    thread_context->running = false;
+
+    /* Prevent the compiler from setting group->running first. */
+    barrier();
+    dsa_device_group_stop(group);
+
+    qemu_cond_signal(&group->task_queue_cond);
+    qemu_mutex_unlock(&group->task_queue_lock);
+
+    qemu_thread_join(&thread_context->thread);
+
+    qemu_sem_destroy(&thread_context->sem_init_done);
+}
+
 /**
  * @brief Check if DSA is running.
  *
@@ -XXX,XX +XXX,XX @@ submit_batch_wi_async(QemuDsaBatchTask *batch_task)
  */
 bool qemu_dsa_is_running(void)
 {
-    return false;
+    return completion_thread.running;
 }
 
 static void
@@ -XXX,XX +XXX,XX @@ void qemu_dsa_start(void)
         return;
     }
     dsa_device_group_start(&dsa_group);
+    dsa_completion_thread_init(&completion_thread, &dsa_group);
 }
 
 /**
@@ -XXX,XX +XXX,XX @@ void qemu_dsa_stop(void)
         return;
     }
 
+    dsa_completion_thread_stop(&completion_thread);
     dsa_empty_task_queue(group);
 }
 
-- 
Yichen Wang

From: Hao Xiang <hao.xiang@linux.dev>

Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
---
 include/qemu/dsa.h |  36 ++++++-
 util/dsa.c         | 240 ++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 270 insertions(+), 6 deletions(-)

diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@
 #define QEMU_DSA_H
 
 #include "qapi/error.h"
+#include "exec/cpu-common.h"
 #include "qemu/thread.h"
 #include "qemu/queue.h"
 
@@ -XXX,XX +XXX,XX @@ typedef struct QemuDsaBatchTask {
     QemuDsaTaskStatus status;
     int batch_size;
     bool *results;
+    /* Address of each pages in pages */
+    ram_addr_t *addr;
     QSIMPLEQ_ENTRY(QemuDsaBatchTask) entry;
 } QemuDsaBatchTask;
 
-
 /**
  * @brief Initializes DSA devices.
  *
@@ -XXX,XX +XXX,XX @@ void qemu_dsa_cleanup(void);
  */
 bool qemu_dsa_is_running(void);
 
+/**
+ * @brief Initializes a buffer zero DSA batch task.
+ *
+ * @param batch_size The number of zero page checking tasks in the batch.
+ * @return A pointer to the zero page checking tasks initialized.
+ */
+QemuDsaBatchTask *
+buffer_zero_batch_task_init(int batch_size);
+
+/**
+ * @brief Performs the proper cleanup on a DSA batch task.
+ *
+ * @param task A pointer to the batch task to cleanup.
+ */
+void buffer_zero_batch_task_destroy(QemuDsaBatchTask *task);
+
 #else
 
+typedef struct QemuDsaBatchTask {} QemuDsaBatchTask;
+
 static inline bool qemu_dsa_is_running(void)
 {
     return false;
@@ -XXX,XX +XXX,XX @@ static inline void qemu_dsa_stop(void) {}
 
 static inline void qemu_dsa_cleanup(void) {}
 
+static inline QemuDsaBatchTask *buffer_zero_batch_task_init(int batch_size)
+{
+    return NULL;
+}
+
+static inline void buffer_zero_batch_task_destroy(QemuDsaBatchTask *task) {}
+
+static inline int
+buffer_is_zero_dsa_batch_sync(QemuDsaBatchTask *batch_task,
+                              const void **buf, size_t count, size_t len)
+{
+    return -1;
+}
+
 #endif
 
 #endif
diff --git a/util/dsa.c b/util/dsa.c
index XXXXXXX..XXXXXXX 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@ uint32_t max_retry_count;
 static QemuDsaDeviceGroup dsa_group;
 static QemuDsaCompletionThread completion_thread;
 
+static void buffer_zero_dsa_completion(void *context);
 
 /**
  * @brief This function opens a DSA device's work queue and
@@ -XXX,XX +XXX,XX @@ dsa_device_group_start(QemuDsaDeviceGroup *group)
  *
  * @param group A pointer to the DSA device group.
  */
-__attribute__((unused))
 static void
 dsa_device_group_stop(QemuDsaDeviceGroup *group)
 {
@@ -XXX,XX +XXX,XX @@ dsa_device_group_cleanup(QemuDsaDeviceGroup *group)
  * @return struct QemuDsaDevice* A pointer to the next available DSA device
  *         in the group.
  */
-__attribute__((unused))
 static QemuDsaDevice *
 dsa_device_group_get_next_device(QemuDsaDeviceGroup *group)
 {
@@ -XXX,XX +XXX,XX @@ dsa_task_enqueue(QemuDsaDeviceGroup *group,
  * @param group A pointer to the DSA device group.
  * @return QemuDsaBatchTask* The DSA task being dequeued.
  */
-__attribute__((unused))
 static QemuDsaBatchTask *
 dsa_task_dequeue(QemuDsaDeviceGroup *group)
 {
@@ -XXX,XX +XXX,XX @@ submit_wi_int(void *wq, struct dsa_hw_desc *descriptor)
  *
  * @return int Zero if successful, non-zero otherwise.
  */
-__attribute__((unused))
 static int
 submit_wi_async(QemuDsaBatchTask *task)
 {
@@ -XXX,XX +XXX,XX @@ submit_wi_async(QemuDsaBatchTask *task)
  *
  * @return int Zero if successful, non-zero otherwise.
  */
-__attribute__((unused))
 static int
 submit_batch_wi_async(QemuDsaBatchTask *batch_task)
 {
@@ -XXX,XX +XXX,XX @@ void qemu_dsa_cleanup(void)
     dsa_device_group_cleanup(&dsa_group);
 }
 
+
+/* Buffer zero comparison DSA task implementations */
+/* =============================================== */
+
+/**
+ * @brief Sets a buffer zero comparison DSA task.
+ *
+ * @param descriptor A pointer to the DSA task descriptor.
+ * @param buf A pointer to the memory buffer.
+ * @param len The length of the buffer.
+ */
+static void
+buffer_zero_task_set_int(struct dsa_hw_desc *descriptor,
+                         const void *buf,
+                         size_t len)
+{
+    struct dsa_completion_record *completion =
+        (struct dsa_completion_record *)descriptor->completion_addr;
+
+    descriptor->xfer_size = len;
+    descriptor->src_addr = (uintptr_t)buf;
+    completion->status = 0;
+    completion->result = 0;
+}
+
+/**
+ * @brief Resets a buffer zero comparison DSA batch task.
+ *
+ * @param task A pointer to the DSA batch task.
+ */
+static void
+buffer_zero_task_reset(QemuDsaBatchTask *task)
+{
+    task->completions[0].status = DSA_COMP_NONE;
+    task->task_type = QEMU_DSA_TASK;
+    task->status = QEMU_DSA_TASK_READY;
+}
+
+/**
+ * @brief Resets a buffer zero comparison DSA batch task.
+ *
+ * @param task A pointer to the batch task.
+ * @param count The number of DSA tasks this batch task will contain.
+ */
+static void
+buffer_zero_batch_task_reset(QemuDsaBatchTask *task, size_t count)
+{
+    task->batch_completion.status = DSA_COMP_NONE;
+    task->batch_descriptor.desc_count = count;
+    task->task_type = QEMU_DSA_BATCH_TASK;
+    task->status = QEMU_DSA_TASK_READY;
+}
+
+/**
+ * @brief Sets a buffer zero comparison DSA task.
+ *
+ * @param task A pointer to the DSA task.
+ * @param buf A pointer to the memory buffer.
+ * @param len The buffer length.
+ */
+static void
+buffer_zero_task_set(QemuDsaBatchTask *task,
+                     const void *buf,
+                     size_t len)
+{
+    buffer_zero_task_reset(task);
+    buffer_zero_task_set_int(&task->descriptors[0], buf, len);
+}
+
+/**
+ * @brief Sets a buffer zero comparison batch task.
+ *
+ * @param batch_task A pointer to the batch task.
+ * @param buf An array of memory buffers.
+ * @param count The number of buffers in the array.
+ * @param len The length of the buffers.
+ */
+static void
+buffer_zero_batch_task_set(QemuDsaBatchTask *batch_task,
+                           const void **buf, size_t count, size_t len)
+{
+    assert(count > 0);
+    assert(count <= batch_task->batch_size);
+
+    buffer_zero_batch_task_reset(batch_task, count);
+    for (int i = 0; i < count; i++) {
+        buffer_zero_task_set_int(&batch_task->descriptors[i], buf[i], len);
+    }
+}
+
+/**
+ * @brief Asychronously perform a buffer zero DSA operation.
+ *
+ * @param task A pointer to the batch task structure.
+ * @param buf A pointer to the memory buffer.
+ * @param len The length of the memory buffer.
+ *
+ * @return int Zero if successful, otherwise an appropriate error code.
+ */
+__attribute__((unused))
+static int
+buffer_zero_dsa_async(QemuDsaBatchTask *task,
+                      const void *buf, size_t len)
+{
+    buffer_zero_task_set(task, buf, len);
+
+    return submit_wi_async(task);
+}
+
+/**
+ * @brief Sends a memory comparison batch task to a DSA device and wait
+ *        for completion.
+ *
+ * @param batch_task The batch task to be submitted to DSA device.
+ * @param buf An array of memory buffers to check for zero.
+ * @param count The number of buffers.
+ * @param len The buffer length.
+ */
+__attribute__((unused))
+static int
+buffer_zero_dsa_batch_async(QemuDsaBatchTask *batch_task,
+                            const void **buf, size_t count, size_t len)
+{
+    assert(count <= batch_task->batch_size);
+    buffer_zero_batch_task_set(batch_task, buf, count, len);
+
+    return submit_batch_wi_async(batch_task);
+}
+
+/**
+ * @brief The completion callback function for buffer zero
+ *        comparison DSA task completion.
+ *
+ * @param context A pointer to the callback context.
+ */
+static void
+buffer_zero_dsa_completion(void *context)
+{
+    assert(context != NULL);
+
+    QemuDsaBatchTask *task = (QemuDsaBatchTask *)context;
+    qemu_sem_post(&task->sem_task_complete);
+}
+
+/**
+ * @brief Wait for the asynchronous DSA task to complete.
+ *
+ * @param batch_task A pointer to the buffer zero comparison batch task.
+ */
+__attribute__((unused))
+static void
+buffer_zero_dsa_wait(QemuDsaBatchTask *batch_task)
+{
+    qemu_sem_wait(&batch_task->sem_task_complete);
+}
+
+/**
+ * @brief Initializes a buffer zero comparison DSA task.
+ *
+ * @param descriptor A pointer to the DSA task descriptor.
+ * @param completion A pointer to the DSA task completion record.
+ */
+static void
+buffer_zero_task_init_int(struct dsa_hw_desc *descriptor,
+                          struct dsa_completion_record *completion)
+{
+    descriptor->opcode = DSA_OPCODE_COMPVAL;
+    descriptor->flags = IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CRAV;
+    descriptor->comp_pattern = (uint64_t)0;
+    descriptor->completion_addr = (uint64_t)completion;
+}
+
+/**
+ * @brief Initializes a buffer zero DSA batch task.
+ *
+ * @param batch_size The number of zero page checking tasks in the batch.
+ * @return A pointer to the zero page checking tasks initialized.
+ */
+QemuDsaBatchTask *
+buffer_zero_batch_task_init(int batch_size)
+{
+    QemuDsaBatchTask *task = qemu_memalign(64, sizeof(QemuDsaBatchTask));
+    int descriptors_size = sizeof(*task->descriptors) * batch_size;
+
+    memset(task, 0, sizeof(*task));
+    task->addr = g_new0(ram_addr_t, batch_size);
+    task->results = g_new0(bool, batch_size);
+    task->batch_size = batch_size;
+    task->descriptors =
+        (struct dsa_hw_desc *)qemu_memalign(64, descriptors_size);
+    memset(task->descriptors, 0, descriptors_size);
+    task->completions = (struct dsa_completion_record *)qemu_memalign(
+        32, sizeof(*task->completions) * batch_size);
+
+    task->batch_completion.status = DSA_COMP_NONE;
+    task->batch_descriptor.completion_addr = (uint64_t)&task->batch_completion;
+    /* TODO: Ensure that we never send a batch with count <= 1 */
+    task->batch_descriptor.desc_count = 0;
+    task->batch_descriptor.opcode = DSA_OPCODE_BATCH;
+    task->batch_descriptor.flags = IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CRAV;
+    task->batch_descriptor.desc_list_addr = (uintptr_t)task->descriptors;
+    task->status = QEMU_DSA_TASK_READY;
+    task->group = &dsa_group;
+    task->device = dsa_device_group_get_next_device(&dsa_group);
+
+    for (int i = 0; i < task->batch_size; i++) {
+        buffer_zero_task_init_int(&task->descriptors[i],
+                                  &task->completions[i]);
+    }
+
+    qemu_sem_init(&task->sem_task_complete, 0);
+    task->completion_callback = buffer_zero_dsa_completion;
+
+    return task;
+}
+
+/**
+ * @brief Performs the proper cleanup on a DSA batch task.
+ *
+ * @param task A pointer to the batch task to cleanup.
+ */
+void
+buffer_zero_batch_task_destroy(QemuDsaBatchTask *task)
+{
+    if (task) {
+        g_free(task->addr);
+        g_free(task->results);
+        qemu_vfree(task->descriptors);
+        qemu_vfree(task->completions);
+        task->results = NULL;
+        qemu_sem_destroy(&task->sem_task_complete);
+        qemu_vfree(task);
+    }
+}
-- 
Yichen Wang

From: Hao Xiang <hao.xiang@linux.dev>

Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
---
 include/qemu/dsa.h |  14 +++++
 util/dsa.c         | 125 +++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 135 insertions(+), 4 deletions(-)

diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@ buffer_zero_batch_task_init(int batch_size);
  */
 void buffer_zero_batch_task_destroy(QemuDsaBatchTask *task);
 
+/**
+ * @brief Performs buffer zero comparison on a DSA batch task synchronously.
+ *
+ * @param batch_task A pointer to the batch task.
+ * @param buf An array of memory buffers.
+ * @param count The number of buffers in the array.
+ * @param len The buffer length.
+ *
+ * @return Zero if successful, otherwise non-zero.
+ */
+int
+buffer_is_zero_dsa_batch_sync(QemuDsaBatchTask *batch_task,
+                              const void **buf, size_t count, size_t len);
+
 #else
 
 typedef struct QemuDsaBatchTask {} QemuDsaBatchTask;
diff --git a/util/dsa.c b/util/dsa.c
index XXXXXXX..XXXXXXX 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@ poll_completion(struct dsa_completion_record *completion,
     return 0;
 }
 
+/**
+ * @brief Helper function to use CPU to complete a single
+ *        zero page checking task.
+ *
+ * @param completion A pointer to a DSA task completion record.
+ * @param descriptor A pointer to a DSA task descriptor.
+ * @param result A pointer to the result of a zero page checking.
+ */
+static void
+task_cpu_fallback_int(struct dsa_completion_record *completion,
+                      struct dsa_hw_desc *descriptor, bool *result)
+{
+    const uint8_t *buf;
+    size_t len;
+
+    if (completion->status == DSA_COMP_SUCCESS) {
+        return;
+    }
+
+    /*
+     * DSA was able to partially complete the operation. Check the
+     * result. If we already know this is not a zero page, we can
+     * return now.
+     */
+    if (completion->bytes_completed != 0 && completion->result != 0) {
+        *result = false;
+        return;
+    }
+
+    /* Let's fallback to use CPU to complete it. */
+    buf = (const uint8_t *)descriptor->src_addr;
+    len = descriptor->xfer_size;
+    *result = buffer_is_zero(buf + completion->bytes_completed,
+                             len - completion->bytes_completed);
+}
+
 /**
  * @brief Complete a single DSA task in the batch task.
  *
@@ -XXX,XX +XXX,XX @@ dsa_completion_loop(void *opaque)
         (QemuDsaCompletionThread *)opaque;
     QemuDsaBatchTask *batch_task;
     QemuDsaDeviceGroup *group = thread_context->group;
-    int ret;
+    int ret = 0;
 
     rcu_register_thread();
 
@@ -XXX,XX +XXX,XX @@ buffer_zero_batch_task_set(QemuDsaBatchTask *batch_task,
  *
  * @return int Zero if successful, otherwise an appropriate error code.
  */
-__attribute__((unused))
 static int
 buffer_zero_dsa_async(QemuDsaBatchTask *task,
                       const void *buf, size_t len)
@@ -XXX,XX +XXX,XX @@ buffer_zero_dsa_async(QemuDsaBatchTask *task,
  * @param count The number of buffers.
  * @param len The buffer length.
  */
-__attribute__((unused))
 static int
 buffer_zero_dsa_batch_async(QemuDsaBatchTask *batch_task,
                             const void **buf, size_t count, size_t len)
@@ -XXX,XX +XXX,XX @@ buffer_zero_dsa_completion(void *context)
  *
  * @param batch_task A pointer to the buffer zero comparison batch task.
  */
-__attribute__((unused))
 static void
 buffer_zero_dsa_wait(QemuDsaBatchTask *batch_task)
 {
     qemu_sem_wait(&batch_task->sem_task_complete);
 }
 
+/**
+ * @brief Use CPU to complete the zero page checking task if DSA
+ *        is not able to complete it.
+ *
+ * @param batch_task A pointer to the batch task.
+ */
+static void
+buffer_zero_cpu_fallback(QemuDsaBatchTask *batch_task)
+{
+    if (batch_task->task_type == QEMU_DSA_TASK) {
+        if (batch_task->completions[0].status == DSA_COMP_SUCCESS) {
+            return;
+        }
+        task_cpu_fallback_int(&batch_task->completions[0],
+                              &batch_task->descriptors[0],
+                              &batch_task->results[0]);
+    } else if (batch_task->task_type == QEMU_DSA_BATCH_TASK) {
+        struct dsa_completion_record *batch_completion =
+            &batch_task->batch_completion;
+        struct dsa_completion_record *completion;
+        uint8_t status;
+        bool *results = batch_task->results;
+        uint32_t count = batch_task->batch_descriptor.desc_count;
+
+        /* DSA is able to complete the entire batch task. */
+        if (batch_completion->status == DSA_COMP_SUCCESS) {
+            assert(count == batch_completion->bytes_completed);
+            return;
+        }
+
+        /*
+         * DSA encounters some error and is not able to complete
+         * the entire batch task. Use CPU fallback.
+         */
+        for (int i = 0; i < count; i++) {
+
+            completion = &batch_task->completions[i];
+            status = completion->status;
+
+            assert(status == DSA_COMP_SUCCESS ||
+                status == DSA_COMP_PAGE_FAULT_NOBOF);
+
+            task_cpu_fallback_int(completion,
+                                  &batch_task->descriptors[i],
+                                  &results[i]);
+        }
+    }
+}
+
 /**
  * @brief Initializes a buffer zero comparison DSA task.
  *
@@ -XXX,XX +XXX,XX @@ buffer_zero_batch_task_destroy(QemuDsaBatchTask *task)
         qemu_vfree(task);
     }
 }
+
+/**
+ * @brief Performs buffer zero comparison on a DSA batch task synchronously.
+ *
+ * @param batch_task A pointer to the batch task.
+ * @param buf An array of memory buffers.
+ * @param count The number of buffers in the array.
+ * @param len The buffer length.
+ *
+ * @return Zero if successful, otherwise non-zero.
+ */
+int
+buffer_is_zero_dsa_batch_sync(QemuDsaBatchTask *batch_task,
+                              const void **buf, size_t count, size_t len)
+{
+    assert(batch_task != NULL);
+    assert(len != 0);
+    assert(buf != NULL);
+
+    if (count <= 0 || count > batch_task->batch_size) {
+        return -1;
+    }
+
+    if (count == 1) {
+        /* DSA doesn't take batch operation with only 1 task. */
+        buffer_zero_dsa_async(batch_task, buf[0], len);
+    } else {
+        buffer_zero_dsa_batch_async(batch_task, buf, count, len);
+    }
+
+    buffer_zero_dsa_wait(batch_task);
+    buffer_zero_cpu_fallback(batch_task);
+
+    return 0;
+}
-- 
Yichen Wang

From: Hao Xiang <hao.xiang@linux.dev>

Intel DSA offloading is an optional feature that turns on if
proper hardware and software stack is available. To turn on
DSA offloading in multifd live migration by setting:

zero-page-detection=dsa-accel
accel-path="dsa:<dsa_dev_path1> dsa:[dsa_dev_path2] ..."

This feature is turned off by default.

Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
Acked-by: Dr. David Alan Gilbert <dave@treblig.org>
---
 hmp-commands.hx                |  2 +-
 include/qemu/dsa.h             | 13 +++++++++++++
 migration/migration-hmp-cmds.c | 20 +++++++++++++++++++-
 migration/options.c            | 30 ++++++++++++++++++++++++++++++
 migration/options.h            |  1 +
 qapi/migration.json            | 32 ++++++++++++++++++++++++++++----
 util/dsa.c                     | 31 +++++++++++++++++++++++++++++++
 7 files changed, 123 insertions(+), 6 deletions(-)

diff --git a/hmp-commands.hx b/hmp-commands.hx
index XXXXXXX..XXXXXXX 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -XXX,XX +XXX,XX @@ ERST
 
     {
         .name       = "migrate_set_parameter",
-        .args_type  = "parameter:s,value:s",
+        .args_type  = "parameter:s,value:S",
         .params     = "parameter value",
         .help       = "Set the parameter for migration",
         .cmd        = hmp_migrate_set_parameter,
diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h
index XXXXXXX..XXXXXXX 100644
--- a/include/qemu/dsa.h
+++ b/include/qemu/dsa.h
@@ -XXX,XX +XXX,XX @@ void qemu_dsa_stop(void);
  */
 void qemu_dsa_cleanup(void);
 
+/**
+ * @brief Check if DSA is supported.
+ *
+ * @return True if DSA is supported, otherwise false.
+ */
+bool qemu_dsa_is_supported(void);
+
 /**
  * @brief Check if DSA is running.
  *
@@ -XXX,XX +XXX,XX @@ buffer_is_zero_dsa_batch_sync(QemuDsaBatchTask *batch_task,
 
 typedef struct QemuDsaBatchTask {} QemuDsaBatchTask;
 
+static inline bool qemu_dsa_is_supported(void)
+{
+    return false;
+}
+
+
 static inline bool qemu_dsa_is_running(void)
 {
     return false;
diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/migration-hmp-cmds.c
+++ b/migration/migration-hmp-cmds.c
@@ -XXX,XX +XXX,XX @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
         monitor_printf(mon, "%s: '%s'\n",
             MigrationParameter_str(MIGRATION_PARAMETER_TLS_AUTHZ),
             params->tls_authz);
-
+        if (params->has_accel_path) {
+            strList *accel_path = params->accel_path;
+            monitor_printf(mon, "%s:",
+                MigrationParameter_str(MIGRATION_PARAMETER_ACCEL_PATH));
+            while (accel_path) {
+                monitor_printf(mon, " '%s'", accel_path->value);
+                accel_path = accel_path->next;
+            }
+            monitor_printf(mon, "\n");
+        }
         if (params->has_block_bitmap_mapping) {
             const BitmapMigrationNodeAliasList *bmnal;
 
@@ -XXX,XX +XXX,XX @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         p->has_x_checkpoint_delay = true;
         visit_type_uint32(v, param, &p->x_checkpoint_delay, &err);
         break;
+    case MIGRATION_PARAMETER_ACCEL_PATH:
+        p->has_accel_path = true;
+        char **strv = g_strsplit(valuestr ? : "", " ", -1);
+        strList **tail = &p->accel_path;
+        for (int i = 0; strv[i]; i++) {
+            QAPI_LIST_APPEND(tail, g_strdup(strv[i]));
+        }
+        g_strfreev(strv);
+        break;
     case MIGRATION_PARAMETER_MULTIFD_CHANNELS:
         p->has_multifd_channels = true;
         visit_type_uint8(v, param, &p->multifd_channels, &err);
diff --git a/migration/options.c b/migration/options.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -XXX,XX +XXX,XX @@
 
 #include "qemu/osdep.h"
 #include "qemu/error-report.h"
+#include "qemu/dsa.h"
 #include "exec/target_page.h"
 #include "qapi/clone-visitor.h"
 #include "qapi/error.h"
@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_creds(void)
     return s->parameters.tls_creds;
 }
 
+const strList *migrate_accel_path(void)
+{
+    MigrationState *s = migrate_get_current();
+
+    return s->parameters.accel_path;
+}
+
 const char *migrate_tls_hostname(void)
 {
     MigrationState *s = migrate_get_current();
@@ -XXX,XX +XXX,XX @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
     params->zero_page_detection = s->parameters.zero_page_detection;
     params->has_direct_io = true;
     params->direct_io = s->parameters.direct_io;
+    params->has_accel_path = true;
+    params->accel_path = QAPI_CLONE(strList, s->parameters.accel_path);
 
     return params;
 }
@@ -XXX,XX +XXX,XX @@ void migrate_params_init(MigrationParameters *params)
 {
     params->tls_hostname = g_strdup("");
     params->tls_creds = g_strdup("");
+    params->accel_path = NULL;
 
     /* Set has_* up only for parameter checks */
     params->has_throttle_trigger_threshold = true;
@@ -XXX,XX +XXX,XX @@ bool migrate_params_check(MigrationParameters *params, Error **errp)
         return false;
     }
 
+    if (params->has_zero_page_detection &&
+        params->zero_page_detection == ZERO_PAGE_DETECTION_DSA_ACCEL) {
+        if (!qemu_dsa_is_supported()) {
+            error_setg(errp, "DSA acceleration is not supported.");
+            return false;
+        }
+    }
+
     return true;
 }
 
@@ -XXX,XX +XXX,XX @@ static void migrate_params_test_apply(MigrateSetParameters *params,
     if (params->has_direct_io) {
         dest->direct_io = params->direct_io;
     }
+
+    if (params->has_accel_path) {
+        dest->has_accel_path = true;
+        dest->accel_path = params->accel_path;
+    }
 }
 
 static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
@@ -XXX,XX +XXX,XX @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
     if (params->has_direct_io) {
         s->parameters.direct_io = params->direct_io;
     }
+    if (params->has_accel_path) {
+        qapi_free_strList(s->parameters.accel_path);
+        s->parameters.has_accel_path = true;
+        s->parameters.accel_path =
+            QAPI_CLONE(strList, params->accel_path);
+    }
 }
 
 void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
diff --git a/migration/options.h b/migration/options.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.h
+++ b/migration/options.h
@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_creds(void);
 const char *migrate_tls_hostname(void);
 uint64_t migrate_xbzrle_cache_size(void);
 ZeroPageDetection migrate_zero_page_detection(void);
+const strList *migrate_accel_path(void);
 
 /* parameters helpers */
 
diff --git a/qapi/migration.json b/qapi/migration.json
index XXXXXXX..XXXXXXX 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -XXX,XX +XXX,XX @@
 #     multifd migration is enabled, else in the main migration thread
 #     as for @legacy.
 #
+# @dsa-accel: Perform zero page checking with the DSA accelerator
+#     offloading in multifd sender thread if multifd migration is
+#     enabled, else in the main migration thread as for @legacy.
+#
 # Since: 9.0
 ##
 { 'enum': 'ZeroPageDetection',
-  'data': [ 'none', 'legacy', 'multifd' ] }
+  'data': [ 'none', 'legacy', 'multifd', 'dsa-accel' ] }
 
 ##
 # @BitmapMigrationBitmapAliasTransform:
@@ -XXX,XX +XXX,XX @@
 #     See description in @ZeroPageDetection.  Default is 'multifd'.
 #     (since 9.0)
 #
+# @accel-path: If enabled, specify the accelerator paths that to be
+#     used in QEMU. For example, enable DSA accelerator for zero page
+#     detection offloading by setting the @zero-page-detection to
+#     dsa-accel, and defines the accel-path to "dsa:<dsa_device path>".
+#     This parameter is default to an empty list.  (Since 10.0)
+#
 # @direct-io: Open migration files with O_DIRECT when possible.  This
 #     only has effect if the @mapped-ram capability is enabled.
 #     (Since 9.1)
@@ -XXX,XX +XXX,XX @@
            'cpu-throttle-initial', 'cpu-throttle-increment',
            'cpu-throttle-tailslow',
            'tls-creds', 'tls-hostname', 'tls-authz', 'max-bandwidth',
-           'avail-switchover-bandwidth', 'downtime-limit',
+           'avail-switchover-bandwidth', 'downtime-limit', 'accel-path',
            { 'name': 'x-checkpoint-delay', 'features': [ 'unstable' ] },
            'multifd-channels',
            'xbzrle-cache-size', 'max-postcopy-bandwidth',
@@ -XXX,XX +XXX,XX @@
 #     See description in @ZeroPageDetection.  Default is 'multifd'.
 #     (since 9.0)
 #
+# @accel-path: If enabled, specify the accelerator paths that to be
+#     used in QEMU. For example, enable DSA accelerator for zero page
+#     detection offloading by setting the @zero-page-detection to
+#     dsa-accel, and defines the accel-path to "dsa:<dsa_device path>".
+#     This parameter is default to an empty list.  (Since 10.0)
+#
 # @direct-io: Open migration files with O_DIRECT when possible.  This
 #     only has effect if the @mapped-ram capability is enabled.
 #     (Since 9.1)
@@ -XXX,XX +XXX,XX @@
             '*vcpu-dirty-limit': 'uint64',
             '*mode': 'MigMode',
             '*zero-page-detection': 'ZeroPageDetection',
-            '*direct-io': 'bool' } }
+            '*direct-io': 'bool',
+            '*accel-path': [ 'str' ] } }
 
 ##
 # @migrate-set-parameters:
@@ -XXX,XX +XXX,XX @@
 #     See description in @ZeroPageDetection.  Default is 'multifd'.
 #     (since 9.0)
 #
+# @accel-path: If enabled, specify the accelerator paths that to be
+#     used in QEMU. For example, enable DSA accelerator for zero page
+#     detection offloading by setting the @zero-page-detection to
+#     dsa-accel, and defines the accel-path to "dsa:<dsa_device path>".
+#     This parameter is default to an empty list.  (Since 10.0)
+#
 # @direct-io: Open migration files with O_DIRECT when possible.  This
 #     only has effect if the @mapped-ram capability is enabled.
 #     (Since 9.1)
@@ -XXX,XX +XXX,XX @@
             '*vcpu-dirty-limit': 'uint64',
             '*mode': 'MigMode',
             '*zero-page-detection': 'ZeroPageDetection',
-            '*direct-io': 'bool' } }
+            '*direct-io': 'bool',
+            '*accel-path': [ 'str' ] } }
 
 ##
 # @query-migrate-parameters:
diff --git a/util/dsa.c b/util/dsa.c
index XXXXXXX..XXXXXXX 100644
--- a/util/dsa.c
+++ b/util/dsa.c
@@ -XXX,XX +XXX,XX @@
 #include "qemu/bswap.h"
 #include "qemu/error-report.h"
 #include "qemu/rcu.h"
+#include <cpuid.h>
 
 #pragma GCC push_options
 #pragma GCC target("enqcmd")
@@ -XXX,XX +XXX,XX @@ static void dsa_completion_thread_stop(void *opaque)
     qemu_sem_destroy(&thread_context->sem_init_done);
 }
 
+/**
+ * @brief Check if DSA is supported.
+ *
+ * @return True if DSA is supported, otherwise false.
+ */
+bool qemu_dsa_is_supported(void)
+{
+    /*
+     * movdir64b is indicated by bit 28 of ecx in CPUID leaf 7, subleaf 0.
+     * enqcmd is indicated by bit 29 of ecx in CPUID leaf 7, subleaf 0.
+     * Doc: https://cdrdv2-public.intel.com/819680/architecture-instruction-\
+     *      set-extensions-programming-reference.pdf
+     */
+    uint32_t eax, ebx, ecx, edx;
+    bool movedirb_enabled;
+    bool enqcmd_enabled;
+
+    __get_cpuid_count(7, 0, &eax, &ebx, &ecx, &edx);
+    movedirb_enabled = (ecx >> 28) & 0x1;
+    if (!movedirb_enabled) {
+        return false;
+    }
+    enqcmd_enabled = (ecx >> 29) & 0x1;
+    if (!enqcmd_enabled) {
+        return false;
+    }
+
+    return true;
+}
+
 /**
  * @brief Check if DSA is running.
  *
-- 
Yichen Wang

From: Hao Xiang <hao.xiang@linux.dev>

Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
---
 migration/multifd-zero-page.c | 149 ++++++++++++++++++++++++++++++----
 migration/multifd.c           |  23 +++++-
 migration/multifd.h           |   6 ++
 migration/options.c           |  13 +++
 migration/options.h           |   1 +
 5 files changed, 176 insertions(+), 16 deletions(-)

diff --git a/migration/multifd-zero-page.c b/migration/multifd-zero-page.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd-zero-page.c
+++ b/migration/multifd-zero-page.c
@@ -XXX,XX +XXX,XX @@
 
 static bool multifd_zero_page_enabled(void)
 {
-    return migrate_zero_page_detection() == ZERO_PAGE_DETECTION_MULTIFD;
+    ZeroPageDetection curMethod = migrate_zero_page_detection();
+    return (curMethod == ZERO_PAGE_DETECTION_MULTIFD ||
+            curMethod == ZERO_PAGE_DETECTION_DSA_ACCEL);
 }
 
 static void swap_page_offset(ram_addr_t *pages_offset, int a, int b)
@@ -XXX,XX +XXX,XX @@ static void swap_page_offset(ram_addr_t *pages_offset, int a, int b)
     pages_offset[b] = temp;
 }
 
+#ifdef CONFIG_DSA_OPT
+
+static void swap_result(bool *results, int a, int b)
+{
+    bool temp;
+
+    if (a == b) {
+        return;
+    }
+
+    temp = results[a];
+    results[a] = results[b];
+    results[b] = temp;
+}
+
 /**
- * multifd_send_zero_page_detect: Perform zero page detection on all pages.
+ * zero_page_detect_dsa: Perform zero page detection using
+ * Intel Data Streaming Accelerator (DSA).
  *
- * Sorts normal pages before zero pages in p->pages->offset and updates
- * p->pages->normal_num.
+ * Sorts normal pages before zero pages in pages->offset and updates
+ * pages->normal_num.
  *
  * @param p A pointer to the send params.
  */
-void multifd_send_zero_page_detect(MultiFDSendParams *p)
+static void zero_page_detect_dsa(MultiFDSendParams *p)
 {
     MultiFDPages_t *pages = &p->data->u.ram;
     RAMBlock *rb = pages->block;
-    int i = 0;
-    int j = pages->num - 1;
+    bool *results = p->dsa_batch_task->results;
 
-    if (!multifd_zero_page_enabled()) {
-        pages->normal_num = pages->num;
-        goto out;
+    for (int i = 0; i < pages->num; i++) {
+        p->dsa_batch_task->addr[i] =
+            (ram_addr_t)(rb->host + pages->offset[i]);
     }
 
+    buffer_is_zero_dsa_batch_sync(p->dsa_batch_task,
+                                  (const void **)p->dsa_batch_task->addr,
+                                  pages->num,
+                                  multifd_ram_page_size());
+
+    int i = 0;
+    int j = pages->num - 1;
+
     /*
      * Sort the page offset array by moving all normal pages to
      * the left and all zero pages to the right of the array.
@@ -XXX,XX +XXX,XX @@ void multifd_send_zero_page_detect(MultiFDSendParams *p)
     while (i <= j) {
         uint64_t offset = pages->offset[i];
 
-        if (!buffer_is_zero(rb->host + offset, multifd_ram_page_size())) {
+        if (!results[i]) {
             i++;
             continue;
         }
 
+        swap_result(results, i, j);
         swap_page_offset(pages->offset, i, j);
         ram_release_page(rb->idstr, offset);
         j--;
     }
 
     pages->normal_num = i;
+}
 
-out:
-    stat64_add(&mig_stats.normal_pages, pages->normal_num);
-    stat64_add(&mig_stats.zero_pages, pages->num - pages->normal_num);
+int multifd_dsa_setup(MigrationState *s, Error *local_err)
+{
+    g_autofree strList *dsa_parameter = g_malloc0(sizeof(strList));
+    migrate_dsa_accel_path(&dsa_parameter);
+    if (qemu_dsa_init(dsa_parameter, &local_err)) {
+        migrate_set_error(s, local_err);
+        return -1;
+    } else {
+        qemu_dsa_start();
+    }
+
+    return 0;
+}
+
+void multifd_dsa_cleanup(void)
+{
+    qemu_dsa_cleanup();
+}
+
+#else
+
+static void zero_page_detect_dsa(MultiFDSendParams *p)
+{
+    g_assert_not_reached();
 }
 
+int multifd_dsa_setup(MigrationState *s, Error *local_err)
+{
+    g_assert_not_reached();
+    return -1;
+}
+
+void multifd_dsa_cleanup(void)
+{
+    return ;
+}
+
+#endif
+
 void multifd_recv_zero_page_process(MultiFDRecvParams *p)
 {
     for (int i = 0; i < p->zero_num; i++) {
@@ -XXX,XX +XXX,XX @@ void multifd_recv_zero_page_process(MultiFDRecvParams *p)
         }
     }
 }
+
+/**
+ * zero_page_detect_cpu: Perform zero page detection using CPU.
+ *
+ * Sorts normal pages before zero pages in p->pages->offset and updates
+ * p->pages->normal_num.
+ *
+ * @param p A pointer to the send params.
+ */
+static void zero_page_detect_cpu(MultiFDSendParams *p)
+{
+    MultiFDPages_t *pages = &p->data->u.ram;
+    RAMBlock *rb = pages->block;
+    int i = 0;
+    int j = pages->num - 1;
+
+    /*
+     * Sort the page offset array by moving all normal pages to
+     * the left and all zero pages to the right of the array.
+     */
+    while (i <= j) {
+        uint64_t offset = pages->offset[i];
+
+        if (!buffer_is_zero(rb->host + offset, multifd_ram_page_size())) {
+            i++;
+            continue;
+        }
+
+        swap_page_offset(pages->offset, i, j);
+        ram_release_page(rb->idstr, offset);
+        j--;
+    }
+
+    pages->normal_num = i;
+}
+
+/**
+ * multifd_send_zero_page_detect: Perform zero page detection on all pages.
+ *
+ * @param p A pointer to the send params.
+ */
+void multifd_send_zero_page_detect(MultiFDSendParams *p)
+{
+    MultiFDPages_t *pages = &p->data->u.ram;
+
+    if (!multifd_zero_page_enabled()) {
+        pages->normal_num = pages->num;
+        goto out;
+    }
+
+    if (qemu_dsa_is_running()) {
+        zero_page_detect_dsa(p);
+    } else {
+        zero_page_detect_cpu(p);
+    }
+
+out:
+    stat64_add(&mig_stats.normal_pages, pages->normal_num);
+    stat64_add(&mig_stats.zero_pages, pages->num - pages->normal_num);
+}
diff --git a/migration/multifd.c b/migration/multifd.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -XXX,XX +XXX,XX @@
 #include "qemu/osdep.h"
 #include "qemu/cutils.h"
 #include "qemu/rcu.h"
+#include "qemu/dsa.h"
 #include "exec/target_page.h"
 #include "system/system.h"
 #include "exec/ramblock.h"
@@ -XXX,XX +XXX,XX @@ static bool multifd_send_cleanup_channel(MultiFDSendParams *p, Error **errp)
     p->name = NULL;
     g_free(p->data);
     p->data = NULL;
+    buffer_zero_batch_task_destroy(p->dsa_batch_task);
+    p->dsa_batch_task = NULL;
     p->packet_len = 0;
     g_free(p->packet);
     p->packet = NULL;
@@ -XXX,XX +XXX,XX @@ void multifd_send_shutdown(void)
 
     multifd_send_terminate_threads();
 
+    multifd_dsa_cleanup();
+
     for (i = 0; i < migrate_multifd_channels(); i++) {
         MultiFDSendParams *p = &multifd_send_state->params[i];
         Error *local_err = NULL;
@@ -XXX,XX +XXX,XX @@ bool multifd_send_setup(void)
     uint32_t page_count = multifd_ram_page_count();
     bool use_packets = multifd_use_packets();
     uint8_t i;
+    Error *local_err = NULL;
 
     if (!migrate_multifd()) {
         return true;
@@ -XXX,XX +XXX,XX @@ bool multifd_send_setup(void)
     qatomic_set(&multifd_send_state->exiting, 0);
     multifd_send_state->ops = multifd_ops[migrate_multifd_compression()];
 
+    if (ret) {
+        goto err;
+    }
+
     for (i = 0; i < thread_count; i++) {
         MultiFDSendParams *p = &multifd_send_state->params[i];
-        Error *local_err = NULL;
 
         qemu_sem_init(&p->sem, 0);
         qemu_sem_init(&p->sem_sync, 0);
@@ -XXX,XX +XXX,XX @@ bool multifd_send_setup(void)
         goto err;
     }
 
+    if (s && ret == 0 &&
+        s->parameters.zero_page_detection == ZERO_PAGE_DETECTION_DSA_ACCEL) {
+        ret = multifd_dsa_setup(s, local_err);
+    }
+
+    if (ret) {
+        goto err;
+    }
+
     for (i = 0; i < thread_count; i++) {
         MultiFDSendParams *p = &multifd_send_state->params[i];
-        Error *local_err = NULL;
 
+        p->dsa_batch_task = buffer_zero_batch_task_init(page_count);
         ret = multifd_send_state->ops->send_setup(p, &local_err);
         if (ret) {
             migrate_set_error(s, local_err);
@@ -XXX,XX +XXX,XX @@ void multifd_recv_cleanup(void)
             qemu_thread_join(&p->thread);
         }
     }
+    multifd_dsa_cleanup();
     for (i = 0; i < migrate_multifd_channels(); i++) {
         multifd_recv_cleanup_channel(&multifd_recv_state->params[i]);
     }
diff --git a/migration/multifd.h b/migration/multifd.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/multifd.h
+++ b/migration/multifd.h
@@ -XXX,XX +XXX,XX @@
 
 #include "exec/target_page.h"
 #include "ram.h"
+#include "qemu/dsa.h"
 
 typedef struct MultiFDRecvData MultiFDRecvData;
 typedef struct MultiFDSendData MultiFDSendData;
@@ -XXX,XX +XXX,XX @@ typedef struct {
     bool pending_sync;
     MultiFDSendData *data;
 
+    /* Zero page checking batch task */
+    QemuDsaBatchTask *dsa_batch_task;
+
     /* thread local variables. No locking required */
 
     /* pointer to the packet */
@@ -XXX,XX +XXX,XX @@ void multifd_send_fill_packet(MultiFDSendParams *p);
 bool multifd_send_prepare_common(MultiFDSendParams *p);
 void multifd_send_zero_page_detect(MultiFDSendParams *p);
 void multifd_recv_zero_page_process(MultiFDRecvParams *p);
+int multifd_dsa_setup(MigrationState *s, Error *local_err);
+void multifd_dsa_cleanup(void);
 
 static inline void multifd_send_prepare_header(MultiFDSendParams *p)
 {
diff --git a/migration/options.c b/migration/options.c
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.c
+++ b/migration/options.c
@@ -XXX,XX +XXX,XX @@ const strList *migrate_accel_path(void)
     return s->parameters.accel_path;
 }
 
+void migrate_dsa_accel_path(strList **dsa_accel_path)
+{
+    MigrationState *s = migrate_get_current();
+    strList *accel_path = s->parameters.accel_path;
+    strList **tail = dsa_accel_path;
+    while (accel_path) {
+        if (strncmp(accel_path->value, "dsa:", 4) == 0) {
+            QAPI_LIST_APPEND(tail, &accel_path->value[4]);
+        }
+        accel_path = accel_path->next;
+    }
+}
+
 const char *migrate_tls_hostname(void)
 {
     MigrationState *s = migrate_get_current();
diff --git a/migration/options.h b/migration/options.h
index XXXXXXX..XXXXXXX 100644
--- a/migration/options.h
+++ b/migration/options.h
@@ -XXX,XX +XXX,XX @@ const char *migrate_tls_hostname(void);
 uint64_t migrate_xbzrle_cache_size(void);
 ZeroPageDetection migrate_zero_page_detection(void);
 const strList *migrate_accel_path(void);
+void migrate_dsa_accel_path(strList **dsa_accel_path);
 
 /* parameters helpers */
 
-- 
Yichen Wang

From: Hao Xiang <hao.xiang@linux.dev>

* Test DSA start and stop path.
* Test DSA configure and cleanup path.
* Test DSA task submission and completion path.

Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
---
 tests/unit/meson.build |   6 +
 tests/unit/test-dsa.c  | 504 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 510 insertions(+)
 create mode 100644 tests/unit/test-dsa.c

diff --git a/tests/unit/meson.build b/tests/unit/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/tests/unit/meson.build
+++ b/tests/unit/meson.build
@@ -XXX,XX +XXX,XX @@ tests = {
   'test-fifo': [],
 }
 
+if config_host_data.get('CONFIG_DSA_OPT')
+  tests += {
+    'test-dsa': [],
+  }
+endif
+
 if have_system or have_tools
   tests += {
     'test-qmp-event': [testqapi],
diff --git a/tests/unit/test-dsa.c b/tests/unit/test-dsa.c
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/unit/test-dsa.c
@@ -XXX,XX +XXX,XX @@
+/*
+ * Test DSA functions.
+ *
+ * Copyright (C) Bytedance Ltd.
+ *
+ * Authors:
+ *  Hao Xiang <hao.xiang@bytedance.com>
+ *  Bryan Zhang <bryan.zhang@bytedance.com>
+ *  Yichen Wang <yichen.wang@bytedance.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/host-utils.h"
+
+#include "qemu/cutils.h"
+#include "qemu/memalign.h"
+#include "qemu/dsa.h"
+
+/*
+ * TODO Communicate that DSA must be configured to support this batch size.
+ * TODO Alternatively, poke the DSA device to figure out batch size.
+ */
+#define batch_size 128
+#define page_size 4096
+
+#define oversized_batch_size (batch_size + 1)
+#define num_devices 2
+#define max_buffer_size (64 * 1024)
+
+/* TODO Make these not-hardcoded. */
+static const strList path1[] = {
+    {.value = (char *)"/dev/dsa/wq4.0", .next = NULL}
+};
+static const strList path2[] = {
+    {.value = (char *)"/dev/dsa/wq4.0", .next = (strList*)&path2[1]},
+    {.value = (char *)"/dev/dsa/wq4.1", .next = NULL}
+};
+
+static Error **errp;
+
+static QemuDsaBatchTask *task;
+
+/* A helper for running a single task and checking for correctness. */
+static void do_single_task(void)
+{
+    task = buffer_zero_batch_task_init(batch_size);
+    char buf[page_size];
+    char *ptr = buf;
+
+    buffer_is_zero_dsa_batch_sync(task,
+                                  (const void **)&ptr,
+                                  1,
+                                  page_size);
+    g_assert(task->results[0] == buffer_is_zero(buf, page_size));
+
+    buffer_zero_batch_task_destroy(task);
+}
+
+static void test_single_zero(void)
+{
+    g_assert(!qemu_dsa_init(path1, errp));
+    qemu_dsa_start();
+
+    task = buffer_zero_batch_task_init(batch_size);
+
+    char buf[page_size];
+    char *ptr = buf;
+
+    memset(buf, 0x0, page_size);
+    buffer_is_zero_dsa_batch_sync(task,
+                                  (const void **)&ptr,
+                                  1, page_size);
+    g_assert(task->results[0]);
+
+    buffer_zero_batch_task_destroy(task);
+
+    qemu_dsa_cleanup();
+}
+
+static void test_single_zero_async(void)
+{
+    test_single_zero();
+}
+
+static void test_single_nonzero(void)
+{
+    g_assert(!qemu_dsa_init(path1, errp));
+    qemu_dsa_start();
+
+    task = buffer_zero_batch_task_init(batch_size);
+
+    char buf[page_size];
+    char *ptr = buf;
+
+    memset(buf, 0x1, page_size);
+    buffer_is_zero_dsa_batch_sync(task,
+                                  (const void **)&ptr,
+                                  1, page_size);
+    g_assert(!task->results[0]);
+
+    buffer_zero_batch_task_destroy(task);
+
+    qemu_dsa_cleanup();
+}
+
+static void test_single_nonzero_async(void)
+{
+    test_single_nonzero();
+}
+
+/* count == 0 should return quickly without calling into DSA. */
+static void test_zero_count_async(void)
+{
+    char buf[page_size];
+    buffer_is_zero_dsa_batch_sync(task,
+                                  (const void **)&buf,
+                                  0,
+                                  page_size);
+}
+
+static void test_null_task_async(void)
+{
+    if (g_test_subprocess()) {
+        g_assert(!qemu_dsa_init(path1, errp));
+
+        char buf[page_size * batch_size];
+        char *addrs[batch_size];
+        for (int i = 0; i < batch_size; i++) {
+            addrs[i] = buf + (page_size * i);
+        }
+
+        buffer_is_zero_dsa_batch_sync(NULL, (const void **)addrs,
+                                      batch_size,
+                                      page_size);
+    } else {
+        g_test_trap_subprocess(NULL, 0, 0);
+        g_test_trap_assert_failed();
+    }
+}
+
+static void test_oversized_batch(void)
+{
+    g_assert(!qemu_dsa_init(path1, errp));
+    qemu_dsa_start();
+
+    task = buffer_zero_batch_task_init(batch_size);
+
+    char buf[page_size * oversized_batch_size];
+    char *addrs[batch_size];
+    for (int i = 0; i < oversized_batch_size; i++) {
+        addrs[i] = buf + (page_size * i);
+    }
+
+    int ret = buffer_is_zero_dsa_batch_sync(task,
+                                            (const void **)addrs,
+                                            oversized_batch_size,
+                                            page_size);
+    g_assert(ret != 0);
+
+    buffer_zero_batch_task_destroy(task);
+
+    qemu_dsa_cleanup();
+}
+
+static void test_oversized_batch_async(void)
+{
+    test_oversized_batch();
+}
+
+static void test_zero_len_async(void)
+{
+    if (g_test_subprocess()) {
+        g_assert(!qemu_dsa_init(path1, errp));
+
+        task = buffer_zero_batch_task_init(batch_size);
+
+        char buf[page_size];
+
+        buffer_is_zero_dsa_batch_sync(task,
+                                      (const void **)&buf,
+                                      1,
+                                      0);
+
+        buffer_zero_batch_task_destroy(task);
+    } else {
+        g_test_trap_subprocess(NULL, 0, 0);
+        g_test_trap_assert_failed();
+    }
+}
+
+static void test_null_buf_async(void)
+{
+    if (g_test_subprocess()) {
+        g_assert(!qemu_dsa_init(path1, errp));
+
+        task = buffer_zero_batch_task_init(batch_size);
+
+        buffer_is_zero_dsa_batch_sync(task, NULL, 1, page_size);
+
+        buffer_zero_batch_task_destroy(task);
+    } else {
+        g_test_trap_subprocess(NULL, 0, 0);
+        g_test_trap_assert_failed();
+    }
+}
+
+static void test_batch(void)
+{
+    g_assert(!qemu_dsa_init(path1, errp));
+    qemu_dsa_start();
+
+    task = buffer_zero_batch_task_init(batch_size);
+
+    char buf[page_size * batch_size];
+    char *addrs[batch_size];
+    for (int i = 0; i < batch_size; i++) {
+        addrs[i] = buf + (page_size * i);
+    }
+
+    /*
+     * Using whatever is on the stack is somewhat random.
+     * Manually set some pages to zero and some to nonzero.
+     */
+    memset(buf + 0, 0, page_size * 10);
+    memset(buf + (10 * page_size), 0xff, page_size * 10);
+
+    buffer_is_zero_dsa_batch_sync(task,
+                                  (const void **)addrs,
+                                  batch_size,
+                                  page_size);
+
+    bool is_zero;
+    for (int i = 0; i < batch_size; i++) {
+        is_zero = buffer_is_zero((const void *)&buf[page_size * i], page_size);
+        g_assert(task->results[i] == is_zero);
+    }
+
+    buffer_zero_batch_task_destroy(task);
+
+    qemu_dsa_cleanup();
+}
+
+static void test_batch_async(void)
+{
+    test_batch();
+}
+
+static void test_page_fault(void)
+{
+    g_assert(!qemu_dsa_init(path1, errp));
+    qemu_dsa_start();
+
+    char *buf[2];
+    int prot = PROT_READ | PROT_WRITE;
+    int flags = MAP_SHARED | MAP_ANON;
+    buf[0] = (char *)mmap(NULL, page_size * batch_size, prot, flags, -1, 0);
+    assert(buf[0] != MAP_FAILED);
+    buf[1] = (char *)malloc(page_size * batch_size);
+    assert(buf[1] != NULL);
+
+    for (int j = 0; j < 2; j++) {
+        task = buffer_zero_batch_task_init(batch_size);
+
+        char *addrs[batch_size];
+        for (int i = 0; i < batch_size; i++) {
+            addrs[i] = buf[j] + (page_size * i);
+        }
+
+        buffer_is_zero_dsa_batch_sync(task,
+                                      (const void **)addrs,
+                                      batch_size,
+                                      page_size);
+
+        bool is_zero;
+        for (int i = 0; i < batch_size; i++) {
+            is_zero = buffer_is_zero((const void *)&buf[j][page_size * i],
+                                      page_size);
+            g_assert(task->results[i] == is_zero);
+        }
+        buffer_zero_batch_task_destroy(task);
+    }
+
+    assert(!munmap(buf[0], page_size * batch_size));
+    free(buf[1]);
+    qemu_dsa_cleanup();
+}
+
+static void test_various_buffer_sizes(void)
+{
+    g_assert(!qemu_dsa_init(path1, errp));
+    qemu_dsa_start();
+
+    char *buf = malloc(max_buffer_size * batch_size);
+    char *addrs[batch_size];
+
+    for (int len = 16; len <= max_buffer_size; len *= 2) {
+        task = buffer_zero_batch_task_init(batch_size);
+
+        for (int i = 0; i < batch_size; i++) {
+            addrs[i] = buf + (len * i);
+        }
+
+        buffer_is_zero_dsa_batch_sync(task,
+                                      (const void **)addrs,
+                                      batch_size,
+                                      len);
+
+        bool is_zero;
+        for (int j = 0; j < batch_size; j++) {
+            is_zero = buffer_is_zero((const void *)&buf[len * j], len);
+            g_assert(task->results[j] == is_zero);
+        }
+
+        buffer_zero_batch_task_destroy(task);
+    }
+
+    free(buf);
+
+    qemu_dsa_cleanup();
+}
+
+static void test_various_buffer_sizes_async(void)
+{
+    test_various_buffer_sizes();
+}
+
+static void test_double_start_stop(void)
+{
+    g_assert(!qemu_dsa_init(path1, errp));
+    /* Double start */
+    qemu_dsa_start();
+    qemu_dsa_start();
+    g_assert(qemu_dsa_is_running());
+    do_single_task();
+
+    /* Double stop */
+    qemu_dsa_stop();
+    g_assert(!qemu_dsa_is_running());
+    qemu_dsa_stop();
+    g_assert(!qemu_dsa_is_running());
+
+    /* Restart */
+    qemu_dsa_start();
+    g_assert(qemu_dsa_is_running());
+    do_single_task();
+    qemu_dsa_cleanup();
+}
+
+static void test_is_running(void)
+{
+    g_assert(!qemu_dsa_init(path1, errp));
+
+    g_assert(!qemu_dsa_is_running());
+    qemu_dsa_start();
+    g_assert(qemu_dsa_is_running());
+    qemu_dsa_stop();
+    g_assert(!qemu_dsa_is_running());
+    qemu_dsa_cleanup();
+}
+
+static void test_multiple_engines(void)
+{
+    g_assert(!qemu_dsa_init(path2, errp));
+    qemu_dsa_start();
+
+    QemuDsaBatchTask *tasks[num_devices];
+    char bufs[num_devices][page_size * batch_size];
+    char *addrs[num_devices][batch_size];
+
+    /*
+     *  This is a somewhat implementation-specific way
+     *  of testing that the tasks have unique engines
+     *  assigned to them.
+     */
+    tasks[0] = buffer_zero_batch_task_init(batch_size);
+    tasks[1] = buffer_zero_batch_task_init(batch_size);
+    g_assert(tasks[0]->device != tasks[1]->device);
+
+    for (int i = 0; i < num_devices; i++) {
+        for (int j = 0; j < batch_size; j++) {
+            addrs[i][j] = bufs[i] + (page_size * j);
+        }
+
+        buffer_is_zero_dsa_batch_sync(tasks[i],
+                                      (const void **)addrs[i],
+                                      batch_size, page_size);
+
+        bool is_zero;
+        for (int j = 0; j < batch_size; j++) {
+            is_zero = buffer_is_zero((const void *)&bufs[i][page_size * j],
+                                     page_size);
+            g_assert(tasks[i]->results[j] == is_zero);
+        }
+    }
+
+    buffer_zero_batch_task_destroy(tasks[0]);
+    buffer_zero_batch_task_destroy(tasks[1]);
+
+    qemu_dsa_cleanup();
+}
+
+static void test_configure_dsa_twice(void)
+{
+    g_assert(!qemu_dsa_init(path2, errp));
+    g_assert(!qemu_dsa_init(path2, errp));
+    qemu_dsa_start();
+    do_single_task();
+    qemu_dsa_cleanup();
+}
+
+static void test_configure_dsa_bad_path(void)
+{
+    const strList *bad_path = &(strList) {
+        .value = (char *)"/not/a/real/path", .next = NULL
+    };
+    g_assert(qemu_dsa_init(bad_path, errp));
+}
+
+static void test_cleanup_before_configure(void)
+{
+    qemu_dsa_cleanup();
+    g_assert(!qemu_dsa_init(path2, errp));
+}
+
+static void test_configure_dsa_num_devices(void)
+{
+    g_assert(!qemu_dsa_init(path1, errp));
+    qemu_dsa_start();
+
+    do_single_task();
+    qemu_dsa_stop();
+    qemu_dsa_cleanup();
+}
+
+static void test_cleanup_twice(void)
+{
+    g_assert(!qemu_dsa_init(path2, errp));
+    qemu_dsa_cleanup();
+    qemu_dsa_cleanup();
+
+    g_assert(!qemu_dsa_init(path2, errp));
+    qemu_dsa_start();
+    do_single_task();
+    qemu_dsa_cleanup();
+}
+
+static int check_test_setup(void)
+{
+    const strList *path[2] = {path1, path2};
+    for (int i = 0; i < sizeof(path) / sizeof(strList *); i++) {
+        if (qemu_dsa_init(path[i], errp)) {
+            return -1;
+        }
+        qemu_dsa_cleanup();
+    }
+    return 0;
+}
+
+int main(int argc, char **argv)
+{
+    g_test_init(&argc, &argv, NULL);
+
+    if (check_test_setup() != 0) {
+        /*
+         * This test requires extra setup. The current
+         * setup is not correct. Just skip this test
+         * for now.
+         */
+        g_test_skip("DSA hardware is not configured properly.");
+        return g_test_run();
+    }
+
+    if (num_devices > 1) {
+        g_test_add_func("/dsa/multiple_engines", test_multiple_engines);
+    }
+
+    g_test_add_func("/dsa/async/batch", test_batch_async);
+    g_test_add_func("/dsa/async/various_buffer_sizes",
+                    test_various_buffer_sizes_async);
+    g_test_add_func("/dsa/async/null_buf", test_null_buf_async);
+    g_test_add_func("/dsa/async/zero_len", test_zero_len_async);
+    g_test_add_func("/dsa/async/oversized_batch", test_oversized_batch_async);
+    g_test_add_func("/dsa/async/zero_count", test_zero_count_async);
+    g_test_add_func("/dsa/async/single_zero", test_single_zero_async);
+    g_test_add_func("/dsa/async/single_nonzero", test_single_nonzero_async);
+    g_test_add_func("/dsa/async/null_task", test_null_task_async);
+    g_test_add_func("/dsa/async/page_fault", test_page_fault);
+
+    g_test_add_func("/dsa/double_start_stop", test_double_start_stop);
+    g_test_add_func("/dsa/is_running", test_is_running);
+
+    g_test_add_func("/dsa/configure_dsa_twice", test_configure_dsa_twice);
+    g_test_add_func("/dsa/configure_dsa_bad_path", test_configure_dsa_bad_path);
+    g_test_add_func("/dsa/cleanup_before_configure",
+                    test_cleanup_before_configure);
+    g_test_add_func("/dsa/configure_dsa_num_devices",
+                    test_configure_dsa_num_devices);
+    g_test_add_func("/dsa/cleanup_twice", test_cleanup_twice);
+
+    return g_test_run();
+}
-- 
Yichen Wang

From: Hao Xiang <hao.xiang@linux.dev>

* Add test case to start and complete multifd live migration with DSA
offloading enabled.
* Add test case to start and cancel multifd live migration with DSA
offloading enabled.

Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Signed-off-by: Hao Xiang <hao.xiang@linux.dev>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
---
 tests/qtest/meson.build           | 10 +++++-
 tests/qtest/migration-test.c      |  3 ++
 tests/qtest/migration/dsa-tests.c | 59 +++++++++++++++++++++++++++++++
 tests/qtest/migration/framework.h |  1 +
 4 files changed, 72 insertions(+), 1 deletion(-)
 create mode 100644 tests/qtest/migration/dsa-tests.c

diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -XXX,XX +XXX,XX @@ if gnutls.found()
   endif
 endif
 
+migration_dsa_files = []
+if config_host_data.get('CONFIG_DSA_OPT')
+  migration_dsa_files = [files(
+    'migration/dsa-tests.c',
+  )]
+endif
+
 qtests = {
   'bios-tables-test': [io, 'boot-sector.c', 'acpi-utils.c', 'tpm-emu.c'],
   'cdrom-test': files('boot-sector.c'),
@@ -XXX,XX +XXX,XX @@ qtests = {
                              'migration/migration-util.c') + dbus_vmstate1,
   'erst-test': files('erst-test.c'),
   'ivshmem-test': [rt, '../../contrib/ivshmem-server/ivshmem-server.c'],
-  'migration-test': migration_files + migration_tls_files,
+  'migration-test': migration_files + migration_tls_files + \
+                    migration_dsa_files,
   'pxe-test': files('boot-sector.c'),
   'pnv-xive2-test': files('pnv-xive2-common.c', 'pnv-xive2-flush-sync.c'),
   'qos-test': [chardev, io, qos_test_ss.apply({}).sources()],
diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration-test.c
+++ b/tests/qtest/migration-test.c
@@ -XXX,XX +XXX,XX @@ int main(int argc, char **argv)
     migration_test_add_precopy(env);
     migration_test_add_cpr(env);
     migration_test_add_misc(env);
+#ifdef CONFIG_DSA_OPT
+    migration_test_add_dsa(env);
+#endif
 
     ret = g_test_run();
 
diff --git a/tests/qtest/migration/dsa-tests.c b/tests/qtest/migration/dsa-tests.c
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/tests/qtest/migration/dsa-tests.c
@@ -XXX,XX +XXX,XX @@
+/*
+ * QTest testcases for DSA accelerator
+ *
+ * Copyright (C) Bytedance Ltd.
+ *   based on the vhost-user-test.c that is:
+ *      Copyright (c) 2014 Virtual Open Systems Sarl.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "migration/framework.h"
+#include "migration/migration-qmp.h"
+#include "migration/migration-util.h"
+
+/*
+ * It requires separate steps to configure and enable DSA device.
+ * This test assumes that the configuration is done already.
+ */
+static const char *dsa_dev_path_p = "['dsa:/dev/dsa/wq4.0']";
+static const char *dsa_dev_path = "/dev/dsa/wq4.0";
+static int test_dsa_setup(void)
+{
+    int fd;
+    fd = open(dsa_dev_path, O_RDWR);
+    if (fd < 0) {
+        return -1;
+    }
+    close(fd);
+    return 0;
+}
+
+static void *test_migrate_precopy_tcp_multifd_start_dsa(QTestState *from,
+                                                        QTestState *to)
+{
+    migrate_set_parameter_str(from, "zero-page-detection", "dsa-accel");
+    migrate_set_parameter_str(from, "accel-path", dsa_dev_path_p);
+    return migrate_hook_start_precopy_tcp_multifd_common(from, to, "none");
+}
+
+static void test_multifd_tcp_zero_page_dsa(void)
+{
+    MigrateCommon args = {
+        .listen_uri = "defer",
+        .start_hook = test_migrate_precopy_tcp_multifd_start_dsa,
+    };
+
+    test_precopy_common(&args);
+}
+
+void migration_test_add_dsa(MigrationTestEnv *env)
+{
+    if (test_dsa_setup() == 0) {
+        migration_test_add("/migration/multifd/tcp/plain/zero-page/dsa",
+                       test_multifd_tcp_zero_page_dsa);
+    }
+}
diff --git a/tests/qtest/migration/framework.h b/tests/qtest/migration/framework.h
index XXXXXXX..XXXXXXX 100644
--- a/tests/qtest/migration/framework.h
+++ b/tests/qtest/migration/framework.h
@@ -XXX,XX +XXX,XX @@ void migration_test_add_file(MigrationTestEnv *env);
 void migration_test_add_precopy(MigrationTestEnv *env);
 void migration_test_add_cpr(MigrationTestEnv *env);
 void migration_test_add_misc(MigrationTestEnv *env);
+void migration_test_add_dsa(MigrationTestEnv *env);
 
 #endif /* TEST_FRAMEWORK_H */
-- 
Yichen Wang

From: Yuan Liu <yuan1.liu@intel.com>

Signed-off-by: Yuan Liu <yuan1.liu@intel.com>
Signed-off-by: Yichen Wang <yichen.wang@bytedance.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
---
 .../migration/dsa-zero-page-detection.rst     | 290 ++++++++++++++++++
 docs/devel/migration/features.rst             |   1 +
 2 files changed, 291 insertions(+)
 create mode 100644 docs/devel/migration/dsa-zero-page-detection.rst

diff --git a/docs/devel/migration/dsa-zero-page-detection.rst b/docs/devel/migration/dsa-zero-page-detection.rst
new file mode 100644
index XXXXXXX..XXXXXXX
--- /dev/null
+++ b/docs/devel/migration/dsa-zero-page-detection.rst
@@ -XXX,XX +XXX,XX @@
+=============================
+DSA-Based Zero Page Detection
+=============================
+Intel Data Streaming Accelerator(``DSA``) is introduced in Intel's 4th
+generation Xeon server, aka Sapphire Rapids(``SPR``). One of the things
+DSA can do is to offload memory comparison workload from CPU to DSA accelerator
+hardware.
+
+The main advantages of using DSA to accelerate zero pages detection include
+
+1. Reduces CPU usage in multifd live migration workflow across all use cases.
+
+2. Reduces migration total time in some use cases.
+
+
+DSA-Based Zero Page Detection Introduction
+==========================================
+
+::
+
+
+  +----------------+       +------------------+
+  | MultiFD Thread |       |accel-config tool |
+  +-+--------+-----+       +--------+---------+
+    |        |                      |
+    |        |  Open DSA            | Setup DSA
+    |        |  Work Queues         | Resources
+    |        |       +-----+-----+  |
+    |        +------>|idxd driver|<-+
+    |                +-----+-----+
+    |                      |
+    |                      |
+    |                +-----+-----+
+    +----------------+DSA Devices|
+      Submit jobs    +-----------+
+      via enqcmd
+
+
+DSA Introduction
+----------------
+Intel Data Streaming Accelerator (DSA) is a high-performance data copy and
+transformation accelerator that is integrated in Intel Xeon processors,
+targeted for optimizing streaming data movement and transformation operations
+common with applications for high-performance storage, networking, persistent
+memory, and various data processing applications.
+
+For more ``DSA`` introduction, please refer to `DSA Introduction
+<https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/data-streaming-accelerator.html>`_
+
+For ``DSA`` specification, please refer to `DSA Specification
+<https://cdrdv2-public.intel.com/671116/341204-intel-data-streaming-accelerator-spec.pdf>`_
+
+For ``DSA`` user guide, please refer to `DSA User Guide
+<https://www.intel.com/content/www/us/en/content-details/759709/intel-data-streaming-accelerator-user-guide.html>`_
+
+DSA Device Management
+---------------------
+
+The number of ``DSA`` devices will vary depending on the Xeon product model.
+On a ``SPR`` server, there can be a maximum of 8 ``DSA`` devices, with up to
+4 devices per socket.
+
+By default, all ``DSA`` devices are disabled and need to be configured and
+enabled by users manually.
+
+Check the number of devices through the following command
+
+.. code-block:: shell
+
+  #lspci -d 8086:0b25
+  6a:01.0 System peripheral: Intel Corporation Device 0b25
+  6f:01.0 System peripheral: Intel Corporation Device 0b25
+  74:01.0 System peripheral: Intel Corporation Device 0b25
+  79:01.0 System peripheral: Intel Corporation Device 0b25
+  e7:01.0 System peripheral: Intel Corporation Device 0b25
+  ec:01.0 System peripheral: Intel Corporation Device 0b25
+  f1:01.0 System peripheral: Intel Corporation Device 0b25
+  f6:01.0 System peripheral: Intel Corporation Device 0b25
+
+
+DSA Device Configuration And Enabling
+-------------------------------------
+
+The ``accel-config`` tool is used to enable ``DSA`` devices and configure
+``DSA`` hardware resources(work queues and engines). One ``DSA`` device
+has 8 work queues and 4 processing engines, multiple engines can be assigned
+to a work queue via ``group`` attribute.
+
+For ``accel-config`` installation, please refer to `accel-config installation
+<https://github.com/intel/idxd-config>`_
+
+One example of configuring and enabling an ``DSA`` device.
+
+.. code-block:: shell
+
+  #accel-config config-engine dsa0/engine0.0 -g 0
+  #accel-config config-engine dsa0/engine0.1 -g 0
+  #accel-config config-engine dsa0/engine0.2 -g 0
+  #accel-config config-engine dsa0/engine0.3 -g 0
+  #accel-config config-wq dsa0/wq0.0 -g 0 -s 128 -p 10 -b 1 -t 128 -m shared -y user -n app1 -d user
+  #accel-config enable-device dsa0
+  #accel-config enable-wq dsa0/wq0.0
+
+- The ``DSA`` device index is 0, use ``ls -lh /sys/bus/dsa/devices/dsa*``
+  command to query the ``DSA`` device index.
+
+- 4 engines and 1 work queue are configured in group 0, so that all zero-page
+  detection jobs submitted to this work queue can be processed by all engines
+  simultaneously.
+
+- Set work queue attributes including the work mode, work queue size and so on.
+
+- Enable the ``dsa0`` device and work queue ``dsa0/wq0.0``
+
+.. note::
+
+   1. ``DSA`` device driver is Intel Data Accelerator Driver (idxd), it is
+      recommended that the minimum version of Linux kernel is 5.18.
+
+   2. Only ``DSA`` shared work queue mode is supported, it needs to add
+      ``"intel_iommu=on,sm_on"`` parameter to kernel command line.
+
+For more detailed configuration, please refer to `DSA Configuration Samples
+<https://github.com/intel/idxd-config/tree/stable/Documentation/accfg>`_
+
+
+Performances
+============
+We use two Intel 4th generation Xeon servers for testing.
+
+::
+
+    Architecture:        x86_64
+    CPU(s):              192
+    Thread(s) per core:  2
+    Core(s) per socket:  48
+    Socket(s):           2
+    NUMA node(s):        2
+    Vendor ID:           GenuineIntel
+    CPU family:          6
+    Model:               143
+    Model name:          Intel(R) Xeon(R) Platinum 8457C
+    Stepping:            8
+    CPU MHz:             2538.624
+    CPU max MHz:         3800.0000
+    CPU min MHz:         800.0000
+
+We perform multifd live migration with below setup:
+
+1. VM has 100GB memory.
+
+2. Use the new migration option multifd-set-normal-page-ratio to control the
+   total size of the payload sent over the network.
+
+3. Use 8 multifd channels.
+
+4. Use tcp for live migration.
+
+5. Use CPU to perform zero page checking as the baseline.
+
+6. Use one DSA device to offload zero page checking to compare with the baseline.
+
+7. Use "perf sched record" and "perf sched timehist" to analyze CPU usage.
+
+
+A) Scenario 1: 50% (50GB) normal pages on an 100GB vm
+-----------------------------------------------------
+
+::
+
+	CPU usage
+
+	|---------------|---------------|---------------|---------------|
+	|		|comm		|runtime(msec)	|totaltime(msec)|
+	|---------------|---------------|---------------|---------------|
+	|Baseline	|live_migration	|5657.58	|		|
+	|		|multifdsend_0	|3931.563	|		|
+	|		|multifdsend_1	|4405.273	|		|
+	|		|multifdsend_2	|3941.968	|		|
+	|		|multifdsend_3	|5032.975	|		|
+	|		|multifdsend_4	|4533.865	|		|
+	|		|multifdsend_5	|4530.461	|		|
+	|		|multifdsend_6	|5171.916	|		|
+	|		|multifdsend_7	|4722.769	|41922		|
+	|---------------|---------------|---------------|---------------|
+	|DSA		|live_migration	|6129.168	|		|
+	|		|multifdsend_0	|2954.717	|		|
+	|		|multifdsend_1	|2766.359	|		|
+	|		|multifdsend_2	|2853.519	|		|
+	|		|multifdsend_3	|2740.717	|		|
+	|		|multifdsend_4	|2824.169	|		|
+	|		|multifdsend_5	|2966.908	|		|
+	|		|multifdsend_6	|2611.137	|		|
+	|		|multifdsend_7	|3114.732	|		|
+	|		|dsa_completion	|3612.564	|32568		|
+	|---------------|---------------|---------------|---------------|
+
+Baseline total runtime is calculated by adding up all multifdsend_X
+and live_migration threads runtime. DSA offloading total runtime is
+calculated by adding up all multifdsend_X, live_migration and
+dsa_completion threads runtime. 41922 msec VS 32568 msec runtime and
+that is 23% total CPU usage savings.
+
+::
+
+	Latency
+	|---------------|---------------|---------------|---------------|---------------|---------------|
+	|		|total time	|down time	|throughput	|transferred-ram|total-ram	|
+	|---------------|---------------|---------------|---------------|---------------|---------------|
+	|Baseline	|10343 ms	|161 ms		|41007.00 mbps	|51583797 kb	|102400520 kb	|
+	|---------------|---------------|---------------|---------------|-------------------------------|
+	|DSA offload	|9535 ms	|135 ms		|46554.40 mbps	|53947545 kb	|102400520 kb	|
+	|---------------|---------------|---------------|---------------|---------------|---------------|
+
+Total time is 8% faster and down time is 16% faster.
+
+
+B) Scenario 2: 100% (100GB) zero pages on an 100GB vm
+-----------------------------------------------------
+
+::
+
+	CPU usage
+	|---------------|---------------|---------------|---------------|
+	|		|comm		|runtime(msec)	|totaltime(msec)|
+	|---------------|---------------|---------------|---------------|
+	|Baseline	|live_migration	|4860.718	|		|
+	|	 	|multifdsend_0	|748.875	|		|
+	|		|multifdsend_1	|898.498	|		|
+	|		|multifdsend_2	|787.456	|		|
+	|		|multifdsend_3	|764.537	|		|
+	|		|multifdsend_4	|785.687	|		|
+	|		|multifdsend_5	|756.941	|		|
+	|		|multifdsend_6	|774.084	|		|
+	|		|multifdsend_7	|782.900	|11154		|
+	|---------------|---------------|-------------------------------|
+	|DSA offloading	|live_migration	|3846.976	|		|
+	|		|multifdsend_0	|191.880	|		|
+	|		|multifdsend_1	|166.331	|		|
+	|		|multifdsend_2	|168.528	|		|
+	|		|multifdsend_3	|197.831	|		|
+	|		|multifdsend_4	|169.580	|		|
+	|		|multifdsend_5	|167.984	|		|
+	|		|multifdsend_6	|198.042	|		|
+	|		|multifdsend_7	|170.624	|		|
+	|		|dsa_completion	|3428.669	|8700		|
+	|---------------|---------------|---------------|---------------|
+
+Baseline total runtime is 11154 msec and DSA offloading total runtime is
+8700 msec. That is 22% CPU savings.
+
+::
+
+	Latency
+	|--------------------------------------------------------------------------------------------|
+	|		|total time	|down time	|throughput	|transferred-ram|total-ram   |
+	|---------------|---------------|---------------|---------------|---------------|------------|
+	|Baseline	|4867 ms	|20 ms		|1.51 mbps	|565 kb		|102400520 kb|
+	|---------------|---------------|---------------|---------------|----------------------------|
+	|DSA offload	|3888 ms	|18 ms		|1.89 mbps	|565 kb		|102400520 kb|
+	|---------------|---------------|---------------|---------------|---------------|------------|
+
+Total time 20% faster and down time 10% faster.
+
+
+How To Use DSA In Migration
+===========================
+
+The migration parameter ``accel-path`` is used to specify the resource
+allocation for DSA. After the user configures
+``zero-page-detection=dsa-accel``, one or more DSA work queues need to be
+specified for migration.
+
+The following example shows two DSA work queues for zero page detection
+
+.. code-block:: shell
+
+   migrate_set_parameter zero-page-detection=dsa-accel
+   migrate_set_parameter accel-path=dsa:/dev/dsa/wq0.0 dsa:/dev/dsa/wq1.0
+
+.. note::
+
+  Accessing DSA resources requires ``sudo`` command or ``root`` privileges
+  by default. Administrators can modify the DSA device node ownership
+  so that QEMU can use DSA with specified user permissions.
+
+  For example:
+
+  #chown -R qemu /dev/dsa
+
diff --git a/docs/devel/migration/features.rst b/docs/devel/migration/features.rst
index XXXXXXX..XXXXXXX 100644
--- a/docs/devel/migration/features.rst
+++ b/docs/devel/migration/features.rst
@@ -XXX,XX +XXX,XX @@ Migration has plenty of features to support different use cases.
    qpl-compression
    uadk-compression
    qatzip-compression
+   dsa-zero-page-detection
-- 
Yichen Wang