This series splits the RDMA live-migration send path into two threads:
- a producer thread (the migration thread) that scans dirty pages and
records which chunks must be migrated, pushing chunk descriptors
(block + chunk index + dirty byte span) onto a lock-free SPSC ring;
and
- a consumer thread that owns the RDMA queue pair: it drains the ring,
posts the RDMA writes and control messages, handles chunk
registration and barriers, and polls the completion queue.
Chunk-granular descriptors on the ring replace the previous per-page
merge logic on the migration thread.
Dependencies / how to apply
---------------------------
Prerequisites (not included; apply first), both from Yanfei Xu:
[PATCH v3 0/4] migration/rdma: Allow multiple writes per chunk
https://lore.kernel.org/qemu-devel/20260913062455.940447-1-yanfei.xu@bytedance.com/
[PATCH v3 0/3] migration/rdma: Misc RDMA migration fixes
https://lore.kernel.org/qemu-devel/20260823021657.2824542-1-yanfei.xu@bytedance.com/
The base-commit below is upstream master; both Yanfei series are recorded
as the prerequisite-patch-id entries after it.
Note: the misc-fixes series here is Yanfei's v3 rebased onto current
master (trace events renamed qemu_rdma_* -> rdma_*), so its
prerequisite-patch-id entries differ from the v3 posting.
Patches 1-12 are the producer/consumer split. Patch 13 (Jack Wang)
negotiates the batched chunk registration capability so a peer that
lacks it falls back to one chunk per request.
Results
-------
64 GiB guest, single-path RDMA, memload workload (one-time RAM fill,
fixed dirty set). "before" is the single-thread send path with
only the nb_sent-stats patch (patch 1) applied; "after" is the full
producer/consumer series, so the delta isolates the split itself.
pin_all=off:
- total migration time: -16.2% (24 GiB fill), -14.2% (60 GiB fill)
- transfer-phase time: -16.3% (24 GiB fill), -14.2% (60 GiB fill)
- RAM throughput: +17.2% (24 GiB fill), +16.6% (60 GiB fill)
- guest downtime: -55.3% (24 GiB fill), -23.1% (60 GiB fill)
pin_all=on (whole-RAM registration at setup is unchanged and dominates
total time, so the split shows mainly in the transfer phase):
- total migration time: -5.0%
- transfer-phase time: -10.9%
- transfer speed: +12.9%
- guest downtime: -1.5%
QP saturation was measured with the new nb_sent send-queue counter:
send-queue "drain to zero" events (queue starved empty) fell by ~99% in
the transfer-bound pin_all=off cases (24 GiB: 26609 -> 276; 60 GiB:
63771 -> 641), while "full-queue" hits rose.
This is the first step toward RDMA multipath migration.
Dmitrii Omelchenko (12):
migration/rdma: Add nb_sent send-queue-depth stats
migration/rdma: Add producer/consumer send-ring types
migration/rdma: Add send-ring lifecycle (no consumer thread yet)
migration/rdma: Add the send consumer thread (drain only)
migration/rdma: Handle write/flush/reg-stop on the consumer
migration/rdma: Extract qemu_rdma_write_file() and dispatch FILE
migration/rdma: Switch the producer onto the send ring
migration/rdma: Add producer/consumer tuning instrumentation
migration/rdma: drain the send ring in batches
migration/rdma: size reg_resp.len by the register request count
migration/rdma: batch chunk registration in one round-trip
migration/rdma: trace batching and register runs
Jack Wang (1):
migration/rdma: negotiate batched chunk registration
migration/rdma.c | 1275 +++++++++++++++++++++++++++++-----------
migration/trace-events | 18 +
2 files changed, 956 insertions(+), 337 deletions(-)
base-commit: 3876503faff51ce3a132dbbd0ab42c10f2269319
prerequisite-patch-id: 52ae8b66be76e06e9c74c2fca1dced7f76a260b0
prerequisite-patch-id: fadde94af9705117610d71a769d67fdd40b9de4f
prerequisite-patch-id: 8f458aab8e4d1ab1f8d1810051a624341cc9faa8
prerequisite-patch-id: a7c1ffb7ac1c0b9a31858aad636616916e963663
prerequisite-patch-id: 8327b7f067c2925781d68a6d506bf0889f31e186
prerequisite-patch-id: e1879a49d37f2a1b6ea797d3bc48e016f865cf9e
prerequisite-patch-id: 3d8d7869b69fa850f37558f3170e95a46517b3f8
--
2.39.5