[PATCH v1 0/6] hw/arm/virt: Support dirty ring
The patches work well on my arm Ampere host.
The test results are as expected.
Testing
=======
(1) kvm-unit-tests/its-pending-migration,
kvm-unit-tests/its-migrate-unmapped-collection
and kvm-unit-tests/its-migration with dirty ring or normal dirty
page tracking
mechanism. All test cases passed.
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64 ACCEL=kvm \
PROCESSOR=host ./its-migration
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64 ACCEL=kvm \
PROCESSOR=host ./its-migrate-unmapped-collection
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64 ACCEL=kvm \
PROCESSOR=host ./its-pending-migration
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64
ACCEL=kvm,dirty-ring-size=65536 \
PROCESSOR=host ./its-migration
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64
ACCEL=kvm,dirty-ring-size=65536 \
PROCESSOR=host ./its-migrate-unmapped-collection
QEMU=/home/zhenyzha/sandbox/qemu/build/qemu-system-aarch64
ACCEL=kvm,dirty-ring-size=65536 \
PROCESSOR=host ./its-pending-migration
(2) Combinations of migration, post-copy migration, e1000e and virtio-net
devices. All test cases passed.
-device '{"driver": "virtio-net-pci", "mac": "9a:97:8f:c7:cc:a6",
"rombar": 0, "netdev": "idDGdh30", "bus": "pcie-root-port-4", "addr":
"0x0"}' \
-netdev tap,id=idDGdh30,vhost=on
-device '{"driver": "e1000e", "mac": "9a:fd:93:f1:97:b1",
"netdev": "idXDOtMA", "bus": "pcie-root-port-4", "addr": "0x0"}' \
-netdev tap,id=idXDOtMA,vhost=on \
(3) Simulate heavy memory pressure scenarios and compare the migration
performance difference between dirty ring and dirty logging.
I gave with a 200G memory guest, 40 vcpus, using 10g NIC as migration
channel. When idle or dirty workload small, I don't observe major
difference on total migration time. When with higher random dirty
workload (Anonymous mapping 180G memory, 256MB/s dirty rate upon).
Total migration time is (in seconds):
|-------------------------+|-------------------------|
| dirty ring (4k entries) | dirty logging |
|-------------------------+|-------------------------|
| 67 | 74 |
| 67 | 74 |
| 66 | 76 |
| 66 | 73 |
| 67 | 76 |
| 67 | 76 |
| 66 | 73 |
| 67 | 74 |
|-------------------------+|-------------------------|
Summary:
dirty ring average: 67s
dirty logging average: 75s
Tested-by: Zhenyu Zhang <zhenzha@redhat.com>
On Mon, Feb 13, 2023 at 8:39 AM Gavin Shan <gshan@redhat.com> wrote:
>
> This series intends to support dirty ring for live migration. The dirty
> ring use discrete buffer to track dirty pages. For ARM64, the speciality
> is to use backup bitmap to track dirty pages when there is no-running-vcpu
> context. It's known that the backup bitmap needs to be synchronized when
> KVM device "kvm-arm-gicv3" or "arm-its-kvm" has been enabled. The backup
> bitmap is collected in the last stage of migration.
>
> PATCH[1] Synchronize linux-headers for dirty ring
> PATCH[2] Introduce indicator of the last stage migration and pass it
> all the way down
> PATCH[3] Synchronize the backup bitmap in the last stage of live migration
> PATCH[4] Introduce helper kvm_dirty_ring_init() to enable the dirty ring
> PATCH[5-6] Enable dirty ring for hw/arm/virt
>
> RFCv1: https://lists.nongnu.org/archive/html/qemu-arm/2023-02/msg00171.html
>
> Testing
> =======
> (1) kvm-unit-tests/its-pending-migration and kvm-unit-tests/its-migration with
> dirty ring or normal dirty page tracking mechanism. All test cases passed.
>
> QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
> ./its-pending-migration
>
> QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm \
> ./its-migration
>
> QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \
> ./its-pending-migration
>
> QEMU=./qemu.main/build/qemu-system-aarch64 ACCEL=kvm,dirty-ring-size=65536 \
> ./its-migration
>
> (2) Combinations of migration, post-copy migration, e1000e and virtio-net
> devices. All test cases passed.
>
> -netdev tap,id=net0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
> -device e1000e,bus=pcie.5,netdev=net0,mac=52:54:00:f1:26:a0
>
> -netdev tap,id=vnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
> -device virtio-net-pci,bus=pcie.6,netdev=vnet0,mac=52:54:00:f1:26:b0
>
> Changelog
> =========
> v1:
> * Combine two patches into one PATCH[v1 2/6] for the last stage indicator (Peter)
> * Drop the secondary bitmap and use the original one directly (Juan)
> * Avoid "goto out" in helper kvm_dirty_ring_init() (Juan)
>
>
> Gavin Shan (6):
> linux-headers: Update for dirty ring
> migration: Add last stage indicator to global dirty log
> synchronization
> kvm: Synchronize the backup bitmap in the last stage
> kvm: Add helper kvm_dirty_ring_init()
> hw/arm/virt: Enable backup bitmap for dirty ring
> kvm: Enable dirty ring for arm64
>
> accel/kvm/kvm-all.c | 95 ++++++++++++++++++++++++-----------
> hw/arm/virt.c | 26 ++++++++++
> include/exec/memory.h | 5 +-
> include/sysemu/kvm_int.h | 1 +
> linux-headers/asm-arm64/kvm.h | 1 +
> linux-headers/linux/kvm.h | 2 +
> migration/dirtyrate.c | 4 +-
> migration/ram.c | 20 ++++----
> softmmu/memory.c | 10 ++--
> target/arm/kvm64.c | 25 +++++++++
> target/arm/kvm_arm.h | 12 +++++
> 11 files changed, 152 insertions(+), 49 deletions(-)
>
> --
> 2.23.0
>