arch/arm64/include/uapi/asm/kvm.h | 10 +++ arch/arm64/kvm/vgic/vgic-kvm-device.c | 113 +++++++++++++++++++++++++++++++--- 2 files changed, 114 insertions(+), 9 deletions(-)
This series adds batch register access support to KVM/arm64 to reduce
syscall overhead during VM live migration.
Currently, QEMU issues one ioctl per register when saving/restoring VGIC
state. On large VM configurations this means tens of thousands of syscalls,
where lock acquisition and context switch overhead dominates migration
downtime. Thus, we provide a batch register method to allow userspace
read/write multiple distributor and redistributor registers in a single call.
In this way, we can significantly reduce syscalls and migration downtime.
Test the VM migration time under pressure conditions.
The VM specifications for migration are as follows:
- VM use 4-K page;
- the number of VCPU is 160;
- the total memory is 320Gigabit;
- use 'Redis SET-benchmark' to pressurize VM;
Performance results (3-run average, ms):
| Metric | Without patch | With patch | Improvement |
|---------------------|---------------|------------|-------------|
| Migration downtime | 536 | 321 | 40% |
| Source (total) | 344 | 230 | 33% |
| - VGIC put | 158 | 40 | 75% |
| - VGIC get | 120 | 19 | 84% |
| Destination (total) | 192 | 91 | 53% |
| - VGIC put | 132 | 27 | 80% |
Yize Wang (2):
KVM: arm64: Add batch group constant and data structure to UAPI header
KVM: arm64: Add VGIC v3 batch register access implementation
arch/arm64/include/uapi/asm/kvm.h | 10 +++
arch/arm64/kvm/vgic/vgic-kvm-device.c | 113 +++++++++++++++++++++++++++++++---
2 files changed, 114 insertions(+), 9 deletions(-)
On Fri, 18 Sep 2026 09:18:13 +0100, Yize Wang <wangyize7@huawei.com> wrote: > > This series adds batch register access support to KVM/arm64 to reduce > syscall overhead during VM live migration. > > Currently, QEMU issues one ioctl per register when saving/restoring VGIC > state. On large VM configurations this means tens of thousands of syscalls, > where lock acquisition and context switch overhead dominates migration > downtime. Thus, we provide a batch register method to allow userspace > read/write multiple distributor and redistributor registers in a single call. > In this way, we can significantly reduce syscalls and migration downtime. > > Test the VM migration time under pressure conditions. > The VM specifications for migration are as follows: > - VM use 4-K page; > - the number of VCPU is 160; > - the total memory is 320Gigabit; > - use 'Redis SET-benchmark' to pressurize VM; > > Performance results (3-run average, ms): > | Metric | Without patch | With patch | Improvement | > |---------------------|---------------|------------|-------------| > | Migration downtime | 536 | 321 | 40% | > | Source (total) | 344 | 230 | 33% | > | - VGIC put | 158 | 40 | 75% | > | - VGIC get | 120 | 19 | 84% | > | Destination (total) | 192 | 91 | 53% | > | - VGIC put | 132 | 27 | 80% | > > Yize Wang (2): > KVM: arm64: Add batch group constant and data structure to UAPI header > KVM: arm64: Add VGIC v3 batch register access implementation Questions: - Why only the MMIO registers? - Why not the sysregs? - Why only the GIC? - Why not all of the state? - Where is the corresponding userspace code? More importantly, since this is about batching system calls: - Why can't this be done with io_uring instead? M. -- Without deviation from the norm, progress is not possible.
在 2026/9/18 20:08, Marc Zyngier 写道: > On Fri, 18 Sep 2026 09:18:13 +0100, > Yize Wang <wangyize7@huawei.com> wrote: >> This series adds batch register access support to KVM/arm64 to reduce >> syscall overhead during VM live migration. >> >> Currently, QEMU issues one ioctl per register when saving/restoring VGIC >> state. On large VM configurations this means tens of thousands of syscalls, >> where lock acquisition and context switch overhead dominates migration >> downtime. Thus, we provide a batch register method to allow userspace >> read/write multiple distributor and redistributor registers in a single call. >> In this way, we can significantly reduce syscalls and migration downtime. >> >> Test the VM migration time under pressure conditions. >> The VM specifications for migration are as follows: >> - VM use 4-K page; >> - the number of VCPU is 160; >> - the total memory is 320Gigabit; >> - use 'Redis SET-benchmark' to pressurize VM; >> >> Performance results (3-run average, ms): >> | Metric | Without patch | With patch | Improvement | >> |---------------------|---------------|------------|-------------| >> | Migration downtime | 536 | 321 | 40% | >> | Source (total) | 344 | 230 | 33% | >> | - VGIC put | 158 | 40 | 75% | >> | - VGIC get | 120 | 19 | 84% | >> | Destination (total) | 192 | 91 | 53% | >> | - VGIC put | 132 | 27 | 80% | >> >> Yize Wang (2): >> KVM: arm64: Add batch group constant and data structure to UAPI header >> KVM: arm64: Add VGIC v3 batch register access implementation > Questions: > > - Why only the MMIO registers? > > - Why not the sysregs? > > - Why only the GIC? > > - Why not all of the state? > > - Where is the corresponding userspace code? > > More importantly, since this is about batching system calls: > > - Why can't this be done with io_uring instead? > > M. Hi, Marc! Thank you for the review. These patches focus on optimizing GICv3 register access during live migration. We found that there are a large number of locks (kvm->lock, vcpus, config_lock) in the GIC, these lock operations wil cost large time waste. The batches of sysreg for vcpu optimization will come in follow as a separate series. And let me address these questions one by one. 1. Why only the MMIO registers? In vgic_v3_batch_access(), we use 'entries' structure to implement batch read/write of register status. The structure is 'struct kvm_dev_arm_vgic_batch_entry', where the group information can be freely specified by userspace. Thus, vgic_v3_batch_access() supports all VGIC device attr groups, not only MMIO registers. 2. Why not the sysregs? The newly added vgic_v3_batch_access() just forwards the groups received fromQEMU in batches to 'vgic_v3_attr_regs_access()'. And this function already has 'KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS' to handle sysregs. Our patch does not modify the existing sysreg handling logic. 3. Why only the GIC? During live migration, the GIC is the device with the largest number of registers, and we found that there are a large number of locks in it. If each register is locked and unlocked individually, it would cause large time consumption. Thus, we want to optimize the GIC register time consumption during live migrate. The experimental results also show that batch processing significantly reduces VGIC handling time during migration. Similarly, vCPU register save/restore also costs significantly time. The patch of vCPU sysreg batch access will be submitted separately in the future. 4. Why not all of the state? As different devices have different lock hierarchies and access paths(e.g., GIC goes through the device fd, CPU regs go through the vcpu fd), it's difficult for us to realize in a single batch handler. Additionally, introducing too many changes at once would make review harder. These patches focus on GIC-related optimization, and a separate series for vCPU batch processing will follow. 5. Where is the corresponding userspace code? The QEMU-side implementation has been posted to qemu-devel. The link is below: https://lore.kernel.org/qemu-devel/20260918092120.370805-1-wangyize7@huawei.com/T/#t 6. Why can't this be done with io_uring instead? The core idea of io_uring is to reduce the number of context switches between user space and kernel space by utilizing two ring buffers. However, each SQE is still processed independently in kernel space. For the VGIC registers, each SQE still require lock -> read/write -> unlock, so the per-register lock overhead remains unchanged. We hope to read/write a set of register states with a single lock operation to reduce the time cost. Therefore, io_uring does not meet our needs.
kindly ping 在 2026/9/20 20:12, Yize Wang 写道: > > 在 2026/9/18 20:08, Marc Zyngier 写道: >> On Fri, 18 Sep 2026 09:18:13 +0100, >> Yize Wang <wangyize7@huawei.com> wrote: >>> This series adds batch register access support to KVM/arm64 to reduce >>> syscall overhead during VM live migration. >>> >>> Currently, QEMU issues one ioctl per register when saving/restoring >>> VGIC >>> state. On large VM configurations this means tens of thousands of >>> syscalls, >>> where lock acquisition and context switch overhead dominates migration >>> downtime. Thus, we provide a batch register method to allow userspace >>> read/write multiple distributor and redistributor registers in a >>> single call. >>> In this way, we can significantly reduce syscalls and migration >>> downtime. >>> >>> Test the VM migration time under pressure conditions. >>> The VM specifications for migration are as follows: >>> - VM use 4-K page; >>> - the number of VCPU is 160; >>> - the total memory is 320Gigabit; >>> - use 'Redis SET-benchmark' to pressurize VM; >>> >>> Performance results (3-run average, ms): >>> | Metric | Without patch | With patch | Improvement | >>> |---------------------|---------------|------------|-------------| >>> | Migration downtime | 536 | 321 | 40% | >>> | Source (total) | 344 | 230 | 33% | >>> | - VGIC put | 158 | 40 | 75% | >>> | - VGIC get | 120 | 19 | 84% | >>> | Destination (total) | 192 | 91 | 53% | >>> | - VGIC put | 132 | 27 | 80% | >>> >>> Yize Wang (2): >>> KVM: arm64: Add batch group constant and data structure to UAPI >>> header >>> KVM: arm64: Add VGIC v3 batch register access implementation >> Questions: >> >> - Why only the MMIO registers? >> >> - Why not the sysregs? >> >> - Why only the GIC? >> >> - Why not all of the state? >> >> - Where is the corresponding userspace code? >> >> More importantly, since this is about batching system calls: >> >> - Why can't this be done with io_uring instead? >> >> M. > > > Hi, Marc! Thank you for the review. > > > These patches focus on optimizing GICv3 register access during live > migration. We found that there are a large number of locks (kvm->lock, > vcpus, config_lock) in the GIC, these lock operations wil cost large > time waste. The batches of sysreg for vcpu optimization will come in > follow as a separate series. And let me address these questions one by > one. > > > 1. Why only the MMIO registers? > > In vgic_v3_batch_access(), we use 'entries' structure to implement > batch read/write of register status. The structure is 'struct > kvm_dev_arm_vgic_batch_entry', where the group information can be > freely specified by userspace. Thus, vgic_v3_batch_access() supports > all VGIC device attr groups, not only MMIO registers. > > > 2. Why not the sysregs? > > The newly added vgic_v3_batch_access() just forwards the groups > received fromQEMU in batches to 'vgic_v3_attr_regs_access()'. And this > function already has 'KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS' to handle > sysregs. Our patch does not modify the existing sysreg handling logic. > > > 3. Why only the GIC? > > During live migration, the GIC is the device with the largest number > of registers, and we found that there are a large number of locks in > it. If each register is locked and unlocked individually, it would > cause large time consumption. Thus, we want to optimize the GIC > register time consumption during live migrate. The experimental > results also show that batch processing significantly reduces VGIC > handling time during migration. > > Similarly, vCPU register save/restore also costs significantly time. > The patch of vCPU sysreg batch access will be submitted separately in > the future. > > > 4. Why not all of the state? > > As different devices have different lock hierarchies and access > paths(e.g., GIC goes through the device fd, CPU regs go through the > vcpu fd), it's difficult for us to realize in a single batch handler. > Additionally, introducing too many changes at once would make review > harder. These patches focus on GIC-related optimization, and a > separate series for vCPU batch processing will follow. > > > 5. Where is the corresponding userspace code? > > The QEMU-side implementation has been posted to qemu-devel. The link > is below: > > https://lore.kernel.org/qemu-devel/20260918092120.370805-1-wangyize7@huawei.com/T/#t > > > > 6. Why can't this be done with io_uring instead? > > The core idea of io_uring is to reduce the number of context switches > between user space and kernel space by utilizing two ring buffers. > However, each SQE is still processed independently in kernel space. > For the VGIC registers, each SQE still require lock -> read/write -> > unlock, so the per-register lock overhead remains unchanged. We hope > to read/write a set of register states with a single lock operation to > reduce the time cost. Therefore, io_uring does not meet our needs.
© 2016 - 2026 Red Hat, Inc.