[PATCH 0/5] optimize the downtime for vfio migration

Longpeng(Mike) posted 5 patches 2 years, 7 months ago
Test checkpatch failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20210825075620.2607-1-longpeng2@huawei.com
Maintainers: Marcelo Tosatti <mtosatti@redhat.com>, Marcel Apfelbaum <marcel.apfelbaum@gmail.com>, Paolo Bonzini <pbonzini@redhat.com>, Alex Williamson <alex.williamson@redhat.com>, "Michael S. Tsirkin" <mst@redhat.com>
There is a newer version of this series
accel/kvm/kvm-all.c    | 10 +++--
accel/stubs/kvm-stub.c |  3 +-
hw/misc/ivshmem.c      |  2 +-
hw/pci/msix.c          |  7 ++--
hw/vfio/pci.c          | 99 ++++++++++++++++++++++++++++++++++++++------------
hw/vfio/pci.h          |  1 +
hw/virtio/virtio-pci.c |  2 +-
include/sysemu/kvm.h   |  4 +-
target/i386/kvm/kvm.c  |  2 +-
9 files changed, 95 insertions(+), 35 deletions(-)
[PATCH 0/5] optimize the downtime for vfio migration
Posted by Longpeng(Mike) 2 years, 7 months ago
In vfio migration resume phase, the cost would increase if the
vfio device has more unmasked vectors. We try to optimize it in
this series.

Patch 1 & 2 are simple code cleanups.
Patch 3 defers to set irqs to vfio core.
Patch 4 & 5 defer to commit the route to KVM core. 

The test VM has 128 vcpus and 8 VF (with 65 vectors enabled),
we mesure the cost of the vfio_msix_enable for each one, and
we can see the total cost can be significantly reduced.

        Origin         Apply Patch 3     Apply Patch 3/4/5   
1st     8              4                 2
2nd     15             11                2
3rd     22             18                2
4th     24             25                3
5th     36             33                2
6th     44             40                3
7th     51             47                3
8th     58             54                4
Total   258ms          232ms             21ms


Longpeng (Mike) (5):
  vfio: use helper to simplfy the failure path in vfio_msi_enable
  msix: simplfy the conditional in msix_set/unset_vector_notifiers
  vfio: defer to enable msix in migration resume phase
  kvm: irqchip: support defer to commit the route
  vfio: defer to commit kvm route in migraiton resume phase

 accel/kvm/kvm-all.c    | 10 +++--
 accel/stubs/kvm-stub.c |  3 +-
 hw/misc/ivshmem.c      |  2 +-
 hw/pci/msix.c          |  7 ++--
 hw/vfio/pci.c          | 99 ++++++++++++++++++++++++++++++++++++++------------
 hw/vfio/pci.h          |  1 +
 hw/virtio/virtio-pci.c |  2 +-
 include/sysemu/kvm.h   |  4 +-
 target/i386/kvm/kvm.c  |  2 +-
 9 files changed, 95 insertions(+), 35 deletions(-)

-- 
1.8.3.1


Re: [PATCH 0/5] optimize the downtime for vfio migration
Posted by Philippe Mathieu-Daudé 2 years, 7 months ago
Cc'ing David/Juan for migration big picture (just in case).

On 8/25/21 9:56 AM, Longpeng(Mike) wrote:
> In vfio migration resume phase, the cost would increase if the
> vfio device has more unmasked vectors. We try to optimize it in
> this series.
> 
> Patch 1 & 2 are simple code cleanups.
> Patch 3 defers to set irqs to vfio core.
> Patch 4 & 5 defer to commit the route to KVM core. 
> 
> The test VM has 128 vcpus and 8 VF (with 65 vectors enabled),
> we mesure the cost of the vfio_msix_enable for each one, and
> we can see the total cost can be significantly reduced.
> 
>         Origin         Apply Patch 3     Apply Patch 3/4/5   
> 1st     8              4                 2
> 2nd     15             11                2
> 3rd     22             18                2
> 4th     24             25                3
> 5th     36             33                2
> 6th     44             40                3
> 7th     51             47                3
> 8th     58             54                4
> Total   258ms          232ms             21ms
> 
> 
> Longpeng (Mike) (5):
>   vfio: use helper to simplfy the failure path in vfio_msi_enable
>   msix: simplfy the conditional in msix_set/unset_vector_notifiers
>   vfio: defer to enable msix in migration resume phase
>   kvm: irqchip: support defer to commit the route
>   vfio: defer to commit kvm route in migraiton resume phase

Overall makes sense and LGTM but migration/KVM are not my area :/


Re: [PATCH 0/5] optimize the downtime for vfio migration
Posted by Longpeng (Mike, Cloud Infrastructure Service Product Dept.) 2 years, 7 months ago

在 2021/8/25 18:05, Philippe Mathieu-Daudé 写道:
> Cc'ing David/Juan for migration big picture (just in case).
> 
> On 8/25/21 9:56 AM, Longpeng(Mike) wrote:
>> In vfio migration resume phase, the cost would increase if the
>> vfio device has more unmasked vectors. We try to optimize it in
>> this series.
>>
>> Patch 1 & 2 are simple code cleanups.
>> Patch 3 defers to set irqs to vfio core.
>> Patch 4 & 5 defer to commit the route to KVM core. 
>>
>> The test VM has 128 vcpus and 8 VF (with 65 vectors enabled),
>> we mesure the cost of the vfio_msix_enable for each one, and
>> we can see the total cost can be significantly reduced.
>>
>>         Origin         Apply Patch 3     Apply Patch 3/4/5   
>> 1st     8              4                 2
>> 2nd     15             11                2
>> 3rd     22             18                2
>> 4th     24             25                3
>> 5th     36             33                2
>> 6th     44             40                3
>> 7th     51             47                3
>> 8th     58             54                4
>> Total   258ms          232ms             21ms
>>
>>
>> Longpeng (Mike) (5):
>>   vfio: use helper to simplfy the failure path in vfio_msi_enable
>>   msix: simplfy the conditional in msix_set/unset_vector_notifiers
>>   vfio: defer to enable msix in migration resume phase
>>   kvm: irqchip: support defer to commit the route
>>   vfio: defer to commit kvm route in migraiton resume phase
> 
> Overall makes sense and LGTM but migration/KVM are not my area :/
>
Thanks all the same :)

> .
> 

Re: [PATCH 0/5] optimize the downtime for vfio migration
Posted by Longpeng (Mike, Cloud Infrastructure Service Product Dept.) 2 years, 6 months ago
Hi maintainers,

Ping...

在 2021/8/25 15:56, Longpeng(Mike) 写道:
> In vfio migration resume phase, the cost would increase if the
> vfio device has more unmasked vectors. We try to optimize it in
> this series.
> 
> Patch 1 & 2 are simple code cleanups.
> Patch 3 defers to set irqs to vfio core.
> Patch 4 & 5 defer to commit the route to KVM core. 
> 
> The test VM has 128 vcpus and 8 VF (with 65 vectors enabled),
> we mesure the cost of the vfio_msix_enable for each one, and
> we can see the total cost can be significantly reduced.
> 
>         Origin         Apply Patch 3     Apply Patch 3/4/5   
> 1st     8              4                 2
> 2nd     15             11                2
> 3rd     22             18                2
> 4th     24             25                3
> 5th     36             33                2
> 6th     44             40                3
> 7th     51             47                3
> 8th     58             54                4
> Total   258ms          232ms             21ms
> 
> 
> Longpeng (Mike) (5):
>   vfio: use helper to simplfy the failure path in vfio_msi_enable
>   msix: simplfy the conditional in msix_set/unset_vector_notifiers
>   vfio: defer to enable msix in migration resume phase
>   kvm: irqchip: support defer to commit the route
>   vfio: defer to commit kvm route in migraiton resume phase
> 
>  accel/kvm/kvm-all.c    | 10 +++--
>  accel/stubs/kvm-stub.c |  3 +-
>  hw/misc/ivshmem.c      |  2 +-
>  hw/pci/msix.c          |  7 ++--
>  hw/vfio/pci.c          | 99 ++++++++++++++++++++++++++++++++++++++------------
>  hw/vfio/pci.h          |  1 +
>  hw/virtio/virtio-pci.c |  2 +-
>  include/sysemu/kvm.h   |  4 +-
>  target/i386/kvm/kvm.c  |  2 +-
>  9 files changed, 95 insertions(+), 35 deletions(-)
>