Test result of multiple Mellanox CX-7 VFs migration: PASS
[1] create two VFs and bind them to mlx5_vfio_pci driver
[2] start a VM with two VFs
[3] migrate the VM
[4] check if the VM works well
Tested-by: YangHang Liu <yanghliu@redhat.com>
Best Regards,
YangHang Liu
On Wed, Aug 2, 2023 at 4:16 PM Avihai Horon <avihaih@nvidia.com> wrote:
>
> Hi all,
>
> The first patch in this series adds a small optimization to VFIO
> migration by moving the STOP_COPY->STOP transition to
> vfio_save_cleanup(). Testing with a ConnectX-7 VFIO device showed that
> this can reduce downtime by up to 6%.
>
> The rest of the series adds P2P support for VFIO migration.
>
> VFIO migration uAPI defines an optional intermediate P2P quiescent
> state. While in the P2P quiescent state, P2P DMA transactions cannot be
> initiated by the device, but the device can respond to incoming ones.
> Additionally, all outstanding P2P transactions are guaranteed to have
> been completed by the time the device enters this state.
>
> The purpose of this state is to support migration of multiple devices
> that might do P2P transactions between themselves.
>
> To implement P2P migration support, all the devices must be transitioned
> to the P2P quiescent state before being stopped or started.
>
> This behavior is achieved by adding an optional prepare callback to
> VMChangeStateEntry. These callbacks are invoked before the main VM state
> change callbacks, transitioning all the VFIO devices to the P2P state,
> and only then are the main callbacks invoked, which stop or start the
> devices.
>
> This will allow migration of multiple VFIO devices if all of them
> support P2P migration.
>
> Thanks.
>
> Changes from v2 [2]:
> * Registered VFIO prepare callback only if P2P migration is supported by
> the device. (Cedric)
> * Added Cedric R-b tags to patches 2,3 and 6.
>
> Changes from v1 [1]:
> * Rebased on latest master branch.
> * Renamed pre_change_cb to prepare_cb and adjusted relevant code
> accordingly. (Cedric)
> * Split VFIO vmstate change handler to two separate handlers, one for
> prepare callback and another for main callback. (Cedric)
> * Renamed vfio_should_block_multiple_devices_migration() to
> vfio_multiple_devices_migration_is_supported() and reverted its logic
> accordingly. (Cedric)
> * Rephrased "that are doing P2P" to "that might do P2P" in docs and
> commit message. (Jason)
> * Added Cedric R-b tag to patch #4.
>
> [1]
> https://lore.kernel.org/qemu-devel/20230716081541.27900-1-avihaih@nvidia.com/
>
> [2]
> https://lore.kernel.org/qemu-devel/20230731102521.15335-1-avihaih@nvidia.com/
>
> Avihai Horon (5):
> vfio/migration: Move from STOP_COPY to STOP in vfio_save_cleanup()
> sysemu: Add prepare callback to struct VMChangeStateEntry
> qdev: Add qdev_add_vm_change_state_handler_full()
> vfio/migration: Add P2P support for VFIO migration
> vfio/migration: Allow migration of multiple P2P supporting devices
>
> Joao Martins (1):
> vfio/migration: Refactor PRE_COPY and RUNNING state checks
>
> docs/devel/vfio-migration.rst | 93 +++++++++++++++++++------------
> include/hw/vfio/vfio-common.h | 2 +
> include/sysemu/runstate.h | 7 +++
> hw/core/vm-change-state-handler.c | 14 ++++-
> hw/vfio/common.c | 50 +++++++++++++----
> hw/vfio/migration.c | 75 ++++++++++++++++++++-----
> softmmu/runstate.c | 40 +++++++++++++
> hw/vfio/trace-events | 1 +
> 8 files changed, 218 insertions(+), 64 deletions(-)
>
> --
> 2.26.3
>
>