Hi all,

The first patch in this series adds a small optimization to VFIO
migration by moving the STOP_COPY->STOP transition to
vfio_save_cleanup(). Testing with a ConnectX-7 VFIO device showed that
this can reduce downtime by up to 6%.

The rest of the series adds P2P support for VFIO migration.

VFIO migration uAPI defines an optional intermediate P2P quiescent
state. While in the P2P quiescent state, P2P DMA transactions cannot be
initiated by the device, but the device can respond to incoming ones.
Additionally, all outstanding P2P transactions are guaranteed to have
been completed by the time the device enters this state.

The purpose of this state is to support migration of multiple devices
that are doing P2P transactions between themselves.

To implement P2P migration support, all the devices must be transitioned
to the P2P quiescent state before being stopped or started.

This behavior is achieved by adding an optional pre VM state change
callback to VMChangeStateEntry. These callbacks are invoked before the
main VM state change callbacks, transitioning all the VFIO devices to
the P2P state, and only then are the main callbacks invoked, which stop
or start the devices.

This will allow migration of multiple VFIO devices if all of them
support P2P migration.

Thanks.

Avihai Horon (5):
  vfio/migration: Move from STOP_COPY to STOP in vfio_save_cleanup()
  sysemu: Add pre VM state change callback
  qdev: Add qdev_add_vm_change_state_handler_full()
  vfio/migration: Add P2P support for VFIO migration
  vfio/migration: Allow migration of multiple P2P supporting devices

Joao Martins (1):
  vfio/migration: Refactor PRE_COPY and RUNNING state checks

 docs/devel/vfio-migration.rst     | 93 +++++++++++++++++++------------
 include/hw/vfio/vfio-common.h     |  2 +
 include/sysemu/runstate.h         |  7 +++
 hw/core/vm-change-state-handler.c | 14 ++++-
 hw/vfio/common.c                  | 50 +++++++++++++----
 hw/vfio/migration.c               | 83 +++++++++++++++++++--------
 softmmu/runstate.c                | 39 +++++++++++++
 hw/vfio/trace-events              |  2 +-
 8 files changed, 217 insertions(+), 73 deletions(-)

-- 
2.26.3

On Sun, Jul 16, 2023 at 11:15:35AM +0300, Avihai Horon wrote:
> Hi all,
> 
> The first patch in this series adds a small optimization to VFIO
> migration by moving the STOP_COPY->STOP transition to
> vfio_save_cleanup(). Testing with a ConnectX-7 VFIO device showed that
> this can reduce downtime by up to 6%.
> 
> The rest of the series adds P2P support for VFIO migration.
> 
> VFIO migration uAPI defines an optional intermediate P2P quiescent
> state. While in the P2P quiescent state, P2P DMA transactions cannot be
> initiated by the device, but the device can respond to incoming ones.
> Additionally, all outstanding P2P transactions are guaranteed to have
> been completed by the time the device enters this state.

For clarity it is "all outstanding DMA transactions are guarenteed to
have been completed" - the device has no idea if something is P2P or
not.

> The purpose of this state is to support migration of multiple devices
> that are doing P2P transactions between themselves.

s/are/might/

Jason

On 18/07/2023 18:46, Jason Gunthorpe wrote:
> On Sun, Jul 16, 2023 at 11:15:35AM +0300, Avihai Horon wrote:
>> Hi all,
>>
>> The first patch in this series adds a small optimization to VFIO
>> migration by moving the STOP_COPY->STOP transition to
>> vfio_save_cleanup(). Testing with a ConnectX-7 VFIO device showed that
>> this can reduce downtime by up to 6%.
>>
>> The rest of the series adds P2P support for VFIO migration.
>>
>> VFIO migration uAPI defines an optional intermediate P2P quiescent
>> state. While in the P2P quiescent state, P2P DMA transactions cannot be
>> initiated by the device, but the device can respond to incoming ones.
>> Additionally, all outstanding P2P transactions are guaranteed to have
>> been completed by the time the device enters this state.
> For clarity it is "all outstanding DMA transactions are guarenteed to
> have been completed" - the device has no idea if something is P2P or
> not.

The uAPI states that [1]: "If the device can identify P2P 
transactionsthen it can stop only P2P DMA, otherwise it must stop all DMA.".

So I will stick to the uAPI definition.

[1] 
https://elixir.bootlin.com/linux/v6.5-rc1/source/include/uapi/linux/vfio.h#L1032

>
>> The purpose of this state is to support migration of multiple devices
>> that are doing P2P transactions between themselves.
> s/are/might/

Sure, will change it.

Thanks!