Current memory operations like pinning may take a lot of time at the
destination.  Currently they are done after the source of the migration is
stopped, and before the workload is resumed at the destination.  This is a
period where neigher traffic can flow, nor the VM workload can continue
(downtime).

We can do better as we know the memory layout of the guest RAM at the
destination from the moment the migration starts.  Moving that operation allows
QEMU to communicate the kernel the maps while the workload is still running in
the source, so Linux can start mapping them.

Also, the destination of the guest memory may finish before the destination
QEMU maps all the memory.  In this case, the rest of the memory will be mapped
at the same time as before applying this series, when the device is starting.
So we're only improving with this series.

RFC TODO: We should be able to not finish the migration while the memory is
still not mapped, but I still need to find how.  Suggestions are welcome.

Note that further devices setup at the end of the migration may alter the guest
memory layout. But same as the previous point, many operations are still done
incrementally, like memory pinning, so we're saving time anyway.

Only tested with vdpa_sim. I'm sending this before full benchmark, as some work
like [1] can be based on it, and Si-Wei agreed on benchmark this series with
his experience.

This needs to be applied on top of [2], which perform some code reorganization
that allows to map the memory without knowing the queue layout the guest
configure on the device.

Future directions on top of this series may include:
* Iterative migration of virtio-net devices, as it may reduce downtime per [1].
  vhost-vdpa net can apply the configuration through CVQ in the destination
  while the source is still migrating.
* Move more things ahead of migration time, like DRIVER_OK.
* Check that the devices of the destination are valid, and cancel the migration
  in case it is not.

RFC v2:
* Delegate map to another thread so it does no block QMP.
* Fix not allocating iova_tree if x-svq=on at the destination.
* Rebased on latest master.
* More cleanups of current code, that might be split from this series too.

[1] https://lore.kernel.org/qemu-devel/6c8ebb97-d546-3f1c-4cdd-54e23a566f61@nvidia.com/T/
[2] https://lists.nongnu.org/archive/html/qemu-devel/2023-11/msg05331.html

Eugenio Pérez (10):
  vdpa: do not set virtio status bits if unneeded
  vdpa: make batch_begin_once early return
  vdpa: merge _begin_batch into _batch_begin_once
  vdpa: extract out _dma_end_batch from _listener_commit
  vdpa: factor out stop path of vhost_vdpa_dev_start
  vdpa: check for iova tree initialized at net_client_start
  vdpa: set backend capabilities at vhost_vdpa_init
  vdpa: add vhost_vdpa_load_setup
  vdpa: add vhost_vdpa_net_load_setup NetClient callback
  virtio_net: register incremental migration handlers

 include/hw/virtio/vhost-vdpa.h |  25 ++++
 include/net/net.h              |   6 +
 hw/net/virtio-net.c            |  35 +++++
 hw/virtio/vhost-vdpa.c         | 257 +++++++++++++++++++++++++++------
 net/vhost-vdpa.c               |  37 ++++-
 5 files changed, 312 insertions(+), 48 deletions(-)

-- 
2.39.3