Up until now QEMU marked any zPCI device as "unmigratable."

This patch series adds support for migrating emulated devices, which are
simpler to migrate. It leaves VFIO devices still unmigratable.

To enable migration, the device state needs to be saved/restored to/from
the migration stream.

There are 2 kinds of emulated devices - those that use the zPCI IOMMU page
table emulation in QEMU (e.g., Intel IGB NIC), and those that don't (virtio).
This is important to note for testing purposes.

This change was tested on IGB, virtio-net and virtio-blk devices by running
I/O on them while performing "virsh managedsave, virsh start" and also live
migration to another host and back.

Changes in v9:
 - patch 1: update "Reviewed-by".
 - patch 14: Fix NULL dereference on error path after s390_pci_ioat_validate().
 - patch 15:
   - add .needed to vmstate_s390_pcihost,
   - add missing destruction steps after err_unlink_pbdev label.

Konstantin Shkolnyy (15):
  s390x/pci: Create function to contain translation status check
  s390x/pci: Move iommu_mr from S390PCIIOMMU to S390PCIBusDevice
  s390x/pci: Move dm_mr from S390PCIIOMMU to S390PCIBusDevice
  s390x/pci: Move iotlb from S390PCIIOMMU to S390PCIBusDevice
  s390x/pci: Remove a ptr to S390PCIBusDevice from S390PCIIOMMU
  s390x/pci: Move/rename enabled from S390PCIIOMMU to S390PCIBusDevice
  s390x/pci: Move dma_limit from S390PCIIOMMU to S390PCIBusDevice
  s390x/pci: Move g_iota from S390PCIIOMMU to S390PCIBusDevice
  s390x/pci: Move pba from S390PCIIOMMU to S390PCIBusDevice
  s390x/pci: Move pal from S390PCIIOMMU to S390PCIBusDevice
  s390x/pci: Move max_dma_limit from S390PCIIOMMU to S390PCIBusDevice
  s390x/pci: Add a comment explaining S390PCIIOMMU purpose
  s390x/pci: Factor ioat sanity checks into a separate function
  s390x/pci: Implement migration for emulated devices
  s390x/pci: Create function to contain fmb_timer start

Matthew Rosato (1):
  s390x/pci: implement IOMMU replay

 hw/s390x/s390-pci-bus.c          | 463 +++++++++++++++++++++++++++----
 hw/s390x/s390-pci-inst.c         | 164 ++++++-----
 hw/s390x/s390-pci-vfio.c         |   4 +-
 hw/s390x/s390-virtio-ccw.c       |   4 +
 include/hw/s390x/s390-pci-bus.h  |  47 +++-
 include/hw/s390x/s390-pci-inst.h |   7 +-
 6 files changed, 546 insertions(+), 143 deletions(-)

-- 
2.34.1

m 23.09.26 um 16:15 schrieb Konstantin Shkolnyy:
> Up until now QEMU marked any zPCI device as "unmigratable."
> 
> This patch series adds support for migrating emulated devices, which are
> simpler to migrate. It leaves VFIO devices still unmigratable.
> 
> To enable migration, the device state needs to be saved/restored to/from
> the migration stream.
> 
> There are 2 kinds of emulated devices - those that use the zPCI IOMMU page
> table emulation in QEMU (e.g., Intel IGB NIC), and those that don't (virtio).
> This is important to note for testing purposes.
> 
> This change was tested on IGB, virtio-net and virtio-blk devices by running
> I/O on them while performing "virsh managedsave, virsh start" and also live
> migration to another host and back.

Did you rebase the series on top of master? (does not look like)

> 
> Changes in v9:
v10?


In general the series looks pretty good to me but I am not that deep into pci.




>   - patch 1: update "Reviewed-by".
>   - patch 14: Fix NULL dereference on error path after s390_pci_ioat_validate().
>   - patch 15:
>     - add .needed to vmstate_s390_pcihost,
>     - add missing destruction steps after err_unlink_pbdev label.
> 
> Konstantin Shkolnyy (15):
>    s390x/pci: Create function to contain translation status check
>    s390x/pci: Move iommu_mr from S390PCIIOMMU to S390PCIBusDevice
>    s390x/pci: Move dm_mr from S390PCIIOMMU to S390PCIBusDevice
>    s390x/pci: Move iotlb from S390PCIIOMMU to S390PCIBusDevice
>    s390x/pci: Remove a ptr to S390PCIBusDevice from S390PCIIOMMU
>    s390x/pci: Move/rename enabled from S390PCIIOMMU to S390PCIBusDevice
>    s390x/pci: Move dma_limit from S390PCIIOMMU to S390PCIBusDevice
>    s390x/pci: Move g_iota from S390PCIIOMMU to S390PCIBusDevice
>    s390x/pci: Move pba from S390PCIIOMMU to S390PCIBusDevice
>    s390x/pci: Move pal from S390PCIIOMMU to S390PCIBusDevice
>    s390x/pci: Move max_dma_limit from S390PCIIOMMU to S390PCIBusDevice
>    s390x/pci: Add a comment explaining S390PCIIOMMU purpose
>    s390x/pci: Factor ioat sanity checks into a separate function
>    s390x/pci: Implement migration for emulated devices
>    s390x/pci: Create function to contain fmb_timer start
> 
> Matthew Rosato (1):
>    s390x/pci: implement IOMMU replay
> 
>   hw/s390x/s390-pci-bus.c          | 463 +++++++++++++++++++++++++++----
>   hw/s390x/s390-pci-inst.c         | 164 ++++++-----
>   hw/s390x/s390-pci-vfio.c         |   4 +-
>   hw/s390x/s390-virtio-ccw.c       |   4 +
>   include/hw/s390x/s390-pci-bus.h  |  47 +++-
>   include/hw/s390x/s390-pci-inst.h |   7 +-
>   6 files changed, 546 insertions(+), 143 deletions(-)
>

Am 23.09.26 um 16:15 schrieb Konstantin Shkolnyy:
> Up until now QEMU marked any zPCI device as "unmigratable."
> 
> This patch series adds support for migrating emulated devices, which are
> simpler to migrate. It leaves VFIO devices still unmigratable.
> 
> To enable migration, the device state needs to be saved/restored to/from
> the migration stream.
> 
> There are 2 kinds of emulated devices - those that use the zPCI IOMMU page
> table emulation in QEMU (e.g., Intel IGB NIC), and those that don't (virtio).
> This is important to note for testing purposes.
> 
> This change was tested on IGB, virtio-net and virtio-blk devices by running
> I/O on them while performing "virsh managedsave, virsh start" and also live
> migration to another host and back.



FWIW, I asked an AI to test this series and I can confirm it works:
Pre-series baseline: the same guest cannot be saved at all: "State blocked by
non-migratable device 'zpci'".

Managedsave and restore: completes, the guest resumes with identical PCI functions,
reads from both virtio-rng devices keep working, and the MSI counter keeps climbing.
The stream carries the enabled state, the IOAT registration, the MSI-X indicators and
ISC, and the FMB address for all three functions, and the FMB sample counter advanced
across the restore.

IOMMU replay confirmed: with translation tracing on the restored instance, the device
configured with iommu_platform=on produced 12,299 translations for a 256 KiB read and
the bypass device none, with six translations already during the load itself from virtio
reading its rings. That is the priority ordering and the replay doing their jobs.

After restore: hotplug of a new function and unplug of an existing one both work, a second
save/restore chain into a third process works, and savevm/loadvm on a running guest works
twice, including with a device that was hotplugged after the snapshot.