Hello,
This series can be found on Github:
https://github.com/shvipin/linux vfio/liveupdate/v4
It adds the base support to enable Live Update feature for VFIO devices.
In v3 of this series, both core PCI subsystem and VFIO changes were
together. Based on the feedbacks on the patches, increasing
complexities and the overall number of patches, we have split the effort
in two parts, PCI and VFIO.
v4 version of PCI work is here:
https://lore.kernel.org/linux-pci/20260423212316.3431746-1-dmatlack@google.com/
This series is the VFIO part, built on top of PCI v4. Going forward,
work on both subsystem will be independent and we may not be making same
version releases in sync. However, VFIO changes will always be on the top of
some version of PCI changes.
Overview
--------
This series made following changes to support VFIO live update:
1. VFIO Live Update feature is put behind VFIO_PCI_LIVEUPDATE config
option. It also requires that VFIO_DEVICE_CDEV config is enabled.
1. To keep things simple in this series, enabling this option will
disable VFIO_PCI_DMABUF. This is temporary.
1. A VFIO cdev FD can be preserved via Live Update session and
retrieved after kexec via the same session.
2. Getting the preserved device FD after kexec via legacy VFIO group
APIs or by directly opening /dev/vfio/devices/vfioX will error out.
User has to get the FD from Live Update subsystem.
3. For now, only FD is preserved, underlying device and its states are
not.
4. To keep the next kernel safe, just before the kexec, VFIO resets
the device, disables BME and restores the original state of the
device. This is temporary and will be removed when we have added
full support of the Live Update.
5. New VFIO selftests have been added which interacts with VFIO
- vfio_pci_liveupdate_uapi_test
- vfio_pci_liveupdate_kexec_test
liveupdate.
6. Documentations have been added for kernel docs.
Dependencies
------------
This series has following dependencies as shown in the diagram
|---------------------------|
| VFIO v4 |
|---------------------------|
| Liveupdate Selftests[1] |
|---------------------------|
| PCI v4 [2] |
|---------------------------|
| Liveupdate Refcount [3] |
|---------------------------|
[1] https://lore.kernel.org/all/20260511201155.1488670-1-vipinsh@google.com/
These selftests changes are needed by other selftests also (iommu, kvm),
so, I have sent them separately.
[2] https://lore.kernel.org/linux-pci/20260423212316.3431746-1-dmatlack@google.com/
This is the PCI v4, which adds the live update support in PCI core.
[3] https://lore.kernel.org/lkml/20260423174032.3140399-1-dmatlack@google.com/
PCI v4 is dependent on this.
Github branch I have mentioned at the top has all of these dependencies.
Testing
-------
Series has been tested using new VFIO live update selftests in both qemu and
bare-metal environment. In bare-metal environment, it was ran using the
Intel DSA PCIe device.
vfio_pci_liveupdate_kexec_test test is executed as following
$ ./setup.sh 0000:00:04.0
$ ./vfio_pci_liveupdate_kexec_test --stage 1 0000:6a:01.0
$ kexec --reuse-cmdline -l -s /boot/vmlinuz
$ kexec -e
$ ./setup.sh 0000:00:04.0
$ ./vfio_pci_liveupdate_kexec_test --stage 2 0000:6a:01.0
Kernel commandline used for booting the VM and testing had following
options passed
- liveupdate=on
- kho=on
- vfio_iommu_type1.allow_unsafe_interrupts=Y
- vfio_pci.enable_sriov=Y
- vfio_pci.disable_denylist=Y
- vfio.enable_unsafe_noiommu_mode=Y
Changelog
---------
v4:
- Addressed sashiko feedback regarding issues in resetting the devize
in freeze
- Dropped the patch which was skipping reset of preserved VFIO device
after kexec when user opens the device. This will be added back in
future but for now not needed as freeze() is also resetting the
device before kexec.
- Kernel doc and comment updates based on feedbacks.
v3: https://lore.kernel.org/kvm/20260323235817.1960573-1-dmatlack@google.com/
v2: https://lore.kernel.org/kvm/20260129212510.967611-1-dmatlack@google.com/
v1: https://lore.kernel.org/kvm/20251126193608.2678510-1-dmatlack@google.com/
rfc: https://lore.kernel.org/kvm/20251018000713.677779-1-vipinsh@google.com/
David Matlack (10):
vfio/pci: Notify PCI subsystem about devices preserved across Live
Update
vfio: Enforce preserved devices are retrieved via
LIVEUPDATE_SESSION_RETRIEVE_FD
vfio/pci: Store incoming Live Update state in struct
vfio_pci_core_device
docs: liveupdate: Add documentation for VFIO PCI
vfio: selftests: Add vfio_pci_liveupdate_uapi_test
vfio: selftests: Add Makefile support for TEST_GEN_PROGS_EXTENDED
vfio: selftests: Expose iommu_modes to tests
vfio: selftests: Expose low-level helper routines for setting up
struct vfio_pci_device
vfio: selftests: Verify that opening VFIO device fails during Live
Update
vfio: selftests: Add continuous DMA to vfio_pci_liveupdate_kexec_test
Vipin Sharma (6):
vfio/pci: Register a file handler with Live Update Orchestrator
vfio/pci: Preserve vfio-pci device files across Live Update
vfio/pci: Retrieve preserved device files after Live Update
vfio: selftests: Build liveupdate library in VFIO selftests
vfio: selftests: Initialize vfio_pci_device using a VFIO cdev FD
vfio: selftests: Add vfio_pci_liveupdate_kexec_test
Documentation/core-api/liveupdate.rst | 1 +
.../driver-api/vfio_pci_liveupdate.rst | 23 ++
MAINTAINERS | 2 +
drivers/vfio/device_cdev.c | 67 +++-
drivers/vfio/group.c | 9 +
drivers/vfio/pci/Kconfig | 16 +-
drivers/vfio/pci/Makefile | 1 +
drivers/vfio/pci/vfio_pci.c | 13 +-
drivers/vfio/pci/vfio_pci_core.c | 62 ++--
drivers/vfio/pci/vfio_pci_liveupdate.c | 349 ++++++++++++++++++
drivers/vfio/pci/vfio_pci_priv.h | 19 +
drivers/vfio/vfio.h | 18 +
drivers/vfio/vfio_main.c | 16 +-
include/linux/kho/abi/vfio_pci.h | 43 +++
include/linux/vfio.h | 13 +
include/linux/vfio_pci_core.h | 2 +
tools/testing/selftests/vfio/Makefile | 23 +-
.../vfio/lib/include/libvfio/iommu.h | 2 +
.../lib/include/libvfio/vfio_pci_device.h | 8 +
tools/testing/selftests/vfio/lib/iommu.c | 4 +-
.../selftests/vfio/lib/vfio_pci_device.c | 60 ++-
.../vfio/vfio_pci_liveupdate_kexec_test.c | 256 +++++++++++++
.../vfio/vfio_pci_liveupdate_uapi_test.c | 93 +++++
23 files changed, 1041 insertions(+), 59 deletions(-)
create mode 100644 Documentation/driver-api/vfio_pci_liveupdate.rst
create mode 100644 drivers/vfio/pci/vfio_pci_liveupdate.c
create mode 100644 include/linux/kho/abi/vfio_pci.h
create mode 100644 tools/testing/selftests/vfio/vfio_pci_liveupdate_kexec_test.c
create mode 100644 tools/testing/selftests/vfio/vfio_pci_liveupdate_uapi_test.c
base-commit: 2a4c0c11c0193889446cdb6f1540cc2b9aff97dd
--
2.54.0.563.g4f69b47b94-goog