Hi,
This is a follow up to the third RFC patchset [0] for supporting multiple
vSMMU instances and using iommufd to propagate DMA mappings to kernel for
VM-assigned host devices in a qemu VM. It builds on the recent libvirt patch
series [1] without support for iommufd and the 'accel' iommu attribute.
This patchset implements support for specifying multiple <iommu> devices
within the VM definition when smmuv3Dev IOMMU model is specified, and is
tested with Shameer's latest qemu RFC for HW-accelerated vSMMU devices [2]
Moreover, it adds a new 'iommufd' attribute for hostdev devices to be
associated with the iommufd object.
For instance, specifying the iommufd object and associated hostdev in a
VM definition with multiple IOMMUs, configured to be routed to
pcie-expander-bus controllers in a way where VFIO device to SMMUv3
associations are matched with the host:
<devices>
...
<controller type='pci' index='1' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='252'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='248'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</controller>
...
<controller type='pci' index='21' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='21' port='0x0'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='22' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='22' port='0xa8'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
...
<hostdev mode='subsystem' type='pci' managed='no'>
<driver iommufd='yes'/>
<source>
<address domain='0x0009' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x15' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='no'>
<driver iommufd='yes'/>
<source>
<address domain='0x0019' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x16' slot='0x00' function='0x0'/>
</hostdev>
<iommu model='smmuv3Dev' parentIdx='1' accel='on'/>
<iommu model='smmuv3Dev' parentIdx='2' accel='on'/>
</devices>
This would get translated to a qemu command line with the arguments below.
Note that libvirt will open the /dev/iommu and VFIO cdev, passing the
associated fd number to qemu:
-device '{"driver":"pxb-pcie","bus_nr":252,"id":"pci.1","bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"pxb-pcie","bus_nr":248,"id":"pci.2","bus":"pcie.0","addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":0,"chassis":21,"id":"pci.21","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"pcie-root-port","port":168,"chassis":22,"id":"pci.22","bus":"pci.2","addr":"0x0"}' \
-object '{"qom-type":"iommufd","id":"iommufd0","fd":"24"}' \
-device '{"driver":"arm-smmuv3","primary-bus":"pci.1","id":"smmuv3.0","accel":true}' \
-device '{"driver":"arm-smmuv3","primary-bus":"pci.2","id":"smmuv3.1","accel":true}' \
-device '{"driver":"vfio-pci","host":"0009:01:00.0","id":"hostdev0","iommufd":"iommufd0","fd":"22","bus":"pci.21","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0019:01:00.0","id":"hostdev1","iommufd":"iommufd0","fd":"25","bus":"pci.22","addr":"0x0"}' \
Changes from RFCv3:
- Change iommufd attribute value type to boolean
- Move iommufd attribute under hostdev PCI driver struct
- Use virDomainDeviceInfoAddressIsEqual() for comparing definitions in virDomainIOMMUDefEquals()
- Added qemuxmlconfdata test cases
- Include seclabel logic for /dev/iommu
This series is on Github:
https://github.com/NathanChenNVIDIA/libvirt/tree/smmuv3Dev-iommufd-09-15-25
Thanks,
Nathan
[0] https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/XXEO2O54IEJPTKRWLDZKCO53DM4RYL27/
[1] https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/3PYS3PTBP2DCOPP4SHD3UZSRELTX7RFC/
[2] https://lore.kernel.org/qemu-devel/20250714155941.22176-1-shameerali.kolothum.thodi@huawei.com/
Nathan Chen (8):
qemu: add IOMMU model smmuv3Dev
conf: Support multiple smmuv3Dev IOMMU devices
tests: qemuxmlconfdata: provide smmuv3Dev sample XML and CLI args
qemu: Implement support for associating iommufd to hostdev
qemu: open iommufd FDs from libvirt backend
qemu: Update Cgroup, namespace, and seclabel for qemu to access
iommufd paths
tests: qemuxmlconfdata: provide iommufd sample XML and CLI args
cover letter: qemu: Implement support for iommufd and multiple vSMMUs
docs/formatdomain.rst | 21 +-
src/conf/device_conf.c | 9 +
src/conf/device_conf.h | 1 +
src/conf/domain_conf.c | 120 ++++++--
src/conf/domain_conf.h | 12 +-
src/conf/domain_validate.c | 58 +++-
src/conf/schemas/basictypes.rng | 5 +
src/conf/schemas/domaincommon.rng | 15 +-
src/libvirt_private.syms | 2 +
src/qemu/qemu_alias.c | 15 +-
src/qemu/qemu_cgroup.c | 61 ++++
src/qemu/qemu_cgroup.h | 1 +
src/qemu/qemu_command.c | 265 +++++++++++++-----
src/qemu/qemu_command.h | 3 +-
src/qemu/qemu_domain.c | 8 +
src/qemu/qemu_domain.h | 7 +
src/qemu/qemu_domain_address.c | 33 ++-
src/qemu/qemu_driver.c | 8 +-
src/qemu/qemu_hotplug.c | 2 +-
src/qemu/qemu_namespace.c | 44 +++
src/qemu/qemu_postparse.c | 11 +-
src/qemu/qemu_process.c | 232 +++++++++++++++
src/qemu/qemu_validate.c | 18 +-
src/security/security_apparmor.c | 15 +
src/security/security_dac.c | 34 +++
src/security/security_selinux.c | 30 ++
src/util/virpci.c | 68 +++++
src/util/virpci.h | 1 +
.../iommu-smmuv3Dev.aarch64-latest.args | 39 +++
.../iommu-smmuv3Dev.aarch64-latest.xml | 62 ++++
tests/qemuxmlconfdata/iommu-smmuv3Dev.xml | 49 ++++
.../iommufd-q35.x86_64-latest.args | 41 +++
.../iommufd-q35.x86_64-latest.xml | 60 ++++
tests/qemuxmlconfdata/iommufd-q35.xml | 38 +++
.../iommufd-virt.aarch64-latest.args | 33 +++
.../iommufd-virt.aarch64-latest.xml | 34 +++
tests/qemuxmlconfdata/iommufd-virt.xml | 22 ++
.../iommufd.x86_64-latest.args | 35 +++
.../qemuxmlconfdata/iommufd.x86_64-latest.xml | 38 +++
tests/qemuxmlconfdata/iommufd.xml | 30 ++
tests/qemuxmlconftest.c | 5 +
41 files changed, 1447 insertions(+), 138 deletions(-)
create mode 100644 tests/qemuxmlconfdata/iommu-smmuv3Dev.aarch64-latest.args
create mode 100644 tests/qemuxmlconfdata/iommu-smmuv3Dev.aarch64-latest.xml
create mode 100644 tests/qemuxmlconfdata/iommu-smmuv3Dev.xml
create mode 100644 tests/qemuxmlconfdata/iommufd-q35.x86_64-latest.args
create mode 100644 tests/qemuxmlconfdata/iommufd-q35.x86_64-latest.xml
create mode 100644 tests/qemuxmlconfdata/iommufd-q35.xml
create mode 100644 tests/qemuxmlconfdata/iommufd-virt.aarch64-latest.args
create mode 100644 tests/qemuxmlconfdata/iommufd-virt.aarch64-latest.xml
create mode 100644 tests/qemuxmlconfdata/iommufd-virt.xml
create mode 100644 tests/qemuxmlconfdata/iommufd.x86_64-latest.args
create mode 100644 tests/qemuxmlconfdata/iommufd.x86_64-latest.xml
create mode 100644 tests/qemuxmlconfdata/iommufd.xml
--
2.43.0