[v6] hw/arm/virt: Add support for user-creatable accelerated SMMUv3

[PATCH v6 00/33] hw/arm/virt: Add support for user-creatable accelerated SMMUv3

Posted by Shameer Kolothum 3 weeks, 2 days ago

Hi,

Changes from v5:
  https://lore.kernel.org/qemu-devel/20251031105005.24618-1-skolothumtho@nvidia.com/

 - Addressed feedback from v5 and picked up R-by tags. Thanks to all!
 - The previously split out _DSM fix mini-series is now accepted [0].
 - Improved documentation about the rationale behind the design choice of
   returning an address space aliased to the system address space for
   vfio-pci endpoint devices (patch #10).
 - Added error propagation support for smmuv3_cmdq_consume() (patch #13).
 - Updated vSTE based HWPT installation to check the SMMU enabled case
   (patch #14).
 - Introduced an optional callback to PCIIOMMUOps to retrieve the MSI
   doorbell GPA directly, allowing us to avoid unsafe page table walks for
   MSI translation in accelerated SMMUv3 cases (patch #16).
 - GBPA-based vSTE update depends on Nicolin's kernel patch [1].
 - VFIO/IOMMUFD has dependency on Zhenzhong's patches: 4/5/8 from the
   pass-through support series [2].

PATCH organization:
 1–26: Enables accelerated SMMUv3 with features based on default QEMU SMMUv3,
       including IORT RMR based MSI support.
 27–29: Adds options for specifying RIL, ATS, and OAS features.
 30–33: Adds PASID support, including VFIO changes.

Tests:
Performed basic sanity tests on an NVIDIA GRACE platform with GPU device
assignments. A CUDA test application was used to verify the SVA use case.
Further tests are always welcome.

Eg: Qemu Cmd line:

qemu-system-aarch64 -machine virt,gic-version=3,highmem-mmio-size=2T \
-cpu host -smp cpus=4 -m size=16G,slots=2,maxmem=66G -nographic \
-bios QEMU_EFI.fd -object iommufd,id=iommufd0 -enable-kvm \
-object memory-backend-ram,size=8G,id=m0 \
-object memory-backend-ram,size=8G,id=m1 \
-numa node,memdev=m0,cpus=0-3,nodeid=0 -numa node,memdev=m1,nodeid=1 \
-numa node,nodeid=2 -numa node,nodeid=3 -numa node,nodeid=4 -numa node,nodeid=5 \
-numa node,nodeid=6 -numa node,nodeid=7 -numa node,nodeid=8 -numa node,nodeid=9 \
-device pxb-pcie,id=pcie.1,bus_nr=1,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.0,accel=on,ats=on,ril=off,pasid=on,oas=48 \
-device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1,pref64-reserve=512G,id=dev0 \
-device vfio-pci,host=0019:06:00.0,rombar=0,id=dev0,iommufd=iommufd0,bus=pcie.port1 \
-object acpi-generic-initiator,id=gi0,pci-dev=dev0,node=2 \
...
-object acpi-generic-initiator,id=gi7,pci-dev=dev0,node=9 \
-device pxb-pcie,id=pcie.2,bus_nr=8,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.2,id=smmuv3.1,accel=on,ats=on,ril=off,pasid=on \
-device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=2,pref64-reserve=512G \
-device vfio-pci,host=0018:06:00.0,rombar=0,id=dev1,iommufd=iommufd0,bus=pcie.port2 \
-device virtio-blk-device,drive=fs \
-drive file=image.qcow2,index=0,media=disk,format=qcow2,if=none,id=fs \
-net none \
-nographic

A complete branch can be found here,
https://github.com/shamiali2008/qemu-master master-smmuv3-accel-v6

Please take a look and let me know your feedback.

Thanks,
Shameer

[0] https://lore.kernel.org/qemu-devel/20251022080639.243965-1-skolothumtho@nvidia.com/
[1] https://lore.kernel.org/linux-iommu/20251103172755.2026145-1-nicolinc@nvidia.com/
[2] https://lore.kernel.org/qemu-devel/20251117093729.1121324-1-zhenzhong.duan@intel.com/

Details from RFCv3 Cover letter:
-------------------------------
https://lore.kernel.org/qemu-devel/20250714155941.22176-1-shameerali.kolothum.thodi@huawei.com/

This patch series introduces initial support for a user-creatable,
accelerated SMMUv3 device (-device arm-smmuv3,accel=on) in QEMU.

This is based on the user-creatable SMMUv3 device series [0].

Why this is needed:

On ARM, to enable vfio-pci pass-through devices in a VM, the host SMMUv3
must be set up in nested translation mode (Stage 1 + Stage 2), with
Stage 1 (S1) controlled by the guest and Stage 2 (S2) managed by the host.

This series introduces an optional accel property for the SMMUv3 device,
indicating that the guest will try to leverage host SMMUv3 features for
acceleration. By default, enabling accel configures the host SMMUv3 in
nested mode to support vfio-pci pass-through.

This new accelerated, user-creatable SMMUv3 device lets you:

 -Set up a VM with multiple SMMUv3s, each tied to a different physical SMMUv3
  on the host. Typically, you’d have multiple PCIe PXB root complexes in the
  VM (one per virtual NUMA node), and each of them can have its own SMMUv3.
  This setup mirrors the host's layout, where each NUMA node has its own
  SMMUv3, and helps build VMs that are more aligned with the host's NUMA
  topology.

 -The host–guest SMMUv3 association results in reduced invalidation broadcasts
  and lookups for devices behind different physical SMMUv3s.

 -Simplifies handling of host SMMUv3s with differing feature sets.

 -Lays the groundwork for additional capabilities like vCMDQ support.
-------------------------------

Eric Auger (2):
  hw/pci-host/gpex: Allow to generate preserve boot config DSM #5
  hw/arm/virt-acpi-build: Add IORT RMR regions to handle MSI nested
    binding

Nicolin Chen (4):
  backends/iommufd: Introduce iommufd_backend_alloc_viommu
  backends/iommufd: Introduce iommufd_backend_alloc_vdev
  hw/arm/smmuv3-accel: Add set/unset_iommu_device callback
  hw/arm/smmuv3-accel: Add nested vSTE install/uninstall support

Shameer Kolothum (26):
  hw/arm/smmu-common: Factor out common helper functions and export
  hw/arm/smmu-common: Make iommu ops part of SMMUState
  hw/arm/smmuv3-accel: Introduce smmuv3 accel device
  hw/arm/smmuv3-accel: Initialize shared system address space
  hw/pci/pci: Move pci_init_bus_master() after adding device to bus
  hw/pci/pci: Add optional supports_address_space() callback
  hw/pci-bridge/pci_expander_bridge: Move TYPE_PXB_PCIE_DEV to header
  hw/arm/smmuv3-accel: Restrict accelerated SMMUv3 to vfio-pci endpoints
    with iommufd
  hw/arm/smmuv3: Implement get_viommu_cap() callback
  hw/arm/smmuv3: propagate smmuv3_cmdq_consume() errors to caller
  hw/arm/smmuv3-accel: Install SMMUv3 GBPA based hwpt
  hw/pci/pci: Introduce a callback to retrieve the MSI doorbell GPA
    directly
  hw/arm/smmuv3: Add support for providing a direct MSI doorbell GPA
  hw/arm/smmuv3-accel: Add support to issue invalidation cmd to host
  hw/arm/smmuv3: Initialize ID registers early during realize()
  hw/arm/smmuv3-accel: Get host SMMUv3 hw info and validate
  hw/arm/virt: Set PCI preserve_config for accel SMMUv3
  tests/qtest/bios-tables-test: Prepare for IORT revison upgrade
  tests/qtest/bios-tables-test: Update IORT blobs after revision upgrade
  hw/arm/smmuv3: Add accel property for SMMUv3 device
  hw/arm/smmuv3-accel: Add a property to specify RIL support
  hw/arm/smmuv3-accel: Add support for ATS
  hw/arm/smmuv3-accel: Add property to specify OAS bits
  backends/iommufd: Retrieve PASID width from
    iommufd_backend_get_device_info()
  Extend get_cap() callback to support PASID
  hw/arm/smmuv3-accel: Add support for PASID enable

Yi Liu (1):
  vfio: Synthesize vPASID capability to VM

 backends/iommufd.c                            |  82 +-
 backends/trace-events                         |   2 +
 hw/arm/Kconfig                                |   5 +
 hw/arm/meson.build                            |   3 +-
 hw/arm/smmu-common.c                          |  51 +-
 hw/arm/smmuv3-accel.c                         | 756 ++++++++++++++++++
 hw/arm/smmuv3-accel.h                         |  86 ++
 hw/arm/smmuv3-internal.h                      |  27 +-
 hw/arm/smmuv3.c                               | 206 ++++-
 hw/arm/trace-events                           |   6 +
 hw/arm/virt-acpi-build.c                      | 127 ++-
 hw/arm/virt.c                                 |  30 +
 hw/i386/intel_iommu.c                         |   8 +-
 hw/pci-bridge/pci_expander_bridge.c           |   1 -
 hw/pci-host/gpex-acpi.c                       |  29 +-
 hw/pci/pci.c                                  |  43 +-
 hw/vfio/container-legacy.c                    |   8 +-
 hw/vfio/iommufd.c                             |   7 +-
 hw/vfio/pci.c                                 |  38 +
 include/hw/arm/smmu-common.h                  |   7 +
 include/hw/arm/smmuv3.h                       |  10 +
 include/hw/arm/virt.h                         |   1 +
 include/hw/iommu.h                            |   1 +
 include/hw/pci-host/gpex.h                    |   1 +
 include/hw/pci/pci.h                          |  34 +
 include/hw/pci/pci_bridge.h                   |   1 +
 include/system/host_iommu_device.h            |  21 +-
 include/system/iommufd.h                      |  29 +-
 target/arm/kvm.c                              |  18 +-
 tests/data/acpi/aarch64/virt/IORT             | Bin 128 -> 128 bytes
 tests/data/acpi/aarch64/virt/IORT.its_off     | Bin 172 -> 172 bytes
 tests/data/acpi/aarch64/virt/IORT.smmuv3-dev  | Bin 364 -> 364 bytes
 .../data/acpi/aarch64/virt/IORT.smmuv3-legacy | Bin 276 -> 276 bytes
 33 files changed, 1536 insertions(+), 102 deletions(-)
 create mode 100644 hw/arm/smmuv3-accel.c
 create mode 100644 hw/arm/smmuv3-accel.h

-- 
2.43.0

RE: [PATCH v6 00/33] hw/arm/virt: Add support for user-creatable accelerated SMMUv3

Posted by Duan, Zhenzhong 5 days, 15 hours ago

Hi Shameer,

>-----Original Message-----
>From: Shameer Kolothum <skolothumtho@nvidia.com>
>Subject: [PATCH v6 00/33] hw/arm/virt: Add support for user-creatable
>accelerated SMMUv3
>
>Hi,
>
>Changes from v5:
>
>https://lore.kernel.org/qemu-devel/20251031105005.24618-1-skolothumtho
>@nvidia.com/
>
> - Addressed feedback from v5 and picked up R-by tags. Thanks to all!
> - The previously split out _DSM fix mini-series is now accepted [0].
> - Improved documentation about the rationale behind the design choice of
>   returning an address space aliased to the system address space for
>   vfio-pci endpoint devices (patch #10).
> - Added error propagation support for smmuv3_cmdq_consume() (patch
>#13).
> - Updated vSTE based HWPT installation to check the SMMU enabled case
>   (patch #14).
> - Introduced an optional callback to PCIIOMMUOps to retrieve the MSI
>   doorbell GPA directly, allowing us to avoid unsafe page table walks for
>   MSI translation in accelerated SMMUv3 cases (patch #16).
> - GBPA-based vSTE update depends on Nicolin's kernel patch [1].
> - VFIO/IOMMUFD has dependency on Zhenzhong's patches: 4/5/8 from the
>   pass-through support series [2].
>
>PATCH organization:
> 1–26: Enables accelerated SMMUv3 with features based on default QEMU
>SMMUv3,
>       including IORT RMR based MSI support.
> 27–29: Adds options for specifying RIL, ATS, and OAS features.
> 30–33: Adds PASID support, including VFIO changes.
>
>Tests:
>Performed basic sanity tests on an NVIDIA GRACE platform with GPU device
>assignments. A CUDA test application was used to verify the SVA use case.
>Further tests are always welcome.

I see PASID capability is exposed to guest but no pasid attachment in this series.
Was the nested hwpt attached to SID instead of pasid? How was page fault handled
in stage1?

Thanks
Zhenzhong

RE: [PATCH v6 00/33] hw/arm/virt: Add support for user-creatable accelerated SMMUv3

Posted by Shameer Kolothum 5 days, 14 hours ago


> -----Original Message-----
> From: Duan, Zhenzhong <zhenzhong.duan@intel.com>
> Sent: 08 December 2025 10:08
> To: Shameer Kolothum <skolothumtho@nvidia.com>; qemu-
> arm@nongnu.org; qemu-devel@nongnu.org
> Cc: eric.auger@redhat.com; peter.maydell@linaro.org; Jason Gunthorpe
> <jgg@nvidia.com>; Nicolin Chen <nicolinc@nvidia.com>; ddutile@redhat.com;
> berrange@redhat.com; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
> <mochs@nvidia.com>; smostafa@google.com; wangzhou1@hisilicon.com;
> jiangkunkun@huawei.com; jonathan.cameron@huawei.com;
> zhangfei.gao@linaro.org; Liu, Yi L <yi.l.liu@intel.com>; Krishnakant Jaju
> <kjaju@nvidia.com>
> Subject: RE: [PATCH v6 00/33] hw/arm/virt: Add support for user-creatable
> accelerated SMMUv3
> 
> External email: Use caution opening links or attachments
> 
> 
> Hi Shameer,
> 
> >-----Original Message-----
> >From: Shameer Kolothum <skolothumtho@nvidia.com>
> >Subject: [PATCH v6 00/33] hw/arm/virt: Add support for user-creatable
> >accelerated SMMUv3
> >
> >Hi,
> >
> >Changes from v5:
> >
> >https://lore.kernel.org/qemu-devel/20251031105005.24618-1-
> skolothumtho
> >@nvidia.com/
> >
> > - Addressed feedback from v5 and picked up R-by tags. Thanks to all!
> > - The previously split out _DSM fix mini-series is now accepted [0].
> > - Improved documentation about the rationale behind the design choice of
> >   returning an address space aliased to the system address space for
> >   vfio-pci endpoint devices (patch #10).
> > - Added error propagation support for smmuv3_cmdq_consume() (patch
> >#13).
> > - Updated vSTE based HWPT installation to check the SMMU enabled case
> >   (patch #14).
> > - Introduced an optional callback to PCIIOMMUOps to retrieve the MSI
> >   doorbell GPA directly, allowing us to avoid unsafe page table walks for
> >   MSI translation in accelerated SMMUv3 cases (patch #16).
> > - GBPA-based vSTE update depends on Nicolin's kernel patch [1].
> > - VFIO/IOMMUFD has dependency on Zhenzhong's patches: 4/5/8 from the
> >   pass-through support series [2].
> >
> >PATCH organization:
> > 1–26: Enables accelerated SMMUv3 with features based on default QEMU
> >SMMUv3,
> >       including IORT RMR based MSI support.
> > 27–29: Adds options for specifying RIL, ATS, and OAS features.
> > 30–33: Adds PASID support, including VFIO changes.
> >
> >Tests:
> >Performed basic sanity tests on an NVIDIA GRACE platform with GPU
> >device assignments. A CUDA test application was used to verify the SVA use
> case.
> >Further tests are always welcome.
> 
> I see PASID capability is exposed to guest but no pasid attachment in this
> series.
> Was the nested hwpt attached to SID instead of pasid?

In ARM world there is no specific PASID attachment. ARM uses a Context
Descriptor (CD) table indexed by PASID(substream in ARM) which is owned by
Guest. Hence, no specific PASID attach handling is required in QEMU.

How was page fault
> handled in stage1?

If you meant PRI CAP that is not supported yet.

However, Zhangfei is maintaining a branch to add support for devices that supports
SMMUv3 STALL feature and handles S1 page fault.
https://github.com/Linaro/qemu/commits/master-smmuv3-accel-v6/

Thanks,
Shameer

RE: [PATCH v6 00/33] hw/arm/virt: Add support for user-creatable accelerated SMMUv3

Posted by Duan, Zhenzhong 4 days, 23 hours ago


>-----Original Message-----
>From: Shameer Kolothum <skolothumtho@nvidia.com>
>Subject: RE: [PATCH v6 00/33] hw/arm/virt: Add support for user-creatable
>accelerated SMMUv3
>
>
>
>> -----Original Message-----
>> From: Duan, Zhenzhong <zhenzhong.duan@intel.com>
>> Sent: 08 December 2025 10:08
>> To: Shameer Kolothum <skolothumtho@nvidia.com>; qemu-
>> arm@nongnu.org; qemu-devel@nongnu.org
>> Cc: eric.auger@redhat.com; peter.maydell@linaro.org; Jason Gunthorpe
>> <jgg@nvidia.com>; Nicolin Chen <nicolinc@nvidia.com>;
>ddutile@redhat.com;
>> berrange@redhat.com; Nathan Chen <nathanc@nvidia.com>; Matt Ochs
>> <mochs@nvidia.com>; smostafa@google.com; wangzhou1@hisilicon.com;
>> jiangkunkun@huawei.com; jonathan.cameron@huawei.com;
>> zhangfei.gao@linaro.org; Liu, Yi L <yi.l.liu@intel.com>; Krishnakant Jaju
>> <kjaju@nvidia.com>
>> Subject: RE: [PATCH v6 00/33] hw/arm/virt: Add support for user-creatable
>> accelerated SMMUv3
>>
>> External email: Use caution opening links or attachments
>>
>>
>> Hi Shameer,
>>
>> >-----Original Message-----
>> >From: Shameer Kolothum <skolothumtho@nvidia.com>
>> >Subject: [PATCH v6 00/33] hw/arm/virt: Add support for user-creatable
>> >accelerated SMMUv3
>> >
>> >Hi,
>> >
>> >Changes from v5:
>> >
>> >https://lore.kernel.org/qemu-devel/20251031105005.24618-1-
>> skolothumtho
>> >@nvidia.com/
>> >
>> > - Addressed feedback from v5 and picked up R-by tags. Thanks to all!
>> > - The previously split out _DSM fix mini-series is now accepted [0].
>> > - Improved documentation about the rationale behind the design choice
>of
>> >   returning an address space aliased to the system address space for
>> >   vfio-pci endpoint devices (patch #10).
>> > - Added error propagation support for smmuv3_cmdq_consume() (patch
>> >#13).
>> > - Updated vSTE based HWPT installation to check the SMMU enabled case
>> >   (patch #14).
>> > - Introduced an optional callback to PCIIOMMUOps to retrieve the MSI
>> >   doorbell GPA directly, allowing us to avoid unsafe page table walks for
>> >   MSI translation in accelerated SMMUv3 cases (patch #16).
>> > - GBPA-based vSTE update depends on Nicolin's kernel patch [1].
>> > - VFIO/IOMMUFD has dependency on Zhenzhong's patches: 4/5/8 from
>the
>> >   pass-through support series [2].
>> >
>> >PATCH organization:
>> > 1–26: Enables accelerated SMMUv3 with features based on default QEMU
>> >SMMUv3,
>> >       including IORT RMR based MSI support.
>> > 27–29: Adds options for specifying RIL, ATS, and OAS features.
>> > 30–33: Adds PASID support, including VFIO changes.
>> >
>> >Tests:
>> >Performed basic sanity tests on an NVIDIA GRACE platform with GPU
>> >device assignments. A CUDA test application was used to verify the SVA
>use
>> case.
>> >Further tests are always welcome.
>>
>> I see PASID capability is exposed to guest but no pasid attachment in this
>> series.
>> Was the nested hwpt attached to SID instead of pasid?
>
>In ARM world there is no specific PASID attachment. ARM uses a Context
>Descriptor (CD) table indexed by PASID(substream in ARM) which is owned by
>Guest. Hence, no specific PASID attach handling is required in QEMU.

I just realized a nested hwpt in ARM is “stage2 hwpt + guest CD table” rather than
“stage2 hwpt + a guest s1 hwpt”. When creating nested hwpt, guest S1ContextPtr
is passed to host rather than a stage1 TTB. Do I understand right?

>
>How was page fault
>> handled in stage1?
>
>If you meant PRI CAP that is not supported yet.
>
>However, Zhangfei is maintaining a branch to add support for devices that
>supports
>SMMUv3 STALL feature and handles S1 page fault.
>https://github.com/Linaro/qemu/commits/master-smmuv3-accel-v6/

Thanks for sharing.

BRs,
Zhenzhong