[RFC PATCH 0/8] hw/arm/smmuv3-accel: Support AUTO properties

Nathan Chen posted 8 patches 1 month ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20260309192119.870186-1-nathanc@nvidia.com
Maintainers: Yi Liu <yi.l.liu@intel.com>, Eric Auger <eric.auger@redhat.com>, Zhenzhong Duan <zhenzhong.duan@intel.com>, Peter Maydell <peter.maydell@linaro.org>, "Michael S. Tsirkin" <mst@redhat.com>, Igor Mammedov <imammedo@redhat.com>, Ani Sinha <anisinha@redhat.com>, Shannon Zhao <shannon.zhaosl@gmail.com>, Paolo Bonzini <pbonzini@redhat.com>, "Daniel P. Berrangé" <berrange@redhat.com>, Eduardo Habkost <eduardo@habkost.net>, Alex Williamson <alex@shazbot.org>, "Cédric Le Goater" <clg@redhat.com>, Eric Blake <eblake@redhat.com>, Markus Armbruster <armbru@redhat.com>
There is a newer version of this series
backends/iommufd.c                       |  15 +++
hw/arm/smmuv3-accel.c                    | 118 ++++++++++++++++++++---
hw/arm/smmuv3-accel.h                    |   2 +
hw/arm/smmuv3.c                          |  43 +++++----
hw/arm/virt-acpi-build.c                 |   2 +-
hw/core/qdev-properties-system.c         |  27 ++++++
hw/vfio/pci.c                            |  63 ++++++++++++
hw/vfio/pci.h                            |   1 +
include/hw/arm/smmuv3.h                  |   9 +-
include/hw/core/qdev-properties-system.h |   6 ++
include/system/host_iommu_device.h       |  10 ++
qapi/misc-arm.json                       |  31 ++++++
qapi/pragma.json                         |   1 +
13 files changed, 292 insertions(+), 36 deletions(-)
[RFC PATCH 0/8] hw/arm/smmuv3-accel: Support AUTO properties
Posted by Nathan Chen 1 month ago
Hi,

This series introduces support for specifying 'auto' for arm-smmuv3
accelerated mode's ATS, RIL, SSIDSIZE, and OAS feature properties.
When set to 'auto', these feature values are derived directly from
host IOMMU capabilities, avoiding the need for management layers to
introspect host settings.

Accelerated SMMUv3 Address Translation Services support is derived
from IDR0, Range Invalidation support is derived from IDR3, Substream
ID size is derived from IDR1, and output address space is derived from
IDR5.

Additionally, an OnOffAuto "ats" property is added for vfio-pci devices,
where setting 'auto' detects the per-device presence of
IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED from the kernel, and the ATS cap can
be advertised or hidden by setting 'on' or 'off'. This is dependent
on Shameer's recent kernel series for reporting effective ATS support
status [0].

The default values are set to 'auto' for all properties.

A complete branch can be found here:
https://github.com/NathanChenNVIDIA/qemu/tree/smmuv3-accel-auto

Please take a look and let me know your feedback.

Thanks,
Nathan

Example usage:
qemu-system-aarch64 \
  -object iommufd,id=iommufd0 \
  -machine virt,accel=kvm,gic-version=3,ras=on,highmem-mmio-size=4T \
  -cpu host -smp cpus=4 -m size=16G -nographic \
  -object memory-backend-ram,size=16G,id=m0 \
  -numa node,memdev=m0,cpus=0-3,nodeid=0 \
  -numa node,nodeid=1 -numa node,nodeid=2 -numa node,nodeid=3 -numa node,nodeid=4 \
  -numa node,nodeid=5 -numa node,nodeid=6 -numa node,nodeid=7 -numa node,nodeid=8 \
  -device pxb-pcie,id=pcie.1,bus_nr=1,bus=pcie.0,numa_node=0 \
  -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1,accel=on,ats=auto,ssidsize=auto,ril=auto,oas=auto \
  -device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1,io-reserve=0 \
  -device vfio-pci-nohotplug,host=0009:06:00.0,bus=pcie.port1,rombar=0,id=dev0,iommufd=iommufd0,ats=auto \
  -object acpi-generic-initiator,id=gi0,pci-dev=dev0,node=1 \
  -object acpi-generic-initiator,id=gi1,pci-dev=dev0,node=2 \
  -object acpi-generic-initiator,id=gi2,pci-dev=dev0,node=3 \
  -object acpi-generic-initiator,id=gi3,pci-dev=dev0,node=4 \
  -object acpi-generic-initiator,id=gi4,pci-dev=dev0,node=5 \
  -object acpi-generic-initiator,id=gi5,pci-dev=dev0,node=6 \
  -object acpi-generic-initiator,id=gi6,pci-dev=dev0,node=7 \
  -object acpi-generic-initiator,id=gi7,pci-dev=dev0,node=8 \
  -bios /usr/share/AAVMF/AAVMF_CODE.fd \
  -device nvme,drive=nvme0,serial=deadbeaf1,bus=pcie.0 \
  -drive file=/var/lib/libvirt/images/guest.qcow2,index=0,media=disk,format=qcow2,if=none,id=nvme0 \
  -device e1000,romfile=/usr/local/share/qemu/efi-e1000.rom,netdev=net0,bus=pcie.0 \
  -netdev user,id=net0,hostfwd=tcp::5558-:22,hostfwd=tcp::5586-:5586

The properties may also be omitted as they are set to auto by default:

  -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1,accel=on \
  -device vfio-pci-nohotplug,host=0009:06:00.0,bus=pcie.port1,rombar=0,id=dev0,iommufd=iommufd0 \

Testing:
Basic sanity testing was performed on an NVIDIA Grace platform with GPU
device assignment and running CUDA test apps on the guest. Observed the
feature properties being set based on host IOMMU capabilities and the
ATS capability for a vfio-pci device reported based on what was reported
from the host. Additional testing and feedback are welcome.

[0] https://lore.kernel.org/all/20260303150348.233997-1-skolothumtho@nvidia.com/#r

Nathan Chen (8):
  hw/arm/smmuv3-accel: Add helper for resolving auto parameters
  hw/arm/smmuv3-accel: Introduce _AUTO support for ATS
  vfio/pci: Add ats property and mask ATS cap when not exposed
  hw/arm/smmuv3-accel: Introduce _AUTO support for RIL
  qdev: Add a SsidSizeMode property
  hw/arm/smmuv3-accel: Introduce _AUTO support for SSID size
  qdev: Add an OasMode property
  hw/arm/smmuv3-accel: Introduce _AUTO support for OAS

 backends/iommufd.c                       |  15 +++
 hw/arm/smmuv3-accel.c                    | 118 ++++++++++++++++++++---
 hw/arm/smmuv3-accel.h                    |   2 +
 hw/arm/smmuv3.c                          |  43 +++++----
 hw/arm/virt-acpi-build.c                 |   2 +-
 hw/core/qdev-properties-system.c         |  27 ++++++
 hw/vfio/pci.c                            |  63 ++++++++++++
 hw/vfio/pci.h                            |   1 +
 include/hw/arm/smmuv3.h                  |   9 +-
 include/hw/core/qdev-properties-system.h |   6 ++
 include/system/host_iommu_device.h       |  10 ++
 qapi/misc-arm.json                       |  31 ++++++
 qapi/pragma.json                         |   1 +
 13 files changed, 292 insertions(+), 36 deletions(-)

-- 
2.43.0
Re: [RFC PATCH 0/8] hw/arm/smmuv3-accel: Support AUTO properties
Posted by Eric Auger 1 month ago

On 3/9/26 8:21 PM, Nathan Chen wrote:
> Hi,
>
> This series introduces support for specifying 'auto' for arm-smmuv3
> accelerated mode's ATS, RIL, SSIDSIZE, and OAS feature properties.
> When set to 'auto', these feature values are derived directly from
> host IOMMU capabilities, avoiding the need for management layers to
> introspect host settings.
>
> Accelerated SMMUv3 Address Translation Services support is derived
> from IDR0, Range Invalidation support is derived from IDR3, Substream
> ID size is derived from IDR1, and output address space is derived from
> IDR5.
>
> Additionally, an OnOffAuto "ats" property is added for vfio-pci devices,
> where setting 'auto' detects the per-device presence of
> IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED from the kernel, and the ATS cap can
> be advertised or hidden by setting 'on' or 'off'. This is dependent
> on Shameer's recent kernel series for reporting effective ATS support
> status [0].
>
> The default values are set to 'auto' for all properties.
>
> A complete branch can be found here:
> https://github.com/NathanChenNVIDIA/qemu/tree/smmuv3-accel-auto

I have just noticed we are missing documentation for smmuv3 accel options in

qemu-options.hx

At the moment we just have:

``-device arm-smmuv3,primary-bus=id``
    This is only supported by ``-machine virt`` (ARM).

    ``primary-bus=id``
        Accepts either the default root complex (pcie.0) or a
        pxb-pcie based root complex.

Eric 
>
> Please take a look and let me know your feedback.
>
> Thanks,
> Nathan
>
> Example usage:
> qemu-system-aarch64 \
>   -object iommufd,id=iommufd0 \
>   -machine virt,accel=kvm,gic-version=3,ras=on,highmem-mmio-size=4T \
>   -cpu host -smp cpus=4 -m size=16G -nographic \
>   -object memory-backend-ram,size=16G,id=m0 \
>   -numa node,memdev=m0,cpus=0-3,nodeid=0 \
>   -numa node,nodeid=1 -numa node,nodeid=2 -numa node,nodeid=3 -numa node,nodeid=4 \
>   -numa node,nodeid=5 -numa node,nodeid=6 -numa node,nodeid=7 -numa node,nodeid=8 \
>   -device pxb-pcie,id=pcie.1,bus_nr=1,bus=pcie.0,numa_node=0 \
>   -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1,accel=on,ats=auto,ssidsize=auto,ril=auto,oas=auto \
>   -device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1,io-reserve=0 \
>   -device vfio-pci-nohotplug,host=0009:06:00.0,bus=pcie.port1,rombar=0,id=dev0,iommufd=iommufd0,ats=auto \
>   -object acpi-generic-initiator,id=gi0,pci-dev=dev0,node=1 \
>   -object acpi-generic-initiator,id=gi1,pci-dev=dev0,node=2 \
>   -object acpi-generic-initiator,id=gi2,pci-dev=dev0,node=3 \
>   -object acpi-generic-initiator,id=gi3,pci-dev=dev0,node=4 \
>   -object acpi-generic-initiator,id=gi4,pci-dev=dev0,node=5 \
>   -object acpi-generic-initiator,id=gi5,pci-dev=dev0,node=6 \
>   -object acpi-generic-initiator,id=gi6,pci-dev=dev0,node=7 \
>   -object acpi-generic-initiator,id=gi7,pci-dev=dev0,node=8 \
>   -bios /usr/share/AAVMF/AAVMF_CODE.fd \
>   -device nvme,drive=nvme0,serial=deadbeaf1,bus=pcie.0 \
>   -drive file=/var/lib/libvirt/images/guest.qcow2,index=0,media=disk,format=qcow2,if=none,id=nvme0 \
>   -device e1000,romfile=/usr/local/share/qemu/efi-e1000.rom,netdev=net0,bus=pcie.0 \
>   -netdev user,id=net0,hostfwd=tcp::5558-:22,hostfwd=tcp::5586-:5586
>
> The properties may also be omitted as they are set to auto by default:
>
>   -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1,accel=on \
>   -device vfio-pci-nohotplug,host=0009:06:00.0,bus=pcie.port1,rombar=0,id=dev0,iommufd=iommufd0 \
>
> Testing:
> Basic sanity testing was performed on an NVIDIA Grace platform with GPU
> device assignment and running CUDA test apps on the guest. Observed the
> feature properties being set based on host IOMMU capabilities and the
> ATS capability for a vfio-pci device reported based on what was reported
> from the host. Additional testing and feedback are welcome.
>
> [0] https://lore.kernel.org/all/20260303150348.233997-1-skolothumtho@nvidia.com/#r
>
> Nathan Chen (8):
>   hw/arm/smmuv3-accel: Add helper for resolving auto parameters
>   hw/arm/smmuv3-accel: Introduce _AUTO support for ATS
>   vfio/pci: Add ats property and mask ATS cap when not exposed
>   hw/arm/smmuv3-accel: Introduce _AUTO support for RIL
>   qdev: Add a SsidSizeMode property
>   hw/arm/smmuv3-accel: Introduce _AUTO support for SSID size
>   qdev: Add an OasMode property
>   hw/arm/smmuv3-accel: Introduce _AUTO support for OAS
>
>  backends/iommufd.c                       |  15 +++
>  hw/arm/smmuv3-accel.c                    | 118 ++++++++++++++++++++---
>  hw/arm/smmuv3-accel.h                    |   2 +
>  hw/arm/smmuv3.c                          |  43 +++++----
>  hw/arm/virt-acpi-build.c                 |   2 +-
>  hw/core/qdev-properties-system.c         |  27 ++++++
>  hw/vfio/pci.c                            |  63 ++++++++++++
>  hw/vfio/pci.h                            |   1 +
>  include/hw/arm/smmuv3.h                  |   9 +-
>  include/hw/core/qdev-properties-system.h |   6 ++
>  include/system/host_iommu_device.h       |  10 ++
>  qapi/misc-arm.json                       |  31 ++++++
>  qapi/pragma.json                         |   1 +
>  13 files changed, 292 insertions(+), 36 deletions(-)
>


Re: [RFC PATCH 0/8] hw/arm/smmuv3-accel: Support AUTO properties
Posted by Nathan Chen 1 month ago

On 3/11/2026 10:43 AM, Eric Auger wrote:
> 
> On 3/9/26 8:21 PM, Nathan Chen wrote:
>> Hi,
>>
>> This series introduces support for specifying 'auto' for arm-smmuv3
>> accelerated mode's ATS, RIL, SSIDSIZE, and OAS feature properties.
>> When set to 'auto', these feature values are derived directly from
>> host IOMMU capabilities, avoiding the need for management layers to
>> introspect host settings.
>>
>> Accelerated SMMUv3 Address Translation Services support is derived
>> from IDR0, Range Invalidation support is derived from IDR3, Substream
>> ID size is derived from IDR1, and output address space is derived from
>> IDR5.
>>
>> Additionally, an OnOffAuto "ats" property is added for vfio-pci devices,
>> where setting 'auto' detects the per-device presence of
>> IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED from the kernel, and the ATS cap can
>> be advertised or hidden by setting 'on' or 'off'. This is dependent
>> on Shameer's recent kernel series for reporting effective ATS support
>> status [0].
>>
>> The default values are set to 'auto' for all properties.
>>
>> A complete branch can be found here:
>> https://github.com/NathanChenNVIDIA/qemu/tree/smmuv3-accel-auto
> I have just noticed we are missing documentation for smmuv3 accel options in
> 
> qemu-options.hx
> 
> At the moment we just have:
> 
> ``-device arm-smmuv3,primary-bus=id``
>      This is only supported by ``-machine virt`` (ARM).
> 
>      ``primary-bus=id``
>          Accepts either the default root complex (pcie.0) or a
>          pxb-pcie based root complex.

I will add a commit to include documentation for the smmuv3 accel 
options in the next refresh.

Thanks,
Nathan
Re: [RFC PATCH 0/8] hw/arm/smmuv3-accel: Support AUTO properties
Posted by Eric Auger 1 month ago

On 3/11/26 6:55 PM, Nathan Chen wrote:
>
>
> On 3/11/2026 10:43 AM, Eric Auger wrote:
>>
>> On 3/9/26 8:21 PM, Nathan Chen wrote:
>>> Hi,
>>>
>>> This series introduces support for specifying 'auto' for arm-smmuv3
>>> accelerated mode's ATS, RIL, SSIDSIZE, and OAS feature properties.
>>> When set to 'auto', these feature values are derived directly from
>>> host IOMMU capabilities, avoiding the need for management layers to
>>> introspect host settings.
>>>
>>> Accelerated SMMUv3 Address Translation Services support is derived
>>> from IDR0, Range Invalidation support is derived from IDR3, Substream
>>> ID size is derived from IDR1, and output address space is derived from
>>> IDR5.
>>>
>>> Additionally, an OnOffAuto "ats" property is added for vfio-pci
>>> devices,
>>> where setting 'auto' detects the per-device presence of
>>> IOMMU_HW_CAP_PCI_ATS_NOT_SUPPORTED from the kernel, and the ATS cap can
>>> be advertised or hidden by setting 'on' or 'off'. This is dependent
>>> on Shameer's recent kernel series for reporting effective ATS support
>>> status [0].
>>>
>>> The default values are set to 'auto' for all properties.
>>>
>>> A complete branch can be found here:
>>> https://github.com/NathanChenNVIDIA/qemu/tree/smmuv3-accel-auto
>> I have just noticed we are missing documentation for smmuv3 accel
>> options in
>>
>> qemu-options.hx
>>
>> At the moment we just have:
>>
>> ``-device arm-smmuv3,primary-bus=id``
>>      This is only supported by ``-machine virt`` (ARM).
>>
>>      ``primary-bus=id``
>>          Accepts either the default root complex (pcie.0) or a
>>          pxb-pcie based root complex.
>
> I will add a commit to include documentation for the smmuv3 accel
> options in the next refresh. 

Great. Thanks!

>
> Thanks,
> Nathan
>