Hi All,

Changes from v4:
https://lore.kernel.org/qemu-devel/20250613144449.60156-1-shameerali.kolothum.thodi@huawei.com/

Major changes from v4:

1. Added stricter validation for PCI buses associated with the SMMU.
   Only the default PCIe Root Complex (pcie.0) and additional root
   complexes created using pxb-pcie (see patch #1) are allowed.

2. While testing this series with a setup involving multiple PCIe root
   complexes using pxb-pcie, I encountered an issue related to IOMMU
   ops resolution. Consider the below configuration, where an
   arm-smmuv3 device is associated with the default root complex pcie.0,
   and an additional pxb-pcie-based root complex (pcie.1) is added
   without any associated SMMU:

   -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.1 \
   ...
   -device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \
   -device pcie-root-port,id=pcie.port1,chassis=2,bus=pcie.1 \
   -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.2 \
   ...
   -device virtio-net-pci,bus=pcie.0,netdev=net0,id=virtionet.0 \
   -device virtio-net-pci,bus=pcie.port1,netdev=net1,id=virtionet.1

   The guest boots fine, and virtionet.0(behind the SMMUV3) bring up
   is successful. However, attempting to bring up virtionet.1
   (behind pcie.1, which lacks a connected SMMU) results in a failure:

   root@ubuntu:/# dhclient enp9s0
   arm-smmu-v3 arm-smmu-v3.0.auto: event 0x02 received:
   arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000090000000002
   arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000
   arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000
   arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000
   arm-smmu-v3 arm-smmu-v3.0.auto: event: C_BAD_STREAMID client: (unassigned sid) sid: 0x900 ssid: 0x0
   virtio_net virtio1 enp9s0: NETDEV WATCHDOG: CPU: 2: transmit queue 0 timed out 5172 ms
   virtio_net virtio1 enp9s0: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 5172000 usecs ago
   ...

   Debug shows that QEMU currently registers IOMMU ops for bus using
   pci_setup_iommu(). However, when retrieving IOMMU ops for a device
   via pci_device_get_iommu_bus_devfn(), the function walks up to the 
   parent_dev and fetches the IOMMU ops from the parent, even if the
   current root bus has none configured.

   This works today because existing IOMMU models in QEMU are globally 
   scoped, and pxb-pcie based extra root complexes can use the
   bypass_iommu property to skip translation as needed.

   However, with this series introducing support for associating
   arm-smmuv3 devices with specific PCIe root complexes, this
   becomes problematic. In QEMU, pxb-pcie is implemented as a synthetic
   root complex whose parent_dev is pcie.0. As a result, even though
   pcie.1 has no SMMU attached, pci_device_get_iommu_bus_devfn()
   incorrectly returns the IOMMU ops associated with pcie.0 due to
   the fallback mechanism via parent_dev. This causes devices on
   pcie.1 to erroneously use the address space from pcie.0's SMMU,
   leading to failures like the one above.

   To address this, patch #6 in the series introduces a new helper 
   function pci_setup_iommu_per_bus(), which explicitly sets the 
   iommu_per_bus field in the PCIBus structure. This allows 
   pci_device_get_iommu_bus_devfn() to retrieve IOMMU ops based 
   on the specific bus.

   Not sure this is the correct approach or not. If there is a better
   way to handle this, please let me know .

3. Picked up few R-by tags where the patch content has not changed much.

4. Dropped T-by from Nathan for some patches as things have changed a bit.
   @Nathan, apprecaite if you have time to rerun the tests.

5. Added a bios table tests for both legacy SMMUv3 and new SMMMv3 devices.
   See last few patches.

Cover letter:

This patch series introduces support for a user-creatable SMMUv3 device
(-device arm-smmuv3) in QEMU.

The implementation is based on feedback received from the RFCv2[0]:
"hw/arm/virt: Add support for user-creatable accelerated SMMUv3"

Currently, QEMU's SMMUv3 emulation (iommu=smmuv3) is tied to the machine
and does not support instantiating multiple SMMUv3 devices—each associated
with a separate PCIe root complex. In contrast, real-world ARM systems
often include multiple SMMUv3 instances, each bound to a different PCIe
root complex.

This series allows to specify multiple SMMUv3 instances as below,

 -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0
  ...
 -device arm-smmuv3,primary-bus=pcie.1,,id=smmuv3.1

The multiple SMMUv3 instance support lays the groundwork for supporting
accelerated SMMUv3, as proposed in the aforementioned RFCv2[0]. The
proposed accelerated support will be an optional property like below,
-device arm-smmuv3,primary-bus=pcie.1,accel=on,..

Please note, the accelerated SMMUv3 support is not part of this series
and will be sent out as a separate series later on top of this one.

This series also,

-Supports either the legacy iommu=smmuv3 option or the new
  "-device arm-smmuv3" model.
  -Adds device tree bindings for the new SMMUv3 device on the arm/virt
   machine only, and only for the default pcie.0 root complex.
   (Note: pxb-pcie root complexes are currently not supported with the
    device tree due to known limitations[1].)

Example usage:
  -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0
  -device virtio-net-pci,bus=pcie.0
  -device pxb-pcie,id=pcie.1,bus_nr=2
  -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1
  -device pcie-root-port,id=pcie.port1,bus=pcie.1
  -device virtio-net-pci,bus=pcie.port1

Please take a look and let me know your feedback.

Thanks,
Shameer
[0]:https://lore.kernel.org/qemu-devel/20250311141045.66620-1-shameerali.kolothum.thodi@huawei.com/
[1]:https://lore.kernel.org/qemu-devel/20230421165037.2506-1-Jonathan.Cameron@huawei.com/

Nicolin Chen (1):
  hw/arm/virt: Add an SMMU_IO_LEN macro

Shameer Kolothum (10):
  hw/arm/smmu-common: Check SMMU has PCIe Root Complex association
  hw/arm/virt-acpi-build: Re-arrange SMMUv3 IORT build
  hw/arm/virt-acpi-build: Update IORT for multiple smmuv3 devices
  hw/arm/virt: Factor out common SMMUV3 dt bindings code
  hw/pci: Introduce pci_setup_iommu_per_bus() for per-bus IOMMU ops
    retrieval
  hw/arm/virt: Allow user-creatable SMMUv3 dev instantiation
  qemu-options.hx: Document the arm-smmuv3 device
  bios-tables-test: Allow for smmuv3 test data.
  qtest/bios-tables-test: Add tests for legacy smmuv3 and smmuv3 device
  qtest/bios-tables-test: Update tables for smmuv3 tests

 hw/arm/smmu-common.c                          |  35 +++-
 hw/arm/smmuv3.c                               |   2 +
 hw/arm/virt-acpi-build.c                      | 164 ++++++++++++++----
 hw/arm/virt.c                                 | 110 +++++++++---
 hw/core/sysbus-fdt.c                          |   3 +
 hw/pci-bridge/pci_expander_bridge.c           |   1 -
 hw/pci/pci.c                                  |  25 +++
 include/hw/arm/smmu-common.h                  |   1 +
 include/hw/arm/virt.h                         |   1 +
 include/hw/pci/pci.h                          |   2 +
 include/hw/pci/pci_bridge.h                   |   1 +
 include/hw/pci/pci_bus.h                      |   1 +
 qemu-options.hx                               |   7 +
 tests/data/acpi/aarch64/virt/DSDT.smmuv3-dev  | Bin 0 -> 10162 bytes
 .../data/acpi/aarch64/virt/DSDT.smmuv3-legacy | Bin 0 -> 10162 bytes
 tests/data/acpi/aarch64/virt/IORT.smmuv3-dev  | Bin 0 -> 364 bytes
 .../data/acpi/aarch64/virt/IORT.smmuv3-legacy | Bin 0 -> 276 bytes
 tests/qtest/bios-tables-test.c                |  86 +++++++++
 18 files changed, 375 insertions(+), 64 deletions(-)
 create mode 100644 tests/data/acpi/aarch64/virt/DSDT.smmuv3-dev
 create mode 100644 tests/data/acpi/aarch64/virt/DSDT.smmuv3-legacy
 create mode 100644 tests/data/acpi/aarch64/virt/IORT.smmuv3-dev
 create mode 100644 tests/data/acpi/aarch64/virt/IORT.smmuv3-legacy

-- 
2.34.1

Hi Shameer,

On 6/23/25 11:42 AM, Shameer Kolothum wrote:
> Hi All,
>
> Changes from v4:
> https://lore.kernel.org/qemu-devel/20250613144449.60156-1-shameerali.kolothum.thodi@huawei.com/
>
> Major changes from v4:
this will need a respin after merge of
f5ec751ee70d  hw/arm/virt: Check bypass iommu is not set for iommu-map
DT property

Thanks

Eric
>
> 1. Added stricter validation for PCI buses associated with the SMMU.
>    Only the default PCIe Root Complex (pcie.0) and additional root
>    complexes created using pxb-pcie (see patch #1) are allowed.
>
> 2. While testing this series with a setup involving multiple PCIe root
>    complexes using pxb-pcie, I encountered an issue related to IOMMU
>    ops resolution. Consider the below configuration, where an
>    arm-smmuv3 device is associated with the default root complex pcie.0,
>    and an additional pxb-pcie-based root complex (pcie.1) is added
>    without any associated SMMU:
>
>    -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.1 \
>    ...
>    -device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \
>    -device pcie-root-port,id=pcie.port1,chassis=2,bus=pcie.1 \
>    -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.2 \
>    ...
>    -device virtio-net-pci,bus=pcie.0,netdev=net0,id=virtionet.0 \
>    -device virtio-net-pci,bus=pcie.port1,netdev=net1,id=virtionet.1
>
>    The guest boots fine, and virtionet.0(behind the SMMUV3) bring up
>    is successful. However, attempting to bring up virtionet.1
>    (behind pcie.1, which lacks a connected SMMU) results in a failure:
>
>    root@ubuntu:/# dhclient enp9s0
>    arm-smmu-v3 arm-smmu-v3.0.auto: event 0x02 received:
>    arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000090000000002
>    arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000
>    arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000
>    arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000
>    arm-smmu-v3 arm-smmu-v3.0.auto: event: C_BAD_STREAMID client: (unassigned sid) sid: 0x900 ssid: 0x0
>    virtio_net virtio1 enp9s0: NETDEV WATCHDOG: CPU: 2: transmit queue 0 timed out 5172 ms
>    virtio_net virtio1 enp9s0: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 5172000 usecs ago
>    ...
>
>    Debug shows that QEMU currently registers IOMMU ops for bus using
>    pci_setup_iommu(). However, when retrieving IOMMU ops for a device
>    via pci_device_get_iommu_bus_devfn(), the function walks up to the 
>    parent_dev and fetches the IOMMU ops from the parent, even if the
>    current root bus has none configured.
>
>    This works today because existing IOMMU models in QEMU are globally 
>    scoped, and pxb-pcie based extra root complexes can use the
>    bypass_iommu property to skip translation as needed.
>
>    However, with this series introducing support for associating
>    arm-smmuv3 devices with specific PCIe root complexes, this
>    becomes problematic. In QEMU, pxb-pcie is implemented as a synthetic
>    root complex whose parent_dev is pcie.0. As a result, even though
>    pcie.1 has no SMMU attached, pci_device_get_iommu_bus_devfn()
>    incorrectly returns the IOMMU ops associated with pcie.0 due to
>    the fallback mechanism via parent_dev. This causes devices on
>    pcie.1 to erroneously use the address space from pcie.0's SMMU,
>    leading to failures like the one above.
>
>    To address this, patch #6 in the series introduces a new helper 
>    function pci_setup_iommu_per_bus(), which explicitly sets the 
>    iommu_per_bus field in the PCIBus structure. This allows 
>    pci_device_get_iommu_bus_devfn() to retrieve IOMMU ops based 
>    on the specific bus.
>
>    Not sure this is the correct approach or not. If there is a better
>    way to handle this, please let me know .
>
> 3. Picked up few R-by tags where the patch content has not changed much.
>
> 4. Dropped T-by from Nathan for some patches as things have changed a bit.
>    @Nathan, apprecaite if you have time to rerun the tests.
>
> 5. Added a bios table tests for both legacy SMMUv3 and new SMMMv3 devices.
>    See last few patches.
>
> Cover letter:
>
> This patch series introduces support for a user-creatable SMMUv3 device
> (-device arm-smmuv3) in QEMU.
>
> The implementation is based on feedback received from the RFCv2[0]:
> "hw/arm/virt: Add support for user-creatable accelerated SMMUv3"
>
> Currently, QEMU's SMMUv3 emulation (iommu=smmuv3) is tied to the machine
> and does not support instantiating multiple SMMUv3 devices—each associated
> with a separate PCIe root complex. In contrast, real-world ARM systems
> often include multiple SMMUv3 instances, each bound to a different PCIe
> root complex.
>
> This series allows to specify multiple SMMUv3 instances as below,
>
>  -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0
>   ...
>  -device arm-smmuv3,primary-bus=pcie.1,,id=smmuv3.1
>
> The multiple SMMUv3 instance support lays the groundwork for supporting
> accelerated SMMUv3, as proposed in the aforementioned RFCv2[0]. The
> proposed accelerated support will be an optional property like below,
> -device arm-smmuv3,primary-bus=pcie.1,accel=on,..
>
> Please note, the accelerated SMMUv3 support is not part of this series
> and will be sent out as a separate series later on top of this one.
>
> This series also,
>
> -Supports either the legacy iommu=smmuv3 option or the new
>   "-device arm-smmuv3" model.
>   -Adds device tree bindings for the new SMMUv3 device on the arm/virt
>    machine only, and only for the default pcie.0 root complex.
>    (Note: pxb-pcie root complexes are currently not supported with the
>     device tree due to known limitations[1].)
>
> Example usage:
>   -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0
>   -device virtio-net-pci,bus=pcie.0
>   -device pxb-pcie,id=pcie.1,bus_nr=2
>   -device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1
>   -device pcie-root-port,id=pcie.port1,bus=pcie.1
>   -device virtio-net-pci,bus=pcie.port1
>
> Please take a look and let me know your feedback.
>
> Thanks,
> Shameer
> [0]:https://lore.kernel.org/qemu-devel/20250311141045.66620-1-shameerali.kolothum.thodi@huawei.com/
> [1]:https://lore.kernel.org/qemu-devel/20230421165037.2506-1-Jonathan.Cameron@huawei.com/
>
> Nicolin Chen (1):
>   hw/arm/virt: Add an SMMU_IO_LEN macro
>
> Shameer Kolothum (10):
>   hw/arm/smmu-common: Check SMMU has PCIe Root Complex association
>   hw/arm/virt-acpi-build: Re-arrange SMMUv3 IORT build
>   hw/arm/virt-acpi-build: Update IORT for multiple smmuv3 devices
>   hw/arm/virt: Factor out common SMMUV3 dt bindings code
>   hw/pci: Introduce pci_setup_iommu_per_bus() for per-bus IOMMU ops
>     retrieval
>   hw/arm/virt: Allow user-creatable SMMUv3 dev instantiation
>   qemu-options.hx: Document the arm-smmuv3 device
>   bios-tables-test: Allow for smmuv3 test data.
>   qtest/bios-tables-test: Add tests for legacy smmuv3 and smmuv3 device
>   qtest/bios-tables-test: Update tables for smmuv3 tests
>
>  hw/arm/smmu-common.c                          |  35 +++-
>  hw/arm/smmuv3.c                               |   2 +
>  hw/arm/virt-acpi-build.c                      | 164 ++++++++++++++----
>  hw/arm/virt.c                                 | 110 +++++++++---
>  hw/core/sysbus-fdt.c                          |   3 +
>  hw/pci-bridge/pci_expander_bridge.c           |   1 -
>  hw/pci/pci.c                                  |  25 +++
>  include/hw/arm/smmu-common.h                  |   1 +
>  include/hw/arm/virt.h                         |   1 +
>  include/hw/pci/pci.h                          |   2 +
>  include/hw/pci/pci_bridge.h                   |   1 +
>  include/hw/pci/pci_bus.h                      |   1 +
>  qemu-options.hx                               |   7 +
>  tests/data/acpi/aarch64/virt/DSDT.smmuv3-dev  | Bin 0 -> 10162 bytes
>  .../data/acpi/aarch64/virt/DSDT.smmuv3-legacy | Bin 0 -> 10162 bytes
>  tests/data/acpi/aarch64/virt/IORT.smmuv3-dev  | Bin 0 -> 364 bytes
>  .../data/acpi/aarch64/virt/IORT.smmuv3-legacy | Bin 0 -> 276 bytes
>  tests/qtest/bios-tables-test.c                |  86 +++++++++
>  18 files changed, 375 insertions(+), 64 deletions(-)
>  create mode 100644 tests/data/acpi/aarch64/virt/DSDT.smmuv3-dev
>  create mode 100644 tests/data/acpi/aarch64/virt/DSDT.smmuv3-legacy
>  create mode 100644 tests/data/acpi/aarch64/virt/IORT.smmuv3-dev
>  create mode 100644 tests/data/acpi/aarch64/virt/IORT.smmuv3-legacy
>

> -----Original Message-----
> From: Eric Auger <eric.auger@redhat.com>
> Sent: Friday, June 27, 2025 1:37 PM
> To: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> qemu-devel@nongnu.org
> Cc: peter.maydell@linaro.org; jgg@nvidia.com; nicolinc@nvidia.com;
> ddutile@redhat.com; berrange@redhat.com; imammedo@redhat.com;
> nathanc@nvidia.com; mochs@nvidia.com; smostafa@google.com;
> gustavo.romero@linaro.org; Linuxarm <linuxarm@huawei.com>; Wangzhou
> (B) <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>;
> zhangfei.gao@linaro.org
> Subject: Re: [PATCH v5 00/11] hw/arm/virt: Add support for user creatable
> SMMUv3 device
> 
> Hi Shameer,
> 
> On 6/23/25 11:42 AM, Shameer Kolothum wrote:
> > Hi All,
> >
> > Changes from v4:
> > https://lore.kernel.org/qemu-devel/20250613144449.60156-1-
> shameerali.kolothum.thodi@huawei.com/
> >
> > Major changes from v4:
> this will need a respin after merge of
> f5ec751ee70d  hw/arm/virt: Check bypass iommu is not set for iommu-map
> DT property

Yes. Missed that. Will do.

Thanks,
Shameer

On 6/30/25 9:12 AM, Shameerali Kolothum Thodi wrote:
>
>> -----Original Message-----
>> From: Eric Auger <eric.auger@redhat.com>
>> Sent: Friday, June 27, 2025 1:37 PM
>> To: Shameerali Kolothum Thodi
>> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
>> qemu-devel@nongnu.org
>> Cc: peter.maydell@linaro.org; jgg@nvidia.com; nicolinc@nvidia.com;
>> ddutile@redhat.com; berrange@redhat.com; imammedo@redhat.com;
>> nathanc@nvidia.com; mochs@nvidia.com; smostafa@google.com;
>> gustavo.romero@linaro.org; Linuxarm <linuxarm@huawei.com>; Wangzhou
>> (B) <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
>> Jonathan Cameron <jonathan.cameron@huawei.com>;
>> zhangfei.gao@linaro.org
>> Subject: Re: [PATCH v5 00/11] hw/arm/virt: Add support for user creatable
>> SMMUv3 device
>>
>> Hi Shameer,
>>
>> On 6/23/25 11:42 AM, Shameer Kolothum wrote:
>>> Hi All,
>>>
>>> Changes from v4:
>>> https://lore.kernel.org/qemu-devel/20250613144449.60156-1-
>> shameerali.kolothum.thodi@huawei.com/
>>> Major changes from v4:
>> this will need a respin after merge of
>> f5ec751ee70d  hw/arm/virt: Check bypass iommu is not set for iommu-map
>> DT property
you may have seen Gustavo's series
[PATCH-for-10.1 v6 0/9] hw/arm: GIC 'its=off' ACPI table fixes
has been applied to target-arm.next. You shall rebase on top of it and
unfortunately it should conflict with some part of your series

Eric
> Yes. Missed that. Will do.
>
> Thanks,
> Shameer
>

>    To address this, patch #6 in the series introduces a new helper 
>    function pci_setup_iommu_per_bus(), which explicitly sets the 
>    iommu_per_bus field in the PCIBus structure. This allows 
>    pci_device_get_iommu_bus_devfn() to retrieve IOMMU ops based 
>    on the specific bus.
>
>    This patch series introduces support for a user-creatable SMMUv3 device
>    (-device arm-smmuv3) in QEMU.

Tested-by: Nathan Chen <nathanc@nvidia.com>

I re-ran the test from v3 [0] and am able to create 16 SMMUv3 devices in 
a qemu VM with emulated devices properly associated with the guest SMMUs 
in guest sysfs - verified with some guest SMMUs having two or three 
emulated NICs assigned to them while other guest SMMUs have a minimum of 
one assigned.

Removing SMMUv3 devices from the VM config described above, I do not 
observe the problematic behavior where devices behind PXBs without SMMUs 
erroneously use the address space from pcie.0's SMMU. I removed SMMUv3 
devices from PXBs with one, two, and three emulated NICs assigned to 
them. Below are the guest topology and qemu command used where SMMUv3 
devices are excluded from the original test:

nvidia@lego-cg1-dvt-59:~$ lspci -tv
-+-[0000:00]-+-00.0  Red Hat, Inc. QEMU PCIe Host bridge
  |           +-01.0  Red Hat, Inc. QEMU NVM Express Controller
  |           +-02.0  Intel Corporation 82540EM Gigabit Ethernet Controller
  |           +-03.0  Red Hat, Inc. Virtio network device
  |           +-04.0  Red Hat, Inc. Virtio network device
  |           +-05.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-06.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-07.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-08.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-09.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0a.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0b.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0c.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0d.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0e.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-0f.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-10.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-11.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           +-12.0  Red Hat, Inc. QEMU PCIe Expander bridge
  |           \-13.0  Red Hat, Inc. QEMU PCIe Expander bridge
  +-[0000:90]-+-00.0-[91]----00.0  Red Hat, Inc. Virtio 1.0 network device
  |           \-01.0-[92]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:94]---00.0-[95]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:98]---00.0-[99]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:9c]---00.0-[9d]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:a0]---00.0-[a1]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:a4]---00.0-[a5]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:a8]---00.0-[a9]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:ac]---00.0-[ad]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:b0]---00.0-[b1]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:b4]---00.0-[b5]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:b8]---00.0-[b9]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:bc]---00.0-[bd]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:c0]---00.0-[c1]----00.0  Red Hat, Inc. Virtio 1.0 network device
  +-[0000:c4]---00.0-[c5]----00.0  Red Hat, Inc. Virtio 1.0 network device
  \-[0000:c8]-+-00.0-[c9]----00.0  Red Hat, Inc. Virtio 1.0 network device
              +-01.0-[ca]----00.0  Red Hat, Inc. Virtio 1.0 network device
              \-02.0-[cb]----00.0  Red Hat, Inc. Virtio 1.0 network device

qemu-system-aarch64 \
    -machine hmat=on -machine virt,accel=kvm,gic-version=3,ras=on \
         -cpu host -smp cpus=4 -m size=16G,slots=4,maxmem=32G -nographic \
         -bios /usr/share/AAVMF/AAVMF_CODE.fd \
         -device nvme,drive=nvme0,serial=deadbeaf1,bus=pcie.0 \
         -drive 
file=/localhome/local-nathanc/noble-server-cloudimg-arm64.qcow2,index=0,media=disk,format=qcow2,if=none,id=nvme0 
\
         -device 
e1000,romfile=/localhome/local-nathanc/efi-e1000.rom,netdev=net0,bus=pcie.0 
\
         -netdev 
user,id=net0,hostfwd=tcp::5558-:22,hostfwd=tcp::5586-:5586 \
         -netdev user,id=net1 \
         -netdev user,id=net2 \
         -netdev user,id=net3 \
         -netdev user,id=net4 \
         -netdev user,id=net5 \
         -netdev user,id=net6 \
         -netdev user,id=net7 \
         -netdev user,id=net8 \
         -netdev user,id=net9 \
         -netdev user,id=net10 \
         -netdev user,id=net11 \
         -netdev user,id=net12 \
         -netdev user,id=net13 \
         -netdev user,id=net14 \
         -netdev user,id=net15 \
         -netdev user,id=net16 \
         -netdev user,id=net17 \
         -netdev user,id=net18 \
         -netdev user,id=net19 \
         -netdev user,id=net20 \
         -device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0 \
         -device virtio-net-pci,bus=pcie.0,netdev=net1 \
         -device virtio-net-pci,bus=pcie.0,netdev=net20 \
         -device pxb-pcie,id=pcie.1,bus_nr=200,bus=pcie.0 \
         -device pcie-root-port,id=pcie.port1,bus=pcie.1,slot=1,chassis=1 \
         -device virtio-net-pci,bus=pcie.port1,netdev=net2 \
         -device 
pcie-root-port,id=pcie.port17,bus=pcie.1,slot=17,chassis=17 \
         -device virtio-net-pci,bus=pcie.port17,netdev=net18 \
         -device 
pcie-root-port,id=pcie.port18,bus=pcie.1,slot=18,chassis=18 \
         -device virtio-net-pci,bus=pcie.port18,netdev=net19 \
         -device pxb-pcie,id=pcie.2,bus_nr=196,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.2,id=smmuv3.2 \
         -device pcie-root-port,id=pcie.port2,bus=pcie.2,slot=2,chassis=2 \
         -device virtio-net-pci,bus=pcie.port2,netdev=net3 \
         -device pxb-pcie,id=pcie.3,bus_nr=192,bus=pcie.0 \
         -device pcie-root-port,id=pcie.port3,bus=pcie.3,slot=3,chassis=3 \
         -device virtio-net-pci,bus=pcie.port3,netdev=net4 \
         -device pxb-pcie,id=pcie.4,bus_nr=188,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.4,id=smmuv3.4 \
         -device pcie-root-port,id=pcie.port4,bus=pcie.4,slot=4,chassis=4 \
         -device virtio-net-pci,bus=pcie.port4,netdev=net5 \
         -device pxb-pcie,id=pcie.5,bus_nr=184,bus=pcie.0 \
         -device pcie-root-port,id=pcie.port5,bus=pcie.5,slot=5,chassis=5 \
         -device virtio-net-pci,bus=pcie.port5,netdev=net6 \
         -device pxb-pcie,id=pcie.6,bus_nr=180,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.6,id=smmuv3.6 \
         -device pcie-root-port,id=pcie.port6,bus=pcie.6,slot=6,chassis=6 \
         -device virtio-net-pci,bus=pcie.port6,netdev=net7 \
         -device pxb-pcie,id=pcie.7,bus_nr=176,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.7,id=smmuv3.7 \
         -device pcie-root-port,id=pcie.port7,bus=pcie.7,slot=7,chassis=7 \
         -device virtio-net-pci,bus=pcie.port7,netdev=net8 \
         -device pxb-pcie,id=pcie.8,bus_nr=172,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.8,id=smmuv3.8 \
         -device pcie-root-port,id=pcie.port8,bus=pcie.8,slot=8,chassis=8 \
         -device virtio-net-pci,bus=pcie.port8,netdev=net9 \
         -device pxb-pcie,id=pcie.9,bus_nr=168,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.9,id=smmuv3.9 \
         -device pcie-root-port,id=pcie.port9,bus=pcie.9,slot=9,chassis=9 \
         -device virtio-net-pci,bus=pcie.port9,netdev=net10 \
         -device pxb-pcie,id=pcie.10,bus_nr=164,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.10,id=smmuv3.10 \
         -device 
pcie-root-port,id=pcie.port10,bus=pcie.10,slot=10,chassis=10 \
         -device virtio-net-pci,bus=pcie.port10,netdev=net11 \
         -device pxb-pcie,id=pcie.11,bus_nr=160,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.11,id=smmuv3.11 \
         -device 
pcie-root-port,id=pcie.port11,bus=pcie.11,slot=11,chassis=11 \
         -device virtio-net-pci,bus=pcie.port11,netdev=net12 \
         -device pxb-pcie,id=pcie.12,bus_nr=156,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.12,id=smmuv3.12 \
         -device 
pcie-root-port,id=pcie.port12,bus=pcie.12,slot=12,chassis=12 \
         -device virtio-net-pci,bus=pcie.port12,netdev=net13 \
         -device pxb-pcie,id=pcie.13,bus_nr=152,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.13,id=smmuv3.13 \
         -device 
pcie-root-port,id=pcie.port13,bus=pcie.13,slot=13,chassis=13 \
         -device virtio-net-pci,bus=pcie.port13,netdev=net14 \
         -device pxb-pcie,id=pcie.14,bus_nr=148,bus=pcie.0 \
         -device arm-smmuv3,primary-bus=pcie.14,id=smmuv3.14 \
         -device 
pcie-root-port,id=pcie.port14,bus=pcie.14,slot=14,chassis=14 \
         -device virtio-net-pci,bus=pcie.port14,netdev=net15 \
         -device pxb-pcie,id=pcie.15,bus_nr=144,bus=pcie.0 \
         -device 
pcie-root-port,id=pcie.port15,bus=pcie.15,slot=15,chassis=15 \
         -device virtio-net-pci,bus=pcie.port15,netdev=net16 \
         -device 
pcie-root-port,id=pcie.port16,bus=pcie.15,slot=16,chassis=16 \
         -device virtio-net-pci,bus=pcie.port16,netdev=net17


[0] 
https://lore.kernel.org/all/90957693-dc12-4731-960f-0ee295d297ec@nvidia.com/

Thanks,
Nathan

> -----Original Message-----
> From: Nathan Chen <nathanc@nvidia.com>
> Sent: Wednesday, July 2, 2025 2:02 AM
> To: qemu-devel@nongnu.org
> Cc: berrange@redhat.com; ddutile@redhat.com; eric.auger@redhat.com;
> gustavo.romero@linaro.org; imammedo@redhat.com; jgg@nvidia.com;
> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; Linuxarm <linuxarm@huawei.com>;
> mochs@nvidia.com; nathanc@nvidia.com; nicolinc@nvidia.com;
> peter.maydell@linaro.org; qemu-arm@nongnu.org; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; smostafa@google.com;
> Wangzhou (B) <wangzhou1@hisilicon.com>; zhangfei.gao@linaro.org
> Subject: Re: [PATCH v5 00/11] hw/arm/virt: Add support for user creatable
> SMMUv3 device
> 
> >    To address this, patch #6 in the series introduces a new helper
> >    function pci_setup_iommu_per_bus(), which explicitly sets the
> >    iommu_per_bus field in the PCIBus structure. This allows
> >    pci_device_get_iommu_bus_devfn() to retrieve IOMMU ops based
> >    on the specific bus.
> >
> >    This patch series introduces support for a user-creatable SMMUv3
> device
> >    (-device arm-smmuv3) in QEMU.
> 
> Tested-by: Nathan Chen <nathanc@nvidia.com>
> 
> I re-ran the test from v3 [0] and am able to create 16 SMMUv3 devices in
> a qemu VM with emulated devices properly associated with the guest
> SMMUs
> in guest sysfs - verified with some guest SMMUs having two or three
> emulated NICs assigned to them while other guest SMMUs have a minimum
> of
> one assigned.
> 
> Removing SMMUv3 devices from the VM config described above, I do not
> observe the problematic behavior where devices behind PXBs without
> SMMUs
> erroneously use the address space from pcie.0's SMMU. I removed SMMUv3
> devices from PXBs with one, two, and three emulated NICs assigned to
> them. Below are the guest topology and qemu command used where
> SMMUv3
> devices are excluded from the original test:

Cool. Appreciate testing that specifically.

Thanks,
Shameer