> -----Original Message-----
> From: Donald Dutile <ddutile@redhat.com>
> Sent: Friday, April 18, 2025 9:34 PM
> To: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> qemu-devel@nongnu.org
> Cc: eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
> nicolinc@nvidia.com; berrange@redhat.com; nathanc@nvidia.com;
> mochs@nvidia.com; smostafa@google.com; Linuxarm
> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
> Subject: Re: [PATCH 0/5] Add support for user creatable SMMUv3 device
> 
> Shameer,
> Hi!
> 
> First off, like the partitioning of these pieces.
> 
> On 4/15/25 4:10 AM, Shameer Kolothum wrote:
> > Hi All,
> >
> > This patch series introduces support for a user-creatable SMMUv3 device
> > (-device arm-smmuv3-dev) in QEMU.
> >
> Can we drop the '-dev', as 'arm-smmu' is sufficient, as is '-device'?
> 
> I know, internal to QEMU, there's already an ARM_SMMU, but as suggested
> later,
> if it uses the same struct, the qemu cmdline syntax is a bit less redundant.

Trust me I tried that😊. The problem is that, the legacy one doesn't have a bus
associated with it and the moment we change that and have bus specified for the
 "-device arm-smmuv3, bus=pcie.0" the legacy smmuv3 creation in virt,

create_smmu() --> qdev_new(TYPE_ARM_SMMUV3)

will fails as it expects the bus type to be specified for it. I couldn't find a way
to solve that.

> 
> > The implementation is based on feedback received from the RFCv2[0]:
> > "hw/arm/virt: Add support for user-creatable accelerated SMMUv3"
> >
> > Currently, QEMU's SMMUv3 emulation (iommu=smmuv3) is tied to the
> machine
> > and does not support instantiating multiple SMMUv3 devices—each
> associated
> > with a separate PCIe root complex. In contrast, real-world ARM systems
> > often include multiple SMMUv3 instances, each bound to a different PCIe
> > root complex.
> >
> > This also lays the groundwork for supporting accelerated SMMUv3, as
> > proposed in the aforementioned RFC. Please note, the
> accelerated SMMUv3
> > support is not part of this series and will be sent out as a separate
> > series later on top of this one.
> >
> > Summary of changes:
> >
> >   -Introduces support for multiple -device arm-smmuv3-dev,bus=pcie.x
> >    instances.
> >
> >    Example usage:
> >
> >    -device arm-smmuv3-dev,bus=pcie.0
> >    -device virtio-net-pci,bus=pcie.0
> >    -device pxb-pcie,id=pcie.1,bus_nr=2
> >    -device arm-smmuv3-dev,bus=pcie.1
> >    -device pcie-root-port,id=pcie.port1,bus=pcie.1
> >    -device virtio-net-pci,bus=pcie.port1
> >
> >    -Supports either the legacy iommu=smmuv3 option or the new
> >    "-device arm-smmuv3-dev" model.
> >
> nice! :)
> Can it support both? i.e., some devices using a system-wide, legacy
> smmuv3,
> and some pcie devices using the -device arm-smmuv3 ?

Please see my reply to patch #2. In short No, it doesn't support both.

Also I think we should slowly deprecate the use of legacy SMMUv3 going 
forward unless there is a strong use case/reason to support it.

> If not, then add a check to min-warn or max-fail, that config.
> I can see old machines being converted/upgraded to new machines,
> and they will want to switch from legacy->device smmuv3, and catching
> an edit/update to a machine change (or enabling automated conversion)
> would be helpful.

Please see Patch #4. It already checks and prevents if incompatible SMMUv3
types are specified. Or is the suggestion here is to add something extra?
Please let me know.

Thanks,
Shameer

On 4/22/25 4:57 AM, Shameerali Kolothum Thodi wrote:
> 
>> -----Original Message-----
>> From: Donald Dutile <ddutile@redhat.com>
>> Sent: Friday, April 18, 2025 9:34 PM
>> To: Shameerali Kolothum Thodi
>> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
>> qemu-devel@nongnu.org
>> Cc: eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
>> nicolinc@nvidia.com; berrange@redhat.com; nathanc@nvidia.com;
>> mochs@nvidia.com; smostafa@google.com; Linuxarm
>> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
>> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
>> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
>> Subject: Re: [PATCH 0/5] Add support for user creatable SMMUv3 device
>>
>> Shameer,
>> Hi!
>>
>> First off, like the partitioning of these pieces.
>>
>> On 4/15/25 4:10 AM, Shameer Kolothum wrote:
>>> Hi All,
>>>
>>> This patch series introduces support for a user-creatable SMMUv3 device
>>> (-device arm-smmuv3-dev) in QEMU.
>>>
>> Can we drop the '-dev', as 'arm-smmu' is sufficient, as is '-device'?
>>
>> I know, internal to QEMU, there's already an ARM_SMMU, but as suggested
>> later,
>> if it uses the same struct, the qemu cmdline syntax is a bit less redundant.
> 
> Trust me I tried that😊. The problem is that, the legacy one doesn't have a bus
> associated with it and the moment we change that and have bus specified for the
>   "-device arm-smmuv3, bus=pcie.0" the legacy smmuv3 creation in virt,
> 
> create_smmu() --> qdev_new(TYPE_ARM_SMMUV3)
> 
> will fails as it expects the bus type to be specified for it. I couldn't find a way
> to solve that.
> 
I guessed that's why there was new syntax for what is basically an extension of previous syntax.
I'll dig more into it and see if there's a way to handle it.
qemu does handle varying added params (id=blah, no id=blah),
so I'm researching how that's done, to figure out how the legacy, smmu-for-all,
and the smmu-for-specific pcie<RC> can both be supported.
Given your stated trials and tribulations, this should be fun. ;-)
- Don

>>
>>> The implementation is based on feedback received from the RFCv2[0]:
>>> "hw/arm/virt: Add support for user-creatable accelerated SMMUv3"
>>>
>>> Currently, QEMU's SMMUv3 emulation (iommu=smmuv3) is tied to the
>> machine
>>> and does not support instantiating multiple SMMUv3 devices—each
>> associated
>>> with a separate PCIe root complex. In contrast, real-world ARM systems
>>> often include multiple SMMUv3 instances, each bound to a different PCIe
>>> root complex.
>>>
>>> This also lays the groundwork for supporting accelerated SMMUv3, as
>>> proposed in the aforementioned RFC. Please note, the
>> accelerated SMMUv3
>>> support is not part of this series and will be sent out as a separate
>>> series later on top of this one.
>>>
>>> Summary of changes:
>>>
>>>    -Introduces support for multiple -device arm-smmuv3-dev,bus=pcie.x
>>>     instances.
>>>
>>>     Example usage:
>>>
>>>     -device arm-smmuv3-dev,bus=pcie.0
>>>     -device virtio-net-pci,bus=pcie.0
>>>     -device pxb-pcie,id=pcie.1,bus_nr=2
>>>     -device arm-smmuv3-dev,bus=pcie.1
>>>     -device pcie-root-port,id=pcie.port1,bus=pcie.1
>>>     -device virtio-net-pci,bus=pcie.port1
>>>
>>>     -Supports either the legacy iommu=smmuv3 option or the new
>>>     "-device arm-smmuv3-dev" model.
>>>
>> nice! :)
>> Can it support both? i.e., some devices using a system-wide, legacy
>> smmuv3,
>> and some pcie devices using the -device arm-smmuv3 ?
> 
> Please see my reply to patch #2. In short No, it doesn't support both.
> 
> Also I think we should slowly deprecate the use of legacy SMMUv3 going
> forward unless there is a strong use case/reason to support it.
> 
>> If not, then add a check to min-warn or max-fail, that config.
>> I can see old machines being converted/upgraded to new machines,
>> and they will want to switch from legacy->device smmuv3, and catching
>> an edit/update to a machine change (or enabling automated conversion)
>> would be helpful.
> 
> Please see Patch #4. It already checks and prevents if incompatible SMMUv3
> types are specified. Or is the suggestion here is to add something extra?
> Please let me know.
> 
> Thanks,
> Shameer
> 

Hi Shameer,

On 4/28/25 8:41 PM, Donald Dutile wrote:
>
>
> On 4/22/25 4:57 AM, Shameerali Kolothum Thodi wrote:
>>
>>> -----Original Message-----
>>> From: Donald Dutile <ddutile@redhat.com>
>>> Sent: Friday, April 18, 2025 9:34 PM
>>> To: Shameerali Kolothum Thodi
>>> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
>>> qemu-devel@nongnu.org
>>> Cc: eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
>>> nicolinc@nvidia.com; berrange@redhat.com; nathanc@nvidia.com;
>>> mochs@nvidia.com; smostafa@google.com; Linuxarm
>>> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
>>> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
>>> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
>>> Subject: Re: [PATCH 0/5] Add support for user creatable SMMUv3 device
>>>
>>> Shameer,
>>> Hi!
>>>
>>> First off, like the partitioning of these pieces.
>>>
>>> On 4/15/25 4:10 AM, Shameer Kolothum wrote:
>>>> Hi All,
>>>>
>>>> This patch series introduces support for a user-creatable SMMUv3
>>>> device
>>>> (-device arm-smmuv3-dev) in QEMU.
>>>>
>>> Can we drop the '-dev', as 'arm-smmu' is sufficient, as is '-device'?
>>>
>>> I know, internal to QEMU, there's already an ARM_SMMU, but as suggested
>>> later,
>>> if it uses the same struct, the qemu cmdline syntax is a bit less
>>> redundant.
>>
>> Trust me I tried that😊. The problem is that, the legacy one doesn't
>> have a bus
>> associated with it and the moment we change that and have bus
>> specified for the
>>   "-device arm-smmuv3, bus=pcie.0" the legacy smmuv3 creation in virt,
>>
>> create_smmu() --> qdev_new(TYPE_ARM_SMMUV3)
I have the impression you can achieve your goal by replacing
sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
by
qdev_realize_and_unref(dev, &bus->qbus, &error_fatal);
See https://github.com/eauger/qemu/tree/arm-smmuv3-dev-rc0

and its top patch.

indeed it would be better if we could reuse the same device.

Eric
>>
>> will fails as it expects the bus type to be specified for it. I
>> couldn't find a way
>> to solve that.
>>
> I guessed that's why there was new syntax for what is basically an
> extension of previous syntax.
> I'll dig more into it and see if there's a way to handle it.
> qemu does handle varying added params (id=blah, no id=blah),
> so I'm researching how that's done, to figure out how the legacy,
> smmu-for-all,
> and the smmu-for-specific pcie<RC> can both be supported.
> Given your stated trials and tribulations, this should be fun. ;-)
> - Don
>
>>>
>>>> The implementation is based on feedback received from the RFCv2[0]:
>>>> "hw/arm/virt: Add support for user-creatable accelerated SMMUv3"
>>>>
>>>> Currently, QEMU's SMMUv3 emulation (iommu=smmuv3) is tied to the
>>> machine
>>>> and does not support instantiating multiple SMMUv3 devices—each
>>> associated
>>>> with a separate PCIe root complex. In contrast, real-world ARM systems
>>>> often include multiple SMMUv3 instances, each bound to a different
>>>> PCIe
>>>> root complex.
>>>>
>>>> This also lays the groundwork for supporting accelerated SMMUv3, as
>>>> proposed in the aforementioned RFC. Please note, the
>>> accelerated SMMUv3
>>>> support is not part of this series and will be sent out as a separate
>>>> series later on top of this one.
>>>>
>>>> Summary of changes:
>>>>
>>>>    -Introduces support for multiple -device arm-smmuv3-dev,bus=pcie.x
>>>>     instances.
>>>>
>>>>     Example usage:
>>>>
>>>>     -device arm-smmuv3-dev,bus=pcie.0
>>>>     -device virtio-net-pci,bus=pcie.0
>>>>     -device pxb-pcie,id=pcie.1,bus_nr=2
>>>>     -device arm-smmuv3-dev,bus=pcie.1
>>>>     -device pcie-root-port,id=pcie.port1,bus=pcie.1
>>>>     -device virtio-net-pci,bus=pcie.port1
>>>>
>>>>     -Supports either the legacy iommu=smmuv3 option or the new
>>>>     "-device arm-smmuv3-dev" model.
>>>>
>>> nice! :)
>>> Can it support both? i.e., some devices using a system-wide, legacy
>>> smmuv3,
>>> and some pcie devices using the -device arm-smmuv3 ?
>>
>> Please see my reply to patch #2. In short No, it doesn't support both.
>>
>> Also I think we should slowly deprecate the use of legacy SMMUv3 going
>> forward unless there is a strong use case/reason to support it.
>>
>>> If not, then add a check to min-warn or max-fail, that config.
>>> I can see old machines being converted/upgraded to new machines,
>>> and they will want to switch from legacy->device smmuv3, and catching
>>> an edit/update to a machine change (or enabling automated conversion)
>>> would be helpful.
>>
>> Please see Patch #4. It already checks and prevents if incompatible
>> SMMUv3
>> types are specified. Or is the suggestion here is to add something
>> extra?
>> Please let me know.
>>
>> Thanks,
>> Shameer
>>
>

Hi Eric,

> -----Original Message-----
> From: Eric Auger <eric.auger@redhat.com>
> Sent: Wednesday, April 30, 2025 6:48 PM
> To: Donald Dutile <ddutile@redhat.com>; Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> qemu-devel@nongnu.org
> Cc: peter.maydell@linaro.org; jgg@nvidia.com; nicolinc@nvidia.com;
> berrange@redhat.com; nathanc@nvidia.com; mochs@nvidia.com;
> smostafa@google.com; Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>;
> zhangfei.gao@linaro.org
> Subject: Re: [PATCH 0/5] Add support for user creatable SMMUv3 device
> 
> Hi Shameer,
> 
> On 4/28/25 8:41 PM, Donald Dutile wrote:
> >
> >
> > On 4/22/25 4:57 AM, Shameerali Kolothum Thodi wrote:
> >>
> >>> -----Original Message-----
> >>> From: Donald Dutile <ddutile@redhat.com>
> >>> Sent: Friday, April 18, 2025 9:34 PM
> >>> To: Shameerali Kolothum Thodi
> >>> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> >>> qemu-devel@nongnu.org
> >>> Cc: eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
> >>> nicolinc@nvidia.com; berrange@redhat.com; nathanc@nvidia.com;
> >>> mochs@nvidia.com; smostafa@google.com; Linuxarm
> >>> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
> >>> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> >>> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
> >>> Subject: Re: [PATCH 0/5] Add support for user creatable SMMUv3
> device
> >>>
> >>> Shameer,
> >>> Hi!
> >>>
> >>> First off, like the partitioning of these pieces.
> >>>
> >>> On 4/15/25 4:10 AM, Shameer Kolothum wrote:
> >>>> Hi All,
> >>>>
> >>>> This patch series introduces support for a user-creatable SMMUv3
> >>>> device
> >>>> (-device arm-smmuv3-dev) in QEMU.
> >>>>
> >>> Can we drop the '-dev', as 'arm-smmu' is sufficient, as is '-device'?
> >>>
> >>> I know, internal to QEMU, there's already an ARM_SMMU, but as
> suggested
> >>> later,
> >>> if it uses the same struct, the qemu cmdline syntax is a bit less
> >>> redundant.
> >>
> >> Trust me I tried that😊. The problem is that, the legacy one doesn't
> >> have a bus
> >> associated with it and the moment we change that and have bus
> >> specified for the
> >>   "-device arm-smmuv3, bus=pcie.0" the legacy smmuv3 creation in virt,
> >>
> >> create_smmu() --> qdev_new(TYPE_ARM_SMMUV3)
> I have the impression you can achieve your goal by replacing
> sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
> by
> qdev_realize_and_unref(dev, &bus->qbus, &error_fatal);
> See https://github.com/eauger/qemu/tree/arm-smmuv3-dev-rc0

Cool!. That does the trick. Thanks.
 
> and its top patch.
> 
> indeed it would be better if we could reuse the same device.

I will respin soon with the above and addressing other comments received.

Thanks,
Shameer