Series comparison

 RE: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable nested SMMUv3
 > -----Original Message-----
-> From: Shameerali Kolothum Thodi
+> From: Peter Maydell <peter.maydell@linaro.org>
-> Sent: Thursday, January 30, 2025 6:09 PM
+> Sent: Friday, December 13, 2024 1:33 PM
-> To: 'Daniel P. Berrangé' <berrange@redhat.com>
+> To: Jason Gunthorpe <jgg@nvidia.com>
-> Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
+> Cc: Daniel P. Berrangé <berrange@redhat.com>; Shameerali Kolothum
-> eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
+> Thodi <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
-> nicolinc@nvidia.com; ddutile@redhat.com; Linuxarm
+> qemu-devel@nongnu.org; eric.auger@redhat.com; nicolinc@nvidia.com;
-> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
+> ddutile@redhat.com; Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
-> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
+> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
-> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
+> Jonathan Cameron <jonathan.cameron@huawei.com>;
-> Subject: RE: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
+> zhangfei.gao@linaro.org
 > Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
 > nested SMMUv3
 >
-> Hi Daniel,
+> On Fri, 13 Dec 2024 at 12:46, Jason Gunthorpe <jgg@nvidia.com> wrote:
 > >
 > > On Fri, Dec 13, 2024 at 12:00:43PM +0000, Daniel P. Berrangé wrote:
 > > > On Fri, Nov 08, 2024 at 12:52:37PM +0000, Shameer Kolothum via wrote:
 > > > > Hi,
 > > > >
 > > > > This series adds initial support for a user-creatable "arm-smmuv3-
 > nested"
 > > > > device to Qemu. At present the Qemu ARM SMMUv3 emulation is per
 > machine
 > > > > and cannot support multiple SMMUv3s.
 > > > >
 > > > > In order to support vfio-pci dev assignment with vSMMUv3, the
 > physical
 > > > > SMMUv3 has to be configured in nested mode. Having a pluggable
 > > > > "arm-smmuv3-nested" device enables us to have multiple vSMMUv3
 > for Guests
 > > > > running on a host with multiple physical SMMUv3s. A few benefits of
 > doing
 > > > > this are,
 > > >
 > > > I'm not very familiar with arm, but from this description I'm not
 > > > really seeing how "nesting" is involved here. You're only talking
 > > > about the host and 1 L1 guest, no L2 guest.
 > >
 > > nesting is the term the iommu side is using to refer to the 2
 > > dimensional paging, ie a guest page table on top of a hypervisor page
 > > table.
 >
-> > -----Original Message-----
+> Isn't that more usually called "two stage" paging? Calling
-> > From: Daniel P. Berrangé <berrange@redhat.com>
+> that "nesting" seems like it is going to be massively confusing...
-> > Sent: Thursday, January 30, 2025 4:00 PM
-> > To: Shameerali Kolothum Thodi
+Yes. This will be renamed in future revisions as arm-smmuv3-accel.
-> <shameerali.kolothum.thodi@huawei.com>
 > > Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
 > > eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
 > > nicolinc@nvidia.com; ddutile@redhat.com; Linuxarm
 > > <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
 > > jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
 > > <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
 > > Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
 > > nested SMMUv3
 > >
 > > On Fri, Nov 08, 2024 at 12:52:37PM +0000, Shameer Kolothum via wrote:
 > > > How to use it(Eg:):
 > > >
 > > > On a HiSilicon platform that has multiple physical SMMUv3s, the ACC
 > ZIP
 > > VF
 > > > devices and HNS VF devices are behind different SMMUv3s. So for a
 > > Guest,
 > > > specify two smmuv3-nested devices each behind a pxb-pcie as below,
 > > >
 > > > ./qemu-system-aarch64 -machine virt,gic-version=3,default-bus-bypass-
 > > iommu=on \
 > > > -enable-kvm -cpu host -m 4G -smp cpus=8,maxcpus=8 \
 > > > -object iommufd,id=iommufd0 \
 > > > -bios QEMU_EFI.fd \
 > > > -kernel Image \
 > > > -device virtio-blk-device,drive=fs \
 > > > -drive if=none,file=rootfs.qcow2,id=fs \
 > > > -device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \
 > > > -device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1 \
 > > > -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
 > > > -device vfio-pci,host=0000:7d:02.1,bus=pcie.port1,iommufd=iommufd0 \
 > > > -device pxb-pcie,id=pcie.2,bus_nr=16,bus=pcie.0 \
 > > > -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=2 \
 > > > -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
 > > > -device vfio-pci,host=0000:75:00.1,bus=pcie.port2,iommufd=iommufd0 \
 > > > -append "rdinit=init console=ttyAMA0 root=/dev/vda2 rw
 > > earlycon=pl011,0x9000000" \
 > > > -device virtio-9p-pci,fsdev=p9fs2,mount_tag=p9,bus=pcie.0 \
 > > > -fsdev local,id=p9fs2,path=p9root,security_model=mapped \
 > > > -net none \
 > > > -nographic
 > >
 > > Above you say the host has 2 SMMUv3 devices, and you've created 2
 > > SMMUv3
 > > guest devices to match.
 > >
 > > The various emails in this thread & libvirt thread, indicate that each
 > > guest SMMUv3 is associated with a host SMMUv3, but I don't see any
 > > property on the command line for 'arm-ssmv3-nested' that tells it which
 > > host eSMMUv3 it is to be associated with.
 > >
 > > How does this association work ?
 >
-> You are right. The association is not very obvious in Qemu. The association
+> Also, how does it relate to what this series seems to be
-> and checking is done implicitly by kernel at the moment.  I will try to
+> doing, where we provide the guest with two separate SMMUs?
-> explain
+> (Are those two SMMUs "nested" in the sense that one is sitting
-> it here.
+> behind the other?)
 >
 > Each "arm-smmuv3-nested" instance, when the first device gets attached
 > to it, will create a S2 HWPT and a corresponding SMMUv3 domain in kernel
 > SMMUv3 driver. This domain will have a pointer representing the physical
 > SMMUv3 that the device belongs. And any other device which belongs to
 > the same physical SMMUv3 can share this S2 domain.
 >
 > If a device that belongs to a different physical SMMUv3 gets attached to
 > the above domain, the HWPT attach will eventually fail as the physical
 > smmuv3 in the domains will have a mismatch,
 > https://elixir.bootlin.com/linux/v6.13/source/drivers/iommu/arm/arm-
 > smmu-v3/arm-smmu-v3.c#L2860
 >
 > And as I mentioned in cover letter, Qemu will report,
 >
 > "
 > Attempt to add the HNS VF to a different SMMUv3 will result in,
 >
 > -device vfio-pci,host=0000:7d:02.2,bus=pcie.port3,iommufd=iommufd0:
 > Unable to attach viommu
 > -device vfio-pci,host=0000:7d:02.2,bus=pcie.port3,iommufd=iommufd0: vfio
 > 0000:7d:02.2:
 >    Failed to set iommu_device: [iommufd=29] error attach 0000:7d:02.2 (38)
 > to id=11: Invalid argument
 >
 > At present Qemu is not doing any extra validation other than the above
 > failure to make sure the user configuration is correct or not. The
 > assumption is libvirt will take care of this.
 > "
 > So in summary, if the libvirt gets it wrong, Qemu will fail with error.
 >
 > If a more explicit association is required, some help from kernel is required
 > to identify the physical SMMUv3 associated with the device.
-Again thinking about this, to have an explicit association in the Qemu command
+I don't think it requires two SMMUs in Guest. The nested or "two
-line between the vSMMUv3 and the phys smmuv3,
+stage" means the stage 1 page table is owned by Guest and stage 2
 by host. And this is achieved by IOMMUFD provided IOCTLs.
-We can possibly add something like,
+There is a precurser to this series where the support for hw accelerated
 stage support is added in Qemu SMMUv3 code.
--device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \
+Please see the complete branch here,
--device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1 \
+https://github.com/hisilicon/qemu/commits/private-smmuv3-nested-dev-rfc-v1/
--device arm-smmuv3-accel,bus=pcie.1,phys-smmuv3= smmu3.0x0000000100000000 \
+And patches prior to this commit adds that support:
--device vfio-pci,host=0000:7d:02.1,bus=pcie.port1,iommufd=iommufd0 \
+ccdbe3: ("cover-letter: Add HW accelerated nesting support for arm
 SMMUv3")
--device pxb-pcie,id=pcie.2,bus_nr=16,bus=pcie.0 \
+Nicolin is soon going to send out those for review. Or I can include
--device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=2 \
+those in this series so that it gives a complete picture. Nicolin?
 -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2, phys-smmuv3= smmu3.0x0000000200000000  \
 -device vfio-pci,host=0000:75:00.1,bus=pcie.port2,iommufd=iommufd0 \
-etc.
+Hope this clarifies any confusion.
 And Qemu does some checking to make sure that the device is indeed associated
 with the specified phys-smmuv3.  This can be done going through the sysfs path checking
 which is what I guess libvirt is currently doing to populate the topology. So basically
 Qemu is just replicating that to validate again.
 Or another option is extending the IOMMU_GET_HW_INFO IOCTL to return the phys
 smmuv3 base address which can avoid going through the sysfs.
 The only difference between the current approach(kernel failing the attach implicitly)
 and the above is, Qemu can provide a validation of inputs and may be report a  better
 error message than just saying " Unable to attach viommu/: Invalid argument".
 If the command line looks Ok, I will go with the sysfs path validation method first in my
 next respin.
 Please let me know.
 Thanks,
 Shameer

> -----Original Message-----
> From: Shameerali Kolothum Thodi
> Sent: Thursday, January 30, 2025 6:09 PM
> To: 'Daniel P. Berrangé' <berrange@redhat.com>
> Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
> nicolinc@nvidia.com; ddutile@redhat.com; Linuxarm
> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
> Subject: RE: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
> nested SMMUv3
> 
> Hi Daniel,
> 
> > -----Original Message-----
> > From: Daniel P. Berrangé <berrange@redhat.com>
> > Sent: Thursday, January 30, 2025 4:00 PM
> > To: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>
> > Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> > eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
> > nicolinc@nvidia.com; ddutile@redhat.com; Linuxarm
> > <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
> > jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> > <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
> > Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
> > nested SMMUv3
> >
> > On Fri, Nov 08, 2024 at 12:52:37PM +0000, Shameer Kolothum via wrote:
> > > How to use it(Eg:):
> > >
> > > On a HiSilicon platform that has multiple physical SMMUv3s, the ACC
> ZIP
> > VF
> > > devices and HNS VF devices are behind different SMMUv3s. So for a
> > Guest,
> > > specify two smmuv3-nested devices each behind a pxb-pcie as below,
> > >
> > > ./qemu-system-aarch64 -machine virt,gic-version=3,default-bus-bypass-
> > iommu=on \
> > > -enable-kvm -cpu host -m 4G -smp cpus=8,maxcpus=8 \
> > > -object iommufd,id=iommufd0 \
> > > -bios QEMU_EFI.fd \
> > > -kernel Image \
> > > -device virtio-blk-device,drive=fs \
> > > -drive if=none,file=rootfs.qcow2,id=fs \
> > > -device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \
> > > -device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1 \
> > > -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
> > > -device vfio-pci,host=0000:7d:02.1,bus=pcie.port1,iommufd=iommufd0 \
> > > -device pxb-pcie,id=pcie.2,bus_nr=16,bus=pcie.0 \
> > > -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=2 \
> > > -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
> > > -device vfio-pci,host=0000:75:00.1,bus=pcie.port2,iommufd=iommufd0 \
> > > -append "rdinit=init console=ttyAMA0 root=/dev/vda2 rw
> > earlycon=pl011,0x9000000" \
> > > -device virtio-9p-pci,fsdev=p9fs2,mount_tag=p9,bus=pcie.0 \
> > > -fsdev local,id=p9fs2,path=p9root,security_model=mapped \
> > > -net none \
> > > -nographic
> >
> > Above you say the host has 2 SMMUv3 devices, and you've created 2
> > SMMUv3
> > guest devices to match.
> >
> > The various emails in this thread & libvirt thread, indicate that each
> > guest SMMUv3 is associated with a host SMMUv3, but I don't see any
> > property on the command line for 'arm-ssmv3-nested' that tells it which
> > host eSMMUv3 it is to be associated with.
> >
> > How does this association work ?
> 
> You are right. The association is not very obvious in Qemu. The association
> and checking is done implicitly by kernel at the moment.  I will try to
> explain
> it here.
> 
> Each "arm-smmuv3-nested" instance, when the first device gets attached
> to it, will create a S2 HWPT and a corresponding SMMUv3 domain in kernel
> SMMUv3 driver. This domain will have a pointer representing the physical
> SMMUv3 that the device belongs. And any other device which belongs to
> the same physical SMMUv3 can share this S2 domain.
> 
> If a device that belongs to a different physical SMMUv3 gets attached to
> the above domain, the HWPT attach will eventually fail as the physical
> smmuv3 in the domains will have a mismatch,
> https://elixir.bootlin.com/linux/v6.13/source/drivers/iommu/arm/arm-
> smmu-v3/arm-smmu-v3.c#L2860
> 
> And as I mentioned in cover letter, Qemu will report,
> 
> "
> Attempt to add the HNS VF to a different SMMUv3 will result in,
> 
> -device vfio-pci,host=0000:7d:02.2,bus=pcie.port3,iommufd=iommufd0:
> Unable to attach viommu
> -device vfio-pci,host=0000:7d:02.2,bus=pcie.port3,iommufd=iommufd0: vfio
> 0000:7d:02.2:
>    Failed to set iommu_device: [iommufd=29] error attach 0000:7d:02.2 (38)
> to id=11: Invalid argument
> 
> At present Qemu is not doing any extra validation other than the above
> failure to make sure the user configuration is correct or not. The
> assumption is libvirt will take care of this.
> "
> So in summary, if the libvirt gets it wrong, Qemu will fail with error.
> 
> If a more explicit association is required, some help from kernel is required
> to identify the physical SMMUv3 associated with the device.

Again thinking about this, to have an explicit association in the Qemu command 
line between the vSMMUv3 and the phys smmuv3,

We can possibly add something like,

-device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \
-device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1 \
-device arm-smmuv3-accel,bus=pcie.1,phys-smmuv3= smmu3.0x0000000100000000 \
-device vfio-pci,host=0000:7d:02.1,bus=pcie.port1,iommufd=iommufd0 \

-device pxb-pcie,id=pcie.2,bus_nr=16,bus=pcie.0 \
-device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=2 \
-device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2, phys-smmuv3= smmu3.0x0000000200000000  \
-device vfio-pci,host=0000:75:00.1,bus=pcie.port2,iommufd=iommufd0 \

etc.

And Qemu does some checking to make sure that the device is indeed associated
with the specified phys-smmuv3.  This can be done going through the sysfs path checking
which is what I guess libvirt is currently doing to populate the topology. So basically
Qemu is just replicating that to validate again.

Or another option is extending the IOMMU_GET_HW_INFO IOCTL to return the phys
smmuv3 base address which can avoid going through the sysfs.

The only difference between the current approach(kernel failing the attach implicitly)
and the above is, Qemu can provide a validation of inputs and may be report a  better
error message than just saying " Unable to attach viommu/: Invalid argument".

If the command line looks Ok, I will go with the sysfs path validation method first in my
next respin.

Please let me know.

Thanks,
Shameer

> -----Original Message-----
> From: Peter Maydell <peter.maydell@linaro.org>
> Sent: Friday, December 13, 2024 1:33 PM
> To: Jason Gunthorpe <jgg@nvidia.com>
> Cc: Daniel P. Berrangé <berrange@redhat.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> qemu-devel@nongnu.org; eric.auger@redhat.com; nicolinc@nvidia.com;
> ddutile@redhat.com; Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>;
> zhangfei.gao@linaro.org
> Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
> nested SMMUv3
> 
> On Fri, 13 Dec 2024 at 12:46, Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > On Fri, Dec 13, 2024 at 12:00:43PM +0000, Daniel P. Berrangé wrote:
> > > On Fri, Nov 08, 2024 at 12:52:37PM +0000, Shameer Kolothum via wrote:
> > > > Hi,
> > > >
> > > > This series adds initial support for a user-creatable "arm-smmuv3-
> nested"
> > > > device to Qemu. At present the Qemu ARM SMMUv3 emulation is per
> machine
> > > > and cannot support multiple SMMUv3s.
> > > >
> > > > In order to support vfio-pci dev assignment with vSMMUv3, the
> physical
> > > > SMMUv3 has to be configured in nested mode. Having a pluggable
> > > > "arm-smmuv3-nested" device enables us to have multiple vSMMUv3
> for Guests
> > > > running on a host with multiple physical SMMUv3s. A few benefits of
> doing
> > > > this are,
> > >
> > > I'm not very familiar with arm, but from this description I'm not
> > > really seeing how "nesting" is involved here. You're only talking
> > > about the host and 1 L1 guest, no L2 guest.
> >
> > nesting is the term the iommu side is using to refer to the 2
> > dimensional paging, ie a guest page table on top of a hypervisor page
> > table.
> 
> Isn't that more usually called "two stage" paging? Calling
> that "nesting" seems like it is going to be massively confusing...

Yes. This will be renamed in future revisions as arm-smmuv3-accel.

> 
> Also, how does it relate to what this series seems to be
> doing, where we provide the guest with two separate SMMUs?
> (Are those two SMMUs "nested" in the sense that one is sitting
> behind the other?)

I don't think it requires two SMMUs in Guest. The nested or "two
stage" means the stage 1 page table is owned by Guest and stage 2
by host. And this is achieved by IOMMUFD provided IOCTLs.

There is a precurser to this series where the support for hw accelerated
2 stage support is added in Qemu SMMUv3 code.

Please see the complete branch here,
https://github.com/hisilicon/qemu/commits/private-smmuv3-nested-dev-rfc-v1/
And patches prior to this commit adds that support: 
4ccdbe3: ("cover-letter: Add HW accelerated nesting support for arm
SMMUv3")

Nicolin is soon going to send out those for review. Or I can include
those in this series so that it gives a complete picture. Nicolin?

Hope this clarifies any confusion.

Thanks,
Shameer