Series comparison

 RE: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable nested SMMUv3
-Hi Daniel,
 > -----Original Message-----
-> From: Daniel P. Berrangé <berrange@redhat.com>
+> From: Jason Gunthorpe <jgg@nvidia.com>
-> Sent: Thursday, January 30, 2025 4:00 PM
+> Sent: Thursday, February 6, 2025 5:59 PM
-> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
+> To: Daniel P. Berrangé <berrange@redhat.com>
-> Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
+> Cc: Shameerali Kolothum Thodi
-> eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
+> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
-> nicolinc@nvidia.com; ddutile@redhat.com; Linuxarm
+> qemu-devel@nongnu.org; eric.auger@redhat.com;
-> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
+> peter.maydell@linaro.org; nicolinc@nvidia.com; ddutile@redhat.com;
-> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
+> Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
-> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
+> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
 > Jonathan Cameron <jonathan.cameron@huawei.com>;
 > zhangfei.gao@linaro.org; nathanc@nvidia.com
 > Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
 > nested SMMUv3
 >
-> On Fri, Nov 08, 2024 at 12:52:37PM +0000, Shameer Kolothum via wrote:
+> On Thu, Feb 06, 2025 at 05:54:57PM +0000, Daniel P. Berrangé wrote:
-> > How to use it(Eg:):
+> > > > We shouldn't assume any VFIO device exists in the QEMU cnofig at the
 > time
 > > > > we realize the virtual ssmu. I expect the SMMU may be cold plugged,
 > while
 > > > > the VFIO devices may be hot plugged arbitrarly later, and we should
 > have
 > > > > the association initialized the SMMU is realized.
 > > >
 > > > This is not supported kernel side, you can't instantiate a vIOMMU
 > > > without a VFIO device that uses it. For security.
 > >
-> > On a HiSilicon platform that has multiple physical SMMUv3s, the ACC ZIP
+> > What are the security concerns here ?
 > VF
 > > devices and HNS VF devices are behind different SMMUv3s. So for a
 > Guest,
 > > specify two smmuv3-nested devices each behind a pxb-pcie as below,
 > >
 > > ./qemu-system-aarch64 -machine virt,gic-version=3,default-bus-bypass-
 > iommu=on \
 > > -enable-kvm -cpu host -m 4G -smp cpus=8,maxcpus=8 \
 > > -object iommufd,id=iommufd0 \
 > > -bios QEMU_EFI.fd \
 > > -kernel Image \
 > > -device virtio-blk-device,drive=fs \
 > > -drive if=none,file=rootfs.qcow2,id=fs \
 > > -device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \
 > > -device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1 \
 > > -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
 > > -device vfio-pci,host=0000:7d:02.1,bus=pcie.port1,iommufd=iommufd0 \
 > > -device pxb-pcie,id=pcie.2,bus_nr=16,bus=pcie.0 \
 > > -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=2 \
 > > -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
 > > -device vfio-pci,host=0000:75:00.1,bus=pcie.port2,iommufd=iommufd0 \
 > > -append "rdinit=init console=ttyAMA0 root=/dev/vda2 rw
 > earlycon=pl011,0x9000000" \
 > > -device virtio-9p-pci,fsdev=p9fs2,mount_tag=p9,bus=pcie.0 \
 > > -fsdev local,id=p9fs2,path=p9root,security_model=mapped \
 > > -net none \
 > > -nographic
 >
-> Above you say the host has 2 SMMUv3 devices, and you've created 2
+> You should not be able to open iommufd and manipulate iommu HW that
-> SMMUv3
+> you don't have a VFIO descriptor for, including creating physical
-> guest devices to match.
+> vIOMMU resources, allocating command queues and whatever else.
 >
-> The various emails in this thread & libvirt thread, indicate that each
+> Some kind of hot plug smmu would have to create a vSMMU without any
-> guest SMMUv3 is associated with a host SMMUv3, but I don't see any
+> kernel backing and then later bind it to a kernel implementation.
 > property on the command line for 'arm-ssmv3-nested' that tells it which
 > host eSMMUv3 it is to be associated with.
 >
 > How does this association work ?
-You are right. The association is not very obvious in Qemu. The association
+Not sure I get the problem with associating vSMMU with a pSMMU. Something
-and checking is done implicitly by kernel at the moment.  I will try to explain
+like an iommu instance id mentioned before,
 it here.
-Each "arm-smmuv3-nested" instance, when the first device gets attached
+-device arm-smmuv3-accel,id=smmuv2,bus=pcie.2,host-smmu=iommu.1
 to it, will create a S2 HWPT and a corresponding SMMUv3 domain in kernel
 SMMUv3 driver. This domain will have a pointer representing the physical
 SMMUv3 that the device belongs. And any other device which belongs to
 the same physical SMMUv3 can share this S2 domain.
-If a device that belongs to a different physical SMMUv3 gets attached to
+This can realize the vSMMU without actually creating a vIOMMU in kernel.
-the above domain, the HWPT attach will eventually fail as the physical
+And when the dev gets attached/realized, check (GET_HW_INFO)the specified
-smmuv3 in the domains will have a mismatch,
+iommu instance id matches or not.
 https://elixir.bootlin.com/linux/v6.13/source/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c#L2860
-And as I mentioned in cover letter, Qemu will report,
+Or the concern here is exporting an iommu instance id to user space?
 "
 Attempt to add the HNS VF to a different SMMUv3 will result in,
 -device vfio-pci,host=0000:7d:02.2,bus=pcie.port3,iommufd=iommufd0: Unable to attach viommu
 -device vfio-pci,host=0000:7d:02.2,bus=pcie.port3,iommufd=iommufd0: vfio 0000:7d:02.2:
    Failed to set iommu_device: [iommufd=29] error attach 0000:7d:02.2 (38) to id=11: Invalid argument
 At present Qemu is not doing any extra validation other than the above
 failure to make sure the user configuration is correct or not. The
 assumption is libvirt will take care of this.
 "
 So in summary, if the libvirt gets it wrong, Qemu will fail with error.
 If a more explicit association is required, some help from kernel is required
 to identify the physical SMMUv3 associated with the device.
 Jason/Nicolin, any thoughts on this?
 Thanks,
 Shameer

Hi Daniel,

> -----Original Message-----
> From: Daniel P. Berrangé <berrange@redhat.com>
> Sent: Thursday, January 30, 2025 4:00 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
> nicolinc@nvidia.com; ddutile@redhat.com; Linuxarm
> <linuxarm@huawei.com>; Wangzhou (B) <wangzhou1@hisilicon.com>;
> jiangkunkun <jiangkunkun@huawei.com>; Jonathan Cameron
> <jonathan.cameron@huawei.com>; zhangfei.gao@linaro.org
> Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
> nested SMMUv3
> 
> On Fri, Nov 08, 2024 at 12:52:37PM +0000, Shameer Kolothum via wrote:
> > How to use it(Eg:):
> >
> > On a HiSilicon platform that has multiple physical SMMUv3s, the ACC ZIP
> VF
> > devices and HNS VF devices are behind different SMMUv3s. So for a
> Guest,
> > specify two smmuv3-nested devices each behind a pxb-pcie as below,
> >
> > ./qemu-system-aarch64 -machine virt,gic-version=3,default-bus-bypass-
> iommu=on \
> > -enable-kvm -cpu host -m 4G -smp cpus=8,maxcpus=8 \
> > -object iommufd,id=iommufd0 \
> > -bios QEMU_EFI.fd \
> > -kernel Image \
> > -device virtio-blk-device,drive=fs \
> > -drive if=none,file=rootfs.qcow2,id=fs \
> > -device pxb-pcie,id=pcie.1,bus_nr=8,bus=pcie.0 \
> > -device pcie-root-port,id=pcie.port1,bus=pcie.1,chassis=1 \
> > -device arm-smmuv3-nested,id=smmuv1,pci-bus=pcie.1 \
> > -device vfio-pci,host=0000:7d:02.1,bus=pcie.port1,iommufd=iommufd0 \
> > -device pxb-pcie,id=pcie.2,bus_nr=16,bus=pcie.0 \
> > -device pcie-root-port,id=pcie.port2,bus=pcie.2,chassis=2 \
> > -device arm-smmuv3-nested,id=smmuv2,pci-bus=pcie.2 \
> > -device vfio-pci,host=0000:75:00.1,bus=pcie.port2,iommufd=iommufd0 \
> > -append "rdinit=init console=ttyAMA0 root=/dev/vda2 rw
> earlycon=pl011,0x9000000" \
> > -device virtio-9p-pci,fsdev=p9fs2,mount_tag=p9,bus=pcie.0 \
> > -fsdev local,id=p9fs2,path=p9root,security_model=mapped \
> > -net none \
> > -nographic
> 
> Above you say the host has 2 SMMUv3 devices, and you've created 2
> SMMUv3
> guest devices to match.
> 
> The various emails in this thread & libvirt thread, indicate that each
> guest SMMUv3 is associated with a host SMMUv3, but I don't see any
> property on the command line for 'arm-ssmv3-nested' that tells it which
> host eSMMUv3 it is to be associated with.
> 
> How does this association work ?

You are right. The association is not very obvious in Qemu. The association
and checking is done implicitly by kernel at the moment.  I will try to explain
it here.

Each "arm-smmuv3-nested" instance, when the first device gets attached
to it, will create a S2 HWPT and a corresponding SMMUv3 domain in kernel
SMMUv3 driver. This domain will have a pointer representing the physical
SMMUv3 that the device belongs. And any other device which belongs to
the same physical SMMUv3 can share this S2 domain.

If a device that belongs to a different physical SMMUv3 gets attached to
the above domain, the HWPT attach will eventually fail as the physical
smmuv3 in the domains will have a mismatch,
https://elixir.bootlin.com/linux/v6.13/source/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c#L2860

And as I mentioned in cover letter, Qemu will report,

"
Attempt to add the HNS VF to a different SMMUv3 will result in,

-device vfio-pci,host=0000:7d:02.2,bus=pcie.port3,iommufd=iommufd0: Unable to attach viommu
-device vfio-pci,host=0000:7d:02.2,bus=pcie.port3,iommufd=iommufd0: vfio 0000:7d:02.2:
   Failed to set iommu_device: [iommufd=29] error attach 0000:7d:02.2 (38) to id=11: Invalid argument

At present Qemu is not doing any extra validation other than the above 
failure to make sure the user configuration is correct or not. The
assumption is libvirt will take care of this.
"
So in summary, if the libvirt gets it wrong, Qemu will fail with error.

If a more explicit association is required, some help from kernel is required
to identify the physical SMMUv3 associated with the device.

Jason/Nicolin, any thoughts on this?

Thanks,
Shameer

> -----Original Message-----
> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, February 6, 2025 5:59 PM
> To: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> qemu-devel@nongnu.org; eric.auger@redhat.com;
> peter.maydell@linaro.org; nicolinc@nvidia.com; ddutile@redhat.com;
> Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>;
> zhangfei.gao@linaro.org; nathanc@nvidia.com
> Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
> nested SMMUv3
> 
> On Thu, Feb 06, 2025 at 05:54:57PM +0000, Daniel P. Berrangé wrote:
> > > > We shouldn't assume any VFIO device exists in the QEMU cnofig at the
> time
> > > > we realize the virtual ssmu. I expect the SMMU may be cold plugged,
> while
> > > > the VFIO devices may be hot plugged arbitrarly later, and we should
> have
> > > > the association initialized the SMMU is realized.
> > >
> > > This is not supported kernel side, you can't instantiate a vIOMMU
> > > without a VFIO device that uses it. For security.
> >
> > What are the security concerns here ?
> 
> You should not be able to open iommufd and manipulate iommu HW that
> you don't have a VFIO descriptor for, including creating physical
> vIOMMU resources, allocating command queues and whatever else.
> 
> Some kind of hot plug smmu would have to create a vSMMU without any
> kernel backing and then later bind it to a kernel implementation.

Not sure I get the problem with associating vSMMU with a pSMMU. Something
like an iommu instance id mentioned before,

-device arm-smmuv3-accel,id=smmuv2,bus=pcie.2,host-smmu=iommu.1

This can realize the vSMMU without actually creating a vIOMMU in kernel.
And when the dev gets attached/realized, check (GET_HW_INFO)the specified
iommu instance id matches or not.

Or the concern here is exporting an iommu instance id to user space?

Thanks,
Shameer