Series comparison

 RE: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable nested SMMUv3
 > -----Original Message-----
-> From: Nathan Chen <nathanc@nvidia.com>
+> From: Jason Gunthorpe <jgg@nvidia.com>
-> Sent: Friday, November 22, 2024 1:42 AM
+> Sent: Thursday, February 6, 2025 6:13 PM
 > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
-> Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
+> Cc: Daniel P. Berrangé <berrange@redhat.com>; qemu-arm@nongnu.org;
-> eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
+> qemu-devel@nongnu.org; eric.auger@redhat.com;
-> ddutile@redhat.com; Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
+> peter.maydell@linaro.org; nicolinc@nvidia.com; ddutile@redhat.com;
 > Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
 > <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
 > Jonathan Cameron <jonathan.cameron@huawei.com>;
-> zhangfei.gao@linaro.org; Nicolin Chen <nicolinc@nvidia.com>
+> zhangfei.gao@linaro.org; nathanc@nvidia.com
 > Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
 > nested SMMUv3
 >
->  >> Also as a heads up, I've added support for auto-inserting PCIe switch
+> On Thu, Feb 06, 2025 at 06:04:57PM +0000, Shameerali Kolothum Thodi
->  >> between the PXB and GPUs in libvirt to attach multiple devices to a
+> wrote:
-> SMMU
+> > > Some kind of hot plug smmu would have to create a vSMMU without
->  >> node per libvirt's documentation - "If you intend to plug multiple
+> any
->  >> devices into a pcie-expander-bus, you must connect a
+> > > kernel backing and then later bind it to a kernel implementation.
->  >> pcie-switch-upstream-port to the pcie-root-port that is plugged into the
+> >
->  >> pcie-expander-bus, and multiple pcie-switch-downstream-ports to the
+> > Not sure I get the problem with associating vSMMU with a pSMMU.
->  >> pcie-switch-upstream-port". Future unit-tests should follow this
+> Something
->  >> topology configuration.
+> > like an iommu instance id mentioned before,
->  >
+> >
->  > Ok. Could you please give me an example Qemu equivalent command
+> > -device arm-smmuv3-accel,id=smmuv2,bus=pcie.2,host-smmu=iommu.1
-> option,
+> >
->  > if possible, for the above case. I am not that familiar with libvirt
+> > This can realize the vSMMU without actually creating a vIOMMU in kernel.
-> and I would
+> > And when the dev gets attached/realized, check (GET_HW_INFO)the
->  > also like to test the above scenario.
+> specified
 > > iommu instance id matches or not.
 > >
 > > Or the concern here is exporting an iommu instance id to user space?
 >
-> You can use "-device x3130-upstream" for the upstream switch port, and
+> Philisophically we do not permit any HW access through iommufd without
-> "-device xio3130-downstream" for the downstream port:
+> a VFIO fd to "prove" the process has rights to touch hardware.
 >
->   -device pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x1 \
+> We don't have any way to prove the process has rights to touch the
->   -device pcie-root-port,id=pci.2,bus=pci.1,addr=0x0 \
+> iommu hardware seperately from VFIO.
 >   -device x3130-upstream,id=pci.3,bus=pci.2,addr=0x0 \
 >   -device xio3130-
 > downstream,id=pci.4,bus=pci.3,addr=0x0,chassis=17,port=1 \
 >   -device vfio-pci,host=0009:01:00.0,id=hostdev0,bus=pci.4,addr=0x0 \
 >   -device arm-smmuv3-nested,pci-bus=pci.1
-Thanks. Just wondering why libvirt mandates usage of pcie-switch for multiple
+It is not. Qemu just instantiates a vSMMU and assigns the IOMMU
-device plugging rather than just using pcie-root-ports?
+instance id to it.
-Please let me if there is any advantage in doing so that you are aware of.
+>
 > So even if you invent an iommu ID we cannot accept it as a handle to
 > create viommu in iommufd.
 Creating the vIOMMU only happens when the user does a  cold/hot plug of
 a VFIO device. At that time Qemu checks whether the assigned id matches
 with whatever the kernel tell it.
 Thanks,
 Shameer

> -----Original Message-----
> From: Nathan Chen <nathanc@nvidia.com>
> Sent: Friday, November 22, 2024 1:42 AM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: qemu-arm@nongnu.org; qemu-devel@nongnu.org;
> eric.auger@redhat.com; peter.maydell@linaro.org; jgg@nvidia.com;
> ddutile@redhat.com; Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>;
> zhangfei.gao@linaro.org; Nicolin Chen <nicolinc@nvidia.com>
> Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
> nested SMMUv3
> 
>  >> Also as a heads up, I've added support for auto-inserting PCIe switch
>  >> between the PXB and GPUs in libvirt to attach multiple devices to a
> SMMU
>  >> node per libvirt's documentation - "If you intend to plug multiple
>  >> devices into a pcie-expander-bus, you must connect a
>  >> pcie-switch-upstream-port to the pcie-root-port that is plugged into the
>  >> pcie-expander-bus, and multiple pcie-switch-downstream-ports to the
>  >> pcie-switch-upstream-port". Future unit-tests should follow this
>  >> topology configuration.
>  >
>  > Ok. Could you please give me an example Qemu equivalent command
> option,
>  > if possible, for the above case. I am not that familiar with libvirt
> and I would
>  > also like to test the above scenario.
> 
> You can use "-device x3130-upstream" for the upstream switch port, and
> "-device xio3130-downstream" for the downstream port:
> 
>   -device pxb-pcie,bus_nr=250,id=pci.1,bus=pcie.0,addr=0x1 \
>   -device pcie-root-port,id=pci.2,bus=pci.1,addr=0x0 \
>   -device x3130-upstream,id=pci.3,bus=pci.2,addr=0x0 \
>   -device xio3130-
> downstream,id=pci.4,bus=pci.3,addr=0x0,chassis=17,port=1 \
>   -device vfio-pci,host=0009:01:00.0,id=hostdev0,bus=pci.4,addr=0x0 \
>   -device arm-smmuv3-nested,pci-bus=pci.1

Thanks. Just wondering why libvirt mandates usage of pcie-switch for multiple
device plugging rather than just using pcie-root-ports?

Please let me if there is any advantage in doing so that you are aware of.

Thanks,
Shameer

> -----Original Message-----
> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, February 6, 2025 6:13 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: Daniel P. Berrangé <berrange@redhat.com>; qemu-arm@nongnu.org;
> qemu-devel@nongnu.org; eric.auger@redhat.com;
> peter.maydell@linaro.org; nicolinc@nvidia.com; ddutile@redhat.com;
> Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>;
> zhangfei.gao@linaro.org; nathanc@nvidia.com
> Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
> nested SMMUv3
> 
> On Thu, Feb 06, 2025 at 06:04:57PM +0000, Shameerali Kolothum Thodi
> wrote:
> > > Some kind of hot plug smmu would have to create a vSMMU without
> any
> > > kernel backing and then later bind it to a kernel implementation.
> >
> > Not sure I get the problem with associating vSMMU with a pSMMU.
> Something
> > like an iommu instance id mentioned before,
> >
> > -device arm-smmuv3-accel,id=smmuv2,bus=pcie.2,host-smmu=iommu.1
> >
> > This can realize the vSMMU without actually creating a vIOMMU in kernel.
> > And when the dev gets attached/realized, check (GET_HW_INFO)the
> specified
> > iommu instance id matches or not.
> >
> > Or the concern here is exporting an iommu instance id to user space?
> 
> Philisophically we do not permit any HW access through iommufd without
> a VFIO fd to "prove" the process has rights to touch hardware.
> 
> We don't have any way to prove the process has rights to touch the
> iommu hardware seperately from VFIO.

It is not. Qemu just instantiates a vSMMU and assigns the IOMMU 
instance id to it.

> 
> So even if you invent an iommu ID we cannot accept it as a handle to
> create viommu in iommufd.

Creating the vIOMMU only happens when the user does a  cold/hot plug of
a VFIO device. At that time Qemu checks whether the assigned id matches
with whatever the kernel tell it.

Thanks,
Shameer