Series comparison

 RE: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable nested SMMUv3
 > -----Original Message-----
-> From: Nicolin Chen <nicolinc@nvidia.com>
+> From: Peter Maydell <peter.maydell@linaro.org>
-> Sent: Thursday, January 23, 2025 4:10 AM
+> Sent: Friday, December 13, 2024 1:33 PM
-> To: Donald Dutile <ddutile@redhat.com>
+> To: Jason Gunthorpe <jgg@nvidia.com>
-> Cc: Shameerali Kolothum Thodi
+> Cc: Daniel P. Berrangé <berrange@redhat.com>; Shameerali Kolothum
-> <shameerali.kolothum.thodi@huawei.com>; eric.auger@redhat.com; Peter
+> Thodi <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
-> Maydell <peter.maydell@linaro.org>; Jason Gunthorpe <jgg@nvidia.com>;
+> qemu-devel@nongnu.org; eric.auger@redhat.com; nicolinc@nvidia.com;
-> Daniel P. Berrangé <berrange@redhat.com>; qemu-arm@nongnu.org;
+> ddutile@redhat.com; Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
-> qemu-devel@nongnu.org; Linuxarm <linuxarm@huawei.com>; Wangzhou
+> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
 > (B) <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
 > Jonathan Cameron <jonathan.cameron@huawei.com>;
 > zhangfei.gao@linaro.org
 > Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
 > nested SMMUv3
 >
-> Hi Don,
+> On Fri, 13 Dec 2024 at 12:46, Jason Gunthorpe <jgg@nvidia.com> wrote:
 > >
 > > On Fri, Dec 13, 2024 at 12:00:43PM +0000, Daniel P. Berrangé wrote:
 > > > On Fri, Nov 08, 2024 at 12:52:37PM +0000, Shameer Kolothum via wrote:
 > > > > Hi,
 > > > >
 > > > > This series adds initial support for a user-creatable "arm-smmuv3-
 > nested"
 > > > > device to Qemu. At present the Qemu ARM SMMUv3 emulation is per
 > machine
 > > > > and cannot support multiple SMMUv3s.
 > > > >
 > > > > In order to support vfio-pci dev assignment with vSMMUv3, the
 > physical
 > > > > SMMUv3 has to be configured in nested mode. Having a pluggable
 > > > > "arm-smmuv3-nested" device enables us to have multiple vSMMUv3
 > for Guests
 > > > > running on a host with multiple physical SMMUv3s. A few benefits of
 > doing
 > > > > this are,
 > > >
 > > > I'm not very familiar with arm, but from this description I'm not
 > > > really seeing how "nesting" is involved here. You're only talking
 > > > about the host and 1 L1 guest, no L2 guest.
 > >
 > > nesting is the term the iommu side is using to refer to the 2
 > > dimensional paging, ie a guest page table on top of a hypervisor page
 > > table.
 >
-> On Fri, Jan 10, 2025 at 11:05:24PM -0500, Donald Dutile wrote:
+> Isn't that more usually called "two stage" paging? Calling
-> > On 1/8/25 11:45 PM, Nicolin Chen wrote:
+> that "nesting" seems like it is going to be massively confusing...
-> > > On Mon, Dec 16, 2024 at 10:01:29AM +0000, Shameerali Kolothum Thodi
-> wrote:
+Yes. This will be renamed in future revisions as arm-smmuv3-accel.
-> > > > And patches prior to this commit adds that support:
 > > > > 4ccdbe3: ("cover-letter: Add HW accelerated nesting support for arm
 > > > > SMMUv3")
 > > > >
 > > > > Nicolin is soon going to send out those for review. Or I can include
 > > > > those in this series so that it gives a complete picture. Nicolin?
 > > >
 > > > Just found that I forgot to reply this one...sorry
 > > >
 > > > I asked Don/Eric to take over that vSMMU series:
 > > > https://lore.kernel.org/qemu-devel/Zy0jiPItu8A3wNTL@Asurada-Nvidia/
 > > > (The majority of my effort has been still on the kernel side:
 > > >   previously vIOMMU/vDEVICE, and now vEVENTQ/MSI/vCMDQ..)
 > > >
 > > > Don/Eric, is there any update from your side?
 > > >
 > > Apologies for delayed response, been at customer site, and haven't been
 > keeping up w/biz email.
 > > Eric is probably waiting for me to get back and chat as well.
 > > Will look to reply early next week.
 >
-> I wonder if we can make some progress in Feb? If so, we can start
+> Also, how does it relate to what this series seems to be
-> to wrap up the iommufd uAPI patches for HWPT, which was a part of
+> doing, where we provide the guest with two separate SMMUs?
-> intel's series but never got sent since their emulated series is
+> (Are those two SMMUs "nested" in the sense that one is sitting
-> seemingly still pending?
+> behind the other?)
-I think these are the  5 patches that we require from Intel pass-through series,
+I don't think it requires two SMMUs in Guest. The nested or "two
 stage" means the stage 1 page table is owned by Guest and stage 2
 by host. And this is achieved by IOMMUFD provided IOCTLs.
-vfio/iommufd: Implement [at|de]tach_hwpt handlers
+There is a precurser to this series where the support for hw accelerated
-vfio/iommufd: Implement HostIOMMUDeviceClass::realize_late() handler
+stage support is added in Qemu SMMUv3 code.
 HostIOMMUDevice: Introduce realize_late callback
 vfio/iommufd: Add properties and handlers to TYPE_HOST_IOMMU_DEVICE_IOMMUFD
 backends/iommufd: Add helpers for invalidating user-managed HWPT
-See the commits from here,
+Please see the complete branch here,
-https://github.com/hisilicon/qemu/commit/bbdc65af38fa5723f1bd9b026e292730901f57b5
+https://github.com/hisilicon/qemu/commits/private-smmuv3-nested-dev-rfc-v1/
 And patches prior to this commit adds that support:
 ccdbe3: ("cover-letter: Add HW accelerated nesting support for arm
 SMMUv3")
-[CC  Zhenzhong]
+Nicolin is soon going to send out those for review. Or I can include
 those in this series so that it gives a complete picture. Nicolin?
-Hi Zhenzhong,
+Hope this clarifies any confusion.
 Just wondering what your plans are for the above patches.  If it make sense and you
 are fine with it, I think it is a good idea one of us can pick up those from that series
 and sent out separately so that it can get some review and take it forward.
 Thanks,
 Shameer

> -----Original Message-----
> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Thursday, January 23, 2025 4:10 AM
> To: Donald Dutile <ddutile@redhat.com>
> Cc: Shameerali Kolothum Thodi
> <shameerali.kolothum.thodi@huawei.com>; eric.auger@redhat.com; Peter
> Maydell <peter.maydell@linaro.org>; Jason Gunthorpe <jgg@nvidia.com>;
> Daniel P. Berrangé <berrange@redhat.com>; qemu-arm@nongnu.org;
> qemu-devel@nongnu.org; Linuxarm <linuxarm@huawei.com>; Wangzhou
> (B) <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>;
> zhangfei.gao@linaro.org
> Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
> nested SMMUv3
> 
> Hi Don,
> 
> On Fri, Jan 10, 2025 at 11:05:24PM -0500, Donald Dutile wrote:
> > On 1/8/25 11:45 PM, Nicolin Chen wrote:
> > > On Mon, Dec 16, 2024 at 10:01:29AM +0000, Shameerali Kolothum Thodi
> wrote:
> > > > And patches prior to this commit adds that support:
> > > > 4ccdbe3: ("cover-letter: Add HW accelerated nesting support for arm
> > > > SMMUv3")
> > > >
> > > > Nicolin is soon going to send out those for review. Or I can include
> > > > those in this series so that it gives a complete picture. Nicolin?
> > >
> > > Just found that I forgot to reply this one...sorry
> > >
> > > I asked Don/Eric to take over that vSMMU series:
> > > https://lore.kernel.org/qemu-devel/Zy0jiPItu8A3wNTL@Asurada-Nvidia/
> > > (The majority of my effort has been still on the kernel side:
> > >   previously vIOMMU/vDEVICE, and now vEVENTQ/MSI/vCMDQ..)
> > >
> > > Don/Eric, is there any update from your side?
> > >
> > Apologies for delayed response, been at customer site, and haven't been
> keeping up w/biz email.
> > Eric is probably waiting for me to get back and chat as well.
> > Will look to reply early next week.
> 
> I wonder if we can make some progress in Feb? If so, we can start
> to wrap up the iommufd uAPI patches for HWPT, which was a part of
> intel's series but never got sent since their emulated series is
> seemingly still pending?

I think these are the  5 patches that we require from Intel pass-through series,

vfio/iommufd: Implement [at|de]tach_hwpt handlers
vfio/iommufd: Implement HostIOMMUDeviceClass::realize_late() handler
HostIOMMUDevice: Introduce realize_late callback
vfio/iommufd: Add properties and handlers to TYPE_HOST_IOMMU_DEVICE_IOMMUFD
backends/iommufd: Add helpers for invalidating user-managed HWPT

See the commits from here,
https://github.com/hisilicon/qemu/commit/bbdc65af38fa5723f1bd9b026e292730901f57b5

[CC  Zhenzhong]

Hi Zhenzhong,

Just wondering what your plans are for the above patches.  If it make sense and you
are fine with it, I think it is a good idea one of us can pick up those from that series
and sent out separately so that it can get some review and take it forward.

Thanks,
Shameer

> -----Original Message-----
> From: Peter Maydell <peter.maydell@linaro.org>
> Sent: Friday, December 13, 2024 1:33 PM
> To: Jason Gunthorpe <jgg@nvidia.com>
> Cc: Daniel P. Berrangé <berrange@redhat.com>; Shameerali Kolothum
> Thodi <shameerali.kolothum.thodi@huawei.com>; qemu-arm@nongnu.org;
> qemu-devel@nongnu.org; eric.auger@redhat.com; nicolinc@nvidia.com;
> ddutile@redhat.com; Linuxarm <linuxarm@huawei.com>; Wangzhou (B)
> <wangzhou1@hisilicon.com>; jiangkunkun <jiangkunkun@huawei.com>;
> Jonathan Cameron <jonathan.cameron@huawei.com>;
> zhangfei.gao@linaro.org
> Subject: Re: [RFC PATCH 0/5] hw/arm/virt: Add support for user-creatable
> nested SMMUv3
> 
> On Fri, 13 Dec 2024 at 12:46, Jason Gunthorpe <jgg@nvidia.com> wrote:
> >
> > On Fri, Dec 13, 2024 at 12:00:43PM +0000, Daniel P. Berrangé wrote:
> > > On Fri, Nov 08, 2024 at 12:52:37PM +0000, Shameer Kolothum via wrote:
> > > > Hi,
> > > >
> > > > This series adds initial support for a user-creatable "arm-smmuv3-
> nested"
> > > > device to Qemu. At present the Qemu ARM SMMUv3 emulation is per
> machine
> > > > and cannot support multiple SMMUv3s.
> > > >
> > > > In order to support vfio-pci dev assignment with vSMMUv3, the
> physical
> > > > SMMUv3 has to be configured in nested mode. Having a pluggable
> > > > "arm-smmuv3-nested" device enables us to have multiple vSMMUv3
> for Guests
> > > > running on a host with multiple physical SMMUv3s. A few benefits of
> doing
> > > > this are,
> > >
> > > I'm not very familiar with arm, but from this description I'm not
> > > really seeing how "nesting" is involved here. You're only talking
> > > about the host and 1 L1 guest, no L2 guest.
> >
> > nesting is the term the iommu side is using to refer to the 2
> > dimensional paging, ie a guest page table on top of a hypervisor page
> > table.
> 
> Isn't that more usually called "two stage" paging? Calling
> that "nesting" seems like it is going to be massively confusing...

Yes. This will be renamed in future revisions as arm-smmuv3-accel.

> 
> Also, how does it relate to what this series seems to be
> doing, where we provide the guest with two separate SMMUs?
> (Are those two SMMUs "nested" in the sense that one is sitting
> behind the other?)

I don't think it requires two SMMUs in Guest. The nested or "two
stage" means the stage 1 page table is owned by Guest and stage 2
by host. And this is achieved by IOMMUFD provided IOCTLs.

There is a precurser to this series where the support for hw accelerated
2 stage support is added in Qemu SMMUv3 code.

Please see the complete branch here,
https://github.com/hisilicon/qemu/commits/private-smmuv3-nested-dev-rfc-v1/
And patches prior to this commit adds that support: 
4ccdbe3: ("cover-letter: Add HW accelerated nesting support for arm
SMMUv3")

Nicolin is soon going to send out those for review. Or I can include
those in this series so that it gives a complete picture. Nicolin?

Hope this clarifies any confusion.

Thanks,
Shameer