> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Thursday, November 2, 2023 8:48 PM
>
> On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote:
> > Hi folks,
> >
> > This series implements the functionality of delivering IO page faults to
> > user space through the IOMMUFD framework for nested translation.
> Nested
> > translation is a hardware feature that supports two-stage translation
> > tables for IOMMU. The second-stage translation table is managed by the
> > host VMM, while the first-stage translation table is owned by user
> > space. This allows user space to control the IOMMU mappings for its
> > devices.
> 
> Having now looked more closely at the ARM requirements it seems we
> will need generic events, not just page fault events to have a
> complete emulation.

Can you elaborate?

> 
> So I'd like to see this generalized into a channel to carry any
> events..
> 
> > User space indicates its capability of handling IO page faults by
> > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a
> > hardware page table (HWPT). IOMMUFD will then set up its infrastructure
> > for page fault delivery. On a successful return of HWPT allocation, the
> > user can retrieve and respond to page faults by reading and writing to
> > the file descriptor (FD) returned in out_fault_fd.
> 
> This is the right way to approach it, and more broadly this shouldn't
> be an iommufd specific thing. Kernel drivers will also need to create
> fault capable PAGING iommu domains.
> 

Are you suggesting a common interface used by both iommufd and
kernel drivers?

but I didn't get the last piece. If those domains are created by kernel
drivers why would they require a uAPI for userspace to specify fault
capable?

On Tue, Nov 07, 2023 at 08:35:10AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@ziepe.ca>
> > Sent: Thursday, November 2, 2023 8:48 PM
> >
> > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote:
> > > Hi folks,
> > >
> > > This series implements the functionality of delivering IO page faults to
> > > user space through the IOMMUFD framework for nested translation.
> > Nested
> > > translation is a hardware feature that supports two-stage translation
> > > tables for IOMMU. The second-stage translation table is managed by the
> > > host VMM, while the first-stage translation table is owned by user
> > > space. This allows user space to control the IOMMU mappings for its
> > > devices.
> > 
> > Having now looked more closely at the ARM requirements it seems we
> > will need generic events, not just page fault events to have a
> > complete emulation.
> 
> Can you elaborate?

There are many events related to object in guest memory or controlled
by the guest, eg C_BAD_CD and C_BAD_STE. These should be relayed or
the emulation is not working well.

> > > User space indicates its capability of handling IO page faults by
> > > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a
> > > hardware page table (HWPT). IOMMUFD will then set up its infrastructure
> > > for page fault delivery. On a successful return of HWPT allocation, the
> > > user can retrieve and respond to page faults by reading and writing to
> > > the file descriptor (FD) returned in out_fault_fd.
> > 
> > This is the right way to approach it, and more broadly this shouldn't
> > be an iommufd specific thing. Kernel drivers will also need to create
> > fault capable PAGING iommu domains.
> 
> Are you suggesting a common interface used by both iommufd and
> kernel drivers?

Yes
 
> but I didn't get the last piece. If those domains are created by kernel
> drivers why would they require a uAPI for userspace to specify fault
> capable?

Not to userspace, but a kapi to request a fault capable domain and to
supply the fault handler. Eg:

 iommu_domain_alloc_faultable(dev, handler);

Jason

> From: Jason Gunthorpe <jgg@ziepe.ca>
> Sent: Wednesday, November 8, 2023 1:54 AM
> 
> On Tue, Nov 07, 2023 at 08:35:10AM +0000, Tian, Kevin wrote:
> > > From: Jason Gunthorpe <jgg@ziepe.ca>
> > > Sent: Thursday, November 2, 2023 8:48 PM
> > >
> > > On Thu, Oct 26, 2023 at 10:49:24AM +0800, Lu Baolu wrote:
> > > > Hi folks,
> > > >
> > > > This series implements the functionality of delivering IO page faults to
> > > > user space through the IOMMUFD framework for nested translation.
> > > Nested
> > > > translation is a hardware feature that supports two-stage translation
> > > > tables for IOMMU. The second-stage translation table is managed by
> the
> > > > host VMM, while the first-stage translation table is owned by user
> > > > space. This allows user space to control the IOMMU mappings for its
> > > > devices.
> > >
> > > Having now looked more closely at the ARM requirements it seems we
> > > will need generic events, not just page fault events to have a
> > > complete emulation.
> >
> > Can you elaborate?
> 
> There are many events related to object in guest memory or controlled
> by the guest, eg C_BAD_CD and C_BAD_STE. These should be relayed or
> the emulation is not working well.

so that's the category of unrecoverable faults?

btw I can understand C_BAD_CD given it's walked by the physical SMMU
in nested configuration. But presumably STE is created by the smmu
driver itself then why would there be an error to be relayed for guest STE?

> 
> > > > User space indicates its capability of handling IO page faults by
> > > > setting the IOMMU_HWPT_ALLOC_IOPF_CAPABLE flag when allocating a
> > > > hardware page table (HWPT). IOMMUFD will then set up its
> infrastructure
> > > > for page fault delivery. On a successful return of HWPT allocation, the
> > > > user can retrieve and respond to page faults by reading and writing to
> > > > the file descriptor (FD) returned in out_fault_fd.
> > >
> > > This is the right way to approach it, and more broadly this shouldn't
> > > be an iommufd specific thing. Kernel drivers will also need to create
> > > fault capable PAGING iommu domains.
> >
> > Are you suggesting a common interface used by both iommufd and
> > kernel drivers?
> 
> Yes
> 
> > but I didn't get the last piece. If those domains are created by kernel
> > drivers why would they require a uAPI for userspace to specify fault
> > capable?
> 
> Not to userspace, but a kapi to request a fault capable domain and to
> supply the fault handler. Eg:
> 
>  iommu_domain_alloc_faultable(dev, handler);
> 

Does it affect SVA too?

On Wed, Nov 08, 2023 at 08:53:00AM +0000, Tian, Kevin wrote:

> > There are many events related to object in guest memory or controlled
> > by the guest, eg C_BAD_CD and C_BAD_STE. These should be relayed or
> > the emulation is not working well.
> 
> so that's the category of unrecoverable faults?

I haven't looked exhaustively but I do have the impression that the
only recoverable fault is the 'page not present' one.

> btw I can understand C_BAD_CD given it's walked by the physical SMMU
> in nested configuration. But presumably STE is created by the smmu
> driver itself then why would there be an error to be relayed for
> guest STE?

If the guest programs a bad STE it should still generate a C_BAD_STE
even if the mediation SW could theoretically sanitize it (but sanitize
it to what? BLOCKED?). Since we have to forward things like C_BAD_CD
and others we may as well just drop an invalid STE and forward the
event like real HW.

> > > but I didn't get the last piece. If those domains are created by kernel
> > > drivers why would they require a uAPI for userspace to specify fault
> > > capable?
> > 
> > Not to userspace, but a kapi to request a fault capable domain and to
> > supply the fault handler. Eg:
> > 
> >  iommu_domain_alloc_faultable(dev, handler);
> 
> Does it affect SVA too?

Inside the driver the SVA should be constructed out of the same fault
handling infrastructure, but a SVA domain allocation should have a
different allocation function.

Jason