Cases like VFIO wish to attach a device to an existing domain that was
not allocated specifically from the device. This raises a condition
where the IOMMU driver can fail the domain attach because the domain and
device are incompatible with each other.

This is a soft failure that can be resolved by using a different domain.

Provide a dedicated errno EINVAL from the IOMMU driver during attach that
the reason attached failed is because of domain incompatability.

VFIO can use this to know attach is a soft failure and it should continue
searching. Otherwise the attach will be a hard failure and VFIO will
return the code to userspace.

Update kdocs first to add rules of return errno to ->attach_dev ops.

Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
 include/linux/iommu.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index ea30f00dc145..c5d7ec0187c7 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -266,6 +266,17 @@ struct iommu_ops {
 /**
  * struct iommu_domain_ops - domain specific operations
  * @attach_dev: attach an iommu domain to a device
+ *              Rules of its return errno:
+ *               EINVAL  - Exclusively, device and domain are incompatible. Must
+ *                         avoid kernel prints along with this errno. Any EINVAL
+ *                         returned from kAPIs must be converted to ENODEV if it
+ *                         is device-specific, or to some other reasonable errno
+ *                         being listed below
+ *               ENOMEM  - Out of memory
+ *               ENOSPC  - No space left on device
+ *               EBUSY   - Device is attached to a domain and cannot be changed
+ *               ENODEV  - Device specific errors, not able to be attached
+ *              <others> - Treated as ENODEV by the caller. Use is discouraged
  * @detach_dev: detach an iommu domain from a device
  * @map: map a physically contiguous memory region to an iommu domain
  * @map_pages: map a physically contiguous set of pages of the same size to
-- 
2.17.1

On 9/13/2022 1:24 AM, Nicolin Chen wrote:
> Cases like VFIO wish to attach a device to an existing domain that was
> not allocated specifically from the device. This raises a condition
> where the IOMMU driver can fail the domain attach because the domain and
> device are incompatible with each other.
> 
> This is a soft failure that can be resolved by using a different domain.
> 
> Provide a dedicated errno EINVAL from the IOMMU driver during attach that
> the reason attached failed is because of domain incompatability.
> 
> VFIO can use this to know attach is a soft failure and it should continue
> searching. Otherwise the attach will be a hard failure and VFIO will
> return the code to userspace.
> 
> Update kdocs first to add rules of return errno to ->attach_dev ops.
> 
> Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---
>   include/linux/iommu.h | 11 +++++++++++
>   1 file changed, 11 insertions(+)
> 
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index ea30f00dc145..c5d7ec0187c7 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -266,6 +266,17 @@ struct iommu_ops {
>   /**
>    * struct iommu_domain_ops - domain specific operations
>    * @attach_dev: attach an iommu domain to a device
> + *              Rules of its return errno:
> + *               EINVAL  - Exclusively, device and domain are incompatible. Must
> + *                         avoid kernel prints along with this errno. Any EINVAL
> + *                         returned from kAPIs must be converted to ENODEV if it
> + *                         is device-specific, or to some other reasonable errno
> + *                         being listed below
> + *               ENOMEM  - Out of memory
> + *               ENOSPC  - No space left on device
> + *               EBUSY   - Device is attached to a domain and cannot be changed
> + *               ENODEV  - Device specific errors, not able to be attached
> + *              <others> - Treated as ENODEV by the caller. Use is discouraged

It is my understanding that kernel-doc won't preserve your formatting 
without extra directives. See the "Note" after 
<https://www.kernel.org/doc/html/latest/doc-guide/kernel-doc.html#return-values>


>    * @detach_dev: detach an iommu domain from a device
>    * @map: map a physically contiguous memory region to an iommu domain
>    * @map_pages: map a physically contiguous set of pages of the same size to

Hi Jeff,

On Tue, Sep 13, 2022 at 11:41:35AM -0700, Jeff Johnson wrote:
> > diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> > index ea30f00dc145..c5d7ec0187c7 100644
> > --- a/include/linux/iommu.h
> > +++ b/include/linux/iommu.h
> > @@ -266,6 +266,17 @@ struct iommu_ops {
> >   /**
> >    * struct iommu_domain_ops - domain specific operations
> >    * @attach_dev: attach an iommu domain to a device
> > + *              Rules of its return errno:
> > + *               EINVAL  - Exclusively, device and domain are incompatible. Must
> > + *                         avoid kernel prints along with this errno. Any EINVAL
> > + *                         returned from kAPIs must be converted to ENODEV if it
> > + *                         is device-specific, or to some other reasonable errno
> > + *                         being listed below
> > + *               ENOMEM  - Out of memory
> > + *               ENOSPC  - No space left on device
> > + *               EBUSY   - Device is attached to a domain and cannot be changed
> > + *               ENODEV  - Device specific errors, not able to be attached
> > + *              <others> - Treated as ENODEV by the caller. Use is discouraged
> 
> It is my understanding that kernel-doc won't preserve your formatting
> without extra directives. See the "Note" after
> <https://www.kernel.org/doc/html/latest/doc-guide/kernel-doc.html#return-values>

Thanks for the review!

Perhaps the following one would work:

diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index ea30f00dc145..14f078b58e17 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -266,6 +266,17 @@ struct iommu_ops {
 /**
  * struct iommu_domain_ops - domain specific operations
  * @attach_dev: attach an iommu domain to a device
+ *  Return:
+ * * 0         - success
+ * * EINVAL    - Exclusively, device and domain are incompatible. Must avoid
+ *               kernel prints along with this errno. Any EINVAL returned from
+ *               a kAPI must be converted to ENODEV if it is device-specific,
+ *               or to some other reasonable errno being listed below
+ * * ENOMEM    - Out of memory
+ * * ENOSPC    - No space left on device
+ * * EBUSY     - Device is attached to a domain and cannot be changed
+ * * ENODEV    - Device specific errors, not able to be attached
+ * * <others>  - Treated as ENODEV by the caller. Use is discouraged
  * @detach_dev: detach an iommu domain from a device
  * @map: map a physically contiguous memory region to an iommu domain
  * @map_pages: map a physically contiguous set of pages of the same size to