There is a need to stage a resetting PCI device to temporally the blocked
domain and then attach back to its previously attached domain after reset.
This can be simply done by keeping the "previously attached domain" in the
iommu_group->domain pointer while adding an iommu_group->resetting_domain,
which gives troubles to IOMMU drivers using the iommu_get_domain_for_dev()
for a device's physical domain in order to program IOMMU hardware.
And in such for-driver use cases, the iommu_group->mutex must be held, so
it doesn't fit in external callers that don't hold the iommu_group->mutex.
Introduce a new iommu_driver_get_domain_for_dev() helper, exclusively for
driver use cases that hold the iommu_group->mutex, to separate from those
external use cases.
Add a lockdep_assert_not_held to the existing iommu_get_domain_for_dev()
and highlight that in a kdoc.
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
---
include/linux/iommu.h | 1 +
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++--
drivers/iommu/iommu.c | 28 +++++++++++++++++++++
3 files changed, 32 insertions(+), 2 deletions(-)
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 801b2bd9e8d49..a42a2d1d7a0b7 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -910,6 +910,7 @@ extern int iommu_attach_device(struct iommu_domain *domain,
extern void iommu_detach_device(struct iommu_domain *domain,
struct device *dev);
extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev);
+struct iommu_domain *iommu_driver_get_domain_for_dev(struct device *dev);
extern struct iommu_domain *iommu_get_dma_domain(struct device *dev);
extern int iommu_map(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, size_t size, int prot, gfp_t gfp);
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index a33fbd12a0dd9..412d1a9b31275 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -3125,7 +3125,8 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
struct arm_smmu_cd *cd, struct iommu_domain *old)
{
- struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev);
+ struct iommu_domain *sid_domain =
+ iommu_driver_get_domain_for_dev(master->dev);
struct arm_smmu_attach_state state = {
.master = master,
.ssid = pasid,
@@ -3191,7 +3192,7 @@ static int arm_smmu_blocking_set_dev_pasid(struct iommu_domain *new_domain,
*/
if (!arm_smmu_ssids_in_use(&master->cd_table)) {
struct iommu_domain *sid_domain =
- iommu_get_domain_for_dev(master->dev);
+ iommu_driver_get_domain_for_dev(master->dev);
if (sid_domain->type == IOMMU_DOMAIN_IDENTITY ||
sid_domain->type == IOMMU_DOMAIN_BLOCKED)
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index 1e322f87b1710..1f4d6ca0937bc 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -2217,6 +2217,15 @@ void iommu_detach_device(struct iommu_domain *domain, struct device *dev)
}
EXPORT_SYMBOL_GPL(iommu_detach_device);
+/**
+ * iommu_get_domain_for_dev() - Return the DMA API domain pointer
+ * @dev - Device to query
+ *
+ * This function can be called within a driver bound to dev. The returned
+ * pointer is valid for the lifetime of the bound driver.
+ *
+ * It should not be called by drivers with driver_managed_dma = true.
+ */
struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
{
/* Caller must be a probed driver on dev */
@@ -2225,10 +2234,29 @@ struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
if (!group)
return NULL;
+ lockdep_assert_not_held(&group->mutex);
+
return group->domain;
}
EXPORT_SYMBOL_GPL(iommu_get_domain_for_dev);
+/**
+ * iommu_driver_get_domain_for_dev() - Return the driver-level domain pointer
+ * @dev - Device to query
+ *
+ * This function can be called by an iommu driver that wants to get the physical
+ * domain within an iommu callback function where group->mutex is held.
+ */
+struct iommu_domain *iommu_driver_get_domain_for_dev(struct device *dev)
+{
+ struct iommu_group *group = dev->iommu_group;
+
+ lockdep_assert_held(&group->mutex);
+
+ return group->domain;
+}
+EXPORT_SYMBOL_GPL(iommu_driver_get_domain_for_dev);
+
/*
* For IOMMU_DOMAIN_DMA implementations which already provide their own
* guarantees that the group and its default domain are valid and correct.
--
2.43.0> From: Nicolin Chen <nicolinc@nvidia.com> > Sent: Tuesday, November 11, 2025 1:13 PM > > There is a need to stage a resetting PCI device to temporally the blocked s/temporally/temporarily/
Hi Nicolin, kernel test robot noticed the following build warnings: [auto build test WARNING on next-20251110] [cannot apply to pci/next pci/for-linus awilliam-vfio/next awilliam-vfio/for-linus rafael-pm/linux-next rafael-pm/bleeding-edge linus/master v6.18-rc5 v6.18-rc4 v6.18-rc3 v6.18-rc5] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch#_base_tree_information] url: https://github.com/intel-lab-lkp/linux/commits/Nicolin-Chen/iommu-Lock-group-mutex-in-iommu_deferred_attach/20251111-131520 base: next-20251110 patch link: https://lore.kernel.org/r/0303739735f3f49bcebc244804e9eeb82b1c41dc.1762835355.git.nicolinc%40nvidia.com patch subject: [PATCH v5 3/5] iommu: Add iommu_driver_get_domain_for_dev() helper config: microblaze-randconfig-r072-20251112 (https://download.01.org/0day-ci/archive/20251112/202511121645.Oi9e0lrt-lkp@intel.com/config) compiler: microblaze-linux-gcc (GCC) 8.5.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251112/202511121645.Oi9e0lrt-lkp@intel.com/reproduce) If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags | Reported-by: kernel test robot <lkp@intel.com> | Closes: https://lore.kernel.org/oe-kbuild-all/202511121645.Oi9e0lrt-lkp@intel.com/ All warnings (new ones prefixed by >>): >> Warning: drivers/iommu/iommu.c:2229 function parameter 'dev' not described in 'iommu_get_domain_for_dev' >> Warning: drivers/iommu/iommu.c:2250 function parameter 'dev' not described in 'iommu_driver_get_domain_for_dev' >> Warning: drivers/iommu/iommu.c:2229 function parameter 'dev' not described in 'iommu_get_domain_for_dev' >> Warning: drivers/iommu/iommu.c:2250 function parameter 'dev' not described in 'iommu_driver_get_domain_for_dev' -- 0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
On 11/11/25 13:12, Nicolin Chen wrote:
> There is a need to stage a resetting PCI device to temporally the blocked
> domain and then attach back to its previously attached domain after reset.
>
> This can be simply done by keeping the "previously attached domain" in the
> iommu_group->domain pointer while adding an iommu_group->resetting_domain,
> which gives troubles to IOMMU drivers using the iommu_get_domain_for_dev()
> for a device's physical domain in order to program IOMMU hardware.
>
> And in such for-driver use cases, the iommu_group->mutex must be held, so
> it doesn't fit in external callers that don't hold the iommu_group->mutex.
>
> Introduce a new iommu_driver_get_domain_for_dev() helper, exclusively for
> driver use cases that hold the iommu_group->mutex, to separate from those
> external use cases.
>
> Add a lockdep_assert_not_held to the existing iommu_get_domain_for_dev()
> and highlight that in a kdoc.
>
> Reviewed-by: Kevin Tian <kevin.tian@intel.com>
> Signed-off-by: Nicolin Chen <nicolinc@nvidia.com>
> ---
> include/linux/iommu.h | 1 +
> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 5 ++--
> drivers/iommu/iommu.c | 28 +++++++++++++++++++++
> 3 files changed, 32 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/iommu.h b/include/linux/iommu.h
> index 801b2bd9e8d49..a42a2d1d7a0b7 100644
> --- a/include/linux/iommu.h
> +++ b/include/linux/iommu.h
> @@ -910,6 +910,7 @@ extern int iommu_attach_device(struct iommu_domain *domain,
> extern void iommu_detach_device(struct iommu_domain *domain,
> struct device *dev);
> extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev);
> +struct iommu_domain *iommu_driver_get_domain_for_dev(struct device *dev);
> extern struct iommu_domain *iommu_get_dma_domain(struct device *dev);
> extern int iommu_map(struct iommu_domain *domain, unsigned long iova,
> phys_addr_t paddr, size_t size, int prot, gfp_t gfp);
> diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> index a33fbd12a0dd9..412d1a9b31275 100644
> --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
> @@ -3125,7 +3125,8 @@ int arm_smmu_set_pasid(struct arm_smmu_master *master,
> struct arm_smmu_domain *smmu_domain, ioasid_t pasid,
> struct arm_smmu_cd *cd, struct iommu_domain *old)
> {
> - struct iommu_domain *sid_domain = iommu_get_domain_for_dev(master->dev);
> + struct iommu_domain *sid_domain =
> + iommu_driver_get_domain_for_dev(master->dev);
> struct arm_smmu_attach_state state = {
> .master = master,
> .ssid = pasid,
> @@ -3191,7 +3192,7 @@ static int arm_smmu_blocking_set_dev_pasid(struct iommu_domain *new_domain,
> */
> if (!arm_smmu_ssids_in_use(&master->cd_table)) {
> struct iommu_domain *sid_domain =
> - iommu_get_domain_for_dev(master->dev);
> + iommu_driver_get_domain_for_dev(master->dev);
>
> if (sid_domain->type == IOMMU_DOMAIN_IDENTITY ||
> sid_domain->type == IOMMU_DOMAIN_BLOCKED)
> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
> index 1e322f87b1710..1f4d6ca0937bc 100644
> --- a/drivers/iommu/iommu.c
> +++ b/drivers/iommu/iommu.c
> @@ -2217,6 +2217,15 @@ void iommu_detach_device(struct iommu_domain *domain, struct device *dev)
> }
> EXPORT_SYMBOL_GPL(iommu_detach_device);
>
> +/**
> + * iommu_get_domain_for_dev() - Return the DMA API domain pointer
> + * @dev - Device to query
> + *
> + * This function can be called within a driver bound to dev. The returned
> + * pointer is valid for the lifetime of the bound driver.
> + *
> + * It should not be called by drivers with driver_managed_dma = true.
"driver_managed_dma != true" means the driver will use the default
domain allocated by the iommu core during iommu probe. The iommu core
ensures that this domain stored at group->domain will not be changed
during the driver's whole lifecycle. That's reasonable.
How about making some code to enforce this requirement? Something like
below ...
> + */
> struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
> {
> /* Caller must be a probed driver on dev */
> @@ -2225,10 +2234,29 @@ struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
> if (!group)
> return NULL;
>
> + lockdep_assert_not_held(&group->mutex);
...
if (WARN_ON(!dev->driver || !group->owner_cnt || group->owner))
return NULL;
> +
> return group->domain;
> }
> EXPORT_SYMBOL_GPL(iommu_get_domain_for_dev);
>
> +/**
> + * iommu_driver_get_domain_for_dev() - Return the driver-level domain pointer
> + * @dev - Device to query
> + *
> + * This function can be called by an iommu driver that wants to get the physical
> + * domain within an iommu callback function where group->mutex is held.
> + */
> +struct iommu_domain *iommu_driver_get_domain_for_dev(struct device *dev)
> +{
> + struct iommu_group *group = dev->iommu_group;
> +
> + lockdep_assert_held(&group->mutex);
> +
> + return group->domain;
> +}
> +EXPORT_SYMBOL_GPL(iommu_driver_get_domain_for_dev);
> +
> /*
> * For IOMMU_DOMAIN_DMA implementations which already provide their own
> * guarantees that the group and its default domain are valid and correct.
Others look good to me.
Reviewed-by: Lu Baolu <baolu.lu@linux.intel.com>
Hi Baolu,
On Wed, Nov 12, 2025 at 01:58:51PM +0800, Baolu Lu wrote:
> On 11/11/25 13:12, Nicolin Chen wrote:
> > +/**
> > + * iommu_get_domain_for_dev() - Return the DMA API domain pointer
> > + * @dev - Device to query
> > + *
> > + * This function can be called within a driver bound to dev. The returned
> > + * pointer is valid for the lifetime of the bound driver.
> > + *
> > + * It should not be called by drivers with driver_managed_dma = true.
>
> "driver_managed_dma != true" means the driver will use the default
> domain allocated by the iommu core during iommu probe.
Hmm, I am not very sure. Jason's remarks pointed out that There
is an exception in host1x_client_iommu_detach():
https://lore.kernel.org/all/20250924191055.GJ2617119@nvidia.com/
Where the group->domain could be NULL, i.e. not attached to the
default domain?
> The iommu core
> ensures that this domain stored at group->domain will not be changed
> during the driver's whole lifecycle. That's reasonable.
>
> How about making some code to enforce this requirement? Something like
> below ...
>
> > + */
> > struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
> > {
> > /* Caller must be a probed driver on dev */
> > @@ -2225,10 +2234,29 @@ struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
> > if (!group)
> > return NULL;
> > + lockdep_assert_not_held(&group->mutex);
>
> ...
> if (WARN_ON(!dev->driver || !group->owner_cnt || group->owner))
> return NULL;
With that, could host1x_client_iommu_detach() trigger WARN_ON?
Thanks!
Nicolin
On Wed, Nov 12, 2025 at 09:41:20AM -0800, Nicolin Chen wrote:
> Hi Baolu,
>
> On Wed, Nov 12, 2025 at 01:58:51PM +0800, Baolu Lu wrote:
> > On 11/11/25 13:12, Nicolin Chen wrote:
> > > +/**
> > > + * iommu_get_domain_for_dev() - Return the DMA API domain pointer
> > > + * @dev - Device to query
> > > + *
> > > + * This function can be called within a driver bound to dev. The returned
> > > + * pointer is valid for the lifetime of the bound driver.
> > > + *
> > > + * It should not be called by drivers with driver_managed_dma = true.
> >
> > "driver_managed_dma != true" means the driver will use the default
> > domain allocated by the iommu core during iommu probe.
>
> Hmm, I am not very sure. Jason's remarks pointed out that There
> is an exception in host1x_client_iommu_detach():
> https://lore.kernel.org/all/20250924191055.GJ2617119@nvidia.com/
>
> Where the group->domain could be NULL, i.e. not attached to the
> default domain?
That is impossible these days, the group->domain is always something.
For host1x it changes the domain and ignores the driver_managed_dma
safety mechanism to do it, so it is kind of broken.
> > > + */
> > > struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
> > > {
> > > /* Caller must be a probed driver on dev */
> > > @@ -2225,10 +2234,29 @@ struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
> > > if (!group)
> > > return NULL;
> > > + lockdep_assert_not_held(&group->mutex);
> >
> > ...
> > if (WARN_ON(!dev->driver || !group->owner_cnt || group->owner))
> > return NULL;
>
> With that, could host1x_client_iommu_detach() trigger WARN_ON?
I don't really know but I strongly suspect that host1x has a NULL
dev->driver in some cases.
I think the best you could do is detect that a driver is bound and
that driver_managed_dma != true, host1x should still be OK with it,
but in practice it only effects VFIO.
Jason
On Wed, Nov 12, 2025 at 09:41:25AM -0800, Nicolin Chen wrote:
> Hi Baolu,
>
> On Wed, Nov 12, 2025 at 01:58:51PM +0800, Baolu Lu wrote:
> > On 11/11/25 13:12, Nicolin Chen wrote:
> > > +/**
> > > + * iommu_get_domain_for_dev() - Return the DMA API domain pointer
> > > + * @dev - Device to query
> > > + *
> > > + * This function can be called within a driver bound to dev. The returned
> > > + * pointer is valid for the lifetime of the bound driver.
> > > + *
> > > + * It should not be called by drivers with driver_managed_dma = true.
> >
> > "driver_managed_dma != true" means the driver will use the default
> > domain allocated by the iommu core during iommu probe.
>
> Hmm, I am not very sure. Jason's remarks pointed out that There
> is an exception in host1x_client_iommu_detach():
> https://lore.kernel.org/all/20250924191055.GJ2617119@nvidia.com/
>
> Where the group->domain could be NULL, i.e. not attached to the
> default domain?
>
> > The iommu core
> > ensures that this domain stored at group->domain will not be changed
> > during the driver's whole lifecycle. That's reasonable.
> >
> > How about making some code to enforce this requirement? Something like
> > below ...
> >
> > > + */
> > > struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
> > > {
> > > /* Caller must be a probed driver on dev */
> > > @@ -2225,10 +2234,29 @@ struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
> > > if (!group)
> > > return NULL;
> > > + lockdep_assert_not_held(&group->mutex);
> >
> > ...
> > if (WARN_ON(!dev->driver || !group->owner_cnt || group->owner))
> > return NULL;
>
> With that, could host1x_client_iommu_detach() trigger WARN_ON?
Hi Baolu,
For v6, I tend to keep this API as-is, trying not to give troubles
to existing callers. Jason suggested a potential followup series:
https://lore.kernel.org/linux-iommu/20250821131304.GM802098@nvidia.com/
That would replace this function, so maybe we can think about that.
If you have a strong feeling about the WARN_ON, please let me know.
Thanks
Nicolin
On 11/18/25 15:02, Nicolin Chen wrote:
> On Wed, Nov 12, 2025 at 09:41:25AM -0800, Nicolin Chen wrote:
>> Hi Baolu,
>>
>> On Wed, Nov 12, 2025 at 01:58:51PM +0800, Baolu Lu wrote:
>>> On 11/11/25 13:12, Nicolin Chen wrote:
>>>> +/**
>>>> + * iommu_get_domain_for_dev() - Return the DMA API domain pointer
>>>> + * @dev - Device to query
>>>> + *
>>>> + * This function can be called within a driver bound to dev. The returned
>>>> + * pointer is valid for the lifetime of the bound driver.
>>>> + *
>>>> + * It should not be called by drivers with driver_managed_dma = true.
>>>
>>> "driver_managed_dma != true" means the driver will use the default
>>> domain allocated by the iommu core during iommu probe.
>>
>> Hmm, I am not very sure. Jason's remarks pointed out that There
>> is an exception in host1x_client_iommu_detach():
>> https://lore.kernel.org/all/20250924191055.GJ2617119@nvidia.com/
>>
>> Where the group->domain could be NULL, i.e. not attached to the
>> default domain?
>>
>>> The iommu core
>>> ensures that this domain stored at group->domain will not be changed
>>> during the driver's whole lifecycle. That's reasonable.
>>>
>>> How about making some code to enforce this requirement? Something like
>>> below ...
>>>
>>>> + */
>>>> struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
>>>> {
>>>> /* Caller must be a probed driver on dev */
>>>> @@ -2225,10 +2234,29 @@ struct iommu_domain *iommu_get_domain_for_dev(struct device *dev)
>>>> if (!group)
>>>> return NULL;
>>>> + lockdep_assert_not_held(&group->mutex);
>>>
>>> ...
>>> if (WARN_ON(!dev->driver || !group->owner_cnt || group->owner))
>>> return NULL;
>>
>> With that, could host1x_client_iommu_detach() trigger WARN_ON?
>
> Hi Baolu,
>
> For v6, I tend to keep this API as-is, trying not to give troubles
> to existing callers. Jason suggested a potential followup series:
> https://lore.kernel.org/linux-iommu/20250821131304.GM802098@nvidia.com/
> That would replace this function, so maybe we can think about that.
>
> If you have a strong feeling about the WARN_ON, please let me know.
>
> Thanks
> Nicolin
No strong feeling. I am fine with it because the comments have already
stated that "This function can be called within a driver bound to dev.".
Thanks,
baolu
On Wed, Nov 19, 2025 at 10:47:26AM +0800, Baolu Lu wrote: > On 11/18/25 15:02, Nicolin Chen wrote: > > For v6, I tend to keep this API as-is, trying not to give troubles > > to existing callers. Jason suggested a potential followup series: > > https://lore.kernel.org/linux-iommu/20250821131304.GM802098@nvidia.com/ > > That would replace this function, so maybe we can think about that. > > > > If you have a strong feeling about the WARN_ON, please let me know. > > > No strong feeling. I am fine with it because the comments have already > stated that "This function can be called within a driver bound to dev.". Ack. Thanks! Nicolin
© 2016 - 2026 Red Hat, Inc.