iommu/dma: Reserve iova ranges for reserved regions of all devices

[PATCH] iommu/dma: Reserve iova ranges for reserved regions of all devices

Posted by Jerry Snitselaar 1 month ago

Only the first device that is passed when the domain is set up will
have its reserved regions reserved in the iova address space.  So if
there are other devices in the group with unique reserved regions,
those regions will not get reserved in the iova address space.  All of
the ranges do get set up in the iopagetables via calls to
iommu_create_device_direct_mappings for all devices in a group.

In the case of vt-d system this resulted in messages like the following:

[ 1632.693264] DMAR: ERROR: DMA PTE for vPFN 0xf1f7e already set (to f1f7e003 not 173025001)

To make sure iova ranges are reserved for the reserved regions all of
the devices, call iova_reserve_iommu_regions in iommu_dma_init_domain
prior to exiting in the case where the domain is already initialized.

Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Joerg Roedel <joro@8bytes.org>
Cc: Will Deacon <will@kernel.org>
Cc: linux-kernel@vger.kernel.org
Cc: stable@vger.kernel.org
Fixes: 7c1b058c8b5a ("iommu/dma: Handle IOMMU API reserved regions")
Signed-off-by: Jerry Snitselaar <jsnitsel@redhat.com>
---
Robin: I wasn't positive if this is the correct solution, or if it should be
       done for the entire group at once.

 drivers/iommu/dma-iommu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 2a9fa0c8cc00..5fd3cccbb233 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -707,7 +707,7 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, struct device *dev
 			goto done_unlock;
 		}
 
-		ret = 0;
+		ret = iova_reserve_iommu_regions(dev, domain);
 		goto done_unlock;
 	}
 
-- 
2.44.0

Re: [PATCH] iommu/dma: Reserve iova ranges for reserved regions of all devices

Posted by Jerry Snitselaar 2 weeks, 4 days ago

On Thu, Oct 24, 2024 at 08:34:12AM -0700, Jerry Snitselaar wrote:
> Only the first device that is passed when the domain is set up will
> have its reserved regions reserved in the iova address space.  So if
> there are other devices in the group with unique reserved regions,
> those regions will not get reserved in the iova address space.  All of
> the ranges do get set up in the iopagetables via calls to
> iommu_create_device_direct_mappings for all devices in a group.
> 
> In the case of vt-d system this resulted in messages like the following:
> 
> [ 1632.693264] DMAR: ERROR: DMA PTE for vPFN 0xf1f7e already set (to f1f7e003 not 173025001)
> 
> To make sure iova ranges are reserved for the reserved regions all of
> the devices, call iova_reserve_iommu_regions in iommu_dma_init_domain
> prior to exiting in the case where the domain is already initialized.
> 
> Cc: Robin Murphy <robin.murphy@arm.com>
> Cc: Joerg Roedel <joro@8bytes.org>
> Cc: Will Deacon <will@kernel.org>
> Cc: linux-kernel@vger.kernel.org
> Cc: stable@vger.kernel.org
> Fixes: 7c1b058c8b5a ("iommu/dma: Handle IOMMU API reserved regions")
> Signed-off-by: Jerry Snitselaar <jsnitsel@redhat.com>
> ---
> Robin: I wasn't positive if this is the correct solution, or if it should be
>        done for the entire group at once.
> 
>  drivers/iommu/dma-iommu.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
> index 2a9fa0c8cc00..5fd3cccbb233 100644
> --- a/drivers/iommu/dma-iommu.c
> +++ b/drivers/iommu/dma-iommu.c
> @@ -707,7 +707,7 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, struct device *dev
>  			goto done_unlock;
>  		}
>  
> -		ret = 0;
> +		ret = iova_reserve_iommu_regions(dev, domain);
>  		goto done_unlock;
>  	}
>  
> -- 
> 2.44.0
> 

Robin,

Any thoughts on this patch? In the case where this originally popped
up it was likely a crap DMAR table in an HPE system with an ilo, as
the RMRR in question had a device in the list that as far as I could
tell didn't actually exist. The 2nd function of the sata controller
was in the list, but not the first, and the first function was the
device where the group/domain was initialized. With some debugging
code I could see it set up the ioptes for the 2nd function, but it
wasn't reserving the range of iovas.


Regards,
Jerry

Re: [PATCH] iommu/dma: Reserve iova ranges for reserved regions of all devices

Posted by Robin Murphy 2 weeks, 3 days ago

On 2024-11-06 7:13 pm, Jerry Snitselaar wrote:
> On Thu, Oct 24, 2024 at 08:34:12AM -0700, Jerry Snitselaar wrote:
>> Only the first device that is passed when the domain is set up will
>> have its reserved regions reserved in the iova address space.  So if
>> there are other devices in the group with unique reserved regions,
>> those regions will not get reserved in the iova address space.  All of
>> the ranges do get set up in the iopagetables via calls to
>> iommu_create_device_direct_mappings for all devices in a group.
>>
>> In the case of vt-d system this resulted in messages like the following:
>>
>> [ 1632.693264] DMAR: ERROR: DMA PTE for vPFN 0xf1f7e already set (to f1f7e003 not 173025001)
>>
>> To make sure iova ranges are reserved for the reserved regions all of
>> the devices, call iova_reserve_iommu_regions in iommu_dma_init_domain
>> prior to exiting in the case where the domain is already initialized.
>>
>> Cc: Robin Murphy <robin.murphy@arm.com>
>> Cc: Joerg Roedel <joro@8bytes.org>
>> Cc: Will Deacon <will@kernel.org>
>> Cc: linux-kernel@vger.kernel.org
>> Cc: stable@vger.kernel.org
>> Fixes: 7c1b058c8b5a ("iommu/dma: Handle IOMMU API reserved regions")
>> Signed-off-by: Jerry Snitselaar <jsnitsel@redhat.com>
>> ---
>> Robin: I wasn't positive if this is the correct solution, or if it should be
>>         done for the entire group at once.
>>
>>   drivers/iommu/dma-iommu.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
>> index 2a9fa0c8cc00..5fd3cccbb233 100644
>> --- a/drivers/iommu/dma-iommu.c
>> +++ b/drivers/iommu/dma-iommu.c
>> @@ -707,7 +707,7 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, struct device *dev
>>   			goto done_unlock;
>>   		}
>>   
>> -		ret = 0;
>> +		ret = iova_reserve_iommu_regions(dev, domain);
>>   		goto done_unlock;
>>   	}
>>   
>> -- 
>> 2.44.0
>>
> 
> Robin,
> 
> Any thoughts on this patch? In the case where this originally popped
> up it was likely a crap DMAR table in an HPE system with an ilo, as
> the RMRR in question had a device in the list that as far as I could
> tell didn't actually exist. The 2nd function of the sata controller
> was in the list, but not the first, and the first function was the
> device where the group/domain was initialized. With some debugging
> code I could see it set up the ioptes for the 2nd function, but it
> wasn't reserving the range of iovas.

Yeah, this one's tricky - the current behaviour is not entirely 
unintentional since there's not really a right answer. It's also 
possible for the second device to get here after the first device has 
already started using the domain, so at that point it's no longer safe 
to use reserve_iova() due to how it merges overlapping and adjacent 
rbtree nodes, which could really screw things up once there are normal 
allocations in the tree as well. So in truth it was rather a case of 
crossing my fingers and quietly hoping this particular set of 
circumstances was unlikely enough to never come up... :/

Thanks,
Robin.