In a PowerNV environment, for devices that supports DMA mask less than
64 bit but larger than 32 bits, iommu is incorrectly bypassing DMA
APIs while allocating and mapping buffers for DMA operations.

Devices are failing with ENOMEN during probe with the following messages

amdgpu 0000:01:00.0: [drm] Detected VRAM RAM=4096M, BAR=4096M
amdgpu 0000:01:00.0: [drm] RAM width 128bits GDDR5
amdgpu 0000:01:00.0: iommu: 64-bit OK but direct DMA is limited by 0
amdgpu 0000:01:00.0: dma_iommu_get_required_mask: returning bypass mask 0xfffffffffffffff
amdgpu 0000:01:00.0:  4096M of VRAM memory ready
amdgpu 0000:01:00.0:  32570M of GTT memory ready.
amdgpu 0000:01:00.0: (-12) failed to allocate kernel bo
amdgpu 0000:01:00.0: [drm] Debug VRAM access will use slowpath MM access
amdgpu 0000:01:00.0: [drm] GART: num cpu pages 4096, num gpu pages 65536
amdgpu 0000:01:00.0: [drm] PCIE GART of 256M enabled (table at 0x000000F4FFF80000).
amdgpu 0000:01:00.0: (-12) failed to allocate kernel bo
amdgpu 0000:01:00.0: (-12) create WB bo failed
amdgpu 0000:01:00.0: amdgpu_device_wb_init failed -12
amdgpu 0000:01:00.0: amdgpu_device_ip_init failed
amdgpu 0000:01:00.0: Fatal error during GPU init
amdgpu 0000:01:00.0: finishing device.
amdgpu 0000:01:00.0: probe with driver amdgpu failed with error -12
amdgpu 0000:01:00.0:  ttm finalized

Fixes: 1471c517cf7d ("powerpc/iommu: bypass DMA APIs for coherent allocations for pre-mapped memory")
Reported-by: Dan Horák <dan@danny.cz>
Closes: https://gitlab.freedesktop.org/drm/amd/-/work_items/5039
Tested-by: Dan Horak <dan@danny.cz>
Signed-off-by: Ritesh Harjani <ritesh.list@gmail.com>
Signed-off-by: Gaurav Batra <gbatra@linux.ibm.com>
---
 arch/powerpc/kernel/dma-iommu.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
index 73e10bd4d56d..8b4de508d2eb 100644
--- a/arch/powerpc/kernel/dma-iommu.c
+++ b/arch/powerpc/kernel/dma-iommu.c
@@ -67,7 +67,7 @@ bool arch_dma_unmap_sg_direct(struct device *dev, struct scatterlist *sg,
 }
 bool arch_dma_alloc_direct(struct device *dev)
 {
-	if (dev->dma_ops_bypass)
+	if (dev->dma_ops_bypass && dev->bus_dma_limit)
 		return true;
 
 	return false;
@@ -75,7 +75,7 @@ bool arch_dma_alloc_direct(struct device *dev)
 
 bool arch_dma_free_direct(struct device *dev, dma_addr_t dma_handle)
 {
-	if (!dev->dma_ops_bypass)
+	if (!dev->dma_ops_bypass || !dev->bus_dma_limit)
 		return false;
 
 	return is_direct_handle(dev, dma_handle);
-- 

I am working on testing the patch in an LPAR with AMDGPU. I will update the
results soon.

++CC few people who would be interested in this fix.

Gaurav Batra <gbatra@linux.ibm.com> writes:

> In a PowerNV environment, for devices that supports DMA mask less than
> 64 bit but larger than 32 bits, iommu is incorrectly bypassing DMA
> APIs while allocating and mapping buffers for DMA operations.
>
> Devices are failing with ENOMEN during probe with the following messages
>
> amdgpu 0000:01:00.0: [drm] Detected VRAM RAM=4096M, BAR=4096M
> amdgpu 0000:01:00.0: [drm] RAM width 128bits GDDR5
> amdgpu 0000:01:00.0: iommu: 64-bit OK but direct DMA is limited by 0
> amdgpu 0000:01:00.0: dma_iommu_get_required_mask: returning bypass mask 0xfffffffffffffff
> amdgpu 0000:01:00.0:  4096M of VRAM memory ready
> amdgpu 0000:01:00.0:  32570M of GTT memory ready.
> amdgpu 0000:01:00.0: (-12) failed to allocate kernel bo
> amdgpu 0000:01:00.0: [drm] Debug VRAM access will use slowpath MM access
> amdgpu 0000:01:00.0: [drm] GART: num cpu pages 4096, num gpu pages 65536
> amdgpu 0000:01:00.0: [drm] PCIE GART of 256M enabled (table at 0x000000F4FFF80000).
> amdgpu 0000:01:00.0: (-12) failed to allocate kernel bo
> amdgpu 0000:01:00.0: (-12) create WB bo failed
> amdgpu 0000:01:00.0: amdgpu_device_wb_init failed -12
> amdgpu 0000:01:00.0: amdgpu_device_ip_init failed
> amdgpu 0000:01:00.0: Fatal error during GPU init
> amdgpu 0000:01:00.0: finishing device.
> amdgpu 0000:01:00.0: probe with driver amdgpu failed with error -12
> amdgpu 0000:01:00.0:  ttm finalized
>
> Fixes: 1471c517cf7d ("powerpc/iommu: bypass DMA APIs for coherent allocations for pre-mapped memory")
> Reported-by: Dan Horák <dan@danny.cz>
> Closes: https://gitlab.freedesktop.org/drm/amd/-/work_items/5039

We could even add the lore link so that in future people can find the
discussion.
Closes: https://lore.kernel.org/linuxppc-dev/20260313142351.609bc4c3efe1184f64ca5f44@danny.cz/

> Tested-by: Dan Horak <dan@danny.cz>
> Signed-off-by: Ritesh Harjani <ritesh.list@gmail.com>

Feel free to change this to the following, I mainly only suggested the fix :)

Suggested-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>


> Signed-off-by: Gaurav Batra <gbatra@linux.ibm.com>
> ---

Generally this info...
> I am working on testing the patch in an LPAR with AMDGPU. I will update the
> results soon.

... goes below the three dashes in here ^^^
This is the same place where people also update the patch change log.

But sure, thanks for updating. As for this patch, it looks good to me.
So, feel free to add:

Reviewed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>


>  arch/powerpc/kernel/dma-iommu.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
> index 73e10bd4d56d..8b4de508d2eb 100644
> --- a/arch/powerpc/kernel/dma-iommu.c
> +++ b/arch/powerpc/kernel/dma-iommu.c
> @@ -67,7 +67,7 @@ bool arch_dma_unmap_sg_direct(struct device *dev, struct scatterlist *sg,
>  }
>  bool arch_dma_alloc_direct(struct device *dev)
>  {
> -	if (dev->dma_ops_bypass)
> +	if (dev->dma_ops_bypass && dev->bus_dma_limit)
>  		return true;
>  
>  	return false;
> @@ -75,7 +75,7 @@ bool arch_dma_alloc_direct(struct device *dev)
>  
>  bool arch_dma_free_direct(struct device *dev, dma_addr_t dma_handle)
>  {
> -	if (!dev->dma_ops_bypass)
> +	if (!dev->dma_ops_bypass || !dev->bus_dma_limit)
>  		return false;
>  
>  	return is_direct_handle(dev, dma_handle);
> -- 
>
> I am working on testing the patch in an LPAR with AMDGPU. I will update the
> results soon.

This should go up, as mentioned above.

-ritesh

On 4/1/26 6:35 AM, Ritesh Harjani (IBM) wrote:
> ++CC few people who would be interested in this fix.
>
> Gaurav Batra <gbatra@linux.ibm.com> writes:
>
>> In a PowerNV environment, for devices that supports DMA mask less than
>> 64 bit but larger than 32 bits, iommu is incorrectly bypassing DMA
>> APIs while allocating and mapping buffers for DMA operations.
>>
>> Devices are failing with ENOMEN during probe with the following messages
>>
>> amdgpu 0000:01:00.0: [drm] Detected VRAM RAM=4096M, BAR=4096M
>> amdgpu 0000:01:00.0: [drm] RAM width 128bits GDDR5
>> amdgpu 0000:01:00.0: iommu: 64-bit OK but direct DMA is limited by 0
>> amdgpu 0000:01:00.0: dma_iommu_get_required_mask: returning bypass mask 0xfffffffffffffff
>> amdgpu 0000:01:00.0:  4096M of VRAM memory ready
>> amdgpu 0000:01:00.0:  32570M of GTT memory ready.
>> amdgpu 0000:01:00.0: (-12) failed to allocate kernel bo
>> amdgpu 0000:01:00.0: [drm] Debug VRAM access will use slowpath MM access
>> amdgpu 0000:01:00.0: [drm] GART: num cpu pages 4096, num gpu pages 65536
>> amdgpu 0000:01:00.0: [drm] PCIE GART of 256M enabled (table at 0x000000F4FFF80000).
>> amdgpu 0000:01:00.0: (-12) failed to allocate kernel bo
>> amdgpu 0000:01:00.0: (-12) create WB bo failed
>> amdgpu 0000:01:00.0: amdgpu_device_wb_init failed -12
>> amdgpu 0000:01:00.0: amdgpu_device_ip_init failed
>> amdgpu 0000:01:00.0: Fatal error during GPU init
>> amdgpu 0000:01:00.0: finishing device.
>> amdgpu 0000:01:00.0: probe with driver amdgpu failed with error -12
>> amdgpu 0000:01:00.0:  ttm finalized
>>
>> Fixes: 1471c517cf7d ("powerpc/iommu: bypass DMA APIs for coherent allocations for pre-mapped memory")
>> Reported-by: Dan Horák <dan@danny.cz>
>> Closes: https://gitlab.freedesktop.org/drm/amd/-/work_items/5039
> We could even add the lore link so that in future people can find the
> discussion.
> Closes: https://lore.kernel.org/linuxppc-dev/20260313142351.609bc4c3efe1184f64ca5f44@danny.cz/
>
>> Tested-by: Dan Horak <dan@danny.cz>
>> Signed-off-by: Ritesh Harjani <ritesh.list@gmail.com>
> Feel free to change this to the following, I mainly only suggested the fix :)
>
> Suggested-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
>
>
>> Signed-off-by: Gaurav Batra <gbatra@linux.ibm.com>
>> ---
> Generally this info...
>> I am working on testing the patch in an LPAR with AMDGPU. I will update the
>> results soon.
> ... goes below the three dashes in here ^^^
> This is the same place where people also update the patch change log.
>
> But sure, thanks for updating. As for this patch, it looks good to me.
> So, feel free to add:
>
> Reviewed-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
I will fix the commit message changes and add it to my fixes-test
since we are at the end of -rc7 window, need to get this out soon

Maddy

>
>
>>   arch/powerpc/kernel/dma-iommu.c | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c
>> index 73e10bd4d56d..8b4de508d2eb 100644
>> --- a/arch/powerpc/kernel/dma-iommu.c
>> +++ b/arch/powerpc/kernel/dma-iommu.c
>> @@ -67,7 +67,7 @@ bool arch_dma_unmap_sg_direct(struct device *dev, struct scatterlist *sg,
>>   }
>>   bool arch_dma_alloc_direct(struct device *dev)
>>   {
>> -	if (dev->dma_ops_bypass)
>> +	if (dev->dma_ops_bypass && dev->bus_dma_limit)
>>   		return true;
>>   
>>   	return false;
>> @@ -75,7 +75,7 @@ bool arch_dma_alloc_direct(struct device *dev)
>>   
>>   bool arch_dma_free_direct(struct device *dev, dma_addr_t dma_handle)
>>   {
>> -	if (!dev->dma_ops_bypass)
>> +	if (!dev->dma_ops_bypass || !dev->bus_dma_limit)
>>   		return false;
>>   
>>   	return is_direct_handle(dev, dma_handle);
>> -- 
>>
>> I am working on testing the patch in an LPAR with AMDGPU. I will update the
>> results soon.
> This should go up, as mentioned above.
>
> -ritesh