[v3] amd_iommu: Cleanups and fixes (PART 2)

[PATCH v3 0/2] amd_iommu: Cleanups and fixes (PART 2)

Posted by Sairaj Kodilkar 4 weeks ago

This series provide fixes for following two issues:

1. AMD IOMMU fails to detect the devices when they are attached to PCI bus with
   bus id != 0.
   e.g. With following command line, dhclient command fails inside the guest

    -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x5 \
    -netdev user,id=USER0,hostfwd=tcp::3333-:22 \
    -device virtio-net-pci,id=vnet0,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER0,bus=pci.1,addr=0 \

2. Current AMD IOMMU supports IOVAs upto 60 bit which cause failure while
   setting up the devices when guest is booted with command line 
   "iommu.forcedac=1".

   One example of the failure is when there are two virtio ethernet devices
   attached to the guest with command line
   
       -netdev user,id=USER0 \
       -netdev user,id=USER1 \
       -device virtio-net-pci,id=vnet0,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER0 \
       -device virtio-net-pci,id=vnet1,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER1 \
   
   In this case dhclient fails for second device with following dmesg
   
   [   24.802644] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 5664000 usecs ago
   [   29.856716] virtio_net virtio0 enp0s1: NETDEV WATCHDOG: CPU: 59: transmit queue 0 timed out 10720 ms
   [   29.858585] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 10720000 usecs ago

-------------------------------------------------------------------------------

Change log:
----------

* Changes since V2:
https://lore.kernel.org/qemu-devel/20251013050046.393-1-sarunkod@amd.com/
P1:
 - Remove cast from the pointer assignment [MST]
 - Rename struct amdvi_as_key to AMDViAsKey [Alejandro, MST]
 - Fix compilation error due to missing 'static' [Alejandro]

P2:
 - Rename struct amdvi_iotlb_key to AMDViIOTLBKey [Alejandro, MST]
 - Fix compilation error [Alejandro]


* Changes Since V1:
https://lore.kernel.org/qemu-devel/20251008164324.21553-1-sarunkod@amd.com/
P1:
 - Use fixed type uint8_t for devfn
 - Use uintptr_t instead of uint64_t
 - Build hash key using lower 56 bits of bus pointer and 8 bits of devfn
 - Use gboolean instead of int for amdvi_find_as_by_devid
 - Update comments
 - Use IOMMU_NOTIFIER_NONE instead of IOMMU_NONE

P2:
 - Reword commit message
 - Correctly initialize `struct amdvi_iotlb_key`
 - Remove unused macro

-------------------------------------------------------------------------------

Base commit: (qemu uptream) eb7abb4a719f

-------------------------------------------------------------------------------


Sairaj Kodilkar (2):
  amd_iommu: Fix handling device on buses != 0
  amd_iommu: Support 64 bit address for IOTLB lookup

 hw/i386/amd_iommu.c | 179 +++++++++++++++++++++++++++-----------------
 hw/i386/amd_iommu.h |   6 +-
 2 files changed, 113 insertions(+), 72 deletions(-)

-- 
2.34.1

Re: [PATCH v3 0/2] amd_iommu: Cleanups and fixes (PART 2)

Posted by Alejandro Jimenez 3 weeks, 4 days ago

Hi Sairaj,

On 10/17/25 2:13 AM, Sairaj Kodilkar wrote:
> This series provide fixes for following two issues:
> 
> 1. AMD IOMMU fails to detect the devices when they are attached to PCI bus with
>     bus id != 0.
>     e.g. With following command line, dhclient command fails inside the guest
> 
>      -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x5 \
>      -netdev user,id=USER0,hostfwd=tcp::3333-:22 \
>      -device virtio-net-pci,id=vnet0,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER0,bus=pci.1,addr=0 \
> 
> 2. Current AMD IOMMU supports IOVAs upto 60 bit which cause failure while
>     setting up the devices when guest is booted with command line
>     "iommu.forcedac=1".
> 
>     One example of the failure is when there are two virtio ethernet devices
>     attached to the guest with command line
>     
>         -netdev user,id=USER0 \
>         -netdev user,id=USER1 \
>         -device virtio-net-pci,id=vnet0,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER0 \
>         -device virtio-net-pci,id=vnet1,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER1 \
>     
>     In this case dhclient fails for second device with following dmesg
>     
>     [   24.802644] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 5664000 usecs ago
>     [   29.856716] virtio_net virtio0 enp0s1: NETDEV WATCHDOG: CPU: 59: transmit queue 0 timed out 10720 ms
>     [   29.858585] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 10720000 usecs ago
> 

Besides the above issues, I can also confirm that in combination with a 
patched VFIO driver in the host kernel, this series addresses the 
problems I mentioned earlier with VFIO passthrough devices + DMA remap 
mode + forcedac=1 on guest reboot. Thank you!

For the series:
Reviewed-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
Tested-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>


> -------------------------------------------------------------------------------
> 
> Change log:
> ----------
> 
> * Changes since V2:
> https://lore.kernel.org/qemu-devel/20251013050046.393-1-sarunkod@amd.com/
> P1:
>   - Remove cast from the pointer assignment [MST]
>   - Rename struct amdvi_as_key to AMDViAsKey [Alejandro, MST]
>   - Fix compilation error due to missing 'static' [Alejandro]
> 
> P2:
>   - Rename struct amdvi_iotlb_key to AMDViIOTLBKey [Alejandro, MST]
>   - Fix compilation error [Alejandro]
> 
> 
> * Changes Since V1:
> https://lore.kernel.org/qemu-devel/20251008164324.21553-1-sarunkod@amd.com/
> P1:
>   - Use fixed type uint8_t for devfn
>   - Use uintptr_t instead of uint64_t
>   - Build hash key using lower 56 bits of bus pointer and 8 bits of devfn
>   - Use gboolean instead of int for amdvi_find_as_by_devid
>   - Update comments
>   - Use IOMMU_NOTIFIER_NONE instead of IOMMU_NONE
> 
> P2:
>   - Reword commit message
>   - Correctly initialize `struct amdvi_iotlb_key`
>   - Remove unused macro
> 
> -------------------------------------------------------------------------------
> 
> Base commit: (qemu uptream) eb7abb4a719f
> 
> -------------------------------------------------------------------------------
> 
> 
> Sairaj Kodilkar (2):
>    amd_iommu: Fix handling device on buses != 0
>    amd_iommu: Support 64 bit address for IOTLB lookup
> 
>   hw/i386/amd_iommu.c | 179 +++++++++++++++++++++++++++-----------------
>   hw/i386/amd_iommu.h |   6 +-
>   2 files changed, 113 insertions(+), 72 deletions(-)
>

Re: [PATCH v3 0/2] amd_iommu: Cleanups and fixes (PART 2)

Posted by Vasant Hegde 3 weeks, 3 days ago

Alejandro,

On 10/19/2025 10:12 PM, Alejandro Jimenez wrote:
> Hi Sairaj,
> 
> On 10/17/25 2:13 AM, Sairaj Kodilkar wrote:
>> This series provide fixes for following two issues:
>>
>> 1. AMD IOMMU fails to detect the devices when they are attached to PCI bus with
>>     bus id != 0.
>>     e.g. With following command line, dhclient command fails inside the guest
>>
>>      -device pcie-root-
>> port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x5 \
>>      -netdev user,id=USER0,hostfwd=tcp::3333-:22 \
>>      -device virtio-net-pci,id=vnet0,iommu_platform=on,disable-
>> legacy=on,romfile=,netdev=USER0,bus=pci.1,addr=0 \
>>
>> 2. Current AMD IOMMU supports IOVAs upto 60 bit which cause failure while
>>     setting up the devices when guest is booted with command line
>>     "iommu.forcedac=1".
>>
>>     One example of the failure is when there are two virtio ethernet devices
>>     attached to the guest with command line
>>             -netdev user,id=USER0 \
>>         -netdev user,id=USER1 \
>>         -device virtio-net-pci,id=vnet0,iommu_platform=on,disable-
>> legacy=on,romfile=,netdev=USER0 \
>>         -device virtio-net-pci,id=vnet1,iommu_platform=on,disable-
>> legacy=on,romfile=,netdev=USER1 \
>>         In this case dhclient fails for second device with following dmesg
>>         [   24.802644] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq:
>> output.0, vq: 0x1, name: output.0, 5664000 usecs ago
>>     [   29.856716] virtio_net virtio0 enp0s1: NETDEV WATCHDOG: CPU: 59:
>> transmit queue 0 timed out 10720 ms
>>     [   29.858585] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq:
>> output.0, vq: 0x1, name: output.0, 10720000 usecs ago
>>
> 
> Besides the above issues, I can also confirm that in combination with a patched
> VFIO driver in the host kernel, this series addresses the problems I mentioned
> earlier with VFIO passthrough devices + DMA remap mode + forcedac=1 on guest
> reboot. Thank you!

Nice!  Thanks for verifying.


-Vasant

> 
> For the series:
> Reviewed-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
> Tested-by: Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
> 
> 
>> -------------------------------------------------------------------------------
>>
>> Change log:
>> ----------
>>
>> * Changes since V2:
>> https://lore.kernel.org/qemu-devel/20251013050046.393-1-sarunkod@amd.com/
>> P1:
>>   - Remove cast from the pointer assignment [MST]
>>   - Rename struct amdvi_as_key to AMDViAsKey [Alejandro, MST]
>>   - Fix compilation error due to missing 'static' [Alejandro]
>>
>> P2:
>>   - Rename struct amdvi_iotlb_key to AMDViIOTLBKey [Alejandro, MST]
>>   - Fix compilation error [Alejandro]
>>
>>
>> * Changes Since V1:
>> https://lore.kernel.org/qemu-devel/20251008164324.21553-1-sarunkod@amd.com/
>> P1:
>>   - Use fixed type uint8_t for devfn
>>   - Use uintptr_t instead of uint64_t
>>   - Build hash key using lower 56 bits of bus pointer and 8 bits of devfn
>>   - Use gboolean instead of int for amdvi_find_as_by_devid
>>   - Update comments
>>   - Use IOMMU_NOTIFIER_NONE instead of IOMMU_NONE
>>
>> P2:
>>   - Reword commit message
>>   - Correctly initialize `struct amdvi_iotlb_key`
>>   - Remove unused macro
>>
>> -------------------------------------------------------------------------------
>>
>> Base commit: (qemu uptream) eb7abb4a719f
>>
>> -------------------------------------------------------------------------------
>>
>>
>> Sairaj Kodilkar (2):
>>    amd_iommu: Fix handling device on buses != 0
>>    amd_iommu: Support 64 bit address for IOTLB lookup
>>
>>   hw/i386/amd_iommu.c | 179 +++++++++++++++++++++++++++-----------------
>>   hw/i386/amd_iommu.h |   6 +-
>>   2 files changed, 113 insertions(+), 72 deletions(-)
>>
>

Re: [PATCH v3 0/2] amd_iommu: Cleanups and fixes (PART 2)

Posted by Sairaj Kodilkar 4 weeks ago


On 10/17/2025 11:43 AM, Sairaj Kodilkar wrote:
> This series provide fixes for following two issues:
>
> 1. AMD IOMMU fails to detect the devices when they are attached to PCI bus with
>     bus id != 0.
>     e.g. With following command line, dhclient command fails inside the guest
>
>      -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x5 \
>      -netdev user,id=USER0,hostfwd=tcp::3333-:22 \
>      -device virtio-net-pci,id=vnet0,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER0,bus=pci.1,addr=0 \
>
> 2. Current AMD IOMMU supports IOVAs upto 60 bit which cause failure while
>     setting up the devices when guest is booted with command line
>     "iommu.forcedac=1".
>
>     One example of the failure is when there are two virtio ethernet devices
>     attached to the guest with command line
>     
>         -netdev user,id=USER0 \
>         -netdev user,id=USER1 \
>         -device virtio-net-pci,id=vnet0,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER0 \
>         -device virtio-net-pci,id=vnet1,iommu_platform=on,disable-legacy=on,romfile=,netdev=USER1 \
>     
>     In this case dhclient fails for second device with following dmesg
>     
>     [   24.802644] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 5664000 usecs ago
>     [   29.856716] virtio_net virtio0 enp0s1: NETDEV WATCHDOG: CPU: 59: transmit queue 0 timed out 10720 ms
>     [   29.858585] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 10720000 usecs ago
>
> -------------------------------------------------------------------------------
>
> Change log:
> ----------
>
> * Changes since V2:
> https://lore.kernel.org/qemu-devel/20251013050046.393-1-sarunkod@amd.com/
> P1:
>   - Remove cast from the pointer assignment [MST]
>   - Rename struct amdvi_as_key to AMDViAsKey [Alejandro, MST]
>   - Fix compilation error due to missing 'static' [Alejandro]
>
> P2:
>   - Rename struct amdvi_iotlb_key to AMDViIOTLBKey [Alejandro, MST]
>   - Fix compilation error [Alejandro]
>
Correction...
This is AMDVIAskey and AMDVIIOTLBKey, not AMDViAsKey and AMDViIOTLBKey

Thanks
Sairaj