Add bypass mode support to assigned device

[PATCH 0/3] Add bypass mode support to assigned device

Posted by Zhenzhong Duan 3 years, 8 months ago

Currently virtio-iommu's logic to support bypass mode works only for
emulated device. For assigned device, no GPA -> HPA mapping is setup
in IOMMU page table.

Host report below error:
[3713481.750944] dmar_fault: 191 callbacks suppressed
[3713481.750953] DMAR: DRHD: handling fault status reg 302
[3713481.750962] DMAR: [DMA Read NO_PASID] Request device [2f:00.1] fault addr 0x7ebb0000 [fault reason 0x06] PTE Read access is not set
[3713481.751003] DMAR: DRHD: handling fault status reg 402
[3713481.751007] DMAR: [DMA Read NO_PASID] Request device [2f:00.1] fault addr 0x7ebb0000 [fault reason 0x06] PTE Read access is not set
[3713481.751023] DMAR: DRHD: handling fault status reg 502
[3713481.751026] DMAR: [DMA Write NO_PASID] Request device [2f:00.1] fault addr 0x7ebb0000 [fault reason 0x05] PTE Write access is not set
[3713481.751072] DMAR: DRHD: handling fault status reg 602

Guest kernel report below error:
[    3.461716] i40e: Intel(R) Ethernet Connection XL710 Network Driver
[    3.462605] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
[    3.464630] i40e 0000:00:04.0: Adding to iommu group 5
[    3.482093] i40e 0000:00:04.0: fw 0.0.00000 api 0.0 nvm 0.00 0x176953ce 28.50.1 [8086:37d3] [8086:35d0]
[    3.484295] i40e 0000:00:04.0: eeprom check failed (-62), Tx/Rx traffic disabled
[    3.487268] i40e 0000:00:04.0: configure_lan_hmc failed: -49
[    3.489066] i40e: probe of 0000:00:04.0 failed with error -2

Fix it by adding switch beween bypass and iommu address space just like
the virtual VT-d implementation, so that in bypass mode, vfio mapping
is setup.

Tested with four combination of qemu's virtio-iommu.boot-bypass=true/false
with guest kernel's iommu=pt/nopt on x86_64 platform.

Zhenzhong Duan (3):
  virtio-iommu: Add bypass mode support to assigned device
  virtio-iommu: Use recursive lock to avoid deadlock
  virtio-iommu: Add an assert check in translate routine

 hw/virtio/trace-events           |   1 +
 hw/virtio/virtio-iommu.c         | 135 ++++++++++++++++++++++++++++---
 include/hw/virtio/virtio-iommu.h |   4 +-
 3 files changed, 130 insertions(+), 10 deletions(-)

-- 
2.25.1

Re: [PATCH 0/3] Add bypass mode support to assigned device

Posted by Eric Auger 3 years, 7 months ago

Hi Duan,

On 6/13/22 08:10, Zhenzhong Duan wrote:
> Currently virtio-iommu's logic to support bypass mode works only for
> emulated device. For assigned device, no GPA -> HPA mapping is setup
> in IOMMU page table.
>
> Host report below error:
> [3713481.750944] dmar_fault: 191 callbacks suppressed
> [3713481.750953] DMAR: DRHD: handling fault status reg 302
> [3713481.750962] DMAR: [DMA Read NO_PASID] Request device [2f:00.1] fault addr 0x7ebb0000 [fault reason 0x06] PTE Read access is not set
> [3713481.751003] DMAR: DRHD: handling fault status reg 402
> [3713481.751007] DMAR: [DMA Read NO_PASID] Request device [2f:00.1] fault addr 0x7ebb0000 [fault reason 0x06] PTE Read access is not set
> [3713481.751023] DMAR: DRHD: handling fault status reg 502
> [3713481.751026] DMAR: [DMA Write NO_PASID] Request device [2f:00.1] fault addr 0x7ebb0000 [fault reason 0x05] PTE Write access is not set
> [3713481.751072] DMAR: DRHD: handling fault status reg 602
>
> Guest kernel report below error:
> [    3.461716] i40e: Intel(R) Ethernet Connection XL710 Network Driver
> [    3.462605] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
> [    3.464630] i40e 0000:00:04.0: Adding to iommu group 5
> [    3.482093] i40e 0000:00:04.0: fw 0.0.00000 api 0.0 nvm 0.00 0x176953ce 28.50.1 [8086:37d3] [8086:35d0]
> [    3.484295] i40e 0000:00:04.0: eeprom check failed (-62), Tx/Rx traffic disabled
> [    3.487268] i40e 0000:00:04.0: configure_lan_hmc failed: -49
> [    3.489066] i40e: probe of 0000:00:04.0 failed with error -2
>
> Fix it by adding switch beween bypass and iommu address space just like
> the virtual VT-d implementation, so that in bypass mode, vfio mapping
> is setup.
>
> Tested with four combination of qemu's virtio-iommu.boot-bypass=true/false
> with guest kernel's iommu=pt/nopt on x86_64 platform.

I know this has already landed uptream (I was off last week) but I have
few comments/questions related to the series.
>
> Zhenzhong Duan (3):
>   virtio-iommu: Add bypass mode support to assigned device
>   virtio-iommu: Use recursive lock to avoid deadlock
This patch may have been squashed into the previous one, as
"virtio-iommu: Add bypass mode support to assigned device" deadlocks.

Eric
>   virtio-iommu: Add an assert check in translate routine
>
>  hw/virtio/trace-events           |   1 +
>  hw/virtio/virtio-iommu.c         | 135 ++++++++++++++++++++++++++++---
>  include/hw/virtio/virtio-iommu.h |   4 +-
>  3 files changed, 130 insertions(+), 10 deletions(-)
>

RE: [PATCH 0/3] Add bypass mode support to assigned device

Posted by Duan, Zhenzhong 3 years, 7 months ago

Hi Eric,

>-----Original Message-----
>From: Eric Auger <eric.auger@redhat.com>
>Sent: Friday, June 24, 2022 12:52 AM
>To: Duan, Zhenzhong <zhenzhong.duan@intel.com>
>Cc: qemu-devel@nongnu.org; mst@redhat.com; jean-philippe@linaro.org;
>pbonzini@redhat.com; Zhang, Yu C <yu.c.zhang@intel.com>; Dong,
>Chuanxiao <chuanxiao.dong@intel.com>; Zhang, Tina
><tina.zhang@intel.com>
>Subject: Re: [PATCH 0/3] Add bypass mode support to assigned device
>
>Hi Duan,
>
>On 6/13/22 08:10, Zhenzhong Duan wrote:
>> Currently virtio-iommu's logic to support bypass mode works only for
>> emulated device. For assigned device, no GPA -> HPA mapping is setup
>> in IOMMU page table.
>>
>> Host report below error:
>> [3713481.750944] dmar_fault: 191 callbacks suppressed [3713481.750953]
>> DMAR: DRHD: handling fault status reg 302 [3713481.750962] DMAR: [DMA
>> Read NO_PASID] Request device [2f:00.1] fault addr 0x7ebb0000 [fault
>> reason 0x06] PTE Read access is not set [3713481.751003] DMAR: DRHD:
>> handling fault status reg 402 [3713481.751007] DMAR: [DMA Read
>> NO_PASID] Request device [2f:00.1] fault addr 0x7ebb0000 [fault reason
>> 0x06] PTE Read access is not set [3713481.751023] DMAR: DRHD: handling
>> fault status reg 502 [3713481.751026] DMAR: [DMA Write NO_PASID]
>> Request device [2f:00.1] fault addr 0x7ebb0000 [fault reason 0x05] PTE
>> Write access is not set [3713481.751072] DMAR: DRHD: handling fault
>> status reg 602
>>
>> Guest kernel report below error:
>> [    3.461716] i40e: Intel(R) Ethernet Connection XL710 Network Driver
>> [    3.462605] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
>> [    3.464630] i40e 0000:00:04.0: Adding to iommu group 5
>> [    3.482093] i40e 0000:00:04.0: fw 0.0.00000 api 0.0 nvm 0.00 0x176953ce
>28.50.1 [8086:37d3] [8086:35d0]
>> [    3.484295] i40e 0000:00:04.0: eeprom check failed (-62), Tx/Rx traffic
>disabled
>> [    3.487268] i40e 0000:00:04.0: configure_lan_hmc failed: -49
>> [    3.489066] i40e: probe of 0000:00:04.0 failed with error -2
>>
>> Fix it by adding switch beween bypass and iommu address space just
>> like the virtual VT-d implementation, so that in bypass mode, vfio
>> mapping is setup.
>>
>> Tested with four combination of qemu's
>> virtio-iommu.boot-bypass=true/false
>> with guest kernel's iommu=pt/nopt on x86_64 platform.
>
>I know this has already landed uptream (I was off last week) but I have few
>comments/questions related to the series.

Sure😊

>>
>> Zhenzhong Duan (3):
>>   virtio-iommu: Add bypass mode support to assigned device
>>   virtio-iommu: Use recursive lock to avoid deadlock
>This patch may have been squashed into the previous one, as
>"virtio-iommu: Add bypass mode support to assigned device" deadlocks.

Yes, that may be better. My initial thought is to make 1st patch slim a bit,
which may be more clear for maintainers to review.

Thanks
Zhenzhong