[Qemu-devel] [RFC v4 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration

Eric Auger posted 29 patches 13 weeks ago
Test docker-clang@ubuntu failed
Test asan failed
Test docker-mingw@fedora passed
Test s390x failed
Test FreeBSD passed
Test checkpatch passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20190711172845.31035-1-eric.auger@redhat.com
Maintainers: Paolo Bonzini <pbonzini@redhat.com>, "Michael S. Tsirkin" <mst@redhat.com>, Alex Williamson <alex.williamson@redhat.com>, Cornelia Huck <cohuck@redhat.com>, Marcel Apfelbaum <marcel.apfelbaum@gmail.com>, Eric Auger <eric.auger@redhat.com>, Peter Maydell <peter.maydell@linaro.org>
hw/arm/smmuv3-internal.h                     |   1 +
hw/arm/smmuv3.c                              | 195 +++++--
hw/arm/trace-events                          |   3 +-
hw/pci/pci.c                                 |  34 ++
hw/vfio/common.c                             | 517 +++++++++++++++----
hw/vfio/pci.c                                | 267 +++++++++-
hw/vfio/pci.h                                |   9 +
hw/vfio/trace-events                         |   9 +-
include/exec/memory.h                        |  59 ++-
include/hw/arm/smmu-common.h                 |   1 +
include/hw/iommu/iommu.h                     |  28 +
include/hw/pci/pci.h                         |  11 +
include/hw/vfio/vfio-common.h                |  16 +
include/standard-headers/asm-x86/bootparam.h |   2 +
include/standard-headers/linux/virtio_ids.h  |   1 -
include/standard-headers/linux/virtio_pmem.h |  34 --
linux-headers/asm-arm/unistd-common.h        |   1 +
linux-headers/asm-arm64/kvm.h                |   7 +
linux-headers/asm-generic/unistd.h           |   4 +-
linux-headers/asm-mips/unistd_n32.h          |   1 +
linux-headers/asm-mips/unistd_n64.h          |   1 +
linux-headers/asm-mips/unistd_o32.h          |   1 +
linux-headers/asm-powerpc/unistd_32.h        |   1 +
linux-headers/asm-powerpc/unistd_64.h        |   1 +
linux-headers/asm-s390/unistd_32.h           |   1 +
linux-headers/asm-s390/unistd_64.h           |   1 +
linux-headers/asm-x86/kvm.h                  |   6 +-
linux-headers/asm-x86/unistd_32.h            |   1 +
linux-headers/asm-x86/unistd_64.h            |   1 +
linux-headers/asm-x86/unistd_x32.h           |   1 +
linux-headers/linux/iommu.h                  | 316 ++++++++++++
linux-headers/linux/psp-sev.h                |   5 +-
linux-headers/linux/vfio.h                   | 109 +++-
memory.c                                     |  19 +-
scripts/update-linux-headers.sh              |   5 +-
35 files changed, 1462 insertions(+), 207 deletions(-)
create mode 100644 include/hw/iommu/iommu.h
delete mode 100644 include/standard-headers/linux/virtio_pmem.h
create mode 100644 linux-headers/linux/iommu.h

[Qemu-devel] [RFC v4 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration

Posted by Eric Auger 13 weeks ago
Up to now vSMMUv3 has not been integrated with VFIO. VFIO
integration requires to program the physical IOMMU consistently
with the guest mappings. However, as opposed to VTD, SMMUv3 has
no "Caching Mode" which allows easy trapping of guest mappings.
This means the vSMMUV3 cannot use the same VFIO integration as VTD.

However SMMUv3 has 2 translation stages. This was devised with
virtualization use case in mind where stage 1 is "owned" by the
guest whereas the host uses stage 2 for VM isolation.

This series sets up this nested translation stage. It only works
if there is one physical SMMUv3 used along with QEMU vSMMUv3 (in
other words, it does not work if there is a physical SMMUv2).

The series uses a new kernel user API [1] not yet upstream.

- We force the host to use stage 2 instead of stage 1, when we
  detect a vSMMUV3 is behind a VFIO device. For a VFIO device
  without any virtual IOMMU, we still use stage 1 as many existing
  SMMUs expect this behavior.
- We use PCIPASIDOps to propage guest stage1 config changes on
  STE (Stream Table Entry) changes.
- We implement a specific UNMAP notifier that conveys guest
  IOTLB invalidations to the host
- We register MSI IOVA/GPA bindings to the host so that this latter
  can build a nested stage translation
- As the legacy MAP notifier is not called anymore, we must make
  sure stage 2 mappings are set. This is achieved through another
  prereg memory listener.
- Physical SMMU stage 1 related faults are reported to the guest
  via en eventfd mechanism and exposed trhough a dedicated VFIO-PCI
  region. Then they are reinjected into the guest.

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5

Kernel Dependencies:
[1] [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
[2] [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part)
branch at: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9

Includes fixes sent separately:
[3] [PATCH-for-4.2 v3 0/5] ARM SMMUv3: Fix spurious notification
    errors and assert with vfio: in case of any comment on those
    patches, please comment the other series.

History:

v4 -> v5:
- Use PCIPASIDOps for config update notifications
- removal of notification for MSI binding which is not needed
  anymore
- Uses fixes in [3]
- Use a single fault region
- use the specific interrupt index

v3 -> v4:
- adapt to changes in uapi (asid cache invalidation)
- check VFIO_PCI_DMA_FAULT_IRQ_INDEX is supported at kernel level
  before attempting to set signaling for it.
- sync on 5.2-rc1 kernel headers + Drew's patch that imports sve_context.h
- fix MSI binding for MSI (not MSIX)
- fix mingw compilation

v2 -> v3:
- rework fault handling
- MSI binding registration done in vfio-pci. MSI binding tear down called
  on container cleanup path
- leaf parameter propagated

v1 -> v2:
- Fixed dual assignment (asid now correctly propagated on TLB invalidations)
- Integrated fault reporting

Andrew Jones (1):
  update-linux-headers: Add sve_context.h to asm-arm64

Eric Auger (27):
  memory: Remove unused memory_region_iommu_replay_all()
  memory: Add IOMMU_ATTR_VFIO_NESTED IOMMU memory region attribute
  hw/vfio/common: Assert in case of nested mode
  hw/arm/smmuv3: Log a guest error when decoding an invalid STE
  hw/arm/smmuv3: Remove spurious error messages on IOVA invalidations
  update-linux-headers: Import iommu.h
  header update against 5.3.0-rc0 and IOMMU/VFIO nested stage APIs
  memory: Add IOMMU_ATTR_MSI_TRANSLATE IOMMU memory region attribute
  memory: Introduce IOMMU Memory Region inject_faults API
  memory: Add arch_id and leaf fields in IOTLBEntry
  iommu: Introduce generic header
  vfio: Force nested if iommu requires it
  vfio: Introduce hostwin_from_range helper
  vfio: Introduce helpers to DMA map/unmap a RAM section
  vfio: Set up nested stage mappings
  vfio: Pass stage 1 MSI bindings to the host
  vfio: Helper to get IRQ info including capabilities
  vfio/pci: Register handler for iommu fault
  vfio/pci: Set up the DMA FAULT region
  vfio/pci: Implement the DMA fault handler
  hw/arm/smmuv3: Advertise MSI_TRANSLATE attribute
  hw/arm/smmuv3: Store the PASID table GPA in the translation config
  hw/arm/smmuv3: Fill the IOTLBEntry arch_id on NH_VA invalidation
  hw/arm/smmuv3: Fill the IOTLBEntry leaf field on NH_VA invalidation
  hw/arm/smmuv3: Pass stage 1 configurations to the host
  hw/arm/smmuv3: Implement fault injection
  vfio: Remove VFIO/SMMUv3 assert

Liu Yi L (1):
  pci: introduce PCIPASIDOps to PCIDevice

 hw/arm/smmuv3-internal.h                     |   1 +
 hw/arm/smmuv3.c                              | 195 +++++--
 hw/arm/trace-events                          |   3 +-
 hw/pci/pci.c                                 |  34 ++
 hw/vfio/common.c                             | 517 +++++++++++++++----
 hw/vfio/pci.c                                | 267 +++++++++-
 hw/vfio/pci.h                                |   9 +
 hw/vfio/trace-events                         |   9 +-
 include/exec/memory.h                        |  59 ++-
 include/hw/arm/smmu-common.h                 |   1 +
 include/hw/iommu/iommu.h                     |  28 +
 include/hw/pci/pci.h                         |  11 +
 include/hw/vfio/vfio-common.h                |  16 +
 include/standard-headers/asm-x86/bootparam.h |   2 +
 include/standard-headers/linux/virtio_ids.h  |   1 -
 include/standard-headers/linux/virtio_pmem.h |  34 --
 linux-headers/asm-arm/unistd-common.h        |   1 +
 linux-headers/asm-arm64/kvm.h                |   7 +
 linux-headers/asm-generic/unistd.h           |   4 +-
 linux-headers/asm-mips/unistd_n32.h          |   1 +
 linux-headers/asm-mips/unistd_n64.h          |   1 +
 linux-headers/asm-mips/unistd_o32.h          |   1 +
 linux-headers/asm-powerpc/unistd_32.h        |   1 +
 linux-headers/asm-powerpc/unistd_64.h        |   1 +
 linux-headers/asm-s390/unistd_32.h           |   1 +
 linux-headers/asm-s390/unistd_64.h           |   1 +
 linux-headers/asm-x86/kvm.h                  |   6 +-
 linux-headers/asm-x86/unistd_32.h            |   1 +
 linux-headers/asm-x86/unistd_64.h            |   1 +
 linux-headers/asm-x86/unistd_x32.h           |   1 +
 linux-headers/linux/iommu.h                  | 316 ++++++++++++
 linux-headers/linux/psp-sev.h                |   5 +-
 linux-headers/linux/vfio.h                   | 109 +++-
 memory.c                                     |  19 +-
 scripts/update-linux-headers.sh              |   5 +-
 35 files changed, 1462 insertions(+), 207 deletions(-)
 create mode 100644 include/hw/iommu/iommu.h
 delete mode 100644 include/standard-headers/linux/virtio_pmem.h
 create mode 100644 linux-headers/linux/iommu.h

-- 
2.20.1


Re: [Qemu-devel] [RFC v4 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration

Posted by Auger Eric 13 weeks ago
Hi,

On 7/11/19 7:28 PM, Eric Auger wrote:
> Up to now vSMMUv3 has not been integrated with VFIO. VFIO
> integration requires to program the physical IOMMU consistently
> with the guest mappings. However, as opposed to VTD, SMMUv3 has
> no "Caching Mode" which allows easy trapping of guest mappings.
> This means the vSMMUV3 cannot use the same VFIO integration as VTD.
> 
> However SMMUv3 has 2 translation stages. This was devised with
> virtualization use case in mind where stage 1 is "owned" by the
> guest whereas the host uses stage 2 for VM isolation.
> 
> This series sets up this nested translation stage. It only works
> if there is one physical SMMUv3 used along with QEMU vSMMUv3 (in
> other words, it does not work if there is a physical SMMUv2).
> 
> The series uses a new kernel user API [1] not yet upstream.
> 
> - We force the host to use stage 2 instead of stage 1, when we
>   detect a vSMMUV3 is behind a VFIO device. For a VFIO device
>   without any virtual IOMMU, we still use stage 1 as many existing
>   SMMUs expect this behavior.
> - We use PCIPASIDOps to propage guest stage1 config changes on
>   STE (Stream Table Entry) changes.
> - We implement a specific UNMAP notifier that conveys guest
>   IOTLB invalidations to the host
> - We register MSI IOVA/GPA bindings to the host so that this latter
>   can build a nested stage translation
> - As the legacy MAP notifier is not called anymore, we must make
>   sure stage 2 mappings are set. This is achieved through another
>   prereg memory listener.
> - Physical SMMU stage 1 related faults are reported to the guest
>   via en eventfd mechanism and exposed trhough a dedicated VFIO-PCI
>   region. Then they are reinjected into the guest.
> 
> Best Regards
> 
> Eric
> 
> This series can be found at:
> https://github.com/eauger/qemu/tree/v4.1.0-rc0-2stage-rfcv5
Please forgive me. This is the v5 of the RFC. Resending it with the
correct prefix ...

Sorry for the spam :-(

Eric
> 
> Kernel Dependencies:
> [1] [PATCH v9 00/11] SMMUv3 Nested Stage Setup (VFIO part)
> [2] [PATCH v9 00/14] SMMUv3 Nested Stage Setup (IOMMU part)
> branch at: https://github.com/eauger/linux/tree/v5.3.0-rc0-2stage-v9
> 
> Includes fixes sent separately:
> [3] [PATCH-for-4.2 v3 0/5] ARM SMMUv3: Fix spurious notification
>     errors and assert with vfio: in case of any comment on those
>     patches, please comment the other series.
> 
> History:
> 
> v4 -> v5:
> - Use PCIPASIDOps for config update notifications
> - removal of notification for MSI binding which is not needed
>   anymore
> - Uses fixes in [3]
> - Use a single fault region
> - use the specific interrupt index
> 
> v3 -> v4:
> - adapt to changes in uapi (asid cache invalidation)
> - check VFIO_PCI_DMA_FAULT_IRQ_INDEX is supported at kernel level
>   before attempting to set signaling for it.
> - sync on 5.2-rc1 kernel headers + Drew's patch that imports sve_context.h
> - fix MSI binding for MSI (not MSIX)
> - fix mingw compilation
> 
> v2 -> v3:
> - rework fault handling
> - MSI binding registration done in vfio-pci. MSI binding tear down called
>   on container cleanup path
> - leaf parameter propagated
> 
> v1 -> v2:
> - Fixed dual assignment (asid now correctly propagated on TLB invalidations)
> - Integrated fault reporting
> 
> Andrew Jones (1):
>   update-linux-headers: Add sve_context.h to asm-arm64
> 
> Eric Auger (27):
>   memory: Remove unused memory_region_iommu_replay_all()
>   memory: Add IOMMU_ATTR_VFIO_NESTED IOMMU memory region attribute
>   hw/vfio/common: Assert in case of nested mode
>   hw/arm/smmuv3: Log a guest error when decoding an invalid STE
>   hw/arm/smmuv3: Remove spurious error messages on IOVA invalidations
>   update-linux-headers: Import iommu.h
>   header update against 5.3.0-rc0 and IOMMU/VFIO nested stage APIs
>   memory: Add IOMMU_ATTR_MSI_TRANSLATE IOMMU memory region attribute
>   memory: Introduce IOMMU Memory Region inject_faults API
>   memory: Add arch_id and leaf fields in IOTLBEntry
>   iommu: Introduce generic header
>   vfio: Force nested if iommu requires it
>   vfio: Introduce hostwin_from_range helper
>   vfio: Introduce helpers to DMA map/unmap a RAM section
>   vfio: Set up nested stage mappings
>   vfio: Pass stage 1 MSI bindings to the host
>   vfio: Helper to get IRQ info including capabilities
>   vfio/pci: Register handler for iommu fault
>   vfio/pci: Set up the DMA FAULT region
>   vfio/pci: Implement the DMA fault handler
>   hw/arm/smmuv3: Advertise MSI_TRANSLATE attribute
>   hw/arm/smmuv3: Store the PASID table GPA in the translation config
>   hw/arm/smmuv3: Fill the IOTLBEntry arch_id on NH_VA invalidation
>   hw/arm/smmuv3: Fill the IOTLBEntry leaf field on NH_VA invalidation
>   hw/arm/smmuv3: Pass stage 1 configurations to the host
>   hw/arm/smmuv3: Implement fault injection
>   vfio: Remove VFIO/SMMUv3 assert
> 
> Liu Yi L (1):
>   pci: introduce PCIPASIDOps to PCIDevice
> 
>  hw/arm/smmuv3-internal.h                     |   1 +
>  hw/arm/smmuv3.c                              | 195 +++++--
>  hw/arm/trace-events                          |   3 +-
>  hw/pci/pci.c                                 |  34 ++
>  hw/vfio/common.c                             | 517 +++++++++++++++----
>  hw/vfio/pci.c                                | 267 +++++++++-
>  hw/vfio/pci.h                                |   9 +
>  hw/vfio/trace-events                         |   9 +-
>  include/exec/memory.h                        |  59 ++-
>  include/hw/arm/smmu-common.h                 |   1 +
>  include/hw/iommu/iommu.h                     |  28 +
>  include/hw/pci/pci.h                         |  11 +
>  include/hw/vfio/vfio-common.h                |  16 +
>  include/standard-headers/asm-x86/bootparam.h |   2 +
>  include/standard-headers/linux/virtio_ids.h  |   1 -
>  include/standard-headers/linux/virtio_pmem.h |  34 --
>  linux-headers/asm-arm/unistd-common.h        |   1 +
>  linux-headers/asm-arm64/kvm.h                |   7 +
>  linux-headers/asm-generic/unistd.h           |   4 +-
>  linux-headers/asm-mips/unistd_n32.h          |   1 +
>  linux-headers/asm-mips/unistd_n64.h          |   1 +
>  linux-headers/asm-mips/unistd_o32.h          |   1 +
>  linux-headers/asm-powerpc/unistd_32.h        |   1 +
>  linux-headers/asm-powerpc/unistd_64.h        |   1 +
>  linux-headers/asm-s390/unistd_32.h           |   1 +
>  linux-headers/asm-s390/unistd_64.h           |   1 +
>  linux-headers/asm-x86/kvm.h                  |   6 +-
>  linux-headers/asm-x86/unistd_32.h            |   1 +
>  linux-headers/asm-x86/unistd_64.h            |   1 +
>  linux-headers/asm-x86/unistd_x32.h           |   1 +
>  linux-headers/linux/iommu.h                  | 316 ++++++++++++
>  linux-headers/linux/psp-sev.h                |   5 +-
>  linux-headers/linux/vfio.h                   | 109 +++-
>  memory.c                                     |  19 +-
>  scripts/update-linux-headers.sh              |   5 +-
>  35 files changed, 1462 insertions(+), 207 deletions(-)
>  create mode 100644 include/hw/iommu/iommu.h
>  delete mode 100644 include/standard-headers/linux/virtio_pmem.h
>  create mode 100644 linux-headers/linux/iommu.h
> 

Re: [Qemu-devel] [RFC v4 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration

Posted by no-reply@patchew.org 13 weeks ago
Patchew URL: https://patchew.org/QEMU/20190711172845.31035-1-eric.auger@redhat.com/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      x86_64-softmmu/hw/virtio/vhost-user-blk-pci.o
  CC      x86_64-softmmu/hw/virtio/vhost-user-input-pci.o
  CC      x86_64-softmmu/hw/virtio/vhost-user-scsi-pci.o
/tmp/qemu-test/src/hw/virtio/virtio-pmem.c:21:10: fatal error: 'standard-headers/linux/virtio_pmem.h' file not found
#include "standard-headers/linux/virtio_pmem.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.


The full log is available at
http://patchew.org/logs/20190711172845.31035-1-eric.auger@redhat.com/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [RFC v4 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration

Posted by no-reply@patchew.org 13 weeks ago
Patchew URL: https://patchew.org/QEMU/20190711172845.31035-1-eric.auger@redhat.com/



Hi,

This series failed build test on s390x host. Please find the details below.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e

echo
echo "=== ENV ==="
env

echo
echo "=== PACKAGES ==="
rpm -qa

echo
echo "=== UNAME ==="
uname -a

CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===

  CC      arm-softmmu/monitor/misc.o
  CC      i386-softmmu/hw/virtio/virtio-pmem.o
  CC      aarch64-softmmu/hw/arm/fsl-imx7.o
/var/tmp/patchew-tester-tmp-oj2zej_6/src/hw/virtio/virtio-pmem.c:21:10: fatal error: standard-headers/linux/virtio_pmem.h: No such file or directory
   21 | #include "standard-headers/linux/virtio_pmem.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.


The full log is available at
http://patchew.org/logs/20190711172845.31035-1-eric.auger@redhat.com/testing.s390x/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [RFC v4 00/29] vSMMUv3/pSMMUv3 2 stage VFIO integration

Posted by no-reply@patchew.org 13 weeks ago
Patchew URL: https://patchew.org/QEMU/20190711172845.31035-1-eric.auger@redhat.com/



Hi,

This series failed build test on s390x host. Please find the details below.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
# Testing script will be invoked under the git checkout with
# HEAD pointing to a commit that has the patches applied on top of "base"
# branch
set -e

echo
echo "=== ENV ==="
env

echo
echo "=== PACKAGES ==="
rpm -qa

echo
echo "=== UNAME ==="
uname -a

CC=$HOME/bin/cc
INSTALL=$PWD/install
BUILD=$PWD/build
mkdir -p $BUILD $INSTALL
SRC=$PWD
cd $BUILD
$SRC/configure --cc=$CC --prefix=$INSTALL
make -j4
# XXX: we need reliable clean up
# make check -j4 V=1
make install
=== TEST SCRIPT END ===

  CC      aarch64-softmmu/hw/arm/msf2-soc.o
  CC      i386-softmmu/hw/virtio/virtio-pmem.o
  CC      aarch64-softmmu/hw/arm/musca.o
/var/tmp/patchew-tester-tmp-r6tvf9a3/src/hw/virtio/virtio-pmem.c:21:10: fatal error: standard-headers/linux/virtio_pmem.h: No such file or directory
   21 | #include "standard-headers/linux/virtio_pmem.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.


The full log is available at
http://patchew.org/logs/20190711172845.31035-1-eric.auger@redhat.com/testing.s390x/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com