MAINTAINERS | 2 + include/hw/i386/intel_iommu.h | 2 + include/hw/pci/pci.h | 38 ++++- include/hw/vfio/vfio-common.h | 8 + include/hw/vfio/vfio-container-base.h | 3 + include/qemu/range.h | 11 ++ include/sysemu/host_iommu_device.h | 91 ++++++++++++ include/sysemu/iommufd.h | 19 +++ backends/host_iommu_device.c | 33 +++++ backends/iommufd.c | 76 ++++++++-- hw/i386/intel_iommu.c | 203 ++++++++++++++++++++------ hw/pci/pci.c | 75 +++++++++- hw/vfio/common.c | 16 +- hw/vfio/container.c | 41 +++++- hw/vfio/helpers.c | 17 +++ hw/vfio/iommufd.c | 37 ++++- hw/vfio/pci.c | 19 ++- backends/meson.build | 1 + 18 files changed, 623 insertions(+), 69 deletions(-) create mode 100644 include/sysemu/host_iommu_device.h create mode 100644 backends/host_iommu_device.c
Hi,
This series introduce a HostIOMMUDevice abstraction and sub-classes.
Also HostIOMMUDeviceCaps structure in HostIOMMUDevice and a new interface
between vIOMMU and HostIOMMUDevice.
A HostIOMMUDevice is an abstraction for an assigned device that is protected
by a physical IOMMU (aka host IOMMU). The userspace interaction with this
physical IOMMU can be done either through the VFIO IOMMU type 1 legacy
backend or the new iommufd backend. The assigned device can be a VFIO device
or a VDPA device. The HostIOMMUDevice is needed to interact with the host
IOMMU that protects the assigned device. It is especially useful when the
device is also protected by a virtual IOMMU as this latter use the translation
services of the physical IOMMU and is constrained by it. In that context the
HostIOMMUDevice can be passed to the virtual IOMMU to collect physical IOMMU
capabilities such as the supported address width. In the future, the virtual
IOMMU will use the HostIOMMUDevice to program the guest page tables in the
first translation stage of the physical IOMMU.
HostIOMMUDeviceClass::realize() is introduced to initialize
HostIOMMUDeviceCaps and other fields of HostIOMMUDevice variants.
HostIOMMUDeviceClass::get_cap() is introduced to query host IOMMU
device capabilities.
The class tree is as below:
HostIOMMUDevice
| .caps
| .realize()
| .get_cap()
|
.-----------------------------------------------.
| | |
HostIOMMUDeviceLegacyVFIO {HostIOMMUDeviceLegacyVDPA} HostIOMMUDeviceIOMMUFD
| | | [.iommufd]
| [.devid]
| [.ioas_id]
| [.attach_hwpt()]
| [.detach_hwpt()]
|
.----------------------.
| |
HostIOMMUDeviceIOMMUFDVFIO {HostIOMMUDeviceIOMMUFDVDPA}
| [.vdev] | {.vdev}
* The attributes in [] will be implemented in nesting series.
* The classes in {} will be implemented in future.
* .vdev in different class points to different agent device,
* i.e., VFIODevice or VDPADevice.
PATCH1-4: Introduce HostIOMMUDevice and its sub classes
PATCH5-10: Implement .realize() and .get_cap() handler
PATCH11-14: Create HostIOMMUDevice instance and pass to vIOMMU
PATCH15-17: Implement compatibility check between host IOMMU and vIOMMU(intel_iommu)
Test done:
make check
vfio device hotplug/unplug with different backend on linux
reboot, kexec
build test on linux and windows11
Qemu code can be found at:
https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_preq_v7
Besides the compatibility check in this series, in nesting series, this
host IOMMU device is extended for much wider usage. For anyone interested
on the nesting series, here is the link:
https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_rfcv2
Thanks
Zhenzhong
Changelog:
v7:
- drop config CONFIG_HOST_IOMMU_DEVICE (Cédric)
- introduce HOST_IOMMU_DEVICE_CAP_AW_BITS_MAX (Eric)
- use iova_ranges method in iommufd.realize() (Eric)
- introduce HostIOMMUDevice::name to facilitate tracing (Eric)
- implement a custom destroy hash function (Cédric)
- drop VTDHostIOMMUDevice and save HostIOMMUDevice in hash table (Eric)
- move patch5 after patch1 (Eric)
- squash patch3 and 4, squash patch12 and 13 (Eric)
- refine comments (Eric)
- collect Eric's R-B
v6:
- open coded host_iommu_device_get_cap() to avoid #ifdef in intel_iommu.c (Cédric)
v5:
- pci_device_set_iommu_device return true (Cédric)
- fix build failure on windows (thanks Cédric found that issue)
v4:
- move properties vdev, iommufd and devid to nesting series where need it (Cédric)
- fix 32bit build with clz64 (Cédric)
- change check_cap naming to get_cap (Cédric)
- return bool if error is passed through errp (Cédric)
- drop HostIOMMUDevice[LegacyVFIO|IOMMUFD|IOMMUFDVFIO] declaration (Cédric)
- drop HOST_IOMMU_DEVICE_CAP_IOMMUFD (Cédric)
- replace include directive with forward declaration (Cédric)
v3:
- refine declaration and doc for HostIOMMUDevice (Cédric, Philippe)
- introduce HostIOMMUDeviceCaps, .realize() and .check_cap() (Cédric)
- introduce helper range_get_last_bit() for range operation (Cédric)
- separate pci_device_get_iommu_bus_devfn() in a prereq patch (Cédric)
- replace HIOD_ abbreviation with HOST_IOMMU_DEVICE_ (Cédric)
- add header in include/sysemu/iommufd.h (Cédric)
v2:
- use QOM to abstract host IOMMU device and its sub-classes (Cédric)
- move host IOMMU device creation in attach_device() (Cédric)
- refine pci_device_set/unset_iommu_device doc further (Eric)
- define host IOMMU info format of different backend
- implement get_host_iommu_info() for different backend (Cédric)
- drop cap/ecap update logic (MST)
- check aw-bits from get_host_iommu_info() in legacy mode
v1:
- use HostIOMMUDevice handle instead of union in VFIODevice (Eric)
- change host_iommu_device_init to host_iommu_device_create
- allocate HostIOMMUDevice in host_iommu_device_create callback
and set the VFIODevice base_hdev handle (Eric)
- refine pci_device_set/unset_iommu_device doc (Eric)
- use HostIOMMUDevice handle instead of union in VTDHostIOMMUDevice (Eric)
- convert HostIOMMUDevice to sub object pointer in vtd_check_hdev
rfcv2:
- introduce common abstract HostIOMMUDevice and sub struct for different BEs (Eric, Cédric)
- remove iommufd_device.[ch] (Cédric)
- remove duplicate iommufd/devid define from VFIODevice (Eric)
- drop the p in aliased_pbus and aliased_pdevfn (Eric)
- assert devfn and iommu_bus in pci_device_get_iommu_bus_devfn (Cédric, Eric)
- use errp in iommufd_device_get_info (Eric)
- split and simplify cap/ecap check/sync code in intel_iommu.c (Cédric)
- move VTDHostIOMMUDevice declaration to intel_iommu_internal.h (Cédric)
- make '(vtd->cap_reg >> 16) & 0x3fULL' a MACRO and add missed '+1' (Cédric)
- block migration if vIOMMU cap/ecap updated based on host IOMMU cap/ecap
- add R-B
Yi Liu (2):
hw/pci: Introduce pci_device_[set|unset]_iommu_device()
intel_iommu: Implement [set|unset]_iommu_device() callbacks
Zhenzhong Duan (15):
backends: Introduce HostIOMMUDevice abstract
backends/host_iommu_device: Introduce HostIOMMUDeviceCaps
vfio/container: Introduce TYPE_HOST_IOMMU_DEVICE_LEGACY_VFIO device
backends/iommufd: Introduce TYPE_HOST_IOMMU_DEVICE_IOMMUFD[_VFIO]
devices
range: Introduce range_get_last_bit()
vfio/container: Implement HostIOMMUDeviceClass::realize() handler
backends/iommufd: Introduce helper function
iommufd_backend_get_device_info()
vfio/iommufd: Implement HostIOMMUDeviceClass::realize() handler
vfio/container: Implement HostIOMMUDeviceClass::get_cap() handler
backends/iommufd: Implement HostIOMMUDeviceClass::get_cap() handler
vfio: Create host IOMMU device instance
hw/pci: Introduce helper function pci_device_get_iommu_bus_devfn()
vfio/pci: Pass HostIOMMUDevice to vIOMMU
intel_iommu: Extract out vtd_cap_init() to initialize cap/ecap
intel_iommu: Check compatibility with host IOMMU capabilities
MAINTAINERS | 2 +
include/hw/i386/intel_iommu.h | 2 +
include/hw/pci/pci.h | 38 ++++-
include/hw/vfio/vfio-common.h | 8 +
include/hw/vfio/vfio-container-base.h | 3 +
include/qemu/range.h | 11 ++
include/sysemu/host_iommu_device.h | 91 ++++++++++++
include/sysemu/iommufd.h | 19 +++
backends/host_iommu_device.c | 33 +++++
backends/iommufd.c | 76 ++++++++--
hw/i386/intel_iommu.c | 203 ++++++++++++++++++++------
hw/pci/pci.c | 75 +++++++++-
hw/vfio/common.c | 16 +-
hw/vfio/container.c | 41 +++++-
hw/vfio/helpers.c | 17 +++
hw/vfio/iommufd.c | 37 ++++-
hw/vfio/pci.c | 19 ++-
backends/meson.build | 1 +
18 files changed, 623 insertions(+), 69 deletions(-)
create mode 100644 include/sysemu/host_iommu_device.h
create mode 100644 backends/host_iommu_device.c
--
2.34.1
Hi Zhenzhong,
On 6/5/24 10:30, Zhenzhong Duan wrote:
> Hi,
>
> This series introduce a HostIOMMUDevice abstraction and sub-classes.
> Also HostIOMMUDeviceCaps structure in HostIOMMUDevice and a new interface
> between vIOMMU and HostIOMMUDevice.
>
> A HostIOMMUDevice is an abstraction for an assigned device that is protected
> by a physical IOMMU (aka host IOMMU). The userspace interaction with this
> physical IOMMU can be done either through the VFIO IOMMU type 1 legacy
> backend or the new iommufd backend. The assigned device can be a VFIO device
> or a VDPA device. The HostIOMMUDevice is needed to interact with the host
> IOMMU that protects the assigned device. It is especially useful when the
> device is also protected by a virtual IOMMU as this latter use the translation
> services of the physical IOMMU and is constrained by it. In that context the
> HostIOMMUDevice can be passed to the virtual IOMMU to collect physical IOMMU
> capabilities such as the supported address width. In the future, the virtual
> IOMMU will use the HostIOMMUDevice to program the guest page tables in the
> first translation stage of the physical IOMMU.
>
> HostIOMMUDeviceClass::realize() is introduced to initialize
> HostIOMMUDeviceCaps and other fields of HostIOMMUDevice variants.
>
> HostIOMMUDeviceClass::get_cap() is introduced to query host IOMMU
> device capabilities.
>
> The class tree is as below:
>
> HostIOMMUDevice
> | .caps
> | .realize()
> | .get_cap()
> |
> .-----------------------------------------------.
> | | |
> HostIOMMUDeviceLegacyVFIO {HostIOMMUDeviceLegacyVDPA} HostIOMMUDeviceIOMMUFD
> | | | [.iommufd]
> | [.devid]
> | [.ioas_id]
> | [.attach_hwpt()]
> | [.detach_hwpt()]
> |
> .----------------------.
> | |
> HostIOMMUDeviceIOMMUFDVFIO {HostIOMMUDeviceIOMMUFDVDPA}
> | [.vdev] | {.vdev}
>
> * The attributes in [] will be implemented in nesting series.
> * The classes in {} will be implemented in future.
> * .vdev in different class points to different agent device,
> * i.e., VFIODevice or VDPADevice.
>
> PATCH1-4: Introduce HostIOMMUDevice and its sub classes
> PATCH5-10: Implement .realize() and .get_cap() handler
> PATCH11-14: Create HostIOMMUDevice instance and pass to vIOMMU
> PATCH15-17: Implement compatibility check between host IOMMU and vIOMMU(intel_iommu)
>
> Test done:
> make check
> vfio device hotplug/unplug with different backend on linux
> reboot, kexec
> build test on linux and windows11
>
> Qemu code can be found at:
> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_preq_v7
>
> Besides the compatibility check in this series, in nesting series, this
> host IOMMU device is extended for much wider usage. For anyone interested
> on the nesting series, here is the link:
> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_rfcv2
>
> Thanks
> Zhenzhong
>
> Changelog:
> v7:
> - drop config CONFIG_HOST_IOMMU_DEVICE (Cédric)
> - introduce HOST_IOMMU_DEVICE_CAP_AW_BITS_MAX (Eric)
> - use iova_ranges method in iommufd.realize() (Eric)
> - introduce HostIOMMUDevice::name to facilitate tracing (Eric)
> - implement a custom destroy hash function (Cédric)
> - drop VTDHostIOMMUDevice and save HostIOMMUDevice in hash table (Eric)
> - move patch5 after patch1 (Eric)
> - squash patch3 and 4, squash patch12 and 13 (Eric)
> - refine comments (Eric)
> - collect Eric's R-B
for the whole series:
Reviewed-by: Eric Auger <eric.auger@redhat.com>
I exercised part of it using the virtio-iommu and this series on top
[RFC v2 0/7] VIRTIO-IOMMU/VFIO: Fix host iommu geometry handling for
hotplugged devices
Thanks
Eric
>
> v6:
> - open coded host_iommu_device_get_cap() to avoid #ifdef in intel_iommu.c (Cédric)
>
> v5:
> - pci_device_set_iommu_device return true (Cédric)
> - fix build failure on windows (thanks Cédric found that issue)
>
> v4:
> - move properties vdev, iommufd and devid to nesting series where need it (Cédric)
> - fix 32bit build with clz64 (Cédric)
> - change check_cap naming to get_cap (Cédric)
> - return bool if error is passed through errp (Cédric)
> - drop HostIOMMUDevice[LegacyVFIO|IOMMUFD|IOMMUFDVFIO] declaration (Cédric)
> - drop HOST_IOMMU_DEVICE_CAP_IOMMUFD (Cédric)
> - replace include directive with forward declaration (Cédric)
>
> v3:
> - refine declaration and doc for HostIOMMUDevice (Cédric, Philippe)
> - introduce HostIOMMUDeviceCaps, .realize() and .check_cap() (Cédric)
> - introduce helper range_get_last_bit() for range operation (Cédric)
> - separate pci_device_get_iommu_bus_devfn() in a prereq patch (Cédric)
> - replace HIOD_ abbreviation with HOST_IOMMU_DEVICE_ (Cédric)
> - add header in include/sysemu/iommufd.h (Cédric)
>
> v2:
> - use QOM to abstract host IOMMU device and its sub-classes (Cédric)
> - move host IOMMU device creation in attach_device() (Cédric)
> - refine pci_device_set/unset_iommu_device doc further (Eric)
> - define host IOMMU info format of different backend
> - implement get_host_iommu_info() for different backend (Cédric)
> - drop cap/ecap update logic (MST)
> - check aw-bits from get_host_iommu_info() in legacy mode
>
> v1:
> - use HostIOMMUDevice handle instead of union in VFIODevice (Eric)
> - change host_iommu_device_init to host_iommu_device_create
> - allocate HostIOMMUDevice in host_iommu_device_create callback
> and set the VFIODevice base_hdev handle (Eric)
> - refine pci_device_set/unset_iommu_device doc (Eric)
> - use HostIOMMUDevice handle instead of union in VTDHostIOMMUDevice (Eric)
> - convert HostIOMMUDevice to sub object pointer in vtd_check_hdev
>
> rfcv2:
> - introduce common abstract HostIOMMUDevice and sub struct for different BEs (Eric, Cédric)
> - remove iommufd_device.[ch] (Cédric)
> - remove duplicate iommufd/devid define from VFIODevice (Eric)
> - drop the p in aliased_pbus and aliased_pdevfn (Eric)
> - assert devfn and iommu_bus in pci_device_get_iommu_bus_devfn (Cédric, Eric)
> - use errp in iommufd_device_get_info (Eric)
> - split and simplify cap/ecap check/sync code in intel_iommu.c (Cédric)
> - move VTDHostIOMMUDevice declaration to intel_iommu_internal.h (Cédric)
> - make '(vtd->cap_reg >> 16) & 0x3fULL' a MACRO and add missed '+1' (Cédric)
> - block migration if vIOMMU cap/ecap updated based on host IOMMU cap/ecap
> - add R-B
>
> Yi Liu (2):
> hw/pci: Introduce pci_device_[set|unset]_iommu_device()
> intel_iommu: Implement [set|unset]_iommu_device() callbacks
>
> Zhenzhong Duan (15):
> backends: Introduce HostIOMMUDevice abstract
> backends/host_iommu_device: Introduce HostIOMMUDeviceCaps
> vfio/container: Introduce TYPE_HOST_IOMMU_DEVICE_LEGACY_VFIO device
> backends/iommufd: Introduce TYPE_HOST_IOMMU_DEVICE_IOMMUFD[_VFIO]
> devices
> range: Introduce range_get_last_bit()
> vfio/container: Implement HostIOMMUDeviceClass::realize() handler
> backends/iommufd: Introduce helper function
> iommufd_backend_get_device_info()
> vfio/iommufd: Implement HostIOMMUDeviceClass::realize() handler
> vfio/container: Implement HostIOMMUDeviceClass::get_cap() handler
> backends/iommufd: Implement HostIOMMUDeviceClass::get_cap() handler
> vfio: Create host IOMMU device instance
> hw/pci: Introduce helper function pci_device_get_iommu_bus_devfn()
> vfio/pci: Pass HostIOMMUDevice to vIOMMU
> intel_iommu: Extract out vtd_cap_init() to initialize cap/ecap
> intel_iommu: Check compatibility with host IOMMU capabilities
>
> MAINTAINERS | 2 +
> include/hw/i386/intel_iommu.h | 2 +
> include/hw/pci/pci.h | 38 ++++-
> include/hw/vfio/vfio-common.h | 8 +
> include/hw/vfio/vfio-container-base.h | 3 +
> include/qemu/range.h | 11 ++
> include/sysemu/host_iommu_device.h | 91 ++++++++++++
> include/sysemu/iommufd.h | 19 +++
> backends/host_iommu_device.c | 33 +++++
> backends/iommufd.c | 76 ++++++++--
> hw/i386/intel_iommu.c | 203 ++++++++++++++++++++------
> hw/pci/pci.c | 75 +++++++++-
> hw/vfio/common.c | 16 +-
> hw/vfio/container.c | 41 +++++-
> hw/vfio/helpers.c | 17 +++
> hw/vfio/iommufd.c | 37 ++++-
> hw/vfio/pci.c | 19 ++-
> backends/meson.build | 1 +
> 18 files changed, 623 insertions(+), 69 deletions(-)
> create mode 100644 include/sysemu/host_iommu_device.h
> create mode 100644 backends/host_iommu_device.c
>
>-----Original Message-----
>From: Eric Auger <eric.auger@redhat.com>
>Subject: Re: [PATCH v7 00/17] Add a host IOMMU device abstraction to
>check with vIOMMU
>
>Hi Zhenzhong,
>
>On 6/5/24 10:30, Zhenzhong Duan wrote:
>> Hi,
>>
>> This series introduce a HostIOMMUDevice abstraction and sub-classes.
>> Also HostIOMMUDeviceCaps structure in HostIOMMUDevice and a new
>interface
>> between vIOMMU and HostIOMMUDevice.
>>
>> A HostIOMMUDevice is an abstraction for an assigned device that is
>protected
>> by a physical IOMMU (aka host IOMMU). The userspace interaction with
>this
>> physical IOMMU can be done either through the VFIO IOMMU type 1
>legacy
>> backend or the new iommufd backend. The assigned device can be a VFIO
>device
>> or a VDPA device. The HostIOMMUDevice is needed to interact with the
>host
>> IOMMU that protects the assigned device. It is especially useful when the
>> device is also protected by a virtual IOMMU as this latter use the
>translation
>> services of the physical IOMMU and is constrained by it. In that context the
>> HostIOMMUDevice can be passed to the virtual IOMMU to collect physical
>IOMMU
>> capabilities such as the supported address width. In the future, the virtual
>> IOMMU will use the HostIOMMUDevice to program the guest page tables
>in the
>> first translation stage of the physical IOMMU.
>>
>> HostIOMMUDeviceClass::realize() is introduced to initialize
>> HostIOMMUDeviceCaps and other fields of HostIOMMUDevice variants.
>>
>> HostIOMMUDeviceClass::get_cap() is introduced to query host IOMMU
>> device capabilities.
>>
>> The class tree is as below:
>>
>> HostIOMMUDevice
>> | .caps
>> | .realize()
>> | .get_cap()
>> |
>> .-----------------------------------------------.
>> | | |
>> HostIOMMUDeviceLegacyVFIO {HostIOMMUDeviceLegacyVDPA}
>HostIOMMUDeviceIOMMUFD
>> | | | [.iommufd]
>> | [.devid]
>> | [.ioas_id]
>> | [.attach_hwpt()]
>> | [.detach_hwpt()]
>> |
>> .----------------------.
>> | |
>> HostIOMMUDeviceIOMMUFDVFIO
>{HostIOMMUDeviceIOMMUFDVDPA}
>> | [.vdev] | {.vdev}
>>
>> * The attributes in [] will be implemented in nesting series.
>> * The classes in {} will be implemented in future.
>> * .vdev in different class points to different agent device,
>> * i.e., VFIODevice or VDPADevice.
>>
>> PATCH1-4: Introduce HostIOMMUDevice and its sub classes
>> PATCH5-10: Implement .realize() and .get_cap() handler
>> PATCH11-14: Create HostIOMMUDevice instance and pass to vIOMMU
>> PATCH15-17: Implement compatibility check between host IOMMU and
>vIOMMU(intel_iommu)
>>
>> Test done:
>> make check
>> vfio device hotplug/unplug with different backend on linux
>> reboot, kexec
>> build test on linux and windows11
>>
>> Qemu code can be found at:
>>
>https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_pre
>q_v7
>>
>> Besides the compatibility check in this series, in nesting series, this
>> host IOMMU device is extended for much wider usage. For anyone
>interested
>> on the nesting series, here is the link:
>>
>https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_rfc
>v2
>>
>> Thanks
>> Zhenzhong
>>
>> Changelog:
>> v7:
>> - drop config CONFIG_HOST_IOMMU_DEVICE (Cédric)
>> - introduce HOST_IOMMU_DEVICE_CAP_AW_BITS_MAX (Eric)
>> - use iova_ranges method in iommufd.realize() (Eric)
>> - introduce HostIOMMUDevice::name to facilitate tracing (Eric)
>> - implement a custom destroy hash function (Cédric)
>> - drop VTDHostIOMMUDevice and save HostIOMMUDevice in hash table
>(Eric)
>> - move patch5 after patch1 (Eric)
>> - squash patch3 and 4, squash patch12 and 13 (Eric)
>> - refine comments (Eric)
>> - collect Eric's R-B
>
>for the whole series:
>Reviewed-by: Eric Auger <eric.auger@redhat.com>
Thanks Eric.
>
>I exercised part of it using the virtio-iommu and this series on top
>[RFC v2 0/7] VIRTIO-IOMMU/VFIO: Fix host iommu geometry handling for
>hotplugged devices
You are super-efficient😊
BRs.
Zhenzhong
On 6/5/24 10:30 AM, Zhenzhong Duan wrote:
> Hi,
>
> This series introduce a HostIOMMUDevice abstraction and sub-classes.
> Also HostIOMMUDeviceCaps structure in HostIOMMUDevice and a new interface
> between vIOMMU and HostIOMMUDevice.
>
> A HostIOMMUDevice is an abstraction for an assigned device that is protected
> by a physical IOMMU (aka host IOMMU). The userspace interaction with this
> physical IOMMU can be done either through the VFIO IOMMU type 1 legacy
> backend or the new iommufd backend. The assigned device can be a VFIO device
> or a VDPA device. The HostIOMMUDevice is needed to interact with the host
> IOMMU that protects the assigned device. It is especially useful when the
> device is also protected by a virtual IOMMU as this latter use the translation
> services of the physical IOMMU and is constrained by it. In that context the
> HostIOMMUDevice can be passed to the virtual IOMMU to collect physical IOMMU
> capabilities such as the supported address width. In the future, the virtual
> IOMMU will use the HostIOMMUDevice to program the guest page tables in the
> first translation stage of the physical IOMMU.
>
> HostIOMMUDeviceClass::realize() is introduced to initialize
> HostIOMMUDeviceCaps and other fields of HostIOMMUDevice variants.
>
> HostIOMMUDeviceClass::get_cap() is introduced to query host IOMMU
> device capabilities.
>
> The class tree is as below:
>
> HostIOMMUDevice
> | .caps
> | .realize()
> | .get_cap()
> |
> .-----------------------------------------------.
> | | |
> HostIOMMUDeviceLegacyVFIO {HostIOMMUDeviceLegacyVDPA} HostIOMMUDeviceIOMMUFD
> | | | [.iommufd]
> | [.devid]
> | [.ioas_id]
> | [.attach_hwpt()]
> | [.detach_hwpt()]
> |
> .----------------------.
> | |
> HostIOMMUDeviceIOMMUFDVFIO {HostIOMMUDeviceIOMMUFDVDPA}
> | [.vdev] | {.vdev}
>
> * The attributes in [] will be implemented in nesting series.
> * The classes in {} will be implemented in future.
> * .vdev in different class points to different agent device,
> * i.e., VFIODevice or VDPADevice.
>
> PATCH1-4: Introduce HostIOMMUDevice and its sub classes
> PATCH5-10: Implement .realize() and .get_cap() handler
> PATCH11-14: Create HostIOMMUDevice instance and pass to vIOMMU
> PATCH15-17: Implement compatibility check between host IOMMU and vIOMMU(intel_iommu)
>
> Test done:
> make check
> vfio device hotplug/unplug with different backend on linux
> reboot, kexec
> build test on linux and windows11
>
> Qemu code can be found at:
> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_preq_v7
>
> Besides the compatibility check in this series, in nesting series, this
> host IOMMU device is extended for much wider usage. For anyone interested
> on the nesting series, here is the link:
> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_rfcv2
>
> Thanks
> Zhenzhong
>
> Changelog:
> v7:
> - drop config CONFIG_HOST_IOMMU_DEVICE (Cédric)
> - introduce HOST_IOMMU_DEVICE_CAP_AW_BITS_MAX (Eric)
> - use iova_ranges method in iommufd.realize() (Eric)
> - introduce HostIOMMUDevice::name to facilitate tracing (Eric)
> - implement a custom destroy hash function (Cédric)
> - drop VTDHostIOMMUDevice and save HostIOMMUDevice in hash table (Eric)
> - move patch5 after patch1 (Eric)
> - squash patch3 and 4, squash patch12 and 13 (Eric)
> - refine comments (Eric)
> - collect Eric's R-B
>
> v6:
> - open coded host_iommu_device_get_cap() to avoid #ifdef in intel_iommu.c (Cédric)
>
> v5:
> - pci_device_set_iommu_device return true (Cédric)
> - fix build failure on windows (thanks Cédric found that issue)
>
> v4:
> - move properties vdev, iommufd and devid to nesting series where need it (Cédric)
> - fix 32bit build with clz64 (Cédric)
> - change check_cap naming to get_cap (Cédric)
> - return bool if error is passed through errp (Cédric)
> - drop HostIOMMUDevice[LegacyVFIO|IOMMUFD|IOMMUFDVFIO] declaration (Cédric)
> - drop HOST_IOMMU_DEVICE_CAP_IOMMUFD (Cédric)
> - replace include directive with forward declaration (Cédric)
>
> v3:
> - refine declaration and doc for HostIOMMUDevice (Cédric, Philippe)
> - introduce HostIOMMUDeviceCaps, .realize() and .check_cap() (Cédric)
> - introduce helper range_get_last_bit() for range operation (Cédric)
> - separate pci_device_get_iommu_bus_devfn() in a prereq patch (Cédric)
> - replace HIOD_ abbreviation with HOST_IOMMU_DEVICE_ (Cédric)
> - add header in include/sysemu/iommufd.h (Cédric)
>
> v2:
> - use QOM to abstract host IOMMU device and its sub-classes (Cédric)
> - move host IOMMU device creation in attach_device() (Cédric)
> - refine pci_device_set/unset_iommu_device doc further (Eric)
> - define host IOMMU info format of different backend
> - implement get_host_iommu_info() for different backend (Cédric)
> - drop cap/ecap update logic (MST)
> - check aw-bits from get_host_iommu_info() in legacy mode
>
> v1:
> - use HostIOMMUDevice handle instead of union in VFIODevice (Eric)
> - change host_iommu_device_init to host_iommu_device_create
> - allocate HostIOMMUDevice in host_iommu_device_create callback
> and set the VFIODevice base_hdev handle (Eric)
> - refine pci_device_set/unset_iommu_device doc (Eric)
> - use HostIOMMUDevice handle instead of union in VTDHostIOMMUDevice (Eric)
> - convert HostIOMMUDevice to sub object pointer in vtd_check_hdev
>
> rfcv2:
> - introduce common abstract HostIOMMUDevice and sub struct for different BEs (Eric, Cédric)
> - remove iommufd_device.[ch] (Cédric)
> - remove duplicate iommufd/devid define from VFIODevice (Eric)
> - drop the p in aliased_pbus and aliased_pdevfn (Eric)
> - assert devfn and iommu_bus in pci_device_get_iommu_bus_devfn (Cédric, Eric)
> - use errp in iommufd_device_get_info (Eric)
> - split and simplify cap/ecap check/sync code in intel_iommu.c (Cédric)
> - move VTDHostIOMMUDevice declaration to intel_iommu_internal.h (Cédric)
> - make '(vtd->cap_reg >> 16) & 0x3fULL' a MACRO and add missed '+1' (Cédric)
> - block migration if vIOMMU cap/ecap updated based on host IOMMU cap/ecap
> - add R-B
>
> Yi Liu (2):
> hw/pci: Introduce pci_device_[set|unset]_iommu_device()
> intel_iommu: Implement [set|unset]_iommu_device() callbacks
>
> Zhenzhong Duan (15):
> backends: Introduce HostIOMMUDevice abstract
> backends/host_iommu_device: Introduce HostIOMMUDeviceCaps
> vfio/container: Introduce TYPE_HOST_IOMMU_DEVICE_LEGACY_VFIO device
> backends/iommufd: Introduce TYPE_HOST_IOMMU_DEVICE_IOMMUFD[_VFIO]
> devices
> range: Introduce range_get_last_bit()
> vfio/container: Implement HostIOMMUDeviceClass::realize() handler
> backends/iommufd: Introduce helper function
> iommufd_backend_get_device_info()
> vfio/iommufd: Implement HostIOMMUDeviceClass::realize() handler
> vfio/container: Implement HostIOMMUDeviceClass::get_cap() handler
> backends/iommufd: Implement HostIOMMUDeviceClass::get_cap() handler
> vfio: Create host IOMMU device instance
> hw/pci: Introduce helper function pci_device_get_iommu_bus_devfn()
> vfio/pci: Pass HostIOMMUDevice to vIOMMU
> intel_iommu: Extract out vtd_cap_init() to initialize cap/ecap
> intel_iommu: Check compatibility with host IOMMU capabilities
>
> MAINTAINERS | 2 +
> include/hw/i386/intel_iommu.h | 2 +
> include/hw/pci/pci.h | 38 ++++-
> include/hw/vfio/vfio-common.h | 8 +
> include/hw/vfio/vfio-container-base.h | 3 +
> include/qemu/range.h | 11 ++
> include/sysemu/host_iommu_device.h | 91 ++++++++++++
> include/sysemu/iommufd.h | 19 +++
> backends/host_iommu_device.c | 33 +++++
> backends/iommufd.c | 76 ++++++++--
> hw/i386/intel_iommu.c | 203 ++++++++++++++++++++------
> hw/pci/pci.c | 75 +++++++++-
> hw/vfio/common.c | 16 +-
> hw/vfio/container.c | 41 +++++-
> hw/vfio/helpers.c | 17 +++
> hw/vfio/iommufd.c | 37 ++++-
> hw/vfio/pci.c | 19 ++-
> backends/meson.build | 1 +
> 18 files changed, 623 insertions(+), 69 deletions(-)
> create mode 100644 include/sysemu/host_iommu_device.h
> create mode 100644 backends/host_iommu_device.c
>
Applied to vfio-next.
Thanks,
C.
On Wed, Jun 05, 2024 at 04:30:26PM +0800, Zhenzhong Duan wrote:
> Hi,
>
> This series introduce a HostIOMMUDevice abstraction and sub-classes.
> Also HostIOMMUDeviceCaps structure in HostIOMMUDevice and a new interface
> between vIOMMU and HostIOMMUDevice.
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Who is merging this? Me? Or Alex?
> A HostIOMMUDevice is an abstraction for an assigned device that is protected
> by a physical IOMMU (aka host IOMMU). The userspace interaction with this
> physical IOMMU can be done either through the VFIO IOMMU type 1 legacy
> backend or the new iommufd backend. The assigned device can be a VFIO device
> or a VDPA device. The HostIOMMUDevice is needed to interact with the host
> IOMMU that protects the assigned device. It is especially useful when the
> device is also protected by a virtual IOMMU as this latter use the translation
> services of the physical IOMMU and is constrained by it. In that context the
> HostIOMMUDevice can be passed to the virtual IOMMU to collect physical IOMMU
> capabilities such as the supported address width. In the future, the virtual
> IOMMU will use the HostIOMMUDevice to program the guest page tables in the
> first translation stage of the physical IOMMU.
>
> HostIOMMUDeviceClass::realize() is introduced to initialize
> HostIOMMUDeviceCaps and other fields of HostIOMMUDevice variants.
>
> HostIOMMUDeviceClass::get_cap() is introduced to query host IOMMU
> device capabilities.
>
> The class tree is as below:
>
> HostIOMMUDevice
> | .caps
> | .realize()
> | .get_cap()
> |
> .-----------------------------------------------.
> | | |
> HostIOMMUDeviceLegacyVFIO {HostIOMMUDeviceLegacyVDPA} HostIOMMUDeviceIOMMUFD
> | | | [.iommufd]
> | [.devid]
> | [.ioas_id]
> | [.attach_hwpt()]
> | [.detach_hwpt()]
> |
> .----------------------.
> | |
> HostIOMMUDeviceIOMMUFDVFIO {HostIOMMUDeviceIOMMUFDVDPA}
> | [.vdev] | {.vdev}
>
> * The attributes in [] will be implemented in nesting series.
> * The classes in {} will be implemented in future.
> * .vdev in different class points to different agent device,
> * i.e., VFIODevice or VDPADevice.
>
> PATCH1-4: Introduce HostIOMMUDevice and its sub classes
> PATCH5-10: Implement .realize() and .get_cap() handler
> PATCH11-14: Create HostIOMMUDevice instance and pass to vIOMMU
> PATCH15-17: Implement compatibility check between host IOMMU and vIOMMU(intel_iommu)
>
> Test done:
> make check
> vfio device hotplug/unplug with different backend on linux
> reboot, kexec
> build test on linux and windows11
>
> Qemu code can be found at:
> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_preq_v7
>
> Besides the compatibility check in this series, in nesting series, this
> host IOMMU device is extended for much wider usage. For anyone interested
> on the nesting series, here is the link:
> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_rfcv2
>
> Thanks
> Zhenzhong
>
> Changelog:
> v7:
> - drop config CONFIG_HOST_IOMMU_DEVICE (Cédric)
> - introduce HOST_IOMMU_DEVICE_CAP_AW_BITS_MAX (Eric)
> - use iova_ranges method in iommufd.realize() (Eric)
> - introduce HostIOMMUDevice::name to facilitate tracing (Eric)
> - implement a custom destroy hash function (Cédric)
> - drop VTDHostIOMMUDevice and save HostIOMMUDevice in hash table (Eric)
> - move patch5 after patch1 (Eric)
> - squash patch3 and 4, squash patch12 and 13 (Eric)
> - refine comments (Eric)
> - collect Eric's R-B
>
> v6:
> - open coded host_iommu_device_get_cap() to avoid #ifdef in intel_iommu.c (Cédric)
>
> v5:
> - pci_device_set_iommu_device return true (Cédric)
> - fix build failure on windows (thanks Cédric found that issue)
>
> v4:
> - move properties vdev, iommufd and devid to nesting series where need it (Cédric)
> - fix 32bit build with clz64 (Cédric)
> - change check_cap naming to get_cap (Cédric)
> - return bool if error is passed through errp (Cédric)
> - drop HostIOMMUDevice[LegacyVFIO|IOMMUFD|IOMMUFDVFIO] declaration (Cédric)
> - drop HOST_IOMMU_DEVICE_CAP_IOMMUFD (Cédric)
> - replace include directive with forward declaration (Cédric)
>
> v3:
> - refine declaration and doc for HostIOMMUDevice (Cédric, Philippe)
> - introduce HostIOMMUDeviceCaps, .realize() and .check_cap() (Cédric)
> - introduce helper range_get_last_bit() for range operation (Cédric)
> - separate pci_device_get_iommu_bus_devfn() in a prereq patch (Cédric)
> - replace HIOD_ abbreviation with HOST_IOMMU_DEVICE_ (Cédric)
> - add header in include/sysemu/iommufd.h (Cédric)
>
> v2:
> - use QOM to abstract host IOMMU device and its sub-classes (Cédric)
> - move host IOMMU device creation in attach_device() (Cédric)
> - refine pci_device_set/unset_iommu_device doc further (Eric)
> - define host IOMMU info format of different backend
> - implement get_host_iommu_info() for different backend (Cédric)
> - drop cap/ecap update logic (MST)
> - check aw-bits from get_host_iommu_info() in legacy mode
>
> v1:
> - use HostIOMMUDevice handle instead of union in VFIODevice (Eric)
> - change host_iommu_device_init to host_iommu_device_create
> - allocate HostIOMMUDevice in host_iommu_device_create callback
> and set the VFIODevice base_hdev handle (Eric)
> - refine pci_device_set/unset_iommu_device doc (Eric)
> - use HostIOMMUDevice handle instead of union in VTDHostIOMMUDevice (Eric)
> - convert HostIOMMUDevice to sub object pointer in vtd_check_hdev
>
> rfcv2:
> - introduce common abstract HostIOMMUDevice and sub struct for different BEs (Eric, Cédric)
> - remove iommufd_device.[ch] (Cédric)
> - remove duplicate iommufd/devid define from VFIODevice (Eric)
> - drop the p in aliased_pbus and aliased_pdevfn (Eric)
> - assert devfn and iommu_bus in pci_device_get_iommu_bus_devfn (Cédric, Eric)
> - use errp in iommufd_device_get_info (Eric)
> - split and simplify cap/ecap check/sync code in intel_iommu.c (Cédric)
> - move VTDHostIOMMUDevice declaration to intel_iommu_internal.h (Cédric)
> - make '(vtd->cap_reg >> 16) & 0x3fULL' a MACRO and add missed '+1' (Cédric)
> - block migration if vIOMMU cap/ecap updated based on host IOMMU cap/ecap
> - add R-B
>
> Yi Liu (2):
> hw/pci: Introduce pci_device_[set|unset]_iommu_device()
> intel_iommu: Implement [set|unset]_iommu_device() callbacks
>
> Zhenzhong Duan (15):
> backends: Introduce HostIOMMUDevice abstract
> backends/host_iommu_device: Introduce HostIOMMUDeviceCaps
> vfio/container: Introduce TYPE_HOST_IOMMU_DEVICE_LEGACY_VFIO device
> backends/iommufd: Introduce TYPE_HOST_IOMMU_DEVICE_IOMMUFD[_VFIO]
> devices
> range: Introduce range_get_last_bit()
> vfio/container: Implement HostIOMMUDeviceClass::realize() handler
> backends/iommufd: Introduce helper function
> iommufd_backend_get_device_info()
> vfio/iommufd: Implement HostIOMMUDeviceClass::realize() handler
> vfio/container: Implement HostIOMMUDeviceClass::get_cap() handler
> backends/iommufd: Implement HostIOMMUDeviceClass::get_cap() handler
> vfio: Create host IOMMU device instance
> hw/pci: Introduce helper function pci_device_get_iommu_bus_devfn()
> vfio/pci: Pass HostIOMMUDevice to vIOMMU
> intel_iommu: Extract out vtd_cap_init() to initialize cap/ecap
> intel_iommu: Check compatibility with host IOMMU capabilities
>
> MAINTAINERS | 2 +
> include/hw/i386/intel_iommu.h | 2 +
> include/hw/pci/pci.h | 38 ++++-
> include/hw/vfio/vfio-common.h | 8 +
> include/hw/vfio/vfio-container-base.h | 3 +
> include/qemu/range.h | 11 ++
> include/sysemu/host_iommu_device.h | 91 ++++++++++++
> include/sysemu/iommufd.h | 19 +++
> backends/host_iommu_device.c | 33 +++++
> backends/iommufd.c | 76 ++++++++--
> hw/i386/intel_iommu.c | 203 ++++++++++++++++++++------
> hw/pci/pci.c | 75 +++++++++-
> hw/vfio/common.c | 16 +-
> hw/vfio/container.c | 41 +++++-
> hw/vfio/helpers.c | 17 +++
> hw/vfio/iommufd.c | 37 ++++-
> hw/vfio/pci.c | 19 ++-
> backends/meson.build | 1 +
> 18 files changed, 623 insertions(+), 69 deletions(-)
> create mode 100644 include/sysemu/host_iommu_device.h
> create mode 100644 backends/host_iommu_device.c
>
> --
> 2.34.1
On 6/24/24 12:26 PM, Michael S. Tsirkin wrote:
> On Wed, Jun 05, 2024 at 04:30:26PM +0800, Zhenzhong Duan wrote:
>> Hi,
>>
>> This series introduce a HostIOMMUDevice abstraction and sub-classes.
>> Also HostIOMMUDeviceCaps structure in HostIOMMUDevice and a new interface
>> between vIOMMU and HostIOMMUDevice.
>
> Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
>
> Who is merging this? Me? Or Alex?
I will and I will include this series also :
[v4] VIRTIO-IOMMU/VFIO: Fix host iommu geometry
https://lore.kernel.org/all/20240614095402.904691-1-eric.auger@redhat.com
Thanks,
C.
>
>
>
>> A HostIOMMUDevice is an abstraction for an assigned device that is protected
>> by a physical IOMMU (aka host IOMMU). The userspace interaction with this
>> physical IOMMU can be done either through the VFIO IOMMU type 1 legacy
>> backend or the new iommufd backend. The assigned device can be a VFIO device
>> or a VDPA device. The HostIOMMUDevice is needed to interact with the host
>> IOMMU that protects the assigned device. It is especially useful when the
>> device is also protected by a virtual IOMMU as this latter use the translation
>> services of the physical IOMMU and is constrained by it. In that context the
>> HostIOMMUDevice can be passed to the virtual IOMMU to collect physical IOMMU
>> capabilities such as the supported address width. In the future, the virtual
>> IOMMU will use the HostIOMMUDevice to program the guest page tables in the
>> first translation stage of the physical IOMMU.
>>
>> HostIOMMUDeviceClass::realize() is introduced to initialize
>> HostIOMMUDeviceCaps and other fields of HostIOMMUDevice variants.
>>
>> HostIOMMUDeviceClass::get_cap() is introduced to query host IOMMU
>> device capabilities.
>>
>> The class tree is as below:
>>
>> HostIOMMUDevice
>> | .caps
>> | .realize()
>> | .get_cap()
>> |
>> .-----------------------------------------------.
>> | | |
>> HostIOMMUDeviceLegacyVFIO {HostIOMMUDeviceLegacyVDPA} HostIOMMUDeviceIOMMUFD
>> | | | [.iommufd]
>> | [.devid]
>> | [.ioas_id]
>> | [.attach_hwpt()]
>> | [.detach_hwpt()]
>> |
>> .----------------------.
>> | |
>> HostIOMMUDeviceIOMMUFDVFIO {HostIOMMUDeviceIOMMUFDVDPA}
>> | [.vdev] | {.vdev}
>>
>> * The attributes in [] will be implemented in nesting series.
>> * The classes in {} will be implemented in future.
>> * .vdev in different class points to different agent device,
>> * i.e., VFIODevice or VDPADevice.
>>
>> PATCH1-4: Introduce HostIOMMUDevice and its sub classes
>> PATCH5-10: Implement .realize() and .get_cap() handler
>> PATCH11-14: Create HostIOMMUDevice instance and pass to vIOMMU
>> PATCH15-17: Implement compatibility check between host IOMMU and vIOMMU(intel_iommu)
>>
>> Test done:
>> make check
>> vfio device hotplug/unplug with different backend on linux
>> reboot, kexec
>> build test on linux and windows11
>>
>> Qemu code can be found at:
>> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_preq_v7
>>
>> Besides the compatibility check in this series, in nesting series, this
>> host IOMMU device is extended for much wider usage. For anyone interested
>> on the nesting series, here is the link:
>> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_rfcv2
>>
>> Thanks
>> Zhenzhong
>>
>> Changelog:
>> v7:
>> - drop config CONFIG_HOST_IOMMU_DEVICE (Cédric)
>> - introduce HOST_IOMMU_DEVICE_CAP_AW_BITS_MAX (Eric)
>> - use iova_ranges method in iommufd.realize() (Eric)
>> - introduce HostIOMMUDevice::name to facilitate tracing (Eric)
>> - implement a custom destroy hash function (Cédric)
>> - drop VTDHostIOMMUDevice and save HostIOMMUDevice in hash table (Eric)
>> - move patch5 after patch1 (Eric)
>> - squash patch3 and 4, squash patch12 and 13 (Eric)
>> - refine comments (Eric)
>> - collect Eric's R-B
>>
>> v6:
>> - open coded host_iommu_device_get_cap() to avoid #ifdef in intel_iommu.c (Cédric)
>>
>> v5:
>> - pci_device_set_iommu_device return true (Cédric)
>> - fix build failure on windows (thanks Cédric found that issue)
>>
>> v4:
>> - move properties vdev, iommufd and devid to nesting series where need it (Cédric)
>> - fix 32bit build with clz64 (Cédric)
>> - change check_cap naming to get_cap (Cédric)
>> - return bool if error is passed through errp (Cédric)
>> - drop HostIOMMUDevice[LegacyVFIO|IOMMUFD|IOMMUFDVFIO] declaration (Cédric)
>> - drop HOST_IOMMU_DEVICE_CAP_IOMMUFD (Cédric)
>> - replace include directive with forward declaration (Cédric)
>>
>> v3:
>> - refine declaration and doc for HostIOMMUDevice (Cédric, Philippe)
>> - introduce HostIOMMUDeviceCaps, .realize() and .check_cap() (Cédric)
>> - introduce helper range_get_last_bit() for range operation (Cédric)
>> - separate pci_device_get_iommu_bus_devfn() in a prereq patch (Cédric)
>> - replace HIOD_ abbreviation with HOST_IOMMU_DEVICE_ (Cédric)
>> - add header in include/sysemu/iommufd.h (Cédric)
>>
>> v2:
>> - use QOM to abstract host IOMMU device and its sub-classes (Cédric)
>> - move host IOMMU device creation in attach_device() (Cédric)
>> - refine pci_device_set/unset_iommu_device doc further (Eric)
>> - define host IOMMU info format of different backend
>> - implement get_host_iommu_info() for different backend (Cédric)
>> - drop cap/ecap update logic (MST)
>> - check aw-bits from get_host_iommu_info() in legacy mode
>>
>> v1:
>> - use HostIOMMUDevice handle instead of union in VFIODevice (Eric)
>> - change host_iommu_device_init to host_iommu_device_create
>> - allocate HostIOMMUDevice in host_iommu_device_create callback
>> and set the VFIODevice base_hdev handle (Eric)
>> - refine pci_device_set/unset_iommu_device doc (Eric)
>> - use HostIOMMUDevice handle instead of union in VTDHostIOMMUDevice (Eric)
>> - convert HostIOMMUDevice to sub object pointer in vtd_check_hdev
>>
>> rfcv2:
>> - introduce common abstract HostIOMMUDevice and sub struct for different BEs (Eric, Cédric)
>> - remove iommufd_device.[ch] (Cédric)
>> - remove duplicate iommufd/devid define from VFIODevice (Eric)
>> - drop the p in aliased_pbus and aliased_pdevfn (Eric)
>> - assert devfn and iommu_bus in pci_device_get_iommu_bus_devfn (Cédric, Eric)
>> - use errp in iommufd_device_get_info (Eric)
>> - split and simplify cap/ecap check/sync code in intel_iommu.c (Cédric)
>> - move VTDHostIOMMUDevice declaration to intel_iommu_internal.h (Cédric)
>> - make '(vtd->cap_reg >> 16) & 0x3fULL' a MACRO and add missed '+1' (Cédric)
>> - block migration if vIOMMU cap/ecap updated based on host IOMMU cap/ecap
>> - add R-B
>>
>> Yi Liu (2):
>> hw/pci: Introduce pci_device_[set|unset]_iommu_device()
>> intel_iommu: Implement [set|unset]_iommu_device() callbacks
>>
>> Zhenzhong Duan (15):
>> backends: Introduce HostIOMMUDevice abstract
>> backends/host_iommu_device: Introduce HostIOMMUDeviceCaps
>> vfio/container: Introduce TYPE_HOST_IOMMU_DEVICE_LEGACY_VFIO device
>> backends/iommufd: Introduce TYPE_HOST_IOMMU_DEVICE_IOMMUFD[_VFIO]
>> devices
>> range: Introduce range_get_last_bit()
>> vfio/container: Implement HostIOMMUDeviceClass::realize() handler
>> backends/iommufd: Introduce helper function
>> iommufd_backend_get_device_info()
>> vfio/iommufd: Implement HostIOMMUDeviceClass::realize() handler
>> vfio/container: Implement HostIOMMUDeviceClass::get_cap() handler
>> backends/iommufd: Implement HostIOMMUDeviceClass::get_cap() handler
>> vfio: Create host IOMMU device instance
>> hw/pci: Introduce helper function pci_device_get_iommu_bus_devfn()
>> vfio/pci: Pass HostIOMMUDevice to vIOMMU
>> intel_iommu: Extract out vtd_cap_init() to initialize cap/ecap
>> intel_iommu: Check compatibility with host IOMMU capabilities
>>
>> MAINTAINERS | 2 +
>> include/hw/i386/intel_iommu.h | 2 +
>> include/hw/pci/pci.h | 38 ++++-
>> include/hw/vfio/vfio-common.h | 8 +
>> include/hw/vfio/vfio-container-base.h | 3 +
>> include/qemu/range.h | 11 ++
>> include/sysemu/host_iommu_device.h | 91 ++++++++++++
>> include/sysemu/iommufd.h | 19 +++
>> backends/host_iommu_device.c | 33 +++++
>> backends/iommufd.c | 76 ++++++++--
>> hw/i386/intel_iommu.c | 203 ++++++++++++++++++++------
>> hw/pci/pci.c | 75 +++++++++-
>> hw/vfio/common.c | 16 +-
>> hw/vfio/container.c | 41 +++++-
>> hw/vfio/helpers.c | 17 +++
>> hw/vfio/iommufd.c | 37 ++++-
>> hw/vfio/pci.c | 19 ++-
>> backends/meson.build | 1 +
>> 18 files changed, 623 insertions(+), 69 deletions(-)
>> create mode 100644 include/sysemu/host_iommu_device.h
>> create mode 100644 backends/host_iommu_device.c
>>
>> --
>> 2.34.1
>
On Mon, Jun 24, 2024 at 05:12:13PM +0200, Cédric Le Goater wrote:
> On 6/24/24 12:26 PM, Michael S. Tsirkin wrote:
> > On Wed, Jun 05, 2024 at 04:30:26PM +0800, Zhenzhong Duan wrote:
> > > Hi,
> > >
> > > This series introduce a HostIOMMUDevice abstraction and sub-classes.
> > > Also HostIOMMUDeviceCaps structure in HostIOMMUDevice and a new interface
> > > between vIOMMU and HostIOMMUDevice.
> >
> > Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
> >
> > Who is merging this? Me? Or Alex?
>
> I will and I will include this series also :
>
> [v4] VIRTIO-IOMMU/VFIO: Fix host iommu geometry
> https://lore.kernel.org/all/20240614095402.904691-1-eric.auger@redhat.com
>
> Thanks,
>
> C.
Sounds good. I sent ack for merging, both reviewed:
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
>
> >
> >
> >
> > > A HostIOMMUDevice is an abstraction for an assigned device that is protected
> > > by a physical IOMMU (aka host IOMMU). The userspace interaction with this
> > > physical IOMMU can be done either through the VFIO IOMMU type 1 legacy
> > > backend or the new iommufd backend. The assigned device can be a VFIO device
> > > or a VDPA device. The HostIOMMUDevice is needed to interact with the host
> > > IOMMU that protects the assigned device. It is especially useful when the
> > > device is also protected by a virtual IOMMU as this latter use the translation
> > > services of the physical IOMMU and is constrained by it. In that context the
> > > HostIOMMUDevice can be passed to the virtual IOMMU to collect physical IOMMU
> > > capabilities such as the supported address width. In the future, the virtual
> > > IOMMU will use the HostIOMMUDevice to program the guest page tables in the
> > > first translation stage of the physical IOMMU.
> > >
> > > HostIOMMUDeviceClass::realize() is introduced to initialize
> > > HostIOMMUDeviceCaps and other fields of HostIOMMUDevice variants.
> > >
> > > HostIOMMUDeviceClass::get_cap() is introduced to query host IOMMU
> > > device capabilities.
> > >
> > > The class tree is as below:
> > >
> > > HostIOMMUDevice
> > > | .caps
> > > | .realize()
> > > | .get_cap()
> > > |
> > > .-----------------------------------------------.
> > > | | |
> > > HostIOMMUDeviceLegacyVFIO {HostIOMMUDeviceLegacyVDPA} HostIOMMUDeviceIOMMUFD
> > > | | | [.iommufd]
> > > | [.devid]
> > > | [.ioas_id]
> > > | [.attach_hwpt()]
> > > | [.detach_hwpt()]
> > > |
> > > .----------------------.
> > > | |
> > > HostIOMMUDeviceIOMMUFDVFIO {HostIOMMUDeviceIOMMUFDVDPA}
> > > | [.vdev] | {.vdev}
> > >
> > > * The attributes in [] will be implemented in nesting series.
> > > * The classes in {} will be implemented in future.
> > > * .vdev in different class points to different agent device,
> > > * i.e., VFIODevice or VDPADevice.
> > >
> > > PATCH1-4: Introduce HostIOMMUDevice and its sub classes
> > > PATCH5-10: Implement .realize() and .get_cap() handler
> > > PATCH11-14: Create HostIOMMUDevice instance and pass to vIOMMU
> > > PATCH15-17: Implement compatibility check between host IOMMU and vIOMMU(intel_iommu)
> > >
> > > Test done:
> > > make check
> > > vfio device hotplug/unplug with different backend on linux
> > > reboot, kexec
> > > build test on linux and windows11
> > >
> > > Qemu code can be found at:
> > > https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_preq_v7
> > >
> > > Besides the compatibility check in this series, in nesting series, this
> > > host IOMMU device is extended for much wider usage. For anyone interested
> > > on the nesting series, here is the link:
> > > https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_rfcv2
> > >
> > > Thanks
> > > Zhenzhong
> > >
> > > Changelog:
> > > v7:
> > > - drop config CONFIG_HOST_IOMMU_DEVICE (Cédric)
> > > - introduce HOST_IOMMU_DEVICE_CAP_AW_BITS_MAX (Eric)
> > > - use iova_ranges method in iommufd.realize() (Eric)
> > > - introduce HostIOMMUDevice::name to facilitate tracing (Eric)
> > > - implement a custom destroy hash function (Cédric)
> > > - drop VTDHostIOMMUDevice and save HostIOMMUDevice in hash table (Eric)
> > > - move patch5 after patch1 (Eric)
> > > - squash patch3 and 4, squash patch12 and 13 (Eric)
> > > - refine comments (Eric)
> > > - collect Eric's R-B
> > >
> > > v6:
> > > - open coded host_iommu_device_get_cap() to avoid #ifdef in intel_iommu.c (Cédric)
> > >
> > > v5:
> > > - pci_device_set_iommu_device return true (Cédric)
> > > - fix build failure on windows (thanks Cédric found that issue)
> > >
> > > v4:
> > > - move properties vdev, iommufd and devid to nesting series where need it (Cédric)
> > > - fix 32bit build with clz64 (Cédric)
> > > - change check_cap naming to get_cap (Cédric)
> > > - return bool if error is passed through errp (Cédric)
> > > - drop HostIOMMUDevice[LegacyVFIO|IOMMUFD|IOMMUFDVFIO] declaration (Cédric)
> > > - drop HOST_IOMMU_DEVICE_CAP_IOMMUFD (Cédric)
> > > - replace include directive with forward declaration (Cédric)
> > >
> > > v3:
> > > - refine declaration and doc for HostIOMMUDevice (Cédric, Philippe)
> > > - introduce HostIOMMUDeviceCaps, .realize() and .check_cap() (Cédric)
> > > - introduce helper range_get_last_bit() for range operation (Cédric)
> > > - separate pci_device_get_iommu_bus_devfn() in a prereq patch (Cédric)
> > > - replace HIOD_ abbreviation with HOST_IOMMU_DEVICE_ (Cédric)
> > > - add header in include/sysemu/iommufd.h (Cédric)
> > >
> > > v2:
> > > - use QOM to abstract host IOMMU device and its sub-classes (Cédric)
> > > - move host IOMMU device creation in attach_device() (Cédric)
> > > - refine pci_device_set/unset_iommu_device doc further (Eric)
> > > - define host IOMMU info format of different backend
> > > - implement get_host_iommu_info() for different backend (Cédric)
> > > - drop cap/ecap update logic (MST)
> > > - check aw-bits from get_host_iommu_info() in legacy mode
> > >
> > > v1:
> > > - use HostIOMMUDevice handle instead of union in VFIODevice (Eric)
> > > - change host_iommu_device_init to host_iommu_device_create
> > > - allocate HostIOMMUDevice in host_iommu_device_create callback
> > > and set the VFIODevice base_hdev handle (Eric)
> > > - refine pci_device_set/unset_iommu_device doc (Eric)
> > > - use HostIOMMUDevice handle instead of union in VTDHostIOMMUDevice (Eric)
> > > - convert HostIOMMUDevice to sub object pointer in vtd_check_hdev
> > >
> > > rfcv2:
> > > - introduce common abstract HostIOMMUDevice and sub struct for different BEs (Eric, Cédric)
> > > - remove iommufd_device.[ch] (Cédric)
> > > - remove duplicate iommufd/devid define from VFIODevice (Eric)
> > > - drop the p in aliased_pbus and aliased_pdevfn (Eric)
> > > - assert devfn and iommu_bus in pci_device_get_iommu_bus_devfn (Cédric, Eric)
> > > - use errp in iommufd_device_get_info (Eric)
> > > - split and simplify cap/ecap check/sync code in intel_iommu.c (Cédric)
> > > - move VTDHostIOMMUDevice declaration to intel_iommu_internal.h (Cédric)
> > > - make '(vtd->cap_reg >> 16) & 0x3fULL' a MACRO and add missed '+1' (Cédric)
> > > - block migration if vIOMMU cap/ecap updated based on host IOMMU cap/ecap
> > > - add R-B
> > >
> > > Yi Liu (2):
> > > hw/pci: Introduce pci_device_[set|unset]_iommu_device()
> > > intel_iommu: Implement [set|unset]_iommu_device() callbacks
> > >
> > > Zhenzhong Duan (15):
> > > backends: Introduce HostIOMMUDevice abstract
> > > backends/host_iommu_device: Introduce HostIOMMUDeviceCaps
> > > vfio/container: Introduce TYPE_HOST_IOMMU_DEVICE_LEGACY_VFIO device
> > > backends/iommufd: Introduce TYPE_HOST_IOMMU_DEVICE_IOMMUFD[_VFIO]
> > > devices
> > > range: Introduce range_get_last_bit()
> > > vfio/container: Implement HostIOMMUDeviceClass::realize() handler
> > > backends/iommufd: Introduce helper function
> > > iommufd_backend_get_device_info()
> > > vfio/iommufd: Implement HostIOMMUDeviceClass::realize() handler
> > > vfio/container: Implement HostIOMMUDeviceClass::get_cap() handler
> > > backends/iommufd: Implement HostIOMMUDeviceClass::get_cap() handler
> > > vfio: Create host IOMMU device instance
> > > hw/pci: Introduce helper function pci_device_get_iommu_bus_devfn()
> > > vfio/pci: Pass HostIOMMUDevice to vIOMMU
> > > intel_iommu: Extract out vtd_cap_init() to initialize cap/ecap
> > > intel_iommu: Check compatibility with host IOMMU capabilities
> > >
> > > MAINTAINERS | 2 +
> > > include/hw/i386/intel_iommu.h | 2 +
> > > include/hw/pci/pci.h | 38 ++++-
> > > include/hw/vfio/vfio-common.h | 8 +
> > > include/hw/vfio/vfio-container-base.h | 3 +
> > > include/qemu/range.h | 11 ++
> > > include/sysemu/host_iommu_device.h | 91 ++++++++++++
> > > include/sysemu/iommufd.h | 19 +++
> > > backends/host_iommu_device.c | 33 +++++
> > > backends/iommufd.c | 76 ++++++++--
> > > hw/i386/intel_iommu.c | 203 ++++++++++++++++++++------
> > > hw/pci/pci.c | 75 +++++++++-
> > > hw/vfio/common.c | 16 +-
> > > hw/vfio/container.c | 41 +++++-
> > > hw/vfio/helpers.c | 17 +++
> > > hw/vfio/iommufd.c | 37 ++++-
> > > hw/vfio/pci.c | 19 ++-
> > > backends/meson.build | 1 +
> > > 18 files changed, 623 insertions(+), 69 deletions(-)
> > > create mode 100644 include/sysemu/host_iommu_device.h
> > > create mode 100644 backends/host_iommu_device.c
> > >
> > > --
> > > 2.34.1
> >
Hello Michael,
On 6/5/24 10:30 AM, Zhenzhong Duan wrote:
> Hi,
>
> This series introduce a HostIOMMUDevice abstraction and sub-classes.
> Also HostIOMMUDeviceCaps structure in HostIOMMUDevice and a new interface
> between vIOMMU and HostIOMMUDevice.
>
> A HostIOMMUDevice is an abstraction for an assigned device that is protected
> by a physical IOMMU (aka host IOMMU). The userspace interaction with this
> physical IOMMU can be done either through the VFIO IOMMU type 1 legacy
> backend or the new iommufd backend. The assigned device can be a VFIO device
> or a VDPA device. The HostIOMMUDevice is needed to interact with the host
> IOMMU that protects the assigned device. It is especially useful when the
> device is also protected by a virtual IOMMU as this latter use the translation
> services of the physical IOMMU and is constrained by it. In that context the
> HostIOMMUDevice can be passed to the virtual IOMMU to collect physical IOMMU
> capabilities such as the supported address width. In the future, the virtual
> IOMMU will use the HostIOMMUDevice to program the guest page tables in the
> first translation stage of the physical IOMMU.
This series has been the subject of reviews and tests on various
architectures and platforms. It prepares ground for more IOMMU changes
related to the new IOMMUFD backend.
I have queued them in the VFIO 9.1 tree for now, awaiting approval
from the PCI maintainers. Could please take look at the pci part which
introduces new IOMMU callbacks ?
Thanks,
C
> HostIOMMUDeviceClass::realize() is introduced to initialize
> HostIOMMUDeviceCaps and other fields of HostIOMMUDevice variants.
>
> HostIOMMUDeviceClass::get_cap() is introduced to query host IOMMU
> device capabilities.
>
> The class tree is as below:
>
> HostIOMMUDevice
> | .caps
> | .realize()
> | .get_cap()
> |
> .-----------------------------------------------.
> | | |
> HostIOMMUDeviceLegacyVFIO {HostIOMMUDeviceLegacyVDPA} HostIOMMUDeviceIOMMUFD
> | | | [.iommufd]
> | [.devid]
> | [.ioas_id]
> | [.attach_hwpt()]
> | [.detach_hwpt()]
> |
> .----------------------.
> | |
> HostIOMMUDeviceIOMMUFDVFIO {HostIOMMUDeviceIOMMUFDVDPA}
> | [.vdev] | {.vdev}
>
> * The attributes in [] will be implemented in nesting series.
> * The classes in {} will be implemented in future.
> * .vdev in different class points to different agent device,
> * i.e., VFIODevice or VDPADevice.
>
> PATCH1-4: Introduce HostIOMMUDevice and its sub classes
> PATCH5-10: Implement .realize() and .get_cap() handler
> PATCH11-14: Create HostIOMMUDevice instance and pass to vIOMMU
> PATCH15-17: Implement compatibility check between host IOMMU and vIOMMU(intel_iommu)
>
> Test done:
> make check
> vfio device hotplug/unplug with different backend on linux
> reboot, kexec
> build test on linux and windows11
>
> Qemu code can be found at:
> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_preq_v7
>
> Besides the compatibility check in this series, in nesting series, this
> host IOMMU device is extended for much wider usage. For anyone interested
> on the nesting series, here is the link:
> https://github.com/yiliu1765/qemu/tree/zhenzhong/iommufd_nesting_rfcv2
>
> Thanks
> Zhenzhong
>
> Changelog:
> v7:
> - drop config CONFIG_HOST_IOMMU_DEVICE (Cédric)
> - introduce HOST_IOMMU_DEVICE_CAP_AW_BITS_MAX (Eric)
> - use iova_ranges method in iommufd.realize() (Eric)
> - introduce HostIOMMUDevice::name to facilitate tracing (Eric)
> - implement a custom destroy hash function (Cédric)
> - drop VTDHostIOMMUDevice and save HostIOMMUDevice in hash table (Eric)
> - move patch5 after patch1 (Eric)
> - squash patch3 and 4, squash patch12 and 13 (Eric)
> - refine comments (Eric)
> - collect Eric's R-B
>
> v6:
> - open coded host_iommu_device_get_cap() to avoid #ifdef in intel_iommu.c (Cédric)
>
> v5:
> - pci_device_set_iommu_device return true (Cédric)
> - fix build failure on windows (thanks Cédric found that issue)
>
> v4:
> - move properties vdev, iommufd and devid to nesting series where need it (Cédric)
> - fix 32bit build with clz64 (Cédric)
> - change check_cap naming to get_cap (Cédric)
> - return bool if error is passed through errp (Cédric)
> - drop HostIOMMUDevice[LegacyVFIO|IOMMUFD|IOMMUFDVFIO] declaration (Cédric)
> - drop HOST_IOMMU_DEVICE_CAP_IOMMUFD (Cédric)
> - replace include directive with forward declaration (Cédric)
>
> v3:
> - refine declaration and doc for HostIOMMUDevice (Cédric, Philippe)
> - introduce HostIOMMUDeviceCaps, .realize() and .check_cap() (Cédric)
> - introduce helper range_get_last_bit() for range operation (Cédric)
> - separate pci_device_get_iommu_bus_devfn() in a prereq patch (Cédric)
> - replace HIOD_ abbreviation with HOST_IOMMU_DEVICE_ (Cédric)
> - add header in include/sysemu/iommufd.h (Cédric)
>
> v2:
> - use QOM to abstract host IOMMU device and its sub-classes (Cédric)
> - move host IOMMU device creation in attach_device() (Cédric)
> - refine pci_device_set/unset_iommu_device doc further (Eric)
> - define host IOMMU info format of different backend
> - implement get_host_iommu_info() for different backend (Cédric)
> - drop cap/ecap update logic (MST)
> - check aw-bits from get_host_iommu_info() in legacy mode
>
> v1:
> - use HostIOMMUDevice handle instead of union in VFIODevice (Eric)
> - change host_iommu_device_init to host_iommu_device_create
> - allocate HostIOMMUDevice in host_iommu_device_create callback
> and set the VFIODevice base_hdev handle (Eric)
> - refine pci_device_set/unset_iommu_device doc (Eric)
> - use HostIOMMUDevice handle instead of union in VTDHostIOMMUDevice (Eric)
> - convert HostIOMMUDevice to sub object pointer in vtd_check_hdev
>
> rfcv2:
> - introduce common abstract HostIOMMUDevice and sub struct for different BEs (Eric, Cédric)
> - remove iommufd_device.[ch] (Cédric)
> - remove duplicate iommufd/devid define from VFIODevice (Eric)
> - drop the p in aliased_pbus and aliased_pdevfn (Eric)
> - assert devfn and iommu_bus in pci_device_get_iommu_bus_devfn (Cédric, Eric)
> - use errp in iommufd_device_get_info (Eric)
> - split and simplify cap/ecap check/sync code in intel_iommu.c (Cédric)
> - move VTDHostIOMMUDevice declaration to intel_iommu_internal.h (Cédric)
> - make '(vtd->cap_reg >> 16) & 0x3fULL' a MACRO and add missed '+1' (Cédric)
> - block migration if vIOMMU cap/ecap updated based on host IOMMU cap/ecap
> - add R-B
>
> Yi Liu (2):
> hw/pci: Introduce pci_device_[set|unset]_iommu_device()
> intel_iommu: Implement [set|unset]_iommu_device() callbacks
>
> Zhenzhong Duan (15):
> backends: Introduce HostIOMMUDevice abstract
> backends/host_iommu_device: Introduce HostIOMMUDeviceCaps
> vfio/container: Introduce TYPE_HOST_IOMMU_DEVICE_LEGACY_VFIO device
> backends/iommufd: Introduce TYPE_HOST_IOMMU_DEVICE_IOMMUFD[_VFIO]
> devices
> range: Introduce range_get_last_bit()
> vfio/container: Implement HostIOMMUDeviceClass::realize() handler
> backends/iommufd: Introduce helper function
> iommufd_backend_get_device_info()
> vfio/iommufd: Implement HostIOMMUDeviceClass::realize() handler
> vfio/container: Implement HostIOMMUDeviceClass::get_cap() handler
> backends/iommufd: Implement HostIOMMUDeviceClass::get_cap() handler
> vfio: Create host IOMMU device instance
> hw/pci: Introduce helper function pci_device_get_iommu_bus_devfn()
> vfio/pci: Pass HostIOMMUDevice to vIOMMU
> intel_iommu: Extract out vtd_cap_init() to initialize cap/ecap
> intel_iommu: Check compatibility with host IOMMU capabilities
>
> MAINTAINERS | 2 +
> include/hw/i386/intel_iommu.h | 2 +
> include/hw/pci/pci.h | 38 ++++-
> include/hw/vfio/vfio-common.h | 8 +
> include/hw/vfio/vfio-container-base.h | 3 +
> include/qemu/range.h | 11 ++
> include/sysemu/host_iommu_device.h | 91 ++++++++++++
> include/sysemu/iommufd.h | 19 +++
> backends/host_iommu_device.c | 33 +++++
> backends/iommufd.c | 76 ++++++++--
> hw/i386/intel_iommu.c | 203 ++++++++++++++++++++------
> hw/pci/pci.c | 75 +++++++++-
> hw/vfio/common.c | 16 +-
> hw/vfio/container.c | 41 +++++-
> hw/vfio/helpers.c | 17 +++
> hw/vfio/iommufd.c | 37 ++++-
> hw/vfio/pci.c | 19 ++-
> backends/meson.build | 1 +
> 18 files changed, 623 insertions(+), 69 deletions(-)
> create mode 100644 include/sysemu/host_iommu_device.h
> create mode 100644 backends/host_iommu_device.c
>
© 2016 - 2024 Red Hat, Inc.