drivers/iommu/intel/Kconfig | 2 +- drivers/iommu/intel/Makefile | 2 +- drivers/iommu/intel/iommu.c | 29 +-- drivers/iommu/intel/iommu.h | 14 +- drivers/iommu/intel/prq.c | 404 +++++++++++++++++++++++++++++++++++ drivers/iommu/intel/svm.c | 397 ---------------------------------- drivers/iommu/iommufd/hw_pagetable.c | 3 +- 7 files changed, 425 insertions(+), 426 deletions(-)
This series makes use of iommufd_hwpt_replace_device to execute
non-pasid/non-svm user space IOPFs. Our main motivation is to enable
user-space driver driven device verification without SVM/PASID.
What?
* Enable IO page fault handling in user space for a non-pasid, non-svm
and non-virtualised use case.
* Move IOMMU_IOPF configuration from INTEL_IOMMU_SVM into INTEL_IOMMU.
* Move all page request queue related logic to a new (prq.c) file.
* Remove PASID checks from PRQ event handling as well as PRQ
initialization.
* Allow execution of IOMMU_HWPT_ALLOC with a valid fault id
(IOMMU_HWPT_FAULT_ID_VALID)
* Insert a zero handle into the PASID array in dev->iommu_group when
replacing the old HWPT with an IOPF enabled HWPT.
Why?
The PCI ATS Extended Capability allows peripheral devices to
participate in the caching of translations when operating under an
IOMMU. Further, the ATS Page Request Interface (PRI) Extension allows
devices to handle missing mappings. Currently, PRI is mainly used in
the context of Shared Virtual Addressing, requiring support for the
Process Address Space Identifier (PASID) capability, but other use
cases such as enabling user-space driver driven device verification
and reducing memory pinning exists. This patchest sets out to enable
these use cases.
Testing?
The non-svm IOPF interface is exercised by first initializing an IOPF
enabled IOAS and then reading the fault file descriptor. Pseudocode on
the iopf initializing and handling is in [3] and [4] (using libvfn).
Supplementary repositories supporting this patchset:
1. A user space library libvfn [1] which is used for testing and
verification (see examples/iopf.c), and
2. Basic emulation of PCIe ATS/PRI and Intel VT-d PRQ in QEMU [2].
Changes in v2:
- Remove "nesting" from wording. This wording is left over from initial
versions that are now irrelevant.
- Dropped "iommu: init pasid array while doing domain_replace and iopf
is active" as the initialization of the pasid_array x-array happens
automatically when an iopf capable domain is replaced on a device.
- Corrected commit message in "iommu/vt-d: Separate page request queue
from SVM"
- Link to v1: https://lore.kernel.org/r/20240904-jag-iopfv8-v1-0-e3549920adf3@samsung.com
V1:
- This is the first version of the series after initial feedback from
the RFC [5].
Comments and feedback are greatly appreciated
Best
Joel
[1] https://github.com/SamsungDS/libvfn/tree/iommufd-fault-queue
[2] https://gitlab.com/birkelund/qemu/-/tree/pcie-ats-pri
[3] Initializing
```
int iopf_init(struct iommu_ioas *ioas, const char *bdf)
{
// open vfio device from bdf
int devfd = open('/dev/vfio/devices/VFIO_DEV', O_RDWR);
struct vfio_device_bind_iommufd bind = {
.argsz = sizeof(bind),
.flags = 0,
.iommufd = __iommufd,
};
ioctl(devfd, VFIO_DEVICE_BIND_IOMMUFD, &bind);
struct iommu_ioas *ioas = ioas;
struct vfio_device_attach_iommufd_pt attach_data = {
.argsz = sizeof(attach_data),
.flags = 0,
.pt_id = ioas->id,
};
ioctl(devfd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach_data);
struct iommu_fault_alloc fault = {
.size = sizeof(fault),
.flags = 0,
};
ioctl(__iommufd, IOMMU_FAULT_QUEUE_ALLOC, &fault);
struct iommu_hwpt_alloc fault_cmd = {
.size = sizeof(fault_cmd),
.flags = IOMMU_HWPT_FAULT_ID_VALID,
.dev_id = bind.out_devid,
.pt_id = ioas->id,
.data_len = 0,
.data_uptr = (uint64_t)NULL,
.fault_id = fault.out_fault_id,
.__reserved = 0,
};
ioctl(__iommufd, IOMMU_HWPT_ALLOC, &fault_cmd);
// This is a re-attach
struct vfio_device_attach_iommufd_pt attach = {
.argsz = sizeof(attach),
.flags = 0,
.pt_id = fault_cmd.out_hwpt_id
};
ioctl(dev_fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach);
}
```
[4] Handling
```
int handle_iopf(void *vaddr, int len, uint64_t iova) {
exec_command(CMD)
int iopf_fd = fault_cmd.fault_id;
struct iommu_hwpt_pgfault pgfault = {0};
if(read(iopf_fd, &pgfault, sizeof(pgfault)) == 0);
return; // no page fault
ret = iommu_map_vaddr(__iommmufd, vaddr, len, &iova)
struct iommu_hwpt_page_response pgfault_response = {
.cookie = pgfault.cookie,
.code = ret ? IOMMUFD_PAGE_RESP_SUCCESS : IOMMUFD_PAGE_RESP_INVALID,
};
write(iopf_fd, &pgfault_response, sizeof(pgfault_response));
return;
}
```
[5] https://lore.kernel.org/20240826-iopf-for-all-v1-0-59174e6a7528@samsung.com
Signed-off-by: Joel Granados <j.granados@samsung.com>
---
Joel Granados (3):
iommu/vt-d: Separate page request queue from SVM
iommu: kconfig: Move IOMMU_IOPF into INTEL_IOMMU
iommufd: Enable PRI when doing the iommufd_hwpt_alloc
Klaus Jensen (2):
iommu/vt-d: Remove the pasid present check in prq_event_thread
iommu/vt-d: drop pasid requirement for prq initialization
drivers/iommu/intel/Kconfig | 2 +-
drivers/iommu/intel/Makefile | 2 +-
drivers/iommu/intel/iommu.c | 29 +--
drivers/iommu/intel/iommu.h | 14 +-
drivers/iommu/intel/prq.c | 404 +++++++++++++++++++++++++++++++++++
drivers/iommu/intel/svm.c | 397 ----------------------------------
drivers/iommu/iommufd/hw_pagetable.c | 3 +-
7 files changed, 425 insertions(+), 426 deletions(-)
---
base-commit: 431c1646e1f86b949fa3685efc50b660a364c2b6
change-id: 20240904-jag-iopfv8-1577fd20422d
Best regards,
--
Joel Granados <j.granados@samsung.com>
> From: Joel Granados via B4 Relay > <devnull+j.granados.samsung.com@kernel.org> > > This series makes use of iommufd_hwpt_replace_device to execute > non-pasid/non-svm user space IOPFs. Our main motivation is to enable > user-space driver driven device verification without SVM/PASID. can you elaborate why IOPFs are necessary to help verify such usage? > > What? > * Enable IO page fault handling in user space for a non-pasid, non-svm > and non-virtualised use case. > * Move IOMMU_IOPF configuration from INTEL_IOMMU_SVM into > INTEL_IOMMU. > * Move all page request queue related logic to a new (prq.c) file. > * Remove PASID checks from PRQ event handling as well as PRQ > initialization. > * Allow execution of IOMMU_HWPT_ALLOC with a valid fault id > (IOMMU_HWPT_FAULT_ID_VALID) > * Insert a zero handle into the PASID array in dev->iommu_group when > replacing the old HWPT with an IOPF enabled HWPT. the last bullet is stale now. btw a selftest is expected too.
On Sat, Sep 14, 2024 at 12:48:31AM +0000, Tian, Kevin wrote: > > From: Joel Granados via B4 Relay > > <devnull+j.granados.samsung.com@kernel.org> > > > > This series makes use of iommufd_hwpt_replace_device to execute > > non-pasid/non-svm user space IOPFs. Our main motivation is to enable > > user-space driver driven device verification without SVM/PASID. > > can you elaborate why IOPFs are necessary to help verify such usage? In retrospect "enable" might not be the best word to use here. We are not "enabling" user-space driver driven device verification as it is already enabled; you could already poke a device from user space. But the whole poke space was not available, you could not test IOPF without having an SVM/PASID capable IOMMU. Therefore a better wording would be "Our main motivation is to expand or facilitate user-space driver driven device verification by enabling IOPF without SMV/PASID". Does this address your concern? > > > > > What? > > * Enable IO page fault handling in user space for a non-pasid, non-svm > > and non-virtualised use case. > > * Move IOMMU_IOPF configuration from INTEL_IOMMU_SVM into > > INTEL_IOMMU. > > * Move all page request queue related logic to a new (prq.c) file. > > * Remove PASID checks from PRQ event handling as well as PRQ > > initialization. > > * Allow execution of IOMMU_HWPT_ALLOC with a valid fault id > > (IOMMU_HWPT_FAULT_ID_VALID) > > * Insert a zero handle into the PASID array in dev->iommu_group when > > replacing the old HWPT with an IOPF enabled HWPT. > > the last bullet is stale now. oops. Missed that one; will correct in next version > > btw a selftest is expected too. I'll figure this out for the next version. Thx for the review Best -- Joel Granados
> From: Joel Granados <j.granados@samsung.com> > Sent: Monday, September 16, 2024 4:51 PM > > On Sat, Sep 14, 2024 at 12:48:31AM +0000, Tian, Kevin wrote: > > > From: Joel Granados via B4 Relay > > > <devnull+j.granados.samsung.com@kernel.org> > > > > > > This series makes use of iommufd_hwpt_replace_device to execute > > > non-pasid/non-svm user space IOPFs. Our main motivation is to enable > > > user-space driver driven device verification without SVM/PASID. > > > > can you elaborate why IOPFs are necessary to help verify such usage? > > In retrospect "enable" might not be the best word to use here. We are not > "enabling" user-space driver driven device verification as it is already > enabled; you could already poke a device from user space. But the whole > poke > space was not available, you could not test IOPF without having an > SVM/PASID > capable IOMMU. Therefore a better wording would be "Our main motivation > is to > expand or facilitate user-space driver driven device verification by enabling > IOPF without SMV/PASID". > hmm did you actually see a IOMMU which supports IOPF only but not SVM/PASID? this series alone has its merit, e.g. postcopy migration might want such notification. But not sure it helps solve a real problem in your side...
On Fri, Sep 20, 2024 at 06:57:04AM +0000, Tian, Kevin wrote: > > From: Joel Granados <j.granados@samsung.com> > > Sent: Monday, September 16, 2024 4:51 PM > > > > On Sat, Sep 14, 2024 at 12:48:31AM +0000, Tian, Kevin wrote: > > > > From: Joel Granados via B4 Relay > > > > <devnull+j.granados.samsung.com@kernel.org> > > > > > > > > This series makes use of iommufd_hwpt_replace_device to execute > > > > non-pasid/non-svm user space IOPFs. Our main motivation is to enable > > > > user-space driver driven device verification without SVM/PASID. > > > > > > can you elaborate why IOPFs are necessary to help verify such usage? > > > > In retrospect "enable" might not be the best word to use here. We are not > > "enabling" user-space driver driven device verification as it is already > > enabled; you could already poke a device from user space. But the whole > > poke > > space was not available, you could not test IOPF without having an > > SVM/PASID > > capable IOMMU. Therefore a better wording would be "Our main motivation > > is to > > expand or facilitate user-space driver driven device verification by enabling > > IOPF without SMV/PASID". > > > > hmm did you actually see a IOMMU which supports IOPF only but > not SVM/PASID? > > this series alone has its merit, e.g. postcopy migration might want > such notification. But not sure it helps solve a real problem in your side... I understand that you want more information about what problem(s) are solved by this patch set from my point of view. right? One of the main motivations is to enable IOPF in use cases where PASID is *not* an option, like NVMe devices. Therefore one of the examples for enabling user-space driver driver device verification are NVMe without PASID. Quick Note: I am adding my new email to the thread so I get the responses routed to the correct inbox. -- Joel Granados
© 2016 - 2024 Red Hat, Inc.