Not all IOMMUs support the same virtual address width as the processor,
for instance older Intel consumer platforms only support 39-bits of
IOMMU address space. On such platforms, using the virtual address as the
IOVA and mappings at the top of the address space both fail.

VFIO and IOMMUFD have facilities for retrieving valid IOVA ranges,
VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE and IOMMU_IOAS_IOVA_RANGES,
respectively. These provide compatible arrays of ranges from which we
can construct a simple allocator.

Use this new allocator in place of reusing the virtual address, and
incorporate the maximum supported IOVA into the limit testing.  This
latter change doesn't test quite the same absolute end-of-address space
behavior but still seems to have some value.

This series is based on Alex Williamson's "Incorporate IOVA range info"
[1] along with feedback from the discussion in David Matlack's "Skip
vfio_dma_map_limit_test if mapping returns -EINVAL" [2].

Given David's plans to split IOMMU concerns from devices as described
in [3], this series' home for `struct iova_allocator` and IOVA
range helpers are likely to be short lived, since they reside in
vfio_pci_device.c. I assume that the rework can move this functionality
to a more appropriate location next to other IOMMU-focused code, once
such a place exists.

[1] https://lore.kernel.org/all/20251108212954.26477-1-alex@shazbot.org/#t
[2] https://lore.kernel.org/all/20251107222058.2009244-1-dmatlack@google.com/
[3] https://lore.kernel.org/all/aRIoKJk0uwLD-yGr@google.com/

To: Alex Williamson <alex@shazbot.org>
To: David Matlack <dmatlack@google.com>
To: Shuah Khan <shuah@kernel.org>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: kvm@vger.kernel.org
Cc: linux-kselftest@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Alex Mastro <amastro@fb.com>

Changes in v3:
- Update capability chain cycle detection
- Clarify the iova=vaddr commit message
- Link to v2: https://lore.kernel.org/r/20251111-iova-ranges-v2-0-0fa267ff9b78@fb.com

Changes in v2:
- Fix various nits
- calloc() where appropriate
- Update overflow test to run regardless of iova range constraints
- Change iova_allocator_init() to return an allocated struct
- Unfold iova_allocator_alloc()
- Fix iova allocator initial state bug
- Update vfio_pci_driver_test to use iova allocator
- Link to v1: https://lore.kernel.org/r/20251110-iova-ranges-v1-0-4d441cf5bf6d@fb.com

---
Alex Mastro (4):
      vfio: selftests: add iova range query helpers
      vfio: selftests: fix map limit tests to use last available iova
      vfio: selftests: add iova allocator
      vfio: selftests: replace iova=vaddr with allocated iovas

 .../testing/selftests/vfio/lib/include/vfio_util.h |  19 +-
 tools/testing/selftests/vfio/lib/vfio_pci_device.c | 246 ++++++++++++++++++++-
 .../testing/selftests/vfio/vfio_dma_mapping_test.c |  20 +-
 .../testing/selftests/vfio/vfio_pci_driver_test.c  |  12 +-
 4 files changed, 288 insertions(+), 9 deletions(-)
---
base-commit: 0ed3a30fd996cb0cac872432cf25185fda7e5316
change-id: 20251110-iova-ranges-1c09549fbf63

Best regards,
-- 
Alex Mastro <amastro@fb.com>

On Tue, 11 Nov 2025 10:48:23 -0800
Alex Mastro <amastro@fb.com> wrote:

> Not all IOMMUs support the same virtual address width as the processor,
> for instance older Intel consumer platforms only support 39-bits of
> IOMMU address space. On such platforms, using the virtual address as the
> IOVA and mappings at the top of the address space both fail.
> 
> VFIO and IOMMUFD have facilities for retrieving valid IOVA ranges,
> VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE and IOMMU_IOAS_IOVA_RANGES,
> respectively. These provide compatible arrays of ranges from which we
> can construct a simple allocator.
> 
> Use this new allocator in place of reusing the virtual address, and
> incorporate the maximum supported IOVA into the limit testing.  This
> latter change doesn't test quite the same absolute end-of-address space
> behavior but still seems to have some value.
> 
> This series is based on Alex Williamson's "Incorporate IOVA range info"
> [1] along with feedback from the discussion in David Matlack's "Skip
> vfio_dma_map_limit_test if mapping returns -EINVAL" [2].
> 
> Given David's plans to split IOMMU concerns from devices as described
> in [3], this series' home for `struct iova_allocator` and IOVA
> range helpers are likely to be short lived, since they reside in
> vfio_pci_device.c. I assume that the rework can move this functionality
> to a more appropriate location next to other IOMMU-focused code, once
> such a place exists.
> 
> [1] https://lore.kernel.org/all/20251108212954.26477-1-alex@shazbot.org/#t
> [2] https://lore.kernel.org/all/20251107222058.2009244-1-dmatlack@google.com/
> [3] https://lore.kernel.org/all/aRIoKJk0uwLD-yGr@google.com/
> 
> To: Alex Williamson <alex@shazbot.org>
> To: David Matlack <dmatlack@google.com>
> To: Shuah Khan <shuah@kernel.org>
> To: Jason Gunthorpe <jgg@ziepe.ca>
> Cc: kvm@vger.kernel.org
> Cc: linux-kselftest@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> Signed-off-by: Alex Mastro <amastro@fb.com>
> 
> Changes in v3:
> - Update capability chain cycle detection
> - Clarify the iova=vaddr commit message
> - Link to v2: https://lore.kernel.org/r/20251111-iova-ranges-v2-0-0fa267ff9b78@fb.com

Applied to vfio for-linus branch for v6.18.  Thanks for the quick
resolution on this!

Alex

On Wed, Nov 12, 2025 at 08:30:05AM -0700, Alex Williamson wrote:
> On Tue, 11 Nov 2025 10:48:23 -0800
> Alex Mastro <amastro@fb.com> wrote:
> 
> > Not all IOMMUs support the same virtual address width as the processor,
> > for instance older Intel consumer platforms only support 39-bits of
> > IOMMU address space. On such platforms, using the virtual address as the
> > IOVA and mappings at the top of the address space both fail.
> > 
> > VFIO and IOMMUFD have facilities for retrieving valid IOVA ranges,
> > VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE and IOMMU_IOAS_IOVA_RANGES,
> > respectively. These provide compatible arrays of ranges from which we
> > can construct a simple allocator.
> > 
> > Use this new allocator in place of reusing the virtual address, and
> > incorporate the maximum supported IOVA into the limit testing.  This
> > latter change doesn't test quite the same absolute end-of-address space
> > behavior but still seems to have some value.
> > 
> > This series is based on Alex Williamson's "Incorporate IOVA range info"
> > [1] along with feedback from the discussion in David Matlack's "Skip
> > vfio_dma_map_limit_test if mapping returns -EINVAL" [2].
> > 
> > Given David's plans to split IOMMU concerns from devices as described
> > in [3], this series' home for `struct iova_allocator` and IOVA
> > range helpers are likely to be short lived, since they reside in
> > vfio_pci_device.c. I assume that the rework can move this functionality
> > to a more appropriate location next to other IOMMU-focused code, once
> > such a place exists.
> > 
> > [1] https://lore.kernel.org/all/20251108212954.26477-1-alex@shazbot.org/#t
> > [2] https://lore.kernel.org/all/20251107222058.2009244-1-dmatlack@google.com/
> > [3] https://lore.kernel.org/all/aRIoKJk0uwLD-yGr@google.com/
> > 
> > To: Alex Williamson <alex@shazbot.org>
> > To: David Matlack <dmatlack@google.com>
> > To: Shuah Khan <shuah@kernel.org>
> > To: Jason Gunthorpe <jgg@ziepe.ca>
> > Cc: kvm@vger.kernel.org
> > Cc: linux-kselftest@vger.kernel.org
> > Cc: linux-kernel@vger.kernel.org
> > Signed-off-by: Alex Mastro <amastro@fb.com>
> > 
> > Changes in v3:
> > - Update capability chain cycle detection
> > - Clarify the iova=vaddr commit message
> > - Link to v2: https://lore.kernel.org/r/20251111-iova-ranges-v2-0-0fa267ff9b78@fb.com
> 
> Applied to vfio for-linus branch for v6.18.  Thanks for the quick
> resolution on this!

No problem, thanks for the reviews!

> 
> Alex

On Tue, Nov 11, 2025 at 10:48 AM Alex Mastro <amastro@fb.com> wrote:
>
> Changes in v3:
> - Update capability chain cycle detection
> - Clarify the iova=vaddr commit message
> - Link to v2: https://lore.kernel.org/r/20251111-iova-ranges-v2-0-0fa267ff9b78@fb.com

All tests are still passing on v3 for me.

On Tue, Nov 11, 2025 at 11:26:43AM -0800, David Matlack wrote:
> On Tue, Nov 11, 2025 at 10:48 AM Alex Mastro <amastro@fb.com> wrote:
> >
> > Changes in v3:
> > - Update capability chain cycle detection
> > - Clarify the iova=vaddr commit message
> > - Link to v2: https://lore.kernel.org/r/20251111-iova-ranges-v2-0-0fa267ff9b78@fb.com
> 
> All tests are still passing on v3 for me.

Hey David, is vfio_pci_driver_test known to be in good shape? Both on the base
commit and after my series, I am seeing below, which results in a KSFT_SKIP.
Invoking other tests in a similar way actually runs things with expected
results (my devices are already bound to vfio-pci before running anything).

base commit: 0ed3a30fd996cb0cac872432cf25185fda7e5316

$ vfio_pci_driver_test -f 0000:05:00.0
No driver found for device 0000:05:00.0

Same thing using the run.sh wrapper

$ sudo ./run.sh -d 0000:05:00.0 ./vfio_pci_driver_test
+ echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
+ echo "vfio-pci" > /sys/bus/pci/devices/0000:05:00.0/driver_override
+ echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

No driver found for device 0000:05:00.0
+ echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
+ echo "" > /sys/bus/pci/devices/0000:05:00.0/driver_override
+ echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

device = vfio_pci_device_init(device_bdf, default_iommu_mode);
if (!device->driver.ops) {
        fprintf(stderr, "No driver found for device %s\n", device_bdf);
        return KSFT_SKIP;
}

Is this meant to be a placeholder for some future testing, or am I holding
things wrong?

On Wed, Nov 12, 2025 at 9:40 AM Alex Mastro <amastro@fb.com> wrote:
>
> Hey David, is vfio_pci_driver_test known to be in good shape? Both on the base
> commit and after my series, I am seeing below, which results in a KSFT_SKIP.
> Invoking other tests in a similar way actually runs things with expected
> results (my devices are already bound to vfio-pci before running anything).
>
> base commit: 0ed3a30fd996cb0cac872432cf25185fda7e5316
>
> $ vfio_pci_driver_test -f 0000:05:00.0
> No driver found for device 0000:05:00.0
>
> Same thing using the run.sh wrapper
>
> $ sudo ./run.sh -d 0000:05:00.0 ./vfio_pci_driver_test
> + echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
> + echo "vfio-pci" > /sys/bus/pci/devices/0000:05:00.0/driver_override
> + echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
>
> No driver found for device 0000:05:00.0
> + echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
> + echo "" > /sys/bus/pci/devices/0000:05:00.0/driver_override
> + echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
>
> device = vfio_pci_device_init(device_bdf, default_iommu_mode);
> if (!device->driver.ops) {
>         fprintf(stderr, "No driver found for device %s\n", device_bdf);
>         return KSFT_SKIP;
> }
>
> Is this meant to be a placeholder for some future testing, or am I holding
> things wrong?

What kind of device are you using?

This test uses the selftests driver framework, so it requires a driver
in tools/testing/selftests/vfio/lib/drivers to function. The driver
framework allows tests to trigger real DMA and MSIs from the device in
a controlled, generic, way.

We currently only have drivers for Intel DSA and Intel CBDMA
devices.So if you're not using one of those devices,
vfio_pci_driver_test exiting with KSFT_SKIP is entirely expected.

I would love to add support for more devices. Jason Gunthrope
suggested supporting a driver for mlx5 class hardware, since it's
broadly available. I've also had some discussions about adding a
simple emulated PCIe device to QEMU for running VFIO selftests within
VMs.

On Wed, Nov 12, 2025 at 09:51:35AM -0800, David Matlack wrote:
> On Wed, Nov 12, 2025 at 9:40 AM Alex Mastro <amastro@fb.com> wrote:
> >
> > Hey David, is vfio_pci_driver_test known to be in good shape? Both on the base
> > commit and after my series, I am seeing below, which results in a KSFT_SKIP.
> > Invoking other tests in a similar way actually runs things with expected
> > results (my devices are already bound to vfio-pci before running anything).
> >
> > base commit: 0ed3a30fd996cb0cac872432cf25185fda7e5316
> >
> > $ vfio_pci_driver_test -f 0000:05:00.0
> > No driver found for device 0000:05:00.0
> >
> > Same thing using the run.sh wrapper
> >
> > $ sudo ./run.sh -d 0000:05:00.0 ./vfio_pci_driver_test
> > + echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
> > + echo "vfio-pci" > /sys/bus/pci/devices/0000:05:00.0/driver_override
> > + echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
> >
> > No driver found for device 0000:05:00.0
> > + echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
> > + echo "" > /sys/bus/pci/devices/0000:05:00.0/driver_override
> > + echo "0000:05:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
> >
> > device = vfio_pci_device_init(device_bdf, default_iommu_mode);
> > if (!device->driver.ops) {
> >         fprintf(stderr, "No driver found for device %s\n", device_bdf);
> >         return KSFT_SKIP;
> > }
> >
> > Is this meant to be a placeholder for some future testing, or am I holding
> > things wrong?
> 
> What kind of device are you using?
> 
> This test uses the selftests driver framework, so it requires a driver
> in tools/testing/selftests/vfio/lib/drivers to function. The driver
> framework allows tests to trigger real DMA and MSIs from the device in
> a controlled, generic, way.

Ah, TIL about that concept. This is with one of our internal compute
accelerators, so not surprising that I'm seeing a skip then.

> We currently only have drivers for Intel DSA and Intel CBDMA
> devices.So if you're not using one of those devices,
> vfio_pci_driver_test exiting with KSFT_SKIP is entirely expected.
> 
> I would love to add support for more devices. Jason Gunthrope
> suggested supporting a driver for mlx5 class hardware, since it's
> broadly available. I've also had some discussions about adding a
> simple emulated PCIe device to QEMU for running VFIO selftests within
> VMs.

I do have access to mlx5 hardware FWIW, so that would be cool.

Thanks for the explanation!

Alex