From nobody Mon Dec 1 22:34:43 2025 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAF54347BB9 for ; Wed, 26 Nov 2025 23:17:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764199082; cv=none; b=Cithk+PjWy2kV8CLpXD3aZRaaRmvDepvCPgP34QLbXSiiv4ZKxYgYxASaYn7tuEN2MfiCJFCkRk+lbCXIau5Lik+1qVjJRpsFbUF5MKrP41NH+BxPK5Ap8TebxPGuYjr8+EfafCtVhoiLVdCEgDbQFtJUl72hJ5nhuiRpp+xqTo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764199082; c=relaxed/simple; bh=T5iM/jn+gEAw6xdUVBY7eQtcRsPFbyXY1T6+O/mET3Q=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kuh3S/FfR9PY3CcoMoH5Asy/owGvb84lSYIWfOWuDE2zVw+90l/j6MOP/zoW/89nW4f0/y85VmM0yVlG7cQgIlyiV3xxGfuvFdcJwbFJhwSSwDIeqILOkOyBZcye5i/xIcIJKHIXdrT0TkAHgF9Nx4RXGXy4Wnbb6u+uFpaz30g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gS3T7VdI; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--dmatlack.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gS3T7VdI" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-7b9c91b814cso649042b3a.2 for ; Wed, 26 Nov 2025 15:17:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1764199079; x=1764803879; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=i21+YoY7Q+KNbCFBTzX46LioB1ZUMHUsqKfP5Y1Z30Y=; b=gS3T7VdIc16OtBOcRRg1KfHKcrj/WsGT6pXBNnzIIgWRJ4V1wgzJAl3mlssysbTRyK Zv1p9XlsdZp046A7cQF0IPncwMJBuKunGZRQTs1sUA4nK+VCXwRI+WOh+06PpZjQXREE X7i0O82S5Aef6Yze5JkYq6vD+phADIGHHpAGtDFvJUHyzrvlbdpPe4n+ZuB/V/ui7E+H L/bWaumvPOS97YoyW5TDvk7YB9p4RtrIBd+VMOpwxNgMqERCUOC0tEV575OHW2AkPNNr yl0U8B9TQq/pu8PlWHzIz+jaC3VWqIRnZpcfV0sZfBmTYzyOv43wOWbI9di4IJT8eGuU GV0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764199079; x=1764803879; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=i21+YoY7Q+KNbCFBTzX46LioB1ZUMHUsqKfP5Y1Z30Y=; b=gV3brxSqFhs0m8bcUspXoNQk3grFTgkg0kNLA2RH+1LSNsF98wBiRR8rAlfa95hPlc 3oCUt0T+AndVk8nCZ4BWOxEB89uIyAZF7X8OVwMZa8FwIadL+DDT8Pmh6xBow1rZeKlh 7vhXEQEERZZ+Z/i2mfVi7kN7b95WgE+22ZTp5LH+a62ZiON/VFBIkjswjSU7+X9NlxO7 NLJ06nod9DGWtGE7pXC2/LYnyffYfA+dQZ4RQ62EJ2m4CeWxfIZQ7zQrsxcMdMBKTvlk /XDVXQqXxlpravYWCYCc7LOj2W+dCcsL+tUk6p8KQROuMAmOgd9bpafYRd87idJXWASE kd6Q== X-Forwarded-Encrypted: i=1; AJvYcCVAWDeJwAnoFoHkEINua0OmQlqN9Bpm2brbipgasPLf6UZ4qyb3ciGH8HQTel9INNNQYooZ7YJ2o0IFGjQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxDbjWg6OiZMZd34Ep6DjHPQ7Z/3adT6bC0yvY66UYFLN2R+S5L ko8JE/QfrxgCfR715WuqkvlIOLreIOPhK8e6aV/PVwIhBcbZsS6urTYPZ1YPYneAPaSwmAwfzYV R8NxN8WvswVNvWg== X-Google-Smtp-Source: AGHT+IE9N/+TaEzjkPjfZfEH/oWXSzgo38MmG1+/bnSvZP5VzyVIf/9mho/62knJRPKDvFVImfnLuRcF5pZigw== X-Received: from pfblu1.prod.google.com ([2002:a05:6a00:7481:b0:7ba:6768:fe53]) (user=dmatlack job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:12dd:b0:7b9:420:cc0f with SMTP id d2e1a72fcca58-7ca87ec9a06mr10423422b3a.14.1764199079179; Wed, 26 Nov 2025 15:17:59 -0800 (PST) Date: Wed, 26 Nov 2025 23:17:26 +0000 In-Reply-To: <20251126231733.3302983-1-dmatlack@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251126231733.3302983-1-dmatlack@google.com> X-Mailer: git-send-email 2.52.0.487.g5c8c507ade-goog Message-ID: <20251126231733.3302983-12-dmatlack@google.com> Subject: [PATCH v4 11/18] vfio: selftests: Move IOMMU library code into iommu.c From: David Matlack To: Alex Williamson Cc: Alex Mastro , David Matlack , Jason Gunthorpe , Josh Hilke , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Raghavendra Rao Ananta , Vipin Sharma Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move all the IOMMU related library code into their own file iommu.c. This provides a better separation between the vfio_pci_device helper code and the iommu code. No function change intended. Reviewed-by: Alex Mastro Tested-by: Alex Mastro Reviewed-by: Raghavendra Rao Ananta Signed-off-by: David Matlack --- .../selftests/vfio/lib/include/vfio_util.h | 74 ++- tools/testing/selftests/vfio/lib/iommu.c | 461 ++++++++++++++++++ tools/testing/selftests/vfio/lib/libvfio.mk | 3 +- .../selftests/vfio/lib/vfio_pci_device.c | 442 ----------------- 4 files changed, 527 insertions(+), 453 deletions(-) create mode 100644 tools/testing/selftests/vfio/lib/iommu.c diff --git a/tools/testing/selftests/vfio/lib/include/vfio_util.h b/tools/t= esting/selftests/vfio/lib/include/vfio_util.h index 7784422116de..f67915d9443e 100644 --- a/tools/testing/selftests/vfio/lib/include/vfio_util.h +++ b/tools/testing/selftests/vfio/lib/include/vfio_util.h @@ -50,6 +50,12 @@ VFIO_LOG_AND_EXIT(_fmt, ##__VA_ARGS__); \ } while (0) =20 +#define ioctl_assert(_fd, _op, _arg) do { \ + void *__arg =3D (_arg); \ + int __ret =3D ioctl((_fd), (_op), (__arg)); \ + VFIO_ASSERT_EQ(__ret, 0, "ioctl(%s, %s, %s) returned %d\n", #_fd, #_op, #= _arg, __ret); \ +} while (0) + #define dev_info(_dev, _fmt, ...) printf("%s: " _fmt, (_dev)->bdf, ##__VA_= ARGS__) #define dev_err(_dev, _fmt, ...) fprintf(stderr, "%s: " _fmt, (_dev)->bdf,= ##__VA_ARGS__) =20 @@ -223,24 +229,52 @@ extern const char *default_iommu_mode; struct iommu *iommu_init(const char *iommu_mode); void iommu_cleanup(struct iommu *iommu); =20 +int __iommu_map(struct iommu *iommu, struct dma_region *region); + +static inline void iommu_map(struct iommu *iommu, struct dma_region *regio= n) +{ + VFIO_ASSERT_EQ(__iommu_map(iommu, region), 0); +} + +int __iommu_unmap(struct iommu *iommu, struct dma_region *region, u64 *unm= apped); + +static inline void iommu_unmap(struct iommu *iommu, struct dma_region *reg= ion) +{ + VFIO_ASSERT_EQ(__iommu_unmap(iommu, region, NULL), 0); +} + +int __iommu_unmap_all(struct iommu *iommu, u64 *unmapped); + +static inline void iommu_unmap_all(struct iommu *iommu) +{ + VFIO_ASSERT_EQ(__iommu_unmap_all(iommu, NULL), 0); +} + +iova_t __iommu_hva2iova(struct iommu *iommu, void *vaddr); +iova_t iommu_hva2iova(struct iommu *iommu, void *vaddr); + +struct iommu_iova_range *iommu_iova_ranges(struct iommu *iommu, u32 *nrang= es); + struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu= *iommu); void vfio_pci_device_cleanup(struct vfio_pci_device *device); =20 void vfio_pci_device_reset(struct vfio_pci_device *device); =20 -struct iommu_iova_range *vfio_pci_iova_ranges(struct vfio_pci_device *devi= ce, - u32 *nranges); +static inline struct iommu_iova_range *vfio_pci_iova_ranges(struct vfio_pc= i_device *device, + u32 *nranges) +{ + return iommu_iova_ranges(device->iommu, nranges); +} =20 struct iova_allocator *iova_allocator_init(struct vfio_pci_device *device); void iova_allocator_cleanup(struct iova_allocator *allocator); iova_t iova_allocator_alloc(struct iova_allocator *allocator, size_t size); =20 -int __vfio_pci_dma_map(struct vfio_pci_device *device, - struct dma_region *region); -int __vfio_pci_dma_unmap(struct vfio_pci_device *device, - struct dma_region *region, - u64 *unmapped); -int __vfio_pci_dma_unmap_all(struct vfio_pci_device *device, u64 *unmapped= ); +static inline int __vfio_pci_dma_map(struct vfio_pci_device *device, + struct dma_region *region) +{ + return __iommu_map(device->iommu, region); +} =20 static inline void vfio_pci_dma_map(struct vfio_pci_device *device, struct dma_region *region) @@ -248,12 +282,25 @@ static inline void vfio_pci_dma_map(struct vfio_pci_d= evice *device, VFIO_ASSERT_EQ(__vfio_pci_dma_map(device, region), 0); } =20 +static inline int __vfio_pci_dma_unmap(struct vfio_pci_device *device, + struct dma_region *region, + u64 *unmapped) +{ + return __iommu_unmap(device->iommu, region, unmapped); +} + static inline void vfio_pci_dma_unmap(struct vfio_pci_device *device, struct dma_region *region) { VFIO_ASSERT_EQ(__vfio_pci_dma_unmap(device, region, NULL), 0); } =20 +static inline int __vfio_pci_dma_unmap_all(struct vfio_pci_device *device, + u64 *unmapped) +{ + return __iommu_unmap_all(device->iommu, unmapped); +} + static inline void vfio_pci_dma_unmap_all(struct vfio_pci_device *device) { VFIO_ASSERT_EQ(__vfio_pci_dma_unmap_all(device, NULL), 0); @@ -319,8 +366,15 @@ static inline void vfio_pci_msix_disable(struct vfio_p= ci_device *device) vfio_pci_irq_disable(device, VFIO_PCI_MSIX_IRQ_INDEX); } =20 -iova_t __to_iova(struct vfio_pci_device *device, void *vaddr); -iova_t to_iova(struct vfio_pci_device *device, void *vaddr); +static inline iova_t __to_iova(struct vfio_pci_device *device, void *vaddr) +{ + return __iommu_hva2iova(device->iommu, vaddr); +} + +static inline iova_t to_iova(struct vfio_pci_device *device, void *vaddr) +{ + return iommu_hva2iova(device->iommu, vaddr); +} =20 static inline bool vfio_pci_device_match(struct vfio_pci_device *device, u16 vendor_id, u16 device_id) diff --git a/tools/testing/selftests/vfio/lib/iommu.c b/tools/testing/selft= ests/vfio/lib/iommu.c new file mode 100644 index 000000000000..3933079fc419 --- /dev/null +++ b/tools/testing/selftests/vfio/lib/iommu.c @@ -0,0 +1,461 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "../../../kselftest.h" +#include + +const char *default_iommu_mode =3D "iommufd"; + +/* Reminder: Keep in sync with FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(). */ +static const struct iommu_mode iommu_modes[] =3D { + { + .name =3D "vfio_type1_iommu", + .container_path =3D "/dev/vfio/vfio", + .iommu_type =3D VFIO_TYPE1_IOMMU, + }, + { + .name =3D "vfio_type1v2_iommu", + .container_path =3D "/dev/vfio/vfio", + .iommu_type =3D VFIO_TYPE1v2_IOMMU, + }, + { + .name =3D "iommufd_compat_type1", + .container_path =3D "/dev/iommu", + .iommu_type =3D VFIO_TYPE1_IOMMU, + }, + { + .name =3D "iommufd_compat_type1v2", + .container_path =3D "/dev/iommu", + .iommu_type =3D VFIO_TYPE1v2_IOMMU, + }, + { + .name =3D "iommufd", + }, +}; + +static const struct iommu_mode *lookup_iommu_mode(const char *iommu_mode) +{ + int i; + + if (!iommu_mode) + iommu_mode =3D default_iommu_mode; + + for (i =3D 0; i < ARRAY_SIZE(iommu_modes); i++) { + if (strcmp(iommu_mode, iommu_modes[i].name)) + continue; + + return &iommu_modes[i]; + } + + VFIO_FAIL("Unrecognized IOMMU mode: %s\n", iommu_mode); +} + +iova_t __iommu_hva2iova(struct iommu *iommu, void *vaddr) +{ + struct dma_region *region; + + list_for_each_entry(region, &iommu->dma_regions, link) { + if (vaddr < region->vaddr) + continue; + + if (vaddr >=3D region->vaddr + region->size) + continue; + + return region->iova + (vaddr - region->vaddr); + } + + return INVALID_IOVA; +} + +iova_t iommu_hva2iova(struct iommu *iommu, void *vaddr) +{ + iova_t iova; + + iova =3D __iommu_hva2iova(iommu, vaddr); + VFIO_ASSERT_NE(iova, INVALID_IOVA, "%p is not mapped into IOMMU\n", vaddr= ); + + return iova; +} + +static int vfio_iommu_map(struct iommu *iommu, struct dma_region *region) +{ + struct vfio_iommu_type1_dma_map args =3D { + .argsz =3D sizeof(args), + .flags =3D VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE, + .vaddr =3D (u64)region->vaddr, + .iova =3D region->iova, + .size =3D region->size, + }; + + if (ioctl(iommu->container_fd, VFIO_IOMMU_MAP_DMA, &args)) + return -errno; + + return 0; +} + +static int iommufd_map(struct iommu *iommu, struct dma_region *region) +{ + struct iommu_ioas_map args =3D { + .size =3D sizeof(args), + .flags =3D IOMMU_IOAS_MAP_READABLE | + IOMMU_IOAS_MAP_WRITEABLE | + IOMMU_IOAS_MAP_FIXED_IOVA, + .user_va =3D (u64)region->vaddr, + .iova =3D region->iova, + .length =3D region->size, + .ioas_id =3D iommu->ioas_id, + }; + + if (ioctl(iommu->iommufd, IOMMU_IOAS_MAP, &args)) + return -errno; + + return 0; +} + +int __iommu_map(struct iommu *iommu, struct dma_region *region) +{ + int ret; + + if (iommu->iommufd) + ret =3D iommufd_map(iommu, region); + else + ret =3D vfio_iommu_map(iommu, region); + + if (ret) + return ret; + + list_add(®ion->link, &iommu->dma_regions); + + return 0; +} + +static int __vfio_iommu_unmap(int fd, u64 iova, u64 size, u32 flags, u64 *= unmapped) +{ + struct vfio_iommu_type1_dma_unmap args =3D { + .argsz =3D sizeof(args), + .iova =3D iova, + .size =3D size, + .flags =3D flags, + }; + + if (ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &args)) + return -errno; + + if (unmapped) + *unmapped =3D args.size; + + return 0; +} + +static int vfio_iommu_unmap(struct iommu *iommu, struct dma_region *region, + u64 *unmapped) +{ + return __vfio_iommu_unmap(iommu->container_fd, region->iova, + region->size, 0, unmapped); +} + +static int __iommufd_unmap(int fd, u64 iova, u64 length, u32 ioas_id, u64 = *unmapped) +{ + struct iommu_ioas_unmap args =3D { + .size =3D sizeof(args), + .iova =3D iova, + .length =3D length, + .ioas_id =3D ioas_id, + }; + + if (ioctl(fd, IOMMU_IOAS_UNMAP, &args)) + return -errno; + + if (unmapped) + *unmapped =3D args.length; + + return 0; +} + +static int iommufd_unmap(struct iommu *iommu, struct dma_region *region, + u64 *unmapped) +{ + return __iommufd_unmap(iommu->iommufd, region->iova, region->size, + iommu->ioas_id, unmapped); +} + +int __iommu_unmap(struct iommu *iommu, struct dma_region *region, u64 *unm= apped) +{ + int ret; + + if (iommu->iommufd) + ret =3D iommufd_unmap(iommu, region, unmapped); + else + ret =3D vfio_iommu_unmap(iommu, region, unmapped); + + if (ret) + return ret; + + list_del_init(®ion->link); + + return 0; +} + +int __iommu_unmap_all(struct iommu *iommu, u64 *unmapped) +{ + int ret; + struct dma_region *curr, *next; + + if (iommu->iommufd) + ret =3D __iommufd_unmap(iommu->iommufd, 0, UINT64_MAX, + iommu->ioas_id, unmapped); + else + ret =3D __vfio_iommu_unmap(iommu->container_fd, 0, 0, + VFIO_DMA_UNMAP_FLAG_ALL, unmapped); + + if (ret) + return ret; + + list_for_each_entry_safe(curr, next, &iommu->dma_regions, link) + list_del_init(&curr->link); + + return 0; +} + +static struct vfio_info_cap_header *next_cap_hdr(void *buf, u32 bufsz, + u32 *cap_offset) +{ + struct vfio_info_cap_header *hdr; + + if (!*cap_offset) + return NULL; + + VFIO_ASSERT_LT(*cap_offset, bufsz); + VFIO_ASSERT_GE(bufsz - *cap_offset, sizeof(*hdr)); + + hdr =3D (struct vfio_info_cap_header *)((u8 *)buf + *cap_offset); + *cap_offset =3D hdr->next; + + return hdr; +} + +static struct vfio_info_cap_header *vfio_iommu_info_cap_hdr(struct vfio_io= mmu_type1_info *info, + u16 cap_id) +{ + struct vfio_info_cap_header *hdr; + u32 cap_offset =3D info->cap_offset; + u32 max_depth; + u32 depth =3D 0; + + if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) + return NULL; + + if (cap_offset) + VFIO_ASSERT_GE(cap_offset, sizeof(*info)); + + max_depth =3D (info->argsz - sizeof(*info)) / sizeof(*hdr); + + while ((hdr =3D next_cap_hdr(info, info->argsz, &cap_offset))) { + depth++; + VFIO_ASSERT_LE(depth, max_depth, "Capability chain contains a cycle\n"); + + if (hdr->id =3D=3D cap_id) + return hdr; + } + + return NULL; +} + +/* Return buffer including capability chain, if present. Free with free() = */ +static struct vfio_iommu_type1_info *vfio_iommu_get_info(int container_fd) +{ + struct vfio_iommu_type1_info *info; + + info =3D malloc(sizeof(*info)); + VFIO_ASSERT_NOT_NULL(info); + + *info =3D (struct vfio_iommu_type1_info) { + .argsz =3D sizeof(*info), + }; + + ioctl_assert(container_fd, VFIO_IOMMU_GET_INFO, info); + VFIO_ASSERT_GE(info->argsz, sizeof(*info)); + + info =3D realloc(info, info->argsz); + VFIO_ASSERT_NOT_NULL(info); + + ioctl_assert(container_fd, VFIO_IOMMU_GET_INFO, info); + VFIO_ASSERT_GE(info->argsz, sizeof(*info)); + + return info; +} + +/* + * Return iova ranges for the device's container. Normalize vfio_iommu_typ= e1 to + * report iommufd's iommu_iova_range. Free with free(). + */ +static struct iommu_iova_range *vfio_iommu_iova_ranges(struct iommu *iommu, + u32 *nranges) +{ + struct vfio_iommu_type1_info_cap_iova_range *cap_range; + struct vfio_iommu_type1_info *info; + struct vfio_info_cap_header *hdr; + struct iommu_iova_range *ranges =3D NULL; + + info =3D vfio_iommu_get_info(iommu->container_fd); + hdr =3D vfio_iommu_info_cap_hdr(info, VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANG= E); + VFIO_ASSERT_NOT_NULL(hdr); + + cap_range =3D container_of(hdr, struct vfio_iommu_type1_info_cap_iova_ran= ge, header); + VFIO_ASSERT_GT(cap_range->nr_iovas, 0); + + ranges =3D calloc(cap_range->nr_iovas, sizeof(*ranges)); + VFIO_ASSERT_NOT_NULL(ranges); + + for (u32 i =3D 0; i < cap_range->nr_iovas; i++) { + ranges[i] =3D (struct iommu_iova_range){ + .start =3D cap_range->iova_ranges[i].start, + .last =3D cap_range->iova_ranges[i].end, + }; + } + + *nranges =3D cap_range->nr_iovas; + + free(info); + return ranges; +} + +/* Return iova ranges of the device's IOAS. Free with free() */ +static struct iommu_iova_range *iommufd_iova_ranges(struct iommu *iommu, + u32 *nranges) +{ + struct iommu_iova_range *ranges; + int ret; + + struct iommu_ioas_iova_ranges query =3D { + .size =3D sizeof(query), + .ioas_id =3D iommu->ioas_id, + }; + + ret =3D ioctl(iommu->iommufd, IOMMU_IOAS_IOVA_RANGES, &query); + VFIO_ASSERT_EQ(ret, -1); + VFIO_ASSERT_EQ(errno, EMSGSIZE); + VFIO_ASSERT_GT(query.num_iovas, 0); + + ranges =3D calloc(query.num_iovas, sizeof(*ranges)); + VFIO_ASSERT_NOT_NULL(ranges); + + query.allowed_iovas =3D (uintptr_t)ranges; + + ioctl_assert(iommu->iommufd, IOMMU_IOAS_IOVA_RANGES, &query); + *nranges =3D query.num_iovas; + + return ranges; +} + +static int iova_range_comp(const void *a, const void *b) +{ + const struct iommu_iova_range *ra =3D a, *rb =3D b; + + if (ra->start < rb->start) + return -1; + + if (ra->start > rb->start) + return 1; + + return 0; +} + +/* Return sorted IOVA ranges of the device. Free with free(). */ +struct iommu_iova_range *iommu_iova_ranges(struct iommu *iommu, u32 *nrang= es) +{ + struct iommu_iova_range *ranges; + + if (iommu->iommufd) + ranges =3D iommufd_iova_ranges(iommu, nranges); + else + ranges =3D vfio_iommu_iova_ranges(iommu, nranges); + + if (!ranges) + return NULL; + + VFIO_ASSERT_GT(*nranges, 0); + + /* Sort and check that ranges are sane and non-overlapping */ + qsort(ranges, *nranges, sizeof(*ranges), iova_range_comp); + VFIO_ASSERT_LT(ranges[0].start, ranges[0].last); + + for (u32 i =3D 1; i < *nranges; i++) { + VFIO_ASSERT_LT(ranges[i].start, ranges[i].last); + VFIO_ASSERT_LT(ranges[i - 1].last, ranges[i].start); + } + + return ranges; +} + +static u32 iommufd_ioas_alloc(int iommufd) +{ + struct iommu_ioas_alloc args =3D { + .size =3D sizeof(args), + }; + + ioctl_assert(iommufd, IOMMU_IOAS_ALLOC, &args); + return args.out_ioas_id; +} + +struct iommu *iommu_init(const char *iommu_mode) +{ + const char *container_path; + struct iommu *iommu; + int version; + + iommu =3D calloc(1, sizeof(*iommu)); + VFIO_ASSERT_NOT_NULL(iommu); + + INIT_LIST_HEAD(&iommu->dma_regions); + + iommu->mode =3D lookup_iommu_mode(iommu_mode); + + container_path =3D iommu->mode->container_path; + if (container_path) { + iommu->container_fd =3D open(container_path, O_RDWR); + VFIO_ASSERT_GE(iommu->container_fd, 0, "open(%s) failed\n", container_pa= th); + + version =3D ioctl(iommu->container_fd, VFIO_GET_API_VERSION); + VFIO_ASSERT_EQ(version, VFIO_API_VERSION, "Unsupported version: %d\n", v= ersion); + } else { + /* + * Require device->iommufd to be >0 so that a simple non-0 check can be + * used to check if iommufd is enabled. In practice open() will never + * return 0 unless stdin is closed. + */ + iommu->iommufd =3D open("/dev/iommu", O_RDWR); + VFIO_ASSERT_GT(iommu->iommufd, 0); + + iommu->ioas_id =3D iommufd_ioas_alloc(iommu->iommufd); + } + + return iommu; +} + +void iommu_cleanup(struct iommu *iommu) +{ + if (iommu->iommufd) + VFIO_ASSERT_EQ(close(iommu->iommufd), 0); + else + VFIO_ASSERT_EQ(close(iommu->container_fd), 0); + + free(iommu); +} diff --git a/tools/testing/selftests/vfio/lib/libvfio.mk b/tools/testing/se= lftests/vfio/lib/libvfio.mk index 3c0cdac30cb6..7ecf2ad75c67 100644 --- a/tools/testing/selftests/vfio/lib/libvfio.mk +++ b/tools/testing/selftests/vfio/lib/libvfio.mk @@ -3,7 +3,8 @@ ARCH ?=3D $(SUBARCH) =20 LIBVFIO_SRCDIR :=3D $(selfdir)/vfio/lib =20 -LIBVFIO_C :=3D vfio_pci_device.c +LIBVFIO_C :=3D iommu.c +LIBVFIO_C +=3D vfio_pci_device.c LIBVFIO_C +=3D vfio_pci_driver.c =20 ifeq ($(ARCH:x86_64=3Dx86),x86) diff --git a/tools/testing/selftests/vfio/lib/vfio_pci_device.c b/tools/tes= ting/selftests/vfio/lib/vfio_pci_device.c index 422ad8dfad95..10fc016a2a3e 100644 --- a/tools/testing/selftests/vfio/lib/vfio_pci_device.c +++ b/tools/testing/selftests/vfio/lib/vfio_pci_device.c @@ -24,184 +24,6 @@ =20 #define PCI_SYSFS_PATH "/sys/bus/pci/devices" =20 -#define ioctl_assert(_fd, _op, _arg) do { \ - void *__arg =3D (_arg); \ - int __ret =3D ioctl((_fd), (_op), (__arg)); \ - VFIO_ASSERT_EQ(__ret, 0, "ioctl(%s, %s, %s) returned %d\n", #_fd, #_op, #= _arg, __ret); \ -} while (0) - -static struct vfio_info_cap_header *next_cap_hdr(void *buf, u32 bufsz, - u32 *cap_offset) -{ - struct vfio_info_cap_header *hdr; - - if (!*cap_offset) - return NULL; - - VFIO_ASSERT_LT(*cap_offset, bufsz); - VFIO_ASSERT_GE(bufsz - *cap_offset, sizeof(*hdr)); - - hdr =3D (struct vfio_info_cap_header *)((u8 *)buf + *cap_offset); - *cap_offset =3D hdr->next; - - return hdr; -} - -static struct vfio_info_cap_header *vfio_iommu_info_cap_hdr(struct vfio_io= mmu_type1_info *info, - u16 cap_id) -{ - struct vfio_info_cap_header *hdr; - u32 cap_offset =3D info->cap_offset; - u32 max_depth; - u32 depth =3D 0; - - if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) - return NULL; - - if (cap_offset) - VFIO_ASSERT_GE(cap_offset, sizeof(*info)); - - max_depth =3D (info->argsz - sizeof(*info)) / sizeof(*hdr); - - while ((hdr =3D next_cap_hdr(info, info->argsz, &cap_offset))) { - depth++; - VFIO_ASSERT_LE(depth, max_depth, "Capability chain contains a cycle\n"); - - if (hdr->id =3D=3D cap_id) - return hdr; - } - - return NULL; -} - -/* Return buffer including capability chain, if present. Free with free() = */ -static struct vfio_iommu_type1_info *vfio_iommu_get_info(struct vfio_pci_d= evice *device) -{ - struct vfio_iommu_type1_info *info; - - info =3D malloc(sizeof(*info)); - VFIO_ASSERT_NOT_NULL(info); - - *info =3D (struct vfio_iommu_type1_info) { - .argsz =3D sizeof(*info), - }; - - ioctl_assert(device->iommu->container_fd, VFIO_IOMMU_GET_INFO, info); - VFIO_ASSERT_GE(info->argsz, sizeof(*info)); - - info =3D realloc(info, info->argsz); - VFIO_ASSERT_NOT_NULL(info); - - ioctl_assert(device->iommu->container_fd, VFIO_IOMMU_GET_INFO, info); - VFIO_ASSERT_GE(info->argsz, sizeof(*info)); - - return info; -} - -/* - * Return iova ranges for the device's container. Normalize vfio_iommu_typ= e1 to - * report iommufd's iommu_iova_range. Free with free(). - */ -static struct iommu_iova_range *vfio_iommu_iova_ranges(struct vfio_pci_dev= ice *device, - u32 *nranges) -{ - struct vfio_iommu_type1_info_cap_iova_range *cap_range; - struct vfio_iommu_type1_info *info; - struct vfio_info_cap_header *hdr; - struct iommu_iova_range *ranges =3D NULL; - - info =3D vfio_iommu_get_info(device); - hdr =3D vfio_iommu_info_cap_hdr(info, VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANG= E); - VFIO_ASSERT_NOT_NULL(hdr); - - cap_range =3D container_of(hdr, struct vfio_iommu_type1_info_cap_iova_ran= ge, header); - VFIO_ASSERT_GT(cap_range->nr_iovas, 0); - - ranges =3D calloc(cap_range->nr_iovas, sizeof(*ranges)); - VFIO_ASSERT_NOT_NULL(ranges); - - for (u32 i =3D 0; i < cap_range->nr_iovas; i++) { - ranges[i] =3D (struct iommu_iova_range){ - .start =3D cap_range->iova_ranges[i].start, - .last =3D cap_range->iova_ranges[i].end, - }; - } - - *nranges =3D cap_range->nr_iovas; - - free(info); - return ranges; -} - -/* Return iova ranges of the device's IOAS. Free with free() */ -static struct iommu_iova_range *iommufd_iova_ranges(struct vfio_pci_device= *device, - u32 *nranges) -{ - struct iommu_iova_range *ranges; - int ret; - - struct iommu_ioas_iova_ranges query =3D { - .size =3D sizeof(query), - .ioas_id =3D device->iommu->ioas_id, - }; - - ret =3D ioctl(device->iommu->iommufd, IOMMU_IOAS_IOVA_RANGES, &query); - VFIO_ASSERT_EQ(ret, -1); - VFIO_ASSERT_EQ(errno, EMSGSIZE); - VFIO_ASSERT_GT(query.num_iovas, 0); - - ranges =3D calloc(query.num_iovas, sizeof(*ranges)); - VFIO_ASSERT_NOT_NULL(ranges); - - query.allowed_iovas =3D (uintptr_t)ranges; - - ioctl_assert(device->iommu->iommufd, IOMMU_IOAS_IOVA_RANGES, &query); - *nranges =3D query.num_iovas; - - return ranges; -} - -static int iova_range_comp(const void *a, const void *b) -{ - const struct iommu_iova_range *ra =3D a, *rb =3D b; - - if (ra->start < rb->start) - return -1; - - if (ra->start > rb->start) - return 1; - - return 0; -} - -/* Return sorted IOVA ranges of the device. Free with free(). */ -struct iommu_iova_range *vfio_pci_iova_ranges(struct vfio_pci_device *devi= ce, - u32 *nranges) -{ - struct iommu_iova_range *ranges; - - if (device->iommu->iommufd) - ranges =3D iommufd_iova_ranges(device, nranges); - else - ranges =3D vfio_iommu_iova_ranges(device, nranges); - - if (!ranges) - return NULL; - - VFIO_ASSERT_GT(*nranges, 0); - - /* Sort and check that ranges are sane and non-overlapping */ - qsort(ranges, *nranges, sizeof(*ranges), iova_range_comp); - VFIO_ASSERT_LT(ranges[0].start, ranges[0].last); - - for (u32 i =3D 1; i < *nranges; i++) { - VFIO_ASSERT_LT(ranges[i].start, ranges[i].last); - VFIO_ASSERT_LT(ranges[i - 1].last, ranges[i].start); - } - - return ranges; -} - struct iova_allocator *iova_allocator_init(struct vfio_pci_device *device) { struct iova_allocator *allocator; @@ -273,33 +95,6 @@ iova_t iova_allocator_alloc(struct iova_allocator *allo= cator, size_t size) } } =20 -iova_t __to_iova(struct vfio_pci_device *device, void *vaddr) -{ - struct dma_region *region; - - list_for_each_entry(region, &device->iommu->dma_regions, link) { - if (vaddr < region->vaddr) - continue; - - if (vaddr >=3D region->vaddr + region->size) - continue; - - return region->iova + (vaddr - region->vaddr); - } - - return INVALID_IOVA; -} - -iova_t to_iova(struct vfio_pci_device *device, void *vaddr) -{ - iova_t iova; - - iova =3D __to_iova(device, vaddr); - VFIO_ASSERT_NE(iova, INVALID_IOVA, "%p is not mapped into device.\n", vad= dr); - - return iova; -} - static void vfio_pci_irq_set(struct vfio_pci_device *device, u32 index, u32 vector, u32 count, int *fds) { @@ -386,142 +181,6 @@ static void vfio_pci_irq_get(struct vfio_pci_device *= device, u32 index, ioctl_assert(device->fd, VFIO_DEVICE_GET_IRQ_INFO, irq_info); } =20 -static int vfio_iommu_dma_map(struct vfio_pci_device *device, - struct dma_region *region) -{ - struct vfio_iommu_type1_dma_map args =3D { - .argsz =3D sizeof(args), - .flags =3D VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE, - .vaddr =3D (u64)region->vaddr, - .iova =3D region->iova, - .size =3D region->size, - }; - - if (ioctl(device->iommu->container_fd, VFIO_IOMMU_MAP_DMA, &args)) - return -errno; - - return 0; -} - -static int iommufd_dma_map(struct vfio_pci_device *device, - struct dma_region *region) -{ - struct iommu_ioas_map args =3D { - .size =3D sizeof(args), - .flags =3D IOMMU_IOAS_MAP_READABLE | - IOMMU_IOAS_MAP_WRITEABLE | - IOMMU_IOAS_MAP_FIXED_IOVA, - .user_va =3D (u64)region->vaddr, - .iova =3D region->iova, - .length =3D region->size, - .ioas_id =3D device->iommu->ioas_id, - }; - - if (ioctl(device->iommu->iommufd, IOMMU_IOAS_MAP, &args)) - return -errno; - - return 0; -} - -int __vfio_pci_dma_map(struct vfio_pci_device *device, - struct dma_region *region) -{ - int ret; - - if (device->iommu->iommufd) - ret =3D iommufd_dma_map(device, region); - else - ret =3D vfio_iommu_dma_map(device, region); - - if (ret) - return ret; - - list_add(®ion->link, &device->iommu->dma_regions); - - return 0; -} - -static int vfio_iommu_dma_unmap(int fd, u64 iova, u64 size, u32 flags, - u64 *unmapped) -{ - struct vfio_iommu_type1_dma_unmap args =3D { - .argsz =3D sizeof(args), - .iova =3D iova, - .size =3D size, - .flags =3D flags, - }; - - if (ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &args)) - return -errno; - - if (unmapped) - *unmapped =3D args.size; - - return 0; -} - -static int iommufd_dma_unmap(int fd, u64 iova, u64 length, u32 ioas_id, - u64 *unmapped) -{ - struct iommu_ioas_unmap args =3D { - .size =3D sizeof(args), - .iova =3D iova, - .length =3D length, - .ioas_id =3D ioas_id, - }; - - if (ioctl(fd, IOMMU_IOAS_UNMAP, &args)) - return -errno; - - if (unmapped) - *unmapped =3D args.length; - - return 0; -} - -int __vfio_pci_dma_unmap(struct vfio_pci_device *device, - struct dma_region *region, u64 *unmapped) -{ - int ret; - - if (device->iommu->iommufd) - ret =3D iommufd_dma_unmap(device->iommu->iommufd, region->iova, - region->size, device->iommu->ioas_id, - unmapped); - else - ret =3D vfio_iommu_dma_unmap(device->iommu->container_fd, - region->iova, region->size, 0, - unmapped); - - if (ret) - return ret; - - list_del_init(®ion->link); - - return 0; -} - -int __vfio_pci_dma_unmap_all(struct vfio_pci_device *device, u64 *unmapped) -{ - int ret; - struct dma_region *curr, *next; - - if (device->iommu->iommufd) - ret =3D iommufd_dma_unmap(device->iommu->iommufd, 0, UINT64_MAX, - device->iommu->ioas_id, unmapped); - else - ret =3D vfio_iommu_dma_unmap(device->iommu->container_fd, 0, 0, - VFIO_DMA_UNMAP_FLAG_ALL, unmapped); - - if (ret) - return ret; - - list_for_each_entry_safe(curr, next, &device->iommu->dma_regions, link) - list_del_init(&curr->link); - - return 0; -} - static void vfio_pci_region_get(struct vfio_pci_device *device, int index, struct vfio_region_info *info) { @@ -711,52 +370,6 @@ const char *vfio_pci_get_cdev_path(const char *bdf) return cdev_path; } =20 -/* Reminder: Keep in sync with FIXTURE_VARIANT_ADD_ALL_IOMMU_MODES(). */ -static const struct iommu_mode iommu_modes[] =3D { - { - .name =3D "vfio_type1_iommu", - .container_path =3D "/dev/vfio/vfio", - .iommu_type =3D VFIO_TYPE1_IOMMU, - }, - { - .name =3D "vfio_type1v2_iommu", - .container_path =3D "/dev/vfio/vfio", - .iommu_type =3D VFIO_TYPE1v2_IOMMU, - }, - { - .name =3D "iommufd_compat_type1", - .container_path =3D "/dev/iommu", - .iommu_type =3D VFIO_TYPE1_IOMMU, - }, - { - .name =3D "iommufd_compat_type1v2", - .container_path =3D "/dev/iommu", - .iommu_type =3D VFIO_TYPE1v2_IOMMU, - }, - { - .name =3D "iommufd", - }, -}; - -const char *default_iommu_mode =3D "iommufd"; - -static const struct iommu_mode *lookup_iommu_mode(const char *iommu_mode) -{ - int i; - - if (!iommu_mode) - iommu_mode =3D default_iommu_mode; - - for (i =3D 0; i < ARRAY_SIZE(iommu_modes); i++) { - if (strcmp(iommu_mode, iommu_modes[i].name)) - continue; - - return &iommu_modes[i]; - } - - VFIO_FAIL("Unrecognized IOMMU mode: %s\n", iommu_mode); -} - static void vfio_device_bind_iommufd(int device_fd, int iommufd) { struct vfio_device_bind_iommufd args =3D { @@ -767,16 +380,6 @@ static void vfio_device_bind_iommufd(int device_fd, in= t iommufd) ioctl_assert(device_fd, VFIO_DEVICE_BIND_IOMMUFD, &args); } =20 -static u32 iommufd_ioas_alloc(int iommufd) -{ - struct iommu_ioas_alloc args =3D { - .size =3D sizeof(args), - }; - - ioctl_assert(iommufd, IOMMU_IOAS_ALLOC, &args); - return args.out_ioas_id; -} - static void vfio_device_attach_iommufd_pt(int device_fd, u32 pt_id) { struct vfio_device_attach_iommufd_pt args =3D { @@ -799,41 +402,6 @@ static void vfio_pci_iommufd_setup(struct vfio_pci_dev= ice *device, const char *b vfio_device_attach_iommufd_pt(device->fd, device->iommu->ioas_id); } =20 -struct iommu *iommu_init(const char *iommu_mode) -{ - const char *container_path; - struct iommu *iommu; - int version; - - iommu =3D calloc(1, sizeof(*iommu)); - VFIO_ASSERT_NOT_NULL(iommu); - - INIT_LIST_HEAD(&iommu->dma_regions); - - iommu->mode =3D lookup_iommu_mode(iommu_mode); - - container_path =3D iommu->mode->container_path; - if (container_path) { - iommu->container_fd =3D open(container_path, O_RDWR); - VFIO_ASSERT_GE(iommu->container_fd, 0, "open(%s) failed\n", container_pa= th); - - version =3D ioctl(iommu->container_fd, VFIO_GET_API_VERSION); - VFIO_ASSERT_EQ(version, VFIO_API_VERSION, "Unsupported version: %d\n", v= ersion); - } else { - /* - * Require device->iommufd to be >0 so that a simple non-0 check can be - * used to check if iommufd is enabled. In practice open() will never - * return 0 unless stdin is closed. - */ - iommu->iommufd =3D open("/dev/iommu", O_RDWR); - VFIO_ASSERT_GT(iommu->iommufd, 0); - - iommu->ioas_id =3D iommufd_ioas_alloc(iommu->iommufd); - } - - return iommu; -} - struct vfio_pci_device *vfio_pci_device_init(const char *bdf, struct iommu= *iommu) { struct vfio_pci_device *device; @@ -880,16 +448,6 @@ void vfio_pci_device_cleanup(struct vfio_pci_device *d= evice) free(device); } =20 -void iommu_cleanup(struct iommu *iommu) -{ - if (iommu->iommufd) - VFIO_ASSERT_EQ(close(iommu->iommufd), 0); - else - VFIO_ASSERT_EQ(close(iommu->container_fd), 0); - - free(iommu); -} - static bool is_bdf(const char *str) { unsigned int s, b, d, f; --=20 2.52.0.487.g5c8c507ade-goog