From nobody Wed Nov 27 19:38:13 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1697622449633939.1500698800529; Wed, 18 Oct 2023 02:47:29 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qt38T-0001L5-NS; Wed, 18 Oct 2023 05:46:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qt38C-0000wt-H6 for qemu-devel@nongnu.org; Wed, 18 Oct 2023 05:46:24 -0400 Received: from mail.ozlabs.org ([2404:9400:2221:ea00::3] helo=gandalf.ozlabs.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qt387-0003Uq-Da for qemu-devel@nongnu.org; Wed, 18 Oct 2023 05:46:24 -0400 Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by gandalf.ozlabs.org (Postfix) with ESMTP id 4S9QwB4ksvz4xbw; Wed, 18 Oct 2023 20:46:18 +1100 (AEDT) Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4S9Qw82BcKz4xbC; Wed, 18 Oct 2023 20:46:16 +1100 (AEDT) From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Yi Liu , Eric Auger , Zhenzhong Duan , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 15/22] vfio/common: Move legacy VFIO backend code into separate container.c Date: Wed, 18 Oct 2023 11:45:24 +0200 Message-ID: <20231018094531.733211-16-clg@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018094531.733211-1-clg@redhat.com> References: <20231018094531.733211-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2404:9400:2221:ea00::3; envelope-from=SRS0=VFJy=GA=redhat.com=clg@ozlabs.org; helo=gandalf.ozlabs.org X-Spam_score_int: -39 X-Spam_score: -4.0 X-Spam_bar: ---- X-Spam_report: (-4.0 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.25, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1697622450400100001 From: Yi Liu Move all the code really dependent on the legacy VFIO container/group into a separate file: container.c. What does remain in common.c is the code related to VFIOAddressSpace, MemoryListeners, migration and all other general operations. Signed-off-by: Eric Auger Signed-off-by: Yi Liu Signed-off-by: Zhenzhong Duan Reviewed-by: C=C3=A9dric Le Goater Signed-off-by: C=C3=A9dric Le Goater --- include/hw/vfio/vfio-common.h | 35 + hw/vfio/common.c | 1155 +------------------------------- hw/vfio/container.c | 1161 +++++++++++++++++++++++++++++++++ hw/vfio/meson.build | 1 + 4 files changed, 1213 insertions(+), 1139 deletions(-) create mode 100644 hw/vfio/container.c diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 54905b9dd4bbb0b2c8bc9b3ee232e539a9a46216..7780b9073a6c71d2aa5e2c5d157= 935c5805ba2f8 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -206,6 +206,30 @@ typedef struct { hwaddr pages; } VFIOBitmap; =20 +void vfio_host_win_add(VFIOContainer *container, + hwaddr min_iova, hwaddr max_iova, + uint64_t iova_pgsizes); +int vfio_host_win_del(VFIOContainer *container, hwaddr min_iova, + hwaddr max_iova); +VFIOAddressSpace *vfio_get_address_space(AddressSpace *as); +void vfio_put_address_space(VFIOAddressSpace *space); +bool vfio_devices_all_running_and_saving(VFIOContainer *container); + +/* container->fd */ +int vfio_dma_unmap(VFIOContainer *container, hwaddr iova, + ram_addr_t size, IOMMUTLBEntry *iotlb); +int vfio_dma_map(VFIOContainer *container, hwaddr iova, + ram_addr_t size, void *vaddr, bool readonly); +int vfio_set_dirty_page_tracking(VFIOContainer *container, bool start); +int vfio_query_dirty_bitmap(VFIOContainer *container, VFIOBitmap *vbmap, + hwaddr iova, hwaddr size); + +int vfio_container_add_section_window(VFIOContainer *container, + MemoryRegionSection *section, + Error **errp); +void vfio_container_del_section_window(VFIOContainer *container, + MemoryRegionSection *section); + void vfio_disable_irqindex(VFIODevice *vbasedev, int index); void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index); void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index); @@ -235,6 +259,10 @@ extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList; typedef QLIST_HEAD(VFIODeviceList, VFIODevice) VFIODeviceList; extern VFIOGroupList vfio_group_list; +extern VFIODeviceList vfio_device_list; + +extern const MemoryListener vfio_memory_listener; +extern int vfio_kvm_device_fd; =20 bool vfio_mig_active(void); int vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **er= rp); @@ -272,4 +300,11 @@ bool vfio_migration_realize(VFIODevice *vbasedev, Erro= r **errp); void vfio_migration_exit(VFIODevice *vbasedev); =20 int vfio_bitmap_alloc(VFIOBitmap *vbmap, hwaddr size); +bool vfio_devices_all_running_and_mig_active(VFIOContainer *container); +bool vfio_devices_all_device_dirty_tracking(VFIOContainer *container); +int vfio_devices_query_dirty_bitmap(VFIOContainer *container, + VFIOBitmap *vbmap, hwaddr iova, + hwaddr size); +int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, + uint64_t size, ram_addr_t ram_addr); #endif /* HW_VFIO_VFIO_COMMON_H */ diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 95bc50bcda4c44612cda2b0d8e9d5782e31ea4c6..9e61de03ee0ee611264ab9943d8= e5abc1320c766 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -46,9 +46,7 @@ #include "migration/qemu-file.h" #include "sysemu/tpm.h" =20 -VFIOGroupList vfio_group_list =3D - QLIST_HEAD_INITIALIZER(vfio_group_list); -static VFIODeviceList vfio_device_list =3D +VFIODeviceList vfio_device_list =3D QLIST_HEAD_INITIALIZER(vfio_device_list); static QLIST_HEAD(, VFIOAddressSpace) vfio_address_spaces =3D QLIST_HEAD_INITIALIZER(vfio_address_spaces); @@ -61,39 +59,13 @@ static QLIST_HEAD(, VFIOAddressSpace) vfio_address_spac= es =3D * initialized, this file descriptor is only released on QEMU exit and * we'll re-use it should another vfio device be attached before then. */ -static int vfio_kvm_device_fd =3D -1; +int vfio_kvm_device_fd =3D -1; #endif =20 -static int vfio_ram_block_discard_disable(VFIOContainer *container, bool s= tate) -{ - switch (container->iommu_type) { - case VFIO_TYPE1v2_IOMMU: - case VFIO_TYPE1_IOMMU: - /* - * We support coordinated discarding of RAM via the RamDiscardMana= ger. - */ - return ram_block_uncoordinated_discard_disable(state); - default: - /* - * VFIO_SPAPR_TCE_IOMMU most probably works just fine with - * RamDiscardManager, however, it is completely untested. - * - * VFIO_SPAPR_TCE_v2_IOMMU with "DMA memory preregistering" does - * completely the opposite of managing mapping/pinning dynamically= as - * required by RamDiscardManager. We would have to special-case se= ctions - * with a RamDiscardManager. - */ - return ram_block_discard_disable(state); - } -} - /* * Device state interfaces */ =20 -static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, - uint64_t size, ram_addr_t ram_addr); - bool vfio_mig_active(void) { VFIODevice *vbasedev; @@ -238,7 +210,7 @@ static bool vfio_devices_all_dirty_tracking(VFIOContain= er *container) return true; } =20 -static bool vfio_devices_all_device_dirty_tracking(VFIOContainer *containe= r) +bool vfio_devices_all_device_dirty_tracking(VFIOContainer *container) { VFIODevice *vbasedev; =20 @@ -255,7 +227,7 @@ static bool vfio_devices_all_device_dirty_tracking(VFIO= Container *container) * Check if all VFIO devices are running and migration is active, which is * essentially equivalent to the migration being in pre-copy phase. */ -static bool vfio_devices_all_running_and_mig_active(VFIOContainer *contain= er) +bool vfio_devices_all_running_and_mig_active(VFIOContainer *container) { VFIODevice *vbasedev; =20 @@ -280,150 +252,8 @@ static bool vfio_devices_all_running_and_mig_active(V= FIOContainer *container) return true; } =20 -static int vfio_dma_unmap_bitmap(VFIOContainer *container, - hwaddr iova, ram_addr_t size, - IOMMUTLBEntry *iotlb) -{ - struct vfio_iommu_type1_dma_unmap *unmap; - struct vfio_bitmap *bitmap; - VFIOBitmap vbmap; - int ret; - - ret =3D vfio_bitmap_alloc(&vbmap, size); - if (ret) { - return ret; - } - - unmap =3D g_malloc0(sizeof(*unmap) + sizeof(*bitmap)); - - unmap->argsz =3D sizeof(*unmap) + sizeof(*bitmap); - unmap->iova =3D iova; - unmap->size =3D size; - unmap->flags |=3D VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP; - bitmap =3D (struct vfio_bitmap *)&unmap->data; - - /* - * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of - * qemu_real_host_page_size to mark those dirty. Hence set bitmap_pgsi= ze - * to qemu_real_host_page_size. - */ - bitmap->pgsize =3D qemu_real_host_page_size(); - bitmap->size =3D vbmap.size; - bitmap->data =3D (__u64 *)vbmap.bitmap; - - if (vbmap.size > container->max_dirty_bitmap_size) { - error_report("UNMAP: Size of bitmap too big 0x%"PRIx64, vbmap.size= ); - ret =3D -E2BIG; - goto unmap_exit; - } - - ret =3D ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); - if (!ret) { - cpu_physical_memory_set_dirty_lebitmap(vbmap.bitmap, - iotlb->translated_addr, vbmap.pages); - } else { - error_report("VFIO_UNMAP_DMA with DIRTY_BITMAP : %m"); - } - -unmap_exit: - g_free(unmap); - g_free(vbmap.bitmap); - - return ret; -} - -/* - * DMA - Mapping and unmapping for the "type1" IOMMU interface used on x86 - */ -static int vfio_dma_unmap(VFIOContainer *container, - hwaddr iova, ram_addr_t size, - IOMMUTLBEntry *iotlb) -{ - struct vfio_iommu_type1_dma_unmap unmap =3D { - .argsz =3D sizeof(unmap), - .flags =3D 0, - .iova =3D iova, - .size =3D size, - }; - bool need_dirty_sync =3D false; - int ret; - - if (iotlb && vfio_devices_all_running_and_mig_active(container)) { - if (!vfio_devices_all_device_dirty_tracking(container) && - container->dirty_pages_supported) { - return vfio_dma_unmap_bitmap(container, iova, size, iotlb); - } - - need_dirty_sync =3D true; - } - - while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { - /* - * The type1 backend has an off-by-one bug in the kernel (71a7d3d7= 8e3c - * v4.15) where an overflow in its wrap-around check prevents us f= rom - * unmapping the last page of the address space. Test for the err= or - * condition and re-try the unmap excluding the last page. The - * expectation is that we've never mapped the last page anyway and= this - * unmap request comes via vIOMMU support which also makes it unli= kely - * that this page is used. This bug was introduced well after typ= e1 v2 - * support was introduced, so we shouldn't need to test for v1. A= fix - * is queued for kernel v5.0 so this workaround can be removed once - * affected kernels are sufficiently deprecated. - */ - if (errno =3D=3D EINVAL && unmap.size && !(unmap.iova + unmap.size= ) && - container->iommu_type =3D=3D VFIO_TYPE1v2_IOMMU) { - trace_vfio_dma_unmap_overflow_workaround(); - unmap.size -=3D 1ULL << ctz64(container->pgsizes); - continue; - } - error_report("VFIO_UNMAP_DMA failed: %s", strerror(errno)); - return -errno; - } - - if (need_dirty_sync) { - ret =3D vfio_get_dirty_bitmap(container, iova, size, - iotlb->translated_addr); - if (ret) { - return ret; - } - } - - return 0; -} - -static int vfio_dma_map(VFIOContainer *container, hwaddr iova, - ram_addr_t size, void *vaddr, bool readonly) -{ - struct vfio_iommu_type1_dma_map map =3D { - .argsz =3D sizeof(map), - .flags =3D VFIO_DMA_MAP_FLAG_READ, - .vaddr =3D (__u64)(uintptr_t)vaddr, - .iova =3D iova, - .size =3D size, - }; - - if (!readonly) { - map.flags |=3D VFIO_DMA_MAP_FLAG_WRITE; - } - - /* - * Try the mapping, if it fails with EBUSY, unmap the region and try - * again. This shouldn't be necessary, but we sometimes see it in - * the VGA ROM space. - */ - if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) =3D=3D 0 || - (errno =3D=3D EBUSY && vfio_dma_unmap(container, iova, size, NULL)= =3D=3D 0 && - ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) =3D=3D 0)) { - return 0; - } - - error_report("VFIO_MAP_DMA failed: %s", strerror(errno)); - return -errno; -} - -static void vfio_host_win_add(VFIOContainer *container, - hwaddr min_iova, hwaddr max_iova, - uint64_t iova_pgsizes) +void vfio_host_win_add(VFIOContainer *container, hwaddr min_iova, + hwaddr max_iova, uint64_t iova_pgsizes) { VFIOHostDMAWindow *hostwin; =20 @@ -444,8 +274,8 @@ static void vfio_host_win_add(VFIOContainer *container, QLIST_INSERT_HEAD(&container->hostwin_list, hostwin, hostwin_next); } =20 -static int vfio_host_win_del(VFIOContainer *container, hwaddr min_iova, - hwaddr max_iova) +int vfio_host_win_del(VFIOContainer *container, + hwaddr min_iova, hwaddr max_iova) { VFIOHostDMAWindow *hostwin; =20 @@ -794,92 +624,6 @@ static bool vfio_get_section_iova_range(VFIOContainer = *container, return true; } =20 -static int vfio_container_add_section_window(VFIOContainer *container, - MemoryRegionSection *section, - Error **errp) -{ - VFIOHostDMAWindow *hostwin; - hwaddr pgsize =3D 0; - int ret; - - if (container->iommu_type !=3D VFIO_SPAPR_TCE_v2_IOMMU) { - return 0; - } - - /* For now intersections are not allowed, we may relax this later */ - QLIST_FOREACH(hostwin, &container->hostwin_list, hostwin_next) { - if (ranges_overlap(hostwin->min_iova, - hostwin->max_iova - hostwin->min_iova + 1, - section->offset_within_address_space, - int128_get64(section->size))) { - error_setg(errp, - "region [0x%"PRIx64",0x%"PRIx64"] overlaps with existing" - "host DMA window [0x%"PRIx64",0x%"PRIx64"]", - section->offset_within_address_space, - section->offset_within_address_space + - int128_get64(section->size) - 1, - hostwin->min_iova, hostwin->max_iova); - return -EINVAL; - } - } - - ret =3D vfio_spapr_create_window(container, section, &pgsize); - if (ret) { - error_setg_errno(errp, -ret, "Failed to create SPAPR window"); - return ret; - } - - vfio_host_win_add(container, section->offset_within_address_space, - section->offset_within_address_space + - int128_get64(section->size) - 1, pgsize); -#ifdef CONFIG_KVM - if (kvm_enabled()) { - VFIOGroup *group; - IOMMUMemoryRegion *iommu_mr =3D IOMMU_MEMORY_REGION(section->mr); - struct kvm_vfio_spapr_tce param; - struct kvm_device_attr attr =3D { - .group =3D KVM_DEV_VFIO_GROUP, - .attr =3D KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE, - .addr =3D (uint64_t)(unsigned long)¶m, - }; - - if (!memory_region_iommu_get_attr(iommu_mr, IOMMU_ATTR_SPAPR_TCE_F= D, - ¶m.tablefd)) { - QLIST_FOREACH(group, &container->group_list, container_next) { - param.groupfd =3D group->fd; - if (ioctl(vfio_kvm_device_fd, KVM_SET_DEVICE_ATTR, &attr))= { - error_setg_errno(errp, errno, - "vfio: failed GROUP_SET_SPAPR_TCE for= " - "KVM VFIO device %d and group fd %d", - param.tablefd, param.groupfd); - return -errno; - } - trace_vfio_spapr_group_attach(param.groupfd, param.tablefd= ); - } - } - } -#endif - return 0; -} - -static void vfio_container_del_section_window(VFIOContainer *container, - MemoryRegionSection *section) -{ - if (container->iommu_type !=3D VFIO_SPAPR_TCE_v2_IOMMU) { - return; - } - - vfio_spapr_remove_window(container, - section->offset_within_address_space); - if (vfio_host_win_del(container, - section->offset_within_address_space, - section->offset_within_address_space + - int128_get64(section->size) - 1) < 0) { - hw_error("%s: Cannot delete missing window at %"HWADDR_PRIx, - __func__, section->offset_within_address_space); - } -} - static void vfio_listener_region_add(MemoryListener *listener, MemoryRegionSection *section) { @@ -1127,33 +871,6 @@ static void vfio_listener_region_del(MemoryListener *= listener, vfio_container_del_section_window(container, section); } =20 -static int vfio_set_dirty_page_tracking(VFIOContainer *container, bool sta= rt) -{ - int ret; - struct vfio_iommu_type1_dirty_bitmap dirty =3D { - .argsz =3D sizeof(dirty), - }; - - if (!container->dirty_pages_supported) { - return 0; - } - - if (start) { - dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_START; - } else { - dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP; - } - - ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); - if (ret) { - ret =3D -errno; - error_report("Failed to set dirty tracking flag 0x%x errno: %d", - dirty.flags, errno); - } - - return ret; -} - typedef struct VFIODirtyRanges { hwaddr min32; hwaddr max32; @@ -1473,9 +1190,9 @@ static int vfio_device_dma_logging_report(VFIODevice = *vbasedev, hwaddr iova, return 0; } =20 -static int vfio_devices_query_dirty_bitmap(VFIOContainer *container, - VFIOBitmap *vbmap, hwaddr iova, - hwaddr size) +int vfio_devices_query_dirty_bitmap(VFIOContainer *container, + VFIOBitmap *vbmap, hwaddr iova, + hwaddr size) { VFIODevice *vbasedev; int ret; @@ -1496,45 +1213,8 @@ static int vfio_devices_query_dirty_bitmap(VFIOConta= iner *container, return 0; } =20 -static int vfio_query_dirty_bitmap(VFIOContainer *container, VFIOBitmap *v= bmap, - hwaddr iova, hwaddr size) -{ - struct vfio_iommu_type1_dirty_bitmap *dbitmap; - struct vfio_iommu_type1_dirty_bitmap_get *range; - int ret; - - dbitmap =3D g_malloc0(sizeof(*dbitmap) + sizeof(*range)); - - dbitmap->argsz =3D sizeof(*dbitmap) + sizeof(*range); - dbitmap->flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP; - range =3D (struct vfio_iommu_type1_dirty_bitmap_get *)&dbitmap->data; - range->iova =3D iova; - range->size =3D size; - - /* - * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of - * qemu_real_host_page_size to mark those dirty. Hence set bitmap's pg= size - * to qemu_real_host_page_size. - */ - range->bitmap.pgsize =3D qemu_real_host_page_size(); - range->bitmap.size =3D vbmap->size; - range->bitmap.data =3D (__u64 *)vbmap->bitmap; - - ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); - if (ret) { - ret =3D -errno; - error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 - " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova, - (uint64_t)range->size, errno); - } - - g_free(dbitmap); - - return ret; -} - -static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, - uint64_t size, ram_addr_t ram_addr) +int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, + uint64_t size, ram_addr_t ram_addr) { bool all_device_dirty_tracking =3D vfio_devices_all_device_dirty_tracking(container); @@ -1723,7 +1403,7 @@ static void vfio_listener_log_sync(MemoryListener *li= stener, } } =20 -static const MemoryListener vfio_memory_listener =3D { +const MemoryListener vfio_memory_listener =3D { .name =3D "vfio", .region_add =3D vfio_listener_region_add, .region_del =3D vfio_listener_region_del, @@ -1732,45 +1412,6 @@ static const MemoryListener vfio_memory_listener =3D= { .log_sync =3D vfio_listener_log_sync, }; =20 -static void vfio_listener_release(VFIOContainer *container) -{ - memory_listener_unregister(&container->listener); - if (container->iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU) { - memory_listener_unregister(&container->prereg_listener); - } -} - -static struct vfio_info_cap_header * -vfio_get_iommu_type1_info_cap(struct vfio_iommu_type1_info *info, uint16_t= id) -{ - if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { - return NULL; - } - - return vfio_get_cap((void *)info, info->cap_offset, id); -} - -bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, - unsigned int *avail) -{ - struct vfio_info_cap_header *hdr; - struct vfio_iommu_type1_info_dma_avail *cap; - - /* If the capability cannot be found, assume no DMA limiting */ - hdr =3D vfio_get_iommu_type1_info_cap(info, - VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL); - if (hdr =3D=3D NULL) { - return false; - } - - if (avail !=3D NULL) { - cap =3D (void *) hdr; - *avail =3D cap->avail; - } - - return true; -} - void vfio_reset_handler(void *opaque) { VFIODevice *vbasedev; @@ -1846,25 +1487,7 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) return 0; } =20 -static void vfio_kvm_device_add_group(VFIOGroup *group) -{ - Error *err =3D NULL; - - if (vfio_kvm_device_add_fd(group->fd, &err)) { - error_reportf_err(err, "group ID %d: ", group->groupid); - } -} - -static void vfio_kvm_device_del_group(VFIOGroup *group) -{ - Error *err =3D NULL; - - if (vfio_kvm_device_del_fd(group->fd, &err)) { - error_reportf_err(err, "group ID %d: ", group->groupid); - } -} - -static VFIOAddressSpace *vfio_get_address_space(AddressSpace *as) +VFIOAddressSpace *vfio_get_address_space(AddressSpace *as) { VFIOAddressSpace *space; =20 @@ -1888,7 +1511,7 @@ static VFIOAddressSpace *vfio_get_address_space(Addre= ssSpace *as) return space; } =20 -static void vfio_put_address_space(VFIOAddressSpace *space) +void vfio_put_address_space(VFIOAddressSpace *space) { if (QLIST_EMPTY(&space->containers)) { QLIST_REMOVE(space, list); @@ -1899,499 +1522,6 @@ static void vfio_put_address_space(VFIOAddressSpace= *space) } } =20 -/* - * vfio_get_iommu_type - selects the richest iommu_type (v2 first) - */ -static int vfio_get_iommu_type(VFIOContainer *container, - Error **errp) -{ - int iommu_types[] =3D { VFIO_TYPE1v2_IOMMU, VFIO_TYPE1_IOMMU, - VFIO_SPAPR_TCE_v2_IOMMU, VFIO_SPAPR_TCE_IOMMU }; - int i; - - for (i =3D 0; i < ARRAY_SIZE(iommu_types); i++) { - if (ioctl(container->fd, VFIO_CHECK_EXTENSION, iommu_types[i])) { - return iommu_types[i]; - } - } - error_setg(errp, "No available IOMMU models"); - return -EINVAL; -} - -static int vfio_init_container(VFIOContainer *container, int group_fd, - Error **errp) -{ - int iommu_type, ret; - - iommu_type =3D vfio_get_iommu_type(container, errp); - if (iommu_type < 0) { - return iommu_type; - } - - ret =3D ioctl(group_fd, VFIO_GROUP_SET_CONTAINER, &container->fd); - if (ret) { - error_setg_errno(errp, errno, "Failed to set group container"); - return -errno; - } - - while (ioctl(container->fd, VFIO_SET_IOMMU, iommu_type)) { - if (iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU) { - /* - * On sPAPR, despite the IOMMU subdriver always advertises v1 = and - * v2, the running platform may not support v2 and there is no - * way to guess it until an IOMMU group gets added to the cont= ainer. - * So in case it fails with v2, try v1 as a fallback. - */ - iommu_type =3D VFIO_SPAPR_TCE_IOMMU; - continue; - } - error_setg_errno(errp, errno, "Failed to set iommu for container"); - return -errno; - } - - container->iommu_type =3D iommu_type; - return 0; -} - -static int vfio_get_iommu_info(VFIOContainer *container, - struct vfio_iommu_type1_info **info) -{ - - size_t argsz =3D sizeof(struct vfio_iommu_type1_info); - - *info =3D g_new0(struct vfio_iommu_type1_info, 1); -again: - (*info)->argsz =3D argsz; - - if (ioctl(container->fd, VFIO_IOMMU_GET_INFO, *info)) { - g_free(*info); - *info =3D NULL; - return -errno; - } - - if (((*info)->argsz > argsz)) { - argsz =3D (*info)->argsz; - *info =3D g_realloc(*info, argsz); - goto again; - } - - return 0; -} - -static struct vfio_info_cap_header * -vfio_get_iommu_info_cap(struct vfio_iommu_type1_info *info, uint16_t id) -{ - struct vfio_info_cap_header *hdr; - void *ptr =3D info; - - if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { - return NULL; - } - - for (hdr =3D ptr + info->cap_offset; hdr !=3D ptr; hdr =3D ptr + hdr->= next) { - if (hdr->id =3D=3D id) { - return hdr; - } - } - - return NULL; -} - -static void vfio_get_iommu_info_migration(VFIOContainer *container, - struct vfio_iommu_type1_info *inf= o) -{ - struct vfio_info_cap_header *hdr; - struct vfio_iommu_type1_info_cap_migration *cap_mig; - - hdr =3D vfio_get_iommu_info_cap(info, VFIO_IOMMU_TYPE1_INFO_CAP_MIGRAT= ION); - if (!hdr) { - return; - } - - cap_mig =3D container_of(hdr, struct vfio_iommu_type1_info_cap_migrati= on, - header); - - /* - * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of - * qemu_real_host_page_size to mark those dirty. - */ - if (cap_mig->pgsize_bitmap & qemu_real_host_page_size()) { - container->dirty_pages_supported =3D true; - container->max_dirty_bitmap_size =3D cap_mig->max_dirty_bitmap_siz= e; - container->dirty_pgsizes =3D cap_mig->pgsize_bitmap; - } -} - -static int vfio_connect_container(VFIOGroup *group, AddressSpace *as, - Error **errp) -{ - VFIOContainer *container; - int ret, fd; - VFIOAddressSpace *space; - - space =3D vfio_get_address_space(as); - - /* - * VFIO is currently incompatible with discarding of RAM insofar as the - * madvise to purge (zap) the page from QEMU's address space does not - * interact with the memory API and therefore leaves stale virtual to - * physical mappings in the IOMMU if the page was previously pinned. = We - * therefore set discarding broken for each group added to a container, - * whether the container is used individually or shared. This provides - * us with options to allow devices within a group to opt-in and allow - * discarding, so long as it is done consistently for a group (for ins= tance - * if the device is an mdev device where it is known that the host ven= dor - * driver will never pin pages outside of the working set of the guest - * driver, which would thus not be discarding candidates). - * - * The first opportunity to induce pinning occurs here where we attemp= t to - * attach the group to existing containers within the AddressSpace. I= f any - * pages are already zapped from the virtual address space, such as fr= om - * previous discards, new pinning will cause valid mappings to be - * re-established. Likewise, when the overall MemoryListener for a new - * container is registered, a replay of mappings within the AddressSpa= ce - * will occur, re-establishing any previously zapped pages as well. - * - * Especially virtio-balloon is currently only prevented from discardi= ng - * new memory, it will not yet set ram_block_discard_set_required() and - * therefore, neither stops us here or deals with the sudden memory - * consumption of inflated memory. - * - * We do support discarding of memory coordinated via the RamDiscardMa= nager - * with some IOMMU types. vfio_ram_block_discard_disable() handles the - * details once we know which type of IOMMU we are using. - */ - - QLIST_FOREACH(container, &space->containers, next) { - if (!ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd)) { - ret =3D vfio_ram_block_discard_disable(container, true); - if (ret) { - error_setg_errno(errp, -ret, - "Cannot set discarding of RAM broken"); - if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, - &container->fd)) { - error_report("vfio: error disconnecting group %d from" - " container", group->groupid); - } - return ret; - } - group->container =3D container; - QLIST_INSERT_HEAD(&container->group_list, group, container_nex= t); - vfio_kvm_device_add_group(group); - return 0; - } - } - - fd =3D qemu_open_old("/dev/vfio/vfio", O_RDWR); - if (fd < 0) { - error_setg_errno(errp, errno, "failed to open /dev/vfio/vfio"); - ret =3D -errno; - goto put_space_exit; - } - - ret =3D ioctl(fd, VFIO_GET_API_VERSION); - if (ret !=3D VFIO_API_VERSION) { - error_setg(errp, "supported vfio version: %d, " - "reported version: %d", VFIO_API_VERSION, ret); - ret =3D -EINVAL; - goto close_fd_exit; - } - - container =3D g_malloc0(sizeof(*container)); - container->space =3D space; - container->fd =3D fd; - container->error =3D NULL; - container->dirty_pages_supported =3D false; - container->dma_max_mappings =3D 0; - QLIST_INIT(&container->giommu_list); - QLIST_INIT(&container->hostwin_list); - QLIST_INIT(&container->vrdl_list); - - ret =3D vfio_init_container(container, group->fd, errp); - if (ret) { - goto free_container_exit; - } - - ret =3D vfio_ram_block_discard_disable(container, true); - if (ret) { - error_setg_errno(errp, -ret, "Cannot set discarding of RAM broken"= ); - goto free_container_exit; - } - - switch (container->iommu_type) { - case VFIO_TYPE1v2_IOMMU: - case VFIO_TYPE1_IOMMU: - { - struct vfio_iommu_type1_info *info; - - ret =3D vfio_get_iommu_info(container, &info); - if (ret) { - error_setg_errno(errp, -ret, "Failed to get VFIO IOMMU info"); - goto enable_discards_exit; - } - - if (info->flags & VFIO_IOMMU_INFO_PGSIZES) { - container->pgsizes =3D info->iova_pgsizes; - } else { - container->pgsizes =3D qemu_real_host_page_size(); - } - - if (!vfio_get_info_dma_avail(info, &container->dma_max_mappings)) { - container->dma_max_mappings =3D 65535; - } - vfio_get_iommu_info_migration(container, info); - g_free(info); - - /* - * FIXME: We should parse VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE - * information to get the actual window extent rather than assume - * a 64-bit IOVA address space. - */ - vfio_host_win_add(container, 0, (hwaddr)-1, container->pgsizes); - - break; - } - case VFIO_SPAPR_TCE_v2_IOMMU: - case VFIO_SPAPR_TCE_IOMMU: - { - struct vfio_iommu_spapr_tce_info info; - bool v2 =3D container->iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU; - - /* - * The host kernel code implementing VFIO_IOMMU_DISABLE is called - * when container fd is closed so we do not call it explicitly - * in this file. - */ - if (!v2) { - ret =3D ioctl(fd, VFIO_IOMMU_ENABLE); - if (ret) { - error_setg_errno(errp, errno, "failed to enable container"= ); - ret =3D -errno; - goto enable_discards_exit; - } - } else { - container->prereg_listener =3D vfio_prereg_listener; - - memory_listener_register(&container->prereg_listener, - &address_space_memory); - if (container->error) { - memory_listener_unregister(&container->prereg_listener); - ret =3D -1; - error_propagate_prepend(errp, container->error, - "RAM memory listener initialization failed: "); - goto enable_discards_exit; - } - } - - info.argsz =3D sizeof(info); - ret =3D ioctl(fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info); - if (ret) { - error_setg_errno(errp, errno, - "VFIO_IOMMU_SPAPR_TCE_GET_INFO failed"); - ret =3D -errno; - if (v2) { - memory_listener_unregister(&container->prereg_listener); - } - goto enable_discards_exit; - } - - if (v2) { - container->pgsizes =3D info.ddw.pgsizes; - /* - * There is a default window in just created container. - * To make region_add/del simpler, we better remove this - * window now and let those iommu_listener callbacks - * create/remove them when needed. - */ - ret =3D vfio_spapr_remove_window(container, info.dma32_window_= start); - if (ret) { - error_setg_errno(errp, -ret, - "failed to remove existing window"); - goto enable_discards_exit; - } - } else { - /* The default table uses 4K pages */ - container->pgsizes =3D 0x1000; - vfio_host_win_add(container, info.dma32_window_start, - info.dma32_window_start + - info.dma32_window_size - 1, - 0x1000); - } - } - } - - vfio_kvm_device_add_group(group); - - QLIST_INIT(&container->group_list); - QLIST_INSERT_HEAD(&space->containers, container, next); - - group->container =3D container; - QLIST_INSERT_HEAD(&container->group_list, group, container_next); - - container->listener =3D vfio_memory_listener; - - memory_listener_register(&container->listener, container->space->as); - - if (container->error) { - ret =3D -1; - error_propagate_prepend(errp, container->error, - "memory listener initialization failed: "); - goto listener_release_exit; - } - - container->initialized =3D true; - - return 0; -listener_release_exit: - QLIST_REMOVE(group, container_next); - QLIST_REMOVE(container, next); - vfio_kvm_device_del_group(group); - vfio_listener_release(container); - -enable_discards_exit: - vfio_ram_block_discard_disable(container, false); - -free_container_exit: - g_free(container); - -close_fd_exit: - close(fd); - -put_space_exit: - vfio_put_address_space(space); - - return ret; -} - -static void vfio_disconnect_container(VFIOGroup *group) -{ - VFIOContainer *container =3D group->container; - - QLIST_REMOVE(group, container_next); - group->container =3D NULL; - - /* - * Explicitly release the listener first before unset container, - * since unset may destroy the backend container if it's the last - * group. - */ - if (QLIST_EMPTY(&container->group_list)) { - vfio_listener_release(container); - } - - if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, &container->fd)) { - error_report("vfio: error disconnecting group %d from container", - group->groupid); - } - - if (QLIST_EMPTY(&container->group_list)) { - VFIOAddressSpace *space =3D container->space; - VFIOGuestIOMMU *giommu, *tmp; - VFIOHostDMAWindow *hostwin, *next; - - QLIST_REMOVE(container, next); - - QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, t= mp) { - memory_region_unregister_iommu_notifier( - MEMORY_REGION(giommu->iommu_mr), &giommu->n); - QLIST_REMOVE(giommu, giommu_next); - g_free(giommu); - } - - QLIST_FOREACH_SAFE(hostwin, &container->hostwin_list, hostwin_next, - next) { - QLIST_REMOVE(hostwin, hostwin_next); - g_free(hostwin); - } - - trace_vfio_disconnect_container(container->fd); - close(container->fd); - g_free(container); - - vfio_put_address_space(space); - } -} - -static VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **er= rp) -{ - VFIOGroup *group; - char path[32]; - struct vfio_group_status status =3D { .argsz =3D sizeof(status) }; - - QLIST_FOREACH(group, &vfio_group_list, next) { - if (group->groupid =3D=3D groupid) { - /* Found it. Now is it already in the right context? */ - if (group->container->space->as =3D=3D as) { - return group; - } else { - error_setg(errp, "group %d used in multiple address spaces= ", - group->groupid); - return NULL; - } - } - } - - group =3D g_malloc0(sizeof(*group)); - - snprintf(path, sizeof(path), "/dev/vfio/%d", groupid); - group->fd =3D qemu_open_old(path, O_RDWR); - if (group->fd < 0) { - error_setg_errno(errp, errno, "failed to open %s", path); - goto free_group_exit; - } - - if (ioctl(group->fd, VFIO_GROUP_GET_STATUS, &status)) { - error_setg_errno(errp, errno, "failed to get group %d status", gro= upid); - goto close_fd_exit; - } - - if (!(status.flags & VFIO_GROUP_FLAGS_VIABLE)) { - error_setg(errp, "group %d is not viable", groupid); - error_append_hint(errp, - "Please ensure all devices within the iommu_grou= p " - "are bound to their vfio bus driver.\n"); - goto close_fd_exit; - } - - group->groupid =3D groupid; - QLIST_INIT(&group->device_list); - - if (vfio_connect_container(group, as, errp)) { - error_prepend(errp, "failed to setup container for group %d: ", - groupid); - goto close_fd_exit; - } - - QLIST_INSERT_HEAD(&vfio_group_list, group, next); - - return group; - -close_fd_exit: - close(group->fd); - -free_group_exit: - g_free(group); - - return NULL; -} - -static void vfio_put_group(VFIOGroup *group) -{ - if (!group || !QLIST_EMPTY(&group->device_list)) { - return; - } - - if (!group->ram_block_discard_allowed) { - vfio_ram_block_discard_disable(group->container, false); - } - vfio_kvm_device_del_group(group); - vfio_disconnect_container(group); - QLIST_REMOVE(group, next); - trace_vfio_put_group(group->fd); - close(group->fd); - g_free(group); -} - struct vfio_device_info *vfio_get_device_info(int fd) { struct vfio_device_info *info; @@ -2415,256 +1545,3 @@ retry: =20 return info; } - -static int vfio_get_device(VFIOGroup *group, const char *name, - VFIODevice *vbasedev, Error **errp) -{ - g_autofree struct vfio_device_info *info =3D NULL; - int fd; - - fd =3D ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, name); - if (fd < 0) { - error_setg_errno(errp, errno, "error getting device from group %d", - group->groupid); - error_append_hint(errp, - "Verify all devices in group %d are bound to vfio- " - "or pci-stub and not already in use\n", group->group= id); - return fd; - } - - info =3D vfio_get_device_info(fd); - if (!info) { - error_setg_errno(errp, errno, "error getting device info"); - close(fd); - return -1; - } - - /* - * Set discarding of RAM as not broken for this group if the driver kn= ows - * the device operates compatibly with discarding. Setting must be - * consistent per group, but since compatibility is really only possib= le - * with mdev currently, we expect singleton groups. - */ - if (vbasedev->ram_block_discard_allowed !=3D - group->ram_block_discard_allowed) { - if (!QLIST_EMPTY(&group->device_list)) { - error_setg(errp, "Inconsistent setting of support for discardi= ng " - "RAM (e.g., balloon) within group"); - close(fd); - return -1; - } - - if (!group->ram_block_discard_allowed) { - group->ram_block_discard_allowed =3D true; - vfio_ram_block_discard_disable(group->container, false); - } - } - - vbasedev->fd =3D fd; - vbasedev->group =3D group; - QLIST_INSERT_HEAD(&group->device_list, vbasedev, next); - - vbasedev->num_irqs =3D info->num_irqs; - vbasedev->num_regions =3D info->num_regions; - vbasedev->flags =3D info->flags; - - trace_vfio_get_device(name, info->flags, info->num_regions, info->num_= irqs); - - vbasedev->reset_works =3D !!(info->flags & VFIO_DEVICE_FLAGS_RESET); - - return 0; -} - -static void vfio_put_base_device(VFIODevice *vbasedev) -{ - if (!vbasedev->group) { - return; - } - QLIST_REMOVE(vbasedev, next); - vbasedev->group =3D NULL; - trace_vfio_put_base_device(vbasedev->fd); - close(vbasedev->fd); -} - -/* - * Interfaces for IBM EEH (Enhanced Error Handling) - */ -static bool vfio_eeh_container_ok(VFIOContainer *container) -{ - /* - * As of 2016-03-04 (linux-4.5) the host kernel EEH/VFIO - * implementation is broken if there are multiple groups in a - * container. The hardware works in units of Partitionable - * Endpoints (=3D=3D IOMMU groups) and the EEH operations naively - * iterate across all groups in the container, without any logic - * to make sure the groups have their state synchronized. For - * certain operations (ENABLE) that might be ok, until an error - * occurs, but for others (GET_STATE) it's clearly broken. - */ - - /* - * XXX Once fixed kernels exist, test for them here - */ - - if (QLIST_EMPTY(&container->group_list)) { - return false; - } - - if (QLIST_NEXT(QLIST_FIRST(&container->group_list), container_next)) { - return false; - } - - return true; -} - -static int vfio_eeh_container_op(VFIOContainer *container, uint32_t op) -{ - struct vfio_eeh_pe_op pe_op =3D { - .argsz =3D sizeof(pe_op), - .op =3D op, - }; - int ret; - - if (!vfio_eeh_container_ok(container)) { - error_report("vfio/eeh: EEH_PE_OP 0x%x: " - "kernel requires a container with exactly one group",= op); - return -EPERM; - } - - ret =3D ioctl(container->fd, VFIO_EEH_PE_OP, &pe_op); - if (ret < 0) { - error_report("vfio/eeh: EEH_PE_OP 0x%x failed: %m", op); - return -errno; - } - - return ret; -} - -static VFIOContainer *vfio_eeh_as_container(AddressSpace *as) -{ - VFIOAddressSpace *space =3D vfio_get_address_space(as); - VFIOContainer *container =3D NULL; - - if (QLIST_EMPTY(&space->containers)) { - /* No containers to act on */ - goto out; - } - - container =3D QLIST_FIRST(&space->containers); - - if (QLIST_NEXT(container, next)) { - /* We don't yet have logic to synchronize EEH state across - * multiple containers */ - container =3D NULL; - goto out; - } - -out: - vfio_put_address_space(space); - return container; -} - -bool vfio_eeh_as_ok(AddressSpace *as) -{ - VFIOContainer *container =3D vfio_eeh_as_container(as); - - return (container !=3D NULL) && vfio_eeh_container_ok(container); -} - -int vfio_eeh_as_op(AddressSpace *as, uint32_t op) -{ - VFIOContainer *container =3D vfio_eeh_as_container(as); - - if (!container) { - return -ENODEV; - } - return vfio_eeh_container_op(container, op); -} - -static int vfio_device_groupid(VFIODevice *vbasedev, Error **errp) -{ - char *tmp, group_path[PATH_MAX], *group_name; - int ret, groupid; - ssize_t len; - - tmp =3D g_strdup_printf("%s/iommu_group", vbasedev->sysfsdev); - len =3D readlink(tmp, group_path, sizeof(group_path)); - g_free(tmp); - - if (len <=3D 0 || len >=3D sizeof(group_path)) { - ret =3D len < 0 ? -errno : -ENAMETOOLONG; - error_setg_errno(errp, -ret, "no iommu_group found"); - return ret; - } - - group_path[len] =3D 0; - - group_name =3D basename(group_path); - if (sscanf(group_name, "%d", &groupid) !=3D 1) { - error_setg_errno(errp, errno, "failed to read %s", group_path); - return -errno; - } - return groupid; -} - -/* - * vfio_attach_device: attach a device to a security context - * @name and @vbasedev->name are likely to be different depending - * on the type of the device, hence the need for passing @name - */ -int vfio_attach_device(char *name, VFIODevice *vbasedev, - AddressSpace *as, Error **errp) -{ - int groupid =3D vfio_device_groupid(vbasedev, errp); - VFIODevice *vbasedev_iter; - VFIOGroup *group; - VFIOContainer *container; - int ret; - - if (groupid < 0) { - return groupid; - } - - trace_vfio_attach_device(vbasedev->name, groupid); - - group =3D vfio_get_group(groupid, as, errp); - if (!group) { - return -ENOENT; - } - - QLIST_FOREACH(vbasedev_iter, &group->device_list, next) { - if (strcmp(vbasedev_iter->name, vbasedev->name) =3D=3D 0) { - error_setg(errp, "device is already attached"); - vfio_put_group(group); - return -EBUSY; - } - } - ret =3D vfio_get_device(group, name, vbasedev, errp); - if (ret) { - vfio_put_group(group); - return ret; - } - - container =3D group->container; - vbasedev->container =3D container; - QLIST_INSERT_HEAD(&container->device_list, vbasedev, container_next); - QLIST_INSERT_HEAD(&vfio_device_list, vbasedev, global_next); - - return ret; -} - -void vfio_detach_device(VFIODevice *vbasedev) -{ - VFIOGroup *group =3D vbasedev->group; - - if (!vbasedev->container) { - return; - } - - QLIST_REMOVE(vbasedev, global_next); - QLIST_REMOVE(vbasedev, container_next); - vbasedev->container =3D NULL; - trace_vfio_detach_device(vbasedev->name, group->groupid); - vfio_put_base_device(vbasedev); - vfio_put_group(group); -} diff --git a/hw/vfio/container.c b/hw/vfio/container.c new file mode 100644 index 0000000000000000000000000000000000000000..adc467210ff73422ad2f897abfb= cf6eddbb22ad2 --- /dev/null +++ b/hw/vfio/container.c @@ -0,0 +1,1161 @@ +/* + * generic functions used by VFIO devices + * + * Copyright Red Hat, Inc. 2012 + * + * Authors: + * Alex Williamson + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + * Based on qemu-kvm device-assignment: + * Adapted for KVM by Qumranet. + * Copyright (c) 2007, Neocleus, Alex Novik (alex@neocleus.com) + * Copyright (c) 2007, Neocleus, Guy Zana (guy@neocleus.com) + * Copyright (C) 2008, Qumranet, Amit Shah (amit.shah@qumranet.com) + * Copyright (C) 2008, Red Hat, Amit Shah (amit.shah@redhat.com) + * Copyright (C) 2008, IBM, Muli Ben-Yehuda (muli@il.ibm.com) + */ + +#include "qemu/osdep.h" +#include +#ifdef CONFIG_KVM +#include +#endif +#include + +#include "hw/vfio/vfio-common.h" +#include "hw/vfio/vfio.h" +#include "exec/address-spaces.h" +#include "exec/memory.h" +#include "exec/ram_addr.h" +#include "hw/hw.h" +#include "qemu/error-report.h" +#include "qemu/range.h" +#include "sysemu/kvm.h" +#include "sysemu/reset.h" +#include "trace.h" +#include "qapi/error.h" +#include "migration/migration.h" + +VFIOGroupList vfio_group_list =3D + QLIST_HEAD_INITIALIZER(vfio_group_list); + +static int vfio_ram_block_discard_disable(VFIOContainer *container, bool s= tate) +{ + switch (container->iommu_type) { + case VFIO_TYPE1v2_IOMMU: + case VFIO_TYPE1_IOMMU: + /* + * We support coordinated discarding of RAM via the RamDiscardMana= ger. + */ + return ram_block_uncoordinated_discard_disable(state); + default: + /* + * VFIO_SPAPR_TCE_IOMMU most probably works just fine with + * RamDiscardManager, however, it is completely untested. + * + * VFIO_SPAPR_TCE_v2_IOMMU with "DMA memory preregistering" does + * completely the opposite of managing mapping/pinning dynamically= as + * required by RamDiscardManager. We would have to special-case se= ctions + * with a RamDiscardManager. + */ + return ram_block_discard_disable(state); + } +} + +static int vfio_dma_unmap_bitmap(VFIOContainer *container, + hwaddr iova, ram_addr_t size, + IOMMUTLBEntry *iotlb) +{ + struct vfio_iommu_type1_dma_unmap *unmap; + struct vfio_bitmap *bitmap; + VFIOBitmap vbmap; + int ret; + + ret =3D vfio_bitmap_alloc(&vbmap, size); + if (ret) { + return ret; + } + + unmap =3D g_malloc0(sizeof(*unmap) + sizeof(*bitmap)); + + unmap->argsz =3D sizeof(*unmap) + sizeof(*bitmap); + unmap->iova =3D iova; + unmap->size =3D size; + unmap->flags |=3D VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP; + bitmap =3D (struct vfio_bitmap *)&unmap->data; + + /* + * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of + * qemu_real_host_page_size to mark those dirty. Hence set bitmap_pgsi= ze + * to qemu_real_host_page_size. + */ + bitmap->pgsize =3D qemu_real_host_page_size(); + bitmap->size =3D vbmap.size; + bitmap->data =3D (__u64 *)vbmap.bitmap; + + if (vbmap.size > container->max_dirty_bitmap_size) { + error_report("UNMAP: Size of bitmap too big 0x%"PRIx64, vbmap.size= ); + ret =3D -E2BIG; + goto unmap_exit; + } + + ret =3D ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + if (!ret) { + cpu_physical_memory_set_dirty_lebitmap(vbmap.bitmap, + iotlb->translated_addr, vbmap.pages); + } else { + error_report("VFIO_UNMAP_DMA with DIRTY_BITMAP : %m"); + } + +unmap_exit: + g_free(unmap); + g_free(vbmap.bitmap); + + return ret; +} + +/* + * DMA - Mapping and unmapping for the "type1" IOMMU interface used on x86 + */ +int vfio_dma_unmap(VFIOContainer *container, hwaddr iova, + ram_addr_t size, IOMMUTLBEntry *iotlb) +{ + struct vfio_iommu_type1_dma_unmap unmap =3D { + .argsz =3D sizeof(unmap), + .flags =3D 0, + .iova =3D iova, + .size =3D size, + }; + bool need_dirty_sync =3D false; + int ret; + + if (iotlb && vfio_devices_all_running_and_mig_active(container)) { + if (!vfio_devices_all_device_dirty_tracking(container) && + container->dirty_pages_supported) { + return vfio_dma_unmap_bitmap(container, iova, size, iotlb); + } + + need_dirty_sync =3D true; + } + + while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { + /* + * The type1 backend has an off-by-one bug in the kernel (71a7d3d7= 8e3c + * v4.15) where an overflow in its wrap-around check prevents us f= rom + * unmapping the last page of the address space. Test for the err= or + * condition and re-try the unmap excluding the last page. The + * expectation is that we've never mapped the last page anyway and= this + * unmap request comes via vIOMMU support which also makes it unli= kely + * that this page is used. This bug was introduced well after typ= e1 v2 + * support was introduced, so we shouldn't need to test for v1. A= fix + * is queued for kernel v5.0 so this workaround can be removed once + * affected kernels are sufficiently deprecated. + */ + if (errno =3D=3D EINVAL && unmap.size && !(unmap.iova + unmap.size= ) && + container->iommu_type =3D=3D VFIO_TYPE1v2_IOMMU) { + trace_vfio_dma_unmap_overflow_workaround(); + unmap.size -=3D 1ULL << ctz64(container->pgsizes); + continue; + } + error_report("VFIO_UNMAP_DMA failed: %s", strerror(errno)); + return -errno; + } + + if (need_dirty_sync) { + ret =3D vfio_get_dirty_bitmap(container, iova, size, + iotlb->translated_addr); + if (ret) { + return ret; + } + } + + return 0; +} + +int vfio_dma_map(VFIOContainer *container, hwaddr iova, + ram_addr_t size, void *vaddr, bool readonly) +{ + struct vfio_iommu_type1_dma_map map =3D { + .argsz =3D sizeof(map), + .flags =3D VFIO_DMA_MAP_FLAG_READ, + .vaddr =3D (__u64)(uintptr_t)vaddr, + .iova =3D iova, + .size =3D size, + }; + + if (!readonly) { + map.flags |=3D VFIO_DMA_MAP_FLAG_WRITE; + } + + /* + * Try the mapping, if it fails with EBUSY, unmap the region and try + * again. This shouldn't be necessary, but we sometimes see it in + * the VGA ROM space. + */ + if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) =3D=3D 0 || + (errno =3D=3D EBUSY && vfio_dma_unmap(container, iova, size, NULL)= =3D=3D 0 && + ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) =3D=3D 0)) { + return 0; + } + + error_report("VFIO_MAP_DMA failed: %s", strerror(errno)); + return -errno; +} + +int vfio_container_add_section_window(VFIOContainer *container, + MemoryRegionSection *section, + Error **errp) +{ + VFIOHostDMAWindow *hostwin; + hwaddr pgsize =3D 0; + int ret; + + if (container->iommu_type !=3D VFIO_SPAPR_TCE_v2_IOMMU) { + return 0; + } + + /* For now intersections are not allowed, we may relax this later */ + QLIST_FOREACH(hostwin, &container->hostwin_list, hostwin_next) { + if (ranges_overlap(hostwin->min_iova, + hostwin->max_iova - hostwin->min_iova + 1, + section->offset_within_address_space, + int128_get64(section->size))) { + error_setg(errp, + "region [0x%"PRIx64",0x%"PRIx64"] overlaps with existing" + "host DMA window [0x%"PRIx64",0x%"PRIx64"]", + section->offset_within_address_space, + section->offset_within_address_space + + int128_get64(section->size) - 1, + hostwin->min_iova, hostwin->max_iova); + return -EINVAL; + } + } + + ret =3D vfio_spapr_create_window(container, section, &pgsize); + if (ret) { + error_setg_errno(errp, -ret, "Failed to create SPAPR window"); + return ret; + } + + vfio_host_win_add(container, section->offset_within_address_space, + section->offset_within_address_space + + int128_get64(section->size) - 1, pgsize); +#ifdef CONFIG_KVM + if (kvm_enabled()) { + VFIOGroup *group; + IOMMUMemoryRegion *iommu_mr =3D IOMMU_MEMORY_REGION(section->mr); + struct kvm_vfio_spapr_tce param; + struct kvm_device_attr attr =3D { + .group =3D KVM_DEV_VFIO_GROUP, + .attr =3D KVM_DEV_VFIO_GROUP_SET_SPAPR_TCE, + .addr =3D (uint64_t)(unsigned long)¶m, + }; + + if (!memory_region_iommu_get_attr(iommu_mr, IOMMU_ATTR_SPAPR_TCE_F= D, + ¶m.tablefd)) { + QLIST_FOREACH(group, &container->group_list, container_next) { + param.groupfd =3D group->fd; + if (ioctl(vfio_kvm_device_fd, KVM_SET_DEVICE_ATTR, &attr))= { + error_setg_errno(errp, errno, + "vfio: failed GROUP_SET_SPAPR_TCE for= " + "KVM VFIO device %d and group fd %d", + param.tablefd, param.groupfd); + return -errno; + } + trace_vfio_spapr_group_attach(param.groupfd, param.tablefd= ); + } + } + } +#endif + return 0; +} + +void vfio_container_del_section_window(VFIOContainer *container, + MemoryRegionSection *section) +{ + if (container->iommu_type !=3D VFIO_SPAPR_TCE_v2_IOMMU) { + return; + } + + vfio_spapr_remove_window(container, + section->offset_within_address_space); + if (vfio_host_win_del(container, + section->offset_within_address_space, + section->offset_within_address_space + + int128_get64(section->size) - 1) < 0) { + hw_error("%s: Cannot delete missing window at %"HWADDR_PRIx, + __func__, section->offset_within_address_space); + } +} + +int vfio_set_dirty_page_tracking(VFIOContainer *container, bool start) +{ + int ret; + struct vfio_iommu_type1_dirty_bitmap dirty =3D { + .argsz =3D sizeof(dirty), + }; + + if (!container->dirty_pages_supported) { + return 0; + } + + if (start) { + dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_START; + } else { + dirty.flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP; + } + + ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); + if (ret) { + ret =3D -errno; + error_report("Failed to set dirty tracking flag 0x%x errno: %d", + dirty.flags, errno); + } + + return ret; +} + +int vfio_query_dirty_bitmap(VFIOContainer *container, VFIOBitmap *vbmap, + hwaddr iova, hwaddr size) +{ + struct vfio_iommu_type1_dirty_bitmap *dbitmap; + struct vfio_iommu_type1_dirty_bitmap_get *range; + int ret; + + dbitmap =3D g_malloc0(sizeof(*dbitmap) + sizeof(*range)); + + dbitmap->argsz =3D sizeof(*dbitmap) + sizeof(*range); + dbitmap->flags =3D VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP; + range =3D (struct vfio_iommu_type1_dirty_bitmap_get *)&dbitmap->data; + range->iova =3D iova; + range->size =3D size; + + /* + * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of + * qemu_real_host_page_size to mark those dirty. Hence set bitmap's pg= size + * to qemu_real_host_page_size. + */ + range->bitmap.pgsize =3D qemu_real_host_page_size(); + range->bitmap.size =3D vbmap->size; + range->bitmap.data =3D (__u64 *)vbmap->bitmap; + + ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); + if (ret) { + ret =3D -errno; + error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 + " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova, + (uint64_t)range->size, errno); + } + + g_free(dbitmap); + + return ret; +} + +static void vfio_listener_release(VFIOContainer *container) +{ + memory_listener_unregister(&container->listener); + if (container->iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU) { + memory_listener_unregister(&container->prereg_listener); + } +} + +static struct vfio_info_cap_header * +vfio_get_iommu_type1_info_cap(struct vfio_iommu_type1_info *info, uint16_t= id) +{ + if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { + return NULL; + } + + return vfio_get_cap((void *)info, info->cap_offset, id); +} + +bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, + unsigned int *avail) +{ + struct vfio_info_cap_header *hdr; + struct vfio_iommu_type1_info_dma_avail *cap; + + /* If the capability cannot be found, assume no DMA limiting */ + hdr =3D vfio_get_iommu_type1_info_cap(info, + VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL); + if (hdr =3D=3D NULL) { + return false; + } + + if (avail !=3D NULL) { + cap =3D (void *) hdr; + *avail =3D cap->avail; + } + + return true; +} + +static void vfio_kvm_device_add_group(VFIOGroup *group) +{ + Error *err =3D NULL; + + if (vfio_kvm_device_add_fd(group->fd, &err)) { + error_reportf_err(err, "group ID %d: ", group->groupid); + } +} + +static void vfio_kvm_device_del_group(VFIOGroup *group) +{ + Error *err =3D NULL; + + if (vfio_kvm_device_del_fd(group->fd, &err)) { + error_reportf_err(err, "group ID %d: ", group->groupid); + } +} + +/* + * vfio_get_iommu_type - selects the richest iommu_type (v2 first) + */ +static int vfio_get_iommu_type(VFIOContainer *container, + Error **errp) +{ + int iommu_types[] =3D { VFIO_TYPE1v2_IOMMU, VFIO_TYPE1_IOMMU, + VFIO_SPAPR_TCE_v2_IOMMU, VFIO_SPAPR_TCE_IOMMU }; + int i; + + for (i =3D 0; i < ARRAY_SIZE(iommu_types); i++) { + if (ioctl(container->fd, VFIO_CHECK_EXTENSION, iommu_types[i])) { + return iommu_types[i]; + } + } + error_setg(errp, "No available IOMMU models"); + return -EINVAL; +} + +static int vfio_init_container(VFIOContainer *container, int group_fd, + Error **errp) +{ + int iommu_type, ret; + + iommu_type =3D vfio_get_iommu_type(container, errp); + if (iommu_type < 0) { + return iommu_type; + } + + ret =3D ioctl(group_fd, VFIO_GROUP_SET_CONTAINER, &container->fd); + if (ret) { + error_setg_errno(errp, errno, "Failed to set group container"); + return -errno; + } + + while (ioctl(container->fd, VFIO_SET_IOMMU, iommu_type)) { + if (iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU) { + /* + * On sPAPR, despite the IOMMU subdriver always advertises v1 = and + * v2, the running platform may not support v2 and there is no + * way to guess it until an IOMMU group gets added to the cont= ainer. + * So in case it fails with v2, try v1 as a fallback. + */ + iommu_type =3D VFIO_SPAPR_TCE_IOMMU; + continue; + } + error_setg_errno(errp, errno, "Failed to set iommu for container"); + return -errno; + } + + container->iommu_type =3D iommu_type; + return 0; +} + +static int vfio_get_iommu_info(VFIOContainer *container, + struct vfio_iommu_type1_info **info) +{ + + size_t argsz =3D sizeof(struct vfio_iommu_type1_info); + + *info =3D g_new0(struct vfio_iommu_type1_info, 1); +again: + (*info)->argsz =3D argsz; + + if (ioctl(container->fd, VFIO_IOMMU_GET_INFO, *info)) { + g_free(*info); + *info =3D NULL; + return -errno; + } + + if (((*info)->argsz > argsz)) { + argsz =3D (*info)->argsz; + *info =3D g_realloc(*info, argsz); + goto again; + } + + return 0; +} + +static struct vfio_info_cap_header * +vfio_get_iommu_info_cap(struct vfio_iommu_type1_info *info, uint16_t id) +{ + struct vfio_info_cap_header *hdr; + void *ptr =3D info; + + if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { + return NULL; + } + + for (hdr =3D ptr + info->cap_offset; hdr !=3D ptr; hdr =3D ptr + hdr->= next) { + if (hdr->id =3D=3D id) { + return hdr; + } + } + + return NULL; +} + +static void vfio_get_iommu_info_migration(VFIOContainer *container, + struct vfio_iommu_type1_info *in= fo) +{ + struct vfio_info_cap_header *hdr; + struct vfio_iommu_type1_info_cap_migration *cap_mig; + + hdr =3D vfio_get_iommu_info_cap(info, VFIO_IOMMU_TYPE1_INFO_CAP_MIGRAT= ION); + if (!hdr) { + return; + } + + cap_mig =3D container_of(hdr, struct vfio_iommu_type1_info_cap_migrati= on, + header); + + /* + * cpu_physical_memory_set_dirty_lebitmap() supports pages in bitmap of + * qemu_real_host_page_size to mark those dirty. + */ + if (cap_mig->pgsize_bitmap & qemu_real_host_page_size()) { + container->dirty_pages_supported =3D true; + container->max_dirty_bitmap_size =3D cap_mig->max_dirty_bitmap_siz= e; + container->dirty_pgsizes =3D cap_mig->pgsize_bitmap; + } +} + +static int vfio_connect_container(VFIOGroup *group, AddressSpace *as, + Error **errp) +{ + VFIOContainer *container; + int ret, fd; + VFIOAddressSpace *space; + + space =3D vfio_get_address_space(as); + + /* + * VFIO is currently incompatible with discarding of RAM insofar as the + * madvise to purge (zap) the page from QEMU's address space does not + * interact with the memory API and therefore leaves stale virtual to + * physical mappings in the IOMMU if the page was previously pinned. = We + * therefore set discarding broken for each group added to a container, + * whether the container is used individually or shared. This provides + * us with options to allow devices within a group to opt-in and allow + * discarding, so long as it is done consistently for a group (for ins= tance + * if the device is an mdev device where it is known that the host ven= dor + * driver will never pin pages outside of the working set of the guest + * driver, which would thus not be discarding candidates). + * + * The first opportunity to induce pinning occurs here where we attemp= t to + * attach the group to existing containers within the AddressSpace. I= f any + * pages are already zapped from the virtual address space, such as fr= om + * previous discards, new pinning will cause valid mappings to be + * re-established. Likewise, when the overall MemoryListener for a new + * container is registered, a replay of mappings within the AddressSpa= ce + * will occur, re-establishing any previously zapped pages as well. + * + * Especially virtio-balloon is currently only prevented from discardi= ng + * new memory, it will not yet set ram_block_discard_set_required() and + * therefore, neither stops us here or deals with the sudden memory + * consumption of inflated memory. + * + * We do support discarding of memory coordinated via the RamDiscardMa= nager + * with some IOMMU types. vfio_ram_block_discard_disable() handles the + * details once we know which type of IOMMU we are using. + */ + + QLIST_FOREACH(container, &space->containers, next) { + if (!ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd)) { + ret =3D vfio_ram_block_discard_disable(container, true); + if (ret) { + error_setg_errno(errp, -ret, + "Cannot set discarding of RAM broken"); + if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, + &container->fd)) { + error_report("vfio: error disconnecting group %d from" + " container", group->groupid); + } + return ret; + } + group->container =3D container; + QLIST_INSERT_HEAD(&container->group_list, group, container_nex= t); + vfio_kvm_device_add_group(group); + return 0; + } + } + + fd =3D qemu_open_old("/dev/vfio/vfio", O_RDWR); + if (fd < 0) { + error_setg_errno(errp, errno, "failed to open /dev/vfio/vfio"); + ret =3D -errno; + goto put_space_exit; + } + + ret =3D ioctl(fd, VFIO_GET_API_VERSION); + if (ret !=3D VFIO_API_VERSION) { + error_setg(errp, "supported vfio version: %d, " + "reported version: %d", VFIO_API_VERSION, ret); + ret =3D -EINVAL; + goto close_fd_exit; + } + + container =3D g_malloc0(sizeof(*container)); + container->space =3D space; + container->fd =3D fd; + container->error =3D NULL; + container->dirty_pages_supported =3D false; + container->dma_max_mappings =3D 0; + QLIST_INIT(&container->giommu_list); + QLIST_INIT(&container->hostwin_list); + QLIST_INIT(&container->vrdl_list); + + ret =3D vfio_init_container(container, group->fd, errp); + if (ret) { + goto free_container_exit; + } + + ret =3D vfio_ram_block_discard_disable(container, true); + if (ret) { + error_setg_errno(errp, -ret, "Cannot set discarding of RAM broken"= ); + goto free_container_exit; + } + + switch (container->iommu_type) { + case VFIO_TYPE1v2_IOMMU: + case VFIO_TYPE1_IOMMU: + { + struct vfio_iommu_type1_info *info; + + ret =3D vfio_get_iommu_info(container, &info); + if (ret) { + error_setg_errno(errp, -ret, "Failed to get VFIO IOMMU info"); + goto enable_discards_exit; + } + + if (info->flags & VFIO_IOMMU_INFO_PGSIZES) { + container->pgsizes =3D info->iova_pgsizes; + } else { + container->pgsizes =3D qemu_real_host_page_size(); + } + + if (!vfio_get_info_dma_avail(info, &container->dma_max_mappings)) { + container->dma_max_mappings =3D 65535; + } + vfio_get_iommu_info_migration(container, info); + g_free(info); + + /* + * FIXME: We should parse VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE + * information to get the actual window extent rather than assume + * a 64-bit IOVA address space. + */ + vfio_host_win_add(container, 0, (hwaddr)-1, container->pgsizes); + + break; + } + case VFIO_SPAPR_TCE_v2_IOMMU: + case VFIO_SPAPR_TCE_IOMMU: + { + struct vfio_iommu_spapr_tce_info info; + bool v2 =3D container->iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU; + + /* + * The host kernel code implementing VFIO_IOMMU_DISABLE is called + * when container fd is closed so we do not call it explicitly + * in this file. + */ + if (!v2) { + ret =3D ioctl(fd, VFIO_IOMMU_ENABLE); + if (ret) { + error_setg_errno(errp, errno, "failed to enable container"= ); + ret =3D -errno; + goto enable_discards_exit; + } + } else { + container->prereg_listener =3D vfio_prereg_listener; + + memory_listener_register(&container->prereg_listener, + &address_space_memory); + if (container->error) { + memory_listener_unregister(&container->prereg_listener); + ret =3D -1; + error_propagate_prepend(errp, container->error, + "RAM memory listener initialization failed: "); + goto enable_discards_exit; + } + } + + info.argsz =3D sizeof(info); + ret =3D ioctl(fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info); + if (ret) { + error_setg_errno(errp, errno, + "VFIO_IOMMU_SPAPR_TCE_GET_INFO failed"); + ret =3D -errno; + if (v2) { + memory_listener_unregister(&container->prereg_listener); + } + goto enable_discards_exit; + } + + if (v2) { + container->pgsizes =3D info.ddw.pgsizes; + /* + * There is a default window in just created container. + * To make region_add/del simpler, we better remove this + * window now and let those iommu_listener callbacks + * create/remove them when needed. + */ + ret =3D vfio_spapr_remove_window(container, info.dma32_window_= start); + if (ret) { + error_setg_errno(errp, -ret, + "failed to remove existing window"); + goto enable_discards_exit; + } + } else { + /* The default table uses 4K pages */ + container->pgsizes =3D 0x1000; + vfio_host_win_add(container, info.dma32_window_start, + info.dma32_window_start + + info.dma32_window_size - 1, + 0x1000); + } + } + } + + vfio_kvm_device_add_group(group); + + QLIST_INIT(&container->group_list); + QLIST_INSERT_HEAD(&space->containers, container, next); + + group->container =3D container; + QLIST_INSERT_HEAD(&container->group_list, group, container_next); + + container->listener =3D vfio_memory_listener; + + memory_listener_register(&container->listener, container->space->as); + + if (container->error) { + ret =3D -1; + error_propagate_prepend(errp, container->error, + "memory listener initialization failed: "); + goto listener_release_exit; + } + + container->initialized =3D true; + + return 0; +listener_release_exit: + QLIST_REMOVE(group, container_next); + QLIST_REMOVE(container, next); + vfio_kvm_device_del_group(group); + vfio_listener_release(container); + +enable_discards_exit: + vfio_ram_block_discard_disable(container, false); + +free_container_exit: + g_free(container); + +close_fd_exit: + close(fd); + +put_space_exit: + vfio_put_address_space(space); + + return ret; +} + +static void vfio_disconnect_container(VFIOGroup *group) +{ + VFIOContainer *container =3D group->container; + + QLIST_REMOVE(group, container_next); + group->container =3D NULL; + + /* + * Explicitly release the listener first before unset container, + * since unset may destroy the backend container if it's the last + * group. + */ + if (QLIST_EMPTY(&container->group_list)) { + vfio_listener_release(container); + } + + if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, &container->fd)) { + error_report("vfio: error disconnecting group %d from container", + group->groupid); + } + + if (QLIST_EMPTY(&container->group_list)) { + VFIOAddressSpace *space =3D container->space; + VFIOGuestIOMMU *giommu, *tmp; + VFIOHostDMAWindow *hostwin, *next; + + QLIST_REMOVE(container, next); + + QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, t= mp) { + memory_region_unregister_iommu_notifier( + MEMORY_REGION(giommu->iommu_mr), &giommu->n); + QLIST_REMOVE(giommu, giommu_next); + g_free(giommu); + } + + QLIST_FOREACH_SAFE(hostwin, &container->hostwin_list, hostwin_next, + next) { + QLIST_REMOVE(hostwin, hostwin_next); + g_free(hostwin); + } + + trace_vfio_disconnect_container(container->fd); + close(container->fd); + g_free(container); + + vfio_put_address_space(space); + } +} + +static VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **er= rp) +{ + VFIOGroup *group; + char path[32]; + struct vfio_group_status status =3D { .argsz =3D sizeof(status) }; + + QLIST_FOREACH(group, &vfio_group_list, next) { + if (group->groupid =3D=3D groupid) { + /* Found it. Now is it already in the right context? */ + if (group->container->space->as =3D=3D as) { + return group; + } else { + error_setg(errp, "group %d used in multiple address spaces= ", + group->groupid); + return NULL; + } + } + } + + group =3D g_malloc0(sizeof(*group)); + + snprintf(path, sizeof(path), "/dev/vfio/%d", groupid); + group->fd =3D qemu_open_old(path, O_RDWR); + if (group->fd < 0) { + error_setg_errno(errp, errno, "failed to open %s", path); + goto free_group_exit; + } + + if (ioctl(group->fd, VFIO_GROUP_GET_STATUS, &status)) { + error_setg_errno(errp, errno, "failed to get group %d status", gro= upid); + goto close_fd_exit; + } + + if (!(status.flags & VFIO_GROUP_FLAGS_VIABLE)) { + error_setg(errp, "group %d is not viable", groupid); + error_append_hint(errp, + "Please ensure all devices within the iommu_grou= p " + "are bound to their vfio bus driver.\n"); + goto close_fd_exit; + } + + group->groupid =3D groupid; + QLIST_INIT(&group->device_list); + + if (vfio_connect_container(group, as, errp)) { + error_prepend(errp, "failed to setup container for group %d: ", + groupid); + goto close_fd_exit; + } + + QLIST_INSERT_HEAD(&vfio_group_list, group, next); + + return group; + +close_fd_exit: + close(group->fd); + +free_group_exit: + g_free(group); + + return NULL; +} + +static void vfio_put_group(VFIOGroup *group) +{ + if (!group || !QLIST_EMPTY(&group->device_list)) { + return; + } + + if (!group->ram_block_discard_allowed) { + vfio_ram_block_discard_disable(group->container, false); + } + vfio_kvm_device_del_group(group); + vfio_disconnect_container(group); + QLIST_REMOVE(group, next); + trace_vfio_put_group(group->fd); + close(group->fd); + g_free(group); +} + +static int vfio_get_device(VFIOGroup *group, const char *name, + VFIODevice *vbasedev, Error **errp) +{ + g_autofree struct vfio_device_info *info =3D NULL; + int fd; + + fd =3D ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, name); + if (fd < 0) { + error_setg_errno(errp, errno, "error getting device from group %d", + group->groupid); + error_append_hint(errp, + "Verify all devices in group %d are bound to vfio- " + "or pci-stub and not already in use\n", group->group= id); + return fd; + } + + info =3D vfio_get_device_info(fd); + if (!info) { + error_setg_errno(errp, errno, "error getting device info"); + close(fd); + return -1; + } + + /* + * Set discarding of RAM as not broken for this group if the driver kn= ows + * the device operates compatibly with discarding. Setting must be + * consistent per group, but since compatibility is really only possib= le + * with mdev currently, we expect singleton groups. + */ + if (vbasedev->ram_block_discard_allowed !=3D + group->ram_block_discard_allowed) { + if (!QLIST_EMPTY(&group->device_list)) { + error_setg(errp, "Inconsistent setting of support for discardi= ng " + "RAM (e.g., balloon) within group"); + close(fd); + return -1; + } + + if (!group->ram_block_discard_allowed) { + group->ram_block_discard_allowed =3D true; + vfio_ram_block_discard_disable(group->container, false); + } + } + + vbasedev->fd =3D fd; + vbasedev->group =3D group; + QLIST_INSERT_HEAD(&group->device_list, vbasedev, next); + + vbasedev->num_irqs =3D info->num_irqs; + vbasedev->num_regions =3D info->num_regions; + vbasedev->flags =3D info->flags; + + trace_vfio_get_device(name, info->flags, info->num_regions, info->num_= irqs); + + vbasedev->reset_works =3D !!(info->flags & VFIO_DEVICE_FLAGS_RESET); + + return 0; +} + +static void vfio_put_base_device(VFIODevice *vbasedev) +{ + if (!vbasedev->group) { + return; + } + QLIST_REMOVE(vbasedev, next); + vbasedev->group =3D NULL; + trace_vfio_put_base_device(vbasedev->fd); + close(vbasedev->fd); +} + +/* + * Interfaces for IBM EEH (Enhanced Error Handling) + */ +static bool vfio_eeh_container_ok(VFIOContainer *container) +{ + /* + * As of 2016-03-04 (linux-4.5) the host kernel EEH/VFIO + * implementation is broken if there are multiple groups in a + * container. The hardware works in units of Partitionable + * Endpoints (=3D=3D IOMMU groups) and the EEH operations naively + * iterate across all groups in the container, without any logic + * to make sure the groups have their state synchronized. For + * certain operations (ENABLE) that might be ok, until an error + * occurs, but for others (GET_STATE) it's clearly broken. + */ + + /* + * XXX Once fixed kernels exist, test for them here + */ + + if (QLIST_EMPTY(&container->group_list)) { + return false; + } + + if (QLIST_NEXT(QLIST_FIRST(&container->group_list), container_next)) { + return false; + } + + return true; +} + +static int vfio_eeh_container_op(VFIOContainer *container, uint32_t op) +{ + struct vfio_eeh_pe_op pe_op =3D { + .argsz =3D sizeof(pe_op), + .op =3D op, + }; + int ret; + + if (!vfio_eeh_container_ok(container)) { + error_report("vfio/eeh: EEH_PE_OP 0x%x: " + "kernel requires a container with exactly one group",= op); + return -EPERM; + } + + ret =3D ioctl(container->fd, VFIO_EEH_PE_OP, &pe_op); + if (ret < 0) { + error_report("vfio/eeh: EEH_PE_OP 0x%x failed: %m", op); + return -errno; + } + + return ret; +} + +static VFIOContainer *vfio_eeh_as_container(AddressSpace *as) +{ + VFIOAddressSpace *space =3D vfio_get_address_space(as); + VFIOContainer *container =3D NULL; + + if (QLIST_EMPTY(&space->containers)) { + /* No containers to act on */ + goto out; + } + + container =3D QLIST_FIRST(&space->containers); + + if (QLIST_NEXT(container, next)) { + /* + * We don't yet have logic to synchronize EEH state across + * multiple containers + */ + container =3D NULL; + goto out; + } + +out: + vfio_put_address_space(space); + return container; +} + +bool vfio_eeh_as_ok(AddressSpace *as) +{ + VFIOContainer *container =3D vfio_eeh_as_container(as); + + return (container !=3D NULL) && vfio_eeh_container_ok(container); +} + +int vfio_eeh_as_op(AddressSpace *as, uint32_t op) +{ + VFIOContainer *container =3D vfio_eeh_as_container(as); + + if (!container) { + return -ENODEV; + } + return vfio_eeh_container_op(container, op); +} + +static int vfio_device_groupid(VFIODevice *vbasedev, Error **errp) +{ + char *tmp, group_path[PATH_MAX], *group_name; + int ret, groupid; + ssize_t len; + + tmp =3D g_strdup_printf("%s/iommu_group", vbasedev->sysfsdev); + len =3D readlink(tmp, group_path, sizeof(group_path)); + g_free(tmp); + + if (len <=3D 0 || len >=3D sizeof(group_path)) { + ret =3D len < 0 ? -errno : -ENAMETOOLONG; + error_setg_errno(errp, -ret, "no iommu_group found"); + return ret; + } + + group_path[len] =3D 0; + + group_name =3D basename(group_path); + if (sscanf(group_name, "%d", &groupid) !=3D 1) { + error_setg_errno(errp, errno, "failed to read %s", group_path); + return -errno; + } + return groupid; +} + +/* + * vfio_attach_device: attach a device to a security context + * @name and @vbasedev->name are likely to be different depending + * on the type of the device, hence the need for passing @name + */ +int vfio_attach_device(char *name, VFIODevice *vbasedev, + AddressSpace *as, Error **errp) +{ + int groupid =3D vfio_device_groupid(vbasedev, errp); + VFIODevice *vbasedev_iter; + VFIOGroup *group; + VFIOContainer *container; + int ret; + + if (groupid < 0) { + return groupid; + } + + trace_vfio_attach_device(vbasedev->name, groupid); + + group =3D vfio_get_group(groupid, as, errp); + if (!group) { + return -ENOENT; + } + + QLIST_FOREACH(vbasedev_iter, &group->device_list, next) { + if (strcmp(vbasedev_iter->name, vbasedev->name) =3D=3D 0) { + error_setg(errp, "device is already attached"); + vfio_put_group(group); + return -EBUSY; + } + } + ret =3D vfio_get_device(group, name, vbasedev, errp); + if (ret) { + vfio_put_group(group); + return ret; + } + + container =3D group->container; + vbasedev->container =3D container; + QLIST_INSERT_HEAD(&container->device_list, vbasedev, container_next); + QLIST_INSERT_HEAD(&vfio_device_list, vbasedev, global_next); + + return ret; +} + +void vfio_detach_device(VFIODevice *vbasedev) +{ + VFIOGroup *group =3D vbasedev->group; + + if (!vbasedev->container) { + return; + } + + QLIST_REMOVE(vbasedev, global_next); + QLIST_REMOVE(vbasedev, container_next); + vbasedev->container =3D NULL; + trace_vfio_detach_device(vbasedev->name, group->groupid); + vfio_put_base_device(vbasedev); + vfio_put_group(group); +} diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index 3746c9f98420b1be949443b5d57f26a9982a37fe..2a6912c94027d6213144f910d64= 625a469cc2b1f 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -2,6 +2,7 @@ vfio_ss =3D ss.source_set() vfio_ss.add(files( 'helpers.c', 'common.c', + 'container.c', 'spapr.c', 'migration.c', )) --=20 2.41.0