From nobody Mon Feb 9 11:56:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=yandex-team.com ARC-Seal: i=1; a=rsa-sha256; t=1666005188; cv=none; d=zohomail.com; s=zohoarc; b=AZQYmvSBIxGJxLEQh1Pepc74XCj9Zxo5ZTIfm8wQVAR7WOta5wQhaRNIPT6IAKp46R4LXgpJ8VCxiA+NAoR+r7ja/yoytaYqJ3G8ZmO9zd2QUXwJRPqPoyofLUcpm6qQWOhWa+c1UCeo4vMu1gc166YjXTYaMMmBHvkGLnkw0rY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1666005188; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=kMM+N5Dh3Fi28eWzt4V2B1pW2mzLtqj1Y0XpSfOP7mY=; b=RRnVZyW4gBetgsHE4dqn3edEKqXqXabNCUNYiQYgiXnJ+hY0TqClnn9zQlg2G+vdGtU/DjWym3xPFjgKja2I7tRZ0iK/yADK89i7FpKpkxL5iQx8WsIT9clPx03u1+wDhKigVEIUAmr0B7dV21VR2yxXN21577T87/5A5pygjjg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 16660051882052.4644673580338576; Mon, 17 Oct 2022 04:13:08 -0700 (PDT) Received: from localhost ([::1]:51304 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1okO3P-00005Z-5J for importer@patchew.org; Mon, 17 Oct 2022 07:13:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33292) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okNvy-0001Ry-8i for qemu-devel@nongnu.org; Mon, 17 Oct 2022 07:05:27 -0400 Received: from forwardcorp1b.mail.yandex.net ([178.154.239.136]:54280) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okNvq-0001IJ-LW for qemu-devel@nongnu.org; Mon, 17 Oct 2022 07:05:22 -0400 Received: from sas1-7470331623bb.qloud-c.yandex.net (sas1-7470331623bb.qloud-c.yandex.net [IPv6:2a02:6b8:c08:bd1e:0:640:7470:3316]) by forwardcorp1b.mail.yandex.net (Yandex) with ESMTP id 65AD160E94; Mon, 17 Oct 2022 13:54:24 +0300 (MSK) Received: from dellarbn.yandex.net (unknown [2a02:6b8:0:107:3e85:844d:5b1d:60a]) by sas1-7470331623bb.qloud-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id PrI3KDUSQF-sM7KnEEA; Mon, 17 Oct 2022 13:54:23 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) Precedence: bulk X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.com; s=default; t=1666004063; bh=kMM+N5Dh3Fi28eWzt4V2B1pW2mzLtqj1Y0XpSfOP7mY=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=WlvErUzyA6X7SkoW/j5092EcTPJ5b2+cqD6rjdLR8+8EnpJ8LfpQbxpnHlsv7b75N hClnNVq6ArnZimO6ihku9LFAAnyEhMgN4+H8bLXHEP+JYDzVnJwgoJEIpWB7crqzvk dgQMRDi/7PaH460MBGJEF4Ah5l8rdyXTpAE2V18s= Authentication-Results: sas1-7470331623bb.qloud-c.yandex.net; dkim=pass header.i=@yandex-team.com From: Andrey Ryabinin To: qemu-devel@nongnu.org Cc: Steve Sistare , yc-core@yandex-team.ru, Andrey Ryabinin , Cornelia Huck , Thomas Huth , Alex Williamson , Tony Krowiak , Halil Pasic , Jason Herne , Eric Farman , Matthew Rosato , Paolo Bonzini , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , Eduardo Habkost , Eric Blake , Markus Armbruster , Cleber Rosa , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , Beraldo Leal Subject: [PATCH 1/4] vfio: add vfio-container user createable object Date: Mon, 17 Oct 2022 13:54:04 +0300 Message-Id: <20221017105407.3858-2-arbn@yandex-team.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221017105407.3858-1-arbn@yandex-team.com> References: <20221017105407.3858-1-arbn@yandex-team.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=178.154.239.136; envelope-from=arbn@yandex-team.com; helo=forwardcorp1b.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @yandex-team.com) X-ZM-MESSAGEID: 1666005190450100001 Content-Type: text/plain; charset="utf-8" Add vfio-container type and allow user to create vfio-container object via '-object' command line argument or 'object-add' qmp command. Add 'fd' parameter to this type to allow user provide file descriptor of a vfio-container. E.g. -object vfio-container,id=3Dct,fd=3D5 Signed-off-by: Andrey Ryabinin --- hw/vfio/common.c | 183 ++++++++++++++++++++++------------ hw/vfio/trace-events | 2 +- include/hw/vfio/vfio-common.h | 4 + qapi/qom.json | 14 +++ 4 files changed, 137 insertions(+), 66 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 6b5d8c0bf69..392057d3025 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -31,9 +31,11 @@ #include "exec/memory.h" #include "exec/ram_addr.h" #include "hw/hw.h" +#include "monitor/monitor.h" #include "qemu/error-report.h" #include "qemu/main-loop.h" #include "qemu/range.h" +#include "qom/object_interfaces.h" #include "sysemu/kvm.h" #include "sysemu/reset.h" #include "sysemu/runstate.h" @@ -2013,7 +2015,7 @@ static int vfio_connect_container(VFIOGroup *group, A= ddressSpace *as, Error **errp) { VFIOContainer *container; - int ret, fd; + int ret; VFIOAddressSpace *space; =20 space =3D vfio_get_address_space(as); @@ -2069,31 +2071,15 @@ static int vfio_connect_container(VFIOGroup *group,= AddressSpace *as, } } =20 - fd =3D qemu_open_old("/dev/vfio/vfio", O_RDWR); - if (fd < 0) { - error_setg_errno(errp, errno, "failed to open /dev/vfio/vfio"); - ret =3D -errno; - goto put_space_exit; - } + container =3D VFIO_CONTAINER(object_new(TYPE_VFIO_CONTAINER)); + container->space =3D space; =20 - ret =3D ioctl(fd, VFIO_GET_API_VERSION); - if (ret !=3D VFIO_API_VERSION) { - error_setg(errp, "supported vfio version: %d, " - "reported version: %d", VFIO_API_VERSION, ret); - ret =3D -EINVAL; - goto close_fd_exit; + user_creatable_complete(USER_CREATABLE(container), errp); + if (*errp) { + ret =3D -1; + goto free_container_exit; } =20 - container =3D g_malloc0(sizeof(*container)); - container->space =3D space; - container->fd =3D fd; - container->error =3D NULL; - container->dirty_pages_supported =3D false; - container->dma_max_mappings =3D 0; - QLIST_INIT(&container->giommu_list); - QLIST_INIT(&container->hostwin_list); - QLIST_INIT(&container->vrdl_list); - ret =3D vfio_init_container(container, group->fd, errp); if (ret) { goto free_container_exit; @@ -2150,7 +2136,7 @@ static int vfio_connect_container(VFIOGroup *group, A= ddressSpace *as, * in this file. */ if (!v2) { - ret =3D ioctl(fd, VFIO_IOMMU_ENABLE); + ret =3D ioctl(container->fd, VFIO_IOMMU_ENABLE); if (ret) { error_setg_errno(errp, errno, "failed to enable container"= ); ret =3D -errno; @@ -2171,7 +2157,7 @@ static int vfio_connect_container(VFIOGroup *group, A= ddressSpace *as, } =20 info.argsz =3D sizeof(info); - ret =3D ioctl(fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info); + ret =3D ioctl(container->fd, VFIO_IOMMU_SPAPR_TCE_GET_INFO, &info); if (ret) { error_setg_errno(errp, errno, "VFIO_IOMMU_SPAPR_TCE_GET_INFO failed"); @@ -2209,7 +2195,6 @@ static int vfio_connect_container(VFIOGroup *group, A= ddressSpace *as, =20 vfio_kvm_device_add_group(group); =20 - QLIST_INIT(&container->group_list); QLIST_INSERT_HEAD(&space->containers, container, next); =20 group->container =3D container; @@ -2223,29 +2208,18 @@ static int vfio_connect_container(VFIOGroup *group,= AddressSpace *as, ret =3D -1; error_propagate_prepend(errp, container->error, "memory listener initialization failed: "); - goto listener_release_exit; + goto free_container_exit; } =20 container->initialized =3D true; =20 return 0; -listener_release_exit: - QLIST_REMOVE(group, container_next); - QLIST_REMOVE(container, next); - vfio_kvm_device_del_group(group); - vfio_listener_release(container); =20 enable_discards_exit: vfio_ram_block_discard_disable(container, false); =20 free_container_exit: - g_free(container); - -close_fd_exit: - close(fd); - -put_space_exit: - vfio_put_address_space(space); + object_unref(OBJECT(container)); =20 return ret; } @@ -2271,32 +2245,7 @@ static void vfio_disconnect_container(VFIOGroup *gro= up) group->groupid); } =20 - if (QLIST_EMPTY(&container->group_list)) { - VFIOAddressSpace *space =3D container->space; - VFIOGuestIOMMU *giommu, *tmp; - VFIOHostDMAWindow *hostwin, *next; - - QLIST_REMOVE(container, next); - - QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, t= mp) { - memory_region_unregister_iommu_notifier( - MEMORY_REGION(giommu->iommu_mr), &giommu->n); - QLIST_REMOVE(giommu, giommu_next); - g_free(giommu); - } - - QLIST_FOREACH_SAFE(hostwin, &container->hostwin_list, hostwin_next, - next) { - QLIST_REMOVE(hostwin, hostwin_next); - g_free(hostwin); - } - - trace_vfio_disconnect_container(container->fd); - close(container->fd); - g_free(container); - - vfio_put_address_space(space); - } + object_unref(OBJECT(container)); } =20 VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) @@ -2628,3 +2577,107 @@ int vfio_eeh_as_op(AddressSpace *as, uint32_t op) } return vfio_eeh_container_op(container, op); } + +static void vfio_container_set_fd(Object *obj, const char *value, + Error **errp) +{ + VFIOContainer *ct =3D VFIO_CONTAINER(obj); + + ct->fd =3D monitor_fd_param(monitor_cur(), value, errp); +} + +static void vfio_container_complete(UserCreatable *uc, Error **errp) +{ + VFIOContainer *container =3D VFIO_CONTAINER(uc); + int ret; + + if (container->fd < 0) { + int fd; + + fd =3D qemu_open_old("/dev/vfio/vfio", O_RDWR); + if (fd < 0) { + error_setg_errno(errp, errno, "failed to open /dev/vfio/vfio"); + return; + } + container->fd =3D fd; + } + + ret =3D ioctl(container->fd, VFIO_GET_API_VERSION); + if (ret !=3D VFIO_API_VERSION) { + error_setg(errp, "supported vfio version: %d, " + "reported version: %d", VFIO_API_VERSION, ret); + return; + } +} + +static void vfio_container_class_init(ObjectClass *class, void *data) +{ + UserCreatableClass *ucc =3D USER_CREATABLE_CLASS(class); + ucc->complete =3D vfio_container_complete; + + object_class_property_add_str(class, "fd", NULL, vfio_container_set_fd= ); +} + +static void vfio_container_instance_init(Object *obj) +{ + VFIOContainer *ct =3D VFIO_CONTAINER(obj); + + ct->dirty_pages_supported =3D false; + ct->dma_max_mappings =3D 0; + ct->fd =3D -1; + QLIST_INIT(&ct->giommu_list); + QLIST_INIT(&ct->hostwin_list); + QLIST_INIT(&ct->group_list); + QLIST_INIT(&ct->vrdl_list); +} + +static void +vfio_container_instance_finalize(Object *obj) +{ + VFIOContainer *container =3D VFIO_CONTAINER(obj); + VFIOAddressSpace *space =3D container->space; + VFIOGuestIOMMU *giommu, *tmp; + VFIOHostDMAWindow *hostwin, *next; + + QLIST_REMOVE(container, next); + + QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, tmp) { + memory_region_unregister_iommu_notifier( + MEMORY_REGION(giommu->iommu_mr), &giommu->n); + QLIST_REMOVE(giommu, giommu_next); + g_free(giommu); + } + QLIST_FOREACH_SAFE(hostwin, &container->hostwin_list, hostwin_next, + next) { + QLIST_REMOVE(hostwin, hostwin_next); + g_free(hostwin); + } + + + trace_vfio_container_instance_finalize(container->fd); + if (container->fd > 0) { + close(container->fd); + } + if (space) { + vfio_put_address_space(space); + } +} + +static const TypeInfo vfio_container_info =3D { + .name =3D TYPE_VFIO_CONTAINER, + .parent =3D TYPE_OBJECT, + .class_init =3D vfio_container_class_init, + .instance_size =3D sizeof(VFIOContainer), + .instance_init =3D vfio_container_instance_init, + .instance_finalize =3D vfio_container_instance_finalize, + .interfaces =3D (InterfaceInfo[]) { + {TYPE_USER_CREATABLE}, + {} + }, +}; + +static void register_vfio_types(void) +{ + type_register_static(&vfio_container_info); +} +type_init(register_vfio_types) diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 73dffe9e00d..8b79cf33a33 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -104,7 +104,7 @@ vfio_known_safe_misalignment(const char *name, uint64_t= iova, uint64_t offset_wi vfio_listener_region_add_no_dma_map(const char *name, uint64_t iova, uint6= 4_t size, uint64_t page_size) "Region \"%s\" 0x%"PRIx64" size=3D0x%"PRIx64"= is not aligned to 0x%"PRIx64" and cannot be mapped for DMA" vfio_listener_region_del_skip(uint64_t start, uint64_t end) "SKIPPING regi= on_del 0x%"PRIx64" - 0x%"PRIx64 vfio_listener_region_del(uint64_t start, uint64_t end) "region_del 0x%"PRI= x64" - 0x%"PRIx64 -vfio_disconnect_container(int fd) "close container->fd=3D%d" +vfio_container_instance_finalize(int fd) "close container->fd=3D%d" vfio_put_group(int fd) "close group->fd=3D%d" vfio_get_device(const char * name, unsigned int flags, unsigned int num_re= gions, unsigned int num_irqs) "Device %s flags: %u, regions: %u, irqs: %u" vfio_put_base_device(int fd) "close vdev->fd=3D%d" diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index e573f5a9f19..0ab99060e44 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -77,6 +77,7 @@ typedef struct VFIOAddressSpace { struct VFIOGroup; =20 typedef struct VFIOContainer { + Object parent; VFIOAddressSpace *space; int fd; /* /dev/vfio/vfio, empowered by the attached groups */ MemoryListener listener; @@ -190,6 +191,9 @@ typedef struct VFIODisplay { } dmabuf; } VFIODisplay; =20 +#define TYPE_VFIO_CONTAINER "vfio-container" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOContainer, VFIO_CONTAINER) + void vfio_put_base_device(VFIODevice *vbasedev); void vfio_disable_irqindex(VFIODevice *vbasedev, int index); void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index); diff --git a/qapi/qom.json b/qapi/qom.json index 80dd419b392..d1a88e10b52 100644 --- a/qapi/qom.json +++ b/qapi/qom.json @@ -734,6 +734,18 @@ { 'struct': 'RemoteObjectProperties', 'data': { 'fd': 'str', 'devid': 'str' } } =20 +## +# @VFIOContainerProperties: +# +# Properties for vfio-container objects. +# +# @fd: file descriptor of vfio container +# +# Since: 7.2 +## +{ 'struct': 'VFIOContainerProperties', + 'data': { 'fd': 'str' } } + ## # @VfioUserServerProperties: # @@ -888,6 +900,7 @@ 'tls-creds-psk', 'tls-creds-x509', 'tls-cipher-suites', + 'vfio-container', { 'name': 'x-remote-object', 'features': [ 'unstable' ] }, { 'name': 'x-vfio-user-server', 'features': [ 'unstable' ] } ] } @@ -953,6 +966,7 @@ 'tls-creds-psk': 'TlsCredsPskProperties', 'tls-creds-x509': 'TlsCredsX509Properties', 'tls-cipher-suites': 'TlsCredsProperties', + 'vfio-container': 'VFIOContainerProperties', 'x-remote-object': 'RemoteObjectProperties', 'x-vfio-user-server': 'VfioUserServerProperties' } } --=20 2.37.3 From nobody Mon Feb 9 11:56:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=yandex-team.com ARC-Seal: i=1; a=rsa-sha256; t=1666005565; cv=none; d=zohomail.com; s=zohoarc; b=V3HRPfkTQPy7Yu4ZlSKv6vltiu8Wi/rTZBY6xwwdEuYyQSrL+E7Suk8oBnqr3EKEcyLU8lPaMmLwNvq7vb1LsDA/7kxrTwxI9ZqgIf+j0QBn3aDaj+emxNR+1ytulYfjDTM72Y4JNhLhquTIDxBItjbBkgcMB+tqRdMo6q/Xpr8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1666005565; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=zo7zMEbss00aqj6jZXqzSYwpPO2/VtpkStNC3EWwfRw=; b=H4AkbS6SVendjF4AIxauvre4BW0h79hJVEEKYBe2iMDmVGkpGVgSxuN3VR+LobF5td83Y1hRMbng3LtmNZEzx44s2WOr19A6RNHmGCnRggyRQhhA1jE0k6/B5qLy3IsN8gJtq9N5oprMLkqaW5r1qfwLlWK3SMCESmnilwfo9UM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1666005565014955.6380239462883; Mon, 17 Oct 2022 04:19:25 -0700 (PDT) Received: from localhost ([::1]:58424 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1okO9S-0000eX-Up for importer@patchew.org; Mon, 17 Oct 2022 07:19:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33290) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okNvy-0001Rx-U4 for qemu-devel@nongnu.org; Mon, 17 Oct 2022 07:05:27 -0400 Received: from forwardcorp1b.mail.yandex.net ([178.154.239.136]:54284) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okNvq-0001IN-Lk for qemu-devel@nongnu.org; Mon, 17 Oct 2022 07:05:20 -0400 Received: from sas1-7470331623bb.qloud-c.yandex.net (sas1-7470331623bb.qloud-c.yandex.net [IPv6:2a02:6b8:c08:bd1e:0:640:7470:3316]) by forwardcorp1b.mail.yandex.net (Yandex) with ESMTP id 0238160EB0; Mon, 17 Oct 2022 13:54:27 +0300 (MSK) Received: from dellarbn.yandex.net (unknown [2a02:6b8:0:107:3e85:844d:5b1d:60a]) by sas1-7470331623bb.qloud-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id PrI3KDUSQF-sO7K69hh; Mon, 17 Oct 2022 13:54:25 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) Precedence: bulk X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.com; s=default; t=1666004065; bh=zo7zMEbss00aqj6jZXqzSYwpPO2/VtpkStNC3EWwfRw=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=m1QiDMN2F0PkPGFnW0/2tYkmLVQqBjSC0XXUxSIXIlRmISQRmlPfyZIIi4aaxBBX0 nusufLJ6dLbhWYPrMZxMsMCT+qhCQ5kKB2I4oW6Vz19APz3tMAmTeioDb3+Ma4ctyd I3SBgnw0YPqk5ink6wwqlW3L5/bO5+tN0NIDuTME= Authentication-Results: sas1-7470331623bb.qloud-c.yandex.net; dkim=pass header.i=@yandex-team.com From: Andrey Ryabinin To: qemu-devel@nongnu.org Cc: Steve Sistare , yc-core@yandex-team.ru, Andrey Ryabinin , Cornelia Huck , Thomas Huth , Alex Williamson , Tony Krowiak , Halil Pasic , Jason Herne , Eric Farman , Matthew Rosato , Eric Blake , Markus Armbruster , Paolo Bonzini , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , Eduardo Habkost , Cleber Rosa , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , Beraldo Leal Subject: [PATCH 2/4] vfio: add vfio-group user createable object Date: Mon, 17 Oct 2022 13:54:05 +0300 Message-Id: <20221017105407.3858-3-arbn@yandex-team.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221017105407.3858-1-arbn@yandex-team.com> References: <20221017105407.3858-1-arbn@yandex-team.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=178.154.239.136; envelope-from=arbn@yandex-team.com; helo=forwardcorp1b.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @yandex-team.com) X-ZM-MESSAGEID: 1666005566764100001 Content-Type: text/plain; charset="utf-8" Add vfio-group type and allow user to create such object via '-object' command line argument or 'object-add' qmp command. Parameters are: - @fd - file descriptor - @container - id of vfio-container object which will be used for this VFIO group - @groupid - number representing IOMMU group (no needed if @fd and @container were provided) E.g.: -object vfio-container,id=3Dct,fd=3D5 \ -object vfio-group,id=3Dgroup,fd=3D6,container=3Dct Signed-off-by: Andrey Ryabinin --- hw/vfio/common.c | 267 +++++++++++++++++++++++----------- hw/vfio/trace-events | 2 +- include/hw/vfio/vfio-common.h | 4 + qapi/qom.json | 15 ++ 4 files changed, 205 insertions(+), 83 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 392057d3025..95722ecf96a 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1911,31 +1911,40 @@ static int vfio_init_container(VFIOContainer *conta= iner, int group_fd, Error **errp) { int iommu_type, ret; + struct vfio_group_status status =3D { .argsz =3D sizeof(status) }; =20 iommu_type =3D vfio_get_iommu_type(container, errp); if (iommu_type < 0) { return iommu_type; } =20 - ret =3D ioctl(group_fd, VFIO_GROUP_SET_CONTAINER, &container->fd); + + ret =3D ioctl(group_fd, VFIO_GROUP_GET_STATUS, &status); if (ret) { - error_setg_errno(errp, errno, "Failed to set group container"); + error_setg_errno(errp, errno, "Failed to get group status"); return -errno; } - - while (ioctl(container->fd, VFIO_SET_IOMMU, iommu_type)) { - if (iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU) { - /* - * On sPAPR, despite the IOMMU subdriver always advertises v1 = and - * v2, the running platform may not support v2 and there is no - * way to guess it until an IOMMU group gets added to the cont= ainer. - * So in case it fails with v2, try v1 as a fallback. - */ - iommu_type =3D VFIO_SPAPR_TCE_IOMMU; - continue; + if (!(status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) { + ret =3D ioctl(group_fd, VFIO_GROUP_SET_CONTAINER, &container->fd); + if (ret) { + error_setg_errno(errp, errno, "Failed to set group container"); + return -errno; + } + + while (ioctl(container->fd, VFIO_SET_IOMMU, iommu_type)) { + if (iommu_type =3D=3D VFIO_SPAPR_TCE_v2_IOMMU) { + /* + * On sPAPR, despite the IOMMU subdriver always advertises= v1 and + * v2, the running platform may not support v2 and there i= s no + * way to guess it until an IOMMU group gets added to the = container. + * So in case it fails with v2, try v1 as a fallback. + */ + iommu_type =3D VFIO_SPAPR_TCE_IOMMU; + continue; + } + error_setg_errno(errp, errno, "Failed to set iommu for contain= er"); + return -errno; } - error_setg_errno(errp, errno, "Failed to set iommu for container"); - return -errno; } =20 container->iommu_type =3D iommu_type; @@ -2050,34 +2059,44 @@ static int vfio_connect_container(VFIOGroup *group,= AddressSpace *as, * with some IOMMU types. vfio_ram_block_discard_disable() handles the * details once we know which type of IOMMU we are using. */ - - QLIST_FOREACH(container, &space->containers, next) { - if (!ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd)) { - ret =3D vfio_ram_block_discard_disable(container, true); - if (ret) { - error_setg_errno(errp, -ret, - "Cannot set discarding of RAM broken"); - if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, - &container->fd)) { - error_report("vfio: error disconnecting group %d from" - " container", group->groupid); + if (!group->container) { + QLIST_FOREACH(container, &space->containers, next) { + if (!ioctl(group->fd, VFIO_GROUP_SET_CONTAINER, &container->fd= )) { + ret =3D vfio_ram_block_discard_disable(container, true); + if (ret) { + error_setg_errno(errp, -ret, + "Cannot set discarding of RAM broken"= ); + if (ioctl(group->fd, VFIO_GROUP_UNSET_CONTAINER, + &container->fd)) { + error_report("vfio: error disconnecting group %d f= rom" + " container", group->groupid); + } + return ret; } - return ret; + group->container =3D container; + QLIST_INSERT_HEAD(&container->group_list, group, container= _next); + vfio_kvm_device_add_group(group); + return 0; } - group->container =3D container; - QLIST_INSERT_HEAD(&container->group_list, group, container_nex= t); - vfio_kvm_device_add_group(group); - return 0; } - } + container =3D VFIO_CONTAINER(object_new(TYPE_VFIO_CONTAINER)); + container->space =3D space; =20 - container =3D VFIO_CONTAINER(object_new(TYPE_VFIO_CONTAINER)); - container->space =3D space; - - user_creatable_complete(USER_CREATABLE(container), errp); - if (*errp) { - ret =3D -1; - goto free_container_exit; + user_creatable_complete(USER_CREATABLE(container), errp); + if (*errp) { + ret =3D -1; + goto free_container_exit; + } + group->container =3D container; + } else if (group->container->initialized) { + object_ref(OBJECT(group->container)); + QLIST_INSERT_HEAD(&group->container->group_list, group, container_= next); + vfio_kvm_device_add_group(group); + return 0; + } else { + container =3D group->container; + container->space =3D space; + object_ref(OBJECT(container)); } =20 ret =3D vfio_init_container(container, group->fd, errp); @@ -2228,6 +2247,10 @@ static void vfio_disconnect_container(VFIOGroup *gro= up) { VFIOContainer *container =3D group->container; =20 + if (!group->container) { + return; + } + QLIST_REMOVE(group, container_next); group->container =3D NULL; =20 @@ -2251,7 +2274,6 @@ static void vfio_disconnect_container(VFIOGroup *grou= p) VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) { VFIOGroup *group; - char path[32]; struct vfio_group_status status =3D { .argsz =3D sizeof(status) }; =20 QLIST_FOREACH(group, &vfio_group_list, next) { @@ -2267,31 +2289,14 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace= *as, Error **errp) } } =20 - group =3D g_malloc0(sizeof(*group)); - - snprintf(path, sizeof(path), "/dev/vfio/%d", groupid); - group->fd =3D qemu_open_old(path, O_RDWR); - if (group->fd < 0) { - error_setg_errno(errp, errno, "failed to open %s", path); - goto free_group_exit; - } - - if (ioctl(group->fd, VFIO_GROUP_GET_STATUS, &status)) { - error_setg_errno(errp, errno, "failed to get group %d status", gro= upid); - goto close_fd_exit; - } - - if (!(status.flags & VFIO_GROUP_FLAGS_VIABLE)) { - error_setg(errp, "group %d is not viable", groupid); - error_append_hint(errp, - "Please ensure all devices within the iommu_grou= p " - "are bound to their vfio bus driver.\n"); - goto close_fd_exit; + group =3D VFIO_GROUP(object_new(TYPE_VFIO_GROUP)); + object_property_set_int(OBJECT(group), "groupid", groupid, errp); + user_creatable_complete(USER_CREATABLE(group), errp); + if (*errp) { + object_unref(OBJECT(group)); + return NULL; } =20 - group->groupid =3D groupid; - QLIST_INIT(&group->device_list); - if (vfio_connect_container(group, as, errp)) { error_prepend(errp, "failed to setup container for group %d: ", groupid); @@ -2302,15 +2307,10 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace= *as, Error **errp) qemu_register_reset(vfio_reset_handler, NULL); } =20 - QLIST_INSERT_HEAD(&vfio_group_list, group, next); - return group; =20 close_fd_exit: - close(group->fd); - -free_group_exit: - g_free(group); + object_unref(OBJECT(group)); =20 return NULL; } @@ -2321,19 +2321,7 @@ void vfio_put_group(VFIOGroup *group) return; } =20 - if (!group->ram_block_discard_allowed) { - vfio_ram_block_discard_disable(group->container, false); - } - vfio_kvm_device_del_group(group); - vfio_disconnect_container(group); - QLIST_REMOVE(group, next); - trace_vfio_put_group(group->fd); - close(group->fd); - g_free(group); - - if (QLIST_EMPTY(&vfio_group_list)) { - qemu_unregister_reset(vfio_reset_handler, NULL); - } + object_unref(OBJECT(group)); } =20 int vfio_get_device(VFIOGroup *group, const char *name, @@ -2676,8 +2664,123 @@ static const TypeInfo vfio_container_info =3D { }, }; =20 +static void vfio_group_set_fd(Object *obj, const char *value, + Error **errp) +{ + VFIOGroup *group =3D VFIO_GROUP(obj); + + group->fd =3D monitor_fd_param(monitor_cur(), value, errp); +} + +static void vfio_group_set_groupid(Object *obj, Visitor *v, + const char *name, void *opaque, + Error **errp) +{ + VFIOGroup *group =3D VFIO_GROUP(obj); + Error *error =3D NULL; + uint32_t groupid; + + visit_type_uint32(v, name, &groupid, &error); + if (error) { + error_propagate(errp, error); + return; + } + + group->groupid =3D groupid; +} + +static void vfio_group_complete(UserCreatable *uc, Error **errp) +{ + VFIOGroup *group =3D VFIO_GROUP(uc); + struct vfio_group_status status =3D { .argsz =3D sizeof(status) }; + + if (group->fd < 0 && group->groupid >=3D 0) { + char path[32]; + + snprintf(path, sizeof(path), "/dev/vfio/%d", group->groupid); + + group->fd =3D qemu_open_old(path, O_RDWR); + if (group->fd < 0) { + error_setg_errno(errp, errno, "failed to open %s", path); + return; + } + } + + if (ioctl(group->fd, VFIO_GROUP_GET_STATUS, &status)) { + error_setg_errno(errp, errno, "failed to get group %d status", gro= up->groupid); + return; + } + + if (!(status.flags & VFIO_GROUP_FLAGS_VIABLE)) { + error_setg(errp, "group %d is not viable", group->groupid); + error_append_hint(errp, + "Please ensure all devices within the iommu_grou= p " + "are bound to their vfio bus driver.\n"); + } +} + +static void vfio_group_class_init(ObjectClass *class, void *data) +{ + UserCreatableClass *ucc =3D USER_CREATABLE_CLASS(class); + ucc->complete =3D vfio_group_complete; + + object_class_property_add_link(class, "container", TYPE_VFIO_CONTAINER, + offsetof(VFIOGroup, container), + object_property_allow_set_link, 0); + object_class_property_add_str(class, "fd", NULL, vfio_group_set_fd); + object_class_property_add(class, "groupid", "int", NULL, + vfio_group_set_groupid, + NULL, NULL); +} + +static void vfio_group_instance_init(Object *obj) +{ + VFIOGroup *group =3D VFIO_GROUP(obj); + + QLIST_INIT(&group->device_list); + group->fd =3D -1; + group->groupid =3D -1; + QLIST_INSERT_HEAD(&vfio_group_list, group, next); +} + +static void +vfio_group_instance_finalize(Object *obj) +{ + VFIOGroup *group =3D VFIO_GROUP(obj); + + if (!group->ram_block_discard_allowed) { + vfio_ram_block_discard_disable(group->container, false); + } + + vfio_kvm_device_del_group(group); + vfio_disconnect_container(group); + QLIST_REMOVE(group, next); + trace_vfio_group_instance_finalize(group->fd); + if (group->fd >=3D 0) { + close(group->fd); + } + + if (QLIST_EMPTY(&vfio_group_list)) { + qemu_unregister_reset(vfio_reset_handler, NULL); + } +} + +static const TypeInfo vfio_group_info =3D { + .name =3D TYPE_VFIO_GROUP, + .parent =3D TYPE_OBJECT, + .class_init =3D vfio_group_class_init, + .instance_size =3D sizeof(VFIOGroup), + .instance_init =3D vfio_group_instance_init, + .instance_finalize =3D vfio_group_instance_finalize, + .interfaces =3D (InterfaceInfo[]) { + {TYPE_USER_CREATABLE}, + {} + }, +}; + static void register_vfio_types(void) { type_register_static(&vfio_container_info); + type_register_static(&vfio_group_info); } type_init(register_vfio_types) diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 8b79cf33a33..6ae0ed09acd 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -105,7 +105,7 @@ vfio_listener_region_add_no_dma_map(const char *name, u= int64_t iova, uint64_t si vfio_listener_region_del_skip(uint64_t start, uint64_t end) "SKIPPING regi= on_del 0x%"PRIx64" - 0x%"PRIx64 vfio_listener_region_del(uint64_t start, uint64_t end) "region_del 0x%"PRI= x64" - 0x%"PRIx64 vfio_container_instance_finalize(int fd) "close container->fd=3D%d" -vfio_put_group(int fd) "close group->fd=3D%d" +vfio_group_instance_finalize(int fd) "close group->fd=3D%d" vfio_get_device(const char * name, unsigned int flags, unsigned int num_re= gions, unsigned int num_irqs) "Device %s flags: %u, regions: %u, irqs: %u" vfio_put_base_device(int fd) "close vdev->fd=3D%d" vfio_region_setup(const char *dev, int index, const char *name, unsigned l= ong flags, unsigned long offset, unsigned long size) "Device %s, region %d = \"%s\", flags: 0x%lx, offset: 0x%lx, size: 0x%lx" diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 0ab99060e44..f2d67093f44 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -156,6 +156,7 @@ struct VFIODeviceOps { }; =20 typedef struct VFIOGroup { + Object parent; int fd; int groupid; VFIOContainer *container; @@ -194,6 +195,9 @@ typedef struct VFIODisplay { #define TYPE_VFIO_CONTAINER "vfio-container" OBJECT_DECLARE_SIMPLE_TYPE(VFIOContainer, VFIO_CONTAINER) =20 +#define TYPE_VFIO_GROUP "vfio-group" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOGroup, VFIO_GROUP) + void vfio_put_base_device(VFIODevice *vbasedev); void vfio_disable_irqindex(VFIODevice *vbasedev, int index); void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index); diff --git a/qapi/qom.json b/qapi/qom.json index d1a88e10b52..f46dd6b8034 100644 --- a/qapi/qom.json +++ b/qapi/qom.json @@ -746,6 +746,19 @@ { 'struct': 'VFIOContainerProperties', 'data': { 'fd': 'str' } } =20 +## +# @VFIOGroupProperties: +# +# Properties for vfio-group objects. +# +# @fd: file descriptor of vfio group +# @container: container +# +# Since: 7.2 +## +{ 'struct': 'VFIOGroupProperties', + 'data': { 'fd': 'str', 'container': 'str'} } + ## # @VfioUserServerProperties: # @@ -901,6 +914,7 @@ 'tls-creds-x509', 'tls-cipher-suites', 'vfio-container', + 'vfio-group', { 'name': 'x-remote-object', 'features': [ 'unstable' ] }, { 'name': 'x-vfio-user-server', 'features': [ 'unstable' ] } ] } @@ -967,6 +981,7 @@ 'tls-creds-x509': 'TlsCredsX509Properties', 'tls-cipher-suites': 'TlsCredsProperties', 'vfio-container': 'VFIOContainerProperties', + 'vfio-group': 'VFIOGroupProperties', 'x-remote-object': 'RemoteObjectProperties', 'x-vfio-user-server': 'VfioUserServerProperties' } } --=20 2.37.3 From nobody Mon Feb 9 11:56:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=yandex-team.com ARC-Seal: i=1; a=rsa-sha256; t=1666005171; cv=none; d=zohomail.com; s=zohoarc; b=KFF6ruODd1bwWBOTgbh7J5pzbUCF/ezPV+zIJAUE6UvD7TmoFd0/ap+DgktKbEwA1siC0WZLHRaGtvnGsw7ERuGfr+Dx+K6tKPoEdkXwNUzyx1IU7P+A01nhWtDca+kFkh6avO0Zv6Ry9c7hV2iSQLN8/LZj0IBnc127FWkTsaA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1666005171; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ecLNYysUvaXK/QBAT76JIStY71ytLFpBTYT85le8ZPo=; b=JKZkpsi4TZs5M/oBRRxZbSXEt8GPaqEMXPKt4T+QB+NurQi+SGyCsPwXXgGcuHiptk0SENTiX28LFI2PZvEv3vYf8BUW6r+LWqLW9kHQY/w0K2U5wMnin6+cZayzurrvkkiMYNYiSu6pozfrZmj0pGBDMT/Qua8IkxUYJ4FLJ8I= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1666005170671463.9071957643454; Mon, 17 Oct 2022 04:12:50 -0700 (PDT) Received: from localhost ([::1]:56322 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1okO37-0007r1-L9 for importer@patchew.org; Mon, 17 Oct 2022 07:12:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33288) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okNvy-0001Rw-Tu for qemu-devel@nongnu.org; Mon, 17 Oct 2022 07:05:27 -0400 Received: from forwardcorp1b.mail.yandex.net ([178.154.239.136]:46354) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okNvn-0001Fp-Et for qemu-devel@nongnu.org; Mon, 17 Oct 2022 07:05:18 -0400 Received: from sas1-7470331623bb.qloud-c.yandex.net (sas1-7470331623bb.qloud-c.yandex.net [IPv6:2a02:6b8:c08:bd1e:0:640:7470:3316]) by forwardcorp1b.mail.yandex.net (Yandex) with ESMTP id A89FA60EBB; Mon, 17 Oct 2022 13:54:28 +0300 (MSK) Received: from dellarbn.yandex.net (unknown [2a02:6b8:0:107:3e85:844d:5b1d:60a]) by sas1-7470331623bb.qloud-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id PrI3KDUSQF-sR7KTbLI; Mon, 17 Oct 2022 13:54:27 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) Precedence: bulk X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.com; s=default; t=1666004067; bh=ecLNYysUvaXK/QBAT76JIStY71ytLFpBTYT85le8ZPo=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=KOVb2sBaA07x55fMBAZBYlPf1jZOHkoYhpwGrjLyYol5NGYvoLYW44HUQccDaektU eoXqVhYRvqSKspvjBoq3q8O4BDia4Nx0KJbw+EwtNaYtN/y7t2KW4gU+hJLHo5CBtB j38sye8/tlyvSBjDsKAy7ya2M+q8ykMb8CkFRAa4= Authentication-Results: sas1-7470331623bb.qloud-c.yandex.net; dkim=pass header.i=@yandex-team.com From: Andrey Ryabinin To: qemu-devel@nongnu.org Cc: Steve Sistare , yc-core@yandex-team.ru, Andrey Ryabinin , Cornelia Huck , Thomas Huth , Tony Krowiak , Halil Pasic , Jason Herne , Alex Williamson , Eric Farman , Matthew Rosato , Paolo Bonzini , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , Eduardo Habkost , Eric Blake , Markus Armbruster , Cleber Rosa , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , Beraldo Leal Subject: [PATCH 3/4] vfio: Add 'group' property to 'vfio-pci' device Date: Mon, 17 Oct 2022 13:54:06 +0300 Message-Id: <20221017105407.3858-4-arbn@yandex-team.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221017105407.3858-1-arbn@yandex-team.com> References: <20221017105407.3858-1-arbn@yandex-team.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=178.154.239.136; envelope-from=arbn@yandex-team.com; helo=forwardcorp1b.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @yandex-team.com) X-ZM-MESSAGEID: 1666005172849100001 Content-Type: text/plain; charset="utf-8" Add 'group' properties to 'vfio-pci' device. This allows to add vfio-pci device using precreated vfio container and group, e.g. -object vfio-container,id=3Dct,fd=3D5 \ -object vfio-group,id=3Dgrp,fd=3D6,container=3Dct \ -device vfio-pci,host=3D05:00.0,group=3Dgrp Signed-off-by: Andrey Ryabinin --- hw/vfio/ap.c | 2 +- hw/vfio/ccw.c | 2 +- hw/vfio/common.c | 45 ++++++++++++++++++++--------------- hw/vfio/pci.c | 10 +++++++- hw/vfio/platform.c | 2 +- include/hw/vfio/vfio-common.h | 2 +- 6 files changed, 39 insertions(+), 24 deletions(-) diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c index e0dd561e85a..2a5c322883b 100644 --- a/hw/vfio/ap.c +++ b/hw/vfio/ap.c @@ -81,7 +81,7 @@ static VFIOGroup *vfio_ap_get_group(VFIOAPDevice *vapdev,= Error **errp) =20 g_free(group_path); =20 - return vfio_get_group(groupid, &address_space_memory, errp); + return vfio_get_group(&vapdev->vdev, groupid, &address_space_memory, e= rrp); } =20 static void vfio_ap_realize(DeviceState *dev, Error **errp) diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c index 0354737666a..500f0f62163 100644 --- a/hw/vfio/ccw.c +++ b/hw/vfio/ccw.c @@ -650,7 +650,7 @@ static VFIOGroup *vfio_ccw_get_group(S390CCWDevice *cde= v, Error **errp) return NULL; } =20 - return vfio_get_group(groupid, &address_space_memory, errp); + return vfio_get_group(NULL, groupid, &address_space_memory, errp); } =20 static void vfio_ccw_realize(DeviceState *dev, Error **errp) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 95722ecf96a..64ace47822d 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -2271,30 +2271,35 @@ static void vfio_disconnect_container(VFIOGroup *gr= oup) object_unref(OBJECT(container)); } =20 -VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) +VFIOGroup *vfio_get_group(VFIODevice *vdev, int groupid, AddressSpace *as,= Error **errp) { VFIOGroup *group; struct vfio_group_status status =3D { .argsz =3D sizeof(status) }; =20 - QLIST_FOREACH(group, &vfio_group_list, next) { - if (group->groupid =3D=3D groupid) { - /* Found it. Now is it already in the right context? */ - if (group->container->space->as =3D=3D as) { - return group; - } else { - error_setg(errp, "group %d used in multiple address spaces= ", - group->groupid); - return NULL; + if (!vdev->group) { + QLIST_FOREACH(group, &vfio_group_list, next) { + if (group->groupid =3D=3D groupid) { + /* Found it. Now is it already in the right context? */ + if (group->container->space->as =3D=3D as) { + return group; + } else { + error_setg(errp, "group %d used in multiple address sp= aces", + group->groupid); + return NULL; + } } } - } - - group =3D VFIO_GROUP(object_new(TYPE_VFIO_GROUP)); - object_property_set_int(OBJECT(group), "groupid", groupid, errp); - user_creatable_complete(USER_CREATABLE(group), errp); - if (*errp) { - object_unref(OBJECT(group)); - return NULL; + group =3D VFIO_GROUP(object_new(TYPE_VFIO_GROUP)); + object_property_set_int(OBJECT(group), "groupid", groupid, errp); + user_creatable_complete(USER_CREATABLE(group), errp); + if (*errp) { + object_unref(OBJECT(group)); + return NULL; + } + } else { + group =3D vdev->group; + group->groupid =3D groupid; + object_ref(OBJECT(group)); } =20 if (vfio_connect_container(group, as, errp)) { @@ -2388,7 +2393,9 @@ void vfio_put_base_device(VFIODevice *vbasedev) if (!vbasedev->group) { return; } - QLIST_REMOVE(vbasedev, next); + if (!QLIST_EMPTY(&vbasedev->group->device_list)) { + QLIST_REMOVE(vbasedev, next); + } vbasedev->group =3D NULL; trace_vfio_put_base_device(vbasedev->fd); close(vbasedev->fd); diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 939dcc3d4a9..d671eb74881 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -29,6 +29,7 @@ #include "hw/qdev-properties.h" #include "hw/qdev-properties-system.h" #include "migration/vmstate.h" +#include "monitor/monitor.h" #include "qapi/qmp/qdict.h" #include "qemu/error-report.h" #include "qemu/main-loop.h" @@ -2902,7 +2903,8 @@ static void vfio_realize(PCIDevice *pdev, Error **err= p) =20 trace_vfio_realize(vbasedev->name, groupid); =20 - group =3D vfio_get_group(groupid, pci_device_iommu_address_space(pdev)= , errp); + group =3D vfio_get_group(&vdev->vbasedev, groupid, + pci_device_iommu_address_space(pdev), errp); if (!group) { goto error; } @@ -3190,6 +3192,7 @@ static void vfio_instance_finalize(Object *obj) static void vfio_exitfn(PCIDevice *pdev) { VFIOPCIDevice *vdev =3D VFIO_PCI(pdev); + VFIOGroup *group =3D vdev->vbasedev.group; =20 vfio_unregister_req_notifier(vdev); vfio_unregister_err_notifier(vdev); @@ -3204,6 +3207,8 @@ static void vfio_exitfn(PCIDevice *pdev) vfio_teardown_msi(vdev); vfio_bars_exit(vdev); vfio_migration_finalize(&vdev->vbasedev); + + vfio_put_group(group); } =20 static void vfio_pci_reset(DeviceState *dev) @@ -3330,6 +3335,9 @@ static void vfio_pci_dev_class_init(ObjectClass *klas= s, void *data) pdc->exit =3D vfio_exitfn; pdc->config_read =3D vfio_pci_read_config; pdc->config_write =3D vfio_pci_write_config; + object_class_property_add_link(klass, "group", TYPE_VFIO_GROUP, + offsetof(VFIOPCIDevice, vbasedev.group), + object_property_allow_set_link, 0); } =20 static const TypeInfo vfio_pci_dev_info =3D { diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c index 5af73f92876..3a72e085026 100644 --- a/hw/vfio/platform.c +++ b/hw/vfio/platform.c @@ -577,7 +577,7 @@ static int vfio_base_device_init(VFIODevice *vbasedev, = Error **errp) =20 trace_vfio_platform_base_device_init(vbasedev->name, groupid); =20 - group =3D vfio_get_group(groupid, &address_space_memory, errp); + group =3D vfio_get_group(vbasedev, groupid, &address_space_memory, err= p); if (!group) { return -ENOENT; } diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index f2d67093f44..79b43eb76b3 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -216,7 +216,7 @@ void vfio_region_unmap(VFIORegion *region); void vfio_region_exit(VFIORegion *region); void vfio_region_finalize(VFIORegion *region); void vfio_reset_handler(void *opaque); -VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp); +VFIOGroup *vfio_get_group(VFIODevice *vdev, int groupid, AddressSpace *as,= Error **errp); void vfio_put_group(VFIOGroup *group); int vfio_get_device(VFIOGroup *group, const char *name, VFIODevice *vbasedev, Error **errp); --=20 2.37.3 From nobody Mon Feb 9 11:56:17 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=yandex-team.com ARC-Seal: i=1; a=rsa-sha256; t=1666005180; cv=none; d=zohomail.com; s=zohoarc; b=Pzeie74i112KWg+kCxN4gbGf4ioMu/o9owtTEgEH9oBbXvvuhTeLxx7Gk/nnelg4EidatnWhyseTh7GtJPU5yAtKHpbB8lXlYdWWrXAMYwtZRx5pBI2ZyQlgw872P/DlcfPK0jL7DE5ebZ52xb0nctwn+sftFaAKUgtUirMEe/c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1666005180; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=43QDqg/hxytt/fGiR7Qd1qh7ZB1ECiEMYfrGwAdxma4=; b=Ctn1ErahnvAD+MXHs6AAYzXAU+wgGYDMy8+imtkb4cYydkiaZVpHbWuzV5LJXMVSuqsyKoWNQquVK4zMLBjIACOu/tTh1vqRhgnf+dPmrJBHT+dhoDUaordtJMsbdaMRU+YMiDguSIT1Bc5GXz9CcV386PZjAexCek+Tqw24MyM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1666005180689378.19596965129836; Mon, 17 Oct 2022 04:13:00 -0700 (PDT) Received: from localhost ([::1]:51260 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1okO3H-0008KC-FH for importer@patchew.org; Mon, 17 Oct 2022 07:12:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33296) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okNvy-0001S0-TY for qemu-devel@nongnu.org; Mon, 17 Oct 2022 07:05:27 -0400 Received: from forwardcorp1b.mail.yandex.net ([178.154.239.136]:54278) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1okNvq-0001IH-G2 for qemu-devel@nongnu.org; Mon, 17 Oct 2022 07:05:24 -0400 Received: from sas1-7470331623bb.qloud-c.yandex.net (sas1-7470331623bb.qloud-c.yandex.net [IPv6:2a02:6b8:c08:bd1e:0:640:7470:3316]) by forwardcorp1b.mail.yandex.net (Yandex) with ESMTP id 785FE60E7D; Mon, 17 Oct 2022 13:54:30 +0300 (MSK) Received: from dellarbn.yandex.net (unknown [2a02:6b8:0:107:3e85:844d:5b1d:60a]) by sas1-7470331623bb.qloud-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id PrI3KDUSQF-sS7Kd5R0; Mon, 17 Oct 2022 13:54:29 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) Precedence: bulk X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.com; s=default; t=1666004069; bh=43QDqg/hxytt/fGiR7Qd1qh7ZB1ECiEMYfrGwAdxma4=; h=Message-Id:Date:In-Reply-To:Cc:Subject:References:To:From; b=tY2RPaCzypziMhqW5U2ov/+u4r2r6x7E14xvhmp//DWc3nfcoPREAlNPHTVVx+x0+ aRWUYdn/gwoX4E2QpROE0t9XXLurBAyY2pwKKA3diNL7umIUenyDgwtlubzfY2+ABy aR2S8SwCMVekaobHt1vMcQKkjLaicwvDWugvWcQQ= Authentication-Results: sas1-7470331623bb.qloud-c.yandex.net; dkim=pass header.i=@yandex-team.com From: Andrey Ryabinin To: qemu-devel@nongnu.org Cc: Steve Sistare , yc-core@yandex-team.ru, Andrey Ryabinin , Tony Krowiak , Halil Pasic , Jason Herne , Alex Williamson , Cornelia Huck , Thomas Huth , Eric Farman , Matthew Rosato , Eric Blake , Markus Armbruster , Paolo Bonzini , =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= , Eduardo Habkost , Cleber Rosa , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Wainer dos Santos Moschetta , Beraldo Leal Subject: [PATCH 4/4] tests/avocado/vfio: add test for vfio devices Date: Mon, 17 Oct 2022 13:54:07 +0300 Message-Id: <20221017105407.3858-5-arbn@yandex-team.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221017105407.3858-1-arbn@yandex-team.com> References: <20221017105407.3858-1-arbn@yandex-team.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=178.154.239.136; envelope-from=arbn@yandex-team.com; helo=forwardcorp1b.mail.yandex.net X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @yandex-team.com) X-ZM-MESSAGEID: 1666005181866100004 Content-Type: text/plain; charset="utf-8" Add avocado test for vfio subsystem. The test uses pci-tesdev as a source PCI device for vfio. So the test supposed to be launched inside QEMU guest launched with '-device pci-testdev' devices. The test creates VFIO container&group, launches QEMU and passes container/group/device to it. Signed-off-by: Andrey Ryabinin --- tests/avocado/vfio.py | 156 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 tests/avocado/vfio.py diff --git a/tests/avocado/vfio.py b/tests/avocado/vfio.py new file mode 100644 index 00000000000..fa0d2b65dc7 --- /dev/null +++ b/tests/avocado/vfio.py @@ -0,0 +1,156 @@ +# tests for QEMU's vfio subsystem +# +# Copyright (c) 2022 Yandex N.V. +# +# This work is licensed under the terms of the GNU GPL, version 2 or later. +# See the COPYING file in the top-level directory. + +from avocado.utils import wait +from avocado import skipUnless +from avocado_qemu import QemuSystemTest +from avocado_qemu import run_cmd +import os +import sys +import subprocess +from fcntl import ioctl +from ctypes import * +import struct + + +VFIO_CMD_PREFIX =3D ord(';') << (4*2) +VFIO_GET_API_VERSION =3D VFIO_CMD_PREFIX | 100 +VFIO_CHECK_EXTENSION =3D VFIO_CMD_PREFIX | 101 +VFIO_SET_IOMMU =3D VFIO_CMD_PREFIX | 102 +VFIO_GROUP_GET_STATUS =3D VFIO_CMD_PREFIX | 103 +VFIO_GROUP_SET_CONTAINER =3D VFIO_CMD_PREFIX | 104 +VFIO_GROUP_GET_DEVICE_FD =3D VFIO_CMD_PREFIX | 106 + +VFIO_TYPE1_IOMMU =3D 1 +VFIO_SPAPR_TCE_IOMMU =3D 2 +VFIO_TYPE1v2_IOMMU =3D 3 +VFIO_SPAPR_TCE_v2_IOMMU =3D 7 + +VFIO_API_VERSION =3D 0 +VFIO_TYPE1_IOMMU =3D 1 +PCI_VENDOR_ID=3D0x1b36 +PCI_DEV_ID=3D0x0005 + +class vfio_group_status(Structure): + _fields_ =3D [("argsz", c_uint32), + ("flags", c_uint32)] + +class vfio_container(object): + def open(self): + self.container_fd =3D os.open("/dev/vfio/vfio", os.O_RDWR) + if ioctl(self.container_fd, VFIO_GET_API_VERSION) !=3D VFIO_API_VE= RSION: + raise Exception("VFIO_GET_API_VERSION: unexpected vfio api ver= sion") + iommu_types =3D [ VFIO_TYPE1v2_IOMMU, VFIO_TYPE1_IOMMU, + VFIO_SPAPR_TCE_v2_IOMMU, VFIO_SPAPR_TCE_IOMMU ]; + for iommu_type in iommu_types: + if ioctl(self.container_fd, VFIO_CHECK_EXTENSION, iommu_type): + self.iommu_type =3D iommu_type + if not self.iommu_type: + raise Exception("No available IOMMU models"); + + def set_iommu(self): + ioctl(self.container_fd, VFIO_SET_IOMMU, self.iommu_type); + +class vfio_group(object): + def __init__(self, container, group_num): + self.ct =3D container + self.group_num =3D group_num + def open(self): + self.group_fd =3D os.open("/dev/vfio/%d" % self.group_num, os.O_RD= WR) + status =3D vfio_group_status(0, 0) + ioctl(self.group_fd, VFIO_GROUP_GET_STATUS, pointer(status)) + ioctl(self.group_fd, VFIO_GROUP_SET_CONTAINER, + c_int(self.ct.container_fd)); + +class vfio_device(object): + def __init__(self, dev, group): + self.dev =3D dev + self.group =3D group + +def pci_testdev_exists(): + for dev in next(os.walk('/sys/bus/pci/devices'))[1]: + with open("/sys/bus/pci/devices/%s/vendor" % dev) as f: + if f.read() !=3D '0x%x\n' % PCI_VENDOR_ID: + continue + with open("/sys/bus/pci/devices/%s/device" % dev) as f: + if f.read() !=3D '0x%.4x\n' % PCI_DEV_ID: + continue + return True + return False + +class VFIOIOMMUTest(QemuSystemTest): + devices =3D [] + groups =3D [] + timeout =3D 900 + ct =3D vfio_container() + + def add_group(self, dev): + tmp =3D os.readlink("/sys/bus/pci/devices/%s/iommu_group" % dev) + group_num =3D int(tmp.split('/')[-1]) + + for g in self.groups: + if g.group_num =3D=3D group_num: + return g + group =3D vfio_group(self.ct, group_num) + self.groups.append(group) + return group + + def setUp(self): + super().setUp() + run_cmd(('modprobe', 'vfio-pci')) + try: + f =3D open("/sys/bus/pci/drivers/vfio-pci/new_id", "a") + f.write("%x %x" % (PCI_VENDOR_ID, PCI_DEV_ID)) + except PermissionError: + pass + except FileExistsError: + pass + for dev in next(os.walk('/sys/bus/pci/devices'))[1]: + with open("/sys/bus/pci/devices/%s/vendor" % dev) as f: + if f.read() !=3D '0x%x\n' % PCI_VENDOR_ID: + continue + with open("/sys/bus/pci/devices/%s/device" % dev) as f: + if f.read() !=3D '0x%.4x\n' % PCI_DEV_ID: + continue + + self.devices.append(vfio_device(dev, self.add_group(dev))) + + def open_dev_fds(self): + self.ct.open() + for group in self.groups: + group.open() + self.ct.set_iommu() + + def close_fds(self): + for group in self.groups: + os.close(group.group_fd) + os.close(self.ct.container_fd) + + def hotplug_devices(self, vm): + vm._qmp.send_fd_scm(self.devices[0].group.ct.container_fd) + vm.command("getfd", fdname=3D"ct") + vm.command("object-add", qom_type=3D"vfio-container", id=3D"ct", f= d=3D"ct") + + for group in self.groups: + vm._qmp.send_fd_scm(group.group_fd) + vm.command("getfd", fdname=3D"group_fd") + vm.command("object-add", qom_type=3D"vfio-group", + id=3D"group%d" % group.group_num, + fd=3D"group_fd", container=3D"ct") + + for i in range(len(self.devices)): + vm.command("device_add", driver=3D"vfio-pci", + host=3Dself.devices[i].dev, id=3D"dev%d" % i, + group=3D"group%d" % self.devices[i].group.group_num) + + @skipUnless(pci_testdev_exists(), "no pci-testdev found") + def test_vfio(self): + self.open_dev_fds() + self.vm.add_args('-nodefaults') + self.vm.launch() + self.hotplug_devices(self.vm) + self.close_fds() --=20 2.37.3