From nobody Sat Nov 15 16:36:06 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1750924260; cv=none; d=zohomail.com; s=zohoarc; b=Yoda8/Ab59rxoEz7sNE0MgIed82XwaAr17FbVkQT2MMD7LTPJerXrciRbVfmspCw7OYSJGcZcO7bLMKqgvsIksXsDpcatanGOBYkDO2akIgtI7ziQoZSikmJETEkbLeIjqZhLcxU3efWaIcPBST1092f77dUCwqIu8sDKey0oFw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1750924260; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=4njr8PwYAhNczIr5/nlvFNXwpt2ofdxrbzvEMNzQSNw=; b=kG0vIMnhplDHVsxcq8WP3ht96p29AcWBwtsmb0yRHyFjShkc1ODN9H+cIMIY33/oo6arKbhvi4L7PIMDoD2ON/DilzAkkm/NCjgiXtQNxNMeKiYDBNJC3iFhu5M8JvNjsVm90g2IF/s1l1gQUJLf6Nk7niQsy57EzDE0gr0VGUY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1750924260456619.7648458650352; Thu, 26 Jun 2025 00:51:00 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uUhJI-0004VJ-OF; Thu, 26 Jun 2025 03:46:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uUhJ6-0004QG-9z for qemu-devel@nongnu.org; Thu, 26 Jun 2025 03:46:08 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uUhJ3-0001wu-Sh for qemu-devel@nongnu.org; Thu, 26 Jun 2025 03:46:04 -0400 Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-193-3eXKpNt4OtG71EttOO_R6g-1; Thu, 26 Jun 2025 03:45:57 -0400 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B316219560AE; Thu, 26 Jun 2025 07:45:55 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.32.51]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 4EFF8180035C; Thu, 26 Jun 2025 07:45:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1750923959; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4njr8PwYAhNczIr5/nlvFNXwpt2ofdxrbzvEMNzQSNw=; b=UJuP5RPGTU+Dwc78tkX8YkTIgicQrYcjs9VQT/vQcenIDtPZpHTAwy3YaI0L7QFASQfbPz dCb56jHdQVurpM3IwkQDkRTflBfi+sgIzDGrvXtT5syJiTimQveD8Em8s8wAYWxXWsQ7pV +ZLinRqafWKAhuvPt1HP725KE62Aucw= X-MC-Unique: 3eXKpNt4OtG71EttOO_R6g-1 X-Mimecast-MFC-AGG-ID: 3eXKpNt4OtG71EttOO_R6g_1750923955 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , John Levon , John Johnson , Elena Ufimtseva , Jagannathan Raman , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 08/25] vfio-user: connect vfio proxy to remote server Date: Thu, 26 Jun 2025 09:45:12 +0200 Message-ID: <20250626074529.1384114-9-clg@redhat.com> In-Reply-To: <20250626074529.1384114-1-clg@redhat.com> References: <20250626074529.1384114-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1750924262950116600 From: John Levon Introduce the vfio-user "proxy": this is the client code responsible for sending and receiving vfio-user messages across the control socket. The new files hw/vfio-user/proxy.[ch] contain some basic plumbing for managing the proxy; initialize the proxy during realization of the VFIOUserPCIDevice instance. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/20250625193012.2316242-3-john.levo= n@nutanix.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio-user/proxy.h | 79 +++++++++++++++++ include/hw/vfio/vfio-device.h | 2 + hw/vfio-user/pci.c | 22 +++++ hw/vfio-user/proxy.c | 162 ++++++++++++++++++++++++++++++++++ hw/vfio-user/meson.build | 1 + 5 files changed, 266 insertions(+) create mode 100644 hw/vfio-user/proxy.h create mode 100644 hw/vfio-user/proxy.c diff --git a/hw/vfio-user/proxy.h b/hw/vfio-user/proxy.h new file mode 100644 index 0000000000000000000000000000000000000000..a9bce82239d0e9dc40a892e1542= 488cfffc4f3da --- /dev/null +++ b/hw/vfio-user/proxy.h @@ -0,0 +1,79 @@ +#ifndef VFIO_USER_PROXY_H +#define VFIO_USER_PROXY_H + +/* + * vfio protocol over a UNIX socket. + * + * Copyright =C2=A9 2018, 2021 Oracle and/or its affiliates. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "io/channel.h" +#include "io/channel-socket.h" + +typedef struct { + int send_fds; + int recv_fds; + int *fds; +} VFIOUserFDs; + +enum msg_type { + VFIO_MSG_NONE, + VFIO_MSG_ASYNC, + VFIO_MSG_WAIT, + VFIO_MSG_NOWAIT, + VFIO_MSG_REQ, +}; + +typedef struct VFIOUserMsg { + QTAILQ_ENTRY(VFIOUserMsg) next; + VFIOUserFDs *fds; + uint32_t rsize; + uint32_t id; + QemuCond cv; + bool complete; + enum msg_type type; +} VFIOUserMsg; + + +enum proxy_state { + VFIO_PROXY_CONNECTED =3D 1, + VFIO_PROXY_ERROR =3D 2, + VFIO_PROXY_CLOSING =3D 3, + VFIO_PROXY_CLOSED =3D 4, +}; + +typedef QTAILQ_HEAD(VFIOUserMsgQ, VFIOUserMsg) VFIOUserMsgQ; + +typedef struct VFIOUserProxy { + QLIST_ENTRY(VFIOUserProxy) next; + char *sockname; + struct QIOChannel *ioc; + void (*request)(void *opaque, VFIOUserMsg *msg); + void *req_arg; + int flags; + QemuCond close_cv; + AioContext *ctx; + QEMUBH *req_bh; + + /* + * above only changed when BQL is held + * below are protected by per-proxy lock + */ + QemuMutex lock; + VFIOUserMsgQ free; + VFIOUserMsgQ pending; + VFIOUserMsgQ incoming; + VFIOUserMsgQ outgoing; + VFIOUserMsg *last_nowait; + enum proxy_state state; +} VFIOUserProxy; + +/* VFIOProxy flags */ +#define VFIO_PROXY_CLIENT 0x1 + +VFIOUserProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); +void vfio_user_disconnect(VFIOUserProxy *proxy); + +#endif /* VFIO_USER_PROXY_H */ diff --git a/include/hw/vfio/vfio-device.h b/include/hw/vfio/vfio-device.h index 959e458d7f7e9facd8216fbd3695d9d97dbc667a..c616652ee72265c637cb5136fdf= fb4444639e0b7 100644 --- a/include/hw/vfio/vfio-device.h +++ b/include/hw/vfio/vfio-device.h @@ -47,6 +47,7 @@ typedef struct VFIOMigration VFIOMigration; =20 typedef struct IOMMUFDBackend IOMMUFDBackend; typedef struct VFIOIOASHwpt VFIOIOASHwpt; +typedef struct VFIOUserProxy VFIOUserProxy; =20 typedef struct VFIODevice { QLIST_ENTRY(VFIODevice) next; @@ -88,6 +89,7 @@ typedef struct VFIODevice { struct vfio_region_info **reginfo; int *region_fds; VFIODeviceCPR cpr; + VFIOUserProxy *proxy; } VFIODevice; =20 struct VFIODeviceOps { diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index 86d705574736586a893feb529a057e799ae610c8..642421e7916aad01a894ee390c4= 3453dd5297356 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -12,6 +12,7 @@ =20 #include "hw/qdev-properties.h" #include "hw/vfio/pci.h" +#include "hw/vfio-user/proxy.h" =20 #define TYPE_VFIO_USER_PCI "vfio-user-pci" OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) @@ -54,6 +55,8 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error = **errp) VFIODevice *vbasedev =3D &vdev->vbasedev; const char *sock_name; AddressSpace *as; + SocketAddress addr; + VFIOUserProxy *proxy; =20 if (!udev->socket) { error_setg(errp, "No socket specified"); @@ -69,6 +72,15 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error= **errp) =20 vbasedev->name =3D g_strdup_printf("vfio-user:%s", sock_name); =20 + memset(&addr, 0, sizeof(addr)); + addr.type =3D SOCKET_ADDRESS_TYPE_UNIX; + addr.u.q_unix.path =3D (char *)sock_name; + proxy =3D vfio_user_connect_dev(&addr, errp); + if (!proxy) { + return; + } + vbasedev->proxy =3D proxy; + /* * vfio-user devices are effectively mdevs (don't use a host iommu). */ @@ -112,8 +124,13 @@ static void vfio_user_instance_init(Object *obj) static void vfio_user_instance_finalize(Object *obj) { VFIOPCIDevice *vdev =3D VFIO_PCI_BASE(obj); + VFIODevice *vbasedev =3D &vdev->vbasedev; =20 vfio_pci_put_device(vdev); + + if (vbasedev->proxy !=3D NULL) { + vfio_user_disconnect(vbasedev->proxy); + } } =20 static const Property vfio_user_pci_dev_properties[] =3D { @@ -133,6 +150,11 @@ static void vfio_user_pci_set_socket(Object *obj, Visi= tor *v, const char *name, VFIOUserPCIDevice *udev =3D VFIO_USER_PCI(obj); bool success; =20 + if (udev->device.vbasedev.proxy) { + error_setg(errp, "Proxy is connected"); + return; + } + qapi_free_SocketAddress(udev->socket); =20 udev->socket =3D NULL; diff --git a/hw/vfio-user/proxy.c b/hw/vfio-user/proxy.c new file mode 100644 index 0000000000000000000000000000000000000000..bb436c9db91b6ebd90542b4121b= 62d56bbbdf4ab --- /dev/null +++ b/hw/vfio-user/proxy.c @@ -0,0 +1,162 @@ +/* + * vfio protocol over a UNIX socket. + * + * Copyright =C2=A9 2018, 2021 Oracle and/or its affiliates. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "qemu/osdep.h" +#include + +#include "hw/vfio/vfio-device.h" +#include "hw/vfio-user/proxy.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "qemu/lockable.h" +#include "system/iothread.h" + +static IOThread *vfio_user_iothread; + +static void vfio_user_shutdown(VFIOUserProxy *proxy); + + +/* + * Functions called by main, CPU, or iothread threads + */ + +static void vfio_user_shutdown(VFIOUserProxy *proxy) +{ + qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL); + qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, NULL, + proxy->ctx, NULL, NULL); +} + +/* + * Functions only called by iothread + */ + +static void vfio_user_cb(void *opaque) +{ + VFIOUserProxy *proxy =3D opaque; + + QEMU_LOCK_GUARD(&proxy->lock); + + proxy->state =3D VFIO_PROXY_CLOSED; + qemu_cond_signal(&proxy->close_cv); +} + + +/* + * Functions called by main or CPU threads + */ + +static QLIST_HEAD(, VFIOUserProxy) vfio_user_sockets =3D + QLIST_HEAD_INITIALIZER(vfio_user_sockets); + +VFIOUserProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) +{ + VFIOUserProxy *proxy; + QIOChannelSocket *sioc; + QIOChannel *ioc; + char *sockname; + + if (addr->type !=3D SOCKET_ADDRESS_TYPE_UNIX) { + error_setg(errp, "vfio_user_connect - bad address family"); + return NULL; + } + sockname =3D addr->u.q_unix.path; + + sioc =3D qio_channel_socket_new(); + ioc =3D QIO_CHANNEL(sioc); + if (qio_channel_socket_connect_sync(sioc, addr, errp)) { + object_unref(OBJECT(ioc)); + return NULL; + } + qio_channel_set_blocking(ioc, false, NULL); + + proxy =3D g_malloc0(sizeof(VFIOUserProxy)); + proxy->sockname =3D g_strdup_printf("unix:%s", sockname); + proxy->ioc =3D ioc; + proxy->flags =3D VFIO_PROXY_CLIENT; + proxy->state =3D VFIO_PROXY_CONNECTED; + + qemu_mutex_init(&proxy->lock); + qemu_cond_init(&proxy->close_cv); + + if (vfio_user_iothread =3D=3D NULL) { + vfio_user_iothread =3D iothread_create("VFIO user", errp); + } + + proxy->ctx =3D iothread_get_aio_context(vfio_user_iothread); + + QTAILQ_INIT(&proxy->outgoing); + QTAILQ_INIT(&proxy->incoming); + QTAILQ_INIT(&proxy->free); + QTAILQ_INIT(&proxy->pending); + QLIST_INSERT_HEAD(&vfio_user_sockets, proxy, next); + + return proxy; +} + +void vfio_user_disconnect(VFIOUserProxy *proxy) +{ + VFIOUserMsg *r1, *r2; + + qemu_mutex_lock(&proxy->lock); + + /* our side is quitting */ + if (proxy->state =3D=3D VFIO_PROXY_CONNECTED) { + vfio_user_shutdown(proxy); + if (!QTAILQ_EMPTY(&proxy->pending)) { + error_printf("vfio_user_disconnect: outstanding requests\n"); + } + } + object_unref(OBJECT(proxy->ioc)); + proxy->ioc =3D NULL; + + proxy->state =3D VFIO_PROXY_CLOSING; + QTAILQ_FOREACH_SAFE(r1, &proxy->outgoing, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->outgoing, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->incoming, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->incoming, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->pending, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->pending, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->free, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->free, r1, next); + g_free(r1); + } + + /* + * Make sure the iothread isn't blocking anywhere + * with a ref to this proxy by waiting for a BH + * handler to run after the proxy fd handlers were + * deleted above. + */ + aio_bh_schedule_oneshot(proxy->ctx, vfio_user_cb, proxy); + qemu_cond_wait(&proxy->close_cv, &proxy->lock); + + /* we now hold the only ref to proxy */ + qemu_mutex_unlock(&proxy->lock); + qemu_cond_destroy(&proxy->close_cv); + qemu_mutex_destroy(&proxy->lock); + + QLIST_REMOVE(proxy, next); + if (QLIST_EMPTY(&vfio_user_sockets)) { + iothread_destroy(vfio_user_iothread); + vfio_user_iothread =3D NULL; + } + + g_free(proxy->sockname); + g_free(proxy); +} diff --git a/hw/vfio-user/meson.build b/hw/vfio-user/meson.build index b82c558252e82d0f492bd68ab6c8b657d5b6b08c..9e85a8ea51c3402d828df0b78d5= 34a4adde193a2 100644 --- a/hw/vfio-user/meson.build +++ b/hw/vfio-user/meson.build @@ -4,6 +4,7 @@ vfio_user_ss =3D ss.source_set() vfio_user_ss.add(files( 'container.c', 'pci.c', + 'proxy.c', )) =20 system_ss.add_all(when: 'CONFIG_VFIO_USER', if_true: vfio_user_ss) --=20 2.49.0