From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072457724774.3688235062667; Fri, 1 Oct 2021 00:14:17 -0700 (PDT) Received: from localhost ([::1]:52470 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCkK-0007Nu-Kd for importer@patchew.org; Fri, 01 Oct 2021 03:14:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36018) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCfL-0006et-Vo for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:09:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:42600) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCfJ-0002RA-Uv for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:09:07 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-412-2C-kbHlFMVmdfsoatg9yXw-1; Fri, 01 Oct 2021 03:07:18 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 77AC2362FB; Fri, 1 Oct 2021 07:07:17 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6F63B652A9; Fri, 1 Oct 2021 07:06:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072145; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0XBf7eYZBNcPvk4C25u+XXOlbYb3AydMit/+nLqh3iM=; b=BtBMfQ2n6vDDZobYkAmtBK5ghq0I1p6IPASiTT2alqIT7pAMEVE+binuz1wVcl2lSUJwyN k0DxZI6ZERbhwEDRdc9Cv1FTW4PcWhq9AK+NjyeQ7S3N9GkPF4eG7G2gHjPy/YglsOHnGz 5LIGuZDuLyrVYmdXDZCNsdzW4j8Yyvc= X-MC-Unique: 2C-kbHlFMVmdfsoatg9yXw-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 01/20] virtio: Add VIRTIO_F_QUEUE_STATE Date: Fri, 1 Oct 2021 09:05:44 +0200 Message-Id: <20211001070603.307037-2-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072459919100001 virtio: Add VIRTIO_F_QUEUE_STATE Implementation of RFC of device state capability: https://lists.oasis-open.org/archives/virtio-comment/202012/msg00005.html With this capability, vdpa device can reset it's index so it can start consuming from guest after disabling shadow virtqueue (SVQ), with state not 0. The use case is to test SVQ with virtio-pci vdpa (vp_vdpa) with nested virtualization. Spawning a L0 qemu with a virtio-net device, use vp_vdpa driver to handle it in the guest, and then spawn a L1 qemu using that vdpa device. When L1 qemu calls device to set a new state though vdpa ioctl, vp_vdpa should set each queue state though virtio VIRTIO_PCI_COMMON_Q_AVAIL_STATE. Since this is only for testing vhost-vdpa, it's added here before of proposing to kernel code. No effort is done for checking that device can actually change its state, its layout, or if the device even supports to change state at all. These will be added in the future. Also, a modified version of vp_vdpa that allows to set these in PCI config is needed. TODO: Check for feature enabled and split in virtio pci config Signed-off-by: Eugenio P=C3=A9rez --- hw/virtio/virtio-pci.h | 1 + include/hw/virtio/virtio.h | 4 +++- include/standard-headers/linux/virtio_config.h | 3 +++ include/standard-headers/linux/virtio_pci.h | 2 ++ hw/virtio/virtio-pci.c | 9 +++++++++ 5 files changed, 18 insertions(+), 1 deletion(-) diff --git a/hw/virtio/virtio-pci.h b/hw/virtio/virtio-pci.h index 2446dcd9ae..019badbd7c 100644 --- a/hw/virtio/virtio-pci.h +++ b/hw/virtio/virtio-pci.h @@ -120,6 +120,7 @@ typedef struct VirtIOPCIQueue { uint32_t desc[2]; uint32_t avail[2]; uint32_t used[2]; + uint16_t state; } VirtIOPCIQueue; =20 struct VirtIOPCIProxy { diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h index 8bab9cfb75..5fe575b8f0 100644 --- a/include/hw/virtio/virtio.h +++ b/include/hw/virtio/virtio.h @@ -289,7 +289,9 @@ typedef struct VirtIORNGConf VirtIORNGConf; DEFINE_PROP_BIT64("iommu_platform", _state, _field, \ VIRTIO_F_IOMMU_PLATFORM, false), \ DEFINE_PROP_BIT64("packed", _state, _field, \ - VIRTIO_F_RING_PACKED, false) + VIRTIO_F_RING_PACKED, false), \ + DEFINE_PROP_BIT64("save_restore_q_state", _state, _field, \ + VIRTIO_F_QUEUE_STATE, true) =20 hwaddr virtio_queue_get_desc_addr(VirtIODevice *vdev, int n); bool virtio_queue_enabled_legacy(VirtIODevice *vdev, int n); diff --git a/include/standard-headers/linux/virtio_config.h b/include/stand= ard-headers/linux/virtio_config.h index 22e3a85f67..59fad3eb45 100644 --- a/include/standard-headers/linux/virtio_config.h +++ b/include/standard-headers/linux/virtio_config.h @@ -90,4 +90,7 @@ * Does the device support Single Root I/O Virtualization? */ #define VIRTIO_F_SR_IOV 37 + +/* Device support save and restore virtqueue state */ +#define VIRTIO_F_QUEUE_STATE 40 #endif /* _LINUX_VIRTIO_CONFIG_H */ diff --git a/include/standard-headers/linux/virtio_pci.h b/include/standard= -headers/linux/virtio_pci.h index db7a8e2fcb..c8d9802a87 100644 --- a/include/standard-headers/linux/virtio_pci.h +++ b/include/standard-headers/linux/virtio_pci.h @@ -164,6 +164,7 @@ struct virtio_pci_common_cfg { uint32_t queue_avail_hi; /* read-write */ uint32_t queue_used_lo; /* read-write */ uint32_t queue_used_hi; /* read-write */ + uint16_t queue_avail_state; /* read-write */ }; =20 /* Fields in VIRTIO_PCI_CAP_PCI_CFG: */ @@ -202,6 +203,7 @@ struct virtio_pci_cfg_cap { #define VIRTIO_PCI_COMMON_Q_AVAILHI 44 #define VIRTIO_PCI_COMMON_Q_USEDLO 48 #define VIRTIO_PCI_COMMON_Q_USEDHI 52 +#define VIRTIO_PCI_COMMON_Q_AVAIL_STATE 56 =20 #endif /* VIRTIO_PCI_NO_MODERN */ =20 diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index b321604d9b..6f30118c4e 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -1216,6 +1216,9 @@ static uint64_t virtio_pci_common_read(void *opaque, = hwaddr addr, case VIRTIO_PCI_COMMON_Q_USEDHI: val =3D proxy->vqs[vdev->queue_sel].used[1]; break; + case VIRTIO_PCI_COMMON_Q_AVAIL_STATE: + val =3D virtio_queue_get_last_avail_idx(vdev, vdev->queue_sel); + break; default: val =3D 0; } @@ -1298,6 +1301,8 @@ static void virtio_pci_common_write(void *opaque, hwa= ddr addr, proxy->vqs[vdev->queue_sel].avail[0], ((uint64_t)proxy->vqs[vdev->queue_sel].used[1]) << = 32 | proxy->vqs[vdev->queue_sel].used[0]); + virtio_queue_set_last_avail_idx(vdev, vdev->queue_sel, + proxy->vqs[vdev->queue_sel].state); proxy->vqs[vdev->queue_sel].enabled =3D 1; } else { virtio_error(vdev, "wrong value for queue_enable %"PRIx64, val= ); @@ -1321,6 +1326,9 @@ static void virtio_pci_common_write(void *opaque, hwa= ddr addr, case VIRTIO_PCI_COMMON_Q_USEDHI: proxy->vqs[vdev->queue_sel].used[1] =3D val; break; + case VIRTIO_PCI_COMMON_Q_AVAIL_STATE: + proxy->vqs[vdev->queue_sel].state =3D val; + break; default: break; } @@ -1909,6 +1917,7 @@ static void virtio_pci_reset(DeviceState *qdev) proxy->vqs[i].desc[0] =3D proxy->vqs[i].desc[1] =3D 0; proxy->vqs[i].avail[0] =3D proxy->vqs[i].avail[1] =3D 0; proxy->vqs[i].used[0] =3D proxy->vqs[i].used[1] =3D 0; + proxy->vqs[i].state =3D 0; } =20 if (pci_is_express(dev)) { --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072548363366.8041001274124; Fri, 1 Oct 2021 00:15:48 -0700 (PDT) Received: from localhost ([::1]:56578 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCln-0001vy-C4 for importer@patchew.org; Fri, 01 Oct 2021 03:15:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35842) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCeJ-0004p7-4p for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:08:03 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:41673) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCeG-0001Xz-6H for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:08:02 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-216-IbTRIXAFMnCK9WYahj7GSg-1; Fri, 01 Oct 2021 03:07:57 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1DFA9100C611; Fri, 1 Oct 2021 07:07:56 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id CDBF6652A3; Fri, 1 Oct 2021 07:07:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fe5PwWm1Kl1k0v6vrZBQCaXgqjhoQkQLDRjOdghm7BA=; b=WeK7mPxESjN7eNm945ja5v77F1GOeErViiJ1prfU7v6nj1j/xH3Ymsi1uJTMmnvOZWyFsG Bwf2VizZRI5iRb03y2lTbw5FidHFWAbDJVhxjMwcafjMGv9CI2K1+IOa2BVwPVI0LzAJyq x7lg7IGtU1AnXUqUja1DAuTrDvdHE+8= X-MC-Unique: IbTRIXAFMnCK9WYahj7GSg-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 02/20] virtio-net: Honor VIRTIO_CONFIG_S_DEVICE_STOPPED Date: Fri, 1 Oct 2021 09:05:45 +0200 Message-Id: <20211001070603.307037-3-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072550696100001 So the guest can stop and start net device. It freely implements the RFC https://lists.oasis-open.org/archives/virtio-comment/202012/msg00027.html To stop (as "pause") the device is required to migrate status and vring addresses between device and SVQ. Once the device is stopped, the driver can request avail_idx, so it can be assigned to SVQ. This is a WIP commit: as with VIRTIO_F_QUEUE_STATE, is introduced in virtio_config.h before of even proposing for the kernel, with no feature flag, and, with no checking in the device. It also needs a modified vp_vdpa driver that supports to set and retrieve status. For virtio-net with qemu device there is no need to restore avail state: Since every tx and rx operation is entirely done in BQL regarding virtio, it would be enough with restore last_avail_idx with used_idx. Doing this way test the vq state part of the rest of the series. Signed-off-by: Eugenio P=C3=A9rez --- include/standard-headers/linux/virtio_config.h | 2 ++ hw/net/virtio-net.c | 6 ++++-- hw/virtio/virtio-pci.c | 7 +++++-- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/include/standard-headers/linux/virtio_config.h b/include/stand= ard-headers/linux/virtio_config.h index 59fad3eb45..b3f6b1365d 100644 --- a/include/standard-headers/linux/virtio_config.h +++ b/include/standard-headers/linux/virtio_config.h @@ -40,6 +40,8 @@ #define VIRTIO_CONFIG_S_DRIVER_OK 4 /* Driver has finished configuring features */ #define VIRTIO_CONFIG_S_FEATURES_OK 8 +/* Device is stopped */ +#define VIRTIO_CONFIG_S_DEVICE_STOPPED 32 /* Device entered invalid state, driver must reset it */ #define VIRTIO_CONFIG_S_NEEDS_RESET 0x40 /* We've given up on this device. */ diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c index bd7958b9f0..e8f55cdeba 100644 --- a/hw/net/virtio-net.c +++ b/hw/net/virtio-net.c @@ -198,6 +198,7 @@ static bool virtio_net_started(VirtIONet *n, uint8_t st= atus) { VirtIODevice *vdev =3D VIRTIO_DEVICE(n); return (status & VIRTIO_CONFIG_S_DRIVER_OK) && + (!(status & VIRTIO_CONFIG_S_DEVICE_STOPPED)) && (n->status & VIRTIO_NET_S_LINK_UP) && vdev->vm_running; } =20 @@ -385,7 +386,7 @@ static void virtio_net_set_status(struct VirtIODevice *= vdev, uint8_t status) qemu_flush_queued_packets(ncs); } =20 - if (!q->tx_waiting) { + if (!q->tx_waiting && !(status & VIRTIO_CONFIG_S_DEVICE_STOPPED)) { continue; } =20 @@ -1503,7 +1504,8 @@ static bool virtio_net_can_receive(NetClientState *nc) } =20 if (!virtio_queue_ready(q->rx_vq) || - !(vdev->status & VIRTIO_CONFIG_S_DRIVER_OK)) { + !(vdev->status & VIRTIO_CONFIG_S_DRIVER_OK) || + vdev->status =3D=3D VIRTIO_CONFIG_S_DEVICE_STOPPED) { return false; } =20 diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index 6f30118c4e..9ed0f62222 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -326,13 +326,15 @@ static void virtio_ioport_write(void *opaque, uint32_= t addr, uint32_t val) } break; case VIRTIO_PCI_STATUS: - if (!(val & VIRTIO_CONFIG_S_DRIVER_OK)) { + if (!(val & VIRTIO_CONFIG_S_DRIVER_OK) || + val & VIRTIO_CONFIG_S_DEVICE_STOPPED) { virtio_pci_stop_ioeventfd(proxy); } =20 virtio_set_status(vdev, val & 0xFF); =20 - if (val & VIRTIO_CONFIG_S_DRIVER_OK) { + if (val & VIRTIO_CONFIG_S_DRIVER_OK && + !(val & VIRTIO_CONFIG_S_DEVICE_STOPPED)) { virtio_pci_start_ioeventfd(proxy); } =20 @@ -1303,6 +1305,7 @@ static void virtio_pci_common_write(void *opaque, hwa= ddr addr, proxy->vqs[vdev->queue_sel].used[0]); virtio_queue_set_last_avail_idx(vdev, vdev->queue_sel, proxy->vqs[vdev->queue_sel].state); + virtio_queue_update_used_idx(vdev, vdev->queue_sel); proxy->vqs[vdev->queue_sel].enabled =3D 1; } else { virtio_error(vdev, "wrong value for queue_enable %"PRIx64, val= ); --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072340601839.2686257492762; Fri, 1 Oct 2021 00:12:20 -0700 (PDT) Received: from localhost ([::1]:43868 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCiR-0001bL-K8 for importer@patchew.org; Fri, 01 Oct 2021 03:12:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35942) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCer-0005oG-UN for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:08:37 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22047) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCeq-00021P-BG for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:08:37 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-425-V4TBdsemNJWq8Gyifeh7KQ-1; Fri, 01 Oct 2021 03:08:34 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3A951835DF2; Fri, 1 Oct 2021 07:08:33 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7438B652A3; Fri, 1 Oct 2021 07:07:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072115; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sgZ38tOaoOZHoH82uyObhvieEWAWbN89505U54jy5E0=; b=Yti47XKRvfmz2lcTOaW5U5XEZ/yTafSffKn0QHxwqV9XEerzl/2v9x59oM2wi+URaDboEA A3wWZiZHmjEsdE7DvzK/C92FBjK2yndy5U1XwRK2FWjjc3bCPMq0YI4NHsgjCfYMcM4ZnL ILkYySmdJXnVMIx3OBL9JGt/1emu0BY= X-MC-Unique: V4TBdsemNJWq8Gyifeh7KQ-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 03/20] virtio: Add virtio_queue_is_host_notifier_enabled Date: Fri, 1 Oct 2021 09:05:46 +0200 Message-Id: <20211001070603.307037-4-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072341162100001 This allows shadow virtqueue code to assert the queue status before making changes. Signed-off-by: Eugenio P=C3=A9rez --- include/hw/virtio/virtio.h | 1 + hw/virtio/virtio.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h index 5fe575b8f0..2fe03f64c6 100644 --- a/include/hw/virtio/virtio.h +++ b/include/hw/virtio/virtio.h @@ -318,6 +318,7 @@ void virtio_device_release_ioeventfd(VirtIODevice *vdev= ); bool virtio_device_ioeventfd_enabled(VirtIODevice *vdev); EventNotifier *virtio_queue_get_host_notifier(VirtQueue *vq); void virtio_queue_set_host_notifier_enabled(VirtQueue *vq, bool enabled); +bool virtio_queue_is_host_notifier_enabled(const VirtQueue *vq); void virtio_queue_host_notifier_read(EventNotifier *n); void virtio_queue_aio_set_host_notifier_handler(VirtQueue *vq, AioContext = *ctx, VirtIOHandleAIOOutput hand= le_output); diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c index ab516ac614..f04de57409 100644 --- a/hw/virtio/virtio.c +++ b/hw/virtio/virtio.c @@ -3591,6 +3591,11 @@ EventNotifier *virtio_queue_get_host_notifier(VirtQu= eue *vq) return &vq->host_notifier; } =20 +bool virtio_queue_is_host_notifier_enabled(const VirtQueue *vq) +{ + return vq->host_notifier_enabled; +} + void virtio_queue_set_host_notifier_enabled(VirtQueue *vq, bool enabled) { vq->host_notifier_enabled =3D enabled; --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072301745279.3434072647716; Fri, 1 Oct 2021 00:11:41 -0700 (PDT) Received: from localhost ([::1]:41910 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCho-0000Gp-Ep for importer@patchew.org; Fri, 01 Oct 2021 03:11:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35982) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCf6-0006HQ-Rk for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:08:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:46301) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCf5-0002De-8Z for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:08:52 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-485-gcWOCzEqNP-RyhHkihZ_tQ-1; Fri, 01 Oct 2021 03:08:49 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 50F445074B; Fri, 1 Oct 2021 07:08:48 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9077B652A3; Fri, 1 Oct 2021 07:08:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072130; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uOCqeYp9qrRUCeUqz4B9/qImndEpz6AxLlPOQZEQNnM=; b=GjeWY1/qL5coi3kZ0gts+L0WMh6nl1htm5xXpI7DRykTU0PxKl0bPUpeR0vdwp44LTg8Sw XYQpkMBfviQH1WKNjkUvIAC24+QqKHOCKFM72RVMsimAZqpX/KEblR3FuKWXQUXvPuO9d9 n6tPj22snXgQXIcUpzS5F2lHuZa0vUk= X-MC-Unique: gcWOCzEqNP-RyhHkihZ_tQ-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 04/20] vhost: Make vhost_virtqueue_{start,stop} public Date: Fri, 1 Oct 2021 09:05:47 +0200 Message-Id: <20211001070603.307037-5-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072303334100001 The easier way to reset vhost behavior after enabling shadow virtqueue is to call vhost_virtqueue_start. Also, vhost_virtqueue_stop provides the simmetrical call to it, and it will be used. Signed-off-by: Eugenio P=C3=A9rez --- include/hw/virtio/vhost.h | 4 ++++ hw/virtio/vhost.c | 12 ++++-------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h index 045d0fd9f2..1cdcded4c5 100644 --- a/include/hw/virtio/vhost.h +++ b/include/hw/virtio/vhost.h @@ -130,6 +130,10 @@ int vhost_net_set_backend(struct vhost_dev *hdev, struct vhost_vring_file *file); =20 int vhost_device_iotlb_miss(struct vhost_dev *dev, uint64_t iova, int writ= e); +int vhost_virtqueue_start(struct vhost_dev *dev, struct VirtIODevice *vdev, + struct vhost_virtqueue *vq, unsigned idx); +void vhost_virtqueue_stop(struct vhost_dev *dev, struct VirtIODevice *vdev, + struct vhost_virtqueue *vq, unsigned idx); int vhost_dev_get_config(struct vhost_dev *hdev, uint8_t *config, uint32_t config_len, Error **errp); int vhost_dev_set_config(struct vhost_dev *dev, const uint8_t *data, diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c index e040f631c6..5d2aa6d72d 100644 --- a/hw/virtio/vhost.c +++ b/hw/virtio/vhost.c @@ -1049,10 +1049,8 @@ out: return ret; } =20 -static int vhost_virtqueue_start(struct vhost_dev *dev, - struct VirtIODevice *vdev, - struct vhost_virtqueue *vq, - unsigned idx) +int vhost_virtqueue_start(struct vhost_dev *dev, struct VirtIODevice *vdev, + struct vhost_virtqueue *vq, unsigned idx) { BusState *qbus =3D BUS(qdev_get_parent_bus(DEVICE(vdev))); VirtioBusState *vbus =3D VIRTIO_BUS(qbus); @@ -1171,10 +1169,8 @@ fail_alloc_desc: return r; } =20 -static void vhost_virtqueue_stop(struct vhost_dev *dev, - struct VirtIODevice *vdev, - struct vhost_virtqueue *vq, - unsigned idx) +void vhost_virtqueue_stop(struct vhost_dev *dev, struct VirtIODevice *vdev, + struct vhost_virtqueue *vq, unsigned idx) { int vhost_vq_index =3D dev->vhost_ops->vhost_get_vq_index(dev, idx); struct vhost_vring_state state =3D { --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072435291928.8862771306923; Fri, 1 Oct 2021 00:13:55 -0700 (PDT) Received: from localhost ([::1]:50210 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCjx-0005sr-Un for importer@patchew.org; Fri, 01 Oct 2021 03:13:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36116) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCfX-0006nY-4X for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:09:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:49887) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCfU-0002a0-8z for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:09:18 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-140-TQIMso0iPBeVRjzMI1HCmg-1; Fri, 01 Oct 2021 03:09:08 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 960FE5074C; Fri, 1 Oct 2021 07:09:07 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id A7502652A3; Fri, 1 Oct 2021 07:08:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072155; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5XBGobM44EFS3izwe6BQDokWeZytbVyZqepmR96WPwI=; b=hcCn6rhe5r244QzYAJczyoryb43N4vTotogJlnNUPvqVajtz/5Q8ZadB6ngDLTImF2fBuk xCTahV0YTujao+JtcdKgAq/pO5SGrO4uXLX9KrXme7XOEwQyzQeu4a4ZC16q0zzRblNNHh ksA3lv1OqjBvO9/FDZ9MimOMaC9tx64= X-MC-Unique: TQIMso0iPBeVRjzMI1HCmg-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 05/20] vhost: Add x-vhost-enable-shadow-vq qmp Date: Fri, 1 Oct 2021 09:05:48 +0200 Message-Id: <20211001070603.307037-6-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072436243100001 Command to enable shadow virtqueue. Signed-off-by: Eugenio P=C3=A9rez --- qapi/net.json | 23 +++++++++++++++++++++++ hw/virtio/vhost-vdpa.c | 8 ++++++++ 2 files changed, 31 insertions(+) diff --git a/qapi/net.json b/qapi/net.json index 7fab2e7cd8..a2c30fd455 100644 --- a/qapi/net.json +++ b/qapi/net.json @@ -79,6 +79,29 @@ { 'command': 'netdev_del', 'data': {'id': 'str'}, 'allow-preconfig': true } =20 +## +# @x-vhost-enable-shadow-vq: +# +# Use vhost shadow virtqueue. +# +# @name: the device name of the VirtIO device +# +# @enable: true to use the alternate shadow VQ notifications +# +# Returns: Always error, since SVQ is not implemented at the moment. +# +# Since: 6.2 +# +# Example: +# +# -> { "execute": "x-vhost-enable-shadow-vq", +# "arguments": { "name": "virtio-net", "enable": false } } +# +## +{ 'command': 'x-vhost-enable-shadow-vq', + 'data': {'name': 'str', 'enable': 'bool'}, + 'if': 'defined(CONFIG_VHOST_KERNEL)' } + ## # @NetLegacyNicOptions: # diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 4fa414feea..c63e311d7c 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -23,6 +23,8 @@ #include "cpu.h" #include "trace.h" #include "qemu-common.h" +#include "qapi/qapi-commands-net.h" +#include "qapi/error.h" =20 static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *secti= on) { @@ -656,6 +658,12 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *= dev) return true; } =20 + +void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **e= rrp) +{ + error_setg(errp, "Shadow virtqueue still not implemented"); +} + const VhostOps vdpa_ops =3D { .backend_type =3D VHOST_BACKEND_TYPE_VDPA, .vhost_backend_init =3D vhost_vdpa_init, --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072603545900.441233641598; Fri, 1 Oct 2021 00:16:43 -0700 (PDT) Received: from localhost ([::1]:58608 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCmg-0003fe-EX for importer@patchew.org; Fri, 01 Oct 2021 03:16:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36150) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCfq-000797-MB for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:09:38 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:33462) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCfn-0002pE-Mr for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:09:37 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-433-9UKpIPfuNjelWYIh4eTW8w-1; Fri, 01 Oct 2021 03:09:33 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7AA9D18414A0; Fri, 1 Oct 2021 07:09:31 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id EBB6A652AE; Fri, 1 Oct 2021 07:09:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072175; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h0/PHssZ4/ql7VOBVkx1sV4rOr7kKacTvOgAX8YEMwo=; b=G5YuCBwiUTLydPqcIAO1rkSFuRA9kbb0vvJYNssWpFv5BJKUS462w6ylhh//sllXDB9FOq klwZrGPoXCLetAPozVud+lQs9y2Hfy3eI6CE1ILLW9PDKq5L71/DmYuqLFU3OvOGFaPskM ZnM/cPwL6k32+Lj5kll9OSC4vZDsMYg= X-MC-Unique: 9UKpIPfuNjelWYIh4eTW8w-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 06/20] vhost: Add VhostShadowVirtqueue Date: Fri, 1 Oct 2021 09:05:49 +0200 Message-Id: <20211001070603.307037-7-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072605626100001 Vhost shadow virtqueue (SVQ) is an intermediate jump for virtqueue notifications and buffers, allowing qemu to track them. While qemu is forwarding the buffers and virtqueue changes, is able to commit the memory it's being dirtied, the same way regular qemu's VirtIO devices do. This commit only exposes basic SVQ allocation and free, so changes regarding different aspects of SVQ (notifications forwarding, buffer forwarding, starting/stopping) are more isolated and easier to bisect. Signed-off-by: Eugenio P=C3=A9rez --- hw/virtio/vhost-shadow-virtqueue.h | 21 ++++++++++ hw/virtio/vhost-shadow-virtqueue.c | 64 ++++++++++++++++++++++++++++++ hw/virtio/meson.build | 2 +- 3 files changed, 86 insertions(+), 1 deletion(-) create mode 100644 hw/virtio/vhost-shadow-virtqueue.h create mode 100644 hw/virtio/vhost-shadow-virtqueue.c diff --git a/hw/virtio/vhost-shadow-virtqueue.h b/hw/virtio/vhost-shadow-vi= rtqueue.h new file mode 100644 index 0000000000..27ac6388fa --- /dev/null +++ b/hw/virtio/vhost-shadow-virtqueue.h @@ -0,0 +1,21 @@ +/* + * vhost shadow virtqueue + * + * SPDX-FileCopyrightText: Red Hat, Inc. 2021 + * SPDX-FileContributor: Author: Eugenio P=C3=A9rez + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#ifndef VHOST_SHADOW_VIRTQUEUE_H +#define VHOST_SHADOW_VIRTQUEUE_H + +#include "hw/virtio/vhost.h" + +typedef struct VhostShadowVirtqueue VhostShadowVirtqueue; + +VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx); + +void vhost_svq_free(VhostShadowVirtqueue *vq); + +#endif diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-vi= rtqueue.c new file mode 100644 index 0000000000..c4826a1b56 --- /dev/null +++ b/hw/virtio/vhost-shadow-virtqueue.c @@ -0,0 +1,64 @@ +/* + * vhost shadow virtqueue + * + * SPDX-FileCopyrightText: Red Hat, Inc. 2021 + * SPDX-FileContributor: Author: Eugenio P=C3=A9rez + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "qemu/osdep.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" + +#include "qemu/error-report.h" +#include "qemu/event_notifier.h" + +/* Shadow virtqueue to relay notifications */ +typedef struct VhostShadowVirtqueue { + /* Shadow kick notifier, sent to vhost */ + EventNotifier kick_notifier; + /* Shadow call notifier, sent to vhost */ + EventNotifier call_notifier; +} VhostShadowVirtqueue; + +/* + * Creates vhost shadow virtqueue, and instruct vhost device to use the sh= adow + * methods and file descriptors. + */ +VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) +{ + g_autofree VhostShadowVirtqueue *svq =3D g_new0(VhostShadowVirtqueue, = 1); + int r; + + r =3D event_notifier_init(&svq->kick_notifier, 0); + if (r !=3D 0) { + error_report("Couldn't create kick event notifier: %s", + strerror(errno)); + goto err_init_kick_notifier; + } + + r =3D event_notifier_init(&svq->call_notifier, 0); + if (r !=3D 0) { + error_report("Couldn't create call event notifier: %s", + strerror(errno)); + goto err_init_call_notifier; + } + + return g_steal_pointer(&svq); + +err_init_call_notifier: + event_notifier_cleanup(&svq->kick_notifier); + +err_init_kick_notifier: + return NULL; +} + +/* + * Free the resources of the shadow virtqueue. + */ +void vhost_svq_free(VhostShadowVirtqueue *vq) +{ + event_notifier_cleanup(&vq->kick_notifier); + event_notifier_cleanup(&vq->call_notifier); + g_free(vq); +} diff --git a/hw/virtio/meson.build b/hw/virtio/meson.build index fbff9bc9d4..8b5a0225fe 100644 --- a/hw/virtio/meson.build +++ b/hw/virtio/meson.build @@ -11,7 +11,7 @@ softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('vhost-= stub.c')) =20 virtio_ss =3D ss.source_set() virtio_ss.add(files('virtio.c')) -virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backe= nd.c')) +virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backe= nd.c', 'vhost-shadow-virtqueue.c')) virtio_ss.add(when: 'CONFIG_VHOST_USER', if_true: files('vhost-user.c')) virtio_ss.add(when: 'CONFIG_VHOST_VDPA', if_true: files('vhost-vdpa.c')) virtio_ss.add(when: 'CONFIG_VIRTIO_BALLOON', if_true: files('virtio-balloo= n.c')) --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 16330723259061016.6192913216737; Fri, 1 Oct 2021 00:12:05 -0700 (PDT) Received: from localhost ([::1]:43430 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCiC-0001JV-Ns for importer@patchew.org; Fri, 01 Oct 2021 03:12:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36162) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCfs-0007B7-VP for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:09:42 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:21704) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCfq-0002rJ-Q1 for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:09:40 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-578-Xv5e4LRUPOusZ76MYWvRIA-1; Fri, 01 Oct 2021 03:09:36 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CE04C802C94; Fri, 1 Oct 2021 07:09:34 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id CFBAA652A3; Fri, 1 Oct 2021 07:09:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072177; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cj8QM05V63c4CeVJDbrPZw5Vo13mWE1Jz+BFYgkYJZc=; b=RJmej8fIHWeiqX8qNjl/ie+LmQ2F5yee7LkqI4HDAoP5oY5PfLAD/tQHjuXRNPIGMwdonO 36iTfOSzdYZGBqX0nI7MCMigQQvXj3wqfiEcebPLn5NedwUafiu6+IwrS5Py6KCoAzMhSx rXHdZSGj7IOMGUl4yCsLwy1G+oBFzfg= X-MC-Unique: Xv5e4LRUPOusZ76MYWvRIA-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 07/20] vdpa: Register vdpa devices in a list Date: Fri, 1 Oct 2021 09:05:50 +0200 Message-Id: <20211001070603.307037-8-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072327389100001 This way QMP command can iterate through them and find the devices it needs to enable shadow virtqueue in. Signed-off-by: Eugenio P=C3=A9rez --- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 9188226d8b..0d565bb5bd 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -12,6 +12,7 @@ #ifndef HW_VIRTIO_VHOST_VDPA_H #define HW_VIRTIO_VHOST_VDPA_H =20 +#include "qemu/queue.h" #include "hw/virtio/virtio.h" =20 typedef struct VhostVDPAHostNotifier { @@ -24,6 +25,7 @@ typedef struct vhost_vdpa { uint32_t msg_type; MemoryListener listener; struct vhost_dev *dev; + QLIST_ENTRY(vhost_vdpa) entry; VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; } VhostVDPA; =20 diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index c63e311d7c..e0dc7508c3 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -26,6 +26,9 @@ #include "qapi/qapi-commands-net.h" #include "qapi/error.h" =20 +static QLIST_HEAD(, vhost_vdpa) vhost_vdpa_devices =3D + QLIST_HEAD_INITIALIZER(vhost_vdpa_devices); + static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *secti= on) { return (!memory_region_is_ram(section->mr) && @@ -280,6 +283,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void = *opaque, Error **errp) dev->opaque =3D opaque ; v->listener =3D vhost_vdpa_memory_listener; v->msg_type =3D VHOST_IOTLB_MSG_V2; + QLIST_INSERT_HEAD(&vhost_vdpa_devices, v, entry); =20 vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | VIRTIO_CONFIG_S_DRIVER); @@ -377,6 +381,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev) trace_vhost_vdpa_cleanup(dev, v); vhost_vdpa_host_notifiers_uninit(dev, dev->nvqs); memory_listener_unregister(&v->listener); + QLIST_REMOVE(v, entry); =20 dev->opaque =3D NULL; return 0; --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072781126185.57714639568735; Fri, 1 Oct 2021 00:19:41 -0700 (PDT) Received: from localhost ([::1]:36986 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCpX-0008IF-Qk for importer@patchew.org; Fri, 01 Oct 2021 03:19:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36174) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCfv-0007C1-AB for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:09:44 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22712) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCft-0002tj-4h for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:09:43 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-603-Q_kwPjGEM0a1F_UT4EldZA-1; Fri, 01 Oct 2021 03:09:39 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4F4151868417; Fri, 1 Oct 2021 07:09:38 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 313F0652AE; Fri, 1 Oct 2021 07:09:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xl2hd1UexJOmNm81T+0SxgNCwG4G9PPSF2XXnl/ztVg=; b=NGpzyMnbCFq2tlUFjaWO6x1thGtpLQzH2HpLq8b9mfDvLzdOUGlF7Vrant5aWCWcoK93D3 DMDIp5f0r5i34lHRekO/zv8IhHab38U8x92HbxU9bZTZCBu3ECeSN+MDOE25AZvrMVZuut wi6evz8WTrJapxYXHChL8tkpW89Xxbw= X-MC-Unique: Q_kwPjGEM0a1F_UT4EldZA-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 08/20] vhost: Route guest->host notification through shadow virtqueue Date: Fri, 1 Oct 2021 09:05:51 +0200 Message-Id: <20211001070603.307037-9-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072783655100001 Shadow virtqueue notifications forwarding is disabled when vhost_dev stops, so code flow follows usual cleanup. Also, host notifiers must be disabled at SVQ start, and they will not start if SVQ has been enabled when device is stopped. This is trivial to address, but it is left out for simplicity at this moment. Signed-off-by: Eugenio P=C3=A9rez --- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.h | 8 ++ include/hw/virtio/vhost-vdpa.h | 4 + hw/virtio/vhost-shadow-virtqueue.c | 138 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 116 +++++++++++++++++++++++- 5 files changed, 264 insertions(+), 4 deletions(-) diff --git a/qapi/net.json b/qapi/net.json index a2c30fd455..fe546b0e7c 100644 --- a/qapi/net.json +++ b/qapi/net.json @@ -88,7 +88,7 @@ # # @enable: true to use the alternate shadow VQ notifications # -# Returns: Always error, since SVQ is not implemented at the moment. +# Returns: Error if failure, or 'no error' for success. # # Since: 6.2 # diff --git a/hw/virtio/vhost-shadow-virtqueue.h b/hw/virtio/vhost-shadow-vi= rtqueue.h index 27ac6388fa..237cfceb9c 100644 --- a/hw/virtio/vhost-shadow-virtqueue.h +++ b/hw/virtio/vhost-shadow-virtqueue.h @@ -14,6 +14,14 @@ =20 typedef struct VhostShadowVirtqueue VhostShadowVirtqueue; =20 +EventNotifier *vhost_svq_get_svq_call_notifier(VhostShadowVirtqueue *svq); +void vhost_svq_set_guest_call_notifier(VhostShadowVirtqueue *svq, int call= _fd); + +bool vhost_svq_start(struct vhost_dev *dev, unsigned idx, + VhostShadowVirtqueue *svq); +void vhost_svq_stop(struct vhost_dev *dev, unsigned idx, + VhostShadowVirtqueue *svq); + VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx); =20 void vhost_svq_free(VhostShadowVirtqueue *vq); diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 0d565bb5bd..48aae59d8e 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -12,6 +12,8 @@ #ifndef HW_VIRTIO_VHOST_VDPA_H #define HW_VIRTIO_VHOST_VDPA_H =20 +#include + #include "qemu/queue.h" #include "hw/virtio/virtio.h" =20 @@ -24,6 +26,8 @@ typedef struct vhost_vdpa { int device_fd; uint32_t msg_type; MemoryListener listener; + bool shadow_vqs_enabled; + GPtrArray *shadow_vqs; struct vhost_dev *dev; QLIST_ENTRY(vhost_vdpa) entry; VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-vi= rtqueue.c index c4826a1b56..21dc99ab5d 100644 --- a/hw/virtio/vhost-shadow-virtqueue.c +++ b/hw/virtio/vhost-shadow-virtqueue.c @@ -9,9 +9,12 @@ =20 #include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" +#include "hw/virtio/vhost.h" + +#include "standard-headers/linux/vhost_types.h" =20 #include "qemu/error-report.h" -#include "qemu/event_notifier.h" +#include "qemu/main-loop.h" =20 /* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue { @@ -19,14 +22,146 @@ typedef struct VhostShadowVirtqueue { EventNotifier kick_notifier; /* Shadow call notifier, sent to vhost */ EventNotifier call_notifier; + + /* + * Borrowed virtqueue's guest to host notifier. + * To borrow it in this event notifier allows to register on the event + * loop and access the associated shadow virtqueue easily. If we use t= he + * VirtQueue, we don't have an easy way to retrieve it. + * + * So shadow virtqueue must not clean it, or we would lose VirtQueue o= ne. + */ + EventNotifier host_notifier; + + /* Guest's call notifier, where SVQ calls guest. */ + EventNotifier guest_call_notifier; + + /* Virtio queue shadowing */ + VirtQueue *vq; } VhostShadowVirtqueue; =20 +/* Forward guest notifications */ +static void vhost_handle_guest_kick(EventNotifier *n) +{ + VhostShadowVirtqueue *svq =3D container_of(n, VhostShadowVirtqueue, + host_notifier); + + if (unlikely(!event_notifier_test_and_clear(n))) { + return; + } + + event_notifier_set(&svq->kick_notifier); +} + +/* + * Obtain the SVQ call notifier, where vhost device notifies SVQ that there + * exists pending used buffers. + * + * @svq Shadow Virtqueue + */ +EventNotifier *vhost_svq_get_svq_call_notifier(VhostShadowVirtqueue *svq) +{ + return &svq->call_notifier; +} + +/* + * Set the call notifier for the SVQ to call the guest + * + * @svq Shadow virtqueue + * @call_fd call notifier + * + * Called on BQL context. + */ +void vhost_svq_set_guest_call_notifier(VhostShadowVirtqueue *svq, int call= _fd) +{ + event_notifier_init_fd(&svq->guest_call_notifier, call_fd); +} + +/* + * Restore the vhost guest to host notifier, i.e., disables svq effect. + */ +static int vhost_svq_restore_vdev_host_notifier(struct vhost_dev *dev, + unsigned vhost_index, + VhostShadowVirtqueue *svq) +{ + EventNotifier *vq_host_notifier =3D virtio_queue_get_host_notifier(svq= ->vq); + struct vhost_vring_file file =3D { + .index =3D vhost_index, + .fd =3D event_notifier_get_fd(vq_host_notifier), + }; + int r; + + /* Restore vhost kick */ + r =3D dev->vhost_ops->vhost_set_vring_kick(dev, &file); + return r ? -errno : 0; +} + +/* + * Start shadow virtqueue operation. + * @dev vhost device + * @hidx vhost virtqueue index + * @svq Shadow Virtqueue + */ +bool vhost_svq_start(struct vhost_dev *dev, unsigned idx, + VhostShadowVirtqueue *svq) +{ + EventNotifier *vq_host_notifier =3D virtio_queue_get_host_notifier(svq= ->vq); + struct vhost_vring_file file =3D { + .index =3D dev->vhost_ops->vhost_get_vq_index(dev, dev->vq_index += idx), + .fd =3D event_notifier_get_fd(&svq->kick_notifier), + }; + int r; + + /* Check that notifications are still going directly to vhost dev */ + assert(virtio_queue_is_host_notifier_enabled(svq->vq)); + + /* + * event_notifier_set_handler already checks for guest's notifications= if + * they arrive in the switch, so there is no need to explicitely check= for + * them. + */ + event_notifier_init_fd(&svq->host_notifier, + event_notifier_get_fd(vq_host_notifier)); + event_notifier_set_handler(&svq->host_notifier, vhost_handle_guest_kic= k); + + r =3D dev->vhost_ops->vhost_set_vring_kick(dev, &file); + if (unlikely(r !=3D 0)) { + error_report("Couldn't set kick fd: %s", strerror(errno)); + goto err_set_vring_kick; + } + + return true; + +err_set_vring_kick: + event_notifier_set_handler(&svq->host_notifier, NULL); + + return false; +} + +/* + * Stop shadow virtqueue operation. + * @dev vhost device + * @idx vhost queue index + * @svq Shadow Virtqueue + */ +void vhost_svq_stop(struct vhost_dev *dev, unsigned idx, + VhostShadowVirtqueue *svq) +{ + int r =3D vhost_svq_restore_vdev_host_notifier(dev, idx, svq); + if (unlikely(r < 0)) { + error_report("Couldn't restore vq kick fd: %s", strerror(-r)); + } + + event_notifier_set_handler(&svq->host_notifier, NULL); +} + /* * Creates vhost shadow virtqueue, and instruct vhost device to use the sh= adow * methods and file descriptors. */ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) { + int vq_idx =3D dev->vq_index + idx; g_autofree VhostShadowVirtqueue *svq =3D g_new0(VhostShadowVirtqueue, = 1); int r; =20 @@ -44,6 +179,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *de= v, int idx) goto err_init_call_notifier; } =20 + svq->vq =3D virtio_get_queue(dev->vdev, vq_idx); return g_steal_pointer(&svq); =20 err_init_call_notifier: diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index e0dc7508c3..36c954a779 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -17,6 +17,7 @@ #include "hw/virtio/vhost.h" #include "hw/virtio/vhost-backend.h" #include "hw/virtio/virtio-net.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" @@ -272,6 +273,16 @@ static void vhost_vdpa_add_status(struct vhost_dev *de= v, uint8_t status) vhost_vdpa_call(dev, VHOST_VDPA_SET_STATUS, &s); } =20 +/** + * Adaptor function to free shadow virtqueue through gpointer + * + * @svq The Shadow Virtqueue + */ +static void vhost_psvq_free(gpointer svq) +{ + vhost_svq_free(svq); +} + static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **er= rp) { struct vhost_vdpa *v; @@ -283,6 +294,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void = *opaque, Error **errp) dev->opaque =3D opaque ; v->listener =3D vhost_vdpa_memory_listener; v->msg_type =3D VHOST_IOTLB_MSG_V2; + v->shadow_vqs =3D g_ptr_array_new_full(dev->nvqs, vhost_psvq_free); QLIST_INSERT_HEAD(&vhost_vdpa_devices, v, entry); =20 vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE | @@ -373,6 +385,17 @@ err: return; } =20 +static void vhost_vdpa_svq_cleanup(struct vhost_dev *dev) +{ + struct vhost_vdpa *v =3D dev->opaque; + size_t idx; + + for (idx =3D 0; idx < v->shadow_vqs->len; ++idx) { + vhost_svq_stop(dev, idx, g_ptr_array_index(v->shadow_vqs, idx)); + } + g_ptr_array_free(v->shadow_vqs, true); +} + static int vhost_vdpa_cleanup(struct vhost_dev *dev) { struct vhost_vdpa *v; @@ -381,6 +404,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev) trace_vhost_vdpa_cleanup(dev, v); vhost_vdpa_host_notifiers_uninit(dev, dev->nvqs); memory_listener_unregister(&v->listener); + vhost_vdpa_svq_cleanup(dev); QLIST_REMOVE(v, entry); =20 dev->opaque =3D NULL; @@ -557,7 +581,9 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, = bool started) if (started) { uint8_t status =3D 0; memory_listener_register(&v->listener, &address_space_memory); - vhost_vdpa_host_notifiers_init(dev); + if (!v->shadow_vqs_enabled) { + vhost_vdpa_host_notifiers_init(dev); + } vhost_vdpa_set_vring_ready(dev); vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK); vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status); @@ -663,10 +689,96 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev = *dev) return true; } =20 +/* + * Start shadow virtqueue. + */ +static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) +{ + struct vhost_vdpa *v =3D dev->opaque; + VhostShadowVirtqueue *svq =3D g_ptr_array_index(v->shadow_vqs, idx); + return vhost_svq_start(dev, idx, svq); +} + +static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) +{ + struct vhost_dev *hdev =3D v->dev; + unsigned n; + + if (enable =3D=3D v->shadow_vqs_enabled) { + return hdev->nvqs; + } + + if (enable) { + /* Allocate resources */ + assert(v->shadow_vqs->len =3D=3D 0); + for (n =3D 0; n < hdev->nvqs; ++n) { + VhostShadowVirtqueue *svq =3D vhost_svq_new(hdev, n); + bool ok; + + if (unlikely(!svq)) { + g_ptr_array_set_size(v->shadow_vqs, 0); + return 0; + } + g_ptr_array_add(v->shadow_vqs, svq); + + ok =3D vhost_vdpa_svq_start_vq(hdev, n); + if (unlikely(!ok)) { + /* Free still not started svqs */ + g_ptr_array_set_size(v->shadow_vqs, n); + enable =3D false; + break; + } + } + } + + v->shadow_vqs_enabled =3D enable; + + if (!enable) { + /* Disable all queues or clean up failed start */ + for (n =3D 0; n < v->shadow_vqs->len; ++n) { + unsigned vq_idx =3D vhost_vdpa_get_vq_index(hdev, n); + VhostShadowVirtqueue *svq =3D g_ptr_array_index(v->shadow_vqs,= n); + vhost_svq_stop(hdev, n, svq); + vhost_virtqueue_start(hdev, hdev->vdev, &hdev->vqs[n], vq_idx); + } + + /* Resources cleanup */ + g_ptr_array_set_size(v->shadow_vqs, 0); + } + + return n; +} =20 void qmp_x_vhost_enable_shadow_vq(const char *name, bool enable, Error **e= rrp) { - error_setg(errp, "Shadow virtqueue still not implemented"); + struct vhost_vdpa *v; + const char *err_cause =3D NULL; + bool r; + + QLIST_FOREACH(v, &vhost_vdpa_devices, entry) { + if (v->dev->vdev && 0 =3D=3D strcmp(v->dev->vdev->name, name)) { + break; + } + } + + if (!v) { + err_cause =3D "Device not found"; + goto err; + } else if (v->notifier[0].addr) { + err_cause =3D "Device has host notifiers enabled"; + goto err; + } + + r =3D vhost_vdpa_enable_svq(v, enable); + if (unlikely(!r)) { + err_cause =3D "Error enabling (see monitor)"; + goto err; + } + +err: + if (err_cause) { + error_setg(errp, "Can't enable shadow vq on %s: %s", name, err_cau= se); + } } =20 const VhostOps vdpa_ops =3D { --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072450713904.7453177694113; Fri, 1 Oct 2021 00:14:10 -0700 (PDT) Received: from localhost ([::1]:51814 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCkD-0006w6-OI for importer@patchew.org; Fri, 01 Oct 2021 03:14:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36318) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCgL-0007wO-Uc for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:10:10 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:42454) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCgK-0003JG-Dn for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:10:09 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-494-NFEs3XFyO-qF1HRWRxbDmw-1; Fri, 01 Oct 2021 03:10:06 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6BA7150751; Fri, 1 Oct 2021 07:10:05 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id A5D85652A3; Fri, 1 Oct 2021 07:09:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072207; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Uk9LgPGhlT5lTNw4EN8/+0R3NDtMJI5HobhN7STl1W8=; b=bvRaKcG8JflRFM/ukFJw0vp8g+fH774ZXRoFBXiA0RNjMepZ1UJZV+DwWAphGhXvZnkxKK OcCFRaj1StsIGW3u3ThM52p94kWECM5soJrBQBr1xLeeuYJiCwzJxUqDHAHwVDeiYzWAlK CETrhaF6ZuXwrZTdBNo5dl8GyL7l0N4= X-MC-Unique: NFEs3XFyO-qF1HRWRxbDmw-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 09/20] vdpa: Save call_fd in vhost-vdpa Date: Fri, 1 Oct 2021 09:05:52 +0200 Message-Id: <20211001070603.307037-10-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072451372100001 We need to know it to switch to Shadow VirtQueue. Signed-off-by: Eugenio P=C3=A9rez --- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 48aae59d8e..fddac248b3 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -30,6 +30,8 @@ typedef struct vhost_vdpa { GPtrArray *shadow_vqs; struct vhost_dev *dev; QLIST_ENTRY(vhost_vdpa) entry; + /* File descriptor the device uses to call VM/SVQ */ + int call_fd[VIRTIO_QUEUE_MAX]; VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX]; } VhostVDPA; =20 diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 36c954a779..57a857444a 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -652,7 +652,12 @@ static int vhost_vdpa_set_vring_kick(struct vhost_dev = *dev, static int vhost_vdpa_set_vring_call(struct vhost_dev *dev, struct vhost_vring_file *file) { + struct vhost_vdpa *v =3D dev->opaque; + int vdpa_idx =3D vhost_vdpa_get_vq_index(dev, file->index); + trace_vhost_vdpa_set_vring_call(dev, file->index, file->fd); + + v->call_fd[vdpa_idx] =3D file->fd; return vhost_vdpa_call(dev, VHOST_SET_VRING_CALL, file); } =20 --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072637811187.8886481912948; Fri, 1 Oct 2021 00:17:17 -0700 (PDT) Received: from localhost ([::1]:60358 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCnE-0004zg-Nb for importer@patchew.org; Fri, 01 Oct 2021 03:17:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36402) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCh7-0000EJ-7g for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:10:58 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:56848) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCh5-00040h-Lw for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:10:56 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-344-7_PWmqmxN9ujmAS4y1AgJA-1; Fri, 01 Oct 2021 03:10:54 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D344C800053; Fri, 1 Oct 2021 07:10:52 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id C3EFE652B1; Fri, 1 Oct 2021 07:10:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=okylH1txMGPK+6wTEQCtSCWo6poZiq+E83xdhql8ca8=; b=hisSXGHET0424fSnceMXlD72d0rM/ueYaqIRt2Sj9hdQEs5q7JiluozmiJMjwz+WeXy67i dKGqEdacKUmveNMVbHgssBYYjUzuEALzto0cN9OVJlmCXVn+hVCfYajgCzkb8BgVawgj4H O1E6HW0G+YThpOWUPyS55W5MkJOXkgo= X-MC-Unique: 7_PWmqmxN9ujmAS4y1AgJA-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 10/20] vhost-vdpa: Take into account SVQ in vhost_vdpa_set_vring_call Date: Fri, 1 Oct 2021 09:05:53 +0200 Message-Id: <20211001070603.307037-11-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072638752100001 Signed-off-by: Eugenio P=C3=A9rez --- hw/virtio/vhost-vdpa.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 57a857444a..bc34de2439 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -649,16 +649,27 @@ static int vhost_vdpa_set_vring_kick(struct vhost_dev= *dev, return vhost_vdpa_call(dev, VHOST_SET_VRING_KICK, file); } =20 +static int vhost_vdpa_set_vring_dev_call(struct vhost_dev *dev, + struct vhost_vring_file *file) +{ + trace_vhost_vdpa_set_vring_call(dev, file->index, file->fd); + return vhost_vdpa_call(dev, VHOST_SET_VRING_CALL, file); +} + static int vhost_vdpa_set_vring_call(struct vhost_dev *dev, struct vhost_vring_file *file) { struct vhost_vdpa *v =3D dev->opaque; int vdpa_idx =3D vhost_vdpa_get_vq_index(dev, file->index); =20 - trace_vhost_vdpa_set_vring_call(dev, file->index, file->fd); - v->call_fd[vdpa_idx] =3D file->fd; - return vhost_vdpa_call(dev, VHOST_SET_VRING_CALL, file); + if (v->shadow_vqs_enabled) { + VhostShadowVirtqueue *svq =3D g_ptr_array_index(v->shadow_vqs, vdp= a_idx); + vhost_svq_set_guest_call_notifier(svq, file->fd); + return 0; + } else { + return vhost_vdpa_set_vring_dev_call(dev, file); + } } =20 static int vhost_vdpa_get_features(struct vhost_dev *dev, --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633073045572902.8358132988934; Fri, 1 Oct 2021 00:24:05 -0700 (PDT) Received: from localhost ([::1]:49148 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCto-000073-ES for importer@patchew.org; Fri, 01 Oct 2021 03:24:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36434) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWChH-0000N2-Fk for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:11:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:45116) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWChF-00049a-8J for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:11:06 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-504-VVFSyDVeNS2wi_5T09gClA-1; Fri, 01 Oct 2021 03:11:03 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3872D10151E0; Fri, 1 Oct 2021 07:11:02 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 34B1053E08; Fri, 1 Oct 2021 07:10:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072264; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4FMWHKJqRbXnJMl22nuaafL4UiVJzbsZAYB/Jd90dFw=; b=BUkmQ9xeBlkfNKaPGPMgtqa46DCxI3ZxGluDUT2nci42PCUxvVvP7m/OH1V+IP78KpTgH+ +uAY9vtkxtFUkl5VW3kCUjcrpveUeZYJU0X0ab6zDCJEsDhx9yatYcAhCG/UXGrRiDV0M8 9qDHWra0um8AzhyDe/zWXSi72CQcasE= X-MC-Unique: VVFSyDVeNS2wi_5T09gClA-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 11/20] vhost: Route host->guest notification through shadow virtqueue Date: Fri, 1 Oct 2021 09:05:54 +0200 Message-Id: <20211001070603.307037-12-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633073046983100003 This will make qemu aware of the device used buffers, allowing it to write the guest memory with its contents if needed. Since the use of vhost_virtqueue_start can unmasks and discard call events, vhost_virtqueue_start should be modified in one of these ways: * Split in two: One of them uses all logic to start a queue with no side effects for the guest, and another one tha actually assumes that the guest has just started the device. Vdpa should use just the former. * Actually store and check if the guest notifier is masked, and do it conditionally. * Left as it is, and duplicate all the logic in vhost-vdpa. Signed-off-by: Eugenio P=C3=A9rez --- hw/virtio/vhost-shadow-virtqueue.c | 19 +++++++++++++++ hw/virtio/vhost-vdpa.c | 38 +++++++++++++++++++++++++++++- 2 files changed, 56 insertions(+), 1 deletion(-) diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-vi= rtqueue.c index 21dc99ab5d..3fe129cf63 100644 --- a/hw/virtio/vhost-shadow-virtqueue.c +++ b/hw/virtio/vhost-shadow-virtqueue.c @@ -53,6 +53,22 @@ static void vhost_handle_guest_kick(EventNotifier *n) event_notifier_set(&svq->kick_notifier); } =20 +/* Forward vhost notifications */ +static void vhost_svq_handle_call_no_test(EventNotifier *n) +{ + VhostShadowVirtqueue *svq =3D container_of(n, VhostShadowVirtqueue, + call_notifier); + + event_notifier_set(&svq->guest_call_notifier); +} + +static void vhost_svq_handle_call(EventNotifier *n) +{ + if (likely(event_notifier_test_and_clear(n))) { + vhost_svq_handle_call_no_test(n); + } +} + /* * Obtain the SVQ call notifier, where vhost device notifies SVQ that there * exists pending used buffers. @@ -180,6 +196,8 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *d= ev, int idx) } =20 svq->vq =3D virtio_get_queue(dev->vdev, vq_idx); + event_notifier_set_handler(&svq->call_notifier, + vhost_svq_handle_call); return g_steal_pointer(&svq); =20 err_init_call_notifier: @@ -195,6 +213,7 @@ err_init_kick_notifier: void vhost_svq_free(VhostShadowVirtqueue *vq) { event_notifier_cleanup(&vq->kick_notifier); + event_notifier_set_handler(&vq->call_notifier, NULL); event_notifier_cleanup(&vq->call_notifier); g_free(vq); } diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index bc34de2439..6c5f4c98b8 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -712,13 +712,40 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev = *dev, unsigned idx) { struct vhost_vdpa *v =3D dev->opaque; VhostShadowVirtqueue *svq =3D g_ptr_array_index(v->shadow_vqs, idx); - return vhost_svq_start(dev, idx, svq); + EventNotifier *vhost_call_notifier =3D vhost_svq_get_svq_call_notifier= (svq); + struct vhost_vring_file vhost_call_file =3D { + .index =3D idx + dev->vq_index, + .fd =3D event_notifier_get_fd(vhost_call_notifier), + }; + int r; + bool b; + + /* Set shadow vq -> guest notifier */ + assert(v->call_fd[idx]); + vhost_svq_set_guest_call_notifier(svq, v->call_fd[idx]); + + b =3D vhost_svq_start(dev, idx, svq); + if (unlikely(!b)) { + return false; + } + + /* Set device -> SVQ notifier */ + r =3D vhost_vdpa_set_vring_dev_call(dev, &vhost_call_file); + if (unlikely(r)) { + error_report("vhost_vdpa_set_vring_call for shadow vq failed"); + return false; + } + + /* Check for pending calls */ + event_notifier_set(vhost_call_notifier); + return true; } =20 static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) { struct vhost_dev *hdev =3D v->dev; unsigned n; + int r; =20 if (enable =3D=3D v->shadow_vqs_enabled) { return hdev->nvqs; @@ -752,9 +779,18 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdp= a *v, bool enable) if (!enable) { /* Disable all queues or clean up failed start */ for (n =3D 0; n < v->shadow_vqs->len; ++n) { + struct vhost_vring_file file =3D { + .index =3D vhost_vdpa_get_vq_index(hdev, n), + .fd =3D v->call_fd[n], + }; + + r =3D vhost_vdpa_set_vring_call(hdev, &file); + assert(r =3D=3D 0); + unsigned vq_idx =3D vhost_vdpa_get_vq_index(hdev, n); VhostShadowVirtqueue *svq =3D g_ptr_array_index(v->shadow_vqs,= n); vhost_svq_stop(hdev, n, svq); + /* TODO: This can unmask or override call fd! */ vhost_virtqueue_start(hdev, hdev->vdev, &hdev->vqs[n], vq_idx); } =20 --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072658533581.6769307355518; Fri, 1 Oct 2021 00:17:38 -0700 (PDT) Received: from localhost ([::1]:33068 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCnZ-0005bc-Dj for importer@patchew.org; Fri, 01 Oct 2021 03:17:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36452) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWChQ-0000Q5-Iy for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:11:17 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:21952) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWChO-0004Jy-Uy for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:11:16 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-510-AeJtO8dSNXKVmdHZOODiEw-1; Fri, 01 Oct 2021 03:11:13 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D5AC7824FA6; Fri, 1 Oct 2021 07:11:11 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8F2D319811; Fri, 1 Oct 2021 07:11:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072274; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=okT+K7yCEHpbCZs4NzHTB6jwBt2kx5s0YjoEQl1sNhM=; b=d415CMCYrGABIV6ZQsdFSTK+KrNKleSkRnTaP7hyXV/KtxlK5om1D48zEvihpPJTDX7LC1 HxFubX29kvboFo5e3qJQYdnQn1IDxwiT3PpK+tUxtHooPl0mjmuh56ZSBcQdooq6vJlTkd lVevqbc1oQRXPGFYOvirg9Arhi77DYI= X-MC-Unique: AeJtO8dSNXKVmdHZOODiEw-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 12/20] virtio: Add vhost_shadow_vq_get_vring_addr Date: Fri, 1 Oct 2021 09:05:55 +0200 Message-Id: <20211001070603.307037-13-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072660015100001 It reports the shadow virtqueue address from qemu virtual address space Signed-off-by: Eugenio P=C3=A9rez --- hw/virtio/vhost-shadow-virtqueue.h | 4 +++ hw/virtio/vhost-shadow-virtqueue.c | 50 ++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/hw/virtio/vhost-shadow-virtqueue.h b/hw/virtio/vhost-shadow-vi= rtqueue.h index 237cfceb9c..2df3d117f5 100644 --- a/hw/virtio/vhost-shadow-virtqueue.h +++ b/hw/virtio/vhost-shadow-virtqueue.h @@ -16,6 +16,10 @@ typedef struct VhostShadowVirtqueue VhostShadowVirtqueue; =20 EventNotifier *vhost_svq_get_svq_call_notifier(VhostShadowVirtqueue *svq); void vhost_svq_set_guest_call_notifier(VhostShadowVirtqueue *svq, int call= _fd); +void vhost_svq_get_vring_addr(const VhostShadowVirtqueue *svq, + struct vhost_vring_addr *addr); +size_t vhost_svq_driver_area_size(const VhostShadowVirtqueue *svq); +size_t vhost_svq_device_area_size(const VhostShadowVirtqueue *svq); =20 bool vhost_svq_start(struct vhost_dev *dev, unsigned idx, VhostShadowVirtqueue *svq); diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-vi= rtqueue.c index 3fe129cf63..5c1899f6af 100644 --- a/hw/virtio/vhost-shadow-virtqueue.c +++ b/hw/virtio/vhost-shadow-virtqueue.c @@ -18,6 +18,9 @@ =20 /* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue { + /* Shadow vring */ + struct vring vring; + /* Shadow kick notifier, sent to vhost */ EventNotifier kick_notifier; /* Shadow call notifier, sent to vhost */ @@ -38,6 +41,9 @@ typedef struct VhostShadowVirtqueue { =20 /* Virtio queue shadowing */ VirtQueue *vq; + + /* Virtio device */ + VirtIODevice *vdev; } VhostShadowVirtqueue; =20 /* Forward guest notifications */ @@ -93,6 +99,35 @@ void vhost_svq_set_guest_call_notifier(VhostShadowVirtqu= eue *svq, int call_fd) event_notifier_init_fd(&svq->guest_call_notifier, call_fd); } =20 +/* + * Get the shadow vq vring address. + * @svq Shadow virtqueue + * @addr Destination to store address + */ +void vhost_svq_get_vring_addr(const VhostShadowVirtqueue *svq, + struct vhost_vring_addr *addr) +{ + addr->desc_user_addr =3D (uint64_t)svq->vring.desc; + addr->avail_user_addr =3D (uint64_t)svq->vring.avail; + addr->used_user_addr =3D (uint64_t)svq->vring.used; +} + +size_t vhost_svq_driver_area_size(const VhostShadowVirtqueue *svq) +{ + uint16_t vq_idx =3D virtio_get_queue_index(svq->vq); + size_t desc_size =3D virtio_queue_get_desc_size(svq->vdev, vq_idx); + size_t avail_size =3D virtio_queue_get_avail_size(svq->vdev, vq_idx); + + return ROUND_UP(desc_size + avail_size, qemu_real_host_page_size); +} + +size_t vhost_svq_device_area_size(const VhostShadowVirtqueue *svq) +{ + uint16_t vq_idx =3D virtio_get_queue_index(svq->vq); + size_t used_size =3D virtio_queue_get_used_size(svq->vdev, vq_idx); + return ROUND_UP(used_size, qemu_real_host_page_size); +} + /* * Restore the vhost guest to host notifier, i.e., disables svq effect. */ @@ -178,6 +213,10 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned id= x, VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) { int vq_idx =3D dev->vq_index + idx; + unsigned num =3D virtio_queue_get_num(dev->vdev, vq_idx); + size_t desc_size =3D virtio_queue_get_desc_size(dev->vdev, vq_idx); + size_t driver_size; + size_t device_size; g_autofree VhostShadowVirtqueue *svq =3D g_new0(VhostShadowVirtqueue, = 1); int r; =20 @@ -196,6 +235,15 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *= dev, int idx) } =20 svq->vq =3D virtio_get_queue(dev->vdev, vq_idx); + svq->vdev =3D dev->vdev; + driver_size =3D vhost_svq_driver_area_size(svq); + device_size =3D vhost_svq_device_area_size(svq); + svq->vring.num =3D num; + svq->vring.desc =3D qemu_memalign(qemu_real_host_page_size, driver_siz= e); + svq->vring.avail =3D (void *)((char *)svq->vring.desc + desc_size); + memset(svq->vring.desc, 0, driver_size); + svq->vring.used =3D qemu_memalign(qemu_real_host_page_size, device_siz= e); + memset(svq->vring.used, 0, device_size); event_notifier_set_handler(&svq->call_notifier, vhost_svq_handle_call); return g_steal_pointer(&svq); @@ -215,5 +263,7 @@ void vhost_svq_free(VhostShadowVirtqueue *vq) event_notifier_cleanup(&vq->kick_notifier); event_notifier_set_handler(&vq->call_notifier, NULL); event_notifier_cleanup(&vq->call_notifier); + qemu_vfree(vq->vring.desc); + qemu_vfree(vq->vring.used); g_free(vq); } --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633072888342494.6517443854543; Fri, 1 Oct 2021 00:21:28 -0700 (PDT) Received: from localhost ([::1]:41574 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCrG-0003ED-Pi for importer@patchew.org; Fri, 01 Oct 2021 03:21:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36540) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWChq-0001Qs-Gz for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:11:42 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:29674) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCho-0004eV-RF for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:11:42 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-335-8RVtreyTM0iKYNvTSX8e-g-1; Fri, 01 Oct 2021 03:11:39 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B58E7801E72; Fri, 1 Oct 2021 07:11:37 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3ABAC652A3; Fri, 1 Oct 2021 07:11:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u2JUZbCJP9MWb2hD4MtOi5qn59JQrKiwOlPWFUeqOe4=; b=YlY9x4kRJjN1PSOBE3/eIw7kL09viEv6MseXNCo5G7nXUbeFI6A/1PRShz98eBdU4UYyc0 6ONvSWBxnen1WVjy+wZdOyUEex2fSKmU6fKlFW4pl8AcyQ4iKrgQxFlbiaUAu9Z1dYpj6R hHnDCTY11y1pLHJlnJioEaEOOr9PzKI= X-MC-Unique: 8RVtreyTM0iKYNvTSX8e-g-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 13/20] vdpa: Save host and guest features Date: Fri, 1 Oct 2021 09:05:56 +0200 Message-Id: <20211001070603.307037-14-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633072889252100001 Those are needed for SVQ: Host ones are needed to check if SVQ knows how to talk with the device and for feature negotiation, and guest ones to know if SVQ can talk with it. Signed-off-by: Eugenio P=C3=A9rez --- include/hw/virtio/vhost-vdpa.h | 2 ++ hw/virtio/vhost-vdpa.c | 31 ++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index fddac248b3..9044ae694b 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -26,6 +26,8 @@ typedef struct vhost_vdpa { int device_fd; uint32_t msg_type; MemoryListener listener; + uint64_t host_features; + uint64_t guest_features; bool shadow_vqs_enabled; GPtrArray *shadow_vqs; struct vhost_dev *dev; diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 6c5f4c98b8..a057e8277d 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -439,10 +439,19 @@ static int vhost_vdpa_set_mem_table(struct vhost_dev = *dev, return 0; } =20 -static int vhost_vdpa_set_features(struct vhost_dev *dev, - uint64_t features) +/** + * Internal set_features() that follows vhost/VirtIO protocol for that + */ +static int vhost_vdpa_backend_set_features(struct vhost_dev *dev, + uint64_t features) { + struct vhost_vdpa *v =3D dev->opaque; + int ret; + if (v->host_features & BIT_ULL(VIRTIO_F_QUEUE_STATE)) { + features |=3D BIT_ULL(VIRTIO_F_QUEUE_STATE); + } + trace_vhost_vdpa_set_features(dev, features); ret =3D vhost_vdpa_call(dev, VHOST_SET_FEATURES, &features); uint8_t status =3D 0; @@ -455,6 +464,17 @@ static int vhost_vdpa_set_features(struct vhost_dev *d= ev, return !(status & VIRTIO_CONFIG_S_FEATURES_OK); } =20 +/** + * Exposed vhost set features + */ +static int vhost_vdpa_set_features(struct vhost_dev *dev, + uint64_t features) +{ + struct vhost_vdpa *v =3D dev->opaque; + v->guest_features =3D features; + return vhost_vdpa_backend_set_features(dev, features); +} + static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) { uint64_t features; @@ -673,12 +693,17 @@ static int vhost_vdpa_set_vring_call(struct vhost_dev= *dev, } =20 static int vhost_vdpa_get_features(struct vhost_dev *dev, - uint64_t *features) + uint64_t *features) { int ret; =20 ret =3D vhost_vdpa_call(dev, VHOST_GET_FEATURES, features); trace_vhost_vdpa_get_features(dev, *features); + + if (ret =3D=3D 0) { + struct vhost_vdpa *v =3D dev->opaque; + v->host_features =3D *features; + } return ret; } =20 --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633073049287921.095782386299; Fri, 1 Oct 2021 00:24:09 -0700 (PDT) Received: from localhost ([::1]:49378 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCts-0000Hi-5q for importer@patchew.org; Fri, 01 Oct 2021 03:24:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36596) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCi5-0002Fl-Fr for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:11:57 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:21777) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCi2-0004qd-Rx for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:11:57 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-572-pG8fovN4PkK46srYuO1tLg-1; Fri, 01 Oct 2021 03:11:53 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0BCB018D6A2A; Fri, 1 Oct 2021 07:11:52 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 17BA3652A3; Fri, 1 Oct 2021 07:11:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v4fBz87yvAJtN9GOUUmJoLj969+76+cO613TUa4xtY4=; b=DEtfbo4VPKiyYoT49H9OODM86UZMiOsYFpxm590os9s1jPsZzqpDqll3skHRS4ScB4xHX7 9SWiuYferjXqEd0tluMwqzM//DwYiQX1vzJzTbYj30UBFNlMTxdjB2ecW1T6YzxFu5lcoi bVWGeSGxMEqyWyzCrYY6xYQ+3yowoD0= X-MC-Unique: pG8fovN4PkK46srYuO1tLg-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 14/20] vhost: Add vhost_svq_valid_device_features to shadow vq Date: Fri, 1 Oct 2021 09:05:57 +0200 Message-Id: <20211001070603.307037-15-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633073051149100001 This allows it to test if the guest has aknowledge an invalid transport feature for SVQ. This will include packed vq layout, invalid descriptors or event idx at the moment we start forwarding buffers. We don't check for device features here since they will be re-negotiated again. This allows SVQ to both use more advanced features of the device when they are available and the guest is not capable of run them, and to make SVQ compatible with future transport features. Signed-off-by: Eugenio P=C3=A9rez --- hw/virtio/vhost-shadow-virtqueue.h | 2 ++ hw/virtio/vhost-shadow-virtqueue.c | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/hw/virtio/vhost-shadow-virtqueue.h b/hw/virtio/vhost-shadow-vi= rtqueue.h index 2df3d117f5..b7baa424a7 100644 --- a/hw/virtio/vhost-shadow-virtqueue.h +++ b/hw/virtio/vhost-shadow-virtqueue.h @@ -14,6 +14,8 @@ =20 typedef struct VhostShadowVirtqueue VhostShadowVirtqueue; =20 +bool vhost_svq_valid_device_features(uint64_t *features); + EventNotifier *vhost_svq_get_svq_call_notifier(VhostShadowVirtqueue *svq); void vhost_svq_set_guest_call_notifier(VhostShadowVirtqueue *svq, int call= _fd); void vhost_svq_get_vring_addr(const VhostShadowVirtqueue *svq, diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-vi= rtqueue.c index 5c1899f6af..34e159d4fd 100644 --- a/hw/virtio/vhost-shadow-virtqueue.c +++ b/hw/virtio/vhost-shadow-virtqueue.c @@ -46,6 +46,12 @@ typedef struct VhostShadowVirtqueue { VirtIODevice *vdev; } VhostShadowVirtqueue; =20 +/* If the device is using some of these, SVQ cannot communicate */ +bool vhost_svq_valid_device_features(uint64_t *dev_features) +{ + return true; +} + /* Forward guest notifications */ static void vhost_handle_guest_kick(EventNotifier *n) { --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633073196529461.4441067183835; Fri, 1 Oct 2021 00:26:36 -0700 (PDT) Received: from localhost ([::1]:57202 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCwF-0005n3-Fg for importer@patchew.org; Fri, 01 Oct 2021 03:26:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36648) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCiM-0002gS-Ab for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:12:14 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:47385) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCiH-00051Q-4D for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:12:13 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-374-_uM6TyHoOBS39GkzzPnb_A-1; Fri, 01 Oct 2021 03:12:05 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 026D8800053; Fri, 1 Oct 2021 07:12:04 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 509C7652A3; Fri, 1 Oct 2021 07:11:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072328; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rYKHfgCNiQQogGwEEeihk1SDwP+VTSklTztWL96nT30=; b=IIkxJzN8gcfT+rj+Roq8vtEG3KMNUg7VybZ2odai3/Q/oL58RCCEIr2AFtxHKtCS6GigNB WgeFP7dm03RiDjSnNSlIv/8l8IHnHfUSZ7v2oksEjXvrqFqS5fhkNZD/Uui1Flr6mtqtis m990r71gkJGutlh+NsvE7DiNTP+f0Lg= X-MC-Unique: _uM6TyHoOBS39GkzzPnb_A-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 15/20] vhost: Shadow virtqueue buffers forwarding Date: Fri, 1 Oct 2021 09:05:58 +0200 Message-Id: <20211001070603.307037-16-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633073197965100001 Initial version of shadow virtqueue that actually forward buffers. There are no iommu support at the moment, and that will be addressed in future patches of this series. Since all vhost-vdpa devices uses forced IOMMU, this means that SVQ is not usable at this point of the series on any device. For simplicity it only supports modern devices, that expects vring in little endian, with split ring and no event idx or indirect descriptors. Support for them will not be added in this series. It reuses the VirtQueue code for the device part. The driver part is based on Linux's virtio_ring driver, but with stripped functionality and optimizations so it's easier to review. Later commits add simpler ones. SVQ uses VIRTIO_CONFIG_S_DEVICE_STOPPED to pause the device and retrieve its status (next available idx the device was going to consume) race-free. It can later reset the device to replace vring addresses etc. When SVQ starts qemu can resume consuming the guest's driver ring from that state, without notice from the latter. This status bit VIRTIO_CONFIG_S_DEVICE_STOPPED is currently discussed in VirtIO, and is implemented in qemu VirtIO-net devices in previous commits. Removal of _S_DEVICE_STOPPED bit (in other words, resuming the device) can be done in the future if an use case arises. At this moment we can just rely on reseting the full device. Signed-off-by: Eugenio P=C3=A9rez --- qapi/net.json | 2 +- hw/virtio/vhost-shadow-virtqueue.c | 237 ++++++++++++++++++++++++++++- hw/virtio/vhost-vdpa.c | 109 ++++++++++++- 3 files changed, 337 insertions(+), 11 deletions(-) diff --git a/qapi/net.json b/qapi/net.json index fe546b0e7c..1f4a55f2c5 100644 --- a/qapi/net.json +++ b/qapi/net.json @@ -86,7 +86,7 @@ # # @name: the device name of the VirtIO device # -# @enable: true to use the alternate shadow VQ notifications +# @enable: true to use the alternate shadow VQ buffers fowarding path # # Returns: Error if failure, or 'no error' for success. # diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-vi= rtqueue.c index 34e159d4fd..df7e6fa3ec 100644 --- a/hw/virtio/vhost-shadow-virtqueue.c +++ b/hw/virtio/vhost-shadow-virtqueue.c @@ -10,6 +10,7 @@ #include "qemu/osdep.h" #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" +#include "hw/virtio/virtio-access.h" =20 #include "standard-headers/linux/vhost_types.h" =20 @@ -44,15 +45,135 @@ typedef struct VhostShadowVirtqueue { =20 /* Virtio device */ VirtIODevice *vdev; + + /* Map for returning guest's descriptors */ + VirtQueueElement **ring_id_maps; + + /* Next head to expose to device */ + uint16_t avail_idx_shadow; + + /* Next free descriptor */ + uint16_t free_head; + + /* Last seen used idx */ + uint16_t shadow_used_idx; + + /* Next head to consume from device */ + uint16_t used_idx; } VhostShadowVirtqueue; =20 /* If the device is using some of these, SVQ cannot communicate */ bool vhost_svq_valid_device_features(uint64_t *dev_features) { - return true; + uint64_t b; + bool r =3D true; + + for (b =3D VIRTIO_TRANSPORT_F_START; b <=3D VIRTIO_TRANSPORT_F_END; ++= b) { + switch (b) { + case VIRTIO_F_NOTIFY_ON_EMPTY: + case VIRTIO_F_ANY_LAYOUT: + /* SVQ is fine with this feature */ + continue; + + case VIRTIO_F_ACCESS_PLATFORM: + /* SVQ needs this feature disabled. Can't continue */ + if (*dev_features & BIT_ULL(b)) { + clear_bit(b, dev_features); + r =3D false; + } + break; + + case VIRTIO_F_VERSION_1: + /* SVQ needs this feature, so can't continue */ + if (!(*dev_features & BIT_ULL(b))) { + set_bit(b, dev_features); + r =3D false; + } + continue; + + default: + /* + * SVQ must disable this feature, let's hope the device is fine + * without it. + */ + if (*dev_features & BIT_ULL(b)) { + clear_bit(b, dev_features); + } + } + } + + return r; +} + +static void vhost_vring_write_descs(VhostShadowVirtqueue *svq, + const struct iovec *iovec, + size_t num, bool more_descs, bool writ= e) +{ + uint16_t i =3D svq->free_head, last =3D svq->free_head; + unsigned n; + uint16_t flags =3D write ? cpu_to_le16(VRING_DESC_F_WRITE) : 0; + vring_desc_t *descs =3D svq->vring.desc; + + if (num =3D=3D 0) { + return; + } + + for (n =3D 0; n < num; n++) { + if (more_descs || (n + 1 < num)) { + descs[i].flags =3D flags | cpu_to_le16(VRING_DESC_F_NEXT); + } else { + descs[i].flags =3D flags; + } + descs[i].addr =3D cpu_to_le64((hwaddr)iovec[n].iov_base); + descs[i].len =3D cpu_to_le32(iovec[n].iov_len); + + last =3D i; + i =3D cpu_to_le16(descs[i].next); + } + + svq->free_head =3D le16_to_cpu(descs[last].next); +} + +static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, + VirtQueueElement *elem) +{ + int head; + unsigned avail_idx; + vring_avail_t *avail =3D svq->vring.avail; + + head =3D svq->free_head; + + /* We need some descriptors here */ + assert(elem->out_num || elem->in_num); + + vhost_vring_write_descs(svq, elem->out_sg, elem->out_num, + elem->in_num > 0, false); + vhost_vring_write_descs(svq, elem->in_sg, elem->in_num, false, true); + + /* + * Put entry in available array (but don't update avail->idx until they + * do sync). + */ + avail_idx =3D svq->avail_idx_shadow & (svq->vring.num - 1); + avail->ring[avail_idx] =3D cpu_to_le16(head); + svq->avail_idx_shadow++; + + /* Update avail index after the descriptor is wrote */ + smp_wmb(); + avail->idx =3D cpu_to_le16(svq->avail_idx_shadow); + + return head; + } =20 -/* Forward guest notifications */ +static void vhost_svq_add(VhostShadowVirtqueue *svq, VirtQueueElement *ele= m) +{ + unsigned qemu_head =3D vhost_svq_add_split(svq, elem); + + svq->ring_id_maps[qemu_head] =3D elem; +} + +/* Handle guest->device notifications */ static void vhost_handle_guest_kick(EventNotifier *n) { VhostShadowVirtqueue *svq =3D container_of(n, VhostShadowVirtqueue, @@ -62,7 +183,74 @@ static void vhost_handle_guest_kick(EventNotifier *n) return; } =20 - event_notifier_set(&svq->kick_notifier); + /* Make available as many buffers as possible */ + do { + if (virtio_queue_get_notification(svq->vq)) { + /* No more notifications until process all available */ + virtio_queue_set_notification(svq->vq, false); + } + + while (true) { + VirtQueueElement *elem =3D virtqueue_pop(svq->vq, sizeof(*elem= )); + if (!elem) { + break; + } + + vhost_svq_add(svq, elem); + event_notifier_set(&svq->kick_notifier); + } + + virtio_queue_set_notification(svq->vq, true); + } while (!virtio_queue_empty(svq->vq)); +} + +static bool vhost_svq_more_used(VhostShadowVirtqueue *svq) +{ + if (svq->used_idx !=3D svq->shadow_used_idx) { + return true; + } + + /* Get used idx must not be reordered */ + smp_rmb(); + svq->shadow_used_idx =3D cpu_to_le16(svq->vring.used->idx); + + return svq->used_idx !=3D svq->shadow_used_idx; +} + +static VirtQueueElement *vhost_svq_get_buf(VhostShadowVirtqueue *svq) +{ + vring_desc_t *descs =3D svq->vring.desc; + const vring_used_t *used =3D svq->vring.used; + vring_used_elem_t used_elem; + uint16_t last_used; + + if (!vhost_svq_more_used(svq)) { + return NULL; + } + + last_used =3D svq->used_idx & (svq->vring.num - 1); + used_elem.id =3D le32_to_cpu(used->ring[last_used].id); + used_elem.len =3D le32_to_cpu(used->ring[last_used].len); + + svq->used_idx++; + if (unlikely(used_elem.id >=3D svq->vring.num)) { + error_report("Device %s says index %u is used", svq->vdev->name, + used_elem.id); + return NULL; + } + + if (unlikely(!svq->ring_id_maps[used_elem.id])) { + error_report( + "Device %s says index %u is used, but it was not available", + svq->vdev->name, used_elem.id); + return NULL; + } + + descs[used_elem.id].next =3D svq->free_head; + svq->free_head =3D used_elem.id; + + svq->ring_id_maps[used_elem.id]->len =3D used_elem.len; + return g_steal_pointer(&svq->ring_id_maps[used_elem.id]); } =20 /* Forward vhost notifications */ @@ -70,8 +258,26 @@ static void vhost_svq_handle_call_no_test(EventNotifier= *n) { VhostShadowVirtqueue *svq =3D container_of(n, VhostShadowVirtqueue, call_notifier); - - event_notifier_set(&svq->guest_call_notifier); + VirtQueue *vq =3D svq->vq; + + /* Make as many buffers as possible used. */ + do { + unsigned i =3D 0; + + /* TODO: Use VRING_AVAIL_F_NO_INTERRUPT */ + while (true) { + g_autofree VirtQueueElement *elem =3D vhost_svq_get_buf(svq); + if (!elem) { + break; + } + + assert(i < svq->vring.num); + virtqueue_fill(vq, elem, elem->len, i++); + } + + virtqueue_flush(vq, i); + event_notifier_set(&svq->guest_call_notifier); + } while (vhost_svq_more_used(svq)); } =20 static void vhost_svq_handle_call(EventNotifier *n) @@ -204,12 +410,25 @@ err_set_vring_kick: void vhost_svq_stop(struct vhost_dev *dev, unsigned idx, VhostShadowVirtqueue *svq) { + int i; int r =3D vhost_svq_restore_vdev_host_notifier(dev, idx, svq); + if (unlikely(r < 0)) { error_report("Couldn't restore vq kick fd: %s", strerror(-r)); } =20 event_notifier_set_handler(&svq->host_notifier, NULL); + + for (i =3D 0; i < svq->vring.num; ++i) { + g_autofree VirtQueueElement *elem =3D svq->ring_id_maps[i]; + /* + * Although the doc says we must unpop in order, it's ok to unpop + * everything. + */ + if (elem) { + virtqueue_unpop(svq->vq, elem, elem->len); + } + } } =20 /* @@ -224,7 +443,7 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *d= ev, int idx) size_t driver_size; size_t device_size; g_autofree VhostShadowVirtqueue *svq =3D g_new0(VhostShadowVirtqueue, = 1); - int r; + int r, i; =20 r =3D event_notifier_init(&svq->kick_notifier, 0); if (r !=3D 0) { @@ -250,6 +469,11 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *= dev, int idx) memset(svq->vring.desc, 0, driver_size); svq->vring.used =3D qemu_memalign(qemu_real_host_page_size, device_siz= e); memset(svq->vring.used, 0, device_size); + for (i =3D 0; i < num - 1; i++) { + svq->vring.desc[i].next =3D cpu_to_le16(i + 1); + } + + svq->ring_id_maps =3D g_new0(VirtQueueElement *, num); event_notifier_set_handler(&svq->call_notifier, vhost_svq_handle_call); return g_steal_pointer(&svq); @@ -269,6 +493,7 @@ void vhost_svq_free(VhostShadowVirtqueue *vq) event_notifier_cleanup(&vq->kick_notifier); event_notifier_set_handler(&vq->call_notifier, NULL); event_notifier_cleanup(&vq->call_notifier); + g_free(vq->ring_id_maps); qemu_vfree(vq->vring.desc); qemu_vfree(vq->vring.used); g_free(vq); diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index a057e8277d..bb7010ddb5 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -19,6 +19,7 @@ #include "hw/virtio/virtio-net.h" #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost-vdpa.h" +#include "hw/virtio/vhost-shadow-virtqueue.h" #include "exec/address-spaces.h" #include "qemu/main-loop.h" #include "cpu.h" @@ -475,6 +476,28 @@ static int vhost_vdpa_set_features(struct vhost_dev *d= ev, return vhost_vdpa_backend_set_features(dev, features); } =20 +/** + * Restore guest features to vdpa device + */ +static int vhost_vdpa_set_guest_features(struct vhost_dev *dev) +{ + struct vhost_vdpa *v =3D dev->opaque; + return vhost_vdpa_backend_set_features(dev, v->guest_features); +} + +/** + * Set shadow virtqueue supported features + */ +static int vhost_vdpa_set_svq_features(struct vhost_dev *dev) +{ + struct vhost_vdpa *v =3D dev->opaque; + uint64_t features =3D v->host_features; + bool b =3D vhost_svq_valid_device_features(&features); + assert(b); + + return vhost_vdpa_backend_set_features(dev, features); +} + static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) { uint64_t features; @@ -730,6 +753,19 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *= dev) return true; } =20 +static int vhost_vdpa_vring_pause(struct vhost_dev *dev) +{ + int r; + uint8_t status; + + vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DEVICE_STOPPED); + do { + r =3D vhost_vdpa_call(dev, VHOST_VDPA_GET_STATUS, &status); + } while (r =3D=3D 0 && !(status & VIRTIO_CONFIG_S_DEVICE_STOPPED)); + + return 0; +} + /* * Start shadow virtqueue. */ @@ -742,9 +778,29 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *= dev, unsigned idx) .index =3D idx + dev->vq_index, .fd =3D event_notifier_get_fd(vhost_call_notifier), }; + struct vhost_vring_addr addr =3D { + .index =3D idx + dev->vq_index, + }; + struct vhost_vring_state num =3D { + .index =3D idx + dev->vq_index, + .num =3D virtio_queue_get_num(dev->vdev, idx), + }; int r; bool b; =20 + vhost_svq_get_vring_addr(svq, &addr); + r =3D vhost_vdpa_set_vring_addr(dev, &addr); + if (unlikely(r)) { + error_report("vhost_set_vring_addr for shadow vq failed"); + return false; + } + + r =3D vhost_vdpa_set_vring_num(dev, &num); + if (unlikely(r)) { + error_report("vhost_vdpa_set_vring_num for shadow vq failed"); + return false; + } + /* Set shadow vq -> guest notifier */ assert(v->call_fd[idx]); vhost_svq_set_guest_call_notifier(svq, v->call_fd[idx]); @@ -781,15 +837,32 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vd= pa *v, bool enable) assert(v->shadow_vqs->len =3D=3D 0); for (n =3D 0; n < hdev->nvqs; ++n) { VhostShadowVirtqueue *svq =3D vhost_svq_new(hdev, n); - bool ok; - if (unlikely(!svq)) { g_ptr_array_set_size(v->shadow_vqs, 0); return 0; } g_ptr_array_add(v->shadow_vqs, svq); + } + } =20 - ok =3D vhost_vdpa_svq_start_vq(hdev, n); + r =3D vhost_vdpa_vring_pause(hdev); + assert(r =3D=3D 0); + + if (enable) { + for (n =3D 0; n < v->shadow_vqs->len; ++n) { + /* Obtain Virtqueue state */ + vhost_virtqueue_stop(hdev, hdev->vdev, &hdev->vqs[n], n); + } + } + + /* Reset device so it can be configured */ + r =3D vhost_vdpa_dev_start(hdev, false); + assert(r =3D=3D 0); + + if (enable) { + int r; + for (n =3D 0; n < v->shadow_vqs->len; ++n) { + bool ok =3D vhost_vdpa_svq_start_vq(hdev, n); if (unlikely(!ok)) { /* Free still not started svqs */ g_ptr_array_set_size(v->shadow_vqs, n); @@ -797,11 +870,19 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vd= pa *v, bool enable) break; } } + + /* Need to ack features to set state in vp_vdpa devices */ + r =3D vhost_vdpa_set_svq_features(hdev); + if (unlikely(r)) { + enable =3D false; + } } =20 v->shadow_vqs_enabled =3D enable; =20 if (!enable) { + vhost_vdpa_set_guest_features(hdev); + /* Disable all queues or clean up failed start */ for (n =3D 0; n < v->shadow_vqs->len; ++n) { struct vhost_vring_file file =3D { @@ -818,7 +899,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdp= a *v, bool enable) /* TODO: This can unmask or override call fd! */ vhost_virtqueue_start(hdev, hdev->vdev, &hdev->vqs[n], vq_idx); } + } =20 + r =3D vhost_vdpa_dev_start(hdev, true); + assert(r =3D=3D 0); + + if (!enable) { /* Resources cleanup */ g_ptr_array_set_size(v->shadow_vqs, 0); } @@ -831,6 +917,7 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, boo= l enable, Error **errp) struct vhost_vdpa *v; const char *err_cause =3D NULL; bool r; + uint64_t svq_features; =20 QLIST_FOREACH(v, &vhost_vdpa_devices, entry) { if (v->dev->vdev && 0 =3D=3D strcmp(v->dev->vdev->name, name)) { @@ -846,6 +933,20 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, bo= ol enable, Error **errp) goto err; } =20 + svq_features =3D v->host_features; + if (!vhost_svq_valid_device_features(&svq_features)) { + error_setg(errp, + "Can't enable shadow vq on %s: Unexpected feature flags (%lx-%= lx)", + name, v->host_features, svq_features); + return; + } else { + /* TODO: Check for virtio_vdpa + IOMMU & modern device */ + } + + if (err_cause) { + goto err; + } + r =3D vhost_vdpa_enable_svq(v, enable); if (unlikely(!r)) { err_cause =3D "Error enabling (see monitor)"; @@ -853,7 +954,7 @@ void qmp_x_vhost_enable_shadow_vq(const char *name, boo= l enable, Error **errp) } =20 err: - if (err_cause) { + if (errp =3D=3D NULL && err_cause) { error_setg(errp, "Can't enable shadow vq on %s: %s", name, err_cau= se); } } --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 163307319128745.722431010763785; Fri, 1 Oct 2021 00:26:31 -0700 (PDT) Received: from localhost ([::1]:56742 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCwA-0005UK-9Q for importer@patchew.org; Fri, 01 Oct 2021 03:26:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36670) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCio-0004K6-S7 for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:12:43 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:36986) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCin-0005Pm-82 for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:12:42 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-250-3dRQ6-m-PaGfBzEIbwGrqQ-1; Fri, 01 Oct 2021 03:12:37 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9933218D6A35; Fri, 1 Oct 2021 07:12:36 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 58310652A3; Fri, 1 Oct 2021 07:12:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vzWYw5TdT4VDxLuD7rJPNEK27A6V1k2WSJKxtaO9E8Q=; b=EEQe23rkPF8eSoUe+AoNMklqb/X66gbzjbVNCZF7ODMJ45hjJ8WW5PNBpx3T9UtNs8qwwx SRkcR1SSEuk5Yukc0ajsnKCcgigw1ZIuKhyX7ArU/Gr99jJ7zrlfv1pr7a3EllcpnHW5ZY ZkjHnjdT0U+IE3VIAPBucp1j5x808P0= X-MC-Unique: 3dRQ6-m-PaGfBzEIbwGrqQ-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 16/20] vhost: Check for device VRING_USED_F_NO_NOTIFY at shadow virtqueue kick Date: Fri, 1 Oct 2021 09:05:59 +0200 Message-Id: <20211001070603.307037-17-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633073193435100003 Signed-off-by: Eugenio P=C3=A9rez --- hw/virtio/vhost-shadow-virtqueue.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-vi= rtqueue.c index df7e6fa3ec..775f8d36a0 100644 --- a/hw/virtio/vhost-shadow-virtqueue.c +++ b/hw/virtio/vhost-shadow-virtqueue.c @@ -173,6 +173,15 @@ static void vhost_svq_add(VhostShadowVirtqueue *svq, V= irtQueueElement *elem) svq->ring_id_maps[qemu_head] =3D elem; } =20 +static void vhost_svq_kick(VhostShadowVirtqueue *svq) +{ + /* Make sure we are reading updated device flag */ + smp_mb(); + if (!(svq->vring.used->flags & VRING_USED_F_NO_NOTIFY)) { + event_notifier_set(&svq->kick_notifier); + } +} + /* Handle guest->device notifications */ static void vhost_handle_guest_kick(EventNotifier *n) { @@ -197,7 +206,7 @@ static void vhost_handle_guest_kick(EventNotifier *n) } =20 vhost_svq_add(svq, elem); - event_notifier_set(&svq->kick_notifier); + vhost_svq_kick(svq); } =20 virtio_queue_set_notification(svq->vq, true); --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633073034426470.68797715911114; Fri, 1 Oct 2021 00:23:54 -0700 (PDT) Received: from localhost ([::1]:48446 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCtd-00085x-7V for importer@patchew.org; Fri, 01 Oct 2021 03:23:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36746) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCjM-0005Uu-KZ for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:13:16 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:52605) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCjL-0005q0-3a for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:13:16 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-400-W5YLp4aZNsSLyvgwVv0_Pw-1; Fri, 01 Oct 2021 03:13:12 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id DAA6F18D6A3E; Fri, 1 Oct 2021 07:13:10 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id F1761652A3; Fri, 1 Oct 2021 07:12:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072393; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YxfcG9Hve63YPITjYpliIf6LuctyQ6i/mHcmM3wdCOw=; b=Z+LjKCQVSRPwfv4xe0WuKR4ECo6KLpUiFR+MhdY/8PYsInJn9tUAA9Vg9pfwD0h6ckbwiY ROn0qytD71uJLwBMFf9ZjmHNw4n++DBxt/f4z0fbTkXab3DW/9C8urhknYP430DnSX8tTy pE/E5oeKHurOQBNZdMY1opO25pPLJog= X-MC-Unique: W5YLp4aZNsSLyvgwVv0_Pw-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 17/20] vhost: Use VRING_AVAIL_F_NO_INTERRUPT at device call on shadow virtqueue Date: Fri, 1 Oct 2021 09:06:00 +0200 Message-Id: <20211001070603.307037-18-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633073036303100001 Signed-off-by: Eugenio P=C3=A9rez --- hw/virtio/vhost-shadow-virtqueue.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-vi= rtqueue.c index 775f8d36a0..2fd0bab75d 100644 --- a/hw/virtio/vhost-shadow-virtqueue.c +++ b/hw/virtio/vhost-shadow-virtqueue.c @@ -60,6 +60,9 @@ typedef struct VhostShadowVirtqueue { =20 /* Next head to consume from device */ uint16_t used_idx; + + /* Cache for the exposed notification flag */ + bool notification; } VhostShadowVirtqueue; =20 /* If the device is using some of these, SVQ cannot communicate */ @@ -105,6 +108,24 @@ bool vhost_svq_valid_device_features(uint64_t *dev_fea= tures) return r; } =20 +static void vhost_svq_set_notification(VhostShadowVirtqueue *svq, bool ena= ble) +{ + uint16_t notification_flag; + + if (svq->notification =3D=3D enable) { + return; + } + + notification_flag =3D cpu_to_le16(VRING_AVAIL_F_NO_INTERRUPT); + + svq->notification =3D enable; + if (enable) { + svq->vring.avail->flags &=3D ~notification_flag; + } else { + svq->vring.avail->flags |=3D notification_flag; + } +} + static void vhost_vring_write_descs(VhostShadowVirtqueue *svq, const struct iovec *iovec, size_t num, bool more_descs, bool writ= e) @@ -273,7 +294,7 @@ static void vhost_svq_handle_call_no_test(EventNotifier= *n) do { unsigned i =3D 0; =20 - /* TODO: Use VRING_AVAIL_F_NO_INTERRUPT */ + vhost_svq_set_notification(svq, false); while (true) { g_autofree VirtQueueElement *elem =3D vhost_svq_get_buf(svq); if (!elem) { @@ -286,6 +307,7 @@ static void vhost_svq_handle_call_no_test(EventNotifier= *n) =20 virtqueue_flush(vq, i); event_notifier_set(&svq->guest_call_notifier); + vhost_svq_set_notification(svq, true); } while (vhost_svq_more_used(svq)); } =20 --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633073339064401.23710197642004; Fri, 1 Oct 2021 00:28:59 -0700 (PDT) Received: from localhost ([::1]:35692 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCyY-00022b-2d for importer@patchew.org; Fri, 01 Oct 2021 03:28:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36776) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCjP-0005X5-6p for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:13:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:20155) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCjN-0005st-35 for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:13:18 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-228-yHXUKOIZPlKfNCnM_MbwFw-1; Fri, 01 Oct 2021 03:13:15 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3EB7F100C630; Fri, 1 Oct 2021 07:13:14 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3C1BB652B8; Fri, 1 Oct 2021 07:13:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072396; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CByrwRYklfGYjTqDhPlmZaxXos/A5kFvAs3larfYmKM=; b=F83HZQEAypW3SjLyz2UX9Ty0PMEmVdb/C+XWq4EV+cnExofdpEjf+6ogIiGjD7cInzWY1D G5Ai8n+RP3xwi/IGVy+WcPyiogyb3MA+xHuKdzVWSAKlRzxzt5ebfm2fPSSIjcZ7+GHNkc rTg9wZelaFKSK1EuuLCUMbcDTKaEMzc= X-MC-Unique: yHXUKOIZPlKfNCnM_MbwFw-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 18/20] vhost: Add VhostIOVATree Date: Fri, 1 Oct 2021 09:06:01 +0200 Message-Id: <20211001070603.307037-19-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633073341286100001 This tree is able to look for a translated address from an IOVA address. At first glance is similar to util/iova-tree. However, SVQ working on devices with limited IOVA space need more capabilities, like allocating IOVA chunks or perform reverse translations (qemu addresses to iova). The allocation capability, as "assign a free IOVA address to this chunk of memory in qemu's address space" allows shadow virtqueue to create a new address space that is not restricted by guest's addressable one, so we can allocate shadow vqs vrings outside of its reachability, nor qemu's one. At the moment, the allocation is just done growing, not allowing deletion. A different name could be used, but ordered searchable array is a little bit long though. It duplicates the array so it can search efficiently both directions, and it will signal overlap if iova or the translated address is present in it's each array. Use of array will be changed to util-iova-tree in future series. Signed-off-by: Eugenio P=C3=A9rez --- hw/virtio/vhost-iova-tree.h | 40 +++++++ hw/virtio/vhost-iova-tree.c | 230 ++++++++++++++++++++++++++++++++++++ hw/virtio/meson.build | 2 +- 3 files changed, 271 insertions(+), 1 deletion(-) create mode 100644 hw/virtio/vhost-iova-tree.h create mode 100644 hw/virtio/vhost-iova-tree.c diff --git a/hw/virtio/vhost-iova-tree.h b/hw/virtio/vhost-iova-tree.h new file mode 100644 index 0000000000..d163a88905 --- /dev/null +++ b/hw/virtio/vhost-iova-tree.h @@ -0,0 +1,40 @@ +/* + * vhost software live migration ring + * + * SPDX-FileCopyrightText: Red Hat, Inc. 2021 + * SPDX-FileContributor: Author: Eugenio P=C3=A9rez + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#ifndef HW_VIRTIO_VHOST_IOVA_TREE_H +#define HW_VIRTIO_VHOST_IOVA_TREE_H + +#include "exec/memory.h" + +typedef struct VhostDMAMap { + void *translated_addr; + hwaddr iova; + hwaddr size; /* Inclusive */ + IOMMUAccessFlags perm; +} VhostDMAMap; + +typedef enum VhostDMAMapNewRC { + VHOST_DMA_MAP_NO_SPACE =3D -3, + VHOST_DMA_MAP_OVERLAP =3D -2, + VHOST_DMA_MAP_INVALID =3D -1, + VHOST_DMA_MAP_OK =3D 0, +} VhostDMAMapNewRC; + +typedef struct VhostIOVATree VhostIOVATree; + +VhostIOVATree *vhost_iova_tree_new(void); +void vhost_iova_tree_unref(VhostIOVATree *iova_rm); +G_DEFINE_AUTOPTR_CLEANUP_FUNC(VhostIOVATree, vhost_iova_tree_unref); + +const VhostDMAMap *vhost_iova_tree_find_iova(const VhostIOVATree *iova_rm, + const VhostDMAMap *map); +VhostDMAMapNewRC vhost_iova_tree_alloc(VhostIOVATree *iova_rm, + VhostDMAMap *map); + +#endif diff --git a/hw/virtio/vhost-iova-tree.c b/hw/virtio/vhost-iova-tree.c new file mode 100644 index 0000000000..c284e27607 --- /dev/null +++ b/hw/virtio/vhost-iova-tree.c @@ -0,0 +1,230 @@ +/* + * vhost software live migration ring + * + * SPDX-FileCopyrightText: Red Hat, Inc. 2021 + * SPDX-FileContributor: Author: Eugenio P=C3=A9rez + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "qemu/osdep.h" +#include "vhost-iova-tree.h" + +#define G_ARRAY_NOT_ZERO_TERMINATED false +#define G_ARRAY_NOT_CLEAR_ON_ALLOC false + +#define iova_min qemu_real_host_page_size + +/** + * VhostIOVATree, able to: + * - Translate iova address + * - Reverse translate iova address (from translated to iova) + * - Allocate IOVA regions for translated range (potentially slow operatio= n) + * + * Note that it cannot remove nodes. + */ +struct VhostIOVATree { + /* Ordered array of reverse translations, IOVA address to qemu memory.= */ + GArray *iova_taddr_map; + + /* + * Ordered array of translations from qemu virtual memory address to i= ova + */ + GArray *taddr_iova_map; +}; + +/** + * Inserts an element after an existing one in garray. + * + * @array The array + * @prev_elem The previous element of array of NULL if prepending + * @map The DMA map + * + * It provides the aditional advantage of being type safe over + * g_array_insert_val, which accepts a reference pointer instead of a value + * with no complains. + */ +static void vhost_iova_tree_insert_after(GArray *array, + const VhostDMAMap *prev_elem, + const VhostDMAMap *map) +{ + size_t pos; + + if (!prev_elem) { + pos =3D 0; + } else { + pos =3D prev_elem - &g_array_index(array, typeof(*prev_elem), 0) += 1; + } + + g_array_insert_val(array, pos, *map); +} + +static gint vhost_iova_tree_cmp_taddr(gconstpointer a, gconstpointer b) +{ + const VhostDMAMap *m1 =3D a, *m2 =3D b; + + if (m1->translated_addr > m2->translated_addr + m2->size) { + return 1; + } + + if (m1->translated_addr + m1->size < m2->translated_addr) { + return -1; + } + + /* Overlapped */ + return 0; +} + +/** + * Find the previous node to a given iova + * + * @array The ascending ordered-by-translated-addr array of VhostDMAMap + * @map The map to insert + * @prev Returned location of the previous map + * + * Return VHOST_DMA_MAP_OK if everything went well, or VHOST_DMA_MAP_OVERL= AP if + * it already exists. It is ok to use this function to check if a given ra= nge + * exists, but it will use a linear search. + * + * TODO: We can use bsearch to locate the entry if we save the state in the + * needle, knowing that the needle is always the first argument to + * compare_func. + */ +static VhostDMAMapNewRC vhost_iova_tree_find_prev(const GArray *array, + GCompareFunc compare_fun= c, + const VhostDMAMap *map, + const VhostDMAMap **prev) +{ + size_t i; + int r; + + *prev =3D NULL; + for (i =3D 0; i < array->len; ++i) { + r =3D compare_func(map, &g_array_index(array, typeof(*map), i)); + if (r =3D=3D 0) { + return VHOST_DMA_MAP_OVERLAP; + } + if (r < 0) { + return VHOST_DMA_MAP_OK; + } + + *prev =3D &g_array_index(array, typeof(**prev), i); + } + + return VHOST_DMA_MAP_OK; +} + +/** + * Create a new IOVA tree + * + * Returns the new IOVA tree + */ +VhostIOVATree *vhost_iova_tree_new(void) +{ + VhostIOVATree *tree =3D g_new(VhostIOVATree, 1); + tree->iova_taddr_map =3D g_array_new(G_ARRAY_NOT_ZERO_TERMINATED, + G_ARRAY_NOT_CLEAR_ON_ALLOC, + sizeof(VhostDMAMap)); + tree->taddr_iova_map =3D g_array_new(G_ARRAY_NOT_ZERO_TERMINATED, + G_ARRAY_NOT_CLEAR_ON_ALLOC, + sizeof(VhostDMAMap)); + return tree; +} + +/** + * Destroy an IOVA tree + * + * @tree The iova tree + */ +void vhost_iova_tree_unref(VhostIOVATree *tree) +{ + g_array_unref(g_steal_pointer(&tree->iova_taddr_map)); + g_array_unref(g_steal_pointer(&tree->taddr_iova_map)); +} + +/** + * Find the IOVA address stored from a memory address + * + * @tree The iova tree + * @map The map with the memory address + * + * Return the stored mapping, or NULL if not found. + */ +const VhostDMAMap *vhost_iova_tree_find_iova(const VhostIOVATree *tree, + const VhostDMAMap *map) +{ + /* + * This can be replaced with g_array_binary_search (Since glib 2.62) w= hen + * that version become common enough. + */ + return bsearch(map, tree->taddr_iova_map->data, tree->taddr_iova_map->= len, + sizeof(*map), vhost_iova_tree_cmp_taddr); +} + +static bool vhost_iova_tree_find_iova_hole(const GArray *iova_map, + const VhostDMAMap *map, + const VhostDMAMap **prev_elem) +{ + size_t i; + hwaddr iova =3D iova_min; + + *prev_elem =3D NULL; + for (i =3D 0; i < iova_map->len; i++) { + const VhostDMAMap *next =3D &g_array_index(iova_map, typeof(*next)= , i); + hwaddr hole_end =3D next->iova; + if (map->size < hole_end - iova) { + return true; + } + + iova =3D next->iova + next->size + 1; + *prev_elem =3D next; + } + + return ((hwaddr)-1 - iova) > iova_map->len; +} + +/** + * Allocate a new mapping + * + * @tree The iova tree + * @map The iova map + * + * Returns: + * - VHOST_DMA_MAP_OK if the map fits in the container + * - VHOST_DMA_MAP_INVALID if the map does not make sense (like size overf= low) + * - VHOST_DMA_MAP_OVERLAP if the tree already contains that map + * - VHOST_DMA_MAP_NO_SPACE if iova_rm cannot allocate more space. + * + * It returns assignated iova in map->iova if return value is VHOST_DMA_MA= P_OK. + */ +VhostDMAMapNewRC vhost_iova_tree_alloc(VhostIOVATree *tree, + VhostDMAMap *map) +{ + const VhostDMAMap *qemu_prev, *iova_prev; + int find_prev_rc; + bool fit; + + if (map->translated_addr + map->size < map->translated_addr || + map->iova + map->size < map->iova || map->perm =3D=3D IOMMU_NONE) { + return VHOST_DMA_MAP_INVALID; + } + + /* Search for a hole in iova space big enough */ + fit =3D vhost_iova_tree_find_iova_hole(tree->iova_taddr_map, map, + &iova_prev); + if (!fit) { + return VHOST_DMA_MAP_NO_SPACE; + } + + map->iova =3D iova_prev ? (iova_prev->iova + iova_prev->size) + 1 : io= va_min; + find_prev_rc =3D vhost_iova_tree_find_prev(tree->taddr_iova_map, + vhost_iova_tree_cmp_taddr, ma= p, + &qemu_prev); + if (find_prev_rc =3D=3D VHOST_DMA_MAP_OVERLAP) { + return VHOST_DMA_MAP_OVERLAP; + } + + vhost_iova_tree_insert_after(tree->iova_taddr_map, iova_prev, map); + vhost_iova_tree_insert_after(tree->taddr_iova_map, qemu_prev, map); + return VHOST_DMA_MAP_OK; +} diff --git a/hw/virtio/meson.build b/hw/virtio/meson.build index 8b5a0225fe..cb306b83c6 100644 --- a/hw/virtio/meson.build +++ b/hw/virtio/meson.build @@ -11,7 +11,7 @@ softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('vhost-= stub.c')) =20 virtio_ss =3D ss.source_set() virtio_ss.add(files('virtio.c')) -virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backe= nd.c', 'vhost-shadow-virtqueue.c')) +virtio_ss.add(when: 'CONFIG_VHOST', if_true: files('vhost.c', 'vhost-backe= nd.c', 'vhost-shadow-virtqueue.c', 'vhost-iova-tree.c')) virtio_ss.add(when: 'CONFIG_VHOST_USER', if_true: files('vhost-user.c')) virtio_ss.add(when: 'CONFIG_VHOST_VDPA', if_true: files('vhost-vdpa.c')) virtio_ss.add(when: 'CONFIG_VIRTIO_BALLOON', if_true: files('virtio-balloo= n.c')) --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633073180943712.3741656481964; Fri, 1 Oct 2021 00:26:20 -0700 (PDT) Received: from localhost ([::1]:55810 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCvz-0004oe-Rs for importer@patchew.org; Fri, 01 Oct 2021 03:26:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36804) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCjS-0005df-BN for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:13:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:32624) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCjQ-0005vm-Is for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:13:22 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-334-o688-fhbPV-e_6AtC9NjVg-1; Fri, 01 Oct 2021 03:13:18 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8960819067E0; Fri, 1 Oct 2021 07:13:17 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id 94D9C1980E; Fri, 1 Oct 2021 07:13:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072400; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3ijz1uGxJNfAOPp7OTbE3hzeXRghB2rBlM3NdVAU5g8=; b=A6wGdxePAhLmxqhQCm8LrwWP8zlTlAKPfBldDk5bL5Ko1MN77vobA2NsFI7Oenl0CNwzez OA7Mf/VJlGgT2MWciLR+F31uHhBalCrcDRvY62yPqTs4SQwx9tFIeXrUf91Rt8Y5Z3ropW DkU+4y2TVx7R930J8PcWpEzoW2BKGNQ= X-MC-Unique: o688-fhbPV-e_6AtC9NjVg-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 19/20] vhost: Use a tree to store memory mappings Date: Fri, 1 Oct 2021 09:06:02 +0200 Message-Id: <20211001070603.307037-20-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633073182990100001 Track memory translations of devices with IOMMU (all vhost-vdpa devices at the moment). It does not work if device has restrictions in its iova range at the moment. Updates to tree are protected by BQL, each one always run from main event loop context. Signed-off-by: Eugenio P=C3=A9rez --- include/hw/virtio/vhost-vdpa.h | 3 ++ hw/virtio/vhost-vdpa.c | 59 ++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 9044ae694b..7353e36884 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -15,6 +15,7 @@ #include =20 #include "qemu/queue.h" +#include "hw/virtio/vhost-iova-tree.h" #include "hw/virtio/virtio.h" =20 typedef struct VhostVDPAHostNotifier { @@ -29,6 +30,8 @@ typedef struct vhost_vdpa { uint64_t host_features; uint64_t guest_features; bool shadow_vqs_enabled; + /* IOVA mapping used by Shadow Virtqueue */ + VhostIOVATree *iova_map; GPtrArray *shadow_vqs; struct vhost_dev *dev; QLIST_ENTRY(vhost_vdpa) entry; diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index bb7010ddb5..a9c680b487 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -395,6 +395,7 @@ static void vhost_vdpa_svq_cleanup(struct vhost_dev *de= v) vhost_svq_stop(dev, idx, g_ptr_array_index(v->shadow_vqs, idx)); } g_ptr_array_free(v->shadow_vqs, true); + g_clear_pointer(&v->iova_map, vhost_iova_tree_unref); } =20 static int vhost_vdpa_cleanup(struct vhost_dev *dev) @@ -753,6 +754,22 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *= dev) return true; } =20 +/** + * Maps QEMU vaddr memory to device in a suitable way for shadow virtqueue: + * - It always reference qemu memory address, not guest's memory. + * - TODO It's always in range of device. + * + * It returns the translated address + */ +static int vhost_vdpa_svq_map(struct vhost_vdpa *v, VhostDMAMap *map) +{ + int r =3D vhost_iova_tree_alloc(v->iova_map, map); + assert(r =3D=3D VHOST_DMA_MAP_OK); + + return vhost_vdpa_dma_map(v, map->iova, map->size, map->translated_add= r, + false); +} + static int vhost_vdpa_vring_pause(struct vhost_dev *dev) { int r; @@ -771,6 +788,7 @@ static int vhost_vdpa_vring_pause(struct vhost_dev *dev) */ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *dev, unsigned idx) { + VhostDMAMap device_region, driver_region; struct vhost_vdpa *v =3D dev->opaque; VhostShadowVirtqueue *svq =3D g_ptr_array_index(v->shadow_vqs, idx); EventNotifier *vhost_call_notifier =3D vhost_svq_get_svq_call_notifier= (svq); @@ -789,6 +807,33 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *= dev, unsigned idx) bool b; =20 vhost_svq_get_vring_addr(svq, &addr); + driver_region =3D (VhostDMAMap) { + .translated_addr =3D (void *)addr.desc_user_addr, + + /* + * DMAMAp.size include the last byte included in the range, while + * sizeof marks one past it. Substract one byte to make them match. + */ + .size =3D vhost_svq_driver_area_size(svq) - 1, + .perm =3D VHOST_ACCESS_RO, + }; + device_region =3D (VhostDMAMap) { + .translated_addr =3D (void *)addr.used_user_addr, + .size =3D vhost_svq_device_area_size(svq) - 1, + .perm =3D VHOST_ACCESS_RW, + }; + + r =3D vhost_vdpa_svq_map(v, &driver_region); + assert(r =3D=3D 0); + r =3D vhost_vdpa_svq_map(v, &device_region); + assert(r =3D=3D 0); + + /* Expose IOVA addresses to vDPA device */ + addr.avail_user_addr =3D driver_region.iova + addr.avail_user_addr + - addr.desc_user_addr; + addr.desc_user_addr =3D driver_region.iova; + addr.used_user_addr =3D device_region.iova; + r =3D vhost_vdpa_set_vring_addr(dev, &addr); if (unlikely(r)) { error_report("vhost_set_vring_addr for shadow vq failed"); @@ -822,6 +867,17 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *= dev, unsigned idx) return true; } =20 +/** + * Enable or disable shadow virtqueue in a vhost vdpa device. + * + * This function is idempotent, to call it many times with the same value = for + * enable_svq will simply return success. + * + * @v The vhost vdpa device + * @enable The value of shadow virtqueue we want. + * + * Returns the number of queues changed. + */ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) { struct vhost_dev *hdev =3D v->dev; @@ -833,6 +889,8 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa= *v, bool enable) } =20 if (enable) { + v->iova_map =3D vhost_iova_tree_new(); + /* Allocate resources */ assert(v->shadow_vqs->len =3D=3D 0); for (n =3D 0; n < hdev->nvqs; ++n) { @@ -907,6 +965,7 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa= *v, bool enable) if (!enable) { /* Resources cleanup */ g_ptr_array_set_size(v->shadow_vqs, 0); + g_clear_pointer(&v->iova_map, vhost_iova_tree_unref); } =20 return n; --=20 2.27.0 From nobody Sun Feb 8 12:19:49 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1633073331026404.7854396556012; Fri, 1 Oct 2021 00:28:51 -0700 (PDT) Received: from localhost ([::1]:35042 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mWCyP-0001aI-Vq for importer@patchew.org; Fri, 01 Oct 2021 03:28:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36830) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCjW-0005gb-C0 for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:13:26 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:33200) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mWCjU-0005ym-2w for qemu-devel@nongnu.org; Fri, 01 Oct 2021 03:13:26 -0400 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-367-T-uAi4ixPwe_0_ef9lexPA-1; Fri, 01 Oct 2021 03:13:22 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0E461802CB5; Fri, 1 Oct 2021 07:13:21 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.193.2]) by smtp.corp.redhat.com (Postfix) with ESMTP id E0D9E4180; Fri, 1 Oct 2021 07:13:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1633072403; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aZu7BC7f5H5b6x/I9dZdrB2IJlDrq/Td+ISYLo6sYo4=; b=akx33quDWvt3jSue0fcIj4pFEt73nxHgzQXYNThsl7f4ol52SqIG29Wy0CDo8aa4Ux7DF3 H7fq1sYqg+KUMUf2KmnVunIaB5ota70TMQJTjBowGQESrSGejHU0vxP7rUyg6A9MYn6dfV R+4wdRscNPyvjgT0Kad5C0c2dMGfFM0= X-MC-Unique: T-uAi4ixPwe_0_ef9lexPA-1 From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: qemu-devel@nongnu.org Subject: [RFC PATCH v4 20/20] vdpa: Add custom IOTLB translations to SVQ Date: Fri, 1 Oct 2021 09:06:03 +0200 Message-Id: <20211001070603.307037-21-eperezma@redhat.com> In-Reply-To: <20211001070603.307037-1-eperezma@redhat.com> References: <20211001070603.307037-1-eperezma@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=216.205.24.124; envelope-from=eperezma@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Parav Pandit , Juan Quintela , Jason Wang , "Michael S. Tsirkin" , Markus Armbruster , virtualization@lists.linux-foundation.org, Harpreet Singh Anand , Xiao W Wang , Stefan Hajnoczi , Eli Cohen , Eric Blake , Michael Lilja , Stefano Garzarella Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1633073332773100001 Use translations added in VhostIOVATree in SVQ. Now every element needs to store the previous address also, so VirtQueue can consume the elements properly. This adds a little overhead per VQ element, having to allocate more memory to stash them. As a possible optimization, this allocation could be avoided if the descriptor is not a chain but a single one, but this is left undone. TODO: iova range should be queried before, and add logic to fail when GPA is outside of its range and memory listener or svq add it. Signed-off-by: Eugenio P=C3=A9rez --- hw/virtio/vhost-shadow-virtqueue.h | 4 +- hw/virtio/vhost-shadow-virtqueue.c | 130 ++++++++++++++++++++++++----- hw/virtio/vhost-vdpa.c | 40 ++++++++- hw/virtio/trace-events | 1 + 4 files changed, 152 insertions(+), 23 deletions(-) diff --git a/hw/virtio/vhost-shadow-virtqueue.h b/hw/virtio/vhost-shadow-vi= rtqueue.h index b7baa424a7..a0e6b5267a 100644 --- a/hw/virtio/vhost-shadow-virtqueue.h +++ b/hw/virtio/vhost-shadow-virtqueue.h @@ -11,6 +11,7 @@ #define VHOST_SHADOW_VIRTQUEUE_H =20 #include "hw/virtio/vhost.h" +#include "hw/virtio/vhost-iova-tree.h" =20 typedef struct VhostShadowVirtqueue VhostShadowVirtqueue; =20 @@ -28,7 +29,8 @@ bool vhost_svq_start(struct vhost_dev *dev, unsigned idx, void vhost_svq_stop(struct vhost_dev *dev, unsigned idx, VhostShadowVirtqueue *svq); =20 -VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx); +VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx, + VhostIOVATree *iova_map); =20 void vhost_svq_free(VhostShadowVirtqueue *vq); =20 diff --git a/hw/virtio/vhost-shadow-virtqueue.c b/hw/virtio/vhost-shadow-vi= rtqueue.c index 2fd0bab75d..9db538547e 100644 --- a/hw/virtio/vhost-shadow-virtqueue.c +++ b/hw/virtio/vhost-shadow-virtqueue.c @@ -11,12 +11,19 @@ #include "hw/virtio/vhost-shadow-virtqueue.h" #include "hw/virtio/vhost.h" #include "hw/virtio/virtio-access.h" +#include "hw/virtio/vhost-iova-tree.h" =20 #include "standard-headers/linux/vhost_types.h" =20 #include "qemu/error-report.h" #include "qemu/main-loop.h" =20 +typedef struct SVQElement { + VirtQueueElement elem; + void **in_sg_stash; + void **out_sg_stash; +} SVQElement; + /* Shadow virtqueue to relay notifications */ typedef struct VhostShadowVirtqueue { /* Shadow vring */ @@ -46,8 +53,11 @@ typedef struct VhostShadowVirtqueue { /* Virtio device */ VirtIODevice *vdev; =20 + /* IOVA mapping if used */ + VhostIOVATree *iova_map; + /* Map for returning guest's descriptors */ - VirtQueueElement **ring_id_maps; + SVQElement **ring_id_maps; =20 /* Next head to expose to device */ uint16_t avail_idx_shadow; @@ -79,13 +89,6 @@ bool vhost_svq_valid_device_features(uint64_t *dev_featu= res) continue; =20 case VIRTIO_F_ACCESS_PLATFORM: - /* SVQ needs this feature disabled. Can't continue */ - if (*dev_features & BIT_ULL(b)) { - clear_bit(b, dev_features); - r =3D false; - } - break; - case VIRTIO_F_VERSION_1: /* SVQ needs this feature, so can't continue */ if (!(*dev_features & BIT_ULL(b))) { @@ -126,6 +129,64 @@ static void vhost_svq_set_notification(VhostShadowVirt= queue *svq, bool enable) } } =20 +static void vhost_svq_stash_addr(void ***stash, const struct iovec *iov, + size_t num) +{ + size_t i; + + if (num =3D=3D 0) { + return; + } + + *stash =3D g_new(void *, num); + for (i =3D 0; i < num; ++i) { + (*stash)[i] =3D iov[i].iov_base; + } +} + +static void vhost_svq_unstash_addr(void **stash, struct iovec *iov, size_t= num) +{ + size_t i; + + if (num =3D=3D 0) { + return; + } + + for (i =3D 0; i < num; ++i) { + iov[i].iov_base =3D stash[i]; + } + g_free(stash); +} + +static void vhost_svq_translate_addr(const VhostShadowVirtqueue *svq, + struct iovec *iovec, size_t num) +{ + size_t i; + + for (i =3D 0; i < num; ++i) { + VhostDMAMap needle =3D { + .translated_addr =3D iovec[i].iov_base, + .size =3D iovec[i].iov_len, + }; + size_t off; + + const VhostDMAMap *map =3D vhost_iova_tree_find_iova(svq->iova_map, + &needle); + /* + * Map cannot be NULL since iova map contains all guest space and + * qemu already has a physical address mapped + */ + assert(map); + + /* + * Map->iova chunk size is ignored. What to do if descriptor + * (addr, size) does not fit is delegated to the device. + */ + off =3D needle.translated_addr - map->translated_addr; + iovec[i].iov_base =3D (void *)(map->iova + off); + } +} + static void vhost_vring_write_descs(VhostShadowVirtqueue *svq, const struct iovec *iovec, size_t num, bool more_descs, bool writ= e) @@ -156,8 +217,9 @@ static void vhost_vring_write_descs(VhostShadowVirtqueu= e *svq, } =20 static unsigned vhost_svq_add_split(VhostShadowVirtqueue *svq, - VirtQueueElement *elem) + SVQElement *svq_elem) { + VirtQueueElement *elem =3D &svq_elem->elem; int head; unsigned avail_idx; vring_avail_t *avail =3D svq->vring.avail; @@ -167,6 +229,12 @@ static unsigned vhost_svq_add_split(VhostShadowVirtque= ue *svq, /* We need some descriptors here */ assert(elem->out_num || elem->in_num); =20 + vhost_svq_stash_addr(&svq_elem->in_sg_stash, elem->in_sg, elem->in_num= ); + vhost_svq_stash_addr(&svq_elem->out_sg_stash, elem->out_sg, elem->out_= num); + + vhost_svq_translate_addr(svq, elem->in_sg, elem->in_num); + vhost_svq_translate_addr(svq, elem->out_sg, elem->out_num); + vhost_vring_write_descs(svq, elem->out_sg, elem->out_num, elem->in_num > 0, false); vhost_vring_write_descs(svq, elem->in_sg, elem->in_num, false, true); @@ -187,7 +255,7 @@ static unsigned vhost_svq_add_split(VhostShadowVirtqueu= e *svq, =20 } =20 -static void vhost_svq_add(VhostShadowVirtqueue *svq, VirtQueueElement *ele= m) +static void vhost_svq_add(VhostShadowVirtqueue *svq, SVQElement *elem) { unsigned qemu_head =3D vhost_svq_add_split(svq, elem); =20 @@ -221,7 +289,7 @@ static void vhost_handle_guest_kick(EventNotifier *n) } =20 while (true) { - VirtQueueElement *elem =3D virtqueue_pop(svq->vq, sizeof(*elem= )); + SVQElement *elem =3D virtqueue_pop(svq->vq, sizeof(*elem)); if (!elem) { break; } @@ -247,7 +315,7 @@ static bool vhost_svq_more_used(VhostShadowVirtqueue *s= vq) return svq->used_idx !=3D svq->shadow_used_idx; } =20 -static VirtQueueElement *vhost_svq_get_buf(VhostShadowVirtqueue *svq) +static SVQElement *vhost_svq_get_buf(VhostShadowVirtqueue *svq) { vring_desc_t *descs =3D svq->vring.desc; const vring_used_t *used =3D svq->vring.used; @@ -279,7 +347,7 @@ static VirtQueueElement *vhost_svq_get_buf(VhostShadowV= irtqueue *svq) descs[used_elem.id].next =3D svq->free_head; svq->free_head =3D used_elem.id; =20 - svq->ring_id_maps[used_elem.id]->len =3D used_elem.len; + svq->ring_id_maps[used_elem.id]->elem.len =3D used_elem.len; return g_steal_pointer(&svq->ring_id_maps[used_elem.id]); } =20 @@ -296,12 +364,19 @@ static void vhost_svq_handle_call_no_test(EventNotifi= er *n) =20 vhost_svq_set_notification(svq, false); while (true) { - g_autofree VirtQueueElement *elem =3D vhost_svq_get_buf(svq); - if (!elem) { + g_autofree SVQElement *svq_elem =3D vhost_svq_get_buf(svq); + VirtQueueElement *elem; + if (!svq_elem) { break; } =20 assert(i < svq->vring.num); + elem =3D &svq_elem->elem; + + vhost_svq_unstash_addr(svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); + vhost_svq_unstash_addr(svq_elem->out_sg_stash, elem->out_sg, + elem->out_num); virtqueue_fill(vq, elem, elem->len, i++); } =20 @@ -451,14 +526,24 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned i= dx, event_notifier_set_handler(&svq->host_notifier, NULL); =20 for (i =3D 0; i < svq->vring.num; ++i) { - g_autofree VirtQueueElement *elem =3D svq->ring_id_maps[i]; + g_autofree SVQElement *svq_elem =3D svq->ring_id_maps[i]; + VirtQueueElement *elem; + + if (!svq_elem) { + continue; + } + + elem =3D &svq_elem->elem; + vhost_svq_unstash_addr(svq_elem->in_sg_stash, elem->in_sg, + elem->in_num); + vhost_svq_unstash_addr(svq_elem->out_sg_stash, elem->out_sg, + elem->out_num); + /* * Although the doc says we must unpop in order, it's ok to unpop * everything. */ - if (elem) { - virtqueue_unpop(svq->vq, elem, elem->len); - } + virtqueue_unpop(svq->vq, elem, elem->len); } } =20 @@ -466,7 +551,8 @@ void vhost_svq_stop(struct vhost_dev *dev, unsigned idx, * Creates vhost shadow virtqueue, and instruct vhost device to use the sh= adow * methods and file descriptors. */ -VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx) +VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev *dev, int idx, + VhostIOVATree *iova_map) { int vq_idx =3D dev->vq_index + idx; unsigned num =3D virtio_queue_get_num(dev->vdev, vq_idx); @@ -500,11 +586,13 @@ VhostShadowVirtqueue *vhost_svq_new(struct vhost_dev = *dev, int idx) memset(svq->vring.desc, 0, driver_size); svq->vring.used =3D qemu_memalign(qemu_real_host_page_size, device_siz= e); memset(svq->vring.used, 0, device_size); + svq->iova_map =3D iova_map; + for (i =3D 0; i < num - 1; i++) { svq->vring.desc[i].next =3D cpu_to_le16(i + 1); } =20 - svq->ring_id_maps =3D g_new0(VirtQueueElement *, num); + svq->ring_id_maps =3D g_new0(SVQElement *, num); event_notifier_set_handler(&svq->call_notifier, vhost_svq_handle_call); return g_steal_pointer(&svq); diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index a9c680b487..f5a12fee9d 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -176,6 +176,18 @@ static void vhost_vdpa_listener_region_add(MemoryListe= ner *listener, vaddr, section->readonly); =20 llsize =3D int128_sub(llend, int128_make64(iova)); + if (v->shadow_vqs_enabled) { + VhostDMAMap mem_region =3D { + .translated_addr =3D vaddr, + .size =3D int128_get64(llsize) - 1, + .perm =3D IOMMU_ACCESS_FLAG(true, section->readonly), + }; + + int r =3D vhost_iova_tree_alloc(v->iova_map, &mem_region); + assert(r =3D=3D VHOST_DMA_MAP_OK); + + iova =3D mem_region.iova; + } =20 ret =3D vhost_vdpa_dma_map(v, iova, int128_get64(llsize), vaddr, section->readonly); @@ -754,6 +766,23 @@ static bool vhost_vdpa_force_iommu(struct vhost_dev *= dev) return true; } =20 +static int vhost_vdpa_get_iova_range(struct vhost_dev *dev, + hwaddr *first, hwaddr *last) +{ + int ret; + struct vhost_vdpa_iova_range range; + + ret =3D vhost_vdpa_call(dev, VHOST_VDPA_GET_IOVA_RANGE, &range); + if (ret !=3D 0) { + return ret; + } + + *first =3D range.first; + *last =3D range.last; + trace_vhost_vdpa_get_iova_range(dev, *first, *last); + return ret; +} + /** * Maps QEMU vaddr memory to device in a suitable way for shadow virtqueue: * - It always reference qemu memory address, not guest's memory. @@ -881,6 +910,7 @@ static bool vhost_vdpa_svq_start_vq(struct vhost_dev *d= ev, unsigned idx) static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa *v, bool enable) { struct vhost_dev *hdev =3D v->dev; + hwaddr iova_first, iova_last; unsigned n; int r; =20 @@ -894,7 +924,7 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa= *v, bool enable) /* Allocate resources */ assert(v->shadow_vqs->len =3D=3D 0); for (n =3D 0; n < hdev->nvqs; ++n) { - VhostShadowVirtqueue *svq =3D vhost_svq_new(hdev, n); + VhostShadowVirtqueue *svq =3D vhost_svq_new(hdev, n, v->iova_m= ap); if (unlikely(!svq)) { g_ptr_array_set_size(v->shadow_vqs, 0); return 0; @@ -903,6 +933,8 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdpa= *v, bool enable) } } =20 + r =3D vhost_vdpa_get_iova_range(hdev, &iova_first, &iova_last); + assert(r =3D=3D 0); r =3D vhost_vdpa_vring_pause(hdev); assert(r =3D=3D 0); =20 @@ -913,6 +945,12 @@ static unsigned vhost_vdpa_enable_svq(struct vhost_vdp= a *v, bool enable) } } =20 + memory_listener_unregister(&v->listener); + if (vhost_vdpa_dma_unmap(v, iova_first, + (iova_last - iova_first) & TARGET_PAGE_MASK))= { + error_report("Fail to invalidate device iotlb"); + } + /* Reset device so it can be configured */ r =3D vhost_vdpa_dev_start(hdev, false); assert(r =3D=3D 0); diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 8ed19e9d0c..650e521e35 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -52,6 +52,7 @@ vhost_vdpa_set_vring_call(void *dev, unsigned int index, = int fd) "dev: %p index: vhost_vdpa_get_features(void *dev, uint64_t features) "dev: %p features: 0= x%"PRIx64 vhost_vdpa_set_owner(void *dev) "dev: %p" vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint6= 4_t avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_add= r: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64 +vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: = %p first: 0x%"PRIx64" last: 0x%"PRIx64 =20 # virtio.c virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned o= ut_num) "elem %p size %zd in_num %u out_num %u" --=20 2.27.0