From nobody Thu Nov 13 20:42:52 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1582891755; cv=none; d=zohomail.com; s=zohoarc; b=XablVRikO4jSh99j5WINPSQ1BRDP9y3VfsufdrG5dmmXOnqywDlq1dobAkBz862aN6WLPazsyLBGim5pD6SCCKJ5/ZFb1D8PA+c7FWU4coQcKxDxMiKOaTwUZBlRYuqX9EpLS8s/1+evQh4JBTfcM6/lXFflLx9UzQKQV0hwD2o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1582891755; h=Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject; bh=rZDNW1yJsNnrcPFLZN2RYuihigc2J7mLIhPr+y1X9Ic=; b=irDsbPQXp2i5ZUOpWYWr2dRzxuQLH+T8KY7/sQze6toLc+HDHEh8FR2Kr4fwztMfNkOWsXMCqViSl83+Bx4HgLJqoSRKuEeI6/m1hqmrzU1PZYOySf4zuWdnGVQ6Z96DXuzslISVBbjK1lBZhx2OQTAMM3JjV+YbrYaOVxWVKO8= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1582891755050709.5243839242623; Fri, 28 Feb 2020 04:09:15 -0800 (PST) Received: from localhost ([::1]:46050 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7eS9-0006MM-4K for importer@patchew.org; Fri, 28 Feb 2020 07:09:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:44689) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7eRa-0005vN-AT for qemu-devel@nongnu.org; Fri, 28 Feb 2020 07:08:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j7eRX-0000fx-GX for qemu-devel@nongnu.org; Fri, 28 Feb 2020 07:08:36 -0500 Received: from mail.sysgo.com ([176.9.12.79]:50022) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1j7eRX-0000fq-7L for qemu-devel@nongnu.org; Fri, 28 Feb 2020 07:08:35 -0500 From: Roman Kapl To: Subject: [PATCH] vfio: avoid SET_ACTION_TRIGGER ioctls Date: Fri, 28 Feb 2020 13:08:00 +0100 Message-Id: <20200228120800.5979-1-rka@sysgo.com> X-Mailer: git-send-email 2.22.0 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 176.9.12.79 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Williamson , qemu-devel@nongnu.org, Roman Kapl Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" Content-Type: text/plain; charset="utf-8" For MSI-X interrupts, remember what the last used eventfd was (KVM bypass vs QEMU) and only call vfio_set_irq_signaling if it has changed. This not only helps with performance, but it seems that interrupts can be lost during VFIO_IRQ_SET_ACTION_TRIGGER. With the 'x-no-kvm-msix' switch and this patch, SET_ACTION_TRIGGER is not called during mask/unmask. This really only affects guests that actively use MSI-X maskin= g. Signed-off-by: Roman Kapl --- This patch scratches my particular itch. I am able to get our guest (which = masks MSI on each interrupt) running, without getting randomly stuck on waiting f= or interrupt. However, the solution is far from perfect (x-no-kvm-msix is requ= ired) and pretty slow. I would be interested in hearing any ideas how to improve = this. Some ideas: 1) Fix the kernel so that SET_ACTION_TRIGGER does not loose interrupts (I t= hink the problem is there, but not 100% sure). I've tested on 5.3.0-40-generic #32~18.04.1-Ubuntu SMP. 2) Add support for MASK/UNMASK for MSI-X in kernel and use that. But I don't know how to do PBA in that case. Another IOCTL? We could look at the real P= BA array, if mapping is supported, but that seems hacky. 3) Twiddle the bits behing kernel's back, if it can be mapped? Still, I think this patch does not hurt anything and could be applied if no= -one can think of a better way. --- hw/vfio/pci.c | 32 ++++++++++++++++++++++---------- hw/vfio/pci.h | 2 ++ 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index e6569a7968..5f7ce91519 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -390,12 +390,16 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, b= ool msix) * MSI-X mask and pending bits are emulated, so we want to use the * KVM signaling path only when configured and unmasked. */ - if (vdev->msi_vectors[i].use) { - if (vdev->msi_vectors[i].virq < 0 || - (msix && msix_is_masked(&vdev->pdev, i))) { - fd =3D event_notifier_get_fd(&vdev->msi_vectors[i].interru= pt); + VFIOMSIVector *vector =3D &vdev->msi_vectors[i]; + if (vector->use) { + if (vector->virq < 0 || + (msix && msix_is_masked(&vdev->pdev, i))) + { + vector->kvm_path_active =3D false; + fd =3D event_notifier_get_fd(&vector->interrupt); } else { - fd =3D event_notifier_get_fd(&vdev->msi_vectors[i].kvm_int= errupt); + vector->kvm_path_active =3D true; + fd =3D event_notifier_get_fd(&vector->kvm_interrupt); } } =20 @@ -509,17 +513,23 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, u= nsigned int nr, } else { Error *err =3D NULL; int32_t fd; + bool kvm_path; =20 if (vector->virq >=3D 0) { fd =3D event_notifier_get_fd(&vector->kvm_interrupt); + kvm_path =3D true; } else { fd =3D event_notifier_get_fd(&vector->interrupt); + kvm_path =3D false; } =20 - if (vfio_set_irq_signaling(&vdev->vbasedev, - VFIO_PCI_MSIX_IRQ_INDEX, nr, - VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err= )) { - error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); + if (vector->kvm_path_active !=3D kvm_path) { + if (vfio_set_irq_signaling(&vdev->vbasedev, + VFIO_PCI_MSIX_IRQ_INDEX, nr, + VFIO_IRQ_SET_ACTION_TRIGGER, fd, &e= rr)) { + error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.nam= e); + } + vector->kvm_path_active =3D kvm_path; } } =20 @@ -555,13 +565,15 @@ static void vfio_msix_vector_release(PCIDevice *pdev,= unsigned int nr) * core will mask the interrupt and set pending bits, allowing it to * be re-asserted on unmask. Nothing to do if already using QEMU mode. */ - if (vector->virq >=3D 0) { + if (vector->virq >=3D 0 && vector->kvm_path_active) { int32_t fd =3D event_notifier_get_fd(&vector->interrupt); Error *err =3D NULL; =20 if (vfio_set_irq_signaling(&vdev->vbasedev, VFIO_PCI_MSIX_IRQ_INDE= X, nr, VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err))= { error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); + } else { + vector->kvm_path_active =3D false; } } } diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index b329d50338..b01d2676cf 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -91,6 +91,8 @@ typedef struct VFIOMSIVector { */ EventNotifier interrupt; EventNotifier kvm_interrupt; + /* Set when the trigger action is set to the KVM bypass FD */ + bool kvm_path_active; struct VFIOPCIDevice *vdev; /* back pointer to device */ int virq; bool use; --=20 2.22.0