From nobody Thu Nov 13 20:44:49 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1582906578; cv=none; d=zohomail.com; s=zohoarc; b=iNf0Lbqp26EFdgGSbUUmI4CxJEm1e5JvE5lb3ohf/9Y1M0H+LcPiRAr8tAaPjS0e0HOTacWqLqRrcd/PoE3vquXwz6XAs8vvBRo+BoUor9nxlBr8aNQ1ODNYigAIf0OVnZB1LHhhc/iIhDxgz/MP8AlSW13fBYyLl28RwgIlKVk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1582906578; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=rX4c44rhuMkGop3AxusgS1sFdsudEKDgbJvrNqbBkWs=; b=CL/I4/9qLFQWXMr4c89qlV5YSj7Q+rVupA0MSewMXVQrkTDDQe3yJnm4yQy0LWCjVp6Phl+LpYI4GFNyMMlEdSUppDQ0cHtk0aUbaUjwwGBVfzct4VaC5VoL0qgAgpRGJNbbLPj3TCm2oYKtMBrtnCFWS9Eg3Elm0I1LQGugNlI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1582906578366703.4383153648657; Fri, 28 Feb 2020 08:16:18 -0800 (PST) Received: from localhost ([::1]:49535 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7iJF-0000Mf-1r for importer@patchew.org; Fri, 28 Feb 2020 11:16:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53003) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7iIG-0007SD-CT for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j7iIF-0006L3-DK for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:16 -0500 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:56226 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1j7iIF-0006Kh-9V for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:15 -0500 Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-207-GZQ59ETzMJuTWCYxL-Fpkg-1; Fri, 28 Feb 2020 11:15:12 -0500 Received: by mail-qk1-f199.google.com with SMTP id m25so3299578qka.4 for ; Fri, 28 Feb 2020 08:15:12 -0800 (PST) Received: from xz-x1.redhat.com ([2607:9880:19c0:32::2]) by smtp.gmail.com with ESMTPSA id b23sm4006290qkc.39.2020.02.28.08.15.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Feb 2020 08:15:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582906514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rX4c44rhuMkGop3AxusgS1sFdsudEKDgbJvrNqbBkWs=; b=FETWvAY4H47B5w5Msf+D1CCvGu+rve5k7gNOa6dU9HvunDiIHvTS2+1xkn/4fmzFvyHENR fhnzBw3FkqyUdMjf/mknf6DVsWVxYDiFfdBnyg+HOEogNeXSbSJTfPKMlYIfhPawZ/1D0g W2G/VK4JHGTJ/1HfiOTYozZrExMGgwE= X-MC-Unique: GZQ59ETzMJuTWCYxL-Fpkg-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HdePRUJwnDwXnzR9QSorBgIKeVRO36+j82UD9j+Yyjs=; b=Hx3RlwxAYo/cvkZHhMRmWpIeQ4v4ZTNVVGuDVobzJt9wqiDtCnwK4iFuDzzI0psy2g 1Xz+M6cKrJCQE+l2GrSu9zpJP0t7LRC4Hpos/1Eg0p96eDfIGzUVyJTqHWYTOuNGTYA9 BkszU7Ap51Smy1y6m138qKb4VWsDnN2lWByWbWLSqIeTpeJkDKNDMjStjIMLvnVq0+LU dguyVGCdaok36i3JE0CFjCfyoJpvbSzx87gqlbVYyd4sFQqVE5FPiaBIu4JOeYq5M5nl h5ZkQTPxrO7rpCD8HFVdT4ePitRqfiAfk3XM5JRt/cv+jCkrLUynNNgeJ+Aa3Ckg/s9b ZI2w== X-Gm-Message-State: APjAAAXGpglMo7kAURieU3rdiTFJc4uoYdzIt/foNXRtOWKHbI8E03Bt B04LnxbubiGKEdQJ05IXilgzYj+APsBbKSQR4qy70funHGl2gJFMa27Pr1epYPVwvRQNFy6m/d2 oVjBDA0AsXOTF98c= X-Received: by 2002:ac8:777b:: with SMTP id h27mr4846865qtu.251.1582906511260; Fri, 28 Feb 2020 08:15:11 -0800 (PST) X-Google-Smtp-Source: APXvYqz4AOy2Eqao6ETvU3u7KrC6KAxNOZx9vzSUv7t7XDWxf0clXkZgLVXbMk18sRPTwFyQ3Isgow== X-Received: by 2002:ac8:777b:: with SMTP id h27mr4846838qtu.251.1582906511029; Fri, 28 Feb 2020 08:15:11 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 1/5] vfio/pci: Disable INTx fast path if using split irqchip Date: Fri, 28 Feb 2020 11:14:59 -0500 Message-Id: <20200228161503.382656-2-peterx@redhat.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200228161503.382656-1-peterx@redhat.com> References: <20200228161503.382656-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 207.211.31.81 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Alex Williamson , Cornelia Huck , peterx@redhat.com, Eric Auger Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" It's currently broken. Let's use the slow path to at least make it functional. Tested-by: Eric Auger Reviewed-by: Eric Auger Signed-off-by: Peter Xu --- hw/vfio/pci.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 5e75a95129..98e0e0c994 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -128,6 +128,18 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, = Error **errp) return; } =20 + if (kvm_irqchip_is_split()) { + /* + * VFIO INTx is currently not working with split kernel + * irqchip for level triggered interrupts. Go the slow path + * as long as split is enabled so we can be at least + * functional (even with poor performance). + * + * TODO: Remove this after all things fixed up. + */ + return; + } + /* Get to a known interrupt state */ qemu_set_fd_handler(irqfd.fd, NULL, NULL, vdev); vfio_mask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); --=20 2.24.1 From nobody Thu Nov 13 20:44:49 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1582906582; cv=none; d=zohomail.com; s=zohoarc; b=C6hYy7axOGhMUng9SGDUHxhcsA+00LPJR+rpK6bwNb6BY1hHZFlbn5MpbMzWVImY7FPpTXSGDpoY4LqgXAr77QjIF+UCnoDqBNXJlLxkLlzCWRQEBtlR6Y+qNhbKYGjg46/qtuETOTMzRUPZIunR0EB7N3b0S+JE5IMNgGa0taI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1582906582; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=4ssCl7iQooFNz+XiN2eUKax5ODBgAkEJO5ZgF5i78Ug=; b=CKUG3bM6hs+wZNqMVlEHjaIvdoh3M4iPUZctARwFU/y/jNuIFYVpi8Ethpcl76e8hCvC2tZum6lpbhOjqwmzIwOSqxCdy8MdNju22AlqoNEKfFTaMwPbM0YN9+T2P74JIR4XdmQLb9vrxbCcC7VmogjY5aazsyjz/ZciijPCmp0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1582906582795397.7377163011769; Fri, 28 Feb 2020 08:16:22 -0800 (PST) Received: from localhost ([::1]:49536 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7iJJ-0000Un-DM for importer@patchew.org; Fri, 28 Feb 2020 11:16:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53033) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7iIM-0007Ue-0n for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:23 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j7iIK-0006Mp-Q6 for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:21 -0500 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:26340 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1j7iIK-0006MX-MQ for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:20 -0500 Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-180-ifZgCyd7PL2aDfSKPDACOA-1; Fri, 28 Feb 2020 11:15:18 -0500 Received: by mail-qk1-f199.google.com with SMTP id e11so3277342qkl.8 for ; Fri, 28 Feb 2020 08:15:17 -0800 (PST) Received: from xz-x1.redhat.com ([2607:9880:19c0:32::2]) by smtp.gmail.com with ESMTPSA id i4sm5429255qkf.111.2020.02.28.08.15.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Feb 2020 08:15:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582906520; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4ssCl7iQooFNz+XiN2eUKax5ODBgAkEJO5ZgF5i78Ug=; b=hPQZ4bce0zrQkVlwou45mrFDmMRK7LD+6s/nZbalQkLT9rc4umPLRBoSRPk84Vzlhzj4Is Lu7CfU779scGRuIlKOBhfvgjuhT04Bu0o8L+8J4ye/tq0OVZWYPCachaMe4pyiwe1l7vsx EYqeiv00lnQt3nvljBB/3U5RV/+pnDM= X-MC-Unique: ifZgCyd7PL2aDfSKPDACOA-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mCBli3NzZY/CtljfJ87qAC58ePqvWGmUWXkzevfXcgs=; b=HPVDnTuXHA1uvWPXujs+2djKjug7OvR2f5gr9jA8SsMkCZjqxtktiLkqqVup1TFgh0 JhMklYEevdrv8QJYjj8K/U8+7gao5lAo2ROBYtz8VSUAkvhwBzmZlwqwWlZfC8DBa1HF cGSm4aW5ywfWItn5toeCNvVAM5gNKOPGVR/31KIct2UlGjHqweZKeHkQs2SzOryjly5S 2XlYl1npCxzBe8R571GWuS6Lb8xwf58mIntFGDb0AwhIdgl6HjNOiQNCSY4yqzmDHWzB vn/GDHZkGpNBldl6Nu3Wm21l5W6ucL3IbgAKoa7NF80QkqJAqErNF9jGWV8wQwit+hB2 DIXg== X-Gm-Message-State: APjAAAWthFyHCmb0anVhghhCYmT2+3yExTSXh1ZS5AsrnmX/6azxZiZw 6ktyVzaeS9ARjav+qWqcAFJFhhqYCdAG/tgnsQw5f2qpMgdR7iHMX8XdVlpN3S5otw0mPCkfekm OQ+RdHSmIudqpMcw= X-Received: by 2002:aed:204d:: with SMTP id 71mr4927613qta.116.1582906516605; Fri, 28 Feb 2020 08:15:16 -0800 (PST) X-Google-Smtp-Source: APXvYqweWgL8hDVU0jD980U1HfJWmQOKlcRwZpBigQWWORlk46rgl9QatSnTV1wuGgAQJZVJGFYQiw== X-Received: by 2002:aed:204d:: with SMTP id 71mr4927590qta.116.1582906516365; Fri, 28 Feb 2020 08:15:16 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 2/5] vfio/pci: Use kvm_irqchip_add_irqfd_notifier_gsi() for irqfds Date: Fri, 28 Feb 2020 11:15:00 -0500 Message-Id: <20200228161503.382656-3-peterx@redhat.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200228161503.382656-1-peterx@redhat.com> References: <20200228161503.382656-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Alex Williamson , Cornelia Huck , peterx@redhat.com, Eric Auger Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" VFIO is currently the only one left that is not using the generic function (kvm_irqchip_add_irqfd_notifier_gsi()) to register irqfds. Let VFIO use the common framework too. Follow up patches will introduce extra features for kvm irqfd, so that VFIO can easily leverage that after the switch. Reviewed-by: Eric Auger Reviewed-by: Cornelia Huck Signed-off-by: Peter Xu Tested-by: Eric Auger --- hw/vfio/pci.c | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 98e0e0c994..09703362df 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -115,11 +115,7 @@ static void vfio_intx_eoi(VFIODevice *vbasedev) static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp) { #ifdef CONFIG_KVM - struct kvm_irqfd irqfd =3D { - .fd =3D event_notifier_get_fd(&vdev->intx.interrupt), - .gsi =3D vdev->intx.route.irq, - .flags =3D KVM_IRQFD_FLAG_RESAMPLE, - }; + int irq_fd =3D event_notifier_get_fd(&vdev->intx.interrupt); Error *err =3D NULL; =20 if (vdev->no_kvm_intx || !kvm_irqfds_enabled() || @@ -141,7 +137,7 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, E= rror **errp) } =20 /* Get to a known interrupt state */ - qemu_set_fd_handler(irqfd.fd, NULL, NULL, vdev); + qemu_set_fd_handler(irq_fd, NULL, NULL, vdev); vfio_mask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); vdev->intx.pending =3D false; pci_irq_deassert(&vdev->pdev); @@ -152,17 +148,18 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev,= Error **errp) goto fail; } =20 - /* KVM triggers it, VFIO listens for it */ - irqfd.resamplefd =3D event_notifier_get_fd(&vdev->intx.unmask); - - if (kvm_vm_ioctl(kvm_state, KVM_IRQFD, &irqfd)) { + if (kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, + &vdev->intx.interrupt, + &vdev->intx.unmask, + vdev->intx.route.irq)) { error_setg_errno(errp, errno, "failed to setup resample irqfd"); goto fail_irqfd; } =20 if (vfio_set_irq_signaling(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX, 0, VFIO_IRQ_SET_ACTION_UNMASK, - irqfd.resamplefd, &err)) { + event_notifier_get_fd(&vdev->intx.unmask), + &err)) { error_propagate(errp, err); goto fail_vfio; } @@ -177,12 +174,12 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev,= Error **errp) return; =20 fail_vfio: - irqfd.flags =3D KVM_IRQFD_FLAG_DEASSIGN; - kvm_vm_ioctl(kvm_state, KVM_IRQFD, &irqfd); + kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, &vdev->intx.interrupt, + vdev->intx.route.irq); fail_irqfd: event_notifier_cleanup(&vdev->intx.unmask); fail: - qemu_set_fd_handler(irqfd.fd, vfio_intx_interrupt, NULL, vdev); + qemu_set_fd_handler(irq_fd, vfio_intx_interrupt, NULL, vdev); vfio_unmask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); #endif } @@ -190,12 +187,6 @@ fail: static void vfio_intx_disable_kvm(VFIOPCIDevice *vdev) { #ifdef CONFIG_KVM - struct kvm_irqfd irqfd =3D { - .fd =3D event_notifier_get_fd(&vdev->intx.interrupt), - .gsi =3D vdev->intx.route.irq, - .flags =3D KVM_IRQFD_FLAG_DEASSIGN, - }; - if (!vdev->intx.kvm_accel) { return; } @@ -209,7 +200,8 @@ static void vfio_intx_disable_kvm(VFIOPCIDevice *vdev) pci_irq_deassert(&vdev->pdev); =20 /* Tell KVM to stop listening for an INTx irqfd */ - if (kvm_vm_ioctl(kvm_state, KVM_IRQFD, &irqfd)) { + if (kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, &vdev->intx.inter= rupt, + vdev->intx.route.irq)) { error_report("vfio: Error: Failed to disable INTx irqfd: %m"); } =20 @@ -217,7 +209,8 @@ static void vfio_intx_disable_kvm(VFIOPCIDevice *vdev) event_notifier_cleanup(&vdev->intx.unmask); =20 /* QEMU starts listening for interrupt events. */ - qemu_set_fd_handler(irqfd.fd, vfio_intx_interrupt, NULL, vdev); + qemu_set_fd_handler(event_notifier_get_fd(&vdev->intx.interrupt), + vfio_intx_interrupt, NULL, vdev); =20 vdev->intx.kvm_accel =3D false; =20 --=20 2.24.1 From nobody Thu Nov 13 20:44:49 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1582906654; cv=none; d=zohomail.com; s=zohoarc; b=L/JpwYqoCbWeXcAGao3ex3L5iQqDlvAdtwZ00xzRyezCLa0M/X5x7HhH0VYaii6MB/EtxEjS8vIaVn01qsKczLamnpuhiC5xU2GqyqK9gR9InroDRqzzkANjzG9kkAU14Xtsxtnt7z5OgXU8z8S1GOpNwx4Ams4Wh+GtEz3kAns= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1582906654; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=TAbf6Xk4iOWYyl7F2sdd+EDJE3ppod0snPr2a8tOFFs=; b=WY/osYWqtvOQTm4Gbzmi00I3FvyOzDgDu7Tme4vhsx3dp2YdCgQB2wByJ2l0MQHbaK3xO+2UxUXew4Rm44byU/S6DXqXVner03lz/B8tPR1dHq3k7OfE/hfAwtpLXhD/nyk5q5nzCNg1bhP3zWk7HpgCdJprJUdTo1TKSxdfRA4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1582906654585731.1847224638449; Fri, 28 Feb 2020 08:17:34 -0800 (PST) Received: from localhost ([::1]:49560 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7iKT-0002fR-FO for importer@patchew.org; Fri, 28 Feb 2020 11:17:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53064) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7iIT-0007kV-RT for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:30 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j7iIS-0006PR-PE for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:29 -0500 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:55445 helo=us-smtp-delivery-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1j7iIS-0006PH-LN for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:28 -0500 Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-186-i8i8limWPCi_8cAc4CJjlg-1; Fri, 28 Feb 2020 11:15:26 -0500 Received: by mail-qt1-f199.google.com with SMTP id o24so3277975qtr.17 for ; Fri, 28 Feb 2020 08:15:25 -0800 (PST) Received: from xz-x1.redhat.com ([2607:9880:19c0:32::2]) by smtp.gmail.com with ESMTPSA id i64sm3529920qke.56.2020.02.28.08.15.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Feb 2020 08:15:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582906528; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TAbf6Xk4iOWYyl7F2sdd+EDJE3ppod0snPr2a8tOFFs=; b=OJOlaAiU/tYwlWTLg+jp8P2Wwh3nPzVNyPvxldVjWmioG9nRP+5+rNvc37Y02dglfCms0h zNQTMzswAUMqpSuAThkJgaOlGOyDRAfaVXKXLKxmvQA+YU4LtjV2M5tln37zR8m2PMsUXz FfXBXHTlBh0XJ7yiiRrFh791IOyfhWI= X-MC-Unique: i8i8limWPCi_8cAc4CJjlg-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2pYteEERbnpshws/vnpLsSSgy0J+668YOCAjLaOgJUQ=; b=U9CI+o1OA+xfCVJBa5sVqutvMrbLnd+PfNnfbUtAs64NettlmflBc/PaRnt5b96/dv EfzwoUaqQKCGzbxByCp8jJXT9fO9masicTzdjiOPJr2qGIE8G/uVpSDuDeZyj+6JZCkq Yz+qhZwXiBkRhOYiuKiNEwfrGHoaHhDZ8wgR590TVTS5eyfGwQxyzjT7SuMQSorgTl06 tMrsX6/NZHJ0Ct6Vwx5HdWwEHXysYLWUbH0/inma74uOVT+U3e7rKzaZfKwi68K5zSeb TFN7lZ75hUAgSKiczx1ZmLp1/R/NlT83EjNW46zJ3hVBI7Re+P5xHTZ2UUFUmmAJGm80 MLBg== X-Gm-Message-State: APjAAAU9+UD/5MxbmSUjv1rOJcRtJheDfELi0oDZe00B0GVXyOyvjNo1 nGGfmb9c3wmobwKKr+2mzUKvL18bCbjcAV8nnZu3ZaXnYptei5oUdQLnbyOt2KyfUmPvDKcLx1o VOAQhbTNN6+JU1uY= X-Received: by 2002:ac8:7b45:: with SMTP id m5mr5044313qtu.360.1582906523755; Fri, 28 Feb 2020 08:15:23 -0800 (PST) X-Google-Smtp-Source: APXvYqwaPZje3agvO21Nn6toxdqXRjqIQnZ0tY+UbrlE4c798qJnVTHBDfs8uSpJlnpkKxmngOn56Q== X-Received: by 2002:ac8:7b45:: with SMTP id m5mr5044148qtu.360.1582906521714; Fri, 28 Feb 2020 08:15:21 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 3/5] KVM: Pass EventNotifier into kvm_irqchip_assign_irqfd Date: Fri, 28 Feb 2020 11:15:01 -0500 Message-Id: <20200228161503.382656-4-peterx@redhat.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200228161503.382656-1-peterx@redhat.com> References: <20200228161503.382656-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Alex Williamson , Cornelia Huck , peterx@redhat.com, Eric Auger Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" So that kvm_irqchip_assign_irqfd() can have access to the EventNotifiers, especially the resample event. It is needed in follow up patch to cache and kick resamplefds from QEMU. Reviewed-by: Eric Auger Signed-off-by: Peter Xu Tested-by: Eric Auger --- accel/kvm/kvm-all.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 439a4efe52..d49b74512a 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -1628,9 +1628,13 @@ int kvm_irqchip_update_msi_route(KVMState *s, int vi= rq, MSIMessage msg, return kvm_update_routing_entry(s, &kroute); } =20 -static int kvm_irqchip_assign_irqfd(KVMState *s, int fd, int rfd, int virq, +static int kvm_irqchip_assign_irqfd(KVMState *s, EventNotifier *event, + EventNotifier *resample, int virq, bool assign) { + int fd =3D event_notifier_get_fd(event); + int rfd =3D resample ? event_notifier_get_fd(resample) : -1; + struct kvm_irqfd irqfd =3D { .fd =3D fd, .gsi =3D virq, @@ -1735,7 +1739,9 @@ int kvm_irqchip_add_hv_sint_route(KVMState *s, uint32= _t vcpu, uint32_t sint) return -ENOSYS; } =20 -static int kvm_irqchip_assign_irqfd(KVMState *s, int fd, int virq, bool as= sign) +static int kvm_irqchip_assign_irqfd(KVMState *s, EventNotifier *event, + EventNotifier *resample, int virq, + bool assign) { abort(); } @@ -1749,15 +1755,13 @@ int kvm_irqchip_update_msi_route(KVMState *s, int v= irq, MSIMessage msg) int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n, EventNotifier *rn, int virq) { - return kvm_irqchip_assign_irqfd(s, event_notifier_get_fd(n), - rn ? event_notifier_get_fd(rn) : -1, virq, true); + return kvm_irqchip_assign_irqfd(s, n, rn, virq, true); } =20 int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n, int virq) { - return kvm_irqchip_assign_irqfd(s, event_notifier_get_fd(n), -1, virq, - false); + return kvm_irqchip_assign_irqfd(s, n, NULL, virq, false); } =20 int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n, --=20 2.24.1 From nobody Thu Nov 13 20:44:49 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1582906650; cv=none; d=zohomail.com; s=zohoarc; b=gdA7Kcd1Lv1vjjhigFYYSLWwQTIgmxLUqWnFTGKt7qad5aKMnXYTzujvVXaz5fRTLaElGclHURWsLCKD8HCzbD8Mh586YHZPxqbBVGgRpIPT4ruReIF3amxPY4gKhoUqRGc5mbGXHRJrBPddUJZk8EHWl85mBzhl3Rcgq69VmUQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1582906650; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=azQI//x1XHTCYM2Kx0VH4YlnHHSvW/fbim95OF/v5Tg=; b=WqVPfGdZKTwBO565zLhsEQ78hzWho5EAgdqBqlVZOKNOIPutZPcF4buGLLNvul6JpvZnbqb8YP2bpGrTb9Gb9+MkW0RUxhsrYMy/j8SBGb6/QVn1JEeEWy2p3s1tAORzyL8j7RKfTUXxbeeysh/sF72nGsChJPeUTfTuzZfJsGY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1582906650987147.13923258716875; Fri, 28 Feb 2020 08:17:30 -0800 (PST) Received: from localhost ([::1]:49558 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7iKP-0002Wi-Pe for importer@patchew.org; Fri, 28 Feb 2020 11:17:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53080) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7iIX-0007u1-I3 for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j7iIV-0006Qa-S4 for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:33 -0500 Received: from us-smtp-2.mimecast.com ([205.139.110.61]:36770 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1j7iIV-0006QQ-N6 for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:31 -0500 Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-20-aqovlrubNnuVdQdBo9MfRA-1; Fri, 28 Feb 2020 11:15:29 -0500 Received: by mail-qk1-f197.google.com with SMTP id t186so3277211qkf.9 for ; Fri, 28 Feb 2020 08:15:29 -0800 (PST) Received: from xz-x1.redhat.com ([2607:9880:19c0:32::2]) by smtp.gmail.com with ESMTPSA id v9sm5172492qkv.79.2020.02.28.08.15.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Feb 2020 08:15:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582906531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=azQI//x1XHTCYM2Kx0VH4YlnHHSvW/fbim95OF/v5Tg=; b=YPK0GPyRyrYGXq4wZY52e3tJNbw4y/MmRsT3FzCZTECgDHWmSaXuSi580+FuhGdFZDFZzE DUQ46Nv3AsP5fnA1KE1n+PW8kHC54OeL+92d8fcHLm3Zb953hTRAuHnT3qL/bo600oQXj0 2erp/XHs6nWL0JeDYiSWT2SzgNXTqLs= X-MC-Unique: aqovlrubNnuVdQdBo9MfRA-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8c1sdU/yqohOemAqqHR7iCiEMoLJYx26Ueh5v2B5KQU=; b=MXFTDG0du7QU3xghhaTo5RV1ziGvhc2ctVfERdhCn80bEt+/g792HRdsnpBP7dyqnM m7TVecE1BOMjksmfgTEiFoQvVOoUNQRlBZ47p15d6ANJh3bZbh1tgPIQsZzZjg5ZCcxt IRLab6c9YNHz5cRcIyk/HmJ5HK42AF8icy9awBQeimF9k2oeDEqKnU1wSNmlflYnaxsJ jYWs8e7YV0SfzSCrquS6QweymgzIXnDSBv0VLRWKAUSfyF9eIx/MrjduGak3BMOmlM30 TaA3S045g8Jnm7Mk2fHs0Yg1GpPTW0/QZJObo0TwV9ZK/jLAqvjEpVU0wyXY3NHUcpL2 LmQA== X-Gm-Message-State: APjAAAUCLOTThn3aKOIDF+Tx7gjhjegz/ddI3/Fn3PabgwdC5wISSN4p Fk1HR24gsN/OFbag9wuMbisBnTAPaio2X2PLyC/gl0odVV9sSPiH6H3vdWA8JHgR2h5YbTJewTa XGioZ4qpb7JTCSF8= X-Received: by 2002:ad4:58b3:: with SMTP id ea19mr4490678qvb.80.1582906528171; Fri, 28 Feb 2020 08:15:28 -0800 (PST) X-Google-Smtp-Source: APXvYqx95/PxjR+vuf+l7cHAZ2lh39ZXkbL4hbTQVDfvUt7LFM+FgRDXHh8xolLoj910tzwwgT4jQw== X-Received: by 2002:ad4:58b3:: with SMTP id ea19mr4490646qvb.80.1582906527743; Fri, 28 Feb 2020 08:15:27 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 4/5] KVM: Kick resamplefd for split kernel irqchip Date: Fri, 28 Feb 2020 11:15:02 -0500 Message-Id: <20200228161503.382656-5-peterx@redhat.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200228161503.382656-1-peterx@redhat.com> References: <20200228161503.382656-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.61 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Alex Williamson , Cornelia Huck , peterx@redhat.com, Eric Auger Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" This is majorly only for X86 because that's the only one that supports split irqchip for now. When the irqchip is split, we face a dilemma that KVM irqfd will be enabled, however the slow irqchip is still running in the userspace. It means that the resamplefd in the kernel irqfds won't take any effect and it will miss to ack INTx interrupts on EOIs. One example is split irqchip with VFIO INTx, which will break if we use the VFIO INTx fast path. This patch can potentially supports the VFIO fast path again for INTx, that the IRQ delivery will still use the fast path, while we don't need to trap MMIOs in QEMU for the device to emulate the EIOs (see the callers of vfio_eoi() hook). However the EOI of the INTx will still need to be done from the userspace by caching all the resamplefds in QEMU and kick properly for IOAPIC EOI broadcast. This is tricky because in this case the userspace ioapic irr & remote-irr will be bypassed. However such a change will greatly boost performance for assigned devices using INTx irqs (TCP_RR boosts 46% after this patch applied). When the userspace is responsible for the resamplefd kickup, don't register it on the kvm_irqfd anymore, because on newer kernels (after commit 654f1f13ea56, 5.2+) the KVM_IRQFD will fail if with both split irqchip and resamplefd. This will make sure that the fast path will work for all supported kernels. https://patchwork.kernel.org/patch/10738541/#22609933 Suggested-by: Paolo Bonzini Signed-off-by: Peter Xu Reviewed-by: Eric Auger Tested-by: Eric Auger --- accel/kvm/kvm-all.c | 85 +++++++++++++++++++++++++++++++++++++++++- accel/kvm/trace-events | 1 + hw/intc/ioapic.c | 23 +++++++++++- include/sysemu/kvm.h | 7 ++++ 4 files changed, 112 insertions(+), 4 deletions(-) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index d49b74512a..89771ea114 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -159,9 +159,65 @@ static const KVMCapabilityInfo kvm_required_capabilite= s[] =3D { static NotifierList kvm_irqchip_change_notifiers =3D NOTIFIER_LIST_INITIALIZER(kvm_irqchip_change_notifiers); =20 +struct KVMResampleFd { + int gsi; + EventNotifier *resample_event; + QLIST_ENTRY(KVMResampleFd) node; +}; +typedef struct KVMResampleFd KVMResampleFd; + +/* + * Only used with split irqchip where we need to do the resample fd + * kick for the kernel from userspace. + */ +static QLIST_HEAD(, KVMResampleFd) kvm_resample_fd_list =3D + QLIST_HEAD_INITIALIZER(kvm_resample_fd_list); + #define kvm_slots_lock(kml) qemu_mutex_lock(&(kml)->slots_lock) #define kvm_slots_unlock(kml) qemu_mutex_unlock(&(kml)->slots_lock) =20 +static inline void kvm_resample_fd_remove(int gsi) +{ + KVMResampleFd *rfd; + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { + if (rfd->gsi =3D=3D gsi) { + QLIST_REMOVE(rfd, node); + g_free(rfd); + break; + } + } +} + +static inline void kvm_resample_fd_insert(int gsi, EventNotifier *event) +{ + KVMResampleFd *rfd =3D g_new0(KVMResampleFd, 1); + + rfd->gsi =3D gsi; + rfd->resample_event =3D event; + + QLIST_INSERT_HEAD(&kvm_resample_fd_list, rfd, node); +} + +bool kvm_resample_fd_notify(int gsi) +{ + KVMResampleFd *rfd; + + if (!kvm_irqchip_is_split()) { + return false; + } + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { + if (rfd->gsi =3D=3D gsi) { + event_notifier_set(rfd->resample_event); + trace_kvm_resample_fd_notify(gsi); + return true; + } + } + + return false; +} + int kvm_get_max_memslots(void) { KVMState *s =3D KVM_STATE(current_accel()); @@ -1642,8 +1698,33 @@ static int kvm_irqchip_assign_irqfd(KVMState *s, Eve= ntNotifier *event, }; =20 if (rfd !=3D -1) { - irqfd.flags |=3D KVM_IRQFD_FLAG_RESAMPLE; - irqfd.resamplefd =3D rfd; + assert(assign); + if (kvm_irqchip_is_split()) { + /* + * When the slow irqchip (e.g. IOAPIC) is in the + * userspace, KVM kernel resamplefd will not work because + * the EOI of the interrupt will be delivered to userspace + * instead, so the KVM kernel resamplefd kick will be + * skipped. The userspace here mimics what the kernel + * provides with resamplefd, remember the resamplefd and + * kick it when we receive EOI of this IRQ. + * + * This is hackery because IOAPIC is mostly bypassed + * (except EOI broadcasts) when irqfd is used. However + * this can bring much performance back for split irqchip + * with INTx IRQs (for VFIO, this gives 93% perf of the + * full fast path, which is 46% perf boost comparing to + * the INTx slow path). + */ + kvm_resample_fd_insert(virq, resample); + } else { + irqfd.flags |=3D KVM_IRQFD_FLAG_RESAMPLE; + irqfd.resamplefd =3D rfd; + } + } else if (!assign) { + if (kvm_irqchip_is_split()) { + kvm_resample_fd_remove(virq); + } } =20 if (!kvm_irqfds_enabled()) { diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events index 4fb6e59d19..a68eb66534 100644 --- a/accel/kvm/trace-events +++ b/accel/kvm/trace-events @@ -16,4 +16,5 @@ kvm_set_ioeventfd_mmio(int fd, uint64_t addr, uint32_t va= l, bool assign, uint32_ kvm_set_ioeventfd_pio(int fd, uint16_t addr, uint32_t val, bool assign, ui= nt32_t size, bool datamatch) "fd: %d @0x%x val=3D0x%x assign: %d size: %d m= atch: %d" kvm_set_user_memory(uint32_t slot, uint32_t flags, uint64_t guest_phys_add= r, uint64_t memory_size, uint64_t userspace_addr, int ret) "Slot#%d flags= =3D0x%x gpa=3D0x%"PRIx64 " size=3D0x%"PRIx64 " ua=3D0x%"PRIx64 " ret=3D%d" kvm_clear_dirty_log(uint32_t slot, uint64_t start, uint32_t size) "slot#%"= PRId32" start 0x%"PRIx64" size 0x%"PRIx32 +kvm_resample_fd_notify(int gsi) "gsi %d" =20 diff --git a/hw/intc/ioapic.c b/hw/intc/ioapic.c index 15747fe2c2..13921b333d 100644 --- a/hw/intc/ioapic.c +++ b/hw/intc/ioapic.c @@ -236,8 +236,27 @@ void ioapic_eoi_broadcast(int vector) for (n =3D 0; n < IOAPIC_NUM_PINS; n++) { entry =3D s->ioredtbl[n]; =20 - if ((entry & IOAPIC_VECTOR_MASK) !=3D vector || - ((entry >> IOAPIC_LVT_TRIGGER_MODE_SHIFT) & 1) !=3D IOAPIC= _TRIGGER_LEVEL) { + if ((entry & IOAPIC_VECTOR_MASK) !=3D vector) { + continue; + } + + /* + * When IOAPIC is in the userspace while APIC is still in + * the kernel (i.e., split irqchip), we have a trick to + * kick the resamplefd logic for registered irqfds from + * userspace to deactivate the IRQ. When that happens, it + * means the irq bypassed userspace IOAPIC (so the irr and + * remote-irr of the table entry should be bypassed too + * even if interrupt come), then we don't need to clear + * the remote-IRR and check irr again because they'll + * always be zeros. + */ + if (kvm_resample_fd_notify(n)) { + continue; + } + + if (((entry >> IOAPIC_LVT_TRIGGER_MODE_SHIFT) & 1) !=3D + IOAPIC_TRIGGER_LEVEL) { continue; } =20 diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index 141342de98..3f0830cc4f 100644 --- a/include/sysemu/kvm.h +++ b/include/sysemu/kvm.h @@ -555,4 +555,11 @@ int kvm_set_one_reg(CPUState *cs, uint64_t id, void *s= ource); int kvm_get_one_reg(CPUState *cs, uint64_t id, void *target); struct ppc_radix_page_info *kvm_get_radix_page_info(void); int kvm_get_max_memslots(void); + +/* + * Notify resamplefd for EOI of specific interrupts. Returns true + * when one resamplefd is notified, false if no such IRQ found. + */ +bool kvm_resample_fd_notify(int gsi); + #endif --=20 2.24.1 From nobody Thu Nov 13 20:44:49 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1582906725; cv=none; d=zohomail.com; s=zohoarc; b=h905fuSZhvYRPQa/sULmmBsk3ouz98ee9xq9UfX5Q4wmkw51BWkVCcbGQDKTZ7Au2EnoTBjAOZ2oeuCr9YzOr35o08UgNSsYW030n0br1j5ev9hRfK8OlHv8Aj/nyk+eIdLj38swLrSMzWW/YVj+LC/4/ftHGiGl7yoor+MBYNw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1582906725; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=67BJoWVgjqRaE3ez8MIcs6ZgskV92ME7JrNNMKV0joI=; b=TlWIYmD2vUUwmhEEJaig+T39s3809w4mr5Bj5tN0EqTxRc48Z15N0Ba+hOz9bb4s347NP4Z/ivfiIZ7kHj/ZSIXpwrpzh+ym/yABoYSC4kAyObibdWa5lW9VdPkUf+UDPje6J2dts6A3XaRxbBx4ZmxHPxAuBBExx0HtogPyMWI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1582906725920824.0163376597525; Fri, 28 Feb 2020 08:18:45 -0800 (PST) Received: from localhost ([::1]:49572 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7iLc-0003zk-NJ for importer@patchew.org; Fri, 28 Feb 2020 11:18:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:53091) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j7iIc-00085Y-Pe for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j7iIb-0006SN-Du for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:38 -0500 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:20666 helo=us-smtp-1.mimecast.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1j7iIb-0006S7-AN for qemu-devel@nongnu.org; Fri, 28 Feb 2020 11:15:37 -0500 Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-76-lM5xUSESM42CwGD_AC5NJQ-1; Fri, 28 Feb 2020 11:15:34 -0500 Received: by mail-qk1-f199.google.com with SMTP id h6so3258259qkj.14 for ; Fri, 28 Feb 2020 08:15:34 -0800 (PST) Received: from xz-x1.redhat.com ([2607:9880:19c0:32::2]) by smtp.gmail.com with ESMTPSA id g62sm5257277qkd.25.2020.02.28.08.15.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Feb 2020 08:15:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582906536; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=67BJoWVgjqRaE3ez8MIcs6ZgskV92ME7JrNNMKV0joI=; b=jNV1R+Lc7RW68IQ+Jp3dOoORpO4OAqAx6Zc6nYKIb0Sa+56Bzc2YEaayEwhqsnwADqTHUp HSRFwnLtIBV5cMbx6Bt7OT/Kwx9O6e+Nka5zRirXPhtBoOl1azTaGWYo9VcVko35qoUt8n t83uDhtntIBzFves9x7e+X/vOkS2cUo= X-MC-Unique: lM5xUSESM42CwGD_AC5NJQ-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wQ1AClskklPLaQx5THQ8KFrAAijnqVsY12r4IQjRE6M=; b=EZ7OTWmzO5YfxE5BGV76i7aXbFyskZqWLA55yluMqrzm71hyPyOVZWoRzIhLpHyuw1 fJPOYDzAlsl3LLAfwK7prIQQBG/bPY6keNUXsYWQJtxjYYCLIUcWU9PES2Ali0FZplfy ExTWUfT9JlGVqyX4aPX3ObSkLOOe3zzDYUOCJUa9gTXJMB0s3VmOT/XLhFBWCG5X6IR1 1fFAeGk9jgd/IcK7aK0xTZUEk9sETY4m9VIzfKUW0KF8CP+AZS+HcCuCX0wsvYMXjThz kToXRA7ypfsVTR8i+IM0de+XwQd5TfpFd2H0lbnqTWqzvuKm3RJj7ccQ/6wqMaNIXu44 rFIA== X-Gm-Message-State: APjAAAWh4daoempmtr/2p45j21NKUGO1bF99Z7WJMtuZqASwiH4hZUqZ oVULdHqoa7sOG7CaHJNA8c2KjI6znYo9HPj76Au6PQLiLhqlRXXYPYfBYal8ebHheqnrbQ34UtC Ot6Ir9SPWwakOw90= X-Received: by 2002:ae9:e115:: with SMTP id g21mr5146093qkm.83.1582906533022; Fri, 28 Feb 2020 08:15:33 -0800 (PST) X-Google-Smtp-Source: APXvYqzhnNx7ZU7ZRIwo9V0nLkFwIxTLpKRePAlVl/GsyK84AfaA/bps4j76vCiMQqZA5qbIMPaKsA== X-Received: by 2002:ae9:e115:: with SMTP id g21mr5146074qkm.83.1582906532826; Fri, 28 Feb 2020 08:15:32 -0800 (PST) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v2 5/5] Revert "vfio/pci: Disable INTx fast path if using split irqchip" Date: Fri, 28 Feb 2020 11:15:03 -0500 Message-Id: <20200228161503.382656-6-peterx@redhat.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200228161503.382656-1-peterx@redhat.com> References: <20200228161503.382656-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 205.139.110.120 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Alex Williamson , Cornelia Huck , peterx@redhat.com, Eric Auger Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" With the resamplefd list introduced, we can savely enable VFIO INTx fast path again with split irqchip so it can still be faster than the complete slow path. Signed-off-by: Peter Xu Reviewed-by: Eric Auger Tested-by: Eric Auger --- hw/vfio/pci.c | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 09703362df..1c0aa27386 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -124,18 +124,6 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, = Error **errp) return; } =20 - if (kvm_irqchip_is_split()) { - /* - * VFIO INTx is currently not working with split kernel - * irqchip for level triggered interrupts. Go the slow path - * as long as split is enabled so we can be at least - * functional (even with poor performance). - * - * TODO: Remove this after all things fixed up. - */ - return; - } - /* Get to a known interrupt state */ qemu_set_fd_handler(irq_fd, NULL, NULL, vdev); vfio_mask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); --=20 2.24.1