From nobody Sun May 19 19:40:27 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1584474723; cv=none; d=zohomail.com; s=zohoarc; b=Ac1RAvRb9xa/qe9Hqeh7gkjI0Y6eXT48DPOefj8ARyOnlJcwxQRkCY7dnZNZ5uJ9dxhKjz5bNaSMvRLR9wdyNXmq1cGn/k00xaLDkaOl4PKqvQixm7nCxEpQTcjtMfbHyBTy9iWbJyb7XZrqEwigtW2pHWBKiATKrY0f5IluCcU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584474723; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=rX4c44rhuMkGop3AxusgS1sFdsudEKDgbJvrNqbBkWs=; b=lyy5bEWRzx2GCj+VnChWPwJ5CSZU8A62vF2UxerYwV4bpjMlkwWW8CiC9paz5uwM2uWlPzeco2beVZ4NBGubGa1NMgYwB50Jx+aro+bEUPxP42cGU3DqlN7qK8pgwnEDazDgaHxh8mMYJDV6vIiWVoZo1m28KDIiVfEC1dBi9MY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584474723509511.9397727817068; Tue, 17 Mar 2020 12:52:03 -0700 (PDT) Received: from localhost ([::1]:40494 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jEIFu-0000KA-Al for importer@patchew.org; Tue, 17 Mar 2020 15:52:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34270) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jEIEq-0007kI-HZ for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:50:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jEIEp-0001qL-JW for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:50:56 -0400 Received: from us-smtp-delivery-74.mimecast.com ([63.128.21.74]:41161) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jEIEp-0001p0-Fh for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:50:55 -0400 Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-58-6R5yaCiDORKM5GGsExcYRg-1; Tue, 17 Mar 2020 15:50:53 -0400 Received: by mail-wr1-f70.google.com with SMTP id t10so4872713wrp.15 for ; Tue, 17 Mar 2020 12:50:53 -0700 (PDT) Received: from xz-x1.redhat.com ([2607:9880:19c0:32::2]) by smtp.gmail.com with ESMTPSA id z22sm596119wmi.1.2020.03.17.12.50.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Mar 2020 12:50:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584474655; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rX4c44rhuMkGop3AxusgS1sFdsudEKDgbJvrNqbBkWs=; b=Gnns0oqFZ7hoRzRvgrVuwn2CnZj8UQr3Obx2n8jXc0iC1IlNQ8Vd0etADD79u8jxQh/S1b QrvkPwVtMgCuZnEYgeS7HQEPPK3vgp5RjUXfVkTdlU3e81JMT1MGz83fxFpWA0bVbG3h5W cmcsisuyQD1zCQd5EvnM+QZt46NwtRQ= X-MC-Unique: 6R5yaCiDORKM5GGsExcYRg-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HdePRUJwnDwXnzR9QSorBgIKeVRO36+j82UD9j+Yyjs=; b=FUzD2JwJBVmW9BMjhB7QOtsN6wy3+nJBccEpXGqSDX0hrv7+fyn+vkJ/H2l87wrkzo cmcEhmRzilYsFF72aDVtgI+QDD9Sc3wkXZaN28mb41EFbWmj5JaZKKAPTonD0yzfllS3 BYuqS+unXwu/VOIfHuBDq1jD7evNE7t09pqN4Ter6jNf2Ua2uWnrGE99aLHRGf8Ga2Gx aa+d8rYWUUjwnbiCnhXPcg0t2JuhSX6yv5UPfmLjvMSZH1JV2KORx5W8Nf5gd6bvL47F iuNTuAXJNlLNPGbEbJeW3GCe1ZhRV1CMA4tGQVFK2RtBJIhYWgMKXajoX4lwHjrZRyba eHHw== X-Gm-Message-State: ANhLgQ2PxeLMg+awg5YfZRE/NBvjzEqSBNl3xTJ30fj4q0qmDEoVPSaj jdE/ZkowKbXlN+dRkHRkJPIkNeRW210j3g9ZeawmxDEEccKr1JXswOkavlUAIsOiVHW9JuJ7IFZ E7/RX6Rip0pLvvY0= X-Received: by 2002:adf:ec82:: with SMTP id z2mr724397wrn.302.1584474652093; Tue, 17 Mar 2020 12:50:52 -0700 (PDT) X-Google-Smtp-Source: ADFU+vvJGi70HBi0Xf5GQp5SQ6NguwAnJKgP9G9b/auYTiWn1NytsHjMbFFk8bzOLn5JuP+g615MNQ== X-Received: by 2002:adf:ec82:: with SMTP id z2mr724370wrn.302.1584474651780; Tue, 17 Mar 2020 12:50:51 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v3 1/5] vfio/pci: Disable INTx fast path if using split irqchip Date: Tue, 17 Mar 2020 15:50:38 -0400 Message-Id: <20200317195042.282977-2-peterx@redhat.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200317195042.282977-1-peterx@redhat.com> References: <20200317195042.282977-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 63.128.21.74 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Alex Williamson , Cornelia Huck , peterx@redhat.com, Eric Auger Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" It's currently broken. Let's use the slow path to at least make it functional. Tested-by: Eric Auger Reviewed-by: Eric Auger Signed-off-by: Peter Xu Acked-by: Alex Williamson Reviewed-by: Alex Williamson --- hw/vfio/pci.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 5e75a95129..98e0e0c994 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -128,6 +128,18 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, = Error **errp) return; } =20 + if (kvm_irqchip_is_split()) { + /* + * VFIO INTx is currently not working with split kernel + * irqchip for level triggered interrupts. Go the slow path + * as long as split is enabled so we can be at least + * functional (even with poor performance). + * + * TODO: Remove this after all things fixed up. + */ + return; + } + /* Get to a known interrupt state */ qemu_set_fd_handler(irqfd.fd, NULL, NULL, vdev); vfio_mask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); --=20 2.24.1 From nobody Sun May 19 19:40:27 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1584474811; cv=none; d=zohomail.com; s=zohoarc; b=DMuPdgTupdil/33otCe3jpnTgvih4hcajk6UB3Ptneoc34qxBgTGs02nr8QRYMUm/LZo3k3PD8pCum5lL0tzOHPKgg1/zoJlKDGNNsJNP/IGvKb7+egQ8uoMDCgsdiw3SVN8ErjVNDSwOC3UaSd3Fi4d2LXaUlZunnaWec+nx/k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584474811; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=4ssCl7iQooFNz+XiN2eUKax5ODBgAkEJO5ZgF5i78Ug=; b=aOAYfk5AfvwTXqc/HoQCrDl1VCnoCToNRnuuS21drHhEXB7TQIySJAgmhl+JxAdiFjZbscxXxuAuWvTp5ioVRFOWqd+aGDu5y5YmWYFcYxy6djp89s+p7pPLYV8xVKi2DOpcxWQY5ayPnNMZBhEHtLZd0rYelB0UskUoegIckmQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584474811991200.62665278052805; Tue, 17 Mar 2020 12:53:31 -0700 (PDT) Received: from localhost ([::1]:40510 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jEIHK-0002if-T8 for importer@patchew.org; Tue, 17 Mar 2020 15:53:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34339) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jEIEw-0007uc-MC for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jEIEv-0002GI-CV for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:02 -0400 Received: from us-smtp-delivery-74.mimecast.com ([63.128.21.74]:37499) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jEIEv-0002Fm-8K for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:01 -0400 Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-480-H3lnBTfGNniU_tWCjyL-Wg-1; Tue, 17 Mar 2020 15:50:58 -0400 Received: by mail-wr1-f72.google.com with SMTP id u12so8742104wrw.10 for ; Tue, 17 Mar 2020 12:50:58 -0700 (PDT) Received: from xz-x1.redhat.com ([2607:9880:19c0:32::2]) by smtp.gmail.com with ESMTPSA id q13sm5761347wrs.91.2020.03.17.12.50.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Mar 2020 12:50:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584474660; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4ssCl7iQooFNz+XiN2eUKax5ODBgAkEJO5ZgF5i78Ug=; b=V0lZdH31TVZGWIftqnOFzZJ2gkiHoqEEoDuVSyXKnCHHU9uWpQ8A1ynLJ8kxsVPbhdRf8U 3GOA1Xdxo5qr6r8YCos5ERm1XZBzOgjWSkwigkGCI/Pdj5NZQSDfHW6Q6+GIGKOf2asBaE FIr+bOnM58iR0gxt5ZoXkM0g4nq0RB0= X-MC-Unique: H3lnBTfGNniU_tWCjyL-Wg-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mCBli3NzZY/CtljfJ87qAC58ePqvWGmUWXkzevfXcgs=; b=Rh4wT095i583PfzsZJhs68zihrJ5J/yotZ+od4+ytSyGA+3dYQNYmuUQUvHJU2IiWC ycnDEL+cqxDNWNIbmowxNowiZs7HNUgCkmuf4QXe8AU/O48UQQyeec2RC3A8246pYYI4 oe/TJxfL+E9Tse057Fa0LrEgibSpDS2yuzFBH3e9VnX1kjA1g4ujKLOma3EiOnHjT3kF ZicuqGuHQRsmu81cMR/KAB9KZCFQiW69sqB+ZizUrqQLPGmONS+1Lzcqj5gszp+SPmlm PyqjSbSInTz8XcooaYq5GHRwZ4SIVHC/CXK+R0Yyvwt9zmE+GQPjH/PFh7hsm7UZgLkz V2mg== X-Gm-Message-State: ANhLgQ0DtfGfVTVKpd+/GPgJnuYXqFgcimd0HvP4x8GCZ8/3ptmGfrM5 wnUomKLCMdEAgIQQ66B+0LV/79q8GMDfclqNqxxkSD3PxOGBquuBb+GIthvydAtBiAUHgpAWSMA EwffX/c45yNy13DE= X-Received: by 2002:adf:f8cf:: with SMTP id f15mr646134wrq.79.1584474656923; Tue, 17 Mar 2020 12:50:56 -0700 (PDT) X-Google-Smtp-Source: ADFU+vv/91gc9c9smkJjbXwswK4kfToqysfihPBdTOefR4xQ4Z9sjShqxs789G0TngmZ5kmZmNrEJw== X-Received: by 2002:adf:f8cf:: with SMTP id f15mr646110wrq.79.1584474656606; Tue, 17 Mar 2020 12:50:56 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v3 2/5] vfio/pci: Use kvm_irqchip_add_irqfd_notifier_gsi() for irqfds Date: Tue, 17 Mar 2020 15:50:39 -0400 Message-Id: <20200317195042.282977-3-peterx@redhat.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200317195042.282977-1-peterx@redhat.com> References: <20200317195042.282977-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 63.128.21.74 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Alex Williamson , Cornelia Huck , peterx@redhat.com, Eric Auger Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" VFIO is currently the only one left that is not using the generic function (kvm_irqchip_add_irqfd_notifier_gsi()) to register irqfds. Let VFIO use the common framework too. Follow up patches will introduce extra features for kvm irqfd, so that VFIO can easily leverage that after the switch. Reviewed-by: Eric Auger Reviewed-by: Cornelia Huck Signed-off-by: Peter Xu Acked-by: Alex Williamson Reviewed-by: Alex Williamson --- hw/vfio/pci.c | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 98e0e0c994..09703362df 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -115,11 +115,7 @@ static void vfio_intx_eoi(VFIODevice *vbasedev) static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, Error **errp) { #ifdef CONFIG_KVM - struct kvm_irqfd irqfd =3D { - .fd =3D event_notifier_get_fd(&vdev->intx.interrupt), - .gsi =3D vdev->intx.route.irq, - .flags =3D KVM_IRQFD_FLAG_RESAMPLE, - }; + int irq_fd =3D event_notifier_get_fd(&vdev->intx.interrupt); Error *err =3D NULL; =20 if (vdev->no_kvm_intx || !kvm_irqfds_enabled() || @@ -141,7 +137,7 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, E= rror **errp) } =20 /* Get to a known interrupt state */ - qemu_set_fd_handler(irqfd.fd, NULL, NULL, vdev); + qemu_set_fd_handler(irq_fd, NULL, NULL, vdev); vfio_mask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); vdev->intx.pending =3D false; pci_irq_deassert(&vdev->pdev); @@ -152,17 +148,18 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev,= Error **errp) goto fail; } =20 - /* KVM triggers it, VFIO listens for it */ - irqfd.resamplefd =3D event_notifier_get_fd(&vdev->intx.unmask); - - if (kvm_vm_ioctl(kvm_state, KVM_IRQFD, &irqfd)) { + if (kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, + &vdev->intx.interrupt, + &vdev->intx.unmask, + vdev->intx.route.irq)) { error_setg_errno(errp, errno, "failed to setup resample irqfd"); goto fail_irqfd; } =20 if (vfio_set_irq_signaling(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX, 0, VFIO_IRQ_SET_ACTION_UNMASK, - irqfd.resamplefd, &err)) { + event_notifier_get_fd(&vdev->intx.unmask), + &err)) { error_propagate(errp, err); goto fail_vfio; } @@ -177,12 +174,12 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev,= Error **errp) return; =20 fail_vfio: - irqfd.flags =3D KVM_IRQFD_FLAG_DEASSIGN; - kvm_vm_ioctl(kvm_state, KVM_IRQFD, &irqfd); + kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, &vdev->intx.interrupt, + vdev->intx.route.irq); fail_irqfd: event_notifier_cleanup(&vdev->intx.unmask); fail: - qemu_set_fd_handler(irqfd.fd, vfio_intx_interrupt, NULL, vdev); + qemu_set_fd_handler(irq_fd, vfio_intx_interrupt, NULL, vdev); vfio_unmask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); #endif } @@ -190,12 +187,6 @@ fail: static void vfio_intx_disable_kvm(VFIOPCIDevice *vdev) { #ifdef CONFIG_KVM - struct kvm_irqfd irqfd =3D { - .fd =3D event_notifier_get_fd(&vdev->intx.interrupt), - .gsi =3D vdev->intx.route.irq, - .flags =3D KVM_IRQFD_FLAG_DEASSIGN, - }; - if (!vdev->intx.kvm_accel) { return; } @@ -209,7 +200,8 @@ static void vfio_intx_disable_kvm(VFIOPCIDevice *vdev) pci_irq_deassert(&vdev->pdev); =20 /* Tell KVM to stop listening for an INTx irqfd */ - if (kvm_vm_ioctl(kvm_state, KVM_IRQFD, &irqfd)) { + if (kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, &vdev->intx.inter= rupt, + vdev->intx.route.irq)) { error_report("vfio: Error: Failed to disable INTx irqfd: %m"); } =20 @@ -217,7 +209,8 @@ static void vfio_intx_disable_kvm(VFIOPCIDevice *vdev) event_notifier_cleanup(&vdev->intx.unmask); =20 /* QEMU starts listening for interrupt events. */ - qemu_set_fd_handler(irqfd.fd, vfio_intx_interrupt, NULL, vdev); + qemu_set_fd_handler(event_notifier_get_fd(&vdev->intx.interrupt), + vfio_intx_interrupt, NULL, vdev); =20 vdev->intx.kvm_accel =3D false; =20 --=20 2.24.1 From nobody Sun May 19 19:40:27 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1584474869; cv=none; d=zohomail.com; s=zohoarc; b=BMpih0CtYFc14NoYuzrfBruObKQjflfkbpQAYtURN2KFBZ8UGJLoa/0RV8Vc0ETb8MvI50Bgj4cbwHeoc3mUXEjAWT/T+TxcQCEnuL5hjx80tSRNB6eOU5Cn/+mpXxqeUrd1wBG3+TttDDTtHsk0c8711lTDvh4rOk++MSrFS28= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584474869; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=TAbf6Xk4iOWYyl7F2sdd+EDJE3ppod0snPr2a8tOFFs=; b=jhrtIK2s6er9v+jAUjLY+ms6Xc4usOOlZA7EIBBTDLy1z/d7WKs24nbLTX01IhlbchZbCZHixDYpWUHIEayFRNM0qXk2wmAf+zYdklKdHUx2p9YDExJYe9DIQFbbz4FVDbg1VQa8q6M2IK0gePyk/0FFtfVG7Avy+lbi4AHEixk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584474869970502.5997312817133; Tue, 17 Mar 2020 12:54:29 -0700 (PDT) Received: from localhost ([::1]:40518 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jEIIH-0003za-09 for importer@patchew.org; Tue, 17 Mar 2020 15:54:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34410) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jEIF1-00082d-OV for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jEIF0-0002Wl-Ms for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:07 -0400 Received: from us-smtp-delivery-74.mimecast.com ([216.205.24.74]:23728) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jEIF0-0002VV-Ij for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:06 -0400 Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-105-B9Rq6U7yNJ29w7LpVZkNuA-1; Tue, 17 Mar 2020 15:51:04 -0400 Received: by mail-wm1-f70.google.com with SMTP id g26so182598wmk.6 for ; Tue, 17 Mar 2020 12:51:03 -0700 (PDT) Received: from xz-x1.redhat.com ([2607:9880:19c0:32::2]) by smtp.gmail.com with ESMTPSA id s15sm6280600wrr.45.2020.03.17.12.51.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Mar 2020 12:51:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584474666; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TAbf6Xk4iOWYyl7F2sdd+EDJE3ppod0snPr2a8tOFFs=; b=eHLnU4wmFt7+rox9CcsZgdvhJGTFEAYwC1yEFnzDJzZt+HHAT0Nts06iFti1u+2zbxTQrI BUpk5LFMK0/ZBbdIFUeO0yPH+F55zVU9UcweESl1DrNaoJEpVzTYdWpFjXVn+QzO4XIasA 7czx8MnVtqs1B53lW1Hzk7Vhxy1TrbE= X-MC-Unique: B9Rq6U7yNJ29w7LpVZkNuA-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2pYteEERbnpshws/vnpLsSSgy0J+668YOCAjLaOgJUQ=; b=lrxbz891neLRYiMXxd7fUR/MTUukWZppyeO63QxtrxzmigN0XJ7I7CbIaLvL7LgmRn OKcyQKGzPLFG1eOUWf5q/mlgb+g6UDzlZTUwzm44Uo80PqbVm6pFdfovYtcyG7a/DJ0K pqxNGqnxfqxNAZmeFcq5jnbqt0u7KrUE1gqWLcEgIgKrvfUUSeLkstu3A2DfUGL9d6nW SDTsTQMwKR/BxSxzpQy66odYwhNge/j7CkrKDLa8RYt3Mv1R+v7cb7obVVzsqA4fjNdK bSy9PnfQaGQDHiFC5Zbcnoq+LAkX9XsWbBzUSVyQtPO9ekC6Mb9uLjE98ih9GhoSyfU6 3KDw== X-Gm-Message-State: ANhLgQ0QlG8TQi05iaT+aUYdEQ/lI39jaDTrK6ZCcWgYRZrezo2dsSi8 tGZRUiQTJlia0+IMaVkoT5xCGeQ2BR4NfdJO7xlAYCX+3rqFwOIQFtVN+a4fDS+d3EarTciCqf0 QoBbMGEc0TzwQa0c= X-Received: by 2002:a5d:68d1:: with SMTP id p17mr760178wrw.56.1584474662600; Tue, 17 Mar 2020 12:51:02 -0700 (PDT) X-Google-Smtp-Source: ADFU+vs6D6pf0etYHZvKIYUdqlTyYAIWMqmaWcH9PgQ+F7rYEviILi0He4rObRA/dsPRunKTx+co3Q== X-Received: by 2002:a5d:68d1:: with SMTP id p17mr760163wrw.56.1584474662406; Tue, 17 Mar 2020 12:51:02 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v3 3/5] KVM: Pass EventNotifier into kvm_irqchip_assign_irqfd Date: Tue, 17 Mar 2020 15:50:40 -0400 Message-Id: <20200317195042.282977-4-peterx@redhat.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200317195042.282977-1-peterx@redhat.com> References: <20200317195042.282977-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 216.205.24.74 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Alex Williamson , Cornelia Huck , peterx@redhat.com, Eric Auger Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" So that kvm_irqchip_assign_irqfd() can have access to the EventNotifiers, especially the resample event. It is needed in follow up patch to cache and kick resamplefds from QEMU. Reviewed-by: Eric Auger Signed-off-by: Peter Xu Reviewed-by: Alex Williamson --- accel/kvm/kvm-all.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index 439a4efe52..d49b74512a 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -1628,9 +1628,13 @@ int kvm_irqchip_update_msi_route(KVMState *s, int vi= rq, MSIMessage msg, return kvm_update_routing_entry(s, &kroute); } =20 -static int kvm_irqchip_assign_irqfd(KVMState *s, int fd, int rfd, int virq, +static int kvm_irqchip_assign_irqfd(KVMState *s, EventNotifier *event, + EventNotifier *resample, int virq, bool assign) { + int fd =3D event_notifier_get_fd(event); + int rfd =3D resample ? event_notifier_get_fd(resample) : -1; + struct kvm_irqfd irqfd =3D { .fd =3D fd, .gsi =3D virq, @@ -1735,7 +1739,9 @@ int kvm_irqchip_add_hv_sint_route(KVMState *s, uint32= _t vcpu, uint32_t sint) return -ENOSYS; } =20 -static int kvm_irqchip_assign_irqfd(KVMState *s, int fd, int virq, bool as= sign) +static int kvm_irqchip_assign_irqfd(KVMState *s, EventNotifier *event, + EventNotifier *resample, int virq, + bool assign) { abort(); } @@ -1749,15 +1755,13 @@ int kvm_irqchip_update_msi_route(KVMState *s, int v= irq, MSIMessage msg) int kvm_irqchip_add_irqfd_notifier_gsi(KVMState *s, EventNotifier *n, EventNotifier *rn, int virq) { - return kvm_irqchip_assign_irqfd(s, event_notifier_get_fd(n), - rn ? event_notifier_get_fd(rn) : -1, virq, true); + return kvm_irqchip_assign_irqfd(s, n, rn, virq, true); } =20 int kvm_irqchip_remove_irqfd_notifier_gsi(KVMState *s, EventNotifier *n, int virq) { - return kvm_irqchip_assign_irqfd(s, event_notifier_get_fd(n), -1, virq, - false); + return kvm_irqchip_assign_irqfd(s, n, NULL, virq, false); } =20 int kvm_irqchip_add_irqfd_notifier(KVMState *s, EventNotifier *n, --=20 2.24.1 From nobody Sun May 19 19:40:27 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1584474757; cv=none; d=zohomail.com; s=zohoarc; b=N/BrjMcGcpcXHc36SnmPuJaIf+AhHpFjen1QdUwpbZItB8H+08LJ4oFI++Q3O1VH6oN1S5zYW3l2hNz9/+mxXIrCzD6OrgeWNnHYd3lX9MFiDTFsmp9GFeLgwPar3QwHEnNzWnabW/j/rSBD1zzJhoPC1pA28/z/C64DC9IxHO0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584474757; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=hzz+2eiKAuPdVSPUO41t5HZSZcKNQlQ7xMJ1TIvtZ0w=; b=f/0Jx/Cl4XX7a2bo6Y6DFyyLU79JLiPWgdwe1n1/h7kPRrPaY1zNqmN+RJDCgtiODeULzmZOfjKmxESanHKE9F2oFz79867I7z7tUBRC3ISeShPAYm9E4EE5efykCWXVMxBM8/Xc3GCPee2Qd4luXdJusKSvO/hDlfnrajDJXco= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584474757318361.5847741960108; Tue, 17 Mar 2020 12:52:37 -0700 (PDT) Received: from localhost ([::1]:40496 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jEIGS-00016m-3i for importer@patchew.org; Tue, 17 Mar 2020 15:52:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34531) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jEIFA-0008Ch-Ns for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jEIF8-0002ty-VO for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:16 -0400 Received: from us-smtp-delivery-74.mimecast.com ([63.128.21.74]:21963) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jEIF8-0002sE-Qk for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:14 -0400 Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-450-l8Gq6SQBOaeadk_90pBXKw-1; Tue, 17 Mar 2020 15:51:09 -0400 Received: by mail-wr1-f72.google.com with SMTP id v7so1734276wrp.0 for ; Tue, 17 Mar 2020 12:51:09 -0700 (PDT) Received: from xz-x1.redhat.com ([2607:9880:19c0:32::2]) by smtp.gmail.com with ESMTPSA id q10sm5387815wrx.12.2020.03.17.12.51.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Mar 2020 12:51:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584474674; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hzz+2eiKAuPdVSPUO41t5HZSZcKNQlQ7xMJ1TIvtZ0w=; b=diNl/5+xNbmk/O6tyOfkf8ZIbgGBXv1+8IKeqEigjcF+nhwi603QYT3PC+aVg5bc+CU5QP 8oTGd+JVBzkzjwoA2YStMoX0LMjtKKES4MgtLfne34kVkj4SVuD9g+3Cihjw2AukbsLWXf o0dXFNC3jK8ghKc/I5NgIJ11Dbkq2cI= X-MC-Unique: l8Gq6SQBOaeadk_90pBXKw-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Bpw63fIiLJg+SU6dEAqbObFLvO3/MJuPyCw87fvjCgw=; b=K2tYkcbRZCpYNqEPGphselvKAut7n992cVrfpzE3Cq+pfpZC/Yuthn0p61nWJPKBXw 7+th0Zeuk0h9NTsxlYeWw6nvUAkmn00DrkhcrxRMfBC9wDU4rFVv9tcESOE4vnezN4Ch S0hhs4DBqg/HvnMu5UmCHq4zvfCF3IzwScrXZLrqZrMsG/5OBC/pMrb57APf4jMNjz7N i7jk+QnxpYHOUWQvfg1OIlfBE8ecmx7VakftD8vmS3JUke5ON8Mb13a+oixSVsdDLTqX I7S1JekIYXFMjzWkrpUEf7GfIXcpuwMtzCyMpyvetfKslRIG/aI/k6snO767meEi+/wK kWeA== X-Gm-Message-State: ANhLgQ2FUL3BofHf/7i7HjPOjVosRIdPBZ/fIQrRyVAKotle1NTXswyd 2c9ET22vVk8SSSY+Q2wkPfI//i2O+3OvAugF+2zn6LtJw9J9jS3KhSkQYit6tP5w/NqbZNfL8vR ZHfdXE9T958OnM64= X-Received: by 2002:a5d:628f:: with SMTP id k15mr677235wru.248.1584474668076; Tue, 17 Mar 2020 12:51:08 -0700 (PDT) X-Google-Smtp-Source: ADFU+vvMHpFqgKnqmEmUBZjffUKV7g7xae+eKemSxo4LapGuCz2BddtqJkKPnpLTyXqUCnsqPUdubw== X-Received: by 2002:a5d:628f:: with SMTP id k15mr677213wru.248.1584474667793; Tue, 17 Mar 2020 12:51:07 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v3 4/5] KVM: Kick resamplefd for split kernel irqchip Date: Tue, 17 Mar 2020 15:50:41 -0400 Message-Id: <20200317195042.282977-5-peterx@redhat.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200317195042.282977-1-peterx@redhat.com> References: <20200317195042.282977-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 63.128.21.74 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Alex Williamson , Cornelia Huck , peterx@redhat.com, Eric Auger Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" This is majorly only for X86 because that's the only one that supports split irqchip for now. When the irqchip is split, we face a dilemma that KVM irqfd will be enabled, however the slow irqchip is still running in the userspace. It means that the resamplefd in the kernel irqfds won't take any effect and it will miss to ack INTx interrupts on EOIs. One example is split irqchip with VFIO INTx, which will break if we use the VFIO INTx fast path. This patch can potentially supports the VFIO fast path again for INTx, that the IRQ delivery will still use the fast path, while we don't need to trap MMIOs in QEMU for the device to emulate the EIOs (see the callers of vfio_eoi() hook). However the EOI of the INTx will still need to be done from the userspace by caching all the resamplefds in QEMU and kick properly for IOAPIC EOI broadcast. This is tricky because in this case the userspace ioapic irr & remote-irr will be bypassed. However such a change will greatly boost performance for assigned devices using INTx irqs (TCP_RR boosts 46% after this patch applied). When the userspace is responsible for the resamplefd kickup, don't register it on the kvm_irqfd anymore, because on newer kernels (after commit 654f1f13ea56, 5.2+) the KVM_IRQFD will fail if with both split irqchip and resamplefd. This will make sure that the fast path will work for all supported kernels. https://patchwork.kernel.org/patch/10738541/#22609933 Suggested-by: Paolo Bonzini Signed-off-by: Peter Xu --- accel/kvm/kvm-all.c | 79 ++++++++++++++++++++++++++++++++++++++++-- accel/kvm/trace-events | 1 + hw/intc/ioapic.c | 25 +++++++++++-- include/sysemu/kvm.h | 4 +++ 4 files changed, 105 insertions(+), 4 deletions(-) diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c index d49b74512a..9a85fd1b8f 100644 --- a/accel/kvm/kvm-all.c +++ b/accel/kvm/kvm-all.c @@ -159,9 +159,59 @@ static const KVMCapabilityInfo kvm_required_capabilite= s[] =3D { static NotifierList kvm_irqchip_change_notifiers =3D NOTIFIER_LIST_INITIALIZER(kvm_irqchip_change_notifiers); =20 +struct KVMResampleFd { + int gsi; + EventNotifier *resample_event; + QLIST_ENTRY(KVMResampleFd) node; +}; +typedef struct KVMResampleFd KVMResampleFd; + +/* + * Only used with split irqchip where we need to do the resample fd + * kick for the kernel from userspace. + */ +static QLIST_HEAD(, KVMResampleFd) kvm_resample_fd_list =3D + QLIST_HEAD_INITIALIZER(kvm_resample_fd_list); + #define kvm_slots_lock(kml) qemu_mutex_lock(&(kml)->slots_lock) #define kvm_slots_unlock(kml) qemu_mutex_unlock(&(kml)->slots_lock) =20 +static inline void kvm_resample_fd_remove(int gsi) +{ + KVMResampleFd *rfd; + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { + if (rfd->gsi =3D=3D gsi) { + QLIST_REMOVE(rfd, node); + g_free(rfd); + break; + } + } +} + +static inline void kvm_resample_fd_insert(int gsi, EventNotifier *event) +{ + KVMResampleFd *rfd =3D g_new0(KVMResampleFd, 1); + + rfd->gsi =3D gsi; + rfd->resample_event =3D event; + + QLIST_INSERT_HEAD(&kvm_resample_fd_list, rfd, node); +} + +void kvm_resample_fd_notify(int gsi) +{ + KVMResampleFd *rfd; + + QLIST_FOREACH(rfd, &kvm_resample_fd_list, node) { + if (rfd->gsi =3D=3D gsi) { + event_notifier_set(rfd->resample_event); + trace_kvm_resample_fd_notify(gsi); + return; + } + } +} + int kvm_get_max_memslots(void) { KVMState *s =3D KVM_STATE(current_accel()); @@ -1642,8 +1692,33 @@ static int kvm_irqchip_assign_irqfd(KVMState *s, Eve= ntNotifier *event, }; =20 if (rfd !=3D -1) { - irqfd.flags |=3D KVM_IRQFD_FLAG_RESAMPLE; - irqfd.resamplefd =3D rfd; + assert(assign); + if (kvm_irqchip_is_split()) { + /* + * When the slow irqchip (e.g. IOAPIC) is in the + * userspace, KVM kernel resamplefd will not work because + * the EOI of the interrupt will be delivered to userspace + * instead, so the KVM kernel resamplefd kick will be + * skipped. The userspace here mimics what the kernel + * provides with resamplefd, remember the resamplefd and + * kick it when we receive EOI of this IRQ. + * + * This is hackery because IOAPIC is mostly bypassed + * (except EOI broadcasts) when irqfd is used. However + * this can bring much performance back for split irqchip + * with INTx IRQs (for VFIO, this gives 93% perf of the + * full fast path, which is 46% perf boost comparing to + * the INTx slow path). + */ + kvm_resample_fd_insert(virq, resample); + } else { + irqfd.flags |=3D KVM_IRQFD_FLAG_RESAMPLE; + irqfd.resamplefd =3D rfd; + } + } else if (!assign) { + if (kvm_irqchip_is_split()) { + kvm_resample_fd_remove(virq); + } } =20 if (!kvm_irqfds_enabled()) { diff --git a/accel/kvm/trace-events b/accel/kvm/trace-events index 4fb6e59d19..a68eb66534 100644 --- a/accel/kvm/trace-events +++ b/accel/kvm/trace-events @@ -16,4 +16,5 @@ kvm_set_ioeventfd_mmio(int fd, uint64_t addr, uint32_t va= l, bool assign, uint32_ kvm_set_ioeventfd_pio(int fd, uint16_t addr, uint32_t val, bool assign, ui= nt32_t size, bool datamatch) "fd: %d @0x%x val=3D0x%x assign: %d size: %d m= atch: %d" kvm_set_user_memory(uint32_t slot, uint32_t flags, uint64_t guest_phys_add= r, uint64_t memory_size, uint64_t userspace_addr, int ret) "Slot#%d flags= =3D0x%x gpa=3D0x%"PRIx64 " size=3D0x%"PRIx64 " ua=3D0x%"PRIx64 " ret=3D%d" kvm_clear_dirty_log(uint32_t slot, uint64_t start, uint32_t size) "slot#%"= PRId32" start 0x%"PRIx64" size 0x%"PRIx32 +kvm_resample_fd_notify(int gsi) "gsi %d" =20 diff --git a/hw/intc/ioapic.c b/hw/intc/ioapic.c index 15747fe2c2..81a17cc2b8 100644 --- a/hw/intc/ioapic.c +++ b/hw/intc/ioapic.c @@ -236,8 +236,29 @@ void ioapic_eoi_broadcast(int vector) for (n =3D 0; n < IOAPIC_NUM_PINS; n++) { entry =3D s->ioredtbl[n]; =20 - if ((entry & IOAPIC_VECTOR_MASK) !=3D vector || - ((entry >> IOAPIC_LVT_TRIGGER_MODE_SHIFT) & 1) !=3D IOAPIC= _TRIGGER_LEVEL) { + if ((entry & IOAPIC_VECTOR_MASK) !=3D vector) { + continue; + } + + /* + * When IOAPIC is in the userspace while APIC is still in + * the kernel (i.e., split irqchip), we have a trick to + * kick the resamplefd logic for registered irqfds from + * userspace to deactivate the IRQ. When that happens, it + * means the irq bypassed userspace IOAPIC (so the irr and + * remote-irr of the table entry should be bypassed too + * even if interrupt come). Still kick the resamplefds if + * they're bound to the IRQ, to make sure to EOI the + * interrupt for the hardware correctly. + * + * Note: We still need to go through the irr & remote-irr + * operations below because we don't know whether there're + * emulated devices that are using/sharing the same IRQ. + */ + kvm_resample_fd_notify(n); + + if (((entry >> IOAPIC_LVT_TRIGGER_MODE_SHIFT) & 1) !=3D + IOAPIC_TRIGGER_LEVEL) { continue; } =20 diff --git a/include/sysemu/kvm.h b/include/sysemu/kvm.h index 141342de98..583a976f8a 100644 --- a/include/sysemu/kvm.h +++ b/include/sysemu/kvm.h @@ -555,4 +555,8 @@ int kvm_set_one_reg(CPUState *cs, uint64_t id, void *so= urce); int kvm_get_one_reg(CPUState *cs, uint64_t id, void *target); struct ppc_radix_page_info *kvm_get_radix_page_info(void); int kvm_get_max_memslots(void); + +/* Notify resamplefd for EOI of specific interrupts. */ +void kvm_resample_fd_notify(int gsi); + #endif --=20 2.24.1 From nobody Sun May 19 19:40:27 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1584474809; cv=none; d=zohomail.com; s=zohoarc; b=KJTXYcDKvduC5V4CXmtU3Op8E1uuK08141OjUoded3dD6RISIeCoIyPsGXoLVpgal+XiYdaCeWGuemfQj1mrkCGil5AaOtcmTHjE8IVS70R73UdQ65po0ueB0Y337LKn0U9GDzcMrY7ILnpBI6jpOpMC/4YSfbAtodWAiIhR8f4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1584474809; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=XzobTypzUqN3M+rDLvFSV4NqJmV33xE7RPCGY0NoN9k=; b=WWdYUOKnlnVBUUt4QoCCP/zo1q6/qzf25rSOrP0TkeAJq+gknPCh5ofSgWpWH3JHV7wUaqGueg1EMLnEUxtkcP4g+L/tZC8D+MFr9TFYvpmfIgLiF8wPmNIknLx04TstZXLV9NM43dqC7aaO9DD2rjMaXB5WBrMuAreVwsbM9sI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1584474809213229.3512294727608; Tue, 17 Mar 2020 12:53:29 -0700 (PDT) Received: from localhost ([::1]:40508 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jEIHI-0002fF-6e for importer@patchew.org; Tue, 17 Mar 2020 15:53:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:34589) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jEIFD-0008H3-Ns for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jEIFC-0003BF-Oa for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:19 -0400 Received: from us-smtp-delivery-74.mimecast.com ([216.205.24.74]:43488) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1jEIFC-0003A0-L7 for qemu-devel@nongnu.org; Tue, 17 Mar 2020 15:51:18 -0400 Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-371-tT3BjMQFPHaTxNS_IEq6pA-1; Tue, 17 Mar 2020 15:51:16 -0400 Received: by mail-wm1-f72.google.com with SMTP id r23so176806wmn.3 for ; Tue, 17 Mar 2020 12:51:16 -0700 (PDT) Received: from xz-x1.redhat.com ([2607:9880:19c0:32::2]) by smtp.gmail.com with ESMTPSA id c23sm4681495wrb.79.2020.03.17.12.51.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Mar 2020 12:51:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1584474678; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XzobTypzUqN3M+rDLvFSV4NqJmV33xE7RPCGY0NoN9k=; b=Xe6lV2V55U8R3hE0OFMdOuMuxsSJI5928kBu0PIp5+4MZlu4DmFLVLto5qrQb2DEe/CWwF DfOSd3RAZoLlCe+MOTEcX06hMQhFfTAmbY8yeJoyEE8hPZnu/y8dm6rqkVx5sPeBzGC9nl uUD3l/cCyOcCC5muecjt7GHTU9R23/g= X-MC-Unique: tT3BjMQFPHaTxNS_IEq6pA-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Y/QoJwARiHNYw+6RfuxR8p/GSGTeGS8OuqmLslSWCWw=; b=Lb5BxuZ2T6OFuKRl+UEdeHlvtgxVdsJhxwu8jyzhhXrTJyZ+vk1xlP9qXEOWgLYLF4 u0PT1+Juo+atGIez8krkgTgiM94mhGgeTNPIHTg570QuS47xv1spNo/SMcRZUKIjcWfM eZFhgfkvAtVzMrGnRsZ9th1KxZ29iW1scgNdBCcWU6vc09HnKr7nHSHK6UeZ1fXb01It BNzwTt5OiqYUU2Bfwf8ta/BN2nwBicq1OQiy2zrvxs2J5yklY+pRIqFf5Ph8x2ylhkBa 260sJikCX3FvHnZSLQ6JBNUp2IjuIgMZfgVaWx70PH03t4sXVRu9r/9Vlr45IgATGbxS QUFA== X-Gm-Message-State: ANhLgQ1b/wZftMmQrOMD/mI7107YZ++R0eiXIuXPsSp7uvmT55cNstpg QrlFAxnM0N2h6FjtX3qzuaDaKW2GJeY5jt70LDunV8zaQ0QhibcuELZtYENyJV6rhSzwdRfIY0A K4zA8RsC3Bksu2wE= X-Received: by 2002:a1c:7f10:: with SMTP id a16mr651837wmd.1.1584474673492; Tue, 17 Mar 2020 12:51:13 -0700 (PDT) X-Google-Smtp-Source: ADFU+vvTf4uG1wz8HGsLnJQ71nGbbF/9DlkV/cKRTkaVnjLkMoZ9c32nus2s8MbG6wbGVucCKRXcKw== X-Received: by 2002:a1c:7f10:: with SMTP id a16mr651822wmd.1.1584474673280; Tue, 17 Mar 2020 12:51:13 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v3 5/5] Revert "vfio/pci: Disable INTx fast path if using split irqchip" Date: Tue, 17 Mar 2020 15:50:42 -0400 Message-Id: <20200317195042.282977-6-peterx@redhat.com> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200317195042.282977-1-peterx@redhat.com> References: <20200317195042.282977-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 216.205.24.74 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Paolo Bonzini , Alex Williamson , Cornelia Huck , peterx@redhat.com, Eric Auger Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" With the resamplefd list introduced, we can savely enable VFIO INTx fast path again with split irqchip so it can still be faster than the complete slow path. Reviewed-by: Eric Auger Signed-off-by: Peter Xu Acked-by: Alex Williamson Reviewed-by: Alex Williamson --- hw/vfio/pci.c | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 09703362df..1c0aa27386 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -124,18 +124,6 @@ static void vfio_intx_enable_kvm(VFIOPCIDevice *vdev, = Error **errp) return; } =20 - if (kvm_irqchip_is_split()) { - /* - * VFIO INTx is currently not working with split kernel - * irqchip for level triggered interrupts. Go the slow path - * as long as split is enabled so we can be at least - * functional (even with poor performance). - * - * TODO: Remove this after all things fixed up. - */ - return; - } - /* Get to a known interrupt state */ qemu_set_fd_handler(irq_fd, NULL, NULL, vdev); vfio_mask_single_irqindex(&vdev->vbasedev, VFIO_PCI_INTX_IRQ_INDEX); --=20 2.24.1