From nobody Tue May 7 07:06:27 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1488180149596176.9316576659079; Sun, 26 Feb 2017 23:22:29 -0800 (PST) Received: from localhost ([::1]:50758 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ciFdY-000118-2p for importer@patchew.org; Mon, 27 Feb 2017 02:22:24 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51873) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ciFcc-0000Tm-05 for qemu-devel@nongnu.org; Mon, 27 Feb 2017 02:21:27 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ciFcY-0003jR-OM for qemu-devel@nongnu.org; Mon, 27 Feb 2017 02:21:25 -0500 Received: from [59.151.112.132] (port=54212 helo=heian.cn.fujitsu.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ciFcX-0003fQ-Nn for qemu-devel@nongnu.org; Mon, 27 Feb 2017 02:21:22 -0500 Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 27 Feb 2017 15:21:08 +0800 Received: from G08CNEXCHPEKD03.g08.fujitsu.local (unknown [10.167.33.85]) by cn.fujitsu.com (Postfix) with ESMTP id 3C0DB47C4EAF; Mon, 27 Feb 2017 15:21:08 +0800 (CST) Received: from G08FNSTD140223.g08.fujitsu.local (10.167.226.69) by G08CNEXCHPEKD03.g08.fujitsu.local (10.167.33.89) with Microsoft SMTP Server (TLS) id 14.3.319.2; Mon, 27 Feb 2017 15:21:12 +0800 X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; d="scan'208";a="15992106" From: Cao jin To: , , Date: Mon, 27 Feb 2017 15:28:43 +0800 Message-ID: <1488180523-18137-1-git-send-email-caoj.fnst@cn.fujitsu.com> X-Mailer: git-send-email 2.1.0 MIME-Version: 1.0 X-Originating-IP: [10.167.226.69] X-yoursite-MailScanner-ID: 3C0DB47C4EAF.AF732 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: caoj.fnst@cn.fujitsu.com X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 59.151.112.132 Subject: [Qemu-devel] [PATCH] vfio pci: kernel support of error recovery only for non fatal error X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: izumi.taku@jp.fujitsu.com, alex.williamson@redhat.com, mst@redhat.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" 0. What happens now (PCIE AER only) Fatal errors cause a link reset. Non fatal errors don't. All errors stop the VM eventually, but not immediately because it's detected and reported asynchronously. Interrupts are forwarded as usual. Correctable errors are not reported to guest at all. Note: PPC EEH is different. This focuses on AER. 1. Correctable errors There is no need to report these to guest. So let's not. 2. Fatal errors It's not easy to handle them gracefully since link reset is needed. As a first step, let's use the existing mechanism in that case. 2. Non-fatal errors Here we could make progress by reporting them to guest and have guest handle them. Issues: a. this behaviour should only be enabled with new userspace, old userspace should work without changes. Suggestion: One way to address this would be to add a new eventfd non_fatal_err_trigger. If not set, invoke err_trigger. b. drivers are supposed to stop MMIO when error is reported, if vm keeps going, we will keep doing MMIO/config. Suggestion 1: ignore this. vm stop happens much later when userspace runs anyway, so we are not making things much worse. Suggestion 2: try to stop MMIO/config, resume on resume call Patch below implements Suggestion 1. Note that although this is really against the documentation, which states error_detected() is the point at which the driver should quies= ce the device and not touch it further (until diagnostic poking at mmio_enabled or full access at resume callback). Fixing this won't be easy. However, this is not a regression. Also note this does nothing about interrupts, documentation suggests returning IRQ_NONE until reset. Again, not a regression. c. PF driver might detect that function is completely broken, if vm keeps going, we will keep doing MMIO/config. Suggestion 1: ignore this. vm stop happens much later when userspace runs anyway, so we are not making things much worse. Suggestion 2: detect this and invoke err_trigger to stop VM. Patch below implements Suggestion 2. Suggested-by: Michael S. Tsirkin Signed-off-by: Cao jin --- drivers/vfio/pci/vfio_pci.c | 38 +++++++++++++++++++++++++++++++++= ++-- drivers/vfio/pci/vfio_pci_intrs.c | 19 +++++++++++++++++++ drivers/vfio/pci/vfio_pci_private.h | 1 + include/uapi/linux/vfio.h | 1 + 4 files changed, 57 insertions(+), 2 deletions(-) diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index 324c52e..3551cc9 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -441,7 +441,8 @@ static int vfio_pci_get_irq_count(struct vfio_pci_devic= e *vdev, int irq_type) =20 return (flags & PCI_MSIX_FLAGS_QSIZE) + 1; } - } else if (irq_type =3D=3D VFIO_PCI_ERR_IRQ_INDEX) { + } else if (irq_type =3D=3D VFIO_PCI_ERR_IRQ_INDEX || + irq_type =3D=3D VFIO_PCI_NON_FATAL_ERR_IRQ_INDEX) { if (pci_is_pcie(vdev->pdev)) return 1; } else if (irq_type =3D=3D VFIO_PCI_REQ_IRQ_INDEX) { @@ -796,6 +797,7 @@ static long vfio_pci_ioctl(void *device_data, case VFIO_PCI_REQ_IRQ_INDEX: break; case VFIO_PCI_ERR_IRQ_INDEX: + case VFIO_PCI_NON_FATAL_ERR_IRQ_INDEX: if (pci_is_pcie(vdev->pdev)) break; /* pass thru to return error */ @@ -1282,7 +1284,9 @@ static pci_ers_result_t vfio_pci_aer_err_detected(str= uct pci_dev *pdev, =20 mutex_lock(&vdev->igate); =20 - if (vdev->err_trigger) + if (state =3D=3D pci_channel_io_normal && vdev->non_fatal_err_trigger) + eventfd_signal(vdev->non_fatal_err_trigger, 1); + else if (vdev->err_trigger) eventfd_signal(vdev->err_trigger, 1); =20 mutex_unlock(&vdev->igate); @@ -1292,8 +1296,38 @@ static pci_ers_result_t vfio_pci_aer_err_detected(st= ruct pci_dev *pdev, return PCI_ERS_RESULT_CAN_RECOVER; } =20 +static pci_ers_result_t vfio_pci_aer_slot_reset(struct pci_dev *pdev) +{ + struct vfio_pci_device *vdev; + struct vfio_device *device; + static pci_ers_result_t err =3D PCI_ERS_RESULT_NONE; + + device =3D vfio_device_get_from_dev(&pdev->dev); + if (!device) + goto err_dev; + + vdev =3D vfio_device_data(device); + if (!vdev) + goto err_data; + + mutex_lock(&vdev->igate); + + if (vdev->err_trigger) + eventfd_signal(vdev->err_trigger, 1); + + mutex_unlock(&vdev->igate); + + err =3D PCI_ERS_RESULT_RECOVERED; + +err_data: + vfio_device_put(device); +err_dev: + return err; +} + static const struct pci_error_handlers vfio_err_handlers =3D { .error_detected =3D vfio_pci_aer_err_detected, + .slot_reset =3D vfio_pci_aer_slot_reset, }; =20 static struct pci_driver vfio_pci_driver =3D { diff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_= intrs.c index 1c46045..270d568 100644 --- a/drivers/vfio/pci/vfio_pci_intrs.c +++ b/drivers/vfio/pci/vfio_pci_intrs.c @@ -611,6 +611,17 @@ static int vfio_pci_set_err_trigger(struct vfio_pci_de= vice *vdev, count, flags, data); } =20 +static int vfio_pci_set_non_fatal_err_trigger(struct vfio_pci_device *vdev, + unsigned index, unsigned start, + unsigned count, uint32_t flags, void *data) +{ + if (index !=3D VFIO_PCI_NON_FATAL_ERR_IRQ_INDEX || start !=3D 0 || count = > 1) + return -EINVAL; + + return vfio_pci_set_ctx_trigger_single(&vdev->non_fatal_err_trigger, + count, flags, data); +} + static int vfio_pci_set_req_trigger(struct vfio_pci_device *vdev, unsigned index, unsigned start, unsigned count, uint32_t flags, void *data) @@ -664,6 +675,14 @@ int vfio_pci_set_irqs_ioctl(struct vfio_pci_device *vd= ev, uint32_t flags, break; } break; + case VFIO_PCI_NON_FATAL_ERR_IRQ_INDEX: + switch (flags & VFIO_IRQ_SET_ACTION_TYPE_MASK) { + case VFIO_IRQ_SET_ACTION_TRIGGER: + if (pci_is_pcie(vdev->pdev)) + func =3D vfio_pci_set_non_fatal_err_trigger; + break; + } + break; case VFIO_PCI_REQ_IRQ_INDEX: switch (flags & VFIO_IRQ_SET_ACTION_TYPE_MASK) { case VFIO_IRQ_SET_ACTION_TRIGGER: diff --git a/drivers/vfio/pci/vfio_pci_private.h b/drivers/vfio/pci/vfio_pc= i_private.h index f561ac1..d30f8a3 100644 --- a/drivers/vfio/pci/vfio_pci_private.h +++ b/drivers/vfio/pci/vfio_pci_private.h @@ -93,6 +93,7 @@ struct vfio_pci_device { struct pci_saved_state *pci_saved_state; int refcnt; struct eventfd_ctx *err_trigger; + struct eventfd_ctx *non_fatal_err_trigger; struct eventfd_ctx *req_trigger; struct list_head dummy_resources_list; }; diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 519eff3..affa973 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -443,6 +443,7 @@ enum { VFIO_PCI_MSIX_IRQ_INDEX, VFIO_PCI_ERR_IRQ_INDEX, VFIO_PCI_REQ_IRQ_INDEX, + VFIO_PCI_NON_FATAL_ERR_IRQ_INDEX, VFIO_PCI_NUM_IRQS }; =20 --=20 1.8.3.1