From nobody Tue Oct 7 03:47:36 2025 Received: from raptorengineering.com (mail.raptorengineering.com [23.155.224.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F1231C4A24; Tue, 15 Jul 2025 21:38:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=23.155.224.40 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752615507; cv=none; b=rSgPA7jQiaC8vXF2QNMwfHFwCmyU62cPSLocFB5p0k/q8xFzFQMyDoZRIibfJzAOt+brY1dqdSVj87fe8/07zoW4CuA/+n0uuhC2HOfDpomw0B5T9shuxBc50ewB9mDHzbKm3XDGmEtYkh2xst9ztJ9ZYNOIwX1nLurS5m9QoMs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1752615507; c=relaxed/simple; bh=U5Lk+VkVJJP5V8zUZPjwKlzGXzRmt+MR/UQ4a+BmKuk=; h=Date:From:To:Cc:Message-ID:In-Reply-To:References:Subject: MIME-Version:Content-Type; b=n2VvfqtFfBFYl2n2VcQ9BTMEY0UuEsNipuAnRknHqOnsc67S3hFfKT95V6/z+AOGciD+B7X6CAJXsbGSOxBZKRmPOu+k3itDixPTsFe3/D9kSbWU2SuQgrYR9q10/gfz0Z3SprTxXgWwFW/BtjSHYfnVpd3ljpqdj5Fc3ermm1o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=raptorengineering.com; spf=pass smtp.mailfrom=raptorengineering.com; dkim=pass (1024-bit key) header.d=raptorengineering.com header.i=@raptorengineering.com header.b=jIxJcX2W; arc=none smtp.client-ip=23.155.224.40 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=raptorengineering.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=raptorengineering.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=raptorengineering.com header.i=@raptorengineering.com header.b="jIxJcX2W" Received: from localhost (localhost [127.0.0.1]) by mail.rptsys.com (Postfix) with ESMTP id C69F88287698; Tue, 15 Jul 2025 16:38:24 -0500 (CDT) Received: from mail.rptsys.com ([127.0.0.1]) by localhost (vali.starlink.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id TrGGvDmz4rZm; Tue, 15 Jul 2025 16:38:23 -0500 (CDT) Received: from localhost (localhost [127.0.0.1]) by mail.rptsys.com (Postfix) with ESMTP id 7D1738288591; Tue, 15 Jul 2025 16:38:23 -0500 (CDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.rptsys.com 7D1738288591 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=raptorengineering.com; s=B8E824E6-0BE2-11E6-931D-288C65937AAD; t=1752615503; bh=AzvxEw34wp1D5TRYWoqBxaNBDOvxPv48PtS4dQyEJ24=; h=Date:From:To:Message-ID:MIME-Version; b=jIxJcX2WRxl4nSywEZGn+vl4wE3YHhYFtu7Gm/FpZepSdPbebbPDrqn78SArsroiq X6EkBK7CYwQT+kYvpMCgue+tLqU5RPET4BMP5gquftgPIicx/L3ssx2BRXAuZc2yhV FH6YaDkAOLnhlrJTYEmzxNaJszDMTHJSsLq63j+k= X-Virus-Scanned: amavisd-new at rptsys.com Received: from mail.rptsys.com ([127.0.0.1]) by localhost (vali.starlink.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id dNDVnC83S7-x; Tue, 15 Jul 2025 16:38:23 -0500 (CDT) Received: from vali.starlink.edu (localhost [127.0.0.1]) by mail.rptsys.com (Postfix) with ESMTP id 3E77D8287698; Tue, 15 Jul 2025 16:38:23 -0500 (CDT) Date: Tue, 15 Jul 2025 16:38:23 -0500 (CDT) From: Timothy Pearson To: Timothy Pearson Cc: linuxppc-dev , linux-kernel , linux-pci , Madhavan Srinivasan , Michael Ellerman , christophe leroy , Naveen N Rao , Bjorn Helgaas , Shawn Anastasio Message-ID: <1334208367.1359861.1752615503144.JavaMail.zimbra@raptorengineeringinc.com> In-Reply-To: <1268570622.1359844.1752615109932.JavaMail.zimbra@raptorengineeringinc.com> References: <1268570622.1359844.1752615109932.JavaMail.zimbra@raptorengineeringinc.com> Subject: [PATCH v3 4/6] powerpc/eeh: Make EEH driver device hotplug safe Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Mailer: Zimbra 8.5.0_GA_3042 (ZimbraWebClient - GC138 (Linux)/8.5.0_GA_3042) Thread-Topic: powerpc/eeh: Make EEH driver device hotplug safe Thread-Index: XyF2OaMn/3q+H+nwsGaxXLVF4U4PF0HA+s8u Content-Type: text/plain; charset="utf-8" Multiple race conditions existed between the PCIe hotplug driver and the EEH driver, leading to a variety of kernel oopses of the same general nature: A second class of oops is also seen when the underling bus disappears during device recovery. Refactor the EEH module to be PCI rescan and remove safe. Also clean up a few minor formatting / readability issues. Signed-off-by: Timothy Pearson --- arch/powerpc/kernel/eeh_driver.c | 48 +++++++++++++++++++++----------- arch/powerpc/kernel/eeh_pe.c | 10 ++++--- 2 files changed, 38 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_dri= ver.c index 7efe04c68f0f..dd50de91c438 100644 --- a/arch/powerpc/kernel/eeh_driver.c +++ b/arch/powerpc/kernel/eeh_driver.c @@ -257,13 +257,12 @@ static void eeh_pe_report_edev(struct eeh_dev *edev, = eeh_report_fn fn, struct pci_driver *driver; enum pci_ers_result new_result; =20 - pci_lock_rescan_remove(); pdev =3D edev->pdev; if (pdev) get_device(&pdev->dev); - pci_unlock_rescan_remove(); if (!pdev) { eeh_edev_info(edev, "no device"); + *result =3D PCI_ERS_RESULT_DISCONNECT; return; } device_lock(&pdev->dev); @@ -304,8 +303,9 @@ static void eeh_pe_report(const char *name, struct eeh_= pe *root, struct eeh_dev *edev, *tmp; =20 pr_info("EEH: Beginning: '%s'\n", name); - eeh_for_each_pe(root, pe) eeh_pe_for_each_dev(pe, edev, tmp) - eeh_pe_report_edev(edev, fn, result); + eeh_for_each_pe(root, pe) + eeh_pe_for_each_dev(pe, edev, tmp) + eeh_pe_report_edev(edev, fn, result); if (result) pr_info("EEH: Finished:'%s' with aggregate recovery state:'%s'\n", name, pci_ers_result_name(*result)); @@ -383,6 +383,8 @@ static void eeh_dev_restore_state(struct eeh_dev *edev,= void *userdata) if (!edev) return; =20 + pci_lock_rescan_remove(); + /* * The content in the config space isn't saved because * the blocked config space on some adapters. We have @@ -393,14 +395,19 @@ static void eeh_dev_restore_state(struct eeh_dev *ede= v, void *userdata) if (list_is_last(&edev->entry, &edev->pe->edevs)) eeh_pe_restore_bars(edev->pe); =20 + pci_unlock_rescan_remove(); return; } =20 pdev =3D eeh_dev_to_pci_dev(edev); - if (!pdev) + if (!pdev) { + pci_unlock_rescan_remove(); return; + } =20 pci_restore_state(pdev); + + pci_unlock_rescan_remove(); } =20 /** @@ -647,9 +654,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct p= ci_bus *bus, if (any_passed || driver_eeh_aware || (pe->type & EEH_PE_VF)) { eeh_pe_dev_traverse(pe, eeh_rmv_device, rmv_data); } else { - pci_lock_rescan_remove(); pci_hp_remove_devices(bus); - pci_unlock_rescan_remove(); } =20 /* @@ -665,8 +670,6 @@ static int eeh_reset_device(struct eeh_pe *pe, struct p= ci_bus *bus, if (rc) return rc; =20 - pci_lock_rescan_remove(); - /* Restore PE */ eeh_ops->configure_bridge(pe); eeh_pe_restore_bars(pe); @@ -674,7 +677,6 @@ static int eeh_reset_device(struct eeh_pe *pe, struct p= ci_bus *bus, /* Clear frozen state */ rc =3D eeh_clear_pe_frozen_state(pe, false); if (rc) { - pci_unlock_rescan_remove(); return rc; } =20 @@ -709,7 +711,6 @@ static int eeh_reset_device(struct eeh_pe *pe, struct p= ci_bus *bus, pe->tstamp =3D tstamp; pe->freeze_count =3D cnt; =20 - pci_unlock_rescan_remove(); return 0; } =20 @@ -843,10 +844,13 @@ void eeh_handle_normal_event(struct eeh_pe *pe) {LIST_HEAD_INIT(rmv_data.removed_vf_list), 0}; int devices =3D 0; =20 + pci_lock_rescan_remove(); + bus =3D eeh_pe_bus_get(pe); if (!bus) { pr_err("%s: Cannot find PCI bus for PHB#%x-PE#%x\n", __func__, pe->phb->global_number, pe->addr); + pci_unlock_rescan_remove(); return; } =20 @@ -1094,10 +1098,15 @@ void eeh_handle_normal_event(struct eeh_pe *pe) eeh_pe_state_clear(pe, EEH_PE_PRI_BUS, true); eeh_pe_dev_mode_mark(pe, EEH_DEV_REMOVED); =20 - pci_lock_rescan_remove(); - pci_hp_remove_devices(bus); - pci_unlock_rescan_remove(); + bus =3D eeh_pe_bus_get(pe); + if (bus) + pci_hp_remove_devices(bus); + else + pr_err("%s: PCI bus for PHB#%x-PE#%x disappeared\n", + __func__, pe->phb->global_number, pe->addr); + /* The passed PE should no longer be used */ + pci_unlock_rescan_remove(); return; } =20 @@ -1114,6 +1123,8 @@ void eeh_handle_normal_event(struct eeh_pe *pe) eeh_clear_slot_attention(edev->pdev); =20 eeh_pe_state_clear(pe, EEH_PE_RECOVERING, true); + + pci_unlock_rescan_remove(); } =20 /** @@ -1132,6 +1143,7 @@ void eeh_handle_special_event(void) unsigned long flags; int rc; =20 + pci_lock_rescan_remove(); =20 do { rc =3D eeh_ops->next_error(&pe); @@ -1171,10 +1183,12 @@ void eeh_handle_special_event(void) =20 break; case EEH_NEXT_ERR_NONE: + pci_unlock_rescan_remove(); return; default: pr_warn("%s: Invalid value %d from next_error()\n", __func__, rc); + pci_unlock_rescan_remove(); return; } =20 @@ -1186,7 +1200,9 @@ void eeh_handle_special_event(void) if (rc =3D=3D EEH_NEXT_ERR_FROZEN_PE || rc =3D=3D EEH_NEXT_ERR_FENCED_PHB) { eeh_pe_state_mark(pe, EEH_PE_RECOVERING); + pci_unlock_rescan_remove(); eeh_handle_normal_event(pe); + pci_lock_rescan_remove(); } else { eeh_for_each_pe(pe, tmp_pe) eeh_pe_for_each_dev(tmp_pe, edev, tmp_edev) @@ -1199,7 +1215,6 @@ void eeh_handle_special_event(void) eeh_report_failure, NULL); eeh_set_channel_state(pe, pci_channel_io_perm_failure); =20 - pci_lock_rescan_remove(); list_for_each_entry(hose, &hose_list, list_node) { phb_pe =3D eeh_phb_pe_get(hose); if (!phb_pe || @@ -1218,7 +1233,6 @@ void eeh_handle_special_event(void) } pci_hp_remove_devices(bus); } - pci_unlock_rescan_remove(); } =20 /* @@ -1228,4 +1242,6 @@ void eeh_handle_special_event(void) if (rc =3D=3D EEH_NEXT_ERR_DEAD_IOC) break; } while (rc !=3D EEH_NEXT_ERR_NONE); + + pci_unlock_rescan_remove(); } diff --git a/arch/powerpc/kernel/eeh_pe.c b/arch/powerpc/kernel/eeh_pe.c index d283d281d28e..e740101fadf3 100644 --- a/arch/powerpc/kernel/eeh_pe.c +++ b/arch/powerpc/kernel/eeh_pe.c @@ -671,10 +671,12 @@ static void eeh_bridge_check_link(struct eeh_dev *ede= v) eeh_ops->write_config(edev, cap + PCI_EXP_LNKCTL, 2, val); =20 /* Check link */ - if (!edev->pdev->link_active_reporting) { - eeh_edev_dbg(edev, "No link reporting capability\n"); - msleep(1000); - return; + if (edev->pdev) { + if (!edev->pdev->link_active_reporting) { + eeh_edev_dbg(edev, "No link reporting capability\n"); + msleep(1000); + return; + } } =20 /* Wait the link is up until timeout (5s) */ --=20 2.39.5