From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C665C77B7A for ; Sun, 11 Jun 2023 17:19:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233280AbjFKRTU (ORCPT ); Sun, 11 Jun 2023 13:19:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229636AbjFKRTR (ORCPT ); Sun, 11 Jun 2023 13:19:17 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [78.133.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 0981BE5F for ; Sun, 11 Jun 2023 10:19:15 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 0EA1A9200B4; Sun, 11 Jun 2023 19:19:15 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 0B5499200B3; Sun, 11 Jun 2023 18:19:15 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:14 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 01/14] PCI: pciehp: Rely on `link_active_reporting' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use `link_active_reporting' to determine whether Data Link Layer Link=20 Active Reporting is available rather than re-retrieving the capability. Signed-off-by: Maciej W. Rozycki Reviewed-by: Lukas Wunner --- NB this has been compile-tested only with PPC64LE and x86-64 configurations. No change from v8. Changes from v7: - Add Reviewed-by: tag by Lukas Wunner. - Reorder from 6/7. No change from v6. New change in v6. --- drivers/pci/hotplug/pciehp_hpc.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) linux-pcie-link-active-reporting-hpc.diff Index: linux-macro/drivers/pci/hotplug/pciehp_hpc.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/hotplug/pciehp_hpc.c +++ linux-macro/drivers/pci/hotplug/pciehp_hpc.c @@ -984,7 +984,7 @@ static inline int pcie_hotplug_depth(str struct controller *pcie_init(struct pcie_device *dev) { struct controller *ctrl; - u32 slot_cap, slot_cap2, link_cap; + u32 slot_cap, slot_cap2; u8 poweron; struct pci_dev *pdev =3D dev->port; struct pci_bus *subordinate =3D pdev->subordinate; @@ -1030,9 +1030,6 @@ struct controller *pcie_init(struct pcie if (dmi_first_match(inband_presence_disabled_dmi_table)) ctrl->inband_presence_disabled =3D 1; =20 - /* Check if Data Link Layer Link Active Reporting is implemented */ - pcie_capability_read_dword(pdev, PCI_EXP_LNKCAP, &link_cap); - /* Clear all remaining event bits in Slot Status register. */ pcie_capability_write_word(pdev, PCI_EXP_SLTSTA, PCI_EXP_SLTSTA_ABP | PCI_EXP_SLTSTA_PFD | @@ -1051,7 +1048,7 @@ struct controller *pcie_init(struct pcie FLAG(slot_cap, PCI_EXP_SLTCAP_EIP), FLAG(slot_cap, PCI_EXP_SLTCAP_NCCS), FLAG(slot_cap2, PCI_EXP_SLTCAP2_IBPD), - FLAG(link_cap, PCI_EXP_LNKCAP_DLLLARC), + FLAG(pdev->link_active_reporting, true), pdev->broken_cmd_compl ? " (with Cmd Compl erratum)" : ""); =20 /* From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A2ACC77B7A for ; Sun, 11 Jun 2023 17:19:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233691AbjFKRTZ (ORCPT ); Sun, 11 Jun 2023 13:19:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233665AbjFKRTV (ORCPT ); Sun, 11 Jun 2023 13:19:21 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 57FD6E68; Sun, 11 Jun 2023 10:19:20 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id B62CE9200BC; Sun, 11 Jun 2023 19:19:19 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id AEEC09200B3; Sun, 11 Jun 2023 18:19:19 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:19 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 02/14] PCI: Export PCIe link retrain timeout In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert LINK_RETRAIN_TIMEOUT from jiffies to milliseconds, accordingly=20 rename to PCIE_LINK_RETRAIN_TIMEOUT_MS, and make available via "pci.h"=20 for PCI drivers to use. Use in `pcie_wait_for_link_delay'. Signed-off-by: Maciej W. Rozycki --- Changes from v8: - Convert LINK_RETRAIN_TIMEOUT from jiffies to milliseconds, rename it to PCIE_LINK_RETRAIN_TIMEOUT_MS rather than PCIE_LINK_RETRAIN_TIMEOUT, and=20 adjust its use accordingly. - Also replace hardcoded 1000 in `pcie_wait_for_link_delay'. - Correct the change heading, s/PCI/PCIe/ for the link reference. Changes from v7: - Reorder from 1/7. No change from v6. No change from v5. New change in v5. --- drivers/pci/pci.c | 2 +- drivers/pci/pci.h | 2 ++ drivers/pci/pcie/aspm.c | 4 +--- 3 files changed, 4 insertions(+), 4 deletions(-) linux-pcie-link-retrain-timeout.diff Index: linux-macro/drivers/pci/pci.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4860,7 +4860,7 @@ static int pci_pm_reset(struct pci_dev * static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active, int delay) { - int timeout =3D 1000; + int timeout =3D PCIE_LINK_RETRAIN_TIMEOUT_MS; bool ret; u16 lnk_status; =20 Index: linux-macro/drivers/pci/pci.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.h +++ linux-macro/drivers/pci/pci.h @@ -11,6 +11,8 @@ =20 #define PCI_VSEC_ID_INTEL_TBT 0x1234 /* Thunderbolt */ =20 +#define PCIE_LINK_RETRAIN_TIMEOUT_MS 1000 + extern const unsigned char pcie_link_speed[]; extern bool pci_early_dump; =20 Index: linux-macro/drivers/pci/pcie/aspm.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pcie/aspm.c +++ linux-macro/drivers/pci/pcie/aspm.c @@ -90,8 +90,6 @@ static const char *policy_str[] =3D { [POLICY_POWER_SUPERSAVE] =3D "powersupersave" }; =20 -#define LINK_RETRAIN_TIMEOUT HZ - /* * The L1 PM substate capability is only implemented in function 0 in a * multi function device. @@ -213,7 +211,7 @@ static bool pcie_retrain_link(struct pci } =20 /* Wait for link training end. Break out after waiting for timeout */ - end_jiffies =3D jiffies + LINK_RETRAIN_TIMEOUT; + end_jiffies =3D jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); do { pcie_capability_read_word(parent, PCI_EXP_LNKSTA, ®16); if (!(reg16 & PCI_EXP_LNKSTA_LT)) From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6839CC7EE45 for ; Sun, 11 Jun 2023 17:19:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233665AbjFKRTb (ORCPT ); Sun, 11 Jun 2023 13:19:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233879AbjFKRT1 (ORCPT ); Sun, 11 Jun 2023 13:19:27 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 1564A10CC; Sun, 11 Jun 2023 10:19:24 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 69A929200BF; Sun, 11 Jun 2023 19:19:23 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 637809200B3; Sun, 11 Jun 2023 18:19:23 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:23 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 03/14] PCI: Execute `quirk_enable_clear_retrain_link' earlier In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make `quirk_enable_clear_retrain_link' `pci_fixup_early' so that any later=20 fixups can rely on `clear_retrain_link' to have been already initialised. Signed-off-by: Maciej W. Rozycki --- No change from v8. Changes from v7: - Reorder from 2/7. No change from v6. No change from v5. New change in v5. --- drivers/pci/quirks.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) linux-pcie-clear-retrain-link-early.diff Index: linux-macro/drivers/pci/quirks.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/quirks.c +++ linux-macro/drivers/pci/quirks.c @@ -2407,9 +2407,9 @@ static void quirk_enable_clear_retrain_l dev->clear_retrain_link =3D 1; pci_info(dev, "Enable PCIe Retrain Link quirk\n"); } -DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_PERICOM, 0xe110, quirk_enable_clear= _retrain_link); -DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_PERICOM, 0xe111, quirk_enable_clear= _retrain_link); -DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_PERICOM, 0xe130, quirk_enable_clear= _retrain_link); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_PERICOM, 0xe110, quirk_enable_clear_= retrain_link); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_PERICOM, 0xe111, quirk_enable_clear_= retrain_link); +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_PERICOM, 0xe130, quirk_enable_clear_= retrain_link); =20 static void fixup_rev1_53c810(struct pci_dev *dev) { From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A5C1C77B7A for ; Sun, 11 Jun 2023 17:19:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229825AbjFKRTl (ORCPT ); Sun, 11 Jun 2023 13:19:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231451AbjFKRTi (ORCPT ); Sun, 11 Jun 2023 13:19:38 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [78.133.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id A93B5E7D for ; Sun, 11 Jun 2023 10:19:28 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id C29129200BF; Sun, 11 Jun 2023 19:19:27 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id C025E9200BB; Sun, 11 Jun 2023 18:19:27 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:27 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 04/14] PCI: Initialize `link_active_reporting' earlier In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Determine whether Data Link Layer Link Active Reporting is available=20 ahead of calling any fixups so that the cached value can be used there=20 and later on. Signed-off-by: Maciej W. Rozycki --- No change from v8. Changes from v7: - Reorder from 3/7. Changes from v6: - Regenerate against 6.3-rc5. New change in v6. --- drivers/pci/probe.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) linux-pcie-link-active-reporting-early.diff Index: linux-macro/drivers/pci/probe.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/probe.c +++ linux-macro/drivers/pci/probe.c @@ -820,7 +820,6 @@ static void pci_set_bus_speed(struct pci =20 pcie_capability_read_dword(bridge, PCI_EXP_LNKCAP, &linkcap); bus->max_bus_speed =3D pcie_link_speed[linkcap & PCI_EXP_LNKCAP_SLS]; - bridge->link_active_reporting =3D !!(linkcap & PCI_EXP_LNKCAP_DLLLARC); =20 pcie_capability_read_word(bridge, PCI_EXP_LNKSTA, &linksta); pcie_update_link_speed(bus, linksta); @@ -1829,6 +1828,7 @@ int pci_setup_device(struct pci_dev *dev int err, pos =3D 0; struct pci_bus_region region; struct resource *res; + u32 linkcap; =20 hdr_type =3D pci_hdr_type(dev); =20 @@ -1876,6 +1876,10 @@ int pci_setup_device(struct pci_dev *dev /* "Unknown power state" */ dev->current_state =3D PCI_UNKNOWN; =20 + /* Set it early to make it available to fixups, etc. */ + pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &linkcap); + dev->link_active_reporting =3D !!(linkcap & PCI_EXP_LNKCAP_DLLLARC); + /* Early fixups, before probing the BARs */ pci_fixup_device(pci_fixup_early, dev); From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 397B1C77B7A for ; Sun, 11 Jun 2023 17:19:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233630AbjFKRTz (ORCPT ); Sun, 11 Jun 2023 13:19:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233938AbjFKRTt (ORCPT ); Sun, 11 Jun 2023 13:19:49 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [78.133.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 36BBEE7A for ; Sun, 11 Jun 2023 10:19:33 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 67DE69200C3; Sun, 11 Jun 2023 19:19:32 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 640209200BB; Sun, 11 Jun 2023 18:19:32 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:32 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 05/14] powerpc/eeh: Rely on `link_active_reporting' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use `link_active_reporting' to determine whether Data Link Layer Link=20 Active Reporting is available rather than re-retrieving the capability. Signed-off-by: Maciej W. Rozycki --- NB this has been compile-tested only with a PPC64LE configuration. No change from v8. Changes from v7: - Reorder from 4/7. No change from v6. New change in v6. --- arch/powerpc/kernel/eeh_pe.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) linux-pcie-link-active-reporting-eeh.diff Index: linux-macro/arch/powerpc/kernel/eeh_pe.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/arch/powerpc/kernel/eeh_pe.c +++ linux-macro/arch/powerpc/kernel/eeh_pe.c @@ -671,9 +671,8 @@ static void eeh_bridge_check_link(struct eeh_ops->write_config(edev, cap + PCI_EXP_LNKCTL, 2, val); =20 /* Check link */ - eeh_ops->read_config(edev, cap + PCI_EXP_LNKCAP, 4, &val); - if (!(val & PCI_EXP_LNKCAP_DLLLARC)) { - eeh_edev_dbg(edev, "No link reporting capability (0x%08x) \n", val); + if (!edev->pdev->link_active_reporting) { + eeh_edev_dbg(edev, "No link reporting capability\n"); msleep(1000); return; } From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 263F4C7EE37 for ; Sun, 11 Jun 2023 17:20:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231451AbjFKRUW (ORCPT ); Sun, 11 Jun 2023 13:20:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230515AbjFKRUU (ORCPT ); Sun, 11 Jun 2023 13:20:20 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 210661BDF; Sun, 11 Jun 2023 10:19:57 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 060279200C4; Sun, 11 Jun 2023 19:19:36 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id F3F579200C0; Sun, 11 Jun 2023 18:19:36 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:36 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 06/14] net/mlx5: Rely on `link_active_reporting' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use `link_active_reporting' to determine whether Data Link Layer Link=20 Active Reporting is available rather than re-retrieving the capability. Signed-off-by: Maciej W. Rozycki --- NB this has been compile-tested only with PPC64LE and x86-64=20 configurations. No change from v8. Changes from v7: - Reorder from 5/7. Changes from v6: - Regenerate against 6.3-rc5. New change in v6. --- drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) linux-pcie-link-active-reporting-mlx5.diff Index: linux-macro/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c +++ linux-macro/drivers/net/ethernet/mellanox/mlx5/core/fw_reset.c @@ -307,7 +307,6 @@ static int mlx5_pci_link_toggle(struct m unsigned long timeout; struct pci_dev *sdev; int cap, err; - u32 reg32; =20 /* Check that all functions under the pci bridge are PFs of * this device otherwise fail this function. @@ -346,11 +345,8 @@ static int mlx5_pci_link_toggle(struct m return err; =20 /* Check link */ - err =3D pci_read_config_dword(bridge, cap + PCI_EXP_LNKCAP, ®32); - if (err) - return err; - if (!(reg32 & PCI_EXP_LNKCAP_DLLLARC)) { - mlx5_core_warn(dev, "No PCI link reporting capability (0x%08x)\n", reg32= ); + if (!bridge->link_active_reporting) { + mlx5_core_warn(dev, "No PCI link reporting capability\n"); msleep(1000); goto restore; } From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CFA4C77B7A for ; Sun, 11 Jun 2023 17:20:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233811AbjFKRUg (ORCPT ); Sun, 11 Jun 2023 13:20:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233505AbjFKRU3 (ORCPT ); Sun, 11 Jun 2023 13:20:29 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9E47E10C1; Sun, 11 Jun 2023 10:20:06 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 267D59200BC; Sun, 11 Jun 2023 19:19:41 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 21FBD9200BB; Sun, 11 Jun 2023 18:19:41 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:41 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 07/14] PCI: Export `pcie_retrain_link' for use outside ASPM In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Export `pcie_retrain_link' for link retrain needs outside ASPM. There=20 is no functional change at this point, but `struct pcie_link_state' is=20 local to ASPM and not used within `pcie_retrain_link' other than to get=20 at the associated PCI device, so change the operand and adjust the lone=20 call site accordingly. Document the interface. Signed-off-by: Maciej W. Rozycki --- There's a missing full stop added in a comment in the course of the move,=20 not worth mentioning in the change description IMHO and not worth its own=20 change either. This comment will go away in a subsequent change anyway. New change in v9. --- drivers/pci/pci.c | 36 ++++++++++++++++++++++++++++++++++++ drivers/pci/pci.h | 1 + drivers/pci/pcie/aspm.c | 32 +------------------------------- 3 files changed, 38 insertions(+), 31 deletions(-) linux-pcie-retrain-link-export.diff Index: linux-macro/drivers/pci/pci.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4912,6 +4912,42 @@ bool pcie_wait_for_link(struct pci_dev * return pcie_wait_for_link_delay(pdev, active, 100); } =20 +/** + * pcie_retrain_link - Request a link retrain and wait for it to complete + * @pdev: Device whose link to retrain. + * + * Return TRUE if successful, or FALSE if training has not completed + * within PCIE_LINK_RETRAIN_TIMEOUT_MS milliseconds. + */ +bool pcie_retrain_link(struct pci_dev *pdev) +{ + unsigned long end_jiffies; + u16 reg16; + + pcie_capability_read_word(pdev, PCI_EXP_LNKCTL, ®16); + reg16 |=3D PCI_EXP_LNKCTL_RL; + pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, reg16); + if (pdev->clear_retrain_link) { + /* + * Due to an erratum in some devices the Retrain Link bit + * needs to be cleared again manually to allow the link + * training to succeed. + */ + reg16 &=3D ~PCI_EXP_LNKCTL_RL; + pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, reg16); + } + + /* Wait for link training end. Break out after waiting for timeout. */ + end_jiffies =3D jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); + do { + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, ®16); + if (!(reg16 & PCI_EXP_LNKSTA_LT)) + break; + msleep(1); + } while (time_before(jiffies, end_jiffies)); + return !(reg16 & PCI_EXP_LNKSTA_LT); +} + /* * Find maximum D3cold delay required by all the devices on the bus. The * spec says 100 ms, but firmware can lower it and we allow drivers to Index: linux-macro/drivers/pci/pci.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.h +++ linux-macro/drivers/pci/pci.h @@ -561,6 +561,7 @@ pci_ers_result_t pcie_do_recovery(struct pci_ers_result_t (*reset_subordinates)(struct pci_dev *pdev)); =20 bool pcie_wait_for_link(struct pci_dev *pdev, bool active); +bool pcie_retrain_link(struct pci_dev *pdev); #ifdef CONFIG_PCIEASPM void pcie_aspm_init_link_state(struct pci_dev *pdev); void pcie_aspm_exit_link_state(struct pci_dev *pdev); Index: linux-macro/drivers/pci/pcie/aspm.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pcie/aspm.c +++ linux-macro/drivers/pci/pcie/aspm.c @@ -191,36 +191,6 @@ static void pcie_clkpm_cap_init(struct p link->clkpm_disable =3D blacklist ? 1 : 0; } =20 -static bool pcie_retrain_link(struct pcie_link_state *link) -{ - struct pci_dev *parent =3D link->pdev; - unsigned long end_jiffies; - u16 reg16; - - pcie_capability_read_word(parent, PCI_EXP_LNKCTL, ®16); - reg16 |=3D PCI_EXP_LNKCTL_RL; - pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16); - if (parent->clear_retrain_link) { - /* - * Due to an erratum in some devices the Retrain Link bit - * needs to be cleared again manually to allow the link - * training to succeed. - */ - reg16 &=3D ~PCI_EXP_LNKCTL_RL; - pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16); - } - - /* Wait for link training end. Break out after waiting for timeout */ - end_jiffies =3D jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); - do { - pcie_capability_read_word(parent, PCI_EXP_LNKSTA, ®16); - if (!(reg16 & PCI_EXP_LNKSTA_LT)) - break; - msleep(1); - } while (time_before(jiffies, end_jiffies)); - return !(reg16 & PCI_EXP_LNKSTA_LT); -} - /* * pcie_aspm_configure_common_clock: check if the 2 ends of a link * could use common clock. If they are, configure them to use the @@ -287,7 +257,7 @@ static void pcie_aspm_configure_common_c reg16 &=3D ~PCI_EXP_LNKCTL_CCC; pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16); =20 - if (pcie_retrain_link(link)) + if (pcie_retrain_link(link->pdev)) return; =20 /* Training failed. Restore common clock configurations */ From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9E2EC77B7A for ; Sun, 11 Jun 2023 17:20:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233820AbjFKRUc (ORCPT ); Sun, 11 Jun 2023 13:20:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233466AbjFKRU2 (ORCPT ); Sun, 11 Jun 2023 13:20:28 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id EF8F810C4; Sun, 11 Jun 2023 10:20:04 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 53DBD9200C5; Sun, 11 Jun 2023 19:19:45 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 4D8169200C1; Sun, 11 Jun 2023 18:19:45 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:45 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 08/14] PCI: Use distinct local vars in `pcie_retrain_link' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use separate local variables to hold the respective values retrieved=20 from the Link Control Register and the Link Status Register. Not only=20 it improves readability, but it makes it possible for the compiler to=20 detect actual uninitialised use should this code change in the future. Signed-off-by: Maciej W. Rozycki --- New change in v9. --- drivers/pci/pci.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) linux-pcie-retrain-link-lnkctl-lnksta.diff Index: linux-macro/drivers/pci/pci.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4922,30 +4922,31 @@ bool pcie_wait_for_link(struct pci_dev * bool pcie_retrain_link(struct pci_dev *pdev) { unsigned long end_jiffies; - u16 reg16; + u16 lnkctl; + u16 lnksta; =20 - pcie_capability_read_word(pdev, PCI_EXP_LNKCTL, ®16); - reg16 |=3D PCI_EXP_LNKCTL_RL; - pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, reg16); + pcie_capability_read_word(pdev, PCI_EXP_LNKCTL, &lnkctl); + lnkctl |=3D PCI_EXP_LNKCTL_RL; + pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, lnkctl); if (pdev->clear_retrain_link) { /* * Due to an erratum in some devices the Retrain Link bit * needs to be cleared again manually to allow the link * training to succeed. */ - reg16 &=3D ~PCI_EXP_LNKCTL_RL; - pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, reg16); + lnkctl &=3D ~PCI_EXP_LNKCTL_RL; + pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, lnkctl); } =20 /* Wait for link training end. Break out after waiting for timeout. */ end_jiffies =3D jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); do { - pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, ®16); - if (!(reg16 & PCI_EXP_LNKSTA_LT)) + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnksta); + if (!(lnksta & PCI_EXP_LNKSTA_LT)) break; msleep(1); } while (time_before(jiffies, end_jiffies)); - return !(reg16 & PCI_EXP_LNKSTA_LT); + return !(lnksta & PCI_EXP_LNKSTA_LT); } =20 /* From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18D98C77B7A for ; Sun, 11 Jun 2023 17:21:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233898AbjFKRU7 (ORCPT ); Sun, 11 Jun 2023 13:20:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230233AbjFKRUz (ORCPT ); Sun, 11 Jun 2023 13:20:55 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7BCA110D0; Sun, 11 Jun 2023 10:20:29 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 90AD99200C1; Sun, 11 Jun 2023 19:19:49 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 8B5119200B3; Sun, 11 Jun 2023 18:19:49 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:49 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 09/14] PCI: Factor our waiting for link training end In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move code polling for the Link Training bit to clear into a function of=20 its own. Signed-off-by: Maciej W. Rozycki --- New change in v9. --- drivers/pci/pci.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) linux-pcie-wait-for-link-status.diff Index: linux-macro/drivers/pci/pci.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4850,6 +4850,28 @@ static int pci_pm_reset(struct pci_dev * } =20 /** + * pcie_wait_for_link_status - Wait for link training end + * @pdev: Device whose link to wait for. + * + * Return TRUE if successful, or FALSE if training has not completed + * within PCIE_LINK_RETRAIN_TIMEOUT_MS milliseconds. + */ +static bool pcie_wait_for_link_status(struct pci_dev *pdev) +{ + unsigned long end_jiffies; + u16 lnksta; + + end_jiffies =3D jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); + do { + pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnksta); + if (!(lnksta & PCI_EXP_LNKSTA_LT)) + break; + msleep(1); + } while (time_before(jiffies, end_jiffies)); + return !(lnksta & PCI_EXP_LNKSTA_LT); +} + +/** * pcie_wait_for_link_delay - Wait until link is active or inactive * @pdev: Bridge device * @active: waiting for active or inactive? @@ -4916,14 +4938,11 @@ bool pcie_wait_for_link(struct pci_dev * * pcie_retrain_link - Request a link retrain and wait for it to complete * @pdev: Device whose link to retrain. * - * Return TRUE if successful, or FALSE if training has not completed - * within PCIE_LINK_RETRAIN_TIMEOUT_MS milliseconds. + * Return TRUE if successful, or FALSE if training has not completed. */ bool pcie_retrain_link(struct pci_dev *pdev) { - unsigned long end_jiffies; u16 lnkctl; - u16 lnksta; =20 pcie_capability_read_word(pdev, PCI_EXP_LNKCTL, &lnkctl); lnkctl |=3D PCI_EXP_LNKCTL_RL; @@ -4938,15 +4957,7 @@ bool pcie_retrain_link(struct pci_dev *p pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, lnkctl); } =20 - /* Wait for link training end. Break out after waiting for timeout. */ - end_jiffies =3D jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); - do { - pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnksta); - if (!(lnksta & PCI_EXP_LNKSTA_LT)) - break; - msleep(1); - } while (time_before(jiffies, end_jiffies)); - return !(lnksta & PCI_EXP_LNKSTA_LT); + return pcie_wait_for_link_status(pdev); } =20 /* From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03598C7EE23 for ; Sun, 11 Jun 2023 17:21:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233960AbjFKRVS (ORCPT ); Sun, 11 Jun 2023 13:21:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233888AbjFKRVN (ORCPT ); Sun, 11 Jun 2023 13:21:13 -0400 X-Greylist: delayed 92 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Sun, 11 Jun 2023 10:20:49 PDT Received: from angie.orcam.me.uk (angie.orcam.me.uk [78.133.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id E45F71700; Sun, 11 Jun 2023 10:20:48 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 4E8B29200C6; Sun, 11 Jun 2023 19:19:53 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 4858E9200B3; Sun, 11 Jun 2023 18:19:53 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:53 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 10/14] PCI: Add support for polling DLLLA to `pcie_retrain_link' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Let the caller of `pcie_retrain_link' specify whether they want to use=20 the LT bit or the DLLLA bit of the Link Status Register to determine if=20 link training has completed. It is up to the caller to verify whether=20 the use of the DLLLA bit, the implementation of which is optional, is=20 valid for the device requested. Signed-off-by: Maciej W. Rozycki --- New change in v9. --- drivers/pci/pci.c | 28 ++++++++++++++++++++-------- drivers/pci/pci.h | 2 +- drivers/pci/pcie/aspm.c | 2 +- 3 files changed, 22 insertions(+), 10 deletions(-) linux-pcie-retrain-link-use-lt.diff Index: linux-macro/drivers/pci/pci.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4850,25 +4850,32 @@ static int pci_pm_reset(struct pci_dev * } =20 /** - * pcie_wait_for_link_status - Wait for link training end + * pcie_wait_for_link_status - Wait for link status change * @pdev: Device whose link to wait for. + * @use_lt: Use the LT bit if TRUE, or the DLLLA bit if FALSE. + * @active: Waiting for active or inactive? * - * Return TRUE if successful, or FALSE if training has not completed - * within PCIE_LINK_RETRAIN_TIMEOUT_MS milliseconds. + * Return TRUE if successful, or FALSE if status has not changed within + * PCIE_LINK_RETRAIN_TIMEOUT_MS milliseconds. */ -static bool pcie_wait_for_link_status(struct pci_dev *pdev) +static bool pcie_wait_for_link_status(struct pci_dev *pdev, + bool use_lt, bool active) { + u16 lnksta_mask, lnksta_match; unsigned long end_jiffies; u16 lnksta; =20 + lnksta_mask =3D use_lt ? PCI_EXP_LNKSTA_LT : PCI_EXP_LNKSTA_DLLLA; + lnksta_match =3D active ? lnksta_mask : 0; + end_jiffies =3D jiffies + msecs_to_jiffies(PCIE_LINK_RETRAIN_TIMEOUT_MS); do { pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnksta); - if (!(lnksta & PCI_EXP_LNKSTA_LT)) + if ((lnksta & lnksta_mask) =3D=3D lnksta_match) break; msleep(1); } while (time_before(jiffies, end_jiffies)); - return !(lnksta & PCI_EXP_LNKSTA_LT); + return (lnksta & lnksta_mask) =3D=3D lnksta_match; } =20 /** @@ -4937,10 +4944,15 @@ bool pcie_wait_for_link(struct pci_dev * /** * pcie_retrain_link - Request a link retrain and wait for it to complete * @pdev: Device whose link to retrain. + * @use_lt: Use the LT bit if TRUE, or the DLLLA bit if FALSE, for status. + * + * Retrain completion status is retrieved from the Link Status Register + * according to @use_lt. It is not verified whether the use of the DLLLA + * bit is valid. * * Return TRUE if successful, or FALSE if training has not completed. */ -bool pcie_retrain_link(struct pci_dev *pdev) +bool pcie_retrain_link(struct pci_dev *pdev, bool use_lt) { u16 lnkctl; =20 @@ -4957,7 +4969,7 @@ bool pcie_retrain_link(struct pci_dev *p pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, lnkctl); } =20 - return pcie_wait_for_link_status(pdev); + return pcie_wait_for_link_status(pdev, use_lt, !use_lt); } =20 /* Index: linux-macro/drivers/pci/pci.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.h +++ linux-macro/drivers/pci/pci.h @@ -561,7 +561,7 @@ pci_ers_result_t pcie_do_recovery(struct pci_ers_result_t (*reset_subordinates)(struct pci_dev *pdev)); =20 bool pcie_wait_for_link(struct pci_dev *pdev, bool active); -bool pcie_retrain_link(struct pci_dev *pdev); +bool pcie_retrain_link(struct pci_dev *pdev, bool use_lt); #ifdef CONFIG_PCIEASPM void pcie_aspm_init_link_state(struct pci_dev *pdev); void pcie_aspm_exit_link_state(struct pci_dev *pdev); Index: linux-macro/drivers/pci/pcie/aspm.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pcie/aspm.c +++ linux-macro/drivers/pci/pcie/aspm.c @@ -257,7 +257,7 @@ static void pcie_aspm_configure_common_c reg16 &=3D ~PCI_EXP_LNKCTL_CCC; pcie_capability_write_word(parent, PCI_EXP_LNKCTL, reg16); =20 - if (pcie_retrain_link(link->pdev)) + if (pcie_retrain_link(link->pdev, true)) return; =20 /* Training failed. Restore common clock configurations */ From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F8BEC77B7A for ; Sun, 11 Jun 2023 17:20:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229449AbjFKRUv (ORCPT ); Sun, 11 Jun 2023 13:20:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230233AbjFKRUs (ORCPT ); Sun, 11 Jun 2023 13:20:48 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 9B5ED199E; Sun, 11 Jun 2023 10:20:21 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id E73559200C7; Sun, 11 Jun 2023 19:19:57 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id E0F309200C0; Sun, 11 Jun 2023 18:19:57 +0100 (BST) Date: Sun, 11 Jun 2023 18:19:57 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 11/14] PCI: Use `pcie_wait_for_link_status' in `pcie_wait_for_link_delay' In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove a DLLLA status bit polling loop from `pcie_wait_for_link_delay'=20 and call almost identical code in `pcie_wait_for_link_status' instead. =20 This reduces the lower bound on the polling interval from 10ms to 1ms,=20 possibly increasing the CPU load on the system in favour to reducing=20 the wait time. Signed-off-by: Maciej W. Rozycki --- New change in v9. --- drivers/pci/pci.c | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) linux-pcie-wait-for-link-delay-status.diff Index: linux-macro/drivers/pci/pci.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4889,16 +4889,14 @@ static bool pcie_wait_for_link_status(st static bool pcie_wait_for_link_delay(struct pci_dev *pdev, bool active, int delay) { - int timeout =3D PCIE_LINK_RETRAIN_TIMEOUT_MS; bool ret; - u16 lnk_status; =20 /* * Some controllers might not implement link active reporting. In this * case, we wait for 1000 ms + any delay requested by the caller. */ if (!pdev->link_active_reporting) { - msleep(timeout + delay); + msleep(PCIE_LINK_RETRAIN_TIMEOUT_MS + delay); return true; } =20 @@ -4913,20 +4911,11 @@ static bool pcie_wait_for_link_delay(str */ if (active) msleep(20); - for (;;) { - pcie_capability_read_word(pdev, PCI_EXP_LNKSTA, &lnk_status); - ret =3D !!(lnk_status & PCI_EXP_LNKSTA_DLLLA); - if (ret =3D=3D active) - break; - if (timeout <=3D 0) - break; - msleep(10); - timeout -=3D 10; - } + ret =3D pcie_wait_for_link_status(pdev, false, active); if (active && ret) msleep(delay); =20 - return ret =3D=3D active; + return ret; } =20 /** From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1FF7C77B7A for ; Sun, 11 Jun 2023 17:21:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233879AbjFKRVE (ORCPT ); Sun, 11 Jun 2023 13:21:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45280 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230156AbjFKRUz (ORCPT ); Sun, 11 Jun 2023 13:20:55 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 21BFFE6F; Sun, 11 Jun 2023 10:20:30 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 529979200C9; Sun, 11 Jun 2023 19:20:02 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 4E1479200C0; Sun, 11 Jun 2023 18:20:02 +0100 (BST) Date: Sun, 11 Jun 2023 18:20:02 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 12/14] PCI: Provide stub failed link recovery for device probing and hot plug In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This now fails unconditionally and will be always optimised away, but=20 provides for quirks to implement recovery for failed links detected in=20 device probing and device hot plug events. Signed-off-by: Maciej W. Rozycki --- New change in v9, factored out from 7/7: - Rename `pcie_downstream_link_retrain' to `pcie_failed_link_retrain'. - Add stub implementation in "pci.h". --- drivers/pci/pci.c | 2 ++ drivers/pci/pci.h | 4 ++++ drivers/pci/probe.c | 2 ++ 3 files changed, 8 insertions(+) linux-pcie-failed-link-retrain.diff Index: linux-macro/drivers/pci/pci.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -4912,6 +4912,8 @@ static bool pcie_wait_for_link_delay(str if (active) msleep(20); ret =3D pcie_wait_for_link_status(pdev, false, active); + if (active && !ret) + ret =3D pcie_failed_link_retrain(pdev); if (active && ret) msleep(delay); =20 Index: linux-macro/drivers/pci/pci.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.h +++ linux-macro/drivers/pci/pci.h @@ -554,6 +554,10 @@ static inline int pci_dev_specific_disab return -ENOTTY; } #endif +static inline bool pcie_failed_link_retrain(struct pci_dev *dev) +{ + return false; +} =20 /* PCI error reporting and recovery */ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, Index: linux-macro/drivers/pci/probe.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/probe.c +++ linux-macro/drivers/pci/probe.c @@ -2549,6 +2549,8 @@ void pci_device_add(struct pci_dev *dev, dma_set_max_seg_size(&dev->dev, 65536); dma_set_seg_boundary(&dev->dev, 0xffffffff); =20 + pcie_failed_link_retrain(dev); + /* Fix up broken headers */ pci_fixup_device(pci_fixup_header, dev); From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B980C7EE23 for ; Sun, 11 Jun 2023 17:21:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233935AbjFKRVY (ORCPT ); Sun, 11 Jun 2023 13:21:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233992AbjFKRVT (ORCPT ); Sun, 11 Jun 2023 13:21:19 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [78.133.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id CCE1719AA; Sun, 11 Jun 2023 10:20:54 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 6DEA69200CB; Sun, 11 Jun 2023 19:20:06 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 66A4F9200CA; Sun, 11 Jun 2023 18:20:06 +0100 (BST) Date: Sun, 11 Jun 2023 18:20:06 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 13/14] PCI: Add failed link recovery for device reset events In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Request failed link recovery with any upstream bridge where a device has=20 not come back after reset within PCI_RESET_WAIT time. Reset the polling=20 interval if recovery succeeded, otherwise continue as usual. Signed-off-by: Maciej W. Rozycki --- New change in v9, factored out from 7/7: - Remove duplicate succesful completion report previously added (not sure=20 where it came from, possibly an unnoticed leftover from experiments). - Make the type of `retrain' variable `bool' rather than `int' and invert the logic used. - Rename `pcie_downstream_link_retrain' to `pcie_failed_link_retrain'. - Rename `pcie_upstream_link_retrain' to `pcie_parent_link_retrain'. Add=20 documentation. --- drivers/pci/pci.c | 39 ++++++++++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 5 deletions(-) linux-pcie-dev-wait-link-retrain.diff Index: linux-macro/drivers/pci/pci.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.c +++ linux-macro/drivers/pci/pci.c @@ -1146,10 +1146,27 @@ void pci_resume_bus(struct pci_bus *bus) pci_walk_bus(bus, pci_resume_one, NULL); } =20 +/** + * pcie_parent_link_retrain - Check and retrain link we are downstream from + * @dev: PCI device to handle. + * + * Return TRUE if the link was retrained, FALSE otherwise. + */ +static bool pcie_parent_link_retrain(struct pci_dev *dev) +{ + struct pci_dev *bridge; + + bridge =3D pci_upstream_bridge(dev); + if (bridge) + return pcie_failed_link_retrain(bridge); + else + return false; +} + static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout) { + bool retrain =3D true; int delay =3D 1; - u32 id; =20 /* * After reset, the device should not silently discard config @@ -1163,21 +1180,33 @@ static int pci_dev_wait(struct pci_dev * * Command register instead of Vendor ID so we don't have to * contend with the CRS SV value. */ - pci_read_config_dword(dev, PCI_COMMAND, &id); - while (PCI_POSSIBLE_ERROR(id)) { + for (;;) { + u32 id; + + pci_read_config_dword(dev, PCI_COMMAND, &id); + if (!PCI_POSSIBLE_ERROR(id)) + break; + if (delay > timeout) { pci_warn(dev, "not ready %dms after %s; giving up\n", delay - 1, reset_type); return -ENOTTY; } =20 - if (delay > PCI_RESET_WAIT) + if (delay > PCI_RESET_WAIT) { + if (retrain) { + retrain =3D false; + if (pcie_parent_link_retrain(dev)) { + delay =3D 1; + continue; + } + } pci_info(dev, "not ready %dms after %s; waiting\n", delay - 1, reset_type); + } =20 msleep(delay); delay *=3D 2; - pci_read_config_dword(dev, PCI_COMMAND, &id); } =20 if (delay > PCI_RESET_WAIT) From nobody Thu Sep 11 12:11:07 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14E4BC7EE37 for ; Sun, 11 Jun 2023 17:21:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234023AbjFKRV2 (ORCPT ); Sun, 11 Jun 2023 13:21:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233952AbjFKRVU (ORCPT ); Sun, 11 Jun 2023 13:21:20 -0400 Received: from angie.orcam.me.uk (angie.orcam.me.uk [78.133.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 458D7E75; Sun, 11 Jun 2023 10:20:55 -0700 (PDT) Received: by angie.orcam.me.uk (Postfix, from userid 500) id 974869200CC; Sun, 11 Jun 2023 19:20:10 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 91D4E9200CA; Sun, 11 Jun 2023 18:20:10 +0100 (BST) Date: Sun, 11 Jun 2023 18:20:10 +0100 (BST) From: "Maciej W. Rozycki" To: Bjorn Helgaas , Mahesh J Salgaonkar , Oliver O'Halloran , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Saeed Mahameed , Leon Romanovsky , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni cc: Alex Williamson , Lukas Wunner , Mika Westerberg , Stefan Roese , Jim Wilson , David Abdurachmanov , =?UTF-8?Q?Pali_Roh=C3=A1r?= , linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-rdma@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v9 14/14] PCI: Work around PCIe link training failures In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Attempt to handle cases such as with a downstream port of the ASMedia=20 ASM2824 PCIe switch where link training never completes and the link=20 continues switching between speeds indefinitely with the data link layer=20 never reaching the active state. It has been observed with a downstream port of the ASMedia ASM2824 Gen 3=20 switch wired to the upstream port of the Pericom PI7C9X2G304 Gen 2=20 switch, using a Delock Riser Card PCI Express x1 > 2 x PCIe x1 device,=20 P/N 41433, wired to a SiFive HiFive Unmatched board. In this setup the=20 switches are supposed to negotiate the link speed of preferably 5.0GT/s,=20 falling back to 2.5GT/s. Instead the link continues oscillating between the two speeds, at the=20 rate of 34-35 times per second, with link training reported repeatedly=20 active ~84% of the time. Forcibly limiting the target link speed to=20 2.5GT/s with the upstream ASM2824 device however makes the two switches=20 communicate correctly. Removing the speed restriction afterwards makes=20 the two devices switch to 5.0GT/s then. Make use of these observations then and detect the inability to train=20 the link, by checking for the Data Link Layer Link Active status bit=20 being off while the Link Bandwidth Management Status indicating that=20 hardware has changed the link speed or width in an attempt to correct=20 unreliable link operation. Restrict the speed to 2.5GT/s then with the Target Link Speed field,=20 request a retrain and wait 200ms for the data link to go up. If this=20 turns out successful, then lift the restriction, letting the devices=20 negotiate a higher speed. Also check for a 2.5GT/s speed restriction the firmware may have already=20 arranged and lift it too with ports of devices known to continue working=20 afterwards, currently the ASM2824 only, that already report their data=20 link being up. Signed-off-by: Maciej W. Rozycki Link: https://lore.kernel.org/r/alpine.DEB.2.21.2203022037020.56670@angie.o= rcam.me.uk/ Link: https://source.denx.de/u-boot/u-boot/-/commit/a398a51ccc68 --- Changes from v8: - Rename `pcie_downstream_link_retrain' to `pcie_failed_link_retrain', add=20 a prototype in "pci.h", moving the stub implementation under !PCI_QUIRKS=20 umbrella. - Move back to quirks.c, though as an internal API call rather than a=20 regular quirk. - Adjust for PCIE_LINK_RETRAIN_TIMEOUT_MS expressed in milliseconds rather=20 than jiffies. - Use a `pcie_retrain_link' call rather than retraining inline, and also=20 use it in the restriction lift path, making it another possible failure=20 point. No changes from v7. Changes from v6: - Regenerate against 6.3-rc5. - Shorten the lore.kernel.org archive link in the change description. Changes from v5: - Move from a quirk into PCI core and call at device probing, hot-plug, reset and resume. Keep the ASMedia part under CONFIG_PCI_QUIRKS. - Rely on `dev->link_active_reporting' rather than re-retrieving the=20 capability. Changes from v4: - Remove inclusion no longer needed. - Make the quirk generic based on probing device features rather than=20 specific to the ASM2824 part only; take the Retrain Link bit erratum=20 into account. - Still lift the 2.5GT/s speed restriction with the ASM2824 only. - Increase retrain timeout from 200ms to 1s (PCIE_LINK_RETRAIN_TIMEOUT). - Remove retrain success notification. - Use PCIe helpers rather than generic PCI functions throughout. - Trim down and update the wording of the change description for the=20 switch from an ASM2824-specific to a generic fixup. Changes from v3: - Remove the entry for the ASM2824. Changes from v2: - Regenerate for 5.17-rc2 for a merge conflict. - Replace BUG_ON for a missing PCI Express capability with WARN_ON and an early return. Changes from v1: - Regenerate for a merge conflict. --- drivers/pci/pci.h | 3 + drivers/pci/quirks.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++= +++++ 2 files changed, 95 insertions(+), 1 deletion(-) linux-pcie-asm2824-manual-retrain.diff Index: linux-macro/drivers/pci/pci.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/pci.h +++ linux-macro/drivers/pci/pci.h @@ -539,6 +539,7 @@ void pci_acs_init(struct pci_dev *dev); int pci_dev_specific_acs_enabled(struct pci_dev *dev, u16 acs_flags); int pci_dev_specific_enable_acs(struct pci_dev *dev); int pci_dev_specific_disable_acs_redir(struct pci_dev *dev); +bool pcie_failed_link_retrain(struct pci_dev *dev); #else static inline int pci_dev_specific_acs_enabled(struct pci_dev *dev, u16 acs_flags) @@ -553,11 +554,11 @@ static inline int pci_dev_specific_disab { return -ENOTTY; } -#endif static inline bool pcie_failed_link_retrain(struct pci_dev *dev) { return false; } +#endif =20 /* PCI error reporting and recovery */ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, Index: linux-macro/drivers/pci/quirks.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- linux-macro.orig/drivers/pci/quirks.c +++ linux-macro/drivers/pci/quirks.c @@ -33,6 +33,99 @@ #include #include "pci.h" =20 +/* + * Retrain the link of a downstream PCIe port by hand if necessary. + * + * This is needed at least where a downstream port of the ASMedia ASM2824 + * Gen 3 switch is wired to the upstream port of the Pericom PI7C9X2G304 + * Gen 2 switch, and observed with the Delock Riser Card PCI Express x1 > + * 2 x PCIe x1 device, P/N 41433, plugged into the SiFive HiFive Unmatched + * board. + * + * In such a configuration the switches are supposed to negotiate the link + * speed of preferably 5.0GT/s, falling back to 2.5GT/s. However the link + * continues switching between the two speeds indefinitely and the data + * link layer never reaches the active state, with link training reported + * repeatedly active ~84% of the time. Forcing the target link speed to + * 2.5GT/s with the upstream ASM2824 device makes the two switches talk to + * each other correctly however. And more interestingly retraining with a + * higher target link speed afterwards lets the two successfully negotiate + * 5.0GT/s. + * + * With the ASM2824 we can rely on the otherwise optional Data Link Layer + * Link Active status bit and in the failed link training scenario it will + * be off along with the Link Bandwidth Management Status indicating that + * hardware has changed the link speed or width in an attempt to correct + * unreliable link operation. For a port that has been left unconnected + * both bits will be clear. So use this information to detect the problem + * rather than polling the Link Training bit and watching out for flips or + * at least the active status. + * + * Since the exact nature of the problem isn't known and in principle this + * could trigger where an ASM2824 device is downstream rather upstream, + * apply this erratum workaround to any downstream ports as long as they + * support Link Active reporting and have the Link Control 2 register. + * Restrict the speed to 2.5GT/s then with the Target Link Speed field, + * request a retrain and wait 200ms for the data link to go up. + * + * If this turns out successful and we know by the Vendor:Device ID it is + * safe to do so, then lift the restriction, letting the devices negotiate + * a higher speed. Also check for a similar 2.5GT/s speed restriction the + * firmware may have already arranged and lift it with ports that already + * report their data link being up. + * + * Return TRUE if the link has been successfully retrained, otherwise FALS= E. + */ +bool pcie_failed_link_retrain(struct pci_dev *dev) +{ + static const struct pci_device_id ids[] =3D { + { PCI_VDEVICE(ASMEDIA, 0x2824) }, /* ASMedia ASM2824 */ + {} + }; + u16 lnksta, lnkctl2; + + if (!pci_is_pcie(dev) || !pcie_downstream_port(dev) || + !pcie_cap_has_lnkctl2(dev) || !dev->link_active_reporting) + return false; + + pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &lnkctl2); + pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); + if ((lnksta & (PCI_EXP_LNKSTA_LBMS | PCI_EXP_LNKSTA_DLLLA)) =3D=3D + PCI_EXP_LNKSTA_LBMS) { + pci_info(dev, "broken device, retraining non-functional downstream link = at 2.5GT/s\n"); + + lnkctl2 &=3D ~PCI_EXP_LNKCTL2_TLS; + lnkctl2 |=3D PCI_EXP_LNKCTL2_TLS_2_5GT; + pcie_capability_write_word(dev, PCI_EXP_LNKCTL2, lnkctl2); + + if (!pcie_retrain_link(dev, false)) { + pci_info(dev, "retraining failed\n"); + return false; + } + + pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); + } + + if ((lnksta & PCI_EXP_LNKSTA_DLLLA) && + (lnkctl2 & PCI_EXP_LNKCTL2_TLS) =3D=3D PCI_EXP_LNKCTL2_TLS_2_5GT && + pci_match_id(ids, dev)) { + u32 lnkcap; + + pci_info(dev, "removing 2.5GT/s downstream link speed restriction\n"); + pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &lnkcap); + lnkctl2 &=3D ~PCI_EXP_LNKCTL2_TLS; + lnkctl2 |=3D lnkcap & PCI_EXP_LNKCAP_SLS; + pcie_capability_write_word(dev, PCI_EXP_LNKCTL2, lnkctl2); + + if (!pcie_retrain_link(dev, false)) { + pci_info(dev, "retraining failed\n"); + return false; + } + } + + return true; +} + static ktime_t fixup_debug_start(struct pci_dev *dev, void (*fn)(struct pci_dev *dev)) {