From nobody Fri Sep 4 05:20:08 2026 Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2B7E38AC65 for ; Fri, 4 Sep 2026 02:48:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.198 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788490112; cv=none; b=Z257/Dv7wD8PKJ6mHwpjDuDKBewBTwsqQ+nqWFzKSxyerwoiYH8RgE5hlA/ezLszXlwkDlkDHnnJOxPV2Q59h72hqTNIkN4yJCfeWqfhIHG4ScAhi42oW+yknkuTM5+XfEO3NowxAxrxrdy0UCyMAnbUIR4VAc6iApWKjPy1R+A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788490112; c=relaxed/simple; bh=4ZavyL42xtglVBipSMRWA6ihGhpqB9JmwP1ZH/ASTvw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qXTELXgyRMc4WUORdrnQG4h5mY6YgjLfs+B8xJVsA+LCP+tkeW8xVUbvOGpqKccm+/oAgbzKcEQyJfYxz6Xaj2eTy2WGdRm50p3Rg0eBJtZF8eg3B10F5NhEcttW+gS/rJ095WJKJ8q6h8AAa8xw4TpO4hxzL1pFP/LZ0ll36EY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--rathodpriyank.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VZS19nsP; arc=none smtp.client-ip=209.85.210.198 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--rathodpriyank.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VZS19nsP" Received: by mail-pf1-f198.google.com with SMTP id d2e1a72fcca58-85321e37104so648856b3a.3 for ; Thu, 03 Sep 2026 19:48:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788490110; x=1789094910; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=ZsPb5bVQxBixryBb9IXokn4XCrzoWNCx2hg/i/D0/QM=; b=VZS19nsPAlk++BgBjodp/6ZR158Fq63p9aLTLCKhKbJ8kUoQQ2tZk9KqIUgttCmfRv u37iiqT+x95KsB/f0AoNTn19GkkhrQFXHNjr0G7lNlK1Kj5r10QvlL7JjlgEiAJB8RW/ G9xHAdRB16nySxukxHvxmVV2FGbToo2hVVE6RFdP09fTVgr56MIEhrmNLqNlRbToD6bY XiA9OZOVTHtOVb1iz0pZtNVsUWo1Y/Ay8JWBT0DiAFlU1+MvhlMcki78un6azM5f1qmn CHw5yKQfYK2dsvRtsAPgun0oEOFYM8PXHCYIjZERoIsrcBElxfN+atdT7PkW2TlM0lAJ hx1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788490110; x=1789094910; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ZsPb5bVQxBixryBb9IXokn4XCrzoWNCx2hg/i/D0/QM=; b=bIlRKq4SzQ1A344wZH0erpOo79z3j1vPREp2u1Grx7PQqkp+kWQ6Zi/dJ5rmHKUGjd ixEsmMVn1TVWAKomb7yC1Y5C2ln9eDI8DNqqQS01oa/+YDGcF4VK3YFj78RmtqRITnhx PJI7DqLofzC8QkJl4qCaFEpZ4gLrOj9qaV28tiMwcvRebnOvjYwfUTwZOy4V8r3sNrnx JPO6Riy7c4nEyNex+r7vYx8vjrAEGP9CWEADL+cI8IFT4AUH+3a37kWiI3WWpRDB4U/P dGEHLSFHIdM24ziaKE57ovets492sxkDO/pc+g9k1r2mFcdTluYKNTUXlzHCKCAVICZr ftEA== X-Forwarded-Encrypted: i=1; AKwUvByDS41QzlQcVx8qObcdNUdYQ8eE96VZRas7ba7Ka+fAbSieokShb1Zu5Gv6uSqYLeTVE0TuQLBtHklTNGI=@vger.kernel.org X-Gm-Message-State: AFuF++n4nchOh3HFbZM8cVgbKvfsylpsCZTN+OqdpltOvvJTNA57VbCe bfaAPETyV8X0jDYCEd8CyjEh8n4AnXfzkziHf7h9hlOl0sUZkU37LIGn2LiQPRw0Ruqe2N09dF8 IZAd1WbD1J6bnf82/4gcKl5vfLkU6/PYW1A== X-Received: from pfbdo7.prod.google.com ([2002:a05:6a00:4a07:b0:845:a3d5:fed]) (user=rathodpriyank job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:4298:b0:857:726d:2706 with SMTP id d2e1a72fcca58-8616c63f134mr5037616b3a.18.1788490110155; Thu, 03 Sep 2026 19:48:30 -0700 (PDT) Date: Fri, 04 Sep 2026 02:48:28 +0000 In-Reply-To: <20260904-pcie-link-endpoints-v2-0-16fcb301a3e4@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260904-pcie-link-endpoints-v2-0-16fcb301a3e4@google.com> X-Mailer: b4 0.14.3 Message-ID: <20260904-pcie-link-endpoints-v2-1-16fcb301a3e4@google.com> Subject: [PATCH v2 1/3] PCI: Add pcie_get_link_endpoints() helper From: Priyank Rathod To: Bjorn Helgaas , Jonathan Corbet , Shuah Khan , Randy Dunlap , Kees Cook , "Gustavo A. R. Silva" Cc: "=?utf-8?q?Ilpo_J=C3=A4rvinen?=" , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, Priyank Rathod , sashiko-bot@kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-hardening@vger.kernel.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable In PCIe topologies, physical links are point-to-point connections between an Upstream Component (Downstream Port, such as a Root Port or Switch Downstream Port) and a Downstream Component (Upstream Port, such as an Endpoint or Switch Upstream Port), per PCIe Base Specification Revision 7.0 / 6.0 sec 1.3.1. Drivers across drivers/pci/ (such as Lane Margining at Receiver and AER recovery) coordinate operations across both ends of a PCIe link: - Lane Margining at Receiver (LMR): requires resolving both link endpoints to establish hierarchical locking and runtime PM pinning. - AER/DPC Recovery: identifies the parent bridge to coordinate link resets and recovery sequences. Introduce pcie_get_link_endpoints() and pcie_put_link_endpoints() in the PCI core to provide a standardized, symmetric, and race-safe helper: - Validates output pointer arguments and initializes them to NULL. - Resolves Physical Functions for VFs via pci_physfn(). - For Endpoints: resolves parent Downstream Port via pci_upstream_bridge(). - Resolves base Function 0 symmetrically on the link subordinate bus for both non-ARI multi-function devices and ARI (Alternative Routing-ID Interpretation) devices where all functions (0..255) belong to the same logical device. - Safely inspects subordinate bus and bus device lists under down_read(&pci_bus_sem), verifying bridge ownership (child->self =3D=3D pdev) to eliminate ABA pointer identity and UAF races with concurrent hot-remove. - Symmetrically acquires references via pci_dev_get() on both resolved endpoints on success, or returns -ENODEV if either end is unresolved. Callers release acquired references using pcie_put_link_endpoints(). Reported-by: sashiko-bot@kernel.org Closes: https://lore.kernel.org/linux-pci/20260831214631.7CD5E1F000E9@smtp.= kernel.org/ Signed-off-by: Priyank Rathod --- drivers/pci/pci.c | 152 ++++++++++++++++++++++++++++++++++++++++++++++++= ++++ include/linux/pci.h | 5 ++ 2 files changed, 157 insertions(+) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index b2879a6be5f8..d1e3d6e3f5b1 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -4654,6 +4654,158 @@ int pcie_retrain_link(struct pci_dev *pdev, bool us= e_lt) return rc; } =20 +/* + * pcie_get_subordinate_bus_locked() - Safely obtain active subordinate bu= s. + * Validates that a child bus matching bridge @pdev is actively linked in + * pdev->bus->children under pci_bus_sem. Verifies bridge ownership + * (child->self =3D=3D pdev) to prevent Use-After-Free or ABA pointer iden= tity + * races during concurrent hot-remove (pci_remove_bus / pci_remove_bus_dev= ice). + */ +static struct pci_bus *pcie_get_subordinate_bus_locked(struct pci_dev *pde= v) +{ + struct pci_bus *child; + + lockdep_assert_held_read(&pci_bus_sem); + + if (!pdev->subordinate) + return NULL; + + list_for_each_entry(child, &pdev->bus->children, node) { + if (child->self =3D=3D pdev) + return child; + } + + return NULL; +} + +/* + * pcie_find_link_upstream_func0() - Find the base Function 0 device on a = link. + * Handles ARI (Alternative Routing-ID Interpretation) and multi-function + * topologies uniformly and symmetrically. + */ +static struct pci_dev *pcie_find_link_upstream_func0(struct pci_bus *bus, + struct pci_dev *hint) +{ + struct pci_dev *child; + + if (!bus) + return NULL; + + if (pci_ari_enabled(bus)) { + /* + * In ARI, all functions (0..255) on the bus belong to the same + * logical device. Base Function 0 is strictly devfn =3D=3D 0. + */ + list_for_each_entry(child, &bus->devices, bus_list) { + if (child->devfn =3D=3D 0) + return pci_dev_get(child); + } + } else if (hint) { + /* Non-ARI: find Function 0 in the same device slot */ + if (PCI_FUNC(hint->devfn) =3D=3D 0) + return pci_dev_get(hint); + + list_for_each_entry(child, &bus->devices, bus_list) { + if (PCI_SLOT(child->devfn) =3D=3D PCI_SLOT(hint->devfn) && + PCI_FUNC(child->devfn) =3D=3D 0) + return pci_dev_get(child); + } + } else { + /* Non-ARI from Downstream Port: prefer devfn 0, then any Func 0 */ + list_for_each_entry(child, &bus->devices, bus_list) { + if (child->devfn =3D=3D 0) + return pci_dev_get(child); + } + list_for_each_entry(child, &bus->devices, bus_list) { + if (PCI_FUNC(child->devfn) =3D=3D 0) + return pci_dev_get(child); + } + } + + /* Fall back to hint or first device on subordinate bus */ + if (hint) + return pci_dev_get(hint); + + child =3D list_first_entry_or_null(&bus->devices, struct pci_dev, bus_lis= t); + return pci_dev_get(child); +} + +/** + * pcie_get_link_endpoints - Identify Upstream and Downstream ends of a PC= Ie link + * @pdev: Any PCIe device on the link (Downstream Port or Endpoint) + * @downstream_port: Output pointer to Downstream Port (Upstream Component) + * @upstream_port: Output pointer to Upstream Port (Downstream Component) + * + * Identifies both ends of a point-to-point PCIe link. Acquires a reference + * (pci_dev_get()) on both discovered endpoints on success. Callers must r= elease + * acquired references with pcie_put_link_endpoints() or pci_dev_put(). + * + * Return: 0 on success, -EINVAL if @pdev is NULL or not PCIe, or -ENODEV = if + * either end of the link cannot be resolved. + */ +int pcie_get_link_endpoints(struct pci_dev *pdev, + struct pci_dev **downstream_port, + struct pci_dev **upstream_port) +{ + struct pci_dev *down =3D NULL, *up =3D NULL; + + if (!downstream_port || !upstream_port) + return -EINVAL; + + *downstream_port =3D NULL; + *upstream_port =3D NULL; + + if (!pdev || !pci_is_pcie(pdev)) + return -EINVAL; + + pdev =3D pci_physfn(pdev); + if (!pdev->bus) + return -ENODEV; + + if (pcie_downstream_port(pdev)) { + struct pci_bus *subordinate; + + down_read(&pci_bus_sem); + subordinate =3D pcie_get_subordinate_bus_locked(pdev); + if (subordinate) + up =3D pcie_find_link_upstream_func0(subordinate, NULL); + up_read(&pci_bus_sem); + down =3D pci_dev_get(pdev); + } else { + down =3D pci_dev_get(pci_upstream_bridge(pdev)); + down_read(&pci_bus_sem); + up =3D pcie_find_link_upstream_func0(pdev->bus, pdev); + up_read(&pci_bus_sem); + if (!up) + up =3D pci_dev_get(pdev); + } + + if (!down || !up || !pci_is_pcie(down) || !pci_is_pcie(up)) { + pci_dev_put(down); + pci_dev_put(up); + return -ENODEV; + } + + *downstream_port =3D down; + *upstream_port =3D up; + + return 0; +} +EXPORT_SYMBOL_GPL(pcie_get_link_endpoints); + +/** + * pcie_put_link_endpoints - Release references acquired by pcie_get_link_= endpoints + * @downstream_port: Downstream Port pointer + * @upstream_port: Upstream Port pointer + */ +void pcie_put_link_endpoints(struct pci_dev *downstream_port, + struct pci_dev *upstream_port) +{ + pci_dev_put(upstream_port); + pci_dev_put(downstream_port); +} +EXPORT_SYMBOL_GPL(pcie_put_link_endpoints); + /** * pcie_wait_for_link_delay - Wait until link is active or inactive * @pdev: Bridge device diff --git a/include/linux/pci.h b/include/linux/pci.h index d31a8d107b1e..671d8db5898e 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1272,6 +1272,11 @@ struct resource *pci_find_parent_resource(const stru= ct pci_dev *dev, u8 pci_swizzle_interrupt_pin(const struct pci_dev *dev, u8 pin); int pci_get_interrupt_pin(struct pci_dev *dev, struct pci_dev **bridge); u8 pci_common_swizzle(struct pci_dev *dev, u8 *pinp); +int pcie_get_link_endpoints(struct pci_dev *pdev, + struct pci_dev **downstream_port, + struct pci_dev **upstream_port); +void pcie_put_link_endpoints(struct pci_dev *downstream_port, + struct pci_dev *upstream_port); struct pci_dev *pci_dev_get(struct pci_dev *dev); void pci_dev_put(struct pci_dev *dev); DEFINE_FREE(pci_dev_put, struct pci_dev *, if (_T) pci_dev_put(_T)) --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 4 05:20:08 2026 Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6F333921DB for ; Fri, 4 Sep 2026 02:48:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788490114; cv=none; b=oNUSzEutJJUkUIln9O2z7H5mbH9yfCNF9el+3DLtPL9r8lZRmKb7MNi3uXaGphELg4a6iEN8ircBRkwl4N3E8q2bIOzaSStWTA5UxA7CkJF0pPIMKaQUKtBbwg6D1bN4Io9nBLuxZLQGea9cK8yH4pvqmNQYbRh3huuVFlpIaNg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788490114; c=relaxed/simple; bh=1M3as6/JbviFnzO+sZjaXmcLIqRvpUkfQlZvAuRD3Wk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=E/TM7cQ6k0uHXFnfDtFNgCfnEMNoljk+MrLu+c/3mRUBNNEDKHZRlI2Ujw5LmeeGLxtrlKp7R1fZbEI55nWrjb2DjBxHRp+KZC7Zw8W6vqAJ/u6LzEtXLC8WOPNr/ILhGA2QGS20AZ3nmWPZ5/OmXe6Ddi25vImtLYMVKyZ5XWI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--rathodpriyank.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=hn0T4dAy; arc=none smtp.client-ip=209.85.210.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--rathodpriyank.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hn0T4dAy" Received: by mail-pf1-f200.google.com with SMTP id d2e1a72fcca58-854f274dd69so311424b3a.1 for ; Thu, 03 Sep 2026 19:48:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788490111; x=1789094911; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=0sFYKtyudLOnvYV5nLPU7PI7+uaLgwyI8nUNXoVpPUU=; b=hn0T4dAyLdjsqyF2Xef1AO7c50OKWtz8oNLtMhE0WO05x0q2fAJgrMAG4Gn2xZoFJ5 GI6pr5arf5d6oPSc9v+S2Y5B5EAqjpO0SNhsCy4sglUHgAVZbP06h17w3s0cIOf1X8Db m8zD0cPnT3UEaiENM/jZnjklfMfVxehJvWp60MId1TLtpI9Tp86JsClcJvuxUCJPjUO4 mDzsoxPTWSGOH27QHgg0xxUAcTVmvwotFlXuCvF5G8W0EQIcW9OT8eJC5XYBHNoTuyby EAVWGd49hYhVJvFOL9pUP4XJrS7mo1B9smmLE/7Hlpw/MgTUEKNujfc7Pi7gNx4hRzgi Fe0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788490111; x=1789094911; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0sFYKtyudLOnvYV5nLPU7PI7+uaLgwyI8nUNXoVpPUU=; b=rqI9NdZj/a+6gMk5yF4KXTcrq21FmmDy7GmXTzhCTToaKuDUkgtCo2rGCzfv6ZK3Nn GXSxrwVnncMN0fSumUI3uwGwWU2O6KZuvUVxqeqX5fxJVd5dd1vabWIgJ4t8l3Xa8laF WurHD/6NN1TdpzaYBcni9x9pWvMoZ51MoMbOEhzT4tTBuBHEtlq07s+i8AqL2UgMMhrt A4q7ByYi3Y2wm0tEajx+bHsJlLxe6XuV6DZ4j7OSHUrFVr6mjUXh0EBGUSoDKhF/N9v0 sR3kTbkXFFb+7GoyUIh2Zb6pRAMoSQ0aM7T0gWOvSO0dRWs/Esd7SsT7LoggTWLfI2ke YoGA== X-Forwarded-Encrypted: i=1; AKwUvBy6R2B2+dbGSA5V0gtzmx/bsx7M6RPkql4a7USbrLstte5bTl2k1VEvg0gTB1HBm2q3YpYQUSe9+f652pw=@vger.kernel.org X-Gm-Message-State: AFuF++nnHM2MJPJ+YeYmiIXfQP2hghssifPvatw7oZfCnJKfuLT0idoF QbsUe4TJAuG7T/iCEdJZwZPhk/gqdxjJe6Z0+AK5LcycDZ554B2HNeqSsNAyyJ1qe7NOKfn/26d t+P040GakS8OPFg86i/r1b6+NhlixtHgvrA== X-Received: from pfbkx35.prod.google.com ([2002:a05:6a00:6f23:b0:848:5395:9a78]) (user=rathodpriyank job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:4289:b0:852:38ea:3fd with SMTP id d2e1a72fcca58-861676b276dmr4658827b3a.11.1788490111027; Thu, 03 Sep 2026 19:48:31 -0700 (PDT) Date: Fri, 04 Sep 2026 02:48:29 +0000 In-Reply-To: <20260904-pcie-link-endpoints-v2-0-16fcb301a3e4@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260904-pcie-link-endpoints-v2-0-16fcb301a3e4@google.com> X-Mailer: b4 0.14.3 Message-ID: <20260904-pcie-link-endpoints-v2-2-16fcb301a3e4@google.com> Subject: [PATCH v2 2/3] PCI/ASPM: Add pci_aspm_inhibit() helper for temporary link state suppression From: Priyank Rathod To: Bjorn Helgaas , Jonathan Corbet , Shuah Khan , Randy Dunlap , Kees Cook , "Gustavo A. R. Silva" Cc: "=?utf-8?q?Ilpo_J=C3=A4rvinen?=" , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, Priyank Rathod , sashiko-bot@kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-hardening@vger.kernel.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Currently, pci_disable_link_state() permanently disables ASPM on a PCIe link by setting link->aspm_disable. However, several kernel subsystems and driver operations require temporary ASPM inhibition without permanently disabling power management: - PCIe Lane Margining at Receiver (LMR): receiver testing requires the link to remain continuously in L0 throughout the test duration. - Device firmware updates (e.g. NICs and NVMe controllers): link latency or L1 transitions during firmware flashing can disrupt device communication. - Secondary Bus Reset and link retraining sequences: prevent unexpected L1/L1SS entry during link reset and recovery. Because pci_disable_link_state() cannot be reversed by pci_enable_link_state() to clear link->aspm_disable, drivers previously attempted manual manipulations of PCI_EXP_LNKCTL (ASPMC). This bypasses the ASPM driver, risking race conditions and state desynchronization. Introduce pci_aspm_inhibit() and pci_aspm_inhibit_locked() in the ASPM driver: - Updates pcie_aspm_get_link() to resolve links for Root Ports, Switch Downstream Ports, and Endpoints uniformly. - Validates PCIe capability and resolves the link state under pci_bus_sem and aspm_lock in __pci_aspm_inhibit() before inspecting aspm_disabled with rate-limited logging, eliminating log floods and TOCTOU races with concurrent device removal. - Tracks inhibition via an aspm_inhibit_cnt reference counter on struct pcie_link_state. - When the first inhibitor requests suppression (aspm_inhibit_cnt =3D=3D = 1), forces the link to L0 via pcie_config_aspm_link(link, 0), which enforces spec-compliant disable sequencing (Downstream Component before Upstream Component per PCIe Base Specification Revision 7.0 sec 7.5.3.7 & Table 7-24 "Link Control Register Description"). - Waits under aspm_lock on initial inhibit to guarantee concurrent callers cannot access the link before the hardware transition stabilizes. - When all inhibitors have released (aspm_inhibit_cnt =3D=3D 0), restores the configured ASPM policy to hardware (Upstream Component before Downstream Component per sec 7.5.3.7 & Table 7-24). Signed-off-by: Priyank Rathod --- drivers/pci/pcie/aspm.c | 100 ++++++++++++++++++++++++++++++++++++++++++++= ++++ include/linux/pci.h | 6 +++ 2 files changed, 106 insertions(+) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 95ac34a34bd5..4a85d029708b 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -245,6 +246,9 @@ struct pcie_link_state { u32 clkpm_enabled:1; /* Current Clock PM state */ u32 clkpm_default:1; /* Default Clock PM state by BIOS */ u32 clkpm_disable:1; /* Clock PM disabled */ + + /* Temporary ASPM Inhibit state */ + unsigned int aspm_inhibit_cnt; /* Reference count for ASPM inhibition */ }; =20 static bool aspm_disabled, aspm_force; @@ -1055,6 +1059,10 @@ static void pcie_config_aspm_link(struct pcie_link_s= tate *link, u32 state) /* Enable only the states that were not explicitly disabled */ state &=3D (link->aspm_capable & ~link->aspm_disable); =20 + /* If ASPM is temporarily inhibited, force state to 0 (L0) */ + if (link->aspm_inhibit_cnt) + state =3D 0; + /* Can't enable any substates if L1 is not enabled */ if (!(state & PCIE_LINK_STATE_L1)) state &=3D ~PCIE_LINK_STATE_L1SS; @@ -1467,6 +1475,9 @@ static struct pcie_link_state *pcie_aspm_get_link(str= uct pci_dev *pdev) if (!pci_is_pcie(pdev)) return NULL; =20 + if (pcie_downstream_port(pdev)) + return pdev->link_state; + bridge =3D pci_upstream_bridge(pdev); if (!bridge || !pci_is_pcie(bridge)) return NULL; @@ -1531,6 +1542,95 @@ static int __pci_disable_link_state(struct pci_dev *= pdev, int state, bool locked return 0; } =20 +/* + * __pci_aspm_inhibit() - Inhibit or restore ASPM L0s/L1 on a PCIe link. + * + * PCIe Base Specification Revision 7.0 sec 7.5.3.7 & Table 7-24 ("Link + * Control Register Description"): + * - To disable ASPM, software on Downstream Component (Endpoint / Upstream + * Port) must disable ASPM prior to disabling ASPM on Upstream Component + * (Root Port / Downstream Port). + * - To enable ASPM, software on Upstream Component (Root Port / Downstream + * Port) must enable ASPM prior to enabling ASPM on Downstream Component + * (Endpoint / Upstream Port). + */ +static int __pci_aspm_inhibit(struct pci_dev *pdev, bool inhibit, bool loc= ked) +{ + struct pcie_link_state *link; + int ret =3D 0; + + if (!pdev || !pci_is_pcie(pdev)) + return -EINVAL; + + pdev =3D pci_physfn(pdev); + + if (!locked) + down_read(&pci_bus_sem); + mutex_lock(&aspm_lock); + + link =3D pcie_aspm_get_link(pdev); + if (!link) { + ret =3D -EINVAL; + goto unlock; + } + + if (aspm_disabled) { + pci_warn_once(pdev, "can't inhibit ASPM; OS doesn't have ASPM control\n"= ); + ret =3D -EPERM; + goto unlock; + } + + if (inhibit) { + link->aspm_inhibit_cnt++; + if (link->aspm_inhibit_cnt =3D=3D 1) { + pcie_config_aspm_link(link, 0); + usleep_range(2000, 3000); + } + } else { + if (WARN_ON_ONCE(link->aspm_inhibit_cnt =3D=3D 0)) { + ret =3D -EINVAL; + goto unlock; + } + + link->aspm_inhibit_cnt--; + if (link->aspm_inhibit_cnt =3D=3D 0) + pcie_config_aspm_link(link, policy_to_aspm_state(link)); + } + +unlock: + mutex_unlock(&aspm_lock); + if (!locked) + up_read(&pci_bus_sem); + + return ret; +} + +int pci_aspm_inhibit_locked(struct pci_dev *pdev, bool inhibit) +{ + lockdep_assert_held_read(&pci_bus_sem); + + return __pci_aspm_inhibit(pdev, inhibit, true); +} +EXPORT_SYMBOL_GPL(pci_aspm_inhibit_locked); + +/** + * pci_aspm_inhibit - Temporarily inhibit or restore ASPM on a PCIe link + * @pdev: PCI device on the link + * @inhibit: True to inhibit ASPM (transition to L0), false to release + * + * Increments/decrements a reference counter on the link's ASPM state. When + * @inhibit is true, forces the link to L0 on the first inhibitor. When @i= nhibit + * is false, restores the configured ASPM state once all inhibitors have + * released their claims. + * + * Return: 0 on success, or a negative errno. + */ +int pci_aspm_inhibit(struct pci_dev *pdev, bool inhibit) +{ + return __pci_aspm_inhibit(pdev, inhibit, false); +} +EXPORT_SYMBOL_GPL(pci_aspm_inhibit); + int pci_disable_link_state_locked(struct pci_dev *pdev, int state) { lockdep_assert_held_read(&pci_bus_sem); diff --git a/include/linux/pci.h b/include/linux/pci.h index 671d8db5898e..b0ce8ee8d622 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1956,6 +1956,8 @@ int pci_disable_link_state(struct pci_dev *pdev, int = state); int pci_disable_link_state_locked(struct pci_dev *pdev, int state); int pci_enable_link_state(struct pci_dev *pdev, int state); int pci_enable_link_state_locked(struct pci_dev *pdev, int state); +int pci_aspm_inhibit(struct pci_dev *pdev, bool inhibit); +int pci_aspm_inhibit_locked(struct pci_dev *pdev, bool inhibit); void pcie_no_aspm(void); bool pcie_aspm_support_enabled(void); bool pcie_aspm_enabled(struct pci_dev *pdev); @@ -1968,6 +1970,10 @@ static inline int pci_enable_link_state(struct pci_d= ev *pdev, int state) { return 0; } static inline int pci_enable_link_state_locked(struct pci_dev *pdev, int s= tate) { return 0; } +static inline int pci_aspm_inhibit(struct pci_dev *pdev, bool inhibit) +{ return 0; } +static inline int pci_aspm_inhibit_locked(struct pci_dev *pdev, bool inhib= it) +{ return 0; } static inline void pcie_no_aspm(void) { } static inline bool pcie_aspm_support_enabled(void) { return false; } static inline bool pcie_aspm_enabled(struct pci_dev *pdev) { return false;= } --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 4 05:20:08 2026 Received: from mail-pj1-f71.google.com (mail-pj1-f71.google.com [209.85.216.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 290123AC0FB for ; Fri, 4 Sep 2026 02:48:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788490118; cv=none; b=IzeYUbU74Cz8O5ord9MsyrsE4L9VVdvGCBom43NtXuCHCEB9NsQrY3zu0KyWyfcJr2hTw/hVp6h5pOkhTB9K3dWO4ztdpdY34z859oGSrSsMUmkzk3hlGOQv0nJsdw65gW0eNsBzE/mk8AetF1Q33utY9OjAbIKb8mt1gzmee9E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788490118; c=relaxed/simple; bh=Kxq935JJP/CqltM1DhVvplI6qCbPwrGJHYtpMz+H1y0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kZUv2PcTht65BiXjc0dddFLsI5MuJ9u5x+LpFslOyx6QakNESosn9EwcymrY72JtPWQFvCO1fMjTVeexF2DWi9YAmnys5n4pD7LjL4/gQtwfFx1HYYwJYcqRHGVUp/IAqP8mT9vfxvWu/2d7uV0xRvDje+qxZI54VeXzjV4h0YU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--rathodpriyank.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=D18j4Q6v; arc=none smtp.client-ip=209.85.216.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--rathodpriyank.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="D18j4Q6v" Received: by mail-pj1-f71.google.com with SMTP id 98e67ed59e1d1-38f283baf1fso654498a91.3 for ; Thu, 03 Sep 2026 19:48:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1788490112; x=1789094912; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=52EZ7jnHXHpIgst2AUL8Hf/DzD4ZmTivxGlVuAX/kx8=; b=D18j4Q6vCTWaYOA+Atq7W3vSfDplqBo+URRKu2/0/+4kgeX2sIfjjrhdpFCjkd7H0x gt3oFGUYnoOSTVZJDHHFvNdpp6Z/2Jrj7mfApRHcZE8Mi8a+liSOU30ekc/4kqpgUL08 QDN3zP/nK+++/ot/UZruQE1q2l0ZOsKtjkniPX12acR/IBGM762gkmenyFv3ukNZIJd4 Ixpb5Lp6jMUdjoFzRwU0p85V93Y2hVpoQcdXQ6O25uIG+rKrhIyF9V0+7M4537aczrzx zmK3sC+mX26GNMjuddvGmOY+j+9POWv9hZ9LiEe6Zg0Fsgxqa574n//gxMQIZRjbQfQ4 QkZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788490112; x=1789094912; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=52EZ7jnHXHpIgst2AUL8Hf/DzD4ZmTivxGlVuAX/kx8=; b=qBwbuwJynCGtvL7MoBe7kfqY7T7DQeez65L2dqatg+tOYlsuGKs/SWBCC+Pm5RGq4s /V8HAQ5psh2kAk+8Hqs7NioWzFNOhCSJABiMl49twgc0b19gpxBEUowlG2GQP5D3wZQg 6qxVeJqoFDTfCCa2u2IcyrjwJCAqc4aR0y22j6TPpZJRUaAMVcuA+g0ahRkdf/2NlRN/ 90sNqJLMM6iU9S1GszpjX6OGPHOig5j447yDrxwY1iSzEpVxILKKxb/QwOkkAZDP/l5O Di1WIWOPFe+uF9s2zS8EsZi4oX7up0Ii2Lw2K15d21LeSaYmQ6RlNQ1Eo34s9OFsY76m zDow== X-Forwarded-Encrypted: i=1; AKwUvByivSln+rbgiQh35fbzQD4vDXr+c5FFmb834gZOQ30Qlv998ZdR33InrMDCCT+n3DN+VlNxfFQJM3ggJUY=@vger.kernel.org X-Gm-Message-State: AFuF++nI/FiMN5oog1IBsRaDFicUbwyNO+mQKCV6R92vfclz532EOq2O 1vjMMeZXxZTo2QGG54jmLCWp6HVgjHf1M0vu9Hmt1nEMYFcwkF/Cj75JsFxjw6sX2/MEqlsJFFp iyNZYbGJvQEWy76EJcpgr6mTOOCHYO31VpQ== X-Received: from pjbbg9.prod.google.com ([2002:a17:90b:d89:b0:398:c7c7:ede1]) (user=rathodpriyank job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5246:b0:39a:e983:d4bd with SMTP id 98e67ed59e1d1-39b262a3cf7mr5364343a91.24.1788490112065; Thu, 03 Sep 2026 19:48:32 -0700 (PDT) Date: Fri, 04 Sep 2026 02:48:30 +0000 In-Reply-To: <20260904-pcie-link-endpoints-v2-0-16fcb301a3e4@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260904-pcie-link-endpoints-v2-0-16fcb301a3e4@google.com> X-Mailer: b4 0.14.3 Message-ID: <20260904-pcie-link-endpoints-v2-3-16fcb301a3e4@google.com> Subject: [PATCH v2 3/3] PCI/pcie: Add PCIe Lane Margining at Receiver (LMR) support From: Priyank Rathod To: Bjorn Helgaas , Jonathan Corbet , Shuah Khan , Randy Dunlap , Kees Cook , "Gustavo A. R. Silva" Cc: "=?utf-8?q?Ilpo_J=C3=A4rvinen?=" , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, Priyank Rathod , sashiko-bot@kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-hardening@vger.kernel.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Per PCIe Base Specification r6.0, sec 8.4.4 ("Lane Margining at Receiver"), PCIe devices operating at 16.0 GT/s (Gen 4) or higher data rates support the Lane Margining at Receiver Extended Capability (ID 0x27), and it is mandatory for receivers operating at 64.0 GT/s (Gen 6) or higher data rates. Lane Margining allows software to evaluate high-speed link margins by measuring timing and voltage steps for each individual physical lane and receiver. Add driver and debugfs support for PCIe Lane Margining at Receiver: - Add Lane Margining at Receiver Extended Capability register definitions (PCI_EXT_CAP_ID_LMR, PCI_LMR_PORT_CAP, PCI_LMR_PORT_STS, PCI_LMR_LANE_CTRL, PCI_LMR_LANE_STS) to . - Add Kconfig option CONFIG_PCIE_LMR (under drivers/pci/pcie/Kconfig) dependent on DEBUG_FS. - Implement drivers/pci/pcie/margin.c to probe the capability on Gen4+ links and expose per-device debugfs entries under: /sys/kernel/debug/pci/pcie_lmr_/ providing control over margining enablement, receiver selection, and execution of timing/voltage margin step commands. Distinguish between missing mandatory LMR capability on Gen6+ vs optional on Gen4/Gen5. - Hook pci_lmr_init() into pci_init_capabilities() during device probe in drivers/pci/probe.c and pci_lmr_exit() into drivers/pci/remove.c. - Add kselftest script under tools/testing/selftests/pcie_lmt/pcie_lmt.sh to test debugfs capability reads, enablement, and stepping. - Add MAINTAINERS entry for PCIe Lane Margining at Receiver (LMR). Signed-off-by: Priyank Rathod --- Documentation/PCI/index.rst | 1 + Documentation/PCI/pcie-lmr.rst | 174 +++ MAINTAINERS | 8 + drivers/pci/pci.h | 8 + drivers/pci/pcie/Kconfig | 12 + drivers/pci/pcie/Makefile | 1 + drivers/pci/pcie/margin.c | 1592 ++++++++++++++++++++++= ++++ drivers/pci/probe.c | 1 + drivers/pci/remove.c | 2 +- include/linux/pci.h | 6 + include/uapi/linux/pci_regs.h | 18 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/pcie_lmt/Makefile | 3 + tools/testing/selftests/pcie_lmt/pcie_lmt.sh | 287 +++++ 14 files changed, 2113 insertions(+), 1 deletion(-) diff --git a/Documentation/PCI/index.rst b/Documentation/PCI/index.rst index 5d720d2a415e..9170c98cbf3f 100644 --- a/Documentation/PCI/index.rst +++ b/Documentation/PCI/index.rst @@ -20,3 +20,4 @@ PCI Bus Subsystem controller/index boot-interrupts tph + pcie-lmr diff --git a/Documentation/PCI/pcie-lmr.rst b/Documentation/PCI/pcie-lmr.rst new file mode 100644 index 000000000000..7f7bb242551e --- /dev/null +++ b/Documentation/PCI/pcie-lmr.rst @@ -0,0 +1,174 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D +PCI Express Lane Margining at Receiver (LMR) Subsystem +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D + +:Author: Priyank Rathod +:Copyright: 2026 Google LLC + +Overview +=3D=3D=3D=3D=3D=3D=3D=3D + +Lane Margining at Receiver (LMR), specified in the PCI Express Base +Specification (Revision 7.0 sec 7.7.11 & sec 8.4.4), +allows system software to evaluate high-speed link physical signal integri= ty and +eye margins. LMR measures available timing (jitter/phase) and voltage marg= in +offsets for each physical lane and receiver independently while the link is +operating in active L0 state. + +Lane Margining Extended Capability (ID 0x27) is optional for links operati= ng at +16.0 GT/s (PCIe Gen 4) and 32.0 GT/s (Gen 5), and is mandatory for receive= rs +operating at 64.0 GT/s (Gen 6) and higher. + +Target Receivers +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Each physical lane can margin up to 7 distinct receivers per PCIe link: + +* **Receiver 0 (Local Receiver)**: The receiver in the immediate link part= ner. +* **Receivers 1 to 6 (Retimers)**: Retimer pseudo-ports along the physical= link + (up to 3 retimers, each with upstream and downstream pseudo-ports). +* **Receiver 7**: Reserved per PCIe Base Specification. + +Kernel Configuration +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Enable the kernel configuration option under PCI support: + +.. code-block:: none + + CONFIG_PCIE_LMR=3Dy (or =3Dm) + +Dependencies: +* ``CONFIG_PCI`` +* ``CONFIG_DEBUG_FS`` + +Debugfs Interface Guide +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +When an LMR-capable device is enumerated on a Gen4+ link, the kernel expos= es +per-device control and status files under debugfs: + +.. code-block:: none + + /sys/kernel/debug/pci/pcie_lmr_::./ + +Device-Level Attributes +----------------------- + +* ``capabilities`` (read-only): + Displays the 16-bit Margining Port Capabilities register and whether the + device uses the Software Ready handshake bit. + +* ``port_status`` (read-only): + Displays the Margining Port Status register, indicating Margining Ready = and + SW Ready states. + +* ``enable`` (read-write): + Enables (``1``) or disables (``0``) Lane Margining on the device. + Enabling margining locks the link into D0, prevents runtime PM suspend, + disables ASPM L0s/L1, disables hardware autonomous link width/speed chan= ges, + and verifies that the link is operating at >=3D 16.0 GT/s. + Disabling margining restores ASPM, hardware autonomous width/speed setti= ngs, + and runtime PM, and returns all lanes to nominal (normal) operating sett= ings. + +Lane-Level Attributes +--------------------- + +For each physical lane (``lane0``, ``lane1``, ...): + +* ``receiver`` (read-write): + Gets or sets the active target receiver number (``0`` for local receiver, + ``1..6`` for retimers). Switching receivers automatically clears previous + offsets back to normal settings per PCIe single-receiver margining requi= rements. + +* ``caps`` (read-only): + Reports the target receiver's margining capabilities (PCIe Base Specific= ation + Revision 7.0 Table 4-77 and Table 8-13): + - Voltage Margining support (supported vs unsupported) + - Independent Left/Right Timing Margining support (independent vs symmet= ric) + - Independent Up/Down Voltage Margining support (independent vs symmetri= c) + - Error Sampler vs Main Sampler (independent error sampler vs intrusive = main sampler) + - Sample Reporting Method (sampling rate vs sample count) + +* ``num_timing_steps`` (read-only): + Maximum timing margin steps supported by the receiver (0..63). + +* ``num_voltage_steps`` (read-only): + Maximum voltage margin steps supported by the receiver (0..127). + +* ``margin_timing`` (read-write): + Applies timing margin step offset (+/-). Writing ``0`` clears timing mar= gin + back to nominal. + +* ``margin_voltage`` (read-write): + Applies voltage margin step offset (+/-). Writing ``0`` clears voltage m= argin + back to nominal. + +Manual Margining Example +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +1. Inspect device capabilities and status: + +.. code-block:: sh + + cat /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/capabilities + cat /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/port_status + +2. Enable Lane Margining mode: + +.. code-block:: sh + + echo 1 > /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/enable + +3. Configure target receiver and inspect step limits on lane 0: + +.. code-block:: sh + + echo 0 > /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/lane0/receiver + cat /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/lane0/caps + cat /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/lane0/num_timing_steps + cat /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/lane0/num_voltage_steps + +4. Apply timing and voltage margin steps: + +.. code-block:: sh + + # Step timing margin +2 steps + echo 2 > /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/lane0/margin_timing + + # Step voltage margin +1 step + echo 1 > /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/lane0/margin_volta= ge + +5. Reset margins back to nominal: + +.. code-block:: sh + + echo 0 > /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/lane0/margin_timing + echo 0 > /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/lane0/margin_volta= ge + +6. Disable Lane Margining when complete: + +.. code-block:: sh + + echo 0 > /sys/kernel/debug/pci/pcie_lmr_0000:01:00.0/enable + +Automated Testing via Kselftest +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D + +The kernel includes an automated kselftest script under +``tools/testing/selftests/pcie_lmt/pcie_lmt.sh`` to probe, validate, and e= xercise +debugfs controls across all enumerated LMR devices. + +Run directly as root: + +.. code-block:: sh + + sudo ./tools/testing/selftests/pcie_lmt/pcie_lmt.sh + +Or run via the kselftest test harness: + +.. code-block:: sh + + make -C tools/testing/selftests TARGETS=3Dpcie_lmt run_tests diff --git a/MAINTAINERS b/MAINTAINERS index 3a19da74d00c..312cd68a66be 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -21293,6 +21293,14 @@ F: Documentation/devicetree/bindings/pci/qcom,sa82= 55p-pcie-ep.yaml F: drivers/pci/controller/dwc/pcie-qcom-common.c F: drivers/pci/controller/dwc/pcie-qcom-ep.c =20 +PCIE LANE MARGINING AT RECEIVER (LMR) +M: Priyank Rathod +L: linux-pci@vger.kernel.org +S: Maintained +F: Documentation/PCI/pcie-lmr.rst +F: drivers/pci/pcie/margin.c +F: tools/testing/selftests/pcie_lmt/ + PCMCIA SUBSYSTEM M: Dominik Brodowski S: Odd Fixes diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index ba3c3fddddc2..0d84b693bc47 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -1071,6 +1071,14 @@ static inline void pci_no_tph(void) { } static inline void pci_tph_init(struct pci_dev *dev) { } #endif =20 +#ifdef CONFIG_PCIE_LMR +void pci_lmr_init(struct pci_dev *dev); +void pci_lmr_exit(struct pci_dev *dev); +#else +static inline void pci_lmr_init(struct pci_dev *dev) { } +static inline void pci_lmr_exit(struct pci_dev *dev) { } +#endif + #ifdef CONFIG_PCIE_PTM void pci_ptm_init(struct pci_dev *dev); void pci_save_ptm_state(struct pci_dev *dev); diff --git a/drivers/pci/pcie/Kconfig b/drivers/pci/pcie/Kconfig index 207c2deae35f..3b021ca2fe84 100644 --- a/drivers/pci/pcie/Kconfig +++ b/drivers/pci/pcie/Kconfig @@ -137,6 +137,18 @@ config PCIE_PTM This is only useful if you have devices that support PTM, but it is safe to enable even if you don't. =20 +config PCIE_LMR + bool "PCI Express Lane Margining at Receiver Support" + depends on DEBUG_FS + help + This enables the PCI Express Lane Margining at Receiver support. + Lane Margining allows software to determine the voltage and + timing margin of each lane on a PCIe link (16.0 GT/s and above). + The margining data is exposed via debugfs. + + This is only useful if you have devices that support lane + margining, but it is safe to enable even if you don't. + config PCIE_EDR bool "PCI Express Error Disconnect Recover support" depends on PCIE_DPC && ACPI diff --git a/drivers/pci/pcie/Makefile b/drivers/pci/pcie/Makefile index b0b43a18c304..aac45ae0402e 100644 --- a/drivers/pci/pcie/Makefile +++ b/drivers/pci/pcie/Makefile @@ -13,4 +13,5 @@ obj-$(CONFIG_PCIEAER_INJECT) +=3D aer_inject.o obj-$(CONFIG_PCIE_PME) +=3D pme.o obj-$(CONFIG_PCIE_DPC) +=3D dpc.o obj-$(CONFIG_PCIE_PTM) +=3D ptm.o +obj-$(CONFIG_PCIE_LMR) +=3D margin.o obj-$(CONFIG_PCIE_EDR) +=3D edr.o diff --git a/drivers/pci/pcie/margin.c b/drivers/pci/pcie/margin.c new file mode 100644 index 000000000000..1dde5aa928c7 --- /dev/null +++ b/drivers/pci/pcie/margin.c @@ -0,0 +1,1592 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * PCI Express Lane Margining at Receiver + * + * Copyright (C) 2026 Google LLC + * Author: Priyank Rathod + * + * Lane Margining at Receiver (PCIe Base Specification Revision 7.0, + * sec 7.7.11 & sec 8.4.4) allows system software to determine the voltage + * and timing margins of each physical lane on a PCIe link. The Extended + * Capability (ID 0x27) is available for receivers operating at 16.0 GT/s + * (Gen4) or higher data rates, and is mandatory for receivers operating at + * 64.0 GT/s (Gen6) or higher data rates. + * + * This driver implements: + * - Probing Extended Capability ID 0x27 and Margining Port Capabilities. + * - Managing ASPM L0s/L1 link states during active margining with + * temporary inhibition (pci_aspm_inhibit). + * - PCIe Base Specification NO_CMD (0x7) clearing handshake per receiver + * and lane. + * - Caching receiver capabilities & step counts to avoid side-effects + * when setting to normal settings. + * - Handling Symmetric vs Independent Left/Right & Up/Down margin steps. + * - Runtime PM protection (D0 enforcement) during active margining. + * - Exposing per-device debugfs interfaces under /sys/kernel/debug/pci/. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../pci.h" + +/* + * Margining Type (MTYPE) field encodings (bits 5:3) in Margining Lane Con= trol + * and Margining Lane Status registers per PCIe Base Specification + * Revision 7.0: + * - Section 7.7.11 "Lane Margining at the Receiver Extended Capability + * (ID 0x27)" (Margining Lane Control & Margining Lane Status Registers) + * - Section 4.2.18.2 "Margin Command and Response Flow" + * (Table 4-77 "Margin Commands and Corresponding Responses") + * + * Encodings: + * 001b (0x1) - Report Margin Control Capabilities + * 010b (0x2) - Set Margining Parameters (Go to Normal Settings, + * Clear Error Log) + * 011b (0x3) - Step Margin Timing + * 100b (0x4) - Step Margin Voltage + * 111b (0x7) - No Command + * (000b, 101b-110b are Reserved) + */ +#define LMR_TYPE_REPORT_CAPS 0x1 /* Report Capabilities */ +#define LMR_TYPE_SET_PARAMS 0x2 /* Set Margining Parameters */ +#define LMR_TYPE_TIMING 0x3 /* Step Margin Timing */ +#define LMR_TYPE_VOLTAGE 0x4 /* Step Margin Voltage */ +#define LMR_TYPE_NO_CMD 0x7 /* No Command */ + +/* Command Payloads per PCIe Base Specification Revision 7.0 Table 4-77 */ +#define LMR_PAYLOAD_REPORT_CAPS 0x88 /* Report Capabilities */ +#define LMR_PAYLOAD_REPORT_VOLT_STEPS 0x89 /* Report Voltage Steps */ +#define LMR_PAYLOAD_REPORT_TIM_STEPS 0x8A /* Report Timing Steps */ +#define LMR_PAYLOAD_GO_TO_NORMAL 0x0F /* Go to Normal Settings */ +#define LMR_PAYLOAD_CLEAR_ERROR_LOG 0x55 /* Clear Error Log */ +#define LMR_PAYLOAD_NO_CMD 0x9C /* No Command */ + +/* LMR command timing parameters */ +#define LMR_CMD_TIMEOUT_MS 150 +#define LMR_CMD_SLEEP_MIN_US 100 +#define LMR_CMD_SLEEP_MAX_US 250 +#define LMR_ENABLE_TIMEOUT_MS 150 +#define LMR_ENABLE_SLEEP_MIN_US 1000 +#define LMR_ENABLE_SLEEP_MAX_US 2000 + +/* + * LMR parameter limits per PCIe Base Specification Revision 7.0: + * - Max lanes (32): sec 7.7.11 & Table 8-13 (MMaxLanes max 31) + * - Receiver numbers 0..6: Table 4-76 (assignment) & Table 4-77 (valid for + * commands) + * - Max timing step (63): sec 4.2.18.1.2, Table 4-77 (8Ah), & Table 8-13 + * - Max voltage step (127): sec 4.2.18.1.2, Table 4-77 (89h), & Table 8-13 + */ +#define LMR_MAX_LANES 32 +#define LMR_MAX_RX_NUM 6 +#define LMR_MAX_TIMING_STEP 63 +#define LMR_MAX_VOLTAGE_STEP 127 + +/* LMR PCIe generation numbers and helper */ +#define LMR_GEN6 6 +#define LMR_GEN5 5 +#define LMR_GEN4 4 + +#define LMR_SPEED_TO_GEN(speed) \ + ((speed) >=3D PCIE_SPEED_64_0GT ? LMR_GEN6 : \ + (speed) >=3D PCIE_SPEED_32_0GT ? LMR_GEN5 : \ + LMR_GEN4) + +/* LMR lane register stride */ +#define LMR_LANE_REG_STRIDE 4 + +/* LMR receivers */ +#define LMR_RX_LOCAL 0 + +/* + * Margining Payload field masks for Step Margin Timing and Step Margin Vo= ltage + * per PCIe Base Specification Revision 7.0 sec 4.2.18.1.2 + * ("Margin Payload for Step Margin Commands"): + * + * Step Margin Timing Payload: + * Bit 7: Reserved (must be 0b) + * Bit 6: Direction (0 =3D Right/Up, 1 =3D Left/Down) + * Bits 5:0: Margin Step (0..63) + * + * Step Margin Voltage Payload: + * Bit 7: Direction (0 =3D Right/Up, 1 =3D Left/Down) + * Bits 6:0: Margin Step (0..127) + */ +#define LMR_TIMING_STEP_MASK GENMASK(5, 0) +#define LMR_TIMING_DIR_MASK BIT(6) +#define LMR_VOLTAGE_STEP_MASK GENMASK(6, 0) +#define LMR_VOLTAGE_DIR_MASK BIT(7) + +/* + * Margin Payload step direction field encodings per PCIe Base Specificati= on + * Revision 7.0 sec 4.2.18.1.2 ("Margin Payload for Step Margin Commands"): + * + * For timing: + * Bit 6: 0b =3D Right of normal setting (also 0b for symmetric marginin= g) + * 1b =3D Left of normal setting (when MIndLeftRightTiming is Set) + * For voltage: + * Bit 7: 0b =3D Up from normal setting (also 0b for symmetric margining) + * 1b =3D Down from normal setting (when MIndUpDownVoltage is Set) + */ +#define LMR_STEP_DIR_RIGHT_OR_UP 0 +#define LMR_STEP_DIR_LEFT_OR_DOWN 1 + +/* + * Report Margin Control Capabilities (Command 88h) response payload bit f= ields + * per PCIe Base Specification Revision 7.0 Table 4-77 & Table 8-13: + * Bit 0: MVoltageSupported (1 =3D Voltage margining supported; + * 0 =3D Not supported) + * Bit 1: MIndUpDownVoltage (1 =3D Independent Up/Down voltage support= ed; + * 0 =3D Symmetric) + * Bit 2: MIndLeftRightTiming (1 =3D Independent Left/Right timing + * supported; 0 =3D Symmetric) + * Bit 3: MSampleReportingMethod (1 =3D Sampling rate supported; + * 0 =3D Sample count supported) + * Bit 4: MIndErrorSampler (1 =3D Independent error sampler; + * 0 =3D Main data sampler) + * Bit 5: MSampleMultipleReceivers (1 =3D Multiple receivers can be sa= mpled + * concurrently; 0 =3D Only one receiver at a time) + * Bits 7:6: Reserved + */ +#define LMR_CAP_VOLTAGE_SUPPORTED BIT(0) +#define LMR_CAP_IND_UP_DOWN_VOLTAGE BIT(1) +#define LMR_CAP_IND_LEFT_RIGHT_TIMING BIT(2) +#define LMR_CAP_SAMPLE_REPORT_METHOD BIT(3) +#define LMR_CAP_IND_ERROR_SAMPLER BIT(4) +#define LMR_CAP_SAMPLE_MULTIPLE_RECEIVERS BIT(5) + +/* + * Step Margin Execution Status (Bits 7:6 of response payload per PCIe Base + * Specification Revision 7.0 sec 4.2.18.1.1 "Step Margin Execution Status= "): + * 00b: Too many errors - Receiver autonomously went back to default setti= ngs + * 01b: Set up for margin in progress + * 10b: Margining in progress + * 11b: NAK - Unsupported Lane Margining command was issued + */ +#define LMR_STS_EXEC_MASK GENMASK(7, 6) +#define LMR_STS_EXEC_TOO_MANY_ERR 0x0 +#define LMR_STS_EXEC_SETUP_IN_PROGRESS 0x1 +#define LMR_STS_EXEC_IN_PROGRESS 0x2 +#define LMR_STS_EXEC_NAK 0x3 +#define LMR_STS_ERR_CNT_MASK GENMASK(5, 0) + +/** + * struct pci_margin_rx_info - Cached Lane Margining receiver capabilities + * @caps_cached: True if receiver capabilities and step limits are cached + * @caps: Margining capabilities byte reported by receiver + * @num_timing_steps: Maximum timing margin steps supported by receiver + * @num_voltage_steps: Maximum voltage margin steps supported by receiver + */ +struct pci_margin_rx_info { + bool caps_cached; + u8 caps; + u8 num_timing_steps; + u8 num_voltage_steps; +}; + +/** + * struct pci_margin_lane - Per-lane margining state + * @mdev: Parent LMR margin device + * @lane: Physical lane index (0..num_lanes - 1) + * @rx: Selected target receiver number (0 =3D local, 1..6 =3D retimers) + * @timing_val: Current applied timing margin step offset (+/-) + * @voltage_val: Current applied voltage margin step offset (+/-) + * @rx_info: Cached receiver capabilities per receiver number + */ +struct pci_margin_lane { + struct pci_margin_dev *mdev; + int lane; + u8 rx; + int timing_val; + int voltage_val; + struct pci_margin_rx_info rx_info[LMR_MAX_RX_NUM + 1]; +}; + +/** + * struct pci_margin_dev - PCIe Lane Margining device instance + * @dev: Underlying PCI device + * @partner: Connected link partner device across the PCIe link + * @cap: Extended capability offset (PCI_EXT_CAP_ID_LMR) + * @debugfs: Root debugfs dentry for this device + * @lock: Mutex protecting LMR hardware access, active margining enablemen= t, + * target receiver selection, and lane margining steps + * @enabled: True if Lane Margining is currently enabled + * @aspm_inhibited: True if ASPM is temporarily inhibited for active margi= ning + * @autonomous_saved: True if original autonomous width/speed configuration + * has been saved + * @saved_dsp_lnkctl_valid: True if Downstream Port Link Control was saved + * @saved_dsp_lnkctl2_valid: True if Downstream Port Link Control 2 was sa= ved + * @saved_usp_lnkctl_valid: True if Upstream Port Link Control was saved + * @saved_usp_lnkctl2_valid: True if Upstream Port Link Control 2 was saved + * @saved_dsp_lnkctl: Saved Link Control register bits for Downstream Port + * @saved_dsp_lnkctl2: Saved Link Control 2 register bits for Downstream P= ort + * @saved_usp_lnkctl: Saved Link Control register bits for Upstream Port + * @saved_usp_lnkctl2: Saved Link Control 2 register bits for Upstream Port + * @num_lanes: Number of lanes on the link + * @lanes: Flexible array of per-lane state structures + */ +struct pci_margin_dev { + struct pci_dev *dev; + struct pci_dev *partner; + u16 cap; + struct dentry *debugfs; + struct mutex lock; + bool enabled; + bool aspm_inhibited; + bool autonomous_saved; + bool saved_dsp_lnkctl_valid; + bool saved_dsp_lnkctl2_valid; + bool saved_usp_lnkctl_valid; + bool saved_usp_lnkctl2_valid; + u16 saved_dsp_lnkctl; + u16 saved_dsp_lnkctl2; + u16 saved_usp_lnkctl; + u16 saved_usp_lnkctl2; + int num_lanes; + struct pci_margin_lane lanes[] __counted_by(num_lanes); +}; + +#if IS_ENABLED(CONFIG_DEBUG_FS) +static DEFINE_MUTEX(pci_debugfs_root_lock); +static struct dentry *pci_debugfs_root_dir; + +static struct dentry *get_pci_debugfs_root(void) +{ + mutex_lock(&pci_debugfs_root_lock); + if (!pci_debugfs_root_dir) + pci_debugfs_root_dir =3D debugfs_lookup("pci", NULL); + if (!pci_debugfs_root_dir) + pci_debugfs_root_dir =3D debugfs_create_dir("pci", NULL); + mutex_unlock(&pci_debugfs_root_lock); + return pci_debugfs_root_dir; +} +#endif + +/* + * pci_lmr_get_ports() - Identify Downstream and Upstream Port link partne= rs + * using the already tracked mdev->dev and mdev->partner devices. + * + * For Root Ports and Switch Downstream Ports, @dev is the Downstream Port= and + * @partner is the Upstream Port. For Endpoints and Switch Upstream Ports, + * @partner is the Downstream Port and @dev is the Upstream Port. + * + * Context: Called with mdev->lock held and partner already established. + * Does NOT acquire pci_bus_sem, preventing lock inversion deadlocks with + * device_lock. + */ +static void pci_lmr_get_ports(struct pci_margin_dev *mdev, + struct pci_dev **downstream_port, + struct pci_dev **upstream_port) +{ + struct pci_dev *dev =3D mdev->dev; + struct pci_dev *partner =3D mdev->partner; + + if (pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_ROOT_PORT || + pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_DOWNSTREAM) { + *downstream_port =3D dev; + *upstream_port =3D partner; + } else { + *downstream_port =3D partner; + *upstream_port =3D dev; + } +} + + + +/* + * Helpers to manage Autonomous Width/Speed transitions per PCIe Base + * Specification Revision 7.0: + * - Section 7.5.3.7 "Link Control Register" (Hardware Autonomous Width + * Disable, bit 9) + * - Section 7.5.3.17 "Link Control 2 Register" (Hardware Autonomous Speed + * Disable, bit 5) + * - Section 4.2.18.4 "Receiver Margin Testing Requirements" + * - Section 8.4.4 "Lane Margining at the Receiver - Electrical Requiremen= ts" + * + * Both Downstream Port and Upstream Port must save and set Hardware Auton= omous + * Width Disable and Hardware Autonomous Speed Disable bits during margini= ng to + * guarantee that the link remains in a stable active L0 state. + */ +static void pci_lmr_disable_autonomous(struct pci_margin_dev *mdev) +{ + struct pci_dev *downstream_port, *upstream_port; + u16 lnkctl, lnkctl2; + int ret; + + if (mdev->autonomous_saved) + return; + + pci_lmr_get_ports(mdev, &downstream_port, &upstream_port); + + /* 1. Downstream Component (upstream_port): Save and Disable FIRST */ + if (upstream_port && pci_is_pcie(upstream_port) && + !pci_dev_is_disconnected(upstream_port) && + upstream_port->current_state =3D=3D PCI_D0) { + ret =3D pcie_capability_read_word(upstream_port, PCI_EXP_LNKCTL, &lnkctl= ); + if (ret =3D=3D PCIBIOS_SUCCESSFUL && !PCI_POSSIBLE_ERROR(lnkctl)) { + mdev->saved_usp_lnkctl =3D lnkctl; + mdev->saved_usp_lnkctl_valid =3D true; + pcie_capability_set_word(upstream_port, PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_HAWD); + } + + ret =3D pcie_capability_read_word(upstream_port, PCI_EXP_LNKCTL2, &lnkct= l2); + if (ret =3D=3D PCIBIOS_SUCCESSFUL && !PCI_POSSIBLE_ERROR(lnkctl2)) { + mdev->saved_usp_lnkctl2 =3D lnkctl2; + mdev->saved_usp_lnkctl2_valid =3D true; + pcie_capability_set_word(upstream_port, PCI_EXP_LNKCTL2, + PCI_EXP_LNKCTL2_HASD); + } + } + + /* 2. Upstream Component (downstream_port): Save and Disable SECOND */ + if (downstream_port && pci_is_pcie(downstream_port) && + !pci_dev_is_disconnected(downstream_port) && + downstream_port->current_state =3D=3D PCI_D0) { + ret =3D pcie_capability_read_word(downstream_port, PCI_EXP_LNKCTL, &lnkc= tl); + if (ret =3D=3D PCIBIOS_SUCCESSFUL && !PCI_POSSIBLE_ERROR(lnkctl)) { + mdev->saved_dsp_lnkctl =3D lnkctl; + mdev->saved_dsp_lnkctl_valid =3D true; + pcie_capability_set_word(downstream_port, PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_HAWD); + } + + ret =3D pcie_capability_read_word(downstream_port, PCI_EXP_LNKCTL2, &lnk= ctl2); + if (ret =3D=3D PCIBIOS_SUCCESSFUL && !PCI_POSSIBLE_ERROR(lnkctl2)) { + mdev->saved_dsp_lnkctl2 =3D lnkctl2; + mdev->saved_dsp_lnkctl2_valid =3D true; + pcie_capability_set_word(downstream_port, PCI_EXP_LNKCTL2, + PCI_EXP_LNKCTL2_HASD); + } + } + + mdev->autonomous_saved =3D mdev->saved_usp_lnkctl_valid || + mdev->saved_usp_lnkctl2_valid || + mdev->saved_dsp_lnkctl_valid || + mdev->saved_dsp_lnkctl2_valid; +} + +static void pci_lmr_restore_autonomous(struct pci_margin_dev *mdev) +{ + struct pci_dev *downstream_port, *upstream_port; + + if (!mdev->autonomous_saved) + return; + + pci_lmr_get_ports(mdev, &downstream_port, &upstream_port); + + /* + * PCIe Base Specification Revision 7.0 sec 7.5.3.7 & Table 7-24: + * 1. Upstream Component (downstream_port) restored FIRST. + */ + if (downstream_port && pci_is_pcie(downstream_port) && + !pci_dev_is_disconnected(downstream_port) && + downstream_port->current_state =3D=3D PCI_D0) { + if (mdev->saved_dsp_lnkctl_valid) + pcie_capability_clear_and_set_word( + downstream_port, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD, + mdev->saved_dsp_lnkctl & PCI_EXP_LNKCTL_HAWD); + if (mdev->saved_dsp_lnkctl2_valid) + pcie_capability_clear_and_set_word( + downstream_port, PCI_EXP_LNKCTL2, PCI_EXP_LNKCTL2_HASD, + mdev->saved_dsp_lnkctl2 & PCI_EXP_LNKCTL2_HASD); + } + + /* + * 2. Downstream Component (upstream_port) restored SECOND. + */ + if (upstream_port && pci_is_pcie(upstream_port) && + !pci_dev_is_disconnected(upstream_port) && + upstream_port->current_state =3D=3D PCI_D0) { + if (mdev->saved_usp_lnkctl_valid) + pcie_capability_clear_and_set_word( + upstream_port, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD, + mdev->saved_usp_lnkctl & PCI_EXP_LNKCTL_HAWD); + if (mdev->saved_usp_lnkctl2_valid) + pcie_capability_clear_and_set_word( + upstream_port, PCI_EXP_LNKCTL2, PCI_EXP_LNKCTL2_HASD, + mdev->saved_usp_lnkctl2 & PCI_EXP_LNKCTL2_HASD); + } + + mdev->saved_dsp_lnkctl_valid =3D false; + mdev->saved_dsp_lnkctl2_valid =3D false; + mdev->saved_usp_lnkctl_valid =3D false; + mdev->saved_usp_lnkctl2_valid =3D false; + mdev->autonomous_saved =3D false; +} + +static inline u8 pci_lmr_sts_payload(u16 sts) +{ + return FIELD_GET(PCI_LMR_LANE_STS_PAYLOAD, sts); +} + +/* + * pci_lmr_get_active_lanes() - Determine the current number of active lan= es + * based on Negotiated Link Width (NLW) in Link Status Register. + * + * Per PCIe Base Specification Revision 7.0 sec 8.4.4, only active lanes + * respond to margining commands. Inactive lanes do not assert Margining R= eady, + * causing command timeouts. + * + * If the link is down (NLW =3D=3D 0 or configuration read fails), returns= 0 to + * prevent 150 ms per-lane timeout stalls on inactive/down hardware. + */ +static int pci_lmr_get_active_lanes(struct pci_margin_dev *mdev) +{ + struct pci_dev *dev =3D mdev->dev; + u16 lnksta; + int ret, nlw; + + ret =3D pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); + if (ret =3D=3D PCIBIOS_SUCCESSFUL && !PCI_POSSIBLE_ERROR(lnksta)) { + nlw =3D FIELD_GET(PCI_EXP_LNKSTA_NLW, lnksta); + if (nlw > 0 && nlw <=3D mdev->num_lanes) + return nlw; + } + return 0; +} + +/* + * pci_lmr_run_cmd() - Issue LMR command to Lane Control and wait for Stat= us. + * Must be called with mdev->lock held. + */ +static int pci_lmr_run_cmd(struct pci_margin_dev *mdev, int lane, u8 rx, u= 8 type, + u8 usage, u8 payload, u16 *status_val) +{ + struct pci_dev *dev; + u16 lmr, ctrl_offset, sts_offset; + u16 ctrl, sts; + unsigned long timeout; + int ret; + + if (!mdev || lane < 0 || lane >=3D mdev->num_lanes || rx > LMR_MAX_RX_NUM) + return -EINVAL; + + if (lane >=3D pci_lmr_get_active_lanes(mdev)) + return -ENODEV; + + dev =3D mdev->dev; + lmr =3D mdev->cap; + ctrl_offset =3D lmr + PCI_LMR_LANE_CTRL + LMR_LANE_REG_STRIDE * lane; + sts_offset =3D lmr + PCI_LMR_LANE_STS + LMR_LANE_REG_STRIDE * lane; + + /* + * Per PCIe Base Specification Revision 7.0 sec 4.2.18.2 & Table 4-77, + * software must issue NO_CMD (0x7) with payload 0x9C targeting the + * specific receiver (rx) to clear MTYPE in Lane Status before issuing + * a subsequent command. + */ + if (type !=3D LMR_TYPE_NO_CMD) { + ctrl =3D FIELD_PREP(PCI_LMR_LANE_CTRL_RX_NUM, rx) | + FIELD_PREP(PCI_LMR_LANE_CTRL_MTYPE, LMR_TYPE_NO_CMD) | + FIELD_PREP(PCI_LMR_LANE_CTRL_USAGE, 0) | + FIELD_PREP(PCI_LMR_LANE_CTRL_PAYLOAD, + LMR_PAYLOAD_NO_CMD); + + ret =3D pci_write_config_word(dev, ctrl_offset, ctrl); + if (ret !=3D PCIBIOS_SUCCESSFUL) + return pcibios_err_to_errno(ret); + + timeout =3D jiffies + msecs_to_jiffies(LMR_CMD_TIMEOUT_MS); + while (1) { + ret =3D pci_read_config_word(dev, sts_offset, &sts); + if (ret !=3D PCIBIOS_SUCCESSFUL) + return pcibios_err_to_errno(ret); + if (PCI_POSSIBLE_ERROR(sts)) + return -ENODEV; + if (FIELD_GET(PCI_LMR_LANE_STS_MTYPE, sts) =3D=3D LMR_TYPE_NO_CMD && + FIELD_GET(PCI_LMR_LANE_STS_RX_NUM, sts) =3D=3D rx) + break; + if (time_after(jiffies, timeout)) + return -ETIMEDOUT; + usleep_range(LMR_CMD_SLEEP_MIN_US, LMR_CMD_SLEEP_MAX_US); + } + } + + ctrl =3D FIELD_PREP(PCI_LMR_LANE_CTRL_RX_NUM, rx) | + FIELD_PREP(PCI_LMR_LANE_CTRL_MTYPE, type) | + FIELD_PREP(PCI_LMR_LANE_CTRL_USAGE, usage) | + FIELD_PREP(PCI_LMR_LANE_CTRL_PAYLOAD, payload); + + ret =3D pci_write_config_word(dev, ctrl_offset, ctrl); + if (ret !=3D PCIBIOS_SUCCESSFUL) + return pcibios_err_to_errno(ret); + + timeout =3D jiffies + msecs_to_jiffies(LMR_CMD_TIMEOUT_MS); + while (1) { + ret =3D pci_read_config_word(dev, sts_offset, &sts); + if (ret !=3D PCIBIOS_SUCCESSFUL) + return pcibios_err_to_errno(ret); + if (PCI_POSSIBLE_ERROR(sts)) + return -ENODEV; + + if (FIELD_GET(PCI_LMR_LANE_STS_MTYPE, sts) =3D=3D type && + FIELD_GET(PCI_LMR_LANE_STS_RX_NUM, sts) =3D=3D rx) { + if (status_val) + *status_val =3D sts; + return 0; + } + + if (time_after(jiffies, timeout)) { + /* + * Per PCIe Base Specification Revision 7.0 sec 4.2.18.2 + * & Table 4-77, if receiver echoes NO_CMD (0x7) after + * command issuance, it indicates NAK. + */ + if (FIELD_GET(PCI_LMR_LANE_STS_MTYPE, sts) =3D=3D LMR_TYPE_NO_CMD && + FIELD_GET(PCI_LMR_LANE_STS_RX_NUM, sts) =3D=3D rx) + return -EOPNOTSUPP; + break; + } + + usleep_range(LMR_CMD_SLEEP_MIN_US, LMR_CMD_SLEEP_MAX_US); + } + + return -ETIMEDOUT; +} + +/* + * pci_lmr_clear_to_normal_lane() - Clear lane margin back to normal setti= ngs + * per PCIe Base Specification Revision 7.0 sec 4.2.18.2 & Table 4-77. + * Issues Set Margining Parameters (MTYPE 010b) with "Go to Normal Setting= s" + * (Payload 0x0F). + */ +static int pci_lmr_clear_to_normal_lane(struct pci_margin_lane *plane) +{ + u16 sts; + int ret; + + if (!plane || !plane->mdev) + return -EINVAL; + + if (plane->lane >=3D pci_lmr_get_active_lanes(plane->mdev)) { + plane->timing_val =3D 0; + plane->voltage_val =3D 0; + return 0; + } + + ret =3D pci_lmr_run_cmd(plane->mdev, plane->lane, plane->rx, + LMR_TYPE_SET_PARAMS, 0, LMR_PAYLOAD_GO_TO_NORMAL, + &sts); + plane->timing_val =3D 0; + plane->voltage_val =3D 0; + return ret; +} + +static int pci_lmr_cache_rx_info(struct pci_margin_lane *plane, u8 rx) +{ + struct pci_margin_rx_info *info; + u16 sts; + int ret; + + if (!plane || rx > LMR_MAX_RX_NUM) + return -EINVAL; + + info =3D &plane->rx_info[rx]; + + if (info->caps_cached) + return 0; + + /* Issuing REPORT_CAPS aborts active margin; clear to normal settings */ + ret =3D pci_lmr_clear_to_normal_lane(plane); + if (ret) + return ret; + + /* Report Capabilities: MTYPE 001b, Payload 0x88 */ + ret =3D pci_lmr_run_cmd(plane->mdev, plane->lane, rx, + LMR_TYPE_REPORT_CAPS, 0, LMR_PAYLOAD_REPORT_CAPS, + &sts); + if (ret) + return ret; + info->caps =3D pci_lmr_sts_payload(sts); + + /* Report Timing Steps: MTYPE 001b, Payload 0x8A */ + ret =3D pci_lmr_run_cmd(plane->mdev, plane->lane, rx, + LMR_TYPE_REPORT_CAPS, 0, + LMR_PAYLOAD_REPORT_TIM_STEPS, &sts); + if (ret) + return ret; + info->num_timing_steps =3D FIELD_GET(LMR_TIMING_STEP_MASK, pci_lmr_sts_pa= yload(sts)); + + /* + * Report Voltage Steps: MTYPE 001b, Payload 0x89. + * Only query if receiver supports voltage margining. Per PCIe Base + * Specification Revision 7.0 Table 4-77, receivers lacking voltage + * margining support (MVoltageSupported =3D 0b) will NAK this command. + */ + if (info->caps & LMR_CAP_VOLTAGE_SUPPORTED) { + ret =3D pci_lmr_run_cmd(plane->mdev, plane->lane, rx, + LMR_TYPE_REPORT_CAPS, 0, + LMR_PAYLOAD_REPORT_VOLT_STEPS, &sts); + if (ret) + return ret; + info->num_voltage_steps =3D FIELD_GET(LMR_VOLTAGE_STEP_MASK, + pci_lmr_sts_payload(sts)); + } else { + info->num_voltage_steps =3D 0; + } + + info->caps_cached =3D true; + return 0; +} + +#if IS_ENABLED(CONFIG_DEBUG_FS) + +static int margin_caps_show(struct seq_file *s, void *v) +{ + struct pci_margin_dev *mdev =3D s->private; + struct pci_dev *dev =3D mdev->dev; + u16 cap; + int ret; + + /* + * Wake the hardware and hold the PM reference before accessing + * registers + */ + ret =3D pm_runtime_resume_and_get(&dev->dev); + if (ret < 0) + return ret; + + ret =3D pci_read_config_word(dev, mdev->cap + PCI_LMR_PORT_CAP, &cap); + pm_runtime_put_sync(&dev->dev); + + if (ret !=3D PCIBIOS_SUCCESSFUL) + return pcibios_err_to_errno(ret); + + seq_printf(s, "Port Capabilities: %#06x\n", cap); + seq_printf(s, " Uses SW Ready: %s\n", + str_yes_no(cap & PCI_LMR_PORT_CAP_USES_SW_READY)); + return 0; +} +DEFINE_SHOW_ATTRIBUTE(margin_caps); + +static int margin_port_status_show(struct seq_file *s, void *v) +{ + struct pci_margin_dev *mdev =3D s->private; + struct pci_dev *dev =3D mdev->dev; + u16 sts; + int ret; + + /* + * Wake the hardware and hold the PM reference before accessing + * registers + */ + ret =3D pm_runtime_resume_and_get(&dev->dev); + if (ret < 0) + return ret; + + ret =3D pci_read_config_word(dev, mdev->cap + PCI_LMR_PORT_STS, &sts); + pm_runtime_put_sync(&dev->dev); + + if (ret !=3D PCIBIOS_SUCCESSFUL) + return pcibios_err_to_errno(ret); + + seq_printf(s, "Port Status: %#06x\n", sts); + seq_printf(s, " Margining Ready: %s\n", + str_yes_no(sts & PCI_LMR_PORT_STS_MARGIN_READY)); + seq_printf(s, " SW Ready: %s\n", + str_yes_no(sts & PCI_LMR_PORT_STS_SW_READY)); + return 0; +} +DEFINE_SHOW_ATTRIBUTE(margin_port_status); + +static int margin_enable_show(struct seq_file *s, void *v) +{ + struct pci_margin_dev *mdev =3D s->private; + + guard(mutex)(&mdev->lock); + seq_printf(s, "%d\n", mdev->enabled); + return 0; +} + +static void pci_lmr_disable_locked(struct pci_margin_dev *mdev) +{ + struct pci_dev *dev; + int active_lanes, i, ret; + u16 sts; + + if (!mdev) + return; + + lockdep_assert_held(&mdev->lock); + + if (!mdev->enabled) + return; + + dev =3D mdev->dev; + active_lanes =3D pci_lmr_get_active_lanes(mdev); + + for (i =3D 0; i < active_lanes; i++) + pci_lmr_clear_to_normal_lane(&mdev->lanes[i]); + + for (i =3D active_lanes; i < mdev->num_lanes; i++) { + mdev->lanes[i].timing_val =3D 0; + mdev->lanes[i].voltage_val =3D 0; + } + + ret =3D pci_read_config_word(dev, mdev->cap + PCI_LMR_PORT_STS, &sts); + if (ret =3D=3D PCIBIOS_SUCCESSFUL) { + sts &=3D ~PCI_LMR_PORT_STS_SW_READY; + pci_write_config_word(dev, mdev->cap + PCI_LMR_PORT_STS, sts); + } + + if (mdev->aspm_inhibited) { + pci_aspm_inhibit_locked(dev, false); + mdev->aspm_inhibited =3D false; + } + + pci_lmr_restore_autonomous(mdev); + + if (mdev->partner) { + pm_runtime_put_sync(&mdev->partner->dev); + pci_dev_put(mdev->partner); + mdev->partner =3D NULL; + } + + pm_runtime_put_sync(&dev->dev); + mdev->enabled =3D false; +} + +static int pci_lmr_enable_locked(struct pci_margin_dev *mdev, + struct pci_dev *downstream_port, + struct pci_dev *upstream_port) +{ + struct pci_dev *dev =3D mdev->dev; + struct pci_dev *partner =3D NULL; + unsigned long timeout; + u16 sts, cap, lnksta; + int active_lanes, ret, i; + + lockdep_assert_held(&mdev->lock); + + /* + * Ensure device is powered (D0) before reading configuration + * registers + */ + ret =3D pm_runtime_resume_and_get(&dev->dev); + if (ret < 0) + return ret; + + partner =3D (dev =3D=3D downstream_port) ? upstream_port : downstream_por= t; + + /* Prevent concurrent LMR on both ends of the same link */ + if (partner && partner->lmr && partner->lmr->enabled) { + ret =3D -EBUSY; + goto err_rpm; + } + + if (partner) { + ret =3D pm_runtime_resume_and_get(&partner->dev); + if (ret < 0) + goto err_rpm; + mdev->partner =3D pci_dev_get(partner); + } + + /* + * PCIe Base Specification Revision 7.0 sec 8.4.4: LMR is physically + * undefined below 16.0 GT/s. Even if a device supports Gen4+, if the + * link is currently trained and operating at Gen1..Gen3 speeds + * (< 16.0 GT/s) in Link Status Register (sec 7.5.3.8, Current Link + * Speed), reject margining. + */ + ret =3D pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); + if (ret) { + ret =3D pcibios_err_to_errno(ret); + goto err_partner_rpm; + } + if (PCI_POSSIBLE_ERROR(lnksta) || + (lnksta & PCI_EXP_LNKSTA_CLS) < PCI_EXP_LNKSTA_CLS_16_0GB) { + ret =3D -EOPNOTSUPP; + goto err_partner_rpm; + } + + active_lanes =3D FIELD_GET(PCI_EXP_LNKSTA_NLW, lnksta); + if (active_lanes =3D=3D 0 || active_lanes > mdev->num_lanes) + active_lanes =3D mdev->num_lanes; + + ret =3D pci_read_config_word(dev, mdev->cap + PCI_LMR_PORT_CAP, &cap); + if (ret !=3D PCIBIOS_SUCCESSFUL) { + ret =3D pcibios_err_to_errno(ret); + goto err_partner_rpm; + } + + /* Disable Autonomous Width and Speed transitions */ + pci_lmr_disable_autonomous(mdev); + + /* + * Inhibit ASPM during margining via the ASPM driver API so the link + * remains continuously in L0. pci_bus_sem is held by caller. + */ + ret =3D pci_aspm_inhibit_locked(dev, true); + if (ret && ret !=3D -EPERM) + goto err_autonomous; + if (!ret) + mdev->aspm_inhibited =3D true; + + if (cap & PCI_LMR_PORT_CAP_USES_SW_READY) { + ret =3D pci_read_config_word(dev, mdev->cap + PCI_LMR_PORT_STS, &sts); + if (ret !=3D PCIBIOS_SUCCESSFUL) { + ret =3D pcibios_err_to_errno(ret); + goto err_aspm; + } + sts |=3D PCI_LMR_PORT_STS_SW_READY; + pci_write_config_word(dev, mdev->cap + PCI_LMR_PORT_STS, sts); + } + + timeout =3D jiffies + msecs_to_jiffies(LMR_ENABLE_TIMEOUT_MS); + while (1) { + ret =3D pci_read_config_word(dev, mdev->cap + PCI_LMR_PORT_STS, &sts); + if (ret !=3D PCIBIOS_SUCCESSFUL) { + ret =3D pcibios_err_to_errno(ret); + goto err_sw_ready; + } + if (PCI_POSSIBLE_ERROR(sts)) { + ret =3D -ENODEV; + goto err_sw_ready; + } + if (sts & PCI_LMR_PORT_STS_MARGIN_READY) + break; + if (time_after(jiffies, timeout)) { + ret =3D -ETIMEDOUT; + goto err_sw_ready; + } + usleep_range(LMR_ENABLE_SLEEP_MIN_US, LMR_ENABLE_SLEEP_MAX_US); + } + + /* Cache capabilities for configured receiver on all active lanes */ + for (i =3D 0; i < active_lanes; i++) { + ret =3D pci_lmr_cache_rx_info(&mdev->lanes[i], mdev->lanes[i].rx); + if (ret) + goto err_sw_ready; + } + mdev->enabled =3D true; + return 0; + +err_sw_ready: + if (cap & PCI_LMR_PORT_CAP_USES_SW_READY) { + u16 clean_sts; + int clean_ret; + + clean_ret =3D pci_read_config_word( + dev, mdev->cap + PCI_LMR_PORT_STS, &clean_sts); + if (clean_ret =3D=3D PCIBIOS_SUCCESSFUL) { + clean_sts &=3D ~PCI_LMR_PORT_STS_SW_READY; + pci_write_config_word(dev, mdev->cap + PCI_LMR_PORT_STS, + clean_sts); + } + } +err_aspm: + if (mdev->aspm_inhibited) { + pci_aspm_inhibit_locked(dev, false); + mdev->aspm_inhibited =3D false; + } +err_autonomous: + pci_lmr_restore_autonomous(mdev); +err_partner_rpm: + if (mdev->partner) { + pm_runtime_put_sync(&mdev->partner->dev); + pci_dev_put(mdev->partner); + mdev->partner =3D NULL; + } +err_rpm: + pm_runtime_put_sync(&dev->dev); + return ret; +} + +static ssize_t margin_enable_write(struct file *file, + const char __user *user_buf, size_t count, + loff_t *ppos) +{ + struct seq_file *s =3D file->private_data; + struct pci_margin_dev *mdev =3D s->private; + struct pci_dev *dev =3D mdev->dev; + struct pci_dev *downstream_port =3D NULL, *upstream_port =3D NULL; + bool enable; + int ret; + + ret =3D kstrtobool_from_user(user_buf, count, &enable); + if (ret) + return ret; + + if (enable) { + ret =3D pcie_get_link_endpoints(dev, &downstream_port, &upstream_port); + if (ret) + return ret; + } else { + struct pci_dev *partner; + + /* + * When disabling LMR, always operate on the link partner that + * was saved when LMR was enabled (mdev->partner). If a hot-swap + * occurred while LMR was active, pcie_get_link_endpoints() + * would resolve to the newly connected device, causing + * pci_lmr_disable_locked() to restore registers on the old + * partner without holding its device_lock. + */ + mutex_lock(&mdev->lock); + if (!mdev->enabled) { + mutex_unlock(&mdev->lock); + return count; + } + partner =3D mdev->partner; + if (partner) + pci_dev_get(partner); + mutex_unlock(&mdev->lock); + + if (pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_ROOT_PORT || + pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_DOWNSTREAM) { + downstream_port =3D pci_dev_get(dev); + upstream_port =3D partner; + } else { + downstream_port =3D partner; + upstream_port =3D pci_dev_get(dev); + } + } + + /* + * Canonical PCI locking hierarchy: + * pci_bus_sem -> Downstream Port (parent) -> Upstream Port (child) -> = mdev->lock. + * Holding pci_bus_sem allows pci_aspm_inhibit_locked() to safely execute + * without lock inversion deadlocks. + */ + down_read(&pci_bus_sem); + if (downstream_port) + pci_dev_lock(downstream_port); + if (upstream_port && upstream_port !=3D downstream_port) + pci_dev_lock(upstream_port); + + mutex_lock(&mdev->lock); + + if (mdev->enabled =3D=3D enable) { + ret =3D count; + } else if (!enable) { + pci_lmr_disable_locked(mdev); + ret =3D count; + } else { + ret =3D pci_lmr_enable_locked(mdev, downstream_port, upstream_port); + if (!ret) + ret =3D count; + } + + mutex_unlock(&mdev->lock); + + if (upstream_port && upstream_port !=3D downstream_port) + pci_dev_unlock(upstream_port); + if (downstream_port) + pci_dev_unlock(downstream_port); + up_read(&pci_bus_sem); + + pcie_put_link_endpoints(downstream_port, upstream_port); + + return ret; +} + +static int margin_enable_open(struct inode *inode, struct file *file) +{ + return single_open(file, margin_enable_show, inode->i_private); +} + +static const struct file_operations margin_enable_fops =3D { + .open =3D margin_enable_open, + .read =3D seq_read, + .write =3D margin_enable_write, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static int margin_lane_receiver_show(struct seq_file *s, void *v) +{ + struct pci_margin_lane *plane =3D s->private; + + guard(mutex)(&plane->mdev->lock); + seq_printf(s, "%d\n", plane->rx); + return 0; +} + +static ssize_t margin_lane_receiver_write(struct file *file, const char __= user *user_buf, + size_t count, loff_t *ppos) +{ + struct seq_file *s =3D file->private_data; + struct pci_margin_lane *plane =3D s->private; + struct pci_margin_dev *mdev =3D plane->mdev; + int ret; + u8 rx; + + ret =3D kstrtou8_from_user(user_buf, count, 0, &rx); + if (ret) + return ret; + + /* + * Valid receiver numbers are 0..6 per PCIe Base Specification + * Revision 7.0 sec 4.2.18.1 & Table 4-76; 7 is reserved. + */ + if (rx > LMR_MAX_RX_NUM) + return -EINVAL; + + guard(mutex)(&mdev->lock); + if (plane->rx =3D=3D rx) + return count; + + if (mdev->enabled) { + /* + * Clear previous receiver to normal settings per + * single-receiver rule + */ + ret =3D pci_lmr_clear_to_normal_lane(plane); + if (ret) + return ret; + ret =3D pci_lmr_cache_rx_info(plane, rx); + if (ret) + return ret; + } + + plane->rx =3D rx; + return count; +} + +static int margin_lane_receiver_open(struct inode *inode, struct file *fil= e) +{ + return single_open(file, margin_lane_receiver_show, inode->i_private); +} + +static const struct file_operations margin_lane_receiver_fops =3D { + .open =3D margin_lane_receiver_open, + .read =3D seq_read, + .write =3D margin_lane_receiver_write, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static int margin_lane_caps_show(struct seq_file *s, void *v) +{ + struct pci_margin_lane *plane =3D s->private; + struct pci_margin_dev *mdev =3D plane->mdev; + struct pci_margin_rx_info *info; + int ret; + u8 val; + + guard(mutex)(&mdev->lock); + if (!mdev->enabled) + return -EBUSY; + + ret =3D pci_lmr_cache_rx_info(plane, plane->rx); + if (ret) + return ret; + + info =3D &plane->rx_info[plane->rx]; + val =3D info->caps; + seq_printf(s, "Lane %d Rx %d Capabilities: %#02x\n", plane->lane, plane->= rx, val); + seq_printf(s, " Voltage Supported: %s\n", + str_yes_no(val & LMR_CAP_VOLTAGE_SUPPORTED)); + seq_printf(s, " Left/Right: %s\n", + (val & LMR_CAP_IND_LEFT_RIGHT_TIMING) ? "independent" : "symmetric"); + seq_printf(s, " Up/Down: %s\n", + (val & LMR_CAP_IND_UP_DOWN_VOLTAGE) ? "independent" : "symmetric"); + seq_printf(s, " Error Sampler: %s\n", + (val & LMR_CAP_IND_ERROR_SAMPLER) ? "independent" : + "main sampler"); + seq_printf(s, " Sample Reporting: %s\n", + (val & LMR_CAP_SAMPLE_REPORT_METHOD) ? "rate" : "count"); + seq_printf(s, " Sample Multiple Receivers: %s\n", + str_yes_no(val & LMR_CAP_SAMPLE_MULTIPLE_RECEIVERS)); + return 0; +} +DEFINE_SHOW_ATTRIBUTE(margin_lane_caps); + +static int margin_lane_steps_show(struct seq_file *s, u8 type) +{ + struct pci_margin_lane *plane =3D s->private; + struct pci_margin_dev *mdev =3D plane->mdev; + struct pci_margin_rx_info *info; + int ret; + + guard(mutex)(&mdev->lock); + if (!mdev->enabled) + return -EBUSY; + + ret =3D pci_lmr_cache_rx_info(plane, plane->rx); + if (ret) + return ret; + + info =3D &plane->rx_info[plane->rx]; + seq_printf(s, "%d\n", (type =3D=3D LMR_TYPE_VOLTAGE) ? + info->num_voltage_steps : info->num_timing_steps); + return 0; +} + +static int margin_lane_timing_steps_show(struct seq_file *s, void *v) +{ + return margin_lane_steps_show(s, LMR_TYPE_TIMING); +} +DEFINE_SHOW_ATTRIBUTE(margin_lane_timing_steps); + +static int margin_lane_voltage_steps_show(struct seq_file *s, void *v) +{ + return margin_lane_steps_show(s, LMR_TYPE_VOLTAGE); +} +DEFINE_SHOW_ATTRIBUTE(margin_lane_voltage_steps); + +/* + * pci_lmr_check_sample_multiple_rx() - Check multi-receiver concurrency. + * Per PCIe Base Specification Revision 7.0 sec 4.2.18.2 & sec 8.4.4: + * "For Receivers where MIndErrorSampler is 0b, at most one such Receiver = is + * permitted to be margined at a time. However, margining may be performed= on + * multiple Lanes simultaneously, as long as it is within the maximum numb= er of + * Lanes the device supports." + * + * If the target receiver uses an independent error sampler (MIndErrorSamp= ler + * =3D=3D 1b), margining will not produce errors in the live data stream, = and + * multiple receivers may be margined concurrently. If MIndErrorSampler is= 0b + * (main data sampler), software must ensure that no other receiver on any= lane + * is currently margined. + */ +static bool pci_lmr_check_sample_multiple_rx(struct pci_margin_dev *mdev, + struct pci_margin_lane *plane) +{ + struct pci_margin_rx_info *info =3D &plane->rx_info[plane->rx]; + int i; + + /* + * If receiver has an independent error sampler or supports sampling + * multiple receivers concurrently, concurrent margining is permitted. + */ + if ((info->caps & LMR_CAP_IND_ERROR_SAMPLER) || + (info->caps & LMR_CAP_SAMPLE_MULTIPLE_RECEIVERS)) + return true; + + for (i =3D 0; i < mdev->num_lanes; i++) { + struct pci_margin_lane *other =3D &mdev->lanes[i]; + struct pci_margin_rx_info *other_info; + + if (i =3D=3D plane->lane) + continue; + + other_info =3D &other->rx_info[other->rx]; + /* + * For receivers using the main data sampler without multiple + * receiver sampling capability, reject only if another lane is + * actively margining a DIFFERENT receiver that ALSO does not + * support concurrent multi-receiver sampling. + */ + if (other->rx !=3D plane->rx && + !(other_info->caps & (LMR_CAP_IND_ERROR_SAMPLER | + LMR_CAP_SAMPLE_MULTIPLE_RECEIVERS)) && + (other->timing_val !=3D 0 || other->voltage_val !=3D 0)) + return false; + } + return true; +} + +static ssize_t margin_lane_step_write(struct file *file, const char __user= *user_buf, + size_t count, u8 type) +{ + struct seq_file *s =3D file->private_data; + struct pci_margin_lane *plane =3D s->private; + struct pci_margin_dev *mdev =3D plane->mdev; + struct pci_margin_rx_info *info; + u8 step, dir, payload; + int max_step, val, ret; + u16 sts; + u8 caps; + + ret =3D kstrtoint_from_user(user_buf, count, 0, &val); + if (ret) + return ret; + + guard(mutex)(&mdev->lock); + if (!mdev->enabled) + return -EBUSY; + + /* + * Reject step operations if the target lane exceeds the currently + * negotiated/active link width (e.g. down-trained link). + */ + ret =3D pcie_capability_read_word(mdev->dev, PCI_EXP_LNKSTA, &sts); + if (ret =3D=3D PCIBIOS_SUCCESSFUL) { + u16 nlw =3D FIELD_GET(PCI_EXP_LNKSTA_NLW, sts); + + if (nlw > 0 && plane->lane >=3D nlw) + return -ENODEV; + } + + if (val =3D=3D 0) { + /* + * Step this specific axis to 0 without resetting the orthogonal + * axis + */ + if (type =3D=3D LMR_TYPE_TIMING) { + if (plane->voltage_val =3D=3D 0) { + ret =3D pci_lmr_clear_to_normal_lane(plane); + } else { + ret =3D pci_lmr_run_cmd(mdev, plane->lane, plane->rx, + LMR_TYPE_TIMING, 0, 0, &sts); + if (!ret) + plane->timing_val =3D 0; + } + } else { + if (plane->timing_val =3D=3D 0) { + ret =3D pci_lmr_clear_to_normal_lane(plane); + } else { + ret =3D pci_lmr_run_cmd(mdev, plane->lane, plane->rx, + LMR_TYPE_VOLTAGE, 0, 0, &sts); + if (!ret) + plane->voltage_val =3D 0; + } + } + return ret ? ret : count; + } + + ret =3D pci_lmr_cache_rx_info(plane, plane->rx); + if (ret) + return ret; + + if (!pci_lmr_check_sample_multiple_rx(mdev, plane)) + return -EBUSY; + + info =3D &plane->rx_info[plane->rx]; + caps =3D info->caps; + + switch (type) { + case LMR_TYPE_TIMING: + if (val < -LMR_MAX_TIMING_STEP || val > LMR_MAX_TIMING_STEP) + return -EINVAL; + if (val < 0) { + if (!(caps & LMR_CAP_IND_LEFT_RIGHT_TIMING)) + return -EINVAL; + step =3D -val; + dir =3D LMR_STEP_DIR_LEFT_OR_DOWN; + } else { + step =3D val; + dir =3D LMR_STEP_DIR_RIGHT_OR_UP; + } + max_step =3D info->num_timing_steps; + if (step > max_step) + return -EINVAL; + + payload =3D FIELD_PREP(LMR_TIMING_DIR_MASK, dir) | + FIELD_PREP(LMR_TIMING_STEP_MASK, step); + ret =3D pci_lmr_run_cmd(mdev, plane->lane, plane->rx, + LMR_TYPE_TIMING, 0, payload, &sts); + if (ret) + return ret; + if (FIELD_GET(LMR_STS_EXEC_MASK, pci_lmr_sts_payload(sts)) =3D=3D + LMR_STS_EXEC_NAK) + return -EOPNOTSUPP; + if (FIELD_GET(LMR_STS_EXEC_MASK, pci_lmr_sts_payload(sts)) =3D=3D + LMR_STS_EXEC_TOO_MANY_ERR) { + plane->timing_val =3D 0; + plane->voltage_val =3D 0; + return -EIO; + } + plane->timing_val =3D val; + break; + + case LMR_TYPE_VOLTAGE: + if (!(caps & LMR_CAP_VOLTAGE_SUPPORTED)) + return -EOPNOTSUPP; + if (val < -LMR_MAX_VOLTAGE_STEP || val > LMR_MAX_VOLTAGE_STEP) + return -EINVAL; + if (val < 0) { + if (!(caps & LMR_CAP_IND_UP_DOWN_VOLTAGE)) + return -EINVAL; + step =3D -val; + dir =3D LMR_STEP_DIR_LEFT_OR_DOWN; + } else { + step =3D val; + dir =3D LMR_STEP_DIR_RIGHT_OR_UP; + } + max_step =3D info->num_voltage_steps; + if (step > max_step) + return -EINVAL; + + payload =3D FIELD_PREP(LMR_VOLTAGE_DIR_MASK, dir) | + FIELD_PREP(LMR_VOLTAGE_STEP_MASK, step); + ret =3D pci_lmr_run_cmd(mdev, plane->lane, plane->rx, + LMR_TYPE_VOLTAGE, 0, payload, &sts); + if (ret) + return ret; + if (FIELD_GET(LMR_STS_EXEC_MASK, pci_lmr_sts_payload(sts)) =3D=3D + LMR_STS_EXEC_NAK) + return -EOPNOTSUPP; + if (FIELD_GET(LMR_STS_EXEC_MASK, pci_lmr_sts_payload(sts)) =3D=3D + LMR_STS_EXEC_TOO_MANY_ERR) { + plane->timing_val =3D 0; + plane->voltage_val =3D 0; + return -EIO; + } + plane->voltage_val =3D val; + break; + + default: + return -EINVAL; + } + + return count; +} + +static ssize_t margin_lane_timing_write(struct file *file, const char __us= er *user_buf, + size_t count, loff_t *ppos) +{ + return margin_lane_step_write(file, user_buf, count, LMR_TYPE_TIMING); +} + +static int margin_lane_step_show(struct seq_file *s, u8 type) +{ + struct pci_margin_lane *plane =3D s->private; + + guard(mutex)(&plane->mdev->lock); + seq_printf(s, "%d\n", (type =3D=3D LMR_TYPE_VOLTAGE) ? + plane->voltage_val : plane->timing_val); + return 0; +} + +static int margin_lane_timing_show(struct seq_file *s, void *v) +{ + return margin_lane_step_show(s, LMR_TYPE_TIMING); +} + +static int margin_lane_timing_open(struct inode *inode, struct file *file) +{ + return single_open(file, margin_lane_timing_show, inode->i_private); +} + +static const struct file_operations margin_lane_timing_fops =3D { + .open =3D margin_lane_timing_open, + .read =3D seq_read, + .write =3D margin_lane_timing_write, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static ssize_t margin_lane_voltage_write(struct file *file, const char __u= ser *user_buf, + size_t count, loff_t *ppos) +{ + return margin_lane_step_write(file, user_buf, count, LMR_TYPE_VOLTAGE); +} + +static int margin_lane_voltage_show(struct seq_file *s, void *v) +{ + return margin_lane_step_show(s, LMR_TYPE_VOLTAGE); +} + +static int margin_lane_voltage_open(struct inode *inode, struct file *file) +{ + return single_open(file, margin_lane_voltage_show, inode->i_private); +} + +static const struct file_operations margin_lane_voltage_fops =3D { + .open =3D margin_lane_voltage_open, + .read =3D seq_read, + .write =3D margin_lane_voltage_write, + .llseek =3D seq_lseek, + .release =3D single_release, +}; + +static void pci_margin_debugfs_init(struct pci_margin_dev *mdev) +{ + struct pci_dev *dev =3D mdev->dev; + struct dentry *parent; + char dirname[64]; + int i; + + parent =3D get_pci_debugfs_root(); + scnprintf(dirname, sizeof(dirname), "pcie_lmr_%s", dev_name(&dev->dev)); + mdev->debugfs =3D debugfs_create_dir(dirname, parent); + + debugfs_create_file("capabilities", 0444, mdev->debugfs, mdev, &margin_ca= ps_fops); + debugfs_create_file("port_status", 0444, mdev->debugfs, mdev, &margin_por= t_status_fops); + debugfs_create_file("enable", 0644, mdev->debugfs, mdev, &margin_enable_f= ops); + + for (i =3D 0; i < mdev->num_lanes; i++) { + struct pci_margin_lane *plane =3D &mdev->lanes[i]; + struct dentry *lane_dir; + char lane_name[16]; + + scnprintf(lane_name, sizeof(lane_name), "lane%d", i); + lane_dir =3D debugfs_create_dir(lane_name, mdev->debugfs); + + debugfs_create_file("receiver", 0644, lane_dir, plane, &margin_lane_rece= iver_fops); + debugfs_create_file("caps", 0444, lane_dir, plane, &margin_lane_caps_fop= s); + debugfs_create_file("num_timing_steps", 0444, lane_dir, plane, + &margin_lane_timing_steps_fops); + debugfs_create_file("num_voltage_steps", 0444, lane_dir, plane, + &margin_lane_voltage_steps_fops); + debugfs_create_file("margin_timing", 0644, lane_dir, plane, + &margin_lane_timing_fops); + debugfs_create_file("margin_voltage", 0644, lane_dir, plane, + &margin_lane_voltage_fops); + } +} + +static void pci_margin_debugfs_remove(struct pci_margin_dev *mdev) +{ + debugfs_remove_recursive(mdev->debugfs); +} + +#else +static inline void pci_margin_debugfs_init(struct pci_margin_dev *mdev) { } +static inline void pci_margin_debugfs_remove(struct pci_margin_dev *mdev) = { } +#endif + +void pci_lmr_init(struct pci_dev *dev) +{ + struct pci_margin_dev *mdev; + enum pci_bus_speed speed; + u32 lnkcap; + u16 lmr; + int num_lanes, ret, i; + + if (WARN_ON_ONCE(!dev) || !pci_is_pcie(dev)) + return; + + /* + * Per PCIe Base Specification Revision 7.0 sec 7.7.11: + * For devices associated with an Upstream Port (Endpoints, + * Legacy Endpoints, and Switch Upstream Ports), the Lane Margining + * Extended Capability must be implemented in Function 0 (and only + * Function 0). + */ + if ((pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_ENDPOINT || + pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_LEG_END || + pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_UPSTREAM) && + PCI_FUNC(dev->devfn) !=3D 0) + return; + + speed =3D pcie_get_speed_cap(dev); + if (speed < PCIE_SPEED_16_0GT || speed =3D=3D PCI_SPEED_UNKNOWN) + return; + + lmr =3D pci_find_ext_capability(dev, PCI_EXT_CAP_ID_LMR); + if (!lmr) { + if (speed >=3D PCIE_SPEED_64_0GT) + pci_warn(dev, + "Missing Lane Margining at Receiver Capability (mandatory for Gen6+)\= n"); + else + pci_dbg(dev, + "Optional Lane Margining at Receiver Capability not found\n"); + return; + } + + /* + * Determine link width: read Maximum Link Width (MLW) from Link + * Capabilities. Sizing data structures and debugfs interfaces to MLW + * ensures all lanes can be margined if the link up-trains dynamically. + * Dynamic Negotiated Link Width (NLW) is queried at runtime via + * pci_lmr_get_active_lanes(). + */ + ret =3D pcie_capability_read_dword(dev, PCI_EXP_LNKCAP, &lnkcap); + if (ret !=3D PCIBIOS_SUCCESSFUL) + return; + num_lanes =3D FIELD_GET(PCI_EXP_LNKCAP_MLW, lnkcap); + if (num_lanes =3D=3D 0 || num_lanes > LMR_MAX_LANES) { + pci_warn(dev, "Invalid link width %d for LMR\n", num_lanes); + return; + } + + dev->lmr_cap =3D lmr; + + mdev =3D kzalloc(struct_size(mdev, lanes, num_lanes), GFP_KERNEL); + if (!mdev) + return; + + mdev->num_lanes =3D num_lanes; + mdev->dev =3D dev; + mdev->cap =3D lmr; + mutex_init(&mdev->lock); + + for (i =3D 0; i < num_lanes; i++) { + mdev->lanes[i].mdev =3D mdev; + mdev->lanes[i].lane =3D i; + mdev->lanes[i].rx =3D LMR_RX_LOCAL; + } + + dev->lmr =3D mdev; + pci_margin_debugfs_init(mdev); + + pci_dbg(dev, "Lane Margining at Receiver (Gen%u) Capability detected\n", + LMR_SPEED_TO_GEN(speed)); +} + +void pci_lmr_exit(struct pci_dev *dev) +{ + struct pci_margin_dev *mdev; + struct pci_dev *partner; + struct pci_dev *downstream_port, *upstream_port; + + if (!dev || !dev->lmr) + return; + + mdev =3D dev->lmr; + + /* + * 1. Tear down user-facing debugfs files FIRST to prevent concurrent + * access. debugfs_remove_recursive() flushes active file operations. + */ + pci_margin_debugfs_remove(mdev); + + /* + * 2. Acquire locks in canonical PCI hierarchy: + * pci_bus_sem -> Downstream Port (parent) -> Upstream Port (child) ->= mdev->lock. + * + * Both ends of the link must hold their device_lock during teardown + * because pci_lmr_disable_locked() restores autonomous width and speed + * registers on both ports. + */ + mutex_lock(&mdev->lock); + partner =3D mdev->partner; + if (partner) + pci_dev_get(partner); + mutex_unlock(&mdev->lock); + + if (pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_ROOT_PORT || + pci_pcie_type(dev) =3D=3D PCI_EXP_TYPE_DOWNSTREAM) { + downstream_port =3D dev; + upstream_port =3D partner; + } else { + downstream_port =3D partner; + upstream_port =3D dev; + } + + down_read(&pci_bus_sem); + if (downstream_port) + pci_dev_lock(downstream_port); + if (upstream_port && upstream_port !=3D downstream_port) + pci_dev_lock(upstream_port); + + mutex_lock(&mdev->lock); + dev->lmr =3D NULL; + if (mdev->enabled) + pci_lmr_disable_locked(mdev); + mutex_unlock(&mdev->lock); + + if (upstream_port && upstream_port !=3D downstream_port) + pci_dev_unlock(upstream_port); + if (downstream_port) + pci_dev_unlock(downstream_port); + up_read(&pci_bus_sem); + + pci_dev_put(partner); + + /* 3. Safe to destroy structures */ + mutex_destroy(&mdev->lock); + kfree(mdev); +} diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 27008e2ea5af..98c84fdb0a97 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2666,6 +2666,7 @@ static void pci_init_capabilities(struct pci_dev *dev) pci_pasid_init(dev); /* Process Address Space ID */ pci_acs_init(dev); /* Access Control Services */ pci_ptm_init(dev); /* Precision Time Measurement */ + pci_lmr_init(dev); /* Lane Margining at Receiver */ pci_aer_init(dev); /* Advanced Error Reporting */ pci_dpc_init(dev); /* Downstream Port Containment */ pci_rcec_init(dev); /* Root Complex Event Collector */ diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c index e711ac1d4e38..f78f0c654011 100644 --- a/drivers/pci/remove.c +++ b/drivers/pci/remove.c @@ -37,12 +37,12 @@ static void pci_destroy_dev(struct pci_dev *dev) platform_pci_remove_wake(dev); pci_doe_sysfs_teardown(dev); pci_npem_remove(dev); - /* * While device is in D0 drop the device from TSM link operations * including unbind and disconnect (IDE + SPDM teardown). */ pci_tsm_destroy(dev); + pci_lmr_exit(dev); =20 device_del(&dev->dev); =20 diff --git a/include/linux/pci.h b/include/linux/pci.h index b0ce8ee8d622..fbb85ad124b5 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -353,6 +353,8 @@ struct rcec_ea; * number resources to allow for hierarchy expansion. * @is_pciehp: PCIe Hot-Plug Capable bridge. */ +struct pci_margin_dev; + struct pci_dev { struct list_head bus_list; /* Node in per-bus list */ struct pci_bus *bus; /* Bus this device is on */ @@ -532,6 +534,10 @@ struct pci_dev { atomic_t ptm_enable_cnt; u8 ptm_granularity; #endif +#ifdef CONFIG_PCIE_LMR + u16 lmr_cap; /* Lane Margining Capability */ + struct pci_margin_dev *lmr; +#endif #ifdef CONFIG_PCI_MSI void __iomem *msix_base; raw_spinlock_t msi_lock; diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h index facaa324bd86..90cbe310e62f 100644 --- a/include/uapi/linux/pci_regs.h +++ b/include/uapi/linux/pci_regs.h @@ -757,6 +757,7 @@ #define PCI_EXT_CAP_ID_VF_REBAR 0x24 /* VF Resizable BAR */ #define PCI_EXT_CAP_ID_DLF 0x25 /* Data Link Feature */ #define PCI_EXT_CAP_ID_PL_16GT 0x26 /* Physical Layer 16.0 GT/s */ +#define PCI_EXT_CAP_ID_LMR 0x27 /* Lane Margining at Receiver */ #define PCI_EXT_CAP_ID_NPEM 0x29 /* Native PCIe Enclosure Management */ #define PCI_EXT_CAP_ID_PL_32GT 0x2A /* Physical Layer 32.0 GT/s */ #define PCI_EXT_CAP_ID_DOE 0x2E /* Data Object Exchange */ @@ -1181,6 +1182,23 @@ #define PCI_PL_16GT_LE_CTRL_USP_TX_PRESET_MASK 0x000000F0 #define PCI_PL_16GT_LE_CTRL_USP_TX_PRESET_SHIFT 4 =20 +/* Lane Margining at Receiver */ +#define PCI_LMR_PORT_CAP 0x04 /* Margining Port Capabilities */ +#define PCI_LMR_PORT_CAP_USES_SW_READY 0x0001 /* Margining Uses Software = Ready */ +#define PCI_LMR_PORT_STS 0x06 /* Margining Port Status */ +#define PCI_LMR_PORT_STS_MARGIN_READY 0x0001 /* Margining Ready */ +#define PCI_LMR_PORT_STS_SW_READY 0x0002 /* Margining SW Ready */ +#define PCI_LMR_LANE_CTRL 0x08 /* Margining Lane Control */ +#define PCI_LMR_LANE_CTRL_RX_NUM 0x0007 /* Receiver Number */ +#define PCI_LMR_LANE_CTRL_MTYPE 0x0038 /* Margining Type */ +#define PCI_LMR_LANE_CTRL_USAGE 0x0040 /* Margining Usage Model */ +#define PCI_LMR_LANE_CTRL_PAYLOAD 0xFF00 /* Margining Payload */ +#define PCI_LMR_LANE_STS 0x0A /* Margining Lane Status */ +#define PCI_LMR_LANE_STS_RX_NUM 0x0007 /* Receiver Number */ +#define PCI_LMR_LANE_STS_MTYPE 0x0038 /* Margining Type */ +#define PCI_LMR_LANE_STS_USAGE 0x0040 /* Margining Usage Model */ +#define PCI_LMR_LANE_STS_PAYLOAD 0xFF00 /* Margining Payload */ + /* Physical Layer 32.0 GT/s */ #define PCI_PL_32GT_LE_CTRL 0x20 /* Lane Equalization Control Register */ =20 diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Mak= efile index 2d960626750e..f762540b500e 100644 --- a/tools/testing/selftests/Makefile +++ b/tools/testing/selftests/Makefile @@ -93,6 +93,7 @@ TARGETS +=3D net/tcp_ao TARGETS +=3D nolibc TARGETS +=3D pci_endpoint TARGETS +=3D pcie_bwctrl +TARGETS +=3D pcie_lmt TARGETS +=3D perf_events TARGETS +=3D pidfd TARGETS +=3D pid_namespace diff --git a/tools/testing/selftests/pcie_lmt/Makefile b/tools/testing/self= tests/pcie_lmt/Makefile new file mode 100644 index 000000000000..36ac85937d78 --- /dev/null +++ b/tools/testing/selftests/pcie_lmt/Makefile @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 +TEST_PROGS =3D pcie_lmt.sh +include ../lib.mk diff --git a/tools/testing/selftests/pcie_lmt/pcie_lmt.sh b/tools/testing/s= elftests/pcie_lmt/pcie_lmt.sh new file mode 100755 index 000000000000..9ea292f0009d --- /dev/null +++ b/tools/testing/selftests/pcie_lmt/pcie_lmt.sh @@ -0,0 +1,287 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (C) 2026 Google LLC +# Author: Priyank Rathod +# +# Kselftest for PCIe Lane Margining at Receiver (LMR / LMT) +# Tests the debugfs interface exposed by drivers/pci/pcie/margin.c +# (/sys/kernel/debug/pci/pcie_lmr_/) + +TESTNAME=3D"pcie_lmt" + +# Kselftest framework requirement - SKIP code is 4. +ksft_skip=3D4 +retval=3D0 +skipmsg=3D"skip all tests:" + +# Rejection test: must be run as root +if [ "$UID" -ne 0 ]; then + echo "$skipmsg must be run as root" >&2 + exit $ksft_skip +fi + +SCRIPT_PATH=3D$(realpath "$0" 2>/dev/null || echo "$0") + +# Test non-root user execution rejection via subshell if unprivileged user= exists +test_non_root_rejection() +{ + local non_root_cmd=3D"" + local exit_code=3D0 + + if command -v runuser >/dev/null 2>&1 && id nobody >/dev/null 2>&1; then + non_root_cmd=3D"runuser -u nobody --" + elif command -v su >/dev/null 2>&1 && id nobody >/dev/null 2>&1; then + non_root_cmd=3D"su -s /bin/bash nobody -c" + fi + + if [ -n "$non_root_cmd" ]; then + $non_root_cmd "$SCRIPT_PATH" >/dev/null 2>&1 || exit_code=3D$? + if [ "$exit_code" -eq "$ksft_skip" ]; then + echo "$TESTNAME: non-root user execution rejection test [PASS]" + else + echo "$TESTNAME: non-root run got $exit_code (exp $ksft_skip) [FAIL]" >= &2 + retval=3D1 + fi + else + echo "$TESTNAME: skipping non-root subshell test (no unprivileged user/s= u)" + fi +} + +test_non_root_rejection + +DEBUGFS=3D$(mount -t debugfs | head -1 | awk '{ print $3 }') +if [ -z "$DEBUGFS" ]; then + if [ -d "/sys/kernel/debug" ]; then + DEBUGFS=3D"/sys/kernel/debug" + else + echo "$skipmsg debugfs is not mounted" >&2 + exit $ksft_skip + fi +fi + +LMR_DEVS=3D$(ls -d $DEBUGFS/pci/pcie_lmr_* $DEBUGFS/pcie_lmr_* 2>/dev/null= || true) +if [ -z "$LMR_DEVS" ]; then + echo "$skipmsg no PCIe LMR devices found in $DEBUGFS/" >&2 + exit $ksft_skip +fi + +cleanup_dev() +{ + local dev=3D"$1" + echo 0 > "$dev/enable" 2>/dev/null || true +} + +assert_write_fail() +{ + local file=3D"$1" + local val=3D"$2" + local desc=3D"$3" + local fname + + fname=3D$(basename "$file") + if (echo "$val" > "$file") 2>/dev/null; then + echo " FAIL: $desc ('$val' -> $fname succeeded, expected fail)" >&2 + retval=3D1 + else + echo " PASS: $desc rejected correctly" + fi +} + +assert_write_success() +{ + local file=3D"$1" + local val=3D"$2" + local desc=3D"$3" + local fname + + fname=3D$(basename "$file") + if ! (echo "$val" > "$file") 2>/dev/null; then + echo " FAIL: $desc ('$val' -> $fname failed, expected success)" >&2 + retval=3D1 + else + echo " PASS: $desc succeeded correctly" + fi +} + +assert_read_fail() +{ + local file=3D"$1" + local desc=3D"$2" + + if cat "$file" >/dev/null 2>&1; then + echo " FAIL: $desc (reading $(basename "$file") succeeded, expected f= ailure)" >&2 + retval=3D1 + else + echo " PASS: $desc rejected correctly" + fi +} + +assert_read_success() +{ + local file=3D"$1" + local desc=3D"$2" + + if ! cat "$file" >/dev/null 2>&1; then + echo " FAIL: $desc (reading $(basename "$file") failed, expected succ= ess)" >&2 + retval=3D1 + else + echo " PASS: $desc succeeded correctly" + fi +} + +echo "$TESTNAME: testing PCIe LMR debugfs entries" + +for dev in $LMR_DEVS; do + dev_name=3D$(basename "$dev") + echo "$TESTNAME: probing device $dev_name" + + if [ ! -r "$dev/capabilities" ] || [ ! -r "$dev/port_status" ] || + [ ! -r "$dev/enable" ] || [ ! -w "$dev/enable" ]; then + echo "$TESTNAME: $dev_name missing mandatory root attributes" >&2 + retval=3D1 + continue + fi + + assert_read_success "$dev/capabilities" "$dev_name: capabilities read" + assert_read_success "$dev/port_status" "$dev_name: port_status read" + + # Negative test: write to read-only root files + echo " $dev_name: testing read-only root attributes" + assert_write_fail "$dev/capabilities" "0" "write to read-only capabilitie= s" + assert_write_fail "$dev/port_status" "0" "write to read-only port_status" + + # Negative test: operations while margining is disabled + echo " $dev_name: testing operations while disabled" + for lane_dir in $(ls -d "$dev"/lane* 2>/dev/null || true); do + lane=3D$(basename "$lane_dir") + assert_write_fail "$lane_dir/margin_timing" "1" \ + "$lane: timing step while disabled" + assert_write_fail "$lane_dir/margin_voltage" "1" \ + "$lane: voltage step while disabled" + assert_read_fail "$lane_dir/caps" \ + "$lane: read caps while disabled" + assert_read_fail "$lane_dir/num_timing_steps" \ + "$lane: read timing steps while disabled" + assert_read_fail "$lane_dir/num_voltage_steps" \ + "$lane: read voltage steps while disabled" + break + done + + # Negative test: invalid enable inputs + echo " $dev_name: testing invalid enable inputs" + assert_write_fail "$dev/enable" "invalid" "enable invalid string" + assert_write_fail "$dev/enable" "2" "enable invalid numeric '2'" + assert_write_fail "$dev/enable" "-1" "enable negative numeric '-1'" + assert_write_fail "$dev/enable" "999" "enable out-of-bounds '999'" + assert_write_fail "$dev/enable" "" "enable empty string" + + trap 'cleanup_dev "$dev"' EXIT INT TERM + + if ! echo 1 > "$dev/enable" 2>/dev/null; then + echo " $dev_name: margining not ready by hardware (skipping active lane= s)" + trap - EXIT INT TERM + continue + fi + + echo " $dev_name: margining enabled OK" + + for lane_dir in $(ls -d "$dev"/lane* 2>/dev/null || true); do + lane=3D$(basename "$lane_dir") + echo " $dev_name: testing $lane (active)" + + # Negative test: write to read-only lane attributes + assert_write_fail "$lane_dir/caps" "0" \ + "$lane: write to read-only caps" + assert_write_fail "$lane_dir/num_timing_steps" "0" \ + "$lane: write to read-only num_timing_steps" + assert_write_fail "$lane_dir/num_voltage_steps" "0" \ + "$lane: write to read-only num_voltage_steps" + + # Negative test: invalid receiver numbers (valid: 0..6, 7 reserved) + assert_write_fail "$lane_dir/receiver" "7" "$lane: receiver 7 (reserved)" + assert_write_fail "$lane_dir/receiver" "8" "$lane: receiver 8 (> 6)" + assert_write_fail "$lane_dir/receiver" "255" "$lane: receiver 255" + assert_write_fail "$lane_dir/receiver" "-1" "$lane: negative receiver -1" + assert_write_fail "$lane_dir/receiver" "invalid" "$lane: non-numeric rec= eiver" + + # Set valid receiver 0 (local receiver) + assert_write_success "$lane_dir/receiver" "0" "$lane: set receiver 0 (lo= cal)" + + # Read capabilities and step limits + assert_read_success "$lane_dir/caps" "$lane: read caps" + assert_read_success "$lane_dir/num_timing_steps" "$lane: read num_timing= _steps" + assert_read_success "$lane_dir/num_voltage_steps" "$lane: read num_volta= ge_steps" + + caps_raw=3D$(cat "$lane_dir/caps" 2>/dev/null || true) + num_timing=3D$(cat "$lane_dir/num_timing_steps" 2>/dev/null || echo 0) + num_voltage=3D$(cat "$lane_dir/num_voltage_steps" 2>/dev/null || echo 0) + + # Negative test: out-of-bounds timing steps (spec limit: 0..63) + assert_write_fail "$lane_dir/margin_timing" "9999" \ + "$lane: timing step 9999 (out of range)" + assert_write_fail "$lane_dir/margin_timing" "-9999" \ + "$lane: timing step -9999 (out of range)" + assert_write_fail "$lane_dir/margin_timing" "64" \ + "$lane: timing step 64 (> spec max 63)" + assert_write_fail "$lane_dir/margin_timing" "-64" \ + "$lane: timing step -64 (< spec min -63)" + assert_write_fail "$lane_dir/margin_timing" "invalid" \ + "$lane: non-numeric timing step" + + if [ -n "$num_timing" ] && [ "$num_timing" -ge 0 ] 2>/dev/null; then + assert_write_fail "$lane_dir/margin_timing" "$((num_timing + 1))" \ + "$lane: timing step > receiver limit ($num_timing)" + fi + + # Negative test: negative timing on symmetric receiver + if echo "$caps_raw" | grep -q "Left/Right: symmetric"; then + assert_write_fail "$lane_dir/margin_timing" "-1" \ + "$lane: negative timing on symmetric receiver" + fi + + # Negative test: out-of-bounds voltage steps (spec limit: 0..127) + assert_write_fail "$lane_dir/margin_voltage" "9999" \ + "$lane: voltage step 9999 (out of range)" + assert_write_fail "$lane_dir/margin_voltage" "-9999" \ + "$lane: voltage step -9999 (out of range)" + assert_write_fail "$lane_dir/margin_voltage" "128" \ + "$lane: voltage step 128 (> spec max 127)" + assert_write_fail "$lane_dir/margin_voltage" "-128" \ + "$lane: voltage step -128 (< spec min -127)" + assert_write_fail "$lane_dir/margin_voltage" "invalid" \ + "$lane: non-numeric voltage step" + + if echo "$caps_raw" | grep -q "Voltage Supported: No"; then + assert_write_fail "$lane_dir/margin_voltage" "1" \ + "$lane: voltage step on unsupported receiver" + else + if [ -n "$num_voltage" ] && [ "$num_voltage" -ge 0 ] 2>/dev/null; then + assert_write_fail "$lane_dir/margin_voltage" \ + "$((num_voltage + 1))" \ + "$lane: voltage step > receiver limit ($num_voltage)" + fi + if echo "$caps_raw" | grep -q "Up/Down: symmetric"; then + assert_write_fail "$lane_dir/margin_voltage" "-1" \ + "$lane: negative voltage on symmetric receiver" + fi + fi + + # Positive test: reset timing and voltage margin to 0 (nominal settings) + assert_write_success "$lane_dir/margin_timing" "0" "$lane: reset timing = to 0" + assert_write_success "$lane_dir/margin_voltage" "0" "$lane: reset voltag= e to 0" + done + + # Cleanly disable margining + echo 0 > "$dev/enable" + trap - EXIT INT TERM + echo " $dev_name: margining disabled OK" +done + +if [ $retval -eq 0 ]; then + echo "$TESTNAME [PASS]" +else + echo "$TESTNAME [FAIL]" +fi + +exit $retval --=20 2.55.0.1003.g10538fe699-goog