Hi Bjorn, Ilpo, and the PCI community,
This patch series introduces upstream Linux kernel support for PCIe Lane Margining
at Receiver (LMR) per PCI Express Base Specification Revision 7.0 / 6.0 (§ 7.7.11
and § 8.4.4).
During the review of the previous v7 standalone LMR submission
(https://lore.kernel.org/linux-pci/20260828-pcie-lmt-v7-1-6012e9e0940a@google.com/),
Ilpo Järvinen pointed out that feature drivers resolving link partners duplicate
link traversal logic that is already present in drivers such as ASPM:
"This feels like duplicating similar functionality with the aspm driver
that also wants to infer ends of the link when giving a pci_dev in.
The aspm driver currently does that within, but it kind of duplicating
pci_bus. It would be nice to avoid the duplication and have something
similar for this in PCI core."
In response to this feedback, the implementation is factored into a clean,
modular 3-patch stack:
1. Patch 1/3 (PCI: Add pcie_get_link_endpoints() helper):
Standardized, race-safe helper in drivers/pci/pci.c and include/linux/pci.h
to identify both ends of a point-to-point PCIe link. Safely inspects the
subordinate bus under down_read(&pci_bus_sem) and acquires a reference
(pci_dev_get()), paired symmetrically with pcie_put_link_endpoints().
Filters for Function 0 (with ARI support), validates bridge ownership
(child->self == pdev) to prevent ABA pointer reuse, and handles RCiEP
and empty downstream ports (-ENODEV).
2. Patch 2/3 (PCI/ASPM: Add pci_aspm_inhibit() helper for temporary link state suppression):
Reference-counted mechanism (aspm_inhibit_cnt) in drivers/pci/pcie/aspm.c
to temporarily disable ASPM state transitions (L0s, L1, L1SS) on an active link.
Guarantees callers hold down_read(&pci_bus_sem) (via lockdep_assert_held_read),
eliminating deadlocks and preventing power-saving transitions while hardware
link characterization is in progress.
3. Patch 3/3 (PCI/pcie: Add PCIe Lane Margining at Receiver (LMR) support):
Implements the core LMR driver (CONFIG_PCIE_LMR) in drivers/pci/pcie/margin.c
exposing a debugfs interface under /sys/kernel/debug/pci/pcie_lmr_<dev>/.
Strictly self-contained: 0 modified lines in drivers/pci/pci.c, pci-driver.c,
or core drivers/pci/pci.h. Exposes clean lifecycle hooks in drivers/pci/probe.c
and drivers/pci/remove.c, supports 2D timing and voltage margining across
physical receivers (0..6), enforces Function 0 endpoint filtering (§ 7.7.11),
and includes an exhaustive kselftest suite (tools/testing/selftests/pcie_lmt/).
- Capability Discovery (§ 7.7.11): Probed via Extended Capability ID 0x27 on
Gen4+ (>= 16 GT/s) physical links. Differentiates optional support on Gen4/Gen5
from mandatory capability presence on Gen6+ (>= 64 GT/s).
- Multi-Function Device Scope (§ 7.7.11): Enforces that capability registration
is strictly restricted to Function 0 on Endpoints / Upstream Ports.
- Payload Decoding (Table 4-77, Cmd 88h): Accurately decodes Margin Control
Capabilities (Bits 0..4); preserves Bits 7:5 as reserved 0b.
- Direction Encodings (§ 4.2.18.1.2): Enforces that Left/Right and Up/Down
direction bits remain 0b for symmetric receivers (Bits 6 & 7 reserved 0b).
- Multi-Receiver Concurrency (§ 4.2.18.2): Governed strictly by MIndErrorSampler.
If MIndErrorSampler == 0b (main data sampler), at most one receiver across the
link is margined at a time.
- Autonomous Speed & Width Transitions (§ 7.5.3.7, § 7.5.3.17, § 8.4.4): Software
sequencing disables Downstream Component before Upstream Component on enablement,
and restores Upstream Component before Downstream Component on teardown.
Restoration is unconditional via pcie_capability_clear_and_set_word()
(serialized by pci_lock), eliminating trylock failure traps.
Ingress paths (margin_enable_write and pci_lmr_exit) strictly follow the canonical
PCI locking DAG:
down_read(&pci_bus_sem)
-> pci_dev_lock(downstream_port)
-> pci_dev_lock(upstream_port)
-> mutex_lock(&mdev->lock)
-> spin_lock_irqsave(&aspm_lock)
Key Concurrency Guarantees:
1. Hot-Swap Identity Invariance: When disabling LMR, dynamic endpoint resolution
is completely bypassed. Teardown binds strictly to the session-saved mdev->partner,
which is read and pinned with pci_dev_get() under mutex_lock(&mdev->lock).
2. Surprise Removal & Fault Hardening: Hardware register accesses check
pci_dev_is_disconnected() and PCI_POSSIBLE_ERROR() guards, preventing MMIO
or config space bus aborts when hardware is pulled.
3. Power Management Synchronization: Both link partners are pinned in D0 using
pm_runtime_resume_and_get() during enablement, and symmetrically released via
pm_runtime_put() on teardown. System suspend hooks into pci_pm_prepare() where
all devices in the hierarchy are guaranteed to be in D0.
4. Security Teardown Order: In pci_destroy_dev(), pci_lmr_exit() is called after
pci_tsm_destroy(), preserving D0 state for TSM link operations (PCIe IDE unbind
and SPDM cryptographic session teardown).
- Build: Clean compile on x86_64 and ARM64; make W=1 drivers/pci/ (0 warnings).
- Linters: checkpatch.pl clean across all 3 patches (0 errors, 0 warnings).
- Kselftest: tools/testing/selftests/pcie_lmt/pcie_lmt.sh expanded to 287 lines,
covering positive and negative boundary tests (syntax clean with bash -n).
- Pre-Commit AI Review (Sashiko AI):
* Patch 1 (f34fdf7f5c6b): Reviewed — Clean (0 issues).
* Patch 2 (6adc680a727f): Reviewed — Clean (0 issues).
* Patch 3 (c62b5f92af63): Reviewed — Clean (0 issues, "No issues found.").
Signed-off-by: Priyank Rathod <rathodpriyank@google.com>
---
Changes in v2:
- Decomposed monolithic LMR driver into a 3-patch stack with dedicated core helpers:
* Patch 1: PCI core link endpoint discovery helper (pcie_get_link_endpoints()).
* Patch 2: Core ASPM temporary inhibition helper (pci_aspm_inhibit()).
* Patch 3: PCIe Lane Margining at Receiver driver and debugfs interface.
- Core Scope Discipline: Confined LMR logic exclusively to drivers/pci/pcie/margin.c
and include/linux/pci.h forward declarations (0 modified lines in pci.c/pci-driver.c).
- Endpoint Discovery Hardening:
* Handled RCiEP and empty downstream buses returning -ENODEV.
* Added Function 0 resolution (pcie_find_link_upstream_func0()) supporting PCIe ARI.
* Added bridge ownership verification (child->self == pdev) to prevent ABA races.
* Added symmetric pcie_put_link_endpoints() to balance pci_dev_get() references.
- ASPM Inhibit Hardening:
* Added lockdep_assert_held_read(&pci_bus_sem) to pci_aspm_inhibit_locked().
* Added reference counting (aspm_inhibit_cnt) to support nested/concurrent calls.
- Hot-Swap & Concurrency Protections:
* Eliminated hot-swap identity mismatch in margin_enable_write() by binding teardown
strictly to session-saved mdev->partner instead of re-evaluating endpoints.
* Added mutex protection when reading and pinning mdev->partner to eliminate UAF races.
* Converted autonomous speed/width restoration to unconditional register write via
pcie_capability_clear_and_set_word(), eliminating trylock failure degradation.
- Specification Alignments (PCIe Base Spec r7.0):
* Aligned Extended Capability ID 0x27 to § 7.7.11 and § 8.4.4.
* Enforced Function 0 restriction on Endpoints / Upstream Ports (§ 7.7.11).
* Preserved Reserved Bits 7:5 in Command 88h capabilities response.
* Implemented MIndErrorSampler-based concurrency for multi-receiver links (§ 4.2.18.2).
* Corrected direction bit encoding to 0b for symmetric receivers (§ 4.2.18.1.2).
- Kselftest Overhaul:
* Extended test coverage from 105 to 287 lines, adding comprehensive boundary
and negative validation tests.
- Link to v1: https://lore.kernel.org/r/20260831-pcie-link-endpoints-v1-1-32c2fd893e9e@google.com
- Link to v7 (monolithic LMR): https://lore.kernel.org/linux-pci/20260828-pcie-lmt-v7-1-6012e9e0940a@google.com/
---
Priyank Rathod (3):
PCI: Add pcie_get_link_endpoints() helper
PCI/ASPM: Add pci_aspm_inhibit() helper for temporary link state suppression
PCI/pcie: Add PCIe Lane Margining at Receiver (LMR) support
Documentation/PCI/index.rst | 1 +
Documentation/PCI/pcie-lmr.rst | 174 +++
MAINTAINERS | 8 +
drivers/pci/pci.c | 152 +++
drivers/pci/pci.h | 8 +
drivers/pci/pcie/Kconfig | 12 +
drivers/pci/pcie/Makefile | 1 +
drivers/pci/pcie/aspm.c | 100 ++
drivers/pci/pcie/margin.c | 1592 ++++++++++++++++++++++++++
drivers/pci/probe.c | 1 +
drivers/pci/remove.c | 2 +-
include/linux/pci.h | 17 +
include/uapi/linux/pci_regs.h | 18 +
tools/testing/selftests/Makefile | 1 +
tools/testing/selftests/pcie_lmt/Makefile | 3 +
tools/testing/selftests/pcie_lmt/pcie_lmt.sh | 287 +++++
16 files changed, 2376 insertions(+), 1 deletion(-)
---
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
change-id: 20260831-pcie-link-endpoints-978e100d5d06
Best regards,
--
Priyank Rathod <rathodpriyank@google.com>