Documentation/PCI/index.rst | 1 + Documentation/PCI/pcie-lmr.rst | 174 +++ MAINTAINERS | 8 + drivers/pci/pci.c | 152 +++ drivers/pci/pci.h | 8 + drivers/pci/pcie/Kconfig | 12 + drivers/pci/pcie/Makefile | 1 + drivers/pci/pcie/aspm.c | 100 ++ drivers/pci/pcie/margin.c | 1682 ++++++++++++++++++++++++++ drivers/pci/probe.c | 1 + drivers/pci/remove.c | 2 +- include/linux/pci.h | 17 + include/uapi/linux/pci_regs.h | 18 + tools/testing/selftests/Makefile | 1 + tools/testing/selftests/pcie_lmt/Makefile | 3 + tools/testing/selftests/pcie_lmt/pcie_lmt.sh | 405 +++++++ 16 files changed, 2584 insertions(+), 1 deletion(-)
Hi Bjorn, Ilpo, and the PCI community,
This patch series introduces upstream Linux kernel support for PCIe Lane Margining
at Receiver (LMR) per PCI Express Base Specification Revision 7.0 / 6.0 (§ 7.7.11
and § 8.4.4).
During the review of the previous v7 standalone LMR submission
(https://lore.kernel.org/linux-pci/20260828-pcie-lmt-v7-1-6012e9e0940a@google.com/),
Ilpo Järvinen pointed out that feature drivers resolving link partners duplicate
link traversal logic that is already present in drivers such as ASPM:
"This feels like duplicating similar functionality with the aspm driver
that also wants to infer ends of the link when giving a pci_dev in.
The aspm driver currently does that within, but it kind of duplicating
pci_bus. It would be nice to avoid the duplication and have something
similar for this in PCI core."
In response to this feedback, the implementation is factored into a clean,
modular 3-patch stack:
1. Patch 1/3 (PCI: Add pcie_get_link_endpoints() helper):
Standardized, race-safe helper in drivers/pci/pci.c and include/linux/pci.h
to identify both ends of a point-to-point PCIe link. Safely inspects the
subordinate bus under down_read(&pci_bus_sem) and acquires a reference
(pci_dev_get()), paired symmetrically with pcie_put_link_endpoints().
Filters for Function 0 (with ARI support), validates bridge ownership
(child->self == pdev) to prevent ABA pointer reuse, and handles RCiEP
and empty downstream ports (-ENODEV).
2. Patch 2/3 (PCI/ASPM: Add pci_aspm_inhibit() helper for temporary link state suppression):
Reference-counted mechanism (aspm_inhibit_cnt) in drivers/pci/pcie/aspm.c
to temporarily disable ASPM state transitions (L0s, L1, L1SS) on an active link.
Guarantees callers hold down_read(&pci_bus_sem) (via lockdep_assert_held_read),
eliminating deadlocks and preventing power-saving transitions while hardware
link characterization is in progress.
3. Patch 3/3 (PCI/pcie: Add PCIe Lane Margining at Receiver (LMR) support):
Implements the core LMR driver (CONFIG_PCIE_LMR) in drivers/pci/pcie/margin.c
exposing a debugfs interface under /sys/kernel/debug/pci/pcie_lmr_<dev>/.
Strictly self-contained: 0 modified lines in drivers/pci/pci.c, pci-driver.c,
or core drivers/pci/pci.h. Exposes clean lifecycle hooks in drivers/pci/probe.c
and drivers/pci/remove.c, supports 2D timing and voltage margining across
physical receivers (0..6), enforces Function 0 endpoint filtering (§ 7.7.11),
and includes an exhaustive kselftest suite (tools/testing/selftests/pcie_lmt/).
- Capability Discovery (§ 7.7.11): Probed via Extended Capability ID 0x27 on
Gen4+ (>= 16 GT/s) physical links. Differentiates optional support on Gen4/Gen5
from mandatory capability presence on Gen6+ (>= 64 GT/s).
- Multi-Function Device Scope (§ 7.7.11): Enforces that capability registration
is strictly restricted to Function 0 on Endpoints / Upstream Ports.
- Payload Decoding (Table 4-77, Cmd 88h): Accurately decodes Margin Control
Capabilities (Bits 0..4); preserves Bits 7:5 as reserved 0b.
- Direction Encodings (§ 4.2.18.1.2): Enforces that Left/Right and Up/Down
direction bits remain 0b for symmetric receivers (Bits 6 & 7 reserved 0b).
- Multi-Receiver Concurrency (§ 4.2.18.2): Governed strictly by MIndErrorSampler.
If MIndErrorSampler == 0b (main data sampler), at most one receiver across the
link is margined at a time.
- Autonomous Speed & Width Transitions (§ 7.5.3.7, § 7.5.3.17, § 8.4.4): Software
sequencing disables Downstream Component before Upstream Component on enablement,
and restores Upstream Component before Downstream Component on teardown.
Restoration is unconditional via pcie_capability_clear_and_set_word()
(serialized by pci_lock), eliminating trylock failure traps.
Ingress paths (margin_enable_write and pci_lmr_exit) strictly follow the canonical
PCI locking DAG:
down_read(&pci_bus_sem)
-> pci_dev_lock(downstream_port)
-> pci_dev_lock(upstream_port)
-> mutex_lock(&mdev->lock)
-> spin_lock_irqsave(&aspm_lock)
Key Concurrency Guarantees:
1. Hot-Swap Identity Invariance: When disabling LMR, dynamic endpoint resolution
is completely bypassed. Teardown binds strictly to the session-saved mdev->partner,
which is read and pinned with pci_dev_get() under mutex_lock(&mdev->lock).
2. Surprise Removal & Fault Hardening: Hardware register accesses check
pci_dev_is_disconnected() and PCI_POSSIBLE_ERROR() guards, preventing MMIO
or config space bus aborts when hardware is pulled.
3. Power Management Synchronization: Both link partners are pinned in D0 using
pm_runtime_resume_and_get() during enablement, and symmetrically released via
pm_runtime_put() on teardown. System suspend hooks into pci_pm_prepare() where
all devices in the hierarchy are guaranteed to be in D0.
4. Security Teardown Order: In pci_destroy_dev(), pci_lmr_exit() is called after
pci_tsm_destroy(), preserving D0 state for TSM link operations (PCIe IDE unbind
and SPDM cryptographic session teardown).
- Build: Clean compile on x86_64 and ARM64; make W=1 drivers/pci/ (0 warnings).
- Linters: checkpatch.pl clean across all 3 patches (0 errors, 0 warnings).
- Kselftest: tools/testing/selftests/pcie_lmt/pcie_lmt.sh expanded to 287 lines,
covering positive and negative boundary tests (syntax clean with bash -n).
- Pre-Commit AI Review (Sashiko AI):
* Patch 1 (f34fdf7f5c6b): Reviewed — Clean (0 issues).
* Patch 2 (6adc680a727f): Reviewed — Clean (0 issues).
* Patch 3 (c62b5f92af63): Reviewed — Clean (0 issues, "No issues found.").
Signed-off-by: Priyank Rathod <rathodpriyank@google.com>
---
Changes in v3:
- Port Classification Modernization:
* Replaced open-coded (PCI_EXP_TYPE_ROOT_PORT || PCI_EXP_TYPE_DOWNSTREAM) checks
in drivers/pci/pcie/margin.c with canonical pcie_downstream_port(dev).
* Why: Open-coded comparisons missed PCI_EXP_TYPE_PCIE_BRIDGE ports, which act
as downstream ports in specific bridge topologies. This caused inverted
downstream/upstream port classification, reversing lock acquisition order and
causing configuration writes to link partners without holding appropriate locks.
* How: Adopted core pcie_downstream_port() helper universally across margin.c.
- Hardware Execution Payload Status Validation:
* In pci_lmr_run_cmd(), added explicit checking of hardware execution status bits
(LMR_STS_EXEC_MASK via pci_lmr_check_exec_status()) instead of relying solely on
PCI configuration bus transaction return codes.
* Why: A configuration register write/read succeeds at the bus transaction level
(ret == 0) even when the physical receiver hardware rejects the command (NAK)
or encounters excessive bit errors during stepping.
* How: Explicitly map LMR_STS_EXEC_NAK (0x3) to -EOPNOTSUPP without mutating software
margin state, and LMR_STS_EXEC_TOO_MANY_ERR (0x0) to -EIO while resetting both
plane->timing_val and plane->voltage_val to nominal 0.
- Hardware Step 0 Synchronization & Orthogonal State Preservation (Table 4-77):
* Aligned step margin zeroing with PCIe Base Specification Revision 7.0 Table 4-77.
* Why: Writing a Step Margin command with payload 0 is explicitly specified as a
hardware NO-OP; the physical receiver sampler ignores it. The only specification-
defined nominal reset is Command 0x0F ("Go to Normal Settings"), but issuing 0x0F
clears both timing and voltage samplers simultaneously, causing orthogonal hardware
drift in 2D margining.
* How: When zeroing an axis (e.g. echo 0 > margin_timing) while the orthogonal axis
is displaced (voltage_val != 0), issue 0x0F and immediately re-apply the saved
orthogonal displacement via pci_lmr_issue_step(), ensuring 100% bit-accurate
hardware/software state synchronization.
- Multi-Receiver Concurrency Guard Hardening (§ 4.2.18.2 & § 8.4.4):
* In pci_lmr_check_sample_multiple_rx(), enforced dual verification of both
LMR_CAP_IND_ERROR_SAMPLER (Bit 4) and LMR_CAP_SAMPLE_MULTIPLE_RECEIVERS (Bit 5).
* Why: If an active receiver uses the live data sampler (Bit 4 = 0) or cannot sample
multiple receivers concurrently (Bit 5 = 0), running margining on another receiver
injects errors into operational data traffic or exceeds hardware capabilities.
* How: Replaced single-bit check with a bidirectional predicate requiring BOTH Bit 4
and Bit 5 to be asserted on both the targeted receiver and all currently active
receivers across all lanes. If either bit is 0, reject with -EBUSY.
- Non-Link Port Type Filtering:
* In pci_lmr_init(), filtered out non-link PCIe port types (PCI_EXP_TYPE_RC_END,
PCI_EXP_TYPE_RC_EC, PCI_EXP_TYPE_PCI_BRIDGE) per PCIe Base Spec § 7.7.11 to
eliminate spurious pci_warn diagnostics on unsupported topologies.
- DebugFS Subsystem Directory Portability:
* Verified debugfs "pci" root directory coexistence on architectures like s390 via
debugfs_lookup("pci", NULL) before debugfs_create_dir("pci", NULL), preventing
-EEXIST from poisoning the root dentry pointer.
- Pre-Submission Verification:
* Pre-commit AI review (Sashiko AI bot): Reviewed — No issues found (0 findings,
0 concerns, Review ID 94722).
* Compiler: make W=1 drivers/pci/ (0 warnings, 0 errors on x86_64 and ARM64).
* Linter: checkpatch.pl clean across all patches (0 warnings, 0 errors).
- Link to v2: https://lore.kernel.org/r/20260904-pcie-link-endpoints-v2-0-16fcb301a3e4@google.com
Changes in v2:
- Decomposed monolithic LMR driver into a 3-patch stack with dedicated core helpers:
* Patch 1: PCI core link endpoint discovery helper (pcie_get_link_endpoints()).
* Patch 2: Core ASPM temporary inhibition helper (pci_aspm_inhibit()).
* Patch 3: PCIe Lane Margining at Receiver driver and debugfs interface.
- Core Scope Discipline: Confined LMR logic exclusively to drivers/pci/pcie/margin.c
and include/linux/pci.h forward declarations (0 modified lines in pci.c/pci-driver.c).
- Endpoint Discovery Hardening:
* Handled RCiEP and empty downstream buses returning -ENODEV.
* Added Function 0 resolution (pcie_find_link_upstream_func0()) supporting PCIe ARI.
* Added bridge ownership verification (child->self == pdev) to prevent ABA races.
* Added symmetric pcie_put_link_endpoints() to balance pci_dev_get() references.
- ASPM Inhibit Hardening:
* Added lockdep_assert_held_read(&pci_bus_sem) to pci_aspm_inhibit_locked().
* Added reference counting (aspm_inhibit_cnt) to support nested/concurrent calls.
- Hot-Swap & Concurrency Protections:
* Eliminated hot-swap identity mismatch in margin_enable_write() by binding teardown
strictly to session-saved mdev->partner instead of re-evaluating endpoints.
* Added mutex protection when reading and pinning mdev->partner to eliminate UAF races.
* Converted autonomous speed/width restoration to unconditional register write via
pcie_capability_clear_and_set_word(), eliminating trylock failure degradation.
- Specification Alignments (PCIe Base Spec r7.0):
* Aligned Extended Capability ID 0x27 to § 7.7.11 and § 8.4.4.
* Enforced Function 0 restriction on Endpoints / Upstream Ports (§ 7.7.11).
* Preserved Reserved Bits 7:5 in Command 88h capabilities response.
* Implemented MIndErrorSampler-based concurrency for multi-receiver links (§ 4.2.18.2).
* Corrected direction bit encoding to 0b for symmetric receivers (§ 4.2.18.1.2).
- Kselftest Overhaul:
* Extended test coverage from 105 to 287 lines, adding comprehensive boundary
and negative validation tests.
- Link to v1: https://lore.kernel.org/r/20260831-pcie-link-endpoints-v1-1-32c2fd893e9e@google.com
- Link to v7 (monolithic LMR): https://lore.kernel.org/linux-pci/20260828-pcie-lmt-v7-1-6012e9e0940a@google.com/
---
Priyank Rathod (3):
PCI: Add pcie_get_link_endpoints() helper
PCI/ASPM: Add pci_aspm_inhibit() helper for temporary link state suppression
PCI/pcie: Add PCIe Lane Margining at Receiver (LMR) support
Documentation/PCI/index.rst | 1 +
Documentation/PCI/pcie-lmr.rst | 174 +++
MAINTAINERS | 8 +
drivers/pci/pci.c | 152 +++
drivers/pci/pci.h | 8 +
drivers/pci/pcie/Kconfig | 12 +
drivers/pci/pcie/Makefile | 1 +
drivers/pci/pcie/aspm.c | 100 ++
drivers/pci/pcie/margin.c | 1682 ++++++++++++++++++++++++++
drivers/pci/probe.c | 1 +
drivers/pci/remove.c | 2 +-
include/linux/pci.h | 17 +
include/uapi/linux/pci_regs.h | 18 +
tools/testing/selftests/Makefile | 1 +
tools/testing/selftests/pcie_lmt/Makefile | 3 +
tools/testing/selftests/pcie_lmt/pcie_lmt.sh | 405 +++++++
16 files changed, 2584 insertions(+), 1 deletion(-)
---
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
change-id: 20260831-pcie-link-endpoints-978e100d5d06
Best regards,
--
Priyank Rathod <rathodpriyank@google.com>
Hi Bjorn, Ilpo,
Gentle ping on this series - it has been two weeks since v3 was posted
and I haven't seen any feedback yet.
v3 restructured the original monolithic LMR driver (v7) into the 3-patch
stack Ilpo suggested during the v7 review, moving the link endpoint
traversal into the PCI core rather than duplicating it in the feature
driver:
1/3 PCI: Add pcie_get_link_endpoints() helper
2/3 PCI/ASPM: Add pci_aspm_inhibit() helper for temporary link state
suppression
3/3 PCI/pcie: Add PCIe Lane Margining at Receiver (LMR) support
The LMR driver itself is now strictly self-contained in
drivers/pci/pcie/margin.c, with no modifications to drivers/pci/pci.c,
drivers/pci/pci-driver.c or drivers/pci/pci.h. The series builds clean
with W=1 and is checkpatch clean on x86_64 and arm64.
Feedback on patches 1 and 2 would be especially useful, since they are
the core helpers patch 3 builds on. I'm happy to respin if the split
isn't what you had in mind, or to rebase onto a newer base if that helps
review.
Thanks,
Priyank
© 2016 - 2026 Red Hat, Inc.