Documentation/ABI/testing/sysfs-bus-pci | 15 + drivers/cxl/Kconfig | 4 + drivers/cxl/core/Makefile | 2 +- drivers/cxl/core/core.h | 8 + drivers/cxl/core/hdm.c | 324 +++--- drivers/cxl/core/pci.c | 7 +- drivers/cxl/core/port.c | 19 + drivers/cxl/core/regs.c | 4 + drivers/cxl/core/resource.c | 1351 +++++++++++++++++++++++ drivers/cxl/cxl.h | 41 - drivers/pci/bus.c | 2 + drivers/pci/pci.c | 8 + drivers/pci/probe.c | 2 + include/cxl/cxl.h | 116 +- include/linux/pci.h | 8 +- include/uapi/linux/pci_regs.h | 14 + tools/testing/cxl/Kbuild | 1 - 17 files changed, 1715 insertions(+), 211 deletions(-) create mode 100644 drivers/cxl/core/resource.c
Hi folks!
This series adds CXL Reset support for CXL Type 2 devices through the
existing PCI reset_method ABI. The reset sequence follows the CXL 4.0
specification [1], including CXL.cache disable, optional cache
writeback, CXL Reset initiation, ResetComplete polling, and ResetError
reporting.
The userspace ABI is the existing PCI reset interface:
/sys/bus/pci/devices/.../reset_method
/sys/bus/pci/devices/.../reset
Userspace can select "cxl_reset" in reset_method and then trigger reset
through the existing reset attribute.
CXL Reset is exposed conservatively. The target must be a
single-function CXL Type 2 device with CXL.cache and CXL.mem enabled,
and the kernel must have cached HDM state sufficient to validate affected
ranges and restore decoder programming. The reset preserves device
memory; it does not request CXL Reset Memory Clear.
Following Dan's v6 feedback, this uses the existing PCI reset_method
interface instead of adding a memdev-specific sysfs ABI.
v11 changes from v10 [2]:
- Rebase onto v7.3-rc1, which includes the CXL Type 2 enabling work, and
use the merged public CXL PCI header.
- Rename the built-in support symbol to CONFIG_CXL_RESET and fix
built-in/module linkage for CONFIG_CXL_BUS=m.
- Fix tools/testing/cxl linkage without compiling production helper
objects twice in the mock cxl_core module.
- Simplify decoder snapshot handling and preserve endpoint DPA skip
ownership under the existing DPA lock.
- Keep policy out of shared decoder parsing by accepting spec-legal
zero-size decoder state, while skipping zero-size ranges during reset
validation.
- Tighten early HDM cache setup, capability checks, BAR bounds, MSE
enable/restore handling, and cache publication locking.
- Keep the conservative multifunction-device rejection requested in
earlier review.
- Harden post-reset failure handling so MSE and BME remain disabled when
HDM state cannot be safely restored.
- Serialize CXL bus-reset HDM restore against region updates and flush
posted HDM writes before PCI_COMMAND can restore bus mastering.
- Address the remaining reviewability and naming feedback from v10.
Motivation:
-----------
- Type 2 devices need a CXL-specific reset mechanism beyond existing PCI
reset methods.
- FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is
the architectural reset mechanism for those protocols.
- The PCI reset_method ABI lets userspace select CXL Reset before
falling back to broader bus reset methods.
Change Description:
-------------------
Patch 1: cxl: Move HDM decoder programming helpers
- Introduce a decoder settings snapshot collected under the DPA lock.
- Move low-level settings-based HDM programming into resource.c.
Patch 2: cxl: Make HDM commit helpers available to reset code
- Build generic HDM resource helpers under CONFIG_CXL_RESET.
- Export the lock and commit helpers needed when cxl_core is modular.
Patch 3: cxl: Share HDM decoder decode logic
- Share HDM decoder decode and validation between normal CXL core
enumeration and early PCI HDM cache setup.
- Decode zero-size committed decoder state without imposing caller policy.
Patch 4: cxl: Cache decoder settings on PCI devices
- Cache CXL core HDM decoder settings in pci_dev->hdm.
- Refresh the cache as decoders are enumerated, committed, or reset.
Patch 5: cxl: Cache endpoint decoder settings during PCI enumeration
- Snapshot endpoint HDM state during PCI enumeration.
- Cache the HDM register locator as BAR-relative metadata and preserve raw
decoder state for reset restore.
Patch 6: cxl: Add CXL Device Reset helper
- Add the internal DVSEC reset sequence.
- Disable CXL.cache, perform cache writeback where supported, initiate
CXL Reset, and wait for completion.
Patch 7: cxl: Validate HDM ranges before CXL reset
- Collect enabled cached HDM ranges, reject busy ranges, and perform CPU
cache invalidation when a runtime backend is available.
Patch 8: PCI/CXL: Reject CXL Reset on multifunction devices
- Conservatively reject CXL Reset where function scope cannot be assumed.
Patch 9: cxl: Restore CXL HDM state after PCI reset
- Restore cached global and per-decoder HDM state after reset while the
target IOMMU reset block remains active.
Patch 10: PCI/CXL: Expose CXL Reset as a PCI reset method
- Add "cxl_reset" to the PCI reset-method table ahead of FLR.
Patch 11: Documentation/ABI: Document CXL Reset PCI reset method
- Document the new reset_method value and reset behavior.
Patch 12: PCI/CXL: Restore HDM state after CXL bus reset
- Restore cached CXL HDM decoder state after cxl_bus reset.
The CPU cache invalidation step depends on
cpu_cache_invalidate_memregion() support for the affected address ranges.
If no runtime provider is available, the kernel emits a warning and
continues after the affected HDM ranges have been reserved.
Example:
echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method
echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset
Testing:
- Built every intermediate patch, plus the v7.3-rc1 baseline, with
CONFIG_CXL_BUS=m. All 12 patches passed a full kernel build and
`make M=tools/testing/cxl/` independently.
- Built and booted the series on an arm64 NVIDIA NVL72 system and
successfully exercised CXL Reset through the PCI reset sysfs ABI.
- Earlier revision testing ran 100 iterations of cxl_reset and exercised
cxl_bus separately with HDM state restoration.
References:
[1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf
[2] https://lore.kernel.org/linux-cxl/20260804192958.1823952-1-smadhavan@nvidia.com/
[3] https://lore.kernel.org/linux-cxl/20260709010304.680422-1-smadhavan@nvidia.com/
[4] https://lore.kernel.org/linux-cxl/20260703220508.546528-1-smadhavan@nvidia.com/
[5] https://lore.kernel.org/linux-cxl/20260623032453.3404772-1-smadhavan@nvidia.com/
Srirangan Madhavan (12):
cxl: Move HDM decoder programming helpers
cxl: Make HDM commit helpers available to reset code
cxl: Share HDM decoder decode logic
cxl: Cache decoder settings on PCI devices
cxl: Cache endpoint decoder settings during PCI enumeration
cxl: Add CXL Device Reset helper
cxl: Validate HDM ranges before CXL reset
PCI/CXL: Reject CXL Reset on multifunction devices
cxl: Restore CXL HDM state after PCI reset
PCI/CXL: Expose CXL Reset as a PCI reset method
Documentation/ABI: Document CXL Reset PCI reset method
PCI/CXL: Restore HDM state after CXL bus reset
Documentation/ABI/testing/sysfs-bus-pci | 15 +
drivers/cxl/Kconfig | 4 +
drivers/cxl/core/Makefile | 2 +-
drivers/cxl/core/core.h | 8 +
drivers/cxl/core/hdm.c | 324 +++---
drivers/cxl/core/pci.c | 7 +-
drivers/cxl/core/port.c | 19 +
drivers/cxl/core/regs.c | 4 +
drivers/cxl/core/resource.c | 1351 +++++++++++++++++++++++
drivers/cxl/cxl.h | 41 -
drivers/pci/bus.c | 2 +
drivers/pci/pci.c | 8 +
drivers/pci/probe.c | 2 +
include/cxl/cxl.h | 116 +-
include/linux/pci.h | 8 +-
include/uapi/linux/pci_regs.h | 14 +
tools/testing/cxl/Kbuild | 1 -
17 files changed, 1715 insertions(+), 211 deletions(-)
create mode 100644 drivers/cxl/core/resource.c
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
--
2.43.0
On Wed, Sep 02, 2026 at 07:27:52AM +0800, Srirangan Madhavan wrote: > Hi folks! > > This series adds CXL Reset support for CXL Type 2 devices through the > existing PCI reset_method ABI. The reset sequence follows the CXL 4.0 > specification [1], including CXL.cache disable, optional cache > writeback, CXL Reset initiation, ResetComplete polling, and ResetError > reporting. > Hi Srirangan, > The userspace ABI is the existing PCI reset interface: > > /sys/bus/pci/devices/.../reset_method > /sys/bus/pci/devices/.../reset > > Userspace can select "cxl_reset" in reset_method and then trigger reset > through the existing reset attribute. > > CXL Reset is exposed conservatively. The target must be a > single-function CXL Type 2 device with CXL.cache and CXL.mem enabled, > and the kernel must have cached HDM state sufficient to validate affected > ranges and restore decoder programming. The reset preserves device > memory; it does not request CXL Reset Memory Clear. > It seems reasonable to me, the whole approach. I wonder if we can document the complete reset-safety contract down ? Especially what guarantees that all CXL.mem users and functions are quiesced ? Besides HDM EP , what are other EP state that should be retained or restored ? I think it will make people understand the mechanism more easily. Best regards, Richard Cheng. > Following Dan's v6 feedback, this uses the existing PCI reset_method > interface instead of adding a memdev-specific sysfs ABI. > > v11 changes from v10 [2]: > - Rebase onto v7.3-rc1, which includes the CXL Type 2 enabling work, and > use the merged public CXL PCI header. > - Rename the built-in support symbol to CONFIG_CXL_RESET and fix > built-in/module linkage for CONFIG_CXL_BUS=m. > - Fix tools/testing/cxl linkage without compiling production helper > objects twice in the mock cxl_core module. > - Simplify decoder snapshot handling and preserve endpoint DPA skip > ownership under the existing DPA lock. > - Keep policy out of shared decoder parsing by accepting spec-legal > zero-size decoder state, while skipping zero-size ranges during reset > validation. > - Tighten early HDM cache setup, capability checks, BAR bounds, MSE > enable/restore handling, and cache publication locking. > - Keep the conservative multifunction-device rejection requested in > earlier review. > - Harden post-reset failure handling so MSE and BME remain disabled when > HDM state cannot be safely restored. > - Serialize CXL bus-reset HDM restore against region updates and flush > posted HDM writes before PCI_COMMAND can restore bus mastering. > - Address the remaining reviewability and naming feedback from v10. > > Motivation: > ----------- > - Type 2 devices need a CXL-specific reset mechanism beyond existing PCI > reset methods. > > - FLR does not reset CXL.cache or CXL.mem protocol state. CXL Reset is > the architectural reset mechanism for those protocols. > > - The PCI reset_method ABI lets userspace select CXL Reset before > falling back to broader bus reset methods. > > Change Description: > ------------------- > > Patch 1: cxl: Move HDM decoder programming helpers > - Introduce a decoder settings snapshot collected under the DPA lock. > - Move low-level settings-based HDM programming into resource.c. > > Patch 2: cxl: Make HDM commit helpers available to reset code > - Build generic HDM resource helpers under CONFIG_CXL_RESET. > - Export the lock and commit helpers needed when cxl_core is modular. > > Patch 3: cxl: Share HDM decoder decode logic > - Share HDM decoder decode and validation between normal CXL core > enumeration and early PCI HDM cache setup. > - Decode zero-size committed decoder state without imposing caller policy. > > Patch 4: cxl: Cache decoder settings on PCI devices > - Cache CXL core HDM decoder settings in pci_dev->hdm. > - Refresh the cache as decoders are enumerated, committed, or reset. > > Patch 5: cxl: Cache endpoint decoder settings during PCI enumeration > - Snapshot endpoint HDM state during PCI enumeration. > - Cache the HDM register locator as BAR-relative metadata and preserve raw > decoder state for reset restore. > > Patch 6: cxl: Add CXL Device Reset helper > - Add the internal DVSEC reset sequence. > - Disable CXL.cache, perform cache writeback where supported, initiate > CXL Reset, and wait for completion. > > Patch 7: cxl: Validate HDM ranges before CXL reset > - Collect enabled cached HDM ranges, reject busy ranges, and perform CPU > cache invalidation when a runtime backend is available. > > Patch 8: PCI/CXL: Reject CXL Reset on multifunction devices > - Conservatively reject CXL Reset where function scope cannot be assumed. > > Patch 9: cxl: Restore CXL HDM state after PCI reset > - Restore cached global and per-decoder HDM state after reset while the > target IOMMU reset block remains active. > > Patch 10: PCI/CXL: Expose CXL Reset as a PCI reset method > - Add "cxl_reset" to the PCI reset-method table ahead of FLR. > > Patch 11: Documentation/ABI: Document CXL Reset PCI reset method > - Document the new reset_method value and reset behavior. > > Patch 12: PCI/CXL: Restore HDM state after CXL bus reset > - Restore cached CXL HDM decoder state after cxl_bus reset. > > The CPU cache invalidation step depends on > cpu_cache_invalidate_memregion() support for the affected address ranges. > If no runtime provider is available, the kernel emits a warning and > continues after the affected HDM ranges have been reserved. > > Example: > > echo cxl_reset > /sys/bus/pci/devices/0000:bb:dd.f/reset_method > echo 1 > /sys/bus/pci/devices/0000:bb:dd.f/reset > > Testing: > - Built every intermediate patch, plus the v7.3-rc1 baseline, with > CONFIG_CXL_BUS=m. All 12 patches passed a full kernel build and > `make M=tools/testing/cxl/` independently. > - Built and booted the series on an arm64 NVIDIA NVL72 system and > successfully exercised CXL Reset through the PCI reset sysfs ABI. > - Earlier revision testing ran 100 iterations of cxl_reset and exercised > cxl_bus separately with HDM state restoration. > > References: > [1] https://computeexpresslink.org/wp-content/uploads/2026/02/CXL-Specification_rev4p0_ver1p0_2026February26_clean_evalcopy_v2.pdf > [2] https://lore.kernel.org/linux-cxl/20260804192958.1823952-1-smadhavan@nvidia.com/ > [3] https://lore.kernel.org/linux-cxl/20260709010304.680422-1-smadhavan@nvidia.com/ > [4] https://lore.kernel.org/linux-cxl/20260703220508.546528-1-smadhavan@nvidia.com/ > [5] https://lore.kernel.org/linux-cxl/20260623032453.3404772-1-smadhavan@nvidia.com/ > > Srirangan Madhavan (12): > cxl: Move HDM decoder programming helpers > cxl: Make HDM commit helpers available to reset code > cxl: Share HDM decoder decode logic > cxl: Cache decoder settings on PCI devices > cxl: Cache endpoint decoder settings during PCI enumeration > cxl: Add CXL Device Reset helper > cxl: Validate HDM ranges before CXL reset > PCI/CXL: Reject CXL Reset on multifunction devices > cxl: Restore CXL HDM state after PCI reset > PCI/CXL: Expose CXL Reset as a PCI reset method > Documentation/ABI: Document CXL Reset PCI reset method > PCI/CXL: Restore HDM state after CXL bus reset > > Documentation/ABI/testing/sysfs-bus-pci | 15 + > drivers/cxl/Kconfig | 4 + > drivers/cxl/core/Makefile | 2 +- > drivers/cxl/core/core.h | 8 + > drivers/cxl/core/hdm.c | 324 +++--- > drivers/cxl/core/pci.c | 7 +- > drivers/cxl/core/port.c | 19 + > drivers/cxl/core/regs.c | 4 + > drivers/cxl/core/resource.c | 1351 +++++++++++++++++++++++ > drivers/cxl/cxl.h | 41 - > drivers/pci/bus.c | 2 + > drivers/pci/pci.c | 8 + > drivers/pci/probe.c | 2 + > include/cxl/cxl.h | 116 +- > include/linux/pci.h | 8 +- > include/uapi/linux/pci_regs.h | 14 + > tools/testing/cxl/Kbuild | 1 - > 17 files changed, 1715 insertions(+), 211 deletions(-) > create mode 100644 drivers/cxl/core/resource.c > > > base-commit: cee9395acd8043be0644b25c34bfa86623f2b935 > -- > 2.43.0
On 9/4/26 2:15 AM, Richard Cheng wrote: >> and the kernel must have cached HDM state sufficient to validate affected >> ranges and restore decoder programming. The reset preserves device >> memory; it does not request CXL Reset Memory Clear. >> > It seems reasonable to me, the whole approach. > I wonder if we can document the complete reset-safety contract down ? > > Especially what guarantees that all CXL.mem users and functions are quiesced ? > Besides HDM EP , what are other EP state that should be retained or restored ? > > I think it will make people understand the mechanism more easily. > > Best regards, > Richard Cheng. Thank you. I expanded the v12 cover letter and ABI documentation to describe the complete reset-safety contract, including the quiescing assumptions, state restoration, and limitations around memory offlining and mapping revocation. -- Regards, Srirangan
On Wed, 2 Sep 2026 07:27:52 +0000 Srirangan Madhavan <smadhavan@nvidia.com> wrote: > Hi folks! > > This series adds CXL Reset support for CXL Type 2 devices through the > existing PCI reset_method ABI. The reset sequence follows the CXL 4.0 > specification [1], including CXL.cache disable, optional cache > writeback, CXL Reset initiation, ResetComplete polling, and ResetError > reporting. > > The userspace ABI is the existing PCI reset interface: > > /sys/bus/pci/devices/.../reset_method > /sys/bus/pci/devices/.../reset > > Userspace can select "cxl_reset" in reset_method and then trigger reset > through the existing reset attribute. > > CXL Reset is exposed conservatively. The target must be a > single-function CXL Type 2 device with CXL.cache and CXL.mem enabled, > and the kernel must have cached HDM state sufficient to validate affected > ranges and restore decoder programming. The reset preserves device > memory; it does not request CXL Reset Memory Clear. How does this provide multi-tenancy then? vfio-pci considers the device to be scrubbed of tenant data after a pci_reset_function() and I don't think Manish's series alters that. Thanks, Alex
On 9/9/26 2:13 PM, Alex Williamson wrote: > How does this provide multi-tenancy then? vfio-pci considers the > device to be scrubbed of tenant data after a pci_reset_function() and > I don't think Manish's series alters that. Thanks, > > Alex Hi Alex! I missed considering this. In V12 I'm exposing cxl_reset when the device advertises CXL Reset Memory Clear capability and sets RST_MEM_CLR_EN when initiating reset. I added the gate to prevent exposing a non-scrubbing reset to VFIO. -- Regards, Srirangan
© 2016 - 2026 Red Hat, Inc.