[RFC PATCH 02/17] cxl: docs/devices - device reference and uefi placeholder

Gregory Price posted 17 patches 9 months, 2 weeks ago
There is a newer version of this series
[RFC PATCH 02/17] cxl: docs/devices - device reference and uefi placeholder
Posted by Gregory Price 9 months, 2 weeks ago
Add a simple device primer sufficient to understand the theory
of operation documentation.

Add carve-out for CDAT with a TODO.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 .../driver-api/cxl/devices/device-types.rst   | 169 ++++++++++++++++++
 Documentation/driver-api/cxl/devices/uefi.rst |   9 +
 Documentation/driver-api/cxl/index.rst        |   2 +
 3 files changed, 180 insertions(+)
 create mode 100644 Documentation/driver-api/cxl/devices/device-types.rst
 create mode 100644 Documentation/driver-api/cxl/devices/uefi.rst

diff --git a/Documentation/driver-api/cxl/devices/device-types.rst b/Documentation/driver-api/cxl/devices/device-types.rst
new file mode 100644
index 000000000000..e8dd051c2c71
--- /dev/null
+++ b/Documentation/driver-api/cxl/devices/device-types.rst
@@ -0,0 +1,169 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Devices and Protocols
+#####################
+
+The type of CXL device (Memory, Accelerator, etc) dictates many configuration steps. This section
+covers some basic background on device types and on-device resources used by the platform and OS
+which impact configuration.
+
+Protocols
+*********
+
+There are three core protocols to CXL.  For the purpose of this documentation,
+we will only discuss very high level definitions as the specific hardware
+details are largely abstracted away from Linux.  See the CXL specification
+for more details.
+
+CXL.io
+======
+The basic interaction protocol, similar to PCIe configuration mechanisms.
+Typically used for initialization, configuration, and I/O access for anything
+other than memory (CXL.mem) or cache (CXL.cache) operations.
+
+The Linux CXL driver exposes access to .io functionalty via the various sysfs
+interfaces and /dev/cxl/ devices (which exposes direct access to device
+mailboxes).
+
+CXL.cache
+=========
+The mechanism by which a device may coherently access and cache host memory.
+
+Largely transparent to Linux once configured.
+
+CXL.mem
+=======
+The mechanism by which the CPU may coherently access and cache device memory.
+
+Largely transparent to Linux once configured.
+
+
+Device Types
+************
+
+Type-1
+======
+
+A Type-1 CXL device:
+
+* Supports cxl.io and cxl.cache protocols
+* Implements a fully coherent cache
+* Allow Device-to-Host coherence and Host-to-Device snoops.
+* Does NOT have host-managed device memory (HDM)
+
+Typical examples of type-1 devices is a Smart NIC - which may want to
+directly operate on host-memory (DMA) to store incoming packets. These
+devices largely rely on CPU-attached memory.
+
+Type-2
+======
+
+A Type-2 CXL Device:
+
+* Supports cxl.io, cxl.cache, and cxl.mem protocols
+* Optionally implements coherent cache and Host-Managed Device Memory
+* Is typically an accelerator device w/ high bandwidth memory.
+
+The primary difference between a type-1 and type-2 device is the presence
+of host-managed device memory, which allows the device to operate on a
+local memory bank - while the CPU sill has coherent DMA to the same memory.
+
+The allows things like GPUs to expose their memory via DAX devices or file
+descriptors, allows drivers and programs direct access to device memory
+rather than use block-transfer semantics.
+
+Type-3
+======
+
+A Type-3 CXL Device
+
+* Supports cxl.io and cxl.mem
+* Implements Host-Managed Device Memory
+* May provide either Volatile or Persistent memory capacity (or both).
+
+A basic example of a type-3 device is a simple memory expanded, whose
+local memory capacity is exposed to the CPU for access directly via
+basic coherent DMA.
+
+Switch
+======
+
+A CXL switch is a device capacity of routing any CXL (and by extension, PCIe)
+protocol between an upstream, downstream, or peer devices.  Many devices, such
+as Multi-Logical Devices, imply the presence of switching in some manner.
+
+Logical Devices and Heads
+=========================
+
+A CXL device may present one or more "Logical Devices" to one or more hosts
+(via physical "Heads").
+
+A Single-Logical Device (SLD) is a device which presents a single device to
+one or more heads.
+
+A Multi-Logical Device (MLD) is a device which may present multiple devices
+to one or more devices.
+
+A Single-Headed Device exposes only a single physical connection.
+
+A Multi-Headed Device exposes multiple physical connections.
+
+MHSLD
+-----
+A Multi-Headed Single-Logical Device (MHSLD) exposes a single logical
+device to multiple heads which may be connected to one or more discrete
+hosts.  An example of this would be a simple memory-pool which may be
+statically configured (prior to boot) to expose portions of its memory
+to Linux via the CEDT ACPI table.
+
+MHMLD
+-----
+A Multi-Headed Multi-Logical Device (MHMLD) exposes multiple logical
+devices to multiple heads which may be connected to one or more discrete
+hosts.  An example of this would be a Dynamic Capacity Device or which
+may be configured at runtime to expose portions of its memory to Linux.
+
+Example Devices
+***************
+
+Memory Expander
+===============
+The simplest form of Type-3 device is a memory expander.  A memory expander
+exposes Host-Managed Device Memory (HDM) to Linux.  This memory may be
+Volatile or Non-Volatile (Persistent).
+
+Memory Expanders will typically be considered a form of Single-Headed,
+Single-Logical Device - as its form factor will typically be an add-in-card
+(AIC) or some other similar form-factor.
+
+The Linux CXL driver provides support for static or dynamic configuration of
+basic memory expanders.  The platform may program decoders prior to OS init
+(e.g. auto-decoders), or the user may program the fabric if the platform
+defers these operations to the OS.
+
+Multiple Memory Expanders may be added to an external chassis and exposed to
+a host via a head attached to a CXL switch.  This is a "memory pool", and
+would be considered an MHSLD or MHMLD depending on the management capabilities
+provided by the switch platform.
+
+As of v6.14, Linux does not provide a formalized interface to manage non-DCD
+MHSLD or MHMLD devices.
+
+Dynamic Capacity Device (DCD)
+=============================
+
+A Dynamic Capacity Device is a Type-3 device which provides dynamic management
+of memory capacity. The basic premise of a DCD to provide an allocator-like
+interface for physical memory capacity to a "Fabric Manager" (an external,
+privileged host with privileges to change configurations for other hosts).
+
+A DCD manages "Memory Extents", which may be volatile or persistent. Extents
+may also be exclusive to a single host or shared across multiple.
+
+As of v6.14, Linux does not provide a formalized interface to manage DCD
+devices, however there is active work on LKML targeting future release.
+
+Example T2 Device
+=================
+
+Todo
diff --git a/Documentation/driver-api/cxl/devices/uefi.rst b/Documentation/driver-api/cxl/devices/uefi.rst
new file mode 100644
index 000000000000..a51583e6c44c
--- /dev/null
+++ b/Documentation/driver-api/cxl/devices/uefi.rst
@@ -0,0 +1,9 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+UEFI Data
+#########
+
+Coherent Device Attribute Table (CDAT)
+**************************************
+
+todo
diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-api/cxl/index.rst
index dfc0a4aa9003..4dc99a6b08bd 100644
--- a/Documentation/driver-api/cxl/index.rst
+++ b/Documentation/driver-api/cxl/index.rst
@@ -19,6 +19,8 @@ that have impacts on each other.  The docs here break up configurations steps.
    :maxdepth: 2
    :caption: Device Reference
 
+   devices/device-types
+   devices/uefi
    devices/theory-of-operation
 
 .. toctree::
-- 
2.49.0
Re: [RFC PATCH 02/17] cxl: docs/devices - device reference and uefi placeholder
Posted by Jonathan Corbet 9 months, 2 weeks ago
Gregory Price <gourry@gourry.net> writes:

> Add a simple device primer sufficient to understand the theory
> of operation documentation.
>
> Add carve-out for CDAT with a TODO.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
>  .../driver-api/cxl/devices/device-types.rst   | 169 ++++++++++++++++++
>  Documentation/driver-api/cxl/devices/uefi.rst |   9 +
>  Documentation/driver-api/cxl/index.rst        |   2 +
>  3 files changed, 180 insertions(+)
>  create mode 100644 Documentation/driver-api/cxl/devices/device-types.rst
>  create mode 100644 Documentation/driver-api/cxl/devices/uefi.rst

I'm glad to see all this documentation!  One little nit...

> diff --git a/Documentation/driver-api/cxl/devices/device-types.rst b/Documentation/driver-api/cxl/devices/device-types.rst
> new file mode 100644
> index 000000000000..e8dd051c2c71
> --- /dev/null
> +++ b/Documentation/driver-api/cxl/devices/device-types.rst
> @@ -0,0 +1,169 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +Devices and Protocols
> +#####################

If you could stick with the section markup conventions we have
established in Documentation/doc-guide/sphinx.rst, I'd appreciate it.

Thanks,

jon
Re: [RFC PATCH 02/17] cxl: docs/devices - device reference and uefi placeholder
Posted by Gregory Price 9 months, 2 weeks ago
On Wed, Apr 30, 2025 at 06:32:27AM -0600, Jonathan Corbet wrote:
> Gregory Price <gourry@gourry.net> writes:
> 
> > Add a simple device primer sufficient to understand the theory
> > of operation documentation.
> >
> > Add carve-out for CDAT with a TODO.
> >
> > Signed-off-by: Gregory Price <gourry@gourry.net>
> > ---
> >  .../driver-api/cxl/devices/device-types.rst   | 169 ++++++++++++++++++
> >  Documentation/driver-api/cxl/devices/uefi.rst |   9 +
> >  Documentation/driver-api/cxl/index.rst        |   2 +
> >  3 files changed, 180 insertions(+)
> >  create mode 100644 Documentation/driver-api/cxl/devices/device-types.rst
> >  create mode 100644 Documentation/driver-api/cxl/devices/uefi.rst
> 
> I'm glad to see all this documentation!  One little nit...
> 
> > diff --git a/Documentation/driver-api/cxl/devices/device-types.rst b/Documentation/driver-api/cxl/devices/device-types.rst
> > new file mode 100644
> > index 000000000000..e8dd051c2c71
> > --- /dev/null
> > +++ b/Documentation/driver-api/cxl/devices/device-types.rst
> > @@ -0,0 +1,169 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +Devices and Protocols
> > +#####################
> 
> If you could stick with the section markup conventions we have
> established in Documentation/doc-guide/sphinx.rst, I'd appreciate it.
>

Will do, mostly wanted to get it out for content first. Will give it
a once over for formatting now.

~Gregory