From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E4FF246786 for ; Mon, 12 May 2025 16:21:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066907; cv=none; b=hifA/E2oygn1JAgSmjS9W3bpKsA/C6pLCN2GznUqxRKVkT9ri/HsoKAZN8TpWyfjSR3EAKUzQo57a3rUxtPCWLApWs8MvQyGmK1E1qbm8XN50ztdcOp+or1ZgYBjopKl4NGCi3fxX8VLRDS3eIRllCA7YuyZSa8lPvKpH6r+wJI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066907; c=relaxed/simple; bh=bc+l52lzMgg34Vw41X2Wy/FxQj+yccNEeALSbRGK27Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=R60xT5l5lXwkbhJohLb9C57zaT4IrxraM4XkubBT8pw5TD7IHvOOrVIfhUET8MT+U8lCWS9kinL4ecrxDQe6bFOF9oXMgFPnhL+yakfc/C9Xvw8Csl9W4QpeE9Kt3ebAdtmXPYxmBaGt+5v89h1WWbs2UziEu+5r1MOCxFDxotU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=PDDmKXyj; arc=none smtp.client-ip=209.85.160.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="PDDmKXyj" Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-476f4e9cf92so38853191cf.3 for ; Mon, 12 May 2025 09:21:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066903; x=1747671703; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=U3rVHINvuxD50BlXL+fWePXRUBl5vPBCCVNAK+caSv0=; b=PDDmKXyjwC0Hi98kXwfZpOo9d1PQrNrP6UxGJAh5Yu4ndqilfWNNpL0rDAz0R9g1gb 8qtgdNlydJ3GaloDRqkRX4pXQTzFfjf7kiNGka/rW2VXWKi5n94pphP2KcJPXrEvsNs7 P2mEXZyVPyXouiH6cYRE1RQnocnbmIB865nr6S6zDQbCA+dCU7gtqutpdi2WkKwhXLl6 FlgFgZJf34QNTU419hu922O41Gyw1l5ETplsWLyE6v/H5mM6F8bvk7MrGT8uojKtUNwA 4e6Hx7KzouoqbHDAG6x9mdW5y3pHPzCILbRH4cDQsS+Sl8YIz0XQN0wjT0/NcmoaP+T5 UVUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066903; x=1747671703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U3rVHINvuxD50BlXL+fWePXRUBl5vPBCCVNAK+caSv0=; b=OxlMkIjQCVgSyQjwcGATixpMIdQ3eEZC+4zIElnmGBeqsvpYyIxIE2Xut5kFY3rgfo c8sh+gMxTYfBH8+AdCoY+PhfT86nlR6f6I/dqP8Jm1yM7y4TBo3IsLFaTmk+x8NBojVG St4e0jpnMcqsNLvXADL7zB+W7FXzbn0fq40cEWW0qaMwsN7M/JdDBJL4eDgJgirrgb9Y XJ4SJaXKoCGzv5Eg2puTyE+rJxbRJ0OdufDXPfI5qQu0kCez+oenTRzZE+xzz9NVqC+/ aDtFwzqa10tmEZmisu/geQP6NTLiqA27NhoeXBX3qJ/KILO/EOKKZTzjCj1/dQ5mmeJw 8LaQ== X-Forwarded-Encrypted: i=1; AJvYcCUk+5wdXml1ObUs0Uj8x8O9zQpqjftnWwosYK47VwSTEogSMoeaPWeCzF+f9Dos114mNGRVXLa7WAhXUfY=@vger.kernel.org X-Gm-Message-State: AOJu0YxVAb59c+msCnrOtEuhQWMXDe4Am3Gn+pKxlo+k1i524N9/Jlzw pIkhHT8zVmYbKkghkdrdlNVBwM48+uoaq5BfBoGNkRzFJd+fH9bw9sQUVbk8I00= X-Gm-Gg: ASbGnctL2Y2HEgmpFxNfJaMyAaX9vsKx4L+kj0nnkbkF8rqRi9JjB5BPLvYMyqC+Lby 7Zi6HKzP2aiwcyJlmUTxabHUagrwhZTDrZVfXv5vLyD1N2pavtSUGodaJaahzQ29XygG4W6E+GO jVdHBST46C/iDc+rIUxJGLZEZcCd/gHB/BRMJwz9muA/UuDk/Jjj5QwdepT/R8RQGv0mPgj73PD 26rxMlVrCXDvhC7Jil9gp5P/iq2N5juqY7epJy51U4mN6SMm3vV6hhvcz976JDPf6jrLMVnwMz8 SULH5Sf9q7c0Bb33jl9UEP+neP0irDr+7RlD4QjsP8hf04/WUkaH/+6yY1E4uSGokCT7J1Tghd9 9dILtvq6uNNWREAlQIHhkt0Rp2jmhqsG7Usqczy+pnZrIDt4= X-Google-Smtp-Source: AGHT+IHXhc25rBNfw4eftHDjphXiDL9wvYrUeit49NMjedMVQ+FbLTR0r4T1UKzJc3WYccg/k21O/A== X-Received: by 2002:a05:622a:553:b0:48a:2122:5047 with SMTP id d75a77b69052e-494527136f4mr201727911cf.8.1747066903326; Mon, 12 May 2025 09:21:43 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:43 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 01/17] cxl: update documentation structure in prep for new docs Date: Mon, 12 May 2025 12:21:18 -0400 Message-ID: <20250512162134.3596150-2-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Restructure the cxl folder to make adding docs per-page cleaner. Signed-off-by: Gregory Price Reviewed-by: Dave Jiang --- Documentation/driver-api/cxl/index.rst | 16 +++++++++++++--- .../cxl/{ =3D> linux}/access-coordinates.rst | 0 ...emory-devices.rst =3D> theory-of-operation.rst} | 10 +++++----- 3 files changed, 18 insertions(+), 8 deletions(-) rename Documentation/driver-api/cxl/{ =3D> linux}/access-coordinates.rst (= 100%) rename Documentation/driver-api/cxl/{memory-devices.rst =3D> theory-of-ope= ration.rst} (98%) diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 965ba90e8fb7..fe1594dc6778 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -4,12 +4,22 @@ Compute Express Link =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 +CXL device configuration has a complex handoff between platform (Hardware, +BIOS, EFI), OS (early boot, core kernel, driver), and user policy decisions +that have impacts on each other. The docs here break up configurations st= eps. + +.. toctree:: + :maxdepth: 2 + :caption: Overview + + theory-of-operation + maturity-map + .. toctree:: :maxdepth: 1 + :caption: Linux Kernel Configuration =20 - memory-devices - access-coordinates + linux/access-coordinates =20 - maturity-map =20 .. only:: subproject and html diff --git a/Documentation/driver-api/cxl/access-coordinates.rst b/Document= ation/driver-api/cxl/linux/access-coordinates.rst similarity index 100% rename from Documentation/driver-api/cxl/access-coordinates.rst rename to Documentation/driver-api/cxl/linux/access-coordinates.rst diff --git a/Documentation/driver-api/cxl/memory-devices.rst b/Documentatio= n/driver-api/cxl/theory-of-operation.rst similarity index 98% rename from Documentation/driver-api/cxl/memory-devices.rst rename to Documentation/driver-api/cxl/theory-of-operation.rst index d732c42526df..32739e253453 100644 --- a/Documentation/driver-api/cxl/memory-devices.rst +++ b/Documentation/driver-api/cxl/theory-of-operation.rst @@ -1,9 +1,9 @@ .. SPDX-License-Identifier: GPL-2.0 .. include:: =20 -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D -Compute Express Link Memory Devices -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Compute Express Link Driver Theory of Operation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 A Compute Express Link Memory Device is a CXL component that implements the CXL.mem protocol. It contains some amount of volatile memory, persistent m= emory, @@ -14,8 +14,8 @@ that optionally define a device's contribution to an inte= rleaved address range across multiple devices underneath a host-bridge or interleaved across host-bridges. =20 -CXL Bus: Theory of Operation -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +The CXL Bus +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Similar to how a RAID driver takes disk objects and assembles them into a = new logical device, the CXL subsystem is tasked to take PCIe and ACPI objects = and assemble them into a CXL.mem decode topology. The need for runtime configu= ration --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C82552522B8 for ; Mon, 12 May 2025 16:21:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066920; cv=none; b=jAF0xrWN4FfS9AZAmhYmhXytJC8BkyQWFvCuXUh9pvOG5U4+dpdK3fjT4KZA5vD+8Iy+DsHlK1+PG03xDMqo8djLCbmQbk+uf8gpDT04t1LHrJQIHkiSj9np+ipjm2n5MrMH0XSOPUSkxAjF0KmgxNM1SZpVGQHLZwa39HiY/ow= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066920; c=relaxed/simple; bh=DoI5kxJikAM/qpIRAtn9fjxrQS8+dT5cbJX/H3hs68w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Y5GDUalXyXDYI/uLfY+vU6w/rHLLeqZYxKcGIId0ziywivfzpZsRKcZNUughVb1CFIt2Z0mTHa6JQb3qwy6FzU1NN2ff1WvjcFO4/XrKDcHzZQtsGVOxckykPV+KVPYX2navGmDC6LwVT3KmJUcRTog/CeE8P7If8Eb4rH8KBAc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=T11VnCbF; arc=none smtp.client-ip=209.85.222.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="T11VnCbF" Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-7caea4bc9e9so893510885a.1 for ; Mon, 12 May 2025 09:21:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066916; x=1747671716; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sNILEeZYlBJQYm8rxVf5XnkRfrtj2f63uIZxnPhs6Qs=; b=T11VnCbFRPyHD4HaXeoavsqFfREJZpXTL/5dnngo7KylDBShtZ6689/TFf2dvDz1fD yih49sm2N9ze/p5IT++yOpphrutlzwkydXKBZp4bA3aAdvHDcnkyJb7RayJsS+Iwuqn7 ivpk7lWohsG7Sa2GLFXeDD9QX7vxmPVbwCO7XcoM44PJZQK5rX0i1e/A6GyApss9TXCv j7HKiCCev5mtJA5Azaav1zSLKUJt4DwtKG+JnyQE2zThGsRFs7MmC+eA7fX8SwlBkgrG jKt23Dbpz0E49qcBvPj7bN2g5u1SCAdRbW7aaoCZNrfNiGzNEbGhdEitg3pIHCu+Qu3X hKtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066916; x=1747671716; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sNILEeZYlBJQYm8rxVf5XnkRfrtj2f63uIZxnPhs6Qs=; b=qWIrIUjYWho/cTPExp3UZOQfg6uLXM0sacqlOlPvVYkZk6vDKXP3f4P659TAL0VWhB 5GkNnrl1Z8pvE/0vBZew6ZapR4vguckNgGLFUUZmx12iyCVnU64Ty/mZUj2xOkW84I3W IuyJpPDA8ODlnmdritiVm+1Caqv7iCbTwITab6A/0lDDJptbWAAaTptDATGcItzyR+hn Om2pumMtOV16M6DWXatmLLmuj2nmFvvGi6wcgDGlI4kjyl5MmO1zc2gm9KjcvCcTTYwl deZayBgbdIofszSsdlH3fdLtA2L1YuWwjFzawDIAgvF6ZcPfG7F3qNRXumMlW2N5J0bA EBYw== X-Forwarded-Encrypted: i=1; AJvYcCXhZ5rnQtq4dyGCe5SFUjFtM5XIlGIPM8zEes4lQUII9pHhcny4yI5KFFFoBSn4A0vYr1cYG8gS8s2nxrg=@vger.kernel.org X-Gm-Message-State: AOJu0Yxo1pXQ3CsyPctVO5WtZraHc//86ffTN0M1mkOgMAtCxeE+cpvz Vra4naOIljDlFBLwOuiAoENfNF5fIXcj/8CAw23H6nFWYGrmk8tx9bhHyqGaEnSw12bnbsEMHUS t X-Gm-Gg: ASbGncsmBygjRwRnPSE68f1gRp9bYjGTICHO+ZJk8bgUHQwKQkJZq2i4JwjWRfg2aXr ht/VhqZXLNi/Pfd0TEwMPEZ1TjkQ6w142XANZxW1dP+cynhn9s9fzaXHN/OVmtbreHmBs9ItAaR O2Wew0IKPn1WdOac75K5kwTCfI6Cwu2nDBcYRkVN4yAABwqX8H9WmwzivsS9p6f7KBeAr79rY7n tYADpIrLnW/cIqJBuyUDWJdC7PFbxsZRf4mT6gAmPOTo21eGBGHzN1j+BXz4ANNCLPqk6PeCTm4 /4UDn9UUX2RBSomBUMMOLp4UBarvzaRGgqmm2eoPgh5H4x3N07GKxbsL14jCL6WA8lsLcYAlMZG e+nQR8lHredCFFCS1NE2gDN7R6PUt1OKN75Rj X-Google-Smtp-Source: AGHT+IFwl36mLDo7FZkCUHoGJHyTy2zewdmXD6A4NvIgp/fn+qZTsyQP+IQVYzuIHMVoFWLDW+xitg== X-Received: by 2002:a05:622a:1aa0:b0:476:8cad:72e0 with SMTP id d75a77b69052e-49452734c55mr177946951cf.15.1747066905232; Mon, 12 May 2025 09:21:45 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:44 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net, Randy Dunlap , Bagas Sanjaya Subject: [PATCH v3 02/17] cxl: docs - access-coordinates doc fixups Date: Mon, 12 May 2025 12:21:19 -0400 Message-ID: <20250512162134.3596150-3-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Place the hierarchy diagram in access-coordinates.rst in a code block. Fix a few grammar issues. Suggested-by: Randy Dunlap Suggested-by: Bagas Sanjaya Signed-off-by: Gregory Price Reviewed-by: Dave Jiang --- .../cxl/linux/access-coordinates.rst | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/Documentation/driver-api/cxl/linux/access-coordinates.rst b/Do= cumentation/driver-api/cxl/linux/access-coordinates.rst index b07950ea30c9..e408ecbc4038 100644 --- a/Documentation/driver-api/cxl/linux/access-coordinates.rst +++ b/Documentation/driver-api/cxl/linux/access-coordinates.rst @@ -26,20 +26,20 @@ There can be multiple switches under an RP. There can b= e multiple RPs under a CXL Host Bridge (HB). There can be multiple HBs under a CXL Fixed Memory Window Structure (CFMWS). =20 -An example hierarchy: +An example hierarchy:: =20 -> CFMWS 0 -> | -> _________|_________ -> | | -> ACPI0017-0 ACPI0017-1 -> GP0/HB0/ACPI0016-0 GP1/HB1/ACPI0016-1 -> | | | | -> RP0 RP1 RP2 RP3 -> | | | | -> SW 0 SW 1 SW 2 SW 3 -> | | | | | | | | -> EP0 EP1 EP2 EP3 EP4 EP5 EP6 EP7 + CFMWS 0 + | + _________|_________ + | | + ACPI0017-0 ACPI0017-1 + GP0/HB0/ACPI0016-0 GP1/HB1/ACPI0016-1 + | | | | + RP0 RP1 RP2 RP3 + | | | | + SW 0 SW 1 SW 2 SW 3 + | | | | | | | | + EP0 EP1 EP2 EP3 EP4 EP5 EP6 EP7 =20 Computation for the example hierarchy: =20 @@ -82,8 +82,8 @@ this point all the bandwidths are aggregated per each hos= t bridge, which is also the index for the resulting xarray. =20 The next step is to take the min() of the per host bridge bandwidth and the -bandwidth from the Generic Port (GP). The bandwidths for the GP is retriev= ed -via ACPI tables SRAT/HMAT. The min bandwidth are aggregated under the same +bandwidth from the Generic Port (GP). The bandwidths for the GP are retrie= ved +via ACPI tables SRAT/HMAT. The minimum bandwidth are aggregated under the = same ACPI0017 device to form a new xarray. =20 Finally, the cxl_region_update_bandwidth() is called and the aggregated --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A17E524C098 for ; Mon, 12 May 2025 16:21:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066922; cv=none; b=LCH+zEvN9u8endvWIQxHT70EyWny2zmf6Dc3K0LJDaeBXFHWwmpRhbIGO2Os0JJXjhz+xQb92lu8S1NptHsqpb7A+cLKVIUG5V3BP1kJGGzb8dkQrR+FnToSCBA5rUkX+ODTNeOJfVdKuVo40BwN+W59A0P13uSgzPmbUhLwhsU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066922; c=relaxed/simple; bh=5OMS59Fls7yssVe/eBHwNamMLgl3UdidmTKUG9C1bZ8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lPxdSWCPebzHRcqt9cQeQn4q/coRxu7ckZ02kbjj/L0r4ASYTpUInjArgXqFYGzkctHo6O4V4yjyhUmLEctWfD5OcOZ/k+qXC/OJVdbvAYMTKgFEyqqAA5TyHTUcFdwomq5f3QeTxP+venYaliA3dDpkQGI5Ki+cZVZVT4wCyyg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=mFsOy4j0; arc=none smtp.client-ip=209.85.222.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="mFsOy4j0" Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7c53b9d66fdso706846685a.3 for ; Mon, 12 May 2025 09:21:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066918; x=1747671718; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Jt/521w64tsazt1m4c92762k12ji8Xm+LaBsIndvwlE=; b=mFsOy4j0KvZ0bocaq58Obuup0nHhCHj3IgvdtUDb5fxmpVd6r7hHigAw9e11FPk6YD Cb4i6nr0MsRUF+4K5SXE9VFbUlefd9f3BB02RwcxX0BW3fbChWVf5xhE0ypCLfiQVZAh gBJQ9DGea3goKPqyg8/xSB/i5VOOYsGyMvZie+swdflErQffxdbgjCH5UNEXIyIi0beR /ePzKNLtwyeCQrfBkDPS0E3PHOdM6yW1EIx+U2H7NUh3s+mmaFuLHqGFkYTtVtZ64Hi+ 4inFfMPOmpncr8e7bzDNg1jv1h6H8PzgxVqaCQGJAQLF+0fFQ0rb2M8EIPsuTrY7jcXn VxWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066918; x=1747671718; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Jt/521w64tsazt1m4c92762k12ji8Xm+LaBsIndvwlE=; b=NCuUFugDlyVv48Ho8RsQW+WUtEkBIQ6LDH8+zxmScUqXl7DKmqvB3Jvkg+Y+0ZHD4q XFeoBaRrj7BHu69ytO3pBsORwUENDiGPrtR07AAeZqkgMhwhwaQ22Vs6rQ53tjoZeZLP RMegxSDRdB7ZfgkE3DdPiHWmEK4/5CLe2VwyKVKuozByNKjTHD7lZnLxQ7pSk76RvG9Y gJmR7t5W+MFUlveAIIW7BhFfiIET1eDwQy8uGxzMw47A/A14uAcAR/zy98+A72c39pzG X5b3yaBwiuuftui8p0b3rtkwPZhw7lcF92TmFI6tn1EmyAlAf/a8yYYsqqFLHl7MCXu3 m4FA== X-Forwarded-Encrypted: i=1; AJvYcCUj6O8cG2YHeNF13kkD9T41wkpGUXTLShxYqrnGEf72ukZ1nMrAGngyc0D6Bn4T0N7ytM97Tcb91YCfcVc=@vger.kernel.org X-Gm-Message-State: AOJu0Yy8SG+nEUFY2k8qBallXiEMrwgMQQ72wWynq9Mi5I4faHY1Qzd+ XgR/omMDTEo7BNd8Q2w5IEXoRqUI0BDEYmA2DxGUvldQt1VvK8cSgvomOJLaLas0+fnxbfQjNNp Z X-Gm-Gg: ASbGncvj+zMHQ0+HlRmS5d1VIsm6kHxUotilkbu6DeQlDxaxJzODzxM2Oso4PBvjmxw 0Y0EGudkDqO9qSa3d2WTNX7NUYHLPOwZBy4yGyXgGWfaJ6CpBPrI9sFVkZVtvIhtLVaTCB1hRe3 WZA8ORDQJpgaClpZfY47w/2FtlMDXlTRt8PzIX92BfCYAqLvfi5GQxqIf1EaD0umlJ5RbkN0HST O1uS6bv/jUNLUXJi5r4e02A2Tz7M9YyzLFtyTaTGFkNMMBRaRa6vJKduR0vEnurtxtW9YUpBJPl FE87WkeimQIHuhmLv1EYHczuhTaveeRYC2wdru6aI6i1bFfhd1iYFI/9DHM45ite6m3xxpcvbQW o+RrfjzpSJPgTGo2TQo/I6w61sXdSvIKF5FzkkgIIbt3hmwk= X-Google-Smtp-Source: AGHT+IGgq4dX8u+VS136dy5RsF6lNNfBNEBuuVYsuUsKinYkKNcF5+J+dn7L3DuanbNeTbM5peQhrw== X-Received: by 2002:a05:6214:400b:b0:6e4:2dcb:33c8 with SMTP id 6a1803df08f44-6f6e4802e14mr217368156d6.29.1747066906925; Mon, 12 May 2025 09:21:46 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:46 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 03/17] cxl: docs/devices - add cxl device and protocol reference Date: Mon, 12 May 2025 12:21:20 -0400 Message-ID: <20250512162134.3596150-4-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a simple device primer sufficient to understand the theory of operation documentation. Signed-off-by: Gregory Price --- .../driver-api/cxl/devices/device-types.rst | 165 ++++++++++++++++++ Documentation/driver-api/cxl/index.rst | 6 + 2 files changed, 171 insertions(+) create mode 100644 Documentation/driver-api/cxl/devices/device-types.rst diff --git a/Documentation/driver-api/cxl/devices/device-types.rst b/Docume= ntation/driver-api/cxl/devices/device-types.rst new file mode 100644 index 000000000000..c70564cf0be3 --- /dev/null +++ b/Documentation/driver-api/cxl/devices/device-types.rst @@ -0,0 +1,165 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Devices and Protocols +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The type of CXL device (Memory, Accelerator, etc) dictates many configurat= ion steps. This section +covers some basic background on device types and on-device resources used = by the platform and OS +which impact configuration. + +Protocols +=3D=3D=3D=3D=3D=3D=3D=3D=3D + +There are three core protocols to CXL. For the purpose of this documentat= ion, +we will only discuss very high level definitions as the specific hardware +details are largely abstracted away from Linux. See the CXL specification +for more details. + +CXL.io +------ +The basic interaction protocol, similar to PCIe configuration mechanisms. +Typically used for initialization, configuration, and I/O access for anyth= ing +other than memory (CXL.mem) or cache (CXL.cache) operations. + +The Linux CXL driver exposes access to .io functionalty via the various sy= sfs +interfaces and /dev/cxl/ devices (which exposes direct access to device +mailboxes). + +CXL.cache +--------- +The mechanism by which a device may coherently access and cache host memor= y. + +Largely transparent to Linux once configured. + +CXL.mem +--------- +The mechanism by which the CPU may coherently access and cache device memo= ry. + +Largely transparent to Linux once configured. + + +Device Types +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Type-1 +------ + +A Type-1 CXL device: + +* Supports cxl.io and cxl.cache protocols +* Implements a fully coherent cache +* Allows Device-to-Host coherence and Host-to-Device snoops. +* Does NOT have host-managed device memory (HDM) + +Typical examples of type-1 devices is a Smart NIC - which may want to +directly operate on host-memory (DMA) to store incoming packets. These +devices largely rely on CPU-attached memory. + +Type-2 +------ + +A Type-2 CXL Device: + +* Supports cxl.io, cxl.cache, and cxl.mem protocols +* Optionally implements coherent cache and Host-Managed Device Memory +* Is typically an accelerator device w/ high bandwidth memory. + +The primary difference between a type-1 and type-2 device is the presence +of host-managed device memory, which allows the device to operate on a +local memory bank - while the CPU sill has coherent DMA to the same memory. + +The allows things like GPUs to expose their memory via DAX devices or file +descriptors, allows drivers and programs direct access to device memory +rather than use block-transfer semantics. + +Type-3 +------ + +A Type-3 CXL Device + +* Supports cxl.io and cxl.mem +* Implements Host-Managed Device Memory +* May provide either Volatile or Persistent memory capacity (or both). + +A basic example of a type-3 device is a simple memory expander, whose +local memory capacity is exposed to the CPU for access directly via +basic coherent DMA. + +Switch +------ + +A CXL switch is a device capacity of routing any CXL (and by extension, PC= Ie) +protocol between an upstream, downstream, or peer devices. Many devices, = such +as Multi-Logical Devices, imply the presence of switching in some manner. + +Logical Devices and Heads +------------------------- + +A CXL device may present one or more "Logical Devices" to one or more hosts +(via physical "Heads"). + +A Single-Logical Device (SLD) is a device which presents a single device to +one or more heads. + +A Multi-Logical Device (MLD) is a device which may present multiple devices +to one or more devices. + +A Single-Headed Device exposes only a single physical connection. + +A Multi-Headed Device exposes multiple physical connections. + +MHSLD +~~~~~ +A Multi-Headed Single-Logical Device (MHSLD) exposes a single logical +device to multiple heads which may be connected to one or more discrete +hosts. An example of this would be a simple memory-pool which may be +statically configured (prior to boot) to expose portions of its memory +to Linux via the CEDT ACPI table. + +MHMLD +~~~~~ +A Multi-Headed Multi-Logical Device (MHMLD) exposes multiple logical +devices to multiple heads which may be connected to one or more discrete +hosts. An example of this would be a Dynamic Capacity Device or which +may be configured at runtime to expose portions of its memory to Linux. + +Example Devices +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Memory Expander +--------------- +The simplest form of Type-3 device is a memory expander. A memory expander +exposes Host-Managed Device Memory (HDM) to Linux. This memory may be +Volatile or Non-Volatile (Persistent). + +Memory Expanders will typically be considered a form of Single-Headed, +Single-Logical Device - as its form factor will typically be an add-in-card +(AIC) or some other similar form-factor. + +The Linux CXL driver provides support for static or dynamic configuration = of +basic memory expanders. The platform may program decoders prior to OS init +(e.g. auto-decoders), or the user may program the fabric if the platform +defers these operations to the OS. + +Multiple Memory Expanders may be added to an external chassis and exposed = to +a host via a head attached to a CXL switch. This is a "memory pool", and +would be considered an MHSLD or MHMLD depending on the management capabili= ties +provided by the switch platform. + +As of v6.14, Linux does not provide a formalized interface to manage non-D= CD +MHSLD or MHMLD devices. + +Dynamic Capacity Device (DCD) +----------------------------- + +A Dynamic Capacity Device is a Type-3 device which provides dynamic manage= ment +of memory capacity. The basic premise of a DCD to provide an allocator-like +interface for physical memory capacity to a "Fabric Manager" (an external, +privileged host with privileges to change configurations for other hosts). + +A DCD manages "Memory Extents", which may be volatile or persistent. Exten= ts +may also be exclusive to a single host or shared across multiple hosts. + +As of v6.14, Linux does not provide a formalized interface to manage DCD +devices, however there is active work on LKML targeting future release. diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index fe1594dc6778..a2d1c5b18a8a 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -15,6 +15,12 @@ that have impacts on each other. The docs here break up= configurations steps. theory-of-operation maturity-map =20 +.. toctree:: + :maxdepth: 2 + :caption: Device Reference + + devices/device-types + .. toctree:: :maxdepth: 1 :caption: Linux Kernel Configuration --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E172524C09C for ; Mon, 12 May 2025 16:21:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066911; cv=none; b=pJnMoiEVETo9aMbTD6BC925qguIrg/GQMRVLA+oLHuNH+qWw3hJAgeL1SlH+XDdZNsz3Zej+zW68ZEFehIG4A+//3BAgqGgcVYj9UiEUV7r9HXs5Krk/rDaRlFTPh2bCUcOS7EFpWGkLzlc6W9MVB4ymeFvUKy7JQEIY1SoIQAs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066911; c=relaxed/simple; bh=wF1Wn6uogxrZLs0egXNeQW8IAGTYKa8Ie8NiQvbbfCA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FJDeftk8WSsocfbFCbsxh7OTNLujCCfIaUuQijR8W+E2HKvGv5pwj0s1kjNYGfnfnEAnOovsj20yTAhUnKaJH0bMFoN4Ka8VjR5YOZOq1pfPBFy9HMoLSNqS5PLzGcZikzhR29RQOHV+x3pgIYMwaaA624wknkG7sxPvK7rctR4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=RW0d/bQD; arc=none smtp.client-ip=209.85.160.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="RW0d/bQD" Received: by mail-qt1-f180.google.com with SMTP id d75a77b69052e-47690a4ec97so59023801cf.2 for ; Mon, 12 May 2025 09:21:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066909; x=1747671709; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RP9DathHldScOCsDJTBc7upQEKyGa+tOLTpLEorT/Ys=; b=RW0d/bQDHQyyBN4VgmEWG4GgIuzZQs8n5J0N3tbazVjgQXrND6c374KB4hzL56DnEi 9AGWR2lNJ4E/9f0Mo6YmIGjv/WYk76jT/T6o+rBYpojdiRVi6853nzMFAXEQOf8vtOES OImCO8SsTmhj8zj4RSYJb7mmswpgxtpm3Dwxmu0Q5wkNERatgTdbnQxFiId5EdlpWgv4 5upw0sCCjXNUGazVo8qtjPUtrPaq/LaT+qxx35e09AKnPRBUG0e+lGIarKsZzj+oLMkN ARSWWChXizR2LIdHpvxefwWNoqvN871WlpY0AwHNpzoZr3JWq1cpnNjCMB0yKBcKl2db aBPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066909; x=1747671709; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RP9DathHldScOCsDJTBc7upQEKyGa+tOLTpLEorT/Ys=; b=sxaFTVZiB8VOIBBag+kS1Li/YFKKyv4HpDoxJON9hWMyIvMWFgE6TYDzxmNW9ZyJnI ZsA/OG4QW/ZSvY8E8mxjYV/A8+Ewqmcj7fOxVTDgenxIDShHN99FmAVomUU2OsI2mT39 wOWVpFMh5R7r/Ly4qG63jn37d1SW2DLIaMh0Qp+HiA9dfwUk/pcrDs2LcaoBIlIMfSek HeUO1ekD8V6j50TcKOjJUwNQwm2qk3F387GMShw1n0dYiY5vJcf64pSTIk6mxlu1mmD3 7gM+6b4ykaveyCllQbuGQQF8/JXaOocKBhrHwTtYVSGqO+RiZLq2ASVET6JAqt5kRIBG erYQ== X-Forwarded-Encrypted: i=1; AJvYcCUaA7yEsjRZsHlzUkXPU+WJVVQT7a1Na7xxi1MikPPF2gN0+XweS/ndGs7Wdiuvcvp9Lr7lMk7HLt98UHc=@vger.kernel.org X-Gm-Message-State: AOJu0YwrdWl9zu4a6AIM4x6DhXh75nVUlJxFZdk9y71kbM/rLpqgXFSZ /UrqoQk5NZrEPssnuhW79en6pAo8LrQc2ldlxsgk2pcgQKjbUQ5Y+CTuEvE7ELM= X-Gm-Gg: ASbGncsUYrHSjvpzFX9igBg23NHd0QOnUQ3HHYGeKQsC4memd8m8b9Z1vvS/qw+Jwqm x0EO7wNHGXsFKaU7URDFTzzhDEvx5ulVG4v05ytlNnG/XdPnr6b9KydN9NOH1yrej9t56/6qeUs CCXg9AuzSagoK4C3BipakpCZg32EtyNQO24qY0SsfuAY6eTg1YUNXSAUYVCDqe9HfIN11wNQF+m j3g40oJj1mxQpHspFlFwtqF9ckovl+NzVUbx8s7WPKJ14P5bqJUhJjpMhKgKKs7BuPhsyPJ1Ma0 rjRcDB2/srHzH2PXYpp3l9nzyWmylux7jFAvaGgC1Apk34FEl4xvgkDWiy/YS7clVeY1sWYvZSw UufqtN6Czd0bsrx9CLE4H9sC14OWUlAnV0RoS X-Google-Smtp-Source: AGHT+IEUp9iPKick/+CTSjfkLaVSn4+KhiJqqKe4wlaHm4OX2t63Mp7RNtbQYu02g8JBVeTg55d1zQ== X-Received: by 2002:ac8:5885:0:b0:476:a3c8:c78d with SMTP id d75a77b69052e-494527637f9mr182733071cf.29.1747066908469; Mon, 12 May 2025 09:21:48 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:48 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 04/17] cxl: docs/platform/bios-and-efi documentation Date: Mon, 12 May 2025 12:21:21 -0400 Message-ID: <20250512162134.3596150-5-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add some docs on CXL configurations done in bios/efi that affect linux configuration - information vendors may care to consider. Signed-off-by: Gregory Price Reviewed-by: Dave Jiang --- Documentation/driver-api/cxl/index.rst | 6 + .../driver-api/cxl/platform/bios-and-efi.rst | 262 ++++++++++++++++++ 2 files changed, 268 insertions(+) create mode 100644 Documentation/driver-api/cxl/platform/bios-and-efi.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index a2d1c5b18a8a..ffa0462ad950 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -21,6 +21,12 @@ that have impacts on each other. The docs here break up= configurations steps. =20 devices/device-types =20 +.. toctree:: + :maxdepth: 2 + :caption: Platform Configuration + + platform/bios-and-efi + .. toctree:: :maxdepth: 1 :caption: Linux Kernel Configuration diff --git a/Documentation/driver-api/cxl/platform/bios-and-efi.rst b/Docum= entation/driver-api/cxl/platform/bios-and-efi.rst new file mode 100644 index 000000000000..552a83992bcc --- /dev/null +++ b/Documentation/driver-api/cxl/platform/bios-and-efi.rst @@ -0,0 +1,262 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +BIOS/EFI Configuration +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +BIOS and EFI are largely responsible for configuring static information ab= out +devices (or potential future devices) such that Linux can build the approp= riate +logical representations of these devices. + +At a high level, this is what occurs during this phase of configuration. + +* The bootloader starts the BIOS/EFI. + +* BIOS/EFI do early device probe to determine static configuration + +* BIOS/EFI creates ACPI Tables that describe static config for the OS + +* BIOS/EFI create the system memory map (EFI Memory Map, E820, etc) + +* BIOS/EFI calls :code:`start_kernel` and begins the Linux Early Boot proc= ess. + +Much of what this section is concerned with is ACPI Table production and +static memory map configuration. More detail on these tables can be found +under Platform Configuration -> ACPI Table Reference. + +.. note:: + Platform Vendors should read carefully, as this sections has recommenda= tions + on physical memory region size and alignment, memory holes, HDM interle= ave, + and what linux expects of HDM decoders trying to work with these featur= es. + +UEFI Settings +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +If your platform supports it, the :code:`uefisettings` command can be used= to +read/write EFI settings. Changes will be reflected on the next reboot. Kex= ec +is not a sufficient reboot. + +One notable configuration here is the EFI_MEMORY_SP (Specific Purpose) bit. +When this is enabled, this bit tells linux to defer management of a memory +region to a driver (in this case, the CXL driver). Otherwise, the memory is +treated as "normal memory", and is exposed to the page allocator during +:code:`__init`. + +uefisettings examples +--------------------- + +:code:`uefisettings identify` :: + + uefisettings identify + + bios_vendor: xxx + bios_version: xxx + bios_release: xxx + bios_date: xxx + product_name: xxx + product_family: xxx + product_version: xxx + +On some AMD platforms, the :code:`EFI_MEMORY_SP` bit is set via the :code:= `CXL +Memory Attribute` field. This may be called something else on your platfo= rm. + +:code:`uefisettings get "CXL Memory Attribute"` :: + + selector: xxx + ... + question: Question { + name: "CXL Memory Attribute", + answer: "Enabled", + ... + } + +Physical Memory Map +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Physical Address Region Alignment +--------------------------------- + +As of Linux v6.14, the hotplug memory system requires memory regions to be +uniform in size and alignment. While the CXL specification allows for mem= ory +regions as small as 256MB, the supported memory block size and alignment f= or +hotplugged memory is architecture-defined. + +A Linux memory blocks may be as small as 128MB and increase in powers of t= wo. + +* On ARM, the default block size and alignment is either 128MB or 256MB. + +* On x86, the default block size is 256MB, and increases to 2GB as the + capacity of the system increases up to 64GB. + +For best support across versions, platform vendors should place CXL memory= at +a 2GB aligned base address, and regions should be 2GB aligned. This also = helps +prevent the creating thousands of memory devices (one per block). + +Memory Holes +------------ + +Holes in the memory map are tricky. Consider a 4GB device located at base +address 0x100000000, but with the following memory map :: + + --------------------- + | 0x100000000 | + | CXL | + | 0x1BFFFFFFF | + --------------------- + | 0x1C0000000 | + | MEMORY HOLE | + | 0x1FFFFFFFF | + --------------------- + | 0x200000000 | + | CXL CONT. | + | 0x23FFFFFFF | + --------------------- + +There are two issues to consider: + +* decoder programming, and +* memory block alignment. + +If your architecture requires 2GB uniform size and aligned memory blocks, = the +only capacity Linux is capable of mapping (as of v6.14) would be the capac= ity +from `0x100000000-0x180000000`. The remaining capacity will be stranded, = as +they are not of 2GB aligned length. + +Assuming your architecture and memory configuration allows 1GB memory bloc= ks, +this memory map is supported and this should be presented as multiple CFMWS +in the CEDT that describe each side of the memory hole separately - along = with +matching decoders. + +Multiple decoders can (and should) be used to manage such a memory hole (s= ee +below), but each chunk of a memory hole should be aligned to a reasonable = block +size (larger alignment is always better). If you intend to have memory ho= les +in the memory map, expect to use one decoder per contiguous chunk of host +physical memory. + +As of v6.14, Linux does provide support for memory hotplug of multiple +physical memory regions separated by a memory hole described by a single +HDM decoder. + + +Decoder Programming +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +If BIOS/EFI intends to program the decoders to be statically configured, +there are a few things to consider to avoid major pitfalls that will +prevent Linux compatibility. Some of these recommendations are not +required "per the specification", but Linux makes no guarantees of support +otherwise. + + +Translation Point +----------------- +Per the specification, the only decoders which **TRANSLATE** Host Physical +Address (HPA) to Device Physical Address (DPA) are the **Endpoint Decoders= **. +All other decoders in the fabric are intended to route accesses without +translating the addresses. + +This is heavily implied by the specification, see: :: + + CXL Specification 3.1 + 8.2.4.20: CXL HDM Decoder Capability Structure + - Implementation Note: CXL Host Bridge and Upstream Switch Port Decoder = Flow + - Implementation Note: Device Decoder Logic + +Given this, Linux makes a strong assumption that decoders between CPU and +endpoint will all be programmed with addresses ranges that are subsets of +their parent decoder. + +Due to some ambiguity in how Architecture, ACPI, PCI, and CXL specificatio= ns +"hand off" responsibility between domains, some early adopting platforms +attempted to do translation at the originating memory controller or host +bridge. This configuration requires a platform specific extension to the +driver and is not officially endorsed - despite being supported. + +It is *highly recommended* **NOT** to do this; otherwise, you are on your = own +to implement driver support for your platform. + +Interleave and Configuration Flexibility +---------------------------------------- +If providing cross-host-bridge interleave, a CFMWS entry in the CEDT must = be +presented with target host-bridges for the interleaved device sets (there = may +be multiple behind each host bridge). + +If providing intra-host-bridge interleaving, only 1 CFMWS entry in the CED= T is +required for that host bridge - if it covers the entire capacity of the de= vices +behind the host bridge. + +If intending to provide users flexibility in programming decoders beyond t= he +root, you may want to provide multiple CFMWS entries in the CEDT intended = for +different purposes. For example, you may want to consider adding: + +1) A CFMWS entry to cover all interleavable host bridges. +2) A CFMWS entry to cover all devices on a single host bridge. +3) A CFMWS entry to cover each device. + +A platform may choose to add all of these, or change the mode based on a B= IOS +setting. For each CFMWS entry, Linux expects descriptions of the described +memory regions in the SRAT to determine the number of NUMA nodes it should +reserve during early boot / init. + +As of v6.14, Linux will create a NUMA node for each CEDT CFMWS entry, even= if +a matching SRAT entry does not exist; however, this is not guaranteed in t= he +future and such a configuration should be avoided. + +Memory Holes +------------ +If your platform includes memory holes intersparsed between your CXL memor= y, it +is recommended to utilize multiple decoders to cover these regions of memo= ry, +rather than try to program the decoders to accept the entire range and exp= ect +Linux to manage the overlap. + +For example, consider the Memory Hole described above :: + + --------------------- + | 0x100000000 | + | CXL | + | 0x1BFFFFFFF | + --------------------- + | 0x1C0000000 | + | MEMORY HOLE | + | 0x1FFFFFFFF | + --------------------- + | 0x200000000 | + | CXL CONT. | + | 0x23FFFFFFF | + --------------------- + +Assuming this is provided by a single device attached directly to a host b= ridge, +Linux would expect the following decoder programming :: + + ----------------------- ----------------------- + | root-decoder-0 | | root-decoder-1 | + | base: 0x100000000 | | base: 0x200000000 | + | size: 0xC0000000 | | size: 0x40000000 | + ----------------------- ----------------------- + | | + ----------------------- ----------------------- + | HB-decoder-0 | | HB-decoder-1 | + | base: 0x100000000 | | base: 0x200000000 | + | size: 0xC0000000 | | size: 0x40000000 | + ----------------------- ----------------------- + | | + ----------------------- ----------------------- + | ep-decoder-0 | | ep-decoder-1 | + | base: 0x100000000 | | base: 0x200000000 | + | size: 0xC0000000 | | size: 0x40000000 | + ----------------------- ----------------------- + +With a CEDT configuration with two CFMWS describing the above root decoder= s. + +Linux makes no guarantee of support for strange memory hole situations. + +Multi-Media Devices +------------------- +The CFMWS field of the CEDT has special restriction bits which describe wh= ether +the described memory region allows volatile or persistent memory (or both)= . If +the platform intends to support either: + +1) A device with multiple medias, or +2) Using a persistent memory device as normal memory + +A platform may wish to create multiple CEDT CFMWS entries to describe the = same +memory, with the intent of allowing the end user flexibility in how that m= emory +is configured. Linux does not presently have strong requirements in this a= rea. --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6680E24E019 for ; Mon, 12 May 2025 16:21:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066914; cv=none; b=BqdIjgwkFYlAvgfAXzO1+UUH6FxDNs3YLGcEhoWrxGTB0Hwx4Gu1R5gYhJ+fryZHJd/0I2vkSFsAXFr9bx7G3UhS0XcBOVb1qPyb72oNqftkZwvan8InNldPshljmDe2k9X8vlhO3aHuwMEidLAzxHfAzKz4Nf2zEzyNfsRWXS0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066914; c=relaxed/simple; bh=RGhVDgyehlt1h7tJJr4eY1F6OZVuspdxbrtcQQV9us0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MbNJ829qvsf6bQouzE/Nzypes4pSNWZa+T56YG3pjShoKI6qQqFuGf5lXQF79WgRRr7vaRVwBHMewMIv+P3/AOJ5oZOzN1Hwnplbq65CnP5Giv0TD3YWT0yATYzsjmutEsMxVaw81aNCypl3hRYXBvRYIxh229MuxgU0wWE22B4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=cSlVw25l; arc=none smtp.client-ip=209.85.160.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="cSlVw25l" Received: by mail-qt1-f182.google.com with SMTP id d75a77b69052e-4774ce422easo51303231cf.1 for ; Mon, 12 May 2025 09:21:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066910; x=1747671710; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZmsySYwzr8ASsnG753Kt6yv0ZfK1vSiBlGJnWd4881Q=; b=cSlVw25l+w5KpIT/BWeIJEHRnZKiatpz+xmn43rD776CiOSklbzIbXO+2jq5qtNTFL o1hDclluHOsYEIEbTtPtvS/mI3A7BAj3O0qPKVe0o48kB1KuiXjr5seV5cPPrERob0s1 btL6GuXGDH9nKt5fn2MZ5nMOPlGjaNH5hPLZpXMonLJ1XRPyyWmh8rQCpuEeo5++bogc KhjyULUogI5AglZcZR4YSkS5feUmQtFGLP8ciTAGW9YDHpNVHV3k0BXbp6Z4hH/t9UEx hwArzv93f+yz/nCyBcx9A/zUrJxCOU4bJNDVSVl65Z/vukckjkZ9iRUrUqZrt/Axsvhd 2laQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066910; x=1747671710; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZmsySYwzr8ASsnG753Kt6yv0ZfK1vSiBlGJnWd4881Q=; b=m91GLODDZoFJO4KhxteRo/HWei6fVLi1h59TqxujGJhfH0TYNTc8cYSsQlcoDyl1gy bArjS0UV+hsQ0VZyjwIiMEE3pcBYqM7J0HgkeQZzJpOBOXQBrsUDkLwwhqTOMqzlGNq0 67yct6PdAM+HEkpy7guwqO1na0i/s6WDUc/jqS9MknLdOKPJicuI0Zpl7ZNuVlMFnH7h YRh5ATJM0qAB5M0CyXYJ5nwLGZ7bmiU0VVo6wWhgyq3EK3jzd/nbhcraenMmDWh/MxyY wMVrSYqmp3VgaKRk7DYztXkctG/jHE1HWz9RhLwsWxhgfQN5O0ZUoEUPRfRSq/BPnugg m6fw== X-Forwarded-Encrypted: i=1; AJvYcCXXps01cBEE2zU7sAcBvrYPaXgoKwlzTLI5prlezW40N/DNafSpkfn8rpb8/rUpSWYZgqevJkjwsquc6l8=@vger.kernel.org X-Gm-Message-State: AOJu0YwtjQD2rsPjDqqbkkFK4UO8GNrIc6etFbKuGKTIp1ehg2JTjw1v R1Z4dUcF5CqpC1VtNKmyiv5Q7VqLZ5mvCcxkQCc49e4ET0ahMtfAZXX/0PJHUI0= X-Gm-Gg: ASbGncvT3xVVkZgyVtaNjHlAkm0BQf2FpzZux3+DWrNQAtIGv2AtGcZFXlopv5LyRM/ wEGWY3AN/yA39Mf7rVErnYv6qrTHUPydigonKl6B3LXKkiy5MC9349RV63MkuumjBifvYxTzZuf NXs2HtQnHpo+Nqv/zcmNHjPird/UO9QcvA8jYuE1pctY7ISE/9kCbExmYpdBjLRLcjpphpPK39E EZ8rNqiVvUk0I0wCAUvLB9P3CiR7qDQgr5A40ZTM2YW474RYJtl06yv+nI4B5N0JVAEtr5hSqxI cG5HPeelQmUbf3I+hN7MXnlUEC5ZPwoqwVhihee6a16NYLctK0Fwh0G1XWyb8VUxnfo31px0bUm jlKAtEBDMAxu0O9l5QOyC+RpehWssOOhsOacD X-Google-Smtp-Source: AGHT+IF+OivR0X4TPDn7AvlMeNBQTyORAez0itQvzGPdTovbk4Yo4gDHd+0NMcBEZbAsD4L74z3+ow== X-Received: by 2002:ac8:5e07:0:b0:476:8eb5:1669 with SMTP id d75a77b69052e-494527d49bfmr219231491cf.32.1747066910149; Mon, 12 May 2025 09:21:50 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:49 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 05/17] cxl: docs/platform/acpi reference documentation Date: Mon, 12 May 2025 12:21:22 -0400 Message-ID: <20250512162134.3596150-6-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add basic ACPI table information needed to understand the CXL driver probe process. Signed-off-by: Gregory Price Reviewed-by: Dave Jiang --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/platform/acpi.rst | 76 +++++++++++++++++++ .../driver-api/cxl/platform/acpi/cedt.rst | 62 +++++++++++++++ .../driver-api/cxl/platform/acpi/dsdt.rst | 28 +++++++ .../driver-api/cxl/platform/acpi/hmat.rst | 32 ++++++++ .../driver-api/cxl/platform/acpi/slit.rst | 21 +++++ .../driver-api/cxl/platform/acpi/srat.rst | 44 +++++++++++ 7 files changed, 264 insertions(+) create mode 100644 Documentation/driver-api/cxl/platform/acpi.rst create mode 100644 Documentation/driver-api/cxl/platform/acpi/cedt.rst create mode 100644 Documentation/driver-api/cxl/platform/acpi/dsdt.rst create mode 100644 Documentation/driver-api/cxl/platform/acpi/hmat.rst create mode 100644 Documentation/driver-api/cxl/platform/acpi/slit.rst create mode 100644 Documentation/driver-api/cxl/platform/acpi/srat.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index ffa0462ad950..336322dc35a0 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -26,6 +26,7 @@ that have impacts on each other. The docs here break up = configurations steps. :caption: Platform Configuration =20 platform/bios-and-efi + platform/acpi =20 .. toctree:: :maxdepth: 1 diff --git a/Documentation/driver-api/cxl/platform/acpi.rst b/Documentation= /driver-api/cxl/platform/acpi.rst new file mode 100644 index 000000000000..ee7e6bd4c43d --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi.rst @@ -0,0 +1,76 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +ACPI Tables +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +ACPI is the "Advanced Configuration and Power Interface", which is a stand= ard +that defines how platforms and OS manage power and configure computer hard= ware. +For the purpose of this theory of operation, when referring to "ACPI" we w= ill +usually refer to "ACPI Tables" - which are the way a platform (BIOS/EFI) +communicates static configuration information to the operation system. + +The Following ACPI tables contain *static* configuration and performance d= ata +about CXL devices. + +.. toctree:: + :maxdepth: 1 + + acpi/cedt.rst + acpi/srat.rst + acpi/hmat.rst + acpi/slit.rst + acpi/dsdt.rst + +The SRAT table may also contain generic port/initiator content that is int= ended +to describe the generic port, but not information about the rest of the pa= th to +the endpoint. + +Linux uses these tables to configure kernel resources for statically confi= gured +(by BIOS/EFI) CXL devices, such as: + +- NUMA nodes +- Memory Tiers +- NUMA Abstract Distances +- SystemRAM Memory Regions +- Weighted Interleave Node Weights + +ACPI Debugging +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The :code:`acpidump -b` command dumps the ACPI tables into binary format. + +The :code:`iasl -d` command disassembles the files into human readable for= mat. + +Example :code:`acpidump -b && iasl -d cedt.dat` :: + + [000h 0000 4] Signature : "CEDT" [CXL Early Discovery Table] + +Common Issues +------------- +Most failures described here result in a failure of the driver to surface +memory as a DAX device and/or kmem. + +* CEDT CFMWS targets list UIDs do not match CEDT CHBS UIDs. +* CEDT CFMWS targets list UIDs do not match DSDT CXL Host Bridge UIDs. +* CEDT CFMWS Restriction Bits are not correct. +* CEDT CFMWS Memory regions are poorly aligned. +* CEDT CFMWS Memory regions spans a platform memory hole. +* CEDT CHBS UIDs do not match DSDT CXL Host Bridge UIDs. +* CEDT CHBS Specification version is incorrect. +* SRAT is missing regions described in CEDT CFMWS. + + * Result: failure to create a NUMA node for the region, or + region is placed in wrong node. + +* HMAT is missing data for regions described in CEDT CFMWS. + + * Result: NUMA node being placed in the wrong memory tier. + +* SLIT has bad data. + + * Result: Lots of performance mechanisms in the kernel will be very unha= ppy. + +All of these issues will appear to users as if the driver is failing to +support CXL - when in reality they are all the failure of a platform to +configure the ACPI tables correctly. diff --git a/Documentation/driver-api/cxl/platform/acpi/cedt.rst b/Document= ation/driver-api/cxl/platform/acpi/cedt.rst new file mode 100644 index 000000000000..1d9c9d3592dc --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi/cedt.rst @@ -0,0 +1,62 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D +CEDT - CXL Early Discovery Table +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D + +The CXL Early Discovery Table is generated by BIOS to describe the CXL mem= ory +regions configured at boot by the BIOS. + +CHBS +=3D=3D=3D=3D +The CXL Host Bridge Structure describes CXL host bridges. Other than desc= ribing +device register information, it reports the specific host bridge UID for t= his +host bridge. These host bridge ID's will be referenced in other tables. + +Example :: + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000007 <- Host bridge _UID + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010370400000 + Register length : 0000000000010000 + +CFMWS +=3D=3D=3D=3D=3D +The CXL Fixed Memory Window structure describes a memory region associated +with one or more CXL host bridges (as described by the CHBS). It addition= ally +describes any inter-host-bridge interleave configuration that may have been +programmed by BIOS. + +Example :: + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 000000C050000000 <- Memory Region + Window size : 0000003CA0000000 + Interleave Members (2^n) : 01 <- Interleave configuration + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 <- Host Bridge _UID + Next Target : 00000006 <- Host Bridge _UID + +The restriction field dictates what this SPA range may be used for (memory= type, +voltile vs persistent, etc). One or more bits may be set. :: + + Bit[0]: CXL Type 2 Memory + Bit[1]: CXL Type 3 Memory + Bit[2]: Volatile Memory + Bit[3]: Persistent Memory + Bit[4]: Fixed Config (HPA cannot be re-used) + +INTRA-host-bridge interleave (multiple devices on one host bridge) is NOT +reported in this structure, and is solely defined via CXL device decoder +programming (host bridge and endpoint decoders). diff --git a/Documentation/driver-api/cxl/platform/acpi/dsdt.rst b/Document= ation/driver-api/cxl/platform/acpi/dsdt.rst new file mode 100644 index 000000000000..b4583b01d67d --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi/dsdt.rst @@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +DSDT - Differentiated system Description Table +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +This table describes what peripherals a machine has. + +This table's UIDs for CXL devices - specifically host bridges, must be +consistent with the contents of the CEDT, otherwise the CXL driver will +fail to probe correctly. + +Example Compute Express Link Host Bridge :: + + Scope (_SB) + { + Device (S0D0) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */)= // _HID: Hardware ID + Name (_CID, Package (0x02) // _CID: Compatible ID + { + EisaId ("PNP0A08") /* PCI Express Bus */, + EisaId ("PNP0A03") /* PCI Bus */ + }) + ... + Name (_UID, 0x05) // _UID: Unique ID + ... + } diff --git a/Documentation/driver-api/cxl/platform/acpi/hmat.rst b/Document= ation/driver-api/cxl/platform/acpi/hmat.rst new file mode 100644 index 000000000000..095a26f02a37 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi/hmat.rst @@ -0,0 +1,32 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +HMAT - Heterogeneous Memory Attribute Table +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The Heterogeneous Memory Attributes Table contains information such as cac= he +attributes and bandwidth and latency details for memory proximity domains. +For the purpose of this document, we will only discuss the SSLIB entry. + +SLLBI +=3D=3D=3D=3D=3D +The System Locality Latency and Bandwidth Information records latency and +bandwidth information for proximity domains. + +This table is used by Linux to configure interleave weights and memory tie= rs. + +Example (Heavily truncated for brevity) :: + + Structure Type : 0001 [SLLBI] + Data Type : 00 <- Latency + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Entry : 0080 <- DRAM LTC + Entry : 0100 <- CXL LTC + + Structure Type : 0001 [SLLBI] + Data Type : 03 <- Bandwidth + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Entry : 1200 <- DRAM BW + Entry : 0200 <- CXL BW diff --git a/Documentation/driver-api/cxl/platform/acpi/slit.rst b/Document= ation/driver-api/cxl/platform/acpi/slit.rst new file mode 100644 index 000000000000..a56768e8fe41 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi/slit.rst @@ -0,0 +1,21 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +SLIT - System Locality Information Table +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The system locality information table provides "abstract distances" between +accessor and memory nodes. Node without initiators (cpus) are infinitely = (FF) +distance away from all other nodes. + +The abstract distance described in this table does not describe any real +latency of bandwidth information. + +Example :: + + Signature : "SLIT" [System Locality Information Table] + Localities : 0000000000000004 + Locality 0 : 10 20 20 30 + Locality 1 : 20 10 30 20 + Locality 2 : FF FF 0A FF + Locality 3 : FF FF FF 0A diff --git a/Documentation/driver-api/cxl/platform/acpi/srat.rst b/Document= ation/driver-api/cxl/platform/acpi/srat.rst new file mode 100644 index 000000000000..56d7bbb18c3b --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi/srat.rst @@ -0,0 +1,44 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +SRAT - Static Resource Affinity Table +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The System/Static Resource Affinity Table describes resource (CPU, Memory) +affinity to "Proximity Domains". This table is technically optional, but f= or +performance information (see "HMAT") to be enumerated by linux it must be +present. + +There is a careful dance between the CEDT and SRAT tables and how NUMA nod= es are +created. If things don't look quite the way you expect - check the SRAT M= emory +Affinity entries and CEDT CFMWS to determine what your platform actually +supports in terms of flexible topologies. + +The SRAT may statically assign portions of a CFMWS SPA range to a specific +proximity domains. See linux numa creation for more information about how +this presents in the NUMA topology. + +Proximity Domain +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +A proximity domain is ROUGHLY equivalent to "NUMA Node" - though a 1-to-1 +mapping is not guaranteed. There are scenarios where "Proximity Domain 4"= may +map to "NUMA Node 3", for example. (See "NUMA Node Creation") + +Memory Affinity +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Generally speaking, if a host does any amount of CXL fabric (decoder) +programming in BIOS - an SRAT entry for that memory needs to be present. + +Example :: + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000001 <- NUMA Node 1 + Reserved1 : 0000 + Base Address : 000000C050000000 <- Physical Memory Region + Address Length : 0000003CA0000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 269CE2500C5 for ; Mon, 12 May 2025 16:21:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066916; cv=none; b=sQJW1nciSfagfgOVCPfAj0GSov9rRqwnVkXWhT/L6NyKQ7rlr+TR+117e31aDhA2vzwpWowzEv57K4JH8O80XTv80Js5gxA271YCykcFe36pNeLPr9AwKUS8QJeMCANxk+M72RBJsUJEHi+ng9Tc5CvsvsymzAlTlIjhQ9lWrkw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066916; c=relaxed/simple; bh=FrqsnCms64xssgqd/+HDCq4f1uWyiTUT0WHx8B6TExI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Fieus/bC5Jkyx+kJaYdmZIAk2ENI05VuScwhjZ+lCIQwQXKzd+9Gi9fzosRxDycS25VXoaUQoXEjHIeM8rdniKaWVubL1+taUaXNuozpa2vFJ9Txba5uEZDqvn1O0xlTM9P9ntUNnqGNQoRmGbKuTmxpy9Vm1VXyq7qVq3dPmKk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=jjs+vKH2; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="jjs+vKH2" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-476b4c9faa2so67521131cf.3 for ; Mon, 12 May 2025 09:21:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066912; x=1747671712; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mlTkMyf1jxpd4EvIj3V+ddlx0K3cVVTKy+9j4XvAUwA=; b=jjs+vKH2RxEVnBQK73P+0WhFTwbIs+qBqZywRYCsmQHneqTV/7OK7rCDv8JQ3w2wjM sd4e+q8ofUZtJnemAhAeYmtg0nL6GdDA2i0eSDbvqEqFy61JHRL5OM7KHkDX9MBBHIV6 s9EIn7TE4IFElsqqfM5d2IzDsUaRFnKBIIJ8Wz/EZkkfWpzai4wK1lAxPqfFxVI08X/1 TtCB2IIzfb/TGYyK3I/wHhCqWGcbo9fEo3vlvczwloAF3IcvQ+4c79nkidPN5WCxl8O5 qEyw2rDPs+NWbSR9nfnt6dLWFI5QxKu5y+e8KHfTgZdng69WvjrvTdvLa9eYlYQe4ayO LhAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066912; x=1747671712; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mlTkMyf1jxpd4EvIj3V+ddlx0K3cVVTKy+9j4XvAUwA=; b=sHIXu7r7qBdtGXHOerPhd8RaDNHFg66PwLTRQ8JWB+YAbfWI/MnaySrItfZvoPWJX0 brjO9rekS32ac8kKxl27Z2YWxFrJ373KZWudjHaV0drsr4+jRDR5Z8/8wwJ5nA+2JsWc pDSPCQggObsu63P353aDoXii9pzBU0jXLek/Ohm8JW4lQxQqIj8B3FCGRn3qsY6xz+Ve IC+ej5h3SE/axVkw7hnStzwOE2SZQXk/ULG3gWO9oiT3E3CK9iznWmiPgXT1xzaNWDSA s5KDmRcnBEXXsXibtZFswcxyDd7Yyy2Xywdn2ZTGqEk86lp6eDZENCe678jCu4YjhKlm 3QcA== X-Forwarded-Encrypted: i=1; AJvYcCW3MvUeXXrzW/YcDGSoGGrQ0BxISusGBRAaHXxJ2PzNdA25/ZIQ/DnNOqb22Ikjq2FrqtklgrdX1GWVmGc=@vger.kernel.org X-Gm-Message-State: AOJu0YzpFbdGD3uSLmfkO1uMZOd7xbXd+9jvePDVBzPgMpTDhn1LDstj 4ZoomVLTDcNHhQgTsN6vteVCc9cPJU61WgOKL2h3BGgj4PS//blfEcHBv3Ec9Ps= X-Gm-Gg: ASbGnctmi7hSx1ENKgb0dU29t8oC4vM850ianyAeqapEgxeta/ei9jymPH6mKnfo3lT HDx6Gc+zoT9evxkoOdI72qO0d/mksFaxwkxoYd6xVTtnv64kPmUkv8aJlnSW7Xy0wdeUKnZydfc D7MZ1COYqPLOJDzNJDhpPTggE0/isMKFyjfpQCPP26PnaUaY6MEIMrkRU1AAc5OjRIkBE/PQMlX cdLjnG/EVXeZOagtw3Q4ITnq/TSNXeYzFrMkWbbOSWw5OZvAzAJHgZke48ab6IcnBFg8IOHwUZ+ AxjrwTBRDbb8RZ4Oe2FI2clAdTOczPQkuIeShvSHn3QXwnqCym9eKAypnFMtHlvF1gQ5VkOxHiH UhKVoICo4+J133EOb0pdeROSHmOeUkvQb1E9P X-Google-Smtp-Source: AGHT+IEfNfUHzJW1gy0tomAJEHsP0Ni2ouf40Q1xEewnGZ8Igtvdw96kyNakqf4sOLk/RAoM0LT4/g== X-Received: by 2002:a05:622a:56:b0:491:1874:bd9d with SMTP id d75a77b69052e-4945275e65dmr200111601cf.28.1747066911623; Mon, 12 May 2025 09:21:51 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:51 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 06/17] cxl: docs/platform/example-configs documentation Date: Mon, 12 May 2025 12:21:23 -0400 Message-ID: <20250512162134.3596150-7-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add example ACPI Table configurations for different sample platforms. Signed-off-by: Gregory Price Reviewed-by: Dave Jiang --- Documentation/driver-api/cxl/index.rst | 1 + .../cxl/platform/example-configs.rst | 13 + .../example-configurations/flexible.rst | 296 ++++++++++++++++++ .../example-configurations/hb-interleave.rst | 107 +++++++ .../multi-dev-per-hb.rst | 90 ++++++ .../example-configurations/one-dev-per-hb.rst | 136 ++++++++ 6 files changed, 643 insertions(+) create mode 100644 Documentation/driver-api/cxl/platform/example-configs.r= st create mode 100644 Documentation/driver-api/cxl/platform/example-configura= tions/flexible.rst create mode 100644 Documentation/driver-api/cxl/platform/example-configura= tions/hb-interleave.rst create mode 100644 Documentation/driver-api/cxl/platform/example-configura= tions/multi-dev-per-hb.rst create mode 100644 Documentation/driver-api/cxl/platform/example-configura= tions/one-dev-per-hb.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 336322dc35a0..6a5fb7e00c52 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -27,6 +27,7 @@ that have impacts on each other. The docs here break up = configurations steps. =20 platform/bios-and-efi platform/acpi + platform/example-configs =20 .. toctree:: :maxdepth: 1 diff --git a/Documentation/driver-api/cxl/platform/example-configs.rst b/Do= cumentation/driver-api/cxl/platform/example-configs.rst new file mode 100644 index 000000000000..90a10d7473c6 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/example-configs.rst @@ -0,0 +1,13 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Example Platform Configurations +############################### + +.. toctree:: + :maxdepth: 1 + :caption: Contents + + example-configurations/one-dev-per-hb.rst + example-configurations/multi-dev-per-hb.rst + example-configurations/hb-interleave.rst + example-configurations/flexible.rst diff --git a/Documentation/driver-api/cxl/platform/example-configurations/f= lexible.rst b/Documentation/driver-api/cxl/platform/example-configurations/= flexible.rst new file mode 100644 index 000000000000..e39daba65fa0 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/example-configurations/flexible= .rst @@ -0,0 +1,296 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Flexible Presentation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +This system has a single socket with two CXL host bridges. Each host bridge +has two CXL memory expanders with a 4GB of memory (32GB total). + +On this system, the platform designer wanted to provide the user flexibili= ty +to configure the memory devices in various interleave or NUMA node +configurations. So they provided every combination. + +Things to note: + +* Cross-Bridge interleave is described in one CFMWS that covers all capaci= ty. +* One CFMWS is also described per-host bridge. +* One CFMWS is also described per-device. +* This SRAT describes one node for each of the above CFMWS. +* The HMAT describes performance for each node in the SRAT. + +CEDT :: + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000007 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010370400000 + Register length : 0000000000010000 + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000006 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010380800000 + Register length : 0000000000010000 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000001000000000 + Window size : 0000000400000000 + Interleave Members (2^n) : 01 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + Second Target : 00000006 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000002000000000 + Window size : 0000000200000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000002200000000 + Window size : 0000000200000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000006 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000003000000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000003100000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000003200000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000006 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000003300000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000006 + +SRAT :: + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000001 + Reserved1 : 0000 + Base Address : 0000001000000000 + Address Length : 0000000400000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000002 + Reserved1 : 0000 + Base Address : 0000002000000000 + Address Length : 0000000200000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000003 + Reserved1 : 0000 + Base Address : 0000002200000000 + Address Length : 0000000200000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000004 + Reserved1 : 0000 + Base Address : 0000003000000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000005 + Reserved1 : 0000 + Base Address : 0000003100000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000006 + Reserved1 : 0000 + Base Address : 0000003200000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000007 + Reserved1 : 0000 + Base Address : 0000003300000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + +HMAT :: + + Structure Type : 0001 [SLLBI] + Data Type : 00 [Latency] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Target Proximity Domain List : 00000003 + Target Proximity Domain List : 00000004 + Target Proximity Domain List : 00000005 + Target Proximity Domain List : 00000006 + Target Proximity Domain List : 00000007 + Entry : 0080 + Entry : 0100 + Entry : 0100 + Entry : 0100 + Entry : 0100 + Entry : 0100 + Entry : 0100 + Entry : 0100 + + Structure Type : 0001 [SLLBI] + Data Type : 03 [Bandwidth] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Target Proximity Domain List : 00000003 + Target Proximity Domain List : 00000004 + Target Proximity Domain List : 00000005 + Target Proximity Domain List : 00000006 + Target Proximity Domain List : 00000007 + Entry : 1200 + Entry : 0400 + Entry : 0200 + Entry : 0200 + Entry : 0100 + Entry : 0100 + Entry : 0100 + Entry : 0100 + +SLIT :: + + Signature : "SLIT" [System Locality Information Table] + Localities : 0000000000000003 + Locality 0 : 10 20 20 20 20 20 20 20 + Locality 1 : FF 0A FF FF FF FF FF FF + Locality 2 : FF FF 0A FF FF FF FF FF + Locality 3 : FF FF FF 0A FF FF FF FF + Locality 4 : FF FF FF FF 0A FF FF FF + Locality 5 : FF FF FF FF FF 0A FF FF + Locality 6 : FF FF FF FF FF FF 0A FF + Locality 7 : FF FF FF FF FF FF FF 0A + +DSDT :: + + Scope (_SB) + { + Device (S0D0) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x07) // _UID: Unique ID + } + ... + Device (S0D5) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x06) // _UID: Unique ID + } + } diff --git a/Documentation/driver-api/cxl/platform/example-configurations/h= b-interleave.rst b/Documentation/driver-api/cxl/platform/example-configurat= ions/hb-interleave.rst new file mode 100644 index 000000000000..ce07e6162f26 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/example-configurations/hb-inter= leave.rst @@ -0,0 +1,107 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +Cross-Host-Bridge Interleave +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +This system has a single socket with two CXL host bridges. Each host bridge +has a single CXL memory expander with a 4GB of memory. + +Things to note: + +* Cross-Bridge interleave is described. +* The expanders are described by a single CFMWS. +* This SRAT describes one node for both host bridges. +* The HMAT describes a single node's performance. + +CEDT :: + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000007 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010370400000 + Register length : 0000000000010000 + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000006 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010380800000 + Register length : 0000000000010000 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000001000000000 + Window size : 0000000200000000 + Interleave Members (2^n) : 01 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + Second Target : 00000006 + +SRAT :: + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000001 + Reserved1 : 0000 + Base Address : 0000001000000000 + Address Length : 0000000200000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + +HMAT :: + + Structure Type : 0001 [SLLBI] + Data Type : 00 [Latency] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Entry : 0080 + Entry : 0100 + + Structure Type : 0001 [SLLBI] + Data Type : 03 [Bandwidth] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Entry : 1200 + Entry : 0400 + +SLIT :: + + Signature : "SLIT" [System Locality Information Table] + Localities : 0000000000000003 + Locality 0 : 10 20 + Locality 1 : FF 0A + +DSDT :: + + Scope (_SB) + { + Device (S0D0) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x07) // _UID: Unique ID + } + ... + Device (S0D5) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x06) // _UID: Unique ID + } + } diff --git a/Documentation/driver-api/cxl/platform/example-configurations/m= ulti-dev-per-hb.rst b/Documentation/driver-api/cxl/platform/example-configu= rations/multi-dev-per-hb.rst new file mode 100644 index 000000000000..6adf7c639490 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/example-configurations/multi-de= v-per-hb.rst @@ -0,0 +1,90 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D +Multiple Devices per Host Bridge +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D + +In this example system we will have a single socket and one CXL host bridg= e. +There are two CXL memory expanders with 4GB attached to the host bridge. + +Things to note: + +* Intra-Bridge interleave is not described here. +* The expanders are described by a single CEDT/CFMWS. +* This CEDT/SRAT describes one node for both devices. +* There is only one proximity domain the HMAT for both devices. + +CEDT :: + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000007 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010370400000 + Register length : 0000000000010000 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000001000000000 + Window size : 0000000200000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + +SRAT :: + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000001 + Reserved1 : 0000 + Base Address : 0000001000000000 + Address Length : 0000000200000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + +HMAT :: + + Structure Type : 0001 [SLLBI] + Data Type : 00 [Latency] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Entry : 0080 + Entry : 0100 + + Structure Type : 0001 [SLLBI] + Data Type : 03 [Bandwidth] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Entry : 1200 + Entry : 0200 + +SLIT :: + + Signature : "SLIT" [System Locality Information Table] + Localities : 0000000000000003 + Locality 0 : 10 20 + Locality 1 : FF 0A + +DSDT :: + + Scope (_SB) + { + Device (S0D0) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x07) // _UID: Unique ID + } + ... + } diff --git a/Documentation/driver-api/cxl/platform/example-configurations/o= ne-dev-per-hb.rst b/Documentation/driver-api/cxl/platform/example-configura= tions/one-dev-per-hb.rst new file mode 100644 index 000000000000..b89ba3cab98f --- /dev/null +++ b/Documentation/driver-api/cxl/platform/example-configurations/one-dev-= per-hb.rst @@ -0,0 +1,136 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D +One Device per Host Bridge +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D + +This system has a single socket with two CXL host bridges. Each host bridge +has a single CXL memory expander with a 4GB of memory. + +Things to note: + +* Cross-Bridge interleave is not being used. +* The expanders are in two separate but adjascent memory regions. +* This CEDT/SRAT describes one node per device +* The expanders have the same performance and will be in the same memory t= ier. + +CEDT :: + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000007 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010370400000 + Register length : 0000000000010000 + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000006 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010380800000 + Register length : 0000000000010000 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000001000000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000001100000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000006 + +SRAT :: + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000001 + Reserved1 : 0000 + Base Address : 0000001000000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000002 + Reserved1 : 0000 + Base Address : 0000001100000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + +HMAT :: + + Structure Type : 0001 [SLLBI] + Data Type : 00 [Latency] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Entry : 0080 + Entry : 0100 + Entry : 0100 + + Structure Type : 0001 [SLLBI] + Data Type : 03 [Bandwidth] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Entry : 1200 + Entry : 0200 + Entry : 0200 + +SLIT :: + + Signature : "SLIT" [System Locality Information Table] + Localities : 0000000000000003 + Locality 0 : 10 20 20 + Locality 1 : FF 0A FF + Locality 2 : FF FF 0A + +DSDT :: + + Scope (_SB) + { + Device (S0D0) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x07) // _UID: Unique ID + } + ... + Device (S0D5) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x06) // _UID: Unique ID + } + } --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 867382AD0B for ; Mon, 12 May 2025 16:21:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066917; cv=none; b=XNDbLvNlkvfiXXMES51val4I5fhJxWLSUFDBkvtxT42U0/hYiDTDIm7vyjsYPaeNUaB+bvibU999lOonyoqRCFhjqKOtWBJLflBa0/AxMziqJv7POpCgQwVfO/zz5WldCkCBNRNCD+ohHL/vSjTbk3OKyYX97N46IjLNR3h8HmU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066917; c=relaxed/simple; bh=X7EJbGzB/GZuOpW1q7+z422dxgR1XteIB0LGULxl5oA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o9ccg6Ymna0eeLjCnr2QFbKyyTNmc3jvu5U4k3T20KvmEh6u3O250RC+HXL9ZL1kMWlcAgr4eia1wdjSfPqW9ltzcXaOdbnRWyDuhHvC2eTwLDJcr3eorlb4JTryvNLxy7bljgqwwi69XwCc8tFFnyhigm4Xfu1sXMtn/GxbmHY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=DwqkKUAA; arc=none smtp.client-ip=209.85.160.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="DwqkKUAA" Received: by mail-qt1-f170.google.com with SMTP id d75a77b69052e-476f4e9cf92so38854671cf.3 for ; Mon, 12 May 2025 09:21:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066913; x=1747671713; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DC0/ofhj7JKy32BqrO/IxOiR9eJ+JSl3w3qlcKUJ5PI=; b=DwqkKUAANeIBbwyvQzlwlIuNpBJVvkgTnpCd0sDKvoe2+0yj6V6DWSf4zetopIVXlv LTR6EGW375cNiPZasD9HJc/4TpfkFn/HV0UWg+/k0lT/Sh1Dg218ru1iAivXhUPvq3kW NfDHAjuRBj8QpkHkC+wKu7wWTVak2UO/piS6//vtpmEF9bqDyzSKVr6sFZq/+FgGY9yA yVfkzXDUP8g42X9FB/US3vnhYDtp/g3LtUBPIiN/LZy/fjodCQWsTWYgeMdNUFNipK3R XchGsjfdhQ3Y5dfADeD8uIQAjaRuyiW1VUo44s/dMeQM98hNwQTlFKFHq1+WzHuSzWOI BIIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066913; x=1747671713; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DC0/ofhj7JKy32BqrO/IxOiR9eJ+JSl3w3qlcKUJ5PI=; b=bYdvg5T7bLs/JQMiA2vHT7x2ssAy1Mpwjfzcjpy9qxd02OhDVLXUE8MDkFsdLyY7GE Zf1vOrzihFlWjCX3cdfzlATyJ0CRBbDSyppOZkzQmO2WYHd3DGCDmtST/5Tvl6+vdOeu R55bcvtTxROSaxH5Q5ymbXqXOKkO0m2H1CaBFIL5hl7MhNvhM8PDxtscZnts8A1lTkDa evv2+Bux2PhfAbM/hXkYUZ8a9dUsb4KIsiFSu0T+OT1dCBkDE4TAvKBmeakt9/cJLhhL LvdKypD4RGHeatKfT1XY22o7Ha3PYCC+fkEDCxn/M+qn4p+iIzEVgOuD5W/gcSWuOwWT C5nw== X-Forwarded-Encrypted: i=1; AJvYcCX2rzlfUR2Vi22CyE7l+zOAAsBL3MRJwaNxgcBcJaGbi5b72ao14+/QtKBTt/eKlvtinGH8nA7x+0L+q8g=@vger.kernel.org X-Gm-Message-State: AOJu0YznGnnY26QsDVavUnlwpchhhM0s71BOCGN1rrccDfgqRwhhkZQ1 gywBqatwXdXgTYDySNdsZ+U8lRy3RjGDvt9W4e0TmcnLI7bJrzGkMwGtlKDqO/c= X-Gm-Gg: ASbGncss5NWTijWjreLF/Z6l6pr+QjrmRdk+umgc1hAsMtuwWfMij1gdic+jT2pYRj5 xpLhlkj4XnWZ2GoznbRgS+f46wMUnQTeHtyhcd0SenDk1owD9g8+sBGmwmBw8RzDdfjAwqF6omY 6Vf/y/7HeRQi30aGY2ZZFAV/Xaji/ohjtijyJHk1S+Qe5bzaYPtkMqvaSZDSakcIuzasPlQeFrt QUspLK7J07UR63m8BHY9y+a3iVLwY5oS9bWJ8gobPAGcLP5erQ8ilnm23vKFicKmDqnvqZ7lCv4 EeuD+oPkFBHVfQ0XOoKIn250hDFaK0ueJeqnA1fL8/A4TgrzeLYkfFXwFsljqUaO+ZAykaqT9a8 mC1SbmfUN29kT6rnlhrF4JmR80Ai1XIWC8cer X-Google-Smtp-Source: AGHT+IHRtIB0fCjA+A9cMknQMzSETec9axcuV/yvNPBERr6Pc5t0djPvfI94QTi3RZ4eSxuBkXdaIw== X-Received: by 2002:ac8:7e95:0:b0:477:64b0:6a26 with SMTP id d75a77b69052e-4945273c329mr213344211cf.22.1747066913103; Mon, 12 May 2025 09:21:53 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:52 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 07/17] cxl: docs/linux - overview Date: Mon, 12 May 2025 12:21:24 -0400 Message-ID: <20250512162134.3596150-8-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add type-3 device configuration overview that explains the probe process for a type-3 device from early-boot through memory-hotplug. Signed-off-by: Gregory Price Reviewed-by: Dave Jiang --- Documentation/driver-api/cxl/index.rst | 3 +- .../driver-api/cxl/linux/overview.rst | 103 ++++++++++++++++++ 2 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 Documentation/driver-api/cxl/linux/overview.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 6a5fb7e00c52..bc2228c77c32 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -30,9 +30,10 @@ that have impacts on each other. The docs here break up= configurations steps. platform/example-configs =20 .. toctree:: - :maxdepth: 1 + :maxdepth: 2 :caption: Linux Kernel Configuration =20 + linux/overview linux/access-coordinates =20 =20 diff --git a/Documentation/driver-api/cxl/linux/overview.rst b/Documentatio= n/driver-api/cxl/linux/overview.rst new file mode 100644 index 000000000000..648beb2c8c83 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/overview.rst @@ -0,0 +1,103 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D +Overview +=3D=3D=3D=3D=3D=3D=3D=3D + +This section presents the configuration process of a CXL Type-3 memory dev= ice, +and how it is ultimately exposed to users as either a :code:`DAX` device or +normal memory pages via the kernel's page allocator. + +Portions marked with a bullet are points at which certain kernel objects +are generated. + +1) Early Boot + + a) BIOS, Build, and Boot Parameters + + i) EFI_MEMORY_SP + ii) CONFIG_EFI_SOFT_RESERVE + iii) CONFIG_MHP_DEFAULT_ONLINE_TYPE + iv) nosoftreserve + + b) Memory Map Creation + + i) EFI Memory Map / E820 Consulted for Soft-Reserved + + * CXL Memory is set aside to be handled by the CXL driver + + * Soft-Reserved IO Resource created for CFMWS entry + + c) NUMA Node Creation + + * Nodes created from ACPI CEDT CFMWS and SRAT Proximity domains (PXM) + + d) Memory Tier Creation + + * A default memory_tier is created with all nodes. + + e) Contiguous Memory Allocation + + * Any requested CMA is allocated from Online nodes + + f) Init Finishes, Drivers start probing + +2) ACPI and PCI Drivers + + a) Detects PCI device is CXL, marking it for probe by CXL driver + +3) CXL Driver Operation + + a) Base device creation + + * root, port, and memdev devices created + * CEDT CFMWS IO Resource creation + + b) Decoder creation + + * root, switch, and endpoint decoders created + + c) Logical device creation + + * memory_region and endpoint devices created + + d) Devices are associated with each other + + * If auto-decoder (BIOS-programmed decoders), driver validates + configurations, builds associations, and locks configs at probe time. + + * If user-configured, validation and associations are built at + decoder-commit time. + + e) Regions surfaced as DAX region + + * dax_region created + + * DAX device created via DAX driver + +4) DAX Driver Operation + + a) DAX driver surfaces DAX region as one of two dax device modes + + * kmem - dax device is converted to hotplug memory blocks + + * DAX kmem IO Resource creation + + * hmem - dax device is left as daxdev to be accessed as a file. + + * If hmem, journey ends here. + + b) DAX kmem surfaces memory region to Memory Hotplug to add to page + allocator as "driver managed memory" + +5) Memory Hotplug + + a) mhp component surfaces a dax device memory region as multiple memory + blocks to the page allocator + + * blocks appear in :code:`/sys/bus/memory/devices` and linked to a NUM= A node + + b) blocks are onlined into the requested zone (NORMAL or MOVABLE) + + * Memory is marked "Driver Managed" to avoid kexec from using it as re= gion + for kernel updates --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC9DE2517AF for ; Mon, 12 May 2025 16:21:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066918; cv=none; b=Ydet1b26l/sbXEKX0vSYBvyFibJw9D9ueVI4Y1qIQuahXlGBOrxDZ7B+UTJ+eS7aBsmuhgaPq08cw7lXB1qh5ucsErh8h3SW1eH/tou29SZPKlI1RKOI4U+kFYnhyMhg3XckzTbyODSCaEutS/p0F9OlbXaj2wlFTxMV/3dNqow= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066918; c=relaxed/simple; bh=LxQydT9NcYzB9rb63wpRt/UEayoX55pNcemCp/ExSaA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=u2xCVcpfgyxQ2R209gk7C4WWkZEWxvxYUAdRgN7bLoYCxbdleCIjfECMzy6OafZJ3+414I13JFTKLbXMx1JdS49OwSHvwGBxrLWUlRdyGWw8L8/mUqdK9qnq24tHuAhl1l8E1nBd8/0s5QO1wsdCordntZenrp2MDIgX+pF3+QY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=po0pglis; arc=none smtp.client-ip=209.85.160.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="po0pglis" Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-477282401b3so51492431cf.1 for ; Mon, 12 May 2025 09:21:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066915; x=1747671715; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=I8UJn+qPm/CqygOdsDrWlSFpuesufqB+7bgKD2LpNho=; b=po0pglis75+crhbyCE2FcAprMvwOg40U9SHx8P42y+uEvjdRKHAtbErMjYNOeWhTPf /+YhXybyGWPnfz2RY1GwX25x+s/JZR9xtyP7cqG9PBaN7m/yyZQ/S2/LB8y/m/+x9RVh zQN1HRj3Ho54UEKAN8TF4QbS+3yf7bQOPcM4YfYUg9LBv9Hkid4uoYNsyOVeM1835u/H 1vvDTxi2DpqyEoh4g5fgrQZzMFX8/GBb4b+uL/e6crq/O+KhUyXuWTekTTfz+2W5hLek IhfStyNH9MmnsJSyVjxkWQF5tV/4KIeyf02DvmgE7W+nCktJ+IqTydhv94WNhX7iEhKt h+2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066915; x=1747671715; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I8UJn+qPm/CqygOdsDrWlSFpuesufqB+7bgKD2LpNho=; b=sZwZHv8SlUBg3OfIZ2O7lGjlKiPByoj3lhQwFxBB5ew4hZgoRO0msbg5FjDun6M7mm wkJd++O0scID+MpDtsq/nUJv/CWmY2Mmx85hB6F2U6UGckste25+E+oPKsUYnk3wJZMB KcQiRUPlRi3+W59Kht4z+NOgid8GPPOzqxpwUbNWFQLWhAnY8M0w8LoiAIfDsOqUg/mD 8p5R0zcmA9J0kFfQUCvkoHWBeRcrZDeRsedDJWhUiDGjJMSAdLgaFvYHUPZRM7Yck3PA Oy4gzzAk+j/t1UgN8qDFH4LotYFNaqQALwKYs8Ua6MF36A5/P9yjm9HBfvYADCgJujWj Ftzw== X-Forwarded-Encrypted: i=1; AJvYcCWnC/3O1kVpYFOm6AySsBSzPKna354PFsc2yti8MUSBxc6qvAp8NALaMb8yuLkIBtLIvV5jP8teq+ARzEI=@vger.kernel.org X-Gm-Message-State: AOJu0Yw53ertahdKC31cdCJoG1geRdLmlA72Ig0K2a/F9i5N8uYCVef4 w0I+hjG1PQOou9XOkB8x7VRSkgmuZloZTISgqPGLtya6mzMzJhfR65rvpaMEMSg= X-Gm-Gg: ASbGncsMPHEhBqq8Fg689dddSVyS+ERJtb4Fbj2FymopC8av5QHRFU4OkW6Zha0b6HY vk4S0yVHSnqlZsUO5uGD2L7gELwMcF0twLN601zc/Sp82um+kd/DTXGKl7rNFldEEHVon3FJlyX r8wwI8RepFuoi/GZ+gjVoVvTAwI9O9d/XQ8CU9f/ZTdH4IZyRFZsa/UDbKQUzkP+g1S3+KhWXY5 v6+slrWtlYc0/zr+IvlWZY5rbW6GsR5aR83Ea1zDF/PjUfc5iG0/6KqCo68d4w+VWxQH0mR4Cfx rRgjgmhI4/N3TZCPBxy+RCOfP2ujPfah9UNSEaB/KFkYTfh9+wA6UsSz7fgsWxPetG0peMUg1Vm n4HM8kPdpKvRs1D+pt452bW9kHc+FlpVm9R9G X-Google-Smtp-Source: AGHT+IF/Ciuu/RnaIzilFe33rOYM7yb8qG2/RD4RLkfHTbIO9NZZVIDaDNHHl1sBRQgVDQ1Sb37DNw== X-Received: by 2002:a05:622a:11d5:b0:48c:428c:5b5b with SMTP id d75a77b69052e-49452712ee1mr237385871cf.1.1747066914598; Mon, 12 May 2025 09:21:54 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:54 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 08/17] cxl: docs/linux - early boot configuration Date: Mon, 12 May 2025 12:21:25 -0400 Message-ID: <20250512162134.3596150-9-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Document __init time configurations that affect CXL driver probe process and memory region configuration. Signed-off-by: Gregory Price Reviewed-by: Dave Jiang --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/linux/early-boot.rst | 131 ++++++++++++++++++ 2 files changed, 132 insertions(+) create mode 100644 Documentation/driver-api/cxl/linux/early-boot.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index bc2228c77c32..d2eefe575604 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -34,6 +34,7 @@ that have impacts on each other. The docs here break up = configurations steps. :caption: Linux Kernel Configuration =20 linux/overview + linux/early-boot linux/access-coordinates =20 =20 diff --git a/Documentation/driver-api/cxl/linux/early-boot.rst b/Documentat= ion/driver-api/cxl/linux/early-boot.rst new file mode 100644 index 000000000000..8c1c497bc772 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/early-boot.rst @@ -0,0 +1,131 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Linux Init (Early Boot) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Linux configuration is split into two major steps: Early-Boot and everythi= ng else. + +During early boot, Linux sets up immutable resources (such as numa nodes),= while +later operations include things like driver probe and memory hotplug. Lin= ux may +read EFI and ACPI information throughout this process to configure logical +representations of the devices. + +During Linux Early Boot stage (functions in the kernel that have the __init +decorator), the system takes the resources created by EFI/BIOS (ACPI table= s) +and turns them into resources that the kernel can consume. + + +BIOS, Build and Boot Options +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D + +There are 4 pre-boot options that need to be considered during kernel build +which dictate how memory will be managed by Linux during early boot. + +* EFI_MEMORY_SP + + * BIOS/EFI Option that dictates whether memory is SystemRAM or + Specific Purpose. Specific Purpose memory will be deferred to + drivers to manage - and not immediately exposed as system RAM. + +* CONFIG_EFI_SOFT_RESERVE + + * Linux Build config option that dictates whether the kernel supports + Specific Purpose memory. + +* CONFIG_MHP_DEFAULT_ONLINE_TYPE + + * Linux Build config that dictates whether and how Specific Purpose memo= ry + converted to a dax device should be managed (left as DAX or onlined as + SystemRAM in ZONE_NORMAL or ZONE_MOVABLE). + +* nosoftreserve + + * Linux kernel boot option that dictates whether Soft Reserve should be + supported. Similar to CONFIG_EFI_SOFT_RESERVE. + +Memory Map Creation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +While the kernel parses the EFI memory map, if :code:`Specific Purpose` me= mory +is supported and detected, it will set this region aside as +:code:`SOFT_RESERVED`. + +If :code:`EFI_MEMORY_SP=3D0`, :code:`CONFIG_EFI_SOFT_RESERVE=3Dn`, or +:code:`nosoftreserve=3Dy` - Linux will default a CXL device memory region = to +SystemRAM. This will expose the memory to the kernel page allocator in +:code:`ZONE_NORMAL`, making it available for use for most allocations (inc= luding +:code:`struct page` and page tables). + +If `Specific Purpose` is set and supported, :code:`CONFIG_MHP_DEFAULT_ONLI= NE_TYPE_*` +dictates whether the memory is onlined by default (:code:`_OFFLINE` or +:code:`_ONLINE_*`), and if online which zone to online this memory to by d= efault +(:code:`_NORMAL` or :code:`_MOVABLE`). + +If placed in :code:`ZONE_MOVABLE`, the memory will not be available for mo= st +kernel allocations (such as :code:`struct page` or page tables). This may +significant impact performance depending on the memory capacity of the sys= tem. + + +NUMA Node Reservation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Linux refers to the proximity domains (:code:`PXM`) defined in the SRAT to +create NUMA nodes in :code:`acpi_numa_init`. Typically, there is a 1:1 rel= ation +between :code:`PXM` and NUMA node IDs. + +SRAT is the only ACPI defined way of defining Proximity Domains. Linux cho= oses +to, at most, map those 1:1 with NUMA nodes. CEDT adds a description of SPA +ranges which Linux may wish to map to one or more NUMA nodes. + +If there are CXL ranges in the CFMWS but not in SRAT, then a fake :code:`P= XM` +is created (as of v6.15). In the future, Linux may reject CFMWS not descri= bed +by SRAT due to the ambiguity of proximity domain association. + +It is important to note that NUMA node creation cannot be done at runtime.= All +possible NUMA nodes are identified at :code:`__init` time, more specifical= ly +during :code:`mm_init`. The CEDT and SRAT must contain sufficient :code:`P= XM` +data for Linux to identify NUMA nodes their associated memory regions. + +The relevant code exists in: :code:`linux/drivers/acpi/numa/srat.c`. + +See the Example Platform Configurations section for more information. + +Memory Tiers Creation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Memory tiers are a collection of NUMA nodes grouped by performance charact= eristics. +During :code:`__init`, Linux initializes the system with a default memory = tier that +contains all nodes marked :code:`N_MEMORY`. + +:code:`memory_tier_init` is called at boot for all nodes with memory onlin= e by +default. :code:`memory_tier_late_init` is called during late-init for node= s setup +during driver configuration. + +Nodes are only marked :code:`N_MEMORY` if they have *online* memory. + +Tier membership can be inspected in :: + + /sys/devices/virtual/memory_tiering/memory_tierN/nodelist + 0-1 + +If nodes are grouped which have clear difference in performance, check the= HMAT +and CDAT information for the CXL nodes. All nodes default to the DRAM tie= r, +unless HMAT/CDAT information is reported to the memory_tier component via +`access_coordinates`. + +Contiguous Memory Allocation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +The contiguous memory allocator (CMA) enables reservation of contiguous me= mory +regions on NUMA nodes during early boot. However, CMA cannot reserve memo= ry +on NUMA nodes that are not online during early boot. :: + + void __init hugetlb_cma_reserve(int order) { + if (!node_online(nid)) + /* do not allow reservations */ + } + +This means if users intend to defer management of CXL memory to the driver= , CMA +cannot be used to guarantee huge page allocations. If enabling CXL memory= as +SystemRAM in `ZONE_NORMAL` during early boot, CMA reservations per-node ca= n be +made with the :code:`cma_pernuma` or :code:`numa_cma` kernel command line +parameters. --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8227F2522AA for ; Mon, 12 May 2025 16:21:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066920; cv=none; b=C/iF4xhVFhCBj+k3JYIcitPQCIJo2dDfgFLeR7nkgXQN5lnzF9wXB4ICR0gTNiY7XbjXoCwJ9f5yRaWSCdQ+o04XzZcnzjr6v6DYj427YWAkZLrYZEEsZhTPKZs742x9Xtc7ZZqpfc3b2ZOgw33SWGoEd9N0PX+gnFXee6A6pvY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066920; c=relaxed/simple; bh=5Ue75lEdqSibBVW5GkwS0ZQh2GWuV+mXYj0OqZr29CQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Lew4fLa20y8HRvH+ySf/eY91xrjkhZ7XJ44P5U6h2um5PMfcoxhF2avheA+VMx/7ybTqPit4ctP6yySAecabivW8kBweh4lg4nNwqL4J+oReTg+C0mVVTqNhV+mew+q5qLhkAhUVhRSvVDnUTUjPDe1uCV1V1MFXjCJhr3noblg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=RU+hlgJR; arc=none smtp.client-ip=209.85.222.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="RU+hlgJR" Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-7c5568355ffso471786085a.0 for ; Mon, 12 May 2025 09:21:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066916; x=1747671716; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TdYPxmRTQBr2SToQVjzj/rfZTRXuhIOxx9QxYTy/CPQ=; b=RU+hlgJRQ5GeZ2RL7rYZ9XWR30VN7Chou7TqsP8eqdPGgiITgxd8rhFmmPNYmciyE6 TUgn3b/kBE85JlsJvX3GY4PmXbWlmn1S9msahfhqiR91wlgig3XYQfijgI3EMhDJWVA3 x/wL1zIuln2bKxrkzsWDVsbNDxtn9QxvBGN2eJhHGgQjJXXqlTk+pF3PY+X+acayz5ye xrTn3kQTn7UYhZZNNmoqIieYz5LWjjSZzEYX+wQSD1yAAsMOPQUQMMzjoyuIALS0Ba2A YZRSPsDcVa7H8W7HHVQBVM0bDoQwLppBsTNBsw6BVRwmpMjMHpJJfLOQdkcRUWxQk9pM wp/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066916; x=1747671716; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TdYPxmRTQBr2SToQVjzj/rfZTRXuhIOxx9QxYTy/CPQ=; b=J4gdH/XbTMLGnKW2T8yndDIBJ8CMiiKmuxU6KrRkKyKokqdUfl2J4foTcf731ja8XC sljPxvcmg40OliOmZOed3BgC5Gk0PDzV6z2GmVcyPrKV4QPmMrEvk3BDbd17vUnXSlzG /g6LKzWXV1AScYlY5D01FGdgqndmpVBMZ+CIzD2RzcA0M88SzrTCLBLN+OT6NNoa6B/t 0Ze0XDHP5tHMlGBMyhA2cQUrmqAzFA0V+7KQRdK4b/M+eKu9YomIy7NtiqAtegWSFxZ3 WTS/Dtdq4q42S+gA9vZiYYfYDB4Zc8Huu6sjb5clRf6y3Bpcu5kk9MO107EO7Ro5B31/ RAhA== X-Forwarded-Encrypted: i=1; AJvYcCWbK1YMj61CBbdy054EFA+NRqk8zZklPTMg9FIpzrD8+/zLNhXrm7Nm9DqADlYBNhHTkZLIwAVYCCudUHE=@vger.kernel.org X-Gm-Message-State: AOJu0YzqO1JI6fR8Qt1NYkDIlhHeGs5rMXIeF071pqtp2NGogHDi7NMb TcUy8xM1vZOHmmOzj9fDD8INlz+rvnlSe6MLGRo/tJIqibUuizJm6IGDwx1INQw= X-Gm-Gg: ASbGncuuyItA6zCjsSKF0kHIo0vOnT/AG1ff0nQusMLJj6c2gJop20+b8Vfal5n7fnD SH9rJZ0baWfxgpquvhFwT3sVKJuskpIGVuf++NN1PMrZ7nSCuWrK0SbqpO6hdHXGu5qMX4dFWe9 Qjx0SvHC7pnyQzDq2yTNmyPslA8lPlo7C1XDpZki/QsTreo3FIrY1lZKVYem17BFvRqYw8fDa06 j/6rmOmXVrNdQq+rlKtOxs/tw8lxO8AUX7YPpMUdtSJFcA/fQPhIteniK5Ta6kGTyIYA3AtXqYP B9/1HHjZdFrAJ4flWrqLekvh/U+L7ocnfJB2kI5VqBdbimbwA0yzCznGjh3dVFdB+NwD3hiPj38 IFckknJ18OB0e2N17UUsT5uoxxYWVjg1+7LAv X-Google-Smtp-Source: AGHT+IFGiC4TMp5febt1S8WPaXPgYyYZtF9kkuCKnul5xWkzo1fW30oS2yCihMZHRO+YNuTqABsugg== X-Received: by 2002:a05:620a:288e:b0:7ca:ca5d:779e with SMTP id af79cd13be357-7cd0116e26fmr2344069285a.47.1747066916138; Mon, 12 May 2025 09:21:56 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:55 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 09/17] cxl: docs/linux - add cxl-driver theory of operation Date: Mon, 12 May 2025 12:21:26 -0400 Message-ID: <20250512162134.3596150-10-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add docs for the CXL driver that explains the base devices, decoder types, region types, mailbox interfaces, and decoder programming. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/linux/cxl-driver.rst | 522 ++++++++++++++++++ 2 files changed, 523 insertions(+) create mode 100644 Documentation/driver-api/cxl/linux/cxl-driver.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index d2eefe575604..df3c7763c79a 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -35,6 +35,7 @@ that have impacts on each other. The docs here break up = configurations steps. =20 linux/overview linux/early-boot + linux/cxl-driver linux/access-coordinates =20 =20 diff --git a/Documentation/driver-api/cxl/linux/cxl-driver.rst b/Documentat= ion/driver-api/cxl/linux/cxl-driver.rst new file mode 100644 index 000000000000..9a9e8ecee578 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/cxl-driver.rst @@ -0,0 +1,522 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +CXL Driver Operation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The devices described in this section are present in :: + + /sys/bus/cxl/devices/ + /dev/cxl/ + +The :code:`cxl-cli` library, maintained as part of the NDTCL project, may +be used to script interactions with these devices. + +Drivers +=3D=3D=3D=3D=3D=3D=3D +The CXL driver is split into a number of drivers. + +* cxl_core - fundamental init interface and core object creation +* cxl_port - initializes root and provides port enumeration interface. +* cxl_acpi - initializes root decoders and interacts with ACPI data. +* cxl_p/mem - initializes memory devices +* cxl_pci - uses cxl_port to enumates the actual fabric hierarchy. + +Driver Devices +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Here is an example from a single-socket system with 4 host bridges. Two ho= st +bridges have a single memory device attached, and the devices are interlea= ved +into a single memory region. The memory region has been converted to dax. = :: + + # ls /sys/bus/cxl/devices/ + dax_region0 decoder3.0 decoder6.0 mem0 port3 + decoder0.0 decoder4.0 decoder6.1 mem1 port4 + decoder1.0 decoder5.0 endpoint5 port1 region0 + decoder2.0 decoder5.1 endpoint6 port2 root0 + +For this section we'll explore the devices present in this configuration, = but +we'll explore more configurations in-depth in example configurations below. + +Base Devices +------------ +Most devices in a CXL fabric are a `port` of some kind (because each +device mostly routes request from one device to the next, rather than +provide a manageable service). + +Root +~~~~ +The `CXL Root` is logical object created by the `cxl_acpi` driver during +:code:`cxl_acpi_probe` - if the :code:`ACPI0017` `Compute Express Link +Root Object` Device Class is found. + +The Root contains links to: + +* `Host Bridge Ports` defined by ACPI CEDT CHBS. + +* `Root Decoders` defined by ACPI CEDT CFMWS. + +:: + + # ls /sys/bus/cxl/devices/root0 + decoder0.0 dport0 dport5 port2 subsystem + decoders_committed dport1 modalias port3 uevent + devtype dport4 port1 port4 uport + + # cat /sys/bus/cxl/devices/root0/devtype + cxl_port + + # cat port1/devtype + cxl_port + + # cat decoder0.0/devtype + cxl_decoder_root + +The root is first `logical port` in the CXL fabric, as presented by the Li= nux +CXL driver. The `CXL root` is a special type of `switch port`, in that it +only has downstream port connections. + +Port +~~~~ +A `port` object is better described as a `switch port`. It may represent a +host bridge to the root or an actual switch port on a switch. A `switch po= rt` +contains one or more decoders used to route memory requests downstream por= ts, +which may be connected to another `switch port` or an `endpoint port`. + +:: + + # ls /sys/bus/cxl/devices/port1 + decoder1.0 dport0 driver parent_dport uport + decoders_committed dport113 endpoint5 subsystem + devtype dport2 modalias uevent + + # cat devtype + cxl_port + + # cat decoder1.0/devtype + cxl_decoder_switch + + # cat endpoint5/devtype + cxl_port + +CXL `Host Bridges` in the fabric are probed during :code:`cxl_acpi_probe` = at +the time the `CXL Root` is probed. The allows for the immediate logical +connection to between the root and host bridge. + +* The root has a downstream port connection to a host bridge + +* The host bridge has an upstream port connection to the root. + +* The host bridge has one or more downstream port connections to switch + or endpoint ports. + +A `Host Bridge` is a special type of CXL `switch port`. It is explicitly +defined in the ACPI specification via `ACPI0016` ID. `Host Bridge` ports +will be probed at `acpi_probe` time, while similar ports on an actual swit= ch +will be probed later. Otherwise, switch and host bridge ports look very +similar - the both contain switch decoders which route accesses between +upstream and downstream ports. + +Endpoint +~~~~~~~~ +An `endpoint` is a terminal port in the fabric. This is a `logical device= `, +and may be one of many `logical devices` presented by a memory device. It +is still considered a type of `port` in the fabric. + +An `endpoint` contains `endpoint decoders` available for use and the +*Coherent Device Attribute Table* (CDAT) used to describe the capabilities +of the device. :: + + # ls /sys/bus/cxl/devices/endpoint5 + CDAT decoders_committed modalias uevent + decoder5.0 devtype parent_dport uport + decoder5.1 driver subsystem + + # cat /sys/bus/cxl/devices/endpoint5/devtype + cxl_port + + # cat /sys/bus/cxl/devices/endpoint5/decoder5.0/devtype + cxl_decoder_endpoint + + +Memory Device (memdev) +~~~~~~~~~~~~~~~~~~~~~~ +A `memdev` is probed and added by the `cxl_pci` driver in :code:`cxl_pci_p= robe` +and is managed by the `cxl_mem` driver. It primarily provides the `IOCTL` +interface to a memory device, via :code:`/dev/cxl/memN`, and exposes vario= us +device configuration data. :: + + # ls /sys/bus/cxl/devices/mem0 + dev firmware_version payload_max security uevent + driver label_storage_size pmem serial + firmware numa_node ram subsystem + + +Decoders +-------- +A `Decoder` is short for a CXL Host-Managed Device Memory (HDM) Decoder. I= t is +a device that routes accesses through the CXL fabric to an endpoint, and at +the endpoint translates a `Host Physical` to `Device Physical` Addressing. + +The CXL 3.1 specification heavily implies that only endpoint decoders shou= ld +engage in translation of `Host Physical Address` to `Device Physical Addre= ss`. +:: + + 8.2.4.20 CXL HDM Decoder Capability Structure + + IMPLEMENTATION NOTE + CXL Host Bridge and Upstream Switch Port Decode Flow + + IMPLEMENTATION NOTE + Device Decode Logic + +These notes imply that there are two logical groups of decoders. + +* Routing Decoder - a decoder which routes accesses but does not translate + addresses from HPA to DPA. + +* Translating Decoder - a decoder which translates accesses from HPA to DPA + for an endpoint to service. + +The CXL drivers distinguish 3 decoder types: root, switch, and endpoint. O= nly +endpoint decoders are Translating Decoders, all others are Routing Decoder= s. + +.. note:: PLATFORM VENDORS BE AWARE + + Linux makes a strong assumption that endpoint decoders are the only dec= oder + in the fabric that actively translates HPA to DPA. Linux assumes routi= ng + decoders pass the HPA unchanged to the next decoder in the fabric. + + It is therefore assumed that any given decoder in the fabric will have = an + address range that is a subset of its upstream port decoder. Any deviat= ion + from this scheme undefined per the specification. Linux prioritizes + spec-defined / architectural behavior. + +Decoders may have one or more `Downstream Targets` if configured to interl= eave +memory accesses. This will be presented in sysfs via the :code:`target_li= st` +parameter. + +Root Decoder +~~~~~~~~~~~~ +A `Root Decoder` is logical construct of the physical address and interlea= ve +configurations present in the ACPI CEDT CFMWS. Linux presents this inform= ation +as a decoder present in the `CXL Root`. We consider this a `Root Decoder`, +though technically it exists on the boundary of the CXL specification and +platform-specific CXL root implementations. + +Linux considers these logical decoders a type of `Routing Decoder`, and is= the +first decoder in the CXL fabric to receive a memory access from the platfo= rm's +memory controllers. + +`Root Decoders` are created during :code:`cxl_acpi_probe`. One root decod= er +is created per CFMWS entry in the ACPI CEDT. + +The :code:`target_list` parameter is filled by the CFMWS target fields. Ta= rgets +of a root decoder are `Host Bridges`, which means interleave done at the r= oot +decoder level is an `Inter-Host-Bridge Interleave`. + +Only root decoders are capable of `Inter-Host-Bridge Interleave`. + +Such interleaves must be configured by the platform and described in the A= CPI +CEDT CFMWS, as the target CXL host bridge UIDs in the CFMWS must match the= CXL +host bridge UIDs in the ACPI CEDT CHBS and ACPI DSDT. + +Interleave settings in a rootdecoder describe how to interleave accesses a= mong +the *immediate downstream targets*, not the entire interleave set. + +The memory range described in the root decoder is used to + +1) Create a memory region (:code:`region0` in this example), and + +2) Associate the region with an IO Memory Resource (:code:`kernel/resource= .c`) + +:: + + # ls /sys/bus/cxl/devices/decoder0.0/ + cap_pmem devtype region0 + cap_ram interleave_granularity size + cap_type2 interleave_ways start + cap_type3 locked subsystem + create_ram_region modalias target_list + delete_region qos_class uevent + + # cat /sys/bus/cxl/devices/decoder0.0/region0/resource + 0xc050000000 + +The IO Memory Resource is created during early boot when the CFMWS region = is +identified in the EFI Memory Map or E820 table (on x86). + +Root decoders are defined as a separate devtype, but are also a type +of `Switch Decoder` due to having downstream targets. :: + + # cat /sys/bus/cxl/devices/decoder0.0/devtype + cxl_decoder_root + +Switch Decoder +~~~~~~~~~~~~~~ +Any non-root, translating decoder is considered a `Switch Decoder`, and wi= ll +present with the type :code:`cxl_decoder_switch`. Both `Host Bridge` and `= CXL +Switch` (device) decoders are of type :code:`cxl_decoder_switch`. :: + + # ls /sys/bus/cxl/devices/decoder1.0/ + devtype locked size target_list + interleave_granularity modalias start target_type + interleave_ways region subsystem uevent + + # cat /sys/bus/cxl/devices/decoder1.0/devtype + cxl_decoder_switch + + # cat /sys/bus/cxl/devices/decoder1.0/region + region0 + +A `Switch Decoder` has associations between a region defined by a root +decoder and downstream target ports. Interleaving done within a switch de= coder +is a multi-downstream-port interleave (or `Intra-Host-Bridge Interleave` f= or +host bridges). + +Interleave settings in a switch decoder describe how to interleave accesses +among the *immediate downstream targets*, not the entire interleave set. + +Switch decoders are created during :code:`cxl_switch_port_probe` in the +:code:`cxl_port` driver, and is created based on a PCI device's DVSEC +registers. + +Switch decoder programming is validated during probe if the platform progr= ams +them during boot (See `Auto Decoders` below), or on commit if programmed at +runtime (See `Runtime Programming` below). + + +Endpoint Decoder +~~~~~~~~~~~~~~~~ +Any decoder attached to a *terminal* point in the CXL fabric (`An Endpoint= `) is +considered an `Endpoint Decoder`. Endpoint decoders are of type +:code:`cxl_decoder_endpoint`. :: + + # ls /sys/bus/cxl/devices/decoder5.0 + devtype locked start + dpa_resource modalias subsystem + dpa_size mode target_type + interleave_granularity region uevent + interleave_ways size + + # cat /sys/bus/cxl/devices/decoder5.0/devtype + cxl_decoder_endpoint + + # cat /sys/bus/cxl/devices/decoder5.0/region + region0 + +An `Endpoint Decoder` has an association with a region defined by a root +decoder and describes the device-local resource associated with this regio= n. + +Unlike root and switch decoders, endpoint decoders translate `Host Physica= l` to +`Device Physical` address ranges. The interleave settings on an endpoint +therefore describe the entire *interleave set*. + +`Device Physical Address` regions must be committed in-order. For example,= the +DPA region starting at 0x80000000 cannot be committed before the DPA region +starting at 0x0. + +As of Linux v6.15, Linux does not support *imbalanced* interleave setups, = all +endpoints in an interleave set are expected to have the same interleave +settings (granularity and ways must be the same). + +Endpoint decoders are created during :code:`cxl_endpoint_port_probe` in the +:code:`cxl_port` driver, and is created based on a PCI device's DVSEC regi= sters. + +Regions +------- + +Memory Region +~~~~~~~~~~~~~ +A `Memory Region` is a logical construct that connects a set of CXL ports = in +the fabric to an IO Memory Resource. It is ultimately used to expose the = memory +on these devices to the DAX subsystem via a `DAX Region`. + +An example RAM region: :: + + # ls /sys/bus/cxl/devices/region0/ + access0 devtype modalias subsystem uuid + access1 driver mode target0 + commit interleave_granularity resource target1 + dax_region0 interleave_ways size uevent + +A memory region can be constructed during endpoint probe, if decoders were +programmed by BIOS/EFI (see `Auto Decoders`), or by creating a region manu= ally +via a `Root Decoder`'s :code:`create_ram_region` or :code:`create_pmem_reg= ion` +interfaces. + +The interleave settings in a `Memory Region` describe the configuration of= the +`Interleave Set` - and are what can be expected to be seen in the endpoint +interleave settings. + + +DAX Region +~~~~~~~~~~ +A `DAX Region` is used to convert a CXL `Memory Region` to a DAX device. A +DAX device may then be accessed directly via a file descriptor interface, = or +converted to System RAM via the DAX kmem driver. See the DAX driver secti= on +for more details. :: + + # ls /sys/bus/cxl/devices/dax_region0/ + dax0.0 devtype modalias uevent + dax_region driver subsystem + + +Mailbox Interfaces +------------------ +A mailbox command interface for each device is exposed in :: + + /dev/cxl/mem0 + /dev/cxl/mem1 + +These mailboxes may receive any specification-defined command. Raw commands +(custom commands) can only be sent to these interfaces if the build config +:code:`CXL_MEM_RAW_COMMANDS` is set. This is considered a debug and/or +development interface, not an officially supported mechanism for creation +of vendor-specific commands (see the `fwctl` subsystem for that). + +Decoder Programming +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Runtime Programming +------------------- +During probe, the only decoders *required* to be programmed are `Root Deco= ders`. +In reality, `Root Decoders` are a logical construct to describe the memory +region and interleave configuration at the host bridge level - as described +in the ACPI CEDT CFMWS. + +All other `Switch` and `Endpoint` decoders may be programmed by the user +at runtime - if the platform supports such configurations. + +This interaction is what creates a `Software Defined Memory` environment. + +See the :code:`cxl-cli` documentation for more information about how to +configure CXL decoders at runtime. + +Auto Decoders +------------- +Auto Decoders are decoders programmed by BIOS/EFI at boot time, and are +almost always locked (cannot be changed). This is done by a platform +which may have a static configuration - or certain quirks which may prevent +dynamic runtime changes to the decoders (such as requiring additional +controller programming within the CPU complex outside the scope of CXL). + +Auto Decoders are probed automatically as long as the devices and memory +regions they are associated with probe without issue. When probing Auto +Decoders, the driver's primary responsibility is to ensure the fabric is +sane - as-if validating runtime programmed regions and decoders. + +If Linux cannot validate auto-decoder configuration, the memory will not +be surfaced as a DAX device - and therefore not be exposed to the page +allocator - effectively stranding it. + +Interleave +---------- + +The Linux CXL driver supports `Cross-Link First` interleave. This dictates +how interleave is programmed at each decoder step, as the driver validates +the relationships between a decoder and it's parent. + +For example, in a `Cross-Link First` interleave setup with 16 endpoints +attached to 4 host bridges, linux expects the following ways/granularity +across the root, host bridge, and endpoints respectively. :: + + ways granularity + root 4 256 + host bridge 4 1024 + endpoint 16 256 + +At the root, every a given access will be routed to the +:code:`((HPA / 256) % 4)th` target host bridge. Within a host bridge, every +:code:`((HPA / 1024) % 4)th` target endpoint. Each endpoint will translate +the access based on the entire 16 device interleave set. + +Unbalanced interleave sets are not supported - decoders at a similar point +in the hierarchy (e.g. all host bridge decoders) must have the same ways a= nd +granularity configuration. + +At Root +~~~~~~~ +Root decoder interleave is defined by the ACPI CEDT CFMWS. The CEDT +may actually define multiple CFMWS configurations to describe the same +physical capacity - with the intent to allow users to decide at runtime +whether to online memory as interleaved or non-interleaved. :: + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Window base address : 0000000100000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + First Target : 00000007 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Window base address : 0000000200000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + First Target : 00000006 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Window base address : 0000000300000000 + Window size : 0000000200000000 + Interleave Members (2^n) : 01 + Interleave Arithmetic : 00 + First Target : 00000007 + Next Target : 00000006 + +In this example, the CFMWS defines two discrete non-interleaved 4GB regions +for each host bridge, and one interleaved 8GB region that targets both. Th= is +would result in 3 root decoders presenting in the root. :: + + # ls /sys/bus/cxl/devices/root0 + decoder0.0 decoder0.1 decoder0.2 + + # cat /sys/bus/cxl/devices/decoder0.0/target_list start size + 7 + 0x100000000 + 0x100000000 + + # cat /sys/bus/cxl/devices/decoder0.1/target_list start size + 6 + 0x200000000 + 0x100000000 + + # cat /sys/bus/cxl/devices/decoder0.2/target_list start size + 7,6 + 0x300000000 + 0x200000000 + +These decoders are not runtime programmable. They are used to generate a +`Memory Region` to bring this memory online with runtime programmed settin= gs +at the `Switch` and `Endpoint` decoders. + +At Host Bridge or Switch +~~~~~~~~~~~~~~~~~~~~~~~~ +`Host Bridge` and `Switch` decoders are programmable via the following fie= lds: + +- :code:`start` - the HPA region associated with the memory region +- :code:`size` - the size of the region +- :code:`target_list` - the list of downstream ports +- :code:`interleave_ways` - the number downstream ports to interleave acro= ss +- :code:`interleave_granularity` - the granularity to interleave at. + +Linux expects the :code:`interleave_granularity` of switch decoders to be +derived from their upstream port connections. In `Cross-Link First` interl= eave +configurations, the :code:`interleave_granularity` of a decoder is equal to +:code:`parent_interleave_granularity * parent_interleave_ways`. + +At Endpoint +~~~~~~~~~~~ +`Endpoint Decoders` are programmed similar to Host Bridge and Switch decod= ers, +with the exception that the ways and granularity are defined by the interl= eave +set (e.g. the interleave settings defined by the associated `Memory Region= `). + +- :code:`start` - the HPA region associated with the memory region +- :code:`size` - the size of the region +- :code:`interleave_ways` - the number endpoints in the interleave set +- :code:`interleave_granularity` - the granularity to interleave at. + +These settings are used by endpoint decoders to *Translate* memory requests +from HPA to DPA. This is why they must be aware of the entire interleave = set. + +Linux does not support unbalanced interleave configurations. As a result,= all +endpoints in an interleave set must have the same ways and granularity. --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67B1325290E for ; Mon, 12 May 2025 16:21:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066923; cv=none; b=sXt/a7Y7S46iSFWikj4laGyzyMjb8NSJTDU3OUYEtQ+TAh4kxBCb4+8xfKWl1iE8d22oEipzg9U2nq2XgtrJEQuwnXnZ2Il0QdS4j3UZFWWEihosoTU1m+0MmJWiQDo6+lcPS085gq4qvCTRPz/ndQ4Sg8AJlqg9gN9/qxzuhPE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066923; c=relaxed/simple; bh=dkaGPWOKn67y1LdrfoERITXLH++9yici8wbfaUCIToA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OIPk9Z7WouPgHKi7BT15KBqZOfWVDqHg+sqwBJ7I8ASUL8GBNhJlXmx31wDXROA1pdgtqrU60m3GOBafz2oOgiCubYPkDIRlbHzN+DATV9LpYg3iNIzW1jqn5l0nqq9zVEwlsj47zpTZfXchZEViaAw9pH5Drv/hM1zf2/l4PYY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=KR3YSgJe; arc=none smtp.client-ip=209.85.160.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="KR3YSgJe" Received: by mail-qt1-f171.google.com with SMTP id d75a77b69052e-4768f90bf36so50331321cf.0 for ; Mon, 12 May 2025 09:21:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066918; x=1747671718; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6FdhR4gF7aeY7QZzxz+jvh/0OURCPCws/RHYlQ0UyrA=; b=KR3YSgJevCKKUokIbOMbhPeN6Q/PEjC2vf+hjqem8HlfIpA6LYdeJ0YBmO3oBrDhBn SR+lwesqTDcOlUOKvhKarKmGO5eOO96pAafM4IJ6z0AmlR0YB8NnH9M5iDfCwj6IZw+8 nWv+lBNW9YzgDQXIpa0WhNaDNBVUuwVh3uIqyMagwpWMAwp5VrokMArL6AmTP2CA+miJ /lGjfgmFbJmT0D+wDImOuBcC0qpGmuvtH1S/TqVvbKZ12b/k14JpMv5+PuxDwnKsldv2 JmIyrMD2bVju5mpAjGjdHMdsxxE+p3S0ZQKJ26KBXVBzqdaLHB4cx/3N1NqXcZ93/WvG Ie2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066918; x=1747671718; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6FdhR4gF7aeY7QZzxz+jvh/0OURCPCws/RHYlQ0UyrA=; b=VjL4y4XUqgAIl94AdDAPIl2n43Vy0sG8OMklTXsNW1SRKIYVUm5qIe0lED8P5Ly5ev u1QkeqdWqyzYP0mDUmnpb7SzFVc6hlchmfiEelf6dS/HokYuWKyU2tmJ7VA4qgv2QTsb heCE8ORsDE8dwSSu2yRJdAdmAs+P0lwy+pF80s3FnOdg8IPZk/kBuOg0NqbmI2TP2M4K SYMxqJ/FEQ1FWdLMdekcJKvZpAVQOQQli8fJCiovqJ/pK/XrkurVzYswKhATzQHuffgM utrnd681iKPv3e9GJ7DTfgkLDUjiU84P52SG1iPao63J4yCGPO7BvlChf/iSPfa0W6U+ pk8w== X-Forwarded-Encrypted: i=1; AJvYcCWTZS6pd4GZedxxTcUJz7pfsGtejsBdAd+gwNsUYy1WaEFzZg0herzHSNaU9z3gPmap9yODVALGroK0tcI=@vger.kernel.org X-Gm-Message-State: AOJu0Yw52C5MMEQkBFUD0K+fQqd/QZdeg3ugt+jg4qhoXyv2YKSgb9lv K169GibFS2gHAVP3A7h+6Uk1ub45RhWHkIfbECT9/hy467wBRZMdryDo8fm+O2A= X-Gm-Gg: ASbGnctQkwq4NHZZYI076uj0Und2rMYRKCE1l4nGyFr/Re/3gLhAVcQE6nG9t5WSIES 3pmeIzh9FaJJwqv+1fvMCLa5w4AJ2qbVCl85T2vj09Xd1juQtKudSeebnwn7fZ0bKW119oZuX8R DMzseZqv+nIpkQh1IXnFzKfMh5p0cHRmuNwvaGky9ZHrPnza+QuCgwzCKuz2n+RSUCGv2MaqN93 0sNdjpXzyc1xNjZKJEvTpl1tuzgjbHhXcrTx3y5yr5Dcxa8LogSOKewayB+OEigMrntFDtyhtpb htItqCztkTJWfUU7/p9TDZlU6nKsfhoiOG7I4mCs5+BELiIOWP2IMIcFqRmuqEJ53DYvqlu1q1J rzdpGh8HPSFiqou4vT/1TRvuej+5a04yNl/CB X-Google-Smtp-Source: AGHT+IH1RVRuM5ahaMr9cwa2/lm9WOHwEChS2vTespRCGjkzQZA4dRtetghuz1xpgoLde/jPXjNNzA== X-Received: by 2002:a05:622a:d1:b0:476:875e:516d with SMTP id d75a77b69052e-494527d5051mr200467781cf.36.1747066917643; Mon, 12 May 2025 09:21:57 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:57 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 10/17] cxl: docs/linux/cxl-driver - add example configurations Date: Mon, 12 May 2025 12:21:27 -0400 Message-ID: <20250512162134.3596150-11-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add 4 example configurations: - single device - cross-host-bridge interleave - intra-host-bridge-interleave - multi-level interleave Signed-off-by: Gregory Price --- .../driver-api/cxl/linux/cxl-driver.rst | 10 + .../example-configurations/hb-interleave.rst | 314 ++++++++++++++ .../intra-hb-interleave.rst | 291 +++++++++++++ .../multi-interleave.rst | 401 ++++++++++++++++++ .../example-configurations/single-device.rst | 246 +++++++++++ 5 files changed, 1262 insertions(+) create mode 100644 Documentation/driver-api/cxl/linux/example-configuratio= ns/hb-interleave.rst create mode 100644 Documentation/driver-api/cxl/linux/example-configuratio= ns/intra-hb-interleave.rst create mode 100644 Documentation/driver-api/cxl/linux/example-configuratio= ns/multi-interleave.rst create mode 100644 Documentation/driver-api/cxl/linux/example-configuratio= ns/single-device.rst diff --git a/Documentation/driver-api/cxl/linux/cxl-driver.rst b/Documentat= ion/driver-api/cxl/linux/cxl-driver.rst index 9a9e8ecee578..486baf8551aa 100644 --- a/Documentation/driver-api/cxl/linux/cxl-driver.rst +++ b/Documentation/driver-api/cxl/linux/cxl-driver.rst @@ -520,3 +520,13 @@ from HPA to DPA. This is why they must be aware of th= e entire interleave set. =20 Linux does not support unbalanced interleave configurations. As a result,= all endpoints in an interleave set must have the same ways and granularity. + +Example Configurations +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +.. toctree:: + :maxdepth: 1 + + example-configurations/single-device.rst + example-configurations/hb-interleave.rst + example-configurations/intra-hb-interleave.rst + example-configurations/multi-interleave.rst diff --git a/Documentation/driver-api/cxl/linux/example-configurations/hb-i= nterleave.rst b/Documentation/driver-api/cxl/linux/example-configurations/h= b-interleave.rst new file mode 100644 index 000000000000..f071490763a2 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/example-configurations/hb-interlea= ve.rst @@ -0,0 +1,314 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +Inter-Host-Bridge Interleave +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +This cxl-cli configuration dump shows the following host configuration: + +* A single socket system with one CXL root +* CXL Root has Four (4) CXL Host Bridges +* Two CXL Host Bridges have a single CXL Memory Expander Attached +* The CXL root is configured to interleave across the two host bridges. + +This output is generated by :code:`cxl list -v` and describes the relation= ships +between objects exposed in :code:`/sys/bus/cxl/devices/`. + +:: + + [ + { + "bus":"root0", + "provider":"ACPI.CXL", + "nr_dports":4, + "dports":[ + { + "dport":"pci0000:00", + "alias":"ACPI0016:01", + "id":0 + }, + { + "dport":"pci0000:a8", + "alias":"ACPI0016:02", + "id":4 + }, + { + "dport":"pci0000:2a", + "alias":"ACPI0016:03", + "id":1 + }, + { + "dport":"pci0000:d2", + "alias":"ACPI0016:00", + "id":5 + } + ], + +This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to = CXL +Host Bridges. The `Root` can be considered the singular upstream port att= ached +to the platform's memory controller - which routes memory requests to it. + +The `ports:root0` section lays out how each of these downstream ports are +configured. If a port is not configured (id's 0 and 1), they are omitted. + +:: + + "ports:root0":[ + { + "port":"port1", + "host":"pci0000:d2", + "depth":1, + "nr_dports":3, + "dports":[ + { + "dport":"0000:d2:01.1", + "alias":"device:02", + "id":0 + }, + { + "dport":"0000:d2:01.3", + "alias":"device:05", + "id":2 + }, + { + "dport":"0000:d2:07.1", + "alias":"device:0d", + "id":113 + } + ], + +This chunk shows the available downstream ports associated with the CXL Ho= st +Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstr= eam +ports: :code:`dport1`, :code:`dport2`, and :code:`dport113`.. + +:: + + "endpoints:port1":[ + { + "endpoint":"endpoint5", + "host":"mem0", + "parent_dport":"0000:d2:01.1", + "depth":2, + "memdev":{ + "memdev":"mem0", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:d3:00.0" + }, + "decoders:endpoint5":[ + { + "decoder":"decoder5.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + +This chunk shows the endpoints attached to the host bridge :code:`port1`. + +:code:`endpoint5` contains a single configured decoder :code:`decoder5.0` +which has the same interleave configuration as :code:`region0` (shown late= r). + +Next we have the decodesr belonging to the host bridge: + +:: + + "decoders:port1":[ + { + "decoder":"decoder1.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":1, + "region":"region0", + "nr_targets":1, + "targets":[ + { + "target":"0000:d2:01.1", + "alias":"device:02", + "position":0, + "id":0 + } + ] + } + ] + }, + +Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`), whose= only +target is :code:`dport1` - which is attached to :code:`endpoint5`. + +The following chunk shows a similar configuration for Host Bridge :code:`p= ort3`, +the second host bridge with a memory device attached. + +:: + + { + "port":"port3", + "host":"pci0000:a8", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:a8:01.1", + "alias":"device:c3", + "id":0 + } + ], + "endpoints:port3":[ + { + "endpoint":"endpoint6", + "host":"mem1", + "parent_dport":"0000:a8:01.1", + "depth":2, + "memdev":{ + "memdev":"mem1", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:a9:00.0" + }, + "decoders:endpoint6":[ + { + "decoder":"decoder6.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + "decoders:port3":[ + { + "decoder":"decoder3.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":1, + "region":"region0", + "nr_targets":1, + "targets":[ + { + "target":"0000:a8:01.1", + "alias":"device:c3", + "position":0, + "id":0 + } + ] + } + ] + }, + + +The next chunk shows the two CXL host bridges without attached endpoints. + +:: + + { + "port":"port2", + "host":"pci0000:00", + "depth":1, + "nr_dports":2, + "dports":[ + { + "dport":"0000:00:01.3", + "alias":"device:55", + "id":2 + }, + { + "dport":"0000:00:07.1", + "alias":"device:5d", + "id":113 + } + ] + }, + { + "port":"port4", + "host":"pci0000:2a", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:2a:01.1", + "alias":"device:d0", + "id":0 + } + ] + } + ], + +Next we have the `Root Decoders` belonging to :code:`root0`. This root de= coder +applies the interleave across the downstream ports :code:`port1` and +:code:`port3` - with a granularity of 256 bytes. + +This information is generated by the CXL driver reading the ACPI CEDT CMFW= S. + +:: + + "decoders:root0":[ + { + "decoder":"decoder0.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "max_available_extent":0, + "volatile_capable":true, + "nr_targets":2, + "targets":[ + { + "target":"pci0000:a8", + "alias":"ACPI0016:02", + "position":1, + "id":4 + }, + { + "target":"pci0000:d2", + "alias":"ACPI0016:00", + "position":0, + "id":5 + } + ], + +Finally we have the `Memory Region` associated with the `Root Decoder` +:code:`decoder0.0`. This region describes the overall interleave configur= ation +of the interleave set. + +:: + + "regions:decoder0.0":[ + { + "region":"region0", + "resource":825975898112, + "size":274877906944, + "type":"ram", + "interleave_ways":2, + "interleave_granularity":256, + "decode_state":"commit", + "mappings":[ + { + "position":1, + "memdev":"mem1", + "decoder":"decoder6.0" + }, + { + "position":0, + "memdev":"mem0", + "decoder":"decoder5.0" + } + ] + } + ] + } + ] + } + ] diff --git a/Documentation/driver-api/cxl/linux/example-configurations/intr= a-hb-interleave.rst b/Documentation/driver-api/cxl/linux/example-configurat= ions/intra-hb-interleave.rst new file mode 100644 index 000000000000..077dfaf8458d --- /dev/null +++ b/Documentation/driver-api/cxl/linux/example-configurations/intra-hb-in= terleave.rst @@ -0,0 +1,291 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +Intra-Host-Bridge Interleave +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +This cxl-cli configuration dump shows the following host configuration: + +* A single socket system with one CXL root +* CXL Root has Four (4) CXL Host Bridges +* One (1) CXL Host Bridges has two CXL Memory Expanders Attached +* The Host bridge decoder is programmed to interleave across the expanders. + +This output is generated by :code:`cxl list -v` and describes the relation= ships +between objects exposed in :code:`/sys/bus/cxl/devices/`. + +:: + + [ + { + "bus":"root0", + "provider":"ACPI.CXL", + "nr_dports":4, + "dports":[ + { + "dport":"pci0000:00", + "alias":"ACPI0016:01", + "id":0 + }, + { + "dport":"pci0000:a8", + "alias":"ACPI0016:02", + "id":4 + }, + { + "dport":"pci0000:2a", + "alias":"ACPI0016:03", + "id":1 + }, + { + "dport":"pci0000:d2", + "alias":"ACPI0016:00", + "id":5 + } + ], + +This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to = CXL +Host Bridges. The `Root` can be considered the singular upstream port att= ached +to the platform's memory controller - which routes memory requests to it. + +The `ports:root0` section lays out how each of these downstream ports are +configured. If a port is not configured (id's 0 and 1), they are omitted. + +:: + + "ports:root0":[ + { + "port":"port1", + "host":"pci0000:d2", + "depth":1, + "nr_dports":3, + "dports":[ + { + "dport":"0000:d2:01.1", + "alias":"device:02", + "id":0 + }, + { + "dport":"0000:d2:01.3", + "alias":"device:05", + "id":2 + }, + { + "dport":"0000:d2:07.1", + "alias":"device:0d", + "id":113 + } + ], + +This chunk shows the available downstream ports associated with the CXL Ho= st +Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstr= eam +ports: :code:`dport1`, :code:`dport2`, and :code:`dport113`.. + +:: + + "endpoints:port1":[ + { + "endpoint":"endpoint5", + "host":"mem0", + "parent_dport":"0000:d2:01.1", + "depth":2, + "memdev":{ + "memdev":"mem0", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:d3:00.0" + }, + "decoders:endpoint5":[ + { + "decoder":"decoder5.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + }, + { + "endpoint":"endpoint6", + "host":"mem1", + "parent_dport":"0000:d2:01.3, + "depth":2, + "memdev":{ + "memdev":"mem1", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:a9:00.0" + }, + "decoders:endpoint6":[ + { + "decoder":"decoder6.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + +This chunk shows the endpoints attached to the host bridge :code:`port1`. + +:code:`endpoint5` contains a single configured decoder :code:`decoder5.0` +which has the same interleave configuration memory region they belong to +(show later). + +Next we have the decoders belonging to the host bridge: + +:: + + "decoders:port1":[ + { + "decoder":"decoder1.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "region":"region0", + "nr_targets":2, + "targets":[ + { + "target":"0000:d2:01.1", + "alias":"device:02", + "position":0, + "id":0 + }, + { + "target":"0000:d2:01.3", + "alias":"device:05", + "position":1, + "id":0 + } + ] + } + ] + }, + +Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`) with t= wo +targets: :code:`dport1` and :code:`dport3` - which are attached to +:code:`endpoint5` and :code:`endpoint6` respectively. + +The host bridge decoder interleaves these devices at a 256 byte granularit= y. + +The next chunk shows the three CXL host bridges without attached endpoints. + +:: + + { + "port":"port2", + "host":"pci0000:00", + "depth":1, + "nr_dports":2, + "dports":[ + { + "dport":"0000:00:01.3", + "alias":"device:55", + "id":2 + }, + { + "dport":"0000:00:07.1", + "alias":"device:5d", + "id":113 + } + ] + }, + { + "port":"port3", + "host":"pci0000:a8", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:a8:01.1", + "alias":"device:c3", + "id":0 + } + ], + }, + { + "port":"port4", + "host":"pci0000:2a", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:2a:01.1", + "alias":"device:d0", + "id":0 + } + ] + } + ], + +Next we have the `Root Decoders` belonging to :code:`root0`. This root de= coder +applies the interleave across the downstream ports :code:`port1` and +:code:`port3` - with a granularity of 256 bytes. + +This information is generated by the CXL driver reading the ACPI CEDT CMFW= S. + +:: + + "decoders:root0":[ + { + "decoder":"decoder0.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":1, + "max_available_extent":0, + "volatile_capable":true, + "nr_targets":2, + "targets":[ + { + "target":"pci0000:a8", + "alias":"ACPI0016:02", + "position":1, + "id":4 + }, + ], + +Finally we have the `Memory Region` associated with the `Root Decoder` +:code:`decoder0.0`. This region describes the overall interleave configur= ation +of the interleave set. + +:: + + "regions:decoder0.0":[ + { + "region":"region0", + "resource":825975898112, + "size":274877906944, + "type":"ram", + "interleave_ways":2, + "interleave_granularity":256, + "decode_state":"commit", + "mappings":[ + { + "position":1, + "memdev":"mem1", + "decoder":"decoder6.0" + }, + { + "position":0, + "memdev":"mem0", + "decoder":"decoder5.0" + } + ] + } + ] + } + ] + } + ] diff --git a/Documentation/driver-api/cxl/linux/example-configurations/mult= i-interleave.rst b/Documentation/driver-api/cxl/linux/example-configuration= s/multi-interleave.rst new file mode 100644 index 000000000000..008f9053c630 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/example-configurations/multi-inter= leave.rst @@ -0,0 +1,401 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Multi-Level Interleave +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +This cxl-cli configuration dump shows the following host configuration: + +* A single socket system with one CXL root +* CXL Root has Four (4) CXL Host Bridges +* Two CXL Host Bridges have a two CXL Memory Expanders Attached each. +* The CXL root is configured to interleave across the two host bridges. +* Each host bridge with expanders interleaves across two endpoints. + +This output is generated by :code:`cxl list -v` and describes the relation= ships +between objects exposed in :code:`/sys/bus/cxl/devices/`. + +:: + + [ + { + "bus":"root0", + "provider":"ACPI.CXL", + "nr_dports":4, + "dports":[ + { + "dport":"pci0000:00", + "alias":"ACPI0016:01", + "id":0 + }, + { + "dport":"pci0000:a8", + "alias":"ACPI0016:02", + "id":4 + }, + { + "dport":"pci0000:2a", + "alias":"ACPI0016:03", + "id":1 + }, + { + "dport":"pci0000:d2", + "alias":"ACPI0016:00", + "id":5 + } + ], + +This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to = CXL +Host Bridges. The `Root` can be considered the singular upstream port att= ached +to the platform's memory controller - which routes memory requests to it. + +The `ports:root0` section lays out how each of these downstream ports are +configured. If a port is not configured (id's 0 and 1), they are omitted. + +:: + + "ports:root0":[ + { + "port":"port1", + "host":"pci0000:d2", + "depth":1, + "nr_dports":3, + "dports":[ + { + "dport":"0000:d2:01.1", + "alias":"device:02", + "id":0 + }, + { + "dport":"0000:d2:01.3", + "alias":"device:05", + "id":2 + }, + { + "dport":"0000:d2:07.1", + "alias":"device:0d", + "id":113 + } + ], + +This chunk shows the available downstream ports associated with the CXL Ho= st +Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstr= eam +ports: :code:`dport0`, :code:`dport2`, and :code:`dport113`. + +:: + + "endpoints:port1":[ + { + "endpoint":"endpoint5", + "host":"mem0", + "parent_dport":"0000:d2:01.1", + "depth":2, + "memdev":{ + "memdev":"mem0", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:d3:00.0" + }, + "decoders:endpoint5":[ + { + "decoder":"decoder5.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":4, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + }, + { + "endpoint":"endpoint6", + "host":"mem1", + "parent_dport":"0000:d2:01.3", + "depth":2, + "memdev":{ + "memdev":"mem1", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:d3:00.0" + }, + "decoders:endpoint6":[ + { + "decoder":"decoder6.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":4, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + +This chunk shows the endpoints attached to the host bridge :code:`port1`. + +:code:`endpoint5` contains a single configured decoder :code:`decoder5.0` +which has the same interleave configuration as :code:`region0` (shown late= r). + +:code:`endpoint6` contains a single configured decoder :code:`decoder5.0` +which has the same interleave configuration as :code:`region0` (shown late= r). + +Next we have the decoders belonging to the host bridge: + +:: + + "decoders:port1":[ + { + "decoder":"decoder1.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":2, + "interleave_granularity":512, + "region":"region0", + "nr_targets":2, + "targets":[ + { + "target":"0000:d2:01.1", + "alias":"device:02", + "position":0, + "id":0 + }, + { + "target":"0000:d2:01.3", + "alias":"device:05", + "position":2, + "id":0 + } + ] + } + ] + }, + +Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`), whose +targets are :code:`dport0` and :code:`dport2` - which are attached to +:code:`endpoint5` and :code:`endpoint6` respectively. + +The following chunk shows a similar configuration for Host Bridge :code:`p= ort3`, +the second host bridge with a memory device attached. + +:: + + { + "port":"port3", + "host":"pci0000:a8", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:a8:01.1", + "alias":"device:c3", + "id":0 + }, + { + "dport":"0000:a8:01.3", + "alias":"device:c5", + "id":0 + } + ], + "endpoints:port3":[ + { + "endpoint":"endpoint7", + "host":"mem2", + "parent_dport":"0000:a8:01.1", + "depth":2, + "memdev":{ + "memdev":"mem2", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:a9:00.0" + }, + "decoders:endpoint7":[ + { + "decoder":"decoder7.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":4, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + }, + { + "endpoint":"endpoint8", + "host":"mem3", + "parent_dport":"0000:a8:01.3", + "depth":2, + "memdev":{ + "memdev":"mem3", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:a9:00.0" + }, + "decoders:endpoint8":[ + { + "decoder":"decoder8.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":4, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + "decoders:port3":[ + { + "decoder":"decoder3.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":2, + "interleave_granularity":512, + "region":"region0", + "nr_targets":1, + "targets":[ + { + "target":"0000:a8:01.1", + "alias":"device:c3", + "position":1, + "id":0 + }, + { + "target":"0000:a8:01.3", + "alias":"device:c5", + "position":3, + "id":0 + } + ] + } + ] + }, + + +The next chunk shows the two CXL host bridges without attached endpoints. + +:: + + { + "port":"port2", + "host":"pci0000:00", + "depth":1, + "nr_dports":2, + "dports":[ + { + "dport":"0000:00:01.3", + "alias":"device:55", + "id":2 + }, + { + "dport":"0000:00:07.1", + "alias":"device:5d", + "id":113 + } + ] + }, + { + "port":"port4", + "host":"pci0000:2a", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:2a:01.1", + "alias":"device:d0", + "id":0 + } + ] + } + ], + +Next we have the `Root Decoders` belonging to :code:`root0`. This root de= coder +applies the interleave across the downstream ports :code:`port1` and +:code:`port3` - with a granularity of 256 bytes. + +This information is generated by the CXL driver reading the ACPI CEDT CMFW= S. + +:: + + "decoders:root0":[ + { + "decoder":"decoder0.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":2, + "interleave_granularity":256, + "max_available_extent":0, + "volatile_capable":true, + "nr_targets":2, + "targets":[ + { + "target":"pci0000:a8", + "alias":"ACPI0016:02", + "position":1, + "id":4 + }, + { + "target":"pci0000:d2", + "alias":"ACPI0016:00", + "position":0, + "id":5 + } + ], + +Finally we have the `Memory Region` associated with the `Root Decoder` +:code:`decoder0.0`. This region describes the overall interleave configur= ation +of the interleave set. So we see there are a total of :code:`4` interleave +targets across 4 endpoint decoders. + +:: + + "regions:decoder0.0":[ + { + "region":"region0", + "resource":825975898112, + "size":549755813888, + "type":"ram", + "interleave_ways":4, + "interleave_granularity":256, + "decode_state":"commit", + "mappings":[ + { + "position":3, + "memdev":"mem3", + "decoder":"decoder8.0" + }, + { + "position":2, + "memdev":"mem1", + "decoder":"decoder6.0" + } + { + "position":1, + "memdev":"mem2", + "decoder":"decoder7.0" + }, + { + "position":0, + "memdev":"mem0", + "decoder":"decoder5.0" + } + ] + } + ] + } + ] + } + ] diff --git a/Documentation/driver-api/cxl/linux/example-configurations/sing= le-device.rst b/Documentation/driver-api/cxl/linux/example-configurations/s= ingle-device.rst new file mode 100644 index 000000000000..5fd38eb0aaf4 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/example-configurations/single-devi= ce.rst @@ -0,0 +1,246 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Single Device +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +This cxl-cli configuration dump shows the following host configuration: + +* A single socket system with one CXL root +* CXL Root has Four (4) CXL Host Bridges +* One CXL Host Bridges has a single CXL Memory Expander Attached +* No interleave is present. + +This output is generated by :code:`cxl list -v` and describes the relation= ships +between objects exposed in :code:`/sys/bus/cxl/devices/`. + +:: + + [ + { + "bus":"root0", + "provider":"ACPI.CXL", + "nr_dports":4, + "dports":[ + { + "dport":"pci0000:00", + "alias":"ACPI0016:01", + "id":0 + }, + { + "dport":"pci0000:a8", + "alias":"ACPI0016:02", + "id":4 + }, + { + "dport":"pci0000:2a", + "alias":"ACPI0016:03", + "id":1 + }, + { + "dport":"pci0000:d2", + "alias":"ACPI0016:00", + "id":5 + } + ], + +This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to = CXL +Host Bridges. The `Root` can be considered the singular upstream port att= ached +to the platform's memory controller - which routes memory requests to it. + +The `ports:root0` section lays out how each of these downstream ports are +configured. If a port is not configured (id's 0, 1, and 4), they are omit= ted. + +:: + + "ports:root0":[ + { + "port":"port1", + "host":"pci0000:d2", + "depth":1, + "nr_dports":3, + "dports":[ + { + "dport":"0000:d2:01.1", + "alias":"device:02", + "id":0 + }, + { + "dport":"0000:d2:01.3", + "alias":"device:05", + "id":2 + }, + { + "dport":"0000:d2:07.1", + "alias":"device:0d", + "id":113 + } + ], + +This chunk shows the available downstream ports associated with the CXL Ho= st +Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstr= eam +ports: :code:`dport1`, :code:`dport2`, and :code:`dport113`.. + +:: + + "endpoints:port1":[ + { + "endpoint":"endpoint5", + "host":"mem0", + "parent_dport":"0000:d2:01.1", + "depth":2, + "memdev":{ + "memdev":"mem0", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:d3:00.0" + }, + "decoders:endpoint5":[ + { + "decoder":"decoder5.0", + "resource":825975898112, + "size":137438953472, + "interleave_ways":1, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + +This chunk shows the endpoints attached to the host bridge :code:`port1`. + +:code:`endpoint5` contains a single configured decoder :code:`decoder5.0` +which has the same interleave configuration as :code:`region0` (shown late= r). + +Next we have the decoders belonging to the host bridge: + +:: + + "decoders:port1":[ + { + "decoder":"decoder1.0", + "resource":825975898112, + "size":137438953472, + "interleave_ways":1, + "region":"region0", + "nr_targets":1, + "targets":[ + { + "target":"0000:d2:01.1", + "alias":"device:02", + "position":0, + "id":0 + } + ] + } + ] + }, + +Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`), whose= only +target is :code:`dport1` - which is attached to :code:`endpoint5`. + +The next chunk shows the three CXL host bridges without attached endpoints. + +:: + + { + "port":"port2", + "host":"pci0000:00", + "depth":1, + "nr_dports":2, + "dports":[ + { + "dport":"0000:00:01.3", + "alias":"device:55", + "id":2 + }, + { + "dport":"0000:00:07.1", + "alias":"device:5d", + "id":113 + } + ] + }, + { + "port":"port3", + "host":"pci0000:a8", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:a8:01.1", + "alias":"device:c3", + "id":0 + } + ] + }, + { + "port":"port4", + "host":"pci0000:2a", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:2a:01.1", + "alias":"device:d0", + "id":0 + } + ] + } + ], + +Next we have the `Root Decoders` belonging to :code:`root0`. This root de= coder +is a pass-through decoder because :code:`interleave_ways` is set to :code:= `1`. + +This information is generated by the CXL driver reading the ACPI CEDT CMFW= S. + +:: + + "decoders:root0":[ + { + "decoder":"decoder0.0", + "resource":825975898112, + "size":137438953472, + "interleave_ways":1, + "max_available_extent":0, + "volatile_capable":true, + "nr_targets":1, + "targets":[ + { + "target":"pci0000:d2", + "alias":"ACPI0016:00", + "position":0, + "id":5 + } + ], + +Finally we have the `Memory Region` associated with the `Root Decoder` +:code:`decoder0.0`. This region describes the discrete region associated +with the lone device. + +:: + + "regions:decoder0.0":[ + { + "region":"region0", + "resource":825975898112, + "size":137438953472, + "type":"ram", + "interleave_ways":1, + "decode_state":"commit", + "mappings":[ + { + "position":0, + "memdev":"mem0", + "decoder":"decoder5.0" + } + ] + } + ] + } + ] + } + ] --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76A7F253347 for ; Mon, 12 May 2025 16:22:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066924; cv=none; b=UUFQEhur7hpU+eBd43zr6JHHmeO6bKkbNJcaElh8XGgSXB1Rfuxb3PYEcLjq5o4R47XC6SOaPCkD+Gy6YarhRV9aL37HsW7X8yCqbQrY2l6DpRsueOZHof3rhSP8o3WjPOv4MZzRGWQ3L79pc0po0WcVxXtGX96cDEXe+ZDYJKI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066924; c=relaxed/simple; bh=BxgsIiIVKTAXyq2szFAwqx4FiL+UZIkTpCPzLKiU6Yk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KePgQRViakYgQDfx0HoM32rgi+3LMFVW7NpAlhUrzPKAespb1/pEFv4Th6IHY0hKXBjuzYxXGQl7FpKmPFcydrG+kcpa2iIyN1mkYXhw4/bBrFLXc5qdEFdiNCmtw68ujMFlYt36NcoDZI+2Cn+tnNq8SI9XfNZXcnsyRruSpnI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=U+cJ7sQq; arc=none smtp.client-ip=209.85.222.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="U+cJ7sQq" Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-7c7913bab2cso541753085a.0 for ; Mon, 12 May 2025 09:22:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066919; x=1747671719; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YMLAIeu20ZZPJ1i/m+n96//lJvh/XtQZo0nMeJDEoyw=; b=U+cJ7sQq/KemHu+QeCECIh104kAm7aPa38hW1gsAtX6ushMG6wZaAPPwhxhV1q2P4d dOFwIpT72dWCAdcm7XYP1PfFYwl1gFDP0kONQrBOY0mPXRaFW5D5AXBihyrnnuNsOU+g 4ZRfLN6/fSkj7V/CVKZeS9mx28qopawYsFDjCeb0ZitHqrOSD95l8mDrl0yiHPeiD1x/ NIBD/TCs1RsJ6OR3dL3g/Wv4jGXFIoDG+E2rqmK+/EdBQOd+nAROpGJN0Ta3VX2nAu0U 3GGxZiVQEhrVsChRQZuHYTxSHlcmTNegihIp2mYfXAeJSEBsi7KsqtlvANTiezuHUWY4 8LCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066919; x=1747671719; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YMLAIeu20ZZPJ1i/m+n96//lJvh/XtQZo0nMeJDEoyw=; b=cHTa+WAaW9tV52xEUr78n7lXYkJW65srn4vhZ9cXIDXkL6tJk4DbdEi0PdU7Vl442a kmCppMmR7QNXKVAKxYjuyHF5fVdTV1Rm9qxwPLaNPZp7mruRW0hoiPxEGgHF5AMaWbK6 m3zGJXeWJLEVGxFSfQhTDFcnmifzrKj16UnoZAQZ691PUr40i6aa8mSWV8hmRTjrJe9h b/hXQwE7j5waBrrqjPqBX6fMlYfiGNZdo+0PLtA9Cz2rJHKK2O49zQ07shpAq8IQD7aL mc2QDSrKC/inzjkdmKzhSj5qDXNpFMq3UblZsDbqyzter8vcjft0Db4v1S2E4oaDZbuf /3uQ== X-Forwarded-Encrypted: i=1; AJvYcCXTwhOP8hikUY6rAgqgYqbGPTQdLX4kR1mVV5KjLYcoL7diDf1eychav0nPC3uVcbj4axnA0Bkq1xIiRJI=@vger.kernel.org X-Gm-Message-State: AOJu0Yyhbci2/YD0hHdgPbSWNUbyFBSkD7VW+G8RNK/w9Sk6o4LvhLUy 5x/86C++XyzZk5we5+j/u0SP/K2FcEu8lC3EN5CjMfHILiuZ912PfHmc5eyg/JTTmtvfdGiA/QS P X-Gm-Gg: ASbGncuLrgdG5btZjzTCT5NuxL0Oeuks0iAkvrorbDGqLC+vSqNZXLJXcR9GD/T00QV sdQBt04lKWdAwNJDLAbXZO6mSznGGXlM5tkv8SLqb4O17GOiT76kwtxVz/m5mwY0O4OuuAQWJeB 2onZ1nmw05dLERvUKj2R5W775M6V83KBF8vDas0Ve/s15+rG/GsbkqIt0ZyJz0YRnpyTGG2VEWT mdNxm7aOR9aKUnBK+s/rNLs4IEXmTmEHAH57FP9nlfljcpZp6srlwIXJORyRLSOIrxb6yHVIQkh vB1omj5XYD6sTX5zf3uReFfqgB75aLoHFCO+Xm/LDYZxLvCLvtMFXfmGlApP1gylZiXQmqIiVk8 /xQafwOgXjbe/VhwhxEdrMMkkOQMA0Tt3EK2l X-Google-Smtp-Source: AGHT+IGK3lfjM4oSpvVjN4nvNUpXvpjzwqjwnlVVwpRdQ+DJ0SjErWBHozx2SdjcQ9/faPIK34NbmQ== X-Received: by 2002:a05:620a:2586:b0:7c7:a5cb:2b65 with SMTP id af79cd13be357-7cd01101edfmr2018895985a.26.1747066919098; Mon, 12 May 2025 09:21:59 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:21:58 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 11/17] cxl: docs/linux/dax-driver documentation Date: Mon, 12 May 2025 12:21:28 -0400 Message-ID: <20250512162134.3596150-12-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation on how the CXL driver interacts with the DAX driver. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/linux/cxl-driver.rst | 115 ++++++++++++++++-- .../driver-api/cxl/linux/dax-driver.rst | 43 +++++++ 3 files changed, 149 insertions(+), 10 deletions(-) create mode 100644 Documentation/driver-api/cxl/linux/dax-driver.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index df3c7763c79a..f2127968ea78 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -36,6 +36,7 @@ that have impacts on each other. The docs here break up = configurations steps. linux/overview linux/early-boot linux/cxl-driver + linux/dax-driver linux/access-coordinates =20 =20 diff --git a/Documentation/driver-api/cxl/linux/cxl-driver.rst b/Documentat= ion/driver-api/cxl/linux/cxl-driver.rst index 486baf8551aa..cf6b397abdb1 100644 --- a/Documentation/driver-api/cxl/linux/cxl-driver.rst +++ b/Documentation/driver-api/cxl/linux/cxl-driver.rst @@ -34,6 +34,32 @@ into a single memory region. The memory region has been = converted to dax. :: decoder1.0 decoder5.0 endpoint5 port1 region0 decoder2.0 decoder5.1 endpoint6 port2 root0 =20 + +.. kernel-render:: DOT + :alt: Digraph of CXL fabric describing host-bridge interleaving + :caption: Diagraph of CXL fabric with a host-bridge interleave memory r= egion + + digraph foo { + "root0" -> "port1"; + "root0" -> "port3"; + "root0" -> "decoder0.0"; + "port1" -> "endpoint5"; + "port3" -> "endpoint6"; + "port1" -> "decoder1.0"; + "port3" -> "decoder3.0"; + "endpoint5" -> "decoder5.0"; + "endpoint6" -> "decoder6.0"; + "decoder0.0" -> "region0"; + "decoder0.0" -> "decoder1.0"; + "decoder0.0" -> "decoder3.0"; + "decoder1.0" -> "decoder5.0"; + "decoder3.0" -> "decoder6.0"; + "decoder5.0" -> "region0"; + "decoder6.0" -> "region0"; + "region0" -> "dax_region0"; + "dax_region0" -> "dax0.0"; + } + For this section we'll explore the devices present in this configuration, = but we'll explore more configurations in-depth in example configurations below. =20 @@ -41,7 +67,7 @@ Base Devices ------------ Most devices in a CXL fabric are a `port` of some kind (because each device mostly routes request from one device to the next, rather than -provide a manageable service). +provide a direct service). =20 Root ~~~~ @@ -53,6 +79,8 @@ The Root contains links to: =20 * `Host Bridge Ports` defined by ACPI CEDT CHBS. =20 +* `Downstream Ports` typically connected to `Host Bridge Ports`. + * `Root Decoders` defined by ACPI CEDT CFMWS. =20 :: @@ -150,6 +178,27 @@ device configuration data. :: driver label_storage_size pmem serial firmware numa_node ram subsystem =20 +A Memory Device is a discrete base object that is not a port. While the +physical device it belongs to may also host an `endpoint`, the relationship +between an `endpoint` and a `memdev` is not captured in sysfs. + +Port Relationships +~~~~~~~~~~~~~~~~~~ +In our example described above, there are four host bridges attached to the +root, and two of the host bridges have one endpoint attached. + +.. kernel-render:: DOT + :alt: Digraph of CXL fabric describing host-bridge interleaving + :caption: Diagraph of CXL fabric with a host-bridge interleave memory r= egion + + digraph foo { + "root0" -> "port1"; + "root0" -> "port2"; + "root0" -> "port3"; + "root0" -> "port4"; + "port1" -> "endpoint5"; + "port3" -> "endpoint6"; + } =20 Decoders -------- @@ -322,6 +371,29 @@ settings (granularity and ways must be the same). Endpoint decoders are created during :code:`cxl_endpoint_port_probe` in the :code:`cxl_port` driver, and is created based on a PCI device's DVSEC regi= sters. =20 +Decoder Relationships +~~~~~~~~~~~~~~~~~~~~~ +In our example described above, there is one root decoder which routes mem= ory +accesses over two host bridges. Each host bridge has a decoder which rout= es +access to their singular endpoint targets. Each endpoint has a decoder wh= ich +translates HPA to DPA and services the memory request. + +The driver validates relationships between ports by decoder programming, so +we can think of decoders being related in a similarly hierarchical fashion= to +ports. + +.. kernel-render:: DOT + :alt: Digraph of hierarchical relationship between root, switch, and en= dpoint decoders. + :caption: Diagraph of CXL root, switch, and endpoint decoders. + + digraph foo { + "root0" -> "decoder0.0"; + "decoder0.0" -> "decoder1.0"; + "decoder0.0" -> "decoder3.0"; + "decoder1.0" -> "decoder5.0"; + "decoder3.0" -> "decoder6.0"; + } + Regions ------- =20 @@ -348,6 +420,17 @@ The interleave settings in a `Memory Region` describe = the configuration of the `Interleave Set` - and are what can be expected to be seen in the endpoint interleave settings. =20 +.. kernel-render:: DOT + :alt: Digraph of CXL memory region relationships between root and endpo= int decoders. + :caption: Regions are created based on root decoder configurations. End= point decoders + must be programmed with the same interleave settings as the r= egion. + + digraph foo { + "root0" -> "decoder0.0"; + "decoder0.0" -> "region0"; + "region0" -> "decoder5.0"; + "region0" -> "decoder6.0"; + } =20 DAX Region ~~~~~~~~~~ @@ -360,7 +443,6 @@ for more details. :: dax0.0 devtype modalias uevent dax_region driver subsystem =20 - Mailbox Interfaces ------------------ A mailbox command interface for each device is exposed in :: @@ -418,17 +500,30 @@ the relationships between a decoder and it's parent. =20 For example, in a `Cross-Link First` interleave setup with 16 endpoints attached to 4 host bridges, linux expects the following ways/granularity -across the root, host bridge, and endpoints respectively. :: +across the root, host bridge, and endpoints respectively. + +.. flat-table:: 4x4 cross-link first interleave settings + + * - decoder + - ways + - granularity =20 - ways granularity - root 4 256 - host bridge 4 1024 - endpoint 16 256 + * - root + - 4 + - 256 + + * - host bridge + - 4 + - 1024 + + * - endpoint + - 16 + - 256 =20 At the root, every a given access will be routed to the :code:`((HPA / 256) % 4)th` target host bridge. Within a host bridge, every -:code:`((HPA / 1024) % 4)th` target endpoint. Each endpoint will translate -the access based on the entire 16 device interleave set. +:code:`((HPA / 1024) % 4)th` target endpoint. Each endpoint translates ba= sed +on the entire 16 device interleave set. =20 Unbalanced interleave sets are not supported - decoders at a similar point in the hierarchy (e.g. all host bridge decoders) must have the same ways a= nd @@ -467,7 +562,7 @@ In this example, the CFMWS defines two discrete non-int= erleaved 4GB regions for each host bridge, and one interleaved 8GB region that targets both. Th= is would result in 3 root decoders presenting in the root. :: =20 - # ls /sys/bus/cxl/devices/root0 + # ls /sys/bus/cxl/devices/root0/decoder* decoder0.0 decoder0.1 decoder0.2 =20 # cat /sys/bus/cxl/devices/decoder0.0/target_list start size diff --git a/Documentation/driver-api/cxl/linux/dax-driver.rst b/Documentat= ion/driver-api/cxl/linux/dax-driver.rst new file mode 100644 index 000000000000..10d953a2167b --- /dev/null +++ b/Documentation/driver-api/cxl/linux/dax-driver.rst @@ -0,0 +1,43 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +DAX Driver Operation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The `Direct Access Device` driver was originally designed to provide a +memory-like access mechanism to memory-like block-devices. It was +extended to support CXL Memory Devices, which provide user-configured +memory devices. + +The CXL subsystem depends on the DAX subsystem to either: + +- Generate a file-like interface to userland via :code:`/dev/daxN.Y`, or +- Engage the memory-hotplug interface to add CXL memory to page allocator. + +The DAX subsystem exposes this ability through the `cxl_dax_region` driver. +A `dax_region` provides the translation between a CXL `memory_region` and +a `DAX Device`. + +DAX Device +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +A `DAX Device` is a file-like interface exposed in :code:`/dev/daxN.Y`. A +memory region exposed via dax device can be accessed via userland software +via the :code:`mmap()` system-call. The result is direct mappings to the +CXL capacity in the task's page tables. + +Users wishing to manually handle allocation of CXL memory should use this +interface. + +kmem conversion +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The :code:`dax_kmem` driver converts a `DAX Device` into a series of `hotp= lug +memory blocks` managed by :code:`kernel/memory-hotplug.c`. This capacity +will be exposed to the kernel page allocator in the user-selected memory +zone. + +The :code:`memmap_on_memory` setting (both global and DAX device local) +dictates where the kernell will allocate the :code:`struct folio` descript= ors +for this memory will come from. If :code:`memmap_on_memory` is set, memory +hotplug will set aside a portion of the memory block capacity to allocate +folios. If unset, the memory is allocated via a normal :code:`GFP_KERNEL` +allocation - and as a result will most likely land on the local NUM node o= f the +CPU executing the hotplug operation. --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23B722505CD for ; Mon, 12 May 2025 16:22:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066934; cv=none; b=AEHtG51bQdPcsuFGcF7wz/2CGHpmNNbTXYcW89JBebfrt0drhuHOuk/3rkXib3X/PraPoZ5yflob1SK5NoLMiI2qtEmws4Qk1/KtIBc1xtoZu8VjERBARrOl5dxPTNI1tzl1oXYos5UYEEHe9ikTLdEkUZ8HIStcFQWWMecnVXI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066934; c=relaxed/simple; bh=AhTLdqyE2De1b9HJN76/C5lAERWGT4jHTFZhEoUDbtk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=a/IUSvu3eC1jt/2qt12JtrlFze9fdpppq9na3UvTYyjLiRO0l5grFUtUMqWTijTILpEmCF1w9Bg8bSqAqKhhnh9OYRj7Afgrl8zIymQzIqp02GqvkgvHLF3dTd6UmemJGzFpKHr7mwlQn9wQ1afqdczeNi1RqH3Nwk/Fh5nYSLo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=naJCUgy1; arc=none smtp.client-ip=209.85.160.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="naJCUgy1" Received: by mail-qt1-f171.google.com with SMTP id d75a77b69052e-476a1acf61eso50527611cf.1 for ; Mon, 12 May 2025 09:22:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066932; x=1747671732; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xF2+54Is9Y/RM3HTLl2GmOBtMHoFbKc3iQ93j5V9CWg=; b=naJCUgy1KWk+rmjdU4UXP/sXKVzlM6e/kOeFIw2bcTa8BKo+5adJjkIjJY3n4Z6le6 IhK4lGAc18HZcr22dV7XR2FxKJvFlFP+8+MxpRLhX+Q4duAppB7QPy8J3AVqNTRwXpDH ue4kGKIsWS9bpethSs0ip7bQc9u5+XOAK6oiyuPKYQKA+WK1h60GZjHmG3UQm8GdfMSR XJy8dWe4ihEVMaufP1U8STA8NL6CU26cOkpivcVCOJV0XRjGxhejghidvf1ruCdB/x8d B76Q1bBOnMv53aAnwOfhtoJS+KwrOx1pMzevELpbqvaOhCQzPrD2uzWOhhLsZoXL0lwJ KDsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066932; x=1747671732; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xF2+54Is9Y/RM3HTLl2GmOBtMHoFbKc3iQ93j5V9CWg=; b=hg1DOz/ullCqfQ7Z3E0uvgEtIYXS3Rowyj/19Dd+n43qtx9V/OenarEtnFlN8UQk2O 44dLA9/pIK5Iw45vWpaiyXa9Ru0k/wH9TKqViyqE6WkpLOnu0KgXWN8kJP3MJX8/H2+V U+KaOXEyluH3493QwzfHTXW0Z3MtzcRYu7/e1LdcKfGsfyflkvYHKlKgtQNVoM/ZdadO 8aSPIEpJgmV79sJtTvHryMYMdN2M+yRIGa2bhmVV5vHCdkkWXwmA6XkWXOTxRJr+Oa0a 4A0+pnAas6OolEOKUq8nzKCjFVXZvZ6gXwwZtAAZbASWjzG0Cru1LFEQOxHA3XKjwBFA fn3w== X-Forwarded-Encrypted: i=1; AJvYcCVhbcSjoS13+YGhRU3QyM3Of3YkOQ4aOsCtghARkmJHlpHa06j8xVF779visPYZYBctfjFKfJLgW5hcM1w=@vger.kernel.org X-Gm-Message-State: AOJu0Yy7GG1Nrc9LjGvBUPxOgEeZ20lv5x2tc16NVBnQnW1+LijZV/ST P3NdmoalqmgvxgKfvkl4YXOXJDEEvbtbR0Ngb8GaawlkGLusidNZ0XkxkAED8luWVc4XWgNd7xS I X-Gm-Gg: ASbGncuQDNLxOSSHzD3WmZ9K3wExyevpGI/qjc6oBe+zTwtfz7Yfk+mQNRy592L/F1R xmei4pF+AmJeQ3l+SqLjGGUOZ3u+qrMiyczzk3OQ9YSJDl/2VlmEy22o0jUqXYHU/vcONqKvaEN 77knLz34WlS29Wu8Lsi4XKNXF21cNG3yCJcm5zpkH9idBBAc9U3ejhxfptCiCweycQhkT2LUoBw EDdNpDZlZLRXLlk9cO+cLVYbWtakbRRCqJ+D9htKZDG8PdAH1ko5LFbwuJdko9QSdyHan1NSNdX rVRXD2aMgcrBXk788PXxWscva5EwAe19eTnhZ/grgNpYxJlbnUzir3cyNNQbDjcgnsi3Wa7tjeH d8K9kOaB02xLiii/YmNpaN2+EHEHzddKngDyO X-Google-Smtp-Source: AGHT+IEUvJWC96dITPrF63xDkehCwcU0FPTtDoj7fUuKu2PGS+4UNxLAuwofc+V4RoH9DNoujCSNzA== X-Received: by 2002:a05:622a:1aa0:b0:476:838c:b0ce with SMTP id d75a77b69052e-4945273c308mr238859421cf.13.1747066920650; Mon, 12 May 2025 09:22:00 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.21.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:22:00 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 12/17] cxl: docs/linux/memory-hotplug Date: Mon, 12 May 2025 12:21:29 -0400 Message-ID: <20250512162134.3596150-13-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation on how the CXL driver surfaces memory through the DAX driver and memory-hotplug. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/linux/memory-hotplug.rst | 78 +++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 Documentation/driver-api/cxl/linux/memory-hotplug.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index f2127968ea78..35c5b0c6f95e 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -37,6 +37,7 @@ that have impacts on each other. The docs here break up = configurations steps. linux/early-boot linux/cxl-driver linux/dax-driver + linux/memory-hotplug linux/access-coordinates =20 =20 diff --git a/Documentation/driver-api/cxl/linux/memory-hotplug.rst b/Docume= ntation/driver-api/cxl/linux/memory-hotplug.rst new file mode 100644 index 000000000000..af368c2bc9cf --- /dev/null +++ b/Documentation/driver-api/cxl/linux/memory-hotplug.rst @@ -0,0 +1,78 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Memory Hotplug +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The final phase of surfacing CXL memory to the kernel page allocator is for +the `DAX` driver to surface a `Driver Managed` memory region via the +memory-hotplug component. + +There are four major configurations to consider: + +1) Default Online Behavior (on/off and zone) +2) Hotplug Memory Block size +3) Memory Map Resource location +4) Driver-Managed Memory Designation + +Default Online Behavior +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The default-online behavior of hotplug memory is dictated by the following, +in order of precedence: + +- :code:`CONFIG_MHP_DEFAULT_ONLINE_TYPE` Build Configuration +- :code:`memhp_default_state` Boot parameter +- :code:`/sys/devices/system/memory/auto_online_blocks` value + +These dictate whether hotplugged memory blocks arrive in one of three stat= es: + +1) Offline +2) Online in :code:`ZONE_NORMAL` +3) Online in :code:`ZONE_MOVABLE` + +:code:`ZONE_NORMAL` implies this capacity may be used for almost any alloc= ation, +while :code:`ZONE_MOVABLE` implies this capacity should only be used for +migratable allocations. + +:code:`ZONE_MOVABLE` attempts to retain the hotplug-ability of a memory bl= ock +so that it the entire region may be hot-unplugged at a later time. Any ca= pacity +onlined into :code:`ZONE_NORMAL` should be considered permanently attached= to +the page allocator. + +Hotplug Memory Block Size +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D +By default, on most architectures, the Hotplug Memory Block Size is either +128MB or 256MB. On x86, the block size increases up to 2GB as total memory +capacity exceeds 64GB. As of v6.15, Linux does not take into account the +size and alignment of the ACPI CEDT CFMWS regions (see Early Boot docs) wh= en +deciding the Hotplug Memory Block Size. + +Memory Map +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The location of :code:`struct folio` allocations to represent the hotplugg= ed +memory capacity are dictated by the following system settings: + +- :code:`/sys_module/memory_hotplug/parameters/memmap_on_memory` +- :code:`/sys/bus/dax/devices/daxN.Y/memmap_on_memory` + +If both of these parameters are set to true, :code:`struct folio` for this +capacity will be carved out of the memory block being onlined. This has +performance implications if the memory is particularly high-latency and +its :code:`struct folio` becomes hotly contended. + +If either parameter is set to false, :code:`struct folio` for this capacity +will be allocated from the local node of the processor running the hotplug +procedure. This capacity will be allocated from :code:`ZONE_NORMAL` on +that node, as it is a :code:`GFP_KERNEL` allocation. + +Systems with extremely large amounts of :code:`ZONE_MOVABLE` memory (e.g. +CXL memory pools) must ensure that there is sufficient local +:code:`ZONE_NORMAL` capacity to host the memory map for the hotplugged cap= acity. + +Driver Managed Memory +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The DAX driver surfaces this memory to memory-hotplug as "Driver Managed".= This +is not a configurable setting, but it's important to note that driver mana= ged +memory is explicitly excluded from use during kexec. This is required to = ensure +any reset or out-of-band operations that the CXL device may be subject to = during +a functional system-reboot (such as a reset-on-probe) will not cause porti= ons of +the kexec kernel to be overwritten. --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5240225D20D for ; Mon, 12 May 2025 16:22:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066925; cv=none; b=A2n4V5ixu5RgSgnbUMxdlQp2nR+VJ5ngxIcJeRaUj0IYJPoREZR/9b9e2n50eqV6FvE44vfq/RqKRdqx5/IRcUxRg+E/Ysbz9QlYOUPvGLHDiMipzR99tNi+kA9SIdJVnROUIWmC5y85AWrAEcee/WwTb4qMTa0D4SmNjCUYWCs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066925; c=relaxed/simple; bh=6kFRAXLyb71mD2Zs2gNFWfCB+AfEcQArhWsCvZFm48U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=M0MopTjIm+9t2DAZ3j5n5SVAmE3J4My3hhnJ4vE7V8uTtaJZjyGvUlbpdvVTRxi0S6BhyYb9j7sBC/WzZyIM8VYflQqjJ/My+ENFMTd2kw0Ld53Yf5NtZUAY08MqbKd+QQq8frmPVREf+Jasvm2JFDc4fWqUzeaTG5WWoR7chdA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=JnQElByY; arc=none smtp.client-ip=209.85.160.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="JnQElByY" Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-476ae781d21so55631201cf.3 for ; Mon, 12 May 2025 09:22:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066922; x=1747671722; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tpBvWJMvNrl2dBGkl5driDL3hYTI6jOnBaaDfES2w1M=; b=JnQElByYJxg6o/n5loj0kZT19DtAktksz+qN61RdF2oaWDqIHKV8aFcNcCvsxW0FHC pYSnQZsUnbxVcWxgAV8kGggKL6eUEishsR5dWy/Rs045fVb90jouUNaOlI/5FjLY5BZy ctnLUGndKKZbn2S0nN15JlLOq9CbGQft3XmZOyYIHQ5aqB8js673f8l304Ro+UwxGQ17 MpTvbphYM1tsFD0QcKsdGxS2c9jhdkEmuDJqAGnyXBFyWjT2xmr7/CZS71MQTZ3UEw9j syopRlDwDO57k9vskFG/k/vdjD1qy34YGj5YaWo3iW0ncj8MPa2sbDLIJTnbEHy1odQn NlBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066922; x=1747671722; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tpBvWJMvNrl2dBGkl5driDL3hYTI6jOnBaaDfES2w1M=; b=Zch9WCC2JZ0jZPJQPKPWmL8vrzdAH5RVKqtgvxra1itpTaMLBFpd2WeFuV9p/dlxvk XN6zCWo0yZDaWG9+34y8pW1wi56CIWZh2pchpWKTzrv9QbBU02wUFjqwYJJdjIz/X8oU 7rIfTlCqo0l4ttZMShFv/k12LWBINZ4nxchMaEs8bm0J5aTcY1CNXPXDgHdiVUH33x/V zM9yppf+jAGFp85Fu1mdNixKohZ7ue/iLWOdVohe+SZiIm5HKZt87Fl6+ZjpR5ZtIKAA uoECJSLAJWsPMzYRbNQHMlg8L+mkUAHKGwFZGbELtZ7kfsp70XG/7BL8woMsyzKODkJi 8EcA== X-Forwarded-Encrypted: i=1; AJvYcCXhakfn98FzICHGG4UyR/oGStS/Sxq7EfDqmJ62BYy95RGNp1T15Ky6qo2jyA5YI0QV1FhXFPjXn/rx3Jg=@vger.kernel.org X-Gm-Message-State: AOJu0YzwwpbkWwTrAFMrc7XA2nN6eRJWOxb40N4SBidJZFhVnNsMnGhg RoxbSBn9S9q7dHiz4V7s1QEp+CzWKJDPPK7Qblxh+yG6nyUatS2YzDAipYgiFiE= X-Gm-Gg: ASbGncuTdEHlXTPGsSWL1KmvolNdxE1sNXLeKcID4CatlU6eJ0AFRh1FpXhJ/XEPBaT SBkHXzFJcBUjaKVqfTCBXfT/OyjV39lOI2CaN/jbTbWfs3zQXkJQp6DrDXahelQW4uxFlIkSMHd /pL/nZUTspVvdiGgPpjpjUJvp4BZ9WLod8XGa+Kl/GGO4yUz0ngHa0A8DM4yzZdqs9YvNy1S9Hv 6pWwvszcKsUXRAf6Tz1s9MQzfZOJy7Z0cBIS4yC4IDZwrgAUC3h7Zhvn+1vxJuHZzH4l1M9N8QW yfuOKjcqK+1KMKhzsaK1C3N5KIuHK1AuuIlJGuzqnrL0++jgHtTFHUwV922I/Od9xZ7HO0hl2HC hGJaOp3nYCnylexiMrOuk3+a5NbMV6xE74yk6 X-Google-Smtp-Source: AGHT+IEgOwqbTVyM2cCqwhYH6oeFz+yp7dgByEYQB2haUlrU8NGA2h0ID7S4kwTF7yVmhY0k+Ia7IQ== X-Received: by 2002:a05:622a:5408:b0:477:c04:b512 with SMTP id d75a77b69052e-4945273c1dbmr225206941cf.16.1747066922158; Mon, 12 May 2025 09:22:02 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.22.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:22:01 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 13/17] cxl: docs/allocation/dax Date: Mon, 12 May 2025 12:21:30 -0400 Message-ID: <20250512162134.3596150-14-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Small example of accessing CXL memory capacity via DAX device Signed-off-by: Gregory Price --- .../driver-api/cxl/allocation/dax.rst | 60 +++++++++++++++++++ Documentation/driver-api/cxl/index.rst | 5 ++ 2 files changed, 65 insertions(+) create mode 100644 Documentation/driver-api/cxl/allocation/dax.rst diff --git a/Documentation/driver-api/cxl/allocation/dax.rst b/Documentatio= n/driver-api/cxl/allocation/dax.rst new file mode 100644 index 000000000000..c6f7a5da832f --- /dev/null +++ b/Documentation/driver-api/cxl/allocation/dax.rst @@ -0,0 +1,60 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +DAX Devices +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +CXL capacity exposed as a DAX device can be accessed directly via mmap. +Users may wish to use this interface mechanism to write their own userland +CXL allocator, or to managed shared or persistent memory regions across mu= ltiple +hosts. + +If the capacity is shared across hosts or persistent, appropriate flushing +mechanisms must be employed unless the region supports Snoop Back-Invalida= te. + +Note that mappings must be aligned (size and base) to the dax device's base +alignment, which is typically 2MB - but maybe be configured larger. + +:: + + #include + #include + #include + #include + #include + #include + + #define DEVICE_PATH "/dev/dax0.0" // Replace DAX device path + #define DEVICE_SIZE (4ULL * 1024 * 1024 * 1024) // 4GB + + int main() { + int fd; + void* mapped_addr; + + /* Open the DAX device */ + fd =3D open(DEVICE_PATH, O_RDWR); + if (fd < 0) { + perror("open"); + return -1; + } + + /* Map the device into memory */ + mapped_addr =3D mmap(NULL, DEVICE_SIZE, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + if (mapped_addr =3D=3D MAP_FAILED) { + perror("mmap"); + close(fd); + return -1; + } + + printf("Mapped address: %p\n", mapped_addr); + + /* You can now access the device through the mapped address */ + uint64_t* ptr =3D (uint64_t*)mapped_addr; + *ptr =3D 0x1234567890abcdef; // Write a value to the device + printf("Value at address %p: 0x%016llx\n", ptr, *ptr); + + /* Clean up */ + munmap(mapped_addr, DEVICE_SIZE); + close(fd); + return 0; + } diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 35c5b0c6f95e..6e7497f4811a 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -40,5 +40,10 @@ that have impacts on each other. The docs here break up= configurations steps. linux/memory-hotplug linux/access-coordinates =20 +.. toctree:: + :maxdepth: 2 + :caption: Memory Allocation + + allocation/dax =20 .. only:: subproject and html --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CAA7257433 for ; Mon, 12 May 2025 16:22:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066928; cv=none; b=EYn8N6vS63OyopWk/YozWATLlFQU81iPfwtHqcu1pwr1Rp9S4AhZgNE4sQPBIhYmrggZ2y0nkWF8Q323Oye2XJ3NHaPoAl7EIAxZUjNrMQJfiyrfWMi5uWNLclpcbj6hqzp7byEipZBgLGuHATlY4fbQXUxQP/dxzgpdCOHo840= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066928; c=relaxed/simple; bh=alRL+BnN7pA7RLLcp2eJAsPEVuznl19NxFJaEka+9wo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IAF+ZVDuY2ZsRBL1Jjfi724GGa4pqXapK1G3JQd3f7LPc3DxcBPQaQJgInxBaX3kuYpMITG48J/ZdwR0d8TGic6RAG17b3jK3kqFGAroXoEX1Ml4dz5M2BGyrANLt+1oEuWBtL+R+CXhxBjnnW+MXkf9GFEondx6bGvCxCahnbo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=ByeXreV3; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="ByeXreV3" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-4774d68c670so78564271cf.0 for ; Mon, 12 May 2025 09:22:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066924; x=1747671724; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8mmNdRpdNT2e5CioCKqjI7jQZuOdPNFPwu/aBvzKK6c=; b=ByeXreV3D7rxuAuI62OkxgSg/zHGh1SWk/Ir8lSJG+Y7G9h6NvZNeKGf3UAQSLg+E3 DjUxOPsbH+hhFqly6ZLnfiDXxdB6EjnbpakrYSTJrjStmR9RandwYPh6pZsfX7qg8Esi Ccg5cYlr6pmTu0J3w9B51h3Niv4QfbXhogsL52xM6sBwdNJ7e3e7MjQiLsTlH1wzleUc /dIV2n3HQfIrDr1AooXQrP8S1/JDaLrExWpN5b3fwOuAX8aefDNIUuaAdpbHKdoELc2f WTajCCuzAflMCFfv6YRMLF68EnGGiz006DzHS0D2IgOBj8qCh+9IJkkahOUrhFGipbti 9MHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066924; x=1747671724; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8mmNdRpdNT2e5CioCKqjI7jQZuOdPNFPwu/aBvzKK6c=; b=su3IsKXnJbMh2NPJ1zSz4XudM2JhDggPPXc5vRwPQ2lzTGrUM19NiHWJgJ5BPoBsgG W5uucIlfooVBn5dUGOyvNB25be3vN/4GK/dEuntGnxPowwZwE2YEZ/Q203p260ovhMNR D5GJLPxy5CX5MB8G7HPx1QJ0zLHkTCWi+MstKrZYISbcfSoxPQonPKXdkNPmBdWptpwR j/nbZyQS/Y2s0oktKBzHjoQhFPhg+Ln6xBvlyOoJqsXXiaBFX8WyXaWX+0++TCubj3UU Q+O01YvK7SSYBOGat9ZRV8xzm3BX/nszkfOlXSH4BHt0SaolnFSYW5MHYyG6lYIrgux8 hIvQ== X-Forwarded-Encrypted: i=1; AJvYcCVux9rR23LULD/2AzZ7hgsVNSjjfXEqZQnHnAkUdHwIjAjsfaWtWQmU/fbOkFvLHPlJaAqBKl9C1YifBFE=@vger.kernel.org X-Gm-Message-State: AOJu0YxDWOsp+cBMtU97q+UmDx8y240rQX5Vcbqb+/2ZwOLVlSUNeQZj tlDsim8/B3lrvLL+2ih/wHVKFsNv9JqCCsXUSK8Im/7z3NeD+PlS1mcVbt2ztp4= X-Gm-Gg: ASbGncss5sqmxaC0SHPHcNhA6a+kRi2NLjziFq7LHEjsOQQl3ZqTpZeLZFbN6s8Plck Beyv0wWuFdXxhIVgPxDj4dzhF6uThUyySd0cEEe+Ab4AW4tZ+a+BGIVIcS4iCcM0sn+q1IS099R CNAO5gnAtHVqhV/pcu13gipaJSg+DKXDxePvv2Gy3RsZJR2mumscDXOHFQvkJpHXFx4lenN1YlJ IBNJVB4kBBTJP+rp0KzQLa+433h2PQHtuiBkbuClLyzjDdDzWndxMhWT8++rnlL20gxI1AAWdfr 0QKn2Ywy3im5BTKVUVYFLmo7WRD/1aNhqs2AmhZepmF4b8OvH6jFJRhCyy4vKwAJTmpYVgZIwXg Q/uGUD9ZViPmf9C21jWCV/RZkW/kOGxVIuDz5 X-Google-Smtp-Source: AGHT+IGWTZBftJjnQ8DcvuqUwpD3a6TCMt8kbFUpNWIl3gGiOiMusJaygR3wQdoazycEU1zccxqVUA== X-Received: by 2002:ac8:57d3:0:b0:476:87f6:3ce4 with SMTP id d75a77b69052e-494527b8112mr197827301cf.39.1747066923532; Mon, 12 May 2025 09:22:03 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.22.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:22:03 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 14/17] cxl: docs/allocation/page-allocator Date: Mon, 12 May 2025 12:21:31 -0400 Message-ID: <20250512162134.3596150-15-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Document some interesting interactions that occur when exposing CXL memory capacity to page allocator. Signed-off-by: Gregory Price --- .../cxl/allocation/page-allocator.rst | 85 +++++++++++++++++++ Documentation/driver-api/cxl/index.rst | 1 + 2 files changed, 86 insertions(+) create mode 100644 Documentation/driver-api/cxl/allocation/page-allocator.= rst diff --git a/Documentation/driver-api/cxl/allocation/page-allocator.rst b/D= ocumentation/driver-api/cxl/allocation/page-allocator.rst new file mode 100644 index 000000000000..7b8fe1b8d5bb --- /dev/null +++ b/Documentation/driver-api/cxl/allocation/page-allocator.rst @@ -0,0 +1,85 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The Page Allocator +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The kernel page allocator services all general page allocation requests, s= uch +as :code:`kmalloc`. CXL configuration steps affect the behavior of the pa= ge +allocator based on the selected `Memory Zone` and `NUMA node` the capacity= is +placed in. + +This section mostly focuses on how these configurations affect the page +allocator (as of Linux v6.15) rather than the overall page allocator behav= ior. + +NUMA nodes and mempolicy +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Unless a task explicitly registers a mempolicy, the default memory policy +of the linux kernel is to allocate memory from the `local NUMA node` first, +and fall back to other nodes only if the local node is pressured. + +Generally, we expect to see local DRAM and CXL memory on separate NUMA nod= es, +with the CXL memory being non-local. Technically, however, it is possible +for a compute node to have no local DRAM, and for CXL memory to be the +`local` capacity for that compute node. + + +Memory Zones +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +CXL capacity may be onlined in :code:`ZONE_NORMAL` or :code:`ZONE_MOVABLE`. + +As of v6.15, the page allocator attempts to allocate from the highest +available and compatible ZONE for an allocation from the local node first. + +An example of a `zone incompatibility` is attempting to service an allocat= ion +marked :code:`GFP_KERNEL` from :code:`ZONE_MOVABLE`. Kernel allocations a= re +typically not migratable, and as a result can only be serviced from +:code:`ZONE_NORMAL` or lower. + +To simplify this, the page allocator will prefer :code:`ZONE_MOVABLE` over +:code:`ZONE_NORMAL` by default, but if :code:`ZONE_MOVABLE` is depleted, it +will fallback to allocate from :code:`ZONE_NORMAL`. + + +Zone and Node Quirks +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Let's consider a configuration where the local DRAM capacity is largely on= lined +into :code:`ZONE_NORMAL`, with no :code:`ZONE_MOVABLE` capacity present. T= he +CXL capacity has the opposite configuration - all onlined in +:code:`ZONE_MOVABLE`. + +Under the default allocation policy, the page allocator will completely sk= ip +:code:`ZONE_MOVABLE` as a valid allocation target. This is because, as of +Linux v6.15, the page allocator does (approximately) the following: :: + + for (each zone in local_node): + + for (each node in fallback_order): + + attempt_allocation(gfp_flags); + +Because the local node does not have :code:`ZONE_MOVABLE`, the CXL node is +functionally unreachable for direct allocation. As a result, the only way +for CXL capacity to be used is via `demotion` in the reclaim path. + +This configuration also means that if the DRAM ndoe has :code:`ZONE_MOVABL= E` +capacity - when that capacity is depleted, the page allocator will actually +prefer CXL :code:`ZONE_MOVABLE` pages over DRAM :code:`ZONE_NORMAL` pages. + +We may wish to invert this priority in future Linux versions. + +If `demotion` and `swap` are disabled, Linux will begin to cause OOM crash= es +when the DRAM nodes are depleted. See the reclaim section for more details. + + +CGroups and CPUSets +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Finally, assuming CXL memory is reachable via the page allocation (i.e. on= lined +in :code:`ZONE_NORMAL`), the :code:`cpusets.mems_allowed` may be used by +containers to limit the accessibility of certain NUMA nodes for tasks in t= hat +container. Users may wish to utilize this in multi-tenant systems where s= ome +tasks prefer not to use slower memory. + +In the reclaim section we'll discuss some limitations of this interface to +prevent demotions of shared data to CXL memory (if demotions are enabled). + diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 6e7497f4811a..7acab7e7df96 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -45,5 +45,6 @@ that have impacts on each other. The docs here break up = configurations steps. :caption: Memory Allocation =20 allocation/dax + allocation/page-allocator =20 .. only:: subproject and html --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67A872571DD for ; Mon, 12 May 2025 16:22:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066928; cv=none; b=sQ21fK9PYJ2nkL8HP/3igKh7rW3+O/30H4X/MUvFjB2TWll//pyWftvvH488EDWji5qAKlJcGXuOnDFJHdof59SMFeTCrEcp5VClhyMcwoM+psuHwR9r/UXP7KQLholZl8SQmHGU9cUNn95J7u7+e2IEVnumb8Y/GEPkcj+oAcg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066928; c=relaxed/simple; bh=C+4bukmEyuUogjFKJzPVkC+0PLpktTTguS9gLaYvHGU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S33rnCU8zCFMUgy9TXSJULCobYgc9NooZKvXqQAdcG94YRzRgC2iUIxLemgBF3rS9iUUT3YFo279f3y50/Pr18jLs6gV691MDW+Kqby0BLpZ+yqD7WCrmKwXyn3ZGD+aIeF1/x/Ei6kKjSOkIP9uIqSkTGOLAfws/EgG8ZWaEvY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=j1uN6XYM; arc=none smtp.client-ip=209.85.222.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="j1uN6XYM" Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-7c5c815f8efso502011485a.2 for ; Mon, 12 May 2025 09:22:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066925; x=1747671725; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RTTxOKtOTvxz/X6EnOnjuA5x3BeIonRn1m9gLvf8T78=; b=j1uN6XYMkXQr1Ef4pnhT81/B2aFvhR9IYRwrf7OnT/XdMNgSgJGFRJkgqB6Ny4p3v0 qEVcEuzodEC/5ZhaSueNRmjlDlLK35QiycRkZ25q9AkOFaLaG5IdeyYRNe8AWE9OmxHt Yiqgj6iJ9cbYk5NvlOC8yDj/X98NXJt/REQKaxmtLT/c0BDSp4yhpwoBAqeiKwqZh6Hi 4F4IZVKyP50dNN59WnNpKu/PWcnng0rujfZCzBG2kfF21S5B3wpg1QaFiqDB20cKDmFF lcF55ErNc2mUL+07j0T4ptplrOTQK1pek68V5kpxRtkSqor+eG9hyBEOiWWm1tuOo50u Wvgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066925; x=1747671725; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RTTxOKtOTvxz/X6EnOnjuA5x3BeIonRn1m9gLvf8T78=; b=qd9CehgafY9XWS6SUwQNSHHjwWwmcuco1C8yk7CQsj8iaq8Sw6GHFx8TEKGNG7DsbE UJQtbAxEbIZpyBzuKMmyOEqTQZwK3K8zEaAp5kRePlLKqAnfK7eFIJzOE9SqPB8tDJz/ Vs0MRIUWymePZFbNed61OABlMmKcg9qbRkBRYpKiAmWcLUFfp1Ux1v7PS3nzQcIKiW0L AI30QPbmfYWEz7So0TT1gFTasd6/8tODJ6Jk3bAkNnm2UskRfWNX1v6dpbNhBXTo8uLs QqUUIOWhk9axSWShPTZjXCG/5M7ZNSv0lGHCeef1xqnIQACL6hO5DoKkRsq3mpF9uc3x kUNw== X-Forwarded-Encrypted: i=1; AJvYcCW9cJ2Xm9dng8rQH0VTD9cwu9fCubg4G8Aulg1DHQ3yXyxmR4q7V/0WWctu5b0p0iSiD7+OpxeviFvfwM4=@vger.kernel.org X-Gm-Message-State: AOJu0YydiBrWIVeVml4UN5yWicldyHv4EzXuiG6Q5wT7q8HxGbVhgVRR s+3uoCaAT9t7ybpYCjVaO4LuxqagmFdrlNSZN5SYzgAJOXXtYOteThJqNETLIqU= X-Gm-Gg: ASbGncvhRBik8wdqNlNkoPW2LlE4/Jx5GoKpFhzrX3bp7uLzskv9S9/wLZEyME8953/ njsOD+2Z4edhE1Pas4xJZajD9s1rnEKF9DWe97yigg0yazahhofbErZxMgSQye7nH/odEeldtIb L1FxkEJ9aTC15t5Yblybbv2EtC/OKks8IIssOtJrqTvA0dyhawBINeF/rVXGwTNFa0ua2hMJNNt IlHKVnds6pQixo5fGjLCaFan/IHbBUDT+pKPlGR9MuUIGsldjDVgjKurS/IE27FPKi839LCCqBf CuCp/nTwHoVKrZWu3xTST4OWJwhuO+bha7TYmM9eI2WnF1DlK4lnOz+HlmE+gFLyfVETzEkG+lg Upu/fK1vFsmjw8dVukrKsJ7D5F1KZjb2kUbze X-Google-Smtp-Source: AGHT+IHV0UN/KhozVIrARpAytA6jarV108Aoyqsz27PknFZSu3HGQ8itoHJAY+cqjXAR3dS9EUlrqQ== X-Received: by 2002:a05:620a:4403:b0:7c5:9a6c:b7d3 with SMTP id af79cd13be357-7cd01157590mr2261077885a.37.1747066924959; Mon, 12 May 2025 09:22:04 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.22.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:22:04 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 15/17] cxl: docs/allocation/reclaim Date: Mon, 12 May 2025 12:21:32 -0400 Message-ID: <20250512162134.3596150-16-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Document a bit about how reclaim interacts with various CXL configurations. Signed-off-by: Gregory Price --- .../driver-api/cxl/allocation/reclaim.rst | 51 +++++++++++++++++++ Documentation/driver-api/cxl/index.rst | 1 + 2 files changed, 52 insertions(+) create mode 100644 Documentation/driver-api/cxl/allocation/reclaim.rst diff --git a/Documentation/driver-api/cxl/allocation/reclaim.rst b/Document= ation/driver-api/cxl/allocation/reclaim.rst new file mode 100644 index 000000000000..f40f1cae391a --- /dev/null +++ b/Documentation/driver-api/cxl/allocation/reclaim.rst @@ -0,0 +1,51 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D +Reclaim +=3D=3D=3D=3D=3D=3D=3D +Another way CXL memory can be utilized *indirectly* is via the reclaim sys= tem +in :code:`mm/vmscan.c`. Reclaim is engaged when memory capacity on the sy= stem +becomes pressured based on global and cgroup-local `watermark` settings. + +In this section we won't discuss the `watermark` configurations, just how = CXL +memory can be consumed by various pieces of reclaim system. + +Demotion +=3D=3D=3D=3D=3D=3D=3D=3D +By default, the reclaim system will prefer swap (or zswap) when reclaiming +memory. Enabling :code:`kernel/mm/numa/demotion_enabled` will cause vmscan +to opportunistically prefer distant NUMA nodes to swap or zswap, if capaci= ty +is available. + +Demotion engages the :code:`mm/memory_tier.c` component to determine the +next demotion node. The next demotion node is based on the :code:`HMAT` +or :code:`CDAT` performance data. + +cpusets.mems_allowed quirk +-------------------------- +In Linux v6.15 and below, demotion does not respect :code:`cpusets.mems_al= lowed` +when migrating pages. As a result, if demotion is enabled, vmscan cannot +guarantee isolation of a container's memory from nodes not set in mems_all= owed. + +In Linux v6.XX and up, demotion does attempt to respect +:code:`cpusets.mems_allowed`; however, certain classes of shared memory +originally instantiated by another cgroup (such as common libraries - e.g. +libc) may still be demoted. As a result, the mems_allowed interface still +cannot provide perfect isolation from the remote nodes. + +ZSwap and Node Preference +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D +In Linux v6.15 and below, ZSwap allocates memory from the local node of the +processor for the new pages being compressed. Since pages being compressed +are typically cold, the result is a cold page becomes promoted - only to +be later demoted as it ages off the LRU. + +In Linux v6.XX, ZSwap tries to prefer the node of the page being compressed +as the allocation target for the compression page. This helps prevent +thrashing. + +Demotion with ZSwap +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +When enabling both Demotion and ZSwap, you create a situation where ZSwap +will prefer the slowest form of CXL memory by default until that tier of +memory is exhausted. diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 7acab7e7df96..d3ab928d4d7c 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -46,5 +46,6 @@ that have impacts on each other. The docs here break up = configurations steps. =20 allocation/dax allocation/page-allocator + allocation/reclaim =20 .. only:: subproject and html --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 908C824EABF for ; Mon, 12 May 2025 16:22:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066929; cv=none; b=NZljEe/6D/5/wydijevb7P3VQVfSHCdyjU+cmEzHmjBX4TNCdBvrhZElQaJSkACMRdENQoKlnmka4mSrBlIRb8ISEXNhMQNr7L+bozK1dw9OBdOWXt4pQwOMbzSpfi5Tnfqh1XABzfIjgklXmDJZYb0l32JPaI1/xqlPevWme54= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066929; c=relaxed/simple; bh=02rFqC3gSUB4Fy4+GT/QyZr7jHPjwxLz9IoPedteAik=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ShsQQoUjoX2AcDly+zsVg/yEUboinLP63jweymoFyYqIyf8hT9j0JjB3mQqhsjiiM40a1P6BbUWPOkIF/48R0pWoKuHzxx1nvwV8Vf+zl8XQRRSfbIq50Sc1f5klAxTtCuOr4MmdoDRi90Ry2kwWyrAaEhw5GZ06sJri9quxdXg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=SbIsS6Yw; arc=none smtp.client-ip=209.85.160.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="SbIsS6Yw" Received: by mail-qt1-f174.google.com with SMTP id d75a77b69052e-47ae894e9b7so87956181cf.3 for ; Mon, 12 May 2025 09:22:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066926; x=1747671726; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gsjZ8+vqh9tLCwEMgliXk7JRqYeLc6j1GH7rHKcC4Fk=; b=SbIsS6YwW57XJuyeYgmxF9B4fxxJ2WETuQcFS/bsiU0dBMwUDvc7stG5P1Y9szPM6Y Ocu1QSlVumnQVcUHIAP3Jc4wJHopKP5uRZexOBBPxxEnoYLeyz90d2GEeiCHJPZtOb65 CRbwYauS8+xL9Gorb/zkDEg2fjRnkFTnR+Spc6tMjDEIyA7Hxibj6ov+7pYtplXM2Xs4 a+3nDfA0G6W5+YI5R5pP9Hcn2xZVO3jWp/Anfvz+ViJUW8ajiPqPRblh7Y04XvDw1shA gzjgSSAPvLWnhJkDjaNkr28M+rr7jp5hyceGpYOUsPuG9+WKbgBRIV7ypJo8KW+kkqrB rZuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066926; x=1747671726; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gsjZ8+vqh9tLCwEMgliXk7JRqYeLc6j1GH7rHKcC4Fk=; b=KirBH3wFbSMGEv9zDpe3jXKP27KMXVk/tAurPrXpvNwWSFtPjI1aW3hdi+5BD5XEKd 9DH8Hxw4InJdUdORblAGjRvkaBhNfQGrdunr1Ta9DfBhA8q6C4Kk85tqObMetiADG9cX r3Aj/P+cv8M6G+1CLOrzXSMIM/SAlvlQg+lz1OfqaEuSTT10pqrJOGoa7es+aWDV28ie UZpiwjlJuhBJ+2GRd0gecBATaxocY/6IGfyskW+sXAvAnVSaaCHGlfkItZK8w/9SJzA+ q9Lz1GS+FJMNTdPjiMrpwfNTYIOdlQ/0gRmkvXzw6RNU1jEVlKrGXiIqxVPz0EsMsDTS NNTg== X-Forwarded-Encrypted: i=1; AJvYcCX8Ce0XpvYJCipSknAB37WVlp+eKDLzCsL1maCx3NJj8QX8XWANk7ofADh/I7YobUzjctRe+6Y4atj/WjU=@vger.kernel.org X-Gm-Message-State: AOJu0YzUlK9HnJ1xlDeywPjN+S6cUTtdqfh9kBj0J9j7Duw1n6KWt0a7 BXpMUaiknXvN3SYnxJl6BWf64wV/4weHRTKwRlmu8GbUsBRdJG+5NlUIZS0555E= X-Gm-Gg: ASbGncsH052LKoIfEriHHFjSOHtqi5g3qGmdqSgyeGFF4NHAsuc5gDHS6QN/iQuFsGl q/LhVwT7g3/qZLsfskY7BncOUZg0chmL/Y7f3oCK8RRP42+ViGIfiNTK/7PJPLnUfi3fEYqGKO5 I8drdJx/pGRaLerF13O1VL+yzRA8U6Wx1j1Cmfet/9yk9xB9ZvHrLwR521ngeIHw7NxeIWj5511 QY+hUok8uiGZR5ZYNOfP0IqqiNdJ5h9vs6OGxl459Fsgu0nrvzpOSXNF3epSBPfxY/Q04L/+IiF FbAW0UXamy/HI/Du+mq5RCP+tELe52gPsYrnSHfoUn3GCTLVIZwhK+WVkRNOo/i4sherBk6OTNf PX1ZSnN9JjZUoUxKJXubTi9uOODSRfLyku1AA X-Google-Smtp-Source: AGHT+IESCJ4wxgoeogGabJwCYGcV8hNihMeEX8lTGFo0c9q1TtRsQ3SAwLcPjqupzQu+Z/KuxiouEA== X-Received: by 2002:a05:622a:110c:b0:476:9377:17aa with SMTP id d75a77b69052e-494527fd87amr209927431cf.52.1747066926366; Mon, 12 May 2025 09:22:06 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.22.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:22:06 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH v3 16/17] cxl: docs/allocation/hugepages Date: Mon, 12 May 2025 12:21:33 -0400 Message-ID: <20250512162134.3596150-17-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add docs on how CXL capacity interacts with CMA and HugeTLB allocation interfaces. Signed-off-by: Gregory Price --- .../driver-api/cxl/allocation/hugepages.rst | 32 +++++++++++++++++++ Documentation/driver-api/cxl/index.rst | 1 + 2 files changed, 33 insertions(+) create mode 100644 Documentation/driver-api/cxl/allocation/hugepages.rst diff --git a/Documentation/driver-api/cxl/allocation/hugepages.rst b/Docume= ntation/driver-api/cxl/allocation/hugepages.rst new file mode 100644 index 000000000000..1023c6922829 --- /dev/null +++ b/Documentation/driver-api/cxl/allocation/hugepages.rst @@ -0,0 +1,32 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Huge Pages +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Contiguous Memory Allocator +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D +CXL Memory onlined as SystemRAM during early boot is eligible for use by C= MA, +as the NUMA node hosting that capacity will be `Online` at the time CMA +carves out contiguous capacity. + +CXL Memory deferred to the CXL Driver for configuration cannot have its +capacity allocated by CMA - as the NUMA node hosting the capacity is `Offl= ine` +at :code:`__init` time - when CMA carves out contiguous capacity. + +HugeTLB +=3D=3D=3D=3D=3D=3D=3D +Different huge page sizes allow different memory configurations. + +2MB Huge Pages +-------------- +All CXL capacity regardless of configuration time or memory zone is eligib= le +for use as 2MB huge pages. + +1GB Huge Pages +-------------- +CXL capacity onlined in :code:`ZONE_NORMAL` is eligible for 1GB Gigantic P= age +allocation. + +CXL capacity onlined in :code:`ZONE_MOVABLE` is not eligible for 1GB Gigan= tic +Page allocation. diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index d3ab928d4d7c..366faf851fc7 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -47,5 +47,6 @@ that have impacts on each other. The docs here break up = configurations steps. allocation/dax allocation/page-allocator allocation/reclaim + allocation/hugepages.rst =20 .. only:: subproject and html --=20 2.49.0 From nobody Sun Feb 8 15:54:08 2026 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C01E02951C9 for ; Mon, 12 May 2025 16:22:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066933; cv=none; b=Fex2j79YunO9qiLBquooih2WWocuodHXlBvakZceVPQSFdr22/jDFRWVGbMrzzX/P/pgZezzEsL7d127/SBntH4SKUztlhCLlL8BZrRnxKxFPcvZjZBaQPKSzSyTiapTJvC8lKMHh/xDxmC8BNyn0D25sX9A/b6G6nAAihsja3Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1747066933; c=relaxed/simple; bh=A6Mies9hUhfrCMV4P20s2B5vlbK+fHovsvjiHQdVYCE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cJ/mY6/4c7cB9Nu+RgO+66y9j4KwhdgCHr2cAu/3+RLT8kbCk7xzV0rCbBnCeEjP4CuEkSPbL2Qj724x7av2hlcqZQmaq17XOwJnNJdF6shUMqGBReN9k8nNkMA5v/I6NIjBfh+lYWC2/EC4nNB5ny4lrso/7Eo3jPVuJakbq5g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=WHZUZlq+; arc=none smtp.client-ip=209.85.160.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="WHZUZlq+" Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-476f4e9cf92so38856911cf.3 for ; Mon, 12 May 2025 09:22:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1747066928; x=1747671728; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ci6f1K89LsZ+DF6ZgFYZsgSAOzS56KH3yJTL4ef3rh0=; b=WHZUZlq+1rTSFZZPWvR1HjwnFQqgSG3OpqUrEt2xKc150AA+9Qj33nRvvGMLLg44YZ TM5csfL4HO9k/iW3wBUT5Ka1omzgO5kVKmNMRqYFDxpRWAnkiNYXigersQl8JmrktOAZ gdGmyMU5tmijl/Ov+l6i0mIydMzZt3r1u0LQPer9wV/QTGT1N3YXHnVmU8c9hbojzHAN P9cwMQ86r0a9728uYlFP4LHAARZRACZCQZxd4fDzVAFfYJH9tUy6y5WATEIM5i/l2nVS Zcy2JDgiHMGnWusOVWtQt/h/j1XH22cfgwbcRDboGZh0IURpUimWtN2QgFUPZerIukcp a5JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747066928; x=1747671728; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ci6f1K89LsZ+DF6ZgFYZsgSAOzS56KH3yJTL4ef3rh0=; b=LFxQHlkNAZ2GBacNvfFiHo0nD48+ryALBoUWABx71fdrDpy2NlVBnfFmViIHrzvkR1 q5T4gvME8lIK7D7ifWxBdNoglj0I2hFSd1i/fAxNehaJcFzpihNlvTAvh0v/UDdXURiC DegKKU7MISOqVpFeXv5R2ILLXZFHsWTfpu7YDFQiUl74J3y4yrjTavAVCGhlJ/JdW18p vrMX9TVhxrtlaZX03qsoC3goHWRb3HC362VkpG8GutwMRlI3jzhH1ilxaiIx4VD12dN3 cvIN6TN5Bi1dBLedMYSP7YltTDHghOqb/4naWDgJJolqtpTOpsRNBmv9Lim7RuyTxZdG 8cVQ== X-Forwarded-Encrypted: i=1; AJvYcCV7fgonqg0yw0VLwzaMK/fGiLX1p/ICxzvuJjNptc8B2rs9qeVIUzOzw/6atJWLxQdP5BC2zoYUvzWRtE0=@vger.kernel.org X-Gm-Message-State: AOJu0YxLyy9FZNdFPReRqmk6Np9UfyXISiehmFG5/Y2O+eh3nzJk1jdK m31vQoQ4euBvDDt2Ts5AxQqkKQ81zInOnGbPsqnij8t0jAHckuxcRh1/gNotza1gK4VOGsGHTg9 G X-Gm-Gg: ASbGncs7rWtRAsje0khhpHhAuHaPIO390Eqt8Yehav40oTaP/RLikq+GOG3uRtQ79CF 3uqWTjyN2NFWUmIBtxJxvtqLvffupqvpKbk4pyDSGU90UK0icFGynaWOeKKenZyBG2Zw8dFrsn+ 0xk2Ku6bUmktIvPakA5fLI8N0FQGqPf2N/f0PrNHVMwOpKdX2sVp4Lbfaoe9s4QNV8ZUUdI9+vM cojlKuJCRfVpFuSCpeWCn8s5P4YujPVN0sNkzKivwAj7eyyRZuw2z9ZrAfDFS9BBTxkgt8Ij3Kr UHbB3lgIFpDDutwBqZ6pcWTNjc1YkduboSZ2/c1lzDB6UhDe0qK9Jj9cdcIEhGpVd1LHe7g/6Wt ENqXAtCX75Q+kP0VzV7pM+tGJt0Jwa9gT39Lj6VbjCVd6LoM= X-Google-Smtp-Source: AGHT+IE6kifyyq68JXmdCyyHqCef0H7CrWN95iUQ8mmLv4UnZnRRmN3QUvUDR9Mvy0G1oQqLngMKog== X-Received: by 2002:a05:622a:1984:b0:476:ac73:c3f3 with SMTP id d75a77b69052e-494527162demr247105181cf.1.1747066928404; Mon, 12 May 2025 09:22:08 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-96-255-20-42.washdc.ftas.verizon.net. [96.255.20.42]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-49452583961sm52461791cf.58.2025.05.12.09.22.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 May 2025 09:22:08 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net, Bagas Sanjaya Subject: [PATCH v3 17/17] cxl: docs - add self-referencing cross-links Date: Mon, 12 May 2025 12:21:34 -0400 Message-ID: <20250512162134.3596150-18-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250512162134.3596150-1-gourry@gourry.net> References: <20250512162134.3596150-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add some crosslinks between pages in the CXL docs - mostly to the ACPI tables. Suggested-by: Bagas Sanjaya Signed-off-by: Gregory Price --- .../driver-api/cxl/devices/device-types.rst | 2 +- .../cxl/linux/access-coordinates.rst | 7 ++-- .../driver-api/cxl/linux/cxl-driver.rst | 35 ++++++++++--------- .../driver-api/cxl/linux/early-boot.rst | 32 ++++++++++------- .../driver-api/cxl/platform/bios-and-efi.rst | 12 +++---- .../example-configurations/flexible.rst | 10 +++--- .../example-configurations/hb-interleave.rst | 10 +++--- .../multi-dev-per-hb.rst | 10 +++--- .../example-configurations/one-dev-per-hb.rst | 10 +++--- 9 files changed, 69 insertions(+), 59 deletions(-) diff --git a/Documentation/driver-api/cxl/devices/device-types.rst b/Docume= ntation/driver-api/cxl/devices/device-types.rst index c70564cf0be3..f5e4330c1cfe 100644 --- a/Documentation/driver-api/cxl/devices/device-types.rst +++ b/Documentation/driver-api/cxl/devices/device-types.rst @@ -115,7 +115,7 @@ A Multi-Headed Single-Logical Device (MHSLD) exposes a = single logical device to multiple heads which may be connected to one or more discrete hosts. An example of this would be a simple memory-pool which may be statically configured (prior to boot) to expose portions of its memory -to Linux via the CEDT ACPI table. +to Linux via :doc:`CEDT <../platform/acpi/cedt>`. =20 MHMLD ~~~~~ diff --git a/Documentation/driver-api/cxl/linux/access-coordinates.rst b/Do= cumentation/driver-api/cxl/linux/access-coordinates.rst index e408ecbc4038..71024fa0f561 100644 --- a/Documentation/driver-api/cxl/linux/access-coordinates.rst +++ b/Documentation/driver-api/cxl/linux/access-coordinates.rst @@ -24,7 +24,7 @@ asymmetry in properties does not happen and all paths to = EPs are equal. =20 There can be multiple switches under an RP. There can be multiple RPs under a CXL Host Bridge (HB). There can be multiple HBs under a CXL Fixed Memory -Window Structure (CFMWS). +Window Structure (CFMWS) in the :doc:`CEDT <../platform/acpi/cedt>`. =20 An example hierarchy:: =20 @@ -83,8 +83,9 @@ also the index for the resulting xarray. =20 The next step is to take the min() of the per host bridge bandwidth and the bandwidth from the Generic Port (GP). The bandwidths for the GP are retrie= ved -via ACPI tables SRAT/HMAT. The minimum bandwidth are aggregated under the = same -ACPI0017 device to form a new xarray. +via ACPI tables (:doc:`SRAT <../platform/acpi/srat>` and +:doc:`HMAT <../platform/acpi/hmat>`). The minimum bandwidth are aggregated +under the same ACPI0017 device to form a new xarray. =20 Finally, the cxl_region_update_bandwidth() is called and the aggregated bandwidth from all the members of the last xarray is updated for the diff --git a/Documentation/driver-api/cxl/linux/cxl-driver.rst b/Documentat= ion/driver-api/cxl/linux/cxl-driver.rst index cf6b397abdb1..9759e90c3cf1 100644 --- a/Documentation/driver-api/cxl/linux/cxl-driver.rst +++ b/Documentation/driver-api/cxl/linux/cxl-driver.rst @@ -77,11 +77,11 @@ Root Object` Device Class is found. =20 The Root contains links to: =20 -* `Host Bridge Ports` defined by ACPI CEDT CHBS. +* `Host Bridge Ports` defined by CHBS in the :doc:`CEDT<../platform/acpi/c= edt>` =20 * `Downstream Ports` typically connected to `Host Bridge Ports`. =20 -* `Root Decoders` defined by ACPI CEDT CFMWS. +* `Root Decoders` defined by CFMWS the :doc:`CEDT<../platform/acpi/cedt>` =20 :: =20 @@ -150,9 +150,8 @@ An `endpoint` is a terminal port in the fabric. This i= s a `logical device`, and may be one of many `logical devices` presented by a memory device. It is still considered a type of `port` in the fabric. =20 -An `endpoint` contains `endpoint decoders` available for use and the -*Coherent Device Attribute Table* (CDAT) used to describe the capabilities -of the device. :: +An `endpoint` contains `endpoint decoders` and the device's Coherent Device +Attribute Table (which describes the device's capabilities). :: =20 # ls /sys/bus/cxl/devices/endpoint5 CDAT decoders_committed modalias uevent @@ -247,17 +246,18 @@ parameter. Root Decoder ~~~~~~~~~~~~ A `Root Decoder` is logical construct of the physical address and interlea= ve -configurations present in the ACPI CEDT CFMWS. Linux presents this inform= ation -as a decoder present in the `CXL Root`. We consider this a `Root Decoder`, -though technically it exists on the boundary of the CXL specification and -platform-specific CXL root implementations. +configurations present in the CFMWS field of the :doc:`CEDT +<../platform/acpi/cedt>`. +Linux presents this information as a decoder present in the `CXL Root`. We +consider this a `Root Decoder`, though technically it exists on the bounda= ry +of the CXL specification and platform-specific CXL root implementations. =20 Linux considers these logical decoders a type of `Routing Decoder`, and is= the first decoder in the CXL fabric to receive a memory access from the platfo= rm's memory controllers. =20 `Root Decoders` are created during :code:`cxl_acpi_probe`. One root decod= er -is created per CFMWS entry in the ACPI CEDT. +is created per CFMWS entry in the :doc:`CEDT <../platform/acpi/cedt>`. =20 The :code:`target_list` parameter is filled by the CFMWS target fields. Ta= rgets of a root decoder are `Host Bridges`, which means interleave done at the r= oot @@ -267,9 +267,11 @@ Only root decoders are capable of `Inter-Host-Bridge I= nterleave`. =20 Such interleaves must be configured by the platform and described in the A= CPI CEDT CFMWS, as the target CXL host bridge UIDs in the CFMWS must match the= CXL -host bridge UIDs in the ACPI CEDT CHBS and ACPI DSDT. +host bridge UIDs in the CHBS field of the :doc:`CEDT +<../platform/acpi/cedt>` and the UID field of CXL Host Bridges defined in +the :doc:`DSDT <../platform/acpi/dsdt>`. =20 -Interleave settings in a rootdecoder describe how to interleave accesses a= mong +Interleave settings in a root decoder describe how to interleave accesses = among the *immediate downstream targets*, not the entire interleave set. =20 The memory range described in the root decoder is used to @@ -531,10 +533,11 @@ granularity configuration. =20 At Root ~~~~~~~ -Root decoder interleave is defined by the ACPI CEDT CFMWS. The CEDT -may actually define multiple CFMWS configurations to describe the same -physical capacity - with the intent to allow users to decide at runtime -whether to online memory as interleaved or non-interleaved. :: +Root decoder interleave is defined by CFMWS field of the :doc:`CEDT +<../platform/acpi/cedt>`. The CEDT may actually define multiple CFMWS +configurations to describe the same physical capacity, with the intent to = allow +users to decide at runtime whether to online memory as interleaved or +non-interleaved. :: =20 Subtable Type : 01 [CXL Fixed Memory Window Structure] Window base address : 0000000100000000 diff --git a/Documentation/driver-api/cxl/linux/early-boot.rst b/Documentat= ion/driver-api/cxl/linux/early-boot.rst index 8c1c497bc772..a7fc6fc85fbe 100644 --- a/Documentation/driver-api/cxl/linux/early-boot.rst +++ b/Documentation/driver-api/cxl/linux/early-boot.rst @@ -12,8 +12,9 @@ read EFI and ACPI information throughout this process to = configure logical representations of the devices. =20 During Linux Early Boot stage (functions in the kernel that have the __init -decorator), the system takes the resources created by EFI/BIOS (ACPI table= s) -and turns them into resources that the kernel can consume. +decorator), the system takes the resources created by EFI/BIOS +(:doc:`ACPI tables <../platform/acpi>`) and turns them into resources that= the +kernel can consume. =20 =20 BIOS, Build and Boot Options @@ -70,13 +71,14 @@ significant impact performance depending on the memory = capacity of the system. NUMA Node Reservation =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 -Linux refers to the proximity domains (:code:`PXM`) defined in the SRAT to -create NUMA nodes in :code:`acpi_numa_init`. Typically, there is a 1:1 rel= ation -between :code:`PXM` and NUMA node IDs. +Linux refers to the proximity domains (:code:`PXM`) defined in the :doc:`S= RAT +<../platform/acpi/srat>` to create NUMA nodes in :code:`acpi_numa_init`. +Typically, there is a 1:1 relation between :code:`PXM` and NUMA node IDs. =20 -SRAT is the only ACPI defined way of defining Proximity Domains. Linux cho= oses -to, at most, map those 1:1 with NUMA nodes. CEDT adds a description of SPA -ranges which Linux may wish to map to one or more NUMA nodes. +The SRAT is the only ACPI defined way of defining Proximity Domains. Linux +chooses to, at most, map those 1:1 with NUMA nodes. +:doc:`CEDT <../platform/acpi/cedt>` adds a description of SPA ranges which +Linux may map to one or more NUMA nodes. =20 If there are CXL ranges in the CFMWS but not in SRAT, then a fake :code:`P= XM` is created (as of v6.15). In the future, Linux may reject CFMWS not descri= bed @@ -89,7 +91,8 @@ data for Linux to identify NUMA nodes their associated me= mory regions. =20 The relevant code exists in: :code:`linux/drivers/acpi/numa/srat.c`. =20 -See the Example Platform Configurations section for more information. +See :doc:`Example Platform Configurations <../platform/example-configs>` +for more info. =20 Memory Tiers Creation =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D @@ -108,10 +111,13 @@ Tier membership can be inspected in :: /sys/devices/virtual/memory_tiering/memory_tierN/nodelist 0-1 =20 -If nodes are grouped which have clear difference in performance, check the= HMAT -and CDAT information for the CXL nodes. All nodes default to the DRAM tie= r, -unless HMAT/CDAT information is reported to the memory_tier component via -`access_coordinates`. +If nodes are grouped which have clear difference in performance, check the +:doc:`HMAT <../platform/acpi/hmat>` and CDAT information for the CXL nodes= . All +nodes default to the DRAM tier, unless HMAT/CDAT information is reported t= o the +memory_tier component via `access_coordinates`. + +For more, see :doc:`CXL access coordinates documentation +<../linux/access-coordinates>`. =20 Contiguous Memory Allocation =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D diff --git a/Documentation/driver-api/cxl/platform/bios-and-efi.rst b/Docum= entation/driver-api/cxl/platform/bios-and-efi.rst index 552a83992bcc..645322632cc9 100644 --- a/Documentation/driver-api/cxl/platform/bios-and-efi.rst +++ b/Documentation/driver-api/cxl/platform/bios-and-efi.rst @@ -22,7 +22,7 @@ At a high level, this is what occurs during this phase of= configuration. =20 Much of what this section is concerned with is ACPI Table production and static memory map configuration. More detail on these tables can be found -under Platform Configuration -> ACPI Table Reference. +at :doc:`ACPI Tables `. =20 .. note:: Platform Vendors should read carefully, as this sections has recommenda= tions @@ -175,9 +175,9 @@ to implement driver support for your platform. =20 Interleave and Configuration Flexibility ---------------------------------------- -If providing cross-host-bridge interleave, a CFMWS entry in the CEDT must = be -presented with target host-bridges for the interleaved device sets (there = may -be multiple behind each host bridge). +If providing cross-host-bridge interleave, a CFMWS entry in the :doc:`CEDT +` must be presented with target host-bridges for the interleaved +device sets (there may be multiple behind each host bridge). =20 If providing intra-host-bridge interleaving, only 1 CFMWS entry in the CED= T is required for that host bridge - if it covers the entire capacity of the de= vices @@ -193,8 +193,8 @@ different purposes. For example, you may want to consi= der adding: =20 A platform may choose to add all of these, or change the mode based on a B= IOS setting. For each CFMWS entry, Linux expects descriptions of the described -memory regions in the SRAT to determine the number of NUMA nodes it should -reserve during early boot / init. +memory regions in the :doc:`SRAT ` to determine the number of +NUMA nodes it should reserve during early boot / init. =20 As of v6.14, Linux will create a NUMA node for each CEDT CFMWS entry, even= if a matching SRAT entry does not exist; however, this is not guaranteed in t= he diff --git a/Documentation/driver-api/cxl/platform/example-configurations/f= lexible.rst b/Documentation/driver-api/cxl/platform/example-configurations/= flexible.rst index e39daba65fa0..dab704b6fcc2 100644 --- a/Documentation/driver-api/cxl/platform/example-configurations/flexible= .rst +++ b/Documentation/driver-api/cxl/platform/example-configurations/flexible= .rst @@ -18,7 +18,7 @@ Things to note: * This SRAT describes one node for each of the above CFMWS. * The HMAT describes performance for each node in the SRAT. =20 -CEDT :: +:doc:`CEDT <../acpi/cedt>`:: =20 Subtable Type : 00 [CXL Host Bridge Structure] Reserved : 00 @@ -137,7 +137,7 @@ CEDT :: QtgId : 0001 First Target : 00000006 =20 -SRAT :: +:doc:`SRAT <../acpi/srat>`:: =20 Subtable Type : 01 [Memory Affinity] Length : 28 @@ -223,7 +223,7 @@ SRAT :: Hot Pluggable : 1 Non-Volatile : 0 =20 -HMAT :: +:doc:`HMAT <../acpi/hmat>`:: =20 Structure Type : 0001 [SLLBI] Data Type : 00 [Latency] @@ -263,7 +263,7 @@ HMAT :: Entry : 0100 Entry : 0100 =20 -SLIT :: +:doc:`SLIT <../acpi/slit>`:: =20 Signature : "SLIT" [System Locality Information Table] Localities : 0000000000000003 @@ -276,7 +276,7 @@ SLIT :: Locality 6 : FF FF FF FF FF FF 0A FF Locality 7 : FF FF FF FF FF FF FF 0A =20 -DSDT :: +:doc:`DSDT <../acpi/dsdt>`:: =20 Scope (_SB) { diff --git a/Documentation/driver-api/cxl/platform/example-configurations/h= b-interleave.rst b/Documentation/driver-api/cxl/platform/example-configurat= ions/hb-interleave.rst index ce07e6162f26..c474dcf09fb0 100644 --- a/Documentation/driver-api/cxl/platform/example-configurations/hb-inter= leave.rst +++ b/Documentation/driver-api/cxl/platform/example-configurations/hb-inter= leave.rst @@ -13,7 +13,7 @@ Things to note: * This SRAT describes one node for both host bridges. * The HMAT describes a single node's performance. =20 -CEDT :: +:doc:`CEDT <../acpi/cedt>`:: =20 Subtable Type : 00 [CXL Host Bridge Structure] Reserved : 00 @@ -48,7 +48,7 @@ CEDT :: First Target : 00000007 Second Target : 00000006 =20 -SRAT :: +:doc:`SRAT <../acpi/srat>`:: =20 Subtable Type : 01 [Memory Affinity] Length : 28 @@ -62,7 +62,7 @@ SRAT :: Hot Pluggable : 1 Non-Volatile : 0 =20 -HMAT :: +:doc:`HMAT <../acpi/hmat>`:: =20 Structure Type : 0001 [SLLBI] Data Type : 00 [Latency] @@ -80,14 +80,14 @@ HMAT :: Entry : 1200 Entry : 0400 =20 -SLIT :: +:doc:`SLIT <../acpi/slit>`:: =20 Signature : "SLIT" [System Locality Information Table] Localities : 0000000000000003 Locality 0 : 10 20 Locality 1 : FF 0A =20 -DSDT :: +:doc:`DSDT <../acpi/dsdt>`:: =20 Scope (_SB) { diff --git a/Documentation/driver-api/cxl/platform/example-configurations/m= ulti-dev-per-hb.rst b/Documentation/driver-api/cxl/platform/example-configu= rations/multi-dev-per-hb.rst index 6adf7c639490..a7854a79dbbd 100644 --- a/Documentation/driver-api/cxl/platform/example-configurations/multi-de= v-per-hb.rst +++ b/Documentation/driver-api/cxl/platform/example-configurations/multi-de= v-per-hb.rst @@ -14,7 +14,7 @@ Things to note: * This CEDT/SRAT describes one node for both devices. * There is only one proximity domain the HMAT for both devices. =20 -CEDT :: +:doc:`CEDT <../acpi/cedt>`:: =20 Subtable Type : 00 [CXL Host Bridge Structure] Reserved : 00 @@ -39,7 +39,7 @@ CEDT :: QtgId : 0001 First Target : 00000007 =20 -SRAT :: +:doc:`SRAT <../acpi/srat>`:: =20 Subtable Type : 01 [Memory Affinity] Length : 28 @@ -53,7 +53,7 @@ SRAT :: Hot Pluggable : 1 Non-Volatile : 0 =20 -HMAT :: +:doc:`HMAT <../acpi/hmat>`:: =20 Structure Type : 0001 [SLLBI] Data Type : 00 [Latency] @@ -69,14 +69,14 @@ HMAT :: Entry : 1200 Entry : 0200 =20 -SLIT :: +:doc:`SLIT <../acpi/slit>`:: =20 Signature : "SLIT" [System Locality Information Table] Localities : 0000000000000003 Locality 0 : 10 20 Locality 1 : FF 0A =20 -DSDT :: +:doc:`DSDT <../acpi/dsdt>`:: =20 Scope (_SB) { diff --git a/Documentation/driver-api/cxl/platform/example-configurations/o= ne-dev-per-hb.rst b/Documentation/driver-api/cxl/platform/example-configura= tions/one-dev-per-hb.rst index b89ba3cab98f..aebda0eb3e17 100644 --- a/Documentation/driver-api/cxl/platform/example-configurations/one-dev-= per-hb.rst +++ b/Documentation/driver-api/cxl/platform/example-configurations/one-dev-= per-hb.rst @@ -14,7 +14,7 @@ Things to note: * This CEDT/SRAT describes one node per device * The expanders have the same performance and will be in the same memory t= ier. =20 -CEDT :: +:doc:`CEDT <../acpi/cedt>`:: =20 Subtable Type : 00 [CXL Host Bridge Structure] Reserved : 00 @@ -62,7 +62,7 @@ CEDT :: QtgId : 0001 First Target : 00000006 =20 -SRAT :: +:doc:`SRAT <../acpi/srat>`:: =20 Subtable Type : 01 [Memory Affinity] Length : 28 @@ -88,7 +88,7 @@ SRAT :: Hot Pluggable : 1 Non-Volatile : 0 =20 -HMAT :: +:doc:`HMAT <../acpi/hmat>`:: =20 Structure Type : 0001 [SLLBI] Data Type : 00 [Latency] @@ -108,7 +108,7 @@ HMAT :: Entry : 0200 Entry : 0200 =20 -SLIT :: +:doc:`SLIT <../acpi/slit>`:: =20 Signature : "SLIT" [System Locality Information Table] Localities : 0000000000000003 @@ -116,7 +116,7 @@ SLIT :: Locality 1 : FF 0A FF Locality 2 : FF FF 0A =20 -DSDT :: +:doc:`DSDT <../acpi/dsdt>`:: =20 Scope (_SB) { --=20 2.49.0