From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B8DD25B1C2 for ; Wed, 30 Apr 2025 18:11:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036664; cv=none; b=qyjAXmuCaMQs/oMANcA4+JHAgpry14xPxX8FmTj5Ry2XQNQ7ZNmcmIjkOaZ8tNPWfD3vOfXQArjYdjCe3gykeJFj0YQsvdRcmif+5TsV3dkopumhOwimpPjDfLE94r7G0cN+HbhsopGCYY69pURtj7Uw7TCYrdAqTWFz4mvNu9k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036664; c=relaxed/simple; bh=bc+l52lzMgg34Vw41X2Wy/FxQj+yccNEeALSbRGK27Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tPFNaYi8pCZf09DZ8G3r4iAPUd3FC0Sl/V7HMpEWZcWmY+yR0DXDiuj+ro4LSOGxia/0YlAAJ4FzW8sFuZDY//kAtNggvl4N8bXhnoj0BBAdptSLnOABf73vICXioH6yIrtzgqm1uxkSgacEbngBV/biM7NMigY9eO9Zl03jbwA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=aqsIxHzr; arc=none smtp.client-ip=209.85.160.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="aqsIxHzr" Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-476977848c4so2562121cf.1 for ; Wed, 30 Apr 2025 11:11:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036662; x=1746641462; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=U3rVHINvuxD50BlXL+fWePXRUBl5vPBCCVNAK+caSv0=; b=aqsIxHzrJDIZL/iYJd6VsqJJjdiFAzqfLxhI4c46o8J6KLCx/Y3bPNgzOBp2E+01Pr OXf43OGl9NJZz4y9nguXpWSkt1s5CD5lV9m3Dm5VxODJDQmXuDJgMd4Wr4VAsu2k+i4b 539NJAljiflbVOnndzTRP6wK3IG6IZ9hZUuHWBdFz/6o61bNb3GwWaymj6UVJ/hOPSh8 PEg9kHMtIRg70gxZ+0VZ4F1nd+6zfNDXGEuGSuMZwQxg+gFaND4xa/FMOVrpiHdiXAAu /G8pMCsxJOeo76u0sC9OdJtBFHpSYzwUxqIaA2/2sqw+LHEE5DORUh41lJn8z+ShKsxn w20g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036662; x=1746641462; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U3rVHINvuxD50BlXL+fWePXRUBl5vPBCCVNAK+caSv0=; b=ocHdv0KvdK+kSK1IwXgL0DWLBXKK8sxUsFsTayToADE2HSfQFHuz2nE0DlO4ESJiDD FLJF3Z67EYyw7w5k1RW2eH0MgN3SzE5r8frdtPZa2Fb/al+4vUek3CPBUdQKsLvdSqbT 2RfPx/o2R1xKZCFkPSmq3qHdixND9IT8N8Whoas0MM7iCcc4QikPWivGV9aBWUOANiqm 44DH+NVaf5zb2gQupgYjh7mRNwsRMZiBZrzWD8wjrXkmNfB3y6s+QVRQXrThaYu7bPjK rPKCfnQIMVzyih9d2Owo7ODWUx/VgNTXgPwo2WPh8cVQPlDX1S3NHXd8VlSIx+eyaFR6 B12g== X-Forwarded-Encrypted: i=1; AJvYcCUhdohp8KljdnqkQjNAj8r6BPt6HPOD4Iyd3sOq0ypS9JdvQNhvoHVs6blSo42WylzDs8LR9ZRW5aCf/xw=@vger.kernel.org X-Gm-Message-State: AOJu0YxV2EMuUCY4f0qcqQ43uRBek6JJGNAhc1oPpl/p3HEoNOS+nbBO q5CMmgoQ8pR6Kqs5E5hcoJNMMV0nl7lQ95Bp/2zsi6YT2NJH07IsM3Tj41RDUXc= X-Gm-Gg: ASbGncvEaxF251VaBrQ2ZlmDZ5/BEfB12+3h7Meb4StV8LQsgzOFlC85QKegEFbxqax +NsrOufmal3lVvpqe0nnI4cEOLH3bLUD6MDMFPYpc/DUwkgyJa6VtFpmuURLxvJ0Y/x7/h6rcKD wdHXcGd8WyQ1fgRNwqrSacZLEUgQaqiwubvlB7qwipo/HX0+7jn6SAORgkCqvO20C4HYaa9w6lL hBHH/vJ5IlD6uonbHLiv353LMTBF/eJeTzaMeXZbxyXPawTrZINEcue/tRq9Vms08XtEzq7CIKL C5MITlBMUyDVD20Z3cGGXPV4uz1KLhJzBwsOAhX000iICS2cWslzEL0W+H6FRrg/ZZAzhKdd8tT 7B7bu6B18ZAVQPOY1WQnJgDbLSlyI X-Google-Smtp-Source: AGHT+IFEFexCZkfsgjT0Bdcj5OkkLcxTMyz8IGrh8K077WV+JwvSahKI4mujBFQCFuESr0+In0WTAQ== X-Received: by 2002:a05:622a:5815:b0:477:e7c:a4c with SMTP id d75a77b69052e-489c5212aebmr68148911cf.39.1746036661817; Wed, 30 Apr 2025 11:11:01 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:01 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 01/18] cxl: update documentation structure in prep for new docs Date: Wed, 30 Apr 2025 14:10:31 -0400 Message-ID: <20250430181048.1197475-2-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Restructure the cxl folder to make adding docs per-page cleaner. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 16 +++++++++++++--- .../cxl/{ =3D> linux}/access-coordinates.rst | 0 ...emory-devices.rst =3D> theory-of-operation.rst} | 10 +++++----- 3 files changed, 18 insertions(+), 8 deletions(-) rename Documentation/driver-api/cxl/{ =3D> linux}/access-coordinates.rst (= 100%) rename Documentation/driver-api/cxl/{memory-devices.rst =3D> theory-of-ope= ration.rst} (98%) diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 965ba90e8fb7..fe1594dc6778 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -4,12 +4,22 @@ Compute Express Link =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 +CXL device configuration has a complex handoff between platform (Hardware, +BIOS, EFI), OS (early boot, core kernel, driver), and user policy decisions +that have impacts on each other. The docs here break up configurations st= eps. + +.. toctree:: + :maxdepth: 2 + :caption: Overview + + theory-of-operation + maturity-map + .. toctree:: :maxdepth: 1 + :caption: Linux Kernel Configuration =20 - memory-devices - access-coordinates + linux/access-coordinates =20 - maturity-map =20 .. only:: subproject and html diff --git a/Documentation/driver-api/cxl/access-coordinates.rst b/Document= ation/driver-api/cxl/linux/access-coordinates.rst similarity index 100% rename from Documentation/driver-api/cxl/access-coordinates.rst rename to Documentation/driver-api/cxl/linux/access-coordinates.rst diff --git a/Documentation/driver-api/cxl/memory-devices.rst b/Documentatio= n/driver-api/cxl/theory-of-operation.rst similarity index 98% rename from Documentation/driver-api/cxl/memory-devices.rst rename to Documentation/driver-api/cxl/theory-of-operation.rst index d732c42526df..32739e253453 100644 --- a/Documentation/driver-api/cxl/memory-devices.rst +++ b/Documentation/driver-api/cxl/theory-of-operation.rst @@ -1,9 +1,9 @@ .. SPDX-License-Identifier: GPL-2.0 .. include:: =20 -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D -Compute Express Link Memory Devices -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Compute Express Link Driver Theory of Operation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 A Compute Express Link Memory Device is a CXL component that implements the CXL.mem protocol. It contains some amount of volatile memory, persistent m= emory, @@ -14,8 +14,8 @@ that optionally define a device's contribution to an inte= rleaved address range across multiple devices underneath a host-bridge or interleaved across host-bridges. =20 -CXL Bus: Theory of Operation -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +The CXL Bus +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Similar to how a RAID driver takes disk objects and assembles them into a = new logical device, the CXL subsystem is tasked to take PCIe and ACPI objects = and assemble them into a CXL.mem decode topology. The need for runtime configu= ration --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C3BE21CFFA for ; Wed, 30 Apr 2025 18:11:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036666; cv=none; b=C/nq7a2rmf/+rB5TgoKHcEo7N+2McS0tTyCm3GtNLn1xskg0gN2vk5dBvIRfb9c3dfkCh0f3pXFfHQBSMUgCes94cgV2GdE6u8DK6ALYdihmh3aGGCPIji4sIgpG0VPnL+P4vYooQMaY0j2ZRVtyPdLWO7Nl5E5vHEMFT0M5zzk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036666; c=relaxed/simple; bh=rRr+qcLZrtnYg2c9lYmkQcR+1K4cfa1JZgb5Te9T7QA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=U67Od+snmPZ8s/ORmgADrT39cLIUIblnDmL63lr71SmYRaiqDXyFxq4eRr9s4zMxcYtnxyYvqDN68b8WhauQD5mCNbF4hze/OfJZWjP7wCW3Bkd8oQ4flxhFrLYHY83KJ8Rf8axoiNi4Q7kPsO70sUVHsjOZLedApS5KeSZhCYc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=eJa5kCIo; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="eJa5kCIo" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-4769aef457bso2100901cf.2 for ; Wed, 30 Apr 2025 11:11:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036663; x=1746641463; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=a/j5BwxlV3klVmfErwodvPQTvZR5n5+OrQaN53q1h6A=; b=eJa5kCIoTFCFvnE7nTJXIJhVlgYPYgIOy0xebPYCoSgy95Reh3685AoHWfrc+Uv1vk /Gx+4VXVZMCh1NvrMWFOKnBLIbhHv7YtfHk1xZlMR8kle/SwND3TvXuAjugG0Hqrk2mY Ajn8Amu03n5mo0FDuCyxXkUmrt/hD3fZYK1S9rJQv7Wisv1JTmqAVe0+wfPwcAIPL2fg HSsflqZcVPmQ60c6LF2tUvaPIrPZ2kWYeRz3ecKSkRQ9b/g5Xla6E5kHvjEi06sK7F6J 0gTQH8QCSIerzr2+TwKkutFuIHUiwzqLr1//UFq4N06Rq4K6TX4G/sDh3vzOIHErutn/ 6q0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036663; x=1746641463; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a/j5BwxlV3klVmfErwodvPQTvZR5n5+OrQaN53q1h6A=; b=CCoSxzIRFeVquIJy4lUZ8CV4C1b4y/m+HhbLb6aIuD08YKL0dnuUnijRzdZwRlpPWY QAlNoQAz6eSJFsm5IkTpBQfonaYJRJ18RWWTcMN6liqKfJndzpPK9YcTxRXyZcsw3sTN 3VrtDVI5h4pRsVYrgpENQRBjhDKAWoRm+YOKr1377oYk9ut6loKZtFHtg+BrEabHMLwB ygQ616mwmAb1nfIlD8bkv/MRPTmxRA91iwBuBEaKTnX1MPoJOo7dpXiOlrm+X2QiGEkT tj4h5sqQSMd5cVw04jIRzeaEEPgmrQAxhZiCxaa4ebhE+QKxGSr/ipCT3Q3lP7kiiSBx v0eA== X-Forwarded-Encrypted: i=1; AJvYcCWjJ+j1OM/dFXGXOwceolcozo+pD22RtrC9ow0z57QUboY9KpovfZGogEibxEmkjTZwAW3hS2MZ2upOdxU=@vger.kernel.org X-Gm-Message-State: AOJu0YyzN1GmIZsMGaTiT4P4ON1ag5bP2wC2lHmJlnWf/oUrdjaxxG3J Vc9xxRwvpmb4HxrUCMfcNtK7J0J1m9hwwiZ8bJk9e+T/7EDk/EniEqJvDn9DgWU= X-Gm-Gg: ASbGncs3kOmK1pWAJ1PjjGY7DFRmWjJzi56EceVunap09YIxtKDB+02J+/e5D9gukAw AJpJ2h8iwWQBkGsF7Y3a/jmi+P+9LTL0IXKGiFFhsity4qY3yS8cA64Duq7smvdjdHlJB6rOam5 er/H+OlDdm9aOKKRr6VqzU1GmBadcc0jmKc6SHTx59QSD4zpJkRUTs45S6raMGpdtbuaM4tW2xN Hh/RLMMtINsiwONxiQnXsDammg6ja6kVaqIWsfCUIh4f2t/6sMxyxF7/h9C2ETYH/NvRxEA2k5z yLmDeLnbocqSm1dIBFgniozzlHY3w0omXmAkfW+1sytmb/LLwQZYU/Z5Tbl33DWK6pZkB9/5eM2 yOfUBXlwrikq7KHhGKV+HZ9HZ6EST X-Google-Smtp-Source: AGHT+IHDZtessbJycNGB3aPl1k1W+xYcbtzzI+zp3HWXp/EAxUK+kz6cS6edtsKGPIExvfbqIBqSiw== X-Received: by 2002:a05:622a:5c1b:b0:47a:e63b:ec60 with SMTP id d75a77b69052e-489c4450e72mr67614731cf.27.1746036663333; Wed, 30 Apr 2025 11:11:03 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:02 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 02/18] cxl: docs/devices - device reference and uefi placeholder Date: Wed, 30 Apr 2025 14:10:32 -0400 Message-ID: <20250430181048.1197475-3-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a simple device primer sufficient to understand the theory of operation documentation. Add carve-out for CDAT with a TODO. Signed-off-by: Gregory Price --- .../driver-api/cxl/devices/device-types.rst | 170 ++++++++++++++++++ Documentation/driver-api/cxl/devices/uefi.rst | 10 ++ Documentation/driver-api/cxl/index.rst | 7 + 3 files changed, 187 insertions(+) create mode 100644 Documentation/driver-api/cxl/devices/device-types.rst create mode 100644 Documentation/driver-api/cxl/devices/uefi.rst diff --git a/Documentation/driver-api/cxl/devices/device-types.rst b/Docume= ntation/driver-api/cxl/devices/device-types.rst new file mode 100644 index 000000000000..dfe8d4711987 --- /dev/null +++ b/Documentation/driver-api/cxl/devices/device-types.rst @@ -0,0 +1,170 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Devices and Protocols +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The type of CXL device (Memory, Accelerator, etc) dictates many configurat= ion steps. This section +covers some basic background on device types and on-device resources used = by the platform and OS +which impact configuration. + +Protocols +=3D=3D=3D=3D=3D=3D=3D=3D=3D + +There are three core protocols to CXL. For the purpose of this documentat= ion, +we will only discuss very high level definitions as the specific hardware +details are largely abstracted away from Linux. See the CXL specification +for more details. + +CXL.io +------ +The basic interaction protocol, similar to PCIe configuration mechanisms. +Typically used for initialization, configuration, and I/O access for anyth= ing +other than memory (CXL.mem) or cache (CXL.cache) operations. + +The Linux CXL driver exposes access to .io functionalty via the various sy= sfs +interfaces and /dev/cxl/ devices (which exposes direct access to device +mailboxes). + +CXL.cache +--------- +The mechanism by which a device may coherently access and cache host memor= y. + +Largely transparent to Linux once configured. + +CXL.mem +--------- +The mechanism by which the CPU may coherently access and cache device memo= ry. + +Largely transparent to Linux once configured. + + +Device Types +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Type-1 +------ + +A Type-1 CXL device: + +* Supports cxl.io and cxl.cache protocols +* Implements a fully coherent cache +* Allow Device-to-Host coherence and Host-to-Device snoops. +* Does NOT have host-managed device memory (HDM) + +Typical examples of type-1 devices is a Smart NIC - which may want to +directly operate on host-memory (DMA) to store incoming packets. These +devices largely rely on CPU-attached memory. + +Type-2 +------ + +A Type-2 CXL Device: + +* Supports cxl.io, cxl.cache, and cxl.mem protocols +* Optionally implements coherent cache and Host-Managed Device Memory +* Is typically an accelerator device w/ high bandwidth memory. + +The primary difference between a type-1 and type-2 device is the presence +of host-managed device memory, which allows the device to operate on a +local memory bank - while the CPU sill has coherent DMA to the same memory. + +The allows things like GPUs to expose their memory via DAX devices or file +descriptors, allows drivers and programs direct access to device memory +rather than use block-transfer semantics. + +Type-3 +------ + +A Type-3 CXL Device + +* Supports cxl.io and cxl.mem +* Implements Host-Managed Device Memory +* May provide either Volatile or Persistent memory capacity (or both). + +A basic example of a type-3 device is a simple memory expanded, whose +local memory capacity is exposed to the CPU for access directly via +basic coherent DMA. + +Switch +------ + +A CXL switch is a device capacity of routing any CXL (and by extension, PC= Ie) +protocol between an upstream, downstream, or peer devices. Many devices, = such +as Multi-Logical Devices, imply the presence of switching in some manner. + +Logical Devices and Heads +------------------------- + +A CXL device may present one or more "Logical Devices" to one or more hosts +(via physical "Heads"). + +A Single-Logical Device (SLD) is a device which presents a single device to +one or more heads. + +A Multi-Logical Device (MLD) is a device which may present multiple devices +to one or more devices. + +A Single-Headed Device exposes only a single physical connection. + +A Multi-Headed Device exposes multiple physical connections. + +MHSLD +~~~~~ +A Multi-Headed Single-Logical Device (MHSLD) exposes a single logical +device to multiple heads which may be connected to one or more discrete +hosts. An example of this would be a simple memory-pool which may be +statically configured (prior to boot) to expose portions of its memory +to Linux via the CEDT ACPI table. + +MHMLD +~~~~~ +A Multi-Headed Multi-Logical Device (MHMLD) exposes multiple logical +devices to multiple heads which may be connected to one or more discrete +hosts. An example of this would be a Dynamic Capacity Device or which +may be configured at runtime to expose portions of its memory to Linux. + +Example Devices +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Memory Expander +--------------- +The simplest form of Type-3 device is a memory expander. A memory expander +exposes Host-Managed Device Memory (HDM) to Linux. This memory may be +Volatile or Non-Volatile (Persistent). + +Memory Expanders will typically be considered a form of Single-Headed, +Single-Logical Device - as its form factor will typically be an add-in-card +(AIC) or some other similar form-factor. + +The Linux CXL driver provides support for static or dynamic configuration = of +basic memory expanders. The platform may program decoders prior to OS init +(e.g. auto-decoders), or the user may program the fabric if the platform +defers these operations to the OS. + +Multiple Memory Expanders may be added to an external chassis and exposed = to +a host via a head attached to a CXL switch. This is a "memory pool", and +would be considered an MHSLD or MHMLD depending on the management capabili= ties +provided by the switch platform. + +As of v6.14, Linux does not provide a formalized interface to manage non-D= CD +MHSLD or MHMLD devices. + +Dynamic Capacity Device (DCD) +----------------------------- + +A Dynamic Capacity Device is a Type-3 device which provides dynamic manage= ment +of memory capacity. The basic premise of a DCD to provide an allocator-like +interface for physical memory capacity to a "Fabric Manager" (an external, +privileged host with privileges to change configurations for other hosts). + +A DCD manages "Memory Extents", which may be volatile or persistent. Exten= ts +may also be exclusive to a single host or shared across multiple. + +As of v6.14, Linux does not provide a formalized interface to manage DCD +devices, however there is active work on LKML targeting future release. + +Type-2 Device +------------- + +Todo diff --git a/Documentation/driver-api/cxl/devices/uefi.rst b/Documentation/= driver-api/cxl/devices/uefi.rst new file mode 100644 index 000000000000..a431a0e3d980 --- /dev/null +++ b/Documentation/driver-api/cxl/devices/uefi.rst @@ -0,0 +1,10 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D +UEFI Data +=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Coherent Device Attribute Table (CDAT) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +todo diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index fe1594dc6778..8e9d024cdfac 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -15,6 +15,13 @@ that have impacts on each other. The docs here break up= configurations steps. theory-of-operation maturity-map =20 +.. toctree:: + :maxdepth: 2 + :caption: Device Reference + + devices/device-types + devices/uefi + .. toctree:: :maxdepth: 1 :caption: Linux Kernel Configuration --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 569E6266B47 for ; Wed, 30 Apr 2025 18:11:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036669; cv=none; b=pr/NWv3T6xRVT5dfN89acW2t81XvYoGSl9M0Wh50kA+YW7p9v6LKostWkbilXY7/OvzNaIvPFBz1E+hXBAnC9W7i82OaPZCiiBoL58yNIFRDd6kfjQoivkKKp+PWshw+6FwRPvHQaYNtBBMw6yTLMsm3NL8v8zynSpmWxxvkxO0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036669; c=relaxed/simple; bh=KvNSFJq9rASJXID87NCuG6RqLE6dY1Gl1jvW5kVlqFg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XT+KImnKsMSAZua/XpTz/MoApl3BLj6xjxc8coKqaArZNWnOqBoOF6Q6vlIyGe7tA/vPgajPS1jslD518XcnRmrdXHp+jedClDn84QFDsgxhRPFfIbcq9Te5aPtws3A4wiM29wvsDyTGE1TliaBldv7zxuNZpAHWNtIPI0eOEG8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=LsO4fOz5; arc=none smtp.client-ip=209.85.160.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="LsO4fOz5" Received: by mail-qt1-f169.google.com with SMTP id d75a77b69052e-4775ccf3e56so16008851cf.0 for ; Wed, 30 Apr 2025 11:11:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036665; x=1746641465; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P4TKqosidfYPUmViNB1FSWtCP5cThhjm46zc8OPnQtc=; b=LsO4fOz5lmO2fb4m0V+RNF8MNJLVchcWWIutmp4DZIdJYlRPYOMdxZ02FTZjH40qeS rIH2u9qFhGe+N9e4zcHLa6STuwy/kn6Hoe+kwcGLibws+qmlIlUHvTZ0LyyUMp9JK1Du BGEEBWN10mY1G8B3BvqZB9YcD2QHH4LrzR3GUGR1Q/0gaPuYyQU0G0uifGzbVMptI/pl yz9+G5j+nOxzSoAMVNhYBPVl9IoRbGS6xO+x8CPXpYNsJbeyYbxxyieMXFQ+v3OhBWFa iNkXsohlWODn885k8xwEtaHdUzX136Jlz/Dw0MO/Da19cvTmOHgFgTW1+rM+QmLPZRrZ /E6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036665; x=1746641465; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P4TKqosidfYPUmViNB1FSWtCP5cThhjm46zc8OPnQtc=; b=DGZ4WRtSSXjK4VH82qPqT/PE4n3H/o+wMQ70Am36oO5cp2xKRxM3q63XTBmPtG0Za3 U+dF8+E9NxjdwMK4+uIcCH1l1m1WMdPWViF6Y+HGLDUAHcVRDTPz2CQlUOxQ+BWsbz9O EVAb1qVWe1JWEpqIdUsnFVh7NX6uzY2ThpMj+RmSUfteNuxPY4iDsaBf8K7NPHzjPGkB pwyMKZuN0S4cUUU7mBrh2J1utJVOSru7RI/eeAHBvJ7/wcdCd3XW880Fas/JCfYYVHZT fDTVU4ml1BIuTzq1D45/LS5wYV1GiEnXJ9k4rxcRzlF6zJ9/dwDRwCcGfurf5gGxbklA GWYw== X-Forwarded-Encrypted: i=1; AJvYcCVTkorQvrFOS7WXcMB52VMPmI8hsR8A9GCsZsiNAp0EVxcln93w3ddFWB5xUjSqLiEWE8bCZBvnGyVLXKs=@vger.kernel.org X-Gm-Message-State: AOJu0Yw3RcTt8otUYdma0dHF7WnqeF3nnTErIfiq9qSVvgwwghpiMZIF g59QlOt86oP3xX79KaAkoxYKOpVftPHXdrU1sX2ouob45U1LW0bSf+Jjv5uTKKw= X-Gm-Gg: ASbGncuzk/UjdAlRO6WVfSM2GQnvu9dPmViH/GMd2/m5OVtlEFsLAZtb83W//HqUdyY ApvKZ/rNn32ACQeXIXTq8/yT7IHfnVb1tYJCd6fNwI36OzDrVJyilmmmzPI5hhVzAitycTm0ZIm iXgoQ4hs8bHnLgRGyRH19rOsj6UVrYZCJCNy9ruH6e/OWS2Zn85MxyT7K4hddY7bdvbNt6sbmid ixy/QszGOHUvl3fIocFDarrMX8HPMV7Rc29ydL/7yA00nFxPqfiYp+j+EH/Txrm9U0wgwN6vKU5 p6maFRFNX58ObMZ3NZozgJVHSZjy8t3VeUJ9PhPonhPQWu2h28R1Fi9JzNeqotPFgsNb1dmSazT rHpuemT+ak2lrchX+rt4WaHJnWCLI X-Google-Smtp-Source: AGHT+IGjywhCYYedP88BeAr8yEhOXWYBcOSmHHv4KQzgFZDxo4I+IaCwfDxqzyJr+OpTscZoDFNxiQ== X-Received: by 2002:a05:622a:5a10:b0:47b:3a2:ff1f with SMTP id d75a77b69052e-48ad89a4684mr8061961cf.12.1746036664771; Wed, 30 Apr 2025 11:11:04 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:04 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 03/18] cxl: docs/platform/bios-and-efi documentation Date: Wed, 30 Apr 2025 14:10:33 -0400 Message-ID: <20250430181048.1197475-4-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add some docs on CXL configurations done in bios/efi that affect linux configuration - information vendors may care to consider. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 6 + .../driver-api/cxl/platform/bios-and-efi.rst | 262 ++++++++++++++++++ 2 files changed, 268 insertions(+) create mode 100644 Documentation/driver-api/cxl/platform/bios-and-efi.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 8e9d024cdfac..b8f82cdf1435 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -22,6 +22,12 @@ that have impacts on each other. The docs here break up= configurations steps. devices/device-types devices/uefi =20 +.. toctree:: + :maxdepth: 2 + :caption: Platform Configuration + + platform/bios-and-efi + .. toctree:: :maxdepth: 1 :caption: Linux Kernel Configuration diff --git a/Documentation/driver-api/cxl/platform/bios-and-efi.rst b/Docum= entation/driver-api/cxl/platform/bios-and-efi.rst new file mode 100644 index 000000000000..552a83992bcc --- /dev/null +++ b/Documentation/driver-api/cxl/platform/bios-and-efi.rst @@ -0,0 +1,262 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +BIOS/EFI Configuration +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +BIOS and EFI are largely responsible for configuring static information ab= out +devices (or potential future devices) such that Linux can build the approp= riate +logical representations of these devices. + +At a high level, this is what occurs during this phase of configuration. + +* The bootloader starts the BIOS/EFI. + +* BIOS/EFI do early device probe to determine static configuration + +* BIOS/EFI creates ACPI Tables that describe static config for the OS + +* BIOS/EFI create the system memory map (EFI Memory Map, E820, etc) + +* BIOS/EFI calls :code:`start_kernel` and begins the Linux Early Boot proc= ess. + +Much of what this section is concerned with is ACPI Table production and +static memory map configuration. More detail on these tables can be found +under Platform Configuration -> ACPI Table Reference. + +.. note:: + Platform Vendors should read carefully, as this sections has recommenda= tions + on physical memory region size and alignment, memory holes, HDM interle= ave, + and what linux expects of HDM decoders trying to work with these featur= es. + +UEFI Settings +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +If your platform supports it, the :code:`uefisettings` command can be used= to +read/write EFI settings. Changes will be reflected on the next reboot. Kex= ec +is not a sufficient reboot. + +One notable configuration here is the EFI_MEMORY_SP (Specific Purpose) bit. +When this is enabled, this bit tells linux to defer management of a memory +region to a driver (in this case, the CXL driver). Otherwise, the memory is +treated as "normal memory", and is exposed to the page allocator during +:code:`__init`. + +uefisettings examples +--------------------- + +:code:`uefisettings identify` :: + + uefisettings identify + + bios_vendor: xxx + bios_version: xxx + bios_release: xxx + bios_date: xxx + product_name: xxx + product_family: xxx + product_version: xxx + +On some AMD platforms, the :code:`EFI_MEMORY_SP` bit is set via the :code:= `CXL +Memory Attribute` field. This may be called something else on your platfo= rm. + +:code:`uefisettings get "CXL Memory Attribute"` :: + + selector: xxx + ... + question: Question { + name: "CXL Memory Attribute", + answer: "Enabled", + ... + } + +Physical Memory Map +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Physical Address Region Alignment +--------------------------------- + +As of Linux v6.14, the hotplug memory system requires memory regions to be +uniform in size and alignment. While the CXL specification allows for mem= ory +regions as small as 256MB, the supported memory block size and alignment f= or +hotplugged memory is architecture-defined. + +A Linux memory blocks may be as small as 128MB and increase in powers of t= wo. + +* On ARM, the default block size and alignment is either 128MB or 256MB. + +* On x86, the default block size is 256MB, and increases to 2GB as the + capacity of the system increases up to 64GB. + +For best support across versions, platform vendors should place CXL memory= at +a 2GB aligned base address, and regions should be 2GB aligned. This also = helps +prevent the creating thousands of memory devices (one per block). + +Memory Holes +------------ + +Holes in the memory map are tricky. Consider a 4GB device located at base +address 0x100000000, but with the following memory map :: + + --------------------- + | 0x100000000 | + | CXL | + | 0x1BFFFFFFF | + --------------------- + | 0x1C0000000 | + | MEMORY HOLE | + | 0x1FFFFFFFF | + --------------------- + | 0x200000000 | + | CXL CONT. | + | 0x23FFFFFFF | + --------------------- + +There are two issues to consider: + +* decoder programming, and +* memory block alignment. + +If your architecture requires 2GB uniform size and aligned memory blocks, = the +only capacity Linux is capable of mapping (as of v6.14) would be the capac= ity +from `0x100000000-0x180000000`. The remaining capacity will be stranded, = as +they are not of 2GB aligned length. + +Assuming your architecture and memory configuration allows 1GB memory bloc= ks, +this memory map is supported and this should be presented as multiple CFMWS +in the CEDT that describe each side of the memory hole separately - along = with +matching decoders. + +Multiple decoders can (and should) be used to manage such a memory hole (s= ee +below), but each chunk of a memory hole should be aligned to a reasonable = block +size (larger alignment is always better). If you intend to have memory ho= les +in the memory map, expect to use one decoder per contiguous chunk of host +physical memory. + +As of v6.14, Linux does provide support for memory hotplug of multiple +physical memory regions separated by a memory hole described by a single +HDM decoder. + + +Decoder Programming +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +If BIOS/EFI intends to program the decoders to be statically configured, +there are a few things to consider to avoid major pitfalls that will +prevent Linux compatibility. Some of these recommendations are not +required "per the specification", but Linux makes no guarantees of support +otherwise. + + +Translation Point +----------------- +Per the specification, the only decoders which **TRANSLATE** Host Physical +Address (HPA) to Device Physical Address (DPA) are the **Endpoint Decoders= **. +All other decoders in the fabric are intended to route accesses without +translating the addresses. + +This is heavily implied by the specification, see: :: + + CXL Specification 3.1 + 8.2.4.20: CXL HDM Decoder Capability Structure + - Implementation Note: CXL Host Bridge and Upstream Switch Port Decoder = Flow + - Implementation Note: Device Decoder Logic + +Given this, Linux makes a strong assumption that decoders between CPU and +endpoint will all be programmed with addresses ranges that are subsets of +their parent decoder. + +Due to some ambiguity in how Architecture, ACPI, PCI, and CXL specificatio= ns +"hand off" responsibility between domains, some early adopting platforms +attempted to do translation at the originating memory controller or host +bridge. This configuration requires a platform specific extension to the +driver and is not officially endorsed - despite being supported. + +It is *highly recommended* **NOT** to do this; otherwise, you are on your = own +to implement driver support for your platform. + +Interleave and Configuration Flexibility +---------------------------------------- +If providing cross-host-bridge interleave, a CFMWS entry in the CEDT must = be +presented with target host-bridges for the interleaved device sets (there = may +be multiple behind each host bridge). + +If providing intra-host-bridge interleaving, only 1 CFMWS entry in the CED= T is +required for that host bridge - if it covers the entire capacity of the de= vices +behind the host bridge. + +If intending to provide users flexibility in programming decoders beyond t= he +root, you may want to provide multiple CFMWS entries in the CEDT intended = for +different purposes. For example, you may want to consider adding: + +1) A CFMWS entry to cover all interleavable host bridges. +2) A CFMWS entry to cover all devices on a single host bridge. +3) A CFMWS entry to cover each device. + +A platform may choose to add all of these, or change the mode based on a B= IOS +setting. For each CFMWS entry, Linux expects descriptions of the described +memory regions in the SRAT to determine the number of NUMA nodes it should +reserve during early boot / init. + +As of v6.14, Linux will create a NUMA node for each CEDT CFMWS entry, even= if +a matching SRAT entry does not exist; however, this is not guaranteed in t= he +future and such a configuration should be avoided. + +Memory Holes +------------ +If your platform includes memory holes intersparsed between your CXL memor= y, it +is recommended to utilize multiple decoders to cover these regions of memo= ry, +rather than try to program the decoders to accept the entire range and exp= ect +Linux to manage the overlap. + +For example, consider the Memory Hole described above :: + + --------------------- + | 0x100000000 | + | CXL | + | 0x1BFFFFFFF | + --------------------- + | 0x1C0000000 | + | MEMORY HOLE | + | 0x1FFFFFFFF | + --------------------- + | 0x200000000 | + | CXL CONT. | + | 0x23FFFFFFF | + --------------------- + +Assuming this is provided by a single device attached directly to a host b= ridge, +Linux would expect the following decoder programming :: + + ----------------------- ----------------------- + | root-decoder-0 | | root-decoder-1 | + | base: 0x100000000 | | base: 0x200000000 | + | size: 0xC0000000 | | size: 0x40000000 | + ----------------------- ----------------------- + | | + ----------------------- ----------------------- + | HB-decoder-0 | | HB-decoder-1 | + | base: 0x100000000 | | base: 0x200000000 | + | size: 0xC0000000 | | size: 0x40000000 | + ----------------------- ----------------------- + | | + ----------------------- ----------------------- + | ep-decoder-0 | | ep-decoder-1 | + | base: 0x100000000 | | base: 0x200000000 | + | size: 0xC0000000 | | size: 0x40000000 | + ----------------------- ----------------------- + +With a CEDT configuration with two CFMWS describing the above root decoder= s. + +Linux makes no guarantee of support for strange memory hole situations. + +Multi-Media Devices +------------------- +The CFMWS field of the CEDT has special restriction bits which describe wh= ether +the described memory region allows volatile or persistent memory (or both)= . If +the platform intends to support either: + +1) A device with multiple medias, or +2) Using a persistent memory device as normal memory + +A platform may wish to create multiple CEDT CFMWS entries to describe the = same +memory, with the intent of allowing the end user flexibility in how that m= emory +is configured. Linux does not presently have strong requirements in this a= rea. --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8956226B96B for ; Wed, 30 Apr 2025 18:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036669; cv=none; b=fWHw5rjHHS+hCOO9NFDZ2LpFDwYKmaYdLI27wCoaAT8LUJFgiXUy9twDJM9Ab726uqdvrsaWHwPCtnPKZwpY7tzsS4AquDBHrYvFaPG4cHV5vSjxSMKIFMwQ/0KtOB4+Mj/qZh9FAnC37N8h39LJQhM4V07TuWglIyL9gZjSNdI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036669; c=relaxed/simple; bh=VWFOqfmgJ+Pn7eDy2V+gE2NZVAyqONSkMZstHobG2uI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lyvCoC3lUf8rfh8sVRdWEj97DKVwwrk9wqDT0mH3XQgxGcT/+KC3yYnEWRkjmzHvTZnzI4inLmoLiEAjmf0dI0CM9OfCIxMMtj2WjnGd3HWf8T+YVHstk73YJRj6ybydMJ5YOel3wbh6Zrchpn19IM+UBEkQtDsYsHJTYq9TlpE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=Fijb56q/; arc=none smtp.client-ip=209.85.160.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="Fijb56q/" Received: by mail-qt1-f171.google.com with SMTP id d75a77b69052e-4774193fdffso3070481cf.1 for ; Wed, 30 Apr 2025 11:11:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036666; x=1746641466; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gVmgGfYBmV/OlbM3vIMatHDFzxM6ptqaZjXn++F71GM=; b=Fijb56q/cb+/f+uOFfN6/ObwwZmEgkpeaDyELb38aVKdlpJm8DPtReElCGAb1mqegf CwIUtT46YHjDWe3HYdwBE+zaNX/ZCqDZif1lrseAN2tU/lrpgomfOk0ix8IBohpGhoXb 1iqI3mj06pvpln3J/BmM0rGhp+wN+03jI0RvGlQ7mv5GXLaAcTsfN9y4knr7ZbDolLnf mSuANqY15EsBN4slValhOAMgJnuIIX89klojcNV6n1ErqMYhPG+Fo2qt+ZH/5dodHOSY 8b+9BNanH1DcjxiBx6afktH3mtKpD3BpFXHAemAUfAyASjUuf2xqT6e8FBYefi0BUbTJ JUQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036666; x=1746641466; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gVmgGfYBmV/OlbM3vIMatHDFzxM6ptqaZjXn++F71GM=; b=XPrym+jCM+/L8D5QN9GtoLH+Gla/5RPsCT8NZ53R8perIYp5mn1/cmqhtxDrWvHugK OrHudcRTWD3CIVpMViKEjojd94lSy70larB+31DVcOIX0zfPrxsY2onV3KwzNvvawK3j Tu1PFZMb/OFUAY40QYdtbmPbk+wqE9/DVl453ADtek5bz2tPAeo1Vvr/hWAAs6jqG/Qs iaLLqt2N7eW4qtjRZQlbg8Mo3u95pgJUtdvi4+T250zhYU6aJYwoXOtvLnw0D+UagXuJ 1+OGiu9HgBDEuvhEY0w+eNcVI+KiCB+lohjo2S1ghaYIykZstMgz5G+aYjr+5iIKwW57 G2IQ== X-Forwarded-Encrypted: i=1; AJvYcCUU+cXlOs8BrKKjxJnAxVHeXeZBPMQhYqobIfeOSlaXPgZDlq0ah6hqkqInSRunteOkW6HJgpTFwnmk5Gk=@vger.kernel.org X-Gm-Message-State: AOJu0YyYUB6o4jvSDAHOAoaSSBm+3RwRCiXuCUEKTUO1rC2SlJbwj9c1 VVLPyx/TeviSLTV8PqcN5FQS7MmGsm0dlmwow7RmzBg/2OCy+OX0lhsUQXWogcc= X-Gm-Gg: ASbGncuEpCDjmjX+v2zgPz/f9EfvUVS2yTV1lR966bP3bc552p/zBFMBTIQMK7s+rDj 4XS+/Zw2QP0CpHO9sogCNZBCFGceX+rwymV6k/QoBkZvXe+ndGbLKKRpze9ERH57G6g0TXMljhY 5F6f4+IZWn8GAcOH2h0zXWaFmHRMfx6kRMQdExYtMRfBvpTRkuyzO1fw8KBN0VTuljm1GgsliJS 6lsLf9aWs62fkYftj4k0sFoYpPUmta6tEwca1W/AGNJGHw4clVFoz2QR800iWRRK5NrcL66tfLy h97BS+QE0dT6dyJWaQnlnFyrLC2xiZiGjA+YexrWKnSs/eRog6GW7UsSySsL0DdTYElQjbnAT6e TZMySQQwV6b/+Tcssd+9mgFN3D57P X-Google-Smtp-Source: AGHT+IGcKMJcIHau07jxZ+o5UwEJXUA3OYvTS/n20sj8SIQ/bAk8uq7MQApMc9gf+SAiRuekyKxnrA== X-Received: by 2002:a05:622a:5808:b0:47a:e6e1:c04f with SMTP id d75a77b69052e-489e67b4e81mr57460491cf.46.1746036666099; Wed, 30 Apr 2025 11:11:06 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:05 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 04/18] cxl: docs/platform/acpi reference documentation Date: Wed, 30 Apr 2025 14:10:34 -0400 Message-ID: <20250430181048.1197475-5-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add basic ACPI table information needed to understand the CXL driver probe process. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/platform/acpi.rst | 76 +++++++++++++++++++ .../driver-api/cxl/platform/acpi/cedt.rst | 53 +++++++++++++ .../driver-api/cxl/platform/acpi/dsdt.rst | 28 +++++++ .../driver-api/cxl/platform/acpi/hmat.rst | 29 +++++++ .../driver-api/cxl/platform/acpi/slit.rst | 18 +++++ .../driver-api/cxl/platform/acpi/srat.rst | 38 ++++++++++ 7 files changed, 243 insertions(+) create mode 100644 Documentation/driver-api/cxl/platform/acpi.rst create mode 100644 Documentation/driver-api/cxl/platform/acpi/cedt.rst create mode 100644 Documentation/driver-api/cxl/platform/acpi/dsdt.rst create mode 100644 Documentation/driver-api/cxl/platform/acpi/hmat.rst create mode 100644 Documentation/driver-api/cxl/platform/acpi/slit.rst create mode 100644 Documentation/driver-api/cxl/platform/acpi/srat.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index b8f82cdf1435..5e5699803d4c 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -27,6 +27,7 @@ that have impacts on each other. The docs here break up = configurations steps. :caption: Platform Configuration =20 platform/bios-and-efi + platform/acpi =20 .. toctree:: :maxdepth: 1 diff --git a/Documentation/driver-api/cxl/platform/acpi.rst b/Documentation= /driver-api/cxl/platform/acpi.rst new file mode 100644 index 000000000000..ee7e6bd4c43d --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi.rst @@ -0,0 +1,76 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +ACPI Tables +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +ACPI is the "Advanced Configuration and Power Interface", which is a stand= ard +that defines how platforms and OS manage power and configure computer hard= ware. +For the purpose of this theory of operation, when referring to "ACPI" we w= ill +usually refer to "ACPI Tables" - which are the way a platform (BIOS/EFI) +communicates static configuration information to the operation system. + +The Following ACPI tables contain *static* configuration and performance d= ata +about CXL devices. + +.. toctree:: + :maxdepth: 1 + + acpi/cedt.rst + acpi/srat.rst + acpi/hmat.rst + acpi/slit.rst + acpi/dsdt.rst + +The SRAT table may also contain generic port/initiator content that is int= ended +to describe the generic port, but not information about the rest of the pa= th to +the endpoint. + +Linux uses these tables to configure kernel resources for statically confi= gured +(by BIOS/EFI) CXL devices, such as: + +- NUMA nodes +- Memory Tiers +- NUMA Abstract Distances +- SystemRAM Memory Regions +- Weighted Interleave Node Weights + +ACPI Debugging +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The :code:`acpidump -b` command dumps the ACPI tables into binary format. + +The :code:`iasl -d` command disassembles the files into human readable for= mat. + +Example :code:`acpidump -b && iasl -d cedt.dat` :: + + [000h 0000 4] Signature : "CEDT" [CXL Early Discovery Table] + +Common Issues +------------- +Most failures described here result in a failure of the driver to surface +memory as a DAX device and/or kmem. + +* CEDT CFMWS targets list UIDs do not match CEDT CHBS UIDs. +* CEDT CFMWS targets list UIDs do not match DSDT CXL Host Bridge UIDs. +* CEDT CFMWS Restriction Bits are not correct. +* CEDT CFMWS Memory regions are poorly aligned. +* CEDT CFMWS Memory regions spans a platform memory hole. +* CEDT CHBS UIDs do not match DSDT CXL Host Bridge UIDs. +* CEDT CHBS Specification version is incorrect. +* SRAT is missing regions described in CEDT CFMWS. + + * Result: failure to create a NUMA node for the region, or + region is placed in wrong node. + +* HMAT is missing data for regions described in CEDT CFMWS. + + * Result: NUMA node being placed in the wrong memory tier. + +* SLIT has bad data. + + * Result: Lots of performance mechanisms in the kernel will be very unha= ppy. + +All of these issues will appear to users as if the driver is failing to +support CXL - when in reality they are all the failure of a platform to +configure the ACPI tables correctly. diff --git a/Documentation/driver-api/cxl/platform/acpi/cedt.rst b/Document= ation/driver-api/cxl/platform/acpi/cedt.rst new file mode 100644 index 000000000000..c8f904603fc6 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi/cedt.rst @@ -0,0 +1,53 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D +CEDT - CXL Early Discovery Table +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D + +The CXL Early Discovery Table is generated by BIOS to describe the CXL mem= ory regions configured at boot by the BIOS. + +CHBS +=3D=3D=3D=3D +The CXL Host Bridge Structure describes CXL host bridges. Other than desc= ribing device register information, it reports the specific host bridge UID= for this host bridge. These host bridge ID's will be referenced in other = tables. + +Example :: + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000007 <- Host bridge _UID + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010370400000 + Register length : 0000000000010000 + +CFMWS +=3D=3D=3D=3D=3D +The CXL Fixed Memory Window structure describes a memory region associated= with one or more CXL host bridges (as described by the CHBS). It addition= ally describes any inter-host-bridge interleave configuration that may have= been programmed by BIOS. + +Example :: + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 000000C050000000 <- Memory Region + Window size : 0000003CA0000000 + Interleave Members (2^n) : 01 <- Interleave configuration + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 <- Host Bridge _UID + Next Target : 00000006 <- Host Bridge _UID + +The restriction field dictates what this SPA range may be used for (memory= type, voltile vs persistent, etc). One or more bits may be set. :: + + Bit[0]: CXL Type 2 Memory + Bit[1]: CXL Type 3 Memory + Bit[2]: Volatile Memory + Bit[3]: Persistent Memory + Bit[4]: Fixed Config (HPA cannot be re-used) + +INTRA-host-bridge interleave (multiple devices on one host bridge) is NOT = reported in this structure, and is solely defined via CXL device decoder pr= ogramming (host bridge and endpoint decoders). diff --git a/Documentation/driver-api/cxl/platform/acpi/dsdt.rst b/Document= ation/driver-api/cxl/platform/acpi/dsdt.rst new file mode 100644 index 000000000000..b4583b01d67d --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi/dsdt.rst @@ -0,0 +1,28 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +DSDT - Differentiated system Description Table +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +This table describes what peripherals a machine has. + +This table's UIDs for CXL devices - specifically host bridges, must be +consistent with the contents of the CEDT, otherwise the CXL driver will +fail to probe correctly. + +Example Compute Express Link Host Bridge :: + + Scope (_SB) + { + Device (S0D0) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */)= // _HID: Hardware ID + Name (_CID, Package (0x02) // _CID: Compatible ID + { + EisaId ("PNP0A08") /* PCI Express Bus */, + EisaId ("PNP0A03") /* PCI Bus */ + }) + ... + Name (_UID, 0x05) // _UID: Unique ID + ... + } diff --git a/Documentation/driver-api/cxl/platform/acpi/hmat.rst b/Document= ation/driver-api/cxl/platform/acpi/hmat.rst new file mode 100644 index 000000000000..a6a6ec9f2a32 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi/hmat.rst @@ -0,0 +1,29 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +HMAT - Heterogeneous Memory Attribute Table +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The Heterogeneous Memory Attributes Table contains information such as cac= he attributes and bandwidth and latency details for memory proximity domain= s. For the purpose of this document, we will only discuss the SSLIB entry. + +SLLBI +=3D=3D=3D=3D=3D +The System Locality Latency and Bandwidth Information records latency and = bandwidth information for proximity domains. + +This table is used by Linux to configure interleave weights and memory tie= rs. + +Example (Heavily truncated for brevity) :: + + Structure Type : 0001 [SLLBI] + Data Type : 00 <- Latency + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Entry : 0080 <- DRAM LTC + Entry : 0100 <- CXL LTC + + Structure Type : 0001 [SLLBI] + Data Type : 03 <- Bandwidth + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Entry : 1200 <- DRAM BW + Entry : 0200 <- CXL BW diff --git a/Documentation/driver-api/cxl/platform/acpi/slit.rst b/Document= ation/driver-api/cxl/platform/acpi/slit.rst new file mode 100644 index 000000000000..8fd61b40a66c --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi/slit.rst @@ -0,0 +1,18 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +SLIT - System Locality Information Table +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The system locality information table provides "abstract distances" betwee= n accessor and memory nodes. Node without initiators (cpus) are infinitely= (FF) distance away from all other nodes. + +The abstract distance described in this table does not describe any real l= atency of bandwidth information. + +Example :: + + Signature : "SLIT" [System Locality Information Table] + Localities : 0000000000000004 + Locality 0 : 10 20 20 30 + Locality 1 : 20 10 30 20 + Locality 2 : FF FF 0A FF + Locality 3 : FF FF FF 0A diff --git a/Documentation/driver-api/cxl/platform/acpi/srat.rst b/Document= ation/driver-api/cxl/platform/acpi/srat.rst new file mode 100644 index 000000000000..552a70969769 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/acpi/srat.rst @@ -0,0 +1,38 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +SRAT - Static Resource Affinity Table +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The System/Static Resource Affinity Table describes resource (CPU, Memory)= affinity to "Proximity Domains". This table is technically optional, but f= or performance information (see "HMAT") to be enumerated by linux it must b= e present. + +There is a careful dance between the CEDT and SRAT tables and how NUMA nod= es are created. If things don't look quite the way you expect - check the = SRAT Memory Affinity entries and CEDT CFMWS to determine what your platform= actually supports in terms of flexible topologies. + +The SRAT may statically assign portions of a CFMWS SPA range to a specific= proximity domains. See linux numa creation for more information about how= this presents in the NUMA topology. + +Proximity Domain +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +A proximity domain is ROUGHLY equivalent to "NUMA Node" - though a 1-to-1 = mapping is not guaranteed. There are scenarios where "Proximity Domain 4" = may map to "NUMA Node 3", for example. (See "NUMA Node Creation") + +Memory Affinity +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Generally speaking, if a host does any amount of CXL fabric (decoder) prog= ramming in BIOS - an SRAT entry for that memory needs to be present. + +Example :: + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000001 <- NUMA Node 1 + Reserved1 : 0000 + Base Address : 000000C050000000 <- Physical Memory Region + Address Length : 0000003CA0000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + +Generic Initiator / Port +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +todo --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31E0B2749CA for ; Wed, 30 Apr 2025 18:11:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036672; cv=none; b=W0LJhD4iDnFGqYoJbIyLZoKDucRbuOq41mwuiW3VoqN5hGLhyxvmG7NQxe/3NWe+n5O1U1j5aStl6b/JcqjpwNIn9Qkya+BnSfZiuolIj+X+Fkq47X3tobD4IA6ijYfsamaFjnPYTdD+mv/NrRTsIiEvNx5YbQ+ecszNICGDzU4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036672; c=relaxed/simple; bh=bMXlR3kt9skr+e9ELWUswYfD9xPZ2tPfw1ObAbC8KUc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QEUieirCa22iWgQxwWJGAYibTnNgozMuWMosh4p3KhwzEBhKw3EvL0bW6u54mDoLJ0fA/NV3EDPRlZ5qhaFam1g/kpApg3qPQOiWbq/6i+KylZTQEt8AF3H6j1KJzAvg9s3rG87CuAfEKhFbBBr/KCTq+rVihVnM74Jqn4fzUto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=Uyv7PWya; arc=none smtp.client-ip=209.85.160.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="Uyv7PWya" Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-47664364628so2283191cf.1 for ; Wed, 30 Apr 2025 11:11:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036668; x=1746641468; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QuUPWWLzKmYqWhInWYNc7CiuufhIISijzghWvAw6RtM=; b=Uyv7PWyaAi78kHmlSyvADm5QjadUKCiAVBUhPg8dkBkzJ29ungrtK7wkLzXIWYdcul 7fKzQKQcAgMOaaUof+VHjASoT6WUl0iUmOk0Ej2Ti6ex+E8rQ1oO71mxeXZlNIh6+cHT pvI8scHPZEo4FnTox9XW7CaMBz32ujpMViUes9AWsylOHjsrQBTJkgDjCQr5IL5er1V7 Qovifsh2LVgfvofhj9vIJNJQGaGrfVVKTcrP2GSlaNLZee1PiMoHSfsqv/8V35uolSJ1 +aGkggL62IjOWxjmr0w1UAng+OGpIdoYSvKrsO/hee2FB3+AbNF9RXOw0XFVRr86LCif vpDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036668; x=1746641468; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QuUPWWLzKmYqWhInWYNc7CiuufhIISijzghWvAw6RtM=; b=PlA5NehNLP87u/eEl0+6NPoY10WL0ZM1Pihq2CRHHcMGEUAb05RZJft0S6ZEL9Ywna seDtwzsW8seGWJ6MpZAa/OoZTwjqBESX0q47M2sF0zur+a7uWw+d42RYrq6wCjixeU6R IuYk3szhQHOk2+HYHQ7xXloOXfT0RoMJJ9y3gTOTLyzwc7+UbeSJ0Bs7fjF1VUCGC0pn kn++wjnYAsOCbKE90AP5tRIt59FC9CqosHbayxqzaPZw07oBbif4lilymbsNgmgaxb3+ +yETONot5proO6Yi3wknrcdewvCkScm50mf3UvVHIp4YI1ZcJVcPy5qdNP3Py3Hh+kWk uYyQ== X-Forwarded-Encrypted: i=1; AJvYcCU/uxKscMK/AOWMldU8LQXQ6JfU7p8fiCKsPIpDFuIDaN1mIHgeaH82meQOl+RRAuB/Kz+u32udOnGnSXk=@vger.kernel.org X-Gm-Message-State: AOJu0YxA6YUZtKtpa9GuFZhJZ2zk1aqHwQfJmFsR3eBfLQP1lePdsBc+ 3Ala8ZJdTF3ZoQT5cga21uycbj0SoCMMiFQo6TjUMs+YAcNNQZWpiLAND7Ezv+4= X-Gm-Gg: ASbGncuW+k79UP3FOrHVgbvOa+Ivd9ciLpeQ99TTnjC/aJ9mBZC42DsJHQU5+5kImSn kxhGRO0AwMurj2l3hQVhXnH0r2KoCwsox1aDGWKwbWpulmEDGkMvOfB5Dk4k8+zm8nWeKQ3nlC9 QWmQErjGVSRXnziDGo432dnGAmIiZrWBgtcWaTRNZ8GGYImuLUaIhAYuO86Wp4M7EQkYGpljZGC nUW7q8NhJZb3WD4tS43Bdfre2YsfGz2pcJ/CpDETCRAMj0kuYShEAwmMoyC94cLvGIumYqxm/Cm cuL2daCZ5/AgQ0APH30+QMfOyYME4RHdeRwivpq5NW4otIO9ezi/2DJ4OxrDnA6HqdiVJ0AVpDR H/+Wa3SfTxazRTYmrX1vhGemavU5o12yC975xQ5c= X-Google-Smtp-Source: AGHT+IEaZIWXv6Bvielxfh+difKI8YG5mPgBV8OQe/8x1U7+Bv7PmZBhST/VtgsnHh0AUwspKKCAzw== X-Received: by 2002:ac8:6f0a:0:b0:47c:67b5:73da with SMTP id d75a77b69052e-48ad879b665mr6405971cf.13.1746036667977; Wed, 30 Apr 2025 11:11:07 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:07 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 05/18] cxl: docs/platform/example-configs documentation Date: Wed, 30 Apr 2025 14:10:35 -0400 Message-ID: <20250430181048.1197475-6-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add example ACPI Table configurations for different sample platforms. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 1 + .../cxl/platform/example-configs.rst | 13 + .../example-configurations/flexible.rst | 296 ++++++++++++++++++ .../example-configurations/hb-interleave.rst | 107 +++++++ .../multi-dev-per-hb.rst | 90 ++++++ .../example-configurations/one-dev-per-hb.rst | 136 ++++++++ 6 files changed, 643 insertions(+) create mode 100644 Documentation/driver-api/cxl/platform/example-configs.r= st create mode 100644 Documentation/driver-api/cxl/platform/example-configura= tions/flexible.rst create mode 100644 Documentation/driver-api/cxl/platform/example-configura= tions/hb-interleave.rst create mode 100644 Documentation/driver-api/cxl/platform/example-configura= tions/multi-dev-per-hb.rst create mode 100644 Documentation/driver-api/cxl/platform/example-configura= tions/one-dev-per-hb.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 5e5699803d4c..5fa66e668b49 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -28,6 +28,7 @@ that have impacts on each other. The docs here break up = configurations steps. =20 platform/bios-and-efi platform/acpi + platform/example-configs =20 .. toctree:: :maxdepth: 1 diff --git a/Documentation/driver-api/cxl/platform/example-configs.rst b/Do= cumentation/driver-api/cxl/platform/example-configs.rst new file mode 100644 index 000000000000..90a10d7473c6 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/example-configs.rst @@ -0,0 +1,13 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Example Platform Configurations +############################### + +.. toctree:: + :maxdepth: 1 + :caption: Contents + + example-configurations/one-dev-per-hb.rst + example-configurations/multi-dev-per-hb.rst + example-configurations/hb-interleave.rst + example-configurations/flexible.rst diff --git a/Documentation/driver-api/cxl/platform/example-configurations/f= lexible.rst b/Documentation/driver-api/cxl/platform/example-configurations/= flexible.rst new file mode 100644 index 000000000000..13a97c03e25a --- /dev/null +++ b/Documentation/driver-api/cxl/platform/example-configurations/flexible= .rst @@ -0,0 +1,296 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Flexible Presentation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +This system has a single socket with two CXL host bridges. Each host bridge +has two CXL memory expanders with a 4GB of memory (32GB total). + +On this system, the platform designer wanted to provide the user flexibili= ty +to configure the memory devices in various interleave or NUMA node +configurations. So they provided every combination. + +Things to note: + +* Cross-Bridge interleave is described in one CFMWS that covers all capaci= ty. +* One CFMWS is also described per-host bridge. +* One CFMWS is also described per-device. +* This SRAT describes one-node for each of the above CFMWS. +* The HMAT describes performance for each node in the SRAT. + +CEDT :: + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000007 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010370400000 + Register length : 0000000000010000 + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000006 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010380800000 + Register length : 0000000000010000 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000001000000000 + Window size : 0000000400000000 + Interleave Members (2^n) : 01 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + Second Target : 00000006 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000002000000000 + Window size : 0000000200000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000002200000000 + Window size : 0000000200000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000006 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000003000000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000003100000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000003200000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000006 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000003300000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000006 + +SRAT :: + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000001 + Reserved1 : 0000 + Base Address : 0000001000000000 + Address Length : 0000000400000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000002 + Reserved1 : 0000 + Base Address : 0000002000000000 + Address Length : 0000000200000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000003 + Reserved1 : 0000 + Base Address : 0000002200000000 + Address Length : 0000000200000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000004 + Reserved1 : 0000 + Base Address : 0000003000000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000005 + Reserved1 : 0000 + Base Address : 0000003100000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000006 + Reserved1 : 0000 + Base Address : 0000003200000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000007 + Reserved1 : 0000 + Base Address : 0000003300000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + +HMAT :: + + Structure Type : 0001 [SLLBI] + Data Type : 00 [Latency] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Target Proximity Domain List : 00000003 + Target Proximity Domain List : 00000004 + Target Proximity Domain List : 00000005 + Target Proximity Domain List : 00000006 + Target Proximity Domain List : 00000007 + Entry : 0080 + Entry : 0100 + Entry : 0100 + Entry : 0100 + Entry : 0100 + Entry : 0100 + Entry : 0100 + Entry : 0100 + + Structure Type : 0001 [SLLBI] + Data Type : 03 [Bandwidth] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Target Proximity Domain List : 00000003 + Target Proximity Domain List : 00000004 + Target Proximity Domain List : 00000005 + Target Proximity Domain List : 00000006 + Target Proximity Domain List : 00000007 + Entry : 1200 + Entry : 0400 + Entry : 0200 + Entry : 0200 + Entry : 0100 + Entry : 0100 + Entry : 0100 + Entry : 0100 + +SLIT :: + + Signature : "SLIT" [System Locality Information Table] + Localities : 0000000000000003 + Locality 0 : 10 20 20 20 20 20 20 20 + Locality 1 : FF 0A FF FF FF FF FF FF + Locality 2 : FF FF 0A FF FF FF FF FF + Locality 3 : FF FF FF 0A FF FF FF FF + Locality 4 : FF FF FF FF 0A FF FF FF + Locality 5 : FF FF FF FF FF 0A FF FF + Locality 6 : FF FF FF FF FF FF 0A FF + Locality 7 : FF FF FF FF FF FF FF 0A + +DSDT :: + + Scope (_SB) + { + Device (S0D0) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x07) // _UID: Unique ID + } + ... + Device (S0D5) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x06) // _UID: Unique ID + } + } diff --git a/Documentation/driver-api/cxl/platform/example-configurations/h= b-interleave.rst b/Documentation/driver-api/cxl/platform/example-configurat= ions/hb-interleave.rst new file mode 100644 index 000000000000..fa0885d82deb --- /dev/null +++ b/Documentation/driver-api/cxl/platform/example-configurations/hb-inter= leave.rst @@ -0,0 +1,107 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +Cross-Host-Bridge Interleave +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +This system has a single socket with two CXL host bridges. Each host bridge +has a single CXL memory expander with a 4GB of memory. + +Things to note: + +* Cross-Bridge interleave is described. +* The expanders are described by a single CFMWS. +* This SRAT describes one-node for both host bridges. +* The HMAT describes a single node's performance. + +CEDT :: + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000007 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010370400000 + Register length : 0000000000010000 + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000006 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010380800000 + Register length : 0000000000010000 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000001000000000 + Window size : 0000000200000000 + Interleave Members (2^n) : 01 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + Second Target : 00000006 + +SRAT :: + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000001 + Reserved1 : 0000 + Base Address : 0000001000000000 + Address Length : 0000000200000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + +HMAT :: + + Structure Type : 0001 [SLLBI] + Data Type : 00 [Latency] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Entry : 0080 + Entry : 0100 + + Structure Type : 0001 [SLLBI] + Data Type : 03 [Bandwidth] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Entry : 1200 + Entry : 0400 + +SLIT :: + + Signature : "SLIT" [System Locality Information Table] + Localities : 0000000000000003 + Locality 0 : 10 20 + Locality 1 : FF 0A + +DSDT :: + + Scope (_SB) + { + Device (S0D0) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x07) // _UID: Unique ID + } + ... + Device (S0D5) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x06) // _UID: Unique ID + } + } diff --git a/Documentation/driver-api/cxl/platform/example-configurations/m= ulti-dev-per-hb.rst b/Documentation/driver-api/cxl/platform/example-configu= rations/multi-dev-per-hb.rst new file mode 100644 index 000000000000..6adf7c639490 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/example-configurations/multi-de= v-per-hb.rst @@ -0,0 +1,90 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D +Multiple Devices per Host Bridge +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D + +In this example system we will have a single socket and one CXL host bridg= e. +There are two CXL memory expanders with 4GB attached to the host bridge. + +Things to note: + +* Intra-Bridge interleave is not described here. +* The expanders are described by a single CEDT/CFMWS. +* This CEDT/SRAT describes one node for both devices. +* There is only one proximity domain the HMAT for both devices. + +CEDT :: + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000007 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010370400000 + Register length : 0000000000010000 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000001000000000 + Window size : 0000000200000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + +SRAT :: + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000001 + Reserved1 : 0000 + Base Address : 0000001000000000 + Address Length : 0000000200000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + +HMAT :: + + Structure Type : 0001 [SLLBI] + Data Type : 00 [Latency] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Entry : 0080 + Entry : 0100 + + Structure Type : 0001 [SLLBI] + Data Type : 03 [Bandwidth] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Entry : 1200 + Entry : 0200 + +SLIT :: + + Signature : "SLIT" [System Locality Information Table] + Localities : 0000000000000003 + Locality 0 : 10 20 + Locality 1 : FF 0A + +DSDT :: + + Scope (_SB) + { + Device (S0D0) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x07) // _UID: Unique ID + } + ... + } diff --git a/Documentation/driver-api/cxl/platform/example-configurations/o= ne-dev-per-hb.rst b/Documentation/driver-api/cxl/platform/example-configura= tions/one-dev-per-hb.rst new file mode 100644 index 000000000000..8b732dc8c5b6 --- /dev/null +++ b/Documentation/driver-api/cxl/platform/example-configurations/one-dev-= per-hb.rst @@ -0,0 +1,136 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D +One Device per Host Bridge +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D + +This system has a single socket with two CXL host bridges. Each host bridge +has a single CXL memory expander with a 4GB of memory. + +Things to note: + +* Cross-Bridge interleave is not being used. +* The expanders are in two separate but adjascent memory regions. +* This CEDT/SRAT describes one-node per device +* The expanders have the same performance and will be in the same memory t= ier. + +CEDT :: + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000007 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010370400000 + Register length : 0000000000010000 + + Subtable Type : 00 [CXL Host Bridge Structure] + Reserved : 00 + Length : 0020 + Associated host bridge : 00000006 + Specification version : 00000001 + Reserved : 00000000 + Register base : 0000010380800000 + Register length : 0000000000010000 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000001000000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000007 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Reserved : 00 + Length : 002C + Reserved : 00000000 + Window base address : 0000001100000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + Reserved : 0000 + Granularity : 00000000 + Restrictions : 0006 + QtgId : 0001 + First Target : 00000006 + +SRAT :: + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000001 + Reserved1 : 0000 + Base Address : 0000001000000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + + Subtable Type : 01 [Memory Affinity] + Length : 28 + Proximity Domain : 00000002 + Reserved1 : 0000 + Base Address : 0000001100000000 + Address Length : 0000000100000000 + Reserved2 : 00000000 + Flags (decoded below) : 0000000B + Enabled : 1 + Hot Pluggable : 1 + Non-Volatile : 0 + +HMAT :: + + Structure Type : 0001 [SLLBI] + Data Type : 00 [Latency] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Entry : 0080 + Entry : 0100 + Entry : 0100 + + Structure Type : 0001 [SLLBI] + Data Type : 03 [Bandwidth] + Target Proximity Domain List : 00000000 + Target Proximity Domain List : 00000001 + Target Proximity Domain List : 00000002 + Entry : 1200 + Entry : 0200 + Entry : 0200 + +SLIT :: + + Signature : "SLIT" [System Locality Information Table] + Localities : 0000000000000003 + Locality 0 : 10 20 20 + Locality 1 : FF 0A FF + Locality 2 : FF FF 0A + +DSDT :: + + Scope (_SB) + { + Device (S0D0) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x07) // _UID: Unique ID + } + ... + Device (S0D5) + { + Name (_HID, "ACPI0016" /* Compute Express Link Host Bridge */) //= _HID: Hardware ID + ... + Name (_UID, 0x06) // _UID: Unique ID + } + } --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 738A227602E for ; Wed, 30 Apr 2025 18:11:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036672; cv=none; b=mFgwW8fMSSD4tnmVEiSin7/qZZX1b9KZcyYPxbN4ogButTQbOEo+IvFl+TOivhUxbUTDTTb8Fe70iMFEiFPZwR63zpkclf1Hx+7udrGYlU/erD3xqFue9JH1QTu9IZbpndEtUDMx/aEriEAAwKPSNqIGCb9lI8p3YioI7IVJ0ks= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036672; c=relaxed/simple; bh=ysuz2dbhIUwSeo5TDdeE8cXgy0TSC86HQ4jDAdk8nAw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xo8ma8hb57U6nzYr9oyzlezgZwkm9lYPDrTuGAcE7+dshAbsYM6D3E8uzli89dCuwL2EGxtvPhUbuVS9215lM11H1DCMEWtAausWeG728vYj4elmbRtp9HFBZWyMf1CAtUSZhsAY/pIJiPpQN51HETr56AzKtVNmSx6CbKA6q0E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=jBXdzvrm; arc=none smtp.client-ip=209.85.160.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="jBXdzvrm" Received: by mail-qt1-f180.google.com with SMTP id d75a77b69052e-4769b16d4fbso1276431cf.2 for ; Wed, 30 Apr 2025 11:11:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036669; x=1746641469; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4wCLWtxPZo+z1qXRRB8VDvRxUJkg0/tm2oCyfTLu558=; b=jBXdzvrmhjYS681cdzRfZOZX8erz+5H7fGiauW0hgRhZ0066NIcnLFRpRIA0A4H+93 jJgkH88krYXL00OjBjEgHRC8KpTzQFV0TNYFW7o1LV6jg7ndLFPtWMNd8kBOiHK/zI1Q oboQckc++P355FS5bvh9COUdmsPKV7LQB/3brr98vqpfWZjFqvKV5ds2plR4t3Qa5DwL SM6Jk7AGwkFJr3bANJuJ+DeWwRP5UnSmIM8C4sWwxKpl6+k6f7wPQUGKrIHB+MOV/TSe t0cM5soxi4AjRRpIRowrZPm+QXbaTzr4CQsPEzgM0LZPwmqsSMdefXoHgY24S23NfrxP rPCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036669; x=1746641469; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4wCLWtxPZo+z1qXRRB8VDvRxUJkg0/tm2oCyfTLu558=; b=vDDYp+YvtH7UorRqQuNdo3a3MMO73TKY5m/3oD9cFGWZiYoMLO1wSZGX8f4dXLRo4F A1+YwSpxTaZpdriJ5hGkW2n+8BdhTfYTcAdsQ6KV7rQW6gkpKZoYsGETzh3vyyReprgv rkemy0VaFhrNOKU5B4BCh1C3gghWjIQpmOQUY4HE3b3Nxsz6Aod2RRk16Z1nOj0owjlQ fGpzCQnLuE2AUJxRCHC8thEhAbF+JVKhAjNhsxon/S0TcKZttu9TlUcHyR9fnx4Hco9x E9fX8FyDa8074cirmUpNkziB2GaDS4J/pyLiDvt0U/kL7t4aHgvUHaUfE5QXwA16W442 nA5A== X-Forwarded-Encrypted: i=1; AJvYcCXq6wreaIJWvP7TrFyBD0trP6Jq4gv7SXQAf/VpCBDHgvwjzoxLfEA0IDjbY+VRU6NramA4c+Kn2XBl41s=@vger.kernel.org X-Gm-Message-State: AOJu0Yyj2uzqOsVpinDVKyi0gRqnw1HhVDI8hMwdAE2jiOeM3gRZRwV1 ZAFxqt/ivamTCk4hkGBkIX7zbzRJjcFQSFW9eUh5c+HcrCbbK1hBp5eEU44/XE3Dw/TPuVbDWw4 U X-Gm-Gg: ASbGncvlJiVvQtcPKe6HJmiPlSSa1EORDOeHqIbtzT9BdIRmREVXeRfbLNZR7KH1R2Z tPoFRyx/D1J9LNk34Oqg6ylHsk/aasvsn4zatVOBBxccD7P8LwKSzxiTCbJ6iH/Lb8jjCFnQF0c VNlgJC4kiQRQBxwZ9nm82JZDLWl0EOl9jYUdxoQZy74ZwFcELqU1MncJeQkTfv4dRFVSDxe3H/N Igi9t0VKLbJPbhTvh7UQLD7XFUT5h9tUUnBC++KJzR/Yv8ycw/Vq4ggKw2j6p0vgiJEmnlD6FsA vQ/smZV9wh1J6eyBjrQpZFv3HonrF75BqPPDjsjIegB6tFNef5Ke8lUzS3BPR5mI1RS+eTxNyf2 djklBgI9rh+Filu6T/Bgz4AU1r8DM X-Google-Smtp-Source: AGHT+IFZXlyw0vAGV8CcwH9mLxar0qE/OpSzEUJceMsLJiqWVmOB/kxElfqTZAE+Wa/7CGDcCw6Uwg== X-Received: by 2002:a05:622a:a15:b0:477:6e6a:7cfe with SMTP id d75a77b69052e-489c0be6448mr59562341cf.0.1746036669411; Wed, 30 Apr 2025 11:11:09 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:09 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 06/18] cxl: docs/linux - overview Date: Wed, 30 Apr 2025 14:10:36 -0400 Message-ID: <20250430181048.1197475-7-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add type-3 device configuration overview that explains the probe process for a type-3 device from early-boot through memory-hotplug. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 3 +- .../driver-api/cxl/linux/overview.rst | 103 ++++++++++++++++++ 2 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 Documentation/driver-api/cxl/linux/overview.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 5fa66e668b49..284ac71bf292 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -31,9 +31,10 @@ that have impacts on each other. The docs here break up= configurations steps. platform/example-configs =20 .. toctree:: - :maxdepth: 1 + :maxdepth: 2 :caption: Linux Kernel Configuration =20 + linux/overview linux/access-coordinates =20 =20 diff --git a/Documentation/driver-api/cxl/linux/overview.rst b/Documentatio= n/driver-api/cxl/linux/overview.rst new file mode 100644 index 000000000000..f440d79d77d1 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/overview.rst @@ -0,0 +1,103 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D +Overview +=3D=3D=3D=3D=3D=3D=3D=3D + +This section presents the configuration process of a CXL Type-3 memory dev= ice, +and how it is ultimately exposed to users as either a :code:`DAX` device or +normal memory pages via the kernel's page allocator. + +Portions marked with a bullet are points at which certain kernel objects +are generated. + +1) Early Boot + + a) BIOS, Build, and Boot Parameters + + i) EFI_MEMORY_SP + ii) CONFIG_EFI_SOFT_RESERVE + iii) CONFIG_MHP_DEFAULT_ONLINE_TYPE + iv) nosoftreserve + + b) Memory Map Creation + + i) EFI Memory Map / E820 Consulted for Soft-Reserved + + * CXL Memory is set aside to be handled by the CXL driver + + * Soft-Reserved IO Resource created for CFMWS entry + + c) NUMA Node Creation + + * Nodes created from ACPI CEDT CFMWS and SRAT Proximity domains (PXM) + + d) Memory Tier Creation + + * A default memory_tier is created with all nodes. + + e) Contiguous Memory Allocation + + * Any requested CMA is allocated from Online nodes + + f) Init Finishes, Drivers start probing + +2) ACPI and PCI Drivers + + a) Detects PCI devices is CXL, marking it for probe by CXL driver + +3) CXL Driver Operation + + a) Base device creation + + * root, port, and memdev devices created + * CEDT CFMWS IO Resource creation + + b) Decoder creation + + * root, switch, and endpoint decoders created + + c) Logical device creation + + * memory_region and endpoint devices created + + d) Devices are associated with each other + + * If auto-decoder (BIOS-programmed decoders), driver validates + configurations, builds associations, and locks configs at probe time. + + * If user-configured, validation and associations are built at + decoder-commit time. + + e) Regions surfaced as DAX region + + * dax_region created + + * DAX device created via DAX driver + +4) DAX Driver Operation + + a) DAX driver surfaces DAX region as one of two dax device modes + + * kmem - dax device is converted to hotplug memory blocks + + * DAX kmem IO Resource creation + + * hmem - dax device is left as daxdev to be accessed as a file. + + * If hmem, journey ends here. + + b) DAX kmem surfaces memory region to Memory Hotplug to add to page + allocator as "driver managed memory" + +5) Memory Hotplug + + a) mhp component surfaces a dax device memory region as multiple memory + blocks to the page allocator + + * blocks appear in :code:`/sys/bus/memory/devices` and linked to a NUM= A node + + b) blocks are onlined into the requested zone (NORMAL or MOVABLE) + + * Memory is marked "Driver Managed" to avoid kexec from using it as re= gion + for kernel updates --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9171E264637 for ; Wed, 30 Apr 2025 18:11:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036675; cv=none; b=EmlO8bRWfOQ4Y5j43ykZhLMlXEWzuYddnmAj83NpHOJpp9I8ZTyOb4EusjXPVvjg1VyFELJPI9qgkZ4AHRQ4lX7ndI30bmyklO3WcKiv4v4xUyha4bU2c840w405dQ8xaZpsrdqWk1VQE92HDA6pKJiliX+nHDTNOtG85YRWv2E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036675; c=relaxed/simple; bh=qXpnbfIY4MHNlBZ+ICKzhTwqxRePbWx527+2t0bvHME=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aDO1gVCDa1PaYEJlK5Rto6DvjGpeQJMrvmrl9giRwLTgrO+aVrUUMNvAWochk9JGU8H2fFFcoCw2XIBMVh4u3W32tAMJrbJpiDWsMOxwSZB3zwQlVb7SMg5igh8AXUtCgUPX92caP+p8CcP9aJTz/hsAn9PB+KMvjBz8ulVlnS4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=Muuz36gD; arc=none smtp.client-ip=209.85.160.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="Muuz36gD" Received: by mail-qt1-f182.google.com with SMTP id d75a77b69052e-47688ae873fso2445531cf.0 for ; Wed, 30 Apr 2025 11:11:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036671; x=1746641471; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fJ/G4knbx3sdIg268xzLvXJt5d2OlhH6CUst4oqvV4o=; b=Muuz36gD8NyyDy0ggpRgiX3x/os7K2CDSD476Ivw2eOdppEtrMmpSIKzovoxCXnKD4 6Cv4VscDCVFT8rBKK0TrGRE9xIDzslx3b3BOb6iGJ+DXqQGB+QITVHSfCLClweNRXKU5 IA0mgiy1COLjDwR7Gd6kgqi8dWDry/DxWxJGXPCwnF0mEVXJW3oRNVAgYYaXFKk7j64s PX18XxN79LztwGVcDAdkSY010bwKUTif1WMf3W7nMv7xbtSNXwuYlZ7lXVsuHaAgtLzm yH20kKxDcf9OKXqsBKabwZBz05Dwzg3Ndy1QiPBWjyUq8R6MqhEaJLu3ueqHkK9zXHrB v/Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036671; x=1746641471; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fJ/G4knbx3sdIg268xzLvXJt5d2OlhH6CUst4oqvV4o=; b=tJkCJ120tB2zJUABIHBufjtoVCilv+5f08UD/afsAkDR2NSR6Ch/B6euKt6dNapuX6 5NbzE88SZ6aTf9EwVSH+bSpmqJENy7Br6LfmjP8uPbyd0J8wo4RT8pjy14lusxhCFYdd bqH10R3WWQOHxHGH5vTuIJh6zMOk5CIODMjKfaRPFKae/+PvvX+JoHVePlq+CCBlUiZu WBPSWGk9qK5L8zOPzee9ipnz05z3RcItX5jx9G0i/SaVTo/BVs7ut5J8oTwDQyk+mPTH a4fQLlXt3ah2NC/OFKYfZMU/Lp+sftlvz9riQ2dznw/G/TwlYvmpUel7cd5pgOYvpHqS Jbzg== X-Forwarded-Encrypted: i=1; AJvYcCUk7yEerfb2UwjkM28won1yFhvYot87uBvZ4QFmcESFAiCF8xiYu9ru53pPz/TzhNKyalg+xaXAkaReCBg=@vger.kernel.org X-Gm-Message-State: AOJu0YymyCgPfEe9dUQb0/54JSqrbvMuvdaEoncbGFzgW0F9FDppFO+2 YFKPvDTKbDJJXFiFueZn92UfmAyt7Y8TQyRAuT+HiQPRSiw9vl1SfhPUG4BZrRY= X-Gm-Gg: ASbGncvA8Dj6wrPZRH4a8urYE60jwb7YMn/u+Dbc8ShVj97/H1lQV2BFAnUGl2eDSvJ JwAqvUU2yIEkt6kFr3xChgF7z0f9aqqkUjKRev7xMn7in3Roh4244/z81o/Nrh8KrIKA6zIJxsz SndABwDfmHcGvdzDAKPIpjEFIgb+fXcvP0ir9nWe4vazrdHs/n92wUclWFvlN33wfDOMmK+dqiz sp8bH9CsQviO+zjmucO/0VmSRiHyYA4ixZ2NQ0L9d7lik1UjzGL5923VyNLZaE5U9WhM1dKAcQA W+59MSxWN6hn0b+ql6zOLd99PlfH68JBFfcN+qWIJyPEJwO01dEn8KzlVzglC+FtGAguP12NlLm vmYbrqWYvBleARTyqmWtgHn2J/Ayg X-Google-Smtp-Source: AGHT+IHIAJajNhKleOD9+KQtZV+Yd7Q03vZYD8x7ZENKbLpEcaPEOkfUzLCBnnhcth+doGjRWmib/A== X-Received: by 2002:a05:622a:2b46:b0:476:9296:80a4 with SMTP id d75a77b69052e-48ae7157635mr3010001cf.7.1746036671289; Wed, 30 Apr 2025 11:11:11 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:10 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 07/18] cxl: docs/linux - early boot configuration Date: Wed, 30 Apr 2025 14:10:37 -0400 Message-ID: <20250430181048.1197475-8-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Document __init time configurations that affect CXL driver probe process and memory region configuration. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/linux/early-boot.rst | 130 ++++++++++++++++++ 2 files changed, 131 insertions(+) create mode 100644 Documentation/driver-api/cxl/linux/early-boot.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 284ac71bf292..fea07c4c2f91 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -35,6 +35,7 @@ that have impacts on each other. The docs here break up = configurations steps. :caption: Linux Kernel Configuration =20 linux/overview + linux/early-boot linux/access-coordinates =20 =20 diff --git a/Documentation/driver-api/cxl/linux/early-boot.rst b/Documentat= ion/driver-api/cxl/linux/early-boot.rst new file mode 100644 index 000000000000..275174d5b0bb --- /dev/null +++ b/Documentation/driver-api/cxl/linux/early-boot.rst @@ -0,0 +1,130 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Linux Init (Early Boot) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Linux configuration is split into two major steps: Early-Boot and everythi= ng else. + +During early boot, Linux sets up immutable resources (such as numa nodes),= while +later operations include things like driver probe and memory hotplug. Lin= ux may +read EFI and ACPI information throughout this process to configure logical +representations of the devices. + +During Linux Early Boot stage (functions in the kernel that have the __init +decorator), the system takes the resources created by EFI/BIOS (ACPI table= s) +and turns them into resources that the kernel can consume. + + +BIOS, Build and Boot Options +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D + +There are 4 pre-boot options that need to be considered during kernel build +which dictate how memory will be managed by Linux during early boot. + +* EFI_MEMORY_SP + + * BIOS/EFI Option that dictates whether memory is SystemRAM or + Specific Purpose. Specific Purpose memory will be deferred to + drivers to manage - and not immediately exposed as system RAM. + +* CONFIG_EFI_SOFT_RESERVE + + * Linux Build config option that dictates whether the kernel supports + Specific Purpose memory. + +* CONFIG_MHP_DEFAULT_ONLINE_TYPE + + * Linux Build config that dictates whether and how Specific Purpose memo= ry + converted to a dax device should be managed (left as DAX or onlined as + SystemRAM in ZONE_NORMAL or ZONE_MOVABLE). + +* nosoftreserve + + * Linux kernel boot option that dictates whether Soft Reserve should be + supported. Similar to CONFIG_EFI_SOFT_RESERVE. + +Memory Map Creation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +While the kernel parses the EFI memory map, if :code:`Specific Purpose` me= mory +is supported and detect, it will set this region aside as :code:`SOFT_RESE= RVED`. + +If :code:`EFI_MEMORY_SP=3D0`, :code:`CONFIG_EFI_SOFT_RESERVE=3Dn`, or +:code:`nosoftreserve=3Dy` - Linux will default a CXL device memory region = to +SystemRAM. This will expose the memory to the kernel page allocator in +:code:`ZONE_NORMAL`, making it available for use for most allocations (inc= luding +:code:`struct page` and page tables). + +If `Specific Purpose` is set and supported, :code:`CONFIG_MHP_DEFAULT_ONLI= NE_TYPE_*` +dictates whether the memory is onlined by default (:code:`_OFFLINE` or +:code:`_ONLINE_*`), and if online which zone to online this memory to by d= efault +(:code:`_NORMAL` or :code:`_MOVABLE`). + +If placed in :code:`ZONE_MOVABLE`, the memory will not be available for mo= st +kernel allocations (such as :code:`struct page` or page tables). This may +significant impact performance depending on the memory capacity of the sys= tem. + + +NUMA Node Reservation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Linux refers to the proximity domains (:code:`PXM`) defined in the SRAT to +create NUMA nodes in :code:`acpi_numa_init`. Typically, there is a 1:1 rel= ation +between :code:`PXM` and NUMA node IDs. + +SRAT is the only ACPI defined way of defining Proximity Domains. Linux cho= oses +to, at most, map those 1:1 with NUMA nodes. CEDT adds a description of SPA +ranges which Linux may wish to map to one or more NUMA nodes + +If there are CXL ranges in the CFMWS but not in SRAT, then a fake :code:`P= XM` +is created (as of v6.15). In the future, Linux may reject CFMWS not descri= bed +by SRAT due to the ambiguity of proximity domain association. + +It is important to note that NUMA node creation cannot be done at runtime.= All +possible NUMA nodes are identified at :code:`__init` time, more specifical= ly +during :code:`mm_init`. The CEDT and SRAT must contain sufficient :code:`P= XM` +data for Linux to identify NUMA nodes their associated memory regions. + +The relevant code exists in: :code:`linux/drivers/acpi/numa/srat.c`. + +See the Example Platform Configurations section for more information. + +Memory Tiers Creation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Memory tiers are a collection of NUMA nodes grouped by performance charact= eristics. +During :code:`__init`, Linux initializes the system with a default memory = tier that +contains all nodes marked :code:`N_MEMORY`. + +:code:`memory_tier_init` is called at boot for all nodes with memory onlin= e by +default. :code:`memory_tier_late_init` is called during late-init for node= s setup +during driver configuration. + +Nodes are only marked :code:`N_MEMORY` if they have *online* memory. + +Tier membership can be inspected in :: + + /sys/devices/virtual/memory_tiering/memory_tierN/nodelist + 0-1 + +If nodes are grouped which have clear difference in performance, check the= HMAT +and CDAT information for the CXL nodes. All nodes default to the DRAM tie= r, +unless HMAT/CDAT information is reported to the memory_tier component via +`access_coordinates`. + +Contiguous Memory Allocation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +The contiguous memory allocator (CMA) enables reservation of contiguous me= mory +regions on NUMA nodes during early boot. However, CMA cannot reserve memo= ry +on NUMA nodes that are not online during early boot. :: + + void __init hugetlb_cma_reserve(int order) { + if (!node_online(nid)) + /* do not allow reservations */ + } + +This means if users intend to defer management of CXL memory to the driver= , CMA +cannot be used to guarantee huge page allocations. If enabling CXL memory= as +SystemRAM in `ZONE_NORMAL` during early boot, CMA reservations per-node ca= n be +made with the :code:`cma_pernuma` or :code:`numa_cma` kernel command line +parameters. --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5289827A131 for ; Wed, 30 Apr 2025 18:11:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036676; cv=none; b=Qx00TDUxNgPh4yExV4oemmqUjn2XX2BGPtZUGJcb+oSUuH80d1ZWHcl3nKfSyLVx6RRrKNj+J95CX5j28lGquSv8bnc2NSghImh5VGge2Dwhw/HReAXgsfrpzj6UoycZrKI406Yt2Lw5VK96PWc+qgXKlpf+q3iHocbDzQQ4SqQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036676; c=relaxed/simple; bh=9r447h8rz44cGBJaDtE+vqcTaiwZOyOK8BN4Vu4OPsM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TsyC6oOydFyRke/AvWtKXJS5iSQ1EDNuPHftBmVjQsPChFI0NYKNhIC7fRVt0wVg1hEN8ZutPbEp2Qi1i6hExFhJVwyUQB//+GhLFaft+rJ8RNVeiDxQ2NeGGq5Ff4Rqrf8kC7L4e00QjqdFYzG131aX9HSRCw9VkwYn9WuKZxA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=lo5aJP0+; arc=none smtp.client-ip=209.85.160.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="lo5aJP0+" Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-477296dce8dso1978701cf.3 for ; Wed, 30 Apr 2025 11:11:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036673; x=1746641473; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gqIkk3ZEIa5D4XmMT+8SvcOyQakok3isu39hfFolXxQ=; b=lo5aJP0+GJd/GFV9AsmKlaFEeL1Yz/GLL16ybSwfvRqW/oyxRzbsECMl/0N1XfcjSs H7pJGP4HsvdJ4hU/VN8+tIoewc0/x9+DtqQ56nv6K0sgscDGJoZ6+XfaBig19qKleQj3 WmbCfB/xPaqixWZx+uJoMBu4ybin0hIsaZPFAuQWIzw75hkxgZrvdRu0Ws6jMMBFaIKx cZ1+d+tZw0t8HrE8YQ3hGF/zD5fGwVT/DI+6ke3jnY/2ZJ23QjAb/nFqUYl/EHYxjtQO 7bDxO/ngfWGdpFvqn5+V/RqjaJMxgSMJvSOAkZA+HEEtrgtw8Z942cQLI2V5GXFZfk7G bQiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036673; x=1746641473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gqIkk3ZEIa5D4XmMT+8SvcOyQakok3isu39hfFolXxQ=; b=JG2kwHXlKNB9x0BBq/jk9xbWZ7ULJPZNhBQt2kASHgmhjpQP4erSmYSn9w4oZKkL72 mlSk1igYBj/zrYjpMYmbAp7XN6XJgctAeO3cWNv58u+04D68SA/Sos5uIG984nVsi3y1 XWSz5EIriKNako1antN76jMT4ngTt0oZxaFLP96hmNP/cZE2yF3DHhv1dhGseEp3M7/f e4xblAQfpVd6ddhkNtePKKsnkzaIf8+lNA57R2p3Km0KCVH8jq1GbSMeZ2hCyE1V1iFS xf6rIqIWGN4YgJVWX1LHTappA8l5BCikigWw6UtgbTpI4ghNPBG3lOvUl3MQcUFjvMOz Ol/A== X-Forwarded-Encrypted: i=1; AJvYcCWuEfbOx4JM5LXhRWFEgDcWx5NIaiP2KHnWxbpSn8LFUrikUzaVUkgSp58LZE6v4fBBuE8Mf/AZol2iawE=@vger.kernel.org X-Gm-Message-State: AOJu0YycUcHBwxnIiqc2fBWqXEhjNh64KDL4mo/EzUl/dpziub/0K1qt Y+TwdRywjydTDV0edqoBcSTgg55Y2TfgHttFsQ5gOtuohgDGVlo6Z+vla1y4dYs= X-Gm-Gg: ASbGncsHlReloV6qVcYpPLL7viyWGqA+seEmaaw36GEVaiynOka68TFrulR9BDmmr0e fkda11xHqKtB39eoBbrsn74ePnWyELU6xqli1nvwojl1IfCCyuPZDba8QmQgdQcITP6H1HI95Qq Ksal2hvW2iA5NevvC3Y5xsv4HlY9wju3O0dayoSjmb2EUpVGXFEkzw5OSOFqV4L6oCwx8vv741B IIHx5dzJ92G93zO46cm+R13Pc5/GXZRiaV+etU9KKM87wAQMbRIhZMLR9QlYqDT0iPzqWyr9hP0 3kCrFVPD7VYiMIHWq9qBbJZKfJGqjWfwplLfpnxHdbLe8OHqjxqB9bhgSg0SdcNKJqTznX5je2i 1kyuMuJOGitDYs8RZgZjCO9cNpFrt X-Google-Smtp-Source: AGHT+IGzZ2ej/wBf/aOTcyn1kpKX1LI5Y+4REqPjskzEdeLsvCHcVxnv1GF5erytX1/f34x8By3Vtg== X-Received: by 2002:a05:622a:4c88:b0:476:a3c8:c78d with SMTP id d75a77b69052e-489c416a10dmr64938331cf.29.1746036672950; Wed, 30 Apr 2025 11:11:12 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:12 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 08/18] cxl: docs/linux - add cxl-driver theory of operation Date: Wed, 30 Apr 2025 14:10:38 -0400 Message-ID: <20250430181048.1197475-9-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add docs for the CXL driver that explains the base devices, decoder types, region types, mailbox interfaces, and decoder programming. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/linux/cxl-driver.rst | 522 ++++++++++++++++++ 2 files changed, 523 insertions(+) create mode 100644 Documentation/driver-api/cxl/linux/cxl-driver.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index fea07c4c2f91..aa01b14862e1 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -36,6 +36,7 @@ that have impacts on each other. The docs here break up = configurations steps. =20 linux/overview linux/early-boot + linux/cxl-driver linux/access-coordinates =20 =20 diff --git a/Documentation/driver-api/cxl/linux/cxl-driver.rst b/Documentat= ion/driver-api/cxl/linux/cxl-driver.rst new file mode 100644 index 000000000000..9a9e8ecee578 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/cxl-driver.rst @@ -0,0 +1,522 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +CXL Driver Operation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The devices described in this section are present in :: + + /sys/bus/cxl/devices/ + /dev/cxl/ + +The :code:`cxl-cli` library, maintained as part of the NDTCL project, may +be used to script interactions with these devices. + +Drivers +=3D=3D=3D=3D=3D=3D=3D +The CXL driver is split into a number of drivers. + +* cxl_core - fundamental init interface and core object creation +* cxl_port - initializes root and provides port enumeration interface. +* cxl_acpi - initializes root decoders and interacts with ACPI data. +* cxl_p/mem - initializes memory devices +* cxl_pci - uses cxl_port to enumates the actual fabric hierarchy. + +Driver Devices +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Here is an example from a single-socket system with 4 host bridges. Two ho= st +bridges have a single memory device attached, and the devices are interlea= ved +into a single memory region. The memory region has been converted to dax. = :: + + # ls /sys/bus/cxl/devices/ + dax_region0 decoder3.0 decoder6.0 mem0 port3 + decoder0.0 decoder4.0 decoder6.1 mem1 port4 + decoder1.0 decoder5.0 endpoint5 port1 region0 + decoder2.0 decoder5.1 endpoint6 port2 root0 + +For this section we'll explore the devices present in this configuration, = but +we'll explore more configurations in-depth in example configurations below. + +Base Devices +------------ +Most devices in a CXL fabric are a `port` of some kind (because each +device mostly routes request from one device to the next, rather than +provide a manageable service). + +Root +~~~~ +The `CXL Root` is logical object created by the `cxl_acpi` driver during +:code:`cxl_acpi_probe` - if the :code:`ACPI0017` `Compute Express Link +Root Object` Device Class is found. + +The Root contains links to: + +* `Host Bridge Ports` defined by ACPI CEDT CHBS. + +* `Root Decoders` defined by ACPI CEDT CFMWS. + +:: + + # ls /sys/bus/cxl/devices/root0 + decoder0.0 dport0 dport5 port2 subsystem + decoders_committed dport1 modalias port3 uevent + devtype dport4 port1 port4 uport + + # cat /sys/bus/cxl/devices/root0/devtype + cxl_port + + # cat port1/devtype + cxl_port + + # cat decoder0.0/devtype + cxl_decoder_root + +The root is first `logical port` in the CXL fabric, as presented by the Li= nux +CXL driver. The `CXL root` is a special type of `switch port`, in that it +only has downstream port connections. + +Port +~~~~ +A `port` object is better described as a `switch port`. It may represent a +host bridge to the root or an actual switch port on a switch. A `switch po= rt` +contains one or more decoders used to route memory requests downstream por= ts, +which may be connected to another `switch port` or an `endpoint port`. + +:: + + # ls /sys/bus/cxl/devices/port1 + decoder1.0 dport0 driver parent_dport uport + decoders_committed dport113 endpoint5 subsystem + devtype dport2 modalias uevent + + # cat devtype + cxl_port + + # cat decoder1.0/devtype + cxl_decoder_switch + + # cat endpoint5/devtype + cxl_port + +CXL `Host Bridges` in the fabric are probed during :code:`cxl_acpi_probe` = at +the time the `CXL Root` is probed. The allows for the immediate logical +connection to between the root and host bridge. + +* The root has a downstream port connection to a host bridge + +* The host bridge has an upstream port connection to the root. + +* The host bridge has one or more downstream port connections to switch + or endpoint ports. + +A `Host Bridge` is a special type of CXL `switch port`. It is explicitly +defined in the ACPI specification via `ACPI0016` ID. `Host Bridge` ports +will be probed at `acpi_probe` time, while similar ports on an actual swit= ch +will be probed later. Otherwise, switch and host bridge ports look very +similar - the both contain switch decoders which route accesses between +upstream and downstream ports. + +Endpoint +~~~~~~~~ +An `endpoint` is a terminal port in the fabric. This is a `logical device= `, +and may be one of many `logical devices` presented by a memory device. It +is still considered a type of `port` in the fabric. + +An `endpoint` contains `endpoint decoders` available for use and the +*Coherent Device Attribute Table* (CDAT) used to describe the capabilities +of the device. :: + + # ls /sys/bus/cxl/devices/endpoint5 + CDAT decoders_committed modalias uevent + decoder5.0 devtype parent_dport uport + decoder5.1 driver subsystem + + # cat /sys/bus/cxl/devices/endpoint5/devtype + cxl_port + + # cat /sys/bus/cxl/devices/endpoint5/decoder5.0/devtype + cxl_decoder_endpoint + + +Memory Device (memdev) +~~~~~~~~~~~~~~~~~~~~~~ +A `memdev` is probed and added by the `cxl_pci` driver in :code:`cxl_pci_p= robe` +and is managed by the `cxl_mem` driver. It primarily provides the `IOCTL` +interface to a memory device, via :code:`/dev/cxl/memN`, and exposes vario= us +device configuration data. :: + + # ls /sys/bus/cxl/devices/mem0 + dev firmware_version payload_max security uevent + driver label_storage_size pmem serial + firmware numa_node ram subsystem + + +Decoders +-------- +A `Decoder` is short for a CXL Host-Managed Device Memory (HDM) Decoder. I= t is +a device that routes accesses through the CXL fabric to an endpoint, and at +the endpoint translates a `Host Physical` to `Device Physical` Addressing. + +The CXL 3.1 specification heavily implies that only endpoint decoders shou= ld +engage in translation of `Host Physical Address` to `Device Physical Addre= ss`. +:: + + 8.2.4.20 CXL HDM Decoder Capability Structure + + IMPLEMENTATION NOTE + CXL Host Bridge and Upstream Switch Port Decode Flow + + IMPLEMENTATION NOTE + Device Decode Logic + +These notes imply that there are two logical groups of decoders. + +* Routing Decoder - a decoder which routes accesses but does not translate + addresses from HPA to DPA. + +* Translating Decoder - a decoder which translates accesses from HPA to DPA + for an endpoint to service. + +The CXL drivers distinguish 3 decoder types: root, switch, and endpoint. O= nly +endpoint decoders are Translating Decoders, all others are Routing Decoder= s. + +.. note:: PLATFORM VENDORS BE AWARE + + Linux makes a strong assumption that endpoint decoders are the only dec= oder + in the fabric that actively translates HPA to DPA. Linux assumes routi= ng + decoders pass the HPA unchanged to the next decoder in the fabric. + + It is therefore assumed that any given decoder in the fabric will have = an + address range that is a subset of its upstream port decoder. Any deviat= ion + from this scheme undefined per the specification. Linux prioritizes + spec-defined / architectural behavior. + +Decoders may have one or more `Downstream Targets` if configured to interl= eave +memory accesses. This will be presented in sysfs via the :code:`target_li= st` +parameter. + +Root Decoder +~~~~~~~~~~~~ +A `Root Decoder` is logical construct of the physical address and interlea= ve +configurations present in the ACPI CEDT CFMWS. Linux presents this inform= ation +as a decoder present in the `CXL Root`. We consider this a `Root Decoder`, +though technically it exists on the boundary of the CXL specification and +platform-specific CXL root implementations. + +Linux considers these logical decoders a type of `Routing Decoder`, and is= the +first decoder in the CXL fabric to receive a memory access from the platfo= rm's +memory controllers. + +`Root Decoders` are created during :code:`cxl_acpi_probe`. One root decod= er +is created per CFMWS entry in the ACPI CEDT. + +The :code:`target_list` parameter is filled by the CFMWS target fields. Ta= rgets +of a root decoder are `Host Bridges`, which means interleave done at the r= oot +decoder level is an `Inter-Host-Bridge Interleave`. + +Only root decoders are capable of `Inter-Host-Bridge Interleave`. + +Such interleaves must be configured by the platform and described in the A= CPI +CEDT CFMWS, as the target CXL host bridge UIDs in the CFMWS must match the= CXL +host bridge UIDs in the ACPI CEDT CHBS and ACPI DSDT. + +Interleave settings in a rootdecoder describe how to interleave accesses a= mong +the *immediate downstream targets*, not the entire interleave set. + +The memory range described in the root decoder is used to + +1) Create a memory region (:code:`region0` in this example), and + +2) Associate the region with an IO Memory Resource (:code:`kernel/resource= .c`) + +:: + + # ls /sys/bus/cxl/devices/decoder0.0/ + cap_pmem devtype region0 + cap_ram interleave_granularity size + cap_type2 interleave_ways start + cap_type3 locked subsystem + create_ram_region modalias target_list + delete_region qos_class uevent + + # cat /sys/bus/cxl/devices/decoder0.0/region0/resource + 0xc050000000 + +The IO Memory Resource is created during early boot when the CFMWS region = is +identified in the EFI Memory Map or E820 table (on x86). + +Root decoders are defined as a separate devtype, but are also a type +of `Switch Decoder` due to having downstream targets. :: + + # cat /sys/bus/cxl/devices/decoder0.0/devtype + cxl_decoder_root + +Switch Decoder +~~~~~~~~~~~~~~ +Any non-root, translating decoder is considered a `Switch Decoder`, and wi= ll +present with the type :code:`cxl_decoder_switch`. Both `Host Bridge` and `= CXL +Switch` (device) decoders are of type :code:`cxl_decoder_switch`. :: + + # ls /sys/bus/cxl/devices/decoder1.0/ + devtype locked size target_list + interleave_granularity modalias start target_type + interleave_ways region subsystem uevent + + # cat /sys/bus/cxl/devices/decoder1.0/devtype + cxl_decoder_switch + + # cat /sys/bus/cxl/devices/decoder1.0/region + region0 + +A `Switch Decoder` has associations between a region defined by a root +decoder and downstream target ports. Interleaving done within a switch de= coder +is a multi-downstream-port interleave (or `Intra-Host-Bridge Interleave` f= or +host bridges). + +Interleave settings in a switch decoder describe how to interleave accesses +among the *immediate downstream targets*, not the entire interleave set. + +Switch decoders are created during :code:`cxl_switch_port_probe` in the +:code:`cxl_port` driver, and is created based on a PCI device's DVSEC +registers. + +Switch decoder programming is validated during probe if the platform progr= ams +them during boot (See `Auto Decoders` below), or on commit if programmed at +runtime (See `Runtime Programming` below). + + +Endpoint Decoder +~~~~~~~~~~~~~~~~ +Any decoder attached to a *terminal* point in the CXL fabric (`An Endpoint= `) is +considered an `Endpoint Decoder`. Endpoint decoders are of type +:code:`cxl_decoder_endpoint`. :: + + # ls /sys/bus/cxl/devices/decoder5.0 + devtype locked start + dpa_resource modalias subsystem + dpa_size mode target_type + interleave_granularity region uevent + interleave_ways size + + # cat /sys/bus/cxl/devices/decoder5.0/devtype + cxl_decoder_endpoint + + # cat /sys/bus/cxl/devices/decoder5.0/region + region0 + +An `Endpoint Decoder` has an association with a region defined by a root +decoder and describes the device-local resource associated with this regio= n. + +Unlike root and switch decoders, endpoint decoders translate `Host Physica= l` to +`Device Physical` address ranges. The interleave settings on an endpoint +therefore describe the entire *interleave set*. + +`Device Physical Address` regions must be committed in-order. For example,= the +DPA region starting at 0x80000000 cannot be committed before the DPA region +starting at 0x0. + +As of Linux v6.15, Linux does not support *imbalanced* interleave setups, = all +endpoints in an interleave set are expected to have the same interleave +settings (granularity and ways must be the same). + +Endpoint decoders are created during :code:`cxl_endpoint_port_probe` in the +:code:`cxl_port` driver, and is created based on a PCI device's DVSEC regi= sters. + +Regions +------- + +Memory Region +~~~~~~~~~~~~~ +A `Memory Region` is a logical construct that connects a set of CXL ports = in +the fabric to an IO Memory Resource. It is ultimately used to expose the = memory +on these devices to the DAX subsystem via a `DAX Region`. + +An example RAM region: :: + + # ls /sys/bus/cxl/devices/region0/ + access0 devtype modalias subsystem uuid + access1 driver mode target0 + commit interleave_granularity resource target1 + dax_region0 interleave_ways size uevent + +A memory region can be constructed during endpoint probe, if decoders were +programmed by BIOS/EFI (see `Auto Decoders`), or by creating a region manu= ally +via a `Root Decoder`'s :code:`create_ram_region` or :code:`create_pmem_reg= ion` +interfaces. + +The interleave settings in a `Memory Region` describe the configuration of= the +`Interleave Set` - and are what can be expected to be seen in the endpoint +interleave settings. + + +DAX Region +~~~~~~~~~~ +A `DAX Region` is used to convert a CXL `Memory Region` to a DAX device. A +DAX device may then be accessed directly via a file descriptor interface, = or +converted to System RAM via the DAX kmem driver. See the DAX driver secti= on +for more details. :: + + # ls /sys/bus/cxl/devices/dax_region0/ + dax0.0 devtype modalias uevent + dax_region driver subsystem + + +Mailbox Interfaces +------------------ +A mailbox command interface for each device is exposed in :: + + /dev/cxl/mem0 + /dev/cxl/mem1 + +These mailboxes may receive any specification-defined command. Raw commands +(custom commands) can only be sent to these interfaces if the build config +:code:`CXL_MEM_RAW_COMMANDS` is set. This is considered a debug and/or +development interface, not an officially supported mechanism for creation +of vendor-specific commands (see the `fwctl` subsystem for that). + +Decoder Programming +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Runtime Programming +------------------- +During probe, the only decoders *required* to be programmed are `Root Deco= ders`. +In reality, `Root Decoders` are a logical construct to describe the memory +region and interleave configuration at the host bridge level - as described +in the ACPI CEDT CFMWS. + +All other `Switch` and `Endpoint` decoders may be programmed by the user +at runtime - if the platform supports such configurations. + +This interaction is what creates a `Software Defined Memory` environment. + +See the :code:`cxl-cli` documentation for more information about how to +configure CXL decoders at runtime. + +Auto Decoders +------------- +Auto Decoders are decoders programmed by BIOS/EFI at boot time, and are +almost always locked (cannot be changed). This is done by a platform +which may have a static configuration - or certain quirks which may prevent +dynamic runtime changes to the decoders (such as requiring additional +controller programming within the CPU complex outside the scope of CXL). + +Auto Decoders are probed automatically as long as the devices and memory +regions they are associated with probe without issue. When probing Auto +Decoders, the driver's primary responsibility is to ensure the fabric is +sane - as-if validating runtime programmed regions and decoders. + +If Linux cannot validate auto-decoder configuration, the memory will not +be surfaced as a DAX device - and therefore not be exposed to the page +allocator - effectively stranding it. + +Interleave +---------- + +The Linux CXL driver supports `Cross-Link First` interleave. This dictates +how interleave is programmed at each decoder step, as the driver validates +the relationships between a decoder and it's parent. + +For example, in a `Cross-Link First` interleave setup with 16 endpoints +attached to 4 host bridges, linux expects the following ways/granularity +across the root, host bridge, and endpoints respectively. :: + + ways granularity + root 4 256 + host bridge 4 1024 + endpoint 16 256 + +At the root, every a given access will be routed to the +:code:`((HPA / 256) % 4)th` target host bridge. Within a host bridge, every +:code:`((HPA / 1024) % 4)th` target endpoint. Each endpoint will translate +the access based on the entire 16 device interleave set. + +Unbalanced interleave sets are not supported - decoders at a similar point +in the hierarchy (e.g. all host bridge decoders) must have the same ways a= nd +granularity configuration. + +At Root +~~~~~~~ +Root decoder interleave is defined by the ACPI CEDT CFMWS. The CEDT +may actually define multiple CFMWS configurations to describe the same +physical capacity - with the intent to allow users to decide at runtime +whether to online memory as interleaved or non-interleaved. :: + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Window base address : 0000000100000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + First Target : 00000007 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Window base address : 0000000200000000 + Window size : 0000000100000000 + Interleave Members (2^n) : 00 + Interleave Arithmetic : 00 + First Target : 00000006 + + Subtable Type : 01 [CXL Fixed Memory Window Structure] + Window base address : 0000000300000000 + Window size : 0000000200000000 + Interleave Members (2^n) : 01 + Interleave Arithmetic : 00 + First Target : 00000007 + Next Target : 00000006 + +In this example, the CFMWS defines two discrete non-interleaved 4GB regions +for each host bridge, and one interleaved 8GB region that targets both. Th= is +would result in 3 root decoders presenting in the root. :: + + # ls /sys/bus/cxl/devices/root0 + decoder0.0 decoder0.1 decoder0.2 + + # cat /sys/bus/cxl/devices/decoder0.0/target_list start size + 7 + 0x100000000 + 0x100000000 + + # cat /sys/bus/cxl/devices/decoder0.1/target_list start size + 6 + 0x200000000 + 0x100000000 + + # cat /sys/bus/cxl/devices/decoder0.2/target_list start size + 7,6 + 0x300000000 + 0x200000000 + +These decoders are not runtime programmable. They are used to generate a +`Memory Region` to bring this memory online with runtime programmed settin= gs +at the `Switch` and `Endpoint` decoders. + +At Host Bridge or Switch +~~~~~~~~~~~~~~~~~~~~~~~~ +`Host Bridge` and `Switch` decoders are programmable via the following fie= lds: + +- :code:`start` - the HPA region associated with the memory region +- :code:`size` - the size of the region +- :code:`target_list` - the list of downstream ports +- :code:`interleave_ways` - the number downstream ports to interleave acro= ss +- :code:`interleave_granularity` - the granularity to interleave at. + +Linux expects the :code:`interleave_granularity` of switch decoders to be +derived from their upstream port connections. In `Cross-Link First` interl= eave +configurations, the :code:`interleave_granularity` of a decoder is equal to +:code:`parent_interleave_granularity * parent_interleave_ways`. + +At Endpoint +~~~~~~~~~~~ +`Endpoint Decoders` are programmed similar to Host Bridge and Switch decod= ers, +with the exception that the ways and granularity are defined by the interl= eave +set (e.g. the interleave settings defined by the associated `Memory Region= `). + +- :code:`start` - the HPA region associated with the memory region +- :code:`size` - the size of the region +- :code:`interleave_ways` - the number endpoints in the interleave set +- :code:`interleave_granularity` - the granularity to interleave at. + +These settings are used by endpoint decoders to *Translate* memory requests +from HPA to DPA. This is why they must be aware of the entire interleave = set. + +Linux does not support unbalanced interleave configurations. As a result,= all +endpoints in an interleave set must have the same ways and granularity. --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B99C527BF8A for ; Wed, 30 Apr 2025 18:11:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036681; cv=none; b=ODlQQImI6qUhxYSdPuvoa2LklOsYNXB9J27UkLz+fPHSsuXuVEB4LCRzjC4JvS+LRTXJcXPv5fn2JbjW5InbO7wDZcAlpxpW5q5MHl72Uv0dpeGjw+bTLgbXqXZjwLr6vkTjpLKkWtSLEVsVuA04SYZ1oeKpHnlMxFFHhNu4oxE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036681; c=relaxed/simple; bh=dkaGPWOKn67y1LdrfoERITXLH++9yici8wbfaUCIToA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=X9chYHupZHPkNqJGSPd7nAglmUfOrAdCK6SojEJcUMYXSg6MyDPVMq8Q7qjO4EI3cPbbubP14374BMgS9gFAiX83yy1cyWYKavIaWovEPe8966qWBadUbVd6UPtTCddQU3WcBLwnCgk3siiQAh8MagAjigai7y+mSNLBbEt4KiE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=Ns/4ms2s; arc=none smtp.client-ip=209.85.160.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="Ns/4ms2s" Received: by mail-qt1-f174.google.com with SMTP id d75a77b69052e-4772f48f516so14873111cf.1 for ; Wed, 30 Apr 2025 11:11:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036675; x=1746641475; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6FdhR4gF7aeY7QZzxz+jvh/0OURCPCws/RHYlQ0UyrA=; b=Ns/4ms2sVjqhKHL5Kz78rMoGHmPHbtsxfeOH9bgZi1T+rhxnsdVMdf4KST4B7hl3j+ 1W+nYVw5HZVZtGxNs7oXNNZuBKncR/8GTFtMgszmmAlXJ4oD+foEit9VJD1pE1n/TdDn NG8YGn4H2P+/GjJYEUnlYu6QZpb9sIpozIy8MDuCC1hj94D9+lrv1mtpvrBzvHa9ULF/ hXze4jzXQAgOydfulsJiYgW2OLkzLCZA+EMkK57a1jlueNy+4KW7qrS4v273irnNSbCT CXG/P1y8CmiUNruabKLTyZZ/coOvxBeTfhhyo79Wod3mbCA3S/OLAGTzj+GDyThbcbnl tQKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036675; x=1746641475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6FdhR4gF7aeY7QZzxz+jvh/0OURCPCws/RHYlQ0UyrA=; b=I7rctVrZHOCIB+hHMIoKELYOJ2XK11U8vu/Jo9eR/Ij2rE8x9n6aoTEuWleFHaAdPr t+UvDZ4cNmTpCKjI8gOy/WSelF4TnR33/49xngNx/YlNTBv1l28DY6JL0inO8AFPVUQz Nm9QQ+n8aN8j6KQ0wtZsDEeXwjFUFutKMdy3J1BXmXobXzkKiuf6RaP8uupe/wSuiZ8s WgOR0CAXsb/HtDwII5rgoSQShlfBFy3GqfaMGRZd00PmEfPO1t2YOCsiftCJ6Yt2D6wf N0LoKhZmLiSoQnfqrP6ioRuJXz8cS7D/+4HzD8S1yvlUSd0eyGH8yvO4NNZGkrEJOvMY bLKQ== X-Forwarded-Encrypted: i=1; AJvYcCVgBpQh8q2yq7JvQRzSF9HfmsEn1s/2PvTkU+MGh9Yvr4xJZ9DaQ0lH/MfPiGUASxfxSAZky9v9IqSqZ70=@vger.kernel.org X-Gm-Message-State: AOJu0YzNnknQ8DUoqs/KmVjY1CWpbigg0wB83m6LUa1ceYjjLMxifSKJ gh7miVGmSmtL/KoTbtXTz6E83sGgfLngjk3A5CPKEAqhbKf39tVoA9/Ilc5TZRs= X-Gm-Gg: ASbGncsB79oyJ+NMPzbMF1eZ/MxA6IHpA7oZLK+F8mIo3VXiFIWaeONpU5GuvUgGY74 jXvKNEcr4KUPoh89Sp/BVSO2/ouRxWKKPKEuaIpvZ0F0Q5w+OLWAZP2zT3mMsrafDsr6E3qeIcb IfdXT4iiVV0m45BzbrXsscuroFTbKAKeBiPLLJc5WQ0ydzh4yWuCtum865oj1yGCXESwG2s3RCl 8hVfrjxresd2Di6D9BaDQWsYZFbRILJ04ypKntlfvv6w+fr/8/iKFo4V2k4AsaMpjmJaDr/W+kj GdMPP3AFzKuDg7+f+D5ekbIx37VDCkWO4GP1e/1C5GmjHDaXUfM/d+MJwt4HCRdnUdJzNUP8H4P 0Le2BRezOERP+pXxpYhKK0m206md1l7tjodw3WhQ= X-Google-Smtp-Source: AGHT+IFixAPXAPJqTEKo83VIAU12buwU0yRkYYrsrJANnyE7K4k8ICcSRtspGho4TLg0PBwcxs2iIg== X-Received: by 2002:a05:622a:4244:b0:477:1dd7:af94 with SMTP id d75a77b69052e-48ad8097752mr8311031cf.2.1746036674516; Wed, 30 Apr 2025 11:11:14 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:14 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 09/18] cxl: docs/linux/cxl-driver - add example configurations Date: Wed, 30 Apr 2025 14:10:39 -0400 Message-ID: <20250430181048.1197475-10-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add 4 example configurations: - single device - cross-host-bridge interleave - intra-host-bridge-interleave - multi-level interleave Signed-off-by: Gregory Price --- .../driver-api/cxl/linux/cxl-driver.rst | 10 + .../example-configurations/hb-interleave.rst | 314 ++++++++++++++ .../intra-hb-interleave.rst | 291 +++++++++++++ .../multi-interleave.rst | 401 ++++++++++++++++++ .../example-configurations/single-device.rst | 246 +++++++++++ 5 files changed, 1262 insertions(+) create mode 100644 Documentation/driver-api/cxl/linux/example-configuratio= ns/hb-interleave.rst create mode 100644 Documentation/driver-api/cxl/linux/example-configuratio= ns/intra-hb-interleave.rst create mode 100644 Documentation/driver-api/cxl/linux/example-configuratio= ns/multi-interleave.rst create mode 100644 Documentation/driver-api/cxl/linux/example-configuratio= ns/single-device.rst diff --git a/Documentation/driver-api/cxl/linux/cxl-driver.rst b/Documentat= ion/driver-api/cxl/linux/cxl-driver.rst index 9a9e8ecee578..486baf8551aa 100644 --- a/Documentation/driver-api/cxl/linux/cxl-driver.rst +++ b/Documentation/driver-api/cxl/linux/cxl-driver.rst @@ -520,3 +520,13 @@ from HPA to DPA. This is why they must be aware of th= e entire interleave set. =20 Linux does not support unbalanced interleave configurations. As a result,= all endpoints in an interleave set must have the same ways and granularity. + +Example Configurations +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +.. toctree:: + :maxdepth: 1 + + example-configurations/single-device.rst + example-configurations/hb-interleave.rst + example-configurations/intra-hb-interleave.rst + example-configurations/multi-interleave.rst diff --git a/Documentation/driver-api/cxl/linux/example-configurations/hb-i= nterleave.rst b/Documentation/driver-api/cxl/linux/example-configurations/h= b-interleave.rst new file mode 100644 index 000000000000..f071490763a2 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/example-configurations/hb-interlea= ve.rst @@ -0,0 +1,314 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +Inter-Host-Bridge Interleave +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +This cxl-cli configuration dump shows the following host configuration: + +* A single socket system with one CXL root +* CXL Root has Four (4) CXL Host Bridges +* Two CXL Host Bridges have a single CXL Memory Expander Attached +* The CXL root is configured to interleave across the two host bridges. + +This output is generated by :code:`cxl list -v` and describes the relation= ships +between objects exposed in :code:`/sys/bus/cxl/devices/`. + +:: + + [ + { + "bus":"root0", + "provider":"ACPI.CXL", + "nr_dports":4, + "dports":[ + { + "dport":"pci0000:00", + "alias":"ACPI0016:01", + "id":0 + }, + { + "dport":"pci0000:a8", + "alias":"ACPI0016:02", + "id":4 + }, + { + "dport":"pci0000:2a", + "alias":"ACPI0016:03", + "id":1 + }, + { + "dport":"pci0000:d2", + "alias":"ACPI0016:00", + "id":5 + } + ], + +This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to = CXL +Host Bridges. The `Root` can be considered the singular upstream port att= ached +to the platform's memory controller - which routes memory requests to it. + +The `ports:root0` section lays out how each of these downstream ports are +configured. If a port is not configured (id's 0 and 1), they are omitted. + +:: + + "ports:root0":[ + { + "port":"port1", + "host":"pci0000:d2", + "depth":1, + "nr_dports":3, + "dports":[ + { + "dport":"0000:d2:01.1", + "alias":"device:02", + "id":0 + }, + { + "dport":"0000:d2:01.3", + "alias":"device:05", + "id":2 + }, + { + "dport":"0000:d2:07.1", + "alias":"device:0d", + "id":113 + } + ], + +This chunk shows the available downstream ports associated with the CXL Ho= st +Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstr= eam +ports: :code:`dport1`, :code:`dport2`, and :code:`dport113`.. + +:: + + "endpoints:port1":[ + { + "endpoint":"endpoint5", + "host":"mem0", + "parent_dport":"0000:d2:01.1", + "depth":2, + "memdev":{ + "memdev":"mem0", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:d3:00.0" + }, + "decoders:endpoint5":[ + { + "decoder":"decoder5.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + +This chunk shows the endpoints attached to the host bridge :code:`port1`. + +:code:`endpoint5` contains a single configured decoder :code:`decoder5.0` +which has the same interleave configuration as :code:`region0` (shown late= r). + +Next we have the decodesr belonging to the host bridge: + +:: + + "decoders:port1":[ + { + "decoder":"decoder1.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":1, + "region":"region0", + "nr_targets":1, + "targets":[ + { + "target":"0000:d2:01.1", + "alias":"device:02", + "position":0, + "id":0 + } + ] + } + ] + }, + +Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`), whose= only +target is :code:`dport1` - which is attached to :code:`endpoint5`. + +The following chunk shows a similar configuration for Host Bridge :code:`p= ort3`, +the second host bridge with a memory device attached. + +:: + + { + "port":"port3", + "host":"pci0000:a8", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:a8:01.1", + "alias":"device:c3", + "id":0 + } + ], + "endpoints:port3":[ + { + "endpoint":"endpoint6", + "host":"mem1", + "parent_dport":"0000:a8:01.1", + "depth":2, + "memdev":{ + "memdev":"mem1", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:a9:00.0" + }, + "decoders:endpoint6":[ + { + "decoder":"decoder6.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + "decoders:port3":[ + { + "decoder":"decoder3.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":1, + "region":"region0", + "nr_targets":1, + "targets":[ + { + "target":"0000:a8:01.1", + "alias":"device:c3", + "position":0, + "id":0 + } + ] + } + ] + }, + + +The next chunk shows the two CXL host bridges without attached endpoints. + +:: + + { + "port":"port2", + "host":"pci0000:00", + "depth":1, + "nr_dports":2, + "dports":[ + { + "dport":"0000:00:01.3", + "alias":"device:55", + "id":2 + }, + { + "dport":"0000:00:07.1", + "alias":"device:5d", + "id":113 + } + ] + }, + { + "port":"port4", + "host":"pci0000:2a", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:2a:01.1", + "alias":"device:d0", + "id":0 + } + ] + } + ], + +Next we have the `Root Decoders` belonging to :code:`root0`. This root de= coder +applies the interleave across the downstream ports :code:`port1` and +:code:`port3` - with a granularity of 256 bytes. + +This information is generated by the CXL driver reading the ACPI CEDT CMFW= S. + +:: + + "decoders:root0":[ + { + "decoder":"decoder0.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "max_available_extent":0, + "volatile_capable":true, + "nr_targets":2, + "targets":[ + { + "target":"pci0000:a8", + "alias":"ACPI0016:02", + "position":1, + "id":4 + }, + { + "target":"pci0000:d2", + "alias":"ACPI0016:00", + "position":0, + "id":5 + } + ], + +Finally we have the `Memory Region` associated with the `Root Decoder` +:code:`decoder0.0`. This region describes the overall interleave configur= ation +of the interleave set. + +:: + + "regions:decoder0.0":[ + { + "region":"region0", + "resource":825975898112, + "size":274877906944, + "type":"ram", + "interleave_ways":2, + "interleave_granularity":256, + "decode_state":"commit", + "mappings":[ + { + "position":1, + "memdev":"mem1", + "decoder":"decoder6.0" + }, + { + "position":0, + "memdev":"mem0", + "decoder":"decoder5.0" + } + ] + } + ] + } + ] + } + ] diff --git a/Documentation/driver-api/cxl/linux/example-configurations/intr= a-hb-interleave.rst b/Documentation/driver-api/cxl/linux/example-configurat= ions/intra-hb-interleave.rst new file mode 100644 index 000000000000..077dfaf8458d --- /dev/null +++ b/Documentation/driver-api/cxl/linux/example-configurations/intra-hb-in= terleave.rst @@ -0,0 +1,291 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +Intra-Host-Bridge Interleave +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D +This cxl-cli configuration dump shows the following host configuration: + +* A single socket system with one CXL root +* CXL Root has Four (4) CXL Host Bridges +* One (1) CXL Host Bridges has two CXL Memory Expanders Attached +* The Host bridge decoder is programmed to interleave across the expanders. + +This output is generated by :code:`cxl list -v` and describes the relation= ships +between objects exposed in :code:`/sys/bus/cxl/devices/`. + +:: + + [ + { + "bus":"root0", + "provider":"ACPI.CXL", + "nr_dports":4, + "dports":[ + { + "dport":"pci0000:00", + "alias":"ACPI0016:01", + "id":0 + }, + { + "dport":"pci0000:a8", + "alias":"ACPI0016:02", + "id":4 + }, + { + "dport":"pci0000:2a", + "alias":"ACPI0016:03", + "id":1 + }, + { + "dport":"pci0000:d2", + "alias":"ACPI0016:00", + "id":5 + } + ], + +This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to = CXL +Host Bridges. The `Root` can be considered the singular upstream port att= ached +to the platform's memory controller - which routes memory requests to it. + +The `ports:root0` section lays out how each of these downstream ports are +configured. If a port is not configured (id's 0 and 1), they are omitted. + +:: + + "ports:root0":[ + { + "port":"port1", + "host":"pci0000:d2", + "depth":1, + "nr_dports":3, + "dports":[ + { + "dport":"0000:d2:01.1", + "alias":"device:02", + "id":0 + }, + { + "dport":"0000:d2:01.3", + "alias":"device:05", + "id":2 + }, + { + "dport":"0000:d2:07.1", + "alias":"device:0d", + "id":113 + } + ], + +This chunk shows the available downstream ports associated with the CXL Ho= st +Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstr= eam +ports: :code:`dport1`, :code:`dport2`, and :code:`dport113`.. + +:: + + "endpoints:port1":[ + { + "endpoint":"endpoint5", + "host":"mem0", + "parent_dport":"0000:d2:01.1", + "depth":2, + "memdev":{ + "memdev":"mem0", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:d3:00.0" + }, + "decoders:endpoint5":[ + { + "decoder":"decoder5.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + }, + { + "endpoint":"endpoint6", + "host":"mem1", + "parent_dport":"0000:d2:01.3, + "depth":2, + "memdev":{ + "memdev":"mem1", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:a9:00.0" + }, + "decoders:endpoint6":[ + { + "decoder":"decoder6.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + +This chunk shows the endpoints attached to the host bridge :code:`port1`. + +:code:`endpoint5` contains a single configured decoder :code:`decoder5.0` +which has the same interleave configuration memory region they belong to +(show later). + +Next we have the decoders belonging to the host bridge: + +:: + + "decoders:port1":[ + { + "decoder":"decoder1.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":2, + "interleave_granularity":256, + "region":"region0", + "nr_targets":2, + "targets":[ + { + "target":"0000:d2:01.1", + "alias":"device:02", + "position":0, + "id":0 + }, + { + "target":"0000:d2:01.3", + "alias":"device:05", + "position":1, + "id":0 + } + ] + } + ] + }, + +Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`) with t= wo +targets: :code:`dport1` and :code:`dport3` - which are attached to +:code:`endpoint5` and :code:`endpoint6` respectively. + +The host bridge decoder interleaves these devices at a 256 byte granularit= y. + +The next chunk shows the three CXL host bridges without attached endpoints. + +:: + + { + "port":"port2", + "host":"pci0000:00", + "depth":1, + "nr_dports":2, + "dports":[ + { + "dport":"0000:00:01.3", + "alias":"device:55", + "id":2 + }, + { + "dport":"0000:00:07.1", + "alias":"device:5d", + "id":113 + } + ] + }, + { + "port":"port3", + "host":"pci0000:a8", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:a8:01.1", + "alias":"device:c3", + "id":0 + } + ], + }, + { + "port":"port4", + "host":"pci0000:2a", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:2a:01.1", + "alias":"device:d0", + "id":0 + } + ] + } + ], + +Next we have the `Root Decoders` belonging to :code:`root0`. This root de= coder +applies the interleave across the downstream ports :code:`port1` and +:code:`port3` - with a granularity of 256 bytes. + +This information is generated by the CXL driver reading the ACPI CEDT CMFW= S. + +:: + + "decoders:root0":[ + { + "decoder":"decoder0.0", + "resource":825975898112, + "size":274877906944, + "interleave_ways":1, + "max_available_extent":0, + "volatile_capable":true, + "nr_targets":2, + "targets":[ + { + "target":"pci0000:a8", + "alias":"ACPI0016:02", + "position":1, + "id":4 + }, + ], + +Finally we have the `Memory Region` associated with the `Root Decoder` +:code:`decoder0.0`. This region describes the overall interleave configur= ation +of the interleave set. + +:: + + "regions:decoder0.0":[ + { + "region":"region0", + "resource":825975898112, + "size":274877906944, + "type":"ram", + "interleave_ways":2, + "interleave_granularity":256, + "decode_state":"commit", + "mappings":[ + { + "position":1, + "memdev":"mem1", + "decoder":"decoder6.0" + }, + { + "position":0, + "memdev":"mem0", + "decoder":"decoder5.0" + } + ] + } + ] + } + ] + } + ] diff --git a/Documentation/driver-api/cxl/linux/example-configurations/mult= i-interleave.rst b/Documentation/driver-api/cxl/linux/example-configuration= s/multi-interleave.rst new file mode 100644 index 000000000000..008f9053c630 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/example-configurations/multi-inter= leave.rst @@ -0,0 +1,401 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Multi-Level Interleave +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +This cxl-cli configuration dump shows the following host configuration: + +* A single socket system with one CXL root +* CXL Root has Four (4) CXL Host Bridges +* Two CXL Host Bridges have a two CXL Memory Expanders Attached each. +* The CXL root is configured to interleave across the two host bridges. +* Each host bridge with expanders interleaves across two endpoints. + +This output is generated by :code:`cxl list -v` and describes the relation= ships +between objects exposed in :code:`/sys/bus/cxl/devices/`. + +:: + + [ + { + "bus":"root0", + "provider":"ACPI.CXL", + "nr_dports":4, + "dports":[ + { + "dport":"pci0000:00", + "alias":"ACPI0016:01", + "id":0 + }, + { + "dport":"pci0000:a8", + "alias":"ACPI0016:02", + "id":4 + }, + { + "dport":"pci0000:2a", + "alias":"ACPI0016:03", + "id":1 + }, + { + "dport":"pci0000:d2", + "alias":"ACPI0016:00", + "id":5 + } + ], + +This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to = CXL +Host Bridges. The `Root` can be considered the singular upstream port att= ached +to the platform's memory controller - which routes memory requests to it. + +The `ports:root0` section lays out how each of these downstream ports are +configured. If a port is not configured (id's 0 and 1), they are omitted. + +:: + + "ports:root0":[ + { + "port":"port1", + "host":"pci0000:d2", + "depth":1, + "nr_dports":3, + "dports":[ + { + "dport":"0000:d2:01.1", + "alias":"device:02", + "id":0 + }, + { + "dport":"0000:d2:01.3", + "alias":"device:05", + "id":2 + }, + { + "dport":"0000:d2:07.1", + "alias":"device:0d", + "id":113 + } + ], + +This chunk shows the available downstream ports associated with the CXL Ho= st +Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstr= eam +ports: :code:`dport0`, :code:`dport2`, and :code:`dport113`. + +:: + + "endpoints:port1":[ + { + "endpoint":"endpoint5", + "host":"mem0", + "parent_dport":"0000:d2:01.1", + "depth":2, + "memdev":{ + "memdev":"mem0", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:d3:00.0" + }, + "decoders:endpoint5":[ + { + "decoder":"decoder5.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":4, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + }, + { + "endpoint":"endpoint6", + "host":"mem1", + "parent_dport":"0000:d2:01.3", + "depth":2, + "memdev":{ + "memdev":"mem1", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:d3:00.0" + }, + "decoders:endpoint6":[ + { + "decoder":"decoder6.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":4, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + +This chunk shows the endpoints attached to the host bridge :code:`port1`. + +:code:`endpoint5` contains a single configured decoder :code:`decoder5.0` +which has the same interleave configuration as :code:`region0` (shown late= r). + +:code:`endpoint6` contains a single configured decoder :code:`decoder5.0` +which has the same interleave configuration as :code:`region0` (shown late= r). + +Next we have the decoders belonging to the host bridge: + +:: + + "decoders:port1":[ + { + "decoder":"decoder1.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":2, + "interleave_granularity":512, + "region":"region0", + "nr_targets":2, + "targets":[ + { + "target":"0000:d2:01.1", + "alias":"device:02", + "position":0, + "id":0 + }, + { + "target":"0000:d2:01.3", + "alias":"device:05", + "position":2, + "id":0 + } + ] + } + ] + }, + +Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`), whose +targets are :code:`dport0` and :code:`dport2` - which are attached to +:code:`endpoint5` and :code:`endpoint6` respectively. + +The following chunk shows a similar configuration for Host Bridge :code:`p= ort3`, +the second host bridge with a memory device attached. + +:: + + { + "port":"port3", + "host":"pci0000:a8", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:a8:01.1", + "alias":"device:c3", + "id":0 + }, + { + "dport":"0000:a8:01.3", + "alias":"device:c5", + "id":0 + } + ], + "endpoints:port3":[ + { + "endpoint":"endpoint7", + "host":"mem2", + "parent_dport":"0000:a8:01.1", + "depth":2, + "memdev":{ + "memdev":"mem2", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:a9:00.0" + }, + "decoders:endpoint7":[ + { + "decoder":"decoder7.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":4, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + }, + { + "endpoint":"endpoint8", + "host":"mem3", + "parent_dport":"0000:a8:01.3", + "depth":2, + "memdev":{ + "memdev":"mem3", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:a9:00.0" + }, + "decoders:endpoint8":[ + { + "decoder":"decoder8.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":4, + "interleave_granularity":256, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + "decoders:port3":[ + { + "decoder":"decoder3.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":2, + "interleave_granularity":512, + "region":"region0", + "nr_targets":1, + "targets":[ + { + "target":"0000:a8:01.1", + "alias":"device:c3", + "position":1, + "id":0 + }, + { + "target":"0000:a8:01.3", + "alias":"device:c5", + "position":3, + "id":0 + } + ] + } + ] + }, + + +The next chunk shows the two CXL host bridges without attached endpoints. + +:: + + { + "port":"port2", + "host":"pci0000:00", + "depth":1, + "nr_dports":2, + "dports":[ + { + "dport":"0000:00:01.3", + "alias":"device:55", + "id":2 + }, + { + "dport":"0000:00:07.1", + "alias":"device:5d", + "id":113 + } + ] + }, + { + "port":"port4", + "host":"pci0000:2a", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:2a:01.1", + "alias":"device:d0", + "id":0 + } + ] + } + ], + +Next we have the `Root Decoders` belonging to :code:`root0`. This root de= coder +applies the interleave across the downstream ports :code:`port1` and +:code:`port3` - with a granularity of 256 bytes. + +This information is generated by the CXL driver reading the ACPI CEDT CMFW= S. + +:: + + "decoders:root0":[ + { + "decoder":"decoder0.0", + "resource":825975898112, + "size":549755813888, + "interleave_ways":2, + "interleave_granularity":256, + "max_available_extent":0, + "volatile_capable":true, + "nr_targets":2, + "targets":[ + { + "target":"pci0000:a8", + "alias":"ACPI0016:02", + "position":1, + "id":4 + }, + { + "target":"pci0000:d2", + "alias":"ACPI0016:00", + "position":0, + "id":5 + } + ], + +Finally we have the `Memory Region` associated with the `Root Decoder` +:code:`decoder0.0`. This region describes the overall interleave configur= ation +of the interleave set. So we see there are a total of :code:`4` interleave +targets across 4 endpoint decoders. + +:: + + "regions:decoder0.0":[ + { + "region":"region0", + "resource":825975898112, + "size":549755813888, + "type":"ram", + "interleave_ways":4, + "interleave_granularity":256, + "decode_state":"commit", + "mappings":[ + { + "position":3, + "memdev":"mem3", + "decoder":"decoder8.0" + }, + { + "position":2, + "memdev":"mem1", + "decoder":"decoder6.0" + } + { + "position":1, + "memdev":"mem2", + "decoder":"decoder7.0" + }, + { + "position":0, + "memdev":"mem0", + "decoder":"decoder5.0" + } + ] + } + ] + } + ] + } + ] diff --git a/Documentation/driver-api/cxl/linux/example-configurations/sing= le-device.rst b/Documentation/driver-api/cxl/linux/example-configurations/s= ingle-device.rst new file mode 100644 index 000000000000..5fd38eb0aaf4 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/example-configurations/single-devi= ce.rst @@ -0,0 +1,246 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Single Device +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +This cxl-cli configuration dump shows the following host configuration: + +* A single socket system with one CXL root +* CXL Root has Four (4) CXL Host Bridges +* One CXL Host Bridges has a single CXL Memory Expander Attached +* No interleave is present. + +This output is generated by :code:`cxl list -v` and describes the relation= ships +between objects exposed in :code:`/sys/bus/cxl/devices/`. + +:: + + [ + { + "bus":"root0", + "provider":"ACPI.CXL", + "nr_dports":4, + "dports":[ + { + "dport":"pci0000:00", + "alias":"ACPI0016:01", + "id":0 + }, + { + "dport":"pci0000:a8", + "alias":"ACPI0016:02", + "id":4 + }, + { + "dport":"pci0000:2a", + "alias":"ACPI0016:03", + "id":1 + }, + { + "dport":"pci0000:d2", + "alias":"ACPI0016:00", + "id":5 + } + ], + +This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to = CXL +Host Bridges. The `Root` can be considered the singular upstream port att= ached +to the platform's memory controller - which routes memory requests to it. + +The `ports:root0` section lays out how each of these downstream ports are +configured. If a port is not configured (id's 0, 1, and 4), they are omit= ted. + +:: + + "ports:root0":[ + { + "port":"port1", + "host":"pci0000:d2", + "depth":1, + "nr_dports":3, + "dports":[ + { + "dport":"0000:d2:01.1", + "alias":"device:02", + "id":0 + }, + { + "dport":"0000:d2:01.3", + "alias":"device:05", + "id":2 + }, + { + "dport":"0000:d2:07.1", + "alias":"device:0d", + "id":113 + } + ], + +This chunk shows the available downstream ports associated with the CXL Ho= st +Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstr= eam +ports: :code:`dport1`, :code:`dport2`, and :code:`dport113`.. + +:: + + "endpoints:port1":[ + { + "endpoint":"endpoint5", + "host":"mem0", + "parent_dport":"0000:d2:01.1", + "depth":2, + "memdev":{ + "memdev":"mem0", + "ram_size":137438953472, + "serial":0, + "numa_node":0, + "host":"0000:d3:00.0" + }, + "decoders:endpoint5":[ + { + "decoder":"decoder5.0", + "resource":825975898112, + "size":137438953472, + "interleave_ways":1, + "region":"region0", + "dpa_resource":0, + "dpa_size":137438953472, + "mode":"ram" + } + ] + } + ], + +This chunk shows the endpoints attached to the host bridge :code:`port1`. + +:code:`endpoint5` contains a single configured decoder :code:`decoder5.0` +which has the same interleave configuration as :code:`region0` (shown late= r). + +Next we have the decoders belonging to the host bridge: + +:: + + "decoders:port1":[ + { + "decoder":"decoder1.0", + "resource":825975898112, + "size":137438953472, + "interleave_ways":1, + "region":"region0", + "nr_targets":1, + "targets":[ + { + "target":"0000:d2:01.1", + "alias":"device:02", + "position":0, + "id":0 + } + ] + } + ] + }, + +Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`), whose= only +target is :code:`dport1` - which is attached to :code:`endpoint5`. + +The next chunk shows the three CXL host bridges without attached endpoints. + +:: + + { + "port":"port2", + "host":"pci0000:00", + "depth":1, + "nr_dports":2, + "dports":[ + { + "dport":"0000:00:01.3", + "alias":"device:55", + "id":2 + }, + { + "dport":"0000:00:07.1", + "alias":"device:5d", + "id":113 + } + ] + }, + { + "port":"port3", + "host":"pci0000:a8", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:a8:01.1", + "alias":"device:c3", + "id":0 + } + ] + }, + { + "port":"port4", + "host":"pci0000:2a", + "depth":1, + "nr_dports":1, + "dports":[ + { + "dport":"0000:2a:01.1", + "alias":"device:d0", + "id":0 + } + ] + } + ], + +Next we have the `Root Decoders` belonging to :code:`root0`. This root de= coder +is a pass-through decoder because :code:`interleave_ways` is set to :code:= `1`. + +This information is generated by the CXL driver reading the ACPI CEDT CMFW= S. + +:: + + "decoders:root0":[ + { + "decoder":"decoder0.0", + "resource":825975898112, + "size":137438953472, + "interleave_ways":1, + "max_available_extent":0, + "volatile_capable":true, + "nr_targets":1, + "targets":[ + { + "target":"pci0000:d2", + "alias":"ACPI0016:00", + "position":0, + "id":5 + } + ], + +Finally we have the `Memory Region` associated with the `Root Decoder` +:code:`decoder0.0`. This region describes the discrete region associated +with the lone device. + +:: + + "regions:decoder0.0":[ + { + "region":"region0", + "resource":825975898112, + "size":137438953472, + "type":"ram", + "interleave_ways":1, + "decode_state":"commit", + "mappings":[ + { + "position":0, + "memdev":"mem0", + "decoder":"decoder5.0" + } + ] + } + ] + } + ] + } + ] --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 600D727CCD3 for ; Wed, 30 Apr 2025 18:11:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036681; cv=none; b=Yn18iNnckV6Uu8osBZJuPDT/GXPQk9xASs3fmQZRffmASVZaHvimsx4Hr1JVij4VnZcysNkgzKCVI4pNZMTvnwTANuXoF8ZbfTMwHC5BVxWB988M0id0LrTU+tmq1SLOuOwWscxc2QcngNNqledcgzqVuODC2vzFcgaFDsP88d4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036681; c=relaxed/simple; bh=E0fGnn1yC6sz6lUQTwhlc9kcw7MwshD1qLPBTYZmzrk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dPTvXjM8yZuYgDSpoq+emT3CTLLKZ/tWyU6YYre7ISJqtkLYPQqWAMCDNUy0kj/UgLAKSplWVw2THyR3mBus4+KfSSI64AaZwz4J0glNMYrWcVprjDV9oIcGw6YnZOQ3ma0Iki/hKQ/bfwmIFTCpaL1vrebtyQPZ5xiX/RCk/iI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=BWUxOYyD; arc=none smtp.client-ip=209.85.160.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="BWUxOYyD" Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-47691d82bfbso3608711cf.0 for ; Wed, 30 Apr 2025 11:11:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036676; x=1746641476; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+TsZ6+zYHaMJgBEdBWYg9601BLk8goGA5gz8eLP6+mE=; b=BWUxOYyD6w3CACtFk9F9qtIrOuAWx5szhGtYjLSnaAtRnVZKYk192cXIaedb0E4oho lLh9cDGHsv/2utdV0speaE2ccauzKt/WhKcViuZCkICuufKZ+H8S3mVOXM/EOll7ziTA z1V+GfjTg0c11XWDPuIyWfOXmHcZisWLggqDkHxjkfzz87H05d3s2YPe6lw3i24KRyiN s6Cr9h7HP1yyqJgK0MxzQbbQ6XOJNTEdj57gh7XxVOGUGxSBxfPRUaBUHAIl3zllBQqV YqPvFH6o55DvJLfg9v44gpA6Btncv7jvvW4T4/AKHCey5F4DBMEuAnh92VQL+8kJl9k7 xD2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036676; x=1746641476; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+TsZ6+zYHaMJgBEdBWYg9601BLk8goGA5gz8eLP6+mE=; b=gumSO63/O7BY7rE2aWzlrkQQs2/uRfHnIsm1qo3mbBUJZ/yEeYlkJE21c7zgv62Cri MJIY3i/oislnM/HWxM9GFTNoKkWRJ7CyUzZz1bj6svGmM8ehF/8C9a7fkzL91caMnwW9 z/IVND4BpP6dyneKz8FwBI5PDQSok8nLeUwQw0FWjC0y4mI/heTKK7PfJzqVxX0Yc+r/ A30uV4XSiw8Hz01TpSfSanr26tnvQlZEpxILiZ2xuU/P0kWX2/W/w859BapMAEbDWDYO Q6rTk06m4xoJKz/q4sOa2mUsjMGSdHFt31WZ0jpSwQVQ3bHcaXQsz2IqxUbV9mli32O7 XSBQ== X-Forwarded-Encrypted: i=1; AJvYcCV5DWYFkjKRvVBGhXxSKPuKviG7M07++dv5peXg8ZQBsW1igWnGqqYWAB3FndGhrHg5hiJwGDOM9lfAlO4=@vger.kernel.org X-Gm-Message-State: AOJu0YwbZZOVbBwuHsFBsUCpvZ+ZvODIiE2pUl4mx3hxub/ZN7ah+xQm dN4xuAId0jUm3o6Ouvoe7UoM/F6syBwLhwtwxWoU2QYcBzPHmIW9O2u87y2pzNs= X-Gm-Gg: ASbGnct84szSVAC602sFAVgRzsEnoycMhUff8zdNb9MHIkFEVZjth8G+Lnta0fwF6j/ phhx1AWBsTVRwA01sbGtN+6drDAYmkNVrty1RDF+aOmD9ioC0GctoQCVnvoMq/UKln5fJIiDlF9 jKRbMVLNaO+hZtvppapvmFtsxlg4S8s8lDFqIpnXL19KDncydiNVOzE12LkNChwYZ+kwaK8+SUd x+cPkQHm82qMoZSX6jZsGF5sxwFeEmMdTXEtosHqOFYv+mIRypk9Lr0uXvZ4RDUoWuzMJQo2ifv 5TNIlG1/UTqdBNeinLrz1gwCR0WNvHfsUO+meFxtzSV05GY7LBhC2Y9UOqzXYOmUs+1TCQWXOYG jKjK/R7lvZIZ2QUwacieSxiQ9j4U0 X-Google-Smtp-Source: AGHT+IG3ItQI6gLiqIMLhoXsiguUrPuCVfwfwwyMO6/R6vP9My6N+B4ggvm/AG1Yhm/Q1yTCvwIDQg== X-Received: by 2002:a05:622a:418c:b0:476:b461:249b with SMTP id d75a77b69052e-48ae7a1feaemr3415971cf.12.1746036676009; Wed, 30 Apr 2025 11:11:16 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:15 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 10/18] cxl: docs/linux/dax-driver documentation Date: Wed, 30 Apr 2025 14:10:40 -0400 Message-ID: <20250430181048.1197475-11-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation on how the CXL driver interacts with the DAX driver. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/linux/cxl-driver.rst | 115 ++++++++++++++++-- .../driver-api/cxl/linux/dax-driver.rst | 43 +++++++ 3 files changed, 149 insertions(+), 10 deletions(-) create mode 100644 Documentation/driver-api/cxl/linux/dax-driver.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index aa01b14862e1..965f133a1c92 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -37,6 +37,7 @@ that have impacts on each other. The docs here break up = configurations steps. linux/overview linux/early-boot linux/cxl-driver + linux/dax-driver linux/access-coordinates =20 =20 diff --git a/Documentation/driver-api/cxl/linux/cxl-driver.rst b/Documentat= ion/driver-api/cxl/linux/cxl-driver.rst index 486baf8551aa..1a354ea1cda4 100644 --- a/Documentation/driver-api/cxl/linux/cxl-driver.rst +++ b/Documentation/driver-api/cxl/linux/cxl-driver.rst @@ -34,6 +34,32 @@ into a single memory region. The memory region has been = converted to dax. :: decoder1.0 decoder5.0 endpoint5 port1 region0 decoder2.0 decoder5.1 endpoint6 port2 root0 =20 + +.. kernel-render:: DOT + :alt: Digraph of CXL fabric describing host-bridge interleaving + :caption: Diagraph of CXL fabric with a host-bridge interleave memory r= egion + + digraph foo { + "root0" -> "port1"; + "root0" -> "port3"; + "root0" -> "decoder0.0"; + "port1" -> "endpoint5"; + "port3" -> "endpoint6"; + "port1" -> "decoder1.0"; + "port3" -> "decoder3.0"; + "endpoint5" -> "decoder5.0"; + "endpoint6" -> "decoder6.0"; + "decoder0.0" -> "region0"; + "decoder0.0" -> "decoder1.0"; + "decoder0.0" -> "decoder3.0"; + "decoder1.0" -> "decoder5.0"; + "decoder3.0" -> "decoder6.0"; + "decoder5.0" -> "region0"; + "decoder6.0" -> "region0"; + "region0" -> "dax_region0"; + "dax_region0" -> "dax0.0"; + } + For this section we'll explore the devices present in this configuration, = but we'll explore more configurations in-depth in example configurations below. =20 @@ -41,7 +67,7 @@ Base Devices ------------ Most devices in a CXL fabric are a `port` of some kind (because each device mostly routes request from one device to the next, rather than -provide a manageable service). +provide a direct service). =20 Root ~~~~ @@ -53,6 +79,8 @@ The Root contains links to: =20 * `Host Bridge Ports` defined by ACPI CEDT CHBS. =20 +* `Downstream Ports` typically connected to `Host Bridge Ports` + * `Root Decoders` defined by ACPI CEDT CFMWS. =20 :: @@ -150,6 +178,27 @@ device configuration data. :: driver label_storage_size pmem serial firmware numa_node ram subsystem =20 +A Memory Device is a discrete base object that is not a port. While it the +physical device it belongs to may host an `endpoint`, this relationship is +not captured in sysfs. + +Port Relationships +~~~~~~~~~~~~~~~~~~ +In our example described above, there are four host bridges attached to the +root, and two of the host bridges have one endpoint attached. + +.. kernel-render:: DOT + :alt: Digraph of CXL fabric describing host-bridge interleaving + :caption: Diagraph of CXL fabric with a host-bridge interleave memory r= egion + + digraph foo { + "root0" -> "port1"; + "root0" -> "port2"; + "root0" -> "port3"; + "root0" -> "port4"; + "port1" -> "endpoint5"; + "port3" -> "endpoint6"; + } =20 Decoders -------- @@ -322,6 +371,29 @@ settings (granularity and ways must be the same). Endpoint decoders are created during :code:`cxl_endpoint_port_probe` in the :code:`cxl_port` driver, and is created based on a PCI device's DVSEC regi= sters. =20 +Decoder Relationships +~~~~~~~~~~~~~~~~~~~~~ +In our example described above, there is one root decoder which routes mem= ory +accesses over two host bridges. Each host bridge has a decoder which rout= es +access to their singular endpoint targets. Each endpoint has an decoder w= hich +translates HPA to DPA and services the memory request. + +The driver validates relationships between ports by decoder programming, so +we can think of decoders being related in a similarly hierarchical fashion= to +ports. + +.. kernel-render:: DOT + :alt: Digraph of hierarchical relationship between root, switch, and en= dpoint decoders. + :caption: Diagraph of CXL root, switch, and endpoint decoders. + + digraph foo { + "root0" -> "decoder0.0"; + "decoder0.0" -> "decoder1.0"; + "decoder0.0" -> "decoder3.0"; + "decoder1.0" -> "decoder5.0"; + "decoder3.0" -> "decoder6.0"; + } + Regions ------- =20 @@ -348,6 +420,17 @@ The interleave settings in a `Memory Region` describe = the configuration of the `Interleave Set` - and are what can be expected to be seen in the endpoint interleave settings. =20 +.. kernel-render:: DOT + :alt: Digraph of CXL memory region relationships between root and endpo= int decoders. + :caption: Regions are created based on root decoder configurations. End= point decoders + must be programmed with the same interleave settings as the r= egion. + + digraph foo { + "root0" -> "decoder0.0"; + "decoder0.0" -> "region0"; + "region0" -> "decoder5.0"; + "region0" -> "decoder6.0"; + } =20 DAX Region ~~~~~~~~~~ @@ -360,7 +443,6 @@ for more details. :: dax0.0 devtype modalias uevent dax_region driver subsystem =20 - Mailbox Interfaces ------------------ A mailbox command interface for each device is exposed in :: @@ -418,17 +500,30 @@ the relationships between a decoder and it's parent. =20 For example, in a `Cross-Link First` interleave setup with 16 endpoints attached to 4 host bridges, linux expects the following ways/granularity -across the root, host bridge, and endpoints respectively. :: +across the root, host bridge, and endpoints respectively. + +.. flat-table:: 4x4 cross-link first interleave settings + + * - decoder + - ways + - granularity =20 - ways granularity - root 4 256 - host bridge 4 1024 - endpoint 16 256 + * - root + - 4 + - 256 + + * - host bridge + - 4 + - 1024 + + * - endpoint + - 16 + - 256 =20 At the root, every a given access will be routed to the :code:`((HPA / 256) % 4)th` target host bridge. Within a host bridge, every -:code:`((HPA / 1024) % 4)th` target endpoint. Each endpoint will translate -the access based on the entire 16 device interleave set. +:code:`((HPA / 1024) % 4)th` target endpoint. Each endpoint translates ba= sed +on the entire 16 device interleave set. =20 Unbalanced interleave sets are not supported - decoders at a similar point in the hierarchy (e.g. all host bridge decoders) must have the same ways a= nd @@ -467,7 +562,7 @@ In this example, the CFMWS defines two discrete non-int= erleaved 4GB regions for each host bridge, and one interleaved 8GB region that targets both. Th= is would result in 3 root decoders presenting in the root. :: =20 - # ls /sys/bus/cxl/devices/root0 + # ls /sys/bus/cxl/devices/root0/decoder* decoder0.0 decoder0.1 decoder0.2 =20 # cat /sys/bus/cxl/devices/decoder0.0/target_list start size diff --git a/Documentation/driver-api/cxl/linux/dax-driver.rst b/Documentat= ion/driver-api/cxl/linux/dax-driver.rst new file mode 100644 index 000000000000..5063d2b675b4 --- /dev/null +++ b/Documentation/driver-api/cxl/linux/dax-driver.rst @@ -0,0 +1,43 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +DAX Driver Operation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The `Direct Access Device` driver was originally designed to provide a +memory-like access mechanism to memory-like block-devices. It was +extended to support CXL Memory Devices, which provide user-configured +memory devices. + +The CXL subsystem depends on the DAX subsystem to generate either: + +- A file-like interface to userland via :code:`/dev/daxN.Y`, or +- Engaging the memory-hotplug interface to add CXL memory to page allocato= r. + +The DAX subsystem exposes this ability through the `cxl_dax_region` driver. +A `dax_region` provides the translation between a CXL `memory_region` and +a `DAX Device`. + +DAX Device +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +A `DAX Device` is a file-like interface exposed in :code:`/dev/daxN.Y`. A +memory region exposed via dax device can be accessed via userland software +via the :code:`mmap()` system-call. The result is direct mappings to the +CXL capacity in the task's page tables. + +Users wishing to manually handle allocation of CXL memory should use this +interface. + +kmem conversion +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The :code:`dax_kmem` driver converts a `DAX Device` into a series of `hotp= lug +memory blocks` managed by :code:`kernel/memory-hotplug.c`. This capacity +will be exposed to the kernel page allocator in the user-selected memory +zone. + +The :code:`memmap_on_memory` setting (both global and DAX device local) di= ctate +where the kernell will allocate the :code:`struct folio` descriptors for t= his +memory will come from. If :code:`memmap_on_memory` is set, memory hotplug +will set aside a portion of the memory block capacity to allocate folios. = If +unset, the memory is allocated via a normal :code:`GFP_KERNEL` allocation - +and as a result will most likely land on the local NUM node of the cpu exe= cuting +the hotplug operation. --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A57A127E7D7 for ; Wed, 30 Apr 2025 18:11:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036682; cv=none; b=BcvFPBnuZt8cY1iXllKd8xs5ABK/Kxkh59otiu5bhEo/xqbdLTDTT6Oqlcdo7FiGiUKPB5rH7/uqmAh5jk0FNBaLOKcfpCTF1KblNVZqU8MvJxzUOP5V/jllqzZQ1Rq2yKhS1+xVscXIbmGDyaOt1LgLW88zlSzGhPD2ORvzfa0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036682; c=relaxed/simple; bh=5+WGsMDzQfixwYnglIBJyZyeOTlzUu6NS9E3xL/s1ds=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i5oJ7G4buP/b4ZL5+1mffcNpGVTboB7BPU54MMzD7Igg5mcRUFyGmG91c+GkOoj3Iv60O0ipH6zGh2N6PIBJCs7QrSdUOAf7AsL7TI1fnCXpY7k7BChajVNX+ckm7ydRUQLVCTXHVEpS/lbW5cWkitnJdJD5Rw70KmxgwmTnukc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=bfIqiZrx; arc=none smtp.client-ip=209.85.222.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="bfIqiZrx" Received: by mail-qk1-f172.google.com with SMTP id af79cd13be357-7c5e1b40f68so10845785a.1 for ; Wed, 30 Apr 2025 11:11:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036677; x=1746641477; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qZSGq1S9E4b7j+VlhOUItxRA7BaImspXLYaizp94BIQ=; b=bfIqiZrxk4mGCeXJe4VldTZnncSdeIhQCOhPkFZquqs8Wuk5dvdQ8XGsPelqdjjczq NSbIgNAV1IEgkyi8RYT1OInh4cRB7zIm+3uNAq5hu8d6QTo3BpY3ilMLxHgmf7zkpaO7 nEcpxQZsPD8pxmMiRkKsTZzOqY9d2dYisQuuTS4VggF9s9nN90+4qedo293dM4TBfkc+ ozlUr0y9Ag1UZplD56VpYrRWH7SRppNkEEiEXFY6b5XqDOuGhpIqZV2uDHzFzBy4nX50 Dpa9DgI/ewizSrIGjS0ZslFS2o1vRLmSkCjypJPORGzfyQCOI6Su/SZSM3XQUXWoyZUo gMmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036677; x=1746641477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qZSGq1S9E4b7j+VlhOUItxRA7BaImspXLYaizp94BIQ=; b=XUuw+jUj+AYMlxvPUvIBzNvXnjfp3RpehFr0mojRu2AS2YD1hzRHhph8CKRyfVmyu3 4OMHY5OweDHH7Fl8o5bAvp9CcoanT4ExXCVxTuMELBTcCdXyKhDgx4BoyKFVlec8VaWC DkfD7LEGLIiAWEVbuUs/QhtjUGodzp041duMLzhUtYFaNijQ1G/THezolTPKQ3b5x5Ms C1VObztjut/4PZEiUDiwWGEtvICNUSTEoflgNT7qugHkq71i/G3+8J6SW/g4yCZB2ZiZ Ls1rT+OI5hkYyPNk8+wKvLJnb6shEa6maWFQpYTlhHMWCGs0XX9G8oHtnjMWGE8pIVG2 gTMw== X-Forwarded-Encrypted: i=1; AJvYcCUrlP8ml68SOQsYIeeXnBdxB9WdjONe/mn1FmBUzElRnpPyCBT7WsqOBNDff4zFS3/T+XMbxTgsjuZGqrA=@vger.kernel.org X-Gm-Message-State: AOJu0Yx1KD3Fug0rFePCuxREa5M9P4h508ZfhAS9z54vOTCtP4Km15sD gmvuZdADIY7CvKaSh5qW92tQr80rQC50JZbzQbz9vRXrNWI9+NhFakWMsv5XHVE= X-Gm-Gg: ASbGncvL8OyvgwLvyeOiiEyl/KTkGP1X+kTDOV+XXT/4/0gs5nwHfhrqo6rsWL3vqX9 qJOZAJIISsX8cVj0FhVBH9fgpLPGo8Yck8q+IRdH42sFk4vdCuF0qLUJKTc476pGNN+dW6KqsMu LjMnaOgmUOf05Bkfz1GChjX+czakxAgc5RA932baujpRWf5UKSbgNCmunwhWAurZcpPdwAU/dKZ amDwfq8u8VkLAhEOcdKahurUtf8yQs+Jwx11xPUsXibQQBEkPdnPe5CrUaTPGTStHfZJV72G8ei W9SAAi6oKszJOJiybzXMQspfvBYOwvaS9gjf1nAQCemy53O+zG6vf0WEvhspva8064CtwWnAZYw NLNsCnu+DFzNpUkgDb/5GXhD03e9N X-Google-Smtp-Source: AGHT+IEHANnvDvH2u9pl8BjqFctHnHP3/Lhe18uJad/mRqqVJmp00Q7PEDHT6jtq9qycvYfH0LpUmQ== X-Received: by 2002:a05:622a:420e:b0:476:9dc9:3c2a with SMTP id d75a77b69052e-489c57f8405mr54102241cf.48.1746036677513; Wed, 30 Apr 2025 11:11:17 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:17 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 11/18] cxl: docs/linux/memory-hotplug Date: Wed, 30 Apr 2025 14:10:41 -0400 Message-ID: <20250430181048.1197475-12-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation on how the CXL driver surfaces memory through the DAX driver and memory-hotplug. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 1 + .../driver-api/cxl/linux/memory-hotplug.rst | 78 +++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 Documentation/driver-api/cxl/linux/memory-hotplug.rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 965f133a1c92..f68a3eb2bb5e 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -38,6 +38,7 @@ that have impacts on each other. The docs here break up = configurations steps. linux/early-boot linux/cxl-driver linux/dax-driver + linux/memory-hotplug linux/access-coordinates =20 =20 diff --git a/Documentation/driver-api/cxl/linux/memory-hotplug.rst b/Docume= ntation/driver-api/cxl/linux/memory-hotplug.rst new file mode 100644 index 000000000000..a757e0e4487e --- /dev/null +++ b/Documentation/driver-api/cxl/linux/memory-hotplug.rst @@ -0,0 +1,78 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Memory Hotplug +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The final phase of surfacing CXL memory to the kernel page allocator is for +the `DAX` driver to surface a `Driver Managed` memory region via the +memory-hotplug component. + +There are four major configurations to consider + +1) Default Online Behavior (on/off and zone) +2) Hotplug Memory Block size +3) Memory Map Resource location +4) Driver-Managed Memory Designation + +Default Online Behavior +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The default-online behavior of hotplug memory is dictated by the following, +in order of precedence: + +- :code:`CONFIG_MHP_DEFAULT_ONLINE_TYPE` Build Configuration +- :code:`memhp_default_state` Boot parameters +- :code:`/sys/devices/system/memory/auto_online_blocks` value + +These dictate whether hotplugged memory blocks arrive in one of three stat= es: + +1) Offline +2) Online in :code:`ZONE_NORMAL` +3) Online in :code:`ZONE_MOVABLE` + +:code:`ZONE_NORMAL` implies this capacity may be used for almost any alloc= ation, +while :code:`ZONE_MOVABLE` implies this capacity should only be used for +migratable allocations. + +:code:`ZONE_MOVABLE` attempts to retain the hotplug-ability of a memory bl= ock +so that it the entire region may be hot-unplugged at a later time. Any ca= pacity +onlined into :code:`ZONE_NORMAL` should be considered permanently attached= to +the page allocator. + +Hotplug Memory Block Size +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D +By default, on most architectures, the Hotplug Memory Block Size is either +128MB or 256MB. On x86, the block size increases up to 2GB as total memory +capacity exceeds 64GB. As of v6.15, Linux does not take into account the +size and alignment of the ACPI CEDT CFMWS regions (see Early Boot docs) wh= en +deciding the Hotplug Memory Block Size. + +Memory Map +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The location of :code:`struct folio` allocations to represent the hotplugg= ed +memory capacity are dicated by the following system settings: + +- :code:`/sys_module/memory_hotplug/parameters/memmap_on_memory` +- :code:`/sys/bus/dax/devices/daxN.Y/memmap_on_memory` + +If both of these parameters are set to true, :code:`struct folio` for this +capacity will be carved out of the memory block being onlined. This has +performance implications if the memory is particularly high-latency and +its :code:`struct folio` becomes hotly contended. + +If either parameter is set to false, :code:`struct folio` for this capacity +will be allocated from the local node of the processor running the hotplug +procedure. This capacity will be allocated from :code:`ZONE_NORMAL` on +that node, as it is a :code:`GFP_KERNEL` allocation. + +Systems with extremely large amounts of :code:`ZONE_MOVABLE` memory (e.g. +CXL memory pools) must ensure that there is sufficient local +:code:`ZONE_NORMAL` capacity to host the memory map for the hotplugged cap= acity. + +Driver Managed Memory +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The DAX driver surfaces this memory to memory-hotplug as "Driver Managed".= This +is not a configurable setting, but it's important to not that driver manag= ed +memory is explicitly excluded from use during kexec. This is required to = ensure +any reset or out-of-band operations that the CXL device may be subject to = during +a functional system-reboot (such as a reset-on-probe) will not cause porti= ons of +the kexec kernel to be overwritten. --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B80327E7F8 for ; Wed, 30 Apr 2025 18:11:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036682; cv=none; b=mFPuFF3xPuRZZkM41jMwmXeejgkkc9hAwCeQL8s3fHcXhNdO+RSuW3WNkDTf+yN3C3w3p0OiUHRvArgnXBBEShF/0Chh29DFQVOqz4L7gMG2Qt58snJLyB0pp7lKQRcQGv1na8nb5Ld/4vcctDCNJSqajOUystNI+aHIdSiLO/g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036682; c=relaxed/simple; bh=Pt1H2NqKNJM2Nyqg1OGfPH7Lob5zRf30NWeWOGzqUs8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jp/Rd3Whg/sZmzp6jjUCLTlGlsUn8hv8FqqgsP/hTn9XsQn8lunbYM+dQavxJT4tM0geyD6tZlc3F2x9ID8VK0Vjv1jY1dn23ZpohjhTAcOoieSEw73M2/cosNZCn6dSJ5IE22XPJ4Fbii4OAktZdaMoFxdhAMraXrvF0eyv+2Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=fyjAs3Fy; arc=none smtp.client-ip=209.85.219.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="fyjAs3Fy" Received: by mail-qv1-f52.google.com with SMTP id 6a1803df08f44-6e8fce04655so2237926d6.3 for ; Wed, 30 Apr 2025 11:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036679; x=1746641479; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Bw3X5p6Hy+2rJiulBTqaYUggRZ+nbTaNRc7v49X1EHE=; b=fyjAs3Fy/C69xZzZkYTkc4/8RP+rg+MZBnGNnJECaMd0aMb/6IfQG875CLrJM8NSv2 nxNPoVIhnlyi5z9Z6aQi5ko9+ruaVuNvMGu2LtBQSe6ms/Ks/EVNLuaWQCmL5m87S0sG vgSc4sfApaVq9PfFlMhBHWFZPe7rAEOZWy71iWnz7YMDGIgWeidYp17LhIZnmWWLdEzy xGdcHpwLpiUxnJO8ZxJyIv37N9lCoScnJTP3fhMrAeMfMhU32hJ2GfpejLevryEWrpG0 5pui5isRqqDOMxP++s8CaVOIR0+3xaD52MRuhSq435IkQRUiBEcGBu3LAd0QDWI+RhF/ KBGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036679; x=1746641479; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Bw3X5p6Hy+2rJiulBTqaYUggRZ+nbTaNRc7v49X1EHE=; b=dxTzFBXAgxHjVCIMhR4NyhxT0DIZ1d+niykKzP2Q4l3R5ANVMXJhFNi3ENMXkmJoo3 YSsYU132YjzFaX4TDThwYmtnBMl7gxCXcK6p+ye+nfXkYWIEDjEXgYntwjqWeiiSEsC+ 7NNTM7CM5RsgpWYPRwa41d6FtN7eKlXMXYkYm9z0vWJ87FdIRMtYRcYAAYx/tJhqtp4T K0jK+yoOd8AzKYwyCYD7hDwKCU9LaWBdnbDTqS4vtJL66CdHzibdgB0ED9YdLl3mFK8i WSLR9tltn8tmgHqJ5KMvPMZYLHlsPz86a/YV62YkWsy6amd+eZLktmf66+bWQUB0fjLL +2eQ== X-Forwarded-Encrypted: i=1; AJvYcCX/YlZ4gxDDPJdRa50O8ZUNe+LuBWVdWkS73ywLYtMmyrjKjpBxEJR3J2FzFYbh42jdMTK86fsMyWDB2LQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yz78EozyfTe8Hz33luHKC9l9DScOFO47FC/VqJTuGjo40TTxgpK KpprqBM7+l8Bq+cPUW6x+4EsI84gP5RZPgmYj5DdxRYZcxpgBARBA+tbJjX7xPkd4XKyMUM8NPd V X-Gm-Gg: ASbGncvQhyPea1fsoX/0R9D5gkW2nIXjO0vRNgvALyWHXGx8HiypUU31Auce3rhOmSY 9FEFvWpGqPDZHn2SSAhzCXccaJTXHVADoRyqigACLcf2FgqBei81PYSa7jlPpiU6pATIXqx+iWL WVmIf5bjNalfTQuHcdTKgnXZdpHOUL9c+cbldZkJZnNCYmTT0AnGUn0LukbYkm/4zBj73AstYdK SvhjMuT7TSKzfyJx0BikXPew4YydreawxA6wwzC2BLWlqLg6DiJkSYO9eeKMlEt05yRBasYUfHY 32R7D7vkcJGGqTyRQ03ZKNydFy+TwxPzTEqliNOxAkH3GFyxz8ml9+GJPI7fPinptS5iif4iajU EforAKD6hYiDI53u+bkbX3L5YT4/1 X-Google-Smtp-Source: AGHT+IF9uQHO8iaM85TiFvA2BcpXkfNkQmvExe8PEqVIRx6LQV1WN3wovyF6KWS5M+cgfmqegzLPvg== X-Received: by 2002:a05:6214:2405:b0:6e8:fa33:2969 with SMTP id 6a1803df08f44-6f4fce7b58cmr86397876d6.10.1746036678981; Wed, 30 Apr 2025 11:11:18 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:18 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 12/18] cxl: docs/allocation/dax Date: Wed, 30 Apr 2025 14:10:42 -0400 Message-ID: <20250430181048.1197475-13-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Small example of accessing CXL memory capacity via DAX device Signed-off-by: Gregory Price --- .../driver-api/cxl/allocation/dax.rst | 60 +++++++++++++++++++ Documentation/driver-api/cxl/index.rst | 5 ++ 2 files changed, 65 insertions(+) create mode 100644 Documentation/driver-api/cxl/allocation/dax.rst diff --git a/Documentation/driver-api/cxl/allocation/dax.rst b/Documentatio= n/driver-api/cxl/allocation/dax.rst new file mode 100644 index 000000000000..c6f7a5da832f --- /dev/null +++ b/Documentation/driver-api/cxl/allocation/dax.rst @@ -0,0 +1,60 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +DAX Devices +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +CXL capacity exposed as a DAX device can be accessed directly via mmap. +Users may wish to use this interface mechanism to write their own userland +CXL allocator, or to managed shared or persistent memory regions across mu= ltiple +hosts. + +If the capacity is shared across hosts or persistent, appropriate flushing +mechanisms must be employed unless the region supports Snoop Back-Invalida= te. + +Note that mappings must be aligned (size and base) to the dax device's base +alignment, which is typically 2MB - but maybe be configured larger. + +:: + + #include + #include + #include + #include + #include + #include + + #define DEVICE_PATH "/dev/dax0.0" // Replace DAX device path + #define DEVICE_SIZE (4ULL * 1024 * 1024 * 1024) // 4GB + + int main() { + int fd; + void* mapped_addr; + + /* Open the DAX device */ + fd =3D open(DEVICE_PATH, O_RDWR); + if (fd < 0) { + perror("open"); + return -1; + } + + /* Map the device into memory */ + mapped_addr =3D mmap(NULL, DEVICE_SIZE, PROT_READ | PROT_WRITE, + MAP_SHARED, fd, 0); + if (mapped_addr =3D=3D MAP_FAILED) { + perror("mmap"); + close(fd); + return -1; + } + + printf("Mapped address: %p\n", mapped_addr); + + /* You can now access the device through the mapped address */ + uint64_t* ptr =3D (uint64_t*)mapped_addr; + *ptr =3D 0x1234567890abcdef; // Write a value to the device + printf("Value at address %p: 0x%016llx\n", ptr, *ptr); + + /* Clean up */ + munmap(mapped_addr, DEVICE_SIZE); + close(fd); + return 0; + } diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index f68a3eb2bb5e..f779dfb0cd64 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -41,5 +41,10 @@ that have impacts on each other. The docs here break up= configurations steps. linux/memory-hotplug linux/access-coordinates =20 +.. toctree:: + :maxdepth: 2 + :caption: Memory Allocation + + allocation/dax =20 .. only:: subproject and html --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 966CB27FD55 for ; Wed, 30 Apr 2025 18:11:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036683; cv=none; b=S70fWXCdhuiMOrF6Wv/uS0TRtlDvMp/unz0ANo5+ORH8/jIdop9YAYLMN9sr5xKOGdBmRc0IrVPNpvyJz1xphBU5K65E0WEK+biHpHGpe6cKZ9WVdqJx2DRwGix+JyPSmHK2lBk+pHui9HDiVw2RafPzWauS5hkt6YG4lE6sg0g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036683; c=relaxed/simple; bh=yfrC0Eme8BmQ1/OHJ9XQ1ZL+snR+lO3XBR/5UUcZJfg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Qlim/rh6+hCHKV3lYXhdOEoZhFXCOEfrFEPIl3Rhj6fRmhM60hr5Vr5k/V63RAXCpl/gkYhiBf2sL2b0M2uRR6eNi4f9Oq3XBbSBJGBUZ956oKHj70EtnVxDnjh+GiG6pKM31APv6hkcqFrbF3FwOZV7JcwrABFzAEb9A5kfEXQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=jKR0MMC9; arc=none smtp.client-ip=209.85.160.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="jKR0MMC9" Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-4772f48f516so14874191cf.1 for ; Wed, 30 Apr 2025 11:11:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036680; x=1746641480; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IvC2rs3uihn14TnY/NB2X/a9ksGTd6adbUOlhi/H5xc=; b=jKR0MMC9vUO+cTHFNR9O/Mq4rW9izXl1/pEOs0z9h5Zbs+UPtLWfdIdTIZOV6EUtAV ohGcItTO1tBCob7Xw6kLk4qX03UHoCnoyrBUKqRoVqQBAkL/c0sZyM5M6JmNEotAJ+6W 4e5Z5QXrd73iVPZMiJSM6J1Kx/0Ar6FNiiw1T1quCWJPMWU5ndXntQBNhYRMa8wDWs8L md/Nmajw2yHPMbRjeINPpwnbxFwVfvGqOlDgT3wnes55TKdcmbf5JjU7gDppnj74N4v9 2zCJa685cVyDVHMK12w9H/IyxcDKCjpLpwZTvbZGG+h5SAQXCLO+9LBlrrbMt7qSrMFh wPCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036680; x=1746641480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IvC2rs3uihn14TnY/NB2X/a9ksGTd6adbUOlhi/H5xc=; b=FU73efwzITRF/WqhAVOr2hQVk38B5KikjAP94uxTtCLxifQlfXlA9ir92QTdL9vx1Z loAeovbzNtvZnDzfn7+EbATrEQBNoM2md/ybOgi+zagxYrGXDWX2dX/yJqGIoZ+rj1L2 29/6oaYBbVDR+ITIM7NftEcslcsxC4RMorE5b7Pgx55I78CyUgTMFUCcjZmZKjTGK9eL 0DhyUKimV8SJyklOFBIlVzkLNZdK9ra2Hrj3CnF3U2dp0eRhtEngbPftZkIozevCrIlQ xsu5mihW7TJhIalPKtocw+fIxusbzYZ8sLAZW8W41JqpUxqqIhJH/ZvBsYf5qVt75OuL YiBA== X-Forwarded-Encrypted: i=1; AJvYcCV3OXLjvoNyDajRZrSl23Hh+mvKbJDzx8KhBglYhPJlDzl5Rfp7zl8wG22zhomB+zKPHTeW2c+XhSUvfDo=@vger.kernel.org X-Gm-Message-State: AOJu0YzCd73BF0YfQ81j51gjz0dvMsSLkbuaj7I/gGjIsFCya8dVdoUd rbw8txd1lviyYOI66oz9dl2vq6Oq1j1dEf7ArDu1R+62wtJLr87hnPP4ZNYkv5I= X-Gm-Gg: ASbGncuJN2hlt1+wn9M2GhHUn9SCbS/e1vCU7Xdds1duvAjRREdoSz3W61r0tHSstdT csuLGzR77Lm8pTemweAWqLxY1SFXl2BYiJ14txeuDqL5iQxe+SjBjB0b4zqWRVMKQwPbVSh6Lch o6HEU7gMxH6LSVhrG6RoNzWDje0RkCrlqqnMtk1bcLdenVBhVedZ+oWFCKgEFMvYSXj0mPxZsnv 7ki21lnPrclOw0E2zN+hl+P8EGXrRT2srOimKBDnZq55nIOJc041PUjYl+dkpIaZCyLf7q9Gt7g axeir0SPpk7QikzbD+r/QUkruHU8zBbQVmKSsWYSP0pkRJoIx4wPBofiAhesFAKOtbkHLj3b8uq jRv0ECgAEwCcnoj9kGtHq1rYD7kRr X-Google-Smtp-Source: AGHT+IFz5WPodsmmRrFPGLR1bzkZlvspw+qAXKgayP3FunFG0J4oFtDaOJOklbA327O2PgMdv3UdIg== X-Received: by 2002:a05:622a:228e:b0:471:80ef:35e7 with SMTP id d75a77b69052e-48ad7e8d24fmr6910221cf.4.1746036680313; Wed, 30 Apr 2025 11:11:20 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:19 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 13/18] cxl: docs/allocation/page-allocator Date: Wed, 30 Apr 2025 14:10:43 -0400 Message-ID: <20250430181048.1197475-14-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Document some interesting interactions that occur when exposing CXL memory capacity to page allocator. Signed-off-by: Gregory Price --- .../cxl/allocation/page-allocator.rst | 86 +++++++++++++++++++ Documentation/driver-api/cxl/index.rst | 1 + 2 files changed, 87 insertions(+) create mode 100644 Documentation/driver-api/cxl/allocation/page-allocator.= rst diff --git a/Documentation/driver-api/cxl/allocation/page-allocator.rst b/D= ocumentation/driver-api/cxl/allocation/page-allocator.rst new file mode 100644 index 000000000000..f5b21d3eb63f --- /dev/null +++ b/Documentation/driver-api/cxl/allocation/page-allocator.rst @@ -0,0 +1,86 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The Page Allocator +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The kernel page allocator services all general page allocation requests, s= uch +as :code:`kmalloc`. CXL configuration steps affect the behavior of the pa= ge +allocator based on the selected `Memory Zone` and `NUMA node` the capacity= is +placed in. + +This section mostly focuses on how these configurations affect the page +allocator (as of Linux v6.15) rather than the overall page allocator behav= ior. + +NUMA nodes and mempolicy +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Unless a task explicitly registers a mempolicy, the default memory policy +of the linux kernel is to allocate memory from the `local NUMA node` first, +and fall back to other nodes only if the local node is pressured. + +Generally, we expect to see local DRAM and CXL memory on separate NUMA nod= es, +with the CXL memory being non-local. Technically, however, it is possible +for a compute node to have no local DRAM, and for CXL memory to be the +`local` capacity for that compute node. + + +Memory Zones +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +CXL capacity may be onlined in :code:`ZONE_NORMAL` or :code:`ZONE_MOVABLE`. + +As of v6.15, the page allocator attempts to allocate from the highest +available and compatible ZONE for an allocation from the local node first. + +An example of a `zone incompatibility` is attempting to service an allocat= ion +marked :code:`GFP_KERNEL` from :code:`ZONE_MOVABLE`. Kernel allocations a= re +typically not migratable, and as a result can only be serviced from +:code:`ZONE_NORMAL` or lower. + +To simplify this, the page allocator will prefer :code:`ZONE_MOVABLE` over +:code:`ZONE_NORMAL` by default, but if :code:`ZONE_MOVABLE` is depleted, it +will fallback to allocate from :code:`ZONE_NORMAL`. + + +Zone and Node Quirks +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Lets consider a configuration where the local DRAM capacity is largely onl= ined +into :code:`ZONE_NORMAL`, with no :code:`ZONE_MOVABLE` capacity present. T= he +CXL capacity has the opposite configuration - all onlined in +:code:`ZONE_MOVABLE`. + +Under the default allocation policy, the page allocator will completely sk= ip +:code:`ZONE_MOVABLE` has a valid allocation target. This is because, as of +Linux v6.15, the page allocator does approximately the following: :: + + for (each zone in local_node): + + for (each node in fallback_order): + + attempt_allocation(gfp_flags); + +Because the local node does not have :code:`ZONE_MOVABLE`, the CXL node is +functionally unreachable for direct allocation. As a result, the only way +for CXL capacity to be used is via `demotion` in the reclaim path. + +This configuration also means that if the DRAM ndoe has :code:`ZONE_MOVABL= E` +capacity - when that capacity is depleted, the page allocator will actually +prefer CXL :code:`ZONE_MOVABLE` pages over DRAM :code:`ZONE_NORMAL` pages. + +We may wish to invert these configurations in future Linux versions. + +If `demotion` and `swap` are disabled, Linux will begin to cause OOM crash= es +when the DRAM nodes are depleted. This will be covered amore in depth in t= he +reclaim section. + + +CGroups and CPUSets +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Finally, assuming CXL memory is reachable via the page allocation (i.e. on= lined +in :code:`ZONE_NORMAL`), the :code:`cpusets.mems_allowed` may be used by +containers to limit the accessibility of certain NUMA nodes for tasks in t= hat +container. Users may wish to utilize this in multi-tenant systems where s= ome +tasks prefer not to use slower memory. + +In the reclaim section we'll discuss some limitations of this interface to +prevent demotions of shared data to CXL memory (if demotions are enabled). + diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index f779dfb0cd64..2be2855e5870 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -46,5 +46,6 @@ that have impacts on each other. The docs here break up = configurations steps. :caption: Memory Allocation =20 allocation/dax + allocation/page-allocator =20 .. only:: subproject and html --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B024283FC1 for ; Wed, 30 Apr 2025 18:11:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036689; cv=none; b=mdAEsJhaOMLAo4ACQ8vHoNUVM2JoGfJ7vd9gxZPINJDCkshOBemnoHTHLPYu8vyRtCDZLQSmogvKsmLPoQCS0RvI02ia472CrFDKyGyeNZUP4KEBA9nzjNlWHrU1mrw3veo794a4nMQTFzro51lpF1PYf8zCglZszafnqmNOMjA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036689; c=relaxed/simple; bh=fSoRlnSmuQfl7ZxHPvsaxZElUPVqtO8ei3VkyP+sGlc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DKm3tiSIttZ1I9p7b4oxgsp/KQHBEhZKwBmdjP5MPKeQoq56hS/W23mUcrePYWX5YCuK7qMsEODE9wGGf7cv4rn9Nxg0Rc1IWNDUQaozM8ok+8yjuGW5uJNf1aECBoCwMec7kFRm5JWgqqP9lXfO+WYzEvz292jF8vJUOuE50Rs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=fZzCLFlR; arc=none smtp.client-ip=209.85.160.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="fZzCLFlR" Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-47692b9d059so2516621cf.3 for ; Wed, 30 Apr 2025 11:11:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036685; x=1746641485; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Tbt8tCPdtGfAIVOyjjS42F8T26gzOQDX+9RCN/0lBgM=; b=fZzCLFlRj67Mt2er8z5Aw5Q5JVrmDB6JZFA5ZARFksGaSjqrWzcbdj+0fcCv/911dj 3FmVCqmjQlO89nzAdJ42VgFaUquyGe7bYwKpK+r0mq2hSjU6ximELepfgKqSUazxZpGW nAA9rRG54GkpIjMlXfv6OHU0PEPcoiUWD5V4CIIP/o0Bnka8L1jK2ryHUHDh+qgE7GAd 0Ey7W68nTyJ6BC3CZX6wP2pVvM5Xrfh0lX7HxI8apFpolJ0c8S56J/qGBNQTSeObiOID xXVvI285iWYu75z9hc7sUCxMIP30aZvrYupW3GoHfa/o0d1aDuBy3D9nYkLWjakvQac7 CxMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036685; x=1746641485; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Tbt8tCPdtGfAIVOyjjS42F8T26gzOQDX+9RCN/0lBgM=; b=C9IzWK55irhaMoHaiuhPEt6nbgwMq39d0+gAMFiQzN3aSOHK52lmdE98h8wJcivC6C HOMZkRgJzKY+Gjp5eFTyPUwP+7gAys/TidNXWZzKzfDNNO39BcaTx60PWRTa0yDnF6L1 d+JBbdehSgpmXPzuIaIdKqi0+ZK/MKqYs4VVnIa7U3YYuNqf52AZcxvcqEpJSwX+u2QP 2j917GdeITqV2bew7s/lRn27cqGyXl4XSmhICfY7oSlw+zZfDuD5LqwdpH+6UDi/kIaB zzzFVvxf0X/g0mK5qylXUSNJ3Y8ZdA/CfoAaHBecZl2neh01RBDrzZZeCP4c5hZlTsRQ 7Nrw== X-Forwarded-Encrypted: i=1; AJvYcCVMmJnuf7ixsPXsWtFoMw1/54D0+rqSQVFeneIIb6o77W6Vddn2CVi0mEBvjsHKoYQxBSu595RG1mq/uWQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxpCTWxe4yd+QdSSqq6XM0udJYuy9hKZJbalkC+SXdjzBevNvb8 O9OKAZas0ZGm0IjWK8q/53oTKmIbW9Ko1BxW6qeJmnNfnOSa3NpU8i2Eq2jD93A= X-Gm-Gg: ASbGnctHHt9SoLIs+jzd4UgXCqaVeSVBo7ZcpyOna+E5FHB4V1l3Kj+HakzCATHE6MV StpvfwMfn4kuWNeX+esxYTpU8UXnGw6IVziEbnqsOZswgbe+fFGus1uVirZ8w6vNKwHG2+nvikm m+9WlTTXhZR2JRKBnlhCiYnoiuvFe0My9IBeiYs90V/6Co59rYM7tXBmpEb4RKMAufFYqrz7T8k NJ/B7RCCtD5WuzfWywnR2SDFDKucLIQmlTF6H992lSzWDURrP/dYqkwE6Ts/DkVYV9Ng855s+jA qY2rFoRy89imvIeZQdvfEjjI6EsuiB53w96aWFqs3ldnXgKTqxPMy1DbRfzDQAxKNTheZk+9rmT s7DnMJUVCEhJmrU1ImmMI3h11C672 X-Google-Smtp-Source: AGHT+IF7lGDo2V4OWxe67pw6pes0roVjPL60qV8ZkhzEjnVw85shlE47W9reKIhwnUyuTehJMjsREw== X-Received: by 2002:a05:622a:4c87:b0:476:739a:5cf3 with SMTP id d75a77b69052e-489e46a4769mr63495651cf.1.1746036685174; Wed, 30 Apr 2025 11:11:25 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:24 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 14/18] cxl: docs/allocation/reclaim Date: Wed, 30 Apr 2025 14:10:44 -0400 Message-ID: <20250430181048.1197475-15-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Document a bit about how reclaim interacts with various CXL configurations. Signed-off-by: Gregory Price --- .../driver-api/cxl/allocation/reclaim.rst | 51 +++++++++++++++++++ Documentation/driver-api/cxl/index.rst | 1 + 2 files changed, 52 insertions(+) create mode 100644 Documentation/driver-api/cxl/allocation/reclaim.rst diff --git a/Documentation/driver-api/cxl/allocation/reclaim.rst b/Document= ation/driver-api/cxl/allocation/reclaim.rst new file mode 100644 index 000000000000..f37c8b1cc3bd --- /dev/null +++ b/Documentation/driver-api/cxl/allocation/reclaim.rst @@ -0,0 +1,51 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D +Reclaim +=3D=3D=3D=3D=3D=3D=3D +Another way CXL memory can be utilized *indirectly* is via the reclaim sys= tem +in :code:`mm/vmscan.c`. Reclaim is engaged when memory capacity on the sy= stem +becomes pressured based on global and cgroup-local `watermark` settings. + +In this section we won't discuss the `watermark` configurations, just how = CXL +memory can be consumed by various pieces of reclaim system. + +Demotion +=3D=3D=3D=3D=3D=3D=3D=3D +By default, the reclaim system will prefer swap (or zswap) when reclaiming +memory. Enabling :code:`kernel/mm/numa/demotion_enabled` will cause vmscan +to opportunistically prefer distant NUMA nodes to swap or zswap, if capaci= ty +is available. + +Demotion engages the :code:`mm/memory_tier.c` component to determine the +next demotion node. The next demotion node is based on the :code:`HMAT` +or :code:`CDAT` performance data. + +cpusets.mems_allowed quirk +-------------------------- +In Linux v6.15 and below, demotion does not respect :code:`cpusets.mems_al= lowed` +when migrating pages. As a result, if demotion is enabled, vmscan cannot +guarantee isolation of a container's memory from nodes not set in mems_all= owed. + +In Linux v6.XX and up, demotion does attempt to respect +:code:`cpusets.mems_allowed`; however, certain classes of shared memory +originally instantiated by another cgroup (such as common libraries - e.g. +libc) may still be demoted. As a result, the mems_allowed interface still +cannot provide perfect isolation from the remote nodes. + +ZSwap and Node Preference +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D +In Linux v6.15 and below, ZSwap allocates memory from the local node of the +processor for the new pages being compressed. Since pages being compressed +are typically cold, the result is a cold page becomes promoted - only to +be later demoted as it ages off the LRU. + +In Linux v6.XX, ZSwap tries to prefer the node of the page being compressed +as the allocation target for the compression page. This helps prevernt +thrashing. + +Demotion with ZSwap +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +When enabling both Demotion and ZSwap, you create a situation where ZSwap +will prefer the slowest form of CXL memory by default until that tier of +memory is exausted. diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index 2be2855e5870..a768144698e9 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -47,5 +47,6 @@ that have impacts on each other. The docs here break up = configurations steps. =20 allocation/dax allocation/page-allocator + allocation/reclaim =20 .. only:: subproject and html --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 03A972882C1 for ; Wed, 30 Apr 2025 18:11:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036689; cv=none; b=DTzCDtHeRlzX+rbs3vbMG8h+zCszmhLccjRteYAjy9vqav0fDJFmJhYeNoWOzVTeSE6+8TOPwM96ZTFCalQ7MUAtBbCAzQUbds6DdEEaevyXXhXlntuu7B9TFi/14UVTf28SAHE7MPUmhAY3o+zEninrAKUAvHDTCgKnH289UVw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036689; c=relaxed/simple; bh=X5+Ukdg7rvV7208banBi4yu0UKjpHItJx+R1YJoxna4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ds0sOaEbhgFqzm+npAooLtrLC2ZYerEYCqxR6NIX+37HiMb0u4sYMxtfolMiz7IXEkLkGQP/V9OEaNYkW/CBLn55Qfm/NEcsCBpr3+7iyTm4nNtc9e/CC/F9loR0p9PEBTPVOLVqPBcX+aMbUOHPiOb3t4/KUZSenqbnjpj+8Os= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=jYWcxVEs; arc=none smtp.client-ip=209.85.219.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="jYWcxVEs" Received: by mail-qv1-f49.google.com with SMTP id 6a1803df08f44-6e900a7ce55so2816136d6.3 for ; Wed, 30 Apr 2025 11:11:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036687; x=1746641487; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kEWVWC2naWxysqXPj2rk0b5ktFtoGcxHBqlN3kboVV0=; b=jYWcxVEsbDJBnoS5QJ51o+vShg3KBRbOkrQYPMwN3yu+TPIk43b650NfEecd16qZ0c knq6XPHLYzGzGDFy3/bDI76jJDPgJtatt2aQ4GMKb8I0yPYQCLGJQVxOimmjO8Wt9lx0 IPbRwtYDTnI7RD0SpXEEU0QxOdl1zyI41ywrUsZryxS7DXeGBqUWJp+8/wpHEOAEv01S hYtHErphWa6cn8urLprX2ZKFkPXrZs0wyRvcxXgQWWVX4Spfo/Ojs8cVQNNfv3V/Fj0A MF0TbsPWLNiOk4GwLCkjnXVnIgSsni43LrrxIughxbj/MEqhJvFnqHQ5rl1XBTbua7Ad 72Dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036687; x=1746641487; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kEWVWC2naWxysqXPj2rk0b5ktFtoGcxHBqlN3kboVV0=; b=sBjvAOSK3+xwL5+kyaGh5YU4Q6e2RA/8CmunAKTN2vZ3fbbK8D6mxfUlQHPfu+iDoj Mu3UdS4V5j5p/C3D4aZfi3uirNTLGeoEeNhkc3qPTAtzAGJQfgDPiRk7vzUdO4X+uTyG 2k7nHu8Iu4urR6JxUpwezfiN6CeeWW+iUlOR9E1cpQYwjUR7x8yZwijcMK4UkJubfLJ+ tDUmqN6ffRkdGNp6ZIaOeGNPnqf3E7WMVr2wbG+6qqU4KD8XeCLqlEf9pSnYpNl8TTLm q6JCCUXBTy7CwuPDlYDCVfN8o90EyY9+myp6QnzImnYhAJDgsc4DRiIvgnxmuiiAOydA QcXA== X-Forwarded-Encrypted: i=1; AJvYcCUpYs449MEwBazUvsELs7crgZHZ8J3qqBoq+9KeQb1pYT98b9xMufkouVUJXRjFzQF49LfeSPk8r1QFzQA=@vger.kernel.org X-Gm-Message-State: AOJu0YxKui46P9xZmH/yqrMEzkJsXqCX7BAnfKuM0JjaiZHL+oiAgtcV dfUErYfSQ8gJdfyUmfflhSHBhqcOog0umS5QP+NPICzjxAk6CFs+Eg5ao68Udgo= X-Gm-Gg: ASbGncv8ocukz2f2UpSfgevbdCwYLUBz3ljR5ai/chNh5VFJis3+0RfYS49ndvULtLM sxoqqUDYK3Z4B/dcnGcZF1iioPyjYCBgrPLbEv1w/hViy7YAgiAklUGodlXLeMQwpJgRsd0uwbk 8d59T99p/vzXBoOMuCc2DWyYnHXcYol4IZ4HpehCQaDxzSnBPOb+sKBOW0sxwlOnVdzJu4kwNSq rW+TtA5N4JU/wB/fwnkOmArCnSADYP1YANOkSSJv+A6hcMVDlXO466Mm4U0CYzcStVcGY1f3MKz Z49nPfnz6+OzqnzesJslIbJennVSjtHgexHV4Ooor576j4rpQpiHF9dYTbJ67TZc46qNagWY4ns KoTfQLqdrM7k4z0gMGwEdICJMESF6 X-Google-Smtp-Source: AGHT+IG3vL7CDX13v14KAJlzknmgSXzqC6Sf41KMMOAFk0Ub1O74GkEtzw6Jz1Yr0ZgIjWvJ/mA6mA== X-Received: by 2002:a05:6214:d88:b0:6e8:fa72:be4c with SMTP id 6a1803df08f44-6f5090dad94mr4412706d6.1.1746036686871; Wed, 30 Apr 2025 11:11:26 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:26 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 15/18] cxl: docs/allocation/hugepages Date: Wed, 30 Apr 2025 14:10:45 -0400 Message-ID: <20250430181048.1197475-16-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add docs on how CXL capacity interacts with CMA and HugeTLB allocation interfaces. Signed-off-by: Gregory Price --- .../driver-api/cxl/allocation/hugepages.rst | 34 +++++++++++++++++++ Documentation/driver-api/cxl/index.rst | 1 + 2 files changed, 35 insertions(+) create mode 100644 Documentation/driver-api/cxl/allocation/hugepages.rst diff --git a/Documentation/driver-api/cxl/allocation/hugepages.rst b/Docume= ntation/driver-api/cxl/allocation/hugepages.rst new file mode 100644 index 000000000000..279970d73a28 --- /dev/null +++ b/Documentation/driver-api/cxl/allocation/hugepages.rst @@ -0,0 +1,34 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Huge Pages +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Contiguous Memory Allocator +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D +CXL Memory onlined as SystemRAM during early boot is eligible for use by C= MA, +as the NUMA node hosting that capacity will be `Online` at the time CMA +carves out contiguous capacity. + +CXL Memory deferred to the CXL Driver for configuration cannot have its +capacity allocated by CMA - as the NUMA node hosting the capacity is `Offl= ine` +at :code:`__init` time - which CMA carves out contiguous capacity. + +HugeTLB +=3D=3D=3D=3D=3D=3D=3D +HugeTLB capacity is reserved from the page allocator. Only CXL memory onli= ned +as SystemRAM during :code:`__init` and via memory-hotplug can be used via +HugeTLB. + +2MB Huge Pages +-------------- +All CXL capacity regardless of configuration time or memory zone is eligib= le +for use as 2MB huge pages. + +1GB Huge Pages +-------------- +CXL capacity onlined in :code:`ZONE_NORMAL` is eligible for 1GB Gigantic P= age +allocation. + +CXL capacity onlined in :code:`ZONE_MOVABLE` is not eligible for 1GB Gigan= tic +Page allocation. diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index a768144698e9..e58d057a16e3 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -48,5 +48,6 @@ that have impacts on each other. The docs here break up = configurations steps. allocation/dax allocation/page-allocator allocation/reclaim + allocation/hugepages.rst =20 .. only:: subproject and html --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 606592882CD for ; Wed, 30 Apr 2025 18:11:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036691; cv=none; b=CuzQrhO/dumx1WT2mv6OBsR5M2vRppDJLKO8tUaSIieRPjVPSalOggLb2vUdr2GKvnswvsUH0+LBb9XajeF5MNQRxf9bw1bVgGhJyEpe0OBuLnx9LF8EP/MtpJxz2D237/LduzGiH8A6W8jFeSFiR4fGAyZpFHG0vsG4OvfX+Jo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036691; c=relaxed/simple; bh=iirzwEeYoxznZHJe7vcY27yiljb3AwlzwC5WRq52rLc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=D85e0f9TolVXLsuygR92HPh9MOCn1BVZ1mweJefTQhey19JGNHlGsrE+Igjcehf6V7Qomo9ndf5caw6M9Se8+UNbQ+mdJ39BI4J2FNEwTXbB7miGOMwwPS0aoD/kuRl4EF0a/gHuyKG4tl7A5tVOXkUYKjdn2DMb+K9YtF/BMTw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=hrnPfHha; arc=none smtp.client-ip=209.85.160.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="hrnPfHha" Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-4775ce8a4b0so3085271cf.1 for ; Wed, 30 Apr 2025 11:11:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036688; x=1746641488; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A39YiB1Z+l6GCsVmrhqArPxiVLmwBB2VQEqP9tXT4Ig=; b=hrnPfHhaQAzbrxbEaJMbYQ9nSMkwK3qabvTrVWPR51RX9eV6xdN4LMIrJNXRu3aLd+ R/lU/gXZceDxMVeebN9N0tpY6DyD92BBpzVIuAP61GPU+efI/QzI8VkZpZCv/AP3Lh5b OM3n4RituY+VnxqBZDUgD3jEamKW54F8pMzMd6PIhVonpvEhqb7AZJisAsnaLVJowC0Q l3OmCs0rkOV1GZo13/lJCLsdNbJ9RRJlLNlsc0PhUq98X9gx/AdFV2xYiolpTdW/l3bf V2itiyeNxsAwbWhgNPTSJ15yN7myDiwjkpgYLzCXsyOpQ1oq9cUuzogUaAi4y9HUfOUb 9DqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036688; x=1746641488; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A39YiB1Z+l6GCsVmrhqArPxiVLmwBB2VQEqP9tXT4Ig=; b=uVvL1y/1V3JIn79G88+7BtHTSOiVny8Pt7o+luXuGd7IS5cvYlL6POrEmEQ26fMnrZ l9nxTXkvugNBAbSN0sWdi4dtELwovilbV7ZRuyURtVzcXRE7fYvOrap2KJVzgH8gJv3v 4J5dyRy2vnfbfsDhkhL+itQ+R/U4H2tWadTFfaGBS1vfz0JupfVDHzSIOBl0KAfEp17m zsKTumedUscqSWVcQSGbY894fT0Z+VLdZ3Tck50IIiQWSI6fkN5taYfYIjxXtuP5SHF0 YvRdlTc0lodpU6pT2mIsH3xTF3L04x2zKar0wZiokVOAjPa9qSVGg24rd1K+S3QwDmTc 8EkA== X-Forwarded-Encrypted: i=1; AJvYcCXtpsbuk7pWSCc5l3Fdp/OwrdFLwOGBEj59WF+4rdftAlNaC2VLzNLAWumoTS40m6s5B58U5xiZf4jF8hg=@vger.kernel.org X-Gm-Message-State: AOJu0YwLVH/V1GUAh31fMp7zF5wjD5IZoWYxW0JHmcMdfnoHNQ/y7hKi 5FHafJObvHiDrcfQnTrKa0sEf9pd0B6p0mzVi6J93EpTj9M8u9NGlnb5fRRbvPQ= X-Gm-Gg: ASbGncs+Mq37OJbmN0YaW290q1xjpfsi/RmHXvbYcZ2jm8RqjeJNLlDmatQPqbXXCFT cSV7jnq3yml7JXR9D+41dnXChfxXO/rwKv/2gjqgSz8skjM3192n9RIALDJ+Dc01NV8Cs4NqMdO qLW2hQQUTgW72BsKYJA2hcwIEDUFsoinMWb+WNQ+ZjsvrneLTUIra0ZcGhWfR0mbkwsAxNSkEPd 8TTSiKIa0qoM5Y118LtZZief6Jia6POK3n3WHIv8fQaVILhGX+AEWBezFATB/1UQKQBil2I4wak i/lfCxwu5NwnfWBYNO6Adr9aC4ht2tPzDVV0IwIEx4eml1jQTwZijh2qATIoMI+cWv28xTnI+HA 5h/X5TK+Lh4lPCNKohZiFE7Sofg12 X-Google-Smtp-Source: AGHT+IEjj5fRlO+G4AeYsj9CnMTHgu74PV6/AnY9a7Hyw1zOc1l/ean48c9t9b0uKuyRkKfZm4P+pg== X-Received: by 2002:a05:622a:4a17:b0:477:644a:72ec with SMTP id d75a77b69052e-48ae7a20044mr4433871cf.13.1746036688396; Wed, 30 Apr 2025 11:11:28 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:28 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 16/18] cxl: docs/allocation/tiering Date: Wed, 30 Apr 2025 14:10:46 -0400 Message-ID: <20250430181048.1197475-17-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Carve out a space for folks to explain existing tiering mechanisms and how CXL capacity interacts with it. Signed-off-by: Gregory Price --- .../driver-api/cxl/allocation/tiering.rst | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 Documentation/driver-api/cxl/allocation/tiering.rst diff --git a/Documentation/driver-api/cxl/allocation/tiering.rst b/Document= ation/driver-api/cxl/allocation/tiering.rst new file mode 100644 index 000000000000..623bb23088be --- /dev/null +++ b/Documentation/driver-api/cxl/allocation/tiering.rst @@ -0,0 +1,31 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Memory Tiering +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +todo + +Memory Tiers +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +todo + + +Transparent Page Placement +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D + +todo + +Data Access MONitor +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +to be updated + +References +---------- + +- `Self-tuned Memory Tiering RFC prototype and its evaluation `_ +- `SK Hynix HMSDK Capacity Expansion `_ +- `kernel documentation `_ +- `project website `_ --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3F26289E03 for ; Wed, 30 Apr 2025 18:11:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036692; cv=none; b=ahg2UmCGwE5xvgQxvEs4NCgn/+yP5Iy8TVtqRJVCuDM9xnbTfJ2aTCuZKL/UGsZ8ihuZXGNqAik1ca5M7R/n6I6lpn+0FdlVwYMpIViSwOZg4nRTHQASqMKYLAmXVhKuJ+HGbr6SI06roOUksoqzR8/k6edNqRfC9VUuAXiF6O8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036692; c=relaxed/simple; bh=y+zXYdm8lR81Xt8s2hy8d74VfZoLwscWMB6SycFNemM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=up/P3ItIBMku/mB7xxkQ5PlcI8d3jrf5UZ7CB6cvFFm2jnpVOqabIaFasDmJs1gxMp73gbFasIC0b0iqqtakbvffYLjKVqO60dc4pnRdRifL7jWFrZjJOOBm5vbin6VZjbig11C+kOwMHupQI+Y1Nd2srntyZrS9koLhpx6OQFE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=doA73Zbz; arc=none smtp.client-ip=209.85.160.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="doA73Zbz" Received: by mail-qt1-f182.google.com with SMTP id d75a77b69052e-476977848c4so2571331cf.1 for ; Wed, 30 Apr 2025 11:11:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036690; x=1746641490; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZA/uzDtyuFWn0NODL0whOKAEDNrO7IK9AQ6WwWNv0fM=; b=doA73ZbzVUZrFPubkg+KlKO3AfNyAz5rQdhhQi5a6rHopWQGXMvZkoBW4TByp2uzaK 1ighbuIXTaJdYI2BPtYP1Zqe+CpPzocB6YpJsaWADV2s/wXjQYp4/l1qmO45I2HhDLNZ jK5Tu/O+roAeJMzc7XvTo7PFk6mS7sjd648KSouXXZUpmsQLEcVazdqpgPMR04U4wbQX hWo3lH8kUMaPm+OYS9x8aatuRxIUNTv+57lsfIVneaG7dOE7hERz8yMmKqNbLllwSulk ywlLTeAWo6XIQoakdU145GBhwlnUFcDe0J/UgWrfDLHlZ8d7+4tNKd07+FVn7n98xa4X Zu3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036690; x=1746641490; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZA/uzDtyuFWn0NODL0whOKAEDNrO7IK9AQ6WwWNv0fM=; b=TPYu5KMmQS/ghWwdLfrZ9enCrHNGEU/4+IZl/OyJuvGnwS+6bbGj15HSpmmNQDYYA6 7dG5fml6EvVad7HxL00xMiHszQZCxTMogf5qUW6pJuAcGvtzkQGPEUKJdbw95olwQ0UQ 0W0KdeVTGAbagmviMav+X3MWyfIPYNM85xxK02R4lrPpId2T3V61dFZ7gYW227gIWxj+ HViqoFp9WXEFwAbM/mTPon/pApe6UTE5UdnODgrPBRWtwNSVXCx+r1nCn2NnlFL4e3rk haRyabXWn9y1IsZ6yB9NGXKtfEbTvUjaAJ+CsDxuLT7BncR1j2J2YedozIer2NurFP40 wPog== X-Forwarded-Encrypted: i=1; AJvYcCUKxkzRRDXW09Dm2ijGkWlaramgBjzAVakkGpBQMGD4/XChfP/OtPyKwGmJbssvyThVNDtPws+tFvZFGEs=@vger.kernel.org X-Gm-Message-State: AOJu0Yx8JrovEaeZPsq643IIeHe9SokEaYoS2B0vtyVrX+6bo5tBqntt NrGwLTyM2v8Xe1VYOF8W6PsVdd66dlTKbW23mrOPN9qAE8ErYmE/q5epZovZT1U= X-Gm-Gg: ASbGncuSKvYqY6Kk6XQzduQmocGfVhlW2icIbtcSWK6fO+vx2OMzSwiF6/z1KBk0J4K DUcmAQFHQfhwmjbIK7HdrNhS+ljfF5Tdce6ggYtqDhiRhGtBUHGP+ty2v2pp0ktCyHmCvVp46bd Hr2CrI8H633ni8STONN98COzqIg/lA/Kq28/1kxSjeGEQjnRtWs4mQq2y5AAfstkspvqlEJcIYN NT16GVa9iYEULhcXEyF06VxDzNOGZ0YdpvtCTic4RwligAIB4Cfqc3NrmMjiPaSnUC8+Nman2Nh yTYGsmtaik0rZDEeIsqWP2b9F9sW0ajGbi3FqMBmYHhRjlG/sels1D4VJxz8I8D2Pi3FLPczVM6 VbHIk2eGdQXCq9yXTk/OwN0TiJBhY X-Google-Smtp-Source: AGHT+IEv91mZMPKNU0PUGx07QOnNDsFCio0WccLHM/KQVlDxKGCYzZ3aQ4Acw83pDA3zj/uh5YlXmQ== X-Received: by 2002:a05:622a:5c1b:b0:476:71d2:61e0 with SMTP id d75a77b69052e-489c4356885mr70289341cf.23.1746036689813; Wed, 30 Apr 2025 11:11:29 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:29 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 17/18] cxl: docs/use-cases Date: Wed, 30 Apr 2025 14:10:47 -0400 Message-ID: <20250430181048.1197475-18-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add carve-outs for use-case documentation. Complete open as to what (if anything) we should explain here, and/or what the structure should be. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/index.rst | 10 ++++++++++ .../cxl/use-case/dynamic-capacity.rst | 20 +++++++++++++++++++ .../cxl/use-case/memory-expansion.rst | 15 ++++++++++++++ .../driver-api/cxl/use-case/shared-memory.rst | 15 ++++++++++++++ .../cxl/use-case/virtual-machines.rst | 19 ++++++++++++++++++ 5 files changed, 79 insertions(+) create mode 100644 Documentation/driver-api/cxl/use-case/dynamic-capacity.= rst create mode 100644 Documentation/driver-api/cxl/use-case/memory-expansion.= rst create mode 100644 Documentation/driver-api/cxl/use-case/shared-memory.rst create mode 100644 Documentation/driver-api/cxl/use-case/virtual-machines.= rst diff --git a/Documentation/driver-api/cxl/index.rst b/Documentation/driver-= api/cxl/index.rst index e58d057a16e3..956434e23e41 100644 --- a/Documentation/driver-api/cxl/index.rst +++ b/Documentation/driver-api/cxl/index.rst @@ -49,5 +49,15 @@ that have impacts on each other. The docs here break up= configurations steps. allocation/page-allocator allocation/reclaim allocation/hugepages.rst + allocation/tiering.rst + +.. toctree:: + :maxdepth: 1 + :caption: Use Cases + + use-case/memory-expansion.rst + use-case/dynamic-capacity.rst + use-case/virtual-machines.rst + use-case/shared-memory.rst =20 .. only:: subproject and html diff --git a/Documentation/driver-api/cxl/use-case/dynamic-capacity.rst b/D= ocumentation/driver-api/cxl/use-case/dynamic-capacity.rst new file mode 100644 index 000000000000..002f30e6eca9 --- /dev/null +++ b/Documentation/driver-api/cxl/use-case/dynamic-capacity.rst @@ -0,0 +1,20 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Dynamic Capacity +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +todo + +For Virtual Machines +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +todo + +For Workload Orchestration +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D +todo + +For Shared Memory +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +todo + + diff --git a/Documentation/driver-api/cxl/use-case/memory-expansion.rst b/D= ocumentation/driver-api/cxl/use-case/memory-expansion.rst new file mode 100644 index 000000000000..2ebe1d92b556 --- /dev/null +++ b/Documentation/driver-api/cxl/use-case/memory-expansion.rst @@ -0,0 +1,15 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Memory Expansion +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +todo + +As Page Cache +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +todo + +As DAX Device +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +todo diff --git a/Documentation/driver-api/cxl/use-case/shared-memory.rst b/Docu= mentation/driver-api/cxl/use-case/shared-memory.rst new file mode 100644 index 000000000000..3b13b30ddd70 --- /dev/null +++ b/Documentation/driver-api/cxl/use-case/shared-memory.rst @@ -0,0 +1,15 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Shared Memory +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +todo + +Coherence +=3D=3D=3D=3D=3D=3D=3D=3D=3D +todo + +Fabric Attached Memory FileSystem (FAMFS) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +todo diff --git a/Documentation/driver-api/cxl/use-case/virtual-machines.rst b/D= ocumentation/driver-api/cxl/use-case/virtual-machines.rst new file mode 100644 index 000000000000..8fdeccdd5c1c --- /dev/null +++ b/Documentation/driver-api/cxl/use-case/virtual-machines.rst @@ -0,0 +1,19 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Virtual Machines +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +todo + +NUMA Passthrough +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +todo + +Flexible Shapes +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +todo + +Datacenter Efficiency +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +todo --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0E6C28A1DE for ; Wed, 30 Apr 2025 18:11:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036695; cv=none; b=kXsc7007FjzwCZQ7RJSj+3luIySaoRgPKoymzFgTazOVIbeMinQIX11n8jhMldrRJVdpmB4x1f/e1xoWbc54cFFy8np3cphqRmFjDYkauEXALMX9woM8Zw2txLjPXCECMeJFAVFAdSpxHCLpdIudoHYLH1Nc7mTSdJZff+Xj094= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746036695; c=relaxed/simple; bh=C3+NE4GL2t1S6bA4q6zdjnZHIEhMLXG6A09ekqGmY6c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kmMyfKoXBUrFDrqysDJtMG/yxeWQ92FvhmuftEfNK/N9Adi86huPxBQkAr5DS7Lfk55kEVQk6De7ImPVj4BEKjdr4PsFDd5N9R2Zlxv9oT0BzSEJfDcDLlGTSbliJ/isXZ9cCo09cosDFa4v+TwIJT0TcaZ3AfO4niZC3ttn2MI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=k6Pnr9W3; arc=none smtp.client-ip=209.85.219.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="k6Pnr9W3" Received: by mail-qv1-f43.google.com with SMTP id 6a1803df08f44-6ed0cc5eca4so15072206d6.1 for ; Wed, 30 Apr 2025 11:11:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746036691; x=1746641491; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Gs5zk7qO3i+Gy6m+o2xD6HywWGNnomysRu1myabmsGg=; b=k6Pnr9W3hvAn7BrJ0iaBuNnOn5lCW2ACNFQRqofc7Vtg67r6ypbM7X0oxDGafVP95k QZStc6HIB5VfIFBfiK/TT6BR1/4Ozy/ZzgAqbQ68h65B/HZil63jlcv5dVJZpqrqzHqB dA9rnq8dAMyO0f8O/d3pLNVGgj3NBXUdjyJoiBmGjUuXhdPXk7oWmFtJi5S9KkeuPSnf 7d6QEOroY4ho4otDxRfCzniGIKzr8Ers4A1hQoxpmKymj4fEJj54LDXRa0fqcUbzlB6I ZYfOZr8LZnDo9mE7uNe1cUGlRqFGEwAljWcdfRm0BK6e9iu4bVVpa2OYHytpwu98UrNM 6YTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746036691; x=1746641491; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Gs5zk7qO3i+Gy6m+o2xD6HywWGNnomysRu1myabmsGg=; b=WorTJZVvniLjOCkvrBMUWytC5DS70l7htgM4NHBlwfomtC5gcRTaoyTZ6faM4yxbi8 baBuJLnNvM4S0RusyNBiSZIPPe1x6jjFhrjrfZHGYR3iR8VexxsZpfJ/vBJB3gFbG+lU oJURYVialr5+UhWEIK4FwA+wubuxFMHJ9+M1ntK9g3ECf7AriZca+8TLdq7GelWoH4d2 bCyy5kCbXz+MYRutXv96t29YQbxav8Qa3FVKYHXniXMXPjKzaxccNE51WAhiDceJCOPu PTBta5DM3n4/NxqyeQ8mXUG2OcviKuMbRPhQpL/NOyqzq9pNtQGion3h2WGbnthI2+Db ZZUQ== X-Forwarded-Encrypted: i=1; AJvYcCVVs7LqHIduA6s/9rJAs6Y3prwR+GSZaVUexyR5eCp9yBNjhl1oTt3nbJG5t0yyjMFt81+w2pOYMYPQPnU=@vger.kernel.org X-Gm-Message-State: AOJu0YyLKhZqZ+CiGjEZNb3WMRLv4+RHXVtxHEzkr9kL03q5TeKQUxA0 27HhzNgnAO1tYflPsWhEZ6QFW0xjUuWVv/sGGT87K49fS7xEanJuElErbmtoZAg= X-Gm-Gg: ASbGncuzfruhQmzUmBe8IjGAdozFJyaFXAR0hIdCJO/gRXuRgss4fncgT3IPM/dKC2h Ia7yGScxj9FGKEOWrusoP+xH1KaS3d5r3a5grZfwFWfx0vffGWiH6Oz7letsvlvFjUDQ3AkXl9x poZaHv/GTy4SNXRxU/uzlDSsc8K3cA47oJL2De4tB5bL9pESf+dVr8RH+aqYRRrFzV00MlA1w7x 2dCK0YsYwO9ly2lvbe3mRywNanNSXHm/hePCsyqT167me0WGPVFItV6SxJLf+3IsHq5VJ+8gy6+ BBCNsg+GwiVVbHw2tUKcIDak/v0EBQx73cFV/yjqffqaJ3O/H4V/XqcuR/KQbJJQdfoVj7pzps7 /hU2EvgkZkfTwMk/uB1qO2in1JS6y X-Google-Smtp-Source: AGHT+IHvKw+43hjj7TQq+F9B1nQszjFX2xfpktZG1aDblkxQcdp18PoqTjnDcTGLxNGttqu63c1aFg== X-Received: by 2002:ad4:5ca5:0:b0:6f0:e2e1:3f7b with SMTP id 6a1803df08f44-6f508c6bcc0mr7245106d6.3.1746036691378; Wed, 30 Apr 2025 11:11:31 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-48a64fe04efsm5897421cf.19.2025.04.30.11.11.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 11:11:31 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [RFC PATCH v2 18/18] cxl: docs - add self-referencing cross-links Date: Wed, 30 Apr 2025 14:10:48 -0400 Message-ID: <20250430181048.1197475-19-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add some crosslinks between pages in the CXL docs - mostly to the ACPI tables. Signed-off-by: Gregory Price --- .../driver-api/cxl/devices/device-types.rst | 2 +- .../cxl/linux/access-coordinates.rst | 8 +++-- .../driver-api/cxl/linux/cxl-driver.rst | 36 ++++++++++--------- .../driver-api/cxl/linux/early-boot.rst | 30 +++++++++------- .../driver-api/cxl/platform/bios-and-efi.rst | 13 +++---- .../example-configurations/flexible.rst | 10 +++--- .../example-configurations/hb-interleave.rst | 10 +++--- .../multi-dev-per-hb.rst | 10 +++--- .../example-configurations/one-dev-per-hb.rst | 10 +++--- 9 files changed, 71 insertions(+), 58 deletions(-) diff --git a/Documentation/driver-api/cxl/devices/device-types.rst b/Docume= ntation/driver-api/cxl/devices/device-types.rst index dfe8d4711987..f33a40d2284d 100644 --- a/Documentation/driver-api/cxl/devices/device-types.rst +++ b/Documentation/driver-api/cxl/devices/device-types.rst @@ -115,7 +115,7 @@ A Multi-Headed Single-Logical Device (MHSLD) exposes a = single logical device to multiple heads which may be connected to one or more discrete hosts. An example of this would be a simple memory-pool which may be statically configured (prior to boot) to expose portions of its memory -to Linux via the CEDT ACPI table. +to Linux via Documentation/driver-api/cxl/platform/acpi/cedt.rst =20 MHMLD ~~~~~ diff --git a/Documentation/driver-api/cxl/linux/access-coordinates.rst b/Do= cumentation/driver-api/cxl/linux/access-coordinates.rst index b07950ea30c9..24db5b41716a 100644 --- a/Documentation/driver-api/cxl/linux/access-coordinates.rst +++ b/Documentation/driver-api/cxl/linux/access-coordinates.rst @@ -24,7 +24,8 @@ asymmetry in properties does not happen and all paths to = EPs are equal. =20 There can be multiple switches under an RP. There can be multiple RPs under a CXL Host Bridge (HB). There can be multiple HBs under a CXL Fixed Memory -Window Structure (CFMWS). +Window Structure (CFMWS) in the +Documentation/driver-api/cxl/platform/acpi/acpi/cedt.rst. =20 An example hierarchy: =20 @@ -83,8 +84,9 @@ also the index for the resulting xarray. =20 The next step is to take the min() of the per host bridge bandwidth and the bandwidth from the Generic Port (GP). The bandwidths for the GP is retriev= ed -via ACPI tables SRAT/HMAT. The min bandwidth are aggregated under the same -ACPI0017 device to form a new xarray. +via ACPI tables Documentation/driver-api/cxl/platform/acpi/srat.rst and +Documentation/driver-api/cxl/platform/acpi/hmat.rst. The min bandwidth are +aggregated under the same ACPI0017 device to form a new xarray. =20 Finally, the cxl_region_update_bandwidth() is called and the aggregated bandwidth from all the members of the last xarray is updated for the diff --git a/Documentation/driver-api/cxl/linux/cxl-driver.rst b/Documentat= ion/driver-api/cxl/linux/cxl-driver.rst index 1a354ea1cda4..b4cb1910adde 100644 --- a/Documentation/driver-api/cxl/linux/cxl-driver.rst +++ b/Documentation/driver-api/cxl/linux/cxl-driver.rst @@ -77,11 +77,11 @@ Root Object` Device Class is found. =20 The Root contains links to: =20 -* `Host Bridge Ports` defined by ACPI CEDT CHBS. +* `Host Bridge Ports` defined by CHBS in Documentation/driver-api/cxl/plat= form/acpi/cedt.rst. =20 * `Downstream Ports` typically connected to `Host Bridge Ports` =20 -* `Root Decoders` defined by ACPI CEDT CFMWS. +* `Root Decoders` defined by CFMWS in Documentation/driver-api/cxl/platfor= m/acpi/cedt.rst. =20 :: =20 @@ -150,9 +150,9 @@ An `endpoint` is a terminal port in the fabric. This i= s a `logical device`, and may be one of many `logical devices` presented by a memory device. It is still considered a type of `port` in the fabric. =20 -An `endpoint` contains `endpoint decoders` available for use and the -*Coherent Device Attribute Table* (CDAT) used to describe the capabilities -of the device. :: +An `endpoint` contains `endpoint decoders` available for use and the CDAT = in +Device's Documentation/driver-api/cxl/devices/uefi.rst used to describe the +capabilities of the device. :: =20 # ls /sys/bus/cxl/devices/endpoint5 CDAT decoders_committed modalias uevent @@ -247,17 +247,18 @@ parameter. Root Decoder ~~~~~~~~~~~~ A `Root Decoder` is logical construct of the physical address and interlea= ve -configurations present in the ACPI CEDT CFMWS. Linux presents this inform= ation -as a decoder present in the `CXL Root`. We consider this a `Root Decoder`, -though technically it exists on the boundary of the CXL specification and -platform-specific CXL root implementations. +configurations present in the CFMWS field of the +Documentation/driver-api/cxl/platform/acpi/cedt.rst. +Linux presents this information as a decoder present in the `CXL Root`. We +consider this a `Root Decoder`, though technically it exists on the bounda= ry +of the CXL specification and platform-specific CXL root implementations. =20 Linux considers these logical decoders a type of `Routing Decoder`, and is= the first decoder in the CXL fabric to receive a memory access from the platfo= rm's memory controllers. =20 `Root Decoders` are created during :code:`cxl_acpi_probe`. One root decod= er -is created per CFMWS entry in the ACPI CEDT. +is created per CFMWS entry in the Documentation/driver-api/cxl/platform/ac= pi/cedt.rst. =20 The :code:`target_list` parameter is filled by the CFMWS target fields. Ta= rgets of a root decoder are `Host Bridges`, which means interleave done at the r= oot @@ -267,9 +268,11 @@ Only root decoders are capable of `Inter-Host-Bridge I= nterleave`. =20 Such interleaves must be configured by the platform and described in the A= CPI CEDT CFMWS, as the target CXL host bridge UIDs in the CFMWS must match the= CXL -host bridge UIDs in the ACPI CEDT CHBS and ACPI DSDT. +host bridge UIDs in the CHBS field of the +Documentation/driver-api/cxl/platform/acpi/cedt.rst and the UID field of C= XL +Host Bridges defined in the Documentation/driver-api/cxl/platform/acpi/dsd= t.rst. =20 -Interleave settings in a rootdecoder describe how to interleave accesses a= mong +Interleave settings in a root decoder describe how to interleave accesses = among the *immediate downstream targets*, not the entire interleave set. =20 The memory range described in the root decoder is used to @@ -531,10 +534,11 @@ granularity configuration. =20 At Root ~~~~~~~ -Root decoder interleave is defined by the ACPI CEDT CFMWS. The CEDT -may actually define multiple CFMWS configurations to describe the same -physical capacity - with the intent to allow users to decide at runtime -whether to online memory as interleaved or non-interleaved. :: +Root decoder interleave is defined by CFMWS field of the +Documentation/driver-api/cxl/platform/acpi/cedt.rst. The CEDT may actually +define multiple CFMWS configurations to describe the same physical capacit= y, +with the intent to allow users to decide at runtime whether to online memo= ry +as interleaved or non-interleaved. :: =20 Subtable Type : 01 [CXL Fixed Memory Window Structure] Window base address : 0000000100000000 diff --git a/Documentation/driver-api/cxl/linux/early-boot.rst b/Documentat= ion/driver-api/cxl/linux/early-boot.rst index 275174d5b0bb..309cc6999c6b 100644 --- a/Documentation/driver-api/cxl/linux/early-boot.rst +++ b/Documentation/driver-api/cxl/linux/early-boot.rst @@ -12,7 +12,8 @@ read EFI and ACPI information throughout this process to = configure logical representations of the devices. =20 During Linux Early Boot stage (functions in the kernel that have the __init -decorator), the system takes the resources created by EFI/BIOS (ACPI table= s) +decorator), the system takes the resources created by EFI/BIOS +(Documentation/driver-api/cxl/platform/acpi.rst) and turns them into resources that the kernel can consume. =20 =20 @@ -69,13 +70,15 @@ significant impact performance depending on the memory = capacity of the system. NUMA Node Reservation =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 -Linux refers to the proximity domains (:code:`PXM`) defined in the SRAT to -create NUMA nodes in :code:`acpi_numa_init`. Typically, there is a 1:1 rel= ation -between :code:`PXM` and NUMA node IDs. +Linux refers to the proximity domains (:code:`PXM`) defined in the +Documentation/driver-api/cxl/platform/acpi/srat.rst to create NUMA nodes in +:code:`acpi_numa_init`. Typically, there is a 1:1 relation between +:code:`PXM` and NUMA node IDs. =20 -SRAT is the only ACPI defined way of defining Proximity Domains. Linux cho= oses -to, at most, map those 1:1 with NUMA nodes. CEDT adds a description of SPA -ranges which Linux may wish to map to one or more NUMA nodes +The SRAT is the only ACPI defined way of defining Proximity Domains. Linux +chooses to, at most, map those 1:1 with NUMA nodes. +Documentation/driver-api/cxl/platform/acpi/cedt.rst +adds a description of SPA ranges which Linux may map to one or more NUMA n= odes =20 If there are CXL ranges in the CFMWS but not in SRAT, then a fake :code:`P= XM` is created (as of v6.15). In the future, Linux may reject CFMWS not descri= bed @@ -88,7 +91,7 @@ data for Linux to identify NUMA nodes their associated me= mory regions. =20 The relevant code exists in: :code:`linux/drivers/acpi/numa/srat.c`. =20 -See the Example Platform Configurations section for more information. +See Documentation/driver-api/cxl/platform/example-configs.rst for more inf= o. =20 Memory Tiers Creation =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D @@ -107,10 +110,13 @@ Tier membership can be inspected in :: /sys/devices/virtual/memory_tiering/memory_tierN/nodelist 0-1 =20 -If nodes are grouped which have clear difference in performance, check the= HMAT -and CDAT information for the CXL nodes. All nodes default to the DRAM tie= r, -unless HMAT/CDAT information is reported to the memory_tier component via -`access_coordinates`. +If nodes are grouped which have clear difference in performance, check the +Documentation/driver-api/cxl/platform/acpi/hmat.rst and CDAT +(Documentation/driver-api/cxl/devices/uefi.rst) information for the CXL no= des. +All nodes default to the DRAM tier, unless HMAT/CDAT information is report= ed +to the memory_tier component via `access_coordinates`. + +For more, see Documentation/driver-api/cxl/linux/access-coordinates.rst. =20 Contiguous Memory Allocation =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D diff --git a/Documentation/driver-api/cxl/platform/bios-and-efi.rst b/Docum= entation/driver-api/cxl/platform/bios-and-efi.rst index 552a83992bcc..cdbba6798079 100644 --- a/Documentation/driver-api/cxl/platform/bios-and-efi.rst +++ b/Documentation/driver-api/cxl/platform/bios-and-efi.rst @@ -22,7 +22,7 @@ At a high level, this is what occurs during this phase of= configuration. =20 Much of what this section is concerned with is ACPI Table production and static memory map configuration. More detail on these tables can be found -under Platform Configuration -> ACPI Table Reference. +at Documentation/driver-api/cxl/platform/acpi.rst. =20 .. note:: Platform Vendors should read carefully, as this sections has recommenda= tions @@ -175,9 +175,10 @@ to implement driver support for your platform. =20 Interleave and Configuration Flexibility ---------------------------------------- -If providing cross-host-bridge interleave, a CFMWS entry in the CEDT must = be -presented with target host-bridges for the interleaved device sets (there = may -be multiple behind each host bridge). +If providing cross-host-bridge interleave, a CFMWS entry in the +Documentation/driver-api/cxl/platform/acpi/cedt.rst +must be presented with target host-bridges for the interleaved device sets +(there may be multiple behind each host bridge). =20 If providing intra-host-bridge interleaving, only 1 CFMWS entry in the CED= T is required for that host bridge - if it covers the entire capacity of the de= vices @@ -193,8 +194,8 @@ different purposes. For example, you may want to consi= der adding: =20 A platform may choose to add all of these, or change the mode based on a B= IOS setting. For each CFMWS entry, Linux expects descriptions of the described -memory regions in the SRAT to determine the number of NUMA nodes it should -reserve during early boot / init. +memory regions in the Documentation/driver-api/cxl/platform/acpi/srat.rst = to +determine the number of NUMA nodes it should reserve during early boot / i= nit. =20 As of v6.14, Linux will create a NUMA node for each CEDT CFMWS entry, even= if a matching SRAT entry does not exist; however, this is not guaranteed in t= he diff --git a/Documentation/driver-api/cxl/platform/example-configurations/f= lexible.rst b/Documentation/driver-api/cxl/platform/example-configurations/= flexible.rst index 13a97c03e25a..b2559d2de225 100644 --- a/Documentation/driver-api/cxl/platform/example-configurations/flexible= .rst +++ b/Documentation/driver-api/cxl/platform/example-configurations/flexible= .rst @@ -18,7 +18,7 @@ Things to note: * This SRAT describes one-node for each of the above CFMWS. * The HMAT describes performance for each node in the SRAT. =20 -CEDT :: +Documentation/driver-api/cxl/platform/acpi/cedt.rst :: =20 Subtable Type : 00 [CXL Host Bridge Structure] Reserved : 00 @@ -137,7 +137,7 @@ CEDT :: QtgId : 0001 First Target : 00000006 =20 -SRAT :: +Documentation/driver-api/cxl/platform/acpi/srat.rst :: =20 Subtable Type : 01 [Memory Affinity] Length : 28 @@ -223,7 +223,7 @@ SRAT :: Hot Pluggable : 1 Non-Volatile : 0 =20 -HMAT :: +Documentation/driver-api/cxl/platform/acpi/hmat.rst :: =20 Structure Type : 0001 [SLLBI] Data Type : 00 [Latency] @@ -263,7 +263,7 @@ HMAT :: Entry : 0100 Entry : 0100 =20 -SLIT :: +Documentation/driver-api/cxl/platform/acpi/slit.rst :: =20 Signature : "SLIT" [System Locality Information Table] Localities : 0000000000000003 @@ -276,7 +276,7 @@ SLIT :: Locality 6 : FF FF FF FF FF FF 0A FF Locality 7 : FF FF FF FF FF FF FF 0A =20 -DSDT :: +Documentation/driver-api/cxl/platform/acpi/dsdt.rst :: =20 Scope (_SB) { diff --git a/Documentation/driver-api/cxl/platform/example-configurations/h= b-interleave.rst b/Documentation/driver-api/cxl/platform/example-configurat= ions/hb-interleave.rst index fa0885d82deb..9cbf3dd44b0f 100644 --- a/Documentation/driver-api/cxl/platform/example-configurations/hb-inter= leave.rst +++ b/Documentation/driver-api/cxl/platform/example-configurations/hb-inter= leave.rst @@ -13,7 +13,7 @@ Things to note: * This SRAT describes one-node for both host bridges. * The HMAT describes a single node's performance. =20 -CEDT :: +Documentation/driver-api/cxl/platform/acpi/cedt.rst :: =20 Subtable Type : 00 [CXL Host Bridge Structure] Reserved : 00 @@ -48,7 +48,7 @@ CEDT :: First Target : 00000007 Second Target : 00000006 =20 -SRAT :: +Documentation/driver-api/cxl/platform/acpi/srat.rst :: =20 Subtable Type : 01 [Memory Affinity] Length : 28 @@ -62,7 +62,7 @@ SRAT :: Hot Pluggable : 1 Non-Volatile : 0 =20 -HMAT :: +Documentation/driver-api/cxl/platform/acpi/hmat.rst :: =20 Structure Type : 0001 [SLLBI] Data Type : 00 [Latency] @@ -80,14 +80,14 @@ HMAT :: Entry : 1200 Entry : 0400 =20 -SLIT :: +Documentation/driver-api/cxl/platform/acpi/slit.rst :: =20 Signature : "SLIT" [System Locality Information Table] Localities : 0000000000000003 Locality 0 : 10 20 Locality 1 : FF 0A =20 -DSDT :: +Documentation/driver-api/cxl/platform/acpi/dsdt.rst :: =20 Scope (_SB) { diff --git a/Documentation/driver-api/cxl/platform/example-configurations/m= ulti-dev-per-hb.rst b/Documentation/driver-api/cxl/platform/example-configu= rations/multi-dev-per-hb.rst index 6adf7c639490..fa24243968ac 100644 --- a/Documentation/driver-api/cxl/platform/example-configurations/multi-de= v-per-hb.rst +++ b/Documentation/driver-api/cxl/platform/example-configurations/multi-de= v-per-hb.rst @@ -14,7 +14,7 @@ Things to note: * This CEDT/SRAT describes one node for both devices. * There is only one proximity domain the HMAT for both devices. =20 -CEDT :: +Documentation/driver-api/cxl/platform/acpi/cedt.rst :: =20 Subtable Type : 00 [CXL Host Bridge Structure] Reserved : 00 @@ -39,7 +39,7 @@ CEDT :: QtgId : 0001 First Target : 00000007 =20 -SRAT :: +Documentation/driver-api/cxl/platform/acpi/srat.rst :: =20 Subtable Type : 01 [Memory Affinity] Length : 28 @@ -53,7 +53,7 @@ SRAT :: Hot Pluggable : 1 Non-Volatile : 0 =20 -HMAT :: +Documentation/driver-api/cxl/platform/acpi/hmat.rst :: =20 Structure Type : 0001 [SLLBI] Data Type : 00 [Latency] @@ -69,14 +69,14 @@ HMAT :: Entry : 1200 Entry : 0200 =20 -SLIT :: +Documentation/driver-api/cxl/platform/acpi/slit.rst :: =20 Signature : "SLIT" [System Locality Information Table] Localities : 0000000000000003 Locality 0 : 10 20 Locality 1 : FF 0A =20 -DSDT :: +Documentation/driver-api/cxl/platform/acpi/dsdt.rst :: =20 Scope (_SB) { diff --git a/Documentation/driver-api/cxl/platform/example-configurations/o= ne-dev-per-hb.rst b/Documentation/driver-api/cxl/platform/example-configura= tions/one-dev-per-hb.rst index 8b732dc8c5b6..ee65b3364c5b 100644 --- a/Documentation/driver-api/cxl/platform/example-configurations/one-dev-= per-hb.rst +++ b/Documentation/driver-api/cxl/platform/example-configurations/one-dev-= per-hb.rst @@ -14,7 +14,7 @@ Things to note: * This CEDT/SRAT describes one-node per device * The expanders have the same performance and will be in the same memory t= ier. =20 -CEDT :: +Documentation/driver-api/cxl/platform/acpi/cedt.rst :: =20 Subtable Type : 00 [CXL Host Bridge Structure] Reserved : 00 @@ -62,7 +62,7 @@ CEDT :: QtgId : 0001 First Target : 00000006 =20 -SRAT :: +Documentation/driver-api/cxl/platform/acpi/srat.rst :: =20 Subtable Type : 01 [Memory Affinity] Length : 28 @@ -88,7 +88,7 @@ SRAT :: Hot Pluggable : 1 Non-Volatile : 0 =20 -HMAT :: +Documentation/driver-api/cxl/platform/acpi/hmat.rst :: =20 Structure Type : 0001 [SLLBI] Data Type : 00 [Latency] @@ -108,7 +108,7 @@ HMAT :: Entry : 0200 Entry : 0200 =20 -SLIT :: +Documentation/driver-api/cxl/platform/acpi/slit.rst :: =20 Signature : "SLIT" [System Locality Information Table] Localities : 0000000000000003 @@ -116,7 +116,7 @@ SLIT :: Locality 1 : FF 0A FF Locality 2 : FF FF 0A =20 -DSDT :: +Documentation/driver-api/cxl/platform/acpi/dsdt.rst :: =20 Scope (_SB) { --=20 2.49.0 From nobody Sun Feb 8 18:08:46 2026 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7BAE221FCF for ; Wed, 30 Apr 2025 23:15:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746054919; cv=none; b=oPpkh2d3byQXRsMXyI7CzOlz9+xgsqBFXRvTcPI5JbIz1WYI9ouvPyeoEtBBcAS1CioxuXKl9wPS+9NRrpiFJ213CXDrqXvo/xGmC+XDzkhsRIYtGtH6nB7gRD+jNk8pThkbkjzOmdNN/+lZBv+cs4H7Ug+XBHjh7XDGD3lL79U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746054919; c=relaxed/simple; bh=1gkvdFh4jhIrw/MBMXml2s/AX8PO3PV1u4aAhoqRtV0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Mx4Ur7cZxFFV2WlBVYC0o31vV4WTr1xLvenUq3HpKb8Eah4GU8zxYXKQ4b0U8UaE2B/mvjdRUelnZU2ytSfAmS9umfkM6Wx6M13J5DLfqaJMHOld1U+IiXYmixN/WOuQhJvKfM+fBUF9xbXffw1Czcy743tK/TdS0i9wB9vIatg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=UY324YwQ; arc=none smtp.client-ip=209.85.222.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="UY324YwQ" Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7c5b2472969so36235585a.1 for ; Wed, 30 Apr 2025 16:15:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1746054914; x=1746659714; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4qevg+gR/jQXOKEn18MBQyQWDOgs+/lqx3wMxZN8WKs=; b=UY324YwQkpa5PE8XyPxuV929zu5EBhgSwBbl9DRzLcDutA0BgRJz5wSR84Fqmi1uR9 uZ+ExucShUBpu+MfPfauehGPNnUxSAnr0KAo6m3Bse8LFJJZkpnij13ljZG2FbnsLcP8 M7o9QKcQynqnws0eEUU+rDdz/qyhXaxBHIAGhZ6KrIYmUK8llHIHawO5sjJ4QUUN35pY BeEJv1FQLJ0wb5ggUT6/stMqzOqs8+Fpy/tzOIywQA7qbfwzjaFZWCdPtkVlEKWHjelL jxDSNK18CHs9aqu/OaYHp3mFxeoos08NsZoixGvJv0LkQjfv2CL8NXP87r1wstJ+l5ui MHog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746054914; x=1746659714; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4qevg+gR/jQXOKEn18MBQyQWDOgs+/lqx3wMxZN8WKs=; b=lomfuxEFDFlZoperBMKjKLXt9MRkvHA7B05ooAcuafG5uTYKLC+QGSkAGwmFzgp329 aFD4Gi1SgJlQ+ZkVfwQF9XA6CC1LR580z8WU+/GRwcgjkt1bdyjdM1yl357b0Vm7jhBT 9Aun44QLoUR/+kuzq89+fydyhu+h2Ed3yBH54EgY4w6HTXmXLqZfpZFksHrO+3cuiKrF ZhRSSe9bv/N1Mlw8DWw+peI5EFQdzWrD5mAxj/eiWkHlu84A9Q4BTY4yZysP6Oa0/ql+ VkmZU2Bw1eHTT13dRme0yoABTTlXEb/BQf06pztDD3IefyXn8OqAd5LIs2og1m1J+Gsl UMXQ== X-Forwarded-Encrypted: i=1; AJvYcCVtHnrBbzJ0EDx+QCFJdmkYTbnPM/EQT/+t6VW9dE0cN9i5hi4sY0KBCcmr0bkJH/6zz9FoMn7dMJHaNAY=@vger.kernel.org X-Gm-Message-State: AOJu0YyVYdmEOSd6n0z5iLX4wAw0SRetXP+0QrhnXitPUNa2gqDFOXrQ oM6RjpsNiU7C+vhIh/mJq/W7MZmyHhqJREcS6pLoFiumsyK2S5Gr5Xpfqh3zkks= X-Gm-Gg: ASbGnctSdae1gajMQTfKzkzJwU7GDQdg19k5mWtHxE8Nw2awPWo1BQfsu44jahZ2FKZ tC3FaBq5yU4A0joQ8lt5JgldGCvzrVDiuW0+n0zrH7GodhOzgLwI/oNgj5x3JDks1ac4/WdefJ3 P9HA4CWW/WwxNC772TkubTjpSJ3qdkYlsKfgtcs2qucSuO4kE4hAQWicnWxpOktBmrHTevp2xaS EfJR81IoiIT9f5s+0tcPke6MouZ/HyXKYLx15XC7aBpHbaNjV+fESu/AjYEg/Y8kp4B18W64hsO sySafU0HeQGMV0bUEw7OX11II0DFXvvvcxD+4dYW+/CckNE8b7/XttWhAFbwEPiOzNXv8UEyIAI +5+ypm2I8tLVoGuyOj3ae2/pGjtTu X-Google-Smtp-Source: AGHT+IGav1geRgSf7A2vLf5AOwcdw9lUMaP9D7RnYvq+QFvpuCaWOqmOKlfgU51P0IinAgHBqb/HUg== X-Received: by 2002:a05:620a:6183:b0:7c5:5596:8457 with SMTP id af79cd13be357-7cac76d7ab2mr754502485a.57.1746054914481; Wed, 30 Apr 2025 16:15:14 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7cacde764c3sm19538485a.33.2025.04.30.16.15.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Apr 2025 16:15:14 -0700 (PDT) From: Gregory Price To: linux-cxl@vger.kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave@stgolabs.net, jonathan.cameron@huawei.com, dave.jiang@intel.com, alison.schofield@intel.com, vishal.l.verma@intel.com, ira.weiny@intel.com, dan.j.williams@intel.com, corbet@lwn.net Subject: [PATCH FIXUP] cxl: docs - fixup cedt.rst reference in access-coordinates Date: Wed, 30 Apr 2025 19:15:09 -0400 Message-ID: <20250430231509.1298122-1-gourry@gourry.net> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250430181048.1197475-1-gourry@gourry.net> References: <20250430181048.1197475-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove additional acpi/ in path to cedt.rst. Signed-off-by: Gregory Price --- Documentation/driver-api/cxl/linux/access-coordinates.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/driver-api/cxl/linux/access-coordinates.rst b/Do= cumentation/driver-api/cxl/linux/access-coordinates.rst index 24db5b41716a..77ba9be9af2d 100644 --- a/Documentation/driver-api/cxl/linux/access-coordinates.rst +++ b/Documentation/driver-api/cxl/linux/access-coordinates.rst @@ -25,7 +25,7 @@ asymmetry in properties does not happen and all paths to = EPs are equal. There can be multiple switches under an RP. There can be multiple RPs under a CXL Host Bridge (HB). There can be multiple HBs under a CXL Fixed Memory Window Structure (CFMWS) in the -Documentation/driver-api/cxl/platform/acpi/acpi/cedt.rst. +Documentation/driver-api/cxl/platform/acpi/cedt.rst. =20 An example hierarchy: =20 --=20 2.49.0