| |
Hi all,
This RFC proposes a new approach to handle the resource conflict between CXL Fixed Memory Windows (CFMW) and "Soft Reserved" memory regions.
I've been thinking about the "Soft Reserved" issue lately and went through the previous proposals [0]. The existing solutions are generally centered around CXL region management. For instance, an early proposal [1] suggested removing the "Soft Reserved" resource during region teardown. The current approach [2] has evolved to remove it after the CXL driver automatically constructs a CXL region.
I haven't found prior discussions centered on removing the "Soft Reserved" region *before* the CFMW resource is even added. If this has been discussed before, please point me to the thread.
My proposal is to remove the "Soft Reserved" resource at the moment we add the root decoder resource (the CFMW) from `CXL_ACPI`.
Here’s why I believe this is feasible and more effective.
#### Background
Currently, CXL memory can be exposed to the users in several ways(1)(2)(3), depending on firmware and driver configurations. The diagram below illustrates the flow:
```
+-----------------------+
| |
| CXL memory |
| |
+----------+------------+ +----------------------+
| | |
| | |
+firmware---------------------+ | +---------v---------+ +--------------+
| v | | | | NO | |
| +----------+-------------+ | | | enable CXL_ACPI? +---------> HMEM |
| | | | | | | | |
| | expose to E820? | | | +-------------------+ +-------+------+
| | | | | |YES |
| +----------+-------------+ | | | |
| | | | +---------v----------+ |
| | YES | | | iomem | |
| v | | | CXL WindowN | |
+-(1)-----------+ | +-----------+-------------+ | | | (4) | |
| iomem | NO | | set | | | +---------+----------+ |
| System Ram +<--------+ EFI_MEMORY_SP attr? | | | | |
| | | | | | | | |
+---------------+ | +-------------------------+ | | +---------v----------+ +-------v--------+
+-----------------------------+ | | | | |
| YES | | CXL region +--------> (2) dax/kmem |
v | | | | (3) device_dax |
+-----------+-------------+ | +--------------------+ +----------------+
| iomem | |
| Soft Reserved +--------+
| |
+-------------------------+
```
The problem we are facing occurs in path **(4)**, where the ACPI-defined `CXL WindowN` (CFMW) overlaps with a `Soft Reserved` region. In this scenario, if we try to destroy the driver-created CXL region to create a new one, the operation fails because the underlying memory range is still claimed by "Soft Reserved".
Here is an example from ` /proc/iomem `:
```
c070000000-fcffffffff : CXL Window 0
c070000000-fcffffffff : Soft Reserved
c070000000-fcffffffff : region0 ### Deleting this will not free the range for a new region
c070000000-fcffffffff : dax0.0
c070000000-fcffffffff : System RAM (kmem)
```
#### Proposal
The fundamental assumption of the CXL driver design appears to be (and I believe this is correct) that once the kernel successfully parses `CEDT.CFMWS` via `CXL_ACPI`, the CXL subsystem is designated to take full control of the corresponding CXL memory device.
If this holds true, it means that the "Soft Reserved" region, which serves as a firmware-level hint to prevent the OS from arbitrarily using this memory, is no longer necessary after the CFMW is identified. The CFMW itself becomes the authoritative definition for this memory range.
Therefore, we can safely remove the "Soft Reserved" resource from the `iomem_resource` tree right before inserting the new CFMW resource.
This approach is simple and highly effective. It decouples the "Soft Reserved" problem from CXL region management entirely. By cleaning up the resource conflict at the earliest possible stage, this solution should be compatible with all CXL region patterns, including scenarios with misconfigured or unconfigured HDM Decoders.
I haven't spent much time working out all the implementation details yet, but a quick proof-of-concept (PoC) shows that this approach appears to work. A rough sketch of the code is below. If this direction is ACKed, I will prepare and send out a complete implementation for review.
```diff
--- a/drivers/cxl/acpi.c
+++ b/drivers/cxl/acpi.c
@@ -780,7 +783,9 @@ static int add_cxl_resources(struct resource *cxl_res)
*/
cxl_set_public_resource(res, new);
- insert_resource_expand_to_fit(&iomem_resource, new);
+ /* Remove Soft Reserved that is fully covered by this window */
+ claim_and_punch_out_soft_reserved(&iomem_resource, new);
next = res->sibling;
while (next && resource_overlaps(new, next)) {
```
And the new helper function in `kernel/resource.c`:
```diff
--- a/kernel/resource.c
+++ b/kernel/resource.c
@@ -364,6 +355,7 @@ int release_resource(struct resource *old)
EXPORT_SYMBOL(release_resource);
+/**
+ * claim_and_punch_out_soft_reserved - Claim a resource region, punching out
+ * any overlapping "Soft Reserved" areas.
+ * @root: The root of the resource tree (e.g., &iomem_resource).
+ * @new: The new resource to insert.
+ *
+ * Description:
+ * This function prepares for the insertion of a new resource (@new) by
+ * resolving conflicts with existing "Soft Reserved" regions. It works as
+ * follows:
+ *
+ * 1. It iterates through the children of @root, searching for any resource
+ * that both overlaps with @new and is identified as "Soft Reserved"
+ * (by its name and the IORES_DESC_SOFT_RESERVED descriptor).
+ * 2. For each conflicting "Soft Reserved" resource found, it removes the
+ * original conflicting resource from the tree.
+ * 3. It then calculates the remaining parts of the original resource that
+ * lie outside the range of @new. This may result in one or two smaller,
+ * non-overlapping parts.
+ * 4. These remaining parts are re-inserted into the resource tree as new,
+ * smaller "Soft Reserved" resources. This action is akin to "punching a
+ * hole" in the original reserved region.
+ * 5. After all conflicts are resolved, the @new resource is inserted into
+ * the tree, claiming the now-available space.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+int claim_and_punch_out_soft_reserved(struct resource *root,
+ struct resource *new);
+
```
Looking forward to your feedback and thoughts on this approach.
Thanks,
Zhijian
-----
**References:**
[0]
[PATCH 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1687568084.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1687568084.git.alison.schofield@intel.com/)
[PATCH v2 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/)
[PATCH v3 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1692638817.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1692638817.git.alison.schofield@intel.com/)
[RFC] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/)
[RFC v2] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241030172751.81392-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241030172751.81392-1-nathan.fontenot@amd.com/)
[PATCH] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241202155542.22111-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241202155542.22111-1-nathan.fontenot@amd.com/)
[PATCH v2 0/4] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/cover.1737046620.git.nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/cover.1737046620.git.nathan.fontenot@amd.com/)
[PATCH v3 0/4] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/20250403183315.286710-1-terry.bowman@amd.com/](https://lore.kernel.org/linux-cxl/20250403183315.286710-1-terry.bowman@amd.com/)
[PATCH v4 0/7] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com/#r](https://lore.kernel.org/linux-cxl/20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com/#r)
[1] [https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/)
[2] [https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/)On 7/9/2025 9:30 PM, Zhijian Li (Fujitsu) wrote:
> Hi all,
>
> This RFC proposes a new approach to handle the resource conflict between CXL Fixed Memory Windows (CFMW) and "Soft Reserved" memory regions.
>
> I've been thinking about the "Soft Reserved" issue lately and went through the previous proposals [0]. The existing solutions are generally centered around CXL region management. For instance, an early proposal [1] suggested removing the "Soft Reserved" resource during region teardown. The current approach [2] has evolved to remove it after the CXL driver automatically constructs a CXL region.
>
> I haven't found prior discussions centered on removing the "Soft Reserved" region *before* the CFMW resource is even added. If this has been discussed before, please point me to the thread.
>
> My proposal is to remove the "Soft Reserved" resource at the moment we add the root decoder resource (the CFMW) from `CXL_ACPI`.
A couple of thoughts on this approach.
If the device_hmem driver loads prior to the cxl_acpi driver this could be an issue. The device
hmem driver init routine makes a copy of all soft reserve resources into hmem_active. It is
this hmem_active list that is used by the hmem driver to create dax devices, this could possibly
create a dax device for a soft reserve that no longer exists, or has been split up.
Perhaps moving the MODULE_SOFTDEP("pre: cxl_acpi") from dax/hmem/hmem.c to dax/hmem/device.c
could solve this to ensure the cxl_acpi code runs prior to building the hmem_active list.
One other aspect that this approach would prevent is the ability to use the memory covered by
a cxl region if there is an error in creating that region. In previous approaches for instance,
if during cxl region discovery we get an error and fail to create the region, the soft reserve
intersecting that failed region would still exist and be available for the dax driver to consume.
-Nathan
>
> Here’s why I believe this is feasible and more effective.
>
> #### Background
>
> Currently, CXL memory can be exposed to the users in several ways(1)(2)(3), depending on firmware and driver configurations. The diagram below illustrates the flow:
>
> ```
> +-----------------------+
> | |
> | CXL memory |
> | |
> +----------+------------+ +----------------------+
> | | |
> | | |
> +firmware---------------------+ | +---------v---------+ +--------------+
> | v | | | | NO | |
> | +----------+-------------+ | | | enable CXL_ACPI? +---------> HMEM |
> | | | | | | | | |
> | | expose to E820? | | | +-------------------+ +-------+------+
> | | | | | |YES |
> | +----------+-------------+ | | | |
> | | | | +---------v----------+ |
> | | YES | | | iomem | |
> | v | | | CXL WindowN | |
> +-(1)-----------+ | +-----------+-------------+ | | | (4) | |
> | iomem | NO | | set | | | +---------+----------+ |
> | System Ram +<--------+ EFI_MEMORY_SP attr? | | | | |
> | | | | | | | | |
> +---------------+ | +-------------------------+ | | +---------v----------+ +-------v--------+
> +-----------------------------+ | | | | |
> | YES | | CXL region +--------> (2) dax/kmem |
> v | | | | (3) device_dax |
> +-----------+-------------+ | +--------------------+ +----------------+
> | iomem | |
> | Soft Reserved +--------+
> | |
> +-------------------------+
> ```
>
> The problem we are facing occurs in path **(4)**, where the ACPI-defined `CXL WindowN` (CFMW) overlaps with a `Soft Reserved` region. In this scenario, if we try to destroy the driver-created CXL region to create a new one, the operation fails because the underlying memory range is still claimed by "Soft Reserved".
>
> Here is an example from ` /proc/iomem `:
>
> ```
> c070000000-fcffffffff : CXL Window 0
> c070000000-fcffffffff : Soft Reserved
> c070000000-fcffffffff : region0 ### Deleting this will not free the range for a new region
> c070000000-fcffffffff : dax0.0
> c070000000-fcffffffff : System RAM (kmem)
> ```
>
> #### Proposal
>
> The fundamental assumption of the CXL driver design appears to be (and I believe this is correct) that once the kernel successfully parses `CEDT.CFMWS` via `CXL_ACPI`, the CXL subsystem is designated to take full control of the corresponding CXL memory device.
>
> If this holds true, it means that the "Soft Reserved" region, which serves as a firmware-level hint to prevent the OS from arbitrarily using this memory, is no longer necessary after the CFMW is identified. The CFMW itself becomes the authoritative definition for this memory range.
>
> Therefore, we can safely remove the "Soft Reserved" resource from the `iomem_resource` tree right before inserting the new CFMW resource.
>
> This approach is simple and highly effective. It decouples the "Soft Reserved" problem from CXL region management entirely. By cleaning up the resource conflict at the earliest possible stage, this solution should be compatible with all CXL region patterns, including scenarios with misconfigured or unconfigured HDM Decoders.
>
> I haven't spent much time working out all the implementation details yet, but a quick proof-of-concept (PoC) shows that this approach appears to work. A rough sketch of the code is below. If this direction is ACKed, I will prepare and send out a complete implementation for review.
> ```diff
> --- a/drivers/cxl/acpi.c
> +++ b/drivers/cxl/acpi.c
> @@ -780,7 +783,9 @@ static int add_cxl_resources(struct resource *cxl_res)
> */
> cxl_set_public_resource(res, new);
>
> - insert_resource_expand_to_fit(&iomem_resource, new);
> + /* Remove Soft Reserved that is fully covered by this window */
> + claim_and_punch_out_soft_reserved(&iomem_resource, new);
>
> next = res->sibling;
> while (next && resource_overlaps(new, next)) {
> ```
>
> And the new helper function in `kernel/resource.c`:
>
> ```diff
> --- a/kernel/resource.c
> +++ b/kernel/resource.c
> @@ -364,6 +355,7 @@ int release_resource(struct resource *old)
>
> EXPORT_SYMBOL(release_resource);
>
> +/**
> + * claim_and_punch_out_soft_reserved - Claim a resource region, punching out
> + * any overlapping "Soft Reserved" areas.
> + * @root: The root of the resource tree (e.g., &iomem_resource).
> + * @new: The new resource to insert.
> + *
> + * Description:
> + * This function prepares for the insertion of a new resource (@new) by
> + * resolving conflicts with existing "Soft Reserved" regions. It works as
> + * follows:
> + *
> + * 1. It iterates through the children of @root, searching for any resource
> + * that both overlaps with @new and is identified as "Soft Reserved"
> + * (by its name and the IORES_DESC_SOFT_RESERVED descriptor).
> + * 2. For each conflicting "Soft Reserved" resource found, it removes the
> + * original conflicting resource from the tree.
> + * 3. It then calculates the remaining parts of the original resource that
> + * lie outside the range of @new. This may result in one or two smaller,
> + * non-overlapping parts.
> + * 4. These remaining parts are re-inserted into the resource tree as new,
> + * smaller "Soft Reserved" resources. This action is akin to "punching a
> + * hole" in the original reserved region.
> + * 5. After all conflicts are resolved, the @new resource is inserted into
> + * the tree, claiming the now-available space.
> + *
> + * Return: 0 on success, or a negative error code on failure.
> + */
> +int claim_and_punch_out_soft_reserved(struct resource *root,
> + struct resource *new);
> +
> ```
>
> Looking forward to your feedback and thoughts on this approach.
>
> Thanks,
> Zhijian
>
> -----
>
> **References:**
>
> [0]
> [PATCH 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1687568084.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1687568084.git.alison.schofield@intel.com/)
> [PATCH v2 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/)
> [PATCH v3 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1692638817.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1692638817.git.alison.schofield@intel.com/)
> [RFC] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/)
> [RFC v2] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241030172751.81392-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241030172751.81392-1-nathan.fontenot@amd.com/)
> [PATCH] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241202155542.22111-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241202155542.22111-1-nathan.fontenot@amd.com/)
> [PATCH v2 0/4] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/cover.1737046620.git.nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/cover.1737046620.git.nathan.fontenot@amd.com/)
> [PATCH v3 0/4] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/20250403183315.286710-1-terry.bowman@amd.com/](https://lore.kernel.org/linux-cxl/20250403183315.286710-1-terry.bowman@amd.com/)
> [PATCH v4 0/7] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com/#r](https://lore.kernel.org/linux-cxl/20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com/#r)
>
> [1] [https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/)
> [2] [https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/)
On 11/07/2025 05:15, Nathan Fontenot wrote:
>> My proposal is to remove the "Soft Reserved" resource at the moment we add the root decoder resource (the CFMW) from `CXL_ACPI`.
> A couple of thoughts on this approach.
>
> If the device_hmem driver loads prior to the cxl_acpi driver this could be an issue. The device
> hmem driver init routine makes a copy of all soft reserve resources into hmem_active. It is
> this hmem_active list that is used by the hmem driver to create dax devices, this could possibly
> create a dax device for a soft reserve that no longer exists, or has been split up.
>
> Perhaps moving the MODULE_SOFTDEP("pre: cxl_acpi") from dax/hmem/hmem.c to dax/hmem/device.c
> could solve this to ensure the cxl_acpi code runs prior to building the hmem_active list.
>
> One other aspect that this approach would prevent is the ability to use the memory covered by
> a cxl region if there is an error in creating that region.
> In previous approaches for instance,
> if during cxl region discovery we get an error and fail to create the region, the soft reserve
> intersecting that failed region would still exist and be available for the dax driver to consume.
It seems this not always works.
For example, when there is a misconfigured HDM decoder, so the cxl_region will be registered(but not readiness).
In this case(with your V4 patch), we would see:
```
5d0000000-6cfffffff : CXL Window 0
5d0000000-6cfffffff : region0
```
Actually, it cannot be consumed by the hmem/dax_hmem neither.
This made me realize that my current idea ignores the case of CXL_ACPI=y && CXL_REGION=n. I quickly tested it(I will take a deeper look later):
After the OS boots, we can see this resource. Can anyone tell me how to create the corresponding dax_region? Because I tried using dax create-device without success.
```
5d0000000-6cfffffff : CXL Window 0
5d0000000-6cfffffff : Soft Reserved
```
Thanks
Zhijian
On Fri, Jul 11, 2025 at 05:34:27AM +0000, Zhijian Li (Fujitsu) wrote: > > > On 11/07/2025 05:15, Nathan Fontenot wrote: > In this case(with your V4 patch), we would see: > ``` > 5d0000000-6cfffffff : CXL Window 0 > 5d0000000-6cfffffff : region0 > ``` > Actually, it cannot be consumed by the hmem/dax_hmem neither. > > This made me realize that my current idea ignores the case of CXL_ACPI=y && CXL_REGION=n. I quickly tested it(I will take a deeper look later): > > After the OS boots, we can see this resource. Can anyone tell me how to create the corresponding dax_region? Because I tried using dax create-device without success. > ``` > 5d0000000-6cfffffff : CXL Window 0 > 5d0000000-6cfffffff : Soft Reserved > ``` > with CXL_REGION=n you cannot create the dax region. region0 --creates-> dax_region0 --creates-> dax0 all you've done here by setting CXL_REGION=N is break the probe chain but otherwise successfully attributed the memory to the CXL driver. ~Gregory
On Thu, Nov 06, 2025 at 01:24:12PM -0500, Gregory Price wrote: > On Fri, Jul 11, 2025 at 05:34:27AM +0000, Zhijian Li (Fujitsu) wrote: > > with CXL_REGION=n you cannot create the dax region. > > region0 --creates-> dax_region0 --creates-> dax0 > > all you've done here by setting CXL_REGION=N is break the probe chain > but otherwise successfully attributed the memory to the CXL driver. > apologies, disregard, this old email somehow managed to make its way to the top of my queue. This code has changed significantly since july lol. Sorry for the noise. ~Gregory
Hi Zhijian
On 7/9/2025 7:30 PM, Zhijian Li (Fujitsu) wrote:
> Hi all,
>
> This RFC proposes a new approach to handle the resource conflict between CXL Fixed Memory Windows (CFMW) and "Soft Reserved" memory regions.
>
> I've been thinking about the "Soft Reserved" issue lately and went through the previous proposals [0]. The existing solutions are generally centered around CXL region management. For instance, an early proposal [1] suggested removing the "Soft Reserved" resource during region teardown. The current approach [2] has evolved to remove it after the CXL driver automatically constructs a CXL region.
>
> I haven't found prior discussions centered on removing the "Soft Reserved" region *before* the CFMW resource is even added. If this has been discussed before, please point me to the thread.
>
> My proposal is to remove the "Soft Reserved" resource at the moment we add the root decoder resource (the CFMW) from `CXL_ACPI`.
>
> Here’s why I believe this is feasible and more effective.
>
> #### Background
>
> Currently, CXL memory can be exposed to the users in several ways(1)(2)(3), depending on firmware and driver configurations. The diagram below illustrates the flow:
>
> ```
> +-----------------------+
> | |
> | CXL memory |
> | |
> +----------+------------+ +----------------------+
> | | |
> | | |
> +firmware---------------------+ | +---------v---------+ +--------------+
> | v | | | | NO | |
> | +----------+-------------+ | | | enable CXL_ACPI? +---------> HMEM |
> | | | | | | | | |
> | | expose to E820? | | | +-------------------+ +-------+------+
> | | | | | |YES |
> | +----------+-------------+ | | | |
> | | | | +---------v----------+ |
> | | YES | | | iomem | |
> | v | | | CXL WindowN | |
> +-(1)-----------+ | +-----------+-------------+ | | | (4) | |
> | iomem | NO | | set | | | +---------+----------+ |
> | System Ram +<--------+ EFI_MEMORY_SP attr? | | | | |
> | | | | | | | | |
> +---------------+ | +-------------------------+ | | +---------v----------+ +-------v--------+
> +-----------------------------+ | | | | |
> | YES | | CXL region +--------> (2) dax/kmem |
> v | | | | (3) device_dax |
> +-----------+-------------+ | +--------------------+ +----------------+
> | iomem | |
> | Soft Reserved +--------+
> | |
> +-------------------------+
> ```
>
> The problem we are facing occurs in path **(4)**, where the ACPI-defined `CXL WindowN` (CFMW) overlaps with a `Soft Reserved` region. In this scenario, if we try to destroy the driver-created CXL region to create a new one, the operation fails because the underlying memory range is still claimed by "Soft Reserved".
>
> Here is an example from ` /proc/iomem `:
>
> ```
> c070000000-fcffffffff : CXL Window 0
> c070000000-fcffffffff : Soft Reserved
> c070000000-fcffffffff : region0 ### Deleting this will not free the range for a new region
> c070000000-fcffffffff : dax0.0
> c070000000-fcffffffff : System RAM (kmem)
> ```
>
> #### Proposal
>
> The fundamental assumption of the CXL driver design appears to be (and I believe this is correct) that once the kernel successfully parses `CEDT.CFMWS` via `CXL_ACPI`, the CXL subsystem is designated to take full control of the corresponding CXL memory device.
>
> If this holds true, it means that the "Soft Reserved" region, which serves as a firmware-level hint to prevent the OS from arbitrarily using this memory, is no longer necessary after the CFMW is identified. The CFMW itself becomes the authoritative definition for this memory range.
I may be wrong, but shouldn't we also guarantee that cxl_region
instances be always populated after the CXL Windows (CFMWS) are setup
through cxl_acpi?
Dan's comments here:
https://lore.kernel.org/all/65e0fcae989d6_1138c7294c2@dwillia2-xfh.jf.intel.com.notmuch/
"After cxl_acpi is loaded Linux knows where all the CXL windows are, but
it does not know if there are cxl_region instances populated into those
windows. That process involves waiting for PCI probe to complete and the
CXL autoassembly process to run its course."
This patchset
https://lore.kernel.org/all/20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com/
does wait for cxl_mem (cxl_pci) to complete but I agree it doesn't
guarantee region readiness. I was thinking to wait on cxl_region_probe
to complete along with cxl_mem in v5.
Thanks
Smita
>
> Therefore, we can safely remove the "Soft Reserved" resource from the `iomem_resource` tree right before inserting the new CFMW resource.
>
> This approach is simple and highly effective. It decouples the "Soft Reserved" problem from CXL region management entirely. By cleaning up the resource conflict at the earliest possible stage, this solution should be compatible with all CXL region patterns, including scenarios with misconfigured or unconfigured HDM Decoders.
>
> I haven't spent much time working out all the implementation details yet, but a quick proof-of-concept (PoC) shows that this approach appears to work. A rough sketch of the code is below. If this direction is ACKed, I will prepare and send out a complete implementation for review.
> ```diff
> --- a/drivers/cxl/acpi.c
> +++ b/drivers/cxl/acpi.c
> @@ -780,7 +783,9 @@ static int add_cxl_resources(struct resource *cxl_res)
> */
> cxl_set_public_resource(res, new);
>
> - insert_resource_expand_to_fit(&iomem_resource, new);
> + /* Remove Soft Reserved that is fully covered by this window */
> + claim_and_punch_out_soft_reserved(&iomem_resource, new);
>
> next = res->sibling;
> while (next && resource_overlaps(new, next)) {
> ```
>
> And the new helper function in `kernel/resource.c`:
>
> ```diff
> --- a/kernel/resource.c
> +++ b/kernel/resource.c
> @@ -364,6 +355,7 @@ int release_resource(struct resource *old)
>
> EXPORT_SYMBOL(release_resource);
>
> +/**
> + * claim_and_punch_out_soft_reserved - Claim a resource region, punching out
> + * any overlapping "Soft Reserved" areas.
> + * @root: The root of the resource tree (e.g., &iomem_resource).
> + * @new: The new resource to insert.
> + *
> + * Description:
> + * This function prepares for the insertion of a new resource (@new) by
> + * resolving conflicts with existing "Soft Reserved" regions. It works as
> + * follows:
> + *
> + * 1. It iterates through the children of @root, searching for any resource
> + * that both overlaps with @new and is identified as "Soft Reserved"
> + * (by its name and the IORES_DESC_SOFT_RESERVED descriptor).
> + * 2. For each conflicting "Soft Reserved" resource found, it removes the
> + * original conflicting resource from the tree.
> + * 3. It then calculates the remaining parts of the original resource that
> + * lie outside the range of @new. This may result in one or two smaller,
> + * non-overlapping parts.
> + * 4. These remaining parts are re-inserted into the resource tree as new,
> + * smaller "Soft Reserved" resources. This action is akin to "punching a
> + * hole" in the original reserved region.
> + * 5. After all conflicts are resolved, the @new resource is inserted into
> + * the tree, claiming the now-available space.
> + *
> + * Return: 0 on success, or a negative error code on failure.
> + */
> +int claim_and_punch_out_soft_reserved(struct resource *root,
> + struct resource *new);
> +
> ```
>
> Looking forward to your feedback and thoughts on this approach.
>
> Thanks,
> Zhijian
>
> -----
>
> **References:**
>
> [0]
> [PATCH 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1687568084.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1687568084.git.alison.schofield@intel.com/)
> [PATCH v2 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/)
> [PATCH v3 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1692638817.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1692638817.git.alison.schofield@intel.com/)
> [RFC] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/)
> [RFC v2] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241030172751.81392-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241030172751.81392-1-nathan.fontenot@amd.com/)
> [PATCH] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241202155542.22111-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241202155542.22111-1-nathan.fontenot@amd.com/)
> [PATCH v2 0/4] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/cover.1737046620.git.nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/cover.1737046620.git.nathan.fontenot@amd.com/)
> [PATCH v3 0/4] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/20250403183315.286710-1-terry.bowman@amd.com/](https://lore.kernel.org/linux-cxl/20250403183315.286710-1-terry.bowman@amd.com/)
> [PATCH v4 0/7] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com/#r](https://lore.kernel.org/linux-cxl/20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com/#r)
>
> [1] [https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/)
> [2] [https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/)
On 11/07/2025 04:39, Koralahalli Channabasappa, Smita wrote:
> Hi Zhijian
>
> On 7/9/2025 7:30 PM, Zhijian Li (Fujitsu) wrote:
>> Hi all,
>>
>> This RFC proposes a new approach to handle the resource conflict between CXL Fixed Memory Windows (CFMW) and "Soft Reserved" memory regions.
>>
>> I've been thinking about the "Soft Reserved" issue lately and went through the previous proposals [0]. The existing solutions are generally centered around CXL region management. For instance, an early proposal [1] suggested removing the "Soft Reserved" resource during region teardown. The current approach [2] has evolved to remove it after the CXL driver automatically constructs a CXL region.
>>
>> I haven't found prior discussions centered on removing the "Soft Reserved" region *before* the CFMW resource is even added. If this has been discussed before, please point me to the thread.
>>
>> My proposal is to remove the "Soft Reserved" resource at the moment we add the root decoder resource (the CFMW) from `CXL_ACPI`.
>>
>> Here’s why I believe this is feasible and more effective.
>>
>> #### Background
>>
>> Currently, CXL memory can be exposed to the users in several ways(1)(2)(3), depending on firmware and driver configurations. The diagram below illustrates the flow:
>>
>> ```
>> +-----------------------+
>> | |
>> | CXL memory |
>> | |
>> +----------+------------+ +----------------------+
>> | | |
>> | | |
>> +firmware---------------------+ | +---------v---------+ +--------------+
>> | v | | | | NO | |
>> | +----------+-------------+ | | | enable CXL_ACPI? +---------> HMEM |
>> | | | | | | | | |
>> | | expose to E820? | | | +-------------------+ +-------+------+
>> | | | | | |YES |
>> | +----------+-------------+ | | | |
>> | | | | +---------v----------+ |
>> | | YES | | | iomem | |
>> | v | | | CXL WindowN | |
>> +-(1)-----------+ | +-----------+-------------+ | | | (4) | |
>> | iomem | NO | | set | | | +---------+----------+ |
>> | System Ram +<--------+ EFI_MEMORY_SP attr? | | | | |
>> | | | | | | | | |
>> +---------------+ | +-------------------------+ | | +---------v----------+ +-------v--------+
>> +-----------------------------+ | | | | |
>> | YES | | CXL region +--------> (2) dax/kmem |
>> v | | | | (3) device_dax |
>> +-----------+-------------+ | +--------------------+ +----------------+
>> | iomem | |
>> | Soft Reserved +--------+
>> | |
>> +-------------------------+
>> ```
>>
>> The problem we are facing occurs in path **(4)**, where the ACPI-defined `CXL WindowN` (CFMW) overlaps with a `Soft Reserved` region. In this scenario, if we try to destroy the driver-created CXL region to create a new one, the operation fails because the underlying memory range is still claimed by "Soft Reserved".
>>
>> Here is an example from ` /proc/iomem `:
>>
>> ```
>> c070000000-fcffffffff : CXL Window 0
>> c070000000-fcffffffff : Soft Reserved
>> c070000000-fcffffffff : region0 ### Deleting this will not free the range for a new region
>> c070000000-fcffffffff : dax0.0
>> c070000000-fcffffffff : System RAM (kmem)
>> ```
>>
>> #### Proposal
>>
>> The fundamental assumption of the CXL driver design appears to be (and I believe this is correct) that once the kernel successfully parses `CEDT.CFMWS` via `CXL_ACPI`, the CXL subsystem is designated to take full control of the corresponding CXL memory device.
>>
>> If this holds true, it means that the "Soft Reserved" region, which serves as a firmware-level hint to prevent the OS from arbitrarily using this memory, is no longer necessary after the CFMW is identified. The CFMW itself becomes the authoritative definition for this memory range.
>
> I may be wrong, but shouldn't we also guarantee that cxl_region instances be always populated after the CXL Windows (CFMWS) are setup through cxl_acpi?
However, my point was that once the CFMWS is correctly parsed (and inserted into iomem_resource), the CXL memory should be handed over to the CXL subsystem.
If the CXL subsystem encounters obstacles in setting up the cxl region, the following actions should be taken either:
- Fix the firmware
- Allow the CXL subsystem to reprogram misconfigured HDM decoders
- Disable the cxl_acpi driver to fall back to the hmem path.
To my understanding, this aligns with the behavior of the current upstream kernel.
Thanks
Zhijian
>
> Dan's comments here:
> https://lore.kernel.org/all/65e0fcae989d6_1138c7294c2@dwillia2-xfh.jf.intel.com.notmuch/
>
> "After cxl_acpi is loaded Linux knows where all the CXL windows are, but
> it does not know if there are cxl_region instances populated into those
> windows. That process involves waiting for PCI probe to complete and the
> CXL autoassembly process to run its course."
>
> This patchset https://lore.kernel.org/all/20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com/ does wait for cxl_mem (cxl_pci) to complete but I agree it doesn't guarantee region readiness. I was thinking to wait on cxl_region_probe to complete along with cxl_mem in v5.
>
> Thanks
> Smita
>>
>> Therefore, we can safely remove the "Soft Reserved" resource from the `iomem_resource` tree right before inserting the new CFMW resource.
>>
>> This approach is simple and highly effective. It decouples the "Soft Reserved" problem from CXL region management entirely. By cleaning up the resource conflict at the earliest possible stage, this solution should be compatible with all CXL region patterns, including scenarios with misconfigured or unconfigured HDM Decoders.
>>
>> I haven't spent much time working out all the implementation details yet, but a quick proof-of-concept (PoC) shows that this approach appears to work. A rough sketch of the code is below. If this direction is ACKed, I will prepare and send out a complete implementation for review.
>> ```diff
>> --- a/drivers/cxl/acpi.c
>> +++ b/drivers/cxl/acpi.c
>> @@ -780,7 +783,9 @@ static int add_cxl_resources(struct resource *cxl_res)
>> */
>> cxl_set_public_resource(res, new);
>> - insert_resource_expand_to_fit(&iomem_resource, new);
>> + /* Remove Soft Reserved that is fully covered by this window */
>> + claim_and_punch_out_soft_reserved(&iomem_resource, new);
>> next = res->sibling;
>> while (next && resource_overlaps(new, next)) {
>> ```
>>
>> And the new helper function in `kernel/resource.c`:
>>
>> ```diff
>> --- a/kernel/resource.c
>> +++ b/kernel/resource.c
>> @@ -364,6 +355,7 @@ int release_resource(struct resource *old)
>> EXPORT_SYMBOL(release_resource);
>> +/**
>> + * claim_and_punch_out_soft_reserved - Claim a resource region, punching out
>> + * any overlapping "Soft Reserved" areas.
>> + * @root: The root of the resource tree (e.g., &iomem_resource).
>> + * @new: The new resource to insert.
>> + *
>> + * Description:
>> + * This function prepares for the insertion of a new resource (@new) by
>> + * resolving conflicts with existing "Soft Reserved" regions. It works as
>> + * follows:
>> + *
>> + * 1. It iterates through the children of @root, searching for any resource
>> + * that both overlaps with @new and is identified as "Soft Reserved"
>> + * (by its name and the IORES_DESC_SOFT_RESERVED descriptor).
>> + * 2. For each conflicting "Soft Reserved" resource found, it removes the
>> + * original conflicting resource from the tree.
>> + * 3. It then calculates the remaining parts of the original resource that
>> + * lie outside the range of @new. This may result in one or two smaller,
>> + * non-overlapping parts.
>> + * 4. These remaining parts are re-inserted into the resource tree as new,
>> + * smaller "Soft Reserved" resources. This action is akin to "punching a
>> + * hole" in the original reserved region.
>> + * 5. After all conflicts are resolved, the @new resource is inserted into
>> + * the tree, claiming the now-available space.
>> + *
>> + * Return: 0 on success, or a negative error code on failure.
>> + */
>> +int claim_and_punch_out_soft_reserved(struct resource *root,
>> + struct resource *new);
>> +
>> ```
>>
>> Looking forward to your feedback and thoughts on this approach.
>>
>> Thanks,
>> Zhijian
>>
>> -----
>>
>> **References:**
>>
>> [0]
>> [PATCH 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1687568084.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1687568084.git.alison.schofield@intel.com/)
>> [PATCH v2 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/)
>> [PATCH v3 0/2] cxl/region: Improve Soft Reserved resource handling: [https://lore.kernel.org/linux-cxl/cover.1692638817.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1692638817.git.alison.schofield@intel.com/)
>> [RFC] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/)
>> [RFC v2] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241030172751.81392-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241030172751.81392-1-nathan.fontenot@amd.com/)
>> [PATCH] cxl: Update Soft Reserved resources upon region creation: [https://lore.kernel.org/linux-cxl/20241202155542.22111-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241202155542.22111-1-nathan.fontenot@amd.com/)
>> [PATCH v2 0/4] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/cover.1737046620.git.nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/cover.1737046620.git.nathan.fontenot@amd.com/)
>> [PATCH v3 0/4] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/20250403183315.286710-1-terry.bowman@amd.com/](https://lore.kernel.org/linux-cxl/20250403183315.286710-1-terry.bowman@amd.com/)
>> [PATCH v4 0/7] Add managed SOFT RESERVE resource handling: [https://lore.kernel.org/linux-cxl/20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com/#r](https://lore.kernel.org/linux-cxl/20250603221949.53272-1-Smita.KoralahalliChannabasappa@amd.com/#r)
>>
>> [1] [https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/](https://lore.kernel.org/linux-cxl/cover.1691176651.git.alison.schofield@intel.com/)
>> [2] [https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/](https://lore.kernel.org/linux-cxl/20241004181754.8960-1-nathan.fontenot@amd.com/)
>
© 2016 - 2025 Red Hat, Inc.