The setting for non-contiguous 1s support in Intel CAT is
hardcoded to false. On these systems, writing non-contiguous
1s into the schemata file will fail before resctrl passes
the value to the hardware.

In Intel CAT CPUID.0x10.1:ECX[3] and CPUID.0x10.2:ECX[3] stopped
being reserved and now carry information about non-contiguous 1s
value support for L3 and L2 cache respectively. The CAT
capacity bitmask (CBM) supports a non-contiguous 1s value if
the bit is set.

Replace the hardcoded non-contiguous support value with
the support learned from the hardware. Add hardcoded non-contiguous
support value to Haswell probe since it can't make use of CPUID for
Cache allocation.

Originally-by: Fenghua Yu <fenghua.yu@intel.com>
Signed-off-by: Wieczor-Retman, Maciej <maciej.wieczor-retman@intel.com>
---
 arch/x86/kernel/cpu/resctrl/core.c        |  9 ++++++---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 11 +++++++----
 arch/x86/kernel/cpu/resctrl/internal.h    |  9 +++++++++
 3 files changed, 22 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 030d3b409768..c783a873147c 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -152,6 +152,7 @@ static inline void cache_alloc_hsw_probe(void)
 	r->cache.cbm_len = 20;
 	r->cache.shareable_bits = 0xc0000;
 	r->cache.min_cbm_bits = 2;
+	r->cache.arch_has_sparse_bitmaps = false;
 	r->alloc_capable = true;
 
 	rdt_alloc_capable = true;
@@ -267,15 +268,18 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	union cpuid_0x10_1_eax eax;
+	union cpuid_0x10_x_ecx ecx;
 	union cpuid_0x10_x_edx edx;
-	u32 ebx, ecx;
+	u32 ebx;
 
-	cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx, &edx.full);
+	cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx.full, &edx.full);
 	hw_res->num_closid = edx.split.cos_max + 1;
 	r->cache.cbm_len = eax.split.cbm_len + 1;
 	r->default_ctrl = BIT_MASK(eax.split.cbm_len + 1) - 1;
 	r->cache.shareable_bits = ebx & r->default_ctrl;
 	r->data_width = (r->cache.cbm_len + 3) / 4;
+	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
+		r->cache.arch_has_sparse_bitmaps = ecx.split.noncont;
 	r->alloc_capable = true;
 }
 
@@ -872,7 +876,6 @@ static __init void rdt_init_res_defs_intel(void)
 
 		if (r->rid == RDT_RESOURCE_L3 ||
 		    r->rid == RDT_RESOURCE_L2) {
-			r->cache.arch_has_sparse_bitmaps = false;
 			r->cache.arch_has_per_cpu_cfg = false;
 			r->cache.min_cbm_bits = 1;
 		} else if (r->rid == RDT_RESOURCE_MBA) {
diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
index b44c487727d4..782e2700290b 100644
--- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
+++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
@@ -87,10 +87,13 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
 
 /*
  * Check whether a cache bit mask is valid.
- * For Intel the SDM says:
- *	Please note that all (and only) contiguous '1' combinations
- *	are allowed (e.g. FFFFH, 0FF0H, 003CH, etc.).
- * Additionally Haswell requires at least two bits set.
+ * On Intel CPUs, non-contiguous 1s value support is indicated by CPUID:
+ *   - CPUID.0x10.1:ECX[3]: L3 non-contiguous 1s value supported if 1
+ *   - CPUID.0x10.2:ECX[3]: L2 non-contiguous 1s value supported if 1
+ *
+ * Additionally Haswell requires at least two bits set. Since it does not
+ * have CPUID enumeration support for Cache allocation the non-contiguous 1s
+ * value support is disabled.
  * AMD allows non-contiguous bitmasks.
  */
 static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
index 85ceaf9a31ac..c47ef2f13e8e 100644
--- a/arch/x86/kernel/cpu/resctrl/internal.h
+++ b/arch/x86/kernel/cpu/resctrl/internal.h
@@ -492,6 +492,15 @@ union cpuid_0x10_3_eax {
 	unsigned int full;
 };
 
+/* CPUID.(EAX=10H, ECX=ResID).ECX */
+union cpuid_0x10_x_ecx {
+	struct {
+		unsigned int reserved:3;
+		unsigned int noncont:1;
+	} split;
+	unsigned int full;
+};
+
 /* CPUID.(EAX=10H, ECX=ResID).EDX */
 union cpuid_0x10_x_edx {
 	struct {
-- 
2.42.0

Hi Maciej,

On 9/1/2023 1:55 AM, Wieczor-Retman, Maciej wrote:
> The setting for non-contiguous 1s support in Intel CAT is
> hardcoded to false. On these systems, writing non-contiguous
> 1s into the schemata file will fail before resctrl passes
> the value to the hardware.
> 
> In Intel CAT CPUID.0x10.1:ECX[3] and CPUID.0x10.2:ECX[3] stopped
> being reserved and now carry information about non-contiguous 1s
> value support for L3 and L2 cache respectively. The CAT
> capacity bitmask (CBM) supports a non-contiguous 1s value if
> the bit is set.
> 
> Replace the hardcoded non-contiguous support value with
> the support learned from the hardware. Add hardcoded non-contiguous
> support value to Haswell probe since it can't make use of CPUID for
> Cache allocation.
> 
> Originally-by: Fenghua Yu <fenghua.yu@intel.com>
> Signed-off-by: Wieczor-Retman, Maciej <maciej.wieczor-retman@intel.com>
> ---
>  arch/x86/kernel/cpu/resctrl/core.c        |  9 ++++++---
>  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 11 +++++++----
>  arch/x86/kernel/cpu/resctrl/internal.h    |  9 +++++++++
>  3 files changed, 22 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
> index 030d3b409768..c783a873147c 100644
> --- a/arch/x86/kernel/cpu/resctrl/core.c
> +++ b/arch/x86/kernel/cpu/resctrl/core.c
> @@ -152,6 +152,7 @@ static inline void cache_alloc_hsw_probe(void)
>  	r->cache.cbm_len = 20;
>  	r->cache.shareable_bits = 0xc0000;
>  	r->cache.min_cbm_bits = 2;
> +	r->cache.arch_has_sparse_bitmaps = false;
>  	r->alloc_capable = true;
>  
>  	rdt_alloc_capable = true;
> @@ -267,15 +268,18 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
>  {
>  	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
>  	union cpuid_0x10_1_eax eax;
> +	union cpuid_0x10_x_ecx ecx;
>  	union cpuid_0x10_x_edx edx;
> -	u32 ebx, ecx;
> +	u32 ebx;
>  
> -	cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx, &edx.full);
> +	cpuid_count(0x00000010, idx, &eax.full, &ebx, &ecx.full, &edx.full);
>  	hw_res->num_closid = edx.split.cos_max + 1;
>  	r->cache.cbm_len = eax.split.cbm_len + 1;
>  	r->default_ctrl = BIT_MASK(eax.split.cbm_len + 1) - 1;
>  	r->cache.shareable_bits = ebx & r->default_ctrl;
>  	r->data_width = (r->cache.cbm_len + 3) / 4;
> +	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
> +		r->cache.arch_has_sparse_bitmaps = ecx.split.noncont;
>  	r->alloc_capable = true;
>  }
>  
> @@ -872,7 +876,6 @@ static __init void rdt_init_res_defs_intel(void)
>  
>  		if (r->rid == RDT_RESOURCE_L3 ||
>  		    r->rid == RDT_RESOURCE_L2) {
> -			r->cache.arch_has_sparse_bitmaps = false;
>  			r->cache.arch_has_per_cpu_cfg = false;
>  			r->cache.min_cbm_bits = 1;
>  		} else if (r->rid == RDT_RESOURCE_MBA) {
> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> index b44c487727d4..782e2700290b 100644
> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> @@ -87,10 +87,13 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
>  
>  /*
>   * Check whether a cache bit mask is valid.
> - * For Intel the SDM says:
> - *	Please note that all (and only) contiguous '1' combinations
> - *	are allowed (e.g. FFFFH, 0FF0H, 003CH, etc.).
> - * Additionally Haswell requires at least two bits set.
> + * On Intel CPUs, non-contiguous 1s value support is indicated by CPUID:
> + *   - CPUID.0x10.1:ECX[3]: L3 non-contiguous 1s value supported if 1
> + *   - CPUID.0x10.2:ECX[3]: L2 non-contiguous 1s value supported if 1
> + *
> + * Additionally Haswell requires at least two bits set. Since it does not
> + * have CPUID enumeration support for Cache allocation the non-contiguous 1s
> + * value support is disabled.

The second sentence can be misinterpreted. non-contiguous 1s value support is
disabled on Haswell because Haswell does not support non-contiguous 1s value,
not because it does not have CPUID enumeration for RDT.

>   * AMD allows non-contiguous bitmasks.

(connecting to comments in following patch on this topic ... notice how the above
comments use the term "bit mask" and "bitmasks")

>   */
>  static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
> index 85ceaf9a31ac..c47ef2f13e8e 100644
> --- a/arch/x86/kernel/cpu/resctrl/internal.h
> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
> @@ -492,6 +492,15 @@ union cpuid_0x10_3_eax {
>  	unsigned int full;
>  };
>  
> +/* CPUID.(EAX=10H, ECX=ResID).ECX */
> +union cpuid_0x10_x_ecx {
> +	struct {
> +		unsigned int reserved:3;
> +		unsigned int noncont:1;
> +	} split;
> +	unsigned int full;
> +};
> +
>  /* CPUID.(EAX=10H, ECX=ResID).EDX */
>  union cpuid_0x10_x_edx {
>  	struct {

The core of the change looks good to me.

Reinette

Hello,

On 2023-09-11 at 13:05:16 -0700, Reinette Chatre wrote:
>Hi Maciej,
>> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>> index b44c487727d4..782e2700290b 100644
>> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>> @@ -87,10 +87,13 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
>>  
>>  /*
>>   * Check whether a cache bit mask is valid.
>> - * For Intel the SDM says:
>> - *	Please note that all (and only) contiguous '1' combinations
>> - *	are allowed (e.g. FFFFH, 0FF0H, 003CH, etc.).
>> - * Additionally Haswell requires at least two bits set.
>> + * On Intel CPUs, non-contiguous 1s value support is indicated by CPUID:
>> + *   - CPUID.0x10.1:ECX[3]: L3 non-contiguous 1s value supported if 1
>> + *   - CPUID.0x10.2:ECX[3]: L2 non-contiguous 1s value supported if 1
>> + *
>> + * Additionally Haswell requires at least two bits set. Since it does not
>> + * have CPUID enumeration support for Cache allocation the non-contiguous 1s
>> + * value support is disabled.
>
>The second sentence can be misinterpreted. non-contiguous 1s value support is
>disabled on Haswell because Haswell does not support non-contiguous 1s value,
>not because it does not have CPUID enumeration for RDT.

Would phrasing it like this:

+ * Additionally Haswell requires at least two bits set. Since it does not
+ * allow the non-contiguous 1s for cache allocation the support is
+ * hardcoded to disabled.

be more sensible?

>>   * AMD allows non-contiguous bitmasks.
>
>(connecting to comments in following patch on this topic ... notice how the above
>comments use the term "bit mask" and "bitmasks")

I'll change it so the naming is unified, thanks.

>>   */
>>  static bool cbm_validate(char *buf, u32 *data, struct rdt_resource *r)
>> diff --git a/arch/x86/kernel/cpu/resctrl/internal.h b/arch/x86/kernel/cpu/resctrl/internal.h
>> index 85ceaf9a31ac..c47ef2f13e8e 100644
>> --- a/arch/x86/kernel/cpu/resctrl/internal.h
>> +++ b/arch/x86/kernel/cpu/resctrl/internal.h
>> @@ -492,6 +492,15 @@ union cpuid_0x10_3_eax {
>>  	unsigned int full;
>>  };
>>  
>> +/* CPUID.(EAX=10H, ECX=ResID).ECX */
>> +union cpuid_0x10_x_ecx {
>> +	struct {
>> +		unsigned int reserved:3;
>> +		unsigned int noncont:1;
>> +	} split;
>> +	unsigned int full;
>> +};
>> +
>>  /* CPUID.(EAX=10H, ECX=ResID).EDX */
>>  union cpuid_0x10_x_edx {
>>  	struct {
>
>The core of the change looks good to me.
>
>Reinette

Thanks for reviewing!

-- 
Kind regards
Maciej Wieczór-Retman

Hi Maciej,

On 9/11/2023 11:53 PM, Maciej Wieczór-Retman wrote:
> On 2023-09-11 at 13:05:16 -0700, Reinette Chatre wrote:
>> Hi Maciej,
>>> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>>> index b44c487727d4..782e2700290b 100644
>>> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>>> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
>>> @@ -87,10 +87,13 @@ int parse_bw(struct rdt_parse_data *data, struct resctrl_schema *s,
>>>  
>>>  /*
>>>   * Check whether a cache bit mask is valid.
>>> - * For Intel the SDM says:
>>> - *	Please note that all (and only) contiguous '1' combinations
>>> - *	are allowed (e.g. FFFFH, 0FF0H, 003CH, etc.).
>>> - * Additionally Haswell requires at least two bits set.
>>> + * On Intel CPUs, non-contiguous 1s value support is indicated by CPUID:
>>> + *   - CPUID.0x10.1:ECX[3]: L3 non-contiguous 1s value supported if 1
>>> + *   - CPUID.0x10.2:ECX[3]: L2 non-contiguous 1s value supported if 1
>>> + *
>>> + * Additionally Haswell requires at least two bits set. Since it does not
>>> + * have CPUID enumeration support for Cache allocation the non-contiguous 1s
>>> + * value support is disabled.
>>
>> The second sentence can be misinterpreted. non-contiguous 1s value support is
>> disabled on Haswell because Haswell does not support non-contiguous 1s value,
>> not because it does not have CPUID enumeration for RDT.
> 
> Would phrasing it like this:
> 
> + * Additionally Haswell requires at least two bits set. Since it does not
> + * allow the non-contiguous 1s for cache allocation the support is
> + * hardcoded to disabled.
> 
> be more sensible?
> 

I think the above would be sensible as a comment of cache_alloc_hsw_probe()
where the support is hardcoded but here the comment is just used to summarize
what the hardware supports.
Maybe just something like:
	Haswell does not support a non-contiguous 1s value and additionally
	requires at least two bits set.

Reinette