.../admin-guide/kernel-parameters.txt | 2 +- Documentation/arch/x86/resctrl.rst | 46 +++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/msr-index.h | 1 + arch/x86/kernel/cpu/cpuid-deps.c | 1 + arch/x86/kernel/cpu/resctrl/core.c | 13 + arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 4 +- arch/x86/kernel/cpu/resctrl/internal.h | 12 + arch/x86/kernel/cpu/resctrl/rdtgroup.c | 346 ++++++++++++++++++ arch/x86/kernel/cpu/scattered.c | 1 + include/linux/resctrl.h | 12 + 11 files changed, 436 insertions(+), 3 deletions(-)
This series adds the support for L3 Smart Data Cache Injection Allocation
Enforcement (SDCIAE) to resctrl infrastructure. It is refered as io_alloc
in resctrl subsystem.
Upcoming AMD hardware implements Smart Data Cache Injection (SDCI).
Smart Data Cache Injection (SDCI) is a mechanism that enables direct
insertion of data from I/O devices into the L3 cache. By directly caching
data from I/O devices rather than first storing the I/O data in DRAM, SDCI
reduces demands on DRAM bandwidth and reduces latency to the processor
consuming the I/O data.
The SDCIAE (SDCI Allocation Enforcement) PQE feature allows system software
to control the portion of the L3 cache used for SDCI devices.
When enabled, SDCIAE forces all SDCI lines to be placed into the L3 cache
partitions identified by the highest-supported L3_MASK_n register, where n
is the maximum supported CLOSID.
The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
Injection Allocation Enforcement (SDCIAE)
Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
The feature requires linux support of TPH (TLP Processing Hints).
The support is available in linux kernel after the commit
48d0fd2b903e3 ("PCI/TPH: Add TPH documentation")
The patches are based on top of commit
258b24fa741684 (tip/master) Merge branch into tip/master: 'x86/mm'
# Linux Implementation
Feature adds following interface files when the resctrl "io_alloc" feature is
supported:
/sys/fs/resctrl/info/L3/io_alloc: Reports the feature status. Feature can be
enabled/disabled by writing to the interface.
/sys/fs/resctrl/info/L3/io_alloc_cbm: Capacity Bit Masks (CBMs) available to SDCI
supported IO devices. CBM can be configured
by writing to the interface in the following
format::
# Examples
a. Check if io_alloc feature is available
#mount -t resctrl resctrl /sys/fs/resctrl/
# cat /sys/fs/resctrl/info/L3/io_alloc
0
b. Enable the io_alloc feature.
# echo 1 > /sys/fs/resctrl/info/L3/io_alloc
c. Check the CBM values for the io_alloc feature.
# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
L3:0=ffff;1=ffff
d. Change the CBM value for the domain 1:
# echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
# cat /sys/fs/resctrl/info/L3/io_alloc_cbm
L3:0=ffff;1=00ff
d. Disable io_alloc feature and exit.
# echo 0 > /sys/fs/resctrl/info/L3/io_alloc
#umount /sys/fs/resctrl/
---
v3: Rewrote commit log for the last 3 patches. Changed the text to bit
more generic than the AMD specific feature. Added AMD feature
specifics in the end.
Renamed the rdt_get_sdciae_alloc_cfg() to rdt_set_io_alloc_capable().
Renamed the _resctrl_io_alloc_enable() to _resctrl_sdciae_enable()
as it is arch specific.
Changed the return to void in _resctrl_sdciae_enable() instead of int.
The number of CLOSIDs is determined based on the minimum supported
across all resources (in closid_init). It needs to match the max
supported on the resource. Added the check to verify if MAX CLOSID
availability on the system.
Added CDP check to make sure io_alloc is configured in CDP_CODE.
Highest CLOSID corresponds to CDP_CODE.
Added resctrl_io_alloc_closid_free() to free the io_alloc CLOSID.
Added errors in few cases when CLOSID allocation fails.
Fixes splat reported when info/L3/bit_usage is accesed when io_alloc is enabled.
https://lore.kernel.org/lkml/SJ1PR11MB60837B532254E7B23BC27E84FC052@SJ1PR11MB6083.namprd11.prod.outlook.com/
v2: Added dependancy on X86_FEATURE_CAT_L3
Removed the "" in CPU feature definition.
Changed sdciae_capable to io_alloc_capable to make it as generic feature.
Moved io_alloc_capable field in struct resctrl_cache.
Changed the name of few arch functions similar to ABMC series.
resctrl_arch_get_io_alloc_enabled()
resctrl_arch_io_alloc_enable()
Renamed the feature to "io_alloc".
Added generic texts for the feature in commit log and resctrl.rst doc.
Added resctrl_io_alloc_init_cat() to initialize io_alloc to default values
when enabled.
Fixed io_alloc interface to show only on L3 resource.
Added the locks while processing io_alloc CBMs.
Previous versions:
v2: https://lore.kernel.org/lkml/cover.1734556832.git.babu.moger@amd.com/
v1: https://lore.kernel.org/lkml/cover.1723824984.git.babu.moger@amd.com/
Babu Moger (7):
x86/cpufeatures: Add support for L3 Smart Data Cache Injection
Allocation Enforcement
x86/resctrl: Add SDCIAE feature in the command line options
x86/resctrl: Detect io_alloc feature
x86/resctrl: Implement "io_alloc" enable/disable handlers
x86/resctrl: Add interface to enable/disable io_alloc feature
x86/resctrl: Introduce interface to display io_alloc CBMs
x86/resctrl: Introduce interface to modify io_alloc Capacity Bit Masks
.../admin-guide/kernel-parameters.txt | 2 +-
Documentation/arch/x86/resctrl.rst | 46 +++
arch/x86/include/asm/cpufeatures.h | 1 +
arch/x86/include/asm/msr-index.h | 1 +
arch/x86/kernel/cpu/cpuid-deps.c | 1 +
arch/x86/kernel/cpu/resctrl/core.c | 13 +
arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 4 +-
arch/x86/kernel/cpu/resctrl/internal.h | 12 +
arch/x86/kernel/cpu/resctrl/rdtgroup.c | 346 ++++++++++++++++++
arch/x86/kernel/cpu/scattered.c | 1 +
include/linux/resctrl.h | 12 +
11 files changed, 436 insertions(+), 3 deletions(-)
--
2.34.1
Hi Babu,
On 1/30/25 1:20 PM, Babu Moger wrote:
>
> This series adds the support for L3 Smart Data Cache Injection Allocation
> Enforcement (SDCIAE) to resctrl infrastructure. It is refered as io_alloc
"refered as" -> "referred to as"?
> in resctrl subsystem.
>
...
>
> # Linux Implementation
>
> Feature adds following interface files when the resctrl "io_alloc" feature is
> supported:
Please note that, for example when L3 supports io_alloc but L2 does not, the
implementation also creates the files for L2 (since it sets flags for all
cache resources). "io_alloc" may just be present for resource that does not
actually support "io_alloc". (more below)
>
> /sys/fs/resctrl/info/L3/io_alloc: Reports the feature status. Feature can be
> enabled/disabled by writing to the interface.
>
> /sys/fs/resctrl/info/L3/io_alloc_cbm: Capacity Bit Masks (CBMs) available to SDCI
> supported IO devices. CBM can be configured
(please use imperative tone throughout)
> by writing to the interface in the following
> format::
("in the following format" is not followed by a format example as expected)
> # Examples
>
> a. Check if io_alloc feature is available
> #mount -t resctrl resctrl /sys/fs/resctrl/
>
> # cat /sys/fs/resctrl/info/L3/io_alloc
> 0
Considering that io_alloc will be present for resources that may not support
io_alloc I think it may be helpful if it instead display "enabled/disabled/not supported".
User space can still interact with the file with typical boolean variants recognized
by kstrtobool()
>
> b. Enable the io_alloc feature.
>
> # echo 1 > /sys/fs/resctrl/info/L3/io_alloc
>
> c. Check the CBM values for the io_alloc feature.
>
> # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
> L3:0=ffff;1=ffff
Is it expected that more resources may support IO alloc?
In above output the resource name is redundant, but if
more resources could support IO allocation then it may
be more intuitive to move io_alloc_cbm one level up in hierarchy
and keep resource name in output in order to potentially
manage multiple resources. I do not see impact on feature
in current design if files are kept within resource directories,
but in that case the name seems redundant.
>
> d. Change the CBM value for the domain 1:
> # echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
>
> # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
> L3:0=ffff;1=00ff
>
> d. Disable io_alloc feature and exit.
>
> # echo 0 > /sys/fs/resctrl/info/L3/io_alloc
> #umount /sys/fs/resctrl/
>
AMD also supports what is exposed to user space as "shareable_bits". According
to APM:
Depending on the implementation, some portions of the L3 Cache may be
shared by other system functions or used for some other purpose not
under the control of the PQOS feature set. The L3 Cache Allocation
Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a
bitmask that represents portions of the L3 that may be shared by those
functions.
Could you please include what (if any) the relationship is between the CBM
discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of
"highest-supported L3_MASK_n register" when SDCIAE is enabled?
On the resctrl interface side the documentation currently states:
"shareable_bits":
Bitmask of shareable resource with other executing
entities (e.g. I/O). User can use this when
setting up exclusive cache partitions. Note that
some platforms support devices that have their
own settings for cache use which can over-ride
these bits.
Even though this was originally used to expose the content of
Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does
seem to also apply to the "io_alloc" CBM also.
I also expect "bit_usage" to appropriately reflect the io_alloc mask.
From what I can tell SDCIAE does not enforce isolation between hardware and
software use of these cache allocations so it does seem to me that the
existing legends apply.
Reinette
Hi Reinette,
Thanks for the feedback. Sorry for the delay.
On 3/21/25 17:50, Reinette Chatre wrote:
> Hi Babu,
>
> On 1/30/25 1:20 PM, Babu Moger wrote:
>>
>> This series adds the support for L3 Smart Data Cache Injection Allocation
>> Enforcement (SDCIAE) to resctrl infrastructure. It is refered as io_alloc
>
> "refered as" -> "referred to as"?
sure.
>
>> in resctrl subsystem.
>>
>
> ...
>
>>
>> # Linux Implementation
>>
>> Feature adds following interface files when the resctrl "io_alloc" feature is
>> supported:
>
> Please note that, for example when L3 supports io_alloc but L2 does not, the
> implementation also creates the files for L2 (since it sets flags for all
> cache resources). "io_alloc" may just be present for resource that does not
> actually support "io_alloc". (more below)
>
>>
>> /sys/fs/resctrl/info/L3/io_alloc: Reports the feature status. Feature can be
>> enabled/disabled by writing to the interface.
>>
>> /sys/fs/resctrl/info/L3/io_alloc_cbm: Capacity Bit Masks (CBMs) available to SDCI
>> supported IO devices. CBM can be configured
>
> (please use imperative tone throughout)
ok.
>
>> by writing to the interface in the following
>> format::
>
> ("in the following format" is not followed by a format example as expected)
Sure.
>
>> # Examples
>>
>> a. Check if io_alloc feature is available
>> #mount -t resctrl resctrl /sys/fs/resctrl/
>>
>> # cat /sys/fs/resctrl/info/L3/io_alloc
>> 0
>
> Considering that io_alloc will be present for resources that may not support
> io_alloc I think it may be helpful if it instead display "enabled/disabled/not supported".
> User space can still interact with the file with typical boolean variants recognized
> by kstrtobool()
Sounds good.
>
>>
>> b. Enable the io_alloc feature.
>>
>> # echo 1 > /sys/fs/resctrl/info/L3/io_alloc
>>
>> c. Check the CBM values for the io_alloc feature.
>>
>> # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
>> L3:0=ffff;1=ffff
>
> Is it expected that more resources may support IO alloc?
No. There is no plan for that.
> In above output the resource name is redundant, but if
> more resources could support IO allocation then it may
> be more intuitive to move io_alloc_cbm one level up in hierarchy
> and keep resource name in output in order to potentially
> manage multiple resources. I do not see impact on feature
> in current design if files are kept within resource directories,
> but in that case the name seems redundant.
Yea. Will remove "L3". Just keep 0=ffff;1=ffff
>
>>
>> d. Change the CBM value for the domain 1:
>> # echo L3:1=FF > /sys/fs/resctrl/info/L3/io_alloc_cbm
>>
>> # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
>> L3:0=ffff;1=00ff
>>
>> d. Disable io_alloc feature and exit.
>>
>> # echo 0 > /sys/fs/resctrl/info/L3/io_alloc
>> #umount /sys/fs/resctrl/
>>
>
> AMD also supports what is exposed to user space as "shareable_bits". According
> to APM:
> Depending on the implementation, some portions of the L3 Cache may be
> shared by other system functions or used for some other purpose not
> under the control of the PQOS feature set. The L3 Cache Allocation
> Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a
> bitmask that represents portions of the L3 that may be shared by those
> functions.
Here is the complete text.
The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by
CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of
the cache which may be shared with other system entities or used for some
other purpose not under the control of the QOS feature set. When software
sets a bit in one of the L3_MASK_n registers at the same bit positions a
bit in the L3ShareAllocMask, processors executing with the corresponding
COS will competitively share that portion of the cache with the other
function. If this mask is all 0’s, then there is no other entity in the
system competing with the processors for use of the L3 cache.
The "L3ShareAllocMask" is always reported as 0 on AMD systems.
> Could you please include what (if any) the relationship is between the CBM
> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of
> "highest-supported L3_MASK_n register" when SDCIAE is enabled?
No. There is no relationship in here.
>
> On the resctrl interface side the documentation currently states:
>
> "shareable_bits":
> Bitmask of shareable resource with other executing
> entities (e.g. I/O). User can use this when
> setting up exclusive cache partitions. Note that
> some platforms support devices that have their
> own settings for cache use which can over-ride
> these bits.
>
> Even though this was originally used to expose the content of
> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does
> seem to also apply to the "io_alloc" CBM also.
It says "shared by other system functions or used for some other purpose
not under the control of the PQOS feature set".
"io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits".
>
> I also expect "bit_usage" to appropriately reflect the io_alloc mask.
>>From what I can tell SDCIAE does not enforce isolation between hardware and
> software use of these cache allocations so it does seem to me that the
> existing legends apply.
>
> Reinette
>
>
--
Thanks
Babu Moger
Hi Babu, On 4/7/25 1:12 PM, Moger, Babu wrote: > On 3/21/25 17:50, Reinette Chatre wrote: >> On 1/30/25 1:20 PM, Babu Moger wrote: >>> >> >> AMD also supports what is exposed to user space as "shareable_bits". According >> to APM: >> Depending on the implementation, some portions of the L3 Cache may be >> shared by other system functions or used for some other purpose not >> under the control of the PQOS feature set. The L3 Cache Allocation >> Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a >> bitmask that represents portions of the L3 that may be shared by those >> functions. > > Here is the complete text. > > The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by > CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of > the cache which may be shared with other system entities or used for some > other purpose not under the control of the QOS feature set. When software > sets a bit in one of the L3_MASK_n registers at the same bit positions a > bit in the L3ShareAllocMask, processors executing with the corresponding > COS will competitively share that portion of the cache with the other > function. If this mask is all 0’s, then there is no other entity in the > system competing with the processors for use of the L3 cache. > > The "L3ShareAllocMask" is always reported as 0 on AMD systems. > >> Could you please include what (if any) the relationship is between the CBM >> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of >> "highest-supported L3_MASK_n register" when SDCIAE is enabled? > > No. There is no relationship in here. > >> >> On the resctrl interface side the documentation currently states: >> >> "shareable_bits": >> Bitmask of shareable resource with other executing >> entities (e.g. I/O). User can use this when >> setting up exclusive cache partitions. Note that >> some platforms support devices that have their >> own settings for cache use which can over-ride >> these bits. >> >> Even though this was originally used to expose the content of >> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does >> seem to also apply to the "io_alloc" CBM also. > > It says "shared by other system functions or used for some other purpose > not under the control of the PQOS feature set". This is a quote from the AMD spec, not the resctrl user interface documentation. Please consider this from resctrl user interface perspective. > > "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits". When I read the resctrl user interface documentation for "shareable_bits" it sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of shareable resource with other executing entities (e.g. I/O)" ... is this not exactly io_alloc? Reinette
Hi Reinette, On 4/8/2025 4:44 PM, Reinette Chatre wrote: > Hi Babu, > > On 4/7/25 1:12 PM, Moger, Babu wrote: >> On 3/21/25 17:50, Reinette Chatre wrote: >>> On 1/30/25 1:20 PM, Babu Moger wrote: > > >>>> >>> >>> AMD also supports what is exposed to user space as "shareable_bits". According >>> to APM: >>> Depending on the implementation, some portions of the L3 Cache may be >>> shared by other system functions or used for some other purpose not >>> under the control of the PQOS feature set. The L3 Cache Allocation >>> Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a >>> bitmask that represents portions of the L3 that may be shared by those >>> functions. >> >> Here is the complete text. >> >> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by >> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of >> the cache which may be shared with other system entities or used for some >> other purpose not under the control of the QOS feature set. When software >> sets a bit in one of the L3_MASK_n registers at the same bit positions a >> bit in the L3ShareAllocMask, processors executing with the corresponding >> COS will competitively share that portion of the cache with the other >> function. If this mask is all 0’s, then there is no other entity in the >> system competing with the processors for use of the L3 cache. >> >> The "L3ShareAllocMask" is always reported as 0 on AMD systems. >> >>> Could you please include what (if any) the relationship is between the CBM >>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of >>> "highest-supported L3_MASK_n register" when SDCIAE is enabled? >> >> No. There is no relationship in here. >> >>> >>> On the resctrl interface side the documentation currently states: >>> >>> "shareable_bits": >>> Bitmask of shareable resource with other executing >>> entities (e.g. I/O). User can use this when >>> setting up exclusive cache partitions. Note that >>> some platforms support devices that have their >>> own settings for cache use which can over-ride >>> these bits. >>> >>> Even though this was originally used to expose the content of >>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does >>> seem to also apply to the "io_alloc" CBM also. >> >> It says "shared by other system functions or used for some other purpose >> not under the control of the PQOS feature set". > > This is a quote from the AMD spec, not the resctrl user interface documentation. > > Please consider this from resctrl user interface perspective. > >> >> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits". > > When I read the resctrl user interface documentation for "shareable_bits" it > sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of > shareable resource with other executing entities (e.g. I/O)" ... is this > not exactly io_alloc? I agree the text is pretty generic. Actually, the whole bit mask (0xfffF) is shareable with io_alloc. The 'shareable_bits' is coming from CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] which is always 0 on AMD systems. It will be bit odd to manipulate these value. Not sure if we have to do it. Thanks Babu
Hi Babu, On 4/8/25 5:41 PM, Moger, Babu wrote: > Hi Reinette, > > On 4/8/2025 4:44 PM, Reinette Chatre wrote: >> Hi Babu, >> >> On 4/7/25 1:12 PM, Moger, Babu wrote: >>> On 3/21/25 17:50, Reinette Chatre wrote: >>>> On 1/30/25 1:20 PM, Babu Moger wrote: >> >> >>>>> >>>> >>>> AMD also supports what is exposed to user space as "shareable_bits". According >>>> to APM: >>>> Depending on the implementation, some portions of the L3 Cache may be >>>> shared by other system functions or used for some other purpose not >>>> under the control of the PQOS feature set. The L3 Cache Allocation >>>> Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a >>>> bitmask that represents portions of the L3 that may be shared by those >>>> functions. >>> >>> Here is the complete text. >>> >>> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by >>> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of >>> the cache which may be shared with other system entities or used for some >>> other purpose not under the control of the QOS feature set. When software >>> sets a bit in one of the L3_MASK_n registers at the same bit positions a >>> bit in the L3ShareAllocMask, processors executing with the corresponding >>> COS will competitively share that portion of the cache with the other >>> function. If this mask is all 0’s, then there is no other entity in the >>> system competing with the processors for use of the L3 cache. >>> >>> The "L3ShareAllocMask" is always reported as 0 on AMD systems. >>> >>>> Could you please include what (if any) the relationship is between the CBM >>>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of >>>> "highest-supported L3_MASK_n register" when SDCIAE is enabled? >>> >>> No. There is no relationship in here. >>> >>>> >>>> On the resctrl interface side the documentation currently states: >>>> >>>> "shareable_bits": >>>> Bitmask of shareable resource with other executing >>>> entities (e.g. I/O). User can use this when >>>> setting up exclusive cache partitions. Note that >>>> some platforms support devices that have their >>>> own settings for cache use which can over-ride >>>> these bits. >>>> >>>> Even though this was originally used to expose the content of >>>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does >>>> seem to also apply to the "io_alloc" CBM also. >>> >>> It says "shared by other system functions or used for some other purpose >>> not under the control of the PQOS feature set". >> >> This is a quote from the AMD spec, not the resctrl user interface documentation. >> >> Please consider this from resctrl user interface perspective. >> >>> >>> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits". >> >> When I read the resctrl user interface documentation for "shareable_bits" it >> sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of >> shareable resource with other executing entities (e.g. I/O)" ... is this >> not exactly io_alloc? > > I agree the text is pretty generic. Actually, the whole bit mask (0xfffF) is shareable with io_alloc. I think the value of "shareable_bits" presented to user space could be the actual io_alloc_cbm value. Thus, not the "possible IO bitmask" but the actual value. This seems to be the best match of what "shareable_bits" represents, which is the region currently used by IO devices. This partners well with the "bit_usage" output, for example, "X" can be used to show which portions of cache are currently used by both software (via schemata of resource groups) and hardware (via io_alloc_cbm). > > The 'shareable_bits' is coming from CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] which is always 0 on AMD systems. > It will be bit odd to manipulate these value. Not sure if we have to do it. It is not clear to me what you mean with "manipulate". "shareable_bits" does currently come from the existing register but AMD now provides more interfaces with which this data can be obtained and it seems appropriate to use it. Reinette
Hi Reinette, On 4/8/2025 8:41 PM, Reinette Chatre wrote: > Hi Babu, > > On 4/8/25 5:41 PM, Moger, Babu wrote: >> Hi Reinette, >> >> On 4/8/2025 4:44 PM, Reinette Chatre wrote: >>> Hi Babu, >>> >>> On 4/7/25 1:12 PM, Moger, Babu wrote: >>>> On 3/21/25 17:50, Reinette Chatre wrote: >>>>> On 1/30/25 1:20 PM, Babu Moger wrote: >>> >>> >>>>>> >>>>> >>>>> AMD also supports what is exposed to user space as "shareable_bits". According >>>>> to APM: >>>>> Depending on the implementation, some portions of the L3 Cache may be >>>>> shared by other system functions or used for some other purpose not >>>>> under the control of the PQOS feature set. The L3 Cache Allocation >>>>> Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a >>>>> bitmask that represents portions of the L3 that may be shared by those >>>>> functions. >>>> >>>> Here is the complete text. >>>> >>>> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by >>>> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of >>>> the cache which may be shared with other system entities or used for some >>>> other purpose not under the control of the QOS feature set. When software >>>> sets a bit in one of the L3_MASK_n registers at the same bit positions a >>>> bit in the L3ShareAllocMask, processors executing with the corresponding >>>> COS will competitively share that portion of the cache with the other >>>> function. If this mask is all 0’s, then there is no other entity in the >>>> system competing with the processors for use of the L3 cache. >>>> >>>> The "L3ShareAllocMask" is always reported as 0 on AMD systems. >>>> >>>>> Could you please include what (if any) the relationship is between the CBM >>>>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of >>>>> "highest-supported L3_MASK_n register" when SDCIAE is enabled? >>>> >>>> No. There is no relationship in here. >>>> >>>>> >>>>> On the resctrl interface side the documentation currently states: >>>>> >>>>> "shareable_bits": >>>>> Bitmask of shareable resource with other executing >>>>> entities (e.g. I/O). User can use this when >>>>> setting up exclusive cache partitions. Note that >>>>> some platforms support devices that have their >>>>> own settings for cache use which can over-ride >>>>> these bits. >>>>> >>>>> Even though this was originally used to expose the content of >>>>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does >>>>> seem to also apply to the "io_alloc" CBM also. >>>> >>>> It says "shared by other system functions or used for some other purpose >>>> not under the control of the PQOS feature set". >>> >>> This is a quote from the AMD spec, not the resctrl user interface documentation. >>> >>> Please consider this from resctrl user interface perspective. >>> >>>> >>>> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits". >>> >>> When I read the resctrl user interface documentation for "shareable_bits" it >>> sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of >>> shareable resource with other executing entities (e.g. I/O)" ... is this >>> not exactly io_alloc? >> >> I agree the text is pretty generic. Actually, the whole bit mask (0xfffF) is shareable with io_alloc. > > I think the value of "shareable_bits" presented to user space could be the > actual io_alloc_cbm value. Thus, not the "possible IO bitmask" but the actual Confused little bit here. The shareable_bits is resource property. io_alloc_cbm is domain specific value. Not sure how to display it. > value. This seems to be the best match of what "shareable_bits" represents, which > is the region currently used by IO devices. This partners well with the "bit_usage" > output, for example, "X" can be used to show which portions of cache are currently > used by both software (via schemata of resource groups) and hardware (via io_alloc_cbm). Haven't looked at this code much. Will look into it. >> >> The 'shareable_bits' is coming from CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] which is always 0 on AMD systems. >> It will be bit odd to manipulate these value. Not sure if we have to do it. > > It is not clear to me what you mean with "manipulate". "shareable_bits" does currently > come from the existing register but AMD now provides more interfaces with which this data > can be obtained and it seems appropriate to use it. Ok. Thanks Babu
Hi Babu, On 4/9/25 5:58 PM, Moger, Babu wrote: > Hi Reinette, > > On 4/8/2025 8:41 PM, Reinette Chatre wrote: >> Hi Babu, >> >> On 4/8/25 5:41 PM, Moger, Babu wrote: >>> Hi Reinette, >>> >>> On 4/8/2025 4:44 PM, Reinette Chatre wrote: >>>> Hi Babu, >>>> >>>> On 4/7/25 1:12 PM, Moger, Babu wrote: >>>>> On 3/21/25 17:50, Reinette Chatre wrote: >>>>>> On 1/30/25 1:20 PM, Babu Moger wrote: >>>> >>>> >>>>>>> >>>>>> >>>>>> AMD also supports what is exposed to user space as "shareable_bits". According >>>>>> to APM: >>>>>> Depending on the implementation, some portions of the L3 Cache may be >>>>>> shared by other system functions or used for some other purpose not >>>>>> under the control of the PQOS feature set. The L3 Cache Allocation >>>>>> Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a >>>>>> bitmask that represents portions of the L3 that may be shared by those >>>>>> functions. >>>>> >>>>> Here is the complete text. >>>>> >>>>> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by >>>>> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of >>>>> the cache which may be shared with other system entities or used for some >>>>> other purpose not under the control of the QOS feature set. When software >>>>> sets a bit in one of the L3_MASK_n registers at the same bit positions a >>>>> bit in the L3ShareAllocMask, processors executing with the corresponding >>>>> COS will competitively share that portion of the cache with the other >>>>> function. If this mask is all 0’s, then there is no other entity in the >>>>> system competing with the processors for use of the L3 cache. >>>>> >>>>> The "L3ShareAllocMask" is always reported as 0 on AMD systems. >>>>> >>>>>> Could you please include what (if any) the relationship is between the CBM >>>>>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of >>>>>> "highest-supported L3_MASK_n register" when SDCIAE is enabled? >>>>> >>>>> No. There is no relationship in here. >>>>> >>>>>> >>>>>> On the resctrl interface side the documentation currently states: >>>>>> >>>>>> "shareable_bits": >>>>>> Bitmask of shareable resource with other executing >>>>>> entities (e.g. I/O). User can use this when >>>>>> setting up exclusive cache partitions. Note that >>>>>> some platforms support devices that have their >>>>>> own settings for cache use which can over-ride >>>>>> these bits. >>>>>> >>>>>> Even though this was originally used to expose the content of >>>>>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does >>>>>> seem to also apply to the "io_alloc" CBM also. >>>>> >>>>> It says "shared by other system functions or used for some other purpose >>>>> not under the control of the PQOS feature set". >>>> >>>> This is a quote from the AMD spec, not the resctrl user interface documentation. >>>> >>>> Please consider this from resctrl user interface perspective. >>>> >>>>> >>>>> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits". >>>> >>>> When I read the resctrl user interface documentation for "shareable_bits" it >>>> sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of >>>> shareable resource with other executing entities (e.g. I/O)" ... is this >>>> not exactly io_alloc? >>> >>> I agree the text is pretty generic. Actually, the whole bit mask (0xfffF) is shareable with io_alloc. >> >> I think the value of "shareable_bits" presented to user space could be the >> actual io_alloc_cbm value. Thus, not the "possible IO bitmask" but the actual > > Confused little bit here. The shareable_bits is resource property. > io_alloc_cbm is domain specific value. Not sure how to display it. ah, right. We should still aim to not let the new "io_alloc" interfaces cause confusion with the existing "shareable_bits" that is already part of interface and used to communicate IO allocation to user space. Perhaps we are just left with needing a modification to the existing documentation surrounding "shareable_bits"? > >> value. This seems to be the best match of what "shareable_bits" represents, which >> is the region currently used by IO devices. This partners well with the "bit_usage" >> output, for example, "X" can be used to show which portions of cache are currently >> used by both software (via schemata of resource groups) and hardware (via io_alloc_cbm). > > Haven't looked at this code much. Will look into it. Thank you. This is per domain so should be a good fit ... but again the documentation creates strong connection with "shareable_bits" that should be reworked. Reinette
Hi Reinette, On 4/9/2025 10:59 PM, Reinette Chatre wrote: > Hi Babu, > > On 4/9/25 5:58 PM, Moger, Babu wrote: >> Hi Reinette, >> >> On 4/8/2025 8:41 PM, Reinette Chatre wrote: >>> Hi Babu, >>> >>> On 4/8/25 5:41 PM, Moger, Babu wrote: >>>> Hi Reinette, >>>> >>>> On 4/8/2025 4:44 PM, Reinette Chatre wrote: >>>>> Hi Babu, >>>>> >>>>> On 4/7/25 1:12 PM, Moger, Babu wrote: >>>>>> On 3/21/25 17:50, Reinette Chatre wrote: >>>>>>> On 1/30/25 1:20 PM, Babu Moger wrote: >>>>> >>>>> >>>>>>>> >>>>>>> >>>>>>> AMD also supports what is exposed to user space as "shareable_bits". According >>>>>>> to APM: >>>>>>> Depending on the implementation, some portions of the L3 Cache may be >>>>>>> shared by other system functions or used for some other purpose not >>>>>>> under the control of the PQOS feature set. The L3 Cache Allocation >>>>>>> Sharing Mask returned by CPUID Fn0000_0010_EBX_x1[L3ShareAllocMask] is a >>>>>>> bitmask that represents portions of the L3 that may be shared by those >>>>>>> functions. >>>>>> >>>>>> Here is the complete text. >>>>>> >>>>>> The L3 Cache allocation sharing mask (L3ShareAllocMask) returned in EBX by >>>>>> CPUID Fn0000_0010 with ECX=1 is a bit vector which represents portions of >>>>>> the cache which may be shared with other system entities or used for some >>>>>> other purpose not under the control of the QOS feature set. When software >>>>>> sets a bit in one of the L3_MASK_n registers at the same bit positions a >>>>>> bit in the L3ShareAllocMask, processors executing with the corresponding >>>>>> COS will competitively share that portion of the cache with the other >>>>>> function. If this mask is all 0’s, then there is no other entity in the >>>>>> system competing with the processors for use of the L3 cache. >>>>>> >>>>>> The "L3ShareAllocMask" is always reported as 0 on AMD systems. >>>>>> >>>>>>> Could you please include what (if any) the relationship is between the CBM >>>>>>> discoverable via Fn0000_0010_EBX_x1[L3ShareAllocMask] and the CBM of >>>>>>> "highest-supported L3_MASK_n register" when SDCIAE is enabled? >>>>>> >>>>>> No. There is no relationship in here. >>>>>> >>>>>>> >>>>>>> On the resctrl interface side the documentation currently states: >>>>>>> >>>>>>> "shareable_bits": >>>>>>> Bitmask of shareable resource with other executing >>>>>>> entities (e.g. I/O). User can use this when >>>>>>> setting up exclusive cache partitions. Note that >>>>>>> some platforms support devices that have their >>>>>>> own settings for cache use which can over-ride >>>>>>> these bits. >>>>>>> >>>>>>> Even though this was originally used to expose the content of >>>>>>> Fn0000_0010_EBX_x1[L3ShareAllocMask] the intent of the content does >>>>>>> seem to also apply to the "io_alloc" CBM also. >>>>>> >>>>>> It says "shared by other system functions or used for some other purpose >>>>>> not under the control of the PQOS feature set". >>>>> >>>>> This is a quote from the AMD spec, not the resctrl user interface documentation. >>>>> >>>>> Please consider this from resctrl user interface perspective. >>>>> >>>>>> >>>>>> "io_alloc" is PQOS feature set. I feel it should not affect "shareable_bits". >>>>> >>>>> When I read the resctrl user interface documentation for "shareable_bits" it >>>>> sounds relevant to io_alloc. The "shareable_bits" contains a bitmask "of >>>>> shareable resource with other executing entities (e.g. I/O)" ... is this >>>>> not exactly io_alloc? >>>> >>>> I agree the text is pretty generic. Actually, the whole bit mask (0xfffF) is shareable with io_alloc. >>> >>> I think the value of "shareable_bits" presented to user space could be the >>> actual io_alloc_cbm value. Thus, not the "possible IO bitmask" but the actual >> >> Confused little bit here. The shareable_bits is resource property. >> io_alloc_cbm is domain specific value. Not sure how to display it. > > ah, right. We should still aim to not let the new "io_alloc" interfaces cause > confusion with the existing "shareable_bits" that is already part of interface and > used to communicate IO allocation to user space. Perhaps we are just left with > needing a modification to the existing documentation surrounding "shareable_bits"? Yes. Agree. Will add the details. Lets go from there. Thanks Babu
© 2016 - 2026 Red Hat, Inc.