This series adds the support for L3 Smart Data Cache Injection Allocation
Enforcement (SDCIAE) to resctrl infrastructure. 

Upcoming AMD hardware implements Smart Data Cache Injection (SDCI).
Smart Data Cache Injection (SDCI) is a mechanism that enables direct
insertion of data from I/O devices into the L3 cache. By directly caching
data from I/O devices rather than first storing the I/O data in DRAM, SDCI
reduces demands on DRAM bandwidth and reduces latency to the processor
consuming the I/O data. The SDCIAE (SDCI Allocation Enforcement) PQE
feature allows system software to limit the portion of the L3 cache used
for SDCI.

The feature details are documented in the APM listed below [1].
[1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
Injection Allocation Enforcement (SDCIAE)
Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537

The feature requires linux support of TPH (TLP Processing Hints).
The support is ongoing and patches are currently under review.
https://lore.kernel.org/lkml/20240717205511.2541693-2-wei.huang2@amd.com/

The patches are based on top of commit
ad1b4b6ed19f (tip/master) Merge branch into tip/master: 'x86/timers'

Babu Moger (7):
  x86/cpufeatures: Add support for L3 Smart Data Cache Injection
    Allocation Enforcement
  x86/resctrl: Add SDCIAE feature in the command line options
  x86/resctrl: Introduce sdciae_capable in rdt_resource
  x86/resctrl: Implement SDCIAE enable/disable
  x86/resctrl: Add interface to enable/disable SDCIAE
  x86/resctrl: Introduce interface to display SDCIAE Capacity Bit Masks
  x86/resctrl: Introduce interface to modify SDCIAE Capacity Bit Masks

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/arch/x86/resctrl.rst            |  29 ++
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/msr-index.h              |   1 +
 arch/x86/kernel/cpu/cpuid-deps.c              |   1 +
 arch/x86/kernel/cpu/resctrl/core.c            |  10 +
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c     |   4 +-
 arch/x86/kernel/cpu/resctrl/internal.h        |  15 +
 arch/x86/kernel/cpu/resctrl/rdtgroup.c        | 298 ++++++++++++++++++
 arch/x86/kernel/cpu/scattered.c               |   1 +
 include/linux/resctrl.h                       |   2 +
 11 files changed, 361 insertions(+), 3 deletions(-)

-- 
2.34.1

Hi Babu,

On 8/16/24 9:16 AM, Babu Moger wrote:
> 
> This series adds the support for L3 Smart Data Cache Injection Allocation
> Enforcement (SDCIAE) to resctrl infrastructure.
> 
> Upcoming AMD hardware implements Smart Data Cache Injection (SDCI).
> Smart Data Cache Injection (SDCI) is a mechanism that enables direct
> insertion of data from I/O devices into the L3 cache. By directly caching
> data from I/O devices rather than first storing the I/O data in DRAM, SDCI
> reduces demands on DRAM bandwidth and reduces latency to the processor
> consuming the I/O data. The SDCIAE (SDCI Allocation Enforcement) PQE
> feature allows system software to limit the portion of the L3 cache used
> for SDCI.
> 

This series introduces new user interface. Could you please describe the
new user interface in the cover letter and how users are expected to interact
with this interface to be able to use this new feature? Please also describe
the impact on existing resctrl interfaces related to cache allocation from
I/O hardware, for example "shareable_bits", "bit_usage", etc. These existing
interfaces are used to communicate to user space how portions of cache are
used by I/O hardware but I cannot tell from this series how this work builds on
this.

How does this feature work with the existing "L3 Cache Allocation Sharing Mask"
that is enumerated as part of CAT feature?

> The feature details are documented in the APM listed below [1].
> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
> Injection Allocation Enforcement (SDCIAE)
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> 
> The feature requires linux support of TPH (TLP Processing Hints).
> The support is ongoing and patches are currently under review.
> https://lore.kernel.org/lkml/20240717205511.2541693-2-wei.huang2@amd.com/

Please note that the cover letter [1] of that series mentions "Cache Injection
allows PCIe endpoints to inject I/O Coherent DMA writes directly into an L2 ..."
while this series implements and refers to L3 only.

> 
> The patches are based on top of commit
> ad1b4b6ed19f (tip/master) Merge branch into tip/master: 'x86/timers'
> 

Reinette

[1] https://lore.kernel.org/lkml/20240717205511.2541693-1-wei.huang2@amd.com/

Hi Reinette,

Thanks for the feedback.

On 9/13/24 15:44, Reinette Chatre wrote:
> Hi Babu,
> 
> On 8/16/24 9:16 AM, Babu Moger wrote:
>>
>> This series adds the support for L3 Smart Data Cache Injection Allocation
>> Enforcement (SDCIAE) to resctrl infrastructure.
>>
>> Upcoming AMD hardware implements Smart Data Cache Injection (SDCI).
>> Smart Data Cache Injection (SDCI) is a mechanism that enables direct
>> insertion of data from I/O devices into the L3 cache. By directly caching
>> data from I/O devices rather than first storing the I/O data in DRAM, SDCI
>> reduces demands on DRAM bandwidth and reduces latency to the processor
>> consuming the I/O data. The SDCIAE (SDCI Allocation Enforcement) PQE
>> feature allows system software to limit the portion of the L3 cache used
>> for SDCI.
>>
> 
> This series introduces new user interface. Could you please describe the
> new user interface in the cover letter and how users are expected to interact
> with this interface to be able to use this new feature? Please also describe
> the impact on existing resctrl interfaces related to cache allocation from
> I/O hardware, for example "shareable_bits", "bit_usage", etc. These existing
> interfaces are used to communicate to user space how portions of cache are
> used by I/O hardware but I cannot tell from this series how this work
> builds on
> this.

Sure. Will add interface details and bit mask usage.

> 
> How does this feature work with the existing "L3 Cache Allocation Sharing
> Mask"
> that is enumerated as part of CAT feature?

Sharing is allowed as it is right now. SDCIAE cannot use the masks that
are exclusive.

> 
>> The feature details are documented in the APM listed below [1].
>> [1] AMD64 Architecture Programmer's Manual Volume 2: System Programming
>> Publication # 24593 Revision 3.41 section 19.4.7 L3 Smart Data Cache
>> Injection Allocation Enforcement (SDCIAE)
>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
>>
>> The feature requires linux support of TPH (TLP Processing Hints).
>> The support is ongoing and patches are currently under review.
>> https://lore.kernel.org/lkml/20240717205511.2541693-2-wei.huang2@amd.com/
> 
> Please note that the cover letter [1] of that series mentions "Cache
> Injection
> allows PCIe endpoints to inject I/O Coherent DMA writes directly into an
> L2 ..."
> while this series implements and refers to L3 only.

SDCIAE only applies to L3 insertion. SDCIAE has no impact on SDCI
insertion into the L2.

SDCI lines will be inserted into the L3 after being evicted by L2 based on
the maximum CLOS(CLOSIS 15) allocation mask when SDCIAE is enabled. By
limiting the number of ways in L3, it will impact the SDCI cache occupancy
in L3.
-- 
Thanks
Babu Moger