Changes since v5?:
 * Allocate memory for kn->priv - MPAM can't repaint cache-ids anymore.
 * Instead of describing the limit on the number of CLOSID, remove it.
 * Fixed an off-by-one in the for_each walker.
 * Kept the rdt_find_domain() argument used to insert to the list.
 * Resource reset is now per-resource.
 * thread_throttle_mode_init() went and came back again.
 * Added support for throttle_mode on SMBA resources
 * Serialised rdtgroup_destroy_root() and check for duplicate calls.
 * Regex tweaking in the python script to cover more cases

~

This is the final series that allows other architectures to implement resctrl.
The final patch to move the code has been omitted, but can be generated using
the python script at the end of the series.
The final move is a bit of a monster. I don't expect that to get merged as part
of this series - we should wait for it to make less impact on other series.

Otherwise this series renames functions and moves code around. With the
exception of invalid configurations for the configurable-events, there should
be no changes in behaviour caused by this series. It is now possible for
throttle_mode to report 'undefined', but no known platform will do this.

The driving pattern is to make things like struct rdtgroup private to resctrl.
Features like pseudo-lock aren't going to work on arm64, the ability to disable
it at compile time is added.

After this, I can start posting the MPAM driver to make use of resctrl on arm64.
(What's MPAM? See the cover letter of the first series. [1])

This series is based on v6.14-rc1 and can be retrieved from:
https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/move_to_fs/v6

As ever - bugs welcome,
Thanks,

James

[v5] https://lore.kernel.org/r/20241004180347.19985-1-james.morse@arm.com
[v4] https://lore.kernel.org/all/20240802172853.22529-1-james.morse@arm.com/
[v3] https://lore.kernel.org/r/20240614150033.10454-1-james.morse@arm.com
[v2] https://lore.kernel.org/r/20240426150537.8094-1-Dave.Martin@arm.com
[v1] https://lore.kernel.org/r/20240321165106.31602-1-james.morse@arm.com
[1] https://lore.kernel.org/lkml/20201030161120.227225-1-james.morse@arm.com/


Amit Singh Tomar (1):
  x86/resctrl: Remove the limit on the number of CLOSID

James Morse (41):
  x86/resctrl: Fix allocation of cleanest CLOSID on platforms with no
    monitors
  x86/resctrl: Add a helper to avoid reaching into the arch code
    resource list
  x86/resctrl: Remove fflags from struct rdt_resource
  x86/resctrl: Use schema type to determine how to parse schema values
  x86/resctrl: Use schema type to determine the schema format string
  x86/resctrl: Remove data_width and the tabular format
  x86/resctrl: Add max_bw to struct resctrl_membw
  x86/resctrl: Generate default_ctrl instead of sharing it
  x86/resctrl: Add helper for setting CPU default properties
  x86/resctrl: Remove rdtgroup from update_cpu_closid_rmid()
  x86/resctrl: Expose resctrl fs's init function to the rest of the
    kernel
  x86/resctrl: Move rdt_find_domain() to be visible to arch and fs code
  x86/resctrl: Move resctrl types to a separate header
  x86/resctrl: Add an arch helper to reset one resource
  x86/resctrl: Move monitor exit work to a resctrl exit call
  x86/resctrl: Move monitor init work to a resctrl init call
  x86/resctrl: Rewrite and move the for_each_*_rdt_resource() walkers
  x86/resctrl: Move the is_mbm_*_enabled() helpers to asm/resctrl.h
  x86/resctrl: Add resctrl_arch_is_evt_configurable() to abstract BMEC
  x86/resctrl: Change mon_event_config_{read,write}() to be arch helpers
  x86/resctrl: Move mba_mbps_default_event init to filesystem code
  x86/resctrl: Move mbm_cfg_mask to struct rdt_resource
  x86/resctrl: Add resctrl_arch_ prefix to pseudo lock functions
  x86/resctrl: Allow an architecture to disable pseudo lock
  x86/resctrl: Make prefetch_disable_bits belong to the arch code
  x86/resctrl: Make resctrl_arch_pseudo_lock_fn() take a plr
  x86/resctrl: Move RFTYPE flags to be managed by resctrl
  x86/resctrl: Handle throttle_mode for SMBA resources
  x86/resctrl: Move get_config_index() to a header
  x86/resctrl: Claim get_{mon,ctrl}_domain_from_cpu() helpers for
    resctrl
  x86/resctrl: Rename resctrl_sched_in() to begin with "resctrl_arch_"
  x86/resctrl: resctrl_exit() teardown resctrl but leave the mount point
  x86/resctrl: Drop __init/__exit on assorted symbols
  x86/resctrl: Move is_mba_sc() out of core.c
  x86/resctrl: Add end-marker to the resctrl_event_id enum
  x86/restrl: Expand the width of dom_id by replacing mon_data_bits
  x86/resctrl: Remove a newline to avoid confusing the code move script
  x86/resctrl: Split trace.h
  fs/resctrl: Add boiler plate for external resctrl code
  x86/resctrl: Move the filesystem bits to headers visible to fs/resctrl
  x86/resctrl: Add python script to move resctrl code to /fs/resctrl

 MAINTAINERS                                   |   2 +
 arch/Kconfig                                  |   8 +
 arch/x86/Kconfig                              |   6 +-
 arch/x86/include/asm/resctrl.h                |  43 +-
 arch/x86/kernel/cpu/resctrl/Makefile          |   8 +-
 arch/x86/kernel/cpu/resctrl/core.c            | 206 ++---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c     |  90 +-
 arch/x86/kernel/cpu/resctrl/internal.h        | 251 ++----
 arch/x86/kernel/cpu/resctrl/monitor.c         | 114 ++-
 arch/x86/kernel/cpu/resctrl/monitor_trace.h   |  31 +
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c     |  59 +-
 .../resctrl/{trace.h => pseudo_lock_trace.h}  |  24 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c        | 388 ++++++---
 arch/x86/kernel/process_32.c                  |   2 +-
 arch/x86/kernel/process_64.c                  |   2 +-
 fs/Kconfig                                    |   1 +
 fs/Makefile                                   |   1 +
 fs/resctrl/Kconfig                            |  37 +
 fs/resctrl/Makefile                           |   6 +
 fs/resctrl/ctrlmondata.c                      |   0
 fs/resctrl/internal.h                         |   0
 fs/resctrl/monitor.c                          |   0
 fs/resctrl/monitor_trace.h                    |   0
 fs/resctrl/pseudo_lock.c                      |   0
 fs/resctrl/pseudo_lock_trace.h                |   0
 fs/resctrl/rdtgroup.c                         |   0
 include/linux/resctrl.h                       | 273 +++++-
 include/linux/resctrl_types.h                 |  59 ++
 resctrl_copy_pasta.py                         | 812 ++++++++++++++++++
 29 files changed, 1821 insertions(+), 602 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/resctrl/monitor_trace.h
 rename arch/x86/kernel/cpu/resctrl/{trace.h => pseudo_lock_trace.h} (56%)
 create mode 100644 fs/resctrl/Kconfig
 create mode 100644 fs/resctrl/Makefile
 create mode 100644 fs/resctrl/ctrlmondata.c
 create mode 100644 fs/resctrl/internal.h
 create mode 100644 fs/resctrl/monitor.c
 create mode 100644 fs/resctrl/monitor_trace.h
 create mode 100644 fs/resctrl/pseudo_lock.c
 create mode 100644 fs/resctrl/pseudo_lock_trace.h
 create mode 100644 fs/resctrl/rdtgroup.c
 create mode 100644 include/linux/resctrl_types.h
 create mode 100644 resctrl_copy_pasta.py

-- 
2.39.2

Hi James,

I'd like to check in on what you said in [1]. It sounded as though you were
planning to look at the assignable counter work from an Arm/MPAM
perspective but that work has since progressed (now at V11 [2]) without 
input from Arm/MPAM perspective. As I understand assignable counters may benefit
MPAM and looking close to settled but it is difficult to gain confidence
in an interface that may (may not?) be used for MPAM without any feedback
from Arm/MPAM. I am trying to prevent future issues when/if MPAM needs to use
this new interface and find it confusing that there does not seem to be
any input from MPAM side. What am I missing?

Reinette

[1] https://lore.kernel.org/lkml/9479c799-86fc-4d9e-addb-66011ecae9c7@arm.com/
[2] https://lore.kernel.org/lkml/cover.1737577229.git.babu.moger@amd.com/

Hi Reinette,

On 10/02/2025 17:24, Reinette Chatre wrote:
> I'd like to check in on what you said in [1]. It sounded as though you were
> planning to look at the assignable counter work from an Arm/MPAM
> perspective but that work has since progressed (now at V11 [2]) without 
> input from Arm/MPAM perspective. As I understand assignable counters may benefit
> MPAM and looking close to settled but it is difficult to gain confidence
> in an interface that may (may not?) be used for MPAM without any feedback
> from Arm/MPAM. I am trying to prevent future issues when/if MPAM needs to use
> this new interface and find it confusing that there does not seem to be
> any input from MPAM side. What am I missing?

Shortly after that some 'new' Spectre issue turned up - unfortunately those rapidly
consume all the time available, and predicting them is, er, the nature of the problem.

This is still on my todo list, I think Dave is planning to look through the ABMC series
too. I had previously sent some comments, (words to the effect "works for me"), and shared
a branch with the MPAM tree rebased on top.

I'm part way though rebasing the MPAM tree on top of Babu's latest version, and still hope
to give some feedback based on testing it...


Thanks,

James

Hi James,

On 2/11/25 10:37 AM, James Morse wrote:
> Hi Reinette,
> 
> On 10/02/2025 17:24, Reinette Chatre wrote:
>> I'd like to check in on what you said in [1]. It sounded as though you were
>> planning to look at the assignable counter work from an Arm/MPAM
>> perspective but that work has since progressed (now at V11 [2]) without 
>> input from Arm/MPAM perspective. As I understand assignable counters may benefit
>> MPAM and looking close to settled but it is difficult to gain confidence
>> in an interface that may (may not?) be used for MPAM without any feedback
>> from Arm/MPAM. I am trying to prevent future issues when/if MPAM needs to use
>> this new interface and find it confusing that there does not seem to be
>> any input from MPAM side. What am I missing?
> 
> Shortly after that some 'new' Spectre issue turned up - unfortunately those rapidly
> consume all the time available, and predicting them is, er, the nature of the problem.

I understand how these things go. At the same time the new version you sent hinted to
me that you were able to come up for air to give some attention to resctrl upstream and
I grasped the opportunity to get your input on this item that moved along while you were
busy elsewhere.  

> 
> This is still on my todo list, I think Dave is planning to look through the ABMC series
> too. I had previously sent some comments, (words to the effect "works for me"), and shared
> a branch with the MPAM tree rebased on top.

To me the primary concern is the new resctrl files introduced by this work. To get an idea
of the work and the new files you need only consider Babu's detailed cover letter. These files
have changed since your previous comments so a fresh "works for me" will be appreciated.

> 
> I'm part way though rebasing the MPAM tree on top of Babu's latest version, and still hope
> to give some feedback based on testing it...

We tried to separate the arch and fs code logically but I am sure that it can be improved once
another architecture actually tries to use it. To me these internals seem a problem that
can be solved after merge of the assignable counter work, but addressing issues with the new
resctrl fs user interface after merge will be hard.

Reinette

Hi Reinette,

On Tue, Feb 11, 2025 at 11:29:18AM -0800, Reinette Chatre wrote:
> Hi James,
> 
> On 2/11/25 10:37 AM, James Morse wrote:
> > Hi Reinette,
> > 
> > On 10/02/2025 17:24, Reinette Chatre wrote:

[...]

> [...] the new version you sent hinted to
> me that you were able to come up for air to give some attention to resctrl upstream and
> I grasped the opportunity to get your input on this item that moved along while you were
> busy elsewhere.  
> 
> > 
> > This is still on my todo list, I think Dave is planning to look through the ABMC series
> > too. I had previously sent some comments, (words to the effect "works for me"), and shared
> > a branch with the MPAM tree rebased on top.
> 
> To me the primary concern is the new resctrl files introduced by this work. To get an idea
> of the work and the new files you need only consider Babu's detailed cover letter. These files
> have changed since your previous comments so a fresh "works for me" will be appreciated.

This is roughly what I was planning to try to do, so I will go ahead
with that.

[...]

Cheers
---Dave

Hi Reinette,

On Mon, Feb 10, 2025 at 6:24 PM Reinette Chatre
<reinette.chatre@intel.com> wrote:
>
> Hi James,
>
> I'd like to check in on what you said in [1]. It sounded as though you were
> planning to look at the assignable counter work from an Arm/MPAM
> perspective but that work has since progressed (now at V11 [2]) without
> input from Arm/MPAM perspective. As I understand assignable counters may benefit
> MPAM and looking close to settled but it is difficult to gain confidence
> in an interface that may (may not?) be used for MPAM without any feedback
> from Arm/MPAM. I am trying to prevent future issues when/if MPAM needs to use
> this new interface and find it confusing that there does not seem to be
> any input from MPAM side. What am I missing?

I've looked into monitor assignment on MPAM a little, so I'll share my findings.

Like with ABMC/BMEC, MPAM's counters can be configured to monitor
reads, writes, or both, so there are situations where it would be
useful to be able to assign 2 counters to the same group to be able to
break down the bandwidth between reads and writes. However, a group's
two assignment slots are called "local" and "total", so if MPAM's
resources only support one of the two, then only one counter can be
assigned to a group.

MPAM does not support any filters that would differentiate between
traffic serviced by local or remote memory, so it's difficult to see
an MBM event other than "total" ever being used. Multiple MSCs
measuring memory bandwidth at an interconnect and a local memory
controller could potentially be used to together to infer the "local"
and "total" counts, but this would require the implementation to
understand the platform-specific relationship between different types
of MSCs and somehow present them as a single rdt_resource to resctrl.
As best as I can tell, the MPAM driver today will choose "local" or
"total"[1] for what it will present to the FS layer as an
rdt_resource.

Based on this, I would prefer the arch/fs refactoring changes go in
first to give us more time to think about how better to abstract
counter assignment on a non-RDTlike implementation. I believe finally
settling on an arch/fs separation for the currently-supported feature
set would make the counter assignment work clearer for everyone
involved. Also, my own users have been using an implementation like
this one successfully for over a year on ARM-based platforms while I'm
still just experimenting with the usage model of ABMC on AMD hardware,
so I consider the MPAM work to be more mature and would not like to
see it delayed on account of ABMC.

Thanks!
-Peter

[1] https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/tree/drivers/platform/arm64/mpam/mpam_resctrl.c?h=mpam/snapshot/v6.14-rc1#n824

>
> Reinette
>
> [1] https://lore.kernel.org/lkml/9479c799-86fc-4d9e-addb-66011ecae9c7@arm.com/
> [2] https://lore.kernel.org/lkml/cover.1737577229.git.babu.moger@amd.com/
>

Hi Peter/Reinette/James,

On 2/11/25 08:36, Peter Newman wrote:
> Hi Reinette,
> 
> On Mon, Feb 10, 2025 at 6:24 PM Reinette Chatre
> <reinette.chatre@intel.com> wrote:
>>
>> Hi James,
>>
>> I'd like to check in on what you said in [1]. It sounded as though you were
>> planning to look at the assignable counter work from an Arm/MPAM
>> perspective but that work has since progressed (now at V11 [2]) without
>> input from Arm/MPAM perspective. As I understand assignable counters may benefit
>> MPAM and looking close to settled but it is difficult to gain confidence
>> in an interface that may (may not?) be used for MPAM without any feedback
>> from Arm/MPAM. I am trying to prevent future issues when/if MPAM needs to use
>> this new interface and find it confusing that there does not seem to be
>> any input from MPAM side. What am I missing?
> 
> I've looked into monitor assignment on MPAM a little, so I'll share my findings.

Thanks.

> 
> Like with ABMC/BMEC, MPAM's counters can be configured to monitor
> reads, writes, or both, so there are situations where it would be
> useful to be able to assign 2 counters to the same group to be able to
> break down the bandwidth between reads and writes. However, a group's
> two assignment slots are called "local" and "total", so if MPAM's
> resources only support one of the two, then only one counter can be
> assigned to a group.

This can be done with current ABMC interface. Only one counter can be
assigned to the group.

> 
> MPAM does not support any filters that would differentiate between
> traffic serviced by local or remote memory, so it's difficult to see
> an MBM event other than "total" ever being used. Multiple MSCs
> measuring memory bandwidth at an interconnect and a local memory
> controller could potentially be used to together to infer the "local"
> and "total" counts, but this would require the implementation to
> understand the platform-specific relationship between different types
> of MSCs and somehow present them as a single rdt_resource to resctrl.
> As best as I can tell, the MPAM driver today will choose "local" or
> "total"[1] for what it will present to the FS layer as an
> rdt_resource.

That is still fine.

> 
> Based on this, I would prefer the arch/fs refactoring changes go in
> first to give us more time to think about how better to abstract
> counter assignment on a non-RDTlike implementation. I believe finally

I don't believe this series is ready for merging yet. It still needs to go
through the review process and a few more revisions. Based on our past
experience, the turnaround time from ARM has not been great, which will
likely delay this series by six to eight months.


> settling on an arch/fs separation for the currently-supported feature
> set would make the counter assignment work clearer for everyone
> involved. Also, my own users have been using an implementation like
> this one successfully for over a year on ARM-based platforms while I'm
> still just experimenting with the usage model of ABMC on AMD hardware,
> so I consider the MPAM work to be more mature and would not like to
> see it delayed on account of ABMC.

We've been working on ABMC for the past year, and it's almost ready for
merging. Now we have to wait? For how long?

On the higher level ABMC and assignment in MPAM looks similar. We added
the assignment interface, assuming it would be easily adapted to  MPAM.
We've incorporated all the requested changes—at least from Peter—but
haven't received much feedback from ARM.

James, could you take some time to review the interface and see if it can
be easily adapted for MPAM?

https://lore.kernel.org/lkml/cover.1737577229.git.babu.moger@amd.com/

I was planning to post the next version this week, but I can wait for
feedback.


> 
> Thanks!
> -Peter
> 
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/tree/drivers/platform/arm64/mpam/mpam_resctrl.c?h=mpam/snapshot/v6.14-rc1#n824
> 
>>
>> Reinette
>>
>> [1] https://lore.kernel.org/lkml/9479c799-86fc-4d9e-addb-66011ecae9c7@arm.com/
>> [2] https://lore.kernel.org/lkml/cover.1737577229.git.babu.moger@amd.com/
>>
> 

-- 
Thanks
Babu Moger

Hi Peter,

On 11/02/2025 14:36, Peter Newman wrote:
> On Mon, Feb 10, 2025 at 6:24 PM Reinette Chatre
> <reinette.chatre@intel.com> wrote:
>> I'd like to check in on what you said in [1]. It sounded as though you were
>> planning to look at the assignable counter work from an Arm/MPAM
>> perspective but that work has since progressed (now at V11 [2]) without
>> input from Arm/MPAM perspective. As I understand assignable counters may benefit
>> MPAM and looking close to settled but it is difficult to gain confidence
>> in an interface that may (may not?) be used for MPAM without any feedback
>> from Arm/MPAM. I am trying to prevent future issues when/if MPAM needs to use
>> this new interface and find it confusing that there does not seem to be
>> any input from MPAM side. What am I missing?
> 
> I've looked into monitor assignment on MPAM a little, so I'll share my findings.
> 
> Like with ABMC/BMEC, MPAM's counters can be configured to monitor
> reads, writes, or both, so there are situations where it would be
> useful to be able to assign 2 counters to the same group to be able to
> break down the bandwidth between reads and writes. However, a group's
> two assignment slots are called "local" and "total", so if MPAM's
> resources only support one of the two, then only one counter can be
> assigned to a group.

Wouldn't this be a problem on AMD too?
... specifically 2 counters with different configurations to the same group ...


I suspect it may be simpler to support complex things like that via perf.
I'd dropped that in favour of ABMC, but one platform has come out of the woodwork where
there are only monitors on the L2 - and I don't think we should expose new counter files
via resctrl...


> MPAM does not support any filters that would differentiate between
> traffic serviced by local or remote memory, so it's difficult to see
> an MBM event other than "total" ever being used.

The driver guesses from the topology! If the counters used are on the L3, chances are they
are local to a NUMA node. If they're on the memory controller, its probably total.

That code does need tightening up to check the cache boundaries match the numa boundaries
- but I haven't found a machine to test the bandwidth counters on at all yet.

I don't see how this would change what resctrl exposes - mbm_local and mbm_total already
exist. It's up to the MPAM driver to best match what it has with what it can exposed to
user-space...


> Multiple MSCs
> measuring memory bandwidth at an interconnect and a local memory
> controller could potentially be used to together to infer the "local"
> and "total" counts, but this would require the implementation to
> understand the platform-specific relationship between different types
> of MSCs and somehow present them as a single rdt_resource to resctrl.
> As best as I can tell, the MPAM driver today will choose "local" or
> "total"[1] for what it will present to the FS layer as an
> rdt_resource.

I think 'both' should fall out of that logic. It should keep moving the 'total' bandwidth
counter down the hierarchy until it reaches the memory controller.
I'd expect a platform that looks like this to have bandwidth monitors on the L3 (or
whatever cache matches the NUMA boundary) and bandwidth monitors on the memory controller.

Having two sets of bandwidth counters that measure different things in the same MSC is not
something that can be described by the firmware tables. (I did ask)

I think the logic here would be contained to the MPAM driver...


Thanks,

James

> Based on this, I would prefer the arch/fs refactoring changes go in
> first to give us more time to think about how better to abstract
> counter assignment on a non-RDTlike implementation. I believe finally
> settling on an arch/fs separation for the currently-supported feature
> set would make the counter assignment work clearer for everyone
> involved. Also, my own users have been using an implementation like
> this one successfully for over a year on ARM-based platforms while I'm
> still just experimenting with the usage model of ABMC on AMD hardware,
> so I consider the MPAM work to be more mature and would not like to
> see it delayed on account of ABMC.