[PATCH 00/33] arm_mpam: Add basic mpam driver

Ben Horgan posted 33 patches 3 months ago
There is a newer version of this series
arch/arm64/Kconfig                  |   25 +
drivers/Kconfig                     |    2 +
drivers/Makefile                    |    1 +
drivers/acpi/arm64/Kconfig          |    3 +
drivers/acpi/arm64/Makefile         |    1 +
drivers/acpi/arm64/mpam.c           |  403 ++++
drivers/acpi/pptt.c                 |  334 +++-
drivers/acpi/tables.c               |    2 +-
drivers/resctrl/Kconfig             |   24 +
drivers/resctrl/Makefile            |    4 +
drivers/resctrl/mpam_devices.c      | 2729 +++++++++++++++++++++++++++
drivers/resctrl/mpam_internal.h     |  656 +++++++
drivers/resctrl/test_mpam_devices.c |  389 ++++
include/linux/acpi.h                |   26 +
include/linux/arm_mpam.h            |   66 +
include/linux/platform_device.h     |    1 +
16 files changed, 4611 insertions(+), 55 deletions(-)
create mode 100644 drivers/acpi/arm64/mpam.c
create mode 100644 drivers/resctrl/Kconfig
create mode 100644 drivers/resctrl/Makefile
create mode 100644 drivers/resctrl/mpam_devices.c
create mode 100644 drivers/resctrl/mpam_internal.h
create mode 100644 drivers/resctrl/test_mpam_devices.c
create mode 100644 include/linux/arm_mpam.h
[PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Ben Horgan 3 months ago
Hi all,

This version of the series comes to you from me as James is otherwise
engaged. I hope I have done his work justice. I've made quite a few
changes, rework, bugs, typos, all the usual. In order to aid review,
as Jonathan suggested, I've split out some patches and made an effort
to minimise the amount of churn between patches.

It would be great to get this taken up quickly. There are lots more
patches to come before we have a working MPAM story and this driver is
hidden behind the expert config. All reviews, comments, testing
welcomed and thank you for all the feedback so far.

See below for a public branch. No public updated version of the
snapshot (the rest of the driver) I'm afraid.

Changelogs in the patches.

Previous cover letter from James:

This is just enough MPAM driver for ACPI. DT got ripped out. If you need DT
support - please share your DTS so the DT folk know the binding is what is
needed.
This doesn't contain any of the resctrl code, meaning you can't actually drive it
from user-space yet. Because of that, its hidden behind CONFIG_EXPERT.
This will change once the user interface is connected up.

This is the initial group of patches that allows the resctrl code to be built
on top. Including that will increase the number of trees that may need to
coordinate, so breaking it up make sense.

The locking got simplified, but is still strange - this is because of the 'mpam-fb'
firmware interface specification that is still alpha. That thing needs to wait for
an interrupt after every system register write, which significantly impacts the
driver. Some features just won't work, e.g. reading the monitor registers via
perf.

I've not found a platform that can test all the behaviours around the monitors,
so this is where I'd expect the most bugs.

The MPAM spec that describes all the system and MMIO registers can be found here:
https://developer.arm.com/documentation/ddi0598/db/?lang=en
(Ignored the 'RETIRED' warning - that is just arm moving the documentation around.
 This document has the best overview)

The expectation is this will go via the arm64 tree.

This series is based on v6.18-rc4, and can be retrieved from:
https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/driver/v4

The rest of the driver can be found here: (no updated version - based on v3)
https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/snapshot/v6.18-rc1

What is MPAM? Set your time-machine to 2020:
https://lore.kernel.org/lkml/20201030161120.227225-1-james.morse@arm.com/

This series was previously posted here:
[v3] https://lore.kernel.org/linux-arm-kernel/20251017185645.26604-1-james.morse@arm.com/
[v2] lore.kernel.org/r/20250910204309.20751-1-james.morse@arm.com
[v1] lore.kernel.org/r/20250822153048.2287-1-james.morse@arm.com
[RFC] lore.kernel.org/r/20250711183648.30766-2-james.morse@arm.com

Ben Horgan (4):
  ACPI / PPTT: Add acpi_pptt_cache_v1_full to use pptt cache as one
    structure
  platform: Define platform_device_put cleanup handler
  ACPI: Define acpi_put_table cleanup handler and acpi_get_table_ret()
    helper
  arm_mpam: Consider overflow in bandwidth counter state

James Morse (27):
  ACPI / PPTT: Add a helper to fill a cpumask from a processor container
  ACPI / PPTT: Stop acpi_count_levels() expecting callers to clear
    levels
  ACPI / PPTT: Find cache level by cache-id
  ACPI / PPTT: Add a helper to fill a cpumask from a cache_id
  arm64: kconfig: Add Kconfig entry for MPAM
  ACPI / MPAM: Parse the MPAM table
  arm_mpam: Add probe/remove for mpam msc driver and kbuild boiler plate
  arm_mpam: Add the class and component structures for firmware
    described ris
  arm_mpam: Add MPAM MSC register layout definitions
  arm_mpam: Add cpuhp callbacks to probe MSC hardware
  arm_mpam: Probe hardware to find the supported partid/pmg values
  arm_mpam: Add helpers for managing the locking around the mon_sel
    registers
  arm_mpam: Probe the hardware features resctrl supports
  arm_mpam: Merge supported features during mpam_enable() into
    mpam_class
  arm_mpam: Reset MSC controls from cpuhp callbacks
  arm_mpam: Add a helper to touch an MSC from any CPU
  arm_mpam: Extend reset logic to allow devices to be reset any time
  arm_mpam: Register and enable IRQs
  arm_mpam: Use a static key to indicate when mpam is enabled
  arm_mpam: Allow configuration to be applied and restored during cpu
    online
  arm_mpam: Probe and reset the rest of the features
  arm_mpam: Add helpers to allocate monitors
  arm_mpam: Add mpam_msmon_read() to read monitor value
  arm_mpam: Track bandwidth counter state for power management
  arm_mpam: Add helper to reset saved mbwu state
  arm_mpam: Add kunit test for bitmap reset
  arm_mpam: Add kunit tests for props_mismatch()

Rohit Mathew (2):
  arm_mpam: Probe for long/lwd mbwu counters
  arm_mpam: Use long MBWU counters if supported

 arch/arm64/Kconfig                  |   25 +
 drivers/Kconfig                     |    2 +
 drivers/Makefile                    |    1 +
 drivers/acpi/arm64/Kconfig          |    3 +
 drivers/acpi/arm64/Makefile         |    1 +
 drivers/acpi/arm64/mpam.c           |  403 ++++
 drivers/acpi/pptt.c                 |  334 +++-
 drivers/acpi/tables.c               |    2 +-
 drivers/resctrl/Kconfig             |   24 +
 drivers/resctrl/Makefile            |    4 +
 drivers/resctrl/mpam_devices.c      | 2729 +++++++++++++++++++++++++++
 drivers/resctrl/mpam_internal.h     |  656 +++++++
 drivers/resctrl/test_mpam_devices.c |  389 ++++
 include/linux/acpi.h                |   26 +
 include/linux/arm_mpam.h            |   66 +
 include/linux/platform_device.h     |    1 +
 16 files changed, 4611 insertions(+), 55 deletions(-)
 create mode 100644 drivers/acpi/arm64/mpam.c
 create mode 100644 drivers/resctrl/Kconfig
 create mode 100644 drivers/resctrl/Makefile
 create mode 100644 drivers/resctrl/mpam_devices.c
 create mode 100644 drivers/resctrl/mpam_internal.h
 create mode 100644 drivers/resctrl/test_mpam_devices.c
 create mode 100644 include/linux/arm_mpam.h

-- 
2.43.0
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Ben Horgan 2 months, 4 weeks ago
Hi all,

On 11/7/25 12:34, Ben Horgan wrote:

> 
> The rest of the driver can be found here: (no updated version - based on v3)
> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/snapshot/v6.18-rc1
> 

James has kindly hosted a branch with my rebase of the rest of the
driver here.
https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/log/?h=mpam/snapshot/v6.18-rc4

This can be used to help with more testing.

Thanks,

Ben
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Drew Fustini 2 months, 3 weeks ago
On Fri, Nov 07, 2025 at 12:34:17PM +0000, Ben Horgan wrote:
> Hi all,
[snip]
> The rest of the driver can be found here: (no updated version - based on v3)
> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/snapshot/v6.18-rc1

Does anyone know of a hosting platform that offers ARM machines that
have MPAM?

I see there are Ampere systems on Oracle but I wasn't sure if those have
MPAM.

I'm getting a RISC-V QoS patch series ready (Ssqosid ext and CBQRI spec)
and I'd like to get a better understanding of how resctrl works on MPAM
in practice.

Thanks,
Drew
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Ben Horgan 2 months, 3 weeks ago
Hi Drew,

On 11/16/25 17:16, Drew Fustini wrote:
> On Fri, Nov 07, 2025 at 12:34:17PM +0000, Ben Horgan wrote:
>> Hi all,
> [snip]
>> The rest of the driver can be found here: (no updated version - based on v3)
>> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/snapshot/v6.18-rc1
> 
> Does anyone know of a hosting platform that offers ARM machines that
> have MPAM?

As far as I know there aren't any.

There is some MPAM support in the Orion Radxa board which is likely the
cheapest option. The MPAM acpi table isn't in the firmware though so
you'd need to load a custom table. James has this working.

> 
> I see there are Ampere systems on Oracle but I wasn't sure if those have
> MPAM.
> 
> I'm getting a RISC-V QoS patch series ready (Ssqosid ext and CBQRI spec)
> and I'd like to get a better understanding of how resctrl works on MPAM
> in practice.
> 
> Thanks,
> Drew


Thanks,

Ben
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Drew Fustini 2 months, 3 weeks ago
On Tue, Nov 18, 2025 at 02:11:31PM +0000, Ben Horgan wrote:
> Hi Drew,
> 
> On 11/16/25 17:16, Drew Fustini wrote:
> > On Fri, Nov 07, 2025 at 12:34:17PM +0000, Ben Horgan wrote:
> >> Hi all,
> > [snip]
> >> The rest of the driver can be found here: (no updated version - based on v3)
> >> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/snapshot/v6.18-rc1
> > 
> > Does anyone know of a hosting platform that offers ARM machines that
> > have MPAM?
> 
> As far as I know there aren't any.
> 
> There is some MPAM support in the Orion Radxa board which is likely the
> cheapest option. The MPAM acpi table isn't in the firmware though so
> you'd need to load a custom table. James has this working.

Thank you, I didn't realize that there was a dev board that supports
MPAM. I didn't want to the expense or noise of a rackable server :)

Drew
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Jonathan Cameron 2 months, 2 weeks ago
On Tue, 18 Nov 2025 14:55:07 -0800
Drew Fustini <fustini@kernel.org> wrote:

> On Tue, Nov 18, 2025 at 02:11:31PM +0000, Ben Horgan wrote:
> > Hi Drew,
> > 
> > On 11/16/25 17:16, Drew Fustini wrote:  
> > > On Fri, Nov 07, 2025 at 12:34:17PM +0000, Ben Horgan wrote:  
> > >> Hi all,  
> > > [snip]  
> > >> The rest of the driver can be found here: (no updated version - based on v3)
> > >> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/snapshot/v6.18-rc1  
> > > 
> > > Does anyone know of a hosting platform that offers ARM machines that
> > > have MPAM?  
> > 
> > As far as I know there aren't any.
> > 
> > There is some MPAM support in the Orion Radxa board which is likely the
> > cheapest option. The MPAM acpi table isn't in the firmware though so
> > you'd need to load a custom table. James has this working.  
> 
> Thank you, I didn't realize that there was a dev board that supports
> MPAM. I didn't want to the expense or noise of a rackable server :)
> 
> Drew
> 
Hi Drew,

Obvious not functional as such, but I did spin qemu emulation with a bunch
of introspection so you could see what was configured.  Aim was to poke
corner cases more easily than with real hardware. Did it's job at the time
and shook out some bugs.

I haven't rebased it recently though.
https://lore.kernel.org/qemu-devel/20230808115713.2613-1-Jonathan.Cameron@huawei.com/

https://gitlab.com/jic23/qemu/-/commits/mpam-2023-sept-01
Has what looks to be a slightly more recent rebase.

No monitor support though.

I might bring this back to poke the rest of this series as it moves forwards
(or if anyone else wants to they are welcome to do so)

FWIW we could in theory hook this up to the cache plugins to get some 'plausible'
numbers, but I never bothered as we have hardware (as seen by tested-by's on this
series).

Jonathan
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Drew Fustini 2 months, 2 weeks ago
On Wed, Nov 19, 2025 at 10:00:51AM +0000, Jonathan Cameron wrote:
> On Tue, 18 Nov 2025 14:55:07 -0800
> Drew Fustini <fustini@kernel.org> wrote:
> 
> > On Tue, Nov 18, 2025 at 02:11:31PM +0000, Ben Horgan wrote:
> > > Hi Drew,
> > > 
> > > On 11/16/25 17:16, Drew Fustini wrote:  
> > > > On Fri, Nov 07, 2025 at 12:34:17PM +0000, Ben Horgan wrote:  
> > > >> Hi all,  
> > > > [snip]  
> > > >> The rest of the driver can be found here: (no updated version - based on v3)
> > > >> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/snapshot/v6.18-rc1  
> > > > 
> > > > Does anyone know of a hosting platform that offers ARM machines that
> > > > have MPAM?  
> > > 
> > > As far as I know there aren't any.
> > > 
> > > There is some MPAM support in the Orion Radxa board which is likely the
> > > cheapest option. The MPAM acpi table isn't in the firmware though so
> > > you'd need to load a custom table. James has this working.  
> > 
> > Thank you, I didn't realize that there was a dev board that supports
> > MPAM. I didn't want to the expense or noise of a rackable server :)
> > 
> > Drew
> > 
> Hi Drew,
> 
> Obvious not functional as such, but I did spin qemu emulation with a bunch
> of introspection so you could see what was configured.  Aim was to poke
> corner cases more easily than with real hardware. Did it's job at the time
> and shook out some bugs.
> 
> I haven't rebased it recently though.
> https://lore.kernel.org/qemu-devel/20230808115713.2613-1-Jonathan.Cameron@huawei.com/
> 
> https://gitlab.com/jic23/qemu/-/commits/mpam-2023-sept-01
> Has what looks to be a slightly more recent rebase.
> 
> No monitor support though.
> 
> I might bring this back to poke the rest of this series as it moves forwards
> (or if anyone else wants to they are welcome to do so)
> 
> FWIW we could in theory hook this up to the cache plugins to get some 'plausible'
> numbers, but I never bothered as we have hardware (as seen by tested-by's on this
> series).
> 
> Jonathan

Thanks for pointing out your series and the branch from James. Qemu is
the cheapest way for me to try MPAM :)

Drew
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Ben Horgan 3 months ago
This is a v4. Apologies for forgetting the version number!

On 11/7/25 12:34, Ben Horgan wrote:
> Hi all,
> 
> This version of the series comes to you from me as James is otherwise
> engaged. I hope I have done his work justice. I've made quite a few
Thanks,

Ben
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Fenghua Yu 3 months ago

On 11/7/25 04:34, Ben Horgan wrote:
> Hi all,
> 
> This version of the series comes to you from me as James is otherwise
> engaged. I hope I have done his work justice. I've made quite a few
> changes, rework, bugs, typos, all the usual. In order to aid review,
> as Jonathan suggested, I've split out some patches and made an effort
> to minimise the amount of churn between patches.

[SNIP]

> This series is based on v6.18-rc4, and can be retrieved from:
> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/driver/v4

Build, boot, ACPI parse, and Kunit tests pass.

> 
> The rest of the driver can be found here: (no updated version - based on v3)
> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/snapshot/v6.18-rc1
[SNIP]

Thanks.

-Fenghua
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Carl Worth 3 months ago
Ben Horgan <ben.horgan@arm.com> writes:
> This version of the series comes to you from me as James is otherwise
> engaged. I hope I have done his work justice. I've made quite a few
> changes, rework, bugs, typos, all the usual. In order to aid review,
> as Jonathan suggested, I've split out some patches and made an effort
> to minimise the amount of churn between patches.

I've built this and booted on an Ampere system. It ends up reporting a
successful message of:

	MPAM enabled with 1 PARTIDs and 1 PMGs

So the code seems happy enough as far as that goes.

But the expected number of PARTIDs on this system is much larger than 1,
(MPAM with a single PARTID would not be useful at all).

> See below for a public branch. No public updated version of the
> snapshot (the rest of the driver) I'm afraid.

Looking closer, it looks like the bogus value of 0 for mpam_partid_max
is because the following patch (which does appear in James' various
snapshots) isn't present yet in the code submitted to this point:

	commit 33c1f50970917ac9f2a8e224d850936374df6173
	Author: James Morse <james.morse@arm.com>
	Date:   Fri Jul 4 14:22:30 2025 +0100
	
	    arm64: mpam: Advertise the CPUs MPAM limits to the driver
	    
	    Requestors need to populate the MPAM fields on the interconnect. For
	    the CPUs these fields are taken from the corresponding MPAMy_ELx
	    register. Each requestor may have a limit on the largest PARTID or
	    PMG value that can be used. The MPAM driver has to determine the
	    system-wide minimum supported PARTID and PMG values.
	    
	    To do this, the driver needs to be told what each requestor's
	    limit is.

So, I guess I'm wondering what more I could do to test this code at this
point, prior to merging it.

I'm very interested in seeing this code land upstream as soon as
possible, and I've got access to an implementation to test whatever I
can.

So let me know what else I can do and I'll be glad to contribute my
Tested-by when I've done it.

-Carl
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Ben Horgan 2 months, 4 weeks ago
Hi Carl,

On 11/7/25 23:22, Carl Worth wrote:
> Ben Horgan <ben.horgan@arm.com> writes:
>> This version of the series comes to you from me as James is otherwise
>> engaged. I hope I have done his work justice. I've made quite a few
>> changes, rework, bugs, typos, all the usual. In order to aid review,
>> as Jonathan suggested, I've split out some patches and made an effort
>> to minimise the amount of churn between patches.
> 
> I've built this and booted on an Ampere system. It ends up reporting a
> successful message of:
> 
> 	MPAM enabled with 1 PARTIDs and 1 PMGs
> 
> So the code seems happy enough as far as that goes.
> 
> But the expected number of PARTIDs on this system is much larger than 1,
> (MPAM with a single PARTID would not be useful at all).
> 
>> See below for a public branch. No public updated version of the
>> snapshot (the rest of the driver) I'm afraid.
> 
> Looking closer, it looks like the bogus value of 0 for mpam_partid_max
> is because the following patch (which does appear in James' various
> snapshots) isn't present yet in the code submitted to this point:
> 
> 	commit 33c1f50970917ac9f2a8e224d850936374df6173
> 	Author: James Morse <james.morse@arm.com>
> 	Date:   Fri Jul 4 14:22:30 2025 +0100
> 	
> 	    arm64: mpam: Advertise the CPUs MPAM limits to the driver
> 	    
> 	    Requestors need to populate the MPAM fields on the interconnect. For
> 	    the CPUs these fields are taken from the corresponding MPAMy_ELx
> 	    register. Each requestor may have a limit on the largest PARTID or
> 	    PMG value that can be used. The MPAM driver has to determine the
> 	    system-wide minimum supported PARTID and PMG values.
> 	    
> 	    To do this, the driver needs to be told what each requestor's
> 	    limit is.

Yes, the driver does barely anything without a requestor.

> 
> So, I guess I'm wondering what more I could do to test this code at this
> point, prior to merging it.
> 
> I'm very interested in seeing this code land upstream as soon as
> possible, and I've got access to an implementation to test whatever I
> can.
> 
> So let me know what else I can do and I'll be glad to contribute my
> Tested-by when I've done it.
> 
> -Carl

Thanks for the quick response and testing.

I've just responded to the cover letter with a branch containing the
rest of the driver. (Just so it's not hidden in this thread) It's
https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/log/?h=mpam/snapshot/v6.18-rc4

With that, you should be able to enable all usable PARTID and PMG, mount
resctrl, add tasks/cpus to resctrl control groups, run benchmarks to
check that the controls are respected and check that the monitors give
expected values.

Thanks,

Ben
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Carl Worth 2 months, 4 weeks ago
Ben Horgan <ben.horgan@arm.com> writes:
> Yes, the driver does barely anything without a requestor.
...
> Thanks for the quick response and testing.

Sure thing!

> I've just responded to the cover letter with a branch containing the
> rest of the driver. (Just so it's not hidden in this thread) It's
> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/log/?h=mpam/snapshot/v6.18-rc4

Yes. I saw that. That's very helpful, so thank you!

> With that, you should be able to enable all usable PARTID and PMG, mount
> resctrl, add tasks/cpus to resctrl control groups, run benchmarks to
> check that the controls are respected and check that the monitors give
> expected values.

Indeed. I now get:

	MPAM enabled with 256 PARTIDs and 2 PMGs

And I have mounted resctrl, added cpus to resctrl control groups, and
then manipulated the L3 cache bitmap control and verified that
benchmarks respond as expected to changes to the control.

I was going to report that I am encountering an unexpected alignment
fault in one of the KUNIT tests, but then I realized that that's in
test_get_mba_granularity which comes in as part of the snapshot, and is
not part of the code that is submitted here. So I'll continue to
invetigate that, but that need not delay the code of interest in the
current series.

Given all of that, for the series:

Tested-by: Carl Worth <carl@os.amperecomputing.com>

-Carl
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Gavin Shan 2 months, 4 weeks ago
On 11/7/25 10:34 PM, Ben Horgan wrote:
[...]

> 
> This series is based on v6.18-rc4, and can be retrieved from:
> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/driver/v4
> 

Run booting and kunit test on NVidia's grace-hopper machine. The result looks good.

Tested-by: Gavin Shan <gshan@redhat.com>

Thanks,
Gavin
Re: [PATCH 00/33] arm_mpam: Add basic mpam driver
Posted by Zeng Heng 2 months, 4 weeks ago
On 2025/11/7 20:34, Ben Horgan wrote:

> 
> The expectation is this will go via the arm64 tree.
> 
> This series is based on v6.18-rc4, and can be retrieved from:
> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git mpam/driver/v4
> 

Run booting and kunit test on Huawei's Kunpeng machine. The result looks 
fine.

Tested-by: Zeng Heng <zengheng4@huawei.com>

Best Regards,
Zeng Heng