[RFC PATCH 04/19] fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation

Babu Moger posted 19 patches 2 weeks, 3 days ago
[RFC PATCH 04/19] fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation
Posted by Babu Moger 2 weeks, 3 days ago
Add the documentation and example to setup Global Memory Bandwidth
Allocation (GMBA) in resctrl filesystem.

Signed-off-by: Babu Moger <babu.moger@amd.com>
---
 Documentation/filesystems/resctrl.rst | 43 +++++++++++++++++++++++++--
 1 file changed, 41 insertions(+), 2 deletions(-)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index 94187dd3c244..6ff6162719e8 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -28,6 +28,7 @@ SMBA (Slow Memory Bandwidth Allocation)				""
 BMEC (Bandwidth Monitoring Event Configuration)			""
 ABMC (Assignable Bandwidth Monitoring Counters)			""
 SDCIAE (Smart Data Cache Injection Allocation Enforcement)	""
+GMBA (Global Memory Bandwidth Allocation)                       ""
 =============================================================== ================================
 
 Historically, new features were made visible by default in /proc/cpuinfo. This
@@ -960,6 +961,21 @@ Memory bandwidth domain is L3 cache.
 
 	MB:<cache_id0>=bw_MiBps0;<cache_id1>=bw_MiBps1;...
 
+Global Memory bandwidth Allocation
+-----------------------------------
+
+AMD hardware supports Global Memory Bandwidth Allocation (GMBA) provides
+a mechanism for software to specify bandwidth limits for groups of threads
+that span across multiple QoS domains. This collection of QOS domains is
+referred to as GMBA control domain. The GMBA control domain is created by
+setting the same GMBA limits in one or more QoS domains. Setting the default
+max_bandwidth excludes the QoS domain from being part of GMBA control domain.
+
+Global Memory b/w domain is L3 cache.
+::
+
+	GMB:<cache_id0>=bandwidth;<cache_id1>=bandwidth;...
+
 Slow Memory Bandwidth Allocation (SMBA)
 ---------------------------------------
 AMD hardware supports Slow Memory Bandwidth Allocation (SMBA).
@@ -997,8 +1013,8 @@ which you wish to change.  E.g.
 Reading/writing the schemata file (on AMD systems)
 --------------------------------------------------
 Reading the schemata file will show the current bandwidth limit on all
-domains. The allocated resources are in multiples of one eighth GB/s.
-When writing to the file, you need to specify what cache id you wish to
+domains. The allocated resources are in multiples of 1/8 GB/s. When
+writing to the file, you need to specify what cache id you wish to
 configure the bandwidth limit.
 
 For example, to allocate 2GB/s limit on the first cache id:
@@ -1014,6 +1030,29 @@ For example, to allocate 2GB/s limit on the first cache id:
     MB:0=2048;1=  16;2=2048;3=2048
     L3:0=ffff;1=ffff;2=ffff;3=ffff
 
+Reading/writing the schemata file (on AMD systems) with GMBA feature
+--------------------------------------------------------------------
+Reading the schemata file will show the current bandwidth limit on all
+domains. The allocated resources are in multiples of 1 GB/s. The GMBA
+control domain is created by setting the same GMBA limits in one or more
+QoS domains.
+
+For example, to configure a GMBA domain consisting of domains 0 and 2
+with an 8 GB/s limit:
+
+::
+
+  # cat schemata
+    GMB:0=2048;1=2048;2=2048;3=2048
+     MB:0=4096;1=4096;2=4096;3=4096
+     L3:0=ffff;1=ffff;2=ffff;3=ffff
+
+  # echo "GMB:0=8;2=8" > schemata
+  # cat schemata
+    GMB:0=   8;1=2048;2=   8;3=2048
+     MB:0=4096;1=4096;2=4096;3=4096
+     L3:0=ffff;1=ffff;2=ffff;3=ffff
+
 Reading/writing the schemata file (on AMD systems) with SMBA feature
 --------------------------------------------------------------------
 Reading and writing the schemata file is the same as without SMBA in
-- 
2.34.1
Re: [RFC PATCH 04/19] fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation
Posted by Luck, Tony 5 days, 8 hours ago
On Wed, Jan 21, 2026 at 03:12:42PM -0600, Babu Moger wrote:
> +Global Memory bandwidth Allocation
> +-----------------------------------
> +
> +AMD hardware supports Global Memory Bandwidth Allocation (GMBA) provides
> +a mechanism for software to specify bandwidth limits for groups of threads
> +that span across multiple QoS domains. This collection of QOS domains is
> +referred to as GMBA control domain. The GMBA control domain is created by
> +setting the same GMBA limits in one or more QoS domains. Setting the default
> +max_bandwidth excludes the QoS domain from being part of GMBA control domain.

I don't see any checks that the user sets the *SAME* GMBA limits.

What happens if the user ignores the dosumentation and sets different
limits?

... snip ...

+  # cat schemata
+    GMB:0=2048;1=2048;2=2048;3=2048
+     MB:0=4096;1=4096;2=4096;3=4096
+     L3:0=ffff;1=ffff;2=ffff;3=ffff
+
+  # echo "GMB:0=8;2=8" > schemata
+  # cat schemata
+    GMB:0=   8;1=2048;2=   8;3=2048
+     MB:0=4096;1=4096;2=4096;3=4096
+     L3:0=ffff;1=ffff;2=ffff;3=ffff

Can the user go on to set:

   # echo "GMB:1=10;3=10" > schemata

and have domains 0 & 2 with a combined 8GB limit,
while domains 1 & 3 run with a combined 10GB limit?
Or is there a single "GMBA domain"?

Will using "2048" as the "this domain isn't limited
by GMBA" value come back to haunt you when some
system has much more than 2TB bandwidth to divide up?

Should resctrl have a non-numeric "unlimited" value
in the schemata file for this?

The "mba_MBps" feature used U32_MAX as the unlimited
value. But it looks somewhat ugly in the schemata
file:

	$ cat schemata
	MB:0=4294967295;1=4294967295
	L3:0=fff;1=fff

so I'm not sure it is a great precedent.


-Tony
Re: [RFC PATCH 04/19] fs/resctrl: Add the documentation for Global Memory Bandwidth Allocation
Posted by Babu Moger 4 days, 15 hours ago
Hi Tony,

On 2/2/26 18:00, Luck, Tony wrote:
> On Wed, Jan 21, 2026 at 03:12:42PM -0600, Babu Moger wrote:
>> +Global Memory bandwidth Allocation
>> +-----------------------------------
>> +
>> +AMD hardware supports Global Memory Bandwidth Allocation (GMBA) provides
>> +a mechanism for software to specify bandwidth limits for groups of threads
>> +that span across multiple QoS domains. This collection of QOS domains is
>> +referred to as GMBA control domain. The GMBA control domain is created by
>> +setting the same GMBA limits in one or more QoS domains. Setting the default
>> +max_bandwidth excludes the QoS domain from being part of GMBA control domain.
> I don't see any checks that the user sets the *SAME* GMBA limits.
>
> What happens if the user ignores the dosumentation and sets different
> limits?

Good point. Adding checks could be challenging when users update each 
schema individually with different values. We don't know which one value 
is the one he is intending to keep.

> ... snip ...
>
> +  # cat schemata
> +    GMB:0=2048;1=2048;2=2048;3=2048
> +     MB:0=4096;1=4096;2=4096;3=4096
> +     L3:0=ffff;1=ffff;2=ffff;3=ffff
> +
> +  # echo "GMB:0=8;2=8" > schemata
> +  # cat schemata
> +    GMB:0=   8;1=2048;2=   8;3=2048
> +     MB:0=4096;1=4096;2=4096;3=4096
> +     L3:0=ffff;1=ffff;2=ffff;3=ffff
>
> Can the user go on to set:
>
>     # echo "GMB:1=10;3=10" > schemata
>
> and have domains 0 & 2 with a combined 8GB limit,
> while domains 1 & 3 run with a combined 10GB limit?
> Or is there a single "GMBA domain"?

In that case, it  is still treated as a single GMBA domain, but the 
behavior becomes unpredictable. The hardware expert mentioned that it 
will default to the lowest value among all inputs in this case, 8GB.


> Will using "2048" as the "this domain isn't limited
> by GMBA" value come back to haunt you when some
> system has much more than 2TB bandwidth to divide up?

It is actually 4096 (4TB). I made a mistake in the example.  I am 
assuming it may not an issue in the current generation.

It is expected to go up in next generation.

GMB:0=4096;1=4096;2=4096;3=4096;
    MB:0=8192;1=8192;2=8192;3=8192;
     L3:0=ffff;1=ffff;2=ffff;3=ffff


>
> Should resctrl have a non-numeric "unlimited" value
> in the schemata file for this?

The value 4096 corresponds to 12th bit set.  It is called U-bit. If the 
U bit is set then that domain is not part of the GMBA domain.

I was thinking of displaying the "U" in those cases.  It may be good 
idea to do something like this.

GMB:0=      8;1=      U;2=     8 ;3=      U;
    MB:0=8192;1=8192;2=8192;3=8192;
     L3:0=ffff;1=ffff;2=ffff;3=ffff


>
> The "mba_MBps" feature used U32_MAX as the unlimited
> value. But it looks somewhat ugly in the schemata
> file:
Yes, I agree. Non-numeric would have been better.

Thanks

Babu