include/linux/mm.h | 4 ++- include/linux/mm_types.h | 7 ++++ kernel/sched/fair.c | 72 +++++++++++++++++++++++++++++----------- mm/huge_memory.c | 3 +- mm/internal.h | 5 +-- mm/mempolicy.c | 30 +++++++++++------ mm/migrate.c | 6 ++-- mm/mprotect.c | 4 ++- 8 files changed, 93 insertions(+), 38 deletions(-)
NUMA balancing uses hinting faults for both task placement and memory-tier
promotion. Several filters designed to avoid unproductive socket-placement
faults can also prevent promotion.
Read-only file mappings and VMAs without recent PID activity may never be
scanned, while lower MM layers reject some shared folios. Hot memory in
these mappings can therefore remain on a slow tier indefinitely.
Separate promotion-only scans from socket-placement scans.
MM_CP_PROT_NUMA_PROMO_ONLY carries that choice for one protection walk, so
the PTE and PMD paths can restrict hinting faults to promotion candidates.
Keeping this transient state in the protection flags avoids per-mm state and
its associated lifetime, concurrency, and VMA identity problems.
- Allow eligible shared folios to be promoted to a fast tier.
- Scan read-only file mappings and PID-inactive VMAs for promotion
without re-enabling placement sampling.
- Track the last placement scan separately so promotion-only scans
cannot postpone the existing placement-starvation fallback.
The series is ordered as follows:
1. Add promotion-only NUMA protection walks without changing behavior.
2. Permit eligible shared folios to be promoted to a fast tier.
3. Scan read-only file mappings using promotion-only scans.
4. Scan PID-inactive VMAs for promotion and account for placement scans
separately.
Tested on a host with 768GB/256GB DRAM/CXL.
Ran 2 ~430GB database workloads with large (>300GB) shmem VMAs.
Before change:
- 150-200GB/s DRAM bandwidth usage
- 40-45GB/s CXL bandwidth usage (maxed out)
- request latencies over 5ms (longer tails)
After chage:
- 250GB/s+ sustained DRAM bandwidth usage
- ~10GB/s sustained CXL bandwidth usage
- request latencies 800us-2ms.
Functional observation:
A 20 GB hash table VMA that previously remained entirely on CXL was
split evenly between DRAM and CXL after the changes - and tier
residency tracked hotness. This was previously affected by the
stavation issue caused by the "unaccessed VMA" filter.
Gregory Price (Meta) (4):
mm: support promotion-only NUMA hinting scans
mm: allow shared folios to be promoted to a fast tier
sched/numa: scan read-only file mappings in tiering mode
sched/numa: do not let VMA PID activity gate promotion
include/linux/mm.h | 4 ++-
include/linux/mm_types.h | 7 ++++
kernel/sched/fair.c | 72 +++++++++++++++++++++++++++++-----------
mm/huge_memory.c | 3 +-
mm/internal.h | 5 +--
mm/mempolicy.c | 30 +++++++++++------
mm/migrate.c | 6 ++--
mm/mprotect.c | 4 ++-
8 files changed, 93 insertions(+), 38 deletions(-)
--
2.55.0
On Thu, 10 Sep 2026 20:18:22 -0400 Gregory Price <gourry@gourry.net> wrote: > NUMA balancing uses hinting faults for both task placement and memory-tier > promotion. Several filters designed to avoid unproductive socket-placement > faults can also prevent promotion. > > Read-only file mappings and VMAs without recent PID activity may never be > scanned, while lower MM layers reject some shared folios. Hot memory in > these mappings can therefore remain on a slow tier indefinitely. > > Separate promotion-only scans from socket-placement scans. > > MM_CP_PROT_NUMA_PROMO_ONLY carries that choice for one protection walk, so > the PTE and PMD paths can restrict hinting faults to promotion candidates. > Keeping this transient state in the protection flags avoids per-mm state and > its associated lifetime, concurrency, and VMA identity problems. > > - Allow eligible shared folios to be promoted to a fast tier. > - Scan read-only file mappings and PID-inactive VMAs for promotion > without re-enabling placement sampling. > - Track the last placement scan separately so promotion-only scans > cannot postpone the existing placement-starvation fallback. > > The series is ordered as follows: > > 1. Add promotion-only NUMA protection walks without changing behavior. > 2. Permit eligible shared folios to be promoted to a fast tier. > 3. Scan read-only file mappings using promotion-only scans. > 4. Scan PID-inactive VMAs for promotion and account for placement scans > separately. > > Tested on a host with 768GB/256GB DRAM/CXL. > Ran 2 ~430GB database workloads with large (>300GB) shmem VMAs. > > Before change: > - 150-200GB/s DRAM bandwidth usage > - 40-45GB/s CXL bandwidth usage (maxed out) > - request latencies over 5ms (longer tails) > > After chage: > - 250GB/s+ sustained DRAM bandwidth usage > - ~10GB/s sustained CXL bandwidth usage > - request latencies 800us-2ms. > > Functional observation: > A 20 GB hash table VMA that previously remained entirely on CXL was > split evenly between DRAM and CXL after the changes - and tier > residency tracked hotness. This was previously affected by the > stavation issue caused by the "unaccessed VMA" filter. This seems very significant? Why cc:stable and Fixes:? Is this something which ran at these sorts of speeds before the offending commits?
On Wed, Sep 16, 2026 at 10:35:25PM -0700, Andrew Morton wrote: > On Thu, 10 Sep 2026 20:18:22 -0400 Gregory Price <gourry@gourry.net> wrote: > > > > > Functional observation: > > A 20 GB hash table VMA that previously remained entirely on CXL was > > split evenly between DRAM and CXL after the changes - and tier > > residency tracked hotness. This was previously affected by the > > stavation issue caused by the "unaccessed VMA" filter. > > This seems very significant? > > Why cc:stable and Fixes:? Is this something which ran at these sorts > of speeds before the offending commits? > It's actually that the changes functionally broke numa balancing to the point that it just became completely ineffective for very common use cases. We only didn't notice because another bug: https://lore.kernel.org/all/20260629163337.1264881-1-hannes@cmpxchg.org/ hid almost all these issues from being apparent on large workloads (which was differently horrendous - that bug made ~70-90% of some database workload memory ineligible for tiering) Once the shmem fix went in, all the starvation and filtering issues become more apparent when the bandwidth numbers stopped making sense (see above). Basically numa balancing has been broken since 2022/2023, and anyone who has done any testing with it since then has been working off of bad data (incomplete VMA coverage). ~Gregory
On 9/17/26 08:59, Gregory Price wrote: > On Wed, Sep 16, 2026 at 10:35:25PM -0700, Andrew Morton wrote: >> On Thu, 10 Sep 2026 20:18:22 -0400 Gregory Price <gourry@gourry.net> wrote: >> >>> >>> Functional observation: >>> A 20 GB hash table VMA that previously remained entirely on CXL was >>> split evenly between DRAM and CXL after the changes - and tier >>> residency tracked hotness. This was previously affected by the >>> stavation issue caused by the "unaccessed VMA" filter. >> >> This seems very significant? >> >> Why cc:stable and Fixes:? Is this something which ran at these sorts >> of speeds before the offending commits? >> > > It's actually that the changes functionally broke numa balancing to the > point that it just became completely ineffective for very common use > cases. > > We only didn't notice because another bug: > https://lore.kernel.org/all/20260629163337.1264881-1-hannes@cmpxchg.org/ > > hid almost all these issues from being apparent on large workloads > (which was differently horrendous - that bug made ~70-90% of some > database workload memory ineligible for tiering) > > Once the shmem fix went in, all the starvation and filtering issues > become more apparent when the bandwidth numbers stopped making sense > (see above). > > Basically numa balancing has been broken since 2022/2023, and anyone > who has done any testing with it since then has been working off of > bad data (incomplete VMA coverage). I'll go through the patches from a MM side. I am not a big expert on kernel/sched/fair.c side of things. Would be good if someone could have a look on that. -- Cheers, David
On Thu, Sep 10, 2026 at 08:18:22PM -0400, Gregory Price wrote:
> NUMA balancing uses hinting faults for both task placement and memory-tier
> promotion. Several filters designed to avoid unproductive socket-placement
> faults can also prevent promotion.
>
sashiko notes - remaining notes appear to be false positives.
patch 2: "... bounce horizontally between slow tiers" <- nonsense
patch 3 and 4: it's existing code extracted into a helper, using the
same checks we're already using. If we want it updated,
that can come in a separate patch.
note: this work has some backport conflicts, but tiered numa balancing
has effectively been broken multiple times since 2022 and 2023 due to
these filtering mechanisms.
~Gregory
On Thu Sep 10, 2026 at 8:18 PM EDT, Gregory Price wrote: > NUMA balancing uses hinting faults for both task placement and memory-tier > promotion. Several filters designed to avoid unproductive socket-placement > faults can also prevent promotion. > > Read-only file mappings and VMAs without recent PID activity may never be > scanned, while lower MM layers reject some shared folios. Hot memory in > these mappings can therefore remain on a slow tier indefinitely. > > Separate promotion-only scans from socket-placement scans. > > MM_CP_PROT_NUMA_PROMO_ONLY carries that choice for one protection walk, so > the PTE and PMD paths can restrict hinting faults to promotion candidates. > Keeping this transient state in the protection flags avoids per-mm state and > its associated lifetime, concurrency, and VMA identity problems. > > - Allow eligible shared folios to be promoted to a fast tier. > - Scan read-only file mappings and PID-inactive VMAs for promotion > without re-enabling placement sampling. > - Track the last placement scan separately so promotion-only scans > cannot postpone the existing placement-starvation fallback. > > The series is ordered as follows: > > 1. Add promotion-only NUMA protection walks without changing behavior. > 2. Permit eligible shared folios to be promoted to a fast tier. > 3. Scan read-only file mappings using promotion-only scans. > 4. Scan PID-inactive VMAs for promotion and account for placement scans > separately. > > Tested on a host with 768GB/256GB DRAM/CXL. > Ran 2 ~430GB database workloads with large (>300GB) shmem VMAs. Do you just launch them without any additional NUMA control or NUMA policy configuration, like using numactl? Just want to understand the scope of the issue and how well the fixes are tested. It might be good to have a list of expected behaviors for people/AI to check against. > > Before change: > - 150-200GB/s DRAM bandwidth usage > - 40-45GB/s CXL bandwidth usage (maxed out) > - request latencies over 5ms (longer tails) > > After chage: > - 250GB/s+ sustained DRAM bandwidth usage > - ~10GB/s sustained CXL bandwidth usage > - request latencies 800us-2ms. > > Functional observation: > A 20 GB hash table VMA that previously remained entirely on CXL was > split evenly between DRAM and CXL after the changes - and tier > residency tracked hotness. This was previously affected by the > stavation issue caused by the "unaccessed VMA" filter. > > Gregory Price (Meta) (4): > mm: support promotion-only NUMA hinting scans > mm: allow shared folios to be promoted to a fast tier > sched/numa: scan read-only file mappings in tiering mode > sched/numa: do not let VMA PID activity gate promotion > > include/linux/mm.h | 4 ++- > include/linux/mm_types.h | 7 ++++ > kernel/sched/fair.c | 72 +++++++++++++++++++++++++++++----------- > mm/huge_memory.c | 3 +- > mm/internal.h | 5 +-- > mm/mempolicy.c | 30 +++++++++++------ > mm/migrate.c | 6 ++-- > mm/mprotect.c | 4 ++- > 8 files changed, 93 insertions(+), 38 deletions(-) -- Best Regards, Yan, Zi
On Fri, Sep 18, 2026 at 04:56:34PM -0400, Zi Yan wrote: > > > > Tested on a host with 768GB/256GB DRAM/CXL. > > Ran 2 ~430GB database workloads with large (>300GB) shmem VMAs. > > Do you just launch them without any additional NUMA control or NUMA > policy configuration, like using numactl? Just want to understand the > scope of the issue and how well the fixes are tested. It might be good > to have a list of expected behaviors for people/AI to check against. > We found these issues while testing fairness controls with Joshua's tiered memcg limits patches [1]. We were seeing very lopsided bandwidth utilization between tiers. When dropped all the memcg extensions and tested these patches by themselves, we found that numa balancing was completely broken - starting with Johannes' shmem fix [2]. Otherwise - No other numa controls are used on these workloads. - no numactl - no mbind()/set_mempolicy() - no cpuset.mems - etc These fixes impact the fault-injecting side (spraying PROT_NONE), not the fault-handling side. So mempolicy and cpuset doesn't really come into the picture. It's more of an internal-consistency issue for numab itself. ~Gregory [1] https://lore.kernel.org/all/20260807202059.2620949-1-joshua.hahnjy@gmail.com/ [2] https://lore.kernel.org/all/20260629163337.1264881-1-hannes@cmpxchg.org/
© 2016 - 2026 Red Hat, Inc.