AMD Zen6 includes IBS Memory Profiler that is independent of the
primary IBS instance. This second, light-weight IBS instance can
be used exclusively for memory access profiling.

This patchset provides a driver for the same. Till now this driver
was being posted as part of pghot patchset [1] where IBS Memory
Profiler was used as an additional source for providing page hotness
information (hardware provided hints source - pghot-hwhints).

Since this driver is x86 specific and also to make review easier,
I am separating out the hardware-hints source into a patchset of
its own. This patchset depends on pghot infrastructure and hence
applies on top of the base pghot patchset. The small bits in the
patchset that modify the pghot documentation and pghot API itself
can be ignored here for the review purposes.

pghot in brief
==============
pghot is a proposed subsystem for hot page tracking and promotion. It
collects memory access information from multiple sources, classifies
hot pages resident in lower-tier memory, and promotes them to faster
tiers via a per-lower-tier-node kernel thread (kmigrated).

Goals of pghot:

- Unify hot page detection from multiple sources like hint faults,
  page table scans, hardware hints (AMD IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lower-tier-node kmigrated kernel
  thread.
- Move promotion rate-limiting and related logic used by numa_balancing=2
  (NUMAB2, the current NUMA balancing-based promotion) from the scheduler
  to pghot for broader reuse.

pghot provides an API (pghot_record_access()) that other
sub-systems/drivers can use to report memory access information
to pghot. IBS Memory Profiler is one such source or driver.

IBS Memory Profiler
===================
- Hardware samples an op every N core cycles or N dispatched ops,
  with a software-programmable period, so sampling overhead and
  coverage can be traded off at runtime.

- Each sample reports both the linear and the physical address of
  the accessed data, so the kernel gets the accessed PFN directly,
  without page-table scanning, hint faults or any address translation
  of its own.

- Samples include a data-source indication that identifies where the
  access was satisfied from (cache, DRAM, external memory etc). This
  lets software distinguish local DRAM traffic from CXL or other
  lower-tier accesses.

- Hardware filtering lets the profiler report only the accesses that
  matter for memory placement: L3-miss-only, a load-latency threshold,
  and exclusion by instruction-address bit 63 (i.e. user vs kernel).
  Filtered-out samples are discarded by hardware and never interrupt
  the kernel.

- The profiler raises its own interrupt, routed through a dedicated
  extended LVT offset that is distinct from the one used by the
  primary IBS instance. It can therefore be used by the kernel without
  interfering with existing IBS users such as perf.

Spec: "AMD64 Zen6 Instruction Based Sampling (IBS) Extensions and
Features", publication 69205 rev 1.00, section "IBS Memory Profiler".

Benchmarks summary
==================
All results are on 3-node tiered systems (DRAM top tier + CPU-less CXL
lower tier). Speedups below are normalized to the base kernel with no
tiering (NUMAB=0). Columns compare mainline hint-fault tiering
(base NUMAB=2) against HW-hint driven promotion via the AMD IBS
memory profiler (pghot-hwhints, NUMAB=0, no NUMA scanning). IBS sample
period is 10000 unless noted. Single run per config except llama.cpp
and microbench (avg of 3).

Benchmark      Metric (higher=better)   base-NUMAB2  pghot-hwhints
---------------------------------------------------------------------
NAS BT (MPI)   Mop/s total                  2.34x     2.15x
Graph500 BFS   harmonic-mean TEPS           2.34x     3.17x
llama.cpp      decode tok/s (tg128)         1.19x     1.22x
Redis+memtier  ops/sec                      1.04x     1.00x
ptr-chase      1/latency (p10k / p5008)     2.24x     1.49x / 3.48x
Microbench     completion time (1/t)        2.55x     2.22x
---------------------------------------------------------------------
(baseline = base kernel, no tiering = 1.00x; pghot-hwhints uses IBS, NUMAB=0)

Headline: IBS/HW-hints recovers most of mainline NUMAB=2 on
bandwidth-heavy promotion (NAS BT 2.15x vs 2.34x, ~92% of NUMAB2
throughput) with zero hint faults or PTE scans; it beats NUMAB2 on
Graph500 (3.17x vs 2.34x, ~1/4 the promotions) and matches it on
llama decode. Coverage is the limiter on very large, uniformly hot
sets: Redis stays flat until sampling captures more of the hot set,
while pointer-chase at period=5008 reaches the lowest latency of the
set (3.48x vs NUMAB2 2.24x; DAMON 2.69x).

- NAS BT (MPI, Class D, 16 ranks): full footprint pre-staged on the
  CXL lower tier, then measured cold; isolates hot-page promotion.
- Graph500 (reference BFS, SCALE=28, edgefactor=16, 128 ranks):
  working set on the CXL node; figure of merit is harmonic-mean TEPS.
- llama.cpp (llama-bench, Mixtral-8x22B Q4_K_M): ~1/3 of the model
  demoted to CXL under memory pressure; decode (tg128) is tier-sensitive.
- Redis + memtier (~64 GB dataset, 62.2M x 1 KB keys): dataset
  migrated to CXL, GET-heavy traffic over 50% of the keyspace.
  IBS at period 10000 and 5008 both ~1.00x vs baseline (hot-set
  coverage ~10% of NUMAB2's ~40 GiB promotion).
- Microbenchmark: 64 threads, random 4K access over 8G on
  the CXL node; latency-bound.
- Pointer-chase (64 threads, 64 GiB buffer, 16 GiB hot, 240 s):
  latency-bound; NUMAB2 places the exact hot set. IBS period=10000
  under-promotes in 240 s; period=5008 promotes 12.0 GiB of the 16 GiB
  hot set and records the best latency (92 ns vs NUMAB2 143 ns,
  DAMON 119 ns).

Detailed per-benchmark tables (throughput/latency + vmstat and pghot
promotion counters) are posted as replies to this thread.

Test reports
============
Compile test matrix
-------------------
The series was compile-tested across the full range of relevant Kconfig
combinations on x86_64 (defconfig-based) and cross-compiled on arm64.
Combinations of PGHOT, PGHOT_PRECISE, NUMA_BALANCING,
NUMA_BALANCING_TIERING, NUMA_MIGRATION, HWMEM_PROFILER/AMD_IBS_MEMPROF,
DEBUG_FS, SYSCTL and PM/PM_SLEEP were verified, including pghot disabled,
pghot standalone (no NUMA balancing), NUMA balancing mode 1 and mode 2
(tiering), precise mode, the AMD IBS memory profiler (including with power
management and suspend/hibernation disabled), NUMA_MIGRATION=n, and pghot
with DEBUG_FS and/or SYSCTL disabled. Kconfig guards were confirmed
(NUMA_BALANCING_TIERING without PGHOT, and HWMEM_PROFILER without a
selecting driver, are dropped). Since PGHOT defaults to y when
NUMA_BALANCING is set and NUMA_BALANCING_TIERING defaults to y, an existing
config with NUMA_BALANCING=y but no pghot symbols resolves, via
olddefconfig, to PGHOT=y and NUMA_BALANCING_TIERING=y and builds cleanly,
while NUMA_BALANCING=n leaves PGHOT off. allnoconfig, tinyconfig,
allyesconfig and allmodconfig were also built. All configurations build with
no new warnings.

CPU hotplug stress testing
--------------------------
The IBS memory profiler's STARTING-phase cpuhp callbacks (arm IBS and
set up the APIC EILVT on startup; mask, disable and drain the per-CPU
sample ring on teardown) were stress tested against concurrent hotplug.

On an AMD Zen6 (256 CPUs, CXL lower tier) with PROVE_LOCKING, PROVE_RCU,
DEBUG_ATOMIC_SLEEP and SLUB_DEBUG_ON, the following ran concurrently for
20 minutes: IBS armed at the minimum sample period (5008) feeding a
node-bound memory workload; 8 parallel CPU offline/online loops plus
periodic mass offline of 96 CPUs at once; and continuous profiler
reconfiguration driving on_each_cpu() re-arm against the changing online
mask.

No oops, warning, lockdep/RCU splat or hung-task was seen. The profiler
kept sampling and pghot kept recording throughout, the per-CPU ring
dropped no samples even at ~2M samples/interval, and every CPU reliably
came back online with per-CPU state correctly reset.

Changes to the driver since it was posted last time with pghot
==============================================================
- Register IBS Memory Profiler cpuhp callbacks on CPUHP_AP_ONLINE_DYN
  instead of a dedicated CPUHP_AP_MM_AMD_IBS_MEMPROF_STARTING state.
  This elimiates the subtle races due to worker getting migrated
  to other CPU before teardown.
- Make SPSC tail handling symmetric by using smp_store_release().
- Mark the clearing of mprof_work_pending with WRITE_ONCE() in
  mprof_work_handler() and x86_amd_ibs_mprof_offline(), so every
  access to the flag is uniformly annotated.
- Take cpus_read_lock() across mprof_publish(). This keeps the online
  set stable so on_each_cpu() cannot race a CPU's ONLINE arm path.
- Drain the per-CPU sample workers before free_percpu(mprof_s) on the init
  error path.
- Add dram accesses stats for far memory accesses too.
- Made reading of data2 optional to L1 and L2 hit.
- Protect debugfs and sysfs readers in ibs-mprof.c with mutex.
- Validate memprofiler EILVT offset at once during init time.
- Ensure the period reads back what is written into the HW IbsMemOpCnt
  - Minimum value changed from 5000 to 5008 to make it 16-bytes aligned
- Reject over-sized writes to IBS Memory Profiler's debugfs tunables.
- Change ibs_mprof_enabled to static definition.
- Include linux/string_choices.h explicitly in mprof for str_enabled_disabled().
- Handle failure path in mprof driver init proper rollback.
- Added suspend/resume support so that EILVT/IBS is properly configured
  for CPU0 post resume.
- Gated mprof init with X86_FEATURE_IBS check so that the driver
  refuses to move further on non-AMD platforms.

Dependencies
============
- [1] pghot patchset:
  https://lore.kernel.org/linux-mm/20260728054356.291998-1-bharata@amd.com/

- IBS Memory Profiler depends on the patchset that increases the
  number of APIC EILVT registers:
  https://lore.kernel.org/lkml/cover.1788425679.git.naveen@kernel.org/

- Full tree with all dependencies included:
  https://github.com/AMDESE/linux-mm/tree/bharata/pghot-mprof-rfcv0

AI assistance was taken to run and summarize the benchmark results,
to run compile test matrix and CPU hotplug stress tests.

Bharata B Rao (3):
  x86/ibs: Move IBS caps definitions into its own header
  x86/mm/ibs: In-kernel driver for AMD IBS Memory Profiler
  x86/mm/ibs: Add runtime controls for IBS memprofiler

 .../admin-guide/kernel-parameters.txt         |   5 +
 Documentation/admin-guide/mm/pghot.rst        |  89 ++
 arch/x86/Kconfig                              |  16 +
 arch/x86/entry/entry_fred.c                   |   1 +
 arch/x86/include/asm/hardirq.h                |   3 +
 arch/x86/include/asm/ibs-caps.h               |  93 ++
 arch/x86/include/asm/ibs-mprof.h              |  61 ++
 arch/x86/include/asm/idtentry.h               |   6 +
 arch/x86/include/asm/irq_vectors.h            |   4 +-
 arch/x86/include/asm/msr-index.h              |   8 +
 arch/x86/include/asm/perf_event.h             |  81 +-
 arch/x86/kernel/idt.c                         |   3 +
 arch/x86/kernel/irq.c                         |   3 +
 arch/x86/mm/Makefile                          |   1 +
 arch/x86/mm/ibs-mprof.c                       | 867 ++++++++++++++++++
 include/linux/vm_event_item.h                 |   8 +
 mm/Kconfig                                    |   9 +
 mm/vmstat.c                                   |   8 +
 18 files changed, 1185 insertions(+), 81 deletions(-)
 create mode 100644 arch/x86/include/asm/ibs-caps.h
 create mode 100644 arch/x86/include/asm/ibs-mprof.h
 create mode 100644 arch/x86/mm/ibs-mprof.c

-- 
2.34.1

On 24-Sep-26 11:52 AM, Bharata B Rao wrote:
> 
> Detailed per-benchmark tables (throughput/latency + vmstat and pghot
> promotion counters) are posted as replies to this thread.
==========================================================================
Redis + memtier: hot-page promotion on a CXL-tiered system
==========================================================================

Benchmark: an in-memory Redis server is loaded with a ~64 GB dataset
(62.2M keys x 1 KB) whose pages are then explicitly migrated to the CXL
node (lower tier); the Redis server and the memtier client run on a top-
tier DRAM node (node 1).  The measurement phase drives GET traffic with
memtier (16 threads x 100 conns, PASSES=12) over 50% of the keyspace so
that repeatedly-accessed lower-tier pages become promotion candidates.
Promotions target node 1 (local to the accessing threads): NUMAB uses
the accessing node, pghot uses pghot_target_nid=1.

System : AMD Zen6. Nodes 0,1 = DRAM (top tier);
         node 2 = CXL (lower tier).  Redis/client pinned to node 1.
Kernels: base=7.3.0-rc2-base+  pghot=7.3.0-rc2-pghot+

Cases:
  C1 base/NUMAB0      base kernel, promotion OFF (baseline)
  C2 base/NUMAB2      base kernel, NUMAB tiering promotion (hint
                      faults); promotes to local accessing node
  C3 pghot/hwhints-10k  pghot, NUMAB=0, source=IBS mprof,
                      period=10000, freq_thr=1, target_nid=1
  C4 pghot/hwhints-5k   as C3 but IBS period=5008 (kernel min,
                      ~2x sampling; 5008)

==========================================================================
Table 1: Benchmark metrics (memtier)
==========================================================================
Case                   Ops/sec   vs C1     Avg     p50     p99   p99.9
                                                           latency (ms) ->
----------------------------------------------------------------------
C1 base/NUMAB0         285,288     ref  179.46  177.15  344.06  358.40
C2 base/NUMAB2         295,475  +3.57%  173.18  168.96  325.63  364.54
C3 pghot/hwhints-10k   285,923  +0.22%  178.91  177.15  344.06  358.40
C4 pghot/hwhints-5k    285,290  +0.00%  179.24  177.15  346.11  358.40
----------------------------------------------------------------------

==========================================================================
Table 2: Page-migration / hotness metrics (vmstat delta)
==========================================================================
Legend: C1=base/NUMAB0  C2=base/NUMAB2  C3=hwhints p=10000
        C4=hwhints p=5008  (both hwhints: target_nid=1)
        ('-' = counter not present on base kernel)

metric                       C1          C2         C3         C4
-----------------------------------------------------------------
pgpromote_success             0  10,433,506    575,010  1,107,371
numa_pte_updates              0  20,333,730          0          0
numa_hint_faults              0  10,433,506          0          0
numa_pages_migrated           0  10,433,506    575,006  1,107,369
pgmigrate_success             0  10,433,506    575,006  1,107,369
pghot_recorded_accesses       -           -    575,677  1,110,760
pghot_reported_hwhints        -           -    964,304  1,967,044
hwhint_total_events           -           -    964,324  1,967,078
hwhint_dram_accesses          -           -    388,027    854,679
hwhint_extmem_accesses        -           -    575,674  1,110,758
hwhint_useful_events          -           -    964,304  1,967,044
pgdemote_kswapd               0           0          0          0
-----------------------------------------------------------------

==========================================================================
Key observations
==========================================================================
1. NUMAB tiering promotion (C2) helps only marginally: 295,475 vs 285,288
ops/sec (+3.57%), avg latency 179.5 -> 173.2 ms, promoting the full hot set
(10.43M pages / ~39.8 GiB) to the local node.

2. pghot with the IBS hwhints source is flat vs baseline at both periods
(C3 +0.22%, C4 +0.00%). numa_pte_updates / numa_hint_faults are
0 (no NUMA balancing); promotion is purely hardware-sample driven.

3. Sampling density scales promotion linearly (period 10000 -> 5008
doubles reported hwhints 964,304 -> 1,967,044 and promotions 575,010 ->
1,107,371 pages, ~2.19 -> ~4.22 GiB); ext-mem/CXL samples map ~1:1 to
promotions (freq_threshold=1), dram/already-toptier samples are not
promotable.

4. But even ~4.22 GiB is only ~10% of the ~40 GiB hot set, so throughput
does not move.  Unlike C2 (which promotes the whole hot set), sampling-
based hwhints at these periods covers too little of the working set within
the run.  Reaching C2's gain needs far denser sampling and/or a longer run
so hwhints promotes a large fraction of the hot set.

On 24-Sep-26 11:52 AM, Bharata B Rao wrote:
> 
> Detailed per-benchmark tables (throughput/latency + vmstat and pghot
> promotion counters) are posted as replies to this thread.
llama.cpp tiering benchmark: base vs pghot-hwhints (kernel 7.3.0-rc2)
====================================================================
3-run average per configuration.

Workload: llama-bench (llama.cpp), Mixtral-8x22B-Instruct Q4_K_M (140.6B
params, 79.7 GB), -t 64 -p 512 -n 128 -r 5 --mmap 0.  pp512 = prefill
(compute-bound); tg128 = decode (memory-bandwidth-bound).

Setup: 3-node box, N0/N1 DRAM (256 GB each), N2 CXL (distances 10/12/50).
Bench pinned to N1 CPUs with MPOL_PREFERRED_MANY({1}); a 200 GB hot hog on
N1 forces kswapd to naturally demote ~1/3 of the model to CXL; each mode is
then measured while the hog keeps N1 under pressure.  pghot target_nid=1,
freq_threshold=1, promote_window=3000ms, rate_limit=65536 MBps, kmigrated
100ms/512.  hwhints arms AMD IBS (l3miss-only=1, period=10000).

Legend (columns)
----------------
  r1 = base / notier        7.3.0-rc2-base   nb=0 src=-
  r2 = base / tier (NUMAB2) 7.3.0-rc2-base   nb=2 src=-
  r3 = pghot / hwhints(IBS) 7.3.0-rc2-pghot  nb=0 src=0x2
  (nb = kernel.numa_balancing; src = pghot_enabled_sources; 3 runs each)

Table 1 - Throughput, 3-run mean +/- stdev (llama-bench tokens/s)
----------------------------------------------------------------
metric                       r1       r2       r3
-------------------------------------------------
pp512 mean                69.43    61.34    57.90
  stdev                    0.85     1.78     2.21
tg128 mean                4.094    4.883    4.975
  stdev                   0.045    0.053    0.023
tg128 vs r1               1.00x    1.19x    1.22x
pp512 vs r1               1.00x    0.88x    0.83x

Table 1b - per-run values (3 runs), shows consistency
-----------------------------------------------------
                             r1       r2       r3
-------------------------------------------------
tg128 run1                4.099    4.856    4.949
tg128 run2                4.037    4.836    5.005
tg128 run3                4.146    4.957    4.970
pp512 run1                70.37    63.37    61.01
pp512 run2                68.31    59.03    56.61
pp512 run3                69.60    61.64    56.07

Table 2 - Key vmstat counters, 3-run mean of run-phase deltas (millions)
-----------------------------------------------------------------------
metric                            r1       r2       r3
------------------------------------------------------
pgpromote_success               0.00     3.07     1.74
pgpromote_candidate             0.00    12.80    12.86
pgdemote_kswapd                 2.58     4.78     4.40
pgmigrate_success               2.58     7.85     6.13
numa_pte_updates                0.00    26.06     0.00
numa_hint_faults                0.00    25.12     0.00
pghot_recorded_accesses         0.00     0.00    12.86
pghot_reported_hwhints          0.00     0.00    30.79
hwhint_total_events             0.00     0.00    30.79
hwhint_dram_accesses            0.00     0.00    17.74
hwhint_extmem_accesses          0.00     0.00    12.86
start N2 % (at SIGCONT)         28.3     28.9     27.6

Key observations (3-run averages)
---------------------------------
1. base/tier (r2) tg128 4.883 is +19.3% over base/notier (r1)
   4.094; pghot/hwhints (r3) 4.975 is +21.5% over r1 and
   +1.9% over r2.
2. Consistency is tight: r3 tg128 4.949-5.005 (stdev 0.023);
   r2 4.836-4.957 (stdev 0.053).
3. hwhints armed IBS: ~35M events, ~13M on CXL; pgpromote_candidate
   12.9M matches pghot_recorded_accesses 12.9M (1.00x).
4. Prefill (pp512) pays a tiering tax: r2 0.88x, r3 0.83x of r1.

Caveats: base and pghot are different kernels (7.3.0-rc2-base vs -pghot);
the natural-overflow setup settled at ~30% CXL at SIGCONT in all runs;
3 runs per configuration (per-run values in Table 1b).

On 24-Sep-26 11:52 AM, Bharata B Rao wrote:
> 
> Detailed per-benchmark tables (throughput/latency + vmstat and pghot
> promotion counters) are posted as replies to this thread.
======================================================================
Tiered-memory promotion: baseline vs NUMA-balancing vs DAMON vs pghot
hwhints (IBS) - multithreaded runs
======================================================================

Runs taken on one machine, same workload and sizes.  This is a self-contained
comparison of four ways to promote a hot working set from a slow (CXL) tier
to a fast (DRAM) tier:

  - baseline        : no promotion.
  - NUMAB2          : the kernel's NUMA-balancing tier promotion
                      (numa_balancing=2), driven by NUMA hint faults.
  - DAMON fine+quota: in-kernel access monitor with a migrate_hot
                      scheme (paddr), driven by the kdamond thread.
  - pghot hwhints   : per-PFN hotness from the IBS Memory Profiler
                      (hardware sampling), promoted by the kmigrated
                      thread; shown at two IBS sample periods.

Machine / topology
------------------
- 256 CPUs, 792 GB RAM.
- NUMA: node 0, node 1 = DRAM (fast tier); node 2 = CXL (slow tier,
  256 GB).  Promotion target = node 0; cold data starts on node 2.
- Kernels: base = 7.3.0-rc2-base+ (baseline, NUMAB2, DAMON);
           pghot = 7.3.0-rc2-pghot+ (pghot hwhints).

Workload
--------
- 64-thread pointer chase, 64 GB total buffer, 16 GB hot set, 240 s
  timed phase.  Buffer relocated to node 2 before the timed phase;
  worker threads bound to node 0.  "hot set" = 16 GB;
  over = (node0 - 16 GB) / 16 GB.

Configurations
--------------
- baseline : numa_balancing=0, no promotion.
- NUMAB2   : numa_balancing=2 (tier promotion via hint faults).
- DAMON    : sysfs defaults (sample 5ms, aggr 100ms, update 60s) +
             migrate_hot (paddr, nr_accesses>=1) + max_nr_regions=
             100000 (min default 10) + quota 128 MiB/1000 ms with
             nr_accesses weighting (hottest first).
- hwhints  : IBS Memory Profiler as the sole hotness source
             (numa_balancing=0), pghot_freq_threshold=1, IBS
             l3miss-only=1; IBS sample period 10000 and 5008.

Metric notes
------------
- lat  : steady access latency, ns/access, mean of last 25% of samples
         (lower is better).
- node0: end-of-run placement on node 0 (numastat -p).
- promoted: bytes migrated to the fast tier (DAMON scheme sz_applied;
         NUMAB2/pghot pgpromote_success).
- CPU  : cumulative cputime over the 240 s run of the single promoter/
         monitor thread - kd = kdamond (DAMON), km = kmigrated (pghot).
         NUMAB2 has no single promoter thread; its cost is distributed
         across the application threads (scanning + hint faults).
- over:  The "over" column is measured against the 16 GB HOT SET, not
         against B0: over = (node0 - 16 GB) / 16 GB.  B0 promotes
         nothing, so its node0 is 0.0 and its over is n/a.


Table.  Multithreaded results (64 GB total / 16 GB hot)
------------------------------------------------------
id   config              kern   lat  node0  over   promoted  promoter
                                (ns) (GB)             (GB)   CPU
---  ------------------  -----  ---  -----  -----  --------  ----------
B0   baseline            base   320   0.0     -        -     -
N2   NUMAB2              base   143  16.0    0%     ~14.6*   distributed
D3   DAMON fine+quota    base   119  23.9  +49%      24.7    kd 13s(5%)
H10  hwhints IBS p10000  pghot  215   4.9  -69%       4.9    km 20s(8%)
H5   hwhints IBS p5008   pghot   92  11.6  -27%      12.0    km 38s(16%)

* N2 pgpromote_success over the timed window is ~14.6 GB; some
  promotion happens during the load phase before the vmstat baseline,
  so the reliable figure is the placement: node 0 = 16.0 GB (exact).

Supporting counters
-------------------
- N2 : numa_hint_faults 3.73M, numa_pte_updates 12.9M.
- D3 : sz_applied 24.7 GB, qt_exceeds 229, regions settled ~21-71.
- H10: hwhint_total_events 1.40M, pgpromote_success 1.29M.
- H5 : hwhint_total_events 5.91M, pgpromote_success 3.15M.

Findings
--------
1. Accuracy (placement):
   - NUMAB2 places exactly the 16 GB hot set (0% over).
   - DAMON over-promotes (+49%, 23.9 GB): a coarse-then-quota scheme
     bounds volume but still moves ~1.5x the hot set.
   - hwhints under-promotes within 240 s (-69% at p10000, -27% at
     p5008) because hardware sampling observes only a fraction of
     accesses; it had not fully converged at 240 s.

2. IBS sample period is the coverage knob.  10000 -> 5008 roughly
   tripled events (1.40M -> 5.91M), nearly tripled placement (4.9 ->
   11.6 GB) and cut latency 215 -> 92 ns.  period=10000 under-samples
   this workload.

3. Latency does not track placement linearly:
   - hwhints p5008 reaches the lowest latency (92 ns) with only 11.6 GB
     on node 0 - IBS targets the hottest pages, so its smaller
     placement is well-chosen.
   - DAMON needs 23.9 GB to reach 119 ns.
   - NUMAB2 places the exact 16 GB but its steady latency (143 ns) is
     higher than its own first interval (119 ns): numa_balancing keeps
     scanning and faulting the workload's pages after promotion, an
     ongoing per-access tax (12.9M pte updates, 3.73M hint faults).

4. Promoter CPU:
   - DAMON kdamond and pghot kmigrated are both out-of-band and cheap
     here (kd 13 s; km 20-38 s over 240 s, i.e. ~8-16% of one core).
   - NUMAB2's cost is not a single thread; it is paid by the workload
     threads via scanning and hint-fault handling.

Caveats
-------
- Two kernels: base (B0/N2/D3) and pghot (H10/H5).  No separate
  no-promotion baseline was taken on the pghot kernel in this set, so
  cross-kernel absolute latency should be read with that in mind;
  placement / over-promotion is the primary accuracy metric.
- The CPU column is the concentrated promoter/monitor thread only.  It
  excludes distributed costs: NUMAB2's scanning + hint-fault handling,
  and IBS interrupt/sample handling for hwhints.
- Latency is not isolated to a single cause (bandwidth, working set,
  selection quality all plausible); placement is the primary accuracy
  metric and latency is supporting.
- hwhints coverage is still ~27% under the 16 GB hot set at 240 s
  (p5008); it was still climbing at the end of the run.

On 24-Sep-26 11:52 AM, Bharata B Rao wrote:
> 
> Detailed per-benchmark tables (throughput/latency + vmstat and pghot
> promotion counters) are posted as replies to this thread.
Benchmark
----------------------------------------------------------------------
NAS Parallel Benchmark BT (Block Tri-diagonal solver), MPI version,
Class D, 16 ranks. Compute threads are bound to node 1 via:

  /usr/bin/mpirun -np 16 /usr/bin/numactl --cpunodebind=1 \
      .../NPB3.4.4/NPB3.4-MPI/bin/bt.F.x

Test system : 3 NUMA nodes. Nodes 0 and 1 are top-tier DRAM
with CPUs (~252 GiB each); node 2 is a CPU-less CXL lower tier
(~252 GiB). Node distances: 0<->1 = 12, 1<->2 = 50. Before the
measurement phase all provisioned memory (~161.4 GiB across ranks)
is migrated to the CXL node 2, then ranks resume and measurement
begins, so the run starts fully cold on slow memory. This isolates
the effect of hot-page promotion (node 2 -> node 1) on a
bandwidth/latency-sensitive workload.

Kernels
----------------------------------------------------------------------
  base  = 7.3.0-rc2-base+   (mainline base)
  pghot = 7.3.0-rc2-pghot+  (pghot + AMD IBS Memory Profiler)

Legend
----------------------------------------------------------------------
  A = base-numab0    Base kernel, NO tiering (numa_balancing=0);
                     reference case
  B = base-numab2    Base kernel, kernel NUMA-balancing tiering
                     (numa_balancing=2, hint-fault driven promotion,
                     demotion enabled)
  C = pghot-hwhints  pghot kernel, promotion driven by the AMD IBS
                     Memory Profiler (hardware access sampling);
                     numa_balancing=0 (no NUMA hint faults / scanning)

IBS Memory Profiler configuration (run C)
----------------------------------------------------------------------
  vm.pghot_enabled_sources     = 2   (HWHINTS source)
  vm.pghot_target_nid          = 1   (promote to compute node 1,
                                       same target NUMAB2 uses)
  vm.pghot_freq_threshold      = 1   (promote on 1st access, like
                                       NUMAB2)
  vm.pghot_promote_rate_limit_MBps = 65536
  vm.pghot_promote_window_ms       = 3000
  /sys/devices/system/cpu/ibs-mprof/enabled = enabled
  ibs-mprof: period=10000, l3miss-only=1, lat-thresh=0, lat-filter=0

Table 1a - Runtime (lower is better)
----------------------------------------------------------------------
Metric                           A           B           C
----------------------------------------------------------------------
NPB Time (s)               7564.04     3232.04     3515.97
Speedup vs A (time)           1.00        2.34        2.15

Table 1b - Throughput (higher is better)
----------------------------------------------------------------------
Metric                           A           B           C
----------------------------------------------------------------------
NPB Mop/s total           51739.90   121088.51   111310.01
Mop/s ratio vs A              1.00        2.34        2.15
Mop/s ratio vs B              0.43        1.00        0.92
Peak prov. mem (GiB)        161.38      161.38      161.38

Table 2 - vmstat counter deltas (after - before measurement window)
----------------------------------------------------------------------
Counter                               A           B           C
----------------------------------------------------------------------
pgpromote_success                     0    42181004    39701482
pgpromote_candidate                   0           0           0
pgpromote_candidate_nrl               0    42181008    39702291
pgdemote_kswapd                       0           0           0
pgdemote_direct                       0           0           0
numa_pte_updates                      0    42252137           0
numa_hint_faults                      0    42182619           0
numa_pages_migrated                   0    42182610    39701482
pgmigrate_success                     0    42182610    39701482
pghot_recorded_accesses               0           0    39793839
pghot_reported_hintfaults             0           0           0
pghot_reported_hwhints                0           0   553943841
hwhint_total_events                   0           0   553950006
hwhint_dram_accesses                  0           0   514144414
hwhint_extmem_accesses                0           0    39794778
hwhint_cache_accesses                 0           0           0
hwhint_useful_events                  0           0   553943842
hwhint_dropped_events                 0           0        6147

Key observations
----------------------------------------------------------------------
1. Tiering is essential for BT. With the whole footprint pinned to
   the CXL lower tier and no promotion (A), BT is 2.34x slower than
   kernel NUMA-balancing tiering (7564 vs 3232 s) and delivers less
   than half the throughput (51.7k vs 121.1k Mop/s). BT is
   bandwidth/latency-bound and much of its footprint is genuinely
   hot, so promotion back to DRAM helps a lot.

2. IBS Memory Profiler drives effective promotion WITHOUT NUMA
   balancing. Run C reaches 3516 s / 111.3k Mop/s = 2.15x over the
   no-tiering baseline, i.e. ~92% of NUMAB2's throughput, while
   numa_pte_updates and numa_hint_faults are both 0 -- no PTE
   scanning and no hint faults are involved. Promotion is instead
   fed entirely by hardware access samples.

3. Comparable migration volume, different mechanism. C promotes
   ~39.70M pages (pgpromote_success == pgmigrate_success ==
   numa_pages_migrated) vs ~42.18M for NUMAB2 (B) -- about 6% fewer,
   consistent with the ~9% runtime gap. Both promote to node 1.

4. IBS sampling is high-volume but precise. With period=10000 and
   l3miss-only=1, the profiler recorded ~554M useful events
   (hwhint_useful_events / pghot_reported_hwhints = 553,943,84x).
   Of ~554M total events, ~514M were DRAM accesses and only
   ~39.79M were external/CXL accesses (hwhint_extmem_accesses).
   That extmem count matches pages promoted (39.70M) and
   pghot_recorded_accesses (39.79M) very closely: nearly every
   distinct external-memory access observed led to a promotion,
   i.e. precise hot-page targeting.

5. Negligible event loss. hwhint_dropped_events = 6,147 out of
   ~554M total (~0.001%), so the SPSC delivery path kept up with
   the hardware sample rate.

6. Migration is promotion-only. pgdemote_* and pgpromote_candidate
   are 0 across all runs; memory is pre-staged on node 2 and DRAM
   has ample headroom (~252 GiB/node vs ~161 GiB), so pages flow
   node 2 -> node 1 with no demotion pressure.

7. Caveats: Verification reported UNSUCCESSFUL in every run
   (consistent across all three, so a benchmark class/build
   artifact rather than a tiering effect). These are single runs
   per config; the ~9% (C vs B) gap should be confirmed with
   repeats to establish variance.

Bottom line: the AMD IBS Memory Profiler source drives pghot
promotion purely from hardware access samples -- no NUMA hint
faults, no PTE scanning -- and recovers ~92% of kernel
NUMA-balancing tiering performance on NAS BT, at 2.15x over the
untiered baseline.

On 24-Sep-26 11:52 AM, Bharata B Rao wrote:
> 
> Detailed per-benchmark tables (throughput/latency + vmstat and pghot
> promotion counters) are posted as replies to this thread.

Graph500 tiering comparison: base vs pghot (hwhints)
====================================================
Kernel(s) : base  = 7.3.0-rc2-base+
            pghot = 7.3.0-rc2-pghot+
Benchmark : Graph500 reference BFS, SCALE=28, edgefactor=16, 128 ranks
Topology  : top-tier NUMA node=1 (CPUs+DRAM), lower-tier node=2 (mem-only/CXL)
Note      : SKIP_VALIDATION=1 (timing-only); all TEPS carry Graph500 (!) flag.
            Figure of merit is harmonic_mean_TEPS (hmean).

Config legend
-------------
C1 : base  kernel, no tiering       (numa_balancing=0)
C2 : base  kernel, NUMAB2 tiering   (numa_balancing=2)
C3 : pghot kernel, hwhints source   (pghot_enabled_sources=2,
     pghot_freq_threshold=1, pghot_target_nid=0 [default],
     IBS mem-profiler on, numa_balancing=0, promotion=off)

Column legend
-------------
hmean   : harmonic_mean_TEPS   (primary Graph500 metric)
hstddev : harmonic_stddev_TEPS
median  : median_TEPS
bfs_t   : mean BFS time (seconds)
spdup   : speedup of hmean vs C1 baseline

Table 1: Performance
--------------------
+--------+------------+----------+------------+---------+--------+
| Config |   hmean    | hstddev  |   median   |  bfs_t  | spdup  |
|        |   (TEPS)   |  (TEPS)  |   (TEPS)   |  (sec)  |        |
+--------+------------+----------+------------+---------+--------+
| C1     | 5.543e+08  | 5.51e+05 | 5.548e+08  |  7.748  | 1.00x  |
| C2     | 1.298e+09  | 6.80e+07 | 1.394e+09  |  3.308  | 2.34x  |
| C3     | 1.755e+09  | 2.32e+07 | 1.805e+09  |  2.447  | 3.17x  |
+--------+------------+----------+------------+---------+--------+

Table 2: Relevant kernel counters (/proc/vmstat deltas over the run)
--------------------------------------------------------------------
Values are accumulated deltas (before -> after) for the whole run.

+----------------------------+-----------+-----------+-----------+
| Counter                    |    C1     |    C2     |    C3     |
+----------------------------+-----------+-----------+-----------+
| numa_pte_updates           |         0 |  25867526 |         0 |
| numa_hint_faults           |         0 |  13318442 |         0 |
| numa_pages_migrated        |         0 |  13318248 |   3702779 |
| pgpromote_success          |         0 |  13317996 |   3702779 |
| pghot_recorded_accesses    |         0 |         0 |   3709015 |
| pghot_reported_hintfaults  |         0 |         0 |         0 |
| pghot_reported_hwhints     |         0 |         0 |  21385228 |
| hwhint_total_events        |         0 |         0 |  21385281 |
| hwhint_dram_accesses       |         0 |         0 |  16747814 |
| hwhint_extmem_accesses     |         0 |         0 |   3697760 |
| hwhint_cache_accesses      |         0 |         0 |         0 |
| hwhint_useful_events       |         0 |         0 |  21385228 |
| hwhint_dropped_events      |         0 |         0 |         0 |
| pgmigrate_success          |  26841377 |  40161043 |  30548434 |
+----------------------------+-----------+-----------+-----------+

Key findings
------------
1. Tiering is the dominant win: C2 (base NUMAB2) reaches 2.34x and C3
   (pghot hwhints) 3.17x over the untiered baseline (C1), where the
   working set is stranded on the lower-tier/CXL node 2.

2. pghot hwhints now clearly leads base NUMAB2 on the official metric:
   C3 1.755e9 (3.17x) vs C2 1.298e9 (2.34x) -> ~+35% hmean_TEPS.

3. hwhints/IBS is far more efficient and stable. C3 reaches its higher
   hmean while:
     - promoting only ~3.70M pages, ~1/4 of C2 (13.32M);
     - issuing zero NUMA hint faults / PTE scans (numa_pte_updates=0);
     - being ~3x more consistent (hstddev 2.32e7 vs 6.80e7).
   IBS reported 21.39M hwhint events (16.75M DRAM + 3.70M ext-mem),
   with hwhint_dropped_events=0.

4. median vs harmonic-mean: C2 median (1.394e9) sits well above its
   hmean (1.298e9), reflecting high per-BFS variance in the fault-driven
   path. C3 median (1.805e9) and hmean (1.755e9) sit close together ->
   low variance.

Caveats
-------
* Single run per configuration; the median-vs-hmean spread for C2
  indicates non-trivial run-to-run variance. Repeat runs are advisable
  before drawing firm quantitative conclusions.
* SKIP_VALIDATION=1 was used (timing-only), so TEPS values are flagged
  invalid (!) by Graph500 and are intended for relative comparison only.

On 24-Sep-26 11:52 AM, Bharata B Rao wrote:
> 
> Detailed per-benchmark tables (throughput/latency + vmstat and pghot
> promotion counters) are posted as replies to this thread.

Micro-benchmark

Multi-threaded application with 64 threads that access memory(8G) at
4K granularity repetitively and randomly. The number of accesses per
thread and the randomness pattern for each thread are fixed beforehand.
The accesses are divided into stores and loads in the ratio of 50:50.

Benchmark threads run on Node 0, while memory is initially provisioned on
CXL node 2 before the accesses start.

Repetitive accesses results in lowertier pages becoming hot and kmigrated
detecting and migrating them. The benchmark score is the time taken to
finish the accesses in microseconds. The sooner it finishes the better it is.
All the numbers shown below are average of 3 runs.

Table 1: Completion time (lower is better), avg of 3 runs
64 threads, random 4K access over 8G, mem on CXL node 2, cpu node 0

Config    Time (us)   Time (s)  Speedup
---------------------------------------
C1       98,485,716       98.5    1.00x
C2       38,636,402       38.6    2.55x
C3       44,270,350       44.3    2.22x

Speedup = C1 (base-NUMAB0) time / config time (>1.00x is faster)

Legend:
  C1 = base kernel, NUMAB0 (hot page promotion disabled)
  C2 = base kernel, NUMAB2 (hot page promotion enabled)
  C3 = pghot kernel, IBS profiler, pghot_freq_threshold=1 (NUMAB0)

Table 2: Page migration / hotness vmstat counters (avg of 3 runs)
'-' = counter not present for that kernel/config

vmstat counter                      C1         C2         C3
------------------------------------------------------------
pgpromote_success                    0  2,097,152  1,981,045
pgpromote_candidate_nrl              0  2,412,042  1,981,045
numa_pages_migrated                  0  2,097,152  1,981,045
numa_pte_updates                     0  2,097,152          0
numa_hint_faults                     0  2,412,042          0
pghot_recorded_accesses              -          -  1,985,269
pghot_reported_hintfaults            -          -          0
pghot_reported_hwhints               -          -  5,991,177
hwhint_total_events                  -          -  5,991,177
hwhint_dram_accesses                 -          -  3,716,462
hwhint_extmem_accesses               -          -  1,886,078
hwhint_cache_accesses                -          -          0
hwhint_useful_events                 -          -  5,991,177
hwhint_dropped_events                -          -          0

Legend:
  C1 = base kernel, NUMAB0 (hot page promotion disabled)
  C2 = base kernel, NUMAB2 (hot page promotion enabled)
  C3 = pghot kernel, IBS profiler, pghot_freq_threshold=1 (NUMAB0)