include/linux/mmzone.h | 16 +++++++++++++++- kernel/sched/fair.c | 5 +++-- mm/vmstat.c | 1 + 3 files changed, 19 insertions(+), 3 deletions(-)
Goto-san reported confusing pgpromote statistics where the
pgpromote_success count significantly exceeded pgpromote_candidate.
On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB):
# Enable demotion only
echo 1 > /sys/kernel/mm/numa/demotion_enabled
numactl -m 0-1 memhog -r200 3500M >/dev/null &
pid=$!
sleep 2
numactl memhog -r100 2500M >/dev/null &
sleep 10
kill -9 $pid # terminate the 1st memhog
# Enable promotion
echo 2 > /proc/sys/kernel/numa_balancing
After a few seconds, we observeed `pgpromote_candidate < pgpromote_success`
$ grep -e pgpromote /proc/vmstat
pgpromote_success 2579
pgpromote_candidate 0
In this scenario, after terminating the first memhog, the conditions for
pgdat_free_space_enough() are quickly met, and triggers promotion.
However, these migrated pages are only counted for in PGPROMOTE_SUCCESS,
not in PGPROMOTE_CANDIDATE.
To solve this confusing statistics, introduce this PGPROMOTE_CANDIDATE_NRL
to count the missed promotion pages. And also, not counting these pages
into PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or
performance of the promotion rate limit.
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Valentin Schneider <vschneid@redhat.com>
Reported-by: Yasunori Gotou (Fujitsu) <y-goto@fujitsu.com>
Suggested-by: Huang Ying <ying.huang@linux.alibaba.com>
Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
Signed-off-by: Ruan Shiyang <ruansy.fnst@fujitsu.com>
---
Changes since RFC v3:
1. change the naming of new added stat to PGPROMOTE_CANDIDATE_NRL.
2. improve the description of the two stats.
---
include/linux/mmzone.h | 16 +++++++++++++++-
kernel/sched/fair.c | 5 +++--
mm/vmstat.c | 1 +
3 files changed, 19 insertions(+), 3 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 283913d42d7b..4345996a7d5a 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -230,7 +230,21 @@ enum node_stat_item {
#endif
#ifdef CONFIG_NUMA_BALANCING
PGPROMOTE_SUCCESS, /* promote successfully */
- PGPROMOTE_CANDIDATE, /* candidate pages to promote */
+ /**
+ * Candidate pages for promotion based on hint fault latency. This
+ * counter is used to control the promotion rate and adjust the hot
+ * threshold.
+ */
+ PGPROMOTE_CANDIDATE,
+ /**
+ * Not rate-limited (NRL) candidate pages for those can be promoted
+ * without considering hot threshold because of enough free pages in
+ * fast-tier node. These promotions bypass the regular hotness checks
+ * and do NOT influence the promotion rate-limiter or
+ * threshold-adjustment logic.
+ * This is for statistics/monitoring purposes.
+ */
+ PGPROMOTE_CANDIDATE_NRL,
#endif
/* PGDEMOTE_*: pages demoted */
PGDEMOTE_KSWAPD,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 7a14da5396fb..4022c9c1f346 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1940,11 +1940,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
struct pglist_data *pgdat;
unsigned long rate_limit;
unsigned int latency, th, def_th;
+ long nr = folio_nr_pages(folio);
pgdat = NODE_DATA(dst_nid);
if (pgdat_free_space_enough(pgdat)) {
/* workload changed, reset hot threshold */
pgdat->nbp_threshold = 0;
+ mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
return true;
}
@@ -1958,8 +1960,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
if (latency >= th)
return false;
- return !numa_promotion_rate_limit(pgdat, rate_limit,
- folio_nr_pages(folio));
+ return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
}
this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
diff --git a/mm/vmstat.c b/mm/vmstat.c
index a78d70ddeacd..bb0d2b330dd5 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1272,6 +1272,7 @@ const char * const vmstat_text[] = {
#ifdef CONFIG_NUMA_BALANCING
"pgpromote_success",
"pgpromote_candidate",
+ "pgpromote_candidate_nrl",
#endif
"pgdemote_kswapd",
"pgdemote_direct",
--
2.43.0On 7/29/25 05:51, Ruan Shiyang wrote:
> Goto-san reported confusing pgpromote statistics where the
> pgpromote_success count significantly exceeded pgpromote_candidate.
>
> On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB):
> # Enable demotion only
> echo 1 > /sys/kernel/mm/numa/demotion_enabled
> numactl -m 0-1 memhog -r200 3500M >/dev/null &
> pid=$!
> sleep 2
> numactl memhog -r100 2500M >/dev/null &
> sleep 10
> kill -9 $pid # terminate the 1st memhog
> # Enable promotion
> echo 2 > /proc/sys/kernel/numa_balancing
>
> After a few seconds, we observeed `pgpromote_candidate < pgpromote_success`
> $ grep -e pgpromote /proc/vmstat
> pgpromote_success 2579
> pgpromote_candidate 0
>
> In this scenario, after terminating the first memhog, the conditions for
> pgdat_free_space_enough() are quickly met, and triggers promotion.
> However, these migrated pages are only counted for in PGPROMOTE_SUCCESS,
> not in PGPROMOTE_CANDIDATE.
>
> To solve this confusing statistics, introduce this PGPROMOTE_CANDIDATE_NRL
> to count the missed promotion pages. And also, not counting these pages
> into PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or
> performance of the promotion rate limit.
>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Juri Lelli <juri.lelli@redhat.com>
> Cc: Vincent Guittot <vincent.guittot@linaro.org>
> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Ben Segall <bsegall@google.com>
> Cc: Mel Gorman <mgorman@suse.de>
> Cc: Valentin Schneider <vschneid@redhat.com>
> Reported-by: Yasunori Gotou (Fujitsu) <y-goto@fujitsu.com>
> Suggested-by: Huang Ying <ying.huang@linux.alibaba.com>
> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
> Signed-off-by: Ruan Shiyang <ruansy.fnst@fujitsu.com>
Besides my nit, LGTM.
Acked-by: Vlastimil Babka <vbabka@suse.cz>
> ---
> Changes since RFC v3:
> 1. change the naming of new added stat to PGPROMOTE_CANDIDATE_NRL.
> 2. improve the description of the two stats.
> ---
> include/linux/mmzone.h | 16 +++++++++++++++-
> kernel/sched/fair.c | 5 +++--
> mm/vmstat.c | 1 +
> 3 files changed, 19 insertions(+), 3 deletions(-)
>
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 283913d42d7b..4345996a7d5a 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -230,7 +230,21 @@ enum node_stat_item {
> #endif
> #ifdef CONFIG_NUMA_BALANCING
> PGPROMOTE_SUCCESS, /* promote successfully */
> - PGPROMOTE_CANDIDATE, /* candidate pages to promote */
> + /**
> + * Candidate pages for promotion based on hint fault latency. This
> + * counter is used to control the promotion rate and adjust the hot
> + * threshold.
> + */
> + PGPROMOTE_CANDIDATE,
> + /**
> + * Not rate-limited (NRL) candidate pages for those can be promoted
> + * without considering hot threshold because of enough free pages in
> + * fast-tier node. These promotions bypass the regular hotness checks
> + * and do NOT influence the promotion rate-limiter or
> + * threshold-adjustment logic.
> + * This is for statistics/monitoring purposes.
> + */
> + PGPROMOTE_CANDIDATE_NRL,
> #endif
> /* PGDEMOTE_*: pages demoted */
> PGDEMOTE_KSWAPD,
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 7a14da5396fb..4022c9c1f346 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -1940,11 +1940,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
> struct pglist_data *pgdat;
> unsigned long rate_limit;
> unsigned int latency, th, def_th;
> + long nr = folio_nr_pages(folio);
>
> pgdat = NODE_DATA(dst_nid);
> if (pgdat_free_space_enough(pgdat)) {
> /* workload changed, reset hot threshold */
> pgdat->nbp_threshold = 0;
> + mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
> return true;
> }
>
> @@ -1958,8 +1960,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
> if (latency >= th)
> return false;
>
> - return !numa_promotion_rate_limit(pgdat, rate_limit,
> - folio_nr_pages(folio));
> + return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
> }
>
> this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index a78d70ddeacd..bb0d2b330dd5 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1272,6 +1272,7 @@ const char * const vmstat_text[] = {
> #ifdef CONFIG_NUMA_BALANCING
> "pgpromote_success",
> "pgpromote_candidate",
> + "pgpromote_candidate_nrl",
> #endif
> "pgdemote_kswapd",
> "pgdemote_direct",
On 7/29/25 05:51, Ruan Shiyang wrote:
A process nit: your RFC v3 had:
From: Li Zhijian <lizhijian@fujitsu.com>
and this one doesn't.
> Goto-san reported confusing pgpromote statistics where the
> pgpromote_success count significantly exceeded pgpromote_candidate.
>
> On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB):
> # Enable demotion only
> echo 1 > /sys/kernel/mm/numa/demotion_enabled
> numactl -m 0-1 memhog -r200 3500M >/dev/null &
> pid=$!
> sleep 2
> numactl memhog -r100 2500M >/dev/null &
> sleep 10
> kill -9 $pid # terminate the 1st memhog
> # Enable promotion
> echo 2 > /proc/sys/kernel/numa_balancing
>
> After a few seconds, we observeed `pgpromote_candidate < pgpromote_success`
> $ grep -e pgpromote /proc/vmstat
> pgpromote_success 2579
> pgpromote_candidate 0
>
> In this scenario, after terminating the first memhog, the conditions for
> pgdat_free_space_enough() are quickly met, and triggers promotion.
> However, these migrated pages are only counted for in PGPROMOTE_SUCCESS,
> not in PGPROMOTE_CANDIDATE.
>
> To solve this confusing statistics, introduce this PGPROMOTE_CANDIDATE_NRL
> to count the missed promotion pages. And also, not counting these pages
> into PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or
> performance of the promotion rate limit.
>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Juri Lelli <juri.lelli@redhat.com>
> Cc: Vincent Guittot <vincent.guittot@linaro.org>
> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Ben Segall <bsegall@google.com>
> Cc: Mel Gorman <mgorman@suse.de>
> Cc: Valentin Schneider <vschneid@redhat.com>
> Reported-by: Yasunori Gotou (Fujitsu) <y-goto@fujitsu.com>
> Suggested-by: Huang Ying <ying.huang@linux.alibaba.com>
> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
So the S-o-b from Li doesn't match anything now.
You can either reinstate that "From: Li ..." or add a "Co-developed-by: Li
..." right above the "S-o-b: Li ..." - that's for you two to decide who is
the main author.
More details in Documentation/process/submitting-patches.rst
> Signed-off-by: Ruan Shiyang <ruansy.fnst@fujitsu.com>
> ---
> Changes since RFC v3:
> 1. change the naming of new added stat to PGPROMOTE_CANDIDATE_NRL.
> 2. improve the description of the two stats.
> ---
> include/linux/mmzone.h | 16 +++++++++++++++-
> kernel/sched/fair.c | 5 +++--
> mm/vmstat.c | 1 +
> 3 files changed, 19 insertions(+), 3 deletions(-)
>
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 283913d42d7b..4345996a7d5a 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -230,7 +230,21 @@ enum node_stat_item {
> #endif
> #ifdef CONFIG_NUMA_BALANCING
> PGPROMOTE_SUCCESS, /* promote successfully */
> - PGPROMOTE_CANDIDATE, /* candidate pages to promote */
> + /**
> + * Candidate pages for promotion based on hint fault latency. This
> + * counter is used to control the promotion rate and adjust the hot
> + * threshold.
> + */
> + PGPROMOTE_CANDIDATE,
> + /**
> + * Not rate-limited (NRL) candidate pages for those can be promoted
> + * without considering hot threshold because of enough free pages in
> + * fast-tier node. These promotions bypass the regular hotness checks
> + * and do NOT influence the promotion rate-limiter or
> + * threshold-adjustment logic.
> + * This is for statistics/monitoring purposes.
> + */
> + PGPROMOTE_CANDIDATE_NRL,
> #endif
> /* PGDEMOTE_*: pages demoted */
> PGDEMOTE_KSWAPD,
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 7a14da5396fb..4022c9c1f346 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -1940,11 +1940,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
> struct pglist_data *pgdat;
> unsigned long rate_limit;
> unsigned int latency, th, def_th;
> + long nr = folio_nr_pages(folio);
>
> pgdat = NODE_DATA(dst_nid);
> if (pgdat_free_space_enough(pgdat)) {
> /* workload changed, reset hot threshold */
> pgdat->nbp_threshold = 0;
> + mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
> return true;
> }
>
> @@ -1958,8 +1960,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
> if (latency >= th)
> return false;
>
> - return !numa_promotion_rate_limit(pgdat, rate_limit,
> - folio_nr_pages(folio));
> + return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
> }
>
> this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index a78d70ddeacd..bb0d2b330dd5 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1272,6 +1272,7 @@ const char * const vmstat_text[] = {
> #ifdef CONFIG_NUMA_BALANCING
> "pgpromote_success",
> "pgpromote_candidate",
> + "pgpromote_candidate_nrl",
> #endif
> "pgdemote_kswapd",
> "pgdemote_direct",
在 2025/8/29 17:08, Vlastimil Babka 写道:
> On 7/29/25 05:51, Ruan Shiyang wrote:
>
> A process nit: your RFC v3 had:
>
> From: Li Zhijian <lizhijian@fujitsu.com>
>
> and this one doesn't.
>
>> Goto-san reported confusing pgpromote statistics where the
>> pgpromote_success count significantly exceeded pgpromote_candidate.
>>
>> On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB):
>> # Enable demotion only
>> echo 1 > /sys/kernel/mm/numa/demotion_enabled
>> numactl -m 0-1 memhog -r200 3500M >/dev/null &
>> pid=$!
>> sleep 2
>> numactl memhog -r100 2500M >/dev/null &
>> sleep 10
>> kill -9 $pid # terminate the 1st memhog
>> # Enable promotion
>> echo 2 > /proc/sys/kernel/numa_balancing
>>
>> After a few seconds, we observeed `pgpromote_candidate < pgpromote_success`
>> $ grep -e pgpromote /proc/vmstat
>> pgpromote_success 2579
>> pgpromote_candidate 0
>>
>> In this scenario, after terminating the first memhog, the conditions for
>> pgdat_free_space_enough() are quickly met, and triggers promotion.
>> However, these migrated pages are only counted for in PGPROMOTE_SUCCESS,
>> not in PGPROMOTE_CANDIDATE.
>>
>> To solve this confusing statistics, introduce this PGPROMOTE_CANDIDATE_NRL
>> to count the missed promotion pages. And also, not counting these pages
>> into PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or
>> performance of the promotion rate limit.
>>
>> Cc: Ingo Molnar <mingo@redhat.com>
>> Cc: Peter Zijlstra <peterz@infradead.org>
>> Cc: Juri Lelli <juri.lelli@redhat.com>
>> Cc: Vincent Guittot <vincent.guittot@linaro.org>
>> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
>> Cc: Steven Rostedt <rostedt@goodmis.org>
>> Cc: Ben Segall <bsegall@google.com>
>> Cc: Mel Gorman <mgorman@suse.de>
>> Cc: Valentin Schneider <vschneid@redhat.com>
>> Reported-by: Yasunori Gotou (Fujitsu) <y-goto@fujitsu.com>
>> Suggested-by: Huang Ying <ying.huang@linux.alibaba.com>
>> Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
>
> So the S-o-b from Li doesn't match anything now.
> You can either reinstate that "From: Li ..." or add a "Co-developed-by: Li
> ..." right above the "S-o-b: Li ..." - that's for you two to decide who is
> the main author.
Thanks for pointing out. I wasn't aware of this.
I'd like to add a Co-developed-by tag:
Co-developed-by: Li Zhijian
Then, should I resend a new version with is tag added? Or you will do that for me?
--
Best regards,
Ruan.
>
> More details in Documentation/process/submitting-patches.rst
>
>> Signed-off-by: Ruan Shiyang <ruansy.fnst@fujitsu.com>
>> ---
>> Changes since RFC v3:
>> 1. change the naming of new added stat to PGPROMOTE_CANDIDATE_NRL.
>> 2. improve the description of the two stats.
>> ---
>> include/linux/mmzone.h | 16 +++++++++++++++-
>> kernel/sched/fair.c | 5 +++--
>> mm/vmstat.c | 1 +
>> 3 files changed, 19 insertions(+), 3 deletions(-)
>>
>> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
>> index 283913d42d7b..4345996a7d5a 100644
>> --- a/include/linux/mmzone.h
>> +++ b/include/linux/mmzone.h
>> @@ -230,7 +230,21 @@ enum node_stat_item {
>> #endif
>> #ifdef CONFIG_NUMA_BALANCING
>> PGPROMOTE_SUCCESS, /* promote successfully */
>> - PGPROMOTE_CANDIDATE, /* candidate pages to promote */
>> + /**
>> + * Candidate pages for promotion based on hint fault latency. This
>> + * counter is used to control the promotion rate and adjust the hot
>> + * threshold.
>> + */
>> + PGPROMOTE_CANDIDATE,
>> + /**
>> + * Not rate-limited (NRL) candidate pages for those can be promoted
>> + * without considering hot threshold because of enough free pages in
>> + * fast-tier node. These promotions bypass the regular hotness checks
>> + * and do NOT influence the promotion rate-limiter or
>> + * threshold-adjustment logic.
>> + * This is for statistics/monitoring purposes.
>> + */
>> + PGPROMOTE_CANDIDATE_NRL,
>> #endif
>> /* PGDEMOTE_*: pages demoted */
>> PGDEMOTE_KSWAPD,
>> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
>> index 7a14da5396fb..4022c9c1f346 100644
>> --- a/kernel/sched/fair.c
>> +++ b/kernel/sched/fair.c
>> @@ -1940,11 +1940,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
>> struct pglist_data *pgdat;
>> unsigned long rate_limit;
>> unsigned int latency, th, def_th;
>> + long nr = folio_nr_pages(folio);
>>
>> pgdat = NODE_DATA(dst_nid);
>> if (pgdat_free_space_enough(pgdat)) {
>> /* workload changed, reset hot threshold */
>> pgdat->nbp_threshold = 0;
>> + mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
>> return true;
>> }
>>
>> @@ -1958,8 +1960,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
>> if (latency >= th)
>> return false;
>>
>> - return !numa_promotion_rate_limit(pgdat, rate_limit,
>> - folio_nr_pages(folio));
>> + return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
>> }
>>
>> this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
>> diff --git a/mm/vmstat.c b/mm/vmstat.c
>> index a78d70ddeacd..bb0d2b330dd5 100644
>> --- a/mm/vmstat.c
>> +++ b/mm/vmstat.c
>> @@ -1272,6 +1272,7 @@ const char * const vmstat_text[] = {
>> #ifdef CONFIG_NUMA_BALANCING
>> "pgpromote_success",
>> "pgpromote_candidate",
>> + "pgpromote_candidate_nrl",
>> #endif
>> "pgdemote_kswapd",
>> "pgdemote_direct",
>
On 8/29/25 11:18, Shiyang Ruan wrote: >> >> So the S-o-b from Li doesn't match anything now. >> You can either reinstate that "From: Li ..." or add a "Co-developed-by: Li >> ..." right above the "S-o-b: Li ..." - that's for you two to decide who is >> the main author. > > Thanks for pointing out. I wasn't aware of this. > > I'd like to add a Co-developed-by tag: > > Co-developed-by: Li Zhijian > > Then, should I resend a new version with is tag added? Or you will do that for me? Yeah it would be best if you sent it to make things clear. Andrew can then replace or update it in mm-unstable. Thanks.
Ruan Shiyang <ruansy.fnst@fujitsu.com> writes: > Goto-san reported confusing pgpromote statistics where the > pgpromote_success count significantly exceeded pgpromote_candidate. > > On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB): > # Enable demotion only > echo 1 > /sys/kernel/mm/numa/demotion_enabled > numactl -m 0-1 memhog -r200 3500M >/dev/null & > pid=$! > sleep 2 > numactl memhog -r100 2500M >/dev/null & > sleep 10 > kill -9 $pid # terminate the 1st memhog > # Enable promotion > echo 2 > /proc/sys/kernel/numa_balancing > > After a few seconds, we observeed `pgpromote_candidate < pgpromote_success` > $ grep -e pgpromote /proc/vmstat > pgpromote_success 2579 > pgpromote_candidate 0 > > In this scenario, after terminating the first memhog, the conditions for > pgdat_free_space_enough() are quickly met, and triggers promotion. > However, these migrated pages are only counted for in PGPROMOTE_SUCCESS, > not in PGPROMOTE_CANDIDATE. > > To solve this confusing statistics, introduce this PGPROMOTE_CANDIDATE_NRL > to count the missed promotion pages. And also, not counting these pages > into PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or > performance of the promotion rate limit. > > Cc: Ingo Molnar <mingo@redhat.com> > Cc: Peter Zijlstra <peterz@infradead.org> > Cc: Juri Lelli <juri.lelli@redhat.com> > Cc: Vincent Guittot <vincent.guittot@linaro.org> > Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> > Cc: Steven Rostedt <rostedt@goodmis.org> > Cc: Ben Segall <bsegall@google.com> > Cc: Mel Gorman <mgorman@suse.de> > Cc: Valentin Schneider <vschneid@redhat.com> > Reported-by: Yasunori Gotou (Fujitsu) <y-goto@fujitsu.com> > Suggested-by: Huang Ying <ying.huang@linux.alibaba.com> > Signed-off-by: Li Zhijian <lizhijian@fujitsu.com> > Signed-off-by: Ruan Shiyang <ruansy.fnst@fujitsu.com> LGTM, feel free to add my Reviewed-by: Huang Ying <ying.huang@linux.alibaba.com> in the future version. [snip] --- Best Regards, Huang, Ying
Goto-san reported confusing pgpromote statistics where the
pgpromote_success count significantly exceeded pgpromote_candidate.
On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB):
# Enable demotion only
echo 1 > /sys/kernel/mm/numa/demotion_enabled
numactl -m 0-1 memhog -r200 3500M >/dev/null &
pid=$!
sleep 2
numactl memhog -r100 2500M >/dev/null &
sleep 10
kill -9 $pid # terminate the 1st memhog
# Enable promotion
echo 2 > /proc/sys/kernel/numa_balancing
After a few seconds, we observeed `pgpromote_candidate < pgpromote_success`
$ grep -e pgpromote /proc/vmstat
pgpromote_success 2579
pgpromote_candidate 0
In this scenario, after terminating the first memhog, the conditions for
pgdat_free_space_enough() are quickly met, and triggers promotion.
However, these migrated pages are only counted for in PGPROMOTE_SUCCESS,
not in PGPROMOTE_CANDIDATE.
To solve these confusing statistics, introduce PGPROMOTE_CANDIDATE_NRL to
count the missed promotion pages. And also, not counting these pages into
PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or
performance of the promotion rate limit.
Link: https://lkml.kernel.org/r/20250729035101.1601407-1-ruansy.fnst@fujitsu.com
Co-developed-by: Li Zhijian <lizhijian@fujitsu.com>
Signed-off-by: Li Zhijian <lizhijian@fujitsu.com>
Signed-off-by: Ruan Shiyang <ruansy.fnst@fujitsu.com>
Reported-by: Yasunori Gotou (Fujitsu) <y-goto@fujitsu.com>
Suggested-by: Huang Ying <ying.huang@linux.alibaba.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
Changes since v2:
1. add 'Co-developed-by: Li Zhijian' followed by 'Signed-off-by' per Vlastimil.
---
include/linux/mmzone.h | 16 +++++++++++++++-
kernel/sched/fair.c | 5 +++--
mm/vmstat.c | 1 +
3 files changed, 19 insertions(+), 3 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 0c5da9141983..9d3ea9085556 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -234,7 +234,21 @@ enum node_stat_item {
#endif
#ifdef CONFIG_NUMA_BALANCING
PGPROMOTE_SUCCESS, /* promote successfully */
- PGPROMOTE_CANDIDATE, /* candidate pages to promote */
+ /**
+ * Candidate pages for promotion based on hint fault latency. This
+ * counter is used to control the promotion rate and adjust the hot
+ * threshold.
+ */
+ PGPROMOTE_CANDIDATE,
+ /**
+ * Not rate-limited (NRL) candidate pages for those can be promoted
+ * without considering hot threshold because of enough free pages in
+ * fast-tier node. These promotions bypass the regular hotness checks
+ * and do NOT influence the promotion rate-limiter or
+ * threshold-adjustment logic.
+ * This is for statistics/monitoring purposes.
+ */
+ PGPROMOTE_CANDIDATE_NRL,
#endif
/* PGDEMOTE_*: pages demoted */
PGDEMOTE_KSWAPD,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b173a059315c..82c8d804c54c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1923,11 +1923,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
struct pglist_data *pgdat;
unsigned long rate_limit;
unsigned int latency, th, def_th;
+ long nr = folio_nr_pages(folio);
pgdat = NODE_DATA(dst_nid);
if (pgdat_free_space_enough(pgdat)) {
/* workload changed, reset hot threshold */
pgdat->nbp_threshold = 0;
+ mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
return true;
}
@@ -1941,8 +1943,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
if (latency >= th)
return false;
- return !numa_promotion_rate_limit(pgdat, rate_limit,
- folio_nr_pages(folio));
+ return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
}
this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 71cd1ceba191..e74f0b2a1021 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1280,6 +1280,7 @@ const char * const vmstat_text[] = {
#ifdef CONFIG_NUMA_BALANCING
[I(PGPROMOTE_SUCCESS)] = "pgpromote_success",
[I(PGPROMOTE_CANDIDATE)] = "pgpromote_candidate",
+ [I(PGPROMOTE_CANDIDATE_NRL)] = "pgpromote_candidate_nrl",
#endif
[I(PGDEMOTE_KSWAPD)] = "pgdemote_kswapd",
[I(PGDEMOTE_DIRECT)] = "pgdemote_direct",
--
2.43.0On Mon, 1 Sep 2025 17:01:22 +0800 Ruan Shiyang <ruansy.fnst@fujitsu.com> wrote: > Goto-san reported confusing pgpromote statistics where the > pgpromote_success count significantly exceeded pgpromote_candidate. > > On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB): > # Enable demotion only > echo 1 > /sys/kernel/mm/numa/demotion_enabled > numactl -m 0-1 memhog -r200 3500M >/dev/null & > pid=$! > sleep 2 > numactl memhog -r100 2500M >/dev/null & > sleep 10 > kill -9 $pid # terminate the 1st memhog > # Enable promotion > echo 2 > /proc/sys/kernel/numa_balancing > > After a few seconds, we observeed `pgpromote_candidate < pgpromote_success` > $ grep -e pgpromote /proc/vmstat > pgpromote_success 2579 > pgpromote_candidate 0 > > In this scenario, after terminating the first memhog, the conditions for > pgdat_free_space_enough() are quickly met, and triggers promotion. > However, these migrated pages are only counted for in PGPROMOTE_SUCCESS, > not in PGPROMOTE_CANDIDATE. > > To solve these confusing statistics, introduce PGPROMOTE_CANDIDATE_NRL to > count the missed promotion pages. And also, not counting these pages into > PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or > performance of the promotion rate limit. > > ... > It would be good to have a Fixes: here, to tell people how far back to backport it. Could be either c6833e10008f or c959924b0dc5 afaict. I'll go with c6833e10008f, OK?
On 9/1/25 21:59, Andrew Morton wrote: > On Mon, 1 Sep 2025 17:01:22 +0800 Ruan Shiyang <ruansy.fnst@fujitsu.com> wrote: > >> Goto-san reported confusing pgpromote statistics where the >> pgpromote_success count significantly exceeded pgpromote_candidate. >> >> On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB): >> # Enable demotion only >> echo 1 > /sys/kernel/mm/numa/demotion_enabled >> numactl -m 0-1 memhog -r200 3500M >/dev/null & >> pid=$! >> sleep 2 >> numactl memhog -r100 2500M >/dev/null & >> sleep 10 >> kill -9 $pid # terminate the 1st memhog >> # Enable promotion >> echo 2 > /proc/sys/kernel/numa_balancing >> >> After a few seconds, we observeed `pgpromote_candidate < pgpromote_success` >> $ grep -e pgpromote /proc/vmstat >> pgpromote_success 2579 >> pgpromote_candidate 0 >> >> In this scenario, after terminating the first memhog, the conditions for >> pgdat_free_space_enough() are quickly met, and triggers promotion. >> However, these migrated pages are only counted for in PGPROMOTE_SUCCESS, >> not in PGPROMOTE_CANDIDATE. >> >> To solve these confusing statistics, introduce PGPROMOTE_CANDIDATE_NRL to >> count the missed promotion pages. And also, not counting these pages into >> PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or >> performance of the promotion rate limit. >> >> ... >> > > It would be good to have a Fixes: here, to tell people how far back to > backport it. > > Could be either c6833e10008f or c959924b0dc5 afaict. I'll go with > c6833e10008f, OK? LGTM as a helpful pointer, but I don't think Cc: stable is necessary for "admin might be confused" kind of thing if that's there since 6.1 and only came up now.
On Mon, 1 Sep 2025 22:34:32 +0200 Vlastimil Babka <vbabka@suse.cz> wrote: > > Could be either c6833e10008f or c959924b0dc5 afaict. I'll go with > > c6833e10008f, OK? > > LGTM as a helpful pointer, but I don't think Cc: stable is necessary for > "admin might be confused" kind of thing if that's there since 6.1 and only > came up now. OK, thanks.
Ruan Shiyang <ruansy.fnst@fujitsu.com> writes: > Goto-san reported confusing pgpromote statistics where the > pgpromote_success count significantly exceeded pgpromote_candidate. > > On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB): > # Enable demotion only > echo 1 > /sys/kernel/mm/numa/demotion_enabled > numactl -m 0-1 memhog -r200 3500M >/dev/null & > pid=$! > sleep 2 > numactl memhog -r100 2500M >/dev/null & > sleep 10 > kill -9 $pid # terminate the 1st memhog > # Enable promotion > echo 2 > /proc/sys/kernel/numa_balancing > > After a few seconds, we observeed `pgpromote_candidate < pgpromote_success` > $ grep -e pgpromote /proc/vmstat > pgpromote_success 2579 > pgpromote_candidate 0 > > In this scenario, after terminating the first memhog, the conditions for > pgdat_free_space_enough() are quickly met, and triggers promotion. > However, these migrated pages are only counted for in PGPROMOTE_SUCCESS, > not in PGPROMOTE_CANDIDATE. > > To solve these confusing statistics, introduce PGPROMOTE_CANDIDATE_NRL to > count the missed promotion pages. And also, not counting these pages into > PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or > performance of the promotion rate limit. > > Link: https://lkml.kernel.org/r/20250729035101.1601407-1-ruansy.fnst@fujitsu.com > Co-developed-by: Li Zhijian <lizhijian@fujitsu.com> > Signed-off-by: Li Zhijian <lizhijian@fujitsu.com> > Signed-off-by: Ruan Shiyang <ruansy.fnst@fujitsu.com> > Reported-by: Yasunori Gotou (Fujitsu) <y-goto@fujitsu.com> > Suggested-by: Huang Ying <ying.huang@linux.alibaba.com> > Acked-by: Vlastimil Babka <vbabka@suse.cz> > Cc: Ingo Molnar <mingo@redhat.com> > Cc: Peter Zijlstra <peterz@infradead.org> > Cc: Juri Lelli <juri.lelli@redhat.com> > Cc: Vincent Guittot <vincent.guittot@linaro.org> > Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> > Cc: Steven Rostedt <rostedt@goodmis.org> > Cc: Ben Segall <bsegall@google.com> > Cc: Mel Gorman <mgorman@suse.de> > Cc: Valentin Schneider <vschneid@redhat.com> > Cc: <stable@vger.kernel.org> > Signed-off-by: Andrew Morton <akpm@linux-foundation.org> LGTM, feel free to add my Reviewed-by: Huang Ying <ying.huang@linux.alibaba.com> in the future versions. [snip] --- Best Regards, Huang, Ying
Goto-san reported confusing pgpromote statistics where the
pgpromote_success count significantly exceeded pgpromote_candidate.
On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB):
# Enable demotion only
echo 1 > /sys/kernel/mm/numa/demotion_enabled
numactl -m 0-1 memhog -r200 3500M >/dev/null &
pid=$!
sleep 2
numactl memhog -r100 2500M >/dev/null &
sleep 10
kill -9 $pid # terminate the 1st memhog
# Enable promotion
echo 2 > /proc/sys/kernel/numa_balancing
After a few seconds, we observeed `pgpromote_candidate < pgpromote_success`
$ grep -e pgpromote /proc/vmstat
pgpromote_success 2579
pgpromote_candidate 0
In this scenario, after terminating the first memhog, the conditions for
pgdat_free_space_enough() are quickly met, and triggers promotion.
However, these migrated pages are only counted for in PGPROMOTE_SUCCESS,
not in PGPROMOTE_CANDIDATE.
To solve these confusing statistics, introduce PGPROMOTE_CANDIDATE_NRL to
count the missed promotion pages. And also, not counting these pages into
PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or
performance of the promotion rate limit.
Link: https://lkml.kernel.org/r/20250729035101.1601407-1-ruansy.fnst@fujitsu.com
Co-developed-by: Li Zhijian <lizhijian@fujitsu.com>
Signed-off-by: Ruan Shiyang <ruansy.fnst@fujitsu.com>
Reported-by: Yasunori Gotou (Fujitsu) <y-goto@fujitsu.com>
Suggested-by: Huang Ying <ying.huang@linux.alibaba.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Juri Lelli <juri.lelli@redhat.com>
Cc: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ben Segall <bsegall@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Valentin Schneider <vschneid@redhat.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
Changes since v1:
1. change Li Zhijian from 'Signed-off-by' to 'Co-developed-by' per Vlastimil.
2. add Acked-by: Vlastimil Babka <vbabka@suse.cz>
---
include/linux/mmzone.h | 16 +++++++++++++++-
kernel/sched/fair.c | 5 +++--
mm/vmstat.c | 1 +
3 files changed, 19 insertions(+), 3 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 0c5da9141983..9d3ea9085556 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -234,7 +234,21 @@ enum node_stat_item {
#endif
#ifdef CONFIG_NUMA_BALANCING
PGPROMOTE_SUCCESS, /* promote successfully */
- PGPROMOTE_CANDIDATE, /* candidate pages to promote */
+ /**
+ * Candidate pages for promotion based on hint fault latency. This
+ * counter is used to control the promotion rate and adjust the hot
+ * threshold.
+ */
+ PGPROMOTE_CANDIDATE,
+ /**
+ * Not rate-limited (NRL) candidate pages for those can be promoted
+ * without considering hot threshold because of enough free pages in
+ * fast-tier node. These promotions bypass the regular hotness checks
+ * and do NOT influence the promotion rate-limiter or
+ * threshold-adjustment logic.
+ * This is for statistics/monitoring purposes.
+ */
+ PGPROMOTE_CANDIDATE_NRL,
#endif
/* PGDEMOTE_*: pages demoted */
PGDEMOTE_KSWAPD,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b173a059315c..82c8d804c54c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1923,11 +1923,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
struct pglist_data *pgdat;
unsigned long rate_limit;
unsigned int latency, th, def_th;
+ long nr = folio_nr_pages(folio);
pgdat = NODE_DATA(dst_nid);
if (pgdat_free_space_enough(pgdat)) {
/* workload changed, reset hot threshold */
pgdat->nbp_threshold = 0;
+ mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr);
return true;
}
@@ -1941,8 +1943,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
if (latency >= th)
return false;
- return !numa_promotion_rate_limit(pgdat, rate_limit,
- folio_nr_pages(folio));
+ return !numa_promotion_rate_limit(pgdat, rate_limit, nr);
}
this_cpupid = cpu_pid_to_cpupid(dst_cpu, current->pid);
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 71cd1ceba191..e74f0b2a1021 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1280,6 +1280,7 @@ const char * const vmstat_text[] = {
#ifdef CONFIG_NUMA_BALANCING
[I(PGPROMOTE_SUCCESS)] = "pgpromote_success",
[I(PGPROMOTE_CANDIDATE)] = "pgpromote_candidate",
+ [I(PGPROMOTE_CANDIDATE_NRL)] = "pgpromote_candidate_nrl",
#endif
[I(PGDEMOTE_KSWAPD)] = "pgdemote_kswapd",
[I(PGDEMOTE_DIRECT)] = "pgdemote_direct",
--
2.43.0On 9/1/25 04:05, Ruan Shiyang wrote: > Goto-san reported confusing pgpromote statistics where the > pgpromote_success count significantly exceeded pgpromote_candidate. > > On a system with three nodes (nodes 0-1: DRAM 4GB, node 2: NVDIMM 4GB): > # Enable demotion only > echo 1 > /sys/kernel/mm/numa/demotion_enabled > numactl -m 0-1 memhog -r200 3500M >/dev/null & > pid=$! > sleep 2 > numactl memhog -r100 2500M >/dev/null & > sleep 10 > kill -9 $pid # terminate the 1st memhog > # Enable promotion > echo 2 > /proc/sys/kernel/numa_balancing > > After a few seconds, we observeed `pgpromote_candidate < pgpromote_success` > $ grep -e pgpromote /proc/vmstat > pgpromote_success 2579 > pgpromote_candidate 0 > > In this scenario, after terminating the first memhog, the conditions for > pgdat_free_space_enough() are quickly met, and triggers promotion. > However, these migrated pages are only counted for in PGPROMOTE_SUCCESS, > not in PGPROMOTE_CANDIDATE. > > To solve these confusing statistics, introduce PGPROMOTE_CANDIDATE_NRL to > count the missed promotion pages. And also, not counting these pages into > PGPROMOTE_CANDIDATE is to avoid changing the existing algorithm or > performance of the promotion rate limit. > > Link: https://lkml.kernel.org/r/20250729035101.1601407-1-ruansy.fnst@fujitsu.com > Co-developed-by: Li Zhijian <lizhijian@fujitsu.com> > Signed-off-by: Ruan Shiyang <ruansy.fnst@fujitsu.com> > Reported-by: Yasunori Gotou (Fujitsu) <y-goto@fujitsu.com> > Suggested-by: Huang Ying <ying.huang@linux.alibaba.com> > Acked-by: Vlastimil Babka <vbabka@suse.cz> > Cc: Ingo Molnar <mingo@redhat.com> > Cc: Peter Zijlstra <peterz@infradead.org> > Cc: Juri Lelli <juri.lelli@redhat.com> > Cc: Vincent Guittot <vincent.guittot@linaro.org> > Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> > Cc: Steven Rostedt <rostedt@goodmis.org> > Cc: Ben Segall <bsegall@google.com> > Cc: Mel Gorman <mgorman@suse.de> > Cc: Valentin Schneider <vschneid@redhat.com> > Cc: <stable@vger.kernel.org> > Signed-off-by: Andrew Morton <akpm@linux-foundation.org> > --- > Changes since v1: > 1. change Li Zhijian from 'Signed-off-by' to 'Co-developed-by' per Vlastimil. Note according to the docs it should be both, Co-developed-by followed by Signed-off-by.
© 2016 - 2025 Red Hat, Inc.