mm/vmscan.c | 98 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 80 insertions(+), 18 deletions(-)
This is an aging speedup series split out from the MGLRU swappiness
series [1], with the inc_min_seq changes separated to make them
easier to review.
Currently, inc_min_seq performance is crucial to both the swappiness
fix and proactive aging. There are two problems with it:
1. It processes each folio one by one, while many operations can be
batched or skipped. For example, a batch of folios can be moved together
from the oldest generation to the second-oldest generation, and the
associated counting can also be done in batches.
2. It may cause potential cold/hot inversion by placing promoted folios
(which have been scanned and found to have young PTEs) behind
non-promoted folios. A similar inversion can also occur among
non-promoted folios, as tail folios from the oldest generation are
placed before head folios when moving them to the second-oldest
generation.
This series tries to batch operations as much as possible and fix the
potential cold/hot inversion by keeping promoted folios ahead of
non-promoted folios, while also preserving the order of non-promoted
folios when moving them from the oldest generation to the second-oldest
generation.
Minor issue: inc_min_seq() also counts protected folios
improperly, as promoted folios should be skipped, as in
sort_folio().
We need a stable workload with a stable number of folios to measure
aging and evaluate the speedup in inc_min_seq(). So I asked ChatGPT
to generate the microbenchmark below. It ages an LRU vec containing
512 MB of memory 100 times:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <time.h>
#include <sys/mman.h>
#define SIZE (512UL * 1024 * 1024)
#define LRU_GEN "/sys/kernel/debug/lru_gen"
#define TARGET_CGROUP "/system.slice/agetest.scope"
#define START_GEN 3
#define END_GEN 103
static long long nsec_diff(const struct timespec *start,
const struct timespec *end)
{
return (end->tv_sec - start->tv_sec) * 1000000000LL +
(end->tv_nsec - start->tv_nsec);
}
static int find_memcg_id(void)
{
FILE *fp;
char line[4096];
int memcg_id;
fp = fopen(LRU_GEN, "r");
if (!fp) {
perror("fopen lru_gen");
return -1;
}
while (fgets(line, sizeof(line), fp)) {
char *p;
if (strncmp(line, "memcg ", 6))
continue;
p = line + 6;
if (sscanf(p, "%d", &memcg_id) != 1)
continue;
/*
* The memcg path follows the numeric ID.
*/
p = strchr(p, ' ');
if (!p)
continue;
if (strstr(p, TARGET_CGROUP)) {
fclose(fp);
return memcg_id;
}
}
fclose(fp);
fprintf(stderr, "Cannot find %s\n", TARGET_CGROUP);
return -1;
}
int main(void)
{
void *addr;
int memcg_id;
int fd;
long long total_ns = 0;
/*
* mmap 512 MB and touch every page.
*/
addr = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap");
return 1;
}
memset(addr, 0x55, SIZE);
printf("mmap: %p, size: %lu MB\n",
addr, SIZE / 1024 / 1024);
/*
* Find the memcg ID automatically.
*/
memcg_id = find_memcg_id();
if (memcg_id < 0)
return 1;
printf("memcg: %d (%s)\n", memcg_id, TARGET_CGROUP);
printf("aging generation %d -> %d\n",
START_GEN, END_GEN);
fd = open(LRU_GEN, O_WRONLY);
if (fd < 0) {
perror("open lru_gen");
return 1;
}
for (int gen = START_GEN; gen <= END_GEN; gen++) {
char buf[128];
int len;
struct timespec start, end;
long long ns;
len = snprintf(buf, sizeof(buf),
"+ %d 0 %d\n", memcg_id, gen);
clock_gettime(CLOCK_MONOTONIC, &start);
if (write(fd, buf, len) != len) {
perror("write lru_gen");
close(fd);
return 1;
}
clock_gettime(CLOCK_MONOTONIC, &end);
ns = nsec_diff(&start, &end);
total_ns += ns;
printf("gen %3d: %8.3f ms\n",
gen, ns / 1000000.0);
fflush(stdout);
}
close(fd);
printf("\nTotal: %.3f ms\n",
total_ns / 1000000.0);
printf("Average: %.3f ms\n",
total_ns / (double)(END_GEN - START_GEN + 1) /
1000000.0);
while (1)
sleep(1);
return 0;
}
Run the above microbenchmark with:
systemd-run --scope --unit=agetest -p MemoryMax=1024M ./agetest
I’m seeing inc_min_seq() become significantly faster:
W/o patch:
Running scope as unit: agetest.scope
mmap: 0x72c1b5a00000, size: 512 MB
memcg: 12673 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen 3: 7.433 ms
gen 4: 0.949 ms
gen 5: 2.535 ms
gen 6: 5.043 ms
gen 7: 5.041 ms
gen 8: 5.027 ms
...
gen 100: 5.035 ms
gen 101: 5.011 ms
gen 102: 5.029 ms
gen 103: 5.056 ms
Total: 503.946 ms
Average: 4.990 ms
W/ patch:
Running scope as unit: agetest.scope
mmap: 0x7c4b1d200000, size: 512 MB
memcg: 12893 (/system.slice/agetest.scope)
aging generation 3 -> 103
gen 3: 7.538 ms
gen 4: 0.937 ms
gen 5: 2.348 ms
gen 6: 2.300 ms
gen 7: 2.302 ms
gen 8: 2.294 ms
gen 9: 2.296 ms
...
gen 100: 2.292 ms
gen 101: 2.307 ms
gen 102: 2.293 ms
gen 103: 2.293 ms
Total: 235.718 ms
Average: 2.334 ms
The average aging time drops from 4.990 ms to 2.334 ms!
[1] https://lore.kernel.org/linux-mm/20260812121658.69965-1-baohua@kernel.org/
Barry Song (Xiaomi) (6):
mm/mglru: batch update lrugen->nr_pages in inc_min_seq()
mm/mglru: batch update lrugen->protected in inc_min_seq()
mm/mglru: enhance cold/hot inversion handling in inc_min_seq()
mm/mglru: exclude folios promoted by aging from protected in
inc_min_seq()
mm/mglru: move folios from oldest gen to second-oldest gen from head
to tail
mm/mglru: batch move folios to the second-oldest gen's LRU
mm/vmscan.c | 98 +++++++++++++++++++++++++++++++++++++++++++----------
1 file changed, 80 insertions(+), 18 deletions(-)
--
2.34.1
On Fri, Aug 21, 2026 at 06:25:32PM +0800, Barry Song (Xiaomi) wrote:
>This is an aging speedup series split out from the MGLRU swappiness
>series [1], with the inc_min_seq changes separated to make them
>easier to review.
>
>Currently, inc_min_seq performance is crucial to both the swappiness
>fix and proactive aging. There are two problems with it:
>
>1. It processes each folio one by one, while many operations can be
>batched or skipped. For example, a batch of folios can be moved together
>from the oldest generation to the second-oldest generation, and the
>associated counting can also be done in batches.
>
>2. It may cause potential cold/hot inversion by placing promoted folios
>(which have been scanned and found to have young PTEs) behind
>non-promoted folios. A similar inversion can also occur among
>non-promoted folios, as tail folios from the oldest generation are
>placed before head folios when moving them to the second-oldest
>generation.
>
>This series tries to batch operations as much as possible and fix the
>potential cold/hot inversion by keeping promoted folios ahead of
>non-promoted folios, while also preserving the order of non-promoted
>folios when moving them from the oldest generation to the second-oldest
>generation.
>
>Minor issue: inc_min_seq() also counts protected folios
>improperly, as promoted folios should be skipped, as in
>sort_folio().
>
>We need a stable workload with a stable number of folios to measure
>aging and evaluate the speedup in inc_min_seq(). So I asked ChatGPT
>to generate the microbenchmark below. It ages an LRU vec containing
>512 MB of memory 100 times:
>
> #define _GNU_SOURCE
>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <stdint.h>
> #include <unistd.h>
> #include <fcntl.h>
> #include <errno.h>
> #include <time.h>
> #include <sys/mman.h>
>
> #define SIZE (512UL * 1024 * 1024)
> #define LRU_GEN "/sys/kernel/debug/lru_gen"
> #define TARGET_CGROUP "/system.slice/agetest.scope"
> #define START_GEN 3
> #define END_GEN 103
>
> static long long nsec_diff(const struct timespec *start,
> const struct timespec *end)
> {
> return (end->tv_sec - start->tv_sec) * 1000000000LL +
> (end->tv_nsec - start->tv_nsec);
> }
>
> static int find_memcg_id(void)
> {
> FILE *fp;
> char line[4096];
> int memcg_id;
>
> fp = fopen(LRU_GEN, "r");
> if (!fp) {
> perror("fopen lru_gen");
> return -1;
> }
>
> while (fgets(line, sizeof(line), fp)) {
> char *p;
>
> if (strncmp(line, "memcg ", 6))
> continue;
>
> p = line + 6;
>
> if (sscanf(p, "%d", &memcg_id) != 1)
> continue;
>
> /*
> * The memcg path follows the numeric ID.
> */
> p = strchr(p, ' ');
> if (!p)
> continue;
>
> if (strstr(p, TARGET_CGROUP)) {
> fclose(fp);
> return memcg_id;
> }
> }
>
> fclose(fp);
>
> fprintf(stderr, "Cannot find %s\n", TARGET_CGROUP);
> return -1;
> }
>
> int main(void)
> {
> void *addr;
> int memcg_id;
> int fd;
> long long total_ns = 0;
>
> /*
> * mmap 512 MB and touch every page.
> */
> addr = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
> MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> if (addr == MAP_FAILED) {
> perror("mmap");
> return 1;
> }
>
> memset(addr, 0x55, SIZE);
>
> printf("mmap: %p, size: %lu MB\n",
> addr, SIZE / 1024 / 1024);
>
> /*
> * Find the memcg ID automatically.
> */
> memcg_id = find_memcg_id();
> if (memcg_id < 0)
> return 1;
>
> printf("memcg: %d (%s)\n", memcg_id, TARGET_CGROUP);
> printf("aging generation %d -> %d\n",
> START_GEN, END_GEN);
>
> fd = open(LRU_GEN, O_WRONLY);
> if (fd < 0) {
> perror("open lru_gen");
> return 1;
> }
>
> for (int gen = START_GEN; gen <= END_GEN; gen++) {
> char buf[128];
> int len;
> struct timespec start, end;
> long long ns;
>
> len = snprintf(buf, sizeof(buf),
> "+ %d 0 %d\n", memcg_id, gen);
>
> clock_gettime(CLOCK_MONOTONIC, &start);
>
> if (write(fd, buf, len) != len) {
> perror("write lru_gen");
> close(fd);
> return 1;
> }
>
> clock_gettime(CLOCK_MONOTONIC, &end);
>
> ns = nsec_diff(&start, &end);
> total_ns += ns;
>
> printf("gen %3d: %8.3f ms\n",
> gen, ns / 1000000.0);
> fflush(stdout);
> }
>
> close(fd);
>
> printf("\nTotal: %.3f ms\n",
> total_ns / 1000000.0);
> printf("Average: %.3f ms\n",
> total_ns / (double)(END_GEN - START_GEN + 1) /
> 1000000.0);
>
> while (1)
> sleep(1);
>
> return 0;
> }
>
>Run the above microbenchmark with:
>systemd-run --scope --unit=agetest -p MemoryMax=1024M ./agetest
>
>I’m seeing inc_min_seq() become significantly faster:
>
>W/o patch:
>
>Running scope as unit: agetest.scope
>mmap: 0x72c1b5a00000, size: 512 MB
>memcg: 12673 (/system.slice/agetest.scope)
>aging generation 3 -> 103
>gen 3: 7.433 ms
>gen 4: 0.949 ms
>gen 5: 2.535 ms
>gen 6: 5.043 ms
>gen 7: 5.041 ms
>gen 8: 5.027 ms
>...
>gen 100: 5.035 ms
>gen 101: 5.011 ms
>gen 102: 5.029 ms
>gen 103: 5.056 ms
>
>Total: 503.946 ms
>Average: 4.990 ms
>
>W/ patch:
>
>Running scope as unit: agetest.scope
>mmap: 0x7c4b1d200000, size: 512 MB
>memcg: 12893 (/system.slice/agetest.scope)
>aging generation 3 -> 103
>gen 3: 7.538 ms
>gen 4: 0.937 ms
>gen 5: 2.348 ms
>gen 6: 2.300 ms
>gen 7: 2.302 ms
>gen 8: 2.294 ms
>gen 9: 2.296 ms
>...
>gen 100: 2.292 ms
>gen 101: 2.307 ms
>gen 102: 2.293 ms
>gen 103: 2.293 ms
>
>Total: 235.718 ms
>Average: 2.334 ms
>
>The average aging time drops from 4.990 ms to 2.334 ms!
Hi Barry,
I tested this series on my arm64 machine (24 cores, 4K base pages)
and reproduced the improvement with the microbenchmark from the
cover letter.
THP=never (PTE):
baseline: 7.644 ms
patched: 2.964 ms (-61.2%)
THP=always (PMD):
baseline: 0.0373 ms
patched: 0.0292 ms (-21.8%)
The PTE-level gain is larger than your x86 numbers (-61.2% vs
-53.2%). I suspect the per-folio cost of inc_min_seq() is higher
on my arm64 machine (or on arm64 in general).
The gain is also larger at the PTE level than at the PMD level
(-61.2% vs -21.8%), matching the much higher folio count
(131072 vs 256).
Tested-by: Xueyuan Chen <xueyuan.chen21@gmail.com>
thanks,
Xueyuan
>[1] https://lore.kernel.org/linux-mm/20260812121658.69965-1-baohua@kernel.org/
>
>Barry Song (Xiaomi) (6):
> mm/mglru: batch update lrugen->nr_pages in inc_min_seq()
> mm/mglru: batch update lrugen->protected in inc_min_seq()
> mm/mglru: enhance cold/hot inversion handling in inc_min_seq()
> mm/mglru: exclude folios promoted by aging from protected in
> inc_min_seq()
> mm/mglru: move folios from oldest gen to second-oldest gen from head
> to tail
> mm/mglru: batch move folios to the second-oldest gen's LRU
>
> mm/vmscan.c | 98 +++++++++++++++++++++++++++++++++++++++++++----------
> 1 file changed, 80 insertions(+), 18 deletions(-)
>
>--
>2.34.1
>
>
© 2016 - 2026 Red Hat, Inc.