We have observed some cases where memory is allocated with GFP_NOIO, so
we cannot reclaim any anon folios unless they are in swapcache. We can
end up spending more than 150 ms looping in `shrink_folio_list()` scanning
non-swapcache folios without reclaiming a single folio. This is pure
overhead.

This is particularly true on systems using zRAM, where swapcache is
relatively rare. So let's check whether anon reclaim is allowed by
GFP_IO and whether there is enough swapcache to make it worthwhile. If
the swapcache is extremely low, we're essentially searching for a
needle in a haystack, so let's avoid scanning anon in the first place.

On Android this is triggered by dm-verity hash-block reads through
dm-bufio, which use GFP_NOIO:

  verity_verify_io -> verity_hash_for_block -> verity_verify_level
    -> dm_bufio_read_with_ioprio -> new_read -> __bufio_new
      -> alloc_buffer
         gfp: GFP_NOIO | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN

Such a reclaimer can land on a memcg with a large, unswapped anon LRU and
a tiny file LRU (e.g. inactive_anon ~335 MB vs inactive_file ~4 MB, with
negligible swapcache). shrink_lruvec() then keeps feeding that huge anon
list into shrink_folio_list() - ~2400 shrink_folio_list() calls, ~93,000
anon folios scanned - where every folio is kept because it needs IO. The
150+ ms above is one such single shrink_lruvec() pass (not accumulated
across a reclaim cycle), and it reclaims nothing; the actual progress
comes entirely from the file side.

Aging anon alongside file does have some value for a later __GFP_IO
reclaimer, so it is not strictly pure overhead. But that aging is only
deferred, not lost: kswapd and other __GFP_IO reclaimers still walk and
age anon. Spending ~168 ms aging memory that this context cannot reclaim
is not a worthwhile trade-off in a latency-sensitive path.

To stay conservative, this only skips anon when the swapcache is really
tiny - below 1/64 of the anon LRU - i.e. when essentially no anon on the
list can be reclaimed without IO. Whenever there is a meaningful amount of
swapcached anon, the normal path is used and anon is scanned and aged as
before.

Signed-off-by: Bo Zhang <zhangbo56@xiaomi.com>
---
v1 -> v2:
 - Use mem_cgroup_lruvec() instead of get_lruvec(), which returns the raw
   node lruvec for a NULL memcg and would be misinterpreted by
   lruvec_page_state()'s container_of() during global reclaim. This also
   drops the get_lruvec() move. (reported by the sashiko bot, suggested
   by Barry Song)
 - Drop the SWAP_CLUSTER_MAX cap on the threshold; the check is purely
   proportional now (swapcache below 1/64 of the anon LRU).
 - Expand the changelog with the workload, the dm-verity/dm-bufio NOIO
   stack, the ~168 ms single shrink_lruvec() breakdown, and the aging
   trade-off discussed with Johannes Weiner.

 mm/vmscan.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 245f68c75b28..e20ac2cb4dd5 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -362,6 +362,23 @@ static bool can_demote(int nid, struct scan_control *sc,
 	return !nodes_empty(allowed_mask);
 }
 
+static inline bool reclaimable_anon_is_low(struct mem_cgroup *memcg,
+		int nid, struct scan_control *sc)
+{
+	struct lruvec *lruvec;
+	unsigned long anon_pages, swapcache;
+
+	if (!sc || (sc->gfp_mask & __GFP_IO))
+		return false;
+
+	lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
+	anon_pages = lruvec_page_state(lruvec, NR_INACTIVE_ANON) +
+		     lruvec_page_state(lruvec, NR_ACTIVE_ANON);
+	swapcache = lruvec_page_state(lruvec, NR_SWAPCACHE);
+
+	return swapcache < (anon_pages >> 6);
+}
+
 static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
 					  int nid,
 					  struct scan_control *sc)
@@ -371,11 +388,13 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
 		 * For non-memcg reclaim, is there
 		 * space in any swap device?
 		 */
-		if (get_nr_swap_pages() > 0)
+		if (get_nr_swap_pages() > 0 &&
+		    !reclaimable_anon_is_low(memcg, nid, sc))
 			return true;
 	} else {
 		/* Is the memcg below its swap limit? */
-		if (mem_cgroup_get_nr_swap_pages(memcg) > 0)
+		if (mem_cgroup_get_nr_swap_pages(memcg) > 0 &&
+		    !reclaimable_anon_is_low(memcg, nid, sc))
 			return true;
 	}
 
-- 
2.34.1

This is the friendly patch-bot of Lorenzo Stoakes.

You have sent him a patch that has triggered this response.

He used to manually respond to these common problems, but in order to save
his sanity (he kept writing the same thing over and over, yet to different
people), I was created.

Hopefully you will not take offence and will fix the problem in your patch
and resubmit it so that it can be accepted into the Linux kernel tree.

When sending emails to mm:

New revisions sent in-reply-to mail

Please always send new revisions of series individually, not in-reply-to
any other email.

The easiest way of accomplishing this is to use b4 [0] (the recommended
way of sending patches to mm), otherwise format patches like this:

For a patch series, e.g. at v2:

$ git format-patch -v2 --cover-letter --thread ...

For an individual patch, e.g. at v2:

$ git format-patch -v2 HEAD~1

[0]: https://b4.docs.kernel.org/en/latest/contributor/send.html

If you wish to discuss this problem further, or you have questions about
how to resolve this issue, please feel free to respond to this email and
Lorenzo will reply once he has dug out from the pending patches received
from other developers.

thanks,

Lorenzo's patch email bot

[ Idea shamelessly stolen from greg-kh ]

--
Cheers, Lorenzo

On Sun, Sep 6, 2026 at 9:18 AM Bo Zhang <zhangbo0325@gmail.com> wrote:
>
> We have observed some cases where memory is allocated with GFP_NOIO, so
> we cannot reclaim any anon folios unless they are in swapcache. We can
> end up spending more than 150 ms looping in `shrink_folio_list()` scanning
> non-swapcache folios without reclaiming a single folio. This is pure
> overhead.
>
> This is particularly true on systems using zRAM, where swapcache is
> relatively rare. So let's check whether anon reclaim is allowed by
> GFP_IO and whether there is enough swapcache to make it worthwhile. If
> the swapcache is extremely low, we're essentially searching for a
> needle in a haystack, so let's avoid scanning anon in the first place.
>
> On Android this is triggered by dm-verity hash-block reads through
> dm-bufio, which use GFP_NOIO:
>
>   verity_verify_io -> verity_hash_for_block -> verity_verify_level
>     -> dm_bufio_read_with_ioprio -> new_read -> __bufio_new
>       -> alloc_buffer
>          gfp: GFP_NOIO | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN
>
> Such a reclaimer can land on a memcg with a large, unswapped anon LRU and
> a tiny file LRU (e.g. inactive_anon ~335 MB vs inactive_file ~4 MB, with
> negligible swapcache). shrink_lruvec() then keeps feeding that huge anon
> list into shrink_folio_list() - ~2400 shrink_folio_list() calls, ~93,000
> anon folios scanned - where every folio is kept because it needs IO. The
> 150+ ms above is one such single shrink_lruvec() pass (not accumulated
> across a reclaim cycle), and it reclaims nothing; the actual progress
> comes entirely from the file side.
>
> Aging anon alongside file does have some value for a later __GFP_IO
> reclaimer, so it is not strictly pure overhead. But that aging is only
> deferred, not lost: kswapd and other __GFP_IO reclaimers still walk and
> age anon. Spending ~168 ms aging memory that this context cannot reclaim
> is not a worthwhile trade-off in a latency-sensitive path.
>
> To stay conservative, this only skips anon when the swapcache is really
> tiny - below 1/64 of the anon LRU - i.e. when essentially no anon on the
> list can be reclaimed without IO. Whenever there is a meaningful amount of
> swapcached anon, the normal path is used and anon is scanned and aged as
> before.

I notice this only fixes the active/inactive LRU case. To address the
MGLRU case, it seems we may need a more fundamental change.

I'm fine with starting by fixing the active/inactive LRU case first.
However, could we mention in the changelog that this patch only
addresses the active/inactive LRU case, and that fixing the MGLRU case
is on the TODO list?

>
> Signed-off-by: Bo Zhang <zhangbo56@xiaomi.com>
> ---
> v1 -> v2:
>  - Use mem_cgroup_lruvec() instead of get_lruvec(), which returns the raw
>    node lruvec for a NULL memcg and would be misinterpreted by
>    lruvec_page_state()'s container_of() during global reclaim. This also
>    drops the get_lruvec() move. (reported by the sashiko bot, suggested
>    by Barry Song)
>  - Drop the SWAP_CLUSTER_MAX cap on the threshold; the check is purely
>    proportional now (swapcache below 1/64 of the anon LRU).
>  - Expand the changelog with the workload, the dm-verity/dm-bufio NOIO
>    stack, the ~168 ms single shrink_lruvec() breakdown, and the aging
>    trade-off discussed with Johannes Weiner.
>
>  mm/vmscan.c | 23 +++++++++++++++++++++--
>  1 file changed, 21 insertions(+), 2 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 245f68c75b28..e20ac2cb4dd5 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -362,6 +362,23 @@ static bool can_demote(int nid, struct scan_control *sc,
>         return !nodes_empty(allowed_mask);
>  }
>
> +static inline bool reclaimable_anon_is_low(struct mem_cgroup *memcg,
> +               int nid, struct scan_control *sc)
> +{
> +       struct lruvec *lruvec;
> +       unsigned long anon_pages, swapcache;
> +
> +       if (!sc || (sc->gfp_mask & __GFP_IO))
> +               return false;
> +
> +       lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
> +       anon_pages = lruvec_page_state(lruvec, NR_INACTIVE_ANON) +
> +                    lruvec_page_state(lruvec, NR_ACTIVE_ANON);
> +       swapcache = lruvec_page_state(lruvec, NR_SWAPCACHE);
> +
> +       return swapcache < (anon_pages >> 6);
> +}
> +
>  static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
>                                           int nid,
>                                           struct scan_control *sc)
> @@ -371,11 +388,13 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
>                  * For non-memcg reclaim, is there
>                  * space in any swap device?
>                  */
> -               if (get_nr_swap_pages() > 0)
> +               if (get_nr_swap_pages() > 0 &&
> +                   !reclaimable_anon_is_low(memcg, nid, sc))

Can we also update the comment above accordingly?

/*
 * For non-memcg reclaim, do we have space on any swap device?
 * For GFP_NOIO, do we also have sufficient swapcache anon folios
 * to reclaim?
 */

>                         return true;
>         } else {
>                 /* Is the memcg below its swap limit? */
> -               if (mem_cgroup_get_nr_swap_pages(memcg) > 0)
> +               if (mem_cgroup_get_nr_swap_pages(memcg) > 0 &&
> +                   !reclaimable_anon_is_low(memcg, nid, sc))

Do we also need to update the comment?

/*
 * Is the memcg above its swap limit, and does it have enough
 * swapcache anon folios to reclaim for GFP_NOIO?
 */

Best Regards
Barry

On Sun, Sep 6, 2026 at 12:56 PM Barry Song <baohua@kernel.org> wrote:
>
> On Sun, Sep 6, 2026 at 9:18 AM Bo Zhang <zhangbo0325@gmail.com> wrote:
> >
> > We have observed some cases where memory is allocated with GFP_NOIO, so
> > we cannot reclaim any anon folios unless they are in swapcache. We can
> > end up spending more than 150 ms looping in `shrink_folio_list()` scanning
> > non-swapcache folios without reclaiming a single folio. This is pure
> > overhead.
> >
> > This is particularly true on systems using zRAM, where swapcache is
> > relatively rare. So let's check whether anon reclaim is allowed by
> > GFP_IO and whether there is enough swapcache to make it worthwhile. If
> > the swapcache is extremely low, we're essentially searching for a
> > needle in a haystack, so let's avoid scanning anon in the first place.
> >
> > On Android this is triggered by dm-verity hash-block reads through
> > dm-bufio, which use GFP_NOIO:
> >
> >   verity_verify_io -> verity_hash_for_block -> verity_verify_level
> >     -> dm_bufio_read_with_ioprio -> new_read -> __bufio_new
> >       -> alloc_buffer
> >          gfp: GFP_NOIO | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN
> >
> > Such a reclaimer can land on a memcg with a large, unswapped anon LRU and
> > a tiny file LRU (e.g. inactive_anon ~335 MB vs inactive_file ~4 MB, with
> > negligible swapcache). shrink_lruvec() then keeps feeding that huge anon
> > list into shrink_folio_list() - ~2400 shrink_folio_list() calls, ~93,000
> > anon folios scanned - where every folio is kept because it needs IO. The
> > 150+ ms above is one such single shrink_lruvec() pass (not accumulated
> > across a reclaim cycle), and it reclaims nothing; the actual progress
> > comes entirely from the file side.
> >
> > Aging anon alongside file does have some value for a later __GFP_IO
> > reclaimer, so it is not strictly pure overhead. But that aging is only
> > deferred, not lost: kswapd and other __GFP_IO reclaimers still walk and
> > age anon. Spending ~168 ms aging memory that this context cannot reclaim
> > is not a worthwhile trade-off in a latency-sensitive path.
> >
> > To stay conservative, this only skips anon when the swapcache is really
> > tiny - below 1/64 of the anon LRU - i.e. when essentially no anon on the
> > list can be reclaimed without IO. Whenever there is a meaningful amount of
> > swapcached anon, the normal path is used and anon is scanned and aged as
> > before.
>
> I notice this only fixes the active/inactive LRU case. To address the
> MGLRU case, it seems we may need a more fundamental change.

It won't be too hard if we just calculate the type and number to scan
upfront, and I believe this is a similar issue due to the same root
cause of the OOM and swappiness issue of MGLRU, which I mentiones
before (see point 4, "force protection of the youngest two gens"):
https://lore.kernel.org/linux-mm/CAMgjq7BoekNjg-Ra3C8M7=8=75su38w=HD782T5E_cxyeCeH_g@mail.gmail.com/

Removing that force protection and calculate the number to scan
upfront, then we can also make use of can_reclaim_anon_pages, shift
all scan budget to file type. Aging won't be triggered at default
priority, resulting in zero overhead. We can then either offload aging
to a worker or defer it if aging isn't helpful for one reclaim cycle.

Thanks Barry.

On Sun, Sep 6, 2026 at 12:53 PM Barry Song <baohua@kernel.org> wrote:
>
> I notice this only fixes the active/inactive LRU case. To address the
> MGLRU case, it seems we may need a more fundamental change.
>
> I'm fine with starting by fixing the active/inactive LRU case first.
> However, could we mention in the changelog that this patch only
> addresses the active/inactive LRU case, and that fixing the MGLRU case
> is on the TODO list?

You're right - MGLRU decides anon vs file scanning in its own path
(get_type_to_scan()/isolate_folios()) and does not go through this
can_reclaim_anon_pages() check for normal reclaim, so it is not covered
here. I'll note in the changelog that this patch only addresses the
active/inactive LRU case and that the MGLRU case is left as a TODO.

> > -               if (get_nr_swap_pages() > 0)
> > +               if (get_nr_swap_pages() > 0 &&
> > +                   !reclaimable_anon_is_low(memcg, nid, sc))
>
> Can we also update the comment above accordingly?
>
> /*
>  * For non-memcg reclaim, do we have space on any swap device?
>  * For GFP_NOIO, do we also have sufficient swapcache anon folios
>  * to reclaim?
>  */

Will do in v3.

> > -               if (mem_cgroup_get_nr_swap_pages(memcg) > 0)
> > +               if (mem_cgroup_get_nr_swap_pages(memcg) > 0 &&
> > +                   !reclaimable_anon_is_low(memcg, nid, sc))
>
> Do we also need to update the comment?
>
> /*
>  * Is the memcg above its swap limit, and does it have enough
>  * swapcache anon folios to reclaim for GFP_NOIO?
>  */

Yes, updated both comments in v3.

Thanks,
Bo

On Sun,  6 Sep 2026 09:18:20 +0800 Bo Zhang <zhangbo0325@gmail.com> wrote:

> We have observed some cases where memory is allocated with GFP_NOIO, so
> we cannot reclaim any anon folios unless they are in swapcache. We can
> end up spending more than 150 ms looping in `shrink_folio_list()` scanning
> non-swapcache folios without reclaiming a single folio. This is pure
> overhead.
> 
> This is particularly true on systems using zRAM, where swapcache is
> relatively rare. So let's check whether anon reclaim is allowed by
> GFP_IO and whether there is enough swapcache to make it worthwhile. If
> the swapcache is extremely low, we're essentially searching for a
> needle in a haystack, so let's avoid scanning anon in the first place.
> 
> On Android this is triggered by dm-verity hash-block reads through
> dm-bufio, which use GFP_NOIO:
> 
>   verity_verify_io -> verity_hash_for_block -> verity_verify_level
>     -> dm_bufio_read_with_ioprio -> new_read -> __bufio_new
>       -> alloc_buffer
>          gfp: GFP_NOIO | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN
> 
> Such a reclaimer can land on a memcg with a large, unswapped anon LRU and
> a tiny file LRU (e.g. inactive_anon ~335 MB vs inactive_file ~4 MB, with
> negligible swapcache). shrink_lruvec() then keeps feeding that huge anon
> list into shrink_folio_list() - ~2400 shrink_folio_list() calls, ~93,000
> anon folios scanned - where every folio is kept because it needs IO. The
> 150+ ms above is one such single shrink_lruvec() pass (not accumulated
> across a reclaim cycle), and it reclaims nothing; the actual progress
> comes entirely from the file side.
> 
> Aging anon alongside file does have some value for a later __GFP_IO
> reclaimer, so it is not strictly pure overhead. But that aging is only
> deferred, not lost: kswapd and other __GFP_IO reclaimers still walk and
> age anon. Spending ~168 ms aging memory that this context cannot reclaim
> is not a worthwhile trade-off in a latency-sensitive path.

Thanks. That sounds like something we want to fix.

> To stay conservative, this only skips anon when the swapcache is really
> tiny - below 1/64 of the anon LRU - i.e. when essentially no anon on the
> list can be reclaimed without IO. Whenever there is a meaningful amount of
> swapcached anon, the normal path is used and anon is scanned and aged as
> before.

Argh.  The thing about magic numbers is that they're always suboptimal
for everyone.  But I understand that a full-on dynamic tuning setup is
a big project and hopefully not worthwhile.  And yet another /proc knob
would require quite some justification.

> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -362,6 +362,23 @@ static bool can_demote(int nid, struct scan_control *sc,
>  	return !nodes_empty(allowed_mask);
>  }
>  
> +static inline bool reclaimable_anon_is_low(struct mem_cgroup *memcg,
> +		int nid, struct scan_control *sc)
> +{
> +	struct lruvec *lruvec;
> +	unsigned long anon_pages, swapcache;
> +
> +	if (!sc || (sc->gfp_mask & __GFP_IO))
> +		return false;
> +
> +	lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid));
> +	anon_pages = lruvec_page_state(lruvec, NR_INACTIVE_ANON) +
> +		     lruvec_page_state(lruvec, NR_ACTIVE_ANON);
> +	swapcache = lruvec_page_state(lruvec, NR_SWAPCACHE);
> +
> +	return swapcache < (anon_pages >> 6);
> +}

I think this function deserves a comment.  One which explains why isn't
doing what it does rather than what it does.  That comment would
highlight the heuristic and explain the thinking behind it.

Also, AI review asks "does reclaimable_anon_is_low() incorrectly use
root memcg statistics instead of node-wide statistics during global
memory reclaim?".

	https://sashiko.dev/#/patchset/20260906011820.382381-1-zhangbo56@xiaomi.com

Thanks a lot for the review, Andrew - much appreciated.

On Sat, 5 Sep 2026 19:46:02 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:
> > To stay conservative, this only skips anon when the swapcache is really
> > tiny - below 1/64 of the anon LRU - i.e. when essentially no anon on the
> > list can be reclaimed without IO. Whenever there is a meaningful amount of
> > swapcached anon, the normal path is used and anon is scanned and aged as
> > before.
>
> Argh.  The thing about magic numbers is that they're always suboptimal
> for everyone.  But I understand that a full-on dynamic tuning setup is
> a big project and hopefully not worthwhile.  And yet another /proc knob
> would require quite some justification.

Agreed - a full dynamic tuning setup would be complex, and I'd rather not
add a knob for this either. For now this uses a conservative threshold to
catch only the case where anon is effectively unreclaimable; the reasoning
is explained in the function comment (below).

> I think this function deserves a comment.  One which explains why isn't
> doing what it does rather than what it does.  That comment would
> highlight the heuristic and explain the thinking behind it.

Done in v3. The comment now explains the "why": a !__GFP_IO reclaimer can
only reclaim anon already in the swapcache, so when swapcache is far below
the anon LRU, scanning anon reclaims nothing and only burns CPU - and the
aging it would have done is merely deferred to later __GFP_IO reclaimers.
It also notes that 1/64 is a conservative "negligible swapcache" threshold.

> Also, AI review asks "does reclaimable_anon_is_low() incorrectly use
> root memcg statistics instead of node-wide statistics during global
> memory reclaim?".

Good catch - it did, and I've fixed it in v3.

For memcg reclaim, can_reclaim_anon_pages() is called per-memcg (memcg is
the concrete cgroup being scanned), so using its lruvec stats is correct.
But for global reclaim it is also called with memcg == NULL - e.g. from
set_initial_priority() - and there mem_cgroup_lruvec(NULL) resolves to the
root memcg, whose stats exclude the child cgroups where most anon lives.
That could make the check fire on the root's tiny stats even when the node
has plenty of anon and swapcache elsewhere.

v3 splits the two cases: use the memcg's lruvec stats when memcg is set,
and node_page_state() when memcg == NULL, matching the node-wide view its
global callers already use for the file side.

I'll send v3 with these changes.

Thanks,
Bo