We have observed some cases where memory is allocated with GFP_NOIO, so
we cannot reclaim any anon folios unless they are in swapcache. We can
end up spending more than 150 ms looping in `shrink_folio_list()` scanning
non-swapcache folios without reclaiming a single folio. This is pure
overhead.

This is particularly true on systems using zRAM, where swapcache is
relatively rare. So let's check whether anon reclaim is allowed by
GFP_IO and whether there is enough swapcache to make it worthwhile. If
the swapcache is extremely low, we're essentially searching for a
needle in a haystack, so let's avoid scanning anon in the first place.

On Android this is triggered by dm-verity hash-block reads through
dm-bufio, which legitimately use GFP_NOIO because they run underneath the
IO path:

  verity_verify_io -> verity_hash_for_block -> verity_verify_level
    -> dm_bufio_read_with_ioprio -> new_read -> __bufio_new
      -> alloc_buffer
         gfp: GFP_NOIO | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN

Such a reclaimer can land on a memcg with a large, unswapped anon LRU and
a tiny file LRU (e.g. inactive_anon ~335 MB vs inactive_file ~4 MB, with
negligible swapcache). shrink_lruvec() then keeps feeding that huge anon
list into shrink_folio_list() - ~2400 shrink_folio_list() calls, ~93,000
anon folios scanned - where every folio is kept because it needs IO. The
150+ ms above is one such single shrink_lruvec() pass (not accumulated
across a reclaim cycle), and it reclaims nothing; the actual progress
comes entirely from the file side.

Aging anon alongside file does have some value for a later __GFP_IO
reclaimer, so it is not strictly pure overhead. But that aging is only
deferred, not lost: kswapd and other __GFP_IO reclaimers still walk and
age anon. Spending ~168 ms aging memory that this context cannot reclaim
is not a worthwhile trade-off in a latency-sensitive path.

To stay conservative, this only skips anon when the swapcache is really
tiny - below 1/64 of the anon LRU - i.e. when essentially no anon on the
list can be reclaimed without IO. Whenever there is a meaningful amount of
swapcached anon, the normal path is used and anon is scanned and aged as
before.

Note this only addresses the traditional active/inactive LRU. MGLRU
selects anon vs file scanning in its own path and is not covered here;
fixing the MGLRU case is left as a TODO.

Signed-off-by: Bo Zhang <zhangbo56@xiaomi.com>
---
v2 -> v3:
 - Fix stats source in reclaimable_anon_is_low(): for global reclaim
   (memcg == NULL, e.g. from set_initial_priority()) use node_page_state()
   instead of mem_cgroup_lruvec(NULL), which resolves to the root memcg and
   excludes the child cgroups where anon actually lives. (reported by the
   sashiko bot / AI review, raised by Andrew Morton)
 - Add a comment explaining the heuristic and its rationale. (Andrew Morton)
 - Update the comments above the can_reclaim_anon_pages() checks. (Barry Song)
 - Note in the changelog that only the traditional LRU is addressed; MGLRU
   is left as a TODO. (Barry Song)

v1 -> v2:
 - Use mem_cgroup_lruvec() instead of get_lruvec(), which returns the raw
   node lruvec for a NULL memcg and would be misinterpreted by
   lruvec_page_state()'s container_of() during global reclaim. This also
   drops the get_lruvec() move. (sashiko bot, Barry Song)
 - Drop the SWAP_CLUSTER_MAX cap on the threshold; the check is purely
   proportional now (swapcache below 1/64 of the anon LRU). (Barry Song)
 - Expand the changelog with the workload, the dm-verity/dm-bufio NOIO
   stack, the single shrink_lruvec() breakdown, and the aging trade-off.
   (Johannes Weiner)

 mm/vmscan.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 47 insertions(+), 5 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 245f68c75b28..e5c07490f5b3 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -362,20 +362,62 @@ static bool can_demote(int nid, struct scan_control *sc,
 	return !nodes_empty(allowed_mask);
 }
 
+static inline bool reclaimable_anon_is_low(struct mem_cgroup *memcg,
+		int nid, struct scan_control *sc)
+{
+	pg_data_t *pgdat = NODE_DATA(nid);
+	unsigned long anon_pages, swapcache;
+
+	/*
+	 * A !__GFP_IO reclaimer can only reclaim anon that is already in the
+	 * swapcache (adding anon to the swapcache needs IO). When swapcache is
+	 * far below the anon LRU, scanning anon reclaims nothing and only burns
+	 * CPU; the aging it would do is merely deferred to later __GFP_IO
+	 * reclaimers. The 1/64 threshold keeps this to the case where anon is
+	 * effectively unreclaimable.
+	 *
+	 * Use the memcg's lruvec for memcg reclaim; for global reclaim
+	 * (memcg == NULL) use node-wide stats. mem_cgroup_lruvec(NULL) would
+	 * only see the root memcg, not the child cgroups where anon lives.
+	 */
+	if (!sc || (sc->gfp_mask & __GFP_IO))
+		return false;
+
+	if (memcg) {
+		struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
+
+		anon_pages = lruvec_page_state(lruvec, NR_INACTIVE_ANON) +
+			     lruvec_page_state(lruvec, NR_ACTIVE_ANON);
+		swapcache = lruvec_page_state(lruvec, NR_SWAPCACHE);
+	} else {
+		anon_pages = node_page_state(pgdat, NR_INACTIVE_ANON) +
+			     node_page_state(pgdat, NR_ACTIVE_ANON);
+		swapcache = node_page_state(pgdat, NR_SWAPCACHE);
+	}
+
+	return swapcache < (anon_pages >> 6);
+}
+
 static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
 					  int nid,
 					  struct scan_control *sc)
 {
 	if (memcg == NULL) {
 		/*
-		 * For non-memcg reclaim, is there
-		 * space in any swap device?
+		 * For non-memcg reclaim, is there space in any swap device?
+		 * And under GFP_NOIO, is there enough swapcached anon to make
+		 * scanning anon worthwhile?
 		 */
-		if (get_nr_swap_pages() > 0)
+		if (get_nr_swap_pages() > 0 &&
+		    !reclaimable_anon_is_low(memcg, nid, sc))
 			return true;
 	} else {
-		/* Is the memcg below its swap limit? */
-		if (mem_cgroup_get_nr_swap_pages(memcg) > 0)
+		/*
+		 * Is the memcg below its swap limit, and under GFP_NOIO does
+		 * it have enough swapcached anon to make scanning worthwhile?
+		 */
+		if (mem_cgroup_get_nr_swap_pages(memcg) > 0 &&
+		    !reclaimable_anon_is_low(memcg, nid, sc))
 			return true;
 	}
 
-- 
2.34.1

On Mon, Sep 7, 2026 at 8:28 PM Bo Zhang <zhangbo0325@gmail.com> wrote:
>
> We have observed some cases where memory is allocated with GFP_NOIO, so
> we cannot reclaim any anon folios unless they are in swapcache. We can
> end up spending more than 150 ms looping in `shrink_folio_list()` scanning
> non-swapcache folios without reclaiming a single folio. This is pure
> overhead.
>
> This is particularly true on systems using zRAM, where swapcache is
> relatively rare. So let's check whether anon reclaim is allowed by
> GFP_IO and whether there is enough swapcache to make it worthwhile. If
> the swapcache is extremely low, we're essentially searching for a
> needle in a haystack, so let's avoid scanning anon in the first place.
>
> On Android this is triggered by dm-verity hash-block reads through
> dm-bufio, which legitimately use GFP_NOIO because they run underneath the
> IO path:
>
>   verity_verify_io -> verity_hash_for_block -> verity_verify_level
>     -> dm_bufio_read_with_ioprio -> new_read -> __bufio_new
>       -> alloc_buffer
>          gfp: GFP_NOIO | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN
>
> Such a reclaimer can land on a memcg with a large, unswapped anon LRU and
> a tiny file LRU (e.g. inactive_anon ~335 MB vs inactive_file ~4 MB, with
> negligible swapcache). shrink_lruvec() then keeps feeding that huge anon
> list into shrink_folio_list() - ~2400 shrink_folio_list() calls, ~93,000
> anon folios scanned - where every folio is kept because it needs IO. The
> 150+ ms above is one such single shrink_lruvec() pass (not accumulated
> across a reclaim cycle), and it reclaims nothing; the actual progress
> comes entirely from the file side.
>
> Aging anon alongside file does have some value for a later __GFP_IO
> reclaimer, so it is not strictly pure overhead. But that aging is only
> deferred, not lost: kswapd and other __GFP_IO reclaimers still walk and
> age anon. Spending ~168 ms aging memory that this context cannot reclaim
> is not a worthwhile trade-off in a latency-sensitive path.
>
> To stay conservative, this only skips anon when the swapcache is really
> tiny - below 1/64 of the anon LRU - i.e. when essentially no anon on the
> list can be reclaimed without IO. Whenever there is a meaningful amount of
> swapcached anon, the normal path is used and anon is scanned and aged as
> before.
>
> Note this only addresses the traditional active/inactive LRU. MGLRU
> selects anon vs file scanning in its own path and is not covered here;
> fixing the MGLRU case is left as a TODO.
>
> Signed-off-by: Bo Zhang <zhangbo56@xiaomi.com>
> ---
> v2 -> v3:
>  - Fix stats source in reclaimable_anon_is_low(): for global reclaim
>    (memcg == NULL, e.g. from set_initial_priority()) use node_page_state()
>    instead of mem_cgroup_lruvec(NULL), which resolves to the root memcg and
>    excludes the child cgroups where anon actually lives. (reported by the
>    sashiko bot / AI review, raised by Andrew Morton)
>  - Add a comment explaining the heuristic and its rationale. (Andrew Morton)
>  - Update the comments above the can_reclaim_anon_pages() checks. (Barry Song)
>  - Note in the changelog that only the traditional LRU is addressed; MGLRU
>    is left as a TODO. (Barry Song)
>
> v1 -> v2:
>  - Use mem_cgroup_lruvec() instead of get_lruvec(), which returns the raw
>    node lruvec for a NULL memcg and would be misinterpreted by
>    lruvec_page_state()'s container_of() during global reclaim. This also
>    drops the get_lruvec() move. (sashiko bot, Barry Song)
>  - Drop the SWAP_CLUSTER_MAX cap on the threshold; the check is purely
>    proportional now (swapcache below 1/64 of the anon LRU). (Barry Song)
>  - Expand the changelog with the workload, the dm-verity/dm-bufio NOIO
>    stack, the single shrink_lruvec() breakdown, and the aging trade-off.
>    (Johannes Weiner)
>
>  mm/vmscan.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 47 insertions(+), 5 deletions(-)
>
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 245f68c75b28..e5c07490f5b3 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -362,20 +362,62 @@ static bool can_demote(int nid, struct scan_control *sc,
>         return !nodes_empty(allowed_mask);
>  }
>
> +static inline bool reclaimable_anon_is_low(struct mem_cgroup *memcg,
> +               int nid, struct scan_control *sc)
> +{
> +       pg_data_t *pgdat = NODE_DATA(nid);
> +       unsigned long anon_pages, swapcache;
> +
> +       /*
> +        * A !__GFP_IO reclaimer can only reclaim anon that is already in the

I'd rather call it a GFP_NOIO reclaimer than a !__GFP_IO reclaimer.
But they're really the same thing. :-)

> +        * swapcache (adding anon to the swapcache needs IO). When swapcache is
> +        * far below the anon LRU, scanning anon reclaims nothing and only burns
> +        * CPU; the aging it would do is merely deferred to later __GFP_IO
> +        * reclaimers. The 1/64 threshold keeps this to the case where anon is
> +        * effectively unreclaimable.

Let's keep it a little shorter. Maybe we can drop
"the aging it would do is merely deferred to later
__GFP_IO reclaimers."

> +        *
> +        * Use the memcg's lruvec for memcg reclaim; for global reclaim
> +        * (memcg == NULL) use node-wide stats. mem_cgroup_lruvec(NULL) would
> +        * only see the root memcg, not the child cgroups where anon lives.

This is already self-documented, so do we need this?

> +        */
> +       if (!sc || (sc->gfp_mask & __GFP_IO))
> +               return false;
> +

Sashiko is concerned that this might negatively affect MGLRU,
so maybe:
/*
 * FIXME: MGLRU doesn't fully respect can_reclaim_anon_pages()
 * for the scanning type
 */
if (!lru_gen_enabled())
    return false;

We’ll need to address MGLRU with a much larger patchset as commented
by Kairui[1].

> +       if (memcg) {
> +               struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
> +
> +               anon_pages = lruvec_page_state(lruvec, NR_INACTIVE_ANON) +
> +                            lruvec_page_state(lruvec, NR_ACTIVE_ANON);
> +               swapcache = lruvec_page_state(lruvec, NR_SWAPCACHE);
> +       } else {
> +               anon_pages = node_page_state(pgdat, NR_INACTIVE_ANON) +
> +                            node_page_state(pgdat, NR_ACTIVE_ANON);
> +               swapcache = node_page_state(pgdat, NR_SWAPCACHE);
> +       }

Sashiko is concerned that `NR_SWAPCACHE` might be undefined for
!CONFIG_SWAP. That seems reasonable to me. I think we can do:

#ifdef CONFIG_SWAP
static inline bool reclaimable_anon_is_low(struct mem_cgroup *memcg,
               int nid, struct scan_control *sc)
{
    ...
}
#else
static inline bool reclaimable_anon_is_low(struct mem_cgroup *memcg,
               int nid, struct scan_control *sc)
{
    return true;
}
#endif

Actually, returning either true or false makes no difference for
`!CONFIG_SWAP`, as `get_nr_swap_pages()` will definitely return zero.
However, returning true seems semantically more correct.

[1] https://lore.kernel.org/linux-mm/CAMgjq7B==MqHXa6H=arrQ9a0bUXWpXG_a5EBxZGjLjYUuEf1gQ@mail.gmail.com/

Best Regards
Barry