From nobody Sat Sep 26 11:01:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40F5D42BEB0; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; cv=none; b=bgbKQDUwdg1aGz8gXhLxbkmvqUM3xAayhj1UMtvPkosoVytmDayGMWJ7yMdl55Z6/NPFPcuS3d+LL++wv41eniPh9Mh+ShAYnneRwzzkWO3xC6O2KHVEzxJs+ojwUc1QLp7Csi9zD1NnUuhODfIXImdKh8Z+MSg1MtPJtsOFWec= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; c=relaxed/simple; bh=vtnU28IgD/9fgw3eMgLAM3NVwlFuTiAvZXPJld3OpeU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=d0Rio9bXSNgsD2fU1Isk3qtq+xufzMFnBxUReIWsHknx860OsKWXnygNmA0z3PbRRfnTlWhMyAagW1VDoztI4T8HZOwGL9nWq1n57rpWh4heJOVCc9pbHw3dAGp//Nx5ZNa1siDT522uvLce/g9PKFZKQvdLhIOa9ve1Kbku9Qg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gByp5JTp; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gByp5JTp" Received: by smtp.kernel.org (Postfix) with ESMTPS id CD5F4C2BCF7; Wed, 2 Sep 2026 09:50:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1788342659; bh=vtnU28IgD/9fgw3eMgLAM3NVwlFuTiAvZXPJld3OpeU=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=gByp5JTpDlVjfW5kz0pfLwTAfCAariuUpiKFZtmgw8E9BcB+R1A1jNKcU6+4HExwC B77g6ObRdX5yrtpzLB1+ZtODjNoEHqpGsBPhRUBxBzx2UkOFWO4mPlER6TKcZvk1xa y5OiBKOiayTOJ9fxnWVGGTOXbKYq5ae/BvZqIa3eYqjARogmmFjgagsS4RktK0BgCN lrVbz/VpSqNy8rycuVlV6d26TDdDHlJqvGvf16a96ImNRzXPyI3ESutqL5/tnujd3D ct0ahjylmIr3RuwZlRAXIH9HkClgmnNjItWy8xSnhAyz7y/I4e+9psM8itACE9J4fm zXezbIcNEpwpQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5A10C624D0; Wed, 2 Sep 2026 09:50:59 +0000 (UTC) From: Kairui Song via B4 Relay Date: Wed, 02 Sep 2026 17:50:54 +0800 Subject: [PATCH v5 1/6] mm/memcontrol: move the lru_zone_size sanity check to the reader side Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260902-mglru-flags-cleanup-v5-1-9db761d779ef@tencent.com> References: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> In-Reply-To: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , Shakeel Butt , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Chris Li , Baolin Wang , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Ridong Chen , Lian Wang , Yu Zhao , Zi Yan , Qi Zheng , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Kairui Song , Kairui Song X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1788342657; l=4138; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=QI6bvhReGHb4fOvBljBjwiJV5Y8Bw1C5l29qeBm+tPA=; b=zq0mgu+IKjbAnDIaSkFgx6KtQr5EysUe2PZDWpe8eFGP89GdZJw9RhoxJ5PBrmbNBR5CRJC7N NNim/6E9hoaCYsBGaC7VgOuRQeEsD+geKaWaOzKjSEnNcKCREhoD5eE X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Instead of using an unsigned long and checking the counter value at the updater side, turn the counter into a signed long and check at the reader side. This reduces overhead and simplifies the code. commit ca707239e8a7 ("mm: update_lru_size warn and reset bad lru_size") added a sanity check for memcg counter underflow: lru_zone_size is unsigned, so an underflow wraps it around and returns an enormously large number, then the memcg shrinker loops almost forever as the calculated number of folios to shrink is huge. It also checked if a zero value matches the empty LRU list, so the positive and negative deltas had to be handled separately. However that emptiness check was already removed by commit b4536f0c829c ("mm, memcg: fix the active list aging for lowmem requests when memcg is enabled"), so handling the deltas separately is no longer needed. The remaining update-side check is costly and cannot really catch the leak it is after anyway. It runs on every LRU folio, and if a folio was removed without updating the counter while other folios remain on the LRU, the WARN only triggers much later, from a likely innocent callsite. While readers are much rarer than writers, only the reclaim and reparenting paths read it, once per batch. Checking at the reader side instead leaves the update path a plain addition, and puts the warning where the value is actually consumed. Note this changes the behavior on underflow: the correction is removed and a negative value is kept. A massive leak of the LRU size counter would indicate that something else has gone very wrong, and one should fix that leaking site instead. Besides, the original behavior might cause false positives, or make things worse if the accounting happens after the actual insertion: the value is not leaked, just delayed, so force-fixing it would cause a bigger problem. The warning now only kicks in when a consumer actually uses it, in which case the reader gets zero. Reviewed-by: Ridong Chen Reviewed-by: Barry Song Signed-off-by: Kairui Song Acked-by: Qi Zheng Acked-by: Shakeel Butt Reviewed-by: Baolin Wang --- include/linux/memcontrol.h | 9 +++++++-- mm/memcontrol.c | 18 +----------------- 2 files changed, 8 insertions(+), 19 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 7d1c0ce189a8..86780ef65eaf 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -113,7 +113,7 @@ struct mem_cgroup_per_node { /* Fields which get updated often at the end. */ struct lruvec lruvec; CACHELINE_PADDING(_pad2_); - unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; + long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; struct mem_cgroup_reclaim_iter iter; =20 /* @@ -902,10 +902,15 @@ static inline unsigned long mem_cgroup_get_zone_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone_idx) { + long val; struct mem_cgroup_per_node *mz; =20 mz =3D container_of(lruvec, struct mem_cgroup_per_node, lruvec); - return READ_ONCE(mz->lru_zone_size[zone_idx][lru]); + val =3D READ_ONCE(mz->lru_zone_size[zone_idx][lru]); + if (WARN_ON_ONCE(val < 0)) + return 0; + + return val; } =20 void __mem_cgroup_handle_over_high(gfp_t gfp_mask); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 856a7d07586c..0a65ab8df27a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1529,28 +1529,12 @@ void mem_cgroup_update_lru_size(struct lruvec *lruv= ec, enum lru_list lru, int zid, long nr_pages) { struct mem_cgroup_per_node *mz; - unsigned long *lru_size; - long size; =20 if (mem_cgroup_disabled()) return; =20 mz =3D container_of(lruvec, struct mem_cgroup_per_node, lruvec); - lru_size =3D &mz->lru_zone_size[zid][lru]; - - if (nr_pages < 0) - *lru_size +=3D nr_pages; - - size =3D *lru_size; - if (WARN_ONCE(size < 0, - "%s(%p, %d, %ld): lru_size %ld\n", - __func__, lruvec, lru, nr_pages, size)) { - VM_BUG_ON(1); - *lru_size =3D 0; - } - - if (nr_pages > 0) - *lru_size +=3D nr_pages; + mz->lru_zone_size[zid][lru] +=3D nr_pages; } =20 /** --=20 2.55.0 From nobody Sat Sep 26 11:01:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 40D9242048C; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; cv=none; b=oEtbbrRaFM+FosdtJhrLBzpb+XfNxiMxQCKqXLyTERwW/Qhz89GgMOLRZxHsurKM8SGXyYu/uhTpt4eQcr4I39wzuNGxmQkUR8zQEjH5lTenJlDrc1QeJ4NX6+wgvsS/dDySHVsRUPYh0I8eh9Cy9klf/MvTIJ5zvLNWhBG02Bs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; c=relaxed/simple; bh=jG5UqbWsyVbezkV41xqUdAasSe0ofR2mWRBqrf7A6ds=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=DG8+NJLA7xYjCwSBRzqIHh4VYyfVoKfQVKAn/2hDe47714q4n+GRNmaoehbnxBNfIGFEkjfmJ6+cy5xyUDXRV9vUCMEn3Q8V89SDi6P0itii3+J2NGngbIgnEazfjL5fY89DdJMvMdxaHIdjPucDNllE7Dx9RtRBv8epCN6QYBs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=n8Ny8sNa; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="n8Ny8sNa" Received: by smtp.kernel.org (Postfix) with ESMTPS id DF483C2BCFF; Wed, 2 Sep 2026 09:50:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1788342660; bh=jG5UqbWsyVbezkV41xqUdAasSe0ofR2mWRBqrf7A6ds=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=n8Ny8sNadRoXZc1wZGhOxdbronaGepGQlLo1lE03Ymq0POBsCcq18MZ8UMFTuRjoh nHd8AQMUrSFPqsBHIEfg88ph2KM+e1L5Vaze8EK7Mdo7QsIqlxOm5+Zmo86xDGe9EH 7azDIaN1BYKJSb+dmH9lNibWguBvmF56DkksY7yE6MXRkNlJQNlN3Y7WscggY+GITH IOM/OrK2cvwY2sc2SKjnnV9j8VY3+r4IVL3X3jGweHIpAVvrvQ28zXM6Jw7FiAHS9o ljx0MOPV/Ay34uLpb3GoP9CUQ9I9u8fh3SoLW9JPMYhIxRhT455j1wnyqAnrifoGg+ SOxtHgnuzhZ6g== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C939EC624D5; Wed, 2 Sep 2026 09:50:59 +0000 (UTC) From: Kairui Song via B4 Relay Date: Wed, 02 Sep 2026 17:50:55 +0800 Subject: [PATCH v5 2/6] mm/mglru: introduce helpers for manipulating gen and refs flags Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260902-mglru-flags-cleanup-v5-2-9db761d779ef@tencent.com> References: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> In-Reply-To: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , Shakeel Butt , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Chris Li , Baolin Wang , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Ridong Chen , Lian Wang , Yu Zhao , Zi Yan , Qi Zheng , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Kairui Song , Kairui Song X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1788342657; l=12787; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=zxYuxsra7NFMR4nkZh6H9MzBo2geXOCAuVHC5Rnh0Mc=; b=mXppRTYUV1N75361XEkMMew93aY2VH4XNSoJQWgx3GqkmrehcYsOhUrwBEYGUx8feLKOvANGK 3ZcaODNft2zDzbrWYv30B9Kxi+a5nWf8DKduc/+8gkwiPLhrZUTrI8r X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Instead of doing bit ops on folio->flags.f, introduce helpers for adjusting a folio's refs and generation info, making the code easier to debug and understand. No functional change is intended: some combined atomic operations are split into two, which only creates harmless transient states. There is no measurable performance impact, and some paths even look slightly better in the generated assembly. Acked-by: Qi Zheng Reviewed-by: Baolin Wang Reviewed-by: Barry Song Signed-off-by: Kairui Song --- include/linux/mm_inline.h | 84 ++++++++++++++++++++++++++++++++++++++++++-= ---- include/linux/mmzone.h | 1 + mm/folio.c | 19 ++++++----- mm/vmscan.c | 61 +++++++++++++++++++--------------- 4 files changed, 122 insertions(+), 43 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 621c8653d8f7..3f4bd5b02b54 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -142,10 +142,66 @@ static inline int lru_tier_from_refs(int refs, bool w= orkingset) return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs); } =20 -static inline int folio_lru_refs(const struct folio *folio) +/** + * lru_set_gen_flags - Set the LRU generation number to specified folio fl= ags. + * @flags: pointer to the folio flags + * @gen: generation number, between 0 and (MAX_NR_GENS - 1), inclusive. + */ +static inline void lru_set_gen_flags(unsigned long *flags, int gen) +{ + BUILD_BUG_ON(LRU_GEN_MASK & LRU_REFS_MASK); + VM_WARN_ON_ONCE(gen >=3D MAX_NR_GENS || gen < 0); + /* Store gen offset by 1, zero means the folio is off-list. */ + *flags &=3D ~LRU_GEN_MASK; + *flags |=3D (gen + 1UL) << LRU_GEN_PGOFF; +} + +/** + * lru_get_gen_flags - Return the LRU generation number from folio flags. + * @flags: folio flags + * + * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns + * -1 if the flags indicate the folio is off the list (e.g., isolated). + */ +static inline int lru_get_gen_flags(unsigned long flags) { - unsigned long flags =3D READ_ONCE(folio->flags.f); + int gen =3D ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; =20 + /* Exclude the legal -1 from the unsigned MAX_NR_GENS comparison */ + VM_WARN_ON_ONCE(gen !=3D -1 && gen >=3D MAX_NR_GENS); + return gen; +} + +/** + * lru_set_refs_flags - Set the LRU referenced count to folio flags. + * @flags: pointer to the folio flags + * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, in= clusive. + * + * For MGLRU, PG_referenced holds the first ref, and the extra bits hold t= he + * remaining refs. For classical LRU the extra bits are not used, so it can + * also be seen as the refs count never exceeds 1. In both cases, refs =3D= =3D 1 + * means PG_referenced is set and the extra bits are zero, and refs =3D=3D= 0 means + * PG_referenced and the extra bits are all unset. + */ +static inline void lru_set_refs_flags(unsigned long *flags, unsigned int r= efs) +{ + VM_WARN_ON_ONCE(refs > LRU_REFS_MAX); + BUILD_BUG_ON(LRU_REFS_MAX !=3D (LRU_REFS_MASK >> LRU_REFS_PGOFF) + 1); + + *flags &=3D ~LRU_REFS_FLAGS; + if (!refs) + return; + *flags |=3D (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF)); +} + +/** + * lru_get_refs_flags - Return LRU referenced / access count from folio fl= ags. + * @flags: folio flags + * + * Reads the LRU referenced count set by lru_set_refs_flags(). + */ +static inline int lru_get_refs_flags(unsigned long flags) +{ if (!(flags & BIT(PG_referenced))) return 0; /* @@ -155,11 +211,24 @@ static inline int folio_lru_refs(const struct folio *= folio) return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1; } =20 -static inline int folio_lru_gen(const struct folio *folio) +static inline int folio_lru_refs(const struct folio *folio) { - unsigned long flags =3D READ_ONCE(folio->flags.f); + return lru_get_refs_flags(READ_ONCE(*const_folio_flags(folio, 0))); +} + +static inline void folio_set_lru_refs(struct folio *folio, unsigned int re= fs) +{ + unsigned long new_flags, old_flags =3D READ_ONCE(*folio_flags(folio, 0)); + + do { + new_flags =3D old_flags; + lru_set_refs_flags(&new_flags, refs); + } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags)); +} =20 - return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; +static inline int folio_lru_gen(const struct folio *folio) +{ + return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0))); } =20 static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen) @@ -270,7 +339,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lru= vec, struct folio *folio, gen =3D lru_gen_from_seq(seq); flags =3D (gen + 1UL) << LRU_GEN_PGOFF; /* see the comment on MIN_NR_GENS about PG_active */ - set_mask_bits(&folio->flags.f, LRU_GEN_MASK | BIT(PG_active), flags); + set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags= ); =20 lru_gen_update_size(lruvec, folio, -1, gen); /* for folio_rotate_reclaimable() */ @@ -295,7 +364,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lru= vec, struct folio *folio, =20 /* for folio_migrate_flags() */ flags =3D !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) = : 0; - flags =3D set_mask_bits(&folio->flags.f, LRU_GEN_MASK, flags); + flags =3D set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags); gen =3D ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; =20 lru_gen_update_size(lruvec, folio, gen, -1); @@ -339,7 +408,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lru= vec, struct folio *folio, =20 static inline void folio_migrate_refs(struct folio *new, const struct foli= o *old) { - } #endif /* CONFIG_LRU_GEN */ =20 diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 94f9c3ff5416..c9ecf370cd9f 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -497,6 +497,7 @@ enum lruvec_flags { =20 #define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) #define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) +#define LRU_REFS_MAX BIT(LRU_REFS_WIDTH) =20 /* * For folios accessed multiple times through file descriptors, diff --git a/mm/folio.c b/mm/folio.c index c02dcea9c03c..fb874fe492b2 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -353,26 +353,28 @@ static void __lru_cache_activate_folio(struct folio *= folio) =20 static void lru_gen_inc_refs(struct folio *folio) { - unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags.f); + unsigned long new_flags, old_flags =3D READ_ONCE(*folio_flags(folio, 0)); + int refs; =20 if (folio_test_unevictable(folio)) return; =20 /* see the comment on LRU_REFS_FLAGS */ - if (!folio_test_referenced(folio)) { - set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced)); + if (!folio_lru_refs(folio)) { + folio_set_lru_refs(folio, 1); return; } =20 do { - if ((old_flags & LRU_REFS_MASK) =3D=3D LRU_REFS_MASK) { + new_flags =3D old_flags; + refs =3D lru_get_refs_flags(old_flags); + if (refs =3D=3D LRU_REFS_MAX) { if (!folio_test_workingset(folio)) folio_set_workingset(folio); return; } - - new_flags =3D old_flags + BIT(LRU_REFS_PGOFF); - } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); + lru_set_refs_flags(&new_flags, refs + 1); + } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags)); } =20 static bool lru_gen_clear_refs(struct folio *folio) @@ -384,7 +386,8 @@ static bool lru_gen_clear_refs(struct folio *folio) if (gen < 0) return true; =20 - set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS | BIT(PG_workingset), 0); + folio_set_lru_refs(folio, 0); + folio_clear_workingset(folio); =20 rcu_read_lock(); seq =3D READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]); diff --git a/mm/vmscan.c b/mm/vmscan.c index fdd13299a04a..24d09711e820 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -843,19 +843,22 @@ static bool lru_gen_set_refs(struct folio *folio, con= st vma_flags_t *vma_flags) if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) { /* Activate file-backed executable folios after first usage. */ if (is_exec_file_folio(folio, vma_flags)) { - set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset)); + folio_set_workingset(folio); + folio_set_lru_refs(folio, 0); return true; } =20 - set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced)); + folio_set_lru_refs(folio, 1); return false; } =20 /* Promote on second access */ - if (folio_lru_refs(folio) > 1) - set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_workingset)); - else + if (folio_lru_refs(folio) > 1) { + folio_set_workingset(folio); + folio_set_lru_refs(folio, 0); + } else { folio_mark_accessed(folio); + } return true; } #else @@ -3266,11 +3269,10 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, = struct ctrl_pos *pv) *************************************************************************= *****/ =20 /* promote pages accessed through page tables */ -static int folio_update_gen(struct folio *folio, int gen, const vma_flags_= t *vma_flags) +static int folio_update_gen(struct folio *folio, int new_gen, const vma_fl= ags_t *vma_flags) { - unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags.f); - - VM_WARN_ON_ONCE(gen >=3D MAX_NR_GENS); + unsigned long new_flags, old_flags =3D READ_ONCE(*folio_flags(folio, 0)); + int old_gen; =20 /* * See the comment on LRU_REFS_FLAGS, and activate file-backed @@ -3279,20 +3281,24 @@ static int folio_update_gen(struct folio *folio, in= t gen, const vma_flags_t *vma */ if (!folio_test_referenced(folio) && !folio_test_workingset(folio) && !is_exec_file_folio(folio, vma_flags)) { - set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced)); + folio_set_lru_refs(folio, 1); return -1; } =20 do { + old_gen =3D lru_get_gen_flags(old_flags); + new_flags =3D old_flags; + /* lru_gen_del_folio() has isolated this page? */ - if (!(old_flags & LRU_GEN_MASK)) - return -1; + if (old_gen < 0) + break; =20 - new_flags =3D old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS); - new_flags |=3D ((gen + 1UL) << LRU_GEN_PGOFF) | BIT(PG_workingset); - } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); + lru_set_gen_flags(&new_flags, new_gen); + lru_set_refs_flags(&new_flags, 0); + new_flags |=3D BIT(PG_workingset); + } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags)); =20 - return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; + return old_gen; } =20 /* protect pages accessed multiple times through file descriptors */ @@ -3301,21 +3307,20 @@ static int folio_inc_gen(struct lruvec *lruvec, str= uct folio *folio) int type =3D folio_is_file_lru(folio); struct lru_gen_folio *lrugen =3D &lruvec->lrugen; int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); - unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags.f); - - VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); + unsigned long new_flags, old_flags =3D READ_ONCE(*folio_flags(folio, 0)); =20 do { - new_gen =3D ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; + new_gen =3D lru_get_gen_flags(old_flags); + /* folio_update_gen() has promoted this page? */ if (new_gen >=3D 0 && new_gen !=3D old_gen) return new_gen; =20 + new_flags =3D old_flags; new_gen =3D (old_gen + 1) % MAX_NR_GENS; - - new_flags =3D old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS); - new_flags |=3D (new_gen + 1UL) << LRU_GEN_PGOFF; - } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); + lru_set_gen_flags(&new_flags, new_gen); + lru_set_refs_flags(&new_flags, 0); + } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags)); =20 lru_gen_update_size(lruvec, folio, old_gen, new_gen); =20 @@ -4716,7 +4721,7 @@ static bool isolate_folio(struct lruvec *lruvec, stru= ct folio *folio, struct sca =20 /* see the comment on LRU_REFS_FLAGS */ if (!folio_test_referenced(folio)) - set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0); + folio_set_lru_refs(folio, 0); =20 success =3D lru_gen_del_folio(lruvec, folio, true); VM_WARN_ON_ONCE_FOLIO(!success, folio); @@ -4932,8 +4937,10 @@ static int evict_folios(unsigned long nr_to_scan, st= ruct lruvec *lruvec, } =20 /* don't add rejected folios to the oldest generation */ - if (lru_gen_folio_seq(lruvec, folio, false) =3D=3D min_seq[type]) - set_mask_bits(&folio->flags.f, LRU_REFS_FLAGS, BIT(PG_active)); + if (lru_gen_folio_seq(lruvec, folio, false) =3D=3D min_seq[type]) { + folio_set_lru_refs(folio, 0); + folio_set_active(folio); + } } =20 move_folios_to_lru(&list); --=20 2.55.0 From nobody Sat Sep 26 11:01:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77EEA42E401; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; cv=none; b=sTT6lkoejRfeTt9TN1uUmy2s3TX2GMAGfljUW0IZBiB9tuwr9/q3Zrb6Q0Fp+fm1GwdeDT90YzilsuBUIpJaBwq2u6sVAAXg1hIzIJXFYkj3SIhVAVztN5nxb1SbhhqBDyCZUGNubPpkPYM1e7zcjZXQLL+c6ITrCPhfFDWCFg4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; c=relaxed/simple; bh=ZFETDp6p9x9uvwIEd+HcIi3/dkthWZDx6+TpdJIY9to=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=jD80QVRiI3dw1NHbbb+wlpnpmrf0T3mBEAIuJKsOwoq6nmBJT9H8TmtB/3WVMGMX3aXUB8koMOvcv5Q4oJiVC+ISpfm1XaGG4wWKUduqPJdhdarCuoNr8UDJC/31/ZB/tjxPDVMxaiByQAiep0BK6zZGx7CobSoH82J+D7Z/QwM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ODwY/Ye6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ODwY/Ye6" Received: by smtp.kernel.org (Postfix) with ESMTPS id F0954C2BCFA; Wed, 2 Sep 2026 09:50:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1788342660; bh=ZFETDp6p9x9uvwIEd+HcIi3/dkthWZDx6+TpdJIY9to=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=ODwY/Ye631Rwhf/3uAzAk/Q5xTBhXfzqlhdZP675Mo8LmaAoMUt01xRD265D0dYau 5gMaJdSNbFmdYbWRn3I0OiknCFUs7Wv4w0jadRepdCkClmNVWj2MP6OoqbEcbewu4h yKzozgREq4rkHWsf4WQMc/NdoLJ7kTbJrInFe4aSTiokhWvNkr/szwJYrPGH83g2wM 45YVqD5+87AbGav0rkRlf+kUsWsm0cQhDHL1RNVfleoRc5BHK6Gzt7AoHCRJRR20AL M7iBBGyxhoQ3+8FFNUFjnTnxE7rFya1V7xhkWtJVfm+W/5NYb4chZ6p6wg2hcXl+CV hzWhBMuugj/BQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDF60C624D7; Wed, 2 Sep 2026 09:50:59 +0000 (UTC) From: Kairui Song via B4 Relay Date: Wed, 02 Sep 2026 17:50:56 +0800 Subject: [PATCH v5 3/6] mm/migrate: copy all referenced state via folio_migrate_lru_refs Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260902-mglru-flags-cleanup-v5-3-9db761d779ef@tencent.com> References: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> In-Reply-To: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , Shakeel Butt , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Chris Li , Baolin Wang , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Ridong Chen , Lian Wang , Yu Zhao , Zi Yan , Qi Zheng , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Kairui Song , Kairui Song X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1788342657; l=3683; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=Ox6zaP/QbfpF5dExxI36sjV5rWUkpJpYAOEG95FC1rw=; b=FEnfQvBL3/fmg1asiheJX1Ar9Ld/4KCcGnDdckAqj+CLkjyccsIX7u5VcGKPNX6YhMjrv98vM Hkt6CXlJifbB7kso51KMQSSbOdxbtfGM4GOe5GnDa2NksnCGAoAPNpT X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song folio_migrate_flags() copies PG_referenced separately from the MGLRU refs counter, which folio_migrate_refs() transfers. Yet under MGLRU, PG_referenced and the refs counter bits together describe the referenced status of a folio. Consolidate the two: rename folio_migrate_refs() to folio_migrate_lru_refs() and let it copy the complete referenced status, i.e., the MGLRU refs count including PG_referenced, or just PG_referenced for the active/inactive LRU. Drop the open-coded PG_referenced copy so the referenced status is transferred in one place. No behavior change is intended: under the active/inactive LRU the extra bits are unused, so operating on them is a noop. Transfer the reference state first, before the destination folio is marked uptodate, so a concurrent lockless reader cannot have its reference update overwritten by the copy. Reviewed-by: Baoquan He Reviewed-by: Baolin Wang Acked-by: David Hildenbrand (Arm) Reviewed-by: Lian Wang Reviewed-by: Barry Song Reviewed-by: Ridong Chen Signed-off-by: Kairui Song Acked-by: Qi Zheng --- include/linux/mm_inline.h | 20 +++++++++++++++----- mm/migrate.c | 6 +++--- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 3f4bd5b02b54..047295ae6e8a 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -373,11 +373,19 @@ static inline bool lru_gen_del_folio(struct lruvec *l= ruvec, struct folio *folio, return true; } =20 -static inline void folio_migrate_refs(struct folio *new, const struct foli= o *old) +/** + * folio_migrate_lru_refs - copy the reference state to a new folio + * @new: the destination folio + * @old: the source folio + * + * Transfer the reference state to @new during migration: the MGLRU + * refs count, including PG_referenced, or just PG_referenced for the + * active/inactive LRU. + */ +static inline void folio_migrate_lru_refs(struct folio *new, const struct = folio *old) { - unsigned long refs =3D READ_ONCE(old->flags.f) & LRU_REFS_MASK; - - set_mask_bits(&new->flags.f, LRU_REFS_MASK, refs); + BUILD_BUG_ON(LRU_REFS_MASK & BIT(PG_referenced)); + folio_set_lru_refs(new, folio_lru_refs(old)); } #else /* !CONFIG_LRU_GEN */ =20 @@ -406,8 +414,10 @@ static inline bool lru_gen_del_folio(struct lruvec *lr= uvec, struct folio *folio, return false; } =20 -static inline void folio_migrate_refs(struct folio *new, const struct foli= o *old) +static inline void folio_migrate_lru_refs(struct folio *new, const struct = folio *old) { + if (folio_test_referenced(old)) + folio_set_referenced(new); } #endif /* CONFIG_LRU_GEN */ =20 diff --git a/mm/migrate.c b/mm/migrate.c index 15b45832bcfa..a369d0c95c38 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -776,8 +776,9 @@ void folio_migrate_flags(struct folio *newfolio, struct= folio *folio) { int cpupid; =20 - if (folio_test_referenced(folio)) - folio_set_referenced(newfolio); + /* Copy the reference state, including PG_referenced */ + folio_migrate_lru_refs(newfolio, folio); + if (folio_test_uptodate(folio)) folio_mark_uptodate(newfolio); if (folio_test_clear_active(folio)) { @@ -807,7 +808,6 @@ void folio_migrate_flags(struct folio *newfolio, struct= folio *folio) if (folio_test_idle(folio)) folio_set_idle(newfolio); =20 - folio_migrate_refs(newfolio, folio); /* * Copy NUMA information to the new page, to prevent over-eager * future migrations of this same page. --=20 2.55.0 From nobody Sat Sep 26 11:01:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6850A42C4FA; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; cv=none; b=aX8a0NncbXeoDpMzDciZ4/jNtE2X5hZT4ABmBsdYhqBrk/ONUYiKUHha5ULLJVFvwyyWqtXOZR5bFRh0vOv6CfOzQ6thzrce1I1+rfxPykf9AZytxu9EFocCRcrOLQbZ2m7YowGOwel+oY5YyuPdFoTEX8uOoki3wes2G+SDjOc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; c=relaxed/simple; bh=ZExROXpyOer/1fjEOvr29buFXezjDIdWZE9Wtr6aFBc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=FKOkpXrmCA6Rv2tKu6bGQCCUsp7X66N1CjkkOfvYPbptu7qJZrTatRtJh+8zxfFl030GABlAbGn3SAmQouLNWU+w6b1UhLmMIvg7PaP6LyZKNYBMAXiyMRQ1NnnEQdjEQCRcxtTLmOoqa/uaSg4UWG+N9SzdWhM84p4ruGDV2/Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BBCLszTC; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BBCLszTC" Received: by smtp.kernel.org (Postfix) with ESMTPS id 12D3EC2BCFD; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1788342660; bh=ZExROXpyOer/1fjEOvr29buFXezjDIdWZE9Wtr6aFBc=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=BBCLszTCCb4aBxUWh0XikYYuXsfONFOW35ddWeGLMCsnqyTT+bqF6dpU4ywybiZKI 817oNndFREKS6RjS1azEkxO5b4Ci/k5d1j8qwIzXecsp6OBdTGJInWTufn4NtelLDg l5E7h0gmAJEXrLBJ9O6iKaVj19g6jC19ET+d9NfQGehU1NE+5LnnyPp+H+nbc/rdfD k6XsKXU7LAMs8CBHxaCoR2+HrLbyfgPczAN+w730FIEIGZCTGyVE1IS8z4PUbtSo0I RzTyJjHdQYKXOK9y8PnejBkV3QaZbqU3H2d2cLTBMATQqObWcvH5dLxLxQBAuavsbb 8mq5nHsRDiYZg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id F05DEC624D0; Wed, 2 Sep 2026 09:50:59 +0000 (UTC) From: Kairui Song via B4 Relay Date: Wed, 02 Sep 2026 17:50:57 +0800 Subject: [PATCH v5 4/6] mm/mglru: move max_seq read into walk_update_folio Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260902-mglru-flags-cleanup-v5-4-9db761d779ef@tencent.com> References: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> In-Reply-To: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , Shakeel Butt , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Chris Li , Baolin Wang , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Ridong Chen , Lian Wang , Yu Zhao , Zi Yan , Qi Zheng , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Kairui Song , Kairui Song X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1788342657; l=5248; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=7TChCfTyyaJ0Q3bIhLSmtfbdyXgREQ74noYCwDfeHq0=; b=jV8xURSe/w44tQhocpoNNDiokQnNheOfpgai7WIEzKSb+pCCY7rv38ODeBRFIPti58LC4eKoX kOO35SR86TCB95HpmWSlqs7HriHb7Tk++yQeqwE7ASe5wFBSJMEDvp8 X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song walk_pte_range(), walk_pmd_range_locked(), and lru_gen_look_around() each read lrugen->max_seq to compute the target generation used by walk_update_folio(), then pass it as a parameter. Move the read into walk_update_folio() itself so the callers no longer need to compute or pass the value. The max_seq read now happens once per folio update rather than once per walk range, so folios always get promoted to the current youngest generation. Reviewed-by: Baoquan He Reviewed-by: Baolin Wang Reviewed-by: Ridong Chen Reviewed-by: Lian Wang Reviewed-by: Barry Song Signed-off-by: Kairui Song Acked-by: Qi Zheng --- mm/vmscan.c | 29 ++++++++++++----------------- 1 file changed, 12 insertions(+), 17 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 24d09711e820..96fdfc14f615 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3517,13 +3517,15 @@ static bool suitable_to_scan(int total, int young) } =20 static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area= _struct *vma, - struct folio *folio, int new_gen, bool dirty) + struct lruvec *lruvec, struct folio *folio, bool dirty) { - int old_gen; + int new_gen, old_gen; =20 if (!folio) return; =20 + new_gen =3D lru_gen_from_seq(READ_ONCE(lruvec->lrugen.max_seq)); + if (dirty && !folio_test_dirty(folio) && !(folio_test_anon(folio) && folio_test_swapbacked(folio) && !folio_test_swapcache(folio))) @@ -3554,8 +3556,6 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long = start, unsigned long end, struct lru_gen_mm_walk *walk =3D args->private; struct mem_cgroup *memcg =3D lruvec_memcg(walk->lruvec); struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); - DEFINE_MAX_SEQ(walk->lruvec); - int gen =3D lru_gen_from_seq(max_seq); unsigned int nr; pmd_t pmdval; =20 @@ -3606,7 +3606,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long = start, unsigned long end, continue; =20 if (last !=3D folio) { - walk_update_folio(walk, args->vma, last, gen, dirty); + walk_update_folio(walk, args->vma, walk->lruvec, last, dirty); =20 last =3D folio; dirty =3D false; @@ -3619,7 +3619,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long = start, unsigned long end, walk->mm_stats[MM_LEAF_YOUNG] +=3D nr; } =20 - walk_update_folio(walk, args->vma, last, gen, dirty); + walk_update_folio(walk, args->vma, walk->lruvec, last, dirty); last =3D NULL; =20 if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &= end)) @@ -3642,8 +3642,6 @@ static void walk_pmd_range_locked(pud_t *pud, unsigne= d long addr, struct vm_area struct lru_gen_mm_walk *walk =3D args->private; struct mem_cgroup *memcg =3D lruvec_memcg(walk->lruvec); struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); - DEFINE_MAX_SEQ(walk->lruvec); - int gen =3D lru_gen_from_seq(max_seq); =20 VM_WARN_ON_ONCE(pud_leaf(*pud)); =20 @@ -3697,7 +3695,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigne= d long addr, struct vm_area goto next; =20 if (last !=3D folio) { - walk_update_folio(walk, vma, last, gen, dirty); + walk_update_folio(walk, vma, walk->lruvec, last, dirty); =20 last =3D folio; dirty =3D false; @@ -3711,7 +3709,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigne= d long addr, struct vm_area i =3D i > MIN_LRU_BATCH ? 0 : find_next_bit(bitmap, MIN_LRU_BATCH, i) + = 1; } while (i <=3D MIN_LRU_BATCH); =20 - walk_update_folio(walk, vma, last, gen, dirty); + walk_update_folio(walk, vma, walk->lruvec, last, dirty); =20 lazy_mmu_mode_disable(); spin_unlock(ptl); @@ -4275,8 +4273,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk = *pvmw, unsigned int nr) struct pglist_data *pgdat =3D folio_pgdat(folio); struct lruvec *lruvec; struct lru_gen_mm_state *mm_state; - unsigned long max_seq; - int gen; =20 lockdep_assert_held(pvmw->ptl); VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); @@ -4313,8 +4309,6 @@ bool lru_gen_look_around(struct page_vma_mapped_walk = *pvmw, unsigned int nr) =20 memcg =3D get_mem_cgroup_from_folio(folio); lruvec =3D mem_cgroup_lruvec(memcg, pgdat); - max_seq =3D READ_ONCE((lruvec)->lrugen.max_seq); - gen =3D lru_gen_from_seq(max_seq); mm_state =3D get_mm_state(lruvec); =20 lazy_mmu_mode_enable(); @@ -4346,7 +4340,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk = *pvmw, unsigned int nr) continue; =20 if (last !=3D folio) { - walk_update_folio(walk, vma, last, gen, dirty); + walk_update_folio(walk, vma, lruvec, last, dirty); =20 last =3D folio; dirty =3D false; @@ -4358,13 +4352,14 @@ bool lru_gen_look_around(struct page_vma_mapped_wal= k *pvmw, unsigned int nr) young +=3D nr; } =20 - walk_update_folio(walk, vma, last, gen, dirty); + walk_update_folio(walk, vma, lruvec, last, dirty); =20 lazy_mmu_mode_disable(); =20 /* feedback from rmap walkers to page table walkers */ if (mm_state && suitable_to_scan(i, young)) - update_bloom_filter(mm_state, max_seq, pvmw->pmd); + update_bloom_filter(mm_state, READ_ONCE(lruvec->lrugen.max_seq), + pvmw->pmd); =20 mem_cgroup_put(memcg); =20 --=20 2.55.0 From nobody Sat Sep 26 11:01:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 685B142D777; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; cv=none; b=ObgHV46L+92AKawY9/Z2afJa1mbkkUuR42SKSJDR4e9dVjwB9p3r1HYier3pzxTih02yojh5C8ypdfrMv/nE83fky4n5u9r8ugnbFInizSE7rDpZiSe0q+qc7jIV7UlnP3wFT0WPMkjDQV+Aa5zVJmUT/QqgGLCiIr82n35IJF4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; c=relaxed/simple; bh=Mo3+8RFfppINKIARqVbgEULhQ/6q8Q5E73vtDdVsNaQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=JQF3e3/2tqhiRFMnhpkDj2z09XUWHYauncTD5WroGAQg0MBHbAttlw6cB6kTL3wAGMX4XjOM9xXRTbZIDUUTdkvplxVPflw+5YAPL9bnP0jvE5/+k2dvgVBhk8j2eY9aMUO7eRlEeBGvLOx75/hj0CxX2tLOsdkP2Gv8r8srdu8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LjN8Jh71; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LjN8Jh71" Received: by smtp.kernel.org (Postfix) with ESMTPS id 2B516C2BD04; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1788342660; bh=Mo3+8RFfppINKIARqVbgEULhQ/6q8Q5E73vtDdVsNaQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=LjN8Jh71ESTtrz0II1QkWFMR6Ekn10yjUiU5wrtpB83llUdV/2TXESJKlYxXGJbFK BjL/hqJHbUqLQ+xQ7JKPNusWEkPdY3H3tSO0Tripls89K0PunwIWQZJCzgkJ01O9VE MF0jUHlUj8tq8RKt4PsFv4Ve+bB3kLb14TmcN12pef+qAjOXu9J5M6q/6aHQAvub+I wtL7Ub+pGmIsjG3jKL3/mxnsnakHKDJj4Ck/bM7+uPcLyFOxE0xC076uHuPhnmg5Ob +JvXfQKRavAYb/WERikZtdLsVUbzNO0i9U6pMdIidmI4xXF5ULRIzKAX1hAGCvxddi o+jRgTBAC38xg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15FB7C61DD6; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) From: Kairui Song via B4 Relay Date: Wed, 02 Sep 2026 17:50:58 +0800 Subject: [PATCH v5 5/6] mm/mglru: use explicit tier range in read_ctrl_pos() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260902-mglru-flags-cleanup-v5-5-9db761d779ef@tencent.com> References: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> In-Reply-To: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , Shakeel Butt , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Chris Li , Baolin Wang , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Ridong Chen , Lian Wang , Yu Zhao , Zi Yan , Qi Zheng , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Kairui Song , Kairui Song X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1788342657; l=3649; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=oVmtU7RtLEHATDqO9A5jftz8sm6P6wAktfihxMqJ5Tw=; b=dPPzLe7vps/4tFzAt1JpcbiW7bZLRHQCFNYaCQ4sCRNhFfKoV8rZMZJk4dpFzSA1X0k61uup7 ZtSlanhdkEYCfpC+U1UuJBSbBsreUiJ79Z2nOzSwDx7LLDmkPwdeE/2 X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song read_ctrl_pos() encodes the tier range in a single "tier" parameter via "tier % MAX_NR_TIERS" as the start and "min(tier, MAX_NR_TIERS-1)" as the end. This is hard to follow, maintain, or extend. Tier values 0..3 select a single tier, while tier =3D=3D MAX_NR_TIERS selects the full range. Replace it with explicit (tier_min, tier_max) parameters using a closed [tier_min, tier_max] interval, and add LRU_TIER_MIN and LRU_TIER_MAX for the tier bounds. The call sites now become self-documenting: - get_tier_idx: (LRU_TIER_MIN, LRU_TIER_MIN) for the first tier, (tier, tier) for each subsequent tier - get_type_to_scan: (LRU_TIER_MIN, LRU_TIER_MAX) for the full range No functional change. Reviewed-by: Baolin Wang Reviewed-by: Baoquan He Reviewed-by: Barry Song Reviewed-by: Ridong Chen Signed-off-by: Kairui Song Acked-by: Qi Zheng --- include/linux/mmzone.h | 2 ++ mm/vmscan.c | 18 ++++++++++-------- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index c9ecf370cd9f..9b27cf53bdbc 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -492,6 +492,8 @@ enum lruvec_flags { * folio->flags, masked by LRU_REFS_MASK. */ #define MAX_NR_TIERS 4U +#define LRU_TIER_MIN 0U +#define LRU_TIER_MAX (MAX_NR_TIERS - 1) =20 #ifndef __GENERATING_BOUNDS_H =20 diff --git a/mm/vmscan.c b/mm/vmscan.c index 96fdfc14f615..714a3f273f49 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3198,8 +3198,8 @@ struct ctrl_pos { int gain; }; =20 -static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier, int g= ain, - struct ctrl_pos *pos) +static void read_ctrl_pos(struct lruvec *lruvec, int type, int tier_min, + int tier_max, int gain, struct ctrl_pos *pos) { int i; struct lru_gen_folio *lrugen =3D &lruvec->lrugen; @@ -3208,7 +3208,7 @@ static void read_ctrl_pos(struct lruvec *lruvec, int = type, int tier, int gain, pos->gain =3D gain; pos->refaulted =3D pos->total =3D 0; =20 - for (i =3D tier % MAX_NR_TIERS; i <=3D min(tier, MAX_NR_TIERS - 1); i++) { + for (i =3D tier_min; i <=3D tier_max; i++) { pos->refaulted +=3D lrugen->avg_refaulted[type][i] + atomic_long_read(&lrugen->refaulted[hist][type][i]); pos->total +=3D lrugen->avg_total[type][i] + @@ -4809,9 +4809,9 @@ static int get_tier_idx(struct lruvec *lruvec, int ty= pe) * This value is chosen because any other tier would have at least twice * as many refaults as the first tier. */ - read_ctrl_pos(lruvec, type, 0, 2, &sp); - for (tier =3D 1; tier < MAX_NR_TIERS; tier++) { - read_ctrl_pos(lruvec, type, tier, 3, &pv); + read_ctrl_pos(lruvec, type, LRU_TIER_MIN, LRU_TIER_MIN, 2, &sp); + for (tier =3D LRU_TIER_MIN + 1; tier <=3D LRU_TIER_MAX; tier++) { + read_ctrl_pos(lruvec, type, tier, tier, 3, &pv); if (!positive_ctrl_err(&sp, &pv)) break; } @@ -4832,8 +4832,10 @@ static int get_type_to_scan(struct lruvec *lruvec, i= nt swappiness) * Compare the sum of all tiers of anon with that of file to determine * which type to scan. */ - read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp); - read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappi= ness, &pv); + read_ctrl_pos(lruvec, LRU_GEN_ANON, LRU_TIER_MIN, LRU_TIER_MAX, + swappiness, &sp); + read_ctrl_pos(lruvec, LRU_GEN_FILE, LRU_TIER_MIN, LRU_TIER_MAX, + MAX_SWAPPINESS - swappiness, &pv); =20 return positive_ctrl_err(&sp, &pv); } --=20 2.55.0 From nobody Sat Sep 26 11:01:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 78D5342E406; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; cv=none; b=EJD6iX4qXIebnV3mMA8SB6hyMtQB/52w9SLXKQPqJ5G9yvzLAKPh/h5F0woHxeL5cQ0yLP6/wFgQ7nTxvrX3W2Nf4f3KJ/z1Ekh36oMksu5eWxZyLyukyxBEjlAqcMzb5glco2tK5PguDNlSP1GCTo+7IM9rO37mZ4uC7Pyfn6I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788342660; c=relaxed/simple; bh=ENRV4v6WgSczjFoQPgCJiVLJfabo2qU5Lymw9U16Hos=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ghHk2IzfwmTn5tcDC+Q5gwyEJHYNlR5y/uDpbof8NRx46XWikylg42RZS7Mq2vcIXlHcX36czLOhJN2Fs9GdsW4Dud/ED50I78IkBSeoxgjoAj6UEn7+yoSnBBWRZfH2eqGUk2ORkuPKavk3z9vcKghRhDl9FpFNSOqk5zT1E/o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=guhwg4Zz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="guhwg4Zz" Received: by smtp.kernel.org (Postfix) with ESMTPS id 45C3CC2BCFC; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1788342660; bh=ENRV4v6WgSczjFoQPgCJiVLJfabo2qU5Lymw9U16Hos=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=guhwg4ZzUREQFbNhN1zYAYuwejMCSgQdNE+tf28bJpEV7u4X5chlzR85mDuMtpVHt sMfpHf3VdOGEYa85xriA1NyTB9ZeJIxBdi0YGLWLtVZxKqHME1sLgJ1cWsG7MqHHzc JOTPtI3Vw1hHhcb1N7jRfFte2OB58/cRpKERMMxnv+uf1BWxEDtKa1HzSyHd9V9auV Q1a3NgEgYNOTTHxO3xVG9iBW93BIbzECEzrwg+9luRC7LhD0+L+08wiX9FmO0M7uMz lo0+mhceUhHkU2SaOmNCRSsHGpPQVwkC3KElxruH4qJIotFmjRXEGLHE8kxa2MAm8A od+yi+mBimLAg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D578C624D6; Wed, 2 Sep 2026 09:51:00 +0000 (UTC) From: Kairui Song via B4 Relay Date: Wed, 02 Sep 2026 17:50:59 +0800 Subject: [PATCH v5 6/6] mm/mglru: fix potential generation folio number leak Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260902-mglru-flags-cleanup-v5-6-9db761d779ef@tencent.com> References: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> In-Reply-To: <20260902-mglru-flags-cleanup-v5-0-9db761d779ef@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Barry Song , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , Shakeel Butt , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Chris Li , Baolin Wang , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Ridong Chen , Lian Wang , Yu Zhao , Zi Yan , Qi Zheng , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Kairui Song , Kairui Song X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1788342657; l=5300; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=BeFRtTv2DUVCnBDdSk2xGGeIVNwza5FMggB+YkSlaTk=; b=WKortzZwxgU1YkMCpiMgIJayfIQsa7teb+grctu7+vTppe/6UAmFcu5HnWG/tveeH6BpxNbyq 4h3I9AloIQiAPCNj/AL/+unCXwYy8KCX1/IJJTmhLHESJl3BqpwQucE X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Each generation of MGLRU accounts anon and file folio numbers separately, and the page table walker updates each generation's counters in batch once the walk is done. The walker promotes a folio's generation with a cmpxchg on folio->flags, and update_batch_size() then reads the live flags again to pick the anon/file column to charge. The walk holds neither the lruvec lock nor the folio lock, so the type can flip between the cmpxchg and that read: the lazyfree path clears PG_swapbacked, and reclaim sets it back on a dirty lazyfree folio. The batched delta pair is then recorded in the wrong type column. Nothing reconciles it afterwards, permanently skewing lrugen->nr_pages and the reclaim budgets derived from it. Fix it by capturing the type from the flags snapshot the cmpxchg linearized against: folio_update_gen() returns the type of the state it transitioned from, and update_batch_size() accounts with that. A folio's type only changes while it is off the LRU list, inside a del/add pair under the lruvec lock, with the gen bits cleared in between. The generation and PG_swapbacked sit in the same folio->flags word, so the cmpxchg snapshot captures them together. Let G be the generation that snapshot captured (old_gen) and G' the one it wrote (new_gen); the CAS can land in only three places: - before the del: the folio is anon at G; the batch records anon G -> G', and the del later removes the folio from the anon counters; - between del and add: gen =3D=3D -1, so folio_update_gen() returns -1 without touching the flags and no batch is recorded; the del/add pair accounts for the move alone; - after the add: the folio is file at the fresh generation the add charged; the batch records file, that gen -> G', matching that charge. Unlike the drift of lazy promotions, which sort_folio() repairs under the lruvec lock, the phantom deltas from before this fix land in a column the folio never occupies again, so nothing ever repairs them. Fixes: bd74fdaea146 ("mm: multi-gen LRU: support page table walks") Signed-off-by: Kairui Song Acked-by: Qi Zheng Reviewed-by: Baolin Wang Reviewed-by: Barry Song --- include/linux/mm_inline.h | 11 +++++++++++ mm/vmscan.c | 13 +++++++------ 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 047295ae6e8a..ab69b9930893 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -30,6 +30,17 @@ static inline int folio_is_file_lru(const struct folio *= folio) return !folio_test_swapbacked(folio); } =20 +/** + * folio_flags_is_file_lru - Should the folio be on a file LRU or anon LRU? + * @flags: The folio's flags. + * + * Just like folio_is_file_lru but take the folio flags directly instead. + */ +static inline int folio_flags_is_file_lru(const unsigned long *flags) +{ + return !test_bit(PG_swapbacked, flags); +} + static __always_inline void __update_lru_size(struct lruvec *lruvec, enum lru_list lru, enum zone_type zid, long nr_pages) diff --git a/mm/vmscan.c b/mm/vmscan.c index 714a3f273f49..e7d379c6e703 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3269,7 +3269,8 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, st= ruct ctrl_pos *pv) *************************************************************************= *****/ =20 /* promote pages accessed through page tables */ -static int folio_update_gen(struct folio *folio, int new_gen, const vma_fl= ags_t *vma_flags) +static int folio_update_gen(struct folio *folio, int new_gen, int *type, + const vma_flags_t *vma_flags) { unsigned long new_flags, old_flags =3D READ_ONCE(*folio_flags(folio, 0)); int old_gen; @@ -3298,6 +3299,7 @@ static int folio_update_gen(struct folio *folio, int = new_gen, const vma_flags_t new_flags |=3D BIT(PG_workingset); } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags)); =20 + *type =3D folio_flags_is_file_lru(&old_flags); return old_gen; } =20 @@ -3328,9 +3330,8 @@ static int folio_inc_gen(struct lruvec *lruvec, struc= t folio *folio) } =20 static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *= folio, - int old_gen, int new_gen) + int old_gen, int new_gen, int type) { - int type =3D folio_is_file_lru(folio); int zone =3D folio_zonenum(folio); int delta =3D folio_nr_pages(folio); =20 @@ -3519,7 +3520,7 @@ static bool suitable_to_scan(int total, int young) static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area= _struct *vma, struct lruvec *lruvec, struct folio *folio, bool dirty) { - int new_gen, old_gen; + int new_gen, old_gen, type; =20 if (!folio) return; @@ -3532,9 +3533,9 @@ static void walk_update_folio(struct lru_gen_mm_walk = *walk, struct vm_area_struc folio_mark_dirty(folio); =20 if (walk) { - old_gen =3D folio_update_gen(folio, new_gen, &vma->flags); + old_gen =3D folio_update_gen(folio, new_gen, &type, &vma->flags); if (old_gen >=3D 0 && old_gen !=3D new_gen) - update_batch_size(walk, folio, old_gen, new_gen); + update_batch_size(walk, folio, old_gen, new_gen, type); } else if (lru_gen_set_refs(folio, &vma->flags)) { old_gen =3D folio_lru_gen(folio); if (old_gen >=3D 0 && old_gen !=3D new_gen) --=20 2.55.0