From nobody Mon Sep 28 13:17:21 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D17F45FFAD for ; Fri, 21 Aug 2026 10:26:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787307991; cv=none; b=ekJTLL1FqVvzubddgek5nbfXVQga/b6lHbjN+XEOgbuwDMyntFIKtLT8A10enYkfgC6zZontHgjovTkxG2IQrt+AJD8RobsrS2yAhrLcAA9aut6TXQRmlBRAAvBmOdhxWaMaf/NxvP+yt4h7FMnulh/2tUfsOkMvuyWa2bfGbM0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787307991; c=relaxed/simple; bh=KHfEnNVVBoTv/IduBv0cvciIROaDh5WATIJDGrPrTK8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KmcN1Y+cgkmiA3LGPG7eHk8O+zMN6qDNOV6h5llzy5zSaKXlpgJ93sXVDuMd+CRgG1N262u+RRcddUn/S4MZ2AYufdje7+1UEz80PECs+IpUnZgrSIyyvRxbr0eTamT4bjlG+8BE7EexsR5PGbJePqcrRFS6j+nKnYi0mxkk3xA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=T9lhzXpN; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="T9lhzXpN" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9A4CE1F00A3A; Fri, 21 Aug 2026 10:26:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787307982; bh=V6QovsgX/8e0d2F73NtdKk1ypNzI0fiIVIiWZ5AR6Sk=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=T9lhzXpNhEP9xFzYvNcj0iz1hYBR48fDyI23wrWar1B+UKVpKzY4C1MbOTT0yltZ8 w8gWnNgfqIHsKYWPIT+2pLWu8F4y2SDN3fOvyUW6apHrkFJArMFAnyeD5sFCQxHg4c +2OSLcGznOKOqe65NdAzMtVtGALQqh6r6Pt+NT7iEe4IIu4JdV/mP4v3nc/wLuYzre xtqueqW3+evvnLuaNDfkfIfuo8ECe2Gx3OOhgNZ4Y5SUEs9zxveu5JwItjtcKM/4MM OoQ9izQ2AbzEeW1Re/qFby5I5RB1eTnFISjN4xtYbAWRVAqdXQwa9TbFIL8UF12n3K 4Fm3+VduFLtcA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" Subject: [PATCH 1/6] mm/mglru: batch update lrugen->nr_pages in inc_min_seq() Date: Fri, 21 Aug 2026 18:25:33 +0800 Message-Id: <20260821102538.22642-2-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260821102538.22642-1-baohua@kernel.org> References: <20260821102538.22642-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero. Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size(). Signed-off-by: Barry Song (Xiaomi) Tested-by: Xueyuan Chen --- mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index c1404a59523d..0d74fc00abd3 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, in= t gen, const vma_flags_t *vma } =20 /* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio) +static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increas= ed) { - int type =3D folio_is_file_lru(folio); - struct lru_gen_folio *lrugen =3D &lruvec->lrugen; - int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags.f); + int new_gen; =20 VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); =20 do { new_gen =3D ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; /* folio_update_gen() has promoted this page? */ - if (new_gen >=3D 0 && new_gen !=3D old_gen) + if (new_gen >=3D 0 && new_gen !=3D old_gen) { + if (increased) + *increased =3D false; return new_gen; + } =20 new_gen =3D (old_gen + 1) % MAX_NR_GENS; =20 @@ -3317,8 +3318,21 @@ static int folio_inc_gen(struct lruvec *lruvec, stru= ct folio *folio) new_flags |=3D (new_gen + 1UL) << LRU_GEN_PGOFF; } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); =20 - lru_gen_update_size(lruvec, folio, old_gen, new_gen); + if (increased) + *increased =3D true; + return new_gen; +} =20 +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio) +{ + int type =3D folio_is_file_lru(folio); + struct lru_gen_folio *lrugen =3D &lruvec->lrugen; + int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); + bool gen_increased; + + new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); + if (gen_increased) + lru_gen_update_size(lruvec, folio, old_gen, new_gen); return new_gen; } =20 @@ -3904,6 +3918,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) struct lru_gen_folio *lrugen =3D &lruvec->lrugen; int hist =3D lru_hist_from_seq(lrugen->min_seq[type]); int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); + int target_gen =3D (old_gen + 1) % MAX_NR_GENS; =20 /* For file type, skip the check if swappiness is anon only */ if (type && (swappiness =3D=3D SWAPPINESS_ANON_ONLY)) @@ -3916,32 +3931,41 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, int swappiness) /* prevent cold/hot inversion if the type is evictable */ for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; + unsigned long delta =3D 0; =20 while (!list_empty(head)) { struct folio *folio =3D lru_to_folio(head); + long nr_pages =3D folio_nr_pages(folio); int refs =3D folio_lru_refs(folio); bool workingset =3D folio_test_workingset(folio); + bool gen_increased; =20 VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 - new_gen =3D folio_inc_gen(lruvec, folio); + new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); - + if (gen_increased) + delta +=3D nr_pages; /* don't count the workingset being lazily promoted */ if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { int tier =3D lru_tier_from_refs(refs, workingset); - int delta =3D folio_nr_pages(folio); =20 WRITE_ONCE(lrugen->protected[hist][type][tier], - lrugen->protected[hist][type][tier] + delta); + lrugen->protected[hist][type][tier] + nr_pages); } =20 if (!--remaining) - return false; + break; } + WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone], + lrugen->nr_pages[old_gen][type][zone] - delta); + WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone], + lrugen->nr_pages[target_gen][type][zone] + delta); + if (!remaining) + return false; } done: reset_ctrl_pos(lruvec, type, true); --=20 2.34.1 From nobody Mon Sep 28 13:17:21 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB5A037D131 for ; Fri, 21 Aug 2026 10:26:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787307994; cv=none; b=c9OYUgUMXyXxiH9ex08VaFE4XQ7HdrTIH7Xxrn3gEQ02BuATMeB1rM2FMuRiC4/kun4lXF0KerA2S37NsghKwu5+Oe9qwNY24lRSEEHo0FmiA8y/At2kQ70+ordGGvfcFXPCgdbuCFNqHpotihKKxlfkW1K/Z5aJey/DPcmt4V4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787307994; c=relaxed/simple; bh=9RQZ2saCVCXg+gV6puYyA4G+l5ZtffcHSGbbMakqqzg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rU/dnXV9wt87MmAUpQrW1M4bgwFC6IrvoDRsw4Gm982LRthV80uzApA8SkZCT6o04FTXDhxjAxzUx22v48Nmms8EkvKf7+1k5eaAbOqt7ItnkRvCAosETMKaUnBMVpB1kago9IBgzLxQc6elsGBGZX3fmXtGSEkarDU3unHPVZQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iF+sSeP3; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iF+sSeP3" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A11C61F00A3D; Fri, 21 Aug 2026 10:26:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787307986; bh=xaIllxSu2Y9u0+sAmWxSLKA+YkGfX6Fr0GdsHrBYVUM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=iF+sSeP3VoDwF86my97C70bwVVWVkjD5QSlZlwlruiEgMQBn/wy4kKdvZxjcR0zf3 A7I6SU/Zc44/mWsYmI1/wNYCD+kqtBjXd8SV+Gt+1qcvjl+0so/AZyoLkULtlkc4i+ Va6Shven2L5OX538x0k7mskcxIQIel19UF43jFTDu/ka1e635+7jped0IeMdt7zC7r ES32xrFLVvOKt8/Gn1G8HlYjR4bFLp3qFI8hXJ3MpEj4MG5eITovty9B6ksi+FEu1g 1R4t7tARNjrks8gs45n5MDZFwGQgQGvpwQFmgRrVCNEAomavSljfze0nWzH1V4np0U jOoythJbV4TXQ== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" Subject: [PATCH 2/6] mm/mglru: batch update lrugen->protected in inc_min_seq() Date: Fri, 21 Aug 2026 18:25:34 +0800 Message-Id: <20260821102538.22642-3-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260821102538.22642-1-baohua@kernel.org> References: <20260821102538.22642-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Avoid updating lrugen->protected with WRITE_ONCE() for each folio, which may prevent potential compiler optimizations. Accumulate the updates locally and apply them in a batch instead. Signed-off-by: Barry Song (Xiaomi) Tested-by: Xueyuan Chen --- mm/vmscan.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 0d74fc00abd3..99ee3c833d54 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3931,7 +3931,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) /* prevent cold/hot inversion if the type is evictable */ for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; - unsigned long delta =3D 0; + unsigned long protected[MAX_NR_TIERS] =3D {}, delta =3D 0; =20 while (!list_empty(head)) { struct folio *folio =3D lru_to_folio(head); @@ -3953,8 +3953,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { int tier =3D lru_tier_from_refs(refs, workingset); =20 - WRITE_ONCE(lrugen->protected[hist][type][tier], - lrugen->protected[hist][type][tier] + nr_pages); + protected[tier] +=3D nr_pages; } =20 if (!--remaining) @@ -3964,6 +3963,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) lrugen->nr_pages[old_gen][type][zone] - delta); WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone], lrugen->nr_pages[target_gen][type][zone] + delta); + for (int tier =3D 0; tier < MAX_NR_TIERS; tier++) + WRITE_ONCE(lrugen->protected[hist][type][tier], + lrugen->protected[hist][type][tier] + protected[tier]); if (!remaining) return false; } --=20 2.34.1 From nobody Mon Sep 28 13:17:21 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DBC247ACF5 for ; Fri, 21 Aug 2026 10:26:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787308003; cv=none; b=khj04yGbRI5LELkw4UPcfvBl8iu4Hx+OX2a6AGbmF2GdxXs+RBq+adDsdo8K0yK/CZ3g2il7wch5eu7U7hywBfhaWnP1rs/Odwcu78F2YH4czB0d9q12YVNEQ22F2dI7Qefc1kkZniD9+bvNVzMn0w/kxEIiDEM6Q2tsbbyGpOQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787308003; c=relaxed/simple; bh=p/ohgA6OKdZ3L9ESB2t92Vdoor9GmobZwSb6/RnlTjI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=n4bow4oxhXzFe5LkEgcL8ZXJV38MId/yFX3FCBNJHUhvsChyhuEVyPtxSHp7SSI2J0YOTGHO7MFhHVuDYUQVmUQVYLTJuOc7pRqTprpBspGcLJpEZeaMSa/lVgS6r+MT38bG4oFGoWN9CHGazkfrtmexmNQfnFiiFx8vdQtw89Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nDycVO7I; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nDycVO7I" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A7F6D1F000E9; Fri, 21 Aug 2026 10:26:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787307990; bh=Sbrxf+PZdylQ4e8d9YhboPwelpbW3kiqOe9jSbLC3Yk=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=nDycVO7Itn7ToZ1y7L33bMcoxz+OF5wNBXiEGMXhkR76hWR+pKRjmDM0iU7wa86b4 +GKnyADWlZ7KUaMSVIlHVhN0aQzb7icUPIrqfTI+tPmfZxzeKt3h5veI0og0/zLI7T K2ablnl4jxHm9M6ioeVn6xWugVgamr7FA2qSURfM5S+ORIXa8dLF/ll9Cw1/zD1N7L AeFZ+fmRcC/VXCpTBQGlbp7T0LT1tlccbdV1P8bN/Z7hLe6BSW/U/F4wD9JTjlOZyR vTYPyZVbKuZMFcb966DQydno5oE9RgHF5Iz0bndfIjG2Z11XY1uQKAv0O+BmzSohsV U8rQmdW1++isw== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" Subject: [PATCH 3/6] mm/mglru: enhance cold/hot inversion handling in inc_min_seq() Date: Fri, 21 Aug 2026 18:25:35 +0800 Message-Id: <20260821102538.22642-4-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260821102538.22642-1-baohua@kernel.org> References: <20260821102538.22642-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation. It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios(). Otherwise, promoted folios could end up behind folios that were never promoted, effectively inverting their hot/cold ordering. Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Kairui Song Tested-by: Xueyuan Chen --- mm/vmscan.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 99ee3c833d54..3b618a51cde2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3946,9 +3946,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int t= ype, int swappiness) VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); - list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); - if (gen_increased) + if (gen_increased) { delta +=3D nr_pages; + list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + } else { + list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); + } /* don't count the workingset being lazily promoted */ if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { int tier =3D lru_tier_from_refs(refs, workingset); --=20 2.34.1 From nobody Mon Sep 28 13:17:21 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04931478E51 for ; Fri, 21 Aug 2026 10:26:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787308001; cv=none; b=M5k8DQ9Q5kw4Jxs9xTiMg73QT2lfMRBh/C0xGQuuQtxo47XB4C9KX4ht2ixZHbbPjYLLffmBHGlvzkbXKFmvh9E2yzfWWkBdAViHen4h+9Jjk+OsCnte7CFfgmqpGJZXCHRoFSIAxRwb5kyrSXmzEBfqAxNu8YJ1vvCnuJrFpXU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787308001; c=relaxed/simple; bh=QBfCMbSODhiumMXzpsnnq/Q/qupUEE06nlRr3gezUNA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RoJl04cAZk9teCA8eH9Ii4y/Jo9cfC66UIGYviydLmYb2icozYHE+gxRO04XWOUF6zXtD7Holxfb53No23HwSSMkC7a/YGbNxObIqRjpl0o7GlcTxmuf0b3GdbOE8/SfOCKIUGv3z6yHBVVUZNCFGjSLW129iXCGvLUijbfx488= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kW2WO3wS; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kW2WO3wS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AECE61F00A3E; Fri, 21 Aug 2026 10:26:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787307994; bh=bXd8/5/IV3VgMHX6Urv7GKHnjSeWG6/GogM3Cg3sztY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=kW2WO3wSoJLtg6Inb8AuPoDn9JZnzj0GUZW9aneFAAlpqUlseMqxCN/Vjt2rwZItD Wmak52jD6QObp8iy5+IXAi2lk9KhwYUN+4httBpijukH+I0275HsnTWQi4Jij9OkmO 9DPBuk/0Fed3ckWV3uSVkA5H7uoQTdjtJdQYrzNhM00kAzfiOJuTueH3W65JYz35hj NDJv0eQQpQWRT/9HsRtyOTBPcNP4rY32LOHA35D6TJ3YrDIq+CpQWDcEtcF/YOffAB FOezdPShO5gmkBzyKjzBKYcJWPnqwqDY053pkCDvAbP1yzVgdQPSQROJmp1VBZCTBs 3G08JcFDPOLzg== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" Subject: [PATCH 4/6] mm/mglru: exclude folios promoted by aging from protected in inc_min_seq() Date: Fri, 21 Aug 2026 18:25:36 +0800 Message-Id: <20260821102538.22642-5-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260821102538.22642-1-baohua@kernel.org> References: <20260821102538.22642-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some folios may have been promoted during aging, so don't count them as protected, similar to sort_folio(). Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Baoquan He Tested-by: Xueyuan Chen --- mm/vmscan.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 3b618a51cde2..7bd01875fade 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3949,16 +3949,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, int swappiness) if (gen_increased) { delta +=3D nr_pages; list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + /* don't count the workingset being lazily promoted */ + if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { + int tier =3D lru_tier_from_refs(refs, workingset); + + protected[tier] +=3D nr_pages; + } } else { list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); } - /* don't count the workingset being lazily promoted */ - if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { - int tier =3D lru_tier_from_refs(refs, workingset); - - protected[tier] +=3D nr_pages; - } - if (!--remaining) break; } --=20 2.34.1 From nobody Mon Sep 28 13:17:21 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27B3C47CA64 for ; Fri, 21 Aug 2026 10:26:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787308010; cv=none; b=Gob9KtaTnXz39SEctXDrVUZHnMo4iLoPeFj9LsHaL6q4V/Z1O1+1pQy4VjxZ9qUlbIhdpXKB+WUMCFxYHWJO44/OO0W0PbH/hiPTYQ7bvP3XOnzMzRYGoHKphSMLxz6hmsXdG5rg6ZmZq5GLilH8Ki98aodaI5u+1nFvW7Zavl8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787308010; c=relaxed/simple; bh=YF8lnrLkcB0qrQEcSpeDBGFiBxT7QOoiq2glgdnd18U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QJpggIBt7Y49bXBbdGLYfbvV5QITO1FJXOZlvyt963Qsw2G60jhmssi13nRn+RXW6kOarKxBt1af/Y5w1AR4LeXvsBnboOamYfSDOkjllcuYwK2edGNjfMXRq0AR+CpkiWnCFp1kscTVLpz4R9NXIZIogYXWqrIq3NVg+V0IC64= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ielwHHfe; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ielwHHfe" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B5A2B1F00A3A; Fri, 21 Aug 2026 10:26:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787307998; bh=DUPbWhkO9uqnpeFMVIsJBlpt8zN0k3f8YPV9jGEypkU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=ielwHHfesCgnNzots4rpLldqZiC88Hm8FP2b2C/4oMMSgPDfDTgfH2gjNE0Ii18zO lLNNSfD0C5FObB2dU+gKoa+DrgFLYnRJOA7kCygQeBw0rr9nBnQOkQqrQJoJ/N8Gl+ hhFEkgRcxVUSfLbyr6uaiXv3+PGa/vVZly6xlnW5G1OHLybXzeuTPMmFVxxHCZTYEO GjsYecacC08cdmzbdAHnkQXRIwmVziWgZQPAS1a+Ddw/SZsP5V9rZ661O76OcEc1gb XCfUP4/Y01cqNsITdHx6okr4MIiBDwQvxRxeeLg+59W9QqAD4q1PAFs8txYCv0AkRc 29SoD1XBg71HQ== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" Subject: [PATCH 5/6] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail Date: Fri, 21 Aug 2026 18:25:37 +0800 Message-Id: <20260821102538.22642-6-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260821102538.22642-1-baohua@kernel.org> References: <20260821102538.22642-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For reclamation, it makes sense to reclaim folios from tail to head, as folios near the head are relatively hot. However, when moving folios from the oldest generation to the second-oldest generation, using the tail-to-head order would effectively cause a cold/hot inversion. Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Baoquan He Tested-by: Xueyuan Chen --- mm/vmscan.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 7bd01875fade..2fd82b2ca4d1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -191,8 +191,20 @@ struct scan_control { prefetchw(&prev->_field); \ } \ } while (0) +#define prefetchw_next_lru_folio(_folio, _base, _field) \ + do { \ + if ((_folio)->lru.next !=3D _base) { \ + struct folio *next; \ + \ + next =3D list_entry((_folio)->lru.next, \ + struct folio, lru); \ + prefetchw(&next->_field); \ + } \ + } while (0) + #else #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0) +#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0) #endif =20 /* @@ -3932,9 +3944,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int t= ype, int swappiness) for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; unsigned long protected[MAX_NR_TIERS] =3D {}, delta =3D 0; + struct list_head *pos =3D head->next; =20 - while (!list_empty(head)) { - struct folio *folio =3D lru_to_folio(head); + while (pos !=3D head) { + struct folio *folio =3D list_entry(pos, struct folio, lru); long nr_pages =3D folio_nr_pages(folio); int refs =3D folio_lru_refs(folio); bool workingset =3D folio_test_workingset(folio); @@ -3945,6 +3958,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 + prefetchw_next_lru_folio(folio, head, flags); + pos =3D pos->next; new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); if (gen_increased) { delta +=3D nr_pages; --=20 2.34.1 From nobody Mon Sep 28 13:17:21 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 193C547B439 for ; Fri, 21 Aug 2026 10:26:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787308010; cv=none; b=FPWivDUTVnAfKuF1+zsypcHensDgvBn9eDABlnccmYicpLr3CsdSCCTGft70ZE9llWj2WDdcHmrr4VUnxfZQzyjG8A1dMNPxrAcO2jpwmw2s4OHB9XIHv00OzAc3B7LdsXHvuxvYatw++jXUSr/R70CwIKo8qaRF/016eGQnWoI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787308010; c=relaxed/simple; bh=pciDKRtQF6o4mfynAYkxrLdnNd52him8nReIYgXRHWE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MQCNIaVf1vQLuFRaZL1NKK/Efj/sFYWas9n08YHG50a5JZFfBjbXuZoBy7sAVc4nXbhAQQfIJjXaX/0LD9Tyr6z5LnAjLKK4E9ze5gqdmvGxl6/4LPFR/84pC7/FvY1KHOxq9uDLhOcq+FoQAR+IfRkB0qCH03pEysu+lnSwONo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BhsHa0kk; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BhsHa0kk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BB87D1F00A3F; Fri, 21 Aug 2026 10:26:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787308002; bh=Ak2P0tLlmHEpiFHwLAXYiUkDNHqN3CdnzgG9lD0dGyU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=BhsHa0kkx6JhUcbZ5bXcyCYJwYlDmAZBdMVPOphK+IvpqwJKqNth/1zrxSvSXY7WS sXWiSoqqLIFc6B2dNm0DDSBMqsafiYoTq679FQJ0N8LzF9RzV+OAunClgn/F5+ryvi uqQy3iq6dJSedc7qBtrLVBxvYMcRVWjUrsIDWIvdXyw9wsygmMRMnPryHjR8rgc4U6 HNfnskuaKTqX1a1zhNJnSAubv5xlWJQ4EN4QZt/WbPhI1GIiZca6NbEL0u0lNAq3nx 3zpt1RGo6AnzavaQWZJe62Wv0/MOx35Pqz1VkxfgpZlynxhgoBxjO+WqHrRPPPott4 paYhCxu1upNuA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" Subject: [PATCH 6/6] mm/mglru: batch move folios to the second-oldest gen's LRU Date: Fri, 21 Aug 2026 18:25:38 +0800 Message-Id: <20260821102538.22642-7-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260821102538.22642-1-baohua@kernel.org> References: <20260821102538.22642-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Detect folios that need to move from the oldest generation to the second-oldest generation, and batch-move them together. This can significantly reduce the sys time of inc_min_seq(), especially when the other type is significantly behind the preferred type. Assisted-by: gemini:gemini-3.6-flash Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Baoquan He =20 Tested-by: Xueyuan Chen --- mm/vmscan.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 2fd82b2ca4d1..996b48344ed0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3923,6 +3923,19 @@ static void clear_mm_walk(void) kfree(walk); } =20 +static inline void flush_lru_batch(struct list_head *head, struct list_hea= d **batch_end, + struct list_head *dst) +{ + LIST_HEAD(movable); + + if (!*batch_end) + return; + + list_cut_position(&movable, head, *batch_end); + list_splice_tail_init(&movable, dst); + *batch_end =3D NULL; +} + static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) { int zone; @@ -3942,9 +3955,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int t= ype, int swappiness) =20 /* prevent cold/hot inversion if the type is evictable */ for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { + struct list_head *target_list =3D &lrugen->folios[target_gen][type][zone= ]; struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; unsigned long protected[MAX_NR_TIERS] =3D {}, delta =3D 0; struct list_head *pos =3D head->next; + struct list_head *batch_end =3D NULL; =20 while (pos !=3D head) { struct folio *folio =3D list_entry(pos, struct folio, lru); @@ -3963,7 +3978,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); if (gen_increased) { delta +=3D nr_pages; - list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + batch_end =3D &folio->lru; + /* don't count the workingset being lazily promoted */ if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { int tier =3D lru_tier_from_refs(refs, workingset); @@ -3971,11 +3987,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, int swappiness) protected[tier] +=3D nr_pages; } } else { + flush_lru_batch(head, &batch_end, target_list); list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); } if (!--remaining) break; } + flush_lru_batch(head, &batch_end, target_list); + WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone], lrugen->nr_pages[old_gen][type][zone] - delta); WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone], --=20 2.34.1