From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 13C7D279DC2 for ; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=rV2Q773nI6mA7mPpst964DAKnVuR6WAnoP3SIp9kmK7VqybB937Ty0nDrl68mRl+lRYSqwSS9aMfZa3eolWBBJLE0pRAdPHYDR5CIHPahzR5rB1yYDThDlgiAMeLwGaLM2a3ud0pk+YBrNz7r6WBih36/dhZz5wOt422UdZa+AI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=pu/4aSrd7KSa0bxFp6wbeBj72fjrUtTNvGBmWMpcIZ0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=YYEaVZc52MBgIEQkfo8upx+oD5ymTvxQIDdVsrvBygsXjAPYJzGZaiV1RA+BRsdR654cRGsXvkve6q3Gh039qDApOQF9GYpKKFQSlHVs0AexA7V/XCRpLgNxycdVKJamaiYyHe2D0zgj72Ww2Ju34xPLJbyramOHH0R1x83eqa4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ouOCkUpg; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ouOCkUpg" Received: by smtp.kernel.org (Postfix) with ESMTPS id B629CC19425; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012499; bh=pu/4aSrd7KSa0bxFp6wbeBj72fjrUtTNvGBmWMpcIZ0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=ouOCkUpgJEZ8yDu5iiO4dONOFGA2rLUNyCloKovqHeNmXeKZ+dsagd8g4gDYc9rtU WUOPJWUQU1R/ITq20WeYQkIUYZf8wIkdey6qMhlsQFUSRm+YmbReqYdhkZm5d9d9eQ Dq2XQhhn4W2wlrl/j0iZDHduewjBC7ZuH0xXa4IhD7FTgZ8qPif3wsLafAyUnFlH2x L6VjIZUbpgk8pJ6ssLBpxDDnxSXUzldaVG0Xz0AKg9PHQVfFYx9CCXXUBpK+lAhhdd SuIfbgagAgeUM/JVjS14y46+I2SS1kOg2jF7D/dHxx1uFKbH3FTGgBO9xH7viJYV4Y X4Mru0iTBgnug== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3806E937EF; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:15 +0800 Subject: [PATCH v5 01/14] mm/mglru: consolidate common code for retrieving evictable size Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-1-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=3179; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=lovGNHm1LZYPq5LXk6kp/WVa26J2TdTx/242vg8RH/A=; b=mm1ALcPtxy72WxszOQTYS6hmftpu2D3p4pu7xLjkN7rS/KWxHwNJRH6AABRR63NWn1FpC9rmK NNXnEKZrk8/DyyBWhy8g+W8dHUQqU+VTYWXsLGdCmkvDAL4NTqVIQJz X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Merge commonly used code for counting evictable folios in a lruvec. No behavior change. Return unsigned long instead of long as suggested [ Axel Rasmussen ] Acked-by: Yuanchu Xie Reviewed-by: Barry Song Reviewed-by: Chen Ridong Reviewed-by: Axel Rasmussen Reviewed-by: Baolin Wang Signed-off-by: Kairui Song --- mm/vmscan.c | 36 ++++++++++++++---------------------- 1 file changed, 14 insertions(+), 22 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5a8c8fcccbfc..adc07501a137 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4084,27 +4084,33 @@ static void set_initial_priority(struct pglist_data= *pgdat, struct scan_control sc->priority =3D clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY); } =20 -static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *= sc) +static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swap= piness) { int gen, type, zone; - unsigned long total =3D 0; - int swappiness =3D get_swappiness(lruvec, sc); + unsigned long seq, total =3D 0; struct lru_gen_folio *lrugen =3D &lruvec->lrugen; - struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); DEFINE_MAX_SEQ(lruvec); DEFINE_MIN_SEQ(lruvec); =20 for_each_evictable_type(type, swappiness) { - unsigned long seq; - for (seq =3D min_seq[type]; seq <=3D max_seq; seq++) { gen =3D lru_gen_from_seq(seq); - for (zone =3D 0; zone < MAX_NR_ZONES; zone++) total +=3D max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); } } =20 + return total; +} + +static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *= sc) +{ + unsigned long total; + int swappiness =3D get_swappiness(lruvec, sc); + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + + total =3D lruvec_evictable_size(lruvec, swappiness); + /* whether the size is big enough to be helpful */ return mem_cgroup_online(memcg) ? (total >> sc->priority) : total; } @@ -4909,9 +4915,6 @@ static int evict_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, int swappiness, unsigned long *nr_to_scan) { - int gen, type, zone; - unsigned long size =3D 0; - struct lru_gen_folio *lrugen =3D &lruvec->lrugen; DEFINE_MIN_SEQ(lruvec); =20 *nr_to_scan =3D 0; @@ -4919,18 +4922,7 @@ static bool should_run_aging(struct lruvec *lruvec, = unsigned long max_seq, if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq) return true; =20 - for_each_evictable_type(type, swappiness) { - unsigned long seq; - - for (seq =3D min_seq[type]; seq <=3D max_seq; seq++) { - gen =3D lru_gen_from_seq(seq); - - for (zone =3D 0; zone < MAX_NR_ZONES; zone++) - size +=3D max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); - } - } - - *nr_to_scan =3D size; + *nr_to_scan =3D lruvec_evictable_size(lruvec, swappiness); /* better to run aging even though eviction is still possible */ return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS =3D=3D max_se= q; } --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 13BF12773C3 for ; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=EVNnktl80amCMH+cLDfGQl/I1FLQYeYOmJCNvYb685f1V7o3QCubUcAaKb568mqbkb7MXXpYKiJcbCaajU+/Vb56Q4Lx1CCFBpb9luhOoFM2wlR3pRrDlRgo3SyvfLbsXa2rOY88Wj27M6qXK1OIFD9iNgERfQt9Z06C1wKtC1E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=nf8UOchn9pxNiBANW4ySdYqzcmGuZiLDqUsnkRCZFnk=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=d/SzbtyUAmKwf/S8AeLS8GbzDLa6BNTL9vVv9xekJei1DfAuuk246YU7QLeDK9E8s71rUPM4Ash9+AujgjFHBOxZTaErpvJYmmJdMsYSqAMWJn5q5FC9AM5aUbw5+cWzaMBdRvc57/nlhfKzmUrYQzlwi3Mk4ZIX+xtrag0iPJk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ErdFdSoh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ErdFdSoh" Received: by smtp.kernel.org (Postfix) with ESMTPS id C1283C2BCB4; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012499; bh=nf8UOchn9pxNiBANW4ySdYqzcmGuZiLDqUsnkRCZFnk=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=ErdFdSohAPG7hEx5fLRcBg4LN6gXekrA5Cbwi++uy3MhUt7eNoeN2vecyo/8tTm1r vPl5xaYNgdR7waZdNGGKnr/RlV/U9z9vbnxxyN+YgGNOdIlYH8+CxjK8IcpYSiVLCC Hp3aYuJO/rrDTCtrSSyijo27RkHj6Aa8lDjTDQ2MGmRjkiiyQF97J4kneTPCMdptJw 2wLMRv8YCHFNecXOBccnFunFMw2lIjGCB3flNlFRunAiU+i9s7tFZegNKikjijmaK8 6NVhccWLH00TmdflNyr1aqRzhWoWnHJga1j/6EOyash25LidtwmxogGAQVxx0lxfIb 5etczCe3Mdpxg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3CB9E937F1; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:16 +0800 Subject: [PATCH v5 02/14] mm/mglru: rename variables related to aging and rotation Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-2-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=2994; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=uhTQOpFFZDYAXHD7SVVg5W6PNjRmTF38550AcmIU3G0=; b=rzmC+e22/YJ/zjhP/Z03E6RXgQMSbo8ah/UP53ZdGbMkSdGPfGN4T7j73W/Nt5qr44FtFOQ/q gu08pkmZOblAnf7vHx4258Vxc1WbCj7bDwRItBsoDqB/sA+a1eU7F1B X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song The current variable name isn't helpful. Make the variable names more meaningful. Only naming change, no behavior change. Suggested-by: Barry Song Reviewed-by: Baolin Wang Reviewed-by: Chen Ridong Reviewed-by: Barry Song Reviewed-by: Axel Rasmussen Signed-off-by: Kairui Song --- mm/vmscan.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index adc07501a137..f336f89a2de6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4934,7 +4934,7 @@ static bool should_run_aging(struct lruvec *lruvec, u= nsigned long max_seq, */ static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,= int swappiness) { - bool success; + bool need_aging; unsigned long nr_to_scan; struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); DEFINE_MAX_SEQ(lruvec); @@ -4942,7 +4942,7 @@ static long get_nr_to_scan(struct lruvec *lruvec, str= uct scan_control *sc, int s if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg)) return -1; =20 - success =3D should_run_aging(lruvec, max_seq, swappiness, &nr_to_scan); + need_aging =3D should_run_aging(lruvec, max_seq, swappiness, &nr_to_scan); =20 /* try to scrape all its memory if this memcg was deleted */ if (nr_to_scan && !mem_cgroup_online(memcg)) @@ -4951,7 +4951,7 @@ static long get_nr_to_scan(struct lruvec *lruvec, str= uct scan_control *sc, int s nr_to_scan =3D apply_proportional_protection(memcg, sc, nr_to_scan); =20 /* try to get away with not aging at the default priority */ - if (!success || sc->priority =3D=3D DEF_PRIORITY) + if (!need_aging || sc->priority =3D=3D DEF_PRIORITY) return nr_to_scan >> sc->priority; =20 /* stop scanning this lruvec as it's low on cold folios */ @@ -5040,7 +5040,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruve= c, struct scan_control *sc) =20 static int shrink_one(struct lruvec *lruvec, struct scan_control *sc) { - bool success; + bool need_rotate; unsigned long scanned =3D sc->nr_scanned; unsigned long reclaimed =3D sc->nr_reclaimed; struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); @@ -5058,7 +5058,7 @@ static int shrink_one(struct lruvec *lruvec, struct s= can_control *sc) memcg_memory_event(memcg, MEMCG_LOW); } =20 - success =3D try_to_shrink_lruvec(lruvec, sc); + need_rotate =3D try_to_shrink_lruvec(lruvec, sc); =20 shrink_slab(sc->gfp_mask, pgdat->node_id, memcg, sc->priority); =20 @@ -5068,10 +5068,10 @@ static int shrink_one(struct lruvec *lruvec, struct= scan_control *sc) =20 flush_reclaim_state(sc); =20 - if (success && mem_cgroup_online(memcg)) + if (need_rotate && mem_cgroup_online(memcg)) return MEMCG_LRU_YOUNG; =20 - if (!success && lruvec_is_sizable(lruvec, sc)) + if (!need_rotate && lruvec_is_sizable(lruvec, sc)) return 0; =20 /* one retry if offlined or too small */ --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1A62A2C0F6D for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=GYb2VZQ0WqlIh0cEPo3kVkwRtOVmvwEWynH/MMkaDUeMBTTIjzskZ1i5L3EcrOFqlUDBmrVDAs2tg89ycmil7KSNwz9PXAToGH81CZvI+xfuCyDgom0oYS/gkZXnsaJlLyok/SjjAh0TE9xP9vndoU6YGBfU2s/UnnRotskfi3I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=vGlaz9ZYHgWaT2xiL6hluWz9quld465IwkSQuDQSsOE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=J1UjEei9ZTtqeEhicYHuM6iTWuGrv8JujfJ9SY6y9lxfYEzNvTjitPkDpu69oXoaJHgBFT+2SdVO7Or9EQLWm3ymSXM9eg1KE0Srn00MTEQTXqMRakT8lZRk43SrjUTCrk0n2Ebp9vq6KmDgw1CfQEoKZRXOjjk1P7/EkDeX/gw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=q0feVht/; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="q0feVht/" Received: by smtp.kernel.org (Postfix) with ESMTPS id D0AEEC2BCB8; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012499; bh=vGlaz9ZYHgWaT2xiL6hluWz9quld465IwkSQuDQSsOE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=q0feVht/ieYfQHfz9VRnXiJA/cAI0LUwjcnksJPb00cw7UPqPlhXrKJKv2uV/fPPZ 9bPtWlAG3Gj4B69O/K0yUH4+L83Vk5kWqiEEju43afbtaz10XT8AH5w2v05gaHgUja LRieqtqXLAnABFOef8vkbLYcDjsZo4VKiy+7VFG4b9MoN8pqqNihDBg4Um6dXr2jcQ LcuwcWWl6kkMgN6Ih3yxKpO2hQLFnlWbTZZ5MhyAtFzWvFQlmWMizWcIdwAHukisT5 TlO22329UC28IiNukLf5CvPu9UFTERX1fGDOMlrNfNVMlI6qqfteENuBWpAPpk2RO4 N90NQS0Pua5Zg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id C48BEE937EE; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:17 +0800 Subject: [PATCH v5 03/14] mm/mglru: relocate the LRU scan batch limit to callers Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-3-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=3315; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=d+i0IK9czNinxb4Rr8UJfKR3t+PD4LIQm30TVC0a5qc=; b=lESOxzGvSKeWuGXwB306fB+1FmM/2Ymy05a3kDTMVQVruVdQxeiH/KjSYudYr9DXjVhQTlSMy w5OEQhOVtlHCXjdzqpUI78/2n8P/JyjJGot+EBThbLAFGbLn93L+LyB X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Same as active / inactive LRU, MGLRU isolates and scans folios in batches. The batch split is done hidden deep in the helper, which makes the code harder to follow. The helper's arguments are also confusing since callers usually request more folios than the batch size, so the helper almost never processes the full requested amount. Move the batch splitting into the top loop to make it cleaner, there should be no behavior change. Reviewed-by: Axel Rasmussen Reviewed-by: Baolin Wang Reviewed-by: Barry Song Reviewed-by: Chen Ridong Signed-off-by: Kairui Song --- mm/vmscan.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index f336f89a2de6..963362523782 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4695,10 +4695,10 @@ static int scan_folios(unsigned long nr_to_scan, st= ruct lruvec *lruvec, int scanned =3D 0; int isolated =3D 0; int skipped =3D 0; - int scan_batch =3D min(nr_to_scan, MAX_LRU_BATCH); - int remaining =3D scan_batch; + unsigned long remaining =3D nr_to_scan; struct lru_gen_folio *lrugen =3D &lruvec->lrugen; =20 + VM_WARN_ON_ONCE(nr_to_scan > MAX_LRU_BATCH); VM_WARN_ON_ONCE(!list_empty(list)); =20 if (get_nr_gens(lruvec, type) =3D=3D MIN_NR_GENS) @@ -4751,7 +4751,7 @@ static int scan_folios(unsigned long nr_to_scan, stru= ct lruvec *lruvec, mod_lruvec_state(lruvec, item, isolated); mod_lruvec_state(lruvec, PGREFILL, sorted); mod_lruvec_state(lruvec, PGSCAN_ANON + type, isolated); - trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, scan_batch, + trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, nr_to_scan, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); if (type =3D=3D LRU_GEN_FILE) @@ -4987,7 +4987,7 @@ static bool should_abort_scan(struct lruvec *lruvec, = struct scan_control *sc) =20 static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_contro= l *sc) { - long nr_to_scan; + long nr_batch, nr_to_scan; unsigned long scanned =3D 0; int swappiness =3D get_swappiness(lruvec, sc); =20 @@ -4998,7 +4998,8 @@ static bool try_to_shrink_lruvec(struct lruvec *lruve= c, struct scan_control *sc) if (nr_to_scan <=3D 0) break; =20 - delta =3D evict_folios(nr_to_scan, lruvec, sc, swappiness); + nr_batch =3D min(nr_to_scan, MAX_LRU_BATCH); + delta =3D evict_folios(nr_batch, lruvec, sc, swappiness); if (!delta) break; =20 @@ -5623,6 +5624,7 @@ static int run_aging(struct lruvec *lruvec, unsigned = long seq, static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct s= can_control *sc, int swappiness, unsigned long nr_to_reclaim) { + int nr_batch; DEFINE_MAX_SEQ(lruvec); =20 if (seq + MIN_NR_GENS > max_seq) @@ -5639,8 +5641,8 @@ static int run_eviction(struct lruvec *lruvec, unsign= ed long seq, struct scan_co if (sc->nr_reclaimed >=3D nr_to_reclaim) return 0; =20 - if (!evict_folios(nr_to_reclaim - sc->nr_reclaimed, lruvec, sc, - swappiness)) + nr_batch =3D min(nr_to_reclaim - sc->nr_reclaimed, MAX_LRU_BATCH); + if (!evict_folios(nr_batch, lruvec, sc, swappiness)) return 0; =20 cond_resched(); --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23FDB2D131D for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=gu+2tdEbvj/Awa5dwmQemWsRMR3q4UDRH54rcLRtLF3HP6e0KjCfY7NsV5B/z9Hy9D5K5Mx+RRJ1i9jOa/Op1dzJyDuErnZ07l4rkmJ6pSVY5neMrwiB87GsyLK3San4Kzt9vRL+uLtpWp9bgXj44MM5sWGf/hwRZ91Q8EvcP58= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=ZqiSG/dFaxT5Oke0XohOS08i2hQITeY46Sc9nlsZp3Q=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=d2g1hSxVBj3op8oHYglACCAimlSVEyvS7SECPrkYqhU3tPTp9L8dRc3avnOhebzhQrF/jRDmLctoav/GJ12jxTveI6Dlo8zM+3s8k2r/op6VsDz+pKyDQB3uMfjpth6/BhulGk028LTBNXHnvVs/lZa3Odb4xMv8oF2KFcN7s7s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nbWSRgn5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nbWSRgn5" Received: by smtp.kernel.org (Postfix) with ESMTPS id DE248C2BCC7; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=ZqiSG/dFaxT5Oke0XohOS08i2hQITeY46Sc9nlsZp3Q=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=nbWSRgn52hS0p/e3xA/vpCjVn3ERWz0K4S6kdTVMS0FiTayAcCvMjqyQ/BXRSmSRa FyHRt9VWvVImDyq/nNz+bNSzfhiKUTZVoG3hEvuRF4rjrPmO0jC7iJLTyABWsvmdG6 aVOrOFfSbQeKrLvLqAG7+at95Y1QAVMsnJZvpn4+aTJUG0dWS2xAWuef9YuulS8GtX eQ7wDAnQIBbfdD+aG+mOtLiuTyaezrNMTzbTNsNoVL6ShXsnXcWIvRSj08R0oiOugj hEx/lQEAg6wxvN+65eosHSXlkrgpCrKbVxTiwDAhOsfyy8oy4lLThgBPWXOS0kdwHp d49nNh6Q9xBZg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id D52D3E937EF; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:18 +0800 Subject: [PATCH v5 04/14] mm/mglru: restructure the reclaim loop Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-4-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=6467; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=IOjfs+eM5SwXYK7QI7na78x/g82TQo2VWPiMfWlsmRI=; b=fenmnPRtbv8GO4DGCvEzdmnmnC/HdGGj7ugg3b5IHXZSYRJ0qivw3G0Paj4SyCYgB+Isfudm6 nDxeL3LBm6wD4+rJWHNsGFacY7W+pfORdJGWtNc3PiT1evq7ih+9mbK X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song The current loop will calculate the scan number on each iteration. The number of folios to scan is based on the LRU length, with some unclear behaviors, eg, the scan number is only shifted by reclaim priority when aging is not needed or when at the default priority, and it couples the number calculation with aging and rotation. Adjust, simplify it, and decouple aging and rotation. Just calculate the scan number for once at the beginning of the reclaim, always respect the reclaim priority, and make the aging and rotation more explicit. This slightly changes how aging and offline memcg reclaim works: Previously, aging was always skipped at DEF_PRIORITY even when eviction was impossible. Now, aging is always triggered when it is necessary to make progress. The old behavior may waste a reclaim iteration only to escalate priority, potentially causing over-reclaim of slab and breaking reclaim balance in multi-cgroup setups. Similar for offline memcg. Previously, offline memcg wouldn't be aged unless it didn't have any evictable folios. Now, we might age it if it has only 3 generations and the reclaim priority is less than DEF_PRIORITY, which should be fine. On one hand, offline memcg might still hold long-term folios, and in fact, a long-existing offline memcg must be pinned by some long-term folios like shmem. These folios might be used by other memcg, so aging them as ordinary memcg seems correct. Besides, aging enables further reclaim of an offlined memcg, which will certainly happen if we keep shrinking it. And offline memcg might soon be no longer an issue with reparenting. And while at it, make it clear that unevictable memcg will get rotated so following reclaim will more likely to skip them, as a optimization. And apply a minimal batch factor when reclaim is running with higher priority. Overall, the memcg LRU rotation, as described in mmzone.h, remains the same. Reviewed-by: Axel Rasmussen Signed-off-by: Kairui Song --- mm/vmscan.c | 72 +++++++++++++++++++++++++++++++++------------------------= ---- 1 file changed, 39 insertions(+), 33 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 963362523782..d4aaaa62056d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4913,49 +4913,41 @@ static int evict_folios(unsigned long nr_to_scan, s= truct lruvec *lruvec, } =20 static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, - int swappiness, unsigned long *nr_to_scan) + struct scan_control *sc, int swappiness) { DEFINE_MIN_SEQ(lruvec); =20 - *nr_to_scan =3D 0; /* have to run aging, since eviction is not possible anymore */ if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq) return true; =20 - *nr_to_scan =3D lruvec_evictable_size(lruvec, swappiness); + /* try to get away with not aging at the default priority */ + if (sc->priority =3D=3D DEF_PRIORITY) + return false; + /* better to run aging even though eviction is still possible */ return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS =3D=3D max_se= q; } =20 -/* - * For future optimizations: - * 1. Defer try_to_inc_max_seq() to workqueues to reduce latency for memcg - * reclaim. - */ -static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,= int swappiness) +static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, + struct mem_cgroup *memcg, int swappiness) { - bool need_aging; - unsigned long nr_to_scan; - struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); - DEFINE_MAX_SEQ(lruvec); + unsigned long nr_to_scan, evictable; =20 - if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg)) - return -1; - - need_aging =3D should_run_aging(lruvec, max_seq, swappiness, &nr_to_scan); + evictable =3D lruvec_evictable_size(lruvec, swappiness); + nr_to_scan =3D evictable; =20 /* try to scrape all its memory if this memcg was deleted */ - if (nr_to_scan && !mem_cgroup_online(memcg)) + if (!mem_cgroup_online(memcg)) return nr_to_scan; =20 nr_to_scan =3D apply_proportional_protection(memcg, sc, nr_to_scan); + nr_to_scan >>=3D sc->priority; =20 - /* try to get away with not aging at the default priority */ - if (!need_aging || sc->priority =3D=3D DEF_PRIORITY) - return nr_to_scan >> sc->priority; + if (!nr_to_scan && sc->priority < DEF_PRIORITY) + nr_to_scan =3D min(evictable, SWAP_CLUSTER_MAX); =20 - /* stop scanning this lruvec as it's low on cold folios */ - return try_to_inc_max_seq(lruvec, max_seq, swappiness, false) ? -1 : 0; + return nr_to_scan; } =20 static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *= sc) @@ -4985,31 +4977,46 @@ static bool should_abort_scan(struct lruvec *lruvec= , struct scan_control *sc) return true; } =20 +/* + * For future optimizations: + * 1. Defer try_to_inc_max_seq() to workqueues to reduce latency for memcg + * reclaim. + */ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_contro= l *sc) { + bool need_rotate =3D false; long nr_batch, nr_to_scan; - unsigned long scanned =3D 0; int swappiness =3D get_swappiness(lruvec, sc); + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + + nr_to_scan =3D get_nr_to_scan(lruvec, sc, memcg, swappiness); + if (!nr_to_scan) + need_rotate =3D true; =20 - while (true) { + while (nr_to_scan > 0) { int delta; + DEFINE_MAX_SEQ(lruvec); =20 - nr_to_scan =3D get_nr_to_scan(lruvec, sc, swappiness); - if (nr_to_scan <=3D 0) + if (mem_cgroup_below_min(sc->target_mem_cgroup, memcg)) { + need_rotate =3D true; break; + } + + if (should_run_aging(lruvec, max_seq, sc, swappiness)) { + if (try_to_inc_max_seq(lruvec, max_seq, swappiness, false)) + need_rotate =3D true; + break; + } =20 nr_batch =3D min(nr_to_scan, MAX_LRU_BATCH); delta =3D evict_folios(nr_batch, lruvec, sc, swappiness); if (!delta) break; =20 - scanned +=3D delta; - if (scanned >=3D nr_to_scan) - break; - if (should_abort_scan(lruvec, sc)) break; =20 + nr_to_scan -=3D delta; cond_resched(); } =20 @@ -5035,8 +5042,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruve= c, struct scan_control *sc) reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK); } =20 - /* whether this lruvec should be rotated */ - return nr_to_scan < 0; + return need_rotate; } =20 static int shrink_one(struct lruvec *lruvec, struct scan_control *sc) --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3AB192EB87F for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=sgJVaCO4mwPzzmOfI0MVI57HOvmNRNFPUL0LVurCBh5AlwBC/MuvMtSpmbKzwi01j0gzZ9LCT8OLNohpuv98dWG82wC0fH3APhu9NAIrccfFocgQk+GtfMKLfnLIO6wPm+vyGCV2LiNkB/kJke7WRlXQNLuMCVk1OlvFKRXuCSk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=NjTAqTMMdZ1d10EgLz4zAuMCvDORj15wAon0rP9Z3vQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Ojc9wdIeqnpDb7xxmuuE8uDqGc/4r/RGhy1eLNrX84CymbI7n06B0pop43031gGY7A84PJPMPTrh/RqwB9J05r7WLtpKmlaYP8CMT7DSjCEVvvA6EiWTvkzdmsrIaObeMgxntxGSNd+J/TlR9etio3UCY7jE/P8Df0d9ATm2dQM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Mq3bfUdU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Mq3bfUdU" Received: by smtp.kernel.org (Postfix) with ESMTPS id F01E4C2BCF6; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=NjTAqTMMdZ1d10EgLz4zAuMCvDORj15wAon0rP9Z3vQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=Mq3bfUdUiEw9ytI3zBjE1N8QQMKIGKZmBgA59X3EGG7dTU+LgHfDTZso/EqvZPwov jJdiRlPolWI/IoIs6azvPLsE3Zy+P0U4v8I/BS1tifWkk2XTIr/hOtddxP0ZRcNRXN EV8tQE5wYp/8z8RxQXBumzgLagYlIN/YK6/gzB6Rpd1CV3/TwLCBki+XQ1bWbl1q/g fSqMUNJvGmVodfTj1qWrHR8IzzhsFAsfWKlFrTQ3yuquQtv9klWqtXY+9A5UlWHAUa bOVkVBYnDUhQbcX5j5HSblAbCtfH8p5NLyfBiQfZKP+uHvLQlmT4hb6ToBtsdp5xBB w+lxNM94WwLzQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6F87E937EB; Sun, 12 Apr 2026 16:48:19 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:19 +0800 Subject: [PATCH v5 05/14] mm/mglru: scan and count the exact number of folios Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-5-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=7191; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=+lNvcxMchl8+IGJt1lhbYz1RQNKHl1cEaDndT2wk7QA=; b=FH2asZ4t2q0DEJKYPRRpyHAPsCtYa07ehnRjHXa8XSN4VQyzPjeihP/52cXVth9KX6Fg90x8g pnvKm0wqYo+CFFQrvFASzZa3lhOte3ut1BlAN9BHJDdqcVTjfFRbkKN X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Make the scan helpers return the exact number of folios being scanned or isolated. Since the reclaim loop now has a natural scan budget that controls the scan progress, returning the scan number and consume the budget should make the scan more accurate and easier to follow. The number of scanned folios for each iteration is always positive and larger than 0, unless the reclaim must stop for a forced aging, so there is no more need for any special handling when there is no progress made: - `return isolated || !remaining ? scanned : 0` in scan_folios: both the function and the call now just return the exact scan count, combined with the scan budget introduced in the previous commit to avoid livelock or under scan. - `scanned +=3D try_to_inc_min_seq` in evict_folios: adding a bool as a scan count was kind of confusing and no longer needed to, as scan number should never be zero as long as there are still evictable gens. We may encounter a empty old gen that return 0 scan count, to avoid that, do a try_to_inc_min_seq before isolation which have slight to none overhead in most cases. - `evictable_min_seq + MIN_NR_GENS > max_seq` guard in evict_folios: the per-type get_nr_gens =3D=3D MIN_NR_GENS check in scan_folios naturally returns 0 when only two gens remain and breaks the loop. Also change try_to_inc_min_seq to return void, as its return value is no longer used by any caller. Move the call before isolate_folios so that any empty gens created by external folio freeing are flushed, and add another call after isolate_folios to also flush empty gens that isolation itself may create. The scan still stops if there are only two gens left as the scan number will be zero, this behavior is same as before. This force gen protection may get removed or softened later to improve the reclaim a bit more. Reviewed-by: Axel Rasmussen Reviewed-by: Chen Ridong Signed-off-by: Kairui Song Reviewed-by: Baolin Wang --- mm/vmscan.c | 60 ++++++++++++++++++++++++++++++---------------------------= --- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d4aaaa62056d..e3b68b008376 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3878,10 +3878,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int t= ype, int swappiness) return true; } =20 -static bool try_to_inc_min_seq(struct lruvec *lruvec, int swappiness) +static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness) { int gen, type, zone; - bool success =3D false; bool seq_inc_flag =3D false; struct lru_gen_folio *lrugen =3D &lruvec->lrugen; DEFINE_MIN_SEQ(lruvec); @@ -3907,11 +3906,10 @@ static bool try_to_inc_min_seq(struct lruvec *lruve= c, int swappiness) =20 /* * If min_seq[type] of both anonymous and file is not increased, - * we can directly return false to avoid unnecessary checking - * overhead later. + * return here to avoid unnecessary checking overhead later. */ if (!seq_inc_flag) - return success; + return; =20 /* see the comment on lru_gen_folio */ if (swappiness && swappiness <=3D MAX_SWAPPINESS) { @@ -3929,10 +3927,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec= , int swappiness) =20 reset_ctrl_pos(lruvec, type, true); WRITE_ONCE(lrugen->min_seq[type], min_seq[type]); - success =3D true; } - - return success; } =20 static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swap= piness) @@ -4686,7 +4681,7 @@ static bool isolate_folio(struct lruvec *lruvec, stru= ct folio *folio, struct sca =20 static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, int type, int tier, - struct list_head *list) + struct list_head *list, int *isolatedp) { int i; int gen; @@ -4756,11 +4751,9 @@ static int scan_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); if (type =3D=3D LRU_GEN_FILE) sc->nr.file_taken +=3D isolated; - /* - * There might not be eligible folios due to reclaim_idx. Check the - * remaining to prevent livelock if it's not making progress. - */ - return isolated || !remaining ? scanned : 0; + + *isolatedp =3D isolated; + return scanned; } =20 static int get_tier_idx(struct lruvec *lruvec, int type) @@ -4804,33 +4797,36 @@ static int get_type_to_scan(struct lruvec *lruvec, = int swappiness) =20 static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, int swappiness, - int *type_scanned, struct list_head *list) + struct list_head *list, int *isolated, + int *isolate_type, int *isolate_scanned) { int i; + int scanned =3D 0; int type =3D get_type_to_scan(lruvec, swappiness); =20 for_each_evictable_type(i, swappiness) { - int scanned; + int type_scan; int tier =3D get_tier_idx(lruvec, type); =20 - *type_scanned =3D type; + type_scan =3D scan_folios(nr_to_scan, lruvec, sc, + type, tier, list, isolated); =20 - scanned =3D scan_folios(nr_to_scan, lruvec, sc, type, tier, list); - if (scanned) - return scanned; + scanned +=3D type_scan; + if (*isolated) { + *isolate_type =3D type; + *isolate_scanned =3D type_scan; + break; + } =20 type =3D !type; } =20 - return 0; + return scanned; } =20 static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, int swappiness) { - int type; - int scanned; - int reclaimed; LIST_HEAD(list); LIST_HEAD(clean); struct folio *folio; @@ -4838,19 +4834,23 @@ static int evict_folios(unsigned long nr_to_scan, s= truct lruvec *lruvec, enum node_stat_item item; struct reclaim_stat stat; struct lru_gen_mm_walk *walk; + int scanned, reclaimed; + int isolated =3D 0, type, type_scanned; bool skip_retry =3D false; - struct lru_gen_folio *lrugen =3D &lruvec->lrugen; struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); =20 lruvec_lock_irq(lruvec); =20 - scanned =3D isolate_folios(nr_to_scan, lruvec, sc, swappiness, &type, &li= st); + /* In case folio deletion left empty old gens, flush them */ + try_to_inc_min_seq(lruvec, swappiness); =20 - scanned +=3D try_to_inc_min_seq(lruvec, swappiness); + scanned =3D isolate_folios(nr_to_scan, lruvec, sc, swappiness, + &list, &isolated, &type, &type_scanned); =20 - if (evictable_min_seq(lrugen->min_seq, swappiness) + MIN_NR_GENS > lrugen= ->max_seq) - scanned =3D 0; + /* Isolation might create empty gen, flush them */ + if (scanned) + try_to_inc_min_seq(lruvec, swappiness); =20 lruvec_unlock_irq(lruvec); =20 @@ -4861,7 +4861,7 @@ static int evict_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, sc->nr.unqueued_dirty +=3D stat.nr_unqueued_dirty; sc->nr_reclaimed +=3D reclaimed; trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, - scanned, reclaimed, &stat, sc->priority, + type_scanned, reclaimed, &stat, sc->priority, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); =20 list_for_each_entry_safe_reverse(folio, next, &list, lru) { --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A5722E8DEB for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=On2wEXUqQs4K16p1Yt/Qx3jcPjnL9ZQ7OTSTdI98v/YZ4Cx0Dx+j68Tx0mYA2d22JDFuvDABxkGUZlPVeehk+B7unXfwrtEdemkvf3S1xHs3sJJChuhzG5ASE+TudjtaZBW66k8/Zp5YOj0y5iNCkx2qnPeA8QBg/PlWixuMOHU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=DH43CHmYFrRpRdSTPsxFpCPHd33dckhbiin1ffPiNag=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=PbsZ5IYPEggYjCbrTiNzGvJLLmvR55Gnnl/BtI5/cX4SOjGSjIjyomiUXzw24v9WjkMzzNcOKkTRaA3PoFQp3V5neZ4InyITTaAOaRqFhs+9D0+bDotGErD6LbRt1gJDclCW0uTKitK/BbMfOebFmNT7+HVEPLPgdDY2+yAUeVU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=vJLEa8i+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="vJLEa8i+" Received: by smtp.kernel.org (Postfix) with ESMTPS id 0B169C2BCFB; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=DH43CHmYFrRpRdSTPsxFpCPHd33dckhbiin1ffPiNag=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=vJLEa8i+aX8QvIlehLGwxOg3T3CMhEFRRo4Co9voOpreBJQsfKcgwlbhKrQSZUDMx uhaNG89qFCwoDcsMVU9aVQ3kgQYirSdzFPGUXwXfiRH0pWpk5RJ/FO8yvJ2iCh7sPs kkKZrgM2NVPzqWBSzS4JeF/CxUEYQBAW+BSunhtTpbxuilrrKiIhhEgq9go3AUh3D1 wquV3yrI3X6+6iMDo1vT3+VpGl4oGE7CUVfpPQ23Z/tRPRYjkgXDLvdb9F/tgX/gWQ wEUXl1KQSpYgp6d/eB8Mg6BUEvJZAJFSB/0/bIdPXA/YwqVl0XT/p5TrmtPcTrfQnd HVQYP7J5/m4qg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03271E937F1; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:20 +0800 Subject: [PATCH v5 06/14] mm/mglru: use a smaller batch for reclaim Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-6-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=1039; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=cCINFuu/NooJEcPm2HlqwvibVWHBQhV3ICWUZupnUcY=; b=czGn94K9K1jdbOOkNLS5zct45nApfHrU6B0v59GrTBrrU4MfT0KsnQXq5eAQKD5I6PEbmH2s9 k0ysFGD0Hs7CuvfJB1cjGcleNjQBgcv8FdK8u54pnz8ntAZZJzIzIHz X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song With a fixed number to reclaim calculated at the beginning, making each following step smaller should reduce the lock contention and avoid over-aggressive reclaim of folios, as it will abort earlier when the number of folios to be reclaimed is reached. Reviewed-by: Axel Rasmussen Reviewed-by: Chen Ridong Reviewed-by: Baolin Wang Reviewed-by: Barry Song Signed-off-by: Kairui Song --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e3b68b008376..cfb10052e350 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -5008,7 +5008,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruve= c, struct scan_control *sc) break; } =20 - nr_batch =3D min(nr_to_scan, MAX_LRU_BATCH); + nr_batch =3D min(nr_to_scan, MIN_LRU_BATCH); delta =3D evict_folios(nr_batch, lruvec, sc, swappiness); if (!delta) break; --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FDAE2F0C74 for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=R3FETK0e7mprozoLELlUWEx7oUwj3CVks6mNftVwP7sEVJnDIor2jmXf+dsDStVjdRYXc0jYNeKDNR/xAgc2fmzQrO5vKSWrgVADWBn5nkafXmaO5uVIx1iJfxLlsmoc8zAfR+FEif5WZ8yT5WiZxdFW9RC9fFnSAqzlpczGMOs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=dA6z1QpGlnN4t7Dr9dhJIinRiuAH8kuko+RPRulPJfA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=uQwyumSpQulmayXEox8FqBdp1Zyy16CdibbGbryNT5KiRLve1ez1FG/HlVLG3R8GHwdW/UxAa0XZugJU1LxsuOiIBe14eo182PHTx1FkNKVem7j5FoRX3xRg2dy8WrgPzSAx9Z9VL6OlXt4SraUxxdgWtc2ZNG9d6WLQ+ksIMjg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tJ1SD7T6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tJ1SD7T6" Received: by smtp.kernel.org (Postfix) with ESMTPS id 1CC60C2BD00; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=dA6z1QpGlnN4t7Dr9dhJIinRiuAH8kuko+RPRulPJfA=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=tJ1SD7T6OvB+ndRzlzLogjl2YR8a3FpkeP5dXARSV3QUtYRK2+pt74WYHCdONIQ6j Pkq4v916u/Gi2hEigWAxbshDAZc3XdIgUIWGM2RPrO4LDfM3wmSbaJo/CbtfNCN/TZ LR007nO/IRDDgt5YwqQI/C3rrYOJ7tFVXflf9p/u54KAereCACJqW/Jy2zRGypaNnd eDGU9SvssVIAiAWyFVgGLKK3E+nu5d44TLUTccFoloJrnX3AAE3l6S0bFnI6fHZP3x fvog4OkNcV2nnRNHZdVKwC5FJy2G8MriCVYpKDNyA9qdPcgU+r78Vq5JWucllqVCWc sVXUFDH8n9Szg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1344AE937EE; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:21 +0800 Subject: [PATCH v5 07/14] mm/mglru: don't abort scan immediately right after aging Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-7-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=3560; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=eyt/3rbFbCzIlzaC3Pqb1f8uETg5r3owZ/Vsdk0rmM4=; b=gL76/idDwn3TVYTnnU90LhS9xqDoJjdRmG8OzvLF9RvRZlwosZiWHI8FvIJjTp73sFfpebQOr SpneHHQ1n0WAM0qSM4L5x9xfhNHzWWdLXHoGbpsVbclp17uapXjDIDV X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Right now, if eviction triggers aging, the reclaimer will abort. This is not the optimal strategy for several reasons. Aborting the reclaim early wastes a reclaim cycle when under pressure, and for concurrent reclaim, if the LRU is under aging, all concurrent reclaimers might fail. And if the age has just finished, new cold folios exposed by the aging are not reclaimed until the next reclaim iteration. What's more, the current aging trigger is quite lenient, having 3 gens with a reclaim priority lower than default will trigger aging, and blocks reclaiming from one memcg. This wastes reclaim retry cycles easily. And in the worst case, if the reclaim is making slower progress and all following attempts fail due to being blocked by aging, it triggers unexpected early OOM. And if a lruvec requires aging, it doesn't mean it's hot. Instead, the lruvec could be idle for quite a while, and hence it might contain lots of cold folios to be reclaimed. While it's helpful to rotate memcg LRU after aging for global reclaim, as global reclaim fairness is coupled with the rotation in shrink_many, memcg fairness is instead handled by cgroup iteration in shrink_node_memcgs. So, for memcg level pressure, this abort is not the key part for keeping the fairness. And in most cases, there is no need to age, and fairness must be achieved by upper-level reclaim control. So instead, just keep the scanning going unless one whole batch of folios failed to be isolated or enough folios have been scanned, which is triggered by evict_folios returning 0. And only abort for global reclaim after one batch, so when there are fewer memcgs, progress is still made, and the fairness mechanism described above still works fine. And in most cases, the one more batch attempt for global reclaim might just be enough to satisfy what the reclaimer needs, hence improving global reclaim performance by reducing reclaim retry cycles. Rotation is still there after the reclaim is done, which still follows the comment in mmzone.h. And fairness still looking good. Reviewed-by: Axel Rasmussen Reviewed-by: Chen Ridong Signed-off-by: Kairui Song Reviewed-by: Barry Song --- mm/vmscan.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index cfb10052e350..5ae6bd967b17 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4984,7 +4984,7 @@ static bool should_abort_scan(struct lruvec *lruvec, = struct scan_control *sc) */ static bool try_to_shrink_lruvec(struct lruvec *lruvec, struct scan_contro= l *sc) { - bool need_rotate =3D false; + bool need_rotate =3D false, should_age =3D false; long nr_batch, nr_to_scan; int swappiness =3D get_swappiness(lruvec, sc); struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); @@ -5005,7 +5005,7 @@ static bool try_to_shrink_lruvec(struct lruvec *lruve= c, struct scan_control *sc) if (should_run_aging(lruvec, max_seq, sc, swappiness)) { if (try_to_inc_max_seq(lruvec, max_seq, swappiness, false)) need_rotate =3D true; - break; + should_age =3D true; } =20 nr_batch =3D min(nr_to_scan, MIN_LRU_BATCH); @@ -5016,6 +5016,10 @@ static bool try_to_shrink_lruvec(struct lruvec *lruv= ec, struct scan_control *sc) if (should_abort_scan(lruvec, sc)) break; =20 + /* For cgroup reclaim, fairness is handled by iterator, not rotation */ + if (root_reclaim(sc) && should_age) + break; + nr_to_scan -=3D delta; cond_resched(); } --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F7ED2FA0DF for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=tpl9dwY1ZxdjwsI6+ZsfmF7hB6eJ3Pvf0lKtfDLOvkihPoX4kTLocWa0idPtM1/vXedMABEYgRkw6ohOoCNjckUgZ+mK5HIfYOAHG4N0RDkpUnF+Ju0t+LmyIlkdc0Hn/MWxcWQe11PvcrM0FJtni6SbvTYsIuMf2LFVdVMG6ic= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=wr+hcSLPsR0JuxYVFW5Y7utpTZ/YZawl7DSSuajsq8o=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=gk0iDuY4OtV9EnRNb6+fwM45dZjE1PVitM68K8S6p1TTIy2/Bi9wPAhKD0mtdK393cBEZkR3Fh9mrRj7NVnrr/bCad8zTP4kIn4yqkfPOiUKxnfPFG/+wXQNCUZT6Musx9xb0Kq4FXOfYeJRnNj5MX4DL9v2dcm8HpFi1iMJBYQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mYy687BH; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mYy687BH" Received: by smtp.kernel.org (Postfix) with ESMTPS id 2BE1EC32782; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=wr+hcSLPsR0JuxYVFW5Y7utpTZ/YZawl7DSSuajsq8o=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=mYy687BH1F0ueUspoQ/h32t9dNYUtRyjC1wRovjKhePzwbRLzKvlWZZABBZ8nfdK6 gLHML5J0qtpIJrsvGhQqjgvSbeZLIuO4IMUTu8LxAUlB0jEtHeY4VYfruO3EFQd/ZR UVgn8DNaXQgBGw4tgrmFq3T6HPwNi+/g+M4+Fc92/kRsmj28n9R6UqZz47mlVoTwuT xe8lUIIX5KGbiIMKuB9PnCwPMLiFc7oUkmnKWMI+b7vc2irUjPzRqmgRD25AggmA87 KQxPc/W7Os1o22Lq4VvIamZC2xFsf/Gffw9io9MN6gPvrM7bMZheyX6Ve7p/AHno2C x9JifYCsVl1RQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22A00E937F2; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:22 +0800 Subject: [PATCH v5 08/14] mm/mglru: remove redundant swap constrained check upon isolation Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-8-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=1702; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=MVC92Hy0dj5HALuLJgtd87f2adt8BuCL8uRItTw3vYU=; b=BdHwbQIBiS2i+6n+htUoGzfEswBTxEHOZJMTtoq75+ktgZsKvBSPgTLapV7ybRUpnA283Yuie ZlIaTuOtMNsDqzFxCpq1BrikWK5mt3l0WkfIiceQmkSfLAl43zkKJiP X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Remove the swap-constrained early reject check upon isolation. This check is a micro optimization when swap IO is not allowed, so folios are rejected early. But it is redundant and overly broad since shrink_folio_list() already handles all these cases with proper granularity. Notably, this check wrongly rejected lazyfree folios, and it doesn't cover all rejection cases. shrink_folio_list() uses may_enter_fs(), which distinguishes non-SWP_FS_OPS devices from filesystem-backed swap and does all the checks after folio is locked, so flags like swap cache are stable. This check also covers dirty file folios, which are not a problem now since sort_folio() already bumps dirty file folios to the next generation, but causes trouble for unifying dirty folio writeback handling. And there should be no performance impact from removing it. We may have lost a micro optimization, but unblocked lazyfree reclaim for NOIO contexts, which is not a common case in the first place. Reviewed-by: Axel Rasmussen Signed-off-by: Kairui Song Reviewed-by: Baolin Wang Reviewed-by: Barry Song Reviewed-by: Chen Ridong --- mm/vmscan.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5ae6bd967b17..965d8905a4fe 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4650,12 +4650,6 @@ static bool isolate_folio(struct lruvec *lruvec, str= uct folio *folio, struct sca { bool success; =20 - /* swap constrained */ - if (!(sc->gfp_mask & __GFP_IO) && - (folio_test_dirty(folio) || - (folio_test_anon(folio) && !folio_test_swapcache(folio)))) - return false; - /* raced with release_pages() */ if (!folio_try_get(folio)) return false; --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A6702FC89C for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=C4r294NSM1JJKHVOzayOaxgkJueXMOR6JCdKVFoFliHqjZ1kXEC7FpjE989rX9GlMDaR66DHYKu7fmOVsmSH8YkM4+jtcGG67bu7X18COJyZhTRfs9w+nZR2pmda1eEfOCY1qzsF9qd6kl7vpjCIjy018Y3Uwc7brx2TVwv4OW0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=3RE8cIiuwEqnweS2KmT3lOO/ndLGSLdeBknMMEpJdGU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=VSVh7wo1xhrIy1nnED9cQMXXNileWAmVFE7gQJNmpBITb1DyU58ngJYkvc+CqhqvTIiPuLywPq6Fi614PVbR5edVUePGtuhzVrUWxy+m9ZwALloBlkqoE+7NIqOwPz4R2FAt/KZtJ15cmb0egoWi9xLk6Loz0LZHj9XZiurvwzU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WuR1xmq6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WuR1xmq6" Received: by smtp.kernel.org (Postfix) with ESMTPS id 3B81EC4AF18; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=3RE8cIiuwEqnweS2KmT3lOO/ndLGSLdeBknMMEpJdGU=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=WuR1xmq69ig1SziU0qHbhRF88CtGLsmJdyPEC7m4dEPXIS0/Lkwfmx7fL+aMqhehV o4kRNCFs1CtNx4Vo1HjbMXignQt+p7WAjGTkcDEiT4JEh3uqEvjjVHoYuRco1sZA6l PE2yK4IgsOz5GJSrNuXwHUBHzfUQFi67dzGzf5Hcj/mAv0M+CiP7XKaP6sOY+cK81s kdrw3lBTFkSMIywfMfNjDNQtTKXNfJCC+i+Z4Vz45f9VE611A83rjeduiAZgQ1mZ7q Fh7rJJG5htuOGHHF9Xyek+p6jGjHmZsIR1h8Dpn64+hYjWioPTlkbZR4T7+6EvMN5q vTG0KEZzcwI4w== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 332A5E937F6; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:23 +0800 Subject: [PATCH v5 09/14] mm/mglru: use the common routine for dirty/writeback reactivation Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-9-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=3145; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=/X4tLveOy+BYFW/Zy6+R0vMXPcNqV+3IgOLhWgyGVaY=; b=O73Zv+vYMZOn0sSZ+5JRY/iW+5z8qjEomyZSYRa7iYd8d7HTuSFm3QlJFJHOQUPFzgVCGADc0 z3J0NZSLEgLA93OaJZwauHtgqcKd97qysgKlk/iHPlY2ttyQqx8TKZE X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Currently MGLRU will move the dirty writeback folios to the second oldest gen instead of reactivate them like the classical LRU. This might help to reduce the LRU contention as it skipped the isolation. But as a result we will see these folios at the LRU tail more frequently leading to inefficient reclaim. Besides, the dirty / writeback check after isolation in shrink_folio_list is more accurate and covers more cases. So instead, just drop the special handling for dirty writeback, use the common routine and re-activate it like the classical LRU. This should in theory improve the scan efficiency. These folios will be rotated back to LRU tail once writeback is done so there is no risk of hotness inversion. And now each reclaim loop will have a higher success rate. This also prepares for unifying the writeback and throttling mechanism with classical LRU, we keep these folios far from tail so detecting the tail batch will have a similar pattern with classical LRU. The micro optimization that avoids LRU contention by skipping the isolation is gone, which should be fine. Compared to IO and writeback cost, the isolation overhead is trivial. And using the common routine also keeps the folio's referenced bits (tier bits), which could improve metrics in the long term. Also no more need to clean reclaim bit as the common routine will make use of it. Note the common routine updates a few throttling and writeback counters, which are not used, and never have been for the MGLRU case. We will start making use of these in later commits. Reviewed-by: Axel Rasmussen Signed-off-by: Kairui Song Reviewed-by: Baolin Wang Reviewed-by: Barry Song --- mm/vmscan.c | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 965d8905a4fe..d63ac03a7266 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4578,7 +4578,6 @@ static bool sort_folio(struct lruvec *lruvec, struct = folio *folio, struct scan_c int tier_idx) { bool success; - bool dirty, writeback; int gen =3D folio_lru_gen(folio); int type =3D folio_is_file_lru(folio); int zone =3D folio_zonenum(folio); @@ -4628,21 +4627,6 @@ static bool sort_folio(struct lruvec *lruvec, struct= folio *folio, struct scan_c return true; } =20 - dirty =3D folio_test_dirty(folio); - writeback =3D folio_test_writeback(folio); - if (type =3D=3D LRU_GEN_FILE && dirty) { - sc->nr.file_taken +=3D delta; - if (!writeback) - sc->nr.unqueued_dirty +=3D delta; - } - - /* waiting for writeback */ - if (writeback || (type =3D=3D LRU_GEN_FILE && dirty)) { - gen =3D folio_inc_gen(lruvec, folio, true); - list_move(&folio->lru, &lrugen->folios[gen][type][zone]); - return true; - } - return false; } =20 @@ -4664,9 +4648,6 @@ static bool isolate_folio(struct lruvec *lruvec, stru= ct folio *folio, struct sca if (!folio_test_referenced(folio)) set_mask_bits(&folio->flags.f, LRU_REFS_MASK, 0); =20 - /* for shrink_folio_list() */ - folio_clear_reclaim(folio); - success =3D lru_gen_del_folio(lruvec, folio, true); VM_WARN_ON_ONCE_FOLIO(!success, folio); =20 --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 971C12FD1CA for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=E83G13/ODG2+hfYMkqruX6hL7upRI9eGoGKlfok1TZ9UZzNttGS9tXnGX7aDHxn6wn4aUYRxhxFUaPJdfAna6SR/qpHxMfKzoAXPTSflFkeR6WMI9E1UWVi+5/PlAz4z6g0pjj1GIiZvbiMG3JVpv9oKWyg3E0ar3wkbhh7vMF0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=Howtx+WY1RxvIrbQAsQfOblRTAE1qNC3ZcMPzNf3NmI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=LLvTncKgs+C+b7rxzAu06RPabqQEWnipmTvR5cFLJJ2Mtr29LDe5m9jJ5KCzw0lAMNPmlMCcuacEOPU6RaqQS4WsYnziWEewbcPfHOcxG1RQVi95F0WdP7q4LwQ6Xz1NulYR2El6d8b5l0bJDX2bx65lDoktICtS9ackjBr+/is= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oaBjutJo; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oaBjutJo" Received: by smtp.kernel.org (Postfix) with ESMTPS id 4D4CBC2BCB7; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=Howtx+WY1RxvIrbQAsQfOblRTAE1qNC3ZcMPzNf3NmI=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=oaBjutJoyZiL7AdLlTPYqiY/xZhPmD/fzcB8RUOr//B6lKxBGhcJ+M8za2RF53xT9 sl+mUYYc0RHqFHx7uFAyWHmVe9yAJjR1Sgsd+gtFixPoYUMJq/+fNWGT2uhePDIYwn TZMecjCk78b+TPLTpWvPDY1jSueIMr3zLp8XppivAlf3A9CKWtqjm0oAJ2nxt2FP5f rzvBMVVDn9x8QLMjauTDzsfWoMUrthJdzFT11/kthix5CNlDFtrKy/Rv39dn1x2OtH vQiB0rNfBRFphssA4//rULbnj8PaZmEaNKpbeXpStgV0lyhl45owCvie4P8xmzxlEx LilgyPgwB2u4Q== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44BBEE937F1; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:24 +0800 Subject: [PATCH v5 10/14] mm/mglru: simplify and improve dirty writeback handling Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-10-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=4353; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=nHbRG5GNajgUzGsrEyC6HM5seFNrRS8JnEpLkfa1ehE=; b=cvPP0SjnNQGrT2ED1jlxu4Fj0AgDqKlQbDXgwEqU9n4LsCraiW0ODDWOP/u+znOJCRlBZg7gq pklzMCNeBG3Cz3RRuXKxKq9aT37d8dvH/jb056V3uSeX21VAUdO2gx3 X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Right now the flusher wakeup mechanism for MGLRU is less responsive and unlikely to trigger compared to classical LRU. The classical LRU wakes the flusher if one batch of folios passed to shrink_folio_list is unevictable due to under writeback. MGLRU instead check and handle this after the whole reclaim loop is done. We previously even saw OOM problems due to passive flusher, which were fixed but still not perfect [1]. We have just unified the dirty folio counting and activation routine, now just move the dirty flush into the loop right after shrink_folio_list. This improves the performance a lot for workloads involving heavy writeback and prepares for throttling too. Test with YCSB workloadb showed a major performance improvement: Before this series: Throughput(ops/sec): 62485.02962831822 AverageLatency(us): 500.9746963330107 pgpgin 159347462 workingset_refault_file 34522071 After this commit: Throughput(ops/sec): 80857.08510208207 AverageLatency(us): 386.653262968934 pgpgin 112233121 workingset_refault_file 19516246 The performance is a lot better with significantly lower refault. We also observed similar or higher performance gain for other real-world workloads. We were concerned that the dirty flush could cause more wear for SSD: that should not be the problem here, since the wakeup condition is when the dirty folios have been pushed to the tail of LRU, which indicates that memory pressure is so high that writeback is blocking the workload already. Reviewed-by: Axel Rasmussen Link: https://lore.kernel.org/linux-mm/20241026115714.1437435-1-jingxiangze= ng.cas@gmail.com/ [1] Signed-off-by: Kairui Song Reviewed-by: Baolin Wang --- mm/vmscan.c | 41 ++++++++++++++++------------------------- 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d63ac03a7266..8072255e3d3a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4724,8 +4724,6 @@ static int scan_folios(unsigned long nr_to_scan, stru= ct lruvec *lruvec, trace_mm_vmscan_lru_isolate(sc->reclaim_idx, sc->order, nr_to_scan, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); - if (type =3D=3D LRU_GEN_FILE) - sc->nr.file_taken +=3D isolated; =20 *isolatedp =3D isolated; return scanned; @@ -4833,12 +4831,27 @@ static int evict_folios(unsigned long nr_to_scan, s= truct lruvec *lruvec, return scanned; retry: reclaimed =3D shrink_folio_list(&list, pgdat, sc, &stat, false, memcg); - sc->nr.unqueued_dirty +=3D stat.nr_unqueued_dirty; sc->nr_reclaimed +=3D reclaimed; trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, type_scanned, reclaimed, &stat, sc->priority, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); =20 + /* + * If too many file cache in the coldest generation can't be evicted + * due to being dirty, wake up the flusher. + */ + if (stat.nr_unqueued_dirty =3D=3D isolated) { + wakeup_flusher_threads(WB_REASON_VMSCAN); + + /* + * For cgroupv1 dirty throttling is achieved by waking up + * the kernel flusher here and later waiting on folios + * which are in writeback to finish (see shrink_folio_list()). + */ + if (!writeback_throttling_sane(sc)) + reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK); + } + list_for_each_entry_safe_reverse(folio, next, &list, lru) { DEFINE_MIN_SEQ(lruvec); =20 @@ -4999,28 +5012,6 @@ static bool try_to_shrink_lruvec(struct lruvec *lruv= ec, struct scan_control *sc) cond_resched(); } =20 - /* - * If too many file cache in the coldest generation can't be evicted - * due to being dirty, wake up the flusher. - */ - if (sc->nr.unqueued_dirty && sc->nr.unqueued_dirty =3D=3D sc->nr.file_tak= en) { - struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); - - wakeup_flusher_threads(WB_REASON_VMSCAN); - - /* - * For cgroupv1 dirty throttling is achieved by waking up - * the kernel flusher here and later waiting on folios - * which are in writeback to finish (see shrink_folio_list()). - * - * Flusher may not be able to issue writeback quickly - * enough for cgroupv1 writeback throttling to work - * on a large system. - */ - if (!writeback_throttling_sane(sc)) - reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK); - } - return need_rotate; } =20 --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A42A42FE042 for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=WBNo0v+S3C6K8acgHgJpm4IWnIgFXwg42oHVhTVK7XNu0O6C6LK5f8NT4ouF7XtbU70e4Y++5CieGwrs41hPIg/s4x3MD86nQV8y3NwOlzgybqrrw9/FhTbi0VgzGbCIuMhKMUqjr+l7CF1UaGPHyB/xAZFyANjnMrCTcfcgjNw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=oMG2Sjwb6KCG3/s/9b3veVJyt69C5GCtWepaQjtL4LI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=MbUfocQMKElnLUhQU7AlWyZ9Hem1Rng+aDG5QHG7SeyXe4xuXcDU3k2ARR5qG85GJdACZZNyoqEolSb0O1MAzg8C2eo8nq4o4k0+bi1lqkvfH40l8PAqX969ujcnMgpceGum42q6rTGK3tuSfB3nb9kNKu+QTghY9Yu1mHdXmR4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bOBPZ5ei; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bOBPZ5ei" Received: by smtp.kernel.org (Postfix) with ESMTPS id 5CCA0C4AF49; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=oMG2Sjwb6KCG3/s/9b3veVJyt69C5GCtWepaQjtL4LI=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=bOBPZ5eiErAwt5FUnn4PIijnFS8H1jEaVxMiFOqAudqfYDHKSY7dFnfaQ2XlItlG1 uPL4A5s4ZY3D5y6TLe+yWZiAw6/35oQ+K72WVtIqmAII9gpt2KaRzwv9Awf2Acc3hP 9yr3aOFbkhsVFq3SvB/BDL6+oy2cxYX/Y72o36zgVwbTCNyOotimrHUV3aS2g8C+M1 GgQ/rF2JjxybE3R+Po0kdQW+0XwFWirPKuBvi/afoqSmgSbsnkw+AiUXW+0pQELG1n Zk71pFqKoEeJtcBRDOxCuHPPupy7vw0n2o6vgRm89EFmkREpTt2w4c58qqoQaNLI1w mD4N6sO4YcE6Q== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 554C6E937EE; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:25 +0800 Subject: [PATCH v5 11/14] mm/mglru: remove no longer used reclaim argument for folio protection Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-11-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=2628; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=zvjDxFYa9NuPhqbFcpkFSi1B2Cv7XMFY1pzNGB2XdZM=; b=Yi/e8MDjTS+iCKm9r5qxslDObbfFgyp0AiLRDbZSZSiGXOEAloq6Gp7vEtlGApoIO4/VhTwPC s1fQd8goCnOCIPhi92FHECGXQ3B4FKgk2nS/xHetar+XKHRzuP9ymwr X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Now dirty reclaim folios are handled after isolation, not before, since dirty reactivation must take the folio off LRU first, and that helps to unify the dirty handling logic. So this argument is no longer needed. Just remove it. Reviewed-by: Axel Rasmussen Signed-off-by: Kairui Song --- mm/vmscan.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 8072255e3d3a..3f6ba34fcd24 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3220,7 +3220,7 @@ static int folio_update_gen(struct folio *folio, int = gen) } =20 /* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool = reclaiming) +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio) { int type =3D folio_is_file_lru(folio); struct lru_gen_folio *lrugen =3D &lruvec->lrugen; @@ -3239,9 +3239,6 @@ static int folio_inc_gen(struct lruvec *lruvec, struc= t folio *folio, bool reclai =20 new_flags =3D old_flags & ~(LRU_GEN_MASK | LRU_REFS_FLAGS); new_flags |=3D (new_gen + 1UL) << LRU_GEN_PGOFF; - /* for folio_end_writeback() */ - if (reclaiming) - new_flags |=3D BIT(PG_reclaim); } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); =20 lru_gen_update_size(lruvec, folio, old_gen, new_gen); @@ -3855,7 +3852,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 - new_gen =3D folio_inc_gen(lruvec, folio, false); + new_gen =3D folio_inc_gen(lruvec, folio); list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); =20 /* don't count the workingset being lazily promoted */ @@ -4607,7 +4604,7 @@ static bool sort_folio(struct lruvec *lruvec, struct = folio *folio, struct scan_c =20 /* protected */ if (tier > tier_idx || refs + workingset =3D=3D BIT(LRU_REFS_WIDTH) + 1) { - gen =3D folio_inc_gen(lruvec, folio, false); + gen =3D folio_inc_gen(lruvec, folio); list_move(&folio->lru, &lrugen->folios[gen][type][zone]); =20 /* don't count the workingset being lazily promoted */ @@ -4622,7 +4619,7 @@ static bool sort_folio(struct lruvec *lruvec, struct = folio *folio, struct scan_c =20 /* ineligible */ if (zone > sc->reclaim_idx) { - gen =3D folio_inc_gen(lruvec, folio, false); + gen =3D folio_inc_gen(lruvec, folio); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACF532FE575 for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=NGtc22KSNTXcXHePgwEokHe9yZWK0CoXUPti7n9YDAu8IF/cT2k1oEEHPY4bYlrsXqdzYIY7oLuZVdhIhpQ8bs5Yz4EwLPJ6Wimze7BRiNOhLAvRvBqXw1v6e6JOaBQ2xPQ2iOo6fDqKh8ekoLW9cf5ANg+6L9OgjaUZx+Fd7tI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=0ogG1jMyQN2qhbr+HuVe+uUiHh4hF43M9AOehbS6L/w=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=KyZt/E05qDQJZ2BlEiwz/eDh8CoW8PlOZIZFhnhWGSG2BDczu2zkbMywOgrI4q/h346/pvYwJZqVEY4StsxhDZ1cSBheS7Psmo5a9+w8NvoHCskToStN20PY0rcayPL8q2XC50N4ptRida4SH2zgvLgJHWlWfW5Z8SDXaTuCL0Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kXS/Dlwa; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kXS/Dlwa" Received: by smtp.kernel.org (Postfix) with ESMTPS id 6D48CC2BCF4; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=0ogG1jMyQN2qhbr+HuVe+uUiHh4hF43M9AOehbS6L/w=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=kXS/DlwaLpCvGRHvqGjzI+5PcPqqzA9F3uJah00fz7NeMsl04VptobPVp/3JwmoJd J9ShtR5PKoFCTgmACX4kK+9nfwoqN2YQR3UV9xYoy2dwJg4NySEAbWTFFO3Vrq1CRT +Awp/FG2Tt7iHl4Lp4u6XYKClzzhMLR3BpRB3zDLdRH0J3MvzJ9guMoO/UKzfp2TMw ORNq1bmQJjyyS1RneatW4nVvermghQ119f1+avtolgEr8N9HYitsqWy9vJWUoIPQCq VPxMoex9nKZ/IQDiFdur4mt9IpIzWZP5xiFBH+d5gKLWsdjXrlSLx3zGRaut7que42 xhfZd7+sF94Cg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6521EE937EF; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:26 +0800 Subject: [PATCH v5 12/14] mm/vmscan: remove sc->file_taken Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-12-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=963; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=o7IfWpYvb69FfM+6QHpHKLjSXJdMZ8mmLPd6WDSOdzk=; b=XwU6oycFtzzEXn83BtJbfuUKeK0+9X4XcCrwsedeyEG5UJ98PBWfmDNuZobUdvF3wfmMTDfSR j2rLk6jQdGACPZ3qZuO0VUVi5jXV2Zf7n3kLENeT1FB/oXzyRSnfY4y X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song No one is using it now, just remove it. Reviewed-by: Axel Rasmussen Reviewed-by: Baolin Wang Signed-off-by: Kairui Song Reviewed-by: Chen Ridong --- mm/vmscan.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 3f6ba34fcd24..b0b66c03d4dc 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -173,7 +173,6 @@ struct scan_control { unsigned int congested; unsigned int writeback; unsigned int immediate; - unsigned int file_taken; unsigned int taken; } nr; =20 @@ -2040,8 +2039,6 @@ static unsigned long shrink_inactive_list(unsigned lo= ng nr_to_scan, sc->nr.writeback +=3D stat.nr_writeback; sc->nr.immediate +=3D stat.nr_immediate; sc->nr.taken +=3D nr_taken; - if (file) - sc->nr.file_taken +=3D nr_taken; =20 trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, nr_scanned, nr_reclaimed, &stat, sc->priority, file); --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC2B12FF65F for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=ZS8KtnQymy+L6huoKeYAYKfZ9DSh/feXBad9jPp/uotLXFYEZRdn6k9Lyk5E83XsDfESt5SUVvG24MjpM64fWz+qkm6F130XMPBJaepdt0DSi2bNjS8q472uWggsXvrk33zqrqEqOAZBGFBVQzc+Ew6kJRxfEMMUsHkOTHyaWX4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=oHmrzr8wi9GlAl3Ja/rW+v7cpNZs1wYuqzM8tpEID4w=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=o1CYGqjZKT8mHRkbfD2zDx/BUbcNmp4LS4NJ6tLBH6AdJS7Tty167l6nBnHBe6JUs+Qv5I8KQDdnc0QyF0WmVTt7TPZ1uFdCo3btuBi3nKs+FVRHY64fkwowmT8dg/4G38dfsHORQO6hFcSffueqkobgCdtr/pfD/EMJz560TmY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NEVK8l/Q; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NEVK8l/Q" Received: by smtp.kernel.org (Postfix) with ESMTPS id 81EABC4AF15; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=oHmrzr8wi9GlAl3Ja/rW+v7cpNZs1wYuqzM8tpEID4w=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=NEVK8l/Qx22tpE9XikCL2DPAKPXH00I4F4+Uhl2PwvIYTsKMWGFMGmUBl/pJHmTO3 ixh/1M3+GSOvJaCUkzZrrfzygArhoiNtQYwP5IGjTuaeS3n+bYJWvHIVHf1Q2KdbLt X1+414y5LFwca2itL3JxkidXnpbUfsgGoIUKOWS1AIOAQx8IFAgcOGWXnZZpdUrBSA s2vg7Q5YEgaOUHfAiY6+i8qpwDy6hrXmKFpk7JTEiOfxwWKm7rTJk/N+M8xvDShPe0 XPDCRPaN4aDDjPxOscCRlZjUTboTn8JOp3nRJs6OEc5oMC1dUi4f76Z4NXgiq3QbJq owh3cjGyHT6iw== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76261E937F2; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:27 +0800 Subject: [PATCH v5 13/14] mm/vmscan: remove sc->unqueued_dirty Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-13-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=1037; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=4jCESMjGGJsoUesOniTyg/KeGpgULItGaxuoRO86zSM=; b=o5S6aT951G1YUPUxSOT4cQ/sdMDzmFkFRmZZqWI/D6b9o0qILFMa3J4GCiB8cgvvpt536euSX zngfkM+CCCoCveBBR8zPIuXZfqeJdPa63Rv7f466qplLtUbaNobVEXl X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song No one is using it now, just remove it. Suggested-by: Axel Rasmussen Reviewed-by: Baolin Wang Reviewed-by: Axel Rasmussen Reviewed-by: Barry Song Signed-off-by: Kairui Song Reviewed-by: Chen Ridong --- mm/vmscan.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b0b66c03d4dc..a431f94ff3a3 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -169,7 +169,6 @@ struct scan_control { =20 struct { unsigned int dirty; - unsigned int unqueued_dirty; unsigned int congested; unsigned int writeback; unsigned int immediate; @@ -2035,7 +2034,6 @@ static unsigned long shrink_inactive_list(unsigned lo= ng nr_to_scan, =20 sc->nr.dirty +=3D stat.nr_dirty; sc->nr.congested +=3D stat.nr_congested; - sc->nr.unqueued_dirty +=3D stat.nr_unqueued_dirty; sc->nr.writeback +=3D stat.nr_writeback; sc->nr.immediate +=3D stat.nr_immediate; sc->nr.taken +=3D nr_taken; --=20 2.53.0 From nobody Sat Jun 20 17:35:31 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D06B72FFF8D for ; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; cv=none; b=Sc/Mj7VcaTFm3V+/ghX36gtVLg7DOheA4egDVihxA/ZzQ3l/A9MEPU8K4HxaVZD5M3cEk7u3zoZqKzD2woKtlGFZUxp2WE+t2q2yL6v5jqwSXqHy3e3CA79o6uSJRO5/jVAOnUReHbM1y/4U8aG3vde3ThM9s7FOpTEXIlqGDPM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776012500; c=relaxed/simple; bh=Qj/B1/z69+cZMySbji0O/P0DX24+zEDw7gVmSm9wO8U=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=KyWYXZR1uJkZJhCsvQVP6tr4UnpSdks2u+T8i6FeagDXIIrm3syPaY3AsZayS0K6c+bEJUE3NA7RR7V4cJJ9XJby7XjntSiZkrwTDmVtrwV1IdSbpToHD6MTzZkho4w/Qv5q+TQ5xqNS+5HgAZLBOLMJLEOK29rumOOiJJ4lQu4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nb2A1ngQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nb2A1ngQ" Received: by smtp.kernel.org (Postfix) with ESMTPS id 9137EC2BD04; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1776012500; bh=Qj/B1/z69+cZMySbji0O/P0DX24+zEDw7gVmSm9wO8U=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=nb2A1ngQdUgDrqtyzYZ/EbCYuQPAyNLKQZsIQMLsE+NN8BM430W0XZpqXhxKnlSb1 PTTyL3EgXFGragQhxKYgXZjMJAZ39esqiKlOMGEz0Cvm1Lnq2pQ65HDVYxTXqzY31f hInPXG5pFrXxy+HDOpGD3IDdqr2VfEFIpqiTfbPHYHV1GsCofdVkRM1NAeI9W/l5op /K1Nzqh9UONGRxui2QPRBknAKD/zetjz6PT/nyjqt9Oas95JfJR6onB8tosrBazKWZ ao6nyMzuPqt1zinXjzlIM6esRPP7ST5tR29O7fp2WnCji9Jr0463DYITWkGT94greN 0Quvf+Jk00NJw== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87417E937F6; Sun, 12 Apr 2026 16:48:20 +0000 (UTC) From: Kairui Song via B4 Relay Date: Mon, 13 Apr 2026 00:48:28 +0800 Subject: [PATCH v5 14/14] mm/vmscan: unify writeback reclaim statistic and throttling Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260413-mglru-reclaim-v5-14-8eaeacbddc44@tencent.com> References: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> In-Reply-To: <20260413-mglru-reclaim-v5-0-8eaeacbddc44@tencent.com> To: linux-mm@kvack.org Cc: Andrew Morton , Axel Rasmussen , Yuanchu Xie , Wei Xu , Johannes Weiner , David Hildenbrand , Michal Hocko , Qi Zheng , Shakeel Butt , Lorenzo Stoakes , Barry Song , David Stevens , Chen Ridong , Leno Hou , Yafang Shao , Yu Zhao , Zicheng Wang , Kalesh Singh , Suren Baghdasaryan , Chris Li , Vernon Yang , linux-kernel@vger.kernel.org, Qi Zheng , Baolin Wang , Kairui Song X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1776012497; l=6741; i=kasong@tencent.com; s=kasong-sign-tencent; h=from:subject:message-id; bh=onAAOpr3vpU2bMTuSUOIQofpAkgauLbJ3WIXe1tb0hQ=; b=FGbRkx/ZwSYXPdD+OeEnzMtVz6XKIkpf+5rfGCSl0b2y9g6+AgDXBdPdqLWrIsmtWIi3rNhwf Hq50xl97Ww5BsPWwtOGW6pmOwpX+5hRXjcT/bMktBkPJ45glAXWWUog X-Developer-Key: i=kasong@tencent.com; a=ed25519; pk=kCdoBuwrYph+KrkJnrr7Sm1pwwhGDdZKcKrqiK8Y1mI= X-Endpoint-Received: by B4 Relay for kasong@tencent.com/kasong-sign-tencent with auth_id=562 X-Original-From: Kairui Song Reply-To: kasong@tencent.com From: Kairui Song Currently MGLRU and non-MGLRU handle the reclaim statistic and writeback handling very differently, especially throttling. Basically MGLRU just ignored the throttling part. Let's just unify this part, use a helper to deduplicate the code so both setups will share the same behavior. Test using following reproducer using bash: echo "Setup a slow device using dm delay" dd if=3D/dev/zero of=3D/var/tmp/backing bs=3D1M count=3D2048 LOOP=3D$(losetup --show -f /var/tmp/backing) mkfs.ext4 -q $LOOP echo "0 $(blockdev --getsz $LOOP) delay $LOOP 0 0 $LOOP 0 1000" | \ dmsetup create slow_dev mkdir -p /mnt/slow && mount /dev/mapper/slow_dev /mnt/slow echo "Start writeback pressure" sync && echo 3 > /proc/sys/vm/drop_caches mkdir /sys/fs/cgroup/test_wb echo 128M > /sys/fs/cgroup/test_wb/memory.max (echo $BASHPID > /sys/fs/cgroup/test_wb/cgroup.procs && \ dd if=3D/dev/zero of=3D/mnt/slow/testfile bs=3D1M count=3D192) echo "Clean up" echo "0 $(blockdev --getsz $LOOP) error" | dmsetup load slow_dev dmsetup resume slow_dev umount -l /mnt/slow && sync dmsetup remove slow_dev Before this commit, `dd` will get OOM killed immediately if MGLRU is enabled. Classic LRU is fine. After this commit, throttling is now effective and no more spin on LRU or premature OOM. Stress test on other workloads also looking good. Global throttling is not here yet, we will fix that separately later. Suggested-by: Chen Ridong Tested-by: Leno Hou Reviewed-by: Axel Rasmussen Reviewed-by: Baolin Wang Signed-off-by: Kairui Song --- mm/vmscan.c | 90 ++++++++++++++++++++++++++++-----------------------------= ---- 1 file changed, 41 insertions(+), 49 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a431f94ff3a3..43a3cadbb586 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1942,6 +1942,44 @@ static int current_may_throttle(void) return !(current->flags & PF_LOCAL_THROTTLE); } =20 +static void handle_reclaim_writeback(unsigned long nr_taken, + struct pglist_data *pgdat, + struct scan_control *sc, + struct reclaim_stat *stat) +{ + /* + * If dirty folios are scanned that are not queued for IO, it + * implies that flushers are not doing their job. This can + * happen when memory pressure pushes dirty folios to the end of + * the LRU before the dirty limits are breached and the dirty + * data has expired. It can also happen when the proportion of + * dirty folios grows not through writes but through memory + * pressure reclaiming all the clean cache. And in some cases, + * the flushers simply cannot keep up with the allocation + * rate. Nudge the flusher threads in case they are asleep. + */ + if (stat->nr_unqueued_dirty =3D=3D nr_taken && nr_taken) { + wakeup_flusher_threads(WB_REASON_VMSCAN); + /* + * For cgroupv1 dirty throttling is achieved by waking up + * the kernel flusher here and later waiting on folios + * which are in writeback to finish (see shrink_folio_list()). + * + * Flusher may not be able to issue writeback quickly + * enough for cgroupv1 writeback throttling to work + * on a large system. + */ + if (!writeback_throttling_sane(sc)) + reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK); + } + + sc->nr.dirty +=3D stat->nr_dirty; + sc->nr.congested +=3D stat->nr_congested; + sc->nr.writeback +=3D stat->nr_writeback; + sc->nr.immediate +=3D stat->nr_immediate; + sc->nr.taken +=3D nr_taken; +} + /* * shrink_inactive_list() is a helper for shrink_node(). It returns the n= umber * of reclaimed pages @@ -2005,39 +2043,7 @@ static unsigned long shrink_inactive_list(unsigned l= ong nr_to_scan, lruvec_lock_irq(lruvec); lru_note_cost_unlock_irq(lruvec, file, stat.nr_pageout, nr_scanned - nr_reclaimed); - - /* - * If dirty folios are scanned that are not queued for IO, it - * implies that flushers are not doing their job. This can - * happen when memory pressure pushes dirty folios to the end of - * the LRU before the dirty limits are breached and the dirty - * data has expired. It can also happen when the proportion of - * dirty folios grows not through writes but through memory - * pressure reclaiming all the clean cache. And in some cases, - * the flushers simply cannot keep up with the allocation - * rate. Nudge the flusher threads in case they are asleep. - */ - if (stat.nr_unqueued_dirty =3D=3D nr_taken) { - wakeup_flusher_threads(WB_REASON_VMSCAN); - /* - * For cgroupv1 dirty throttling is achieved by waking up - * the kernel flusher here and later waiting on folios - * which are in writeback to finish (see shrink_folio_list()). - * - * Flusher may not be able to issue writeback quickly - * enough for cgroupv1 writeback throttling to work - * on a large system. - */ - if (!writeback_throttling_sane(sc)) - reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK); - } - - sc->nr.dirty +=3D stat.nr_dirty; - sc->nr.congested +=3D stat.nr_congested; - sc->nr.writeback +=3D stat.nr_writeback; - sc->nr.immediate +=3D stat.nr_immediate; - sc->nr.taken +=3D nr_taken; - + handle_reclaim_writeback(nr_taken, pgdat, sc, &stat); trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, nr_scanned, nr_reclaimed, &stat, sc->priority, file); return nr_reclaimed; @@ -4824,26 +4830,11 @@ static int evict_folios(unsigned long nr_to_scan, s= truct lruvec *lruvec, retry: reclaimed =3D shrink_folio_list(&list, pgdat, sc, &stat, false, memcg); sc->nr_reclaimed +=3D reclaimed; + handle_reclaim_writeback(isolated, pgdat, sc, &stat); trace_mm_vmscan_lru_shrink_inactive(pgdat->node_id, type_scanned, reclaimed, &stat, sc->priority, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); =20 - /* - * If too many file cache in the coldest generation can't be evicted - * due to being dirty, wake up the flusher. - */ - if (stat.nr_unqueued_dirty =3D=3D isolated) { - wakeup_flusher_threads(WB_REASON_VMSCAN); - - /* - * For cgroupv1 dirty throttling is achieved by waking up - * the kernel flusher here and later waiting on folios - * which are in writeback to finish (see shrink_folio_list()). - */ - if (!writeback_throttling_sane(sc)) - reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK); - } - list_for_each_entry_safe_reverse(folio, next, &list, lru) { DEFINE_MIN_SEQ(lruvec); =20 @@ -4886,6 +4877,7 @@ static int evict_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, =20 if (!list_empty(&list)) { skip_retry =3D true; + isolated =3D 0; goto retry; } =20 --=20 2.53.0