From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66D5A43C7D3 for ; Wed, 12 Aug 2026 12:18:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537098; cv=none; b=Ep7tBmTelYNuKQEYgW3fbkKv+IDQ8f1hhKT8IwjmUhUr6sZRVxxdi9Ch5sXVle7HL7Lw5268HvalfzkYEUIkpyDwnyernF0rzEGZvXx6xnMfA9SIyouIYBuzeGFTQF9nMXH8LumC9D3d/uGFi65d+oX8IWDjQIhGo0R0mbmGCpw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537098; c=relaxed/simple; bh=3zZOiXijHvymRhTyLFfoNXKxs5IL+xmm5ocYX6k0c1w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cgLtRsmKViBeQtRoMqX7TlGYmeXQwyqXLA0GTE3eRrbyy1gNuJKZUR3mT6djqBLRrZJiHp+rwbKmQcVJRL1WGi46WHMczZn5JVf6z92HJoWEXyuudtsUGDwXH5xIxzauKZyUbv+PxjOxITW8c9Sr6ZZ5TevKuCLfoxjs697QAQY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mOf3kSGC; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mOf3kSGC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5D3941F00A3F; Wed, 12 Aug 2026 12:18:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537088; bh=JfAFNjFFahuQ/02Rkg04vR1FEvVXtWN3blDMb1aBDxU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=mOf3kSGCfSxxfdiJl7wZodylrQcmlpLnTU/EWOJ2bdq3O0M1rLSC+lwTxkhQFdree FOgtXEMi1Ux59cCoHRWmf+Xv17LYSspapX612WkMbcTBYkcPhsdoiAr4YGEl98OPtI hyH6RPUjS8oPpEJE3a0UGkNCjXSVNLn6/v0RgolnW8QozyqbEe3roMnuI4WcPMSXX8 dORxyvZXF3y0WlD5JEH59V6xfchTrVsDkOZ1mr4igXkHACBOAmSB9oqb77iS8qDM8j sILeNtf+alBzo4UvQ+fpV6kfuqk/kJdb9laFKzmTPLyG2Eb+W6pP5dUSAOSg/6d4JB IPuP33JgEA01w== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, Barry Song Subject: [RFC PATCH v4 01/16] mm/mglru: improve readability of isolate_folios() Date: Wed, 12 Aug 2026 20:16:43 +0800 Message-Id: <20260812121658.69965-2-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ridong Chen The for_each_evictable_type() loop in isolate_folios() is misleading: it does not actually iterate over each evictable type. Instead, get_type_to_scan() selects the type to scan, while the iterator `i` merely bounds the number of attempts. Signed-off-by: Ridong Chen Co-developed-by: Barry Song (Xiaomi) Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 17d2b793cbfc..ea058692b9a5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4839,35 +4839,41 @@ static int get_type_to_scan(struct lruvec *lruvec, = int swappiness) return positive_ctrl_err(&sp, &pv); } =20 +static inline bool is_single_type_reclaim(int swappiness) +{ + return swappiness =3D=3D MIN_SWAPPINESS || + swappiness =3D=3D SWAPPINESS_ANON_ONLY; +} + static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, int swappiness, struct list_head *list, int *isolated, int *isolate_type, int *isolate_scanned) { - int i; - int total_scanned =3D 0; + bool type_fallback_allowed =3D !is_single_type_reclaim(swappiness); int type =3D get_type_to_scan(lruvec, swappiness); + int total_scanned =3D 0, scanned, tier; =20 - for_each_evictable_type(i, swappiness) { - int scanned; - int tier =3D get_tier_idx(lruvec, type); +retry: + tier =3D get_tier_idx(lruvec, type); + scanned =3D scan_folios(nr_to_scan, lruvec, sc, + type, tier, list, isolated); =20 - scanned =3D scan_folios(nr_to_scan, lruvec, sc, - type, tier, list, isolated); + total_scanned +=3D scanned; + if (*isolated) { + *isolate_type =3D type; + *isolate_scanned =3D scanned; + return total_scanned; + } =20 - total_scanned +=3D scanned; - if (*isolated) { - *isolate_type =3D type; - *isolate_scanned =3D scanned; - break; - } - /* - * If scanned > 0 and isolated =3D=3D 0, avoid falling back to the - * other type, as this type remains sufficient. Falling back - * too readily can disrupt the positive_ctrl_err() bias. - */ - if (!scanned) - type =3D !type; + /* + * We are running out of the current reclaim type. Fall back to + * the other type if allowed. + */ + if (!scanned && type_fallback_allowed) { + type =3D !type; + type_fallback_allowed =3D false; + goto retry; } =20 return total_scanned; --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71305438012 for ; Wed, 12 Aug 2026 12:18:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537093; cv=none; b=UdrhbizQAQzIYM0roYQoR08PJoZp4p90KbFmAACIyOJyQG4WUpapg760nqnusGZ7jJle0jSlKJ1KtiQcaadoXlRPqFRn5tCqdB0erWMP5mn7j+pg8TmLgqUdFUih0lYhuuJKzW27lrb/Ur3ULME0uj1+86/ONK/s95ahC8SaCGY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537093; c=relaxed/simple; bh=OjPX9d2RuV/5aW/+BD5iDBQlRpzcCsE0LYUYF09K2wk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BKecxx1lpMCV3SikyLjLttPlIbvr1RStj3evSeJ64/Rt2xKJIVDTkPeXRshj4NqJdbV1DVQSykzdqpb87brGVNxvovH7LZtZc+IhFNqFEA9p5XjL+1AYGTxU7U63F5JGsX7yF46KkZXptmRdaZVI4VSPb3zBYnCz7LTRcF72lA8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FY75tQDd; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FY75tQDd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 640BF1F000E9; Wed, 12 Aug 2026 12:18:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537092; bh=zTdKsivSVml3nMVoycYBOOaMCkJqeu1cLwVQEk8xkYQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=FY75tQDd1dy7KRMYkQVZVOWKUDTzrMbOQF8/iIjIN9Otz6yx5FHINrncJKvdrVmR1 CWkJt1HhXEs4EKGktK9a9lKaup8GD6nadgSz1h/gZpHgSgmxrPWE2TRd15/SV+BzHn HQRYrIX18vLxNtH0PDxH03qbdWd+xHCX4H2JAn1AvzrnZUNxigrMVZ0v7xP9pSewAC SBQVYW2xg6yP0p4gGMM5UbZdp6rOBsdRCGDj7VQorqtUK1+DtUBrjOgmGSkhpabzNn pDMmd+Urq82UZiczyGt6H82IiWhhSlB2PUaVkbcMoxpjSAZypoWP83zPnSsluFSaTg jL4Fnvr1I3Lqw== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 02/16] mm/mglru: improve scan_folios() exhaustion detection Date: Wed, 12 Aug 2026 20:16:44 +0800 Message-Id: <20260812121658.69965-3-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit 16b475d2ac3c ("mm/mglru: avoid reclaim type fall back when isolation makes no progress") uses scanned =3D=3D 0 to determine whether scan_folios() has exhausted a reclaim type. However, this is not always sufficient. It is possible for scanned > 0, while the oldest reclaimable generation is exhausted after the first scan_folios() call. We detect early_stop in scan_folios(). If we stop early for any reason, it means the current reclaim type is not exhausted yet. If early_stop is never reached, it means we have exhausted the current oldest generation without hitting any scanning limit. Another issue is that if the lruvec has 4 generations, we might have exhausted the oldest generation while the second oldest generation is still reclaimable. In that case, this type is not exhausted yet. Add an exhausted output argument to scan_folios() so it can explicitly report whether the reclaimable lists for the current type have been exhausted. Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index ea058692b9a5..0670a25d3a7e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4727,7 +4727,8 @@ static bool isolate_folio(struct lruvec *lruvec, stru= ct folio *folio, struct sca =20 static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, int type, int tier, - struct list_head *list, int *isolatedp) + struct list_head *list, int *isolatedp, + bool *exhausted) { int i; int gen; @@ -4738,12 +4739,15 @@ static int scan_folios(unsigned long nr_to_scan, st= ruct lruvec *lruvec, int skipped =3D 0; unsigned long remaining =3D nr_to_scan; struct lru_gen_folio *lrugen =3D &lruvec->lrugen; + bool early_stop =3D false; =20 VM_WARN_ON_ONCE(nr_to_scan > MAX_LRU_BATCH); VM_WARN_ON_ONCE(!list_empty(list)); =20 - if (get_nr_gens(lruvec, type) =3D=3D MIN_NR_GENS) + if (get_nr_gens(lruvec, type) =3D=3D MIN_NR_GENS) { + *exhausted =3D true; return 0; + } =20 gen =3D lru_gen_from_seq(lrugen->min_seq[type]); =20 @@ -4774,8 +4778,10 @@ static int scan_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, skipped_zone +=3D delta; } =20 - if (!--remaining || max(isolated, skipped_zone) >=3D MIN_LRU_BATCH) + if (!--remaining || max(isolated, skipped_zone) >=3D MIN_LRU_BATCH) { + early_stop =3D true; break; + } } =20 if (skipped_zone) { @@ -4784,8 +4790,10 @@ static int scan_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, skipped +=3D skipped_zone; } =20 - if (!remaining || isolated >=3D MIN_LRU_BATCH) + if (!remaining || isolated >=3D MIN_LRU_BATCH) { + early_stop =3D true; break; + } } =20 item =3D PGSCAN_KSWAPD + reclaimer_offset(sc); @@ -4796,6 +4804,13 @@ static int scan_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); =20 + /* + * If we didn't stop early, all reclaimable folios in the current + * generation have been scanned. We are exhausted if this is the last + * reclaimable generation. + */ + *exhausted =3D !early_stop && + lrugen->min_seq[type] + MIN_NR_GENS =3D=3D lrugen->max_seq; *isolatedp =3D isolated; return scanned; } @@ -4853,11 +4868,12 @@ static int isolate_folios(unsigned long nr_to_scan,= struct lruvec *lruvec, bool type_fallback_allowed =3D !is_single_type_reclaim(swappiness); int type =3D get_type_to_scan(lruvec, swappiness); int total_scanned =3D 0, scanned, tier; + bool exhausted; =20 retry: tier =3D get_tier_idx(lruvec, type); scanned =3D scan_folios(nr_to_scan, lruvec, sc, - type, tier, list, isolated); + type, tier, list, isolated, &exhausted); =20 total_scanned +=3D scanned; if (*isolated) { @@ -4870,7 +4886,7 @@ static int isolate_folios(unsigned long nr_to_scan, s= truct lruvec *lruvec, * We are running out of the current reclaim type. Fall back to * the other type if allowed. */ - if (!scanned && type_fallback_allowed) { + if (exhausted && type_fallback_allowed) { type =3D !type; type_fallback_allowed =3D false; goto retry; --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E5A6438012 for ; Wed, 12 Aug 2026 12:18:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537097; cv=none; b=q3ua7RS1LCoPv/PWHa7nNuGhBhHNnjJZjSeXZ47esqPxPed/TevLPHLkGZDkhdZ+12h7Y52H+UGCJzUEH7OWpg7fX4PjxbWQCFN136Hpr5sjaSXvnlmcUgLtI0yGj1hWb2ziX9YjRrIvjAb3u/cGwH4R1DdT6EfznuoUgfMHzdU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537097; c=relaxed/simple; bh=UQK3XK6c5Wr4mcmzYRvd3zgI8+nyrnhmwRcECofVmPY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JMUZ7n2XMu9I0/SZUhYtKCCUlL8+ClkdkzCCH6y98VwbhDusZ6dF5K7hRXl9C1SpVHrrR/lVBqoNdf+lChjvNNsfow8mKxeTgNglwsaZ0b1OzkbWyP2NEg/qkTH2lxRQcLiUtd6TRlwhsEt32HnLoVQ5ZjZHwyRwsaCkmR0tQxk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=R9S12A8I; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="R9S12A8I" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6CCFC1F00ACA; Wed, 12 Aug 2026 12:18:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537096; bh=uepm22g2J/5iiwM0/TymuO6Dk8rItX1/CDBy+v0l+eE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=R9S12A8IO8zHeVVFYk3oGKcmS8dHnKtd493X3Or8A9aMUAnd6TNwa1+X4f5MGtMO4 HZkg1bXcsT54Nq1o0ZUngNCkE7ad4jVS6yhmG9vec/HJLix348ZdwSfOffTFYWzUQn hlQ8+kbywEs6WbtaDYMCUFY6Jt3Y92l0pxWPusHszQozoihM8HV18KN4Jf2RSME68h FDSuRqkmtnRbAzctC9qysMudGB9lb8be25nzo46e9D+vU6CUqdrBdL9FlU9F6/ePLR jV1Dtr/Bon9GdcngDpxypLFv611Zo0azHucegMQWobd0dcSjZXFMZa9UenFCzkaosC 0XnW0QYQJb98A== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 03/16] mm/mglru: retry the same type once if isolation fails due to races Date: Wed, 12 Aug 2026 20:16:45 +0800 Message-Id: <20260812121658.69965-4-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If we are not exhausted (i.e., there are still folios in the reclaimable generations) but fail to isolate any folios due to promotions, protection, or races, give this type one more chance to avoid going through the outer loop again. Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 0670a25d3a7e..dddd2ecf970a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4868,7 +4868,7 @@ static int isolate_folios(unsigned long nr_to_scan, s= truct lruvec *lruvec, bool type_fallback_allowed =3D !is_single_type_reclaim(swappiness); int type =3D get_type_to_scan(lruvec, swappiness); int total_scanned =3D 0, scanned, tier; - bool exhausted; + bool exhausted, tried =3D false; =20 retry: tier =3D get_tier_idx(lruvec, type); @@ -4891,6 +4891,14 @@ static int isolate_folios(unsigned long nr_to_scan, = struct lruvec *lruvec, type_fallback_allowed =3D false; goto retry; } + /* + * We are not exhausted, but failed to isolate any folios due to + * races. Give this type one more chance to avoid a larger loop. + */ + if (!exhausted && !tried) { + tried =3D true; + goto retry; + } =20 return total_scanned; } --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3962443C054 for ; Wed, 12 Aug 2026 12:18:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537101; cv=none; b=e2LwY0rLRgiwKflqNKZGfRQsglgGg88e3uM773eUE6YmTWfs5vI7QwUi9GmwwrH3hjWjeapE8GIgrbRMLSVJY+RUpTVn560ALfPfuiyoIMohYDK5lXknLUvRfhTZXN655QKkVlF6aE4wH5myIeZsPRpapeifYiRvM+Po5GwX1AE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537101; c=relaxed/simple; bh=tyeXHomLNM40QC+mTekcVRiaWV4us5wp5V5r5jrWOxQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qvMs1bHqGBYDEFOlTgSeh62AG4G0oh6zjmoHYPVtVv8F1G1QUYUQXkViR8AvDEU83o2r3McC/QalcxmZSCQCfz3fKjTsH4MTXSId9BI04EH5P30+Tc+gUfFuyje/Jwe70NHCM9JmmzctoYfhI5C2mqjqOvnnrI2LYyuszuL2IvY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Yeol5q/2; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Yeol5q/2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 759FD1F000E9; Wed, 12 Aug 2026 12:18:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537100; bh=8SPnYkw1Tpm0RdC6K4fEpBajDKizXkw5kF84LatlXqs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Yeol5q/25N+xnu1SXOdyiCkZybpJAjBsoTTDhfAYWyjY9mCkLZ/5oJczeJoI9uuWK AonvvY6ikwAM7pWjaw009NzdXeUKBSVAg1uqAp2JpFSQ3dFZwZWK0vb3hnJxyOt4wy 4b3eYJwBF9l/LJqGetsBymGuqJEllDg7uKjG1blNYo9vBYdIrm8q8wNstwZdDW9+Dk zJs4Y1mnlnmt+T43CaOzHcgwuF3hUVCYnvhIAJ7NSlrzAF5G2Xak3LmVCRcJoY8Cw4 QZWlIgZ5+B0+gHvlC4D5RQu02T4qP2hrhjGiBt2qRxIs5HMezylYX6kf5CXQ+zR3Jm yfuATH/+e7d0A== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 04/16] mm/mglru: boost swappiness responsiveness in get_type_to_scan() Date: Wed, 12 Aug 2026 20:16:46 +0800 Message-Id: <20260812121658.69965-5-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The linear weight in get_type_to_scan() can easily be overwhelmed by historical refault costs when swappiness deviates from neutral (100). As a result, user-configured swappiness often feels unresponsive under memory pressure. Apply a smooth quadratic boost based on the distance from the neutral balance point (MAX_SWAPPINESS / 2). This amplifies the weight of the preferred scanning type (anon vs file) while avoiding hardware division via bitwise shift. Assisted-by: gemini:gemini-3.6-flash Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index dddd2ecf970a..2fd1e06eb192 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4838,18 +4838,42 @@ static int get_tier_idx(struct lruvec *lruvec, int = type) static int get_type_to_scan(struct lruvec *lruvec, int swappiness) { struct ctrl_pos sp, pv =3D {}; + int anon_gain, file_gain; =20 if (swappiness <=3D MIN_SWAPPINESS + 1) return LRU_GEN_FILE; =20 if (swappiness >=3D MAX_SWAPPINESS) return LRU_GEN_ANON; + + /* + * Apply a quadratic boost based on the distance from the neutral + * balance point (swappiness =3D MAX_SWAPPINESS / 2). + * + * A linear weight is easily overwhelmed by historical refault cost + * when swappiness deviates from neutral. The quadratic scaling + * amplifies the weight of the preferred type smoothly. + */ + if (swappiness < MAX_SWAPPINESS / 2) { + int delta =3D (MAX_SWAPPINESS / 2) - swappiness; + int boost =3D (delta * delta) >> 4; + + anon_gain =3D swappiness; + file_gain =3D (MAX_SWAPPINESS - swappiness) + boost; + } else { + int delta =3D swappiness - (MAX_SWAPPINESS / 2); + int boost =3D (delta * delta) >> 4; + + anon_gain =3D swappiness + boost; + file_gain =3D MAX_SWAPPINESS - swappiness; + } + /* * Compare the sum of all tiers of anon with that of file to determine * which type to scan. */ - read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, swappiness, &sp); - read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, MAX_SWAPPINESS - swappi= ness, &pv); + read_ctrl_pos(lruvec, LRU_GEN_ANON, MAX_NR_TIERS, anon_gain, &sp); + read_ctrl_pos(lruvec, LRU_GEN_FILE, MAX_NR_TIERS, file_gain, &pv); =20 return positive_ctrl_err(&sp, &pv); } --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4231E43B4B5 for ; Wed, 12 Aug 2026 12:18:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537105; cv=none; b=YbNkMtiE9Sii2tQWPRa+XhqOanomVR0/PXBSxCZw7AQSO4NHKgZhwTri7RJ+oGb4/EY6mhIau6hSCnaa2C1T/lgcg38F+FtGO6bPTELjjPCOb44Ds9uSyLfqbx4/jMBh57gdU1reFF5APPkMRmBXw2Zo3KgL4Iq7W2dLDAJkH04= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537105; c=relaxed/simple; bh=gnreT11CPAXENKEyEEEVDO1A02uJWW7lg7OOSQTUHfM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hXZ4ckUiB++062kRSljAKlxYeE6sPImGTicIlPiSZiVBYn45IfTiG7w7ZxjRAjbQuqo4F0DB85mSZ7VuoXlZ8MwLRMReec2v1zYytvscO1p1DjbaexForLuP9ABBJmFM5ZtyOoK7YEfV0ARGMxmTnkZyexqMjxSZvJw2UPMOyAE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=klp1mR5j; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="klp1mR5j" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 826091F00A3A; Wed, 12 Aug 2026 12:18:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537104; bh=YTbE3zCASBCCzvf13wSpm/yIS5j72vwri6qsbtqBEHs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=klp1mR5jnzOgqpzhCOj0WFwg1FgPJMsPSYNOwa6lgfEEIQFXYfuMOitVf9M1XYkq6 IGUAspcT+VDB83shIuawyyQD76502CYv0vKgxgM4BxAefQK2MfCXbbO1iLLsINdsjh KlslEpEoitr+BYqJBjI3e/ETQkdLUZvQ4xyje68bjFyGmUyXvU1JnGORIXnpvvpr7m Jt4in8dWHpPjxlQL2+RyHHtT8kKLQsWuqaIaJ7j25YV47di0urpRAy+FuWXJYitzYx yFJE97Sh5w8wuy/bCnJh4CWWpjWg1Sp7HRetTJknpR4bxWXx1JwSfwSrSFrjdNKOQf bNTadbY0LBN+Q== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 05/16] mm/mglru: batch update lrugen->nr_pages in inc_min_seq() Date: Wed, 12 Aug 2026 20:16:47 +0800 Message-Id: <20260812121658.69965-6-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero. Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size(). Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 46 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 2fd1e06eb192..c6e3b92c8cae 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3295,20 +3295,21 @@ static int folio_update_gen(struct folio *folio, in= t gen, const vma_flags_t *vma } =20 /* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio) +static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increas= ed) { - int type =3D folio_is_file_lru(folio); - struct lru_gen_folio *lrugen =3D &lruvec->lrugen; - int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags.f); + int new_gen; =20 VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); =20 do { new_gen =3D ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; /* folio_update_gen() has promoted this page? */ - if (new_gen >=3D 0 && new_gen !=3D old_gen) + if (new_gen >=3D 0 && new_gen !=3D old_gen) { + if (increased) + *increased =3D false; return new_gen; + } =20 new_gen =3D (old_gen + 1) % MAX_NR_GENS; =20 @@ -3316,8 +3317,21 @@ static int folio_inc_gen(struct lruvec *lruvec, stru= ct folio *folio) new_flags |=3D (new_gen + 1UL) << LRU_GEN_PGOFF; } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); =20 - lru_gen_update_size(lruvec, folio, old_gen, new_gen); + if (increased) + *increased =3D true; + return new_gen; +} =20 +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio) +{ + int type =3D folio_is_file_lru(folio); + struct lru_gen_folio *lrugen =3D &lruvec->lrugen; + int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); + bool gen_increased; + + new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); + if (gen_increased) + lru_gen_update_size(lruvec, folio, old_gen, new_gen); return new_gen; } =20 @@ -3903,6 +3917,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) struct lru_gen_folio *lrugen =3D &lruvec->lrugen; int hist =3D lru_hist_from_seq(lrugen->min_seq[type]); int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); + int target_gen =3D (old_gen + 1) % MAX_NR_GENS; =20 /* For file type, skip the check if swappiness is anon only */ if (type && (swappiness =3D=3D SWAPPINESS_ANON_ONLY)) @@ -3915,32 +3930,41 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, int swappiness) /* prevent cold/hot inversion if the type is evictable */ for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; + unsigned long delta =3D 0; =20 while (!list_empty(head)) { struct folio *folio =3D lru_to_folio(head); + long nr_pages =3D folio_nr_pages(folio); int refs =3D folio_lru_refs(folio); bool workingset =3D folio_test_workingset(folio); + bool gen_increased; =20 VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 - new_gen =3D folio_inc_gen(lruvec, folio); + new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); - + if (gen_increased) + delta +=3D nr_pages; /* don't count the workingset being lazily promoted */ if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { int tier =3D lru_tier_from_refs(refs, workingset); - int delta =3D folio_nr_pages(folio); =20 WRITE_ONCE(lrugen->protected[hist][type][tier], - lrugen->protected[hist][type][tier] + delta); + lrugen->protected[hist][type][tier] + nr_pages); } =20 if (!--remaining) - return false; + break; } + WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone], + lrugen->nr_pages[old_gen][type][zone] - delta); + WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone], + lrugen->nr_pages[target_gen][type][zone] + delta); + if (!remaining) + return false; } done: reset_ctrl_pos(lruvec, type, true); --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 941463E8695 for ; Wed, 12 Aug 2026 12:18:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537109; cv=none; b=t3G1g5edPQ7SETXhgky6oIAayeIAC+TWdPjYRoy/xJ6DOtVWdzZYsAaC8SYKg78REZD+Z2D0HKAiUw0xDPw8Tte8GAvQtbysKkwfqRd547D1axJsXwZGNejigUfFJzir/EHiHlwOUIwNOaBzp7a0ZxB4PKTzv0XhoTAaFsFc80o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537109; c=relaxed/simple; bh=bjTlo5q4inqWpZchLdWAj07VC2VV4wp6sZQCGMSy6hU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RPZgSOTKlt5l6iA5/sVvC2G/IAlzLy2AhZZyX10kq4LJ5u/zmkYBHK6un2ks+Vfj/nEscK1SCMiraDmkDWLVI4EQL5qlaTxmwm6H20hshFFqzA3WtQ3ssPl65a8LostU3cbZ9t1l9zCTcONNOLUWt9uaWFLCS+D0KBOL+MdAy7I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hlfLM8xQ; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hlfLM8xQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B1E41F000E9; Wed, 12 Aug 2026 12:18:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537108; bh=//ieBRIhh4nll3qPbLkaxIxKpZguGkr7hzBbi75fbu4=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=hlfLM8xQi8KQaXw+J7ZyMU+xzBNGMgaQJ38nanoZDFLF92B7RFmSJxOSjC6btYMfB jNcHmL0DSGsLmeaLjaNVO7gk9eIy7/f2tvDXVxVJ/H/4sdMLRpaip0ODjNNxmnR1gi p6KKiBiOnd1Jmvq0r/4MEIKAcNNvRQ30kd6uaalEdXvoArBlDGcWhQRHbszHeHRDui PBwcUJc2MXMKyi4PURpglOsBz5RTBqtC+n3MctaQz5nXJxclzkk9631VKhIsK9vb/c RcN8AYZqgMq3arGkjouIxZQ5CGb1rC46IlLuAUjVMR8qyF+BpOUW3DLDDj/R4p2g0F SHz7g8ARwdt5Q== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 06/16] mm/mglru: batch update lrugen->protected in inc_min_seq() Date: Wed, 12 Aug 2026 20:16:48 +0800 Message-Id: <20260812121658.69965-7-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Avoid updating lrugen->protected with WRITE_ONCE() for each folio, which may prevent potential compiler optimizations. Accumulate the updates locally and apply them in a batch instead. Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index c6e3b92c8cae..10f690389f5d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3930,7 +3930,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) /* prevent cold/hot inversion if the type is evictable */ for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; - unsigned long delta =3D 0; + unsigned long protected[MAX_NR_TIERS] =3D {}, delta =3D 0; =20 while (!list_empty(head)) { struct folio *folio =3D lru_to_folio(head); @@ -3952,8 +3952,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { int tier =3D lru_tier_from_refs(refs, workingset); =20 - WRITE_ONCE(lrugen->protected[hist][type][tier], - lrugen->protected[hist][type][tier] + nr_pages); + protected[tier] +=3D nr_pages; } =20 if (!--remaining) @@ -3963,6 +3962,9 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) lrugen->nr_pages[old_gen][type][zone] - delta); WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone], lrugen->nr_pages[target_gen][type][zone] + delta); + for (int tier =3D 0; tier < MAX_NR_TIERS; tier++) + WRITE_ONCE(lrugen->protected[hist][type][tier], + lrugen->protected[hist][type][tier] + protected[tier]); if (!remaining) return false; } --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 529503E8695 for ; Wed, 12 Aug 2026 12:18:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537113; cv=none; b=KbHGpQ/b0CNPYETsWd1iZwX1dlzFAlQuPHTa+vMOVpWTymhFdeoNBdvgZoITnOC8Aa8MF2xjtMAtS6tqkr7v+9lXNhzPewmTSk2C/UQEIAktJaIyWAXTNDvsgIasQ1/QEPnJBghCL/AdnmHa8eo5lZjyacb7ez9EPnl7ffUb650= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537113; c=relaxed/simple; bh=btWGq5vav8v0MNojsl686lGB1R9KU1D7U8gq75kk7yA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mlCHVHuQz1JfiYsGWcOlN78GLU0eNaeygoMwRxVeDxzUoxB/QB2ieA67qJ8IoBHkfCqNC67dnDl5eAlQAzCE8+WgPKo2CCL9wEbFmPZA0nOCANhT+OBiINI825h7b4hd0IjwzgreDllmtWvyGYxMJvB7GbRSPmYfa9B3z+wWwCI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=enrNi23S; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="enrNi23S" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 91F591F00A3A; Wed, 12 Aug 2026 12:18:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537112; bh=sjn66HBL5BcMiFsq1jE87/70ep2gVw0orOdl941ZWGU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=enrNi23SNkmgjfzNIYWqLPhi43g/P7K8nbDsnMcP+6+uxoBixq/bV9gVYeWsuyguJ ppEnvFW69IqTOvOvfr1omUrs8ZeVdrDQFUzGNtX1JEdD11iCbix4vid3E5446+lStn nMlgghr4dI694h/VFrm26NTY1WdEkPAwF96QzLq41P+KSnUZSWymwWa7ylOM8sGEEM E2OKPZstuIotUFg8CmJgeGVVU7VFD6AFnjdktFo6Mpm6r3JtD7rSrBY8H7wFigegAp vbA9TNxxTriMqfC3Io7VCrSUgCT2v2Yreq+8oXqlRgOaNsJ+Ji/HLuO1JjWWCSpWd8 uNt/QOqywdcHA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 07/16] mm/mglru: enhance cold/hot inversion handling in inc_min_seq() Date: Wed, 12 Aug 2026 20:16:49 +0800 Message-Id: <20260812121658.69965-8-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation. It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios(). Otherwise, promoted folios could end up behind folios that were never promoted, effectively inverting their hot/cold ordering. Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 10f690389f5d..e0625e6ec919 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3945,9 +3945,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int t= ype, int swappiness) VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); - list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); - if (gen_increased) + if (gen_increased) { delta +=3D nr_pages; + list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + } else { + list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); + } /* don't count the workingset being lazily promoted */ if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { int tier =3D lru_tier_from_refs(refs, workingset); --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5978D3E8695 for ; Wed, 12 Aug 2026 12:18:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537117; cv=none; b=WJmmvrA6ECF1jBAMULSp9kG6rI7fGIlcxWU4BaX+GsAY809eVndLURszN3v0MK/jXGN00iA8oiZjcP76k/f7Sltozbms/NNGZTMLjva0PEfMPo6/R9SfPGMsnpkoTBnwWa1ScwnyT6bZb9H/c7L0J5C03Rc5BjBT2l2AH2lQQHc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537117; c=relaxed/simple; bh=YDpMDfAluWaXhaUYKO8VKxIGp7D7Z7+JpXYlxCFhFd4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oke3N7S7/47PFBOei+vLkSn8kHkUfqZjkOB5BoYUcagzHO0RRTpzG5xUAoTC2da0FYkBzusVlPMPhCylDiQfDHPm6I5j3eIasR5V46KSfWGpYsszfbtp3RDXna637Qt9nRPxLQXElZjU5nH87Lmp7/AgtXdXNpVksmdepI6fSPk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JdxppSf9; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JdxppSf9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9BBE71F00A3D; Wed, 12 Aug 2026 12:18:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537116; bh=LWQXEeuSvxK5SjOlteLHcHA1IM+jE7w1Hqw3xSPnRB8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=JdxppSf94PuOpF0SADdQMVWKqkvDo+ItECrjI0+o4AY68ly6P/63VQSpV9X1KgGvN WLuQ8gsxYp4Ngn2vqpqZ1K1tfeslEpCisrI8jcqa7HKPK1lT1stozgK6kIHf0FbywM cxfjEiWnER0RFxYX8Lt5WcfX5410UHVstMh5YMZTBwunWRre6eisUZfCHh4UeVawua RJga6ONZ3Ezai3ZWO8u2ezQw4XdrGTVtQKrjHJj7rZDHnxVuY9KQ0fQ3LF4VjxJAc5 f28gSROyPaRzuegmbNo0x1tPvmYc4CHjDJfv4nYfDjr6Shy4V+dk1pCv/GMuqpaJO6 EB4ah+nlQkUqA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 08/16] mm/mglru: exclude folios promoted by aging from protected in inc_min_seq() Date: Wed, 12 Aug 2026 20:16:50 +0800 Message-Id: <20260812121658.69965-9-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some folios may have been promoted during aging, so don't count them as protected, similar to sort_folio(). Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e0625e6ec919..27623d3bad95 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3948,16 +3948,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, int swappiness) if (gen_increased) { delta +=3D nr_pages; list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + /* don't count the workingset being lazily promoted */ + if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { + int tier =3D lru_tier_from_refs(refs, workingset); + + protected[tier] +=3D nr_pages; + } } else { list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); } - /* don't count the workingset being lazily promoted */ - if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { - int tier =3D lru_tier_from_refs(refs, workingset); - - protected[tier] +=3D nr_pages; - } - if (!--remaining) break; } --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE8BC424664 for ; Wed, 12 Aug 2026 12:18:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537121; cv=none; b=TsnZno8wc8Pw709anCT4cLxawJM6hucz6H5fxcYv5a/0TprZmQ6ifMTmrCmVxxD41E2HUBfBhEvC9Yu3MTczQsqfGSao88XO1jXdwl1UtithLT9Cqxm5xuNE6SZEydAx1Xzx0ty2abtmVSFQ5MJdmv60xTxXL6kmOxm8yvVrCOU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537121; c=relaxed/simple; bh=YHjzv4WB4NI+PX/Via7hripVyOrsxTsZTgw7P+y4oXc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jWBouIAJpGEkJr7V20+Qj5cvoAOvGc3X5x8IadQcUamUjTf2hralacf7abvVG1q4md2vLWvtuDoOYGTEgb+2227/Q7VVLw3pQ2wnVZphbYuVLD7aLUCsEMdT+Cf8LYcr6mt26WgefoNtFILqEGG41JxdZDVUM0Mp99oIbPI/524= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JkxTYDQC; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JkxTYDQC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A29271F000E9; Wed, 12 Aug 2026 12:18:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537120; bh=c/HS6jeoNkCyji18aZKv6rudVAWLmw8SzAirPydXjGI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=JkxTYDQCLOuxfjIVK740eZCL70XL0BrkAWHVF8LXo5EPShGtbz6Wlsgzcn9T8kzjh 8Zka2Q3iylt2GAX9xst0IU8Y/Rcx5fCLyE+KF005pPWfFdu8KtlUCNPFACO4dfA3DX ElrwzhHLbC5iGlbsLcI4M8ojeYUzgn1Vc5w34z/tsQWKkpFUYbovv04YXLaUvBEN66 Nj3g34foMtUhtdisxEiU1aZD/VD73VXAj3c/YuOiYcaIWYZgBWnIgeWkUnRuO6WcIn OMhxsX0gcoe/ZdgTA83FBiTH7O0XFRvPoa9WEpVx1EUbGFNZYcUNeCCVQK4RbqJaPy ethfRhdyn/pcw== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 09/16] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail Date: Wed, 12 Aug 2026 20:16:51 +0800 Message-Id: <20260812121658.69965-10-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For reclamation, it makes sense to reclaim folios from tail to head, as folios near the head are relatively hot. However, when moving folios from the oldest generation to the second-oldest generation, using the tail-to-head order would effectively cause a cold/hot inversion. Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 27623d3bad95..17357b16d1a7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -190,8 +190,20 @@ struct scan_control { prefetchw(&prev->_field); \ } \ } while (0) +#define prefetchw_next_lru_folio(_folio, _base, _field) \ + do { \ + if ((_folio)->lru.next !=3D _base) { \ + struct folio *next; \ + \ + next =3D list_entry((_folio)->lru.next, \ + struct folio, lru); \ + prefetchw(&next->_field); \ + } \ + } while (0) + #else #define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0) +#define prefetchw_next_lru_folio(_folio, _base, _field) do { } while (0) #endif =20 /* @@ -3931,9 +3943,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int t= ype, int swappiness) for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; unsigned long protected[MAX_NR_TIERS] =3D {}, delta =3D 0; + struct list_head *pos =3D head->next; =20 - while (!list_empty(head)) { - struct folio *folio =3D lru_to_folio(head); + while (pos !=3D head) { + struct folio *folio =3D list_entry(pos, struct folio, lru); long nr_pages =3D folio_nr_pages(folio); int refs =3D folio_lru_refs(folio); bool workingset =3D folio_test_workingset(folio); @@ -3944,6 +3957,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 + prefetchw_next_lru_folio(folio, head, flags); + pos =3D pos->next; new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); if (gen_increased) { delta +=3D nr_pages; --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7018843B3FF for ; Wed, 12 Aug 2026 12:18:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537125; cv=none; b=GO2nVVa5BR1+QgVoMJ6yyUQ2RVBc/HgGICwCvF8g8j9I5RTLKhHp/svtE3Tce5NpQgPbHgTWkifY/zfoTD2c+hwJYjCH3+mMF8jN0FnTpkFmHoN1b+4+s51ir29A6ET/iJN/iYKGeC5+wOXFUU6jccHKiRbopiwFmPVa/GsgqMY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537125; c=relaxed/simple; bh=ZXzzA4BA5h7s6CyPi5Vt5iioNBmHdY3MGLTqW7AGs7o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=osLHzB/3rimSb+JaJuenL3AemDEMV6PSkkmDqJPM8hKwAI7ztPY3QuDSWSxKYmZ0fBuLkiHdc5HB2nkDeD1Td2HWOx4kE7RmrDCE/H/QHPlVo+BOpAcUmKX3dA/pkht5ZUiK+OGVICzsA/WYLxy0JnwuMxwGJomdink+XCtIx6Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CSCdLo2l; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CSCdLo2l" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AC4DB1F00A3D; Wed, 12 Aug 2026 12:18:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537124; bh=emxjHhjwy8XERbEMKSGImjrtQvsEUGjXFQEU42dLbqE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=CSCdLo2lMszB+gAPNugQg54MeL7QysW31P6OWjRQq9cOcOyMtBbRENzoISFusdZC7 yDbqUl23M4nN3rsQXMg1HlY1eOjDdTYed9qr3wScscQGuILuFPQCKxPxqXMP9qS97/ iTgfwTji37orfI88Ay+z0i7yO8mIvkzIomW5YymLrAoBoFrSI3aGuc2oz78TQNLOEl ceKJWs5rwxF/f6hYcsc6oWXnt/tqZa0taIiPfxKUvQEdfasL5hBWCKBs0cqgiCQWAW LdNa0w/m0X6lIDnkn7pxyWmwxbxon3NBjm6FMAybMzYbawnfwnL3Ij/B/mXIAx5aFb 5I1i6WbT/ICQw== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 10/16] mm/mglru: batch move folios to the second-oldest gen's LRU Date: Wed, 12 Aug 2026 20:16:52 +0800 Message-Id: <20260812121658.69965-11-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Detect folios that need to move from the oldest generation to the second-oldest generation, and batch-move them together. This can significantly reduce the sys time of inc_min_seq(), especially when the other type is significantly behind the preferred type. Assisted-by: gemini:gemini-3.6-flash Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 17357b16d1a7..b6c17ece3b3f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3922,6 +3922,19 @@ static void clear_mm_walk(void) kfree(walk); } =20 +static inline void flush_lru_batch(struct list_head *head, struct list_hea= d **batch_end, + struct list_head *dst) +{ + LIST_HEAD(movable); + + if (!*batch_end) + return; + + list_cut_position(&movable, head, *batch_end); + list_splice_tail_init(&movable, dst); + *batch_end =3D NULL; +} + static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) { int zone; @@ -3941,9 +3954,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int t= ype, int swappiness) =20 /* prevent cold/hot inversion if the type is evictable */ for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { + struct list_head *target_list =3D &lrugen->folios[target_gen][type][zone= ]; struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; unsigned long protected[MAX_NR_TIERS] =3D {}, delta =3D 0; struct list_head *pos =3D head->next; + struct list_head *batch_end =3D NULL; =20 while (pos !=3D head) { struct folio *folio =3D list_entry(pos, struct folio, lru); @@ -3962,7 +3977,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); if (gen_increased) { delta +=3D nr_pages; - list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + batch_end =3D &folio->lru; + /* don't count the workingset being lazily promoted */ if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { int tier =3D lru_tier_from_refs(refs, workingset); @@ -3970,11 +3986,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, int swappiness) protected[tier] +=3D nr_pages; } } else { + flush_lru_batch(head, &batch_end, target_list); list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); } if (!--remaining) break; } + flush_lru_batch(head, &batch_end, target_list); + WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone], lrugen->nr_pages[old_gen][type][zone] - delta); WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone], --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 878A943B6CD for ; Wed, 12 Aug 2026 12:18:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537129; cv=none; b=Zskg7RcdwN+W/Ah+tu8GCi+Nqp3ox5enc/VyvoSdNMbFF+h2nMh2JTUJVj5qbd3Nda7urjD/ZnEIzepXW5PrJiBQidU8cKq1jvdom6ToXCURuc4/N7MU/VPVhn0Po9Ihhi+WRe3xy6EKGjTsKgZUPD5SbTyH9tX0tuvrThYlXfo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537129; c=relaxed/simple; bh=cL6TeHp3CJhfddBWBKj5iQj5eHcGqe1BtId8PCgl7o0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=U6Bdg6zAro7C8frtMWwFjSwD/T3Bt52k9ahhBxaJOcGbDJOTflCAQaF/T6QVXeAXcPWIzjaaWbNWo1wC66N3IDO3Su8x8eKcYbGj1O1QkUjd91e6UOOM4KDULH8BACq8r1XovwmvWl4r+8L6kcHVjpqZpTHNlktWH7GH0uOGpAc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TDHRF0AA; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TDHRF0AA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B70D21F000E9; Wed, 12 Aug 2026 12:18:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537128; bh=/7y18sjno+mqqfvWT6M0Hi+njMwHX33SQLnmAtAwqGs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=TDHRF0AAx2OOlCm8oAlPxzA5y9oDBvujnLCSFyymC/pSzwTLhFqIjrF1qgCL+S6NC DHXGcfnZOjAWJKb84DOaCDsJa6v9Gu9OqqMyJ2+T3GCeV3qQEx9DDbXCIAEZgw50bn 17NeIAML5u6HK4eslYC7gpncgsV1041VbVBJYK7xUxCteC+Tbfd9AjKmrQZjrrAQfN B/cFvSsV8oAcNQy+IuR55UQXb+4kBk2WH29LN9D+8nTMRvCw9/liM8Hv+zItrxJCDK Vc1XiuvW3qT8RJBbT+SrqzX7T00HoUhIIhrjrvk8rwYlIuZ1yx3cGqqhE0O/J3zXCl Ao2pt8sE1CI7g== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 11/16] mm/mglru: skip gentle reclaim at DEF_PRIORITY for extreme swappiness Date: Wed, 12 Aug 2026 20:16:53 +0800 Message-Id: <20260812121658.69965-12-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For extreme swappiness (<=3D MIN_SWAPPINESS + 1 or >=3D MAX_SWAPPINESS), skip gentle reclaim. Falling back too easily, even at DEF_PRIORITY, undermines the effect of extreme swappiness. This matches the logic in get_type_to_scan(), where we have: if (swappiness <=3D MIN_SWAPPINESS + 1) return LRU_GEN_FILE; if (swappiness >=3D MAX_SWAPPINESS) return LRU_GEN_ANON; Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b6c17ece3b3f..48a068b50678 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -281,6 +281,13 @@ static inline bool is_exec_file_folio(const struct fol= io *folio, return vma_flags_test(vma_flags, VMA_EXEC_BIT) && folio_is_file_lru(folio= ); } =20 +/* See get_type_to_scan(): these values always select FILE or ANON */ +static inline bool is_extreme_swappiness(int swappiness) +{ + return swappiness <=3D MIN_SWAPPINESS + 1 || + swappiness >=3D MAX_SWAPPINESS; +} + static void set_task_reclaim_state(struct task_struct *task, struct reclaim_state *rs) { @@ -5094,8 +5101,11 @@ static bool should_run_aging(struct lruvec *lruvec, = unsigned long max_seq, if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq) return true; =20 - /* try to avoid aging, do gentle reclaim at the default priority */ - if (sc->priority =3D=3D DEF_PRIORITY) + /* + * Try to avoid aging by doing gentle reclaim at the default + * priority. Skip gentle reclaim for extreme swappiness. + */ + if (sc->priority =3D=3D DEF_PRIORITY && !is_extreme_swappiness(swappiness= )) return false; =20 /* better to run aging even though eviction is still possible */ --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 685EA440A34 for ; Wed, 12 Aug 2026 12:18:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537136; cv=none; b=ikPAHNSooMaHdCL8f9MxoLae+JJsIZkKe+EXMyY4LO+Cuzti/UcwC4Khna+HLZ0HGDLz5VT4ybJjhNpRI1yj9iqHS0/0k4sXaMHf83pxTBPetj6rj/gWLH0Yns+bHMKISVem7SYZRmxNgnZYVV/jtdOC2QjgVWSFgoV7mx6Qqzc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537136; c=relaxed/simple; bh=H5ELtPOVpxqId5aU4K0Mma3n2wXR9Lf19mxIs/3ZFks=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PpnZpdko0Kd3lHIuuN+JZMQHNqnyIvCG1bhHHlvubATYjCZSTP7vrWxtNFPHpKGdvvwGujYvX5+L305vb0H3S1k2iFF6CtRPGY8PpXkQP3rTmId2V/08eoAmqp6e9TblOnqzql9GDrFM1FpfJWXUecW+K3MCe9FWTEY+Ab2ZWHo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WoXwSgKa; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WoXwSgKa" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E58E1F00A3A; Wed, 12 Aug 2026 12:18:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537132; bh=1NLPR+0L7qhjEJJeaSxotIiJRVW8SFQExxb6Lq3NwJc=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=WoXwSgKa0CZv7qDgc1Ukq/vS07Gm7Y1XZ0lvRuWfv9ssh6JlfQt+0bAcsRK7khMp/ lpiVJn224v8r5LLmDuDF1H6BGMeTOoiq580CTgK8JOeMhG41g6/VBdYEdYixFNt69X Zho0YLfwVt82Ud/8dbQKGFJkj1yTRmfLvNjenz2oAvRENz1/mSyoegkKDv/lP0effl undfLVTkDZ4HX/EGIUwDlQRAzGDrdwKRnkFlc42GzS7PUq76FCmgUffzifhPgsGUo3 +oM8D+PbsmNuOXYCBfKaRiFTQdqeu81YA3aU+qGVPPAi344fSqOAxm2AttTeOghPew wA8qvUI/dyf6Q== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 12/16] mm/mglru: run aging if the preferred type has no reclaimable gens Date: Wed, 12 Aug 2026 20:16:54 +0800 Message-Id: <20260812121658.69965-13-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Respect the type selected by get_type_to_scan(). If there are no reclaimable gens left for that type, run aging. Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/vmscan.c b/mm/vmscan.c index 48a068b50678..50f0bcf9576c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -5095,12 +5095,17 @@ static int evict_folios(unsigned long nr_to_scan, s= truct lruvec *lruvec, static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, struct scan_control *sc, int swappiness) { + int type =3D get_type_to_scan(lruvec, swappiness); DEFINE_MIN_SEQ(lruvec); =20 /* have to run aging, since eviction is not possible anymore */ if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq) return true; =20 + /* run aging if the preferred type is exhausted */ + if (min_seq[type] + MIN_NR_GENS > max_seq) + return true; + /* * Try to avoid aging by doing gentle reclaim at the default * priority. Skip gentle reclaim for extreme swappiness. --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6AB8843BDA1 for ; Wed, 12 Aug 2026 12:18:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537139; cv=none; b=PY6kTOA3sSK3i9/NvQSzYEjplTYjml+hOh4XvFwMB0uMZiBhHiCL0wjYHljeUlbMli+2SGkdnRvfoq7A+UrT8rr/fmiADOSridEUOMsKgJj4DQM6cK5U0o0vt44cyWyHvToF3S6oBLbvNnmSeaJgfo0FKr4AuvVdSfqFnO6p1mw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537139; c=relaxed/simple; bh=7EYCiJRcVCzHthjgLI3vVoahusfiqrUgf7UEd10aNGg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gs+nAHgl2pnmmjt9bPK9nIeP08laQ3bBsI9QV9id7y4KvczROcsuJ2PNnCkB5+93IjsmbFzp6lLmwHr2YtUmqdnp2QTFqL8TWLVXUZN9LnEoghsgVLD7T12TEiNt8yuEBY21FDzDQ4Cx5bAFg+BSbShQ7hqsP51HhFi9pVVzKF4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KIQ1U2Fr; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KIQ1U2Fr" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A67211F00A3E; Wed, 12 Aug 2026 12:18:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537136; bh=JQPoDdD5OLaDoL3cib3mD06LEWmyAGJqyFzrLrL7XwM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=KIQ1U2Frf0x661YHoG8mZNUjmwleI0PxWMk3/LU8npUPFUiSPNtjx/IhiwDh7OgiX Gna9MI2ipdDabXPGk2MZ91adiLzJg1kHYBPremNpuE3v6EL12cPGXSdBHDASEC8iNV Qp/Dfxv/F6TdJnIulo/qgrr3ssGVOqPTtG3rZTqp9ZeCvW7c3pqA19KzAfi30tOIMD 0sLlN7u/TSEy5ddjRa6FKZ9O4Tw2uLSMRscyDKiQYnqq01Gpg1QImxLgjIEPdFuBNr cSBHwzsKqccw2OO8LCBmR+FfZoUFkXziyxcH2qgSk5PyZi3C8CjG7j0hv0WyoB9PpW t7s/ACJ85q3Zw== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, Barry Song Subject: [RFC PATCH v4 13/16] mm/mglru: remove redundant gens <= MIN_NR_GENS check in should_run_aging() Date: Wed, 12 Aug 2026 20:16:55 +0800 Message-Id: <20260812121658.69965-14-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Bo Zhang If the following condition is true, if (min_seq[type] + MIN_NR_GENS > max_seq) return true; then this one must also be true, if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq) return true; The converse is not true, so the evictable_min_seq() check is redundant and can be removed. Signed-off-by: Bo Zhang Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 50f0bcf9576c..a276560bc66b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -5098,10 +5098,6 @@ static bool should_run_aging(struct lruvec *lruvec, = unsigned long max_seq, int type =3D get_type_to_scan(lruvec, swappiness); DEFINE_MIN_SEQ(lruvec); =20 - /* have to run aging, since eviction is not possible anymore */ - if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS > max_seq) - return true; - /* run aging if the preferred type is exhausted */ if (min_seq[type] + MIN_NR_GENS > max_seq) return true; --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BB6F4398F1 for ; Wed, 12 Aug 2026 12:19:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537145; cv=none; b=ihlSG4j5DiX/XzrszTQkj0vGso4522RMKZdql8+K7bmWTBoRx+lWFEoxJpOMkay620wqb3J17dNZidKSoWddmYzCoTTV1S2FeTsi4fdo8ZhV0CaEu/qbIB5UFeit/n5l9vpaJHGKSfdD6SHbI5RkfW/eTkI9eqlVlsu28nhkuZA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537145; c=relaxed/simple; bh=5Vi0aACqDNHGS+E+qWnp4LuFVGcj4UIZjf0cvj5xagQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BYH2BmpCPemSVfgS2rogH/c/Lx29jzK0f5H4Lkk6bqzdRHLn1DPIFWghz800FOmp2WtX1JYX/gGPhQtN+ouyVBDbgjjAsajetu6w5+eVJslM0eLcwayEBSG/jRrcegEP2fiy7Ij/HLzDxOonn3tN3EXKDFv9Jc4kzi9YCkai3ks= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LBhdAmMP; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LBhdAmMP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B32641F00A3A; Wed, 12 Aug 2026 12:18:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537140; bh=EjEFNnCpD4D3Lv/lQ5ONSKNhGjrzUmBvcQ2L3a1JEFU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=LBhdAmMPlplf3Bg92sH4iyezEuXbZq+Aty9njEgnfcU6LnlXUhz4A21zPKV9HLkMl ER86pfSDbHephWBRVfUOxQvwVHdGdvrvYsrFhzgMiiCK61wNOrHku4XGoc7hmcaY/o ZX7/uoYReatWEjiHsybiI12sdGVsB4Ms0nZx6VgVrbriv28ehJ8nwzBX9i5MToUJhb x07/d5PhUaaUT5WGDgYJzmegz+wWP2umpkoX0Hh7gr1a0XU+3a6NmK9/M6lkkkciXo FBsv1wRRSEbgWzyGCbtY/bpL1I4z3uiN07VoDfmKzkjIESjlPG09aV94gLQz9wua8X FykVKjgdS6+cg== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, Barry Song Subject: [RFC PATCH v4 14/16] mm/mglru: run aging when pages are severely imbalanced across gens Date: Wed, 12 Aug 2026 20:16:56 +0800 Message-Id: <20260812121658.69965-15-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: lyugaofei This partially restores the reclaim behavior introduced in Yu Zhao's initial MGLRU commit, ac35a4902370 ("mm: multi-gen LRU: minimal implementation"): /* * It's also ideal to spread pages out evenly, i.e., 1/(MIN_NR_GENS+1) * of the total number of pages for each generation. A reasonable range * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The * aging cares about the upper bound of hot pages, while the eviction * cares about the lower bound of cold pages. */ if (young * MIN_NR_GENS > total) return true; if (old * (MIN_NR_GENS + 2) < total) return true; But with a stricter condition: the younger generations must contain at least MAX_NR_GENS times as many folios as the older generations. We also consider the cost of inc_min_seq(). If the oldest generation of the other type has fallen significantly behind, pulling those folios from the oldest generation to the second oldest generation can be very expensive. In this case, skip imbalance aging unless extreme swappiness is in use. Signed-off-by: lyugaofei Co-developed-by: Barry Song (Xiaomi) Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 50 insertions(+), 7 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a276560bc66b..d6fac5b91ac1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4219,20 +4219,28 @@ static void set_initial_priority(struct pglist_data= *pgdat, struct scan_control sc->priority =3D clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY); } =20 +static inline unsigned long lruvec_gen_size(struct lru_gen_folio *lrugen, + int type, unsigned long seq) +{ + int gen =3D lru_gen_from_seq(seq); + unsigned long size =3D 0; + + for (int zone =3D 0; zone < MAX_NR_ZONES; zone++) + size +=3D max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); + return size; +} + static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swap= piness) { - int gen, type, zone; + int type; unsigned long seq, total =3D 0; struct lru_gen_folio *lrugen =3D &lruvec->lrugen; DEFINE_MAX_SEQ(lruvec); DEFINE_MIN_SEQ(lruvec); =20 for_each_evictable_type(type, swappiness) { - for (seq =3D min_seq[type]; seq <=3D max_seq; seq++) { - gen =3D lru_gen_from_seq(seq); - for (zone =3D 0; zone < MAX_NR_ZONES; zone++) - total +=3D max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L); - } + for (seq =3D min_seq[type]; seq <=3D max_seq; seq++) + total +=3D lruvec_gen_size(lrugen, type, seq); } =20 return total; @@ -5092,6 +5100,37 @@ static int evict_folios(unsigned long nr_to_scan, st= ruct lruvec *lruvec, return scanned; } =20 +static bool lru_gen_imbalanced(struct lruvec *lruvec, unsigned long max_se= q, + struct scan_control *sc, int type, int swappiness) +{ + struct lru_gen_folio *lrugen =3D &lruvec->lrugen; + unsigned long young =3D 0, old =3D 0, lag =3D 0; + DEFINE_MIN_SEQ(lruvec); + + /* we still have enough generations to reclaim */ + if (min_seq[type] + MIN_NR_GENS < max_seq) + return false; + + /* + * Trigger aging if the preferred type is running low on reclaimable + * folios, provided the generation lag of the other type remains small + * enough that inc_min_seq() introduces negligible overhead + */ + for (unsigned long seq =3D min_seq[type]; seq <=3D max_seq; seq++) { + unsigned long size =3D lruvec_gen_size(lrugen, type, seq); + + if (seq + MIN_NR_GENS > max_seq) + young +=3D size; + else + old +=3D size; + } + if (min_seq[!type] + MAX_NR_GENS =3D=3D max_seq + 1) + lag +=3D lruvec_gen_size(lrugen, !type, min_seq[!type]); + + return young > old * MAX_NR_GENS && (lag < MAX_LRU_BATCH || + (is_extreme_swappiness(swappiness) && sc->priority > 2)); +} + static bool should_run_aging(struct lruvec *lruvec, unsigned long max_seq, struct scan_control *sc, int swappiness) { @@ -5110,7 +5149,11 @@ static bool should_run_aging(struct lruvec *lruvec, = unsigned long max_seq, return false; =20 /* better to run aging even though eviction is still possible */ - return evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS =3D=3D max_se= q; + if (evictable_min_seq(min_seq, swappiness) + MIN_NR_GENS =3D=3D max_seq) + return true; + + /* Run aging if the preferred type is severely imbalanced across gens */ + return lru_gen_imbalanced(lruvec, max_seq, sc, type, swappiness); } =20 static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc, --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8773A444708 for ; Wed, 12 Aug 2026 12:19:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537147; cv=none; b=esfwIdH0tQV9mnN5X2H0GkPRCTwMh3NloN+RjCQTGD+XwP9ADwvXdwWBX4JL9nVUqb1XMcu/+nWhj17eLB4tKhemWqkCMo9PB7zwoUyIrI4P1nMtgnIx/Gp8TTygOSC8rk04Mz2XllD/DHoVwslmgb1m3kZrY2eAeWLaTEWLKsc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537147; c=relaxed/simple; bh=FUdDUnzXr+2z6a3eQ4V+h/9C27f/kTZTPh0P8QioeeI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AwgPHdLWPphCFQXL0L/K13S13a7Mn+QNezSHgiEKq8Q0vjtLoCeL3pxIJf2zU50RZ7Z4G4N3FSvEh/rDJ6SKrlNZE9TJGDtc5gV8Ay4XYvyX7F+RhELXvaEXhHnd/syEn3rNeQefcjbYI+TEqHR1L9X5pGJYPkJpChIBvzawJyY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=njo3WFSy; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="njo3WFSy" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BDE751F000E9; Wed, 12 Aug 2026 12:19:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537144; bh=3bXsQJXJBgnmO/j5UVCnEEv0tpGrhoZfdxf5g4cVqXw=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=njo3WFSyGDaScgvd6IJHBjqnoSSXc8eHsmqYVtIT+z+kGiY32zHCIHLG1dZdKTzkf WsZF140sZ7HXeZZVQtvkV1HLI/EkPJ83p2zLnENNBOMKYNJoq55pjYH7TDSVkmlZlj Tl0wAonC52TwGwAIy/4hFNR0O554QLM2aGdnQcYe81OocBi8AUfRlATAaXVym5qxFs OgaCK6kKvE4kxmUDINcInx3KnAkZF79yEGzDsgTzFDKJhmUlJkdQShEcuwmFoGe0Bm BM8Ry2v+SmHs/ttJbHE6zToz9DjNEnXqjJ4Ze8a8gAp0gG0om5TAHkDywVs5qapZ3X YKB018bfePxyw== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 15/16] mm/mglru: dynamically scale aging threshold in lru_gen_imbalanced() Date: Wed, 12 Aug 2026 20:16:57 +0800 Message-Id: <20260812121658.69965-16-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, lru_gen_imbalanced() uses a fixed threshold (MAX_NR_GENS) to determine whether the young-to-old folio ratio warrants aging the preferred LRU type. This fixed threshold may not scale well with large memory systems. Borrow the adaptive ratio calculation from inactive_is_low() by scaling the threshold with the square root of the memory size in GB. Suggested-by: Zicheng Wang Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d6fac5b91ac1..c0350a8b61d1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -5105,6 +5105,7 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec,= unsigned long max_seq, { struct lru_gen_folio *lrugen =3D &lruvec->lrugen; unsigned long young =3D 0, old =3D 0, lag =3D 0; + unsigned long inactive_ratio, gb; DEFINE_MIN_SEQ(lruvec); =20 /* we still have enough generations to reclaim */ @@ -5127,7 +5128,13 @@ static bool lru_gen_imbalanced(struct lruvec *lruvec= , unsigned long max_seq, if (min_seq[!type] + MAX_NR_GENS =3D=3D max_seq + 1) lag +=3D lruvec_gen_size(lrugen, !type, min_seq[!type]); =20 - return young > old * MAX_NR_GENS && (lag < MAX_LRU_BATCH || + /* + * Borrow the adaptive ratio from inactive_is_low(), and scale + * it by sqrt(MAX_NR_GENS) to make aging less aggressive + */ + gb =3D (young + old) >> (30 - PAGE_SHIFT); + inactive_ratio =3D gb ? int_sqrt(10 * gb * MAX_NR_GENS) : MAX_NR_GENS; + return young > old * inactive_ratio && (lag < MAX_LRU_BATCH || (is_extreme_swappiness(swappiness) && sc->priority > 2)); } =20 --=20 2.34.1 From nobody Tue Sep 29 04:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 966524483B4 for ; Wed, 12 Aug 2026 12:19:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537150; cv=none; b=Q803Ydn6uYfvwf4tnwShi4NdGtcWSC84DlqIcEAs2Y0EvjxwUbrtgGS+oPggHA4D6iCRbCH1CRrW6sAX8crzbZKQrf7AmWMC4LptInECVO5ZpsHS00ZNrbgPzTcD85r8PfiJmxujy2N7658WvPKykQ9tcuH9Yze8MIxfdhFtrQg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786537150; c=relaxed/simple; bh=jFSf4JY9UOydcxq5jAdVNUTNf1xAMWVhGYmVKnID0GY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HNx/emvibT8AtOhqMOFVjWi57tIDX8s2Yvg1h10CjJKlad/Msjdu8ZKCWWh938GYQ/FmQMq0+ESM1aOIjFsmz0MKioTw85GUU2BOHwQJbEmW5zPXyBj6b6r+nIsdMCm7UtdGTgCMeN0ZYnBVsvA03fHdSfOOZq37s3rTNYeuqQk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BsFRzI2n; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BsFRzI2n" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C99A31F00A3E; Wed, 12 Aug 2026 12:19:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1786537148; bh=SOSrpdrgv6m75PZgqu/T7iS0CPPWanT5qI9R6irTmB0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=BsFRzI2nY/1moZp4Yo1jlnOlYcl3kc9gOA06lA5u6dIR0/qRsH/qbcaotVI9o6wVc D7f2GopsSziedN0lyRn9BqFZ7s2cLl4ptJQVjlnLdsGkT0JaPDJqYjYH7uOrugOp06 +MazIwctBjQKUerp5d4ILwNXf74z6BkGGmtr23nDuDeWS6qRXK6aHL/nlazZekMNAo JtEaAspJfrtH0EgLKph41ZGAWAfIUUSx+A47N4QFWZfBlBFtZo70kSZ3vA7ZOFpRRo 3HI2LzekNYZsaoANA0beFRJaWuyBIGBJZiEXxjQ9fFzaCTF+KjHJaAKWQSp600T82H 94rX0SYL89juA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, "Barry Song (Xiaomi)" Subject: [RFC PATCH v4 16/16] mm/mglru: reduce folios pulled from the oldest gen in inc_min_seq() Date: Wed, 12 Aug 2026 20:16:58 +0800 Message-Id: <20260812121658.69965-17-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260812121658.69965-1-baohua@kernel.org> References: <20260812121658.69965-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Pulling folios from the oldest generation to the second-oldest generation with MAX_LRU_BATCH can take a long time, as the spinlock is held throughout the operation and blocks other users. Reduce the remaining batch size significantly. This reduces the lock hold time, allowing concurrent reclaim, lru_cache_drain, and other operations to make progress. Signed-off-by: Barry Song (Xiaomi) --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index c0350a8b61d1..8e6fcefe0353 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3945,7 +3945,7 @@ static inline void flush_lru_batch(struct list_head *= head, struct list_head **ba static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) { int zone; - int remaining =3D MAX_LRU_BATCH; + int remaining =3D MAX_LRU_BATCH / (is_extreme_swappiness(swappiness) ? 2 = : 8); struct lru_gen_folio *lrugen =3D &lruvec->lrugen; int hist =3D lru_hist_from_seq(lrugen->min_seq[type]); int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); --=20 2.34.1