From nobody Mon Sep 28 16:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DD1DC387371 for ; Thu, 20 Aug 2026 04:57:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787201825; cv=none; b=NCEpOdNJSkixZ+x9jf/TsfiTqSkhCPG/pUY781LHGs3l68/LzWqbBLtBkCwj0YvTk5tCb7X7oTc7QayyINcJxgZ8Xd2rPSVcbVVVWi8JpO/xI+dioTnJAs7Q3fy4/mH/U91JfEc2zZFODb1aJ7/N3pewcPakHmiE/zzi1gC8YXw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787201825; c=relaxed/simple; bh=Il/QPT8I8KLH9DxQsynuoV5Cl8hwv6pjB//hiP3RD2E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Z6BXenPKSfcMVZSbrtGe7Kv6/TVzbHCAw+h1FUVljWp6Q7BnbQSJ8yahFd2ylaSx89nnQDmfK32xYKyDGSrxI8ONaVURUJgUVkoXvkUo46b3lYkDYEtIDErbA8jmDsM2biFhrb7rtcq4c/O77Jlqio70p2EVSJFTOXoknERA0Fo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Sz5NyFbw; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Sz5NyFbw" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F42331F00A3A; Thu, 20 Aug 2026 04:56:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787201823; bh=migX9AimFf4wvpqfBtI4GwlSnq6tIejoAluPFOiC/J4=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Sz5NyFbwN2J4MMhQTyIvIxHNG8WSaelqyeR9V2UqarMofhJfHfpA0yGE/5BYbjgN6 /Sf6OYsPdEFbl8s5wk+cZkll9yhKUfYCQfAFV3dY8AqkhZZJt8cq9ED/MluPOatJmf j1i7BRj76XWTMzulyczJjbuwxND4gRMhH15HUKznwBXH2ceV0E6NRWteBIlhE23fs0 tl/CY+BZ3nfJF8ujtVZT3oQ6cuZrJXLC6bpfzs/fyl1EW5Fdp6bljzEZhU4NwEK/LT rEkzIkO74KJwivg7V97pP0luoGdP4lA+UPfWdOmgRET8RnrW0nyxVPrEE/vWI21EKS sz92nkve/+HAA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, Barry Song Subject: [PATCH 1/3] mm/mglru: improve readability of isolate_folios() Date: Thu, 20 Aug 2026 12:56:01 +0800 Message-Id: <20260820045603.68809-2-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260820045603.68809-1-baohua@kernel.org> References: <20260820045603.68809-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Ridong Chen The for_each_evictable_type() loop in isolate_folios() is misleading: it does not actually iterate over each evictable type. Instead, get_type_to_scan() selects the type to scan, while the iterator `i` merely bounds the number of attempts. Make the fallback behavior explicit in the code and remove the opaque for_each_evictable_type(i, swappiness). Signed-off-by: Ridong Chen Co-developed-by: Barry Song (Xiaomi) Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Baolin Wang Reviewed-by: Kairui Song Reviewed-by: Lian Wang --- mm/vmscan.c | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index c1404a59523d..d5cc30b667ad 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4833,35 +4833,41 @@ static int get_type_to_scan(struct lruvec *lruvec, = int swappiness) return positive_ctrl_err(&sp, &pv); } =20 +static inline bool is_single_type_reclaim(int swappiness) +{ + return swappiness =3D=3D MIN_SWAPPINESS || + swappiness =3D=3D SWAPPINESS_ANON_ONLY; +} + static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, int swappiness, struct list_head *list, int *isolated, int *isolate_type, int *isolate_scanned) { - int i; - int total_scanned =3D 0; + bool type_fallback_allowed =3D !is_single_type_reclaim(swappiness); int type =3D get_type_to_scan(lruvec, swappiness); + int total_scanned =3D 0, scanned, tier; =20 - for_each_evictable_type(i, swappiness) { - int scanned; - int tier =3D get_tier_idx(lruvec, type); +retry: + tier =3D get_tier_idx(lruvec, type); + scanned =3D scan_folios(nr_to_scan, lruvec, sc, + type, tier, list, isolated); =20 - scanned =3D scan_folios(nr_to_scan, lruvec, sc, - type, tier, list, isolated); + total_scanned +=3D scanned; + if (*isolated) { + *isolate_type =3D type; + *isolate_scanned =3D scanned; + return total_scanned; + } =20 - total_scanned +=3D scanned; - if (*isolated) { - *isolate_type =3D type; - *isolate_scanned =3D scanned; - break; - } - /* - * If scanned > 0 and isolated =3D=3D 0, avoid falling back to the - * other type, as this type remains sufficient. Falling back - * too readily can disrupt the positive_ctrl_err() bias. - */ - if (!scanned) - type =3D !type; + /* + * We are running out of the current reclaim type. Fall back to + * the other type if allowed. + */ + if (!scanned && type_fallback_allowed) { + type =3D !type; + type_fallback_allowed =3D false; + goto retry; } =20 return total_scanned; --=20 2.34.1 From nobody Mon Sep 28 16:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9859387371 for ; Thu, 20 Aug 2026 04:57:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787201829; cv=none; b=JeD6Bn0vzggEWyuuW2MQqsK+0N6yNc23IbkpToANXMSQtm/Is+nG4GlAedLFyYPMqA55Jr+2WCQzruQ3YhmHp6fx+zucw53uFxY2Hc+CyqFaEHvnTsUw6l4CH4wIteG8N2TlvDdZ40TwTTGDiNkT8nMaVSVrsfIaUn09e8GOuP0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787201829; c=relaxed/simple; bh=TSZk6F9Zw8iSkjmTGUUIVvN5ixlyLuF9ep8kx50IgmE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fnTwpzUJVPuJR4rlIxyEQgKt5LXX57pdQScStTs8AWStK+cHdCx46Ls25FWQFvQsanhbnfHHQFTeE42FTUNbrXUQyMoqWtEUtZjukYjsuQDSGVQEVy/LkUPWppi1TPD3v0lBmVkt7n2nwTiU/0wxi59/eo7lwWyoNPJYQOGVhwI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WEu0ANyV; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WEu0ANyV" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E08BD1F00A3D; Thu, 20 Aug 2026 04:57:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787201827; bh=L/wIz6ML7okFnhQHn92A6v7algKS0sqhHpGu7c56urE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=WEu0ANyVg5sufXEXgQXZ8Au81gfHVhKXozFhGTGmNlmE7UxANI9x5BG3Z2rDfPLkb j/r1D31sKJvTTs4HsmlK64aOy0CzjHta2ur1HdOq37AakH4TMc+/BD1VD6sx5ZRh6c sF50jzK1+3pZl/G4Z27dHqW6eh2HbvTtlPr7skKofsYJXZopMcNH1uIJu4OL7SK4XJ PrUQ760Joovr8SgYju+nlsYQxwNJhGbnsAcdlKH7OGINshuSU1kiFY/+WrEYk0SjGG yRV1Xv9XtG6mFWbJ9Tt6ko2TxBLG4vwKEVcktLC3MQOgt13+KAowh7VwLLlzJq3Jun ra5gjG+aCJCfQ== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, "Barry Song (Xiaomi)" Subject: [PATCH 2/3] mm/mglru: improve scan_folios() exhaustion detection Date: Thu, 20 Aug 2026 12:56:02 +0800 Message-Id: <20260820045603.68809-3-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260820045603.68809-1-baohua@kernel.org> References: <20260820045603.68809-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit 16b475d2ac3c ("mm/mglru: avoid reclaim type fall back when isolation makes no progress") uses scanned =3D=3D 0 to determine whether scan_folios() has exhausted a reclaim type. However, this is not always sufficient. It is possible for scanned > 0, while the oldest reclaimable generation is exhausted after the first scan_folios() call. We detect early_stop in scan_folios(). If we stop early for any reason, it means the current reclaim type is not exhausted yet. If early_stop is never reached, it means we have exhausted the current oldest generation without hitting any scanning limit. Another issue is that if the lruvec has 4 generations, we might have exhausted the oldest generation while the second oldest generation is still reclaimable. In that case, this type is not exhausted yet. Add an exhausted output argument to scan_folios() so it can explicitly report whether the reclaimable lists for the current type have been exhausted. Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Baolin Wang Reviewed-by: Lian Wang Reviewed-by: Ridong Chen --- mm/vmscan.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d5cc30b667ad..1f2e574b0061 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4721,7 +4721,8 @@ static bool isolate_folio(struct lruvec *lruvec, stru= ct folio *folio, struct sca =20 static int scan_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, int type, int tier, - struct list_head *list, int *isolatedp) + struct list_head *list, int *isolatedp, + bool *exhausted) { int i; int gen; @@ -4732,12 +4733,15 @@ static int scan_folios(unsigned long nr_to_scan, st= ruct lruvec *lruvec, int skipped =3D 0; unsigned long remaining =3D nr_to_scan; struct lru_gen_folio *lrugen =3D &lruvec->lrugen; + bool early_stop =3D false; =20 VM_WARN_ON_ONCE(nr_to_scan > MAX_LRU_BATCH); VM_WARN_ON_ONCE(!list_empty(list)); =20 - if (get_nr_gens(lruvec, type) =3D=3D MIN_NR_GENS) + if (get_nr_gens(lruvec, type) =3D=3D MIN_NR_GENS) { + *exhausted =3D true; return 0; + } =20 gen =3D lru_gen_from_seq(lrugen->min_seq[type]); =20 @@ -4768,8 +4772,10 @@ static int scan_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, skipped_zone +=3D delta; } =20 - if (!--remaining || max(isolated, skipped_zone) >=3D MIN_LRU_BATCH) + if (!--remaining || max(isolated, skipped_zone) >=3D MIN_LRU_BATCH) { + early_stop =3D true; break; + } } =20 if (skipped_zone) { @@ -4778,8 +4784,10 @@ static int scan_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, skipped +=3D skipped_zone; } =20 - if (!remaining || isolated >=3D MIN_LRU_BATCH) + if (!remaining || isolated >=3D MIN_LRU_BATCH) { + early_stop =3D true; break; + } } =20 item =3D PGSCAN_KSWAPD + reclaimer_offset(sc); @@ -4790,6 +4798,13 @@ static int scan_folios(unsigned long nr_to_scan, str= uct lruvec *lruvec, scanned, skipped, isolated, type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON); =20 + /* + * If we didn't stop early, all reclaimable folios in the current + * generation have been scanned. We are exhausted if this is the last + * reclaimable generation. + */ + *exhausted =3D !early_stop && + lrugen->min_seq[type] + MIN_NR_GENS =3D=3D lrugen->max_seq; *isolatedp =3D isolated; return scanned; } @@ -4847,11 +4862,12 @@ static int isolate_folios(unsigned long nr_to_scan,= struct lruvec *lruvec, bool type_fallback_allowed =3D !is_single_type_reclaim(swappiness); int type =3D get_type_to_scan(lruvec, swappiness); int total_scanned =3D 0, scanned, tier; + bool exhausted; =20 retry: tier =3D get_tier_idx(lruvec, type); scanned =3D scan_folios(nr_to_scan, lruvec, sc, - type, tier, list, isolated); + type, tier, list, isolated, &exhausted); =20 total_scanned +=3D scanned; if (*isolated) { @@ -4864,7 +4880,7 @@ static int isolate_folios(unsigned long nr_to_scan, s= truct lruvec *lruvec, * We are running out of the current reclaim type. Fall back to * the other type if allowed. */ - if (!scanned && type_fallback_allowed) { + if (exhausted && type_fallback_allowed) { type =3D !type; type_fallback_allowed =3D false; goto retry; --=20 2.34.1 From nobody Mon Sep 28 16:39:03 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 760F2388E55 for ; Thu, 20 Aug 2026 04:57:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787201832; cv=none; b=Ye38PSY8zAgyJHYfm3o75mEIHZN9XQ8y+cZPq9Zx+9RvuOx9ZOBe7c8lqL/M1aoS9wIMl8BrbWfsqs9gVZj+MTyJeouUPhQb1Jpy4Iclk5eyC4tE7nyup3iUgs4Fl1X88vC2ibe8nAu3FDYnjoSl6C8cNQ7K2o0Dd5r7Q+Rdl4U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787201832; c=relaxed/simple; bh=xU4S5wt0aBlVzzELqWgwbpX+y/fXS6BoJaGYVZGJtB8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=doQeYr8RTWwIz1op5EcvjTF+ucERq60a9jpscUTg4Vwx4IghfnzCI74UwUzu4p5/LqGYgSnWfJIpbtW4DxR3S/wbG0JhXUg5ROJmokqHPd/HL06QKQurE1ANOLKmLJp39QBtrrzg1BdGgSabdPN9w+36cW92fDEwfbbagGWjIKs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AjJX9t8d; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AjJX9t8d" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D3DD81F00A3A; Thu, 20 Aug 2026 04:57:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787201831; bh=leejdAoqEqZUEy4AxrXKtUzDLcGvX9RnYsTPxVQMljU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=AjJX9t8d1iHhdRU7OOIHyWRfxUYNBT7+IhGAeNv/mDa1oB/ISuwI9IO1gpOW/tnio CyWqy6AAx1CojOSTz0zOrLBZQuLaMFT4Zx4X8WDSeHRV2oEDVBquJnP6zoXoqRk5tM 4yBnWVrn9gS4iUXDxccgX7dVxI5PG4xKRgYO2aqHc/qB+fEzTq3qzSiinyGgU6Fndx tD3JUR+C7VkgdPQeNgkWRdtSAI+zOTqujXtJzk5mfifgeKat5dDMZuXI6B3MY/KsJM C+ZOStoNFjopd1olV3zRTrjJYrufhClOETEn2FQ9+5LNn6rDcnHCbM/q2htaeQtJPH /EwJdB3S9Z7Ag== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, lianux.mm@gmail.com, linux-kernel@vger.kernel.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, "Barry Song (Xiaomi)" Subject: [PATCH 3/3] mm/mglru: retry the same type once if isolation fails due to races Date: Thu, 20 Aug 2026 12:56:03 +0800 Message-Id: <20260820045603.68809-4-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260820045603.68809-1-baohua@kernel.org> References: <20260820045603.68809-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If we are not exhausted (i.e., there are still folios in the reclaimable generations) but fail to isolate any folios due to promotions, protections, or races, retry once more to avoid going through the outer loop again. Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Baolin Wang Reviewed-by: Lian Wang Reviewed-by: Ridong Chen --- mm/vmscan.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 1f2e574b0061..1f302386d8ab 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4862,7 +4862,7 @@ static int isolate_folios(unsigned long nr_to_scan, s= truct lruvec *lruvec, bool type_fallback_allowed =3D !is_single_type_reclaim(swappiness); int type =3D get_type_to_scan(lruvec, swappiness); int total_scanned =3D 0, scanned, tier; - bool exhausted; + bool exhausted, tried =3D false; =20 retry: tier =3D get_tier_idx(lruvec, type); @@ -4885,6 +4885,14 @@ static int isolate_folios(unsigned long nr_to_scan, = struct lruvec *lruvec, type_fallback_allowed =3D false; goto retry; } + /* + * We are not exhausted, but failed to isolate any folios due to + * promotions, protections, or races. Retry once to avoid a larger loop. + */ + if (!exhausted && !tried) { + tried =3D true; + goto retry; + } =20 return total_scanned; } --=20 2.34.1