From nobody Sun Sep 27 00:36:28 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 738D633D4E4 for ; Thu, 27 Aug 2026 23:47:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874437; cv=none; b=AeZMNoC9vIpjRZV1hF1VZnScWn9+TdsiNKZBIf26KafZgXoa873BnzyEY5EzduSNi4c6NHPoeflXD+2I1Soj1gwFJ1XAvD+5ANKH0H2MXpvZBTL+XJHZ9rebKn3TRmhrKaSg3DAb5DLdfKFwizjcunEQ/eVXSOwuEk56prQtz0k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874437; c=relaxed/simple; bh=5KzYUWK2iMjN/jkCYyA0SSNyFYYVrGI616WrTii3wdk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=S1I9w196CCj5NusqeVivUIWeMM8cxjFU4AyDHBoHFI/iyaMStKbbM3cIEU8cTm3/HbUMQoQpcJDW0PRU7dyMdIut1RTIP6szjt7H5H/pCaGcUKpuZuNc/t2r196iYKoYR/wK4D7lmX/r3G8ZdQtzexAuNnR9jOx59ygUl6Lv1S4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=INwDxHpo; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="INwDxHpo" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9C6021F00A3A; Thu, 27 Aug 2026 23:47:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787874435; bh=ENEA/CG+RwJeijNigZCVNh2GWKHCE7Bxn/GSW/9uiZQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=INwDxHpoZfJUz25nZdJH2zfsSzwfFzAyCq2FqmEWlBP/GIKCFg/m2jHIsLx3VwVxS xh/43XUdKqF5D4nTevs3HGd144LxVkFnODKo4C95dPRhqb4SpHum2Y3vEJbEsw1NAp V77++kFgREKKbbAmEs+92hhZhKGorzALq19VoZmMIkIVt/JihqdY5X30FXuDwllrFN Nth4Umk8k030Ui9vMATIdgnPOCR1RUN+snvhn/XI/JLehg/Fb2/eLr/G4g72q6viLE 4faxZXpglvH1nlx5TQlL951jrYNTDKOgBVmextsDHvp/YuLtcUOc4BjXaOWHBPM5dk DWuu0pYhXa/kg== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, lianux.mm@gmail.com Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" , Xueyuan Chen Subject: [PATCH v2 1/7] mm/mglru: separate folio generation update from LRU accounting Date: Fri, 28 Aug 2026 07:46:58 +0800 Message-Id: <20260827234704.63163-2-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260827234704.63163-1-baohua@kernel.org> References: <20260827234704.63163-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" folio_inc_gen() currently updates both the folio's generation and the LRU size accounting. This makes it difficult to batch the LRU size updates when moving multiple folios. Extract the generation update into __folio_inc_gen(), which only updates the folio's generation and reports whether the generation was actually increased. Keep folio_inc_gen() as the wrapper that performs the LRU size accounting when needed. This separates the per-folio generation update from LRU accounting and allows the latter to be batched by subsequent changes. Signed-off-by: Barry Song (Xiaomi) Tested-by: Xueyuan Chen Reviewed-by: Lian Wang --- mm/vmscan.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index fdd13299a04a..8f187d296b8e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3296,20 +3296,21 @@ static int folio_update_gen(struct folio *folio, in= t gen, const vma_flags_t *vma } =20 /* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio) +static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increas= ed) { - int type =3D folio_is_file_lru(folio); - struct lru_gen_folio *lrugen =3D &lruvec->lrugen; - int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags.f); + int new_gen; =20 VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); =20 do { new_gen =3D ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; /* folio_update_gen() has promoted this page? */ - if (new_gen >=3D 0 && new_gen !=3D old_gen) + if (new_gen >=3D 0 && new_gen !=3D old_gen) { + if (increased) + *increased =3D false; return new_gen; + } =20 new_gen =3D (old_gen + 1) % MAX_NR_GENS; =20 @@ -3317,8 +3318,21 @@ static int folio_inc_gen(struct lruvec *lruvec, stru= ct folio *folio) new_flags |=3D (new_gen + 1UL) << LRU_GEN_PGOFF; } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); =20 - lru_gen_update_size(lruvec, folio, old_gen, new_gen); + if (increased) + *increased =3D true; + return new_gen; +} + +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio) +{ + int type =3D folio_is_file_lru(folio); + struct lru_gen_folio *lrugen =3D &lruvec->lrugen; + int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); + bool gen_increased; =20 + new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); + if (gen_increased) + lru_gen_update_size(lruvec, folio, old_gen, new_gen); return new_gen; } =20 --=20 2.34.1 From nobody Sun Sep 27 00:36:28 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFD7E38911B for ; Thu, 27 Aug 2026 23:47:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874441; cv=none; b=U+AtvQyB6fkTyILrwtcAC8jJHQmA5Ly5mXU1FutkZvpCfN2oVTpIRiEEKOweAhl6v/yOf80SJ7Na1LT/CNZRPwPiKwg6pItdfh69e4WON8CHwN4fM/yXqFTDQZhlq2qPe6CQNzMqn8nvjmd7eR1fvZ5WBdW1iT2B5joBeeKdXLU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874441; c=relaxed/simple; bh=GkIw99uXkH9v3W0L1NC22q6jYwGKDnEXx7If3PIKqfU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Js3Uw2RQfOZ1k5ZHKAF9kFAZIBRYkPP24AxdJG9Mxu2jDN4zzg8o9cO9Yu5OqVhvXo0/C8VVQgS0UDBF2oUhWFTpcKnoUZ1jNBpknAx+AUAPmf96tLi0KeXC+JmhqZBig6Vj/HPYEXustnkkCSYj9scpyITZZnebsIHC3wiK0Bc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DDO+VEQq; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DDO+VEQq" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C8C961F000E9; Thu, 27 Aug 2026 23:47:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787874439; bh=2f/gBP+lL9kMBCKcIXbFZ0t49f0KGcv6ig1olyg9Fw0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=DDO+VEQqRCzS1cfmBrLQJ50LaIDOsEC9LaOiC2Uzo0Oe5gk2eVQTcRoMVN1zAyqqH fvpVlSo2AcbAjO9Zc6Nj5hvfD/m/DjOhHfH0SiWfwDA5ruE+kOFT+j8t2fP0lE3XTL ul7ftEd6ky1ssOvHyZJdvtjbHDta0Sh4YZlPvt5tVrLJDKvE0idjmq/R66KcZLuMoR JU3/ZgTDPYDfBR++CSNPaGaGlM8Mk1CJQfQH236S4kYn069XhkEHc1JTsReafDxX2q FWuILq5Cw9zObiqIOjwB9L4/WICG/G21Vdq7lpDxejMxmQDCeeVwq3MEOFPm1j7vrh Bh+2FMPr1QPsw== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, lianux.mm@gmail.com Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" , Xueyuan Chen Subject: [PATCH v2 2/7] mm/mglru: batch update lrugen->nr_pages in inc_min_seq() Date: Fri, 28 Aug 2026 07:46:59 +0800 Message-Id: <20260827234704.63163-3-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260827234704.63163-1-baohua@kernel.org> References: <20260827234704.63163-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, folio_inc_gen() updates lrugen->nr_pages for every folio as it advances generations. Instead, accumulate the size changes and update lrugen->nr_pages in a batch after scanning the entire oldest generation, or when the scan stops because remaining reaches zero. Since we only move folios from the oldest generation to the second oldest generation, the active/inactive state cannot change. We can therefore skip __lru_update_size(). Signed-off-by: Barry Song (Xiaomi) Tested-by: Xueyuan Chen Reviewed-by: Kunwu Chan Reviewed-by: Lian Wang --- mm/vmscan.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 8f187d296b8e..07c22d51debd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3918,6 +3918,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) struct lru_gen_folio *lrugen =3D &lruvec->lrugen; int hist =3D lru_hist_from_seq(lrugen->min_seq[type]); int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); + int target_gen =3D (old_gen + 1) % MAX_NR_GENS; =20 /* For file type, skip the check if swappiness is anon only */ if (type && (swappiness =3D=3D SWAPPINESS_ANON_ONLY)) @@ -3927,35 +3928,47 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, int swappiness) if (!type && !swappiness) goto done; =20 + VM_WARN_ON_ONCE(get_nr_gens(lruvec, type) !=3D MAX_NR_GENS); + VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) !=3D + lru_gen_is_active(lruvec, target_gen)); /* prevent cold/hot inversion if the type is evictable */ for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; + long delta =3D 0; =20 while (!list_empty(head)) { struct folio *folio =3D lru_to_folio(head); + long nr_pages =3D folio_nr_pages(folio); int refs =3D folio_lru_refs(folio); bool workingset =3D folio_test_workingset(folio); + bool gen_increased; =20 VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 - new_gen =3D folio_inc_gen(lruvec, folio); + new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); - + if (gen_increased) + delta +=3D nr_pages; /* don't count the workingset being lazily promoted */ if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { int tier =3D lru_tier_from_refs(refs, workingset); - int delta =3D folio_nr_pages(folio); =20 WRITE_ONCE(lrugen->protected[hist][type][tier], - lrugen->protected[hist][type][tier] + delta); + lrugen->protected[hist][type][tier] + nr_pages); } =20 if (!--remaining) - return false; + break; } + WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone], + lrugen->nr_pages[old_gen][type][zone] - delta); + WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone], + lrugen->nr_pages[target_gen][type][zone] + delta); + if (!remaining) + return false; } done: reset_ctrl_pos(lruvec, type, true); --=20 2.34.1 From nobody Sun Sep 27 00:36:28 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28B82340A6B for ; Thu, 27 Aug 2026 23:47:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874445; cv=none; b=DJGEnzQOKjawDK1SDqR5GgWvdRm76+8OPQPizp9/e9svnuimU+oWR3/rBerIXpvl1b4QcyKhg6gkFbYUzh+nBzA+Fj2ErNC88iLdmPzW8tmLVqDh7Rc8boZ80b+Po0gUqdF2mAZONmS4WQ5+r9eBAtEKmjsx57GPvACl8DyxaNo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874445; c=relaxed/simple; bh=/qDZiKT7hh+9+kPFKspP5ZM8yMb/8Mu5BOvXS7K8Aww=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=T040aY1ST2umDRhp3NgNz7a7huY3K/QqT5Gg0ckoIFTa5JSvuGr3COkK7R28WGw/qksf3D81yJFwuBe4xBllOvSYuXJN3pbQRgBRxkrdCIrZhVsIjzDWxPiLe7tIS0IVM2gy5/XbRG0DfcYnYIuDxAE6ZRxR7HAmaa7n5o+FKR0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iR+edBfU; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iR+edBfU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 029261F00A3A; Thu, 27 Aug 2026 23:47:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787874443; bh=7I7JY1zLaktW057z/Fnc7oBx6JyoNkQhe3RQzTBptJo=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=iR+edBfUOenZjti8VZvvvWuIvpP/92rNVd4C6HyLhRCj88IVyA7OyH+hpUms1EV64 VS87H5+jUuRKZ4sV0kClRiHjOI8dWihw+Gx7T9ahwu+BrGyo5mtj5OIGlQPr7NMn/b eqdkuOIK7cAoXWkut1uGIa5LpVgtoTotp0W/e3FyHCPw+HvGPnbTg0LkaEulNE7Hg1 sf/BsT6+Q46Q1oZcRN7PB/5oidPcbDHbLX6BxTwBn7X6eRCO2B6z2kyVA0EsP14Ksm zVVmw6o/xJHgVE5p9A3sdlNgBHeaLQT+F2rw5yVh6Iq2TYAlAPoHgbUoj7+FNEifqh 4nl0jkmQwZDCw== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, lianux.mm@gmail.com Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" , Xueyuan Chen Subject: [PATCH v2 3/7] mm/mglru: enhance cold/hot inversion handling in inc_min_seq() Date: Fri, 28 Aug 2026 07:47:00 +0800 Message-Id: <20260827234704.63163-4-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260827234704.63163-1-baohua@kernel.org> References: <20260827234704.63163-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" During aging, a folio's generation may already have been updated by folio_update_gen(), even though it has not yet been moved to the corresponding generation list. Such folios are hotter than those already in that generation. It makes sense for inc_min_seq() to increment the generation of folios that were never promoted during aging and move them to the tail of the new oldest generation. However, folios that were already promoted should instead be moved to the head of their updated generation, just as sort_folio() does in scan_folios(). Otherwise, promoted folios could end up behind folios that were never promoted, effectively inverting their hot/cold ordering. Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Kairui Song Tested-by: Xueyuan Chen Reviewed-by: Lian Wang Reviewed-by: Ridong Chen --- mm/vmscan.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 07c22d51debd..b10d1703d907 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3949,9 +3949,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int t= ype, int swappiness) VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); - list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); - if (gen_increased) + if (gen_increased) { delta +=3D nr_pages; + list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + } else { + list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); + } /* don't count the workingset being lazily promoted */ if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { int tier =3D lru_tier_from_refs(refs, workingset); --=20 2.34.1 From nobody Sun Sep 27 00:36:28 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53D4F344031 for ; Thu, 27 Aug 2026 23:47:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874449; cv=none; b=P5eUT/TUDYDOA4Iu3kAvqpE8Mb6m8+2cQ9oHPm0JjH+Zn2jmAvjAuajfkkY1e661cI3eubwSjVbEr9HUEaJPjN2zG49K691p7Ma9DWa6vxUltIzlDu8UFzdmOso4e5uuqUIG/NoObIYSKkTq6bByTFEmDNI5i0upDoWfQha9rGY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874449; c=relaxed/simple; bh=XBhPG+6Dr6CO0ZYGyASnz1zIW8XrRj6w2+sUmw0QPyw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mfnSI57TqeulSgkzosgTKplvJKN4Hnlc6FWc5us0hfBo4VN5NzEY6vql+PwbJHumlQfgtPj9If2z0KoPiYwCzvIVcdBkP9pnQ7AtyRKKM+3OW7uuI2EGJfsccqoUi2u9h1u3jL2mzbQGq9JM5KTG3ib1yjZkkdaVjVKNI5Tp3JU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XhR8wbY7; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XhR8wbY7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E7ED1F000E9; Thu, 27 Aug 2026 23:47:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787874448; bh=XpoXXZeU8kWFe5s2ICH1kueGMPGyd+PONtE3ej8VrGI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=XhR8wbY755E4SSkqcE9MoEeN68jEYcQaM1A4b08q+UG4rIUxm2uAv78LbMZvoh5d+ g6gehELG5vSwKTnG3B5FcTDJIUvvrx1QIMwyfn5Fj4EIbmoSFehR1i0enLupWQPP/n qKsntasHkCzEdm6QEYabfxmLMIEFDkavBd1LMsseR8AA9Cfoh+/mNJf+ljS2s71GFT 84kmA6eZXYM2jYuSgOgeIFPSMnCvGM9XFK2ifqLhr7LC7Bryn8yZ0Fgtk2ROpUJt2q 1+sjhqBDeCGIJ3BBxOy/CeZ9CJo49A+dEPKpafC9wKqCAisBjBXuw+ie/Z5ZpEQo7a Wvp1+J8J+xLnA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, lianux.mm@gmail.com Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" , Xueyuan Chen Subject: [PATCH v2 4/7] mm/mglru: exclude folios promoted by aging from protected in inc_min_seq() Date: Fri, 28 Aug 2026 07:47:01 +0800 Message-Id: <20260827234704.63163-5-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260827234704.63163-1-baohua@kernel.org> References: <20260827234704.63163-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some folios may have been promoted during aging, so don't count them as protected, similar to sort_folio(). Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Baoquan He Tested-by: Xueyuan Chen Reviewed-by: Lian Wang --- mm/vmscan.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b10d1703d907..2b33d3f44682 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3952,17 +3952,17 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, int swappiness) if (gen_increased) { delta +=3D nr_pages; list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + + /* don't count the workingset being lazily promoted */ + if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { + int tier =3D lru_tier_from_refs(refs, workingset); + + WRITE_ONCE(lrugen->protected[hist][type][tier], + lrugen->protected[hist][type][tier] + nr_pages); + } } else { list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); } - /* don't count the workingset being lazily promoted */ - if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { - int tier =3D lru_tier_from_refs(refs, workingset); - - WRITE_ONCE(lrugen->protected[hist][type][tier], - lrugen->protected[hist][type][tier] + nr_pages); - } - if (!--remaining) break; } --=20 2.34.1 From nobody Sun Sep 27 00:36:28 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 880D337F006 for ; Thu, 27 Aug 2026 23:47:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874453; cv=none; b=fcJwL00yXakA6jBhhfkk5sEJAQVD6Vt+WXMKZH2uK08yubM7bQPuWym6KRA+1X2Ty468UPgmLvnWjUBO2OhxgaN4f9UTuddWASdIiFvrTV+O3ZlyrECoglRSyNOFtiQIFw1sDbweR3wX6mELP3b7NzZjjuYHrCi2EUTfvuoiQXw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874453; c=relaxed/simple; bh=CR1YMm5eq3PxF2FFFWNolcPx2l/5uCA6SnVt9OPlXAg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Gm1smNLUHrN4TRA30QwcknWOKgpKu6E3ptM0NV3fhZD8VUnF0Em3Kge4IPOWkXcDKQKUyCIMffphRqxRCW0C1JtNy46V3smsgemBqJ++yvys6TlgYQyCtNcRDNQRU0sJ38S69UNhR4GLRYDay1An7l9cFT1qDdtSzzRTJNop354= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fQ476tXi; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fQ476tXi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A7321F00A3A; Thu, 27 Aug 2026 23:47:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787874452; bh=r2jDlzFvOsWktkoLH2gzIr7hlSlT/K6vhqtCTnrj+Wo=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=fQ476tXii4CioaYamLqQ/awsDE6PQpgfETtgAKJTmpSICmIKTCgdvm//WaUr3jg35 r+F1JgTYduERgjwWe0TRzqGHloyHBpL62wvTcBogYzWxiqm55kGsSFisQa6/ZcIHKT EP0/3uMS6bVS2tMqssEZA/uYXO3Xor6TFds1WvJNEWzn6H7iSjKlvhvuWsCLV+EA3I N45O+WAqd9CvsWDYujs5XJg4Qm9GiayzuUvglk2euF3DKtZMQuxWzHA0o6S2jdb71H Ru7Nh53BHtTGJ+NSToSLHu6Agf5fHHH/uY+QoddJMFVGnUA+4IYfy5Z4SYBA2lWwUP gxU7MUtAEbKVg== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, lianux.mm@gmail.com Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" , Xueyuan Chen Subject: [PATCH v2 5/7] mm/mglru: make LRU folio prefetch helper an inline function Date: Fri, 28 Aug 2026 07:47:02 +0800 Message-Id: <20260827234704.63163-6-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260827234704.63163-1-baohua@kernel.org> References: <20260827234704.63163-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" `prefetchw_prev_lru_folio()` is currently implemented as a macro with a potentially unused argument. This makes the helper harder to read and can also trigger checkpatch warnings about unused macro arguments. Make it a `static inline` function and remove the unnecessary `_field` argument. The helper always prefetches the previous folio's `flags`, so there is no need to make the field configurable. Signed-off-by: Barry Song (Xiaomi) Tested-by: Xueyuan Chen Reviewed-by: Lian Wang --- mm/vmscan.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 2b33d3f44682..81f95a968447 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -182,17 +182,21 @@ struct scan_control { }; =20 #ifdef ARCH_HAS_PREFETCHW -#define prefetchw_prev_lru_folio(_folio, _base, _field) \ - do { \ - if ((_folio)->lru.prev !=3D _base) { \ - struct folio *prev; \ - \ - prev =3D lru_to_folio(&(_folio->lru)); \ - prefetchw(&prev->_field); \ - } \ - } while (0) +static inline void prefetchw_prev_lru_folio(struct folio *folio, + struct list_head *base) +{ + if (folio->lru.prev !=3D base) { + struct folio *prev; + + prev =3D lru_to_folio(&folio->lru); + prefetchw(&prev->flags); + } +} #else -#define prefetchw_prev_lru_folio(_folio, _base, _field) do { } while (0) +static inline void prefetchw_prev_lru_folio(struct folio *folio, + struct list_head *base) +{ +} #endif =20 /* @@ -1695,7 +1699,7 @@ static unsigned long isolate_lru_folios(unsigned long= nr_to_scan, struct folio *folio; =20 folio =3D lru_to_folio(src); - prefetchw_prev_lru_folio(folio, src, flags); + prefetchw_prev_lru_folio(folio, src); =20 nr_pages =3D folio_nr_pages(folio); total_scan +=3D nr_pages; --=20 2.34.1 From nobody Sun Sep 27 00:36:28 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD22B37F006 for ; Thu, 27 Aug 2026 23:47:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874458; cv=none; b=dFwFEUe3KeLo194yDVmYcvaSDcwI9TcqAzqWOWqf0WhfFrYlp7NK8wnNp+2i4dGfx14mTU72iA7kAUwGyu+7wkpSuNIWFIScYlpPVXHGl1dpX9q77umh2tUABU/eMKmTkXSt5zsjSmz8wGDPM8PyhPFcEk2S2N8/Ihz7n0Qj2xE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874458; c=relaxed/simple; bh=+wQDlABALUKVLcpowu9fKE0POTtsYkOHESyPGipBNW4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cLo/WqZ2gL/8l+LkEHDv0/ukq1aj5s/gQBqAHzhIU5Zly54FKIYMWZTKH+xx8krTYm5ox1FNwhVEJoeZf4NHnQIo0mXcAwtio1C6K63rszaVjsIM5JEDiCFxcr3rIbm8oMiGh60kjQEgyJX1XYijpmoKOs0axACpmmURln/pt4M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=msIgs9KZ; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="msIgs9KZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8C0D71F000E9; Thu, 27 Aug 2026 23:47:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787874457; bh=yvhE1//hn5gBVVZ/BGXLx4fVEuO6Hu7ZhZmDmlWmdWg=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=msIgs9KZMVZ+2npfWCXzmLUEkiRpmyAtOsS0AFJ0OwwqPCU2r+7MrirFQ2cERxpsH Pqao3cc4b1bs+wT4OWt7kD/joBx5pcSl7Dt3LzWEpeL9CU7zdeU6qarkcdRyOsL2m2 tT9wdg5lsmQ/feBqyFEJa0P0tvJ9oP5LOqjV+GTuwr+lIFo7IN6wDsimfwAofQz4qw rXciMXo/qmNK6NMDlvkaIBB2siFBlctjCrRSE/vYS3ovx6ho+P4c8Mbi8ew/KIUsns swIwVNoJuh6G+XqypKeRwSd8Sk2VnJoP5bPVK6VEXMYiK77ceLCsQ6WX+EZzqOOhps /DNrpEtCgXLXQ== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, lianux.mm@gmail.com Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" , Xueyuan Chen Subject: [PATCH v2 6/7] mm/mglru: move folios from oldest gen to second-oldest gen from head to tail Date: Fri, 28 Aug 2026 07:47:03 +0800 Message-Id: <20260827234704.63163-7-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260827234704.63163-1-baohua@kernel.org> References: <20260827234704.63163-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For reclamation, it makes sense to reclaim folios from tail to head, as folios near the head are relatively hot. However, when moving folios from the oldest generation to the second-oldest generation, using the tail-to-head order would effectively cause a cold/hot inversion. Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Baoquan He Tested-by: Xueyuan Chen Reviewed-by: Lian Wang --- mm/vmscan.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 81f95a968447..17524e96fe64 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -192,11 +192,27 @@ static inline void prefetchw_prev_lru_folio(struct fo= lio *folio, prefetchw(&prev->flags); } } + +static inline void prefetchw_next_lru_folio(struct folio *folio, + struct list_head *base) +{ + if (folio->lru.next !=3D base) { + struct folio *next; + + next =3D list_entry(folio->lru.next, struct folio, lru); + prefetchw(&next->flags); + } +} #else static inline void prefetchw_prev_lru_folio(struct folio *folio, struct list_head *base) { } + +static inline void prefetchw_next_lru_folio(struct folio *folio, + struct list_head *base) +{ +} #endif =20 /* @@ -3938,10 +3954,11 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, int swappiness) /* prevent cold/hot inversion if the type is evictable */ for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; + struct list_head *pos =3D head->next; long delta =3D 0; =20 - while (!list_empty(head)) { - struct folio *folio =3D lru_to_folio(head); + while (pos !=3D head) { + struct folio *folio =3D list_entry(pos, struct folio, lru); long nr_pages =3D folio_nr_pages(folio); int refs =3D folio_lru_refs(folio); bool workingset =3D folio_test_workingset(folio); @@ -3952,6 +3969,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 + prefetchw_next_lru_folio(folio, head); + pos =3D pos->next; new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); if (gen_increased) { delta +=3D nr_pages; --=20 2.34.1 From nobody Sun Sep 27 00:36:28 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93A3D37F006 for ; Thu, 27 Aug 2026 23:47:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874462; cv=none; b=uljW3WQePXBVhNH+JzCdUtDH056nAianNHja94HxjGad+53MgUFMMoXit4Ce5muTqvzrNywrlIkhP+ne8d8329zNnqNE7hhDWouKv78b+vRQL4kLfmiBIVVgZy+h2cRe5Vs+62luw4/KUgfuzTKDyu6uFTcUUcUyvw9E9ga9eu0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787874462; c=relaxed/simple; bh=Hxje9xumeitE6b5gN/GluFPkClYPXrs+5wmzMTCtK84=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=W97/s//niU+YIi9rzGBldOSWwzb4VAPz+d4L3VI0Qu5NX6jAxsr2761NdqgSqsG6C9/XcONssc9blYUDTe2wCJTqgij0VT38BU7PlKHObM+T2ysjvw6sdd1OTmUo78LCg7v16QXzDsnTU9JlGRJfgAnHYDqKob9Nb72ClDzp5Gk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Vjf35Uvk; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Vjf35Uvk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AFAA61F00A3D; Thu, 27 Aug 2026 23:47:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787874461; bh=qItSZf2IMrcadC3doUN+qm71ZNZWkB8tfnyRrPNJHJY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Vjf35UvktgdHiWQIABTxMqaWim3kr4E58NO+7gveoHzr9gSUuODU+e/oPsz6xDDHU d6molKBbiRPKf84hwQk5MKnfAuzKDUvOceSBI8UxSANtm2F+olNrsbn5M7PiwmhDF4 1kKleza67ZT3+22y7hBv665q2bHi7AxrnTMwcJ4denIPhChL6ThTf0gCjWYRftQ07V 6neNqlatpZ5vqdsoIPzu+ElSTTYbCqaIJy0yQceQLtwjMRP4rUSfNMSOMd6BiyFW1r fWy8QvSUkhZ5kR2k0z0bhGFCEUppN14Hvwa8EoskbbyiCtYlmsixo0qaEdRzsuzG2M f8+Xe0GMMdlrA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, lianux.mm@gmail.com Cc: axelrasmussen@google.com, baolin.wang@linux.alibaba.com, baoquan.he@linux.dev, chenridong@xiaomi.com, david@kernel.org, hannes@cmpxchg.org, kasong@tencent.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, ljs@kernel.org, lyugaofei@xiaomi.com, mhocko@kernel.org, qi.zheng@linux.dev, shakeel.butt@linux.dev, stevensd@chromium.org, wangzicheng@honor.com, weixugc@google.com, yuanchu@google.com, zhangbo56@xiaomi.com, "Barry Song (Xiaomi)" , Xueyuan Chen Subject: [PATCH v2 7/7] mm/mglru: batch move folios to the second-oldest gen's LRU Date: Fri, 28 Aug 2026 07:47:04 +0800 Message-Id: <20260827234704.63163-8-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260827234704.63163-1-baohua@kernel.org> References: <20260827234704.63163-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Detect folios that need to move from the oldest generation to the second-oldest generation, and batch-move them together. This can significantly reduce the sys time of inc_min_seq(), especially when the other type is significantly behind the preferred type. Assisted-by: gemini:gemini-3.6-flash Signed-off-by: Barry Song (Xiaomi) Reviewed-by: Baoquan He Tested-by: Xueyuan Chen Reviewed-by: Lian Wang --- mm/vmscan.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 17524e96fe64..2b6f3f05ce60 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3931,6 +3931,19 @@ static void clear_mm_walk(void) kfree(walk); } =20 +static inline void flush_lru_batch(struct list_head *head, struct list_hea= d **batch_end, + struct list_head *dst) +{ + LIST_HEAD(movable); + + if (!*batch_end) + return; + + list_cut_position(&movable, head, *batch_end); + list_splice_tail_init(&movable, dst); + *batch_end =3D NULL; +} + static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness) { int zone; @@ -3953,8 +3966,10 @@ static bool inc_min_seq(struct lruvec *lruvec, int t= ype, int swappiness) lru_gen_is_active(lruvec, target_gen)); /* prevent cold/hot inversion if the type is evictable */ for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { + struct list_head *target_list =3D &lrugen->folios[target_gen][type][zone= ]; struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; struct list_head *pos =3D head->next; + struct list_head *batch_end =3D NULL; long delta =3D 0; =20 while (pos !=3D head) { @@ -3974,7 +3989,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, int swappiness) new_gen =3D __folio_inc_gen(folio, old_gen, &gen_increased); if (gen_increased) { delta +=3D nr_pages; - list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + batch_end =3D &folio->lru; =20 /* don't count the workingset being lazily promoted */ if (refs + workingset !=3D BIT(LRU_REFS_WIDTH) + 1) { @@ -3984,11 +3999,14 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, int swappiness) lrugen->protected[hist][type][tier] + nr_pages); } } else { + flush_lru_batch(head, &batch_end, target_list); list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); } if (!--remaining) break; } + flush_lru_batch(head, &batch_end, target_list); + WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone], lrugen->nr_pages[old_gen][type][zone] - delta); WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone], --=20 2.34.1