From nobody Mon Sep 28 18:33:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12261381EA3 for ; Tue, 18 Aug 2026 22:59:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787093964; cv=none; b=aFalkc/LEPOgk3zUYfz3Dm2rc5T1yjhwG4dVoKXF+B5mrx+BqOdd5hfbyriRqR/vmz5vbQYaf9w/4F25RYHVNX7CX7zB2UyvbBKSFYAFSSsjQImI+4kTvUN+2PAR49/dc8QiW8uZV5/1rlUpLRvPgXhwai7xI4NQcjiku+AIE+k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787093964; c=relaxed/simple; bh=vxbSwwoLKzRzirA80Nl45YrIWdi8uFuuKfYYc+bCIoA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=T9GB8r2A1/PCsHZrt/cpPLVS7u5QogKblEJ1UZ4lUyZMiPrGY2tQcHnU0Fh0/2dSo/CYyJPaCpWHC+wlPdT9YQOlzuavUxmzEOCSnV7uQ8TPy2yp0oXuWjJ5eCHi87qpqDshaBpVSsN1kI5jTl5/xBNaMldh9Y0hH1VjyP9kBwU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TyX8nUis; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TyX8nUis" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1DA8E1F00A3D; Tue, 18 Aug 2026 22:59:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787093962; bh=HX1DWIadQ6nIRR1fyXkWC0pb34Vk39QSCu5CRyVrb0A=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=TyX8nUis49ca4jtpbJ98TM2fg/c4aU445zPNr7AmIN7lFk2ZW8SS9+OnUsAYqofR9 Yy9kscqbH5fQFNtNq19IH2IPrSk/zdBlAGEa5PrDfnxFt/yWNSe//1UQ0+rdd2dHnQ USWpKcr5QBR84gMw/mwE9X0cnqHvFO/7p7NQ+77CXfLHm7o6RFTQ3ptqpydBk2d/1i 46CL2IasQhIqDnFrQJoXtI7QrzOF1FttzzVk+yFPTrqTq3uh6vVaKxqbdMJ4feI2wM qJitG39O7M/ngE89xcIjRj5En6Rj4Gc3Ebq/HqBkc7Hkrr6wFwVvFbad8xdXV8PNvO LC5MinjdK4MxA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, david@kernel.org, dev.jain@arm.com, lance.yang@linux.dev, liam@infradead.org, linux-kernel@vger.kernel.org, ljs@kernel.org, mhocko@suse.com, npache@redhat.com, rppt@kernel.org, ryan.roberts@arm.com, surenb@google.com, vbabka@kernel.org, ziy@nvidia.com, hughd@google.com, ackerleytng@google.com, usama.arif@linux.dev, joannelkoong@gmail.com, hannes@cmpxchg.org, "Barry Song (Xiaomi)" Subject: [RFC PATCH v3 1/4] mm: allow smaller large folios to use lru_cache Date: Wed, 19 Aug 2026 06:59:01 +0800 Message-Id: <20260818225904.55236-2-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260818225904.55236-1-baohua@kernel.org> References: <20260818225904.55236-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For systems that primarily use smaller-order large folios, enabling the lru_cache can help reduce lock contention. For higher-order large folios, the number of folios involved is likely to be smaller, making lock contention less significant. This patch enables the lru_cache for large folios whose `nr_pages` is smaller than `FOLIO_BATCH_SIZE`. To avoid holding too many pages in the lru_cache, which could affect accounting and reclamation, we also limit the total number of pages in the cache to `FOLIO_BATCH_SIZE`. To track the number of pages, this patch adds an `unsigned short nr_pages` field to `struct folio_batch`. It cannot overflow because the batch contains at most `FOLIO_BATCH_SIZE` folios, each of which has fewer than `FOLIO_BATCH_SIZE` pages. For non-LRU caches, `folio_batch` only needs to track the number of folios, so `nr_pages` is left at zero. Signed-off-by: Barry Song (Xiaomi) Tested-by: Shivank Garg --- include/linux/folio_batch.h | 25 +++++++++++++++++++++++++ mm/folio.c | 10 +++++++++- mm/internal.h | 4 ++-- 3 files changed, 36 insertions(+), 3 deletions(-) diff --git a/include/linux/folio_batch.h b/include/linux/folio_batch.h index b45946adc50b..ffc7de091fa3 100644 --- a/include/linux/folio_batch.h +++ b/include/linux/folio_batch.h @@ -10,6 +10,7 @@ #define _LINUX_FOLIO_BATCH_H =20 #include +#include =20 /* 31 pointers + header align the folio_batch structure to a power of two = */ #define FOLIO_BATCH_SIZE 31 @@ -28,6 +29,7 @@ struct folio; struct folio_batch { unsigned char nr; unsigned char i; + unsigned short nr_pages; bool percpu_pvec_drained; struct folio *folios[FOLIO_BATCH_SIZE]; }; @@ -42,6 +44,7 @@ static inline void folio_batch_init(struct folio_batch *f= batch) { fbatch->nr =3D 0; fbatch->i =3D 0; + fbatch->nr_pages =3D 0; fbatch->percpu_pvec_drained =3D false; } =20 @@ -49,6 +52,7 @@ static inline void folio_batch_reinit(struct folio_batch = *fbatch) { fbatch->nr =3D 0; fbatch->i =3D 0; + fbatch->nr_pages =3D 0; } =20 static inline unsigned int folio_batch_count(const struct folio_batch *fba= tch) @@ -78,6 +82,27 @@ static inline unsigned folio_batch_add(struct folio_batc= h *fbatch, return folio_batch_space(fbatch); } =20 +/** + * folio_batch_add_lru_cache() - Add a folio to a batch of lru_cache + * @fbatch: The folio batch. + * @folio: The folio to add. + * + * The folio is added to the end of the batch. + * The batch must have previously been initialised using folio_batch_init(= ). + * + * Return: 0 if the lru_cache is filled with more than FOLIO_BATCH_SIZE + * pages; otherwise, the number of available slots. + */ +static inline unsigned folio_batch_add_lru_cache(struct folio_batch *fbatc= h, + struct folio *folio) +{ + fbatch->folios[fbatch->nr++] =3D folio; + fbatch->nr_pages +=3D (unsigned short)folio_nr_pages(folio); + if (fbatch->nr_pages > FOLIO_BATCH_SIZE) + return 0; + return folio_batch_space(fbatch); +} + /** * folio_batch_next - Return the next folio to process. * @fbatch: The folio batch being processed. diff --git a/mm/folio.c b/mm/folio.c index 59c477120b9a..e5820d7263e8 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -219,7 +219,7 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, else local_lock(&cpu_fbatches.lock); =20 - if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || + if (!folio_batch_add_lru_cache(this_cpu_ptr(fbatch), folio) || !folio_may_be_lru_cached(folio) || lru_cache_disabled()) folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); =20 @@ -981,6 +981,7 @@ void folios_put_refs(struct folio_batch *folios, unsign= ed int *refs) int i, j; struct lruvec *lruvec =3D NULL; unsigned long flags =3D 0; + unsigned long nr_pages =3D 0; =20 for (i =3D 0, j =3D 0; i < folios->nr; i++) { struct folio *folio =3D folios->folios[i]; @@ -1020,6 +1021,7 @@ void folios_put_refs(struct folio_batch *folios, unsi= gned int *refs) =20 if (j !=3D i) folios->folios[j] =3D folio; + nr_pages +=3D folio_nr_pages(folio); j++; } if (lruvec) @@ -1030,6 +1032,12 @@ void folios_put_refs(struct folio_batch *folios, uns= igned int *refs) } =20 folios->nr =3D j; + /* + * For lru_cache, track the number of pages; for non-LRU caches, + * folio_batch->nr_pages is always 0. + */ + if (folios->nr_pages > 0) + folios->nr_pages =3D nr_pages; mem_cgroup_uncharge_folios(folios); free_unref_folios(folios); } diff --git a/mm/internal.h b/mm/internal.h index 38b1165212c9..06adf78e13a2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -48,9 +48,9 @@ static inline bool folio_may_be_lru_cached(const struct f= olio *folio) /* * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting. * Holding small numbers of low-order mTHP folios in per-CPU LRU cache - * will be sensible, but nobody has implemented and tested that yet. + * will be sensible. */ - return !folio_test_large(folio); + return folio_nr_pages(folio) < FOLIO_BATCH_SIZE; } =20 static inline void lru_cache_enable(void) --=20 2.34.1 From nobody Mon Sep 28 18:33:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D17137C0E6 for ; Tue, 18 Aug 2026 22:59:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787093968; cv=none; b=ENN976nJXzDsdADDyfZ6S6zVaPNfXasBqYj721sYliw6NYwzsiAOCuF76wwgzSVhxHUnyhW3QhcE7o3DjWiODu3GZiIqgRCTIdGq4W1eOA40I7wa9yv1YMK0JDUd53HdHsAj/W1/l4JdYoRZWoSL4mxLPhRSnhwupdnDYzmZkOE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787093968; c=relaxed/simple; bh=QOVu8KeQ2sCJjPQBhVXY9WkAUtIxG1RoWv3Uido1YsU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nCTU2wBAkaGWEgjHiib54bqvM3A/2fSTzzJWmYjI78y2J2QdnCCRNAGudMVjFTu3WwY4JyKNulUdmznPIdiFgH5AcSu57mIVdRTRyLHmV8yLVwGJzSJvqMXnCSTZKuGwCqRA4IfYEScUzhZtKf9+7bWxozCHDQxlv0OKed/zios= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KxELYPG8; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KxELYPG8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 253B41F00A3A; Tue, 18 Aug 2026 22:59:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787093966; bh=yUofJYBF2fURCSwAJxaMXCKNOjPJX7Leg5CtfjoSIGM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=KxELYPG8qHof9ElVwEt5AvXIWstk4g2vuciaR9mz+HHHKZAevxhwpjn3PC7g+5b8u idDhwphSoISj/ZtlrAzOg9bWnp4G0RCu8zsYDffKb0DboP5ViKLkO6cCETn7to87Zr NanootF6LqHwSCIM1p4Rto2judFYjXHYmuKarh74Yqfc/9cM8WpI34vZ7thJbK5/yA f69rD8w80OA6DT9AcwvGlMwKFFEZpMaDWV5nDb8DeSAl0ypHiEbeNzit20+cIYBp1R xKllKcv8abq6uN1QYDJR5cPxORgjL07/2TTKOK8tjnFDAkSiun6UC0TYTzeqWT5EQ5 iIx2jSDEic36Q== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, david@kernel.org, dev.jain@arm.com, lance.yang@linux.dev, liam@infradead.org, linux-kernel@vger.kernel.org, ljs@kernel.org, mhocko@suse.com, npache@redhat.com, rppt@kernel.org, ryan.roberts@arm.com, surenb@google.com, vbabka@kernel.org, ziy@nvidia.com, hughd@google.com, ackerleytng@google.com, usama.arif@linux.dev, joannelkoong@gmail.com, hannes@cmpxchg.org, "Barry Song (Xiaomi)" Subject: [RFC PATCH v3 2/4] mm: improve large folio reuse for LRU-cached folios Date: Wed, 19 Aug 2026 06:59:02 +0800 Message-Id: <20260818225904.55236-3-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260818225904.55236-1-baohua@kernel.org> References: <20260818225904.55236-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Large folios may now reside in the per-CPU LRU cache. Before attempting to reuse them, drain the local LRU cache, which can still be beneficial in cases where the folios are likely to remain in this CPU's LRU cache: int main(int argc, char *argv[]) { int i; while (1) { volatile int *p =3D mmap(0, SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); for (int i =3D 0; i < SIZE / sizeof(int); i++) p[i] =3D i; madvise((void *)p, SIZE, MADV_PAGEOUT); if (!fork()) _exit(0); for (int i =3D 0; i < SIZE / sizeof(int); i++) p[i] =3D i; munmap((void *)p, SIZE); } return 0; } Signed-off-by: Barry Song (Xiaomi) Tested-by: Shivank Garg --- mm/memory.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 4134ac607ee0..efdf82b3c418 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4275,6 +4275,12 @@ static bool __wp_can_reuse_large_anon_folio(struct f= olio *folio, folio_unlock(folio); } =20 + if (folio_may_be_lru_cached(folio) && !folio_test_lru(folio)) { + if (folio_ref_count(folio) !=3D folio_large_mapcount(folio) + 1) + return false; + lru_add_drain(); + } + if (folio_large_mapcount(folio) !=3D folio_ref_count(folio)) return false; =20 --=20 2.34.1 From nobody Mon Sep 28 18:33:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22D0237C0E6 for ; Tue, 18 Aug 2026 22:59:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787093972; cv=none; b=s5bz9ApjxgQfmfQiQu27myFRQw9v1AdHB1z/94O44KWbdWO5iahc5KKoFN95XretaMtzF80AplelI36Bb8wE+mEy1PsSo5TPnfk2WmSHUxdtE1AN850878evnC/+7ugUfHmtduOFx56IxOxJ7HWsqi0ihZbTjC3DYwt4jt+h85Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787093972; c=relaxed/simple; bh=9dNKqqkF3YTCYZlCRCOG6L1lSYyF2lX7+PgR52Vwogc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=euw7ha4b075gGsj7r6n3yB62xWNqqjBZsrmbIYnXJVyhTr1e6BG3EmBEfpRYQDXMyokWnbceIUmq5HWfg+NHWykoTYJWkbjbuyRVbJ53MTEA1GzFQWdElIgrxiwDSiJMBH8BR/f7qAjm319yegb0TF7sV70VpLD3+6gs+pGTvac= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FHZPBPs2; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FHZPBPs2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2B1621F00A3E; Tue, 18 Aug 2026 22:59:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787093970; bh=B1x0THuwlQHV75X2L7TyoBUXAkoOlg2uj72r2UXh84o=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=FHZPBPs2ZsXHZEEPdxNKFFIHv6g+omPTLAwrOnfk72DZKY4Rvl+/XwHk4K9ZBE0K3 bu10OqjJC2CTYjB+W+qX+mpQEQSwwn6KIViALPI91C+W4hUADJ+7LiM709GGQRBC0G h340onceStKr9OqWRfQtIyxXWApPymKH1q9HxBoDZg1Qt/M+F/qxUfi/hgMTuJfonE t0ycSOZ71tG/jTdwzkS56JNJEfp9SIB7qezvxJIuY64jIXCLdXN9LOOXhM38MhrKNp 5Evhw3EXpr3vP2BtRHRxIT1FLeeHt/OCp8lvaUybSRv7qUdHVNEtchkfVKv9wC2WB5 bNuoBEuGb+Rww== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, david@kernel.org, dev.jain@arm.com, lance.yang@linux.dev, liam@infradead.org, linux-kernel@vger.kernel.org, ljs@kernel.org, mhocko@suse.com, npache@redhat.com, rppt@kernel.org, ryan.roberts@arm.com, surenb@google.com, vbabka@kernel.org, ziy@nvidia.com, hughd@google.com, ackerleytng@google.com, usama.arif@linux.dev, joannelkoong@gmail.com, hannes@cmpxchg.org, "Barry Song (Xiaomi)" Subject: [RFC PATCH v3 3/4] mm: drain LRU cache if necessary for splitting large folios Date: Wed, 19 Aug 2026 06:59:03 +0800 Message-Id: <20260818225904.55236-4-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260818225904.55236-1-baohua@kernel.org> References: <20260818225904.55236-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Smaller large folios might now be present in the LRU cache. Use David's new lru_cache_drain_for_folio() helper to drain the LRU cache before splitting a folio, ensuring that the folio can be split successfully. Also, we only perform the drain when it may actually help, assuming that the lru_cache holds an extra reference. Signed-off-by: Barry Song (Xiaomi) Tested-by: Shivank Garg --- mm/huge_memory.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ced400f72d43..263ef9b6949d 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4201,6 +4201,9 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, if (shmem_mapping(mapping)) end =3D shmem_fallocend(mapping->host, end); } + if (folio_ref_count(folio) =3D=3D folio_expected_ref_count(folio) + 1 + + folio_may_be_lru_cached(folio)) + lru_cache_drain_for_folio(folio, 1, NULL); =20 /* * Racy check if we can split the page, before unmap_folio() will @@ -4325,6 +4328,9 @@ int folio_split_unmapped(struct folio *folio, unsigne= d int new_order) VM_WARN_ON_ONCE_FOLIO(!folio_test_large(folio), folio); VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio); =20 + if (folio_ref_count(folio) =3D=3D folio_expected_ref_count(folio) + 1 + + folio_may_be_lru_cached(folio)) + lru_cache_drain_for_folio(folio, 1, NULL); if (folio_expected_ref_count(folio) !=3D folio_ref_count(folio) - 1) return -EAGAIN; =20 @@ -4805,6 +4811,10 @@ static int split_huge_pages_pid(int pid, unsigned lo= ng vaddr_start, goto next; =20 total++; + + if (folio_ref_count(folio) =3D=3D folio_expected_ref_count(folio) + + folio_may_be_lru_cached(folio)) + lru_cache_drain_for_folio(folio, 0, NULL); /* * For folios with private, split_huge_page_to_list_to_order() * will try to drop it before split and then check if the folio --=20 2.34.1 From nobody Mon Sep 28 18:33:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3848C37AA8B for ; Tue, 18 Aug 2026 22:59:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787093976; cv=none; b=IolpfoviuD6ySmzBBT5cyX7lgDlE1DU46HXdzRamRXTJG0jh3Hjc6pXnXunfDYUSVfud0BNUrDGCcMGjXnmqEXDWEniFGlIIiV4rlTDDw83Ak5/i42pVD3bfAyD1Cod4NDU6U5bF1pUQUVL8NSfuRj045S8riNyXu7DBAzaMx5M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787093976; c=relaxed/simple; bh=bgJnNmMdOReYz84TpWPYu4ZsUV781gI1wfB3D7Gsqx0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ixnoexcjDmm5iBqNUv58zZBQJZNyn1alS09IXNV+AzPvchHiEiqnM2/kCTClRqoqVk9A0twq+cuBzQidFUS+IxbA4lITzj0C3UJqJhJEWwcF1vqLwiduGeP0O60swQ6g8GtKTomNYh4SQUIT7LjPlYHRkp9OcwhiI7tbyEu/pME= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Uf7uLp0G; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Uf7uLp0G" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33D821F000E9; Tue, 18 Aug 2026 22:59:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1787093974; bh=4aVhpGd68r1Svhk9UlnG+UFWV77hKbotB3T4QNwx1S8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Uf7uLp0GpZigC8Y5cRL6A8Gz8Jk/Vkjh8NM6RbMLIxU0PqW6ohlKIhZaD4ha3efb0 9S482ouY+T64y3pjTDdmXHkPSfspoY6PZrcU3GTpmEo1yA5Em9kwBhBgkAshDJl837 d8DZ+6ksTmIIkXvJlFXFA2cBfAiSLxziS0Owt2EFMGMhc/jszU9saDwWUWq4O+/AAz t6Sz87oictEI4wwBuolMSpL952R4tUXl9roESqZ5F56hRbE1fhHJwXzl2U0b1qbxK/ XR7r4/OliS3mh2ixRQfwFSH+m+rDb+i4CyrwXzdhKyKbEsx1b9jq7GcJaivvozpxkG cvrVpE5CZVfWg== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, david@kernel.org, dev.jain@arm.com, lance.yang@linux.dev, liam@infradead.org, linux-kernel@vger.kernel.org, ljs@kernel.org, mhocko@suse.com, npache@redhat.com, rppt@kernel.org, ryan.roberts@arm.com, surenb@google.com, vbabka@kernel.org, ziy@nvidia.com, hughd@google.com, ackerleytng@google.com, usama.arif@linux.dev, joannelkoong@gmail.com, hannes@cmpxchg.org, "Barry Song (Xiaomi)" Subject: [RFC PATCH v3 4/4] mm: batch lru_cache draining in deferred_split_scan Date: Wed, 19 Aug 2026 06:59:04 +0800 Message-Id: <20260818225904.55236-5-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260818225904.55236-1-baohua@kernel.org> References: <20260818225904.55236-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" deferred_split_scan() splits a batch of folios, so we only need to drain the lru_cache once for the entire batch. Signed-off-by: Barry Song (Xiaomi) Tested-by: Shivank Garg --- mm/huge_memory.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 263ef9b6949d..e797f6d1837e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4085,6 +4085,8 @@ static int __folio_freeze_and_split_unmapped(struct f= olio *folio, unsigned int n * @lock_at: a page within @folio to be left locked to caller * @list: after-split folios will be put on it if non NULL * @split_type: perform uniform split or not (non-uniform split) + * @lru_cache_drained: whether lru_cache has been drained locally + * or on all CPUs for a batch of folios * * It calls __split_unmapped_folio() to perform uniform and non-uniform sp= lit. * It is in charge of checking whether the split is supported or not and @@ -4100,7 +4102,8 @@ static int __folio_freeze_and_split_unmapped(struct f= olio *folio, unsigned int n */ static int __folio_split(struct folio *folio, unsigned int new_order, struct page *split_at, struct page *lock_at, - struct list_head *list, enum split_type split_type) + struct list_head *list, enum split_type split_type, + enum lru_cache_drained *lru_cache_drained) { XA_STATE(xas, &folio->mapping->i_pages, folio->index); struct folio *end_folio =3D folio_next(folio); @@ -4203,7 +4206,7 @@ static int __folio_split(struct folio *folio, unsigne= d int new_order, } if (folio_ref_count(folio) =3D=3D folio_expected_ref_count(folio) + 1 + folio_may_be_lru_cached(folio)) - lru_cache_drain_for_folio(folio, 1, NULL); + lru_cache_drain_for_folio(folio, 1, lru_cache_drained); =20 /* * Racy check if we can split the page, before unmap_folio() will @@ -4395,7 +4398,7 @@ int __split_huge_page_to_list_to_order(struct page *p= age, struct list_head *list struct folio *folio =3D page_folio(page); =20 return __folio_split(folio, new_order, &folio->page, page, list, - SPLIT_TYPE_UNIFORM); + SPLIT_TYPE_UNIFORM, NULL); } =20 /** @@ -4426,7 +4429,7 @@ int folio_split(struct folio *folio, unsigned int new= _order, struct page *split_at, struct list_head *list) { return __folio_split(folio, new_order, split_at, &folio->page, list, - SPLIT_TYPE_NON_UNIFORM); + SPLIT_TYPE_NON_UNIFORM, NULL); } =20 /** @@ -4622,6 +4625,7 @@ static unsigned long deferred_split_scan(struct shrin= ker *shrink, struct folio *folio, *next; int split =3D 0; unsigned long isolated; + enum lru_cache_drained drained =3D LRU_CACHE_NOT_DRAINED; =20 isolated =3D list_lru_shrink_walk_irq(&deferred_split_lru, sc, deferred_split_isolate, &dispose); @@ -4646,7 +4650,8 @@ static unsigned long deferred_split_scan(struct shrin= ker *shrink, } if (!folio_trylock(folio)) goto requeue; - if (!split_folio(folio)) { + if (!__folio_split(folio, 0, &folio->page, &folio->page, NULL, + SPLIT_TYPE_UNIFORM, &drained)) { did_split =3D true; if (underused) count_vm_event(THP_UNDERUSED_SPLIT_PAGE); --=20 2.34.1