From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792424973; b=eWx2HGmJ2DcWbOm2G0Gsk3W8GhNi/PyHQRaVWSxoooF0AuO+saU9gPAkQQEwWGTBH4K6 JqLcQ6nB072wqKGk7+1TR70y/9HbUYiDmmWQcCuTL66uWQlrcjMk+9tRa2IIP/A2sSC1S NVgHhvCDF+RImJm4TkW0hzGZKOknBxJL8Gb2tZh14gCZZIw3cqJtbJMeqX7R9AHjGPPZf E/xU9dlRPQmnAGC19k+7cTxMFd3eLXEQBtQ8p3YVa5uNfWnw3Trr4YCy1Q3yvtW6KRTuB 7EkZaxDOBIzx1HfjvrqWZQ+M3EXA6RKjzrZPQiSSP+BkHPJ01eVhk9jml6Rw18OKvmA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792424973; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=5Nw+dRp3BsamXDt/gXHq20wL6FIVWvA/eL2Wq+/2wtQ=; b=t0Bh8UqXsiB57SAHmrsRX72K2+ZzV5g0HLMjm0T2tvWmYY9N9h9QJsIcjoGaD1DaspEh MOu4FtrwCcmkwz4I7EG4+ARhwN9qJz4RQ8POJISJaL1yLd3lZdVt63gBriXNCQzkp/QwU VHsaKQJz0XIQcXZHCT65DlJJd6ZrUEfCj+QZjUtGOAQmq/UwwGG2qjZq6l5cCm/CoGVyZ vydCOteEMTNDl2nr1CI9Tc+Shb2+xP2xWndTsowx5MEyG1mLvrlUIN9ocRh1+yX4XOY9N ZsAoLuGE6bSc/eFvOvY5AVPT/h7/1n/jpQh1C7fG/KY/UxOjbluAvbu/lr2STkldmsQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=A2SWU1YD; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E6716078840 for ; Mon, 19 Oct 2026 15:49:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=A2SWU1YD Received: by smtp.kernel.org (Postfix) with ESMTPSA id C05651F00ADF; Mon, 19 Oct 2026 15:49:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792424972; bh=5Nw+dRp3BsamXDt/gXHq20wL6FIVWvA/eL2Wq+/2wtQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=A2SWU1YD2gmUSdWCu5DLVytZUw5Py0fBrWGuTB0H5kdVTyDuVh9DuTyMPkLzwu5qE J/MSvca/T8QWhr+3Ax0FXhcZly1nrzFloa2VYh6xUM8qYRfCegiNlOHWvQu/hxzrLF c8zkYzZ7bMgT1qlUKlmt8C/PFkaSIaGmPNbaFVCSxw7Nk0gnGqEC7qSloqN7UC5kOi Mobb7DCoyYkHWtv9dvFffZ9Z7ysyoKabAxVGb4nbc6KAy+5HqtPjC5KfBsiNRYfHtK CsttFzVG4NX8Ysr/J3DRrrURNY2l5KkCWMQVvNzOt0kXWVXOSWrvf7HKyMg7AyZzuv q/1JgL0FL+S3w== Received: from phl-compute-11.internal (phl-compute-11.internal [10.202.2.51]) by mailfauth.ams.internal (Postfix) with ESMTP id 46C781980047; Mon, 19 Oct 2026 11:49:23 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-11.internal (MEProxy); Mon, 19 Oct 2026 11:49:28 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTESqR4vElUNfwe4Wb1kabMg9MdYUtHyNIvlHi5jfGRsEd0tjiYfzpqh5FemuLlMuA +QzWL/iolJzzDzQk8vzuoT2/u2xyL/YZG/CihU2eaSQe3uAw7YKu/Z/hQfgMIJpEQrBQkK hH3KJVpCmjg+gNXnnOIkdBMKpJa1pvTLqzuojrt68Uovke10kpOzVhZrNEf3d58NDgoR+6 GBwzdAiHoLFFDGJo3/r+XBHeUTBbw+IeFaYdgC0j5JEQlKAOMB5I7tBUfVMuw5hMfUnXf2 vE1DwOk7N/x2FbtU++Y3QX/1ODQhD5sTVGp3EMvK4jUTItTZrn4RBPsK+PhnbeHfl8JcOK eC93Uux4gzDYkGL3LunO0coPzmDCOalx65wUsbcyE59TK2N8vbmnx/aORMfTFkjXU6EQa2 mvfPQs1C5tE6wzAWQ4KD9dF07RoozXA2WCG/KEiETgzvG/4lg3sjD1CtDs+wka96IlgryC P3OuWIy05RuEXEfkgI1DdKyHfBIuYnnpY2rMFCYc0GWetFEbb36i18ArZE6KO7h4T1BDlj 9xsAsKEhI2XclfxVGU9Cb7sYpyUQzlQ1i5HYHsKck+DBb3JNGeyVRVXwi5QapHcy8Fzr8l zk24zapjBXVNhkeLB6VEhlwqLebKeilb4kWe1F/FHBK5qEyPFy0X/65SepTA X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:49:22 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com, "Kiryl Shutsemau (Meta)" Subject: [RFC PATCH 01/18] mm: free PG_rcu_free folios only after an RCU grace period Date: Mon, 19 Oct 2026 16:48:54 +0100 Message-ID: <20261019154913.1765974-2-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Linus Torvalds Reading a page cache folio without taking a reference would keep CPUs that read the same file from bouncing its refcount. But without a reference the folio can be freed, and its memory reused, while the reader copies from it. Add PG_rcu_free: the free of a folio with the flag waits for an RCU grace period. A reader that saw the flag under rcu_read_lock(), and then still found the folio where it looked, can read it until rcu_read_unlock(). A split copies the flag, as such a reader may still be reading the new folios. The folios wait on per-CPU lists, one grace period in flight per CPU: call_rcu() per folio would queue a callback for every folio of a truncated file. A work, which can reschedule, frees each batch. Reclaim depends on these frees, so the workqueue is WQ_MEM_RECLAIM. The flag is 64-bit only (CONFIG_PAGE_RCU_FREE): page flags are scarce on 32-bit, and the contention this avoids is a problem of large machines. pgfree_rcu in /proc/vmstat counts the deferred pages. Nothing sets the flag yet. Assisted-by: LLM Signed-off-by: Linus Torvalds [kas: rework for flagging at read time, 64-bit only, shorter comments and changelog] Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/page-flags.h | 21 +++++++ include/linux/vm_event_item.h | 1 + include/trace/events/mmflags.h | 7 +++ mm/Kconfig | 12 ++++ mm/huge_memory.c | 1 + mm/memory-failure.c | 4 ++ mm/page_alloc.c | 105 +++++++++++++++++++++++++++++++++ mm/vmstat.c | 1 + 8 files changed, 152 insertions(+) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 7a863572adce..a1e52a1f8a52 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -111,6 +111,9 @@ enum pageflags { PG_swapbacked, /* Page is backed by RAM/swap */ PG_unevictable, /* Page is "unevictable" */ PG_dropbehind, /* drop pages on IO completion */ +#ifdef CONFIG_PAGE_RCU_FREE + PG_rcu_free, /* free only after an RCU grace period */ +#endif #ifdef CONFIG_MMU PG_mlocked, /* Page is vma mlocked */ #endif @@ -602,6 +605,24 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) FOLIO_TEST_CLEAR_FLAG(dropbehind, FOLIO_HEAD_PAGE) __FOLIO_SET_FLAG(dropbehind, FOLIO_HEAD_PAGE) =20 +/* + * A lockless lookup that sees PG_rcu_free and then still finds the folio + * where it looked may read it until rcu_read_unlock(). Set under a + * reference, cleared only by the free. Not for hugetlb or + * free_pages_nolock(). + */ +#ifdef CONFIG_PAGE_RCU_FREE +FOLIO_TEST_FLAG(rcu_free, FOLIO_HEAD_PAGE) + FOLIO_TEST_SET_FLAG(rcu_free, FOLIO_HEAD_PAGE) + __FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE) +#define __PG_RCU_FREE (1UL << PG_rcu_free) +#else +FOLIO_TEST_FLAG_FALSE(rcu_free) + FOLIO_TEST_SET_FLAG_FALSE(rcu_free) + __FOLIO_CLEAR_FLAG_NOOP(rcu_free) +#define __PG_RCU_FREE 0 +#endif + #ifdef CONFIG_HIGHMEM /* * Must use a macro here due to header dependency issues. page_zone() is n= ot diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 2628ccda076a..cbe849701653 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -36,6 +36,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, FOR_ALL_ZONES(ALLOCSTALL) FOR_ALL_ZONES(PGSCAN_SKIP) PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE, + PGFREE_RCU, PGFAULT, PGMAJFAULT, PGLAZYFREED, PGREUSE, diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 935893e5ea53..47c3411a75e8 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -117,6 +117,12 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT); #define IF_HAVE_PG_IDLE(_name) #endif =20 +#ifdef CONFIG_PAGE_RCU_FREE +#define IF_HAVE_PG_RCU_FREE(_name) ,{1UL << PG_##_name, __stringify(_name)} +#else +#define IF_HAVE_PG_RCU_FREE(_name) +#endif + #ifdef CONFIG_ARCH_USES_PG_ARCH_2 #define IF_HAVE_PG_ARCH_2(_name) ,{1UL << PG_##_name, __stringify(_name)} #else @@ -152,6 +158,7 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT); DEF_PAGEFLAG_NAME(swapbacked), \ DEF_PAGEFLAG_NAME(unevictable), \ DEF_PAGEFLAG_NAME(dropbehind) \ +IF_HAVE_PG_RCU_FREE(rcu_free) \ IF_HAVE_PG_MLOCK(mlocked) \ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ diff --git a/mm/Kconfig b/mm/Kconfig index 604c58199acb..5af00db23f90 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -718,6 +718,18 @@ config PCP_BATCH_SCALE_MAX latency. This option sets the upper limit of scale factor to limit the maximum latency. =20 +config PAGE_RCU_FREE + def_bool 64BIT + help + Reads from the page cache can copy from folios that are read + again and again without taking a reference on them, so small + reads of one file from many CPUs do not contend on the folio's + reference count. The cost is that memory freed from such + folios becomes reusable only after an RCU grace period. + + 64-bit only: it takes a page flag, and those are scarce on + 32-bit. + config PHYS_ADDR_T_64BIT def_bool 64BIT =20 diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1e5d68acf62a..25082949a4ca 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3743,6 +3743,7 @@ static void __split_folio_to_order(struct folio *foli= o, int old_order, #endif (1L << PG_dirty) | (1L << PG_dropbehind) | + __PG_RCU_FREE | LRU_GEN_MASK | LRU_REFS_MASK)); =20 new_folio->mapping =3D folio->mapping; diff --git a/mm/memory-failure.c b/mm/memory-failure.c index a8b03e2920ba..7dde7c3c9a0c 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -211,6 +211,10 @@ static bool page_handle_poison(struct page *page, bool= hugepage_or_freepage, boo return false; } =20 + /* The free of a hwpoisoned page is not deferred: wait for readers */ + if (release && folio_test_rcu_free(page_folio(page))) + synchronize_rcu(); + SetPageHWPoison(page); if (release) put_page(page); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 12fac9084c48..a371e92da273 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1313,6 +1313,96 @@ static inline void pgalloc_tag_sub_pages(struct allo= c_tag *tag, unsigned int nr) =20 #endif /* CONFIG_MEM_ALLOC_PROFILING */ =20 +/* + * Each CPU has at most one grace period in flight (@queued), covering + * @batch. Folios freed meanwhile wait on @pending for the next one. + */ +struct rcu_free_pcpu { + spinlock_t lock; + bool queued; + struct list_head pending; + struct list_head batch; + struct rcu_work rwork; +}; + +static DEFINE_PER_CPU(struct rcu_free_pcpu, rcu_free_pcpu); + +static struct workqueue_struct *rcu_free_wq __ro_after_init; + +static void rcu_free_start(struct rcu_free_pcpu *r) +{ + lockdep_assert_held(&r->lock); + + if (r->queued || list_empty(&r->pending)) + return; + + list_splice_init(&r->pending, &r->batch); + r->queued =3D true; + WARN_ON_ONCE(!queue_rcu_work(rcu_free_wq, &r->rwork)); +} + +static void rcu_free_pages(struct list_head *list) +{ + struct folio_batch fbatch; + struct folio *folio, *next; + + folio_batch_init(&fbatch); + list_for_each_entry_safe(folio, next, list, lru) { + if (!folio_batch_add(&fbatch, folio)) { + free_unref_folios(&fbatch); + cond_resched_tasks_rcu_qs(); + } + } + if (folio_batch_count(&fbatch)) + free_unref_folios(&fbatch); +} + +static void rcu_free_workfn(struct work_struct *work) +{ + struct rcu_free_pcpu *r =3D container_of(to_rcu_work(work), + struct rcu_free_pcpu, rwork); + LIST_HEAD(done); + + spin_lock_irq(&r->lock); + list_splice_init(&r->batch, &done); + r->queued =3D false; + rcu_free_start(r); + spin_unlock_irq(&r->lock); + + rcu_free_pages(&done); +} + +static void rcu_free_defer(struct folio *folio) +{ + struct rcu_free_pcpu *r =3D raw_cpu_ptr(&rcu_free_pcpu); + unsigned long flags; + + spin_lock_irqsave(&r->lock, flags); + list_add_tail(&folio->lru, &r->pending); + rcu_free_start(r); + spin_unlock_irqrestore(&r->lock, flags); +} + +static void __init rcu_free_init(void) +{ + int cpu; + + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return; + + rcu_free_wq =3D alloc_workqueue("rcu_free", + WQ_MEM_RECLAIM | WQ_PERCPU, 0); + + for_each_possible_cpu(cpu) { + struct rcu_free_pcpu *r =3D per_cpu_ptr(&rcu_free_pcpu, cpu); + + spin_lock_init(&r->lock); + INIT_LIST_HEAD(&r->pending); + INIT_LIST_HEAD(&r->batch); + INIT_RCU_WORK(&r->rwork, rcu_free_workfn); + } +} + static __always_inline bool __free_pages_prepare(struct page *page, unsigned int order, fpi_t fpi_flags) { @@ -1327,6 +1417,19 @@ static __always_inline bool __free_pages_prepare(str= uct page *page, =20 VM_BUG_ON_PAGE(PageTail(page), page); =20 + /* + * Not for a hwpoisoned order-0 page: it never reaches the allocator, + * and page_handle_poison() takes a reference right after this free. + */ + if (unlikely(folio_test_rcu_free(folio)) && + !(PageHWPoison(page) && !order)) { + VM_WARN_ON_ONCE(fpi_flags & FPI_NOLOCK); + __folio_clear_rcu_free(folio); + count_vm_events(PGFREE_RCU, 1 << order); + rcu_free_defer(folio); + return false; + } + trace_mm_page_free(page, order); kmsan_free_page(page, order); =20 @@ -6401,6 +6504,8 @@ void __init setup_per_cpu_pageset(void) for_each_online_pgdat(pgdat) pgdat->per_cpu_nodestats =3D alloc_percpu(struct per_cpu_nodestat); + + rcu_free_init(); } =20 __meminit void zone_pcp_init(struct zone *zone) diff --git a/mm/vmstat.c b/mm/vmstat.c index cb57714539fb..66ed017bde46 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1343,6 +1343,7 @@ const char * const vmstat_text[] =3D { [I(PGACTIVATE)] =3D "pgactivate", [I(PGDEACTIVATE)] =3D "pgdeactivate", [I(PGLAZYFREE)] =3D "pglazyfree", + [I(PGFREE_RCU)] =3D "pgfree_rcu", =20 [I(PGFAULT)] =3D "pgfault", [I(PGMAJFAULT)] =3D "pgmajfault", --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792424980; b=ozDXUaeP963fupMrur7mJ4X/5dge13mHmOUrUyj1SGWshbq9jX9bVFK5bBoEqnIsiAia IjumOCGtGAnrwxiI7YznZ51qrtun5ZkrN2aSdEkUKHfADBxpHr6VwUwkQcYfJFZbto0HM xTofKwxEurbAfB0lf/hDgZwLO8DjyRU8vg2P69me6/Ti69XUkl1VGAPsQSUDBz4Qsb8pi x0/Xobf0WtEFuX76hY2Qew17AQJgW1SQ25cb2TykcYZ8ZQCxE4BSpagCjTD4sHSrsRof5 G9aBYadNKcMiSWRR+8QbusoigcNkUDHLEzT7M1Q2M6IXnzaDW83pNOxI0QrV8Rf/omQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792424980; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=U4ryK0uFT6LOFyuJdmAuxXhuh94KeEdmaAIvPF6s5TM=; b=TOgyi4Z86iWb6jG7Skl9oiF0lI5S7C4wDO8+r977rV4lLhsusbxlF42zbcOOfB1Wb3Dz mIZCMIb/4s/WaBCtgeU5xuzPOGl1aH+3Dcr45Z638/UfbuIGc6xZJdiIg8i4WMW74BST4 NGX4dJpiq7O1olL9Xu+KV6gu+Rm1lqP9IXSz7lJTeaKZAdFI2kv5Gj0Xz+ogzrmELgS7K ryBmTiLiM+sHZbDKOb0Kptwqfp2d0fIkc0PM+Kyn7JES7erPe09ogjcsqzOJnU3uLwOtX xaRnJyc/73pzkZ8dijcaWNhr8lsMGaAWQWpiIpE7njtwYyTRptevFnshcHHcnaAR4ew== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b="SuW/ZU3r"; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55FB76086044 for ; Mon, 19 Oct 2026 15:49:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=SuW/ZU3r Received: by smtp.kernel.org (Postfix) with ESMTPSA id 73A9B1F000CA; Mon, 19 Oct 2026 15:49:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792424980; bh=U4ryK0uFT6LOFyuJdmAuxXhuh94KeEdmaAIvPF6s5TM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=SuW/ZU3r2cVDI0MsGtAqY7lp1e4JKP84OSelU1jDLrUdnhardwOiGsL6ZdNWe/sJX XxCOanD+yswhINbioTu/uBwI+GZnxfFRI7tAyVQEbaF7LAUhpvpGi0gtkUVutwsmWI Ztg2U6F4nqZKClyQRTTG/zTVX6Bd3BG5GLE8vjbfXf2sC88/k5ywfQZIReR+X5UZGJ uSfqezTEcgvMZ1by6DdXowbRMURfv/jElF1RmPM7DHV8kWYGN9XKZaT9n/k35Oi1+w 7cB+KGBMUoOzyiaVffGm4MQyuLHNbZ5T2xDigWr5J+TpZUhKDzhwSihAtl5Vuo82DX f/eh0dCaUH+XA== Received: from phl-compute-06.internal (phl-compute-06.internal [10.202.2.46]) by mailfauth.ams.internal (Postfix) with ESMTP id EABA8198003A; Mon, 19 Oct 2026 11:49:30 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Mon, 19 Oct 2026 11:49:34 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFpjrG+2jfVBfklNRE95oThC60xlhUtv5sioyJrfpDDHbEB/2cXkTI411oJriXTLs N7RLfOUdBbmoibSFu4zT/f1BKPPyxkEYUp5eDeafGhPQjcnfJv/29ZtYwE6q77Fr7A6zQu C94bEu3EPqJJ50fLC5xP9PtYZOq3sS+jMfKkgkZFNUAdZxkDiyHArLijwBk9MwzijLWdvR 4raSXvkSyIGvp9XYd2doZUGuEANzDQejgJzzg3ARZCjBL2yZfXIvLZCg0AMujDeTHEemYv cvWt94P7AS+P1mtikHZvyS9eAaK3Fy2/TwDlvaPXULPDsZK9f0sj0aca+uCBBoNs4d3uIW 6UtJ41xqrTh41J9CGKF5HMdWWNPlzyNWgWQzyeIKO+iqcaYPlEF9Y5+OBdY51W8gzONadz MxZF+NNp24tKwLD4/n1vy2Ovyn5wcR3tpqUHyQQpOTfDPZ4IwklH+mHOUVdzlFtnjgYkaQ qoNdWC/Qz3DlVKJX3IEmP6IDgyqMgqiPpwb6q5YyYOFoysGo7gHd2QNO9WcnhuKVK+bLic /u4BPVbK5+aqDmA+vVjbOvePAVA1tvWTAf2718PxhSMiEPspjNO8JwFB57p35ewgTxBW7f y5dRgjXgQxXNs9kr1jbCq9G/mob8EFCcMV/U+ZRpNrKfSXjGnRlgT2m+VOxQ X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:49:29 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com, "Kiryl Shutsemau (Meta)" Subject: [RFC PATCH 02/18] splice: don't steal page cache folios Date: Mon, 19 Oct 2026 16:48:55 +0100 Message-ID: <20261019154913.1765974-3-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Linus Torvalds PG_rcu_free holds back the reuse of a folio's memory until a grace period after the folio is freed. A stolen page cache pipe buffer is reused without being freed: remove_mapping() takes the folio out of the page cache and the caller keeps it. fuse_try_move_folio() puts it into a fuse file, where writes change it under a reader that found it before the steal. Drop ->try_steal from page_cache_pipe_buf_ops. fuse_try_move_folio() and virtio_console's pipe_to_sg() then copy, as they already do when stealing fails. SPLICE_F_MOVE from the page cache into a fuse file or a virtio console now copies. Assisted-by: LLM Signed-off-by: Linus Torvalds [kas: rewrote the changelog] Signed-off-by: Kiryl Shutsemau (Meta) --- fs/splice.c | 51 --------------------------------------------------- 1 file changed, 51 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 9d8f63e2fd1a..0133c2348da0 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -55,56 +55,6 @@ static noinline void pipe_clear_nowait(struct file *file) } while (!try_cmpxchg(&file->f_mode, &fmode, fmode & ~FMODE_NOWAIT)); } =20 -/* - * Attempt to steal a page from a pipe buffer. This should perhaps go into - * a vm helper function, it's already simplified quite a bit by the - * addition of remove_mapping(). If success is returned, the caller may - * attempt to reuse this page for another destination. - */ -static bool page_cache_pipe_buf_try_steal(struct pipe_inode_info *pipe, - struct pipe_buffer *buf) -{ - struct folio *folio =3D page_folio(buf->page); - struct address_space *mapping; - - folio_lock(folio); - - mapping =3D folio_mapping(folio); - if (mapping) { - WARN_ON(!folio_test_uptodate(folio)); - - /* - * At least for ext2 with nobh option, we need to wait on - * writeback completing on this folio, since we'll remove it - * from the pagecache. Otherwise truncate wont wait on the - * folio, allowing the disk blocks to be reused by someone else - * before we actually wrote our data to them. fs corruption - * ensues. - */ - folio_wait_writeback(folio); - - if (!filemap_release_folio(folio, GFP_KERNEL)) - goto out_unlock; - - /* - * If we succeeded in removing the mapping, set LRU flag - * and return good. - */ - if (remove_mapping(mapping, folio)) { - buf->flags |=3D PIPE_BUF_FLAG_LRU; - return true; - } - } - - /* - * Raced with truncate or failed to remove folio from current - * address space, unlock and return failure. - */ -out_unlock: - folio_unlock(folio); - return false; -} - static void page_cache_pipe_buf_release(struct pipe_inode_info *pipe, struct pipe_buffer *buf) { @@ -155,7 +105,6 @@ static int page_cache_pipe_buf_confirm(struct pipe_inod= e_info *pipe, const struct pipe_buf_operations page_cache_pipe_buf_ops =3D { .confirm =3D page_cache_pipe_buf_confirm, .release =3D page_cache_pipe_buf_release, - .try_steal =3D page_cache_pipe_buf_try_steal, .get =3D generic_pipe_buf_get, }; =20 --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792424986; b=quq/i1GNrCP6HZH3dZRvTg7wkrmNa03BnAps9maeZN/VvbJd4LW9e/nbphBExK8GZvBB e4gvqjDNco1idRsocsYDS3E7Qk7liMD56KUv02zejxsUQdMvXe7T2MsvZOxUVSGSoVhPh 1+bgzr//3+okCkqgegdh1aoFZzPnCdkejm2h8Pv8TSnxW1GLfBKxoZGYZCtbVaFkIPGaK PD/nHXc4hBtamlQuHy+DZDW8oaq5txKbaQCwy7pKcUi5BagUpeJPrXowgIh+owKF7ssqf JI0zwthcrgT+DJ6DEk62OTrrAwR9fkausOR4P+dOjYdmY1mYzOuj3yzzHSQ8K473rLg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792424986; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=Kbgj/cbxLipXGxpjxY36dW9YWtUesmr6PYKnaG71cGs=; b=qe2ZL9T0RuKMeZ6gzpxOBDRCmEg2DAyMeYwOqBDmll7uAlJ7M7lIjEReQbyLQkPRTbRS ln+trqCAtTIPyusbI4JbxWEx8Zf1UdTE2ePoyxN/pZueudERZ8ssVcQubQyjvTkggGZ/j GijqO5DshDFKShiLaRaLJJCQz9vvqfb9jtWO8lxkRv0+bS2Ktnb7ru5NHgs7xfrvii2XJ Tq14Tp3FRrTYEU68xaHMU0xLV150irzjUbVQKqJiMQ6USrJ2Uupyfb/iIUuJ5iG8UGd5q fQuBzubiQzSILu48o4TJlaN9R4CxEKXDJ4RStoNSnbUWLX+ahJ7j5C6w78CAGe1NEvQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=UpAdjgpJ; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90B3A6078840; Mon, 19 Oct 2026 15:49:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=UpAdjgpJ Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9A49A1F000CB; Mon, 19 Oct 2026 15:49:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792424986; bh=Kbgj/cbxLipXGxpjxY36dW9YWtUesmr6PYKnaG71cGs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=UpAdjgpJLRrZt5vfapGYdm4s0/UzT09h2EUGkaB2CvZLAUbX7ETxtYHlcqaWNpg1g eXbT5UAX6QRWY1N0d7O2jyoj+FytTUf/zuWrdHOBHp9K7lcvo2+qUHcjUj6mtIrIIQ g2Lil7vJ5nw3PLkI/zpVBy7eeSwCPW5q+OHhZeRwH1BPDPIK6E/kKZpAkimGd2i61q aEyTkBE8TLTHOPAdDdibTE7PkGGNx/ofoUxyZbNjYcSKlZQlm5ulqA7mqCsTzTY/8V rhFsuVbHxzNj8PCeL3V+TGHpxS7tnF3GEXUuPQDMBG9QPEqSeH4BJpBcCeOdpd55Y7 5kzc6h820HMFg== Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfauth.ams.internal (Postfix) with ESMTP id 6B6CF1980047; Mon, 19 Oct 2026 11:49:37 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Mon, 19 Oct 2026 11:49:41 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTESqR4vElUNfwe4Wb1kabMg9MdYUtHyNIvlHi5jfGRsEd0tjiYfzpqh5FemuLlMuA +QzWL/iolJzzDzQk8vzuoT2/u2xyL/YZG/CihU2eaSQe3uAw7YKu/Z/hQfgMIJpEQrBQkK hH3KJVpCmjg+gNXnnOIkdBMKpJa1pvTLqzuojrt68Uovke10kpOzVhZrNEf3d58NDgoR+6 GBwzdAiHoLFFDGJo3/r+XBHeUTBbw+IeFaYdgC0j5JEQlKAOMB5I7tBUfVMuw5hMfUnXf2 vE1DwOk7N/x2FbtU++Y3QX/1ODQhD5sTVGp3EMvK4jUTItTZrn4RBPsK+PhnbeHfl8Jcd/ yYYEKmAGT8vWogHsS/MLpG4mGVeUeUcl8fzs7/zHA9gd8BbEgXpb1CoSUIZuL9AsGdcCeL +gXC3GBZVeiw6q1+GKFOmx8OVmgLnX3toJ5chCtufQcBsun6EpjJaG+M15ijPymLjtuB1N FD7FJ16WFMZH3bHDTYwl4ViatPxmQL4lcXqk/pVAVOGizVyQLMIwi0unAXmKltJ2BEqr5S 32YVsgd9l8nrPOu7EChRRHchU7/xpjfB5k04bMWbcc2A+1JkfyBbqlxnUaxqdwKP/Ztfj7 OS9iW/9oAv4Gm0mAtI4sFlKSdYDo0XfmQz3m1GzlmlQx0HIQJKcwJvH46hFA X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:49:36 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 03/18] fuse: don't move a folio with PG_rcu_free into a fuse file Date: Mon, 19 Oct 2026 16:48:56 +0100 Message-ID: <20261019154913.1765974-4-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" fuse_try_move_folio() takes the folio of a pipe buffer and puts it into a fuse file, where writes change it. A folio with PG_rcu_free may still be copied by a lockless page cache read that found it before it left the page cache. Such a folio reaches fuse when a mapped file page is gifted with vmsplice(SPLICE_F_GIFT) and its file is then truncated. fuse_check_folio() refuses it only because the flag is not among those it allows, and with a dump_page(). Copy such a folio instead, as fuse does whenever it cannot take one. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- fs/fuse/dev.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 4fec31fc0b84..526929c7b3f1 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -1152,6 +1152,10 @@ static int fuse_try_move_folio(struct fuse_copy_stat= e *cs, struct folio **foliop =20 newfolio =3D page_folio(buf->page); =20 + /* A lockless page cache read may still be copying from it */ + if (folio_test_rcu_free(newfolio)) + goto out_fallback_unlock; + folio_clear_uptodate(newfolio); folio_clear_mappedtodisk(newfolio); =20 --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792424992; b=2692bfY/nhnfayaI4DLrZ2wLNp64esAc8xI7ayC2t3s3pRJyhVJODyvLmDTxy9aQCBLQ d7eVbvqrYnzyVerPUTJieq4n8zP2q5Oyu2WD0Ubh0urXF+rDBgcW5mJim9d2p71gz2woX HbXHd1/sCV49is8FTD/ICvHyDX/Y1m5ZC2m4PcUd0lXz1NKrsx7xR7m0fuWs/wQeV4Wqa j+a5JuxDgxA70m08Cnq1fzuruRHksDMKMuvJeRRk0qCThodSsmCj4LRSkDzEMrz8rESAz C0CYv4Kb0SrYV81/lNyKs8E4POq2Vb+N9YxS26QiQF876rxK4GINMZIF1YMRL9MFwiQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792424992; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=Ed7Ai2/v+4UUMnde8ZmMAQuE9IFX/8DJI4RqAxylpU4=; b=o06ozxnFPkceYR39IUPgV28Bvi0yQ5+2FoZb2im4f5z426ZvhbvzzrfKy9nt0CJ9VuSX hNwIQQdPd22oSd3SJW6UMOgSqbwMNgivVIIkAS+ohTg2FU6gV6FSZyUL7j6wxPtUNwyki oyfgCB1+CezLiKGkAbp80kdp+q4vVwlACjihXs1ou071sDg8IXHfQArOny7splZ4ZaJRy w3N7lHdtHTzevtGddxdQJUji7JbSDEsD5Pd2HVHkAOce9K/93Fjvv5Bo84uHykBG9SJ34 edcg7pr+yBLPv3I/PvIxeljAYZM5d7jOxq7tBbdTGhs0XrHQlYG63juajuuml449Sxw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=QATB7jCZ; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 32580606AFAD for ; Mon, 19 Oct 2026 15:49:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=QATB7jCZ Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3AA6F1F000C8; Mon, 19 Oct 2026 15:49:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792424991; bh=Ed7Ai2/v+4UUMnde8ZmMAQuE9IFX/8DJI4RqAxylpU4=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=QATB7jCZ9GC+p/xOCU4/r80jMnEHJ3IGAgM3NU+u6I3pBVOTcLCwE1dmSk0y0K1EJ 3r+krUEHzAFTZERtH9bIx2vMwuw8Rjs0tAXXx2bEOmBG54nohjew7NGXlmkMHCIh5l IrnGDfwxl/S9N4c5eCqN14ObXaZ77/sHnzu6VfUJLIjSazs1rzLC6kR/JBZOGbsz8Q HA7+Z6RetZhWrmrsxHCFmLBefoUo8+u/qmQoE0xAAfaOk5dOrKCDKb6WPJN/gkHJwv zyrOsdKa5+pt3peyv3sTCP0h56PTzZhpr1qCjN0gCLTdNGX4CVX1yArKzydfYBDnMa 2OA6vxmjFVLIQ== Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfauth.ams.internal (Postfix) with ESMTP id 273C5198003A; Mon, 19 Oct 2026 11:49:44 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-02.internal (MEProxy); Mon, 19 Oct 2026 11:49:47 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTESqR4vElUNfwe4Wb1kabMg9MdYUtHyNIvlHi5jfGRsEd0tjiYfzpqh5FemuLlMuA +QzWL/iolJzzDzQk8vzuoT2/u2xyL/YZG/CihU2eaSQe3uAw7YKu/Z/hQfgMIJpEQrBQkK hH3KJVpCmjg+gNXnnOIkdBMKpJa1pvTLqzuojrt68Uovke10kpOzVhZrNEf3d58NDgoR+6 GBwzdAiHoLFFDGJo3/r+XBHeUTBbw+IeFaYdgC0j5JEQlKAOMB5I7tBUfVMuw5hMfUnXf2 vE1DwOk7N/x2FbtU++Y3QX/1ODQhD5sTVGp3EMvK4jUTItTZrn4RBPsK+PhnbeHfl8JcGw mU66hr8sFGivOaQBw7U2atwUF2x3djrLkSB9Q3mnaguXHMvDkpm7rhqSFA0ohGFqfUUH1u dxHChWUAxw09NAyZDTfJM81AqrT9LU2Vh5Twlr1hurWQ1i4hXLhrWSLYZwnEQegkEOZsBW 2ssW6OwYwuRhSe6+I65NTvx7Vr9rCb4mR/BhnMrDjmQO1o94/n4xqYxXyl9y9h78FGtvT7 ndUWzFlFeeSkNndSFl40rBwl3kdQCD+HNz5+1/tpvq8eqN0/fc81tAtElmtYU/Zb+HGtUw CDw5XP+G/fbauiZeq64n2XnYIQyFVBZPncpoGy7aFOK5fl3W6hqS5IiGwv9Q X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:49:42 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 04/18] mm: count the pages waiting for a grace period Date: Mon, 19 Oct 2026 16:48:57 +0100 Message-ID: <20261019154913.1765974-5-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Deferred folios have refcount 0 and are not on the free lists, so no counter shows them. Memory that waits for a grace period is missing from the free count and from every LRU count. Count it per zone in NR_FREE_RCU_PAGES: nr_free_rcu in /proc/vmstat, free_rcu in show_free_areas(). Preparation for freeing the waiting pages sooner when they hold back memory that is needed. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/mmzone.h | 1 + mm/page_alloc.c | 4 ++++ mm/show_mem.c | 9 ++++++--- mm/vmstat.c | 2 ++ 4 files changed, 13 insertions(+), 3 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 94f9c3ff5416..3336b9e5ee99 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -228,6 +228,7 @@ enum zone_stat_item { NR_ZSPAGES, /* allocated in zsmalloc */ #endif NR_FREE_CMA_PAGES, + NR_FREE_RCU_PAGES, /* freed, waiting for an RCU grace period */ #ifdef CONFIG_UNACCEPTED_MEMORY NR_UNACCEPTED, #endif diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a371e92da273..15bb66e36304 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1348,6 +1348,8 @@ static void rcu_free_pages(struct list_head *list) =20 folio_batch_init(&fbatch); list_for_each_entry_safe(folio, next, list, lru) { + mod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES, + -folio_nr_pages(folio)); if (!folio_batch_add(&fbatch, folio)) { free_unref_folios(&fbatch); cond_resched_tasks_rcu_qs(); @@ -1375,9 +1377,11 @@ static void rcu_free_workfn(struct work_struct *work) static void rcu_free_defer(struct folio *folio) { struct rcu_free_pcpu *r =3D raw_cpu_ptr(&rcu_free_pcpu); + struct zone *zone =3D folio_zone(folio); unsigned long flags; =20 spin_lock_irqsave(&r->lock, flags); + __mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio)); list_add_tail(&folio->lru, &r->pending); rcu_free_start(r); spin_unlock_irqrestore(&r->lock, flags); diff --git a/mm/show_mem.c b/mm/show_mem.c index b938cbcd774a..0d3bddbda108 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -204,7 +204,7 @@ static void show_free_areas(unsigned int filter, const = nodemask_t *nodemask, " mapped:%lu shmem:%lu pagetables:%lu\n" " sec_pagetables:%lu bounce:%lu\n" " kernel_misc_reclaimable:%lu\n" - " free:%lu free_pcp:%lu free_cma:%lu\n", + " free:%lu free_pcp:%lu free_cma:%lu free_rcu:%lu\n", global_node_page_state(NR_ACTIVE_ANON), global_node_page_state(NR_INACTIVE_ANON), global_node_page_state(NR_ISOLATED_ANON), @@ -224,7 +224,8 @@ static void show_free_areas(unsigned int filter, const = nodemask_t *nodemask, global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE), global_zone_page_state(NR_FREE_PAGES), free_pcp, - global_zone_page_state(NR_FREE_CMA_PAGES)); + global_zone_page_state(NR_FREE_CMA_PAGES), + global_zone_page_state(NR_FREE_RCU_PAGES)); =20 for_each_online_pgdat(pgdat) { if (show_mem_node_skip(filter, pgdat->node_id, nodemask)) @@ -325,6 +326,7 @@ static void show_free_areas(unsigned int filter, const = nodemask_t *nodemask, " free_pcp:%lukB" " local_pcp:%lukB" " free_cma:%lukB" + " free_rcu:%lukB" "\n", zone->name, K(zone_page_state(zone, NR_FREE_PAGES)), @@ -351,7 +353,8 @@ static void show_free_areas(unsigned int filter, const = nodemask_t *nodemask, 0UL, K(free_pcp), K((unsigned long)this_cpu_read(zone->per_cpu_pageset->count)), - K(zone_page_state(zone, NR_FREE_CMA_PAGES))); + K(zone_page_state(zone, NR_FREE_CMA_PAGES)), + K(zone_page_state(zone, NR_FREE_RCU_PAGES))); printk("lowmem_reserve[]:"); for (i =3D 0; i < MAX_NR_ZONES; i++) printk(KERN_CONT " %ld", zone->lowmem_reserve[i]); diff --git a/mm/vmstat.c b/mm/vmstat.c index 66ed017bde46..64068d73d6d2 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1212,6 +1212,7 @@ const char * const vmstat_text[] =3D { [I(NR_ZSPAGES)] =3D "nr_zspages", #endif [I(NR_FREE_CMA_PAGES)] =3D "nr_free_cma", + [I(NR_FREE_RCU_PAGES)] =3D "nr_free_rcu", #ifdef CONFIG_UNACCEPTED_MEMORY [I(NR_UNACCEPTED)] =3D "nr_unaccepted", #endif @@ -2023,6 +2024,7 @@ static int vmstat_refresh(const struct ctl_table *tab= le, int write, switch (i) { case NR_ZONE_WRITE_PENDING: case NR_FREE_CMA_PAGES: + case NR_FREE_RCU_PAGES: continue; } val =3D atomic_long_read(&vm_zone_stat[i]); --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425000; b=1z6ujXwZJqFewu4dpEaWyDRUV4a4/RsFFfcbPs9KkEhXMNwKJ4cz1y8vzZtRf39Qqkr6 gdkgP3VhBJ/kpqM0U4l2J1buwsrRA7NCiXuWYTqd4FQwTPz30KRHu95fL7UHoHL85ta6D /CEkLuUK74uCjSZ6IbuV/N3FF4AcMZuKXk05XEu3855CuN9qS47RrmlM5rbhAynWx1cQ4 Og51ecs3GBWU45/1TZQwQHlcGwRDVGfruoL1353nXtFUQUxNv2d+P7rk/G7lFkxggEHjU 12J6RPB7RlP5kVdjyKczreewWLiDHjQ/pyWZGLdWzjUttXGuRtM5Cr3TZ7lTt4T2/7w== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425000; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=aT5uOJVtAeeBUAkpAajdbtjwvX4raqbOzSyMQgzblSU=; b=Ny7Y0PdN/9RQns2wEzOqOfKkJ0uSLNoc1sZHxc3/wlRYmR56BoGYNesxHcQmeYQ+Rjz7 TcamSMV4mGlId6Ow5NKbKszymxyq/rp1x4fvNr2NX1nfJ2vZIDmmuuJX5oQWs3WC3ZLb2 ltOoNK8XGjrn3tIDShRnGq3wS5+1Pef6cg3GFrGTvgke6UequBMOlKBr2rSwonFiCOhIg jYxygbcMB8aoQnerT+ie5ppGf907qHxV2Yi636vIh2d6VMoyglHmogmj1lnsQMQciaXQK uI1n6NweSwNfZwMGmRAC5IJ9w6EIWgw422sr/LpOKVc2dQhU6zFbeX5D3HUFh8iJX6A== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=osf7yqoJ; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1CE2609A551; Mon, 19 Oct 2026 15:49:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=osf7yqoJ Received: by smtp.kernel.org (Postfix) with ESMTPSA id C327D1F000CB; Mon, 19 Oct 2026 15:49:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792424999; bh=aT5uOJVtAeeBUAkpAajdbtjwvX4raqbOzSyMQgzblSU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=osf7yqoJhKlrME8EhhoSbtGjAm6gNPmqKAsmvl2kCGjexpFvrN8nHScNaqxrShnWs rx3pQs0xuoOLnh5Bk7/rgrzAPYwYcJHMP1X7+DeOs9eDwQREbuS0pw/fa/evwvP96u I4uQ+MN+FcvOcyvZzvvcViGnKdfaP0JIXI+oDtH3AlwWeTC+UfsZhCFWQrG43VzuRm DSy6SAGU8+HHl3aJR+g44pVkVL8BDXWnqJhIShLBw6WNeI/pqXGajX7YlMQI0UWzBG YIGRzvOZ2TvtVAqi57Thvnc/H7wchLUka9mybYRKPQD1tEWoLbL4MjqJHa7ba2azba Q9/+FUsq0JeFA== Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfauth.ams.internal (Postfix) with ESMTP id 620B51980047; Mon, 19 Oct 2026 11:49:50 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Mon, 19 Oct 2026 11:49:54 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTESqR4vElUNfwe4Wb1kabMg9MdYUtHyNIvlHi5jfGRsEd0tjiYfzpqh5FemuLlMuA +QzWL/iolJzzDzQk8vzuoT2/u2xyL/YZG/CihU2eaSQe3uAw7YKu/Z/hQfgMIJpEQrBQkK hH3KJVpCmjg+gNXnnOIkdBMKpJa1pvTLqzuojrt68Uovke10kpOzVhZrNEf3d58NDgoR+6 GBwzdAiHoLFFDGJo3/r+XBHeUTBbw+IeFaYdgC0j5JEQlKAOMB5I7tBUfVMuw5hMfUnXf2 vE1DwOk7N/x2FbtU++Y3QX/1ODQhD5sTVGp3EMvK4jUTItTZrn4RBPsK+PhnbeHfl8JcRn 8N9/3E1xHLQzkStTAM1kYcniVBbKweqHzIBqUZ0NEcBBHDz6IupfY1aF3rRGHQ7LEKorpQ gVm1/ZBoA9VHbQ2f0U4WL3yW8ieJn87dOng4oTtEeVM77LlnanTJPqYvdg7zkq63FOQgKl 3kot45HjaQuhK8dJORgnME5BEO93itG6PuiaOezy3CdEINnT/QZfplvecCCoMpOpirmntr okr8m3b9OCWqKNQ40j0/FGLxSIYnrEYpesykRn74vh+bLDj7UqIqVtDJTDUCObB8Dmfwl9 3IkUZlxtEaJ6iiXBMcXuT4X8iGRYOeLYeWrgJmoHU/RRiZrC5LOhzjWwUwYQ X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:49:49 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com, "Kiryl Shutsemau (Meta)" Subject: [RFC PATCH 05/18] mm: expedite the grace period when a CPU has many deferred frees Date: Mon, 19 Oct 2026 16:48:58 +0100 Message-ID: <20261019154913.1765974-6-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Linus Torvalds A deferred folio waits one to two grace periods on its CPU's list, where the page allocator cannot use it. A CPU that frees many flagged folios at once, for example when a file is truncated, can hold a lot of memory there. When more than 4 MiB is pending on a CPU, a work takes both of the CPU's lists and frees them after an expedited grace period. That grace period starts after the take, so it covers every reader that found one of the folios. The task that frees them does not wait. An expedited grace period interrupts every CPU, so keep it for when the memory matters. That is when the zone is below its high watermark, or when the pages waiting in it are more than an eighth of its free pages. The watermark alone comes too late for a bulk free. A truncate or fadvise(POSIX_FADV_DONTNEED) of a hot range defers all of it while the zone still has memory. When the readers that refill the range then run short, nothing is freed that would check again. With rcu_normal, as on PREEMPT_RT, the expedited grace period is a normal one. pgfree_rcu_expedite in /proc/vmstat counts the expedite runs that freed something. Assisted-by: LLM Signed-off-by: Linus Torvalds [kas: take both lists, expedite only when the memory matters, rewrote the changelog] Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/vm_event_item.h | 2 +- mm/page_alloc.c | 61 ++++++++++++++++++++++++++++++++++- mm/vmstat.c | 1 + 3 files changed, 62 insertions(+), 2 deletions(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index cbe849701653..45527fa7d0c1 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -36,7 +36,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, FOR_ALL_ZONES(ALLOCSTALL) FOR_ALL_ZONES(PGSCAN_SKIP) PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE, - PGFREE_RCU, + PGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFAULT, PGMAJFAULT, PGLAZYFREED, PGREUSE, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 15bb66e36304..a9058d414f92 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1313,6 +1313,8 @@ static inline void pgalloc_tag_sub_pages(struct alloc= _tag *tag, unsigned int nr) =20 #endif /* CONFIG_MEM_ALLOC_PROFILING */ =20 +#define RCU_FREE_EXPEDITE (SZ_4M >> PAGE_SHIFT) + /* * Each CPU has at most one grace period in flight (@queued), covering * @batch. Folios freed meanwhile wait on @pending for the next one. @@ -1322,7 +1324,9 @@ struct rcu_free_pcpu { bool queued; struct list_head pending; struct list_head batch; + unsigned long nr_pending; struct rcu_work rwork; + struct work_struct expedite; }; =20 static DEFINE_PER_CPU(struct rcu_free_pcpu, rcu_free_pcpu); @@ -1337,17 +1341,29 @@ static void rcu_free_start(struct rcu_free_pcpu *r) return; =20 list_splice_init(&r->pending, &r->batch); + r->nr_pending =3D 0; r->queued =3D true; WARN_ON_ONCE(!queue_rcu_work(rcu_free_wq, &r->rwork)); } =20 -static void rcu_free_pages(struct list_head *list) +static void rcu_free_take(struct rcu_free_pcpu *r, struct list_head *list) +{ + lockdep_assert_held(&r->lock); + + list_splice_tail_init(&r->batch, list); + list_splice_tail_init(&r->pending, list); + r->nr_pending =3D 0; +} + +static unsigned long rcu_free_pages(struct list_head *list) { struct folio_batch fbatch; struct folio *folio, *next; + unsigned long nr =3D 0; =20 folio_batch_init(&fbatch); list_for_each_entry_safe(folio, next, list, lru) { + nr +=3D folio_nr_pages(folio); mod_zone_page_state(folio_zone(folio), NR_FREE_RCU_PAGES, -folio_nr_pages(folio)); if (!folio_batch_add(&fbatch, folio)) { @@ -1357,6 +1373,17 @@ static void rcu_free_pages(struct list_head *list) } if (folio_batch_count(&fbatch)) free_unref_folios(&fbatch); + return nr; +} + +/* Free what rcu_free_take() took: the grace period starts after that */ +static unsigned long rcu_free_flush(struct list_head *list) +{ + if (list_empty(list)) + return 0; + + synchronize_rcu_expedited(); + return rcu_free_pages(list); } =20 static void rcu_free_workfn(struct work_struct *work) @@ -1374,17 +1401,48 @@ static void rcu_free_workfn(struct work_struct *wor= k) rcu_free_pages(&done); } =20 +static void rcu_free_expedite_workfn(struct work_struct *work) +{ + struct rcu_free_pcpu *r =3D container_of(work, struct rcu_free_pcpu, + expedite); + LIST_HEAD(list); + + spin_lock_irq(&r->lock); + rcu_free_take(r, &list); + spin_unlock_irq(&r->lock); + + if (rcu_free_flush(&list)) + count_vm_event(PGFREE_RCU_EXPEDITE); +} + static void rcu_free_defer(struct folio *folio) { struct rcu_free_pcpu *r =3D raw_cpu_ptr(&rcu_free_pcpu); struct zone *zone =3D folio_zone(folio); + unsigned long free, waiting; unsigned long flags; =20 spin_lock_irqsave(&r->lock, flags); __mod_zone_page_state(zone, NR_FREE_RCU_PAGES, folio_nr_pages(folio)); list_add_tail(&folio->lru, &r->pending); + r->nr_pending +=3D folio_nr_pages(folio); rcu_free_start(r); spin_unlock_irqrestore(&r->lock, flags); + + /* Unlocked: a stale count delays the expedite by one free */ + if (data_race(r->nr_pending) <=3D RCU_FREE_EXPEDITE || + work_pending(&r->expedite)) + return; + + /* + * An expedited grace period interrupts every CPU: only while the + * zone is short of memory, or while more than an eighth of what it + * has free waits here. + */ + free =3D zone_page_state(zone, NR_FREE_PAGES); + waiting =3D zone_page_state(zone, NR_FREE_RCU_PAGES); + if (free < high_wmark_pages(zone) || waiting > free / 8) + queue_work(rcu_free_wq, &r->expedite); } =20 static void __init rcu_free_init(void) @@ -1404,6 +1462,7 @@ static void __init rcu_free_init(void) INIT_LIST_HEAD(&r->pending); INIT_LIST_HEAD(&r->batch); INIT_RCU_WORK(&r->rwork, rcu_free_workfn); + INIT_WORK(&r->expedite, rcu_free_expedite_workfn); } } =20 diff --git a/mm/vmstat.c b/mm/vmstat.c index 64068d73d6d2..aa7280b5e69c 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1345,6 +1345,7 @@ const char * const vmstat_text[] =3D { [I(PGDEACTIVATE)] =3D "pgdeactivate", [I(PGLAZYFREE)] =3D "pglazyfree", [I(PGFREE_RCU)] =3D "pgfree_rcu", + [I(PGFREE_RCU_EXPEDITE)] =3D "pgfree_rcu_expedite", =20 [I(PGFAULT)] =3D "pgfault", [I(PGMAJFAULT)] =3D "pgmajfault", --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425005; b=x4DthK7caMbxMkmAjkNuu5DIxwCg5Ds4iQaS9mfcaLctBC8wFlOxBXGGAuc7WLaKL7t6 G1DQcrSa/czpql3fjXad1vrzFIxbVX68+r3hKi8H+jJVFY6ByNQuN0e/c5+I0Js8wnZ6q IYMmeeFgVWbdVUCns7pF1ImgOmo7eRe4YG4W3VaIjzaBT5HA24tjN1WztSJKt11UVHfA6 yw2vUkzbPzcjvArg802KELecEh+j3qxkHVH7wxulMcI7FWfbWb+SWoKEVocL42PjHHnAE 2nsNuEZlyRYViZ3blmA7h7SYhweQ+d1Q6s+qopvRYqhhxkxdpHvcViA8eESwIVuCdIQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425005; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=n+S/ugjb/OJ8R7PrPj5yXY/1bn/4OHkMP3wOgkiMBWs=; b=IbM5Zmool280QPNquGT77OUBG1LhZdMfFLqJ30L+O68VrhRPy5fJ3oUf8xMUZD5hxVCv DoZjKb6k+o69BHhQDTY6v58SsQvfHoiKVK7hu4byyVt0j01oHmDgl1q1kzRrVl8ToNzTV Dv2PGE7/N04XEcDFvJSCg4eOXQ/R2/Yi58j6xkARAItB/CDjoFE+JN6ot4+BlEds2VB/o k6oHZNPxRiCMKvfTmck2LEnN/rFYFn2S5sLBA0eVy1s67lsntukWex6nHnRYljZhU8l9z DGhaMx4HjNdX5tFaJcQU2l68ksKg3L+bQ5XUAtyMVUhMb4lVxLSMj/73MNFlG1BK3yQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=m8LxZuuc; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B07E609A552; Mon, 19 Oct 2026 15:50:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=m8LxZuuc Received: by smtp.kernel.org (Postfix) with ESMTPSA id E33F61F000CB; Mon, 19 Oct 2026 15:50:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425005; bh=n+S/ugjb/OJ8R7PrPj5yXY/1bn/4OHkMP3wOgkiMBWs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=m8LxZuuclx4RHUS5jJjxSkOswRLlo5pjpQ9wupaEyFZ89DFiWI5oq+M9ZzjgAB5R1 WPcMXMgZzy1vi4i/u5ggPMYZkLowHSr74F/i92yHV0QEVfNmxhyg8G2NeUkQZt799w feBYtEiHDm8syHRZsZGzw2kebSdhmFvt1+EMyEWvBGtehsIUQw+oa+wP7VGuomIw7Y 2bEWbG6aITVhZMmPQKe416GSU50vDxW5pvwPhZwh6VKxax2HyQVGDCYHLo0qyFn5Dn uB7w65BUXz74SDKmQpLLGupvtAB6qxNXykZJ6azmmFMNxxmAOvK9eZbCPZmHmxrHPP 8j2EK1MsXYTSA== Received: from phl-compute-11.internal (phl-compute-11.internal [10.202.2.51]) by mailfauth.ams.internal (Postfix) with ESMTP id A05B3198003A; Mon, 19 Oct 2026 11:49:57 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-11.internal (MEProxy); Mon, 19 Oct 2026 11:50:01 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFpjrG+2jfVBfklNRE95oThC60xlhUtv5sioyJrfpDDHbEB/2cXkTI411oJriXTLs N7RLfOUdBbmoibSFu4zT/f1BKPPyxkEYUp5eDeafGhPQjcnfJv/29ZtYwE6q77Fr7A6zQu C94bEu3EPqJJ50fLC5xP9PtYZOq3sS+jMfKkgkZFNUAdZxkDiyHArLijwBk9MwzijLWdvR 4raSXvkSyIGvp9XYd2doZUGuEANzDQejgJzzg3ARZCjBL2yZfXIvLZCg0AMujDeTHEemYv cvWt94P7AS+P1mtikHZvyS9eAaK3Fy2/TwDlvaPXULPDsZK9f0sj0aca+uCBBoNs4d3uVo YdtzlwPs9YNSDvT8I3PUckOtBZwF77w7yengukH2a2sZdb3tiP6Qlt9T5excPi6bNA+J9g mqbobw5Mi702UDCjkOsfr5wroidnBhzWPbrlA+FCXpkk1xOe0/EVkUvBzzMoafJuBeCCLL WBPgNlBt2yek1h9VvQvgT19ztxgajydWWVggL5ja5MkU8lXwa1Ls+occorai2u4Q8LDn1P 8mDhUG6AEuUB2cC14L0ld1brKkx1KVX6VyYLkLq4OMYy5fXiCB2XT9oTKPtorFOMkSTp2b he8TCnhUR6vRkWMBzCEpOD/1xv5Omsu8k9kWVo4e6H2Xrswc560yXC96kwaQ X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:49:56 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 06/18] mm: free the pages waiting for a grace period before OOM Date: Mon, 19 Oct 2026 16:48:59 +0100 Message-ID: <20261019154913.1765974-7-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When reclaim fails, should_reclaim_retry() looks only at free and reclaimable pages. It can hand the allocation to the OOM killer while enough memory is about to come back from the pages that wait for a grace period. Before should_reclaim_retry() gives up, free the waiting pages if they would bring an eligible zone above its min watermark, and retry. That costs an expedited grace period, which is cheap next to the OOM killer. The retries are bounded per no-progress streak, so a drain that does not help still ends in an OOM kill. pgfree_rcu_drain in /proc/vmstat counts the drains that freed something. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/vm_event_item.h | 2 +- mm/page_alloc.c | 88 +++++++++++++++++++++++++++++++++-- mm/vmstat.c | 1 + 3 files changed, 86 insertions(+), 5 deletions(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 45527fa7d0c1..7626a3e3a07a 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -36,7 +36,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, FOR_ALL_ZONES(ALLOCSTALL) FOR_ALL_ZONES(PGSCAN_SKIP) PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE, - PGFREE_RCU, PGFREE_RCU_EXPEDITE, + PGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFREE_RCU_DRAIN, PGFAULT, PGMAJFAULT, PGLAZYFREED, PGREUSE, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a9058d414f92..bec9128d80db 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1445,6 +1445,29 @@ static void rcu_free_defer(struct folio *folio) queue_work(rcu_free_wq, &r->expedite); } =20 +/* + * Free what every CPU's lists hold. A concurrent drain holds the lock + * until it has freed what it took. + */ +static void rcu_free_drain(void) +{ + static DEFINE_MUTEX(lock); + LIST_HEAD(list); + int cpu; + + mutex_lock(&lock); + for_each_possible_cpu(cpu) { + struct rcu_free_pcpu *r =3D per_cpu_ptr(&rcu_free_pcpu, cpu); + + spin_lock_irq(&r->lock); + rcu_free_take(r, &list); + spin_unlock_irq(&r->lock); + } + if (rcu_free_flush(&list)) + count_vm_event(PGFREE_RCU_DRAIN); + mutex_unlock(&lock); +} + static void __init rcu_free_init(void) { int cpu; @@ -4792,6 +4815,52 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask) return !!__gfp_pfmemalloc_flags(gfp_mask); } =20 +/* + * Reclaim is about to give up: if the pages waiting for a grace period + * would bring an eligible zone above its min watermark, free them and + * retry. Not for a dying task, which frees memory by exiting. + */ +static bool rcu_free_wait(gfp_t gfp_mask, unsigned int order, + struct alloc_context *ac, int alloc_flags, + int *rcu_free_waits) +{ + struct zone *zone; + struct zoneref *z; + + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return false; + if (tsk_is_oom_victim(current) || fatal_signal_pending(current)) + return false; + if (*rcu_free_waits >=3D MAX_RECLAIM_RETRIES || + order > PAGE_ALLOC_COSTLY_ORDER) + return false; + + for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, + ac->highest_zoneidx, ac->nodemask) { + unsigned long waiting, available; + + if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) && + !__cpuset_zone_allowed(zone, gfp_mask)) + continue; + + waiting =3D zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES); + if (!waiting) + continue; + + available =3D waiting + + zone_page_state_snapshot(zone, NR_FREE_PAGES); + if (__zone_watermark_ok(zone, order, min_wmark_pages(zone), + ac->highest_zoneidx, alloc_flags, + available)) { + (*rcu_free_waits)++; + rcu_free_drain(); + return true; + } + } + + return false; +} + /* * Checks whether it makes sense to retry the reclaim to make a forward pr= ogress * for the given allocation request. @@ -4805,7 +4874,8 @@ bool gfp_pfmemalloc_allowed(gfp_t gfp_mask) static inline bool should_reclaim_retry(gfp_t gfp_mask, unsigned order, struct alloc_context *ac, int alloc_flags, - bool did_some_progress, int *no_progress_loops) + bool did_some_progress, int *no_progress_loops, + int *rcu_free_waits) { struct zone *zone; struct zoneref *z; @@ -4816,10 +4886,12 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, * their order will become available due to high fragmentation so * always increment the no progress counter for them */ - if (did_some_progress && order <=3D PAGE_ALLOC_COSTLY_ORDER) + if (did_some_progress && order <=3D PAGE_ALLOC_COSTLY_ORDER) { *no_progress_loops =3D 0; - else + *rcu_free_waits =3D 0; + } else { (*no_progress_loops)++; + } =20 if (*no_progress_loops > MAX_RECLAIM_RETRIES) goto out; @@ -4872,6 +4944,10 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order, else cond_resched(); out: + if (!ret && rcu_free_wait(gfp_mask, order, ac, alloc_flags, + rcu_free_waits)) + return true; + /* Before OOM, exhaust highatomic_reserve */ if (!ret) return unreserve_highatomic_pageblock(ac, true); @@ -4961,6 +5037,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int o= rder, enum compact_result compact_result; int compaction_retries; int no_progress_loops; + int rcu_free_waits; unsigned int cpuset_mems_cookie; unsigned int zonelist_iter_cookie; int reserve_flags; @@ -4985,6 +5062,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int o= rder, restart: compaction_retries =3D 0; no_progress_loops =3D 0; + rcu_free_waits =3D 0; compact_result =3D COMPACT_SKIPPED; compact_priority =3D DEF_COMPACT_PRIORITY; cpuset_mems_cookie =3D read_mems_allowed_begin(); @@ -5163,7 +5241,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int o= rder, goto restart; =20 if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags, - did_some_progress > 0, &no_progress_loops)) + did_some_progress > 0, &no_progress_loops, + &rcu_free_waits)) goto retry; =20 /* @@ -5206,6 +5285,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int o= rder, /* Retry as long as the OOM killer is making progress */ if (did_some_progress) { no_progress_loops =3D 0; + rcu_free_waits =3D 0; goto retry; } =20 diff --git a/mm/vmstat.c b/mm/vmstat.c index aa7280b5e69c..d994de1fd739 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1346,6 +1346,7 @@ const char * const vmstat_text[] =3D { [I(PGLAZYFREE)] =3D "pglazyfree", [I(PGFREE_RCU)] =3D "pgfree_rcu", [I(PGFREE_RCU_EXPEDITE)] =3D "pgfree_rcu_expedite", + [I(PGFREE_RCU_DRAIN)] =3D "pgfree_rcu_drain", =20 [I(PGFAULT)] =3D "pgfault", [I(PGMAJFAULT)] =3D "pgmajfault", --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425011; b=mdcWCjrjGihuI8oowkZmfDFSyDl5gFnusDS2c6cP2S4xVMpvX/R2t30C1zn4GtjBr+2F FPMzHVxdALBQ26Sl44Z40qLlObcMuVZOwRnxIUMWsYizOOxXeysZ5ZTNZOMoNTMd9XJ+L YIy3kzXI+QiYEJC1XZRZUUZE/XBcrJsPpuLDKP1vucjhoAKBgfgxLJ8GERX2rhuYFx40L lYPxzX1CEGctlqQtUvH/11IyB49YvN6VZM+uh3/mFmmJivIp/EJcoV7QPBNbkso9AR3xr MW7shKozzh12Vhw15/Jq1gl7fDlj/cnHJMyEKAq7wODRfg7Y1r6LIjcFHc8Lf5gJumw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425011; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=JIYW0lKY8r5WbgU97oH0sqdYNOmLWzVfRBL2C79VgiY=; b=1o2AMvJtCabq4E2z7o27T3kLMCeY/EFXhcXNOVD88nKeWci1Jny1daYlUpYOOZwYIGCN JgdYnX0nly36VdGwkApXQ6bnvNooi/O8PsGKU2O5aUwli4/kdUnPHrQyzLrPXjKfl7omC 1dKWeOGnJpfRJItY6m0QhYG273yvIaaK7MEcFOrZ9gda+O01BB4wNjPxN7352MLj+QvCl Yd/vWLwbbfQl+kHM4rD6oyzeUBMQ7B949NvzBK8ofcIiIghdtkAolQ5itgmJ5IfrfAj4T 9xjo5f7r1eLc1j4CJW1BCxm6OjjwwIIkHiOyf+2NZKOiZMIbkgA6a/zRyQLA7qMSBuQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=IO1bSDKP; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CE656080A8F for ; Mon, 19 Oct 2026 15:50:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=IO1bSDKP Received: by smtp.kernel.org (Postfix) with ESMTPSA id 86F321F000C6; Mon, 19 Oct 2026 15:50:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425010; bh=JIYW0lKY8r5WbgU97oH0sqdYNOmLWzVfRBL2C79VgiY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=IO1bSDKPLcp5IZ4j9DF2iUY/nVHpZ2Ua0hkxJ6hhDfn4XuI+p75xy/G4r12EwpmwA u9N5UFn4VQTIPcGtGn+FunQM+L/IUiwrGwWSdASYj22meM02+hUe9iJdx4w2g9/hpm 6EvB+V2jVdcwqhAuOd1RGButeU0lqT8lGyfFWLVsHnWAMzjvnX/fXhxjElIFl1wDRL CJTXpI5NNpvcFFihmZEID+cEwWNKcW1me9dnqOYGh90fzo+k4IG0wHY+XhfWK3ozkn F2STF+Yn1sN26tiX/PEvx0pNYiohQ5Ov6fO8BwCeRW0izBfF0UVQjPYxlSmBYAE0q7 w7OKXhO8OSpqg== Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfauth.ams.internal (Postfix) with ESMTP id 5C68E1980047; Mon, 19 Oct 2026 11:50:04 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-05.internal (MEProxy); Mon, 19 Oct 2026 11:50:08 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFpjrG+2jfVBfklNRE95oThC60xlhUtv5sioyJrfpDDHbEB/2cXkTI411oJriXTLs N7RLfOUdBbmoibSFu4zT/f1BKPPyxkEYUp5eDeafGhPQjcnfJv/29ZtYwE6q77Fr7A6zQu C94bEu3EPqJJ50fLC5xP9PtYZOq3sS+jMfKkgkZFNUAdZxkDiyHArLijwBk9MwzijLWdvR 4raSXvkSyIGvp9XYd2doZUGuEANzDQejgJzzg3ARZCjBL2yZfXIvLZCg0AMujDeTHEemYv cvWt94P7AS+P1mtikHZvyS9eAaK3Fy2/TwDlvaPXULPDsZK9f0sj0aca+uCBBoNs4d3uh7 RoQzewG/TY6u0dk9M/RDUi+4K1yv+yK67j7jmtVzymHX894+Uihg0lcvs6emCinqbM6wyi +gUxb773pqkq1jVVcdmSvAhK/XxD/KPCg7Hxilx6E+9oYQ5WMSaJWUvVlcniwS0w+KhrFB aoikZIMYd0Nt7GpLtgtIaE3vQKln+pJStVB0vtonatrWvsHTRAgHwTcxLdOTPe/K6HQPJR iuTIfSH64MefXniZjMOLCO5Col3kGmDlFai90qTMVtRjt8vCPwHCFc3lid8qi/aWvaM4wi UDelk5nM3XO9TE5wjafiWBoWVJEtmtWZsXCTxgC48VOvrEqJLpl2xs9xf8yA X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:50:02 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 07/18] mm: free the pages waiting for a grace period before isolation fails Date: Mon, 19 Oct 2026 16:49:00 +0100 Message-ID: <20261019154913.1765974-8-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" alloc_contig_range() and offline_pages() migrate the folios out of a range and then check with test_pages_isolated() that every page is free. A migrated folio that had PG_rcu_free waits for a grace period after its free, so the check fails. alloc_contig_range() returns -EBUSY, and a CMA or gigantic page allocation can fail; offline_pages() retries until the grace period ends. When the check fails and the zone has pages waiting, free them first. alloc_contig_range() then finds the start of the free range again, as the freed folio can merge with a free buddy below it. offline_pages() does so only after a pass that migrated folios: an offline stuck on a busy page would otherwise start an expedited grace period on every retry. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- mm/memory_hotplug.c | 7 +++++++ mm/page_alloc.c | 28 +++++++++++++++++++++++++++- mm/page_alloc.h | 1 + 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 226ab9cb078a..a8f0c95627ee 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1967,6 +1967,7 @@ int offline_pages(unsigned long start_pfn, unsigned l= ong nr_pages, .nid =3D NUMA_NO_NODE, }; unsigned long flags; + bool migrated; char *reason; int ret; unsigned long normal_pages =3D 0; @@ -2050,6 +2051,7 @@ int offline_pages(unsigned long start_pfn, unsigned l= ong nr_pages, =20 do { pfn =3D start_pfn; + migrated =3D false; do { /* * Historically we always checked for any signal and @@ -2071,6 +2073,7 @@ int offline_pages(unsigned long start_pfn, unsigned l= ong nr_pages, * out */ do_migrate_range(pfn, end_pfn); + migrated =3D true; } } while (!ret); =20 @@ -2092,6 +2095,10 @@ int offline_pages(unsigned long start_pfn, unsigned = long nr_pages, =20 ret =3D test_pages_isolated(start_pfn, end_pfn, PB_ISOLATE_MODE_MEM_OFFLINE); + /* What this pass migrated can wait for a grace period */ + if (ret && migrated && + zone_page_state_snapshot(zone, NR_FREE_RCU_PAGES)) + rcu_free_sync(); =20 } while (ret); =20 diff --git a/mm/page_alloc.c b/mm/page_alloc.c index bec9128d80db..e63628e64cbb 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1468,6 +1468,23 @@ static void rcu_free_drain(void) mutex_unlock(&lock); } =20 +/* Free every waiting folio, including those a work took off the lists */ +void rcu_free_sync(void) +{ + int cpu; + + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return; + + rcu_free_drain(); + for_each_possible_cpu(cpu) { + struct rcu_free_pcpu *r =3D per_cpu_ptr(&rcu_free_pcpu, cpu); + + flush_work(&r->rwork.work); + flush_work(&r->expedite); + } +} + static void __init rcu_free_init(void) { int cpu; @@ -7632,7 +7649,16 @@ int alloc_contig_frozen_range_noprof(unsigned long s= tart, unsigned long end, outer_start =3D find_large_buddy(start); =20 /* Make sure the range is really isolated. */ - if (test_pages_isolated(outer_start, end, mode)) { + ret =3D test_pages_isolated(outer_start, end, mode); + if (ret && zone_page_state_snapshot(cc.zone, NR_FREE_RCU_PAGES)) { + rcu_free_sync(); + + /* A freed folio can merge with a free buddy below start */ + outer_start =3D find_large_buddy(start); + ret =3D test_pages_isolated(outer_start, end, mode); + } + + if (ret) { ret =3D -EBUSY; goto done; } diff --git a/mm/page_alloc.h b/mm/page_alloc.h index b9259deddb59..f9e200581e2f 100644 --- a/mm/page_alloc.h +++ b/mm/page_alloc.h @@ -235,6 +235,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, uns= igned int order, int nid, alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__)) void free_frozen_pages(struct page *page, unsigned int order); void free_unref_folios(struct folio_batch *fbatch); +void rcu_free_sync(void); =20 #ifdef CONFIG_NUMA struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order); --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425019; b=qpCs7fZhe4VnDwpZuT8D/9YGhFazQfv0uReCBlHc3kO2MDcdg44fxPK7rHwSCfiRU4/8 95tpWnkJo+P7LA8FBz5OzGbclsoTnKSJBGcfF68W8BQqujR5L2Qfd21wet/OFaV702pu3 05FR4YKzTHv/jsU0s0wMdZjEX4InD4BGLQeWq6Qv7miNVMTrCL0TSMX1D61WHc2lr4KT8 8wW2WhhjxeE+gU6onA8DOH07UUquN0ae0yKFFTsuEJomQejhuA+ehyJ8qJMyU1e2Eav02 TCuElrECr0YLXy3iLLmEwJ1+BHEq0rjEDlTsAPRHJq5t8IIqdfHNhIeOoOr0m3zHSRA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425019; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=PMTfPTsjqmbKCxNwaWHz1dgOx+TTOWHrc04S/ITwtbQ=; b=swaT2wobjZ6kDWEBhk4G3CC7i0TpZjSO9XJuWvT5gLBwf+3WKuHME5OLgl9qlEItVMvr 76iXmoID98twMAYBZhdgNcEYI/Uc3nRmR3EV/fmEczuXPJjWYudy0b1PxFw5rC08EYcvX QRkvYE7Z6oEgiI720uyroddAC0ox4tptTsXscXtqTLZUmN6wbcn4FHJVZUgptyfLmpQjz TSQgHYbEL0DD33SsRJ2Kvs/81A/MM8JR4ofGg23YTlRxDDiiFEeCiSKB0Xk4eJQuZ2J2i AEsbChjl2rgQAU2/yGDgu5NAtICLuzJTFVdGlfRjyXlV3r/4aK9kYBI4PMsC+HpPwaA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=hImCGeam; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD5BB606EFD2 for ; Mon, 19 Oct 2026 15:50:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=hImCGeam Received: by smtp.kernel.org (Postfix) with ESMTPSA id DF0911F000CB; Mon, 19 Oct 2026 15:50:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425018; bh=PMTfPTsjqmbKCxNwaWHz1dgOx+TTOWHrc04S/ITwtbQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=hImCGeamAmp+FYmHCgjT0+aC2HP0OJhG9IfHgF2YTc+c/Lg1JCdqfhZsR48XBx/EK g3218Ry6T4ZZJTTcb04LT7FE0brELiShRrpFIUFsUjvL5LWlhY5eRutVseYHR4uQim /1X7v5Yq2742O208tEu+CX/t9ehB8m3QfcjOO32lSugvlWN1qsOOKEp1WoJSBK6VCy udNboFaQbCEaKpzrZxbQ40MR0luvhb8si/ivsMBy9ofJ7g7IOjCDWnW+3SAOceChHh 9cSsYkLERtYijtq+YdBCF4584X657Nl6PaGOoH8k+rQek3DNxzhqYyH+xBdCJ44VcS hCsk8uz+r0voQ== Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfauth.ams.internal (Postfix) with ESMTP id BFEF3198003A; Mon, 19 Oct 2026 11:50:10 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Mon, 19 Oct 2026 11:50:14 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFpjrG+2jfVBfklNRE95oThC60xlhUtv5sioyJrfpDDHbEB/2cXkTI411oJriXTLs N7RLfOUdBbmoibSFu4zT/f1BKPPyxkEYUp5eDeafGhPQjcnfJv/29ZtYwE6q77Fr7A6zQu C94bEu3EPqJJ50fLC5xP9PtYZOq3sS+jMfKkgkZFNUAdZxkDiyHArLijwBk9MwzijLWdvR 4raSXvkSyIGvp9XYd2doZUGuEANzDQejgJzzg3ARZCjBL2yZfXIvLZCg0AMujDeTHEemYv cvWt94P7AS+P1mtikHZvyS9eAaK3Fy2/TwDlvaPXULPDsZK9f0sj0aca+uCBBoNs4d3uHs jMUJ1Iozqk5UdmIvSWusRKiLxZJibibQQ4io5vi2zZE/MO/E51//OOpVWfhv0zTt87YwNC iT4vrAPyd6Zw38/TgZPAZuDM5MOQ5uy92jrk9+kPwi9FArfqH5nSplQoca7j1ODXWJuKjo Dzl3htmBf/kiSZk95GYec6x/otNAVITvtIg8Nf+oD+dUiJA4EkTHkc1Gpwfmk27qWoZJfQ yFrAjVY2J4RzUnQGa94QZsGOXiqq3U5awoY8NMbMgVRjboeQ00T7MEBaY6re66zsU4Ec9D 0R4U/ITBWynGFo5znxlH1QA0ft/B0hdJAgXaFWzh6VKsSmugq2I9X73PmpLw X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:50:09 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 08/18] mm/memory-failure: handle folios with PG_rcu_free Date: Mon, 19 Oct 2026 16:49:01 +0100 Message-ID: <20261019154913.1765974-9-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A page whose free waits for an RCU grace period has refcount 0 and is neither on the LRU nor in the buddy allocator. memory_failure() and soft_offline_page() cannot handle such a page: get_any_page() retries a few times and gives up. Soft offline of the page fails, and a hard error is ignored. If the page is part of a large folio, the folio is later freed with the poisoned page in it, and that page can be handed out again. Free the waiting pages first when the page looks like one of them. memory_failure() has to do it before it sets PG_hwpoison. The free of a poisoned order-0 page drops the page instead of returning it to the buddy allocator, so a later drain would still leave the error ignored. A folio with PG_rcu_free that is still in use may be copied by a reader that holds no reference. When memory_failure() succeeds, its caller can unmap the page from the direct map under such a reader. Wait for a grace period before returning success. By then the page has left the page cache, or is an order-0 folio with PG_hwpoison, which a reader that starts after the grace period has to check. Wait at the end, not right after setting PG_hwpoison. A large folio is split later, and until the new folios are in the page cache, a reader can still copy the poisoned page as part of the large one. Both callers hold mf_mutex and can wait for the grace period. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- mm/memory-failure.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 7dde7c3c9a0c..dc66cac05b1f 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1565,6 +1565,21 @@ static int get_hwpoison_page(struct page *p, unsigne= d long flags) return ret; } =20 +/* A page waiting for a grace period is not in the buddy yet: free it */ +static void mf_free_rcu_pages(struct page *p) +{ + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return; + if (page_count(p)) + return; + if (!zone_page_state_snapshot(page_zone(p), NR_FREE_RCU_PAGES)) + return; + if (is_free_buddy_page(p)) + return; + + rcu_free_sync(); +} + /* * The caller must guarantee the folio isn't large folio, except hugetlb. * try_to_unmap() can't handle it. @@ -2440,6 +2455,7 @@ int memory_failure(unsigned long pfn, int flags) if (res !=3D -ENOENT) goto unlock_mutex; =20 + mf_free_rcu_pages(p); if (TestSetPageHWPoison(p)) { res =3D -EHWPOISON; if (flags & MF_ACTION_REQUIRED) @@ -2612,6 +2628,13 @@ int memory_failure(unsigned long pfn, int flags) =20 identify_page_state: res =3D identify_page_state(pfn, p, page_flags); + /* + * The caller may unmap the page once this returns 0. Lockless + * readers that started before the page got PG_hwpoison, or before a + * split left it in a folio of its own, may still be copying it. + */ + if (!res && folio_test_rcu_free(folio)) + synchronize_rcu(); mutex_unlock(&mf_mutex); return res; unlock_page: @@ -3003,6 +3026,7 @@ int soft_offline_page(unsigned long pfn, int flags) =20 retry: get_online_mems(); + mf_free_rcu_pages(page); ret =3D get_hwpoison_page(page, flags | MF_SOFT_OFFLINE); put_online_mems(); =20 --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425025; b=JhBowMAZo12J700lWEWj+fxXcpHZmx3xRsHuYmhLkB9bhMRhYYnHFomrRL5/XcNlyUWU +sLI6TQAlO3pV4f0NWiwlXDZpRqfHEPl3OOfRw2nNIEiTPehg1K+aIYFqtKSPYz+VUkSY lQ5LUBwkpZ8YFRYtZWy5EjLM48ANueuxStRlx+BDp5ae6GjumI5f5FoXCRjy0dR1ePEFm ikNxakF3+Rx6tf8tUaNAAjOc/ALuJB4dlKPW7aiQ5D/zDAFRDLIfR3yBuaC7m1VdOpJ2R XwArsE7cqKH4rpQ6tR2s8h+3SOJyGvXf8IS21Xh5c5meDvFwjm8bdmqpBGlfsyeTo+Q== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425025; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=+5Nfzrde1GD9FfRh74qOmGM0CUmDocdXRs1qdcRag3k=; b=Vwt9VifGF1Kj+7/GNqbp8aZfzoc511kIJrYJXabHvL/rWR/oWFPTdFtDuKx2knTHd07B ioycfiIHZwJLcVjGyX946CCrZv3bhPwNZhyeHjV7hBskIeCTLtuB278iJjWYVdUC8knRE mri02qQ0CzWKSZFPJ+ELIAptaQmLMvyfw5xdd7iqNuF4qXGOwU2bAsbxnQVxkuio7Bsep YUMkIpP+EnRY0nuoE7FQw2BaclXF0P2oDXpMuH+EPcgxRHc5MlpjuW8Dg1YqcoV+uGnXQ /QhpLgCmB1NfScCIbDDwqrfrN1p/Lbu+C0tew6eG72w+V7mdJlB7ge2+FIZo9W7F3SA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=ljvEE+u3; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43766607D342; Mon, 19 Oct 2026 15:50:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=ljvEE+u3 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 55D361F000C6; Mon, 19 Oct 2026 15:50:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425025; bh=+5Nfzrde1GD9FfRh74qOmGM0CUmDocdXRs1qdcRag3k=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=ljvEE+u3oeDjYDCjGhqAkTws6/4QJfEktpl7VIAzkm1wbv/mLvVPT7PJyAWLDq5Gt Dx8v1Zw9QfsqO7syFOlR4E6YWIoNdoSWU0CEKKtHX4PPP55zLI04atrDRCVajHT+CL U0tJRg4MDCiT9DcKIT5hlRuTHsgHPMa0pqQwpU+1ieemLEpvbjGXkbCBJOcquo+RKj M1G4CbV319nSBW1W/1kGoRI19m1eGKJlY09uVX5xEgZejFwiN3BOoyLHr6DfkdGrIn XvWXI1/CNypi4ePfPPquLRjRyIpeqEG4vTh1CMZE0hu6JLrzZL7dJJkvNsKD1AFPgK icz1lv5eA/kIw== Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfauth.ams.internal (Postfix) with ESMTP id 2DA0E1980047; Mon, 19 Oct 2026 11:50:17 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Mon, 19 Oct 2026 11:50:20 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTE+kjgISKw/D5xbtovMvE/5+MXw9OAGtDBfqR9Zk5dZNOqhNWGTcEGO5Lmn7qsaMJ P+2+K5eT4SfShSSEvxMo9+0NhMtUrbV4W6Uy0A3XpuEB672w8089GbqFaxEF0wWgdpl5Mt ZGUplA1E9XDXzNgkO2MrFgMfcNS4u61SieDRP0v46gjH0l2eZ2DL9zme+jYfCIiYlLA1YJ nvbn1HssXPAJFHKdL+OCiEp78jaTWZC2fJHjA4vCnfghQrymIXyUGAHupLMJJEm2bjt0nv kEsuBoXYnW/I4u0Q3SZjAxl07Y4ERxrcHOAeD/N6wYko4+jO11wXgs98kOmjLgJ2rMOJIZ dfeJ0xQ74xoR6DfShFlSEAqdM3aQWhYGU9iaTUYTePGvrkWjscCqx8LuiCBIrLxzWO5esr XIWYeR8T4YsFXzTV8r5ByFLLce3aWQVDs/AH8HBc2svQkWdXOwpi0Xzyr1iXxxy3YDQxuu d+vqpBfcAJrom7RgY/UM1LcB98A5Ic6xZOKqm1cV0yUmWcjUHVykTDmFvPPPI9Ab4nRoMb UX70np05MwHA6FfDOUprjmrpo5WFE5+GZaMR+0EHllstbAYlSw/gD9O0Orj1fDNJWYImOg jGxc6OHbvGHcn3reOVmwkwpuisL9nJUWe/8+U8Oz7SBLCzpL0Si1/Pkv3cNQ X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:50:15 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com, "Kiryl Shutsemau (Meta)" Subject: [RFC PATCH 09/18] mm: have folio_mark_accessed() return whether the folio is hot Date: Mon, 19 Oct 2026 16:49:02 +0100 Message-ID: <20261019154913.1765974-10-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Linus Torvalds Each call of folio_mark_accessed() moves a folio one step up what the LRU records of its accesses. Once the folio is at the top, the call leaves it as it was: the folio is as hot as the LRU can tell. Return whether the call found the folio there. On the classic LRU that is active, referenced and not idle. On the multi-gen LRU it is PG_referenced, all the refs bits and PG_workingset, which lru_gen_inc_refs() now reports. An unevictable folio counts as hot once it is referenced, and on the multi-gen LRU from the first call: it stays in memory in any case. A dropbehind folio is not hot: it is dropped after the read. No caller uses the value yet. Preparation for reading the folios that it calls hot without a folio reference. Assisted-by: LLM Signed-off-by: Linus Torvalds [kas: shorter changelog] Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/swap.h | 2 +- mm/folio.c | 46 ++++++++++++++++++++++++++++++-------------- 2 files changed, 33 insertions(+), 15 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 5658a1634b85..67aa5271ae6c 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -295,7 +295,7 @@ extern unsigned long totalreserve_pages; =20 /* linux/mm/folio.c */ void folio_add_lru(struct folio *folio); -void folio_mark_accessed(struct folio *folio); +bool folio_mark_accessed(struct folio *folio); void lru_add_drain_all(void); =20 enum lru_cache_drained { diff --git a/mm/folio.c b/mm/folio.c index 50a6dbe55998..6144b9cf301b 100644 --- a/mm/folio.c +++ b/mm/folio.c @@ -352,28 +352,34 @@ static void __lru_cache_activate_folio(struct folio *= folio) =20 #ifdef CONFIG_LRU_GEN =20 -static void lru_gen_inc_refs(struct folio *folio) +/* + * Return true if @folio is hot: unevictable, or its flags already held as + * many accesses as they can. + */ +static bool lru_gen_inc_refs(struct folio *folio) { unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags.f); =20 if (folio_test_unevictable(folio)) - return; + return true; =20 /* see the comment on LRU_REFS_FLAGS */ if (!folio_test_referenced(folio)) { set_mask_bits(&folio->flags.f, LRU_REFS_MASK, BIT(PG_referenced)); - return; + return false; } =20 do { if ((old_flags & LRU_REFS_MASK) =3D=3D LRU_REFS_MASK) { - if (!folio_test_workingset(folio)) - folio_set_workingset(folio); - return; + if (folio_test_workingset(folio)) + return true; + folio_set_workingset(folio); + return false; } =20 new_flags =3D old_flags + BIT(LRU_REFS_PGOFF); } while (!try_cmpxchg(&folio->flags.f, &old_flags, new_flags)); + return false; } =20 static bool lru_gen_clear_refs(struct folio *folio) @@ -396,8 +402,9 @@ static bool lru_gen_clear_refs(struct folio *folio) =20 #else /* !CONFIG_LRU_GEN */ =20 -static void lru_gen_inc_refs(struct folio *folio) +static bool lru_gen_inc_refs(struct folio *folio) { + return false; } =20 static bool lru_gen_clear_refs(struct folio *folio) @@ -419,15 +426,20 @@ static bool lru_gen_clear_refs(struct folio *folio) * * When a newly allocated folio is not yet visible, so safe for non-atomic= ops, * __folio_set_referenced() may be substituted for folio_mark_accessed(). + * + * Return: true if @folio is hot: it was already as accessed as the LRU + * records. On the classic LRU that is active or unevictable, referenced + * and not idle; on the multi-gen LRU, unevictable, or PG_referenced, all + * the refs bits and PG_workingset set. False for a dropbehind folio. */ -void folio_mark_accessed(struct folio *folio) +bool folio_mark_accessed(struct folio *folio) { + bool hot =3D false; + if (folio_test_dropbehind(folio)) - return; - if (lru_gen_enabled()) { - lru_gen_inc_refs(folio); - return; - } + return false; + if (lru_gen_enabled()) + return lru_gen_inc_refs(folio); =20 if (!folio_test_referenced(folio)) { folio_set_referenced(folio); @@ -437,6 +449,7 @@ void folio_mark_accessed(struct folio *folio) * this list is never rotated or maintained, so marking an * unevictable page accessed has no effect. */ + hot =3D true; } else if (!folio_test_active(folio)) { /* * If the folio is on the LRU, queue it for activation via @@ -450,9 +463,14 @@ void folio_mark_accessed(struct folio *folio) __lru_cache_activate_folio(folio); folio_clear_referenced(folio); workingset_activation(folio); + } else { + hot =3D true; } - if (folio_test_idle(folio)) + if (folio_test_idle(folio)) { folio_clear_idle(folio); + hot =3D false; + } + return hot; } EXPORT_SYMBOL(folio_mark_accessed); =20 --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425033; b=H9RHPbkZ1Wh0TbS1kZMThm+i277wBgsd8KyH0GE0klgLVf8Qq8971So8wxVWK9xT9oeW 7WnICFQVK0xHGzBP8p4uJNSfC9ygInsFMka71OQi3GsOoNNJVy244toLwLuCppYq05qLA fnRtK9EDM4DiQNbhDfrxaEBmeGYtn6xm+vskMFNbhTCPdveq/xpcsEKHJ/iZA/NkVaEpR Ht1XPlasGYinvOYYIm+oHuqWPgTnX2AtjDfj7RFexOqXCrZzVUOp+Ynzquu02Kwbkbrjn mSSu76cxZvB1e2dNedwqqXnAlD8QXY/KQHvWDyMSZyeQxkbdYNnO+He0elGkRwPgDTw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425033; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=HyeMP2UW39CYOeaVl1H1MlIga1p/omRrDkZGp5zmZrQ=; b=ID4yS8Tu2KgEt4I0nnrYLROfu2J1jYVHqNsJ8f6fyGgjan0B7sid8smsD5TiOdaCf7Zm 6usEsPK/cgWciaMIAcfyt4yNK+wLAeUsd4l5Ejt/bt4Yj0AL2vNyKU4ZGds1834kpI9Dz fA3WUSjzhyfvVaKL7/oy6MNYclp2YH9kmT/WgdCqB3cqyYga8r3GzKb/UWpHn0qT45nKH YjJFiwSvjiKejZaaR9k5bzuXjfbFG+KDCrG/3GKWk5bjYHvYHhJ+V/s0X7wYVwIIFMrTy wsn9Oq8kNKYk6BMa5Xb0MZJLsqqK2NwAzAj+yIg6ggadZrVxoh7pRjTYo6pZs+qo5ww== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=mTvAz6sX; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0CDA960A382E for ; Mon, 19 Oct 2026 15:50:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=mTvAz6sX Received: by smtp.kernel.org (Postfix) with ESMTPSA id BB04A1F000CB; Mon, 19 Oct 2026 15:50:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425032; bh=HyeMP2UW39CYOeaVl1H1MlIga1p/omRrDkZGp5zmZrQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=mTvAz6sXCbY9ZEblFh5LzU7G8fSbdmLuLKLKT5q5a5ucl5PadxgglYQrrnQE8ObUP Oo4bg/n843IxlUNHIOXKu2x83+boXNWYjYNmyva0GimQ7dD/ULJ/671O8gVFWydzR0 nCJH6z+7qOTP3bxjKC+Niv3ZGlbzZvc5aU9pmJQqgOmA9np7bmKVSGWZCEP6zW1MqW MFrKN40XP7BXVsy8CNlSj2+2nsnYdOOVwtpI/g2s/Y0P4dYDDwsyJ2sVlHFREKamVC 2RmKtNUQrJL0MeV3F1+S+BGJbLcFA516l58hlJeSt2tTVafUqkVVyYQnKlMtbOSYr/ ktJoTcPfDd7Jg== Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfauth.ams.internal (Postfix) with ESMTP id 8D186198003A; Mon, 19 Oct 2026 11:50:23 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Mon, 19 Oct 2026 11:50:27 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFIZYy10wotytG+4dUkb/TrOx7ODFzNTx6LAAAJtb9SdlnEFJwy1XcImD4gyhna2/ zAC+EEqvmQ9OD1vfTU8mvYiDS0ghuns47Q4+ryXmkVilrv+pqFomIC1gUoiZOfFCnK2t+P 9O9avF7uz974hExF8BDh4AlkO2Kz5b3etknTZfQTy3eRHnd2C5TvNt8woK5q07pcssriFF 0X7YzIQRQ9gOQcIZd4ixgjAJQuYNSGk/RFHFd5/NuxCtenwYPdQO132Kw78QGPjvUTnfF3 VZ20iGzDVcdiQzqlFcp/qqA68rjIK9UtFNtsCnjCSpXpVKtOfu4ZJHZy8EM6a32BgsvOLv GTts5CoIR3ht4mxHxWO6GedS/BYYQFbtcXgGZZZ3koz+37rksmGM/qSKHSSY4u7hGUQHmQ rAidDUTvlwtc5G79J2GHONIeDMRXVPObyUry2h6EpUB/+Vk/YU4fyni8LK0vUQdhhBy2E2 wmoClH/DVozueqSCqPryKyW6Mr6WC/UsdiI/R+v2Rm+6VeTFfrHiMhVAOfWNly4VJkyS6V 1FqCqwaj3etyYvizowlf8vKHJR0w6sN3HwndbOlAo1xYgonV/h4TLvQbqE5aC/TMsWEFVA cL7rCb4wFu7n08oVJvqB68O0OYR7d5RXtC6OBk0vSwV+oN7+WFP8nCzHwLVQ X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:50:22 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com, "Kiryl Shutsemau (Meta)" Subject: [RFC PATCH 10/18] mm/filemap: mark a folio accessed in one place in filemap_read() Date: Mon, 19 Oct 2026 16:49:03 +0100 Message-ID: <20261019154913.1765974-11-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Linus Torvalds filemap_read() calls folio_mark_accessed() in two places. Before the copy loop it marks the first folio of the batch, unless the previous read through the file ended in that folio. In the loop it marks every later folio. The test made for the first folio also says "mark" for every later one. The loop gets to a later folio only after the folio before it was copied to its end, so last_pos is at the later folio's first byte and pos_same_folio() is false. Do both in one helper, filemap_read_accessed(), called for each folio in the loop. For the first folio the call moves below the test of end_offset against the folio's start, which is false for it: the folio holds ki_pos, and end_offset is beyond ki_pos. No change in behaviour intended. Preparation for deciding in the helper which folios are worth reading without a reference. Assisted-by: LLM Signed-off-by: Linus Torvalds [kas: shorter changelog] Signed-off-by: Kiryl Shutsemau (Meta) --- mm/filemap.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 00fd89cf6f55..3508eea44ad9 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2759,6 +2759,20 @@ static inline bool pos_same_folio(loff_t pos1, loff_= t pos2, struct folio *folio) return (pos1 >> shift =3D=3D pos2 >> shift); } =20 +/* + * filemap_read() is about to copy from @folio at @pos. @last_pos is where + * the previous read of this open file ended, or how far this one has got. + * + * Reading on in the folio that the last read ended in is not another acce= ss + * to it: a folio read in small pieces is marked accessed once. + */ +static void filemap_read_accessed(struct folio *folio, loff_t pos, + loff_t last_pos) +{ + if (!pos_same_folio(pos, last_pos - 1, folio)) + folio_mark_accessed(folio); +} + static void filemap_end_dropbehind_read(struct folio *folio) { if (!folio_test_dropbehind(folio)) @@ -2844,14 +2858,6 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_= iter *iter, */ writably_mapped =3D mapping_writably_mapped(mapping); =20 - /* - * When a read accesses the same folio several times, only - * mark it as accessed the first time. - */ - if (!pos_same_folio(iocb->ki_pos, last_pos - 1, - fbatch.folios[0])) - folio_mark_accessed(fbatch.folios[0]); - for (i =3D 0; i < folio_batch_count(&fbatch); i++) { struct folio *folio =3D fbatch.folios[i]; size_t fsize =3D folio_size(folio); @@ -2862,8 +2868,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_i= ter *iter, =20 if (end_offset < folio_pos(folio)) break; - if (i > 0) - folio_mark_accessed(folio); + filemap_read_accessed(folio, iocb->ki_pos, last_pos); /* * If users can be writing to this folio using arbitrary * virtual addresses, take care of potential aliasing --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425039; b=5cKqAElcqAtCqWpsI3Oy2/YTl/3mr6K697HpOEiwKyZs/f7ck7b46HawWQyAIC4m67Jo Jd+5+lO0gzA0fniolboy8jCpJH4PB+s/4oNtJIBuwN5RkNRKF0H84EdMHdZAVNDrdp0u+ 1rj5DP7/LBoY7uSQOlcbJZAFW77h4zppsDyYPpvhBfxGpDm25iRof9+b21Qf25eJ+S9PC UlTi/VENF5zxPczPKH5WnZO5lu4k3OBkKYLZlWbmEdMNYkbINu77gcjwqELBeybi8DcA6 x1fqtnfDNEHGGAqXf0AgpLUXdprB1QVymSry0CzZKsYSQBh2p7VovlFCZQQjiHkuE/A== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425039; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=UMGOxPNVHDCZHjJ/y4wrnwe474JMBsZe9H8zYo9bB5E=; b=2zVjeCO3M4qPCt8nsR+V2tF5lXU6FRUD+2StePR3zkLeBgo13Yh1tJ/rQQPlM+np+fbD jclzaSJhttqEGyl299gBmPlSHGKH/O++AK21KuzZ2wRrkBm0zZtmXrTJdq8k8uHQTjOJw ho69FfpRpW0Mt0ivYFh2eupY8K6MkvVuTOC0MbGk5F8diAJmjQfmixZ/wpFGunHhZuvy4 p7eeYgdYtD5ZnTM/AXdhgQ+MjlVFddogfmZdBiUc9bN09uz7xuVBfA3ea0ClQAwrfm70j Hji5SYaZLaDJpwFOqyFzcAfPA6W3efnVsIvYbsqBBK/N7fj6Qauk0wfdijHoeyiyVYQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=X9QxtGhz; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B14EC60A4ADB; Mon, 19 Oct 2026 15:50:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=X9QxtGhz Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6C5A61F000CA; Mon, 19 Oct 2026 15:50:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425038; bh=UMGOxPNVHDCZHjJ/y4wrnwe474JMBsZe9H8zYo9bB5E=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=X9QxtGhz8KvqIeUtwDkIIOYomC6/ZgSlsRd5UoKKs/6pi0dt0Tsf8LGIvjnJhN4K8 we6RwHUC2sEbKWcSgXEvy9/+uRLmZOuuOjcN9flfvsahfLJIMsKEZKopS0u2mtexbN +41FbQggEkUu2T+H/9R7EuO5NicJ75Xy7egIDPUDZEiPyEgLWe39ukKki+9gDdRmAK 6nJfSdJ2e9ZimCrYjLL9yIGPeeeD8ZXb6ziLl63tqG8hjvp+sAqZOkM41yVnTlbUZ3 SQOG5Wizg/R7MKwyfSTWDD6nfDWAoox2GFl2qDyTIyqC41IKAZPPRjEfaCTLkud1g2 DWYfiLRAhz+gw== Received: from phl-compute-06.internal (phl-compute-06.internal [10.202.2.46]) by mailfauth.ams.internal (Postfix) with ESMTP id 486601980047; Mon, 19 Oct 2026 11:50:30 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Mon, 19 Oct 2026 11:50:34 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTE+kjgISKw/D5xbtovMvE/5+MXw9OAGtDBfqR9Zk5dZNOqhNWGTcEGO5Lmn7qsaMJ P+2+K5eT4SfShSSEvxMo9+0NhMtUrbV4W6Uy0A3XpuEB672w8089GbqFaxEF0wWgdpl5Mt ZGUplA1E9XDXzNgkO2MrFgMfcNS4u61SieDRP0v46gjH0l2eZ2DL9zme+jYfCIiYlLA1YJ nvbn1HssXPAJFHKdL+OCiEp78jaTWZC2fJHjA4vCnfghQrymIXyUGAHupLMJJEm2bjt0nv kEsuBoXYnW/I4u0Q3SZjAxl07Y4ERxrcHOAeD/N6wYko4+jO11wXgs98kOmjLgJ2rMOJLr 4FP7CSXILgep+V+bnFblzLb2zNX5VOi9HTV33Avp2CU2x/3slL9UpGUX8QP3dV/RyCFMFG 7imraoXV3UkaOEWfGmHh0OoG9j6M1axwhaJN6nA2HyYr2TdqTNigjbOhwoon6MyVdyyZ+U 9DK1iVoT/QBuGfOV2J/7tFEPGkQChXo/3aa8Xv7Il7b4za6LWrtfdMTzCHeTrHOuCuyBi+ 7sZr0XoHceQEgIL5FXJB/u/mTl9LrcQBv3ckBqQ40FYXHHMxv6YS0bNZIBfxbsZ3EpISc7 FhCZ+mKuDN/rcicdBgn/vjQLHJDQGooDmhyDm5AYtLzSQEnR+OVRpfY12e7g X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:50:28 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com, "Kiryl Shutsemau (Meta)" Subject: [RFC PATCH 11/18] mm/filemap: set PG_rcu_free on the folios that reads keep coming back to Date: Mon, 19 Oct 2026 16:49:04 +0100 Message-ID: <20261019154913.1765974-12-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Linus Torvalds A folio with PG_rcu_free can be read without a reference, but its memory is held back for a grace period after it is freed. That trade is worth making only for folios that readers look up again and again. Flagging every page cache folio would also delay the free of streaming reads, truncated files and reclaimed cold data, which no reader comes back to. Set the flag in filemap_read(), under the read's reference, when the folio turns out to be hot: - folio_mark_accessed() finds it as accessed as the LRU records, or - the read goes back to bytes that the previous read through the same open file has just read. filemap_read() does not mark accessed the folio that the previous read ended in, so re-reads through one open file do not make a folio hotter. The second test is for that folio. Leave alone a folio whose next read needs a reference anyway: not uptodate, readahead or dropbehind. Leave alone unevictable folios too. folio_mark_accessed() calls them hot after at most one access, so every ramfs folio that is read would get the flag. An initramfs deleted at switch_root would then have the free of all its folios deferred. AS_RCU_READ records that an inode has had a folio flagged, so that a lockless read can skip the inodes that never had one. pgfree_rcu_mark in /proc/vmstat counts the pages that get the flag. Assisted-by: LLM Signed-off-by: Linus Torvalds [kas: count the marks, AS_RCU_READ, 64-bit only, shorter comment, rewrote the changelog] Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/pagemap.h | 12 +++++++++++ include/linux/vm_event_item.h | 1 + mm/filemap.c | 40 +++++++++++++++++++++++++++++++++-- mm/vmstat.c | 1 + 4 files changed, 52 insertions(+), 2 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0adfa6605653..6c7f58803eee 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -210,6 +210,7 @@ enum mapping_flags { AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM =3D 9, AS_KERNEL_FILE =3D 10, /* mapping for a fake kernel file that shouldn't account usage to user cgroups */ + AS_RCU_READ =3D 11, /* a folio has had PG_rcu_free */ /* Bits 16-25 are used for FOLIO_ORDER */ AS_FOLIO_ORDER_BITS =3D 5, AS_FOLIO_ORDER_MIN =3D 16, @@ -269,6 +270,17 @@ static inline bool mapping_unevictable(const struct ad= dress_space *mapping) return mapping && test_bit(AS_UNEVICTABLE, &mapping->flags); } =20 +static inline void mapping_set_rcu_read(struct address_space *mapping) +{ + if (!test_bit(AS_RCU_READ, &mapping->flags)) + set_bit(AS_RCU_READ, &mapping->flags); +} + +static inline bool mapping_rcu_read(const struct address_space *mapping) +{ + return test_bit(AS_RCU_READ, &mapping->flags); +} + static inline void mapping_set_exiting(struct address_space *mapping) { set_bit(AS_EXITING, &mapping->flags); diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 7626a3e3a07a..c1a63f070aa7 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -37,6 +37,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, FOR_ALL_ZONES(PGSCAN_SKIP) PGFREE, PGACTIVATE, PGDEACTIVATE, PGLAZYFREE, PGFREE_RCU, PGFREE_RCU_EXPEDITE, PGFREE_RCU_DRAIN, + PGFREE_RCU_MARK, PGFAULT, PGMAJFAULT, PGLAZYFREED, PGREUSE, diff --git a/mm/filemap.c b/mm/filemap.c index 3508eea44ad9..b732ecaa78a9 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2759,18 +2759,54 @@ static inline bool pos_same_folio(loff_t pos1, loff= _t pos2, struct folio *folio) return (pos1 >> shift =3D=3D pos2 >> shift); } =20 +/* + * Reads keep coming back to @folio: free it only after an RCU grace period + * from now on. Not if its next read needs a reference anyway. Not if it + * is unevictable: folio_mark_accessed() calls such a folio hot after at + * most one access, so every ramfs folio that is read would get the flag. + */ +static void filemap_mark_hot(struct folio *folio) +{ + struct address_space *mapping; + + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + return; + if (folio_test_rcu_free(folio) || !folio_test_uptodate(folio) || + folio_test_readahead(folio) || folio_test_dropbehind(folio) || + folio_test_unevictable(folio)) + return; + if (folio_test_set_rcu_free(folio)) + return; + + count_vm_events(PGFREE_RCU_MARK, folio_nr_pages(folio)); + /* NULL if the folio was truncated meanwhile */ + mapping =3D READ_ONCE(folio->mapping); + if (mapping) + mapping_set_rcu_read(mapping); +} + /* * filemap_read() is about to copy from @folio at @pos. @last_pos is where * the previous read of this open file ended, or how far this one has got. * * Reading on in the folio that the last read ended in is not another acce= ss * to it: a folio read in small pieces is marked accessed once. + * + * The folio is hot when folio_mark_accessed() finds it as accessed as the + * LRU records, or when this open file goes back to data that it has just + * read. */ static void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_pos) { - if (!pos_same_folio(pos, last_pos - 1, folio)) - folio_mark_accessed(folio); + bool hot; + + if (pos_same_folio(pos, last_pos - 1, folio)) + hot =3D pos < last_pos; + else + hot =3D folio_mark_accessed(folio); + if (hot) + filemap_mark_hot(folio); } =20 static void filemap_end_dropbehind_read(struct folio *folio) diff --git a/mm/vmstat.c b/mm/vmstat.c index d994de1fd739..c27caad017e3 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1347,6 +1347,7 @@ const char * const vmstat_text[] =3D { [I(PGFREE_RCU)] =3D "pgfree_rcu", [I(PGFREE_RCU_EXPEDITE)] =3D "pgfree_rcu_expedite", [I(PGFREE_RCU_DRAIN)] =3D "pgfree_rcu_drain", + [I(PGFREE_RCU_MARK)] =3D "pgfree_rcu_mark", =20 [I(PGFAULT)] =3D "pgfault", [I(PGMAJFAULT)] =3D "pgmajfault", --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425045; b=yWushCv0OZGVjZxAz7qnSmXFW6azyMUR5gMjhdVARJAXlv7lRaZ6Ne2yyYd/gNsoXWiP 8RfzHddu1f3k2qJADbQbxqZbZObKoCsy65njxKN+BwhORLWtV9SlX/bArFUefUNvppMLU 4GTjZ+0NJRDPfeajdgXJevQ1Mj9W+Sd5lm2ppLhfTv0mHAhbm+ax693ERs2JuvkKA27Ll c/mbGHSt3sqCqxEkRwtaA6/8uGYZQNi1SSMM3pUSFpQwz09hkFGnlM3H2x22l4PVa2mdL nGh3An3SjxkDR8vJ33tvlunQVBdzCFLSgEvQC6lXQomNis0l5lwVOZDaScWuRC50seg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425045; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=84kciOlrBZyXkODv4mERfw9QO7WVRKrOSp1zoHtEMTA=; b=12m+4SoWbvA5Nk7l/2vgaNrYWkvWC2LPMp7ShAO0HUVmqzgMvWNA3UuVwdMUk1s8np8o 2l8+9PzUnglNRD2uY/IRYS7lXrboM0QTr1zcmVEM4ZlFdv4iwk1jPbS26/HGb+Q3XcYBY CSMq4U5qXkKLYwcYlMQSatMRLoYOhHV4/f7J0oBg66VamE9laoWlhbISN09FSY53MVlkJ MCTS/ueMiG4FqFIUoDVNblmItNYH94YnKpppV5eXSETJG4o+5GtjJtnEUGO77K5L8rhuw K8WY1IHBkheW/bFcPQiGPyoeOjBq7fcTzjASX/a6/bNDa2XkZ3VslxP+RMvVF0dvLpw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b="FcFoz3t/"; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4FE7606C900; Mon, 19 Oct 2026 15:50:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=FcFoz3t/ Received: by smtp.kernel.org (Postfix) with ESMTPSA id E15511F000C8; Mon, 19 Oct 2026 15:50:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425044; bh=84kciOlrBZyXkODv4mERfw9QO7WVRKrOSp1zoHtEMTA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=FcFoz3t/L7qTPpklFErAcrar3ZSEdD4kbcutljvZlQbdPPmrbEwganGTEy/F0rOdm w/rxlzz2CqfPX9ErUCt6kVObwU6fHmiXNtqC8HAfibJx6H94LUxiF3Huq71bteGC7n 15j8fJFXNUdfQ5mQJiGLiWIyRhacwAwqLFxZg8bcEDl59VSBXRPurf7y9hgPvIVv2K aLgjlMBP6NFkrHNqL0BOt1NtBV/sWdDCcEg1pBnqz4UON78dFNQIMOv7eR7pW4j6Aa hUPBLCG2K+fw1mYnLuJy59gQYV6YaduyCCjEqS5SsRRjY8aa8zswNYkbbyi3sWpsOp 3sehmCa9MVXpw== Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfauth.ams.internal (Postfix) with ESMTP id E2028198003A; Mon, 19 Oct 2026 11:50:36 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Mon, 19 Oct 2026 11:50:40 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTE+kjgISKw/D5xbtovMvE/5+MXw9OAGtDBfqR9Zk5dZNOqhNWGTcEGO5Lmn7qsaMJ P+2+K5eT4SfShSSEvxMo9+0NhMtUrbV4W6Uy0A3XpuEB672w8089GbqFaxEF0wWgdpl5Mt ZGUplA1E9XDXzNgkO2MrFgMfcNS4u61SieDRP0v46gjH0l2eZ2DL9zme+jYfCIiYlLA1YJ nvbn1HssXPAJFHKdL+OCiEp78jaTWZC2fJHjA4vCnfghQrymIXyUGAHupLMJJEm2bjt0nv kEsuBoXYnW/I4u0Q3SZjAxl07Y4ERxrcHOAeD/N6wYko4+jO11wXgs98kOmjLgJ2rMOJsz 18DGIkmN8sCfzQCV7sEJC2pxZj7RC8eMNg1UeU3okF34a7lwK/lwdys+qzGdOtcOrJ/aTC NyiQb2ojE9CGVMmdY3XRIUnsiIK+vh2Qp4y529vGZS2ky5yIHL3l90Z6GWGm/WOIIfg5oK Xg26A/pynhkh6hF6E3O78PaDtNLzaswXz7QJDAJiscbJuWQaSXJGKaFS8MYUIp92Gs9Bqi sxgAWUCcGgxBqOARvxJ8DYrqTKwa3N7LrLH3zC2I6q2SQjdzl3j6BRuxDXQTf9D/aeV5Nz GTRyl5naroztNsJRmBnZqOQ+gKUFxhFup+Qqs7PCujlpUVIZc9ZejZctTQDA X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:50:35 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 12/18] mm/filemap: don't defer the free of folios of an inode being evicted Date: Mon, 19 Oct 2026 16:49:05 +0100 Message-ID: <20261019154913.1765974-13-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A hot folio keeps PG_rcu_free until it is freed. Deleting a file that was read a lot, or evicting its inode under memory pressure or at umount, then defers the free of all of its hot folios. No lockless reader can be in the page cache of an inode being evicted: the read holds the file open, and the open file holds the inode. Clear the flag when the final truncate removes the folio, which mapping_exiting() tells apart. A truncate of a file that may still be open keeps the deferral. Suggested-by: Linus Torvalds Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/page-flags.h | 5 ++++- mm/filemap.c | 6 ++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a1e52a1f8a52..4916b5713a74 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -608,17 +608,20 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) /* * A lockless lookup that sees PG_rcu_free and then still finds the folio * where it looked may read it until rcu_read_unlock(). Set under a - * reference, cleared only by the free. Not for hugetlb or + * reference. Cleared by the free, or when the folio's inode is evicted: + * no lockless lookup can reach it then. Not for hugetlb or * free_pages_nolock(). */ #ifdef CONFIG_PAGE_RCU_FREE FOLIO_TEST_FLAG(rcu_free, FOLIO_HEAD_PAGE) FOLIO_TEST_SET_FLAG(rcu_free, FOLIO_HEAD_PAGE) + FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE) __FOLIO_CLEAR_FLAG(rcu_free, FOLIO_HEAD_PAGE) #define __PG_RCU_FREE (1UL << PG_rcu_free) #else FOLIO_TEST_FLAG_FALSE(rcu_free) FOLIO_TEST_SET_FLAG_FALSE(rcu_free) + FOLIO_CLEAR_FLAG_NOOP(rcu_free) __FOLIO_CLEAR_FLAG_NOOP(rcu_free) #define __PG_RCU_FREE 0 #endif diff --git a/mm/filemap.c b/mm/filemap.c index b732ecaa78a9..53ec92279977 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -138,6 +138,9 @@ static void page_cache_delete(struct address_space *map= ping, nr =3D folio_nr_pages(folio); =20 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + /* No lockless reader can be in an evicted inode */ + if (folio_test_rcu_free(folio) && mapping_exiting(mapping)) + folio_clear_rcu_free(folio); =20 xas_store(&xas, shadow); xas_init_marks(&xas); @@ -306,6 +309,9 @@ static void page_cache_delete_batch(struct address_spac= e *mapping, } =20 WARN_ON_ONCE(!folio_test_locked(folio)); + /* No lockless reader can be in an evicted inode */ + if (folio_test_rcu_free(folio) && mapping_exiting(mapping)) + folio_clear_rcu_free(folio); =20 folio->mapping =3D NULL; /* Leave folio->index set: truncation lookup relies on it */ --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425052; b=l3pAIsADRaJNSo/qB0RsF5QrHIqIEb1JqJAs4oe2cT0lfqtIEXVWHjIwQZrOyZ8kjoOL ZYdZimg3O9Ck6cXvMn2OZt4NNsON1TsjCEwOonhGe4FyaUQq3b4DiTA0qPK+JB+3rp86A yjuPOUPVNtIKPMPLy90cj6DnJAcR0aHxjbVYoB22bAUtrj2WNqVCG2kd2noHEpFAtc0CV nLR4EZS3TBjNY/hu3rUzi05lbPJqKwgGqy/TzKjT6xMuQnPs9yLj20udNntMa8CArUjBz gsE/Q2noL1NlVB/cVxdcK97f4C8EGu9+Z2WxJWmhZ6kraj0Rsh1AmrO1NglGi9+ZR9Q== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425052; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=1tklyEFR6gzCJ5Mm0LyngFaJuzsqMluVwnmmNtY4Bjo=; b=v73jvIUIZXRVwhOrW4gEYjMw5F0ojJCHmXs9DuIt/s9ByAkzu0pkwo0mk9A7Ndnb2Kdm MNgfXnZNxJQBe/nrCzO8Wj1PE6EBkg9HFPJJMvc1T4IXRI9N9NZdPA3mur7DNmCrT/9sc F/xx9WEVWN376Ar3d3BvZl2KlcRYsM5TKiTp9Eq2BSW9am7xVF5tXNY1++9f2Ie0fWjKj UCb2VinEl8pWV8SEQneZmkxUGhuge0AwKx7YniYDP2N4N4XpwQFL7nEsKXdNThn/iYv9O Z+Yi/S8bqne7N8oBCYU5DSIb6866mmT3pYCSjLD307dMHeNxbBqAcUjEYDbJXtgi57Q== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=hsXIhuWW; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DE7C606AAE8; Mon, 19 Oct 2026 15:50:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=hsXIhuWW Received: by smtp.kernel.org (Postfix) with ESMTPSA id 621B01F000CB; Mon, 19 Oct 2026 15:50:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425051; bh=1tklyEFR6gzCJ5Mm0LyngFaJuzsqMluVwnmmNtY4Bjo=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=hsXIhuWW5cj0PTnvYiQiB6X2pHdnoes0T9srZXVcGrGCLignDLmHgupnUcoAgQ6sL zNEpP2bzdCxqRAzHHta6tZqeBuB/Ybnq7NpmSLKcyxol4ZNb5HZjMVH3Fi6HN2lIa8 RT6361ta0KFbAm1A7d4ohT1mxTZGqmPad99jfEOBeVsQPuMb5cqHMTV/F1S2gNpWn9 36gbsPRKMmo/fkjzpLprlWSRnqpI4UNcBnFS2ZcF2CKMRpSARaxRs88W2d25DKTwpP bYwnV06hz7rhXb4KqmlQ97hCoZQGDTLCKj1DWLvkR47g5T1eFJyfX3ZzR4maur/kXM c/oH0GAq61XAg== Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfauth.ams.internal (Postfix) with ESMTP id 4FBAE1980047; Mon, 19 Oct 2026 11:50:43 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-05.internal (MEProxy); Mon, 19 Oct 2026 11:50:47 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEfcY8Sedpzpe5Eju9n5oayhWKPyUV/WN4ORzd354gnHcUNRyDxnfDrVtF0Alji6y FwZ4Dnqmp2xFA5TRzvzZXvYk9IWIcIACK0GXlQFj7v3uEIoT3YE0GAecfgycvJObnsWgyU omy/lOe9Ul2f7WNFeSYzOhn9Xhyn3eRZ1/u1NlpfN8iOBPXkq7lLkPZY3eLvZtE4xL/1Z+ 12+yKL0HGegFnoIlCDAqs/SztneWlDn0ZvTxx6bOoXiDCEows+EqK1vqCcT2wFPXdzHPq0 s7OKMwp4WgMV4JhSm776pUXmDzbWh34FJ6orb+IJbBIpKyqJoy5T7ZZsX3UOdwDn/V3zkn ciLaL3SK5aczqSEptHZ258YgE9enzFCIUpka6iwuWD4uOZeMrkBJzpGIh4KYoSTKKtrZvs 4xlScy5Hvhm0g7xX4aojdn8ZeOHImmKyG4/X0NM0BEhBJZwbtMoljvktLExWcSAkjo8TeY M9ZgiJ4yApWw5S7yHaoM4S1qhuWXr76hmoCzlz/14P2STYv2JmsxWXbP8qgk3L+yEEfm4+ 3MerERvzt9ihX2Y+062QX7gG/Ixr1M6EPT/QUzWtcZAGVChJyuJxYkvUbIu8P8Xs/1zdh2 jr3zr60rV2eRRKH1RKef8o1ShHXZiOm4tHdbt5LbIl8o+Sy3moybVLTrAMFg X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:50:41 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 13/18] mm/filemap: store the end of a read in f_ra only when it changes Date: Mon, 19 Oct 2026 16:49:06 +0100 Message-ID: <20261019154913.1765974-14-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" filemap_read() writes the end of every read to file->f_ra.prev_pos. Threads that read through one open file all write that cache line, on every read. The value often stays the same, as for reads that start at the same offset each time. Store it only when it changes. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- mm/filemap.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 53ec92279977..960d53b61d4b 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2941,7 +2941,9 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_i= ter *iter, } while (iov_iter_count(iter) && iocb->ki_pos < isize && !error); =20 file_accessed(filp); - ra->prev_pos =3D last_pos; + /* Readers sharing the file would bounce the line for nothing */ + if (ra->prev_pos !=3D last_pos) + ra->prev_pos =3D last_pos; return already_read ? already_read : error; } EXPORT_SYMBOL_GPL(filemap_read); --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425059; b=arvN3Ibybs2S8fHl3vsqGYZPWF4G9oWu+jGMfFStWHu9dTKwJkQOSQZn9GZEsPpwS2Nr k8vGGnDENNuC5KOUOKl7Yt2kyMlrPrA+T8oujBOBHTqdcqBzRUEmN36zT/gBUGuZY8yQx PM9Hx+hzvyhXp18LSrOGohJXkkFWVhdaFGR8vgzet0ABdJIz3BCBaZm1HwPdb1GSfKve7 EiaDoOeMJTmMy4BcAQua4bErnqOO4RaRnna+0VtCIDrujjdqARMGeNqIA9z0w9NtZxudK dLz0XrofOexqbV2kKYkkPmnZtAI1XV+kVBUJ2QqDZZ5Z1AuhVfhBYjLVpdqW4JqUSbA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425059; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=aHFgwB1QQdeguUBHh1LZeUKmP2EVveCA6ywncN+4824=; b=Sk/VzCdv+3aw5R3k3/73PBwCnUtZy+PDR7pHDj8Ce+o7dGVkKDBrSeVaN2U5pAv8rr3r BEgqmpBj0kFnySnSlYXki9DlSY2kLua7yuq7SWUA73CRp9ERTY8ikNjfX9E0iPFnX+8h5 TvImJ99XSQF+mSoGyXhZcEH8Sf8XMHItdtiJxcS3Dd8Tw1IgSaoR6weoFYVoud9J+md0B 9D94I5Kag3VwrKoieLvVhzMWwKJKb5OdAB/ZYMYn4k2AA1h7SfsIDHQ7ILYo5LQMrYs+Q XCKWxWShPZpnvnnmlGRKetCoiHgaWWAjLxfRLjYiCmyDRBkUuBqKpsql8XMquaIzn/w== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=jp4vxbWn; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7CE48606AAE8; Mon, 19 Oct 2026 15:50:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=jp4vxbWn Received: by smtp.kernel.org (Postfix) with ESMTPSA id DB2211F000C5; Mon, 19 Oct 2026 15:50:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425059; bh=aHFgwB1QQdeguUBHh1LZeUKmP2EVveCA6ywncN+4824=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=jp4vxbWn/yUn9LtCGVPYeakTPX4tsFvUU8YEjhzMI69M/pjH6/p8DSxtcx1C4St4U RXbBFZuCtQB4RFNmOQZsv0T73MCepho+ltYc6dNsTSFubutrPYmOtMw93qvE6HNQ0q ZQpcKYr7r0OTsf46NJoCBJEbReflGrlaQaGiI0pVPQApA3LIDyFISlFYY07coARWX9 8XEXEh/T3YNljIgALCohedAoA6sFnRZc3GDGTGXdBXqFh0MmScqYgA/dbCx8Hq/7np 1KUPsn7FfTMikCHOF+xskyIhBmuuw89MOgRDsUHBJFLB0r7TY75tTL2/PqNKq6iFNk fzQtVF/R7DBqQ== Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfauth.ams.internal (Postfix) with ESMTP id 9D590198003A; Mon, 19 Oct 2026 11:50:49 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Mon, 19 Oct 2026 11:50:53 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEPqwHCv5IVDOfp+gppQpoSvl4iKuiR2RK5Y+mCZyyWO88dcIgYD60QXY0H8fSG9y +i/Fi66FpUhP2Ikf7ONks1QGsKRJt/NoxWDxl5HXxNylv75I6mtGV03osn/+lvccKEBd4W KnTaLXmzxvblXRRW+RSShg/hHxynElJxqWehmOFUYq4U1x7ifoitgX+ne3vp8F3h8AQ/N0 /m2r4CY1m8XbTmLiUOTFaC6q0oQJ3XXcpirfEfr7IPSxVFCOUTfbQW2d5jjoyFzJhvq7h+ uIBkQwMZCXw2y5U40pZhX7XPPqGR7Z3fiyae5UDORfmXvndjR9BDrzyAINzfPzRUg9LHlX k0IkKTk9iNht3cIGRTIZ2lUZssI02DI2+r4CLzf9ALJvkCYS2731J2VpdbOY6W6ssdpZaq q61aKe5jdOQRiHmCzGbTuNUnnXQMa1s1MYT7kWNp2XXUjLgPn0z5+PtISNXTEW1SsTC/56 +/AywSZLo+AmKJldqc/DJBr3AuzCUlzce5s0eOrs/wPTX9AWPQRJ2MMmZjPzhLopP1eoxw 0j3F+Ye0cf+APkDpVznsTTVhYZQ/Y+ausVHhYcDlznuhBSu6aojjhGD6qDJ/85Df8I+5JH JS6uR59Nu8GhniM7xqxNRP1zUGEDjgGFmqzlorj6gQzkBYvu0Z9eOmDJZvKw X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:50:48 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com, "Kiryl Shutsemau (Meta)" Subject: [RFC PATCH 14/18] mm/filemap: read from the page cache without folio references Date: Mon, 19 Oct 2026 16:49:07 +0100 Message-ID: <20261019154913.1765974-15-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Linus Torvalds Small reads of one file from many CPUs all take and drop a reference on the same folio, and its refcount bounces between the CPUs. Before taking a reference, copy what can be copied under rcu_read_lock() from folios with PG_rcu_free, with page faults disabled. A flagged folio is not reused before rcu_read_unlock(), so a folio still in its slot after the flag test is this file's page until then. A folio that leaves the page cache during the copy reads as the file did when it left. Read i_size between the flag and the second look at the slot: a write that extends the file fills the slot before it moves i_size. Leave to the slow path a folio that is not uptodate, needs readahead or dropbehind, is not hot, or has a poisoned page. Leave large folios there too: looking one up for every page is slower than one reference for all of its pages. Copy at most 512 KiB per rcu_read_lock() section, and skip files that never had a folio flagged or that are writably mapped where flush_dcache_folio() is needed. Assisted-by: LLM Signed-off-by: Linus Torvalds [kas: i_size before the second look at the slot, poisoned folio check, large folios left to the slow path, 512 KiB sections, shorter comments and changelog] Signed-off-by: Kiryl Shutsemau (Meta) --- mm/filemap.c | 169 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 960d53b61d4b..55fc471c0489 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2827,6 +2827,165 @@ static void filemap_end_dropbehind_read(struct foli= o *folio) } } =20 +/* + * Whether filemap_read_rcu_batch() can copy from @folio, or has to leave = it + * to the slow path, which holds a reference for readahead, dropbehind and + * folio_mark_accessed(), and deals with poisoned pages. @last_index is t= he + * page the previous copy ended in. + */ +static bool filemap_read_rcu_can_copy(struct folio *folio, pgoff_t index, + pgoff_t last_index) +{ + if (!folio_test_uptodate(folio)) + return false; + if (folio_test_readahead(folio) || folio_test_dropbehind(folio)) + return false; + /* memory_failure() waits only for readers that got past this */ + if (folio_contain_hwpoisoned_page(folio)) + return false; + /* One reference covers all of a large folio's pages */ + if (folio_test_large(folio)) + return false; + + /* + * The slow path does not call folio_mark_accessed() on the folio that + * the previous read ended in, so copying from it skips nothing. The + * folio's size can change without a reference: compare pages instead. + */ + if (index =3D=3D last_index) + return true; + + /* Otherwise only if folio_mark_accessed() would leave it as it is */ + if (lru_gen_enabled()) { + unsigned long all =3D LRU_REFS_FLAGS | BIT(PG_workingset); + + return folio_test_unevictable(folio) || + (READ_ONCE(folio->flags.f) & all) =3D=3D all; + } + return folio_test_referenced(folio) && !folio_test_idle(folio) && + (folio_test_active(folio) || folio_test_unevictable(folio)); +} + +/* + * Return the folio at @pos if it is this file's page until rcu_read_unloc= k(), + * and read i_size into @isize. A folio with PG_rcu_free is freed only af= ter + * an RCU grace period. Without a reference, the folio that xas_load() + * returned can be freed and its memory reused as another folio before the + * flag is tested, so look at the slot again after the flag. + */ +static struct folio *filemap_read_rcu_lookup(struct xa_state *xas, + struct inode *inode, loff_t pos, loff_t *isize) +{ + struct folio *folio; + + xas_set(xas, pos >> PAGE_SHIFT); + folio =3D xas_load(xas); + if (xas_retry(xas, folio) || !folio || xa_is_value(folio)) + return NULL; + + /* Not folio_test_rcu_free(): it may not be a folio any more */ + if (!(READ_ONCE(folio->flags.f) & __PG_RCU_FREE)) + return NULL; + /* Read i_size and slot after the flag: it implies uptodate */ + smp_rmb(); + *isize =3D i_size_read(inode); + if (pos >=3D *isize) + return NULL; + /* i_size_read() is an acquire; a write fills the slot first */ + if (xas_reload(xas) !=3D folio) + return NULL; + /* Read the rest of the folio only once the slot holds it */ + smp_rmb(); + return folio; +} + +/* Bytes one RCU read section copies at most: it holds up grace periods */ +#define FILEMAP_RCU_READ_MAX SZ_512K + +/* + * Copy from folios with PG_rcu_free without taking references, at most + * FILEMAP_RCU_READ_MAX bytes under one rcu_read_lock(). Return true if t= hat + * much was copied and the read is not done. *last_index is the page the + * previous copy ended in, set as the slow path sets last_pos. + */ +static bool filemap_read_rcu_batch(struct kiocb *iocb, struct iov_iter *it= er, + pgoff_t *last_index, size_t *copied) +{ + struct address_space *mapping =3D iocb->ki_filp->f_mapping; + struct inode *inode =3D mapping->host; + XA_STATE(xas, &mapping->i_pages, 0); + size_t budget =3D FILEMAP_RCU_READ_MAX; + bool more =3D false; + + /* flush_dcache_folio() is not known to be safe without a reference */ + if (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE && + mapping_writably_mapped(mapping)) + return false; + + /* A user fault cannot sleep in the RCU read section: it ends the copy */ + rcu_read_lock(); + pagefault_disable(); + while (iov_iter_count(iter)) { + loff_t pos =3D iocb->ki_pos; + pgoff_t index =3D pos >> PAGE_SHIFT; + size_t offset =3D offset_in_page(pos); + struct folio *folio; + size_t bytes, n; + loff_t isize; + + if (!budget) { + more =3D true; + break; + } + folio =3D filemap_read_rcu_lookup(&xas, inode, pos, &isize); + if (!folio) + break; + if (!filemap_read_rcu_can_copy(folio, index, *last_index)) + break; + + bytes =3D min_t(loff_t, isize - pos, iov_iter_count(iter)); + bytes =3D min3(bytes, budget, PAGE_SIZE - offset); + /* Not _nofault(): its usercopy check reads the folio's order */ + n =3D copy_page_to_iter(folio_page(folio, index - folio->index), + offset, bytes, iter); + *copied +=3D n; + iocb->ki_pos +=3D n; + budget -=3D n; + if (n) + *last_index =3D (pos + n - 1) >> PAGE_SHIFT; + if (n < bytes) + break; + } + pagefault_enable(); + rcu_read_unlock(); + + return more; +} + +/* + * Copy from the page cache without folio references, in batches, and + * return how much was copied. The slow path reads the rest. + */ +static size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter, + loff_t last_pos) +{ + /* The page the previous read ended in; ULONG_MAX if none */ + pgoff_t last_index =3D (last_pos - 1) >> PAGE_SHIFT; + size_t copied =3D 0; + + /* Not on an inode that never had a folio flagged */ + if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE) || + !mapping_rcu_read(iocb->ki_filp->f_mapping)) + return 0; + + while (filemap_read_rcu_batch(iocb, iter, &last_index, &copied)) { + if (fatal_signal_pending(current)) + break; + cond_resched(); + } + return copied; +} + /** * filemap_read - Read data from the page cache. * @iocb: The iocb to read. @@ -2852,6 +3011,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_i= ter *iter, bool writably_mapped; loff_t isize, end_offset; loff_t last_pos =3D ra->prev_pos; + size_t fast_read; =20 if (unlikely(iocb->ki_pos < 0)) return -EINVAL; @@ -2863,6 +3023,14 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_= iter *iter, iov_iter_truncate(iter, inode->i_sb->s_maxbytes - iocb->ki_pos); folio_batch_init(&fbatch); =20 + fast_read =3D filemap_read_rcu(iocb, iter, last_pos); + if (fast_read) { + already_read +=3D fast_read; + last_pos =3D iocb->ki_pos; + if (!iov_iter_count(iter)) + goto out; + } + do { cond_resched(); =20 @@ -2940,6 +3108,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_i= ter *iter, folio_batch_init(&fbatch); } while (iov_iter_count(iter) && iocb->ki_pos < isize && !error); =20 +out: file_accessed(filp); /* Readers sharing the file would bounce the line for nothing */ if (ra->prev_pos !=3D last_pos) --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425065; b=QrvWgR093xf6h5uN0qvHO0dLDtuytYnIPdGkX1Js1GSJpMp8nGUy0W/B5OcFrgpJ159E NWGCVZ1tW4rQxMp0JBSdD3WPLNC4qGcAjaRsDhSirNr8tCVn7KE3tMTsEL1+SeUmBHYG0 HIScsUl/ZpJkstz46gy10q6wetZzmKdLg0VLM4nWLJofb/Hu9beZBs8E1w+j4uQ/zZnCY VY7n8O+GhbAjkavKJXktK5YTGQBDXvnz5E5lU5WZwyDjm0NCN+AaoHCFhrVTC4j1I+1Ga SZ0POI5+oSkPx8S//oh7lq/XBxSlT4wNJm1P8lecLN7Uy6tDlxGsU7NZphCPkpNCl1g== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425065; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=U8qRbxVMl9WjIylP4uF+PXHiNMFQO2d35UvAtzhx6sE=; b=OpeJFf0cEtCJKAcs61WYGE0PfIbRyh6w7Wjkt88ObS7bCuptHbH6/DGPVWYGpg2IPAkY 4uXZqJnzzoGGe/sb5a6HjM1pF80QOSZUYsEw1SD9VOZYLHqndHOh9IfJxzdLLbhwk/dAZ LQAvrR513rwAoKgFz5nhQlZMAqvkDSMsIrlt0q2oiQRC5XXsjYYTDLpsCYPmI1cXym+X2 RBbgY09OankZvzu8OwlC2A/2Jq0OI4cduXvDYm6y+WW7yRQdqVcFdSCJeAZuo5VfVNIao tNoZWYu53cmxle7cv5LsvuHBoWYz7E21kLpKKNYEGe8Sj6InA69YUPYvydfFcBqgBpQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=Nxt2g2oi; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 889B9606AAC6 for ; Mon, 19 Oct 2026 15:51:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=Nxt2g2oi Received: by smtp.kernel.org (Postfix) with ESMTPSA id 719EE1F000C7; Mon, 19 Oct 2026 15:51:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425065; bh=U8qRbxVMl9WjIylP4uF+PXHiNMFQO2d35UvAtzhx6sE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Nxt2g2oiPGhnjIWMFd6cbSzerN9RvuAWTochm/XIy+E3YgUf1ZKNc8ln+V1stW6pX ZcRt0aztuO/x4TKZwVt8iBbJ6Kxejpr3xo3MDRJ0jKzal9WgbiB9Uc/un90xx3FDrT pPcpiOK/8jZsztT3XHl9wl3XRRn1bEhRNSK9HC4iHctRg9PYH6/IUpXxQl5WfpeGZj 4YvYG/Ro99+XS/8h9VgRiIe7rpith9RJsbUBI/xyfAdcBbw61aav957VdLN35pfLkH bvsk6kTZEBfE15MbvrhHYoAMi1cgOHa1QNnfeOHRCrJ1olgyKBF2Jqf7b5YssNzsN+ OVEYF68UZGAOg== Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfauth.ams.internal (Postfix) with ESMTP id 1D1F2198004A; Mon, 19 Oct 2026 11:50:56 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Mon, 19 Oct 2026 11:51:01 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEKM7cQ34PANJkOaDpnT+IXyH0pLg7IoM6i34Jbx05genX6Ndq2TdcGNb9654u2H8 41WED7y/zjj5LaonH0VFTkPtnxIjhuRVEkvuk8kM9rm/ofB3Z9Tg4QxquxaJFHbDWQqeu/ Vblnly/EuZwo3Wz9CpGJqqgTrStLmT9zjqH9hVTJAPs00+uZifgMXlnluxpV6wuH9uu7MT BObPpnz8l78pdcpT4Vufdwi//iHA9W9CmvvMWThM1WI/4eq4v7bHyLxAHLcqSlwPUMMG0k NFb5bNvmq2vpdgPX6RraAkdwYnZfp8UPDvRR9XmyDlbwFnPxU3uIs+BDzqJ7Yv2QnX71lk SqTihz8KrnAWYbCEV29gvySR44BJOhM620UDi7RgBFyzLt2FX77tl1bIEBd8zUsl2vqnNF nFaX7oJnw9jzl4Im4TmEN0FGPZ+hsD/Hpjbxq0aPtS9IEAD2eUm0nsjguUdSMoJhg0gvJg WclWlsxxO7a/TyCOSLZLOk2ezO+K1Pjj6gnvCHiC4xJ7A/NLQkpZIIyLZ6fIYskJfVezB1 JTKjUjaYrzuI6H1hc1SKjxsKl3ECQXuoGoOaFfx3Txr+D8UvGA/dyqgPe86Zmx4yohNOZO g36OTZ7vqw89oUDxbVcG9dYAawcjetiW5em9S+Vas4yhNDrb4krv/lDYi4Bw X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:50:55 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 15/18] mm/filemap: read large folios without folio references Date: Mon, 19 Oct 2026 16:49:08 +0100 Message-ID: <20261019154913.1765974-16-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A large folio is in all the slots of its range or in none. Once the slot of the last page to copy holds it too, the pages in between are this folio's: copy them after one lookup. The folio's size is read without a reference and only picks the page to check; a split after the check copies PG_rcu_free to the new folios. Copy page by page with _copy_to_iter(): the hardened usercopy check of copy_to_iter() reads the folio's order, which a split can change. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- mm/filemap.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 57 insertions(+), 7 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 55fc471c0489..ace6b1fd6b9e 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2843,9 +2843,6 @@ static bool filemap_read_rcu_can_copy(struct folio *f= olio, pgoff_t index, /* memory_failure() waits only for readers that got past this */ if (folio_contain_hwpoisoned_page(folio)) return false; - /* One reference covers all of a large folio's pages */ - if (folio_test_large(folio)) - return false; =20 /* * The slow path does not call folio_mark_accessed() on the folio that @@ -2899,6 +2896,60 @@ static struct folio *filemap_read_rcu_lookup(struct = xa_state *xas, return folio; } =20 +/* + * How much of @bytes, starting @offset into page @index, can be copied fr= om + * @folio. A folio is in all the slots of its range or in none: when the = slot + * of the last page holds it too, so do the slots in between, and the pages + * are this folio's. + */ +static size_t filemap_read_rcu_size(struct xa_state *xas, struct folio *fo= lio, + pgoff_t index, size_t offset, size_t bytes) +{ + pgoff_t end =3D index + DIV_ROUND_UP(offset + bytes, PAGE_SIZE); + pgoff_t next; + + if (end - index > 1 && folio_test_large(folio)) { + /* Read without a reference: it only picks the page to check */ + next =3D folio_next_index(folio); + if (next > index && next < end) + end =3D next; + xas_set(xas, end - 1); + /* A split after this copies PG_rcu_free to the new folios */ + if (xas_load(xas) =3D=3D folio) + return min(bytes, (end - index) * PAGE_SIZE - offset); + } + return min(bytes, PAGE_SIZE - offset); +} + +/* + * Copy @bytes from @folio, starting @offset into its page at @index. Page + * by page, as copy_page_to_iter() does, without its checks for each call. + * Not copy_to_iter(): its usercopy check reads the folio's order, which a + * split can change. + */ +static size_t filemap_read_rcu_copy(struct folio *folio, pgoff_t index, + size_t offset, size_t bytes, struct iov_iter *iter) +{ + struct page *page =3D folio_page(folio, index - folio->index); + size_t n =3D 0; + + while (n < bytes) { + size_t len =3D min(bytes - n, PAGE_SIZE - offset); + size_t done; + void *kaddr; + + kaddr =3D kmap_local_page(page); + done =3D _copy_to_iter(kaddr + offset, len, iter); + kunmap_local(kaddr); + n +=3D done; + if (done < len) + break; + offset =3D 0; + page++; + } + return n; +} + /* Bytes one RCU read section copies at most: it holds up grace periods */ #define FILEMAP_RCU_READ_MAX SZ_512K =20 @@ -2944,10 +2995,9 @@ static bool filemap_read_rcu_batch(struct kiocb *ioc= b, struct iov_iter *iter, break; =20 bytes =3D min_t(loff_t, isize - pos, iov_iter_count(iter)); - bytes =3D min3(bytes, budget, PAGE_SIZE - offset); - /* Not _nofault(): its usercopy check reads the folio's order */ - n =3D copy_page_to_iter(folio_page(folio, index - folio->index), - offset, bytes, iter); + bytes =3D filemap_read_rcu_size(&xas, folio, index, offset, + min(bytes, budget)); + n =3D filemap_read_rcu_copy(folio, index, offset, bytes, iter); *copied +=3D n; iocb->ki_pos +=3D n; budget -=3D n; --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425070; b=dP17+Y6gXAyewgJ7brDAGTlcd7HZRVPXSuuyd+znS2I3mtaKk+PSCVSd8bkQ2pBJkDby /9IuB7HU8Y3zBKezgPMhXlj6u5ofsk3XLJM3NmTvxYdKefPI2TY/trH6dMZ1VTnmB2CaN N4jgcwwaqBINSj09AMlspvQxNWCXlbY4D+UcXdJQdu2w9Es//KPf89Nd4cN/cNY0z0m4+ iNiKib7EfujMlpkGS+8QTYTwKdZcPNYWmHz5t/DM9br/cpwCShxFRsFtZMUFjvJFK/faI a9Dy7/ECAalmC1R+cP1dQ3obA2v1sV5RNHoc/05gAk3qxPipMjx/lvPA19rKpIs9JZw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425070; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=nzf5AOZecE/pgMusAub/Smm2ReUG3xW+PWHZuYHUyt4=; b=ePBeMUiMdfcu9Vvjd98YkXuIPZ/V4eo4HX+cKJTBsBWeGxj+ZNsG43uXyaM/NYagDuDK /yluGBQZi0DrwVU+LrQp3QDlqIrFkdZLa0sC0hWQ3p+o5F7e/ts+OASJj905Yp+h9p5Vs nSnxIUrKl1lU4IfR5wGBH+ZfU68W8yiNRH5x7uaAS931H+8PV9rWBoZx/RQDHNOmvdXxE dl6YK1naPV2TuSc7ty32gSXo326LhQ/jBb8wD9h+uXFFwZ6dfxOGXR2+qVTLtk87rSq6t seGD5nZqLK4uiFBESZfTHq/Dv6V30gKIOnUW2/l2o7NIaI847MBGo7GxtM2+Mjg8IrA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=P76XqEwv; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55FE5606C911 for ; Mon, 19 Oct 2026 15:51:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=P76XqEwv Received: by smtp.kernel.org (Postfix) with ESMTPSA id A78451F000CF; Mon, 19 Oct 2026 15:51:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425070; bh=nzf5AOZecE/pgMusAub/Smm2ReUG3xW+PWHZuYHUyt4=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=P76XqEwvYvtRiChGcMWUkKRntAV/ce5/WAbDoQUHDxuyq/aB6KdTUrY78u0cXtIP6 epFp7w4oJ6TMIyqBYj5OwvupZvIxx0sVNv4byNU0buBm+RiKpLfEHmKJSfAOG/Wvm8 dWNlpOHY9hcSDqH8t/aAoNR9F+d1gHDYB1HIylmtVlV/y+P+pwexEONeAlTgLhIeqX n44jzEdIsOgrDlvXmwtwKaCZp3TblM1Qe2Ptl0PKcvwZQFDv5pQyvSvCvTHuLzqDMN yJOUDq31Ztf8n9aWV5p9MFOraPQhZN/taiRlQxrki7CSdHLaBr9ku45PVpfKHgQLOD mT9xN3yE6tpsA== Received: from phl-compute-03.internal (phl-compute-03.internal [10.202.2.43]) by mailfauth.ams.internal (Postfix) with ESMTP id 9F85D198003A; Mon, 19 Oct 2026 11:51:03 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Mon, 19 Oct 2026 11:51:07 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTESqR4vElUNfwe4Wb1kabMg9MdYUtHyNIvlHi5jfGRsEd0tjiYfzpqh5FemuLlMuA +QzWL/iolJzzDzQk8vzuoT2/u2xyL/YZG/CihU2eaSQe3uAw7YKu/Z/hQfgMIJpEQrBQkK hH3KJVpCmjg+gNXnnOIkdBMKpJa1pvTLqzuojrt68Uovke10kpOzVhZrNEf3d58NDgoR+6 GBwzdAiHoLFFDGJo3/r+XBHeUTBbw+IeFaYdgC0j5JEQlKAOMB5I7tBUfVMuw5hMfUnXf2 vE1DwOk7N/x2FbtU++Y3QX/1ODQhD5sTVGp3EMvK4jUTItTZrn4RBPsK+PhnbeHfl8JcpJ c2LUHcVVlnTmrxA40olgviWRl9T+26cUEoz3aZJIdT1EfpFsU+0thKqzxFC9q6r9alkT7P UjijYajoIbQcUwCvES00HGkFx6tcOfQlzfxTtLaIAWuifAZVmXruOnjSB2KfmIGd1Tp9h8 pT7XR7J6tU0+2PVC3G7YK2c6UKb8RdB87PPzk5qtMsP2YH1aNuaOcR9AoL/bstgEKQzsbg t6Ubf6j8Uiwe5o955Rjad4bmK7153G+aKl6kgw9QxprBMlsBHQpSij80iErZkMlHPp5FCS lHriB7oqc0/oTiW+ZSU9MPqkYvzaUVlCHKrJhXKPuBMUw+OJ0rG24eS8p+yg X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:51:02 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 16/18] mm/shmem: count accesses to a folio the way filemap_read() does Date: Mon, 19 Oct 2026 16:49:09 +0100 Message-ID: <20261019154913.1765974-17-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" shmem_file_read_iter() marks a folio accessed only when a read starts at the beginning of the folio. Small reads that start anywhere else never count, so a folio that is read over and over at one offset ages as if it were never read. Count accesses with filemap_read_accessed(), as filemap_read() does: a read marks the folio accessed unless it goes on in the folio that the previous read through the file ended in. Reading the same bytes again through one open file counts once, as for other files, where it used to count each time it started at the beginning of the folio. The end of the previous read is kept in f_ra.prev_pos, as for other files. This also sets PG_rcu_free on the tmpfs folios that reads keep coming back to. On a shmem folio the PG_readahead bit is PG_reclaim, and shmem does no readahead, so do not let that bit keep the flag off. A write marks a tmpfs folio accessed too, so a folio written in several pieces can get the flag on its first read. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- mm/filemap.c | 9 +++++---- mm/internal.h | 1 + mm/shmem.c | 11 ++++++----- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index ace6b1fd6b9e..4b5b7ba78f39 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2778,8 +2778,10 @@ static void filemap_mark_hot(struct folio *folio) if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) return; if (folio_test_rcu_free(folio) || !folio_test_uptodate(folio) || - folio_test_readahead(folio) || folio_test_dropbehind(folio) || - folio_test_unevictable(folio)) + folio_test_dropbehind(folio) || folio_test_unevictable(folio)) + return; + /* shmem does no readahead: there the bit is PG_reclaim */ + if (folio_test_readahead(folio) && !folio_test_swapbacked(folio)) return; if (folio_test_set_rcu_free(folio)) return; @@ -2802,8 +2804,7 @@ static void filemap_mark_hot(struct folio *folio) * LRU records, or when this open file goes back to data that it has just * read. */ -static void filemap_read_accessed(struct folio *folio, loff_t pos, - loff_t last_pos) +void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_po= s) { bool hot; =20 diff --git a/mm/internal.h b/mm/internal.h index 38b1165212c9..18391dc50418 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1248,6 +1248,7 @@ struct migration_target_control { */ size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, struct folio *folio, loff_t fpos, size_t size); +void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_po= s); =20 static inline bool vma_is_single_threaded_private(struct vm_area_struct *v= ma) { diff --git a/mm/shmem.c b/mm/shmem.c index 848316eaa7f4..43ddb50974a5 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3309,6 +3309,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *ioc= b, struct iov_iter *to) struct file *file =3D iocb->ki_filp; struct inode *inode =3D file_inode(file); struct address_space *mapping =3D inode->i_mapping; + loff_t last_pos =3D file->f_ra.prev_pos; pgoff_t index; unsigned long offset; int error =3D 0; @@ -3378,11 +3379,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *io= cb, struct iov_iter *to) flush_dcache_page(page); } =20 - /* - * Mark the folio accessed if we read the beginning. - */ - if (!offset) - folio_mark_accessed(folio); + filemap_read_accessed(folio, iocb->ki_pos, last_pos); /* * Ok, we have the page, and it's up-to-date, so * now we can copy it to user space... @@ -3410,6 +3407,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *ioc= b, struct iov_iter *to) =20 retval +=3D ret; iocb->ki_pos +=3D ret; + last_pos =3D iocb->ki_pos; =20 if (!iov_iter_count(to)) break; @@ -3421,6 +3419,9 @@ static ssize_t shmem_file_read_iter(struct kiocb *ioc= b, struct iov_iter *to) } =20 file_accessed(file); + /* Readers sharing the file would bounce the line for nothing */ + if (file->f_ra.prev_pos !=3D last_pos) + file->f_ra.prev_pos =3D last_pos; return retval ? retval : error; } =20 --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425078; b=eYQbH4/IzTjTu7Nk42PMYiwF9K4I4V5uZGRBZDxCVs3zpM9Z2vf0dZp9oilwleyEBpPc ivxlmUCVQd29RDxqlDlJoaC6V47XBpGRB35gdUr/JH2A/XccJFcwgUh2SEeungJ+EX1k9 7x2uSuk6uU3fMViTURA9+fxWUHjJOzufJGW3fQIabk6FX8VT1seAr4EpQ4vkqBXDyR6K3 HokhwW/NT1x4VxQIXWHGt4GCoDGb6YjfsUDHOcA92qwQq0e3NXAJ5EJ2VBeTyMv9sbZmd kMd2h5Vxy5w0znogaZDetRB+1O5ZOyG+qIbpW+MIpBLgxBVX9D/evWtCMMK43htw7tw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425078; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=RERAHv0f1FeePlwah3UOiQF6OoFt+Qe2xFbGw7JJygo=; b=JgM9pjKuRNTR5oEQWClMKEvnCmleHx7HpeGD+qfGL3fRrS30Mdy+Ar85U/KgPss08HRC nLuufnY/KzfWYFTDyaZ7Haj8CcGXig5yoiQW53T/vkYsNjSSuX1fUFMRbQiZotGkVGzH1 jVvBBTrtDl15Hb3WJ/yGf1xeJg/jgIFjykKEPYmJvNKBf6fYiqg291BcC5tm97TQn6JS3 A+H6l8kIVcXFD4ISgmk9I10ZZgQlHDGKmc4iTQmVCDR3bjkbyIR/2Y5LqH0eUby2hsaWr QVvYsVx2/G6O/PGE5BEAPbsp2ZW/j2XF2LQ+qY+tLCYzJijHWvvz9054uM8G4VOSJDA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=c46yUrpm; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B31360ACCD4 for ; Mon, 19 Oct 2026 15:51:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=c46yUrpm Received: by smtp.kernel.org (Postfix) with ESMTPSA id 724C81F000C6; Mon, 19 Oct 2026 15:51:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425078; bh=RERAHv0f1FeePlwah3UOiQF6OoFt+Qe2xFbGw7JJygo=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=c46yUrpmiNDYms68gjJ5VPLSDDYcCdSjcpdtH2gabOzII827HbCJd9nz4yl+oevfH MbWgZfU830+ekEGLyyGZh1q/SrxcRvcn1mwcpSkiotcotcZh2QF2fu99DZj25mk8Cy SMuZmq4kuDqpHAdkva2KvMl4CFCV85eoPDHEvtJN3AJUJvboJ6AacePQkBdf2TF/SX 2RfCvFbPk+H53YY6kJGbDtDvCP2r2ZzZCfJ6y42EyWnwZI/b3Lg0TgRUMeTP8bpqlK TI8CxvCMEBW+ikRL/p87DwZ079jeka/YWLPfsEdaFS/pP88j7pSmdzKIIyfBXtGXgX qhIB+YyDNs/yg== Received: from phl-compute-12.internal (phl-compute-12.internal [10.202.2.52]) by mailfauth.ams.internal (Postfix) with ESMTP id 04F7D1980047; Mon, 19 Oct 2026 11:51:09 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-12.internal (MEProxy); Mon, 19 Oct 2026 11:51:14 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTGFA7RcREpaZWsLyw0Mr0XLVEScxscceCzrzQCctey9GkLKywZksIPHQL++5Ez1l4 8ZtK4uQQ55DP6KC6KY2Tq4MmUijTRf9wNpB+eBiDYQyyOrNdkCvLnyLRoWmHqIJi9QgC5s a05lQI5Z36HZWoOti16iaeg1yeaoNWQf/xGUk9VWOqtF6qbSZNTrI+y5DzMoADYW/WYpe6 xi0sGgMnsbIx31el0gfeshdIZno0srcJI8ohhNflL/0R7FrINqZHMwN5cb6lyav8SgTLVW L8jKHwxa2hVAuYmQV812wpYXni4+G+KXXKip3711Z5uNAji8ERrYRXaOa0gQ3V/ELjwdMB DY6RU5nX4lcmyZdq4kUuM+SZbr4DMh70PPeGkUpDgnX4Opf3mcxVPJ3HgHr4XJ/Sk5tPy7 8xkCP9KYYUIlOnoYhtjVR3wPGHzGSuWnUPEZbh6vSNUWpd9lY+9iEhBZe7K6FJUPkMT8gq 6W7cpQ3gxgFCXjGoNwpbmy2ccVtcSyoNpgIHVodlO0k+HOn8cFjzadooNdrrWVeM+CdYcC HTGsHHA9WhN6V4Ey1PUvmL6yOlYHbYD6AQv6Dda/+zzOicvimkllzttVSDEHTLijtjuhHY lrffnLRpPewMkqkSl+5MIOwX6fRCrGvVBPPSbjlkDw0USvz1zHci4cNsdx5w X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:51:08 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 17/18] mm/shmem: read without folio references Date: Mon, 19 Oct 2026 16:49:10 +0100 Message-ID: <20261019154913.1765974-18-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" shmem_file_read_iter() takes a reference on every folio it reads, so small reads of one tmpfs file from many CPUs bounce the folio's refcount. Copy with filemap_read_rcu() first, and continue with the existing loop where it stops. Holes, swap entries and fallocated folios that are not uptodate all end the copy. So does a hwpoisoned page, which shmem keeps in the page cache and reports with -EIO. This drops the folio lock as well as the reference. shmem_get_folio() held it only for the lookup, and the copy ran after folio_unlock(), so a read was never atomic against a write or a hole punch. Now a read also does not wait for one that holds the lock, as filemap_read() does not. shmem does no readahead, and on its folios the PG_readahead bit is PG_reclaim, which drop_caches can leave set on dirty tmpfs folios. Do not let it end the copy there. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- mm/filemap.c | 7 ++++--- mm/internal.h | 2 ++ mm/shmem.c | 11 +++++++++-- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 4b5b7ba78f39..cde7bc68b32d 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2837,9 +2837,10 @@ static void filemap_end_dropbehind_read(struct folio= *folio) static bool filemap_read_rcu_can_copy(struct folio *folio, pgoff_t index, pgoff_t last_index) { - if (!folio_test_uptodate(folio)) + if (!folio_test_uptodate(folio) || folio_test_dropbehind(folio)) return false; - if (folio_test_readahead(folio) || folio_test_dropbehind(folio)) + /* shmem does no readahead: there the bit is PG_reclaim */ + if (folio_test_readahead(folio) && !folio_test_swapbacked(folio)) return false; /* memory_failure() waits only for readers that got past this */ if (folio_contain_hwpoisoned_page(folio)) @@ -3017,7 +3018,7 @@ static bool filemap_read_rcu_batch(struct kiocb *iocb= , struct iov_iter *iter, * Copy from the page cache without folio references, in batches, and * return how much was copied. The slow path reads the rest. */ -static size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter, +size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter, loff_t last_pos) { /* The page the previous read ended in; ULONG_MAX if none */ diff --git a/mm/internal.h b/mm/internal.h index 18391dc50418..ae9a4e511c82 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1249,6 +1249,8 @@ struct migration_target_control { size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, struct folio *folio, loff_t fpos, size_t size); void filemap_read_accessed(struct folio *folio, loff_t pos, loff_t last_po= s); +size_t filemap_read_rcu(struct kiocb *iocb, struct iov_iter *iter, + loff_t last_pos); =20 static inline bool vma_is_single_threaded_private(struct vm_area_struct *v= ma) { diff --git a/mm/shmem.c b/mm/shmem.c index 43ddb50974a5..e6c18e745f81 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3313,7 +3313,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *io= cb, struct iov_iter *to) pgoff_t index; unsigned long offset; int error =3D 0; - ssize_t retval =3D 0; + ssize_t retval; + + retval =3D filemap_read_rcu(iocb, to, last_pos); + if (retval) { + last_pos =3D iocb->ki_pos; + if (!iov_iter_count(to)) + goto out; + } =20 for (;;) { struct folio *folio =3D NULL; @@ -3417,7 +3424,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *ioc= b, struct iov_iter *to) } cond_resched(); } - +out: file_accessed(file); /* Readers sharing the file would bounce the line for nothing */ if (file->f_ra.prev_pos !=3D last_pos) --=20 2.54.0 From nobody Mon Oct 19 22:56:21 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792425085; b=qVC5DSccWjOeWjleTpZ6n2ceGuKPEdDuf4b77wAp+eXS1kvvNoKfGWe+l/Q7eWS84ONp XqRwyiWEL3ctE8zPgSfyHdOvS4CNJZc5DpXP8hwCxwGTshod+r6/r7dit1E/ZYQaBTrgF gIV5OxV9FTfxm1aHytQvXB1WkOfGfLYYvshdetETi2OZ8Diba9oHlBCWlRPVUylKLP6Tg hFWRXvWC/cciJ4LBmhut8PH3e7v6mxv26rC1+qtxvHtScstdfBzGnA99uA9z/8VFNgDUk 6R8W5aIeiKGBz03nyi05Cc8UvMA7mnmKx0+v/Eyx4+okASaAADyP+g8MJoeySTnleDw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792425085; h=Received:DKIM-Signature:Received:Received:X-ME-Sender:X-ME-Received: X-ME-Proxy-Cause:X-ME-Proxy:Feedback-ID:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding; bh=V7LU7B522G/Nt7R0FiWPQqjn9CdYZiUxBthyqwKOD9Q=; b=P3Qr+XFSW0MgBicKrhGpmfE4oXjTV05D087GhH90l0HF9/PEZ8btZ6BGdjAxYgQUd0s1 AaYhadlDv8MAGn6NbcgkxFaxcQ2sxhnVNY2DJZ6+mPtul5akMosb7Tj8bFkJ/PqLjTU2n JOkJpSUaiPXRnSf95+30zMfDBT1B/3BSCx30wV60LRI/uIZJy40spouX/ozNnbBKqlg9Z Vnlro+lNep4pKxqzwuzNLsVt5tQw1GTsNLun8zeRXSzG96w3DPx+kgPNivWKbuqCef87x FJBy4wqjvrPjgPTLM3NrvlmCOZUAYkrymkuKqqgrlxm1OQPadsvl4KKXDBBIFoYStFA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=LJXvjlOi; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76428606C904 for ; Mon, 19 Oct 2026 15:51:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20260515 header.b=LJXvjlOi Received: by smtp.kernel.org (Postfix) with ESMTPSA id C066E1F000C9; Mon, 19 Oct 2026 15:51:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1792425085; bh=V7LU7B522G/Nt7R0FiWPQqjn9CdYZiUxBthyqwKOD9Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=LJXvjlOiGVIL/rET/7zf9cAWFEk8SSq0w7A8f3zdNlf+9l6L5vcKK0xaSe86HZn8r J8vpLTL6Mdy+TR6egVN44MM8sw9GoqrDV6ELxDd9D7Q+JHxxFfZgDYJJc+OEEzLkzU yHlvsgD55GZczfXBkfcGbfpZCT8FBlkQckYjMBJyCM/orIYzVk6+zT4HUXKr2IuDYy VhrP+5r1lXWYa+ST+zC5r1MtoPEYWft2LxmKdwxuD6CFgKyqf9J42g1Kw3DLg3j+4N C/lCCGeDgjbNYk6wE9rnxZT2yPHa1wsq/qJlA9SOqLZzpEFmThA7bumeDSEDGlo8jR U+IGRtQqgUPSg== Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfauth.ams.internal (Postfix) with ESMTP id A1B74198003A; Mon, 19 Oct 2026 11:51:16 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-05.internal (MEProxy); Mon, 19 Oct 2026 11:51:20 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTETM5HXVFLzPUxauAkBxER+Z+lTRQkwgIjXhOFN5THGlO0HVgKRB8Wd4BSXaHA5Bq 7Hki4AkN29JlsW7Oz4whntE9FU2sRh3n5MV6IVyinOcMZqrY6k/BF1c/BxQBvNlfSnbHPl QW/SZQekiA5aLb3GJIUgrXyjACrHk/syVIVjTZfD7jaDpyMg1O/dYDf4I6wLOAEwx8xESw vWlPZzevy9W/Bx8Vy1OTZuF/dkabbhGDCVd6eGxAa35wXpSJUVy0LPc7oPWN1fg8RR92+D ECpc3fgxZvfIHD4VEls/NZQ+K0gffqxM1pWJS3eaXhIg/AZa9YCFosiagiRdAJgbrCIzoM o7gldHTvIc7eDhtIlvlXL3C53Ed6FccoFxHkqxXLx6DesQwcRIq2OCKsz71WAjXT1lm5TM Kg1S+IlqYIxSJe9HD2GKza9Xs/DT9IRo/iXi4/lAIHvx7zS5svU1pmtsJwFoiXOifXFOJS c8+Oeh3NCwMTZdMLywvQlmAwW/Y7WDCES6B0jmV7pMkS4HcOooYUKfO5TR9lAZ5VoW9a8r /Sd6cK2NtDUvPD7I9Une5gCLCi06S4ZQdRtRBV9yijBm9YZ31pf3G36MsRdGWua/hTa0oz IXvg1wkV9fllJ5EjieDzrGee0JsCZ2XwYMKjT3W5rPPzBpPVPhNKsKgec/1w X-ME-Proxy: Feedback-ID: i10464835:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Oct 2026 11:51:15 -0400 (EDT) From: "Kiryl Shutsemau (Meta)" To: Andrew Morton , Linus Torvalds , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Lorenzo Stoakes , Jan Kara , Hugh Dickins , Christian Brauner , Alexander Viro Cc: "Kiryl Shutsemau (Meta)" , Miklos Szeredi , Miaohe Lin , Naoya Horiguchi , Oscar Salvador , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Johannes Weiner , Zi Yan , Baolin Wang , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Kairui Song , Qi Zheng , Shakeel Butt , Axel Rasmussen , Yuanchu Xie , Wei Xu , Baoquan He , "Paul E . McKenney" , Pedro Falcato , Harry Yoo , Luis Chamberlain , Ilya Gladyshev , Jaegeuk Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, fuse-devel@lists.linux.dev, kernel-team@meta.com Subject: [RFC PATCH 18/18] mm/filemap: add a debugfs switch for lockless page cache reads Date: Mon, 19 Oct 2026 16:49:11 +0100 Message-ID: <20261019154913.1765974-19-kas@kernel.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20261019154913.1765974-1-kas@kernel.org> References: <20261019154913.1765974-1-kas@kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Reading hot folios without a reference defers the free of their memory by an RCU grace period. A workload that loses on that trade, or a bug in the lockless read, has no way to turn it off: CONFIG_PAGE_RCU_FREE is on in every 64-bit kernel. Add lockless_page_cache_read to debugfs. Writing 0 stops filemap_read() and shmem_file_read_iter() from setting PG_rcu_free and from copying without a reference. A static key keeps the test off the read path. debugfs, not a sysctl: the switch is not ABI and can go once the reads have proven themselves. Folios that have the flag keep it, and their free stays deferred: a read that tested the key before the switch may still be copying from them. Assisted-by: LLM Signed-off-by: Kiryl Shutsemau (Meta) --- mm/Kconfig | 3 +++ mm/filemap.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 51 insertions(+), 3 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 5af00db23f90..1ad1cb7969e7 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -730,6 +730,9 @@ config PAGE_RCU_FREE 64-bit only: it takes a page flag, and those are scarce on 32-bit. =20 + Writing 0 to lockless_page_cache_read in debugfs turns these + reads off. + config PHYS_ADDR_T_64BIT def_bool 64BIT =20 diff --git a/mm/filemap.c b/mm/filemap.c index cde7bc68b32d..7021ed2ab6c9 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -48,6 +48,7 @@ #include #include #include +#include #include =20 #include @@ -2765,6 +2766,50 @@ static inline bool pos_same_folio(loff_t pos1, loff_= t pos2, struct folio *folio) return (pos1 >> shift =3D=3D pos2 >> shift); } =20 +#ifdef CONFIG_PAGE_RCU_FREE +/* Writing 0 to lockless_page_cache_read in debugfs turns the reads off */ +static DEFINE_STATIC_KEY_TRUE(lockless_read_key); + +static bool lockless_read_enabled(void) +{ + return static_branch_likely(&lockless_read_key); +} + +#ifdef CONFIG_DEBUG_FS +static int lockless_read_get(void *data, u64 *val) +{ + *val =3D static_key_enabled(&lockless_read_key); + return 0; +} + +static int lockless_read_set(void *data, u64 val) +{ + if (val > 1) + return -EINVAL; + if (val) + static_branch_enable(&lockless_read_key); + else + static_branch_disable(&lockless_read_key); + return 0; +} +DEFINE_DEBUGFS_ATTRIBUTE(lockless_read_fops, lockless_read_get, + lockless_read_set, "%llu\n"); + +static int __init lockless_read_debugfs_init(void) +{ + debugfs_create_file_unsafe("lockless_page_cache_read", 0644, NULL, NULL, + &lockless_read_fops); + return 0; +} +late_initcall(lockless_read_debugfs_init); +#endif +#else +static bool lockless_read_enabled(void) +{ + return false; +} +#endif + /* * Reads keep coming back to @folio: free it only after an RCU grace period * from now on. Not if its next read needs a reference anyway. Not if it @@ -2775,7 +2820,7 @@ static void filemap_mark_hot(struct folio *folio) { struct address_space *mapping; =20 - if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE)) + if (!lockless_read_enabled()) return; if (folio_test_rcu_free(folio) || !folio_test_uptodate(folio) || folio_test_dropbehind(folio) || folio_test_unevictable(folio)) @@ -3025,8 +3070,8 @@ size_t filemap_read_rcu(struct kiocb *iocb, struct io= v_iter *iter, pgoff_t last_index =3D (last_pos - 1) >> PAGE_SHIFT; size_t copied =3D 0; =20 - /* Not on an inode that never had a folio flagged */ - if (!IS_ENABLED(CONFIG_PAGE_RCU_FREE) || + /* Not when switched off, or on an inode that never had a folio flagged */ + if (!lockless_read_enabled() || !mapping_rcu_read(iocb->ki_filp->f_mapping)) return 0; =20 --=20 2.54.0