Small reads of one file from many CPUs all take and drop a reference on
the same folio, and its refcount bounces between the CPUs. This series
lets filemap_read() and shmem copy from the page cache without taking a
reference. It is Linus' RFC [1], reworked: a folio that such a reader
may copy from is freed only after an RCU grace period.
Only the folios that reads keep coming back to get the flag,
PG_rcu_free: those that folio_mark_accessed() finds as accessed as the
LRU records. Streaming reads and data that never gets hot are not
deferred, so the memory held back stays small and most frees are not
touched.
1 PG_rcu_free and the deferred free (64-bit only)
2-3 splice and fuse no longer reuse page cache folios that a
lockless reader may still copy from
4-7 NR_FREE_RCU_PAGES; an expedited grace period when a CPU's
deferred frees pile up and memory runs short; the waiting
pages are freed before the allocator, alloc_contig_range() or
memory offline give up
8 memory_failure() and soft offline deal with such pages
9-11 filemap_read() marks a folio accessed in one place and sets
PG_rcu_free on the folios that reads keep coming back to
12 the folios of an inode being evicted are freed at once
13 f_ra.prev_pos is written only when it changes
14 filemap_read() copies small folios without a reference
15 ... and large folios, with one more lookup per folio
16-17 shmem does the same
18 a debugfs switch to turn the reads off
Numbers: 64-vCPU KVM guest on a Threadripper 7975WX (Zen 4), 2G, no
debug options; base is v7.3-rc7.
With the default mitigations, every read on this machine pays ~400 ns
at the user/kernel boundary (TSA ~170 ns, SRSO ~230 ns): a single thread
does ~1.8M reads/s. With mitigations=off it does 7.1-8.8M, and the
64-thread gain on ext4 grows from 4.5x to 13x. Linus measured 7.4-7.8M
for one thread and 13.3M -> 304M for 64 threads on a Threadripper 3970X
[1]. Rows without "mit=off" use the default mitigations.
16-byte pread() at offset 0, each thread with its own fd, M reads/s:
base this series
ext4, 64 threads 15.0-15.7 68.9-70.1
ext4, 64 threads, mit=off 17.2 217-220
tmpfs, 64 threads 1.6-1.7 71-72
tmpfs, 64 threads, mit=off 1.8-1.9 230-239
XFS, 64 threads 11.3-11.4 20.4-20.5
block dev, 64 threads 14.7 28.8-28.9
1 thread, any of them 1.6-1.9 1.7-2.0
The mit=off rows are from an earlier version with the same small-read
path. Base tmpfs takes the folio lock on every read: 64 threads read
as fast as one. XFS and the block device take i_rwsem shared on every
read (XFS_IOLOCK_SHARED, inode_lock_shared() in blkdev_read_iter()),
and its count bounces in place of the refcount.
64 threads reading 64K at offset 0 of one file, one large folio,
k MiB/s:
base this series
huge tmpfs 96-100 1370-1390
ext4 756-775 1355-1373
Large sequential reads of a 256M file: single-thread within ~1% of
base, 16 threads at base.
Next to a process that touches 1400M in a loop: no OOM kill. A load
that drops 1G of hot data with FADV_DONTNEED every round runs between
9% below and 1% above base, which itself spans 63-75k MiB/s there. A
load that keeps refaulting loses 4-6%, as Linus' RFC does.
Open:
- Patch 12 relies on mapping_exiting() meaning that no file holds the
inode open. f2fs called truncate_inode_pages_final() on open inodes
for atomic writes; a fix is on its way [2].
- Patch 17 drops the folio lock from tmpfs reads. shmem_get_folio()
held it only for the lookup, so a read was never atomic against a
write or a hole punch; now a read also does not wait for one that
holds the lock, as filemap_read() does not. I believe this is safe,
but would like Hugh to double check.
- A deferred folio is uncharged from its memcg before it waits, as every
page cache folio is. For a grace period it counts towards no cgroup.
- A migrated flagged folio is freed after a grace period, so a pageblock
that compaction just emptied is free only then. defrag_mode leans on
free pageblocks; not measured yet.
- The flag is a page flag, 64-bit only. On large configs it costs MGLRU
one refs bit.
- On writably mapped files, architectures that implement
flush_dcache_page() take the slow path.
- filemap_mark_hot() still tests uptodate. Willy pointed out that a
filesystem clearing it under a reader is a filesystem bug; the test is
cheap.
The series is also at:
https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git rcu-free/rfc-v1
[1] https://lore.kernel.org/all/CAHk-=wjLjgfwgSZ1RDk4YdNFcV0ntosMO5AmBoHD76Zb3HFY7Q@mail.gmail.com/
[2] https://lore.kernel.org/all/20261015161714.1586334-1-daeho43@gmail.com/
Kiryl Shutsemau (Meta) (11):
fuse: don't move a folio with PG_rcu_free into a fuse file
mm: count the pages waiting for a grace period
mm: free the pages waiting for a grace period before OOM
mm: free the pages waiting for a grace period before isolation fails
mm/memory-failure: handle folios with PG_rcu_free
mm/filemap: don't defer the free of folios of an inode being evicted
mm/filemap: store the end of a read in f_ra only when it changes
mm/filemap: read large folios without folio references
mm/shmem: count accesses to a folio the way filemap_read() does
mm/shmem: read without folio references
mm/filemap: add a debugfs switch for lockless page cache reads
Linus Torvalds (7):
mm: free PG_rcu_free folios only after an RCU grace period
splice: don't steal page cache folios
mm: expedite the grace period when a CPU has many deferred frees
mm: have folio_mark_accessed() return whether the folio is hot
mm/filemap: mark a folio accessed in one place in filemap_read()
mm/filemap: set PG_rcu_free on the folios that reads keep coming back
to
mm/filemap: read from the page cache without folio references
fs/fuse/dev.c | 4 +
fs/splice.c | 51 -----
include/linux/mmzone.h | 1 +
include/linux/page-flags.h | 24 +++
include/linux/pagemap.h | 12 ++
include/linux/swap.h | 2 +-
include/linux/vm_event_item.h | 2 +
include/trace/events/mmflags.h | 7 +
mm/Kconfig | 15 ++
mm/filemap.c | 337 +++++++++++++++++++++++++++++++--
mm/folio.c | 46 +++--
mm/huge_memory.c | 1 +
mm/internal.h | 3 +
mm/memory-failure.c | 28 +++
mm/memory_hotplug.c | 7 +
mm/page_alloc.c | 284 ++++++++++++++++++++++++++-
mm/page_alloc.h | 1 +
mm/shmem.c | 22 ++-
mm/show_mem.c | 9 +-
mm/vmstat.c | 6 +
20 files changed, 770 insertions(+), 92 deletions(-)
base-commit: c03c1a14507dfaf8b856c2608f9ac216021fc222
--
2.54.0