__access_remote_vm() holds mmap_read_lock() for the whole transfer. On
large machines with large multi-threaded applications, the mmap_lock
is often contended due to mixed accesses from readers and writers, like
mmap and munmap. When a lock holder gets stuck, system monitoring software
can get stuck behind that, resulting in a failure to log that the system
is in trouble.
Take the per-VMA lock in __access_remote_vm() when the access falls
entirely within a single VMA. Fall back to the mmap lock when the access
crosses a VMA boundary, or when the page cannot be reached under the
per-VMA lock: a dropped fault, a userfaultfd VMA, a hard error, or memory
with no struct page that has to go through vma->vm_ops->access().
The bulk of the work is a new gup helper. __access_remote_vm() needs a
single page from a VMA it has already looked up and locked, faulting it in
when necessary, under either lock.
get_user_pages_remote() does not fit: it hard codes the mmap lock and
re-derives the VMA. get_user_page_vma() walks the page tables, faults a
missing page in, and returns it with a reference and the caller's lock
still held.
The per-VMA path also closes a pre-existing gap. A COWed page in a
VM_IO/VM_PFNMAP VMA has a struct page, but the old code routed it to
->access(), where generic_access_phys() ioremaps the PFN and ioremap of RAM
is rejected, so the read came up short.
get_user_page_vma() now returns that page normally. Raw PFNs with no struct
page still reach ->access() under the mmap lock, as before.
The series is arranged as:
1-2: untag the remote address in the VMA lookup without the mmap lock,
on x86 and riscv.
3: rename get_user_page_vma_remote() to get_user_page_lookup_vma().
4: add get_user_page_vma().
5: switch __access_remote_vm() to the per-VMA lock.
6: add selftest coverage.
Changes since v2 [1]. The per-VMA fast path is reworked to build on the GUP
lookup+fault path instead of a custom walker, per David Hildenbrand's
and Lorenzo Stoakes' review, and now faults pages in rather than only
reading resident ones.
- Drop the folio_walk_start(FW_VMA_LOCKED) walk. Add get_user_page_vma()
in mm/gup.c (patch 4): a trimmed __get_user_pages() that walks with
follow_page_mask(), faults in with faultin_page(), and returns the page
with the caller's lock held. Safe under the per-VMA lock because page
tables are RCU-freed, so interrupts need not be disabled. (David)
- Fault pages in on the fast path. v2 fell back to the mmap lock for any
not-present page; now it falls back only on -EAGAIN. (David)
- Turn the v2 READ_ONCE/WRITE_ONCE untag_mask change into an
untagged_addr_remote_unlocked() helper (patch 1), and add the riscv
pointer-masking equivalent (patch 2), which v2 did not cover.
- Read COWed pages in VM_IO/VM_PFNMAP VMAs; raw PFNs still fall back to
->access() under the mmap lock.
- Add the get_user_page_lookup_vma() rename (patch 3) and selftest
coverage for the struct-page and raw-PFN paths (patch 6).
[1] https://lore.kernel.org/all/20260625015053.2445008-1-riel@surriel.com/
Rik van Riel (6):
x86/mm: add untagged_addr_remote_unlocked()
riscv/mm: add untagged_addr_remote_unlocked()
mm: rename get_user_page_vma_remote() to get_user_page_lookup_vma()
mm/gup: add get_user_page_vma() to fault in a page under a held lock
mm: use per-VMA lock in __access_remote_vm() for single-VMA accesses
selftests/mm: cover /proc/pid/mem access to VM_PFNMAP memory
arch/arm64/kernel/mte.c | 2 +-
arch/riscv/include/asm/mmu_context.h | 4 +-
arch/riscv/include/asm/uaccess.h | 10 +-
arch/riscv/kernel/process.c | 12 +-
arch/x86/include/asm/mmu_context.h | 6 +-
arch/x86/include/asm/uaccess_64.h | 14 ++-
arch/x86/kernel/process_64.c | 4 +-
arch/x86/kernel/uprobes.c | 2 +-
include/linux/mm.h | 2 +-
include/linux/uaccess.h | 7 ++
mm/gup.c | 156 ++++++++++++++++++++++----
mm/internal.h | 7 +-
mm/memory.c | 171 ++++++++++++++++++++-------
mm/rmap.c | 2 +-
tools/testing/selftests/mm/pfnmap.c | 66 +++++++++++
15 files changed, 375 insertions(+), 90 deletions(-)
base-commit: 0f26556c5eeea62cc934fa8938b148aa5844a6b6
--
2.47.0