From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8110A470EA1 for ; Fri, 24 Jul 2026 22:30:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932216; cv=none; b=VX0kbcaGGpfgIfxrzmBoFWGwdwW2ofmjgMYzmc2kTCkrSybQCCiLvm5MVD08rhHIWC9P7JhTYsK8HaFNWwQRigTiguYPyhSaOCnSonymuj9yPd9Vij/wm/5dM/yGix7IIIRBVRPHY7kOmQDZbe35UjD8CwQVHeeXqGPcSw7auhs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932216; c=relaxed/simple; bh=KDF8zmn8nnQ5o+HE9p0R2eQN+l5v8f/djAorIliGAN4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ses1pDHzhl0CeEBg2hpvrBk4NV/sS0fpJ4ubBia8Gv736PFEoVZiw65tuYiDJFTlgLO9T3sPkkcacbb2tn0kFbWoAHsdWwl4FNNb+a1gO5xacO7eCgb5e2xA9PsldHwOGzfgHSgfXkoZ8Kg+R4iAcvVpTKi0IvdVz5AlC/phug0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=KaYyL4Mb; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="KaYyL4Mb" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=C7NU1MfovW+7lx1xP2l6wfJPEmWK+PLGNyfdih7cuN8=; b=KaYyL4MbCf5uM4PBwpzJonMKl9 cyXNE0D+OJUbxfv34AEjHgCSvXMJ8SgkagJY/CwSHuKatkwmtyO+J0MZYTRJuRK35fyAfLK4C5loJ bgEanDnDhIa18xkaNlqkFxBqRPp1XE7YbGrjwSbNQ2RJafZt9rjXhfmE6t3q/CBf0yUcnf8kaLj4v FSxcISpKkWhv3uZi5KgsIhYZQHey7sxnKewn2AG300IBLTrCuw0WfcbGYJKGlCbGqqmZOfQ4tIH/6 aZL4Q/572xDEJjOBhMPboLFBSluS4UpJxEBa+UjlzkqKyazeVe7cVtI3JTaWXgCcIJtnRzKMvWX4q z6mERBQg==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-0xfn; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Usama Arif Subject: [PATCH RFC v4 01/12] x86/mm: add untagged_addr_remote_unlocked() Date: Fri, 24 Jul 2026 18:29:23 -0400 Message-ID: <20260724222934.1463812-2-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __access_remote_vm() reads another process's memory under the mmap lock. On large machines running big multi-threaded applications, that lock is contended between readers and writers: an mmap() or munmap() in one thread stalls readers like /proc/PID/cmdline, /proc/PID/environ, or process_vm_readv(), even though the memory they read is almost always resident and reachable under the per-VMA lock instead. Looking up the VMA first requires untagging the address. untagged_addr_remote() asserts the mmap lock only because it reads mm->context.untag_mask, which can race with the write in mm_enable_lam(). That mask changes only while the target is single-threaded, when it enables LAM or resets it at exec. The read need not be stable: it is already read locklessly elsewhere, and a remote untag is best-effort. Add untagged_addr_remote_unlocked() for callers that have not taken the mmap lock, and annotate access to mm->context.untag_mask with READ_ONCE() and WRITE_ONCE() so the existing lockless reads are explicit and KCSAN-clean. untagged_addr_remote() keeps its assertion and shares the same code. Assisted-by: Claude:claude-opus-4.8 Acked-by: Usama Arif Signed-off-by: Rik van Riel --- arch/x86/include/asm/mmu_context.h | 6 +++--- arch/x86/include/asm/uaccess_64.h | 15 ++++++++++++--- arch/x86/kernel/process_64.c | 4 ++-- include/linux/uaccess.h | 7 +++++++ 4 files changed, 24 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_= context.h index ef5b507de34e..cee710f64658 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -100,18 +100,18 @@ static inline unsigned long mm_lam_cr3_mask(struct mm= _struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask =3D oldmm->context.lam_cr3_mask; - mm->context.untag_mask =3D oldmm->context.untag_mask; + WRITE_ONCE(mm->context.untag_mask, READ_ONCE(oldmm->context.untag_mask)); } =20 #define mm_untag_mask mm_untag_mask static inline unsigned long mm_untag_mask(struct mm_struct *mm) { - return mm->context.untag_mask; + return READ_ONCE(mm->context.untag_mask); } =20 static inline void mm_reset_untag_mask(struct mm_struct *mm) { - mm->context.untag_mask =3D -1UL; + WRITE_ONCE(mm->context.untag_mask, -1UL); } =20 #define arch_pgtable_dma_compat arch_pgtable_dma_compat diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uacce= ss_64.h index 20de34cc9aa6..a095165eb648 100644 --- a/arch/x86/include/asm/uaccess_64.h +++ b/arch/x86/include/asm/uaccess_64.h @@ -39,18 +39,27 @@ static inline unsigned long __untagged_addr(unsigned lo= ng addr) (__force __typeof__(addr))__untagged_addr(__addr); \ }) =20 +/* + * mm->context.untag_mask changes only when the target enables LAM or exec= s, + * always single-threaded. An unlocked remote reader can race that; READ_O= NCE + * keeps the value whole and the untag is best-effort. + */ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigned long addr) { - mmap_assert_locked(mm); - return addr & (mm)->context.untag_mask; + return addr & READ_ONCE(mm->context.untag_mask); } =20 -#define untagged_addr_remote(mm, addr) ({ \ +#define untagged_addr_remote_unlocked(mm, addr) ({ \ unsigned long __addr =3D (__force unsigned long)(addr); \ (__force __typeof__(addr))__untagged_addr_remote(mm, __addr); \ }) =20 +#define untagged_addr_remote(mm, addr) ({ \ + mmap_assert_locked(mm); \ + untagged_addr_remote_unlocked(mm, addr); \ +}) + #endif =20 #define valid_user_address(x) \ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index d44afbe005bb..9fa659117f38 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -814,7 +814,7 @@ static void enable_lam_func(void *__mm) static void mm_enable_lam(struct mm_struct *mm) { mm->context.lam_cr3_mask =3D X86_CR3_LAM_U57; - mm->context.untag_mask =3D ~GENMASK(62, 57); + WRITE_ONCE(mm->context.untag_mask, ~GENMASK(62, 57)); =20 /* * Even though the process must still be single-threaded at this @@ -952,7 +952,7 @@ long do_arch_prctl_64(struct task_struct *task, int opt= ion, unsigned long arg2) #endif #ifdef CONFIG_ADDRESS_MASKING case ARCH_GET_UNTAG_MASK: - return put_user(task->mm->context.untag_mask, + return put_user(mm_untag_mask(task->mm), (unsigned long __user *)arg2); case ARCH_ENABLE_TAGGED_ADDR: return prctl_enable_tagged_addr(task->mm, arg2); diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index eddbbb65ccc4..7e6e4c89184c 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -34,6 +34,13 @@ }) #endif =20 +#ifndef untagged_addr_remote_unlocked +#define untagged_addr_remote_unlocked(mm, addr) ({ \ + (void)(mm); \ + untagged_addr(addr); \ +}) +#endif + #ifdef masked_user_access_begin #define can_do_masked_user_access() 1 # ifndef masked_user_write_access_begin base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DEE2C470120 for ; Fri, 24 Jul 2026 22:30:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932218; cv=none; b=U+sEWVhCKwwgxH1oWlpVz05QrRThVjOb8Q99UEfepL2PNiNOi/YYFV7sQ5ggXXx4VAbR9KCMgciHEdBuCsC1CRsic1zRopT7wWj/kWl510k/FmxkN0/CPN0Gr1VO0KmtN7zFntEaG1KYKu0erTSMzc5P1BoryrBR13KOd03joXQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932218; c=relaxed/simple; bh=VEwMMIEFAluWDJ0OS1P3q5nheOQxX3odzvtWygaRusE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DFQcz8unAji00e/rEnOF1j5WQ+5N6L4js++OCvD4oY9MUn7liPLyLYjia7NyMy/+6p2ZJSd17VlLdD7lousND5bgZKDxO8gB36POSuw2VqfL+AE4gNAq4A6pQYfoDje5T4SSbuKwZOihXTn/eRlzGv+/Q8cg2mdABOxeZNa3cD0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=DwRMlYPQ; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="DwRMlYPQ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=IF4h0IOGesCPD6JnizKVFYQDqwHWpzKQAHqpitemykk=; b=DwRMlYPQpIsFsEsXZzPPL3sxm/ XwEli1sW2qLlJIwTseXX2o/q1glQCXOW/6qb3eWO+YB9NsZJ21DXr8JDOmcD4G8gCuqrbySUNPK0X AeN9ZFPsQawYFLRUwxaTF0qcs/+YwkcQm3g17KMh1EuuHv6EHqkdEfHW5IDK3KUSQ5zpTkQ75MAe7 xMin7ON9QwNgwRZWPvemLN7ReOBz/2+blnX+hqbzOcZ+CVBqoJItKziZOvlfBfywfefvI80kHSH7u 48Irw4hOCzBUYEdb5XsYKbjnzXSQK4vHCnAHc0nabbE43UPB8NzmeRB55+/i970oo+hEwo0d6bduS oRzqlBOA==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-14Iy; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , linux-riscv@lists.infradead.org Subject: [PATCH RFC v4 02/12] riscv/mm: add untagged_addr_remote_unlocked() Date: Fri, 24 Jul 2026 18:29:24 -0400 Message-ID: <20260724222934.1463812-3-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __access_remote_vm() reads another task's memory under the mmap lock. That lock is contended on large multi-threaded workloads, where an mmap() or munmap() stalls readers like /proc/PID/cmdline or process_vm_readv() even though the target memory is resident and reachable under the per-VMA lock. Reaching it under the per-VMA lock means looking up the VMA first, which requires untagging the remote address without the mmap lock. riscv's untagged_addr_remote() asserts that lock and has no unlocked variant; the generic untagged_addr_remote_unlocked() falls back to untagged_addr(), which masks against current->mm, not the target mm. Add untagged_addr_remote_unlocked(), which masks @addr against the target mm without asserting the lock, and redefine untagged_addr_remote() on top of it so it keeps the assertion. The mask width is mm->context.pmlen, written while a program is single threaded, and frozen with MM_CONTEXT_LOCK_PMLEN once it spawns a second thread. That read need not be stable: pmlen is already read locklessly by untagged_addr() and mm_untag_mask(), and a remote untag is best-effort. Annotate the lockless reads with READ_ONCE() and the writes with WRITE_ONCE(), so a reader sees one whole value and the accesses are KCSAN-clean. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Rik van Riel --- arch/riscv/include/asm/mmu_context.h | 4 ++-- arch/riscv/include/asm/uaccess.h | 10 +++++++--- arch/riscv/kernel/process.c | 12 +++++++----- 3 files changed, 16 insertions(+), 10 deletions(-) diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/= mmu_context.h index dbf27a78df6c..3ce16796e5a2 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -21,7 +21,7 @@ static inline void activate_mm(struct mm_struct *prev, struct mm_struct *next) { #ifdef CONFIG_RISCV_ISA_SUPM - next->context.pmlen =3D 0; + WRITE_ONCE(next->context.pmlen, 0); #endif switch_mm(prev, next, NULL); } @@ -44,7 +44,7 @@ DECLARE_STATIC_KEY_FALSE(use_asid_allocator); #define mm_untag_mask mm_untag_mask static inline unsigned long mm_untag_mask(struct mm_struct *mm) { - return -1UL >> mm->context.pmlen; + return -1UL >> READ_ONCE(mm->context.pmlen); } #endif =20 diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uacc= ess.h index 5d4ec15584cf..53806e0f7dcf 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -16,7 +16,7 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, u= nsigned long addr) { if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SUPM)) { - u8 pmlen =3D mm->context.pmlen; + u8 pmlen =3D READ_ONCE(mm->context.pmlen); =20 /* Virtual addresses are sign-extended; physical addresses are zero-exte= nded. */ if (IS_ENABLED(CONFIG_MMU)) @@ -33,12 +33,16 @@ static inline unsigned long __untagged_addr_remote(stru= ct mm_struct *mm, unsigne (__force __typeof__(addr))__untagged_addr_remote(current->mm, __addr); \ }) =20 -#define untagged_addr_remote(mm, addr) ({ \ +#define untagged_addr_remote_unlocked(mm, addr) ({ \ unsigned long __addr =3D (__force unsigned long)(addr); \ - mmap_assert_locked(mm); \ (__force __typeof__(addr))__untagged_addr_remote(mm, __addr); \ }) =20 +#define untagged_addr_remote(mm, addr) ({ \ + mmap_assert_locked(mm); \ + untagged_addr_remote_unlocked(mm, addr); \ +}) + #define access_ok(addr, size) likely(__access_ok(untagged_addr(addr), size= )) #else #define untagged_addr(addr) (addr) diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index b2df7f72241a..6ae7552fed09 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -357,13 +357,15 @@ long set_tagged_addr_ctrl(struct task_struct *task, u= nsigned long arg) if (mmap_write_lock_killable(mm)) return -EINTR; =20 - if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags) && mm->context.pm= len !=3D pmlen) { - mmap_write_unlock(mm); - return -EBUSY; + if (test_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags)) { + if (READ_ONCE(mm->context.pmlen) !=3D pmlen) { + mmap_write_unlock(mm); + return -EBUSY; + } } =20 envcfg_update_bits(task, ENVCFG_PMM, pmm); - mm->context.pmlen =3D pmlen; + WRITE_ONCE(mm->context.pmlen, pmlen); =20 mmap_write_unlock(mm); =20 @@ -394,7 +396,7 @@ long get_tagged_addr_ctrl(struct task_struct *task) break; } =20 - if (task->mm->context.pmlen) + if (READ_ONCE(task->mm->context.pmlen)) ret |=3D PR_TAGGED_ADDR_ENABLE; =20 return ret; --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B633470114; Fri, 24 Jul 2026 22:30:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932221; cv=none; b=QzpAgXHzbWKVQ7zxjcvSkF4pP7/sf4c9JlrvUI0Bs+EUHZRId45uZQDS2GXrgPS2rz7C44UhE0Jnb/1wqipJCZX426HpocgJlMxWKzKl/mUdZmIDsEoO678NjaMIrF5Xr6mMOdRyTpXSxAdiG443nlvauIm8djHpaJGDevB/ObI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932221; c=relaxed/simple; bh=SxU/YkSx9/aDHrnwD+QcDLHCxGTwlDX2cC7MmSC4v78=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d4tmE2p5p79Znshx0jfshOarkMB8nQIaktKk8GQCDY4AO/wAF7+Wis40aM7nb2Igdl71b/TC8PZQzEIOmHr5nYtl52kuNGjJFPgUisP2AjTd/h1PXisfDHmR/p0Bb8IELw10vLLuiZGfSVxnCm4m5zk3jCewS09fhaerMHBcIcM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=RVCkNQ0h; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="RVCkNQ0h" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=qDlbQ/k/gKhextbIPqwqCwZ5BabZN9oEFBEjCjmG3eE=; b=RVCkNQ0hfcZsXJGNqnk2s4AJat 5GNYVKosxlBdUigBnazluK5OVF0UCUomWI4E0gm1sKSyQ2DpZ+Ve3tIRlSFSnuh2kflmXApNDjUUy gxxxgxZDK+Zu5kiplzcnWTHxBVCkgfs3azyY4shlHCrtJXqFOOpJwEQwoT59m8dBOlzuDGVHaelLx 9vu7z4jFwPmPhdj9bAGq53MEVutyT93H6mHz4OREXM9GZBMKTOl1XR7MZjlGNdhiKzL/CVQTn0RWx GCawa8mEsly9K9s3yOHmXaRVfdOenUeerOK0giEoty7aWhdLSK/ThtyzZ3pM42YvYQPfRKES+Rp9i Oa/5MXxQ==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-1CWq; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel , Catalin Marinas , Will Deacon , Masami Hiramatsu , Oleg Nesterov , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , Harry Yoo , Jann Horn , Lance Yang , linux-arm-kernel@lists.infradead.org, linux-trace-kernel@vger.kernel.org, Usama Arif Subject: [PATCH RFC v4 03/12] mm: rename get_user_page_vma_remote() to get_user_page_lookup_vma() Date: Fri, 24 Jul 2026 18:29:25 -0400 Message-ID: <20260724222934.1463812-4-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" get_user_page_vma_remote() faults in the page at @addr in a remote mm and also looks up the VMA that covers it, handing both back to the caller. This cleans up the namespace for adding a get_user_page_vma() variant where the caller already has the vma. No functional change intended. Assisted-by: Claude:claude-opus-4.8 Acked-by: Usama Arif Signed-off-by: Rik van Riel --- arch/arm64/kernel/mte.c | 2 +- arch/x86/kernel/uprobes.c | 2 +- include/linux/mm.h | 2 +- mm/memory.c | 4 ++-- mm/rmap.c | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 1a9aad6ef22a..7a6ecc3d9294 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -459,7 +459,7 @@ static int __access_remote_tags(struct mm_struct *mm, u= nsigned long addr, struct vm_area_struct *vma; unsigned long tags, offset; void *maddr; - struct page *page =3D get_user_page_vma_remote(mm, addr, + struct page *page =3D get_user_page_lookup_vma(mm, addr, gup_flags, &vma); struct folio *folio; =20 diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c index 3af979fb41d3..329efac0cfb3 100644 --- a/arch/x86/kernel/uprobes.c +++ b/arch/x86/kernel/uprobes.c @@ -1036,7 +1036,7 @@ static int copy_from_vaddr(struct mm_struct *mm, unsi= gned long vaddr, void *dst, struct vm_area_struct *vma; struct page *page; =20 - page =3D get_user_page_vma_remote(mm, vaddr, gup_flags, &vma); + page =3D get_user_page_lookup_vma(mm, vaddr, gup_flags, &vma); if (IS_ERR(page)) return PTR_ERR(page); uprobe_copy_from_page(page, vaddr, dst, len); diff --git a/include/linux/mm.h b/include/linux/mm.h index 485df9c2dbdd..24ead14b4790 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3238,7 +3238,7 @@ long pin_user_pages_remote(struct mm_struct *mm, /* * Retrieves a single page alongside its VMA. Does not support FOLL_NOWAIT. */ -static inline struct page *get_user_page_vma_remote(struct mm_struct *mm, +static inline struct page *get_user_page_lookup_vma(struct mm_struct *mm, unsigned long addr, int gup_flags, struct vm_area_struct **vmap) diff --git a/mm/memory.c b/mm/memory.c index ff338c2abe92..3b86eeaf084f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7039,7 +7039,7 @@ static int __access_remote_vm(struct mm_struct *mm, u= nsigned long addr, void *maddr; struct folio *folio; struct vm_area_struct *vma =3D NULL; - struct page *page =3D get_user_page_vma_remote(mm, addr, + struct page *page =3D get_user_page_lookup_vma(mm, addr, gup_flags, &vma); =20 if (IS_ERR(page)) { @@ -7167,7 +7167,7 @@ static int __copy_remote_vm_str(struct mm_struct *mm,= unsigned long addr, struct page *page; struct vm_area_struct *vma =3D NULL; =20 - page =3D get_user_page_vma_remote(mm, addr, gup_flags, &vma); + page =3D get_user_page_lookup_vma(mm, addr, gup_flags, &vma); if (IS_ERR(page)) { /* * Treat as a total failure for now until we decide how diff --git a/mm/rmap.c b/mm/rmap.c index 1c77d5dc06e9..b36f2e219b8f 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2838,7 +2838,7 @@ struct page *make_device_exclusive(struct mm_struct *= mm, unsigned long addr, * (non-device-exclusive) PTE and issue a MMU_NOTIFY_EXCLUSIVE. */ retry: - page =3D get_user_page_vma_remote(mm, addr, + page =3D get_user_page_lookup_vma(mm, addr, FOLL_GET | FOLL_WRITE | FOLL_SPLIT_PMD, &vma); if (IS_ERR(page)) --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93FC9470EA5 for ; Fri, 24 Jul 2026 22:30:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932215; cv=none; b=NegIl23tkVVeYsLPGrpkfV8JnZusED0Xmkzpt+4qPxc7eLjSRgzp/i0PmkykUMBD3nbUpLhzwgLazS2vJZAOmkHyUI+2jC0DgruHxXMtY+d/XlrBO+eCxDhzxS591m+VVMbqy8P6m20YHmYLTsZmWDiKkvR9pMLljIInwY9wa40= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932215; c=relaxed/simple; bh=x5HLJeYpQRCYKSIXjkXM7jBp6L9hp+hP0Jolmm3Gx2I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XpVfqFFpsfMdULp2ufbJrQa/OVtWbZisfU8zTPu2eEQQgVXlyRnl7y4R3t3KCCy0v+OoAzKTO+stsCEm4ahf5j7iQOPjIcn5YE/+V/LzvCx0etArtikNKogehbZYDYwnDJFCs1Dv7Bm33gt1xzC8T0PC6FTOgKtCr7CiIhzxVz8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=jlvsnQJ6; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="jlvsnQJ6" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=VoZs+loVAhBwkjc5A6PJBaB7eEBwxAF4Y2TdD3Br7Qs=; b=jlvsnQJ6OfYza2IkSyBJe1yYTQ djHcHMT23M8KZMD8bBxVN78RuuiujT9B3csAmMEx29Wj4UcB3QOV6j7K+R5W/zDKp0EjWijdzTsis NQFFdlIGG0OQko1SKqWL6/hLu3MIkF9H9M9k/jR7sssLzOBWj1+ZPre8jmZKanZzDWikQKi3tvwo1 c9UaZWccOxUWHRbiKwR8/70hRWTz81a2lKjg2t77yv0C2LbAmYwks1lkn9TQGBzLwDziLvM+xsukY yV7UEJdIzukIrnqT9F48pjx8ZFtSd6m+n/pbY3mvux3pv+tAmDo8z/3UrDf4kQI7FqonhJK6ZBfxy VB/qlljw==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-1JWz; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel Subject: [PATCH RFC v4 04/12] mm/gup: let check_vma_flags() ignore selected VMA flags Date: Fri, 24 Jul 2026 18:29:26 -0400 Message-ID: <20260724222934.1463812-5-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" check_vma_flags() rejects a VMA whose flags gup cannot handle. A caller that wants to reach a COWed page in a VM_IO or VM_PFNMAP VMA, where that page does have a struct page, cannot express the exception and has to repeat the flag test itself. Add an ignore_flags argument that check_vma_flags() clears from its local copy of vma->vm_flags before the flag checks. A caller can drop a single rejection this way while keeping every other check. All current callers pass 0. This prepares for get_user_page_vma(), which passes VM_IO | VM_PFNMAP to reach those COWed pages. No functional change intended. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Rik van Riel --- mm/gup.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 0692119b7904..dcece7b5253c 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1197,13 +1197,21 @@ static bool writable_file_mapping_allowed(struct vm= _area_struct *vma, return !vma_needs_dirty_tracking(vma); } =20 -static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_f= lags) +static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_f= lags, + vm_flags_t ignore_flags) { vm_flags_t vm_flags =3D vma->vm_flags; int write =3D (gup_flags & FOLL_WRITE); int foreign =3D (gup_flags & FOLL_REMOTE); bool vma_anon =3D vma_is_anonymous(vma); =20 + /* + * Opt out of the flag checks that read this local copy (the + * VM_IO/VM_PFNMAP gate and the write/read/cow bits); checks that + * re-read vma->vm_flags through helpers are unaffected. + */ + vm_flags &=3D ~ignore_flags; + if (vm_flags & (VM_IO | VM_PFNMAP)) return -EFAULT; =20 @@ -1387,7 +1395,7 @@ static long __get_user_pages(struct mm_struct *mm, ret =3D -ENOMEM; goto out; } - if (check_vma_flags(vma, gup_flags)) { + if (check_vma_flags(vma, gup_flags, 0)) { ret =3D -EINVAL; goto out; } @@ -1408,7 +1416,7 @@ static long __get_user_pages(struct mm_struct *mm, ret =3D -EFAULT; goto out; } - ret =3D check_vma_flags(vma, gup_flags); + ret =3D check_vma_flags(vma, gup_flags, 0); if (ret) goto out; } --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BADF1471417 for ; Fri, 24 Jul 2026 22:30:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932221; cv=none; b=l1y5UUjeQ5LHdJvS18z9jVHB1SKGKV2W9mDciye5a5yt9SJS4x3T19j4nMmYURYKZBoB08P4NqOK5CWqV3vqzqVqUNchrTTPKsbtkF0ecv/FWL9B9H0xRJKHCcKUybhGw1Y/iw+zus31y0dqZlTBPSUTiNnZ9/QSFNRJUrmuCbk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932221; c=relaxed/simple; bh=ApVF97hyRB7OpTXhr+KJeYzT3X1y71moxCjxt9uWULY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EHObd6468pUetKjzUzTK5G2PSIiwQ/ccPlgWVcmxKwgO+m84TS95XRIsYZTo/DksvDSo9sIDHVewTcyn1CCCQR00oegEti39bDG60o1WAArC3aTmXYqz9VoocxqbfAUp6Xk5+gLkVIHZS1cjNxCni3024GBW1Ou9IuLAkSwO21A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=ablmeSFS; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="ablmeSFS" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=+hHdI4LWlDC8ULGYHra6LN9uCKn7OJXzJpyk4YtJG+A=; b=ablmeSFS7MEeH3jzONlP/3XcqT 2eSf4O1EWC2jWrOLkzzY3D4oXJ+MehzOMZ3Zy4ejrUQAmaG0N/LzqSyZoOyL3oYuFiYDNBCE4bw5u e0yV5uIKRfVNjiDca92SDcRWItCQyf73coJrAQhqL9YlyFoU738gA6+9Pp06GQ8TIxZZFWKJSDK1c xeSiK9+JNCLLeXE1ItuOhOM3CGJfDDzMRUo5dI+i0s1i/I6lL+xmkKwvvVxdHGUKoawSdopc95wWU XkDLRq2gKGKBGP24gK11aSoE/+maegVNFUhPw6mTOJzOrwSKKnaUfDLxcSI2YgWWAeLy2YbrG+6MI 5lS/lSJQ==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-1Q3T; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel Subject: [PATCH RFC v4 05/12] mm/gup: add get_user_page_vma() to fault in a page under a held lock Date: Fri, 24 Jul 2026 18:29:27 -0400 Message-ID: <20260724222934.1463812-6-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __access_remote_vm() needs a single page from a VMA it has already looked up and locked, faulting it in when necessary, under either the mmap lock or the per-VMA lock. get_user_pages_remote() does not fit: it hard codes the mmap lock and re-looks-up the VMA, neither of which is wanted here. Add get_user_page_vma(), a simplified __get_user_pages() that walks the page tables with follow_page_mask(), faults a missing page in with faultin_page(), and on success returns it with a reference and the caller's lock still held. Like __get_user_pages() it runs check_vma_flags(), so callers need not pre-check the VMA. A VM_IO/VM_PFNMAP VMA is the exception to that check: it can still hold COWed pages that have a struct page, so follow_page_mask() is allowed to look for one. Memory with no struct page -- a raw PFN, or a present PFN reported as -EEXIST -- is returned as -EFAULT, so the caller can reach it through vma->vm_ops->access(). The caller sets FOLL_VMA_LOCK when it holds the per-VMA lock rather than the mmap lock, which reaches the fault code as FAULT_FLAG_VMA_LOCK. Anything that cannot complete under the per-VMA lock -- a dropped fault, a userfaultfd VMA (uffd assumes current is the faulting task), a hard error, or ->access() memory -- releases the lock and returns -EAGAIN, so the caller retries under the mmap lock. faultin_page() reports these retries as -EAGAIN for both lock types; only the mmap caller records the dropped lock in *locked. A VM_FAULT_ERROR decoding to no errno now returns -EFAULT instead of BUG(), warning under CONFIG_DEBUG_VM. This relaxes the same case on the existing __get_user_pages() path too. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Rik van Riel --- mm/gup.c | 176 +++++++++++++++++++++++++++++++++++++++++++------- mm/internal.h | 8 ++- 2 files changed, 159 insertions(+), 25 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index dcece7b5253c..cb7245b7542f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1080,9 +1080,16 @@ static int get_gate_page(struct mm_struct *mm, unsig= ned long address, } =20 /* - * mmap_lock must be held on entry. If @flags has FOLL_UNLOCKABLE but not - * FOLL_NOWAIT, the mmap_lock may be released. If it is, *@locked will be= set - * to 0 and -EBUSY returned. + * The caller holds either the mmap lock, or the per-VMA lock (with + * FOLL_VMA_LOCK), on entry. If @flags has FOLL_UNLOCKABLE but not FOLL_NO= WAIT, + * the mmap_lock may be released. If it is, *@locked will be set to 0 and + * -EAGAIN returned. + * + * The return value does not depend on the lock type: a fault that made + * progress but needs a retry (VM_FAULT_RETRY / VM_FAULT_COMPLETED) is rep= orted + * as -EAGAIN for both the mmap lock and the per-VMA lock (FOLL_VMA_LOCK).= Only + * the *@locked side effect is lock-type specific, as the per-VMA lock pat= h has + * no unlockable mmap_lock to drop. */ static int faultin_page(struct vm_area_struct *vma, unsigned long address, unsigned int flags, bool unshare, @@ -1097,6 +1104,8 @@ static int faultin_page(struct vm_area_struct *vma, fault_flags |=3D FAULT_FLAG_WRITE; if (flags & FOLL_REMOTE) fault_flags |=3D FAULT_FLAG_REMOTE; + if (flags & FOLL_VMA_LOCK) + fault_flags |=3D FAULT_FLAG_VMA_LOCK; if (flags & FOLL_UNLOCKABLE) { fault_flags |=3D FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; /* @@ -1125,41 +1134,160 @@ static int faultin_page(struct vm_area_struct *vma, =20 ret =3D handle_mm_fault(vma, address, fault_flags, NULL); =20 + /* + * A fully completed fault (VM_FAULT_COMPLETED) or one that needs a retry + * (VM_FAULT_RETRY) has released the lock it was holding. Report both as + * -EAGAIN so the caller retries: the mmap lock caller retakes it here, + * the per-VMA lock caller (FOLL_VMA_LOCK) falls back to the mmap lock. + * + * Dropping the mmap lock is recorded in *@locked. There is no such lock + * to drop under the per-VMA lock, where @locked is not used, so leave it + * alone in that case. + */ if (ret & VM_FAULT_COMPLETED) { - /* - * With FAULT_FLAG_RETRY_NOWAIT we'll never release the - * mmap lock in the page fault handler. Sanity check this. - */ - WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT); - *locked =3D 0; - - /* - * We should do the same as VM_FAULT_RETRY, but let's not - * return -EBUSY since that's not reflecting the reality of - * what has happened - we've just fully completed a page - * fault, with the mmap lock released. Use -EAGAIN to show - * that we want to take the mmap lock _again_. - */ + if (!(flags & FOLL_VMA_LOCK)) { + /* + * With FAULT_FLAG_RETRY_NOWAIT we'll never release the + * mmap lock in the page fault handler. Sanity check this. + */ + WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT); + *locked =3D 0; + } return -EAGAIN; } =20 if (ret & VM_FAULT_ERROR) { int err =3D vm_fault_to_errno(ret, flags); =20 - if (err) - return err; - BUG(); + /* + * VM_FAULT_ERROR always decodes to an errno; a zero here would + * mean handle_mm_fault() returned an unexpected combination. + * Report -EFAULT rather than crash: under the per-VMA lock the + * mmap lock retry produces the definitive result. + */ + VM_WARN_ON_ONCE(!err); + return err ? err : -EFAULT; } =20 if (ret & VM_FAULT_RETRY) { - if (!(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) + if (!(flags & FOLL_VMA_LOCK) && + !(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) *locked =3D 0; - return -EBUSY; + return -EAGAIN; } =20 return 0; } =20 +/* + * get_user_page_vma - get one page from @vma, whose lock the caller alrea= dy + * holds: the mmap lock, or (with FOLL_VMA_LOCK) the per-VMA lock. Walks t= he + * page tables, faulting the page in if needed, and on success returns it = with + * a reference and the lock still held. + * + * Runs check_vma_flags() like __get_user_pages(), so callers need not pre= -check + * the VMA; most rejections are returned as their error. A VM_IO/VM_PFNMAP= VMA + * is the exception: a COWed page with a struct page is returned, while a = raw + * PFN has none and yields -EFAULT, to be reached via vma->vm_ops->access(= ). + * + * Under FOLL_VMA_LOCK, anything that cannot be finished under the per-VMA= lock + * (a dropped fault, userfaultfd, a hard error, or ->access() memory) rele= ases + * the lock and returns -EAGAIN, so the caller retries under the mmap lock. + */ +struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long a= ddr, + unsigned int gup_flags) +{ + bool vma_locked =3D gup_flags & FOLL_VMA_LOCK; + unsigned long page_mask; + struct page *page; + int locked =3D 1; + bool pfnmap; + int ret; + + /* + * Two lock modes are supported: the mmap lock, with neither flag, or + * the per-VMA lock, with both FOLL_VMA_LOCK and FOLL_UNLOCKABLE. An + * unlockable mmap fault would drop the lock and report it in *locked, + * which this function does not relay, so reject a lone flag. + */ + VM_WARN_ON_ONCE(!(gup_flags & FOLL_VMA_LOCK) !=3D !(gup_flags & FOLL_UNLO= CKABLE)); + + /* + * Validate the VMA up front, like __get_user_pages(). A VM_IO/VM_PFNMAP + * VMA is not rejected outright: it can hold COWed pages that have a + * struct page; let follow_page_mask() look for them, and treat only + * its struct-page-less PFNs as unreachable. Other rejections + * (secretmem, permissions, etc) result in immediate failure. + */ + ret =3D check_vma_flags(vma, gup_flags, VM_IO | VM_PFNMAP); + if (ret) + goto fail; + pfnmap =3D vma->vm_flags & (VM_IO | VM_PFNMAP); + + for (;;) { + if (fatal_signal_pending(current)) { + ret =3D -EINTR; + goto fail; + } + cond_resched(); + + /* follow_page_mask() requires @page_mask; it is unused here. */ + page =3D follow_page_mask(vma, addr, + gup_flags | FOLL_TOUCH | FOLL_GET, + &page_mask); + if (!IS_ERR_OR_NULL(page)) { + /* Match __get_user_pages(): flush for VIVT/aliasing caches. */ + flush_anon_page(vma, page, addr); + flush_dcache_page(page); + return page; + } + + /* + * No struct page: a raw PFN of a VM_IO/VM_PFNMAP VMA, whether + * seen by the up-front check (@pfnmap) or reported as -EEXIST + * for a present PFN. Return -EFAULT so the caller reaches it + * through vma->vm_ops->access(). + */ + if (pfnmap || PTR_ERR(page) =3D=3D -EEXIST) { + ret =3D -EFAULT; + goto fail; + } + /* A hard error from the walk itself. */ + if (page && PTR_ERR(page) !=3D -EMLINK) { + ret =3D PTR_ERR(page); + goto fail; + } + + /* + * The page is not present, or needs unsharing. A remote fault + * under the per-VMA lock cannot deliver userfaultfd (which + * assumes current is the faulting task), so fall back for those. + */ + if (vma_locked && userfaultfd_armed(vma)) { + ret =3D -EAGAIN; + goto fail; + } + ret =3D faultin_page(vma, addr, gup_flags | FOLL_REMOTE | FOLL_GET, + PTR_ERR(page) =3D=3D -EMLINK, &locked); + if (ret =3D=3D -EAGAIN) + return ERR_PTR(-EAGAIN); /* fault released the per-VMA lock */ + if (ret) + goto fail; + } + +fail: + /* + * Under the per-VMA lock the caller cannot reach ->access() or act on a + * hard error (both need the mmap lock), so release the lock and have it + * retry there; the mmap-lock pass produces the definitive error. + */ + if (vma_locked) { + vma_end_read(vma); + return ERR_PTR(-EAGAIN); + } + return ERR_PTR(ret); +} + /* * Writing to file-backed mappings which require folio dirty tracking usin= g GUP * is a fundamentally broken operation, as kernel write access to GUP mapp= ings @@ -1197,8 +1325,8 @@ static bool writable_file_mapping_allowed(struct vm_a= rea_struct *vma, return !vma_needs_dirty_tracking(vma); } =20 -static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_f= lags, - vm_flags_t ignore_flags) +int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags, + vm_flags_t ignore_flags) { vm_flags_t vm_flags =3D vma->vm_flags; int write =3D (gup_flags & FOLL_WRITE); diff --git a/mm/internal.h b/mm/internal.h index 181e79f1d6a2..706f7f08fd18 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1595,6 +1595,10 @@ struct vm_struct *__get_vm_area_node(unsigned long s= ize, */ int __must_check try_grab_folio(struct folio *folio, int refs, unsigned int flags); +int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags, + vm_flags_t ignore_flags); +struct page *get_user_page_vma(struct vm_area_struct *vma, unsigned long a= ddr, + unsigned int gup_flags); =20 /* * mm/huge_memory.c @@ -1641,11 +1645,13 @@ enum { FOLL_UNLOCKABLE =3D 1 << 21, /* VMA lookup+checks compatible with MADV_POPULATE_(READ|WRITE) */ FOLL_MADV_POPULATE =3D 1 << 22, + /* caller holds the per-VMA lock, not the mmap lock */ + FOLL_VMA_LOCK =3D 1 << 23, }; =20 #define INTERNAL_GUP_FLAGS (FOLL_TOUCH | FOLL_TRIED | FOLL_REMOTE | FOLL_P= IN | \ FOLL_FAST_ONLY | FOLL_UNLOCKABLE | \ - FOLL_MADV_POPULATE) + FOLL_MADV_POPULATE | FOLL_VMA_LOCK) =20 /* * Indicates for which pages that are write-protected in the page table, --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 155C747127D for ; Fri, 24 Jul 2026 22:30:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932218; cv=none; b=buxzJzTiN5Z4ZjtKg0/QMz+YI5a2o7iHWIPrcR9n6X0zi6KtJ280WN7Pnyr7FdneBD9s1OWOPOgUJyYec9d9Hy9KH5XhGNIC/IzC84duO6tqp2LO8E2fWFk8Qi97aZT2MBrUQ19Ho7SIKr4Ej38/lu7nrlFlJjlUAOh2sUF9df8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932218; c=relaxed/simple; bh=MZKEAz8yWCM/6wn59o6VET0NBltuoUh1GyhKZx78hRg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CIawI+i3wkCbvTdYiBAqNf2kJUSf8t8t4P4cfFSHE1g4lebsMhWrzqwasfbhDCzcb8KWqJyH9IOI33w8t/Q/nPLKN4a+4+u2mzPu41Y5dW/h3I5gST4nB6LOEdt/sVPoBOsR1+BM4HOdaQggsE0pQV1TXrazyan/d2tARk5AWGs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=lkmV8VVh; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="lkmV8VVh" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=Tct12KgmC1tTE0HVtqWbI1n/3fyI3GSxYNmtgTnfJDk=; b=lkmV8VVhvXXhz16hhgrH1uM+hK m9gans4bdPiCtqRwGtEHzzakQj33b1jxIc1sgMTOh6GxwBEbcQvteFKDJnDO3kSjLmM2Y4ph29dJc CkkwlJ6R7CW/PI10sREsZTlcJAU9XtRnzIG2zjJokgGLxQ6Gz++cvE2ddG/KoMdAUaO7g6Fg61Wis lD3I0sz+fbwO2qEdzKrt6GB6xow76++/KDaeeqvu74hR1/Ews4GPvMtHp0tneWwB9qCmDn3x25S5K GwyWHaNLgwrmCSWvo2afptja5wHYgnuZvYbATAHo9xEnXiebgXMgpSDekFbsHVIzdrcCTC/LiMqAq aFVsrEPg==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-1X40; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel Subject: [PATCH RFC v4 06/12] mm: use per-VMA lock in __access_remote_vm() for single-VMA accesses Date: Fri, 24 Jul 2026 18:29:28 -0400 Message-ID: <20260724222934.1463812-7-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __access_remote_vm() holds mmap_read_lock() for the whole transfer. On large machines running big multi-threaded applications, that lock is contended between readers and writers: an mmap() or munmap() stalls readers like /proc/PID/cmdline, /proc/PID/environ, or /proc/PID/mem, even though the memory they read is almost always resident. Use the per-VMA lock to access a remote process's memory when the access fits in one VMA. Fall back to the mmap_lock if the access crosses VMA boundaries, or when get_user_page_vma() cannot finish the access under the per-VMA lock. Factor the walk into remote_vm_walk(): it selects the lock, faults in each page with get_user_page_vma(), and hands the page to a per-page action. __access_remote_vm() passes access_vm_page(), which copies it; a later patch shares the walk for the remote string reader. remote_access_lock() and remote_access_unlock() keep the lock selection out of the walk. remote_access_lock() takes the per-VMA lock when the range fits one VMA whose flags permit the access, and the mmap lock otherwise, returning an ERR_PTR() when the mmap lock cannot be taken. Looking up the VMA first untags the remote address with untagged_addr_remote_unlocked(), added earlier in this series, so the untag needs no mmap lock. Walking the page tables under only the per-VMA lock is safe against both page table freeing and THP collapse. munmap() frees a VMA's page tables through free_pgtables(), which uses neither the page table lock nor RCU. But it runs only after the VMA is detached under the VMA write lock, which our per-VMA read lock excludes, so no page table of this VMA is torn down under the walk. THP collapse instead needs neither lock: file-backed collapse retracts a page table under i_mmap_lock and the page table lock alone. It frees the retracted PTE page by RCU, and pte_offset_map() holds the RCU read lock, so the page stays valid for the walk. follow_page_pte() takes that same page table lock through pte_offset_map_lock() and rechecks the pmd, so it either walks an intact table or sees the collapsed pmd and faults the THP in. get_user_page_vma() returns -EFAULT for memory with no struct page: the raw PFNs of a VM_IO/VM_PFNMAP VMA, ioremapped device memory reached through ptrace and /proc/PID/mem. access_vm_page() reaches that memory under the mmap lock through vma->vm_ops->access(), via a new access_remote_vma_ops() helper, as get_user_pages_remote() and the old ->access() fallback did before. A COWed page in such a VMA now reads normally through get_user_page_vma(); previously it was routed to ->access(), whose generic_access_phys() rejects the ioremap of a RAM page. A single-threaded microbenchmark reading a remote process's memory through /proc/PID/mem shows the per-VMA path costs less than the old mmap_read_lock() plus get_user_pages_remote() route. The gain grows with the read size as the per-page VMA re-lookup drops away. Median of three pinned repetitions in a VM: read size baseline per-VMA throughput 8 B 201.7 ns 170.6 ns ~15% faster 4 KB 495 ns 442 ns +12% (8266 -> 9265 MB/s) 64 KB 4115 ns 3355 ns +23% (15927 -> 19536 MB/s) 1 MB 75902 ns 63641 ns +19% (13815 -> 16477 MB/s) This does not measure the multi-threaded reader-versus-writer contention that motivates the per-VMA lock; that case remains to be quantified. Assisted-by: Claude:claude-opus-4.8 Suggested-by: Suren Baghdasaryan Signed-off-by: Rik van Riel --- mm/memory.c | 268 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 213 insertions(+), 55 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 3b86eeaf084f..273dfe12bc6d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7015,86 +7015,244 @@ EXPORT_SYMBOL_GPL(generic_access_phys); #endif =20 /* - * Access another process' address space as given in mm. + * VM_IO / VM_PFNMAP memory, such as an ioremapped device mapping, maps + * PFNs that have no struct page, so get_user_page_vma() cannot fetch it + * even though the page tables are populated. It can still be reached + * through vma->vm_ops->access(). + * + * Returns the number of bytes transferred, or <=3D 0 if @vma cannot be + * accessed this way. */ -static int __access_remote_vm(struct mm_struct *mm, unsigned long addr, - void *buf, int len, unsigned int gup_flags) +static int access_remote_vma_ops(struct vm_area_struct *vma, unsigned long= addr, + void *buf, int len, int write) +{ +#ifdef CONFIG_HAVE_IOREMAP_PROT + if (vma->vm_ops && vma->vm_ops->access) + return vma->vm_ops->access(vma, addr, buf, len, write); +#endif + return 0; +} + +/* + * Lock @mm to reach the remote range [@addr, @addr + @len). + * + * Take the per-VMA lock when the whole range fits in a single VMA whose + * flags permit the access. The RCU freed page tables then keep page table + * memory from being reused with unexpected contents while the lock is hel= d. + * Otherwise fall back to the mmap lock, which also covers multi-VMA range= s, + * stack expansion, and ->access() memory. + * + * Return whether the mmap lock is held. The per-VMA locked VMA, when one = is + * taken, is stored in *@vmap; it is NULL on the mmap lock path. *@vmap is + * set to an ERR_PTR() when the mmap lock could not be taken, so callers m= ust + * check IS_ERR(*@vmap) before using either result. + */ +static bool remote_access_lock(struct mm_struct *mm, unsigned long addr, + int len, unsigned int gup_flags, + struct vm_area_struct **vmap) +{ + struct vm_area_struct *vma =3D NULL; + +#if defined(CONFIG_PER_VMA_LOCK) && defined(CONFIG_MMU_GATHER_RCU_TABLE_FR= EE) + vma =3D lock_vma_under_rcu(mm, addr); + if (vma) { + /* addr + len must not wrap, and must fit within the one VMA. */ + if (addr + len < addr || addr + len > vma->vm_end || + check_vma_flags(vma, gup_flags, 0)) { + vma_end_read(vma); + vma =3D NULL; + } + } +#endif + + if (!vma) { + if (mmap_read_lock_killable(mm)) { + *vmap =3D ERR_PTR(-EINTR); + return false; + } + *vmap =3D NULL; + return true; + } + + *vmap =3D vma; + return false; +} + +/* Release the lock taken by remote_access_lock(). */ +static void remote_access_unlock(struct mm_struct *mm, + struct vm_area_struct *vma, bool have_mmap_lock) +{ + if (have_mmap_lock) + mmap_read_unlock(mm); + else if (vma) + vma_end_read(vma); +} + +/* + * Per-page action for a remote VM walk. Handle up to @len bytes at @addr = on + * @page, advancing *@buf past the bytes read from or written to it. @page= is + * NULL for struct-page-less memory (VM_IO / VM_PFNMAP) reached under the = mmap + * lock. + * + * Return the number of source bytes handled at @addr, 0 to end the walk (a + * string reached its NUL, or ->access() memory could not be reached), or a + * negative errno to abort. + */ +typedef int (*remote_vm_action)(struct vm_area_struct *vma, struct page *p= age, + unsigned long addr, void **buf, int len, + int write); + +/* + * Walk the remote range [@addr, @addr + @len) of @mm, handing each page to + * @action. Use the per-VMA lock when the range fits one VMA, and fall bac= k to + * the mmap lock for multi-VMA ranges, stack expansion (when @can_expand_s= tack + * is set), or when the per-VMA lock cannot finish a fault. + * + * Each page is faulted in with get_user_page_vma() under whichever lock is + * held. Return the number of bytes @action consumed; *@err is a negative + * errno when the walk aborted, else 0. + */ +static int remote_vm_walk(struct mm_struct *mm, unsigned long addr, void *= buf, + int len, unsigned int gup_flags, bool can_expand_stack, + remote_vm_action action, int *err) { void *old_buf =3D buf; int write =3D gup_flags & FOLL_WRITE; + bool have_mmap_lock; + struct vm_area_struct *vma; =20 - if (mmap_read_lock_killable(mm)) - return 0; + *err =3D 0; =20 - /* Untag the address before looking up the VMA */ - addr =3D untagged_addr_remote(mm, addr); + /* + * Set FOLL_REMOTE so check_vma_flags() applies the same protection key + * rules as get_user_pages_remote() did: the current PKRU is not checked + * against a VMA reached on @mm's behalf. + */ + gup_flags |=3D FOLL_REMOTE; =20 - /* Avoid triggering the temporary warning in __get_user_pages */ - if (!vma_lookup(mm, addr) && !expand_stack(mm, addr)) + addr =3D untagged_addr_remote_unlocked(mm, addr); + + have_mmap_lock =3D remote_access_lock(mm, addr, len, gup_flags, &vma); + if (IS_ERR(vma)) { + *err =3D -EFAULT; return 0; + } =20 - /* ignore errors, just check how much was successfully transferred */ while (len) { - int bytes, offset; - void *maddr; - struct folio *folio; - struct vm_area_struct *vma =3D NULL; - struct page *page =3D get_user_page_lookup_vma(mm, addr, - gup_flags, &vma); + unsigned int foll_flags =3D gup_flags; + struct page *page; + int ret; =20 - if (IS_ERR(page)) { - /* We might need to expand the stack to access it */ + if (!vma || addr >=3D vma->vm_end) { + /* Any lookup here holds the mmap lock. */ + VM_BUG_ON(!have_mmap_lock); vma =3D vma_lookup(mm, addr); - if (!vma) { + if (!vma && can_expand_stack) { + /* expand_stack() drops the mmap lock if it fails */ vma =3D expand_stack(mm, addr); - - /* mmap_lock was dropped on failure */ if (!vma) - return buf - old_buf; - - /* Try again if stack expansion worked */ - continue; + have_mmap_lock =3D false; } + if (!vma) { + *err =3D -EFAULT; + break; + } + } =20 + /* + * FOLL_UNLOCKABLE lets the per-VMA fault retry, dropping the + * lock, so the walk can fall back to the mmap lock. + */ + if (!have_mmap_lock) + foll_flags |=3D FOLL_VMA_LOCK | FOLL_UNLOCKABLE; + + page =3D get_user_page_vma(vma, addr, foll_flags); + if (IS_ERR(page)) { /* - * Check if this is a VM_IO | VM_PFNMAP VMA, which - * we can access using slightly different code. + * get_user_page_vma() returns -EAGAIN, with the per-VMA + * lock released, for anything it could not finish under + * it; retake the mmap lock and retry. A different error + * therefore only arrives under the mmap lock, where + * struct-page-less memory can be reached via ->access(). */ - bytes =3D 0; -#ifdef CONFIG_HAVE_IOREMAP_PROT - if (vma->vm_ops && vma->vm_ops->access) - bytes =3D vma->vm_ops->access(vma, addr, buf, - len, write); -#endif - if (bytes <=3D 0) - break; - } else { - folio =3D page_folio(page); - bytes =3D len; - offset =3D addr & (PAGE_SIZE-1); - if (bytes > PAGE_SIZE-offset) - bytes =3D PAGE_SIZE-offset; - - maddr =3D kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SI= ZE); - if (write) { - copy_to_user_page(vma, page, addr, - maddr + offset, buf, bytes); - folio_mark_dirty_lock(folio); - } else { - copy_from_user_page(vma, page, addr, - buf, maddr + offset, bytes); + if (PTR_ERR(page) =3D=3D -EAGAIN) { + vma =3D NULL; + if (mmap_read_lock_killable(mm)) { + *err =3D -EFAULT; + break; + } + have_mmap_lock =3D true; + continue; } - folio_release_kmap(folio, maddr); + if (WARN_ON_ONCE(!have_mmap_lock)) + break; + page =3D NULL; } - len -=3D bytes; - buf +=3D bytes; - addr +=3D bytes; + + ret =3D action(vma, page, addr, &buf, len, write); + if (ret <=3D 0) { + if (ret < 0) + *err =3D ret; + break; + } + addr +=3D ret; + len -=3D ret; } - mmap_read_unlock(mm); + + remote_access_unlock(mm, vma, have_mmap_lock); =20 return buf - old_buf; } =20 +/* + * Copy one page's worth of [@addr, @addr + @len) to or from *@buf. Reaches + * struct-page-less VM_IO / VM_PFNMAP memory through vma->vm_ops->access(). + */ +static int access_vm_page(struct vm_area_struct *vma, struct page *page, + unsigned long addr, void **buf, int len, int write) +{ + struct folio *folio; + int bytes, offset; + void *maddr; + + if (!page) { + bytes =3D access_remote_vma_ops(vma, addr, *buf, len, write); + if (bytes > 0) + *buf +=3D bytes; + return bytes; + } + + bytes =3D len; + offset =3D addr & (PAGE_SIZE - 1); + if (bytes > PAGE_SIZE - offset) + bytes =3D PAGE_SIZE - offset; + + folio =3D page_folio(page); + maddr =3D kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE= ); + if (write) { + copy_to_user_page(vma, page, addr, maddr + offset, *buf, bytes); + folio_mark_dirty_lock(folio); + } else { + copy_from_user_page(vma, page, addr, *buf, maddr + offset, bytes); + } + folio_release_kmap(folio, maddr); + + *buf +=3D bytes; + return bytes; +} + +/* + * Access another process' address space as given in mm. + */ +static int __access_remote_vm(struct mm_struct *mm, unsigned long addr, + void *buf, int len, unsigned int gup_flags) +{ + int err; + + return remote_vm_walk(mm, addr, buf, len, gup_flags, true, + access_vm_page, &err); +} + /** * access_remote_vm - access another process' address space * @mm: the mm_struct of the target address space --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D5D5470131; Fri, 24 Jul 2026 22:30:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932217; cv=none; b=kLyu1pYTks/Uz0QRd3iGkwsjDfQ6V2TcvsIdGpPzB1X8dK6D8kdvtqtVCYHn4pPJtsUW5mAhBrJeVxm3ogf6MCZLxSv9b5J98SHnPEB7B0mxExBg7LuGJQt8+7Ydr7oNxqnPsZtz0Rw2fC+UByv+t2Zc80CJl+YZQE6Jl+8jWHI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932217; c=relaxed/simple; bh=F+4QHVMTdUxWVRG4L1XBerMtnFJQb/EMdPg2rQqLd68=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PlipCO5xgGg0/x07cdJ1w5t1iaoPI7pDRX1p4HUDKBIhko7ZXykXNfpVneb9c2v0X0fSCYtXqHgzAD4TiAUMjoxU7zd/ajN1xdxOtp0tijZqI8iM2AcBRXWVqCUIa7FOGivoqZC6MecbyD6xZdrLBTrFpUYWrvvLahe2K60P1Sg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=MS9beTn9; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="MS9beTn9" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=laFb2+2OJHBjt42yZcZ3hsZQhNcm9lH5QpuUTZr6hlo=; b=MS9beTn9E8RTJJ4e2NxUs4APQF lAXN0Zcyhkx0y+7Mtv/RePRaBeHwPGyHKX5pv22oE72E+K3e2CemBHeOJQ9PfLCo/p58tVLwnJIAf rV4awksMHY2ke6Lar0t7F7nBQEmsaHCdS4g0vM7Jm8fSYiL9FSVy8o6AWz03JvGdI93FHR8NU6YK9 PgHLj0T4iLAF9pqDWeSR/4GWTTlGKBm7U70OsI6A75EWGs7LGXi0x4MRr/g4r6Fus+Cfsi3RujiBg BE1K/icV7oT4ieExMWcBru/LRH81jK04qt/z40d1zgRltnZXmSiMAEp2LBrmvW4kg0HpsyfNRtfR/ BoUcWQYQ==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-1eMx; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel , bpf@vger.kernel.org Subject: [PATCH RFC v4 07/12] mm: read remote strings under the per-VMA lock Date: Fri, 24 Jul 2026 18:29:29 -0400 Message-ID: <20260724222934.1463812-8-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __copy_remote_vm_str() reads another process's memory under the mmap read lock for the whole copy, and looks the VMA up again for every page through get_user_page_lookup_vma(). This is the mmap-lock-only form that __access_remote_vm() used before it moved to the per-VMA lock, and it shares the same contention: an mmap() or munmap() stalls a bpf_copy_from_user_task_str() reader even though the string is resident. Read the string through remote_vm_walk(), the walk __access_remote_vm() already uses, with a copy_vm_str() action that strscpy()s each page and stops at the NUL. Bounding the copy by its maximum length keeps the per-VMA path inside a single VMA, so no VMA is looked up again while that lock is held. A copy the per-VMA lock cannot finish falls back to the mmap lock: a fault that dropped the lock, or a string running past the VMA. Memory with no struct page (VM_IO/VM_PFNMAP) and stack expansion stay on the mmap lock path and still return -EFAULT, as before; ->access() support is left for later. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Rik van Riel --- mm/memory.c | 111 ++++++++++++++++++++++------------------------------ 1 file changed, 47 insertions(+), 64 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 273dfe12bc6d..aaf620017f34 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -7296,84 +7296,67 @@ EXPORT_SYMBOL_GPL(access_process_vm); =20 #ifdef CONFIG_BPF_SYSCALL /* - * Copy a string from another process's address space as given in mm. - * If there is any error return -EFAULT. + * Copy a NUL-terminated string from @addr into *@buf, up to @len bytes, + * stopping at the NUL. strscpy() always NUL terminates, so recopy the last + * byte of a page when more pages follow. A string is never read from + * struct-page-less VM_IO / VM_PFNMAP memory. */ -static int __copy_remote_vm_str(struct mm_struct *mm, unsigned long addr, - void *buf, int len, unsigned int gup_flags) +static int copy_vm_str(struct vm_area_struct *vma, struct page *page, + unsigned long addr, void **buf, int len, int write) { - void *old_buf =3D buf; - int err =3D 0; - - *(char *)buf =3D '\0'; + struct folio *folio; + int bytes, offset, retval; + void *maddr; =20 - if (mmap_read_lock_killable(mm)) + if (!page) return -EFAULT; =20 - addr =3D untagged_addr_remote(mm, addr); + bytes =3D len; + offset =3D addr & (PAGE_SIZE - 1); + if (bytes > PAGE_SIZE - offset) + bytes =3D PAGE_SIZE - offset; =20 - /* Avoid triggering the temporary warning in __get_user_pages */ - if (!vma_lookup(mm, addr)) { - err =3D -EFAULT; - goto out; + folio =3D page_folio(page); + maddr =3D kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZE= ); + retval =3D strscpy(*buf, maddr + offset, bytes); + if (retval >=3D 0) { + /* Found the end of the string. */ + *buf +=3D retval; + folio_release_kmap(folio, maddr); + return 0; } =20 - while (len) { - int bytes, offset, retval; - void *maddr; - struct folio *folio; - struct page *page; - struct vm_area_struct *vma =3D NULL; + *buf +=3D bytes - 1; + if (bytes !=3D len) { + copy_from_user_page(vma, page, addr + bytes - 1, *buf, + maddr + (PAGE_SIZE - 1), 1); + *buf +=3D 1; + } + folio_release_kmap(folio, maddr); =20 - page =3D get_user_page_lookup_vma(mm, addr, gup_flags, &vma); - if (IS_ERR(page)) { - /* - * Treat as a total failure for now until we decide how - * to handle the CONFIG_HAVE_IOREMAP_PROT case and - * stack expansion. - */ - *(char *)buf =3D '\0'; - err =3D -EFAULT; - goto out; - } + return bytes; +} =20 - folio =3D page_folio(page); - bytes =3D len; - offset =3D addr & (PAGE_SIZE - 1); - if (bytes > PAGE_SIZE - offset) - bytes =3D PAGE_SIZE - offset; - - maddr =3D kmap_local_folio(folio, folio_page_idx(folio, page) * PAGE_SIZ= E); - retval =3D strscpy(buf, maddr + offset, bytes); - if (retval >=3D 0) { - /* Found the end of the string */ - buf +=3D retval; - folio_release_kmap(folio, maddr); - break; - } +/* + * Copy a string from another process's address space as given in mm. + * If there is any error return -EFAULT. + */ +static int __copy_remote_vm_str(struct mm_struct *mm, unsigned long addr, + void *buf, int len, unsigned int gup_flags) +{ + int bytes, err; =20 - buf +=3D bytes - 1; - /* - * Because strscpy always NUL terminates we need to - * copy the last byte in the page if we are going to - * load more pages - */ - if (bytes !=3D len) { - addr +=3D bytes - 1; - copy_from_user_page(vma, page, addr, buf, maddr + (PAGE_SIZE - 1), 1); - buf +=3D 1; - addr +=3D 1; - } - len -=3D bytes; + *(char *)buf =3D '\0'; =20 - folio_release_kmap(folio, maddr); + bytes =3D remote_vm_walk(mm, addr, buf, len, gup_flags, false, + copy_vm_str, &err); + if (err) { + /* The contract guarantees a terminated buffer even on error. */ + ((char *)buf)[bytes] =3D '\0'; + return err; } =20 -out: - mmap_read_unlock(mm); - if (err) - return err; - return buf - old_buf; + return bytes; } =20 /** --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 94A70471249; Fri, 24 Jul 2026 22:30:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932214; cv=none; b=IPdt4dVtTZn+/6lz1pasLMoIEhDfrFNUlGMvIa4pfQ20GH6cZ/D9ys4BorKYy3P+oPo/f39XA/WLEeASXT1Bh+DZ2lNpDLSfLYwzSnpj4FWMgSwHnVSI4Tg48i5DIXWQW2jO1fGf3KsJsWR1V3bzfjPTnAIZDiJVUULubQIZIk4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932214; c=relaxed/simple; bh=8/w2kfEV4dpxs1e04E9KZKjsCZ7Gf858pYTInPu8Xdk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JVyRJ7CjK/E3caMzGOIJV342OVZBvATw/L6XA1rnWX1Vy/XpgKHDVgJTQ8DoElGoHP3Ms5Un6S76q1y8mM9AVaBq3W26fEkcqWGM146I8J038MPMbDilbENdA7k2UlnFGXZYy+hdtdEQGRekYzSlFsreJeXB13BsWzVtCk1nO1Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=lDGkdWSg; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="lDGkdWSg" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=phv1B9Yxat6YIEopEPUTNjYYzL54eFSY9r4GoCdruM0=; b=lDGkdWSgenq2WUBSSLwgIoU2bb iq5fO5pwGwjNj4bxQlt4EaiKdpGhKGWYtAlH8V4HscBe+rLdz3LpbJ1rz4Eo+8VHi5Cf2YozwpoKa /EOf7b6Fj1trVqdHRewh1VGmUqt2Uelwy48tNNXJNpUrXhxN2zVXjdupFpQVDCV2E7dnQLmXL8+iR L2uoBhWRMCi66UhaWFlC12UuNvV8vGwQzSm9dIYb5f6FEjYNYvE+3V+Vz3IqDYqTDL0zb61Ac86UM /peYQvZJXj0L62/5Os+HC3VsJDdWWxk6zGgLEgSYvF8x6yeOIqsGogSrLBPrd15tBYdhgtMl7L8QJ 6fXo9UZA==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-1kjG; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel , Shuah Khan , linux-kselftest@vger.kernel.org Subject: [PATCH RFC v4 08/12] selftests/mm: cover /proc/pid/mem access to VM_PFNMAP memory Date: Fri, 24 Jul 2026 18:29:30 -0400 Message-ID: <20260724222934.1463812-9-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Reading a VM_PFNMAP mapping through /proc/pid/mem exercises __access_remote_vm() two ways: a COWed page has a struct page and is returned by get_user_page_vma(), while a raw PFN has none and is reached through vma->vm_ops->access(). Add two tests to pfnmap.c, both reading VM_PFNMAP memory through /proc/self/mem. procmem_cow_read maps the file MAP_PRIVATE and writable, writes to COW a page, then reads it back. Without the struct-page path in get_user_page_vma() this read is short: the access falls back to generic_access_phys(), which ioremaps the PFN, and ioremap of a COWed RAM page is rejected. procmem_pfn_read reads a raw PFN back through ->access(). ioremap rejects RAM, so it runs only for genuine device memory and is skipped for the default /dev/mem System RAM target. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Rik van Riel --- tools/testing/selftests/mm/pfnmap.c | 66 +++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/tools/testing/selftests/mm/pfnmap.c b/tools/testing/selftests/= mm/pfnmap.c index 4f550822385a..6ff5d1029517 100644 --- a/tools/testing/selftests/mm/pfnmap.c +++ b/tools/testing/selftests/mm/pfnmap.c @@ -31,6 +31,7 @@ static sigjmp_buf sigjmp_buf_env; static char *file =3D "/dev/mem"; static off_t file_offset; static int fd; +static int target_is_ram; =20 static void signal_handler(int sig) { @@ -113,6 +114,7 @@ static void pfnmap_init(void) if (err) ksft_exit_skip("Cannot find ram target in '/proc/iomem': %s\n", strerror(-err)); + target_is_ram =3D 1; } else { file_offset =3D 0; } @@ -271,6 +273,70 @@ TEST_F(pfnmap, fork) ASSERT_EQ(ret, 0); } =20 +TEST_F(pfnmap, procmem_cow_read) +{ + char *priv, *buf; + ssize_t rc; + int mem_fd; + + /* + * A COWed page in a VM_PFNMAP mapping has a struct page, so reading it + * through /proc/self/mem -- __access_remote_vm() -> get_user_page_vma() + * -- returns it directly, instead of routing to vma->vm_ops->access(), + * which ioremaps the PFN and cannot reach a COWed RAM page. + * + * Map the file MAP_PRIVATE and writable, write to COW a page into anon + * memory, then read the page back through /proc/self/mem. + */ + self->size2 =3D self->pagesize; + self->addr2 =3D mmap(NULL, self->size2, PROT_READ | PROT_WRITE, + MAP_PRIVATE, fd, file_offset); + if (self->addr2 =3D=3D MAP_FAILED) + SKIP(return, "Cannot create a writable private pfnmap mapping"); + priv =3D self->addr2; + + /* COW the page and stamp known bytes into the anon copy. */ + priv[0] =3D 0x42; + priv[self->pagesize - 1] =3D 0x24; + + buf =3D malloc(self->pagesize); + ASSERT_NE(buf, NULL); + + mem_fd =3D open("/proc/self/mem", O_RDONLY); + ASSERT_GE(mem_fd, 0); + rc =3D pread(mem_fd, buf, self->pagesize, (off_t)(uintptr_t)priv); + close(mem_fd); + + ASSERT_EQ(rc, (ssize_t)self->pagesize); + EXPECT_EQ(buf[0], 0x42); + EXPECT_EQ(buf[self->pagesize - 1], 0x24); + + free(buf); +} + +TEST_F(pfnmap, procmem_pfn_read) +{ + char buf[64]; + ssize_t rc; + int mem_fd; + + /* + * A raw PFN of a VM_IO/VM_PFNMAP mapping has no struct page, so + * __access_remote_vm() reaches it through vma->vm_ops->access() + * (generic_access_phys()). That ioremaps the PFN, which is rejected for + * RAM, so this only applies to genuine device memory. + */ + if (target_is_ram) + SKIP(return, "Target is System RAM; ->access() cannot ioremap RAM"); + + mem_fd =3D open("/proc/self/mem", O_RDONLY); + ASSERT_GE(mem_fd, 0); + rc =3D pread(mem_fd, buf, sizeof(buf), (off_t)(uintptr_t)self->addr1); + close(mem_fd); + + ASSERT_EQ(rc, (ssize_t)sizeof(buf)); +} + int main(int argc, char **argv) { for (int i =3D 1; i < argc; i++) { --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 94944471244 for ; Fri, 24 Jul 2026 22:30:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932216; cv=none; b=kBBVjssQWmCPNZ1lvYAJZr85bZyx9YuH+k8tDnkCsx2XrQ3qz6j6QwXs99e7W7U5fuIS8Sk9LnRfgj7l6crpZ7qI4oYYRyOQ1JOtIkA70y2kXdk8CfpiZ/PTpPq9HpUROYnZFZujCI7xq733oKKZ0EhhBXPMLTUIzvpLRY0N8s0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932216; c=relaxed/simple; bh=zUs2junJ53RLXCIf8fipO1HujNYSssD1Mj7OidARQgU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hij1CGBPcuty57rJ/orRaOqqwxtb5EG3oSVwz92i0R165givQ8Kwn0wQhKuJy5V1loHPDAprfdLabdvrNJ5gsBsoAns9As/SmUHv0JX5oz0fqEZmCZ1mBObhqK2e+qJHHNdlZfJ7SH2G/CDkXpOpqOGRWF0GtZnYDGfw2nrGDAU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=JcETxnSL; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="JcETxnSL" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=tJmogI2ctJBh6SOVkxDE96kh4IjYWZ9BC1mV32p1UTU=; b=JcETxnSL13yfACuT5wJax44zN/ afXj97fG6rxSc4BWLIay+Y+tZIvC7hP4oXzrCa+FyZx8pEAxJvp+33ydfnTYscdjvM7MfJwgTxHQK 2LrjzIyde9cwAU8/bWQ4NypPTORvqUjoP4vsimMZr0UFALlgSRsBDZO0+AaU2773SRylr8Qard72R MZ3sUBNHUmv+v2qNvxULOIwbMjPWR/TnNVea+xkj6ES12y9zhfNhPM5NgpM6b3qOwlu4YSQ8Ne0FK HhANRkEHdiX0qFzctkUUXErkHpyLhtIAhJAw8B1pFP5f2XsjfeFP4LURMtNZxc44WeytiTkmEvAcp kzC47CpQ==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-1rBH; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel Subject: [PATCH RFC v4 09/12] mm/gup: build get_user_page_lookup_vma() on get_user_page_vma() Date: Fri, 24 Jul 2026 18:29:31 -0400 Message-ID: <20260724222934.1463812-10-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" get_user_page_lookup_vma() faults in one page of a remote mm and returns it together with the VMA that covers it. It open-codes that with get_user_pages_remote() followed by vma_lookup(), duplicating the single-page walk that get_user_page_vma() now provides. Every caller already holds the mmap lock, and the helper needs it anyway: get_user_pages_remote(locked=3DNULL) and vma_lookup() both rely on it. Look the VMA up under the held mmap lock and fault the page in through get_user_page_vma() without FOLL_VMA_LOCK, so the lock is held for the whole call and never dropped. Force FOLL_REMOTE and FOLL_TOUCH to preserve the foreign-access permission check and accessed-bit behavior that get_user_pages_remote() applied before. Add mmap_assert_locked() so the lock requirement is documented and any future caller that forgets it trips the assertion rather than walking page tables unlocked. Reject FOLL_UNLOCKABLE alongside the existing FOLL_NOWAIT check: both let the fault handler drop the mmap lock, which would invalidate the VMA looked up here. get_user_page_vma() passes neither, so the lock is held for the whole walk and the returned VMA stays valid. The one behavioral change is that the old path could drop and retake the mmap lock across a fault, while get_user_page_vma() holds it throughout. None of the callers rely on the lock being dropped. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Rik van Riel --- include/linux/mm.h | 32 +++----------------------------- mm/gup.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+), 29 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 24ead14b4790..0f66d76e6ca7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3235,35 +3235,9 @@ long pin_user_pages_remote(struct mm_struct *mm, unsigned int gup_flags, struct page **pages, int *locked); =20 -/* - * Retrieves a single page alongside its VMA. Does not support FOLL_NOWAIT. - */ -static inline struct page *get_user_page_lookup_vma(struct mm_struct *mm, - unsigned long addr, - int gup_flags, - struct vm_area_struct **vmap) -{ - struct page *page; - struct vm_area_struct *vma; - int got; - - if (WARN_ON_ONCE(unlikely(gup_flags & FOLL_NOWAIT))) - return ERR_PTR(-EINVAL); - - got =3D get_user_pages_remote(mm, addr, 1, gup_flags, &page, NULL); - - if (got < 0) - return ERR_PTR(got); - - vma =3D vma_lookup(mm, addr); - if (WARN_ON_ONCE(!vma)) { - put_page(page); - return ERR_PTR(-EINVAL); - } - - *vmap =3D vma; - return page; -} +struct page *get_user_page_lookup_vma(struct mm_struct *mm, unsigned long = addr, + int gup_flags, + struct vm_area_struct **vmap); =20 long get_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages); diff --git a/mm/gup.c b/mm/gup.c index cb7245b7542f..7f4107f7ff10 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1288,6 +1288,41 @@ struct page *get_user_page_vma(struct vm_area_struct= *vma, unsigned long addr, return ERR_PTR(ret); } =20 +/* + * get_user_page_lookup_vma - fault in one page of a remote mm and hand ba= ck the + * page along with the VMA that covers it. The caller must hold the mmap_l= ock. + * Returns with the mmap_lock still held. + * + * Looks up the VMA, and gets a reference to the page through + * get_user_page_vma(), faulting in the page if needed. + * + * FOLL_NOWAIT and FOLL_UNLOCKABLE are rejected: both let the fault handler + * drop the mmap lock, which could invalidate the looked-up VMA. + */ +struct page *get_user_page_lookup_vma(struct mm_struct *mm, unsigned long = addr, + int gup_flags, + struct vm_area_struct **vmap) +{ + struct vm_area_struct *vma; + struct page *page; + + if (WARN_ON_ONCE(unlikely(gup_flags & (FOLL_NOWAIT | FOLL_UNLOCKABLE)))) + return ERR_PTR(-EINVAL); + + mmap_assert_locked(mm); + + vma =3D vma_lookup(mm, addr); + if (!vma) + return ERR_PTR(-EFAULT); + + page =3D get_user_page_vma(vma, addr, gup_flags | FOLL_REMOTE | FOLL_TOUC= H); + if (IS_ERR(page)) + return page; + + *vmap =3D vma; + return page; +} + /* * Writing to file-backed mappings which require folio dirty tracking usin= g GUP * is a fundamentally broken operation, as kernel write access to GUP mapp= ings --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5D2647010B for ; Fri, 24 Jul 2026 22:30:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932216; cv=none; b=HTmuzgoocS03BoHYYvJ5weoG6u7+RYZpijUd/sQr+wrlo8MVoL6DtITXTr6Ro/q2XSS4Sx03Nlr3OSGgjpRf1/8IqKi+9v+qB02RNeDxL1NoPenNoNufd1DxyvCsZpplD4DDro+P8eRRD7nYjQlF9oJ+MWSuWvxrEB21yqO1+Zw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932216; c=relaxed/simple; bh=u8ZmOtr7v7kOmoXtHsr92G52DzHZNi7awWkrO/4xaWs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JRAl5ltdQiNRaax0rJNKs8OiolT4LNqDOX+NhDU8hTyYBTfQzGXota72dH0LsroWWBY8UBbCH05JdpPN2YOfPA+BIH3tPhsl2j6TvIDQKyxnDo/bTkQBny5UCdzCPNJFExVUfhUP96DSIYb8KS0XxG8QfLQcaTI2yBxU1OXZSNM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=EfkmCcu6; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="EfkmCcu6" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=uWyDOLxEr139JST0LPn3P6hJ2pi3B91Laa+5/sQT2jw=; b=EfkmCcu6UjqwvmHokFq/BB/q+Q gPqMopzT/+YNSB6yVdiT/pKeZDWxT7f8LDs2kaCaCfHxpsuhsSrwUnuD50CJB4hYFrRagxEH5Th7F hVnbCgzJG7ECtC6Y4JnrE7GJutVka3iswTkbIDFvPVcdz35kQrgqvvw4Tfg3V/HUAHkZ1AbYZvRYc rTpIs7UUU8oxCw8mWiYUKfj/YxBUgQcTdBKWB7ZdRGaChZb4v+B0Lf3BEFmfbMDUy3A81oXrh3EI8 svZ9pU1efMXiuFt5yeLXUd5yoCXgb3t9+3PYb+nQe6b5ACYLw0Z/pzA2F3zY+GWVs5Z8ynhNt4Vl6 caekI+YQ==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-1xxW; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel Subject: [PATCH RFC v4 10/12] mm/gup: pass an end address to follow_page_mask() and return a page count Date: Fri, 24 Jul 2026 18:29:32 -0400 Message-ID: <20260724222934.1463812-11-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" follow_page_mask() reports how many pages the returned page covers with a page_mask: a bitmask of the enclosing naturally-aligned huge page. Callers turn that into a stride, which assumes the run is a power-of-two block aligned to its own size. That form cannot describe an arbitrary contiguous run, which a later patch needs to batch PTE-mapped large folios. Replace the page_mask output with nr_pages, the number of contiguous pages the returned page covers starting at @address, and add an @end argument bounding how far the walk looks, so a small request does not scan an entire large folio. The huge PMD and PUD paths report the same stride as before, now as a count clamped to @end. __get_user_pages() uses the count directly. get_user_page_vma() passes @address + PAGE_SIZE and still returns one page. No functional change intended. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Rik van Riel --- mm/gup.c | 102 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 56 insertions(+), 46 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 7f4107f7ff10..96b00ddf7e04 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -647,8 +647,9 @@ static inline bool can_follow_write_pud(pud_t pud, stru= ct page *page, } =20 static struct page *follow_huge_pud(struct vm_area_struct *vma, - unsigned long addr, pud_t *pudp, - int flags, unsigned long *page_mask) + unsigned long addr, unsigned long end, + pud_t *pudp, int flags, + unsigned long *nr_pages) { struct mm_struct *mm =3D vma->vm_mm; struct page *page; @@ -672,10 +673,13 @@ static struct page *follow_huge_pud(struct vm_area_st= ruct *vma, return ERR_PTR(-EMLINK); =20 ret =3D try_grab_folio(page_folio(page), 1, flags); - if (ret) + if (ret) { page =3D ERR_PTR(ret); - else - *page_mask =3D HPAGE_PUD_NR - 1; + } else { + unsigned long off =3D (addr & ~PUD_MASK) >> PAGE_SHIFT; + + *nr_pages =3D min(HPAGE_PUD_NR - off, (end - addr) >> PAGE_SHIFT); + } =20 return page; } @@ -699,9 +703,9 @@ static inline bool can_follow_write_pmd(pmd_t pmd, stru= ct page *page, } =20 static struct page *follow_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, pmd_t *pmd, - unsigned int flags, - unsigned long *page_mask) + unsigned long addr, unsigned long end, + pmd_t *pmd, unsigned int flags, + unsigned long *nr_pages) { struct mm_struct *mm =3D vma->vm_mm; pmd_t pmdval =3D *pmd; @@ -738,23 +742,25 @@ static struct page *follow_huge_pmd(struct vm_area_st= ruct *vma, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ =20 page +=3D (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT; - *page_mask =3D HPAGE_PMD_NR - 1; + *nr_pages =3D min(HPAGE_PMD_NR - ((addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT), + (end - addr) >> PAGE_SHIFT); =20 return page; } =20 #else /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ static struct page *follow_huge_pud(struct vm_area_struct *vma, - unsigned long addr, pud_t *pudp, - int flags, unsigned long *page_mask) + unsigned long addr, unsigned long end, + pud_t *pudp, int flags, + unsigned long *nr_pages) { return NULL; } =20 static struct page *follow_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, pmd_t *pmd, - unsigned int flags, - unsigned long *page_mask) + unsigned long addr, unsigned long end, + pmd_t *pmd, unsigned int flags, + unsigned long *nr_pages) { return NULL; } @@ -800,7 +806,8 @@ static inline bool can_follow_write_pte(pte_t pte, stru= ct page *page, } =20 static struct page *follow_page_pte(struct vm_area_struct *vma, - unsigned long address, pmd_t *pmd, unsigned int flags) + unsigned long address, unsigned long end, pmd_t *pmd, + unsigned int flags, unsigned long *nr_pages) { struct mm_struct *mm =3D vma->vm_mm; struct folio *folio; @@ -885,6 +892,7 @@ static struct page *follow_page_pte(struct vm_area_stru= ct *vma, */ folio_mark_accessed(folio); } + out: pte_unmap_unlock(ptep, ptl); return page; @@ -896,9 +904,9 @@ static struct page *follow_page_pte(struct vm_area_stru= ct *vma, } =20 static struct page *follow_pmd_mask(struct vm_area_struct *vma, - unsigned long address, pud_t *pudp, - unsigned int flags, - unsigned long *page_mask) + unsigned long address, unsigned long end, + pud_t *pudp, unsigned int flags, + unsigned long *nr_pages) { pmd_t *pmd, pmdval; spinlock_t *ptl; @@ -912,7 +920,7 @@ static struct page *follow_pmd_mask(struct vm_area_stru= ct *vma, if (!pmd_present(pmdval)) return no_page_table(vma, flags, address); if (likely(!pmd_leaf(pmdval))) - return follow_page_pte(vma, address, pmd, flags); + return follow_page_pte(vma, address, end, pmd, flags, nr_pages); =20 if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) return no_page_table(vma, flags, address); @@ -925,24 +933,24 @@ static struct page *follow_pmd_mask(struct vm_area_st= ruct *vma, } if (unlikely(!pmd_leaf(pmdval))) { spin_unlock(ptl); - return follow_page_pte(vma, address, pmd, flags); + return follow_page_pte(vma, address, end, pmd, flags, nr_pages); } if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) { spin_unlock(ptl); split_huge_pmd(vma, pmd, address); /* If pmd was left empty, stuff a page table in there quickly */ return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) : - follow_page_pte(vma, address, pmd, flags); + follow_page_pte(vma, address, end, pmd, flags, nr_pages); } - page =3D follow_huge_pmd(vma, address, pmd, flags, page_mask); + page =3D follow_huge_pmd(vma, address, end, pmd, flags, nr_pages); spin_unlock(ptl); return page; } =20 static struct page *follow_pud_mask(struct vm_area_struct *vma, - unsigned long address, p4d_t *p4dp, - unsigned int flags, - unsigned long *page_mask) + unsigned long address, unsigned long end, + p4d_t *p4dp, unsigned int flags, + unsigned long *nr_pages) { pud_t *pudp, pud; spinlock_t *ptl; @@ -955,7 +963,7 @@ static struct page *follow_pud_mask(struct vm_area_stru= ct *vma, return no_page_table(vma, flags, address); if (pud_leaf(pud)) { ptl =3D pud_lock(mm, pudp); - page =3D follow_huge_pud(vma, address, pudp, flags, page_mask); + page =3D follow_huge_pud(vma, address, end, pudp, flags, nr_pages); spin_unlock(ptl); if (page) return page; @@ -964,13 +972,13 @@ static struct page *follow_pud_mask(struct vm_area_st= ruct *vma, if (unlikely(pud_bad(pud))) return no_page_table(vma, flags, address); =20 - return follow_pmd_mask(vma, address, pudp, flags, page_mask); + return follow_pmd_mask(vma, address, end, pudp, flags, nr_pages); } =20 static struct page *follow_p4d_mask(struct vm_area_struct *vma, - unsigned long address, pgd_t *pgdp, - unsigned int flags, - unsigned long *page_mask) + unsigned long address, unsigned long end, + pgd_t *pgdp, unsigned int flags, + unsigned long *nr_pages) { p4d_t *p4dp, p4d; =20 @@ -981,15 +989,16 @@ static struct page *follow_p4d_mask(struct vm_area_st= ruct *vma, if (!p4d_present(p4d) || p4d_bad(p4d)) return no_page_table(vma, flags, address); =20 - return follow_pud_mask(vma, address, p4dp, flags, page_mask); + return follow_pud_mask(vma, address, end, p4dp, flags, nr_pages); } =20 /** * follow_page_mask - look up a page descriptor from a user-virtual address * @vma: vm_area_struct mapping @address * @address: virtual address to look up + * @end: virtual address at which to stop batching contiguous pages * @flags: flags modifying lookup behaviour - * @page_mask: a pointer to output page_mask + * @nr_pages: output; number of contiguous pages the caller can read * * @flags can have FOLL_ flags set, defined in * @@ -998,15 +1007,17 @@ static struct page *follow_p4d_mask(struct vm_area_s= truct *vma, * trigger a fault with FAULT_FLAG_UNSHARE set. Note that unsharing is only * relevant with FOLL_PIN and !FOLL_WRITE. * - * On output, @page_mask is set according to the size of the page. + * On output, @nr_pages holds how many contiguous pages the folio that inc= ludes + * the returned page has mapped into this process, so the caller can advan= ce + * over a large folio in one step. * * Return: the mapped (struct page *), %NULL if no mapping exists, or * an error pointer if there is a mapping to something not represented * by a page descriptor (see also vm_normal_page()). */ static struct page *follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - unsigned long *page_mask) + unsigned long address, unsigned long end, + unsigned int flags, unsigned long *nr_pages) { pgd_t *pgd; struct mm_struct *mm =3D vma->vm_mm; @@ -1014,13 +1025,13 @@ static struct page *follow_page_mask(struct vm_area= _struct *vma, =20 vma_pgtable_walk_begin(vma); =20 - *page_mask =3D 0; + *nr_pages =3D 1; pgd =3D pgd_offset(mm, address); =20 if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) page =3D no_page_table(vma, flags, address); else - page =3D follow_p4d_mask(vma, address, pgd, flags, page_mask); + page =3D follow_p4d_mask(vma, address, end, pgd, flags, nr_pages); =20 vma_pgtable_walk_end(vma); =20 @@ -1198,7 +1209,7 @@ struct page *get_user_page_vma(struct vm_area_struct = *vma, unsigned long addr, unsigned int gup_flags) { bool vma_locked =3D gup_flags & FOLL_VMA_LOCK; - unsigned long page_mask; + unsigned long nr_pages; struct page *page; int locked =3D 1; bool pfnmap; @@ -1231,10 +1242,10 @@ struct page *get_user_page_vma(struct vm_area_struc= t *vma, unsigned long addr, } cond_resched(); =20 - /* follow_page_mask() requires @page_mask; it is unused here. */ - page =3D follow_page_mask(vma, addr, + /* This helper hands back a single page; cap the batch at one. */ + page =3D follow_page_mask(vma, addr, addr + PAGE_SIZE, gup_flags | FOLL_TOUCH | FOLL_GET, - &page_mask); + &nr_pages); if (!IS_ERR_OR_NULL(page)) { /* Match __get_user_pages(): flush for VIVT/aliasing caches. */ flush_anon_page(vma, page, addr); @@ -1529,7 +1540,6 @@ static long __get_user_pages(struct mm_struct *mm, { long ret =3D 0, i =3D 0; struct vm_area_struct *vma =3D NULL; - unsigned long page_mask =3D 0; =20 if (!nr_pages) return 0; @@ -1544,7 +1554,7 @@ static long __get_user_pages(struct mm_struct *mm, =20 do { struct page *page; - unsigned int page_increm; + unsigned long page_increm; =20 /* first iteration or cross vma bound */ if (!vma || start >=3D vma->vm_end) { @@ -1571,7 +1581,7 @@ static long __get_user_pages(struct mm_struct *mm, pages ? &page : NULL); if (ret) goto out; - page_mask =3D 0; + page_increm =3D 1; goto next_page; } =20 @@ -1594,7 +1604,8 @@ static long __get_user_pages(struct mm_struct *mm, } cond_resched(); =20 - page =3D follow_page_mask(vma, start, gup_flags, &page_mask); + page =3D follow_page_mask(vma, start, start + nr_pages * PAGE_SIZE, + gup_flags, &page_increm); if (!page || PTR_ERR(page) =3D=3D -EMLINK) { ret =3D faultin_page(vma, start, gup_flags, PTR_ERR(page) =3D=3D -EMLINK, locked); @@ -1627,7 +1638,6 @@ static long __get_user_pages(struct mm_struct *mm, goto out; } next_page: - page_increm =3D 1 + (~(start >> PAGE_SHIFT) & page_mask); if (page_increm > nr_pages) page_increm =3D nr_pages; =20 --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DEE946EF6E for ; Fri, 24 Jul 2026 22:30:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932218; cv=none; b=DALQS+95Ae/mDe9zdjIUgSJIk4+oczgS+j2uZU8+1fSMaVIstKVZnUQYOozd763eTSvNYYB2l6NbBEHvMDQDxXQqAq52MrfutbIPFb9QAEzyndkiYHHODcXIrhxEADtkzcrkXPtjxM+ZwN5Q8Shz6cALr+BbutufNb4pkcurUfE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932218; c=relaxed/simple; bh=0AM3kERdoP79xy3BLW7VnaebqOk3m38Tp33w3HtyIhk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dHOrCl5Taplg8gQJ0Vsu9scGXwUsNPxbk0Z0LHl33qgd5HPOWDCOCg4yWYC5APmKbK11eP9drHZcMFq0OIYIW18P2Wog2eoctfsbsaRsL0YaKbUkQHS90LXgVcEEFnfmM2TI8jE1j2YNgcYf5sXXyyXtjYIuC4dDD50O3j/WleE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=OJksZob7; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="OJksZob7" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=/nMsMXQjXiyA5qAYl7QT0yp1WELxZnCfyNHaXsTe27c=; b=OJksZob72/FPZ/ELTbhArp8gh9 7UMXiNlaPpudxMXOBVJMUrfH+0aQhfhpbgKO5m3p6KJ+qYxLMuUeykJu14bVfe/Ju1UpbMLNT/Qpa JEKIKeIgeB81zgqkDkbdQOIqNwsfs0Zq4gmvITvR+9M8DH0FZS0HrBtO6PampJ6y2eQJDWsY38kvo 86CVxhLsA43csRlQ5zd8+eqyGIFWYVas87aTFTGQLAk4aPUxHkl2g91w/AJAY2uR0AN9gJyEcv0LZ Cqb0CMQqR4pl5ZjnUMPZSUfOZZSZ9ajq7qGIf5+2+h2c+Zqygt7/PaQ83o9STEM9CcLRVBnuztl5e 23N8lVDA==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-28o2; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel Subject: [PATCH RFC v4 11/12] mm/gup: batch contiguous PTE-mapped large folios in follow_page_mask() Date: Fri, 24 Jul 2026 18:29:33 -0400 Message-ID: <20260724222934.1463812-12-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" follow_page_mask() returns one page per call for a PTE-mapped large folio, so __get_user_pages() re-walks the page tables for every page of an mTHP even though the folio maps a contiguous run. The huge PMD and PUD paths already return the whole mapping in one step. Report the contiguous run for the PTE case too. follow_pte_batch() uses folio_pte_batch_flags() to count consecutive present PTEs that map consecutive pages of the same folio with a uniform write bit, bounded by the page table, @end, the VMA, and the folio itself. Keep the per-PTE guarantees that follow_page_pte() makes for the head page. folio_pte_batch_flags() with FPB_RESPECT_WRITE stops the run at a change in the write bit, so the whole run matches the head. A writable run is safe for any access: a writable anon page is exclusive, so gup_must_unshare() cannot fire, and FOLL_WRITE is satisfied. A read-only run is batched only for a plain read, since FOLL_WRITE would need a COW fault per page and FOLL_PIN would need a per-page gup_must_unshare() check. Measured with mm/gup_test.c (PIN_LONGTERM_BENCHMARK, the slow pin_user_pages() path) on a 256 MB MADV_HUGEPAGE anonymous region in a 4 CPU VM, median get time over 16 iterations. Each folio size was confirmed through the per-size anon_fault_alloc counters (4096 folios for 64 kB, 128 for 2 MB): gup_test -L -m 256 -n 65536 -r 16 -t before after 64 kB mTHP 3140 us 412 us (7.6x) 2 MB THP (control) 78 us 76 us 4 kB base (control) 3010 us 3042 us The PMD-mapped 2 MB THP already returns the whole mapping in one step, so it stays fast and unchanged. The 4 kB baseline shows the per-page walk cost that the 64 kB case paid before this change; only the PTE-mapped large folio case improves. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Rik van Riel --- mm/gup.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index 96b00ddf7e04..4910c93ba003 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -805,6 +805,43 @@ static inline bool can_follow_write_pte(pte_t pte, str= uct page *page, return !userfaultfd_pte_wp(vma, pte); } =20 +/* + * Count the pages, starting at @address and bounded by @end, that a PTE-m= apped + * large @folio maps contiguously and that can be returned together with t= he + * page at @address: consecutive present PTEs mapping consecutive pages of + * @folio with a uniform write bit, within this VMA and a single page tabl= e. + * Returns at least 1. + * + * gup_must_unshare() and the write-fault check are per PTE. A writable ru= n is + * always safe: a writable anon page is exclusive, and FOLL_WRITE is satis= fied. + * A read-only run is only safe for a plain read; FOLL_WRITE would need a = COW + * fault per page and FOLL_PIN would need a per-page gup_must_unshare() ch= eck, + * so those fall back to a single page. + */ +static unsigned long follow_pte_batch(struct vm_area_struct *vma, + unsigned long address, unsigned long end, struct folio *folio, + struct page *page, pte_t *ptep, pte_t pte, unsigned int flags) +{ + pte_t batch_pte =3D pte; + unsigned long max; + + if (!pte_write(pte) && (flags & (FOLL_WRITE | FOLL_PIN))) + return 1; + + /* + * folio_pte_batch_flags() scans forward from @ptep, so the run must + * stay within this page table: bound it by the PMD as well as @end and + * the VMA, since a large folio can be PTE-mapped across a PMD boundary. + */ + max =3D min((pmd_addr_end(address, end) - address) >> PAGE_SHIFT, + (vma->vm_end - address) >> PAGE_SHIFT); + if (max <=3D 1) + return 1; + + return folio_pte_batch_flags(folio, vma, ptep, &batch_pte, max, + FPB_RESPECT_WRITE); +} + static struct page *follow_page_pte(struct vm_area_struct *vma, unsigned long address, unsigned long end, pmd_t *pmd, unsigned int flags, unsigned long *nr_pages) @@ -893,6 +930,14 @@ static struct page *follow_page_pte(struct vm_area_str= uct *vma, folio_mark_accessed(folio); } =20 + /* + * A PTE-mapped large folio can be handed back as a contiguous batch, + * so the caller advances over the whole run in one step instead of + * walking the page tables for every page. + */ + if (folio_test_large(folio)) + *nr_pages =3D follow_pte_batch(vma, address, end, folio, page, + ptep, pte, flags); out: pte_unmap_unlock(ptep, ptl); return page; --=20 2.53.0-Meta From nobody Sat Jul 25 05:59:23 2026 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A3D7D471CE4; Fri, 24 Jul 2026 22:30:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=96.67.55.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932223; cv=none; b=gA+qC8J0kphJs27t0AJky3rElHOp3Fvyds/iQNoRFtFGnbeTvZTQqagD6579Wushlj+2tDKZe53rQWfaCqlSsxtGYZc0xnRhahpHZ691/atGunMLdSvGvC+ZMzwGL+q+KSYmw2YkmNN396Z6+9+kruBsD6kj4jL87pZDkoxNUdk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784932223; c=relaxed/simple; bh=12LwphxrahrvDgBEuP7aeSaCsbNQvpBj8hd/rK7JMZc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EpFAilRLEeWYiKn6joEFbh8XQIg1b3YlqcevrW1oSBnSUsL0cTpvhKy8N1iVmO7AwrwQZDqiEzjWkSaaI6TO1I4wI+AYGq2eK3XQ9yk8riU68k8hAFFSLj78eNAzoDyShkREP/i10ldUUjOMc3q76qM+DU6g3S6VIjL1TEDBnSw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com; spf=pass smtp.mailfrom=surriel.com; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b=j9VXakYJ; arc=none smtp.client-ip=96.67.55.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=surriel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=surriel.com header.i=@surriel.com header.b="j9VXakYJ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=surriel.com ; s=mail; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=QoasW0kcbLRMqvkHf+Serx6XJgI4VH5CVkhXEbwylBk=; b=j9VXakYJJ/PDqAUv2OdbSq8VPc aERUc+tTW4sVfW6m5IUzJR0guy+ydhBDDxWUluYcqM7Ez8y6hdimGk6KrqTs70NkVhRpuwOoHQNTu ZZFPqJl8o4utuNCC6WlF8tlaR4Ubj2Tjw00gPLLHnWPX6qdCkT0AxX0w/OqaGqTOFzfZw4nZdfstu xLIJ5Fcem+GwhEfkimifB3M4xGgfxPk59FCH6H7OwCGjg9ldqWmBsf0njYcEjFIuLlHsJL3NFrnli 2ZWQkHIA+9Gx53I7ymyXSAOSVoUw9mxYolW5RZrDGQtBj1iyOyEb8mGiLC/FUEeMRVyh2lwbiF3Ba jEqHE4Ig==; Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1wnOOs-0000000027h-2FDa; Fri, 24 Jul 2026 18:29:50 -0400 From: Rik van Riel To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com, Dave Hansen , Peter Zijlstra , Suren Baghdasaryan , Lorenzo Stoakes , Vlastimil Babka , David Hildenbrand , "Liam R. Howlett" , Mike Rapoport , Michal Hocko , Jason Gunthorpe , John Hubbard , Peter Xu , Matthew Wilcox , Usama Arif , Rik van Riel , Shuah Khan , linux-kselftest@vger.kernel.org Subject: [PATCH RFC v4 12/12] selftests/mm: add a slow-GUP content and COW test for mTHP Date: Fri, 24 Jul 2026 18:29:34 -0400 Message-ID: <20260724222934.1463812-13-riel@surriel.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260724222934.1463812-1-riel@surriel.com> References: <20260724222934.1463812-1-riel@surriel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" follow_page_mask() now batches a PTE-mapped large folio (mTHP) into one contiguous run for the slow get_user_pages() path. A mis-batched run would hand back the wrong pages or a stale COW copy, which the existing tests do not catch: gup_test checks only pin/unpin integrity, and cow exercises COW mostly at PMD size. Add mthp_gup_cow_test. It forces 64kB-only mTHP, writes a per-page-distinct pattern, then pins the region on the slow path (PIN_LONGTERM without USE_FAST) and compares the bytes the kernel copies back from the pinned pages against that pattern. The test covers a read pin, a write pin, and COW after fork(): a child write-pins to force per-page unshare and checks the copied contents, then rewrites its copy while the parent verifies its own contents are intact. It also reports how many pages sit in contiguous large-folio runs, so a kernel without mTHP shows light coverage rather than passing vacuously. Assisted-by: Claude:claude-opus-4.8 Signed-off-by: Rik van Riel --- tools/testing/selftests/mm/Makefile | 1 + .../testing/selftests/mm/mthp_gup_cow_test.c | 213 ++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 1 + 3 files changed, 215 insertions(+) create mode 100644 tools/testing/selftests/mm/mthp_gup_cow_test.c diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/= mm/Makefile index e6df968f0971..6b917a4f73e0 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -83,6 +83,7 @@ TEST_GEN_FILES +=3D mrelease_test TEST_GEN_FILES +=3D mremap_dontunmap TEST_GEN_FILES +=3D mremap_test TEST_GEN_FILES +=3D mseal_test +TEST_GEN_FILES +=3D mthp_gup_cow_test TEST_GEN_FILES +=3D on-fault-limit TEST_GEN_FILES +=3D pagemap_ioctl TEST_GEN_FILES +=3D pfnmap diff --git a/tools/testing/selftests/mm/mthp_gup_cow_test.c b/tools/testing= /selftests/mm/mthp_gup_cow_test.c new file mode 100644 index 000000000000..52ee329d8641 --- /dev/null +++ b/tools/testing/selftests/mm/mthp_gup_cow_test.c @@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Verify that the slow GUP path (pin_user_pages -> follow_page_mask -> + * follow_pte_batch) returns the correct pages for a PTE-mapped large folio + * (mTHP), including that COW copies produce the right content. + * + * Uses the CONFIG_GUP_TEST PIN_LONGTERM interface: START pins a range on = the + * slow path, READ copies the pinned pages' bytes back so we can compare t= hem + * against the pattern we wrote. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0])) + +#define GUP_DEV "/sys/kernel/debug/gup_test" + +#define PIN_LONGTERM_TEST_START _IOW('g', 7, struct pin_longterm_test) +#define PIN_LONGTERM_TEST_STOP _IO('g', 8) +#define PIN_LONGTERM_TEST_READ _IOW('g', 9, __u64) +#define USE_WRITE 1 +#define USE_FAST 2 + +struct pin_longterm_test { + __u64 addr; + __u64 size; + __u32 flags; +}; + +#define ORDER_KB 64 +#define NR_FOLIOS 16 +#define REGION ((size_t)ORDER_KB * 1024 * NR_FOLIOS) + +static long PS; +static int fails; +static int tap; + +static void ok(int cond, const char *desc) +{ + printf("%s %d %s\n", cond ? "ok" : "not ok", ++tap, desc); + if (!cond) + fails++; +} + +/* Deterministic, per-page-distinct pattern so any mis-order or leak shows= . */ +static void fill(char *base, size_t sz, uint32_t salt) +{ + for (size_t off =3D 0; off < sz; off +=3D PS) { + uint32_t k =3D off / PS; + uint64_t v =3D ((uint64_t)salt << 32) ^ (k * 0x9E3779B1u + 0x1234); + + for (size_t i =3D 0; i < PS; i +=3D sizeof(v)) + memcpy(base + off + i, &v, sizeof(v)); + } +} + +static int wsysfs(const char *path, const char *val) +{ + int fd =3D open(path, O_WRONLY); + + if (fd < 0) + return -1; + int r =3D write(fd, val, strlen(val)); + + close(fd); + return r < 0 ? -1 : 0; +} + +/* Force sub-PMD 64kB mTHP only, so faults produce PTE-mapped large folios= . */ +static void setup_mthp(void) +{ + const char *thp =3D "/sys/kernel/mm/transparent_hugepage"; + char p[256]; + static const int kb[] =3D { 16, 32, 64, 128, 256, 512, 1024, 2048 }; + + wsysfs("/sys/kernel/mm/transparent_hugepage/enabled", "never"); + for (unsigned int i =3D 0; i < ARRAY_SIZE(kb); i++) { + snprintf(p, sizeof(p), "%s/hugepages-%dkB/enabled", thp, kb[i]); + wsysfs(p, kb[i] =3D=3D ORDER_KB ? "always" : "never"); + } +} + +/* Count how many pages sit in a contiguous >=3DORDER_KB PFN run (via page= map). */ +static int count_large_pages(char *base, size_t sz) +{ + int pm =3D open("/proc/self/pagemap", O_RDONLY); + size_t n =3D sz / PS, large =3D 0; + uint64_t *pfn =3D calloc(n, sizeof(*pfn)); + + if (pm < 0) + return -1; + for (size_t k =3D 0; k < n; k++) { + uint64_t ent; + off_t idx =3D ((uintptr_t)base + k * PS) / PS * sizeof(ent); + + if (pread(pm, &ent, sizeof(ent), idx) !=3D sizeof(ent) || + !(ent & (1ULL << 63))) + pfn[k] =3D 0; + else + pfn[k] =3D ent & ((1ULL << 55) - 1); + } + close(pm); + for (size_t k =3D 0; k < n; k++) + if (k + 1 < n && pfn[k] && pfn[k + 1] =3D=3D pfn[k] + 1) + large++; + free(pfn); + return large; +} + +/* Pin @base..@sz on the slow path, read the pinned bytes back, compare to= exp. */ +static int pin_verify(int fd, char *base, size_t sz, uint32_t wr, char *ex= p) +{ + struct pin_longterm_test a =3D { + .addr =3D (uintptr_t)base, .size =3D sz, + .flags =3D wr ? USE_WRITE : 0, /* USE_FAST unset =3D> slow */ + }; + char *got =3D mmap(NULL, sz, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + __u64 ga =3D (uintptr_t)got; + int rc =3D -1; + + if (got =3D=3D MAP_FAILED) + return -1; + if (ioctl(fd, PIN_LONGTERM_TEST_START, &a)) { + fprintf(stderr, "START(%s) failed: %s\n", + wr ? "write" : "read", strerror(errno)); + goto out; + } + if (ioctl(fd, PIN_LONGTERM_TEST_READ, &ga)) { + fprintf(stderr, "READ failed: %s\n", strerror(errno)); + ioctl(fd, PIN_LONGTERM_TEST_STOP); + goto out; + } + ioctl(fd, PIN_LONGTERM_TEST_STOP); + rc =3D memcmp(got, exp, sz) ? 1 : 0; +out: + munmap(got, sz); + return rc; +} + +int main(void) +{ + PS =3D sysconf(_SC_PAGESIZE); + setup_mthp(); + + int fd =3D open(GUP_DEV, O_RDWR); + + if (fd < 0) { + fprintf(stderr, "open %s: %s (CONFIG_GUP_TEST?)\n", + GUP_DEV, strerror(errno)); + return 2; + } + + char *r =3D mmap(NULL, REGION, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (r =3D=3D MAP_FAILED) { + perror("mmap"); + return 2; + } + fill(r, REGION, 0xA1); /* pattern P */ + char *expP =3D malloc(REGION); + + memcpy(expP, r, REGION); + + int large =3D count_large_pages(r, REGION); + + printf("# %d/%zu pages in contiguous large-folio runs\n", + large, REGION / PS); + if (large < (int)(REGION / PS) / 4) + printf("# WARN: little mTHP backing; batch path lightly covered\n"); + + /* A: read pin over writable mTHP -> batches -> content must equal P. */ + ok(pin_verify(fd, r, REGION, 0, expP) =3D=3D 0, + "slow read-pin of mTHP returns correct contents"); + + /* B: write pin -> FOLL_WRITE batch path -> content must equal P. */ + ok(pin_verify(fd, r, REGION, 1, expP) =3D=3D 0, + "slow write-pin of mTHP returns correct contents"); + + /* C: COW isolation. Child write-pins (unshares) then rewrites; parent P.= */ + pid_t pid =3D fork(); + + if (pid =3D=3D 0) { + int cfd =3D open(GUP_DEV, O_RDWR); + int a =3D pin_verify(cfd, r, REGION, 1, expP); /* COW copy =3D=3D P */ + + fill(r, REGION, 0xB2); /* child writes Q */ + _exit(a =3D=3D 0 ? 0 : 1); + } + int st =3D 0; + + waitpid(pid, &st, 0); + ok(WIFEXITED(st) && WEXITSTATUS(st) =3D=3D 0, + "child write-pin after COW returns correct (copied) contents"); + ok(memcmp(r, expP, REGION) =3D=3D 0, + "parent contents intact after child COW writes"); + /* D: parent read-pin again after the COW split still correct. */ + ok(pin_verify(fd, r, REGION, 0, expP) =3D=3D 0, + "parent slow read-pin after COW still correct"); + + printf("# totals: pass:%d fail:%d\n", tap - fails, fails); + return fails ? 1 : 0; +} diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/self= tests/mm/run_vmtests.sh index 8c296dedf047..b49a3eb0c205 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -289,6 +289,7 @@ fi # Dump pages 0, 19, and 4096, using pin_user_pages: CATEGORY=3D"gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000 CATEGORY=3D"gup_test" run_test ./gup_longterm +CATEGORY=3D"gup_test" run_test ./mthp_gup_cow_test =20 CATEGORY=3D"userfaultfd" run_test ./uffd-unit-tests uffd_stress_bin=3D./uffd-stress --=20 2.53.0-Meta