From nobody Fri Oct 2 10:07:37 2026 Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 292233D75B1 for ; Sun, 2 Aug 2026 21:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.198 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785707711; cv=none; b=I59t+laOqv577NIMfVa2hs0YCis5Czr9YzlnHzZVYlth5bVdsnj3earj3wzROBeN5qFMu6H50m5giN8lTqtdyBNsIpkGtVIKCjeJagHK3pCUGzzQbbdTLiqOUiCcDedEA/V+5i5qh8Yn9SVbNZqY1BxU9PiKBTdQKAnduAtgi2M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785707711; c=relaxed/simple; bh=JEr8XSBPEJgKjcqJ6BfzYYPE1yrV4P82WnQsEogT7WE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gu9muRM5oaqF4WdwQHovWbIRpnh2ataGPop5eVb6TOWPAOLcuIWDi9Y/YwoU+hRkv1ceEk4OI3IbyyQCGFSCa0HJTAxfYLjVKb1A1skmlBGFYSDJYEmSqn/QOrLyupzG77J1BrLvo5mIJ46wACO4h8waal8uTRYNX7X/0UbP8Ug= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=JkgtxUdv; arc=none smtp.client-ip=209.85.215.198 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JkgtxUdv" Received: by mail-pg1-f198.google.com with SMTP id 41be03b00d2f7-cab041eced3so4386368a12.1 for ; Sun, 02 Aug 2026 14:55:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1785707706; x=1786312506; darn=vger.kernel.org; h=content-transfer-encoding:content-type:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date:from:to:cc :subject:date:message-id:reply-to:content-type; bh=5WSGwwxyjjinJPQxJkdVG/Ds1Cy6oHGCMAupIBKVInM=; b=JkgtxUdvPlDDO+no6Um2ClaqtihSgh7I0JBKRWYIvekHqR07tF6/AUJyBCxeMnihiD ZmaXrxALICG0El38jBoUINmg2NZieCaKCmx35Iclp6QRAsoIMLF0Fh2xx8OXQg/7daAN 5dJxQB6F+oiYZxp39Br0qmK6qWehTVgTr+Q9kUrcJhkQkI8eSkFiBbLOTnthCwwyA2ua /8x5YlPJ2mXUzK9UHjc1/CZ2fRDfhaLP1jssDENb+1LNl8ypAVlB5S0eq9TZaekr4zuJ Ig32wjLixpJDkF8Y+9wST8G5FOJtNpp5Bz1L0kE9ByJam5YkzuvzWrBNOS2JMXooNvo7 /nfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785707706; x=1786312506; h=content-transfer-encoding:content-type:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=5WSGwwxyjjinJPQxJkdVG/Ds1Cy6oHGCMAupIBKVInM=; b=rBImuV1kySZrzb2lA7lY5toJ+GnDZv8+nLQfw/FLzc3nb3rvnyzcQA3HWjr7PWTcEp rUpALlAQpQznOyET2pHlAPT4BIrWI4ygtgb/00M+uyq9hdoAmi82VJOqyTkxWQ18kAWj u65aIhAmMyH1HJf6Gd9mD80eEo20QE1AfZ4art3p5XhtrXhNtsawyDTMf8kH7hYpf54P OhigqEPKXan0UbPCDB7HHf18Y18iQvbU8+WauTZUnI218RqeYx5P/MG6280WS3uJLkeu WXYFyBiuf3SHSFMyO1waSF9WGy1Ki2v2Ajek11Lf9V6Or1cl+Gbr3wPP89PtsjE1Yh8X q24A== X-Forwarded-Encrypted: i=1; AHgh+RrdGPou7bJXRJGXSdCRDFgCSBrtBMS+lA0onQYaw06Yg4y/luCvSuE5982pH2+PG/nfoPUcCTLrSQ/UIO0=@vger.kernel.org X-Gm-Message-State: AOJu0YwZeEdPHn+aKiACGzEJRdFCPcICeJ3ZC+4JSm14CR6CLV/BhUlA V5ffUijKlmhIXk7nkQYBU4MBYYidnvfWyxQPb8gw6O/2rUqtSjoIcMK45Tmq6m7Z0Z5kfXOUTFe RbqzL7Q== X-Received: from dybfx23.prod.google.com ([2002:a05:7300:c397:b0:30f:1818:9ce5]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:431d:b0:3c3:bbe6:95c9 with SMTP id adf61e73a8af0-3c92a524e5cmr6759651637.16.1785707705981; Sun, 02 Aug 2026 14:55:05 -0700 (PDT) Date: Sun, 2 Aug 2026 14:54:55 -0700 In-Reply-To: <20260802215459.2769283-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260802215459.2769283-1-surenb@google.com> X-Mailer: git-send-email 2.55.0.508.g3f0d502094-goog Message-ID: <20260802215459.2769283-2-surenb@google.com> Subject: [PATCH v3 1/5] mm: Make per-VMA locks available universally From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: dave.hansen@linux.intel.com, Liam.Howlett@oracle.com, ljs@kernel.org, david@redhat.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@kernel.org, jannh@google.com, aliceryhl@google.com, arve@android.com, cmllamas@google.com, christian@brauner.io, tkjos@android.com, dsahern@kernel.org, davem@davemloft.net, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Dave Hansen The per-VMA locks have been around for several years. They've had some bugs worked out of them and have seen quite wide use. However, they are still only available when architectures explicitly enable them. Remove the conditional compilation around the per-VMA locks, making them available on all architectures and configs. The approach up to now seemed to be to add ARCH_SUPPORTS_PER_VMA_LOCK when the architecture started using per-VMA locks in the fault handler. But, contrary to the naming, the Kconfig option does not really indicate whether the architecture supports per-VMA locks or not. It is more of a marker for whether the architecture is likely to benefit from per-VMA locks. To me, the most important thing side-effect of universal availability is letting per-VMA locks be used in SMP=3Dn configs. This lets us use per-VMA locking in all x86 code without fallbacks. Overall, this just generally makes the kernel simpler. Just look at the diffstat. It also opens the door to users that want to use the per-VMA locks in common code. Doing *that* brings additional simplifications. The downside of this is adding some fields to vm_area_struct and mm_struct. There are likely ways to optimize this, especially for things like SMP=3Dn configs. For now, do the simplest thing: use the same implementation everywhere. Signed-off-by: Dave Hansen Signed-off-by: Suren Baghdasaryan Cc: Suren Baghdasaryan Cc: Andrew Morton Cc: "Liam R. Howlett" Cc: Lorenzo Stoakes Cc: Vlastimil Babka Cc: Shakeel Butt Cc: linux-mm@kvack.org Cc: Greg Kroah-Hartman Cc: Arve Hj=C3=B8nnev=C3=A5g Cc: Todd Kjos Cc: Christian Brauner Cc: Carlos Llamas Cc: Alice Ryhl Cc: "David S. Miller" Cc: David Ahern Cc: netdev@vger.kernel.org Reviewed-by: Jann Horn Reviewed-by: Lorenzo Stoakes (ARM) Reviewed-by: Vlastimil Babka (SUSE) --- arch/arm/Kconfig | 1 - arch/arm64/Kconfig | 1 - arch/loongarch/Kconfig | 1 - arch/powerpc/platforms/powernv/Kconfig | 1 - arch/powerpc/platforms/pseries/Kconfig | 1 - arch/riscv/Kconfig | 1 - arch/s390/Kconfig | 1 - arch/x86/Kconfig | 2 - fs/proc/internal.h | 2 - fs/proc/task_mmu.c | 93 -------------------------- include/linux/mm.h | 12 ---- include/linux/mm_types.h | 8 +-- include/linux/mmap_lock.h | 50 -------------- kernel/bpf/stackmap.c | 16 +---- kernel/bpf/task_iter.c | 5 -- kernel/fork.c | 2 - mm/Kconfig | 13 ---- mm/Kconfig.debug | 1 - mm/debug.c | 4 -- mm/init-mm.c | 2 - mm/memory.c | 2 - mm/mmap_lock.c | 24 ------- mm/pagewalk.c | 2 - mm/rmap.c | 2 - mm/userfaultfd.c | 55 --------------- rust/kernel/mm.rs | 22 +++--- tools/testing/vma/include/dup.h | 5 +- tools/testing/vma/vma_internal.h | 1 - 28 files changed, 13 insertions(+), 317 deletions(-) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 9187240a02db..f815209167cd 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -41,7 +41,6 @@ config ARM select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_CFI select ARCH_SUPPORTS_HUGETLBFS if ARM_LPAE - select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_SUPPORTS_RT select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_CMPXCHG_LOCKREF diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 11a9c534b7b4..21eb64b24a2c 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -81,7 +81,6 @@ config ARM64 select ARCH_HAS_PTE_PROTNONE select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK - select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select ARCH_SUPPORTS_RT select ARCH_SUPPORTS_SCHED_SMT diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index e20acbe5fe7b..7741e39eca2b 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -69,7 +69,6 @@ config LOONGARCH select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS select ARCH_HAS_PTE_PROTNONE if 64BIT select ARCH_SUPPORTS_NUMA_BALANCING if NUMA - select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_SUPPORTS_RT select ARCH_SUPPORTS_SCHED_SMT if SMP select ARCH_SUPPORTS_SCHED_MC if SMP diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platform= s/powernv/Kconfig index b5ad7c173ef0..dd8f6060fb7a 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -17,7 +17,6 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP - select ARCH_SUPPORTS_PER_VMA_LOCK select PPC_RADIX_BROADCAST_TLBIE if PPC_RADIX_MMU default y =20 diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platform= s/pseries/Kconfig index 74910ce3a541..7d125e288f6e 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -23,7 +23,6 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB - select ARCH_SUPPORTS_PER_VMA_LOCK select PPC_RADIX_BROADCAST_TLBIE if PPC_RADIX_MMU default y =20 diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 7b9c373d82fa..faa85a031fe5 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -70,7 +70,6 @@ config RISCV select ARCH_SUPPORTS_LTO_CLANG_THIN select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS if 64BIT && MMU select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU - select ARCH_SUPPORTS_PER_VMA_LOCK if MMU select ARCH_HAS_PTE_PROTNONE if MMU select ARCH_SUPPORTS_RT select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index ab8fccc2cc4e..d1274bca8c39 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -151,7 +151,6 @@ config S390 select ARCH_HAS_PTE_PROTNONE select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK - select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_CMPXCHG_LOCKREF select ARCH_USE_SYM_ANNOTATIONS diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index fb298e219179..79479d29576f 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,7 +27,6 @@ config X86_64 select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_MSEAL_SYSTEM_MAPPINGS select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 - select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA @@ -1846,7 +1845,6 @@ config X86_USER_SHADOW_STACK bool "X86 userspace shadow stack" depends on AS_WRUSS depends on X86_64 - depends on PER_VMA_LOCK select ARCH_USES_HIGH_VMA_FLAGS select ARCH_HAS_USER_SHADOW_STACK select X86_CET diff --git a/fs/proc/internal.h b/fs/proc/internal.h index b232e1098117..6713757da099 100644 --- a/fs/proc/internal.h +++ b/fs/proc/internal.h @@ -385,10 +385,8 @@ struct mem_size_stats; =20 struct proc_maps_locking_ctx { struct mm_struct *mm; -#ifdef CONFIG_PER_VMA_LOCK bool mmap_locked; struct vm_area_struct *locked_vma; -#endif }; =20 struct proc_maps_private { diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 817e3e0f9194..096bf0b0b9e0 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -130,8 +130,6 @@ static void release_task_mempolicy(struct proc_maps_pri= vate *priv) } #endif =20 -#ifdef CONFIG_PER_VMA_LOCK - static inline int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx) { int ret =3D mmap_read_lock_killable(lock_ctx->mm); @@ -233,46 +231,6 @@ static inline void reacquire_rcu(struct proc_maps_priv= ate *priv) vma_iter_set(&priv->iter, priv->lock_ctx.locked_vma->vm_end); } =20 -#else /* CONFIG_PER_VMA_LOCK */ - -static inline int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx) -{ - return mmap_read_lock_killable(lock_ctx->mm); -} - -static inline void unlock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx) -{ - mmap_read_unlock(lock_ctx->mm); -} - -static inline bool lock_vma_range(struct seq_file *m, - struct proc_maps_locking_ctx *lock_ctx) -{ - return lock_ctx_mm(lock_ctx) =3D=3D 0; -} - -static inline void unlock_vma_range(struct proc_maps_locking_ctx *lock_ctx) -{ - unlock_ctx_mm(lock_ctx); -} - -static struct vm_area_struct *get_next_vma(struct proc_maps_private *priv, - loff_t last_pos) -{ - return vma_next(&priv->iter); -} - -static inline bool fallback_to_mmap_lock(struct proc_maps_private *priv, - loff_t pos) -{ - return false; -} - -static inline void drop_rcu(struct proc_maps_private *priv) {} -static inline void reacquire_rcu(struct proc_maps_private *priv) {} - -#endif /* CONFIG_PER_VMA_LOCK */ - static struct vm_area_struct *proc_get_vma(struct seq_file *m, loff_t *ppo= s) { struct proc_maps_private *priv =3D m->private; @@ -560,8 +518,6 @@ static int pid_maps_open(struct inode *inode, struct fi= le *file) PROCMAP_QUERY_VMA_FLAGS \ ) =20 -#ifdef CONFIG_PER_VMA_LOCK - static int query_vma_setup(struct proc_maps_locking_ctx *lock_ctx) { reset_lock_ctx(lock_ctx); @@ -612,26 +568,6 @@ static struct vm_area_struct *query_vma_find_by_addr(s= truct proc_maps_locking_ct return vma; } =20 -#else /* CONFIG_PER_VMA_LOCK */ - -static int query_vma_setup(struct proc_maps_locking_ctx *lock_ctx) -{ - return mmap_read_lock_killable(lock_ctx->mm); -} - -static void query_vma_teardown(struct proc_maps_locking_ctx *lock_ctx) -{ - mmap_read_unlock(lock_ctx->mm); -} - -static struct vm_area_struct *query_vma_find_by_addr(struct proc_maps_lock= ing_ctx *lock_ctx, - unsigned long addr) -{ - return find_vma(lock_ctx->mm, addr); -} - -#endif /* CONFIG_PER_VMA_LOCK */ - static struct vm_area_struct *query_matching_vma(struct proc_maps_locking_= ctx *lock_ctx, unsigned long addr, u32 flags) { @@ -1314,8 +1250,6 @@ static const struct mm_walk_ops smaps_shmem_walk_ops = =3D { .walk_lock =3D PGWALK_RDLOCK, }; =20 -#ifdef CONFIG_PER_VMA_LOCK - static const struct mm_walk_ops smaps_walk_vma_lock_ops =3D { .pmd_entry =3D smaps_pte_range, .hugetlb_entry =3D smaps_hugetlb_range, @@ -1345,22 +1279,6 @@ get_smaps_shmem_walk_ops(struct proc_maps_private *p= riv) return &smaps_shmem_walk_vma_lock_ops; } =20 -#else /* CONFIG_PER_VMA_LOCK */ - -static inline const struct mm_walk_ops * -get_smaps_walk_ops(struct proc_maps_private *priv) -{ - return &smaps_walk_ops; -} - -static inline const struct mm_walk_ops * -get_smaps_shmem_walk_ops(struct proc_maps_private *priv) -{ - return &smaps_shmem_walk_ops; -} - -#endif /* CONFIG_PER_VMA_LOCK */ - /* * Gather mem stats from @vma with the indicated beginning * address @start, and keep them in @mss. @@ -3497,7 +3415,6 @@ static const struct mm_walk_ops show_numa_ops =3D { .walk_lock =3D PGWALK_RDLOCK, }; =20 -#ifdef CONFIG_PER_VMA_LOCK static const struct mm_walk_ops show_numa_vma_lock_ops =3D { .hugetlb_entry =3D gather_hugetlb_stats, .pmd_entry =3D gather_pte_stats, @@ -3512,16 +3429,6 @@ get_show_numa_ops(struct proc_maps_private *priv) return &show_numa_vma_lock_ops; } =20 -#else /* CONFIG_PER_VMA_LOCK */ - -static inline const struct mm_walk_ops * -get_show_numa_ops(struct proc_maps_private *priv) -{ - return &show_numa_ops; -} - -#endif /* CONFIG_PER_VMA_LOCK */ - /* * Display pages allocated per node and memory policy via /proc. */ diff --git a/include/linux/mm.h b/include/linux/mm.h index 7fabe6c66b4b..d9850f846242 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -931,7 +931,6 @@ static inline void vma_numab_state_free(struct vm_area_= struct *vma) {} * These must be here rather than mmap_lock.h as dependent on vm_fault typ= e, * declared in this header. */ -#ifdef CONFIG_PER_VMA_LOCK static inline void release_fault_lock(struct vm_fault *vmf) { if (vmf->flags & FAULT_FLAG_VMA_LOCK) @@ -947,17 +946,6 @@ static inline void assert_fault_locked(const struct vm= _fault *vmf) else mmap_assert_locked(vmf->vma->vm_mm); } -#else -static inline void release_fault_lock(struct vm_fault *vmf) -{ - mmap_read_unlock(vmf->vma->vm_mm); -} - -static inline void assert_fault_locked(const struct vm_fault *vmf) -{ - mmap_assert_locked(vmf->vma->vm_mm); -} -#endif /* CONFIG_PER_VMA_LOCK */ =20 static inline bool mm_flags_test(int flag, const struct mm_struct *mm) { diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index b5d4cd3b067b..d8e246fd09d3 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -950,7 +950,6 @@ struct vm_area_struct { vma_flags_t flags; }; =20 -#ifdef CONFIG_PER_VMA_LOCK /* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) @@ -966,7 +965,7 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; -#endif + /* * Low 32-bits of virtual page offset. * See vma_start_virt_pgoff() comment for details. @@ -1003,7 +1002,6 @@ struct vm_area_struct { #ifdef CONFIG_NUMA_BALANCING struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif -#ifdef CONFIG_PER_VMA_LOCK /* * Used to keep track of firstly, whether the VMA is attached, secondly, * if attached, how many read locks are taken, and thirdly, if the @@ -1046,7 +1044,6 @@ struct vm_area_struct { #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map vmlock_dep_map; #endif -#endif #ifdef CONFIG_64BIT /* * High 32-bits of virtual page offset. @@ -1254,7 +1251,6 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ -#ifdef CONFIG_PER_VMA_LOCK struct rcuwait vma_writer_wait; /* * This field has lock-like semantics, meaning it is sometimes @@ -1274,7 +1270,7 @@ struct mm_struct { * mmap_lock. */ seqcount_t mm_lock_seq; -#endif + struct futex_mm_data futex; =20 unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 87f77e3da77f..eb32b482434e 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -76,8 +76,6 @@ static inline void mmap_assert_write_locked(const struct = mm_struct *mm) rwsem_assert_held_write(&mm->mmap_lock); } =20 -#ifdef CONFIG_PER_VMA_LOCK - #ifdef CONFIG_LOCKDEP #define __vma_lockdep_map(vma) (&vma->vmlock_dep_map) #else @@ -484,54 +482,6 @@ struct vm_area_struct *lock_next_vma(struct mm_struct = *mm, struct vma_iterator *iter, unsigned long address); =20 -#else /* CONFIG_PER_VMA_LOCK */ - -static inline void mm_lock_seqcount_init(struct mm_struct *mm) {} -static inline void mm_lock_seqcount_begin(struct mm_struct *mm) {} -static inline void mm_lock_seqcount_end(struct mm_struct *mm) {} - -static inline bool mmap_lock_speculate_try_begin(struct mm_struct *mm, uns= igned int *seq) -{ - return false; -} - -static inline bool mmap_lock_speculate_retry(struct mm_struct *mm, unsigne= d int seq) -{ - return true; -} -static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_re= fcnt) {} -static inline void vma_end_read(struct vm_area_struct *vma) {} -static inline void vma_start_write(struct vm_area_struct *vma) {} -static inline __must_check -int vma_start_write_killable(struct vm_area_struct *vma) { return 0; } -static inline void vma_assert_write_locked(struct vm_area_struct *vma) - { mmap_assert_write_locked(vma->vm_mm); } -static inline bool vma_is_attached(struct vm_area_struct *vma) - { return true; } -static inline void vma_assert_attached(struct vm_area_struct *vma) {} -static inline void vma_assert_detached(struct vm_area_struct *vma) {} -static inline void vma_mark_attached(struct vm_area_struct *vma) {} -static inline void vma_mark_detached(struct vm_area_struct *vma) {} - -static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *= mm, - unsigned long address) -{ - return NULL; -} - -static inline void vma_assert_locked(struct vm_area_struct *vma) -{ - mmap_assert_locked(vma->vm_mm); -} - -static inline void vma_assert_stabilised(struct vm_area_struct *vma) -{ - /* If no VMA locks, then either mmap lock suffices to stabilise. */ - mmap_assert_locked(vma->vm_mm); -} - -#endif /* CONFIG_PER_VMA_LOCK */ - static inline void vma_assert_can_modify(struct vm_area_struct *vma) { if (vma_is_attached(vma)) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index 41fe87d7302f..8848e26ef581 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -272,13 +272,8 @@ struct stack_map_vma_lock { /* * Acquire a stable read-side reference on the VMA covering @ip. * - * With CONFIG_PER_VMA_LOCK=3Dy this returns a VMA with its per-VMA read - * lock held and mmap_lock dropped, so the caller may sleep. - * - * With CONFIG_PER_VMA_LOCK=3Dn it returns a VMA with mmap_lock still - * held; the caller must snapshot any fields it needs and pin vm_file - * with get_file() before stack_map_unlock_vma() drops mmap_lock, as - * the VMA may be split, merged, or freed after that. + * This returns a VMA with its per-VMA read lock held and mmap_lock + * dropped, so the caller may sleep. * * Returns NULL on failure, in which case no lock is held. */ @@ -288,7 +283,6 @@ stack_map_lock_vma(struct stack_map_vma_lock *lock, uns= igned long ip) struct mm_struct *mm =3D lock->mm; struct vm_area_struct *vma; =20 - /* noop under !CONFIG_PER_VMA_LOCK */ vma =3D lock_vma_under_rcu(mm, ip); if (vma) { lock->vma =3D vma; @@ -308,13 +302,11 @@ stack_map_lock_vma(struct stack_map_vma_lock *lock, u= nsigned long ip) return NULL; } =20 -#ifdef CONFIG_PER_VMA_LOCK if (!vma_start_read_locked(vma)) { mmap_read_unlock(mm); return NULL; } mmap_read_unlock(mm); -#endif =20 lock->vma =3D vma; return vma; @@ -322,11 +314,7 @@ stack_map_lock_vma(struct stack_map_vma_lock *lock, un= signed long ip) =20 static void stack_map_unlock_vma(struct stack_map_vma_lock *lock) { -#ifdef CONFIG_PER_VMA_LOCK vma_end_read(lock->vma); -#else - mmap_read_unlock(lock->mm); -#endif lock->vma =3D NULL; } =20 diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c index e791ae065c39..6cf815bc84be 100644 --- a/kernel/bpf/task_iter.c +++ b/kernel/bpf/task_iter.c @@ -835,11 +835,6 @@ __bpf_kfunc int bpf_iter_task_vma_new(struct bpf_iter_= task_vma *it, BUILD_BUG_ON(sizeof(struct bpf_iter_task_vma_kern) !=3D sizeof(struct bpf= _iter_task_vma)); BUILD_BUG_ON(__alignof__(struct bpf_iter_task_vma_kern) !=3D __alignof__(= struct bpf_iter_task_vma)); =20 - if (!IS_ENABLED(CONFIG_PER_VMA_LOCK)) { - kit->data =3D NULL; - return -EOPNOTSUPP; - } - /* * Reject irqs-disabled contexts including NMI. Operations used * by _next() and _destroy() (vma_end_read, fput, bpf_iter_mmput_async) diff --git a/kernel/fork.c b/kernel/fork.c index f0e2e131a9a5..ff91f5f66c80 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1077,9 +1077,7 @@ static void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); mm_lock_seqcount_init(mm); -#ifdef CONFIG_PER_VMA_LOCK rcuwait_init(&mm->vma_writer_wait); -#endif } =20 static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct = *p) diff --git a/mm/Kconfig b/mm/Kconfig index 060190e12bce..77103b46b679 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1425,19 +1425,6 @@ config LRU_GEN_STATS config LRU_GEN_WALKS_MMU def_bool y depends on LRU_GEN && ARCH_HAS_HW_PTE_YOUNG -# } - -config ARCH_SUPPORTS_PER_VMA_LOCK - def_bool n - -config PER_VMA_LOCK - def_bool y - depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP - help - Allow per-vma locking during page fault handling. - - This feature allows locking each virtual memory area separately when - handling page faults instead of taking mmap_lock. =20 config LOCK_MM_AND_FIND_VMA bool diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index 5737a504efbb..1dd150edfe71 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -310,7 +310,6 @@ config DEBUG_KMEMLEAK_VERBOSE =20 config PER_VMA_LOCK_STATS bool "Statistics for per-vma locks" - depends on PER_VMA_LOCK help Say Y here to enable success, retry and failure counters of page faults handled under protection of per-vma locks. When enabled, the diff --git a/mm/debug.c b/mm/debug.c index 9a0297b3988d..655e6bcc0e8d 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -157,17 +157,13 @@ void dump_vma(const struct vm_area_struct *vma) pr_emerg("vma %px start %px end %px mm %px\n" "prot %lx anon_vma %px vm_ops %px\n" "pgoff %lx file %px private_data %px\n" -#ifdef CONFIG_PER_VMA_LOCK "refcnt %x\n" -#endif "flags: %#lx(%pGv)\n", vma, (void *)vma->vm_start, (void *)vma->vm_end, vma->vm_mm, (unsigned long)pgprot_val(vma->vm_page_prot), vma->anon_vma, vma->vm_ops, vma_start_pgoff(vma), vma->vm_file, vma->vm_private_data, -#ifdef CONFIG_PER_VMA_LOCK refcount_read(&vma->vm_refcnt), -#endif vma->vm_flags, &vma->vm_flags); } EXPORT_SYMBOL(dump_vma); diff --git a/mm/init-mm.c b/mm/init-mm.c index 3e792aad7626..a1bb2c2d0284 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -39,10 +39,8 @@ struct mm_struct init_mm =3D { .page_table_lock =3D __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock =3D __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist =3D LIST_HEAD_INIT(init_mm.mmlist), -#ifdef CONFIG_PER_VMA_LOCK .vma_writer_wait =3D __RCUWAIT_INITIALIZER(init_mm.vma_writer_wait), .mm_lock_seq =3D SEQCNT_ZERO(init_mm.mm_lock_seq), -#endif #ifdef CONFIG_SCHED_MM_CID .mm_cid.lock =3D __RAW_SPIN_LOCK_UNLOCKED(init_mm.mm_cid.lock), #endif diff --git a/mm/memory.c b/mm/memory.c index a620d425ec95..f109f6c87b28 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6813,7 +6813,6 @@ static vm_fault_t sanitize_fault_flags(struct vm_area= _struct *vma, !is_cow_mapping(vma->vm_flags))) return VM_FAULT_SIGSEGV; } -#ifdef CONFIG_PER_VMA_LOCK /* * Per-VMA locks can't be used with FAULT_FLAG_RETRY_NOWAIT because of * the assumption that lock is dropped on VM_FAULT_RETRY. @@ -6822,7 +6821,6 @@ static vm_fault_t sanitize_fault_flags(struct vm_area= _struct *vma, (FAULT_FLAG_VMA_LOCK | FAULT_FLAG_RETRY_NOWAIT)) =3D=3D (FAULT_FLAG_VMA_LOCK | FAULT_FLAG_RETRY_NOWAIT))) return VM_FAULT_SIGSEGV; -#endif =20 return 0; } diff --git a/mm/mmap_lock.c b/mm/mmap_lock.c index 898c2ef1e958..e20d01e8d38f 100644 --- a/mm/mmap_lock.c +++ b/mm/mmap_lock.c @@ -43,9 +43,6 @@ void __mmap_lock_do_trace_released(struct mm_struct *mm, = bool write) EXPORT_SYMBOL(__mmap_lock_do_trace_released); #endif /* CONFIG_TRACING */ =20 -#ifdef CONFIG_MMU -#ifdef CONFIG_PER_VMA_LOCK - /* State shared across __vma_[start, end]_exclude_readers. */ struct vma_exclude_readers_state { /* Input parameters. */ @@ -431,7 +428,6 @@ struct vm_area_struct *lock_next_vma(struct mm_struct *= mm, =20 return vma; } -#endif /* CONFIG_PER_VMA_LOCK */ =20 #ifdef CONFIG_LOCK_MM_AND_FIND_VMA #include @@ -548,23 +544,3 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_= struct *mm, return NULL; } #endif /* CONFIG_LOCK_MM_AND_FIND_VMA */ - -#else /* CONFIG_MMU */ - -/* - * At least xtensa ends up having protection faults even with no - * MMU.. No stack expansion, at least. - */ -struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm, - unsigned long addr, struct pt_regs *regs) -{ - struct vm_area_struct *vma; - - mmap_read_lock(mm); - vma =3D vma_lookup(mm, addr); - if (!vma) - mmap_read_unlock(mm); - return vma; -} - -#endif /* CONFIG_MMU */ diff --git a/mm/pagewalk.c b/mm/pagewalk.c index ed4860c01936..fbcf64c59a97 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -446,7 +446,6 @@ static inline void process_mm_walk_lock(struct mm_struc= t *mm, static inline void process_vma_walk_lock(struct vm_area_struct *vma, enum page_walk_lock walk_lock) { -#ifdef CONFIG_PER_VMA_LOCK switch (walk_lock) { case PGWALK_WRLOCK: vma_start_write(vma); @@ -461,7 +460,6 @@ static inline void process_vma_walk_lock(struct vm_area= _struct *vma, /* PGWALK_RDLOCK is handled by process_mm_walk_lock */ break; } -#endif } =20 /* diff --git a/mm/rmap.c b/mm/rmap.c index b917431759ee..4e4a4b747977 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -260,11 +260,9 @@ static void check_anon_vma_clone(struct vm_area_struct= *dst, /* For the anon_vma to be compatible, it can only be singular. */ VM_WARN_ON_ONCE(operation =3D=3D VMA_OP_MERGE_UNFAULTED && !list_is_singular(&src->anon_vma_chain)); -#ifdef CONFIG_PER_VMA_LOCK /* Only merging an unfaulted VMA leaves the destination attached. */ VM_WARN_ON_ONCE(operation !=3D VMA_OP_MERGE_UNFAULTED && vma_is_attached(dst)); -#endif } =20 static void maybe_reuse_anon_vma(struct vm_area_struct *dst, diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 258b03182a78..edd90892f8cc 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -122,7 +122,6 @@ struct vm_area_struct *find_vma_and_prepare_anon(struct= mm_struct *mm, return vma; } =20 -#ifdef CONFIG_PER_VMA_LOCK /* * uffd_lock_vma() - Lookup and lock vma corresponding to @address. * @mm: mm to search vma in. @@ -182,34 +181,6 @@ static void uffd_mfill_unlock(struct vm_area_struct *v= ma) vma_end_read(vma); } =20 -#else - -static struct vm_area_struct *uffd_mfill_lock(struct mm_struct *dst_mm, - unsigned long dst_start, - unsigned long len) -{ - struct vm_area_struct *dst_vma; - - mmap_read_lock(dst_mm); - dst_vma =3D find_vma_and_prepare_anon(dst_mm, dst_start); - if (IS_ERR(dst_vma)) - goto out_unlock; - - if (validate_dst_vma(dst_vma, dst_start + len)) - return dst_vma; - - dst_vma =3D ERR_PTR(-ENOENT); -out_unlock: - mmap_read_unlock(dst_mm); - return dst_vma; -} - -static void uffd_mfill_unlock(struct vm_area_struct *vma) -{ - mmap_read_unlock(vma->vm_mm); -} -#endif - static void mfill_put_vma(struct mfill_state *state) { if (!state->vma) @@ -1852,7 +1823,6 @@ int find_vmas_mm_locked(struct mm_struct *mm, return 0; } =20 -#ifdef CONFIG_PER_VMA_LOCK static int uffd_move_lock(struct mm_struct *mm, unsigned long dst_start, unsigned long src_start, @@ -1927,31 +1897,6 @@ static void uffd_move_unlock(struct vm_area_struct *= dst_vma, vma_end_read(dst_vma); } =20 -#else - -static int uffd_move_lock(struct mm_struct *mm, - unsigned long dst_start, - unsigned long src_start, - struct vm_area_struct **dst_vmap, - struct vm_area_struct **src_vmap) -{ - int err; - - mmap_read_lock(mm); - err =3D find_vmas_mm_locked(mm, dst_start, src_start, dst_vmap, src_vmap); - if (err) - mmap_read_unlock(mm); - return err; -} - -static void uffd_move_unlock(struct vm_area_struct *dst_vma, - struct vm_area_struct *src_vma) -{ - mmap_assert_locked(src_vma->vm_mm); - mmap_read_unlock(dst_vma->vm_mm); -} -#endif - /** * move_pages - move arbitrary anonymous pages of an existing vma * @ctx: pointer to the userfaultfd context diff --git a/rust/kernel/mm.rs b/rust/kernel/mm.rs index 4764d7b68f2a..2633e704c83d 100644 --- a/rust/kernel/mm.rs +++ b/rust/kernel/mm.rs @@ -174,26 +174,20 @@ pub unsafe fn from_raw<'a>(ptr: *const bindings::mm_s= truct) -> &'a MmWithUser { /// When per-vma locks are disabled, this always returns `None`. #[inline] pub fn lock_vma_under_rcu(&self, vma_addr: usize) -> Option> { - #[cfg(CONFIG_PER_VMA_LOCK)] { // SAFETY: Calling `bindings::lock_vma_under_rcu` is always ok= ay given an mm where // `mm_users` is non-zero. let vma =3D unsafe { bindings::lock_vma_under_rcu(self.as_raw(= ), vma_addr) }; - if !vma.is_null() { - return Some(VmaReadGuard { - // SAFETY: If `lock_vma_under_rcu` returns a non-null = ptr, then it points at a - // valid vma. The vma is stable for as long as the vma= read lock is held. - vma: unsafe { VmaRef::from_raw(vma) }, - _nts: NotThreadSafe, - }); + if vma.is_null() { + return None; } + Some(VmaReadGuard { + // SAFETY: If `lock_vma_under_rcu` returns a non-null ptr,= then it points at a + // valid vma. The vma is stable for as long as the vma rea= d lock is held. + vma: unsafe { VmaRef::from_raw(vma) }, + _nts: NotThreadSafe, + }) } - - // Silence warnings about unused variables. - #[cfg(not(CONFIG_PER_VMA_LOCK))] - let _ =3D vma_addr; - - None } =20 /// Lock the mmap read lock. diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/du= p.h index 800e5fa02d78..362feda28526 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -582,7 +582,6 @@ struct vm_area_struct { vma_flags_t flags; }; =20 -#ifdef CONFIG_PER_VMA_LOCK /* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) @@ -598,7 +597,7 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; -#endif + unsigned int __vm_virt_pgoff_lo; =20 /* @@ -632,10 +631,8 @@ struct vm_area_struct { #ifdef CONFIG_NUMA_BALANCING struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif -#ifdef CONFIG_PER_VMA_LOCK /* Unstable RCU readers are allowed to read this. */ refcount_t vm_refcnt; -#endif #ifdef CONFIG_64BIT unsigned int __vm_virt_pgoff_hi; #endif diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_inter= nal.h index 8a48b231aa7a..54d5c3360aa2 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -15,7 +15,6 @@ #include =20 #define CONFIG_MMU 1 -#define CONFIG_PER_VMA_LOCK 1 =20 #ifdef __CONCAT #undef __CONCAT --=20 2.55.0.508.g3f0d502094-goog From nobody Fri Oct 2 10:07:37 2026 Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E64473DBD41 for ; Sun, 2 Aug 2026 21:55:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785707710; cv=none; b=GC7DDsTE4ckp7/yexF34DcVBp7o4YQSQ7VTSA6oOp06cAYz0obgJv3lhIhTZw0GlBTVcCnB30Vn5OXiMssPKM4/RHOtkhVGnQ1wfCqtQnlhCsm1yEEml/cb5JyEDLTpWKTxfJctMgZODcS4CTpCtxTKBmJMGB825SzohsvWh+oc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785707710; c=relaxed/simple; bh=p62l3PJjOIdYZzi6CGFj9/z2xtIpPh0sBcEeuCMuHo4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hcrVtGZkF9TZzIiUPces8QEndTj5fLx/9Qhsiun/ATrL9fOKcCfs3nPxEcqlVlyb7t0YpbshRu0x54ra8lj0kAkZhMNpQLsfe670o5eucV31Epvdw3HofJR5uxP3sVW1KSQRbBVaVXVWs6X6j231ZfnP40fTKMIh+nq6hmRcICk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=DO81pyuu; arc=none smtp.client-ip=209.85.215.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="DO81pyuu" Received: by mail-pg1-f199.google.com with SMTP id 41be03b00d2f7-cb4cdf95b5aso5595922a12.1 for ; Sun, 02 Aug 2026 14:55:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1785707708; x=1786312508; darn=vger.kernel.org; h=content-transfer-encoding:content-type:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date:from:to:cc :subject:date:message-id:reply-to:content-type; bh=G37mEYyNGIkcVcWDIKuo53Q1LK9xSdA30i8azc+FTGc=; b=DO81pyuuJuR85ndcsnnbbXyEXpLmh0o2wqdUHtllFk8xKcKZi2pG4Cs7EIzVdUBVMb gxCwQp0HE6H1AjQJ/E5ufllP1Kk+fTqm0L05wFnwXJarKRpBb7t7Dfj82MKBgAKDPKdK XJC179A/JLx0H5jAUjRZSb1HrHmQkrs6VjyYl49/iA5T0QC5tt0lsSGsyJsIIhTTpl4V 00kiG6ujSDxbQAH6tZUObAAfAww0SegRkVPz+QrJDxQUs7GfQvTDeUDolZu/0B7kA7TV O7JHMNy8J5k/Fa1r6DjNllPwNWCGf8sdZiiChPgpuzYtFDUXK/10pw5Nv/RxpJLpJjL9 uROQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785707708; x=1786312508; h=content-transfer-encoding:content-type:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=G37mEYyNGIkcVcWDIKuo53Q1LK9xSdA30i8azc+FTGc=; b=hsxc35D99N2Wzs0S4LNA3OByfmSOh2eAmdhVVwPHKsMqtuJ4d+zRpqjypo7MizyNx/ KUfZVW5I3evTuoa929fC+WrnCI5PBD37PxMwgbdmDY9wg9CYIB1AuATbTqXfJYwV32e9 jwUMKq1bbprT4nsWE7D1MPZb8qZTc41XPiYJqTBWQucxZ/HbJSgMuGxQfiAkbSPS3Twg h+SKZR0bHwzWnxT/8d8SY7bQ57qgobvZgkvg7Morc2yBDk5quJpCxqXwxSxusjOzWbX3 LKUDO7NWSqJBaPBzsy9GIysGEjZOVHqNdyvamzaPqi7FEUnEKxB0mW0wOBWEPXNXUa1Z DaEA== X-Forwarded-Encrypted: i=1; AHgh+Rov0FpCMWTdPXLG8kBe2Jvalh42yjM6bS7OXnOhkBnGKsJbmgu1iAd/L2BzkbLD77rK2R9DLucGVKkSozc=@vger.kernel.org X-Gm-Message-State: AOJu0YxCDqUK2O47KG3pk22N0skaR+jRpEsiAI8zKUmL4mPUFfd7zVac CMov7OlQTq1S+Kq9VA3DMwxfM1BmnCEprdZRWAottg86h0n5KyUgfbfVP39ukSJDOcID1gs1aRK DgA86uQ== X-Received: from dlag2.prod.google.com ([2002:a05:701b:2502:b0:139:b9a8:f222]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:6f15:b0:3c4:1493:6822 with SMTP id adf61e73a8af0-3c92a59aca9mr7723441637.18.1785707708099; Sun, 02 Aug 2026 14:55:08 -0700 (PDT) Date: Sun, 2 Aug 2026 14:54:56 -0700 In-Reply-To: <20260802215459.2769283-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260802215459.2769283-1-surenb@google.com> X-Mailer: git-send-email 2.55.0.508.g3f0d502094-goog Message-ID: <20260802215459.2769283-3-surenb@google.com> Subject: [PATCH v3 2/5] binder: Make shrinker rely solely on per-VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: dave.hansen@linux.intel.com, Liam.Howlett@oracle.com, ljs@kernel.org, david@redhat.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@kernel.org, jannh@google.com, aliceryhl@google.com, arve@android.com, cmllamas@google.com, christian@brauner.io, tkjos@android.com, dsahern@kernel.org, davem@davemloft.net, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Dave Hansen tl;dr: lock_vma_under_rcu() is already a trylock. No need to do both it and mmap_read_trylock(). Long Version: =3D=3D Background =3D=3D Historically, binder used an mmap_read_trylock() in its shrinker code. This ensures that reclaim is not blocked on an mmap_lock. Commit 95bc2d4a9020 ("binder: use per-vma lock in page reclaiming") added support for the per-VMA lock, but left mmap_read_trylock() as a fallback. This was presumably because the per-VMA locking can fail for several reasons and most (all?) lock_vma_under_rcu() callers have a fallback to mmap_read_trylock(). =3D=3D Problem =3D=3D The fallback is not worth the complexity here. lock_vma_under_rcu() is essentially already a non-blocking trylock. The main reason it fails is also the reason mmap_read_trylock() fails: something is holding mmap_write_lock(). The only remedy for a collision with mmap_write_lock() is to wait, which this code can not do. So the "fallback" after lock_vma_under_rcu() failure is not really a fallback: it is really likely to just be retrying in vain. That retry in an of itself isn't horrible. But it adds complexity. =3D=3D Solution =3D=3D Now that per-VMA locks are universally available, lock_vma_under_rcu() will not persistently fail. Rely on it alone and simplify the code. Full disclosure: I originally tried to do this with lock_vma_under_rcu_wait(), but it did not fit well with the mmap_lock trylock semantics. Claude caught this in a review and suggested the approach in this path. It seemed sane to me. So, Suggesed-by: Claude, I guess. Signed-off-by: Dave Hansen Signed-off-by: Suren Baghdasaryan Cc: Andrew Morton Cc: "Liam R. Howlett" Cc: Vlastimil Babka Cc: Shakeel Butt Cc: linux-mm@kvack.org Cc: Greg Kroah-Hartman Cc: Arve Hj=C3=B8nnev=C3=A5g Cc: Todd Kjos Cc: Christian Brauner Cc: Carlos Llamas Cc: Alice Ryhl Cc: "David S. Miller" Cc: David Ahern Cc: netdev@vger.kernel.org Reviewed-by: Alice Ryhl --- drivers/android/binder_alloc.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index e4488ad86a65..84104ba04e30 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -1142,7 +1142,6 @@ enum lru_status binder_alloc_free_page(struct list_he= ad *item, struct vm_area_struct *vma; struct page *page_to_free; unsigned long page_addr; - int mm_locked =3D 0; size_t index; =20 if (!mmget_not_zero(mm)) @@ -1151,14 +1150,20 @@ enum lru_status binder_alloc_free_page(struct list_= head *item, index =3D mdata->page_index; page_addr =3D alloc->vm_start + index * PAGE_SIZE; =20 - /* attempt per-vma lock first */ + /* + * Attempt per-vma lock. This is essentially a + * "trylock". It can fail even if the VMA exists + * for 'page_addr'. + */ vma =3D lock_vma_under_rcu(mm, page_addr); if (!vma) { - /* fall back to mmap_lock */ - if (!mmap_read_trylock(mm)) - goto err_mmap_read_lock_failed; - mm_locked =3D 1; - vma =3D vma_lookup(mm, page_addr); + /* + * If the vma exists, we can't continue because we cannot + * remove the page from the vma. However, if the vma was + * unmapped, it's okay to continue. + */ + if (binder_alloc_is_mapped(alloc)) + goto err_vma_lock_failed; } =20 if (!mutex_trylock(&alloc->mutex)) @@ -1191,9 +1196,7 @@ enum lru_status binder_alloc_free_page(struct list_he= ad *item, } =20 mutex_unlock(&alloc->mutex); - if (mm_locked) - mmap_read_unlock(mm); - else + if (vma) vma_end_read(vma); mmput_async(mm); binder_free_page(page_to_free); @@ -1203,11 +1206,9 @@ enum lru_status binder_alloc_free_page(struct list_h= ead *item, err_invalid_vma: mutex_unlock(&alloc->mutex); err_get_alloc_mutex_failed: - if (mm_locked) - mmap_read_unlock(mm); - else + if (vma) vma_end_read(vma); -err_mmap_read_lock_failed: +err_vma_lock_failed: mmput_async(mm); err_mmget: return LRU_SKIP; --=20 2.55.0.508.g3f0d502094-goog From nobody Fri Oct 2 10:07:37 2026 Received: from mail-pj1-f70.google.com (mail-pj1-f70.google.com [209.85.216.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DE1D3DCD86 for ; Sun, 2 Aug 2026 21:55:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785707713; cv=none; b=U0IUPSaRgvSq51oZcykK8Rm6ZwvQgs5bHN++yCtonUEenCyX5Jp2fBrfCvGSLizp4Lbwy2A4BO6gY2MPfoInsDlAkRPT8VCgHzSgQUIZ7ev/SqaLlMY7Ut1tKyfCPXDfeVvkqzMttUztcic21EcjH71uhcUKhMO58vN2+FZws7s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785707713; c=relaxed/simple; bh=+YqZSzP/I2a03BPXHHL01Weg11cKi/MFACPBRAnB+Jk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WO6Tv/xnn7W51uW7vKCb+kuoKjPRs+kV5wkcNaQqXg1rypb0ktblbEXmGLywJfUGDuDgRF84ypKVz1j8nUJBU7iAsETrYPhaFZ8Ie8OqFb44dioUsVVyRNrQL5nwAIXiGSezBo8YIPdZzMmci5BkdoDPOLk2mIf+UTfKd/oZziw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NRGRHJ4E; arc=none smtp.client-ip=209.85.216.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NRGRHJ4E" Received: by mail-pj1-f70.google.com with SMTP id 98e67ed59e1d1-38ea32e57e2so1146999a91.1 for ; Sun, 02 Aug 2026 14:55:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1785707710; x=1786312510; darn=vger.kernel.org; h=content-transfer-encoding:content-type:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date:from:to:cc :subject:date:message-id:reply-to:content-type; bh=PbBMXqyTQfOdEQP+TBH4NONBT8Q85IiQo7bFMy00X9U=; b=NRGRHJ4EaWt7f9EonWSdkutLRSqgmZXj9+ytdpE43RSjDD9RJrTbvZTG6/x954oumj 1kCWh5IQWd+bDnx785Y0jvUCiSrPUPFAs3rooWWFpUWHzCanC3LUoNUYsV+JRGWPL44g 046aPjmghgTUbwLqu1/VZWp/Ni696/u/u31+nt9rYRJk42arLBPCKZruddspAZYnXTaR VR2gQSl23fJPjH8VJ6/1xsdMmNUF+9ANPqpDPPZmMUQIO4qndVEU619ziVjwc+BSLEUS GvPlOSn7c/aKby7b5crJa4p7l0Uuq+dDOYD7s6WtZeTSQeEkNj6KHc1JCuq25dverjMo mSaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785707710; x=1786312510; h=content-transfer-encoding:content-type:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=PbBMXqyTQfOdEQP+TBH4NONBT8Q85IiQo7bFMy00X9U=; b=AdUgxVCRbo0a19UWIzcviWbaZPgam1KwAo4pJrxYn9l7aXANuBa/TCuWeayx8KK4Mv Aiq/2rcrr1zLQMLAnbZdlAi7q/SpXNCuPSDMgURnS2GBMUur6VhpRIGNT3ouhTCTSXV3 ZVVMLOvDHizaBoMlms7ilBjXmFGnsXvzrDnn3tHmZ0ZQ6h72bSWeYKXdKKY3MmHiu63e oSRO/Vb0GgwUyrCUv7JqQpWz9U1gyhimHtZ234x9z0wgLIXecaJDgMYzlCtT4RFJKVgX irSlG8MvPfoXOO39tv7sMS2dYKKNJBPKWKQy29CppyFiwf3sO1FkKxK6kTlpCs5kAlFU XrXg== X-Forwarded-Encrypted: i=1; AHgh+RpZ0v+bpebMeYCI1A9pE74mY07uMJ1JOh3695xScrqycR+X5JLtuAaOn2pGcIjpeZ/yUKgw3h01gwyI1jA=@vger.kernel.org X-Gm-Message-State: AOJu0YyvaneSE/EyBCn19v8zVycq0xZdRzjywU2ttm9YBjWQ7O9+hU2H mk7a4H8t05CCbIDpbz+EkbUx9ACzOH85ww/r7eS+XzGtnOeMsqGWALBMr6YwT8NWI12ewNT5F1Z fXnZHQg== X-Received: from dlbou23.prod.google.com ([2002:a05:7022:1117:b0:13e:6e68:2a1]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4d0c:b0:37f:c69d:ce69 with SMTP id 98e67ed59e1d1-38fbc432bb3mr6817011a91.10.1785707710250; Sun, 02 Aug 2026 14:55:10 -0700 (PDT) Date: Sun, 2 Aug 2026 14:54:57 -0700 In-Reply-To: <20260802215459.2769283-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260802215459.2769283-1-surenb@google.com> X-Mailer: git-send-email 2.55.0.508.g3f0d502094-goog Message-ID: <20260802215459.2769283-4-surenb@google.com> Subject: [PATCH v3 3/5] mm: Add RCU-based VMA lookup helper that waits for writers From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: dave.hansen@linux.intel.com, Liam.Howlett@oracle.com, ljs@kernel.org, david@redhat.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@kernel.org, jannh@google.com, aliceryhl@google.com, arve@android.com, cmllamas@google.com, christian@brauner.io, tkjos@android.com, dsahern@kernel.org, davem@davemloft.net, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Dave Hansen =3D=3D Background =3D=3D There are basically two parallel ways to look up a VMA: the traditional way, which is protected by mmap_read_lock, and the RCU-based per-VMA lock way which is based on RCU and refcounts. =3D=3D Problem =3D=3D The mmap_lock one is more straightforward to use but it has a big disadvantage in that it can not be mixed with page faults since those can take mmap_lock for read, which can deadlock when mixed with nested page faults and parallel writers. For example: mmap_read_lock(mm); // Another thread does mmap_write_lock(). // New mmap_lock readers are blocked. vma =3D vma_lookup(mm, address); // This deadlocks on mmap_read_lock() if it faults: copy_from_user(address); mmap_read_unlock(mm); The per-VMA lock can be mixed with faults, but they can fail and need to be able to fall back to the traditional way. =3D=3D Solution =3D=3D Add a variant of the RCU-based lookup that waits for writers. This is basically the same as the existing RCU-based lookup, but on a failure to lock it temporarily takes mmap_lock for read and waits for writers to finish before locking the VMA, dropping the mmap_lock and returning the locked VMA. This has some advantages: 1. Callers do not need to have a fallback path for when they collide with writers. 2. It can be used in contexts where page faults can happen because it can take the mmap_lock for read but never *holds* it. 3. Its fast path does not require taking mmap_lock for read. Basically, when applied correctly, this approach results in faster *and* simpler code. Signed-off-by: Dave Hansen Signed-off-by: Suren Baghdasaryan Cc: Suren Baghdasaryan Cc: Andrew Morton Cc: "Liam R. Howlett" Cc: Lorenzo Stoakes Cc: Vlastimil Babka Cc: Shakeel Butt Cc: linux-mm@kvack.org Cc: Greg Kroah-Hartman Cc: Arve Hj=C3=B8nnev=C3=A5g Cc: Todd Kjos Cc: Christian Brauner Cc: Carlos Llamas Cc: Alice Ryhl Cc: "David S. Miller" Cc: David Ahern Cc: netdev@vger.kernel.org Suggested-by: Lorenzo Stoakes (ARM) --- include/linux/mmap_lock.h | 15 +++++++++++---- mm/mmap_lock.c | 29 +++++++++++++++++++++++++++++ mm/userfaultfd.c | 6 ++++-- 3 files changed, 44 insertions(+), 6 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index eb32b482434e..fdd8f5cf5722 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -228,10 +228,12 @@ static inline void vma_refcount_put(struct vm_area_st= ruct *vma) } =20 /* - * Use only while holding mmap read lock which guarantees that locking wil= l not - * fail (nobody can concurrently write-lock the vma). vma_start_read() sho= uld + * Use only while holding mmap read lock which guarantees that vma lock is= not + * contended (nobody can concurrently write-lock the vma). vma_start_read(= ) should * not be used in such cases because it might fail due to mm_lock_seq over= flow. * This functionality is used to obtain vma read lock and drop the mmap re= ad lock. + * VMA can't be detached while we are holding mmap lock, therefore in prac= tice this + * function can fail only when there are so many readers that vm_refcnt ov= erflows. */ static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma= , int subclass) { @@ -247,16 +249,21 @@ static inline bool vma_start_read_locked_nested(struc= t vm_area_struct *vma, int } =20 /* - * Use only while holding mmap read lock which guarantees that locking wil= l not - * fail (nobody can concurrently write-lock the vma). vma_start_read() sho= uld + * Use only while holding mmap read lock which guarantees that vma lock is= not + * contended (nobody can concurrently write-lock the vma). vma_start_read(= ) should * not be used in such cases because it might fail due to mm_lock_seq over= flow. * This functionality is used to obtain vma read lock and drop the mmap re= ad lock. + * VMA can't be detached while we are holding mmap lock, therefore in prac= tice this + * function can fail only when there are so many readers that vm_refcnt ov= erflows. */ static inline bool vma_start_read_locked(struct vm_area_struct *vma) { return vma_start_read_locked_nested(vma, 0); } =20 +struct vm_area_struct *vma_start_read_unlocked(struct mm_struct *mm, + unsigned long address); + static inline void vma_end_read(struct vm_area_struct *vma) { vma_refcount_put(vma); diff --git a/mm/mmap_lock.c b/mm/mmap_lock.c index e20d01e8d38f..6ff05e68e61b 100644 --- a/mm/mmap_lock.c +++ b/mm/mmap_lock.c @@ -338,6 +338,35 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_st= ruct *mm, return NULL; } =20 +/* + * Find the VMA covering 'address' and lock it for reading. Waits for writ= ers to + * finish if the VMA is being modified. Returns NULL if there is no VMA co= vering + * 'address'. + * + * Use only in code paths where no mmap_lock and no VMA lock is held. + * + * The fast path does not take mmap_lock. + */ +struct vm_area_struct *vma_start_read_unlocked(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma; + + /* Fast path: return stable VMA covering 'address': */ + vma =3D lock_vma_under_rcu(mm, address); + if (vma) + return vma; + + /* Slow path: preclude VMA writers by temporarily getting mmap read lock.= */ + mmap_read_lock(mm); + vma =3D vma_lookup(mm, address); + if (vma && !vma_start_read_locked(vma)) + vma =3D NULL; + mmap_read_unlock(mm); + + return vma; +} + static struct vm_area_struct *lock_next_vma_under_mmap_lock(struct mm_stru= ct *mm, struct vma_iterator *vmi, unsigned long from_addr) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index edd90892f8cc..c3a0c38a3dc3 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -129,8 +129,10 @@ struct vm_area_struct *find_vma_and_prepare_anon(struc= t mm_struct *mm, * * Should be called without holding mmap_lock. * - * Return: A locked vma containing @address, -ENOENT if no vma is found, or - * -ENOMEM if anon_vma couldn't be allocated. + * Return: A locked vma containing @address, -ENOENT if no vma is found, + * -ENOMEM if anon_vma couldn't be allocated, or -EAGAIN if vma refcount + * overflow happened due to high number of readers and the caller should + * retry later. */ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, unsigned long address) --=20 2.55.0.508.g3f0d502094-goog From nobody Fri Oct 2 10:07:37 2026 Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA3913D810D for ; Sun, 2 Aug 2026 21:55:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785707717; cv=none; b=f45Fbkd+K+6ygq/311iDiFZRivdIGINIyhCIl2W2f/3FtsiZSDCOHKsZ2I5tZ2jNZo23b/jfTy9S3MJJN5h918wQ2S8zprvXrTr0EU3YrM2gcWNJfQW6v1fyFbrMUYL2MuPgxzgAOowvBM7TEfdcG4tdRruihownZqmc6ish/IA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785707717; c=relaxed/simple; bh=DjZ68FKnv53M8cPQ2cNyW+Y/DBTCZnr07T9sJL4GD0w=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NROZCVm/V1YDl5+393DqY3sR2bigM/dEKwKoptKID34/fxB0MLolGPO1PDqHWz1KV3xKuTAMr2Zll+czuNT1+2anbJlmbYx4/J6A57CJyR0KzvpcNciQ+x3ad0lVuRpLWBJxmM+B2QFpMzO9OZV7LlS2qj4TeEnGErMnKj0UztE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=KoPWbdpL; arc=none smtp.client-ip=209.85.215.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="KoPWbdpL" Received: by mail-pg1-f200.google.com with SMTP id 41be03b00d2f7-cb4cdf95b5aso5595966a12.1 for ; Sun, 02 Aug 2026 14:55:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1785707712; x=1786312512; darn=vger.kernel.org; h=content-transfer-encoding:content-type:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date:from:to:cc :subject:date:message-id:reply-to:content-type; bh=xyBuTBv3WW28OhNQPFgbAciWJmDFVPYFH/JX2U0eD1I=; b=KoPWbdpLIAlyYxGBFlDim95OE+DgpRXppgt0/g3OB6F/8AEzCVx+9b/KxJ3Dwz34Ds twxZHHKK5eyO+qE/h9Z45POih/qqe28s0C84HdtguC1gddbHjr3nm/9alsIr58QZy/HY 6QDD2pTjj/a+8oCM+X5QMf9GmSDDasAAhqKZY1fYAKihw5gL/hTumkKNu55FNUheAvU/ Q6ooHkS+HFhs6GvyzckuklNQZvd5yOBY19bxPZqpMxNpmv27NHrOHGp3gpvV6aE3ZoSE CIVU49Aody9lWhZsxL1Rwz3HFoBciRG74hH8nI2AC/Ttde6/yB0hdwaO53Of9o7heZ4f ARPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785707712; x=1786312512; h=content-transfer-encoding:content-type:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=xyBuTBv3WW28OhNQPFgbAciWJmDFVPYFH/JX2U0eD1I=; b=U9f3DPhvEn54oPeXOH84LbzVkDDU5Sdjd64/iIibhIyszYkHfUKBUonJjCK8/IOk2E 8B0Tp1ExQis1UO8ChtVHkri72lQE4nqj9mk306O7NtnEFzsyVOs3WsCt8TXggwuG2hIZ hLXbOMkgJo3295pXcTSLl9QM/VfLI8DO0Xq3SxixraSlOX7yX8cWjAPvCFTLN5zhql86 j+dcHkJONlq1oKYjheIYjt7N4es2QXS2h/eEqV48nMUEyexP/kvvI28M3+GgNpKF/BRL vOMmXYjnoaicEm7BX8yXlukfbmdwMe3b0XN+ZkwSLMKN7g/G934sjIuhzo1oqM+3MMmh SfaQ== X-Forwarded-Encrypted: i=1; AHgh+RqHl568aj5RaxulgOI4ak3+TmcixH5GwHv3JayeQkzS5EnCy1NsKWs3MPsIuzGDPr3j46v0Kch8hYTW+s8=@vger.kernel.org X-Gm-Message-State: AOJu0YyFEV8ThsU5gxNHZH3fnR1C3cn44ypUa5d9nlvUyGsPajns98le EWeIXR/sV15HpY9FQqMF7RyNCN1GcUKQJL6aiJIlIQXK3IsM9HfHuJnwwUOLkW5chpL9a3tlQsT ID3aRkA== X-Received: from dled7-n2.prod.google.com ([2002:a05:701b:42c7:20b0:13b:962e:3bc8]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:505:b0:3c3:9a32:70b6 with SMTP id adf61e73a8af0-3c92a85f682mr8949124637.44.1785707712418; Sun, 02 Aug 2026 14:55:12 -0700 (PDT) Date: Sun, 2 Aug 2026 14:54:58 -0700 In-Reply-To: <20260802215459.2769283-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260802215459.2769283-1-surenb@google.com> X-Mailer: git-send-email 2.55.0.508.g3f0d502094-goog Message-ID: <20260802215459.2769283-5-surenb@google.com> Subject: [PATCH v3 4/5] binder: Remove mmap_lock fallback From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: dave.hansen@linux.intel.com, Liam.Howlett@oracle.com, ljs@kernel.org, david@redhat.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@kernel.org, jannh@google.com, aliceryhl@google.com, arve@android.com, cmllamas@google.com, christian@brauner.io, tkjos@android.com, dsahern@kernel.org, davem@davemloft.net, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, surenb@google.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Dave Hansen Previously, the per-VMA locking could fail in the face of writers which necessitate a fallback to mmap_lock. The new vma_start_read_unlocked() will wait for writers instead of failing. Use the new helper. Wait for writers. Remove the fallback to mmap_lock. Signed-off-by: Dave Hansen Signed-off-by: Suren Baghdasaryan Cc: Andrew Morton Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Shakeel Butt Cc: linux-mm@kvack.org Cc: Greg Kroah-Hartman Cc: Arve Hj=C3=B8nnev=C3=A5g Cc: Todd Kjos Cc: Christian Brauner Cc: Carlos Llamas Cc: Alice Ryhl Cc: David S. Miller Cc: David Ahern Cc: netdev@vger.kernel.org Acked-by: Lorenzo Stoakes (ARM) Reviewed-by: Alice Ryhl --- drivers/android/binder/page_range.rs | 19 +++---------------- drivers/android/binder_alloc.c | 17 +++++------------ rust/kernel/mm.rs | 18 ++++++++++++++++++ 3 files changed, 26 insertions(+), 28 deletions(-) diff --git a/drivers/android/binder/page_range.rs b/drivers/android/binder/= page_range.rs index e82a5523804f..f7ad88a0d806 100644 --- a/drivers/android/binder/page_range.rs +++ b/drivers/android/binder/page_range.rs @@ -439,22 +439,9 @@ unsafe fn use_page_slow(&self, i: usize) -> Result<()>= { // workqueue. let mm =3D MmWithUser::into_mmput_async(self.mm.mmget_not_zero().o= k_or(ESRCH)?); { - let vma_read; - let mmap_read; - let vma =3D if let Some(ret) =3D mm.lock_vma_under_rcu(vma_add= r) { - vma_read =3D ret; - check_vma(&vma_read, self) - } else { - mmap_read =3D mm.mmap_read_lock(); - mmap_read - .vma_lookup(vma_addr) - .and_then(|vma| check_vma(vma, self)) - }; - - match vma { - Some(vma) =3D> vma.vm_insert_page(user_page_addr, &new_pag= e)?, - None =3D> return Err(ESRCH), - } + let vma_read_guard =3D mm.vma_start_read_unlocked(vma_addr).ok= _or(ESRCH)?; + let vma =3D check_vma(&vma_read_guard, self).ok_or(ESRCH)?; + vma.vm_insert_page(user_page_addr, &new_page)?; } =20 let inner =3D self.lock.lock(); diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 84104ba04e30..519dcded19b2 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -259,21 +259,14 @@ static int binder_page_insert(struct binder_alloc *al= loc, struct vm_area_struct *vma; int ret =3D -ESRCH; =20 - /* attempt per-vma lock first */ - vma =3D lock_vma_under_rcu(mm, addr); - if (vma) { - if (binder_alloc_is_mapped(alloc)) - ret =3D vm_insert_page(vma, addr, page); - vma_end_read(vma); + vma =3D vma_start_read_unlocked(mm, addr); + if (!vma) return ret; - } =20 - /* fall back to mmap_lock */ - mmap_read_lock(mm); - vma =3D vma_lookup(mm, addr); - if (vma && binder_alloc_is_mapped(alloc)) + if (binder_alloc_is_mapped(alloc)) ret =3D vm_insert_page(vma, addr, page); - mmap_read_unlock(mm); + + vma_end_read(vma); =20 return ret; } diff --git a/rust/kernel/mm.rs b/rust/kernel/mm.rs index 2633e704c83d..877fad68be9c 100644 --- a/rust/kernel/mm.rs +++ b/rust/kernel/mm.rs @@ -190,6 +190,24 @@ pub fn lock_vma_under_rcu(&self, vma_addr: usize) -> O= ption> { } } =20 + /// Find the VMA covering 'address' and lock it for reading. Waits for= writers to finish if the + /// VMA is being modified. + #[inline] + pub fn vma_start_read_unlocked(&self, vma_addr: usize) -> Option> { + // SAFETY: We may invoke `vma_start_read_unlocked` because we know= this `mm` has non-zero + // `mm_users`. + let vma =3D unsafe { bindings::vma_start_read_unlocked(self.as_raw= (), vma_addr) }; + if vma.is_null() { + return None; + } + Some(VmaReadGuard { + // SAFETY: If `vma_start_read_unlocked` returns a non-null ptr= , then it points at a + // valid vma. The vma is stable for as long as the vma read lo= ck is held. + vma: unsafe { VmaRef::from_raw(vma) }, + _nts: NotThreadSafe, + }) + } + /// Lock the mmap read lock. #[inline] pub fn mmap_read_lock(&self) -> MmapReadGuard<'_> { --=20 2.55.0.508.g3f0d502094-goog From nobody Fri Oct 2 10:07:37 2026 Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 63887396B98 for ; Sun, 2 Aug 2026 21:55:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.198 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785707720; cv=none; b=lza9Kg7lAUGkzipKRlRbC3OKpZHrByGNA30RxwcU5OVf1GG0XaC+g6NFelKC88nm4rpd1Ea2tKpTrWXqFpkAsUdYcvgaBoYL31KO+35JkLG2QXHzmA048DIsEEmUWp8Oqy16k8TywXtAryDZRr4xRZXNl2OmGGNvMDk8hmLriOQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785707720; c=relaxed/simple; bh=ehiROkTUONMb1XfowgmGCN5Etu/lait0K9h2IywdBVw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XPaEHfjmTYsFsj64G/yH3n3g1P31Z0L73d0HhESSthEVROhJvE/HacVTeP/e7PQpvMUgHl8Q2RwRv/HR6ZayyFi2OQw0eOuk91Hrp23+RLzMvWNMqr8Ys0nmsC3nHvSRjLXE19EPjGSmMAiXbplvW3KzDutgBXIq07xChG9fPvk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VMKyf0mU; arc=none smtp.client-ip=209.85.215.198 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--surenb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VMKyf0mU" Received: by mail-pg1-f198.google.com with SMTP id 41be03b00d2f7-cb835525b10so3612111a12.2 for ; Sun, 02 Aug 2026 14:55:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1785707715; x=1786312515; darn=vger.kernel.org; h=content-transfer-encoding:content-type:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date:from:to:cc :subject:date:message-id:reply-to:content-type; bh=FNzfa/Q2B0yr5cHe2iHTydpaM2L8dTGY4PL8h7AQ+vM=; b=VMKyf0mU/9SO7w/E6ybQZ4ENACjFKRDBksqBCXWpemQN/KzAMU5F1k1AzZKdmdL4vz GbT/eqY/LBIzayWtik7o6xGHolJrryrf/tBhI134vqW+kiWbLcwObmJqUzvLqEfCjHZD JiStzo6wdtLIxTNWKPrxXrpGcjJqaqG9r5LCfobm+nQMfJu+qV7vITkbsmPX7eFfw0pG eAy2UzVLQBjAK+m9t/nYBFSuHLYkUm+gnV5si98lNT7yefrnVrNbD80VO1B4srkHrsFH XOV854NmNpNzSaqjggq8zelePSasPHmOXnooOqoKmN4b5dCa4VcQg1hg7InlOtUZ9tak Fl3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785707715; x=1786312515; h=content-transfer-encoding:content-type:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=FNzfa/Q2B0yr5cHe2iHTydpaM2L8dTGY4PL8h7AQ+vM=; b=gRrSZHfbsunuP9sFjSq08ZARepTPu0R3D+In5Aqy3NrcI7mVDA7w/ZoIUnK2HNa36L 7gnuNMn9ITI5BKBoCHnYBDECRPZgX5fBJaMRaFmCaznAGozh9Y7pXJPB8LV1ZUIbdPJi PthhCJO8czYTdhVo3dFePGhYWb4Z19CDRLxVlKzjMHc1LGQ0Hp/0e2F2bqrO2zmiZmUi iEtFj9kChT8G2i+awjVi4c8FsQNQwZfNjYincE1KQNrmDXrU/gBg9OcPvb6QJCNjpaaA eFJqH3SZYFbcjAeHfIT4gU0kAl7toat+bYVmxqcmm64+0d7aerTvKEZoOZ9/cM6TmCqo phEA== X-Forwarded-Encrypted: i=1; AHgh+RrILzoYTkPL8pY0sv0OVQrBimJzwmvbOp1mOC1LBBwGPLGYdI6iRST2Sbc+uDzKs77pBNydkwRFHG+WfFk=@vger.kernel.org X-Gm-Message-State: AOJu0YycJNInDvZbn7An+pnIMhsqlq7uKDw5r9CTxesm1t0aiIkoMKWT z/HLQjz1WS9VRTTOeVfCujwPJxSYppt9wvOfNICPUgo0c39qsQmKulCbOWf8NFxASOYjskeu9xD 7G876Iw== X-Received: from dlep14-n1.prod.google.com ([2002:a05:701b:458e:10b0:13d:2928:7996]) (user=surenb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:d80f:b0:3ba:cd5b:3dc8 with SMTP id adf61e73a8af0-3c92a5f1c89mr7273098637.31.1785707714551; Sun, 02 Aug 2026 14:55:14 -0700 (PDT) Date: Sun, 2 Aug 2026 14:54:59 -0700 In-Reply-To: <20260802215459.2769283-1-surenb@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260802215459.2769283-1-surenb@google.com> X-Mailer: git-send-email 2.55.0.508.g3f0d502094-goog Message-ID: <20260802215459.2769283-6-surenb@google.com> Subject: [PATCH v3 5/5] tcp: Remove mmap_lock fallback path From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: dave.hansen@linux.intel.com, Liam.Howlett@oracle.com, ljs@kernel.org, david@redhat.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@kernel.org, jannh@google.com, aliceryhl@google.com, arve@android.com, cmllamas@google.com, christian@brauner.io, tkjos@android.com, dsahern@kernel.org, davem@davemloft.net, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, surenb@google.com, syzbot@syzkaller.appspotmail.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Dave Hansen Previously, the per-VMA locking could fail in the face of writers which necessitates a fallback to mmap_lock. The new lock_vma_under_rcu_wait() will wait for writers instead of failing. Use the new helper. Wait for writers. Remove the fallback to mmap_lock. This really is a nice cleanup. It removes the need to pass the lock state back and forth to find_tcp_vma(). Signed-off-by: Dave Hansen Signed-off-by: Suren Baghdasaryan Acked-by: Lorenzo Stoakes Acked-by: Vlastimil Babka (SUSE) Tested-by: syzbot@syzkaller.appspotmail.com Cc: Andrew Morton Cc: Liam R. Howlett Cc: Vlastimil Babka Cc: Shakeel Butt Cc: linux-mm@kvack.org Cc: Greg Kroah-Hartman Cc: Arve Hj=C3=B8nnev=C3=A5g Cc: Todd Kjos Cc: Christian Brauner Cc: Carlos Llamas Cc: Alice Ryhl Cc: David S. Miller Cc: David Ahern Cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 31 +++++++++---------------------- 1 file changed, 9 insertions(+), 22 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 455441f1b694..62defe70f3ce 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2167,27 +2167,18 @@ static void tcp_zc_finalize_rx_tstamp(struct sock *= sk, } =20 static struct vm_area_struct *find_tcp_vma(struct mm_struct *mm, - unsigned long address, - bool *mmap_locked) + unsigned long address) { - struct vm_area_struct *vma =3D lock_vma_under_rcu(mm, address); + struct vm_area_struct *vma =3D vma_start_read_unlocked(mm, address); =20 - if (vma) { - if (vma->vm_ops !=3D &tcp_vm_ops) { - vma_end_read(vma); - return NULL; - } - *mmap_locked =3D false; - return vma; - } + if (!vma) + return NULL; =20 - mmap_read_lock(mm); - vma =3D vma_lookup(mm, address); - if (!vma || vma->vm_ops !=3D &tcp_vm_ops) { - mmap_read_unlock(mm); + if (vma->vm_ops !=3D &tcp_vm_ops) { + vma_end_read(vma); return NULL; } - *mmap_locked =3D true; + return vma; } =20 @@ -2208,7 +2199,6 @@ static int tcp_zerocopy_receive(struct sock *sk, u32 seq =3D tp->copied_seq; u32 total_bytes_to_map; int inq =3D tcp_inq(sk); - bool mmap_locked; int ret; =20 zc->copybuf_len =3D 0; @@ -2233,7 +2223,7 @@ static int tcp_zerocopy_receive(struct sock *sk, return 0; } =20 - vma =3D find_tcp_vma(current->mm, address, &mmap_locked); + vma =3D find_tcp_vma(current->mm, address); if (!vma) return -EINVAL; =20 @@ -2315,10 +2305,7 @@ static int tcp_zerocopy_receive(struct sock *sk, zc, total_bytes_to_map); } out: - if (mmap_locked) - mmap_read_unlock(current->mm); - else - vma_end_read(vma); + vma_end_read(vma); /* Try to copy straggler data. */ if (!ret) copylen =3D tcp_zc_handle_leftover(zc, sk, skb, &seq, copybuf_len, tss); --=20 2.55.0.508.g3f0d502094-goog