[PATCH v3 00/40] mm: make VMA flag semantics explicit, eliminate VM_SPECIAL

Lorenzo Stoakes (ARM) posted 40 patches 1 week ago
Documentation/filesystems/mmap_prepare.rst |  81 +++++++++
arch/arm64/kvm/mmu.c                       |   4 +-
arch/powerpc/mm/book3s64/radix_tlb.c       |   6 +-
arch/powerpc/mm/nohash/e500_hugetlbpage.c  |   2 +-
arch/powerpc/mm/nohash/tlb.c               |   2 +-
arch/riscv/kvm/mmu.c                       |   2 +-
arch/riscv/mm/tlbflush.c                   |   2 +-
arch/s390/mm/gmap_helpers.c                |   6 +-
arch/sparc/mm/init_64.c                    |   2 +-
arch/x86/kernel/uprobes.c                  |   2 +-
drivers/gpu/drm/drm_gpusvm.c               |   5 +-
drivers/hsi/clients/cmt_speech.c           |  33 +---
drivers/infiniband/hw/hfi1/file_ops.c      |  83 +++------
drivers/scsi/sg.c                          | 115 ++++++-------
drivers/usb/mon/mon_bin.c                  |  82 +++++----
drivers/video/fbdev/core/fb_defio.c        |   6 +-
drivers/video/fbdev/ssd1307fb.c            |   2 +
fs/coredump.c                              |   6 +-
fs/fuse/dax.c                              |   2 +-
fs/hugetlbfs/inode.c                       |   2 +-
fs/proc/task_mmu.c                         |   8 +-
include/asm-generic/tlb.h                  |   4 +-
include/linux/hugetlb.h                    |   5 +-
include/linux/hugetlb_inline.h             |  28 ---
include/linux/mm.h                         | 266 +++++++++++++++++++++++++++--
include/linux/mm_types.h                   |  50 +++++-
include/linux/pagemap.h                    |   1 -
include/linux/rmap.h                       |   2 +-
include/linux/userfaultfd_k.h              |   1 -
kernel/bpf/arena.c                         |   3 +-
kernel/events/core.c                       |   2 +-
kernel/events/uprobes.c                    |   4 +-
kernel/sched/fair.c                        |   3 +-
mm/folio.c                                 |   2 +-
mm/gup.c                                   |  15 +-
mm/hmm.c                                   |   3 +-
mm/huge_memory.c                           |  31 ++--
mm/hugetlb.c                               |  14 +-
mm/internal.h                              |  81 +++++----
mm/ksm.c                                   |   4 +-
mm/madvise.c                               |  28 +--
mm/memory.c                                | 142 ++++++++++++---
mm/mempolicy.c                             |   5 +-
mm/migrate_device.c                        |  12 +-
mm/mlock.c                                 |  51 +++---
mm/mmap.c                                  |   2 +-
mm/mmu_gather.c                            |   2 +-
mm/mprotect.c                              |   5 +-
mm/mremap.c                                |  11 +-
mm/page_vma_mapped.c                       |   4 +-
mm/pagewalk.c                              |   2 +-
mm/rmap.c                                  |   4 +-
mm/swapfile.c                              |   2 +-
mm/userfaultfd.c                           |  45 +++--
mm/util.c                                  |  29 +++-
mm/vma.c                                   | 246 +++++++++++++++++++-------
mm/vma.h                                   |  31 +++-
mm/vma_internal.h                          |   1 -
mm/vmscan.c                                |   9 +-
security/selinux/selinuxfs.c               |  11 +-
sound/core/pcm_native.c                    |  38 ++---
tools/testing/vma/include/dup.h            |  80 +++++++--
tools/testing/vma/include/stubs.h          |   2 +-
tools/testing/vma/tests/merge.c            |  10 +-
64 files changed, 1169 insertions(+), 575 deletions(-)
[PATCH v3 00/40] mm: make VMA flag semantics explicit, eliminate VM_SPECIAL
Posted by Lorenzo Stoakes (ARM) 1 week ago
The VM_SPECIAL / VMA_SPECIAL_FLAGS mask conflates several unrelated
properties:

* Is this kernel-owned, whether MMIO, kernel-allocated pages, or ordinary
  pages a driver maps itself?
* Can it be expanded or merged?
* Is this a 'weird' case like mlock where migration might race and we
  'have' to set invalid flags to notify?
* Is it another 'weird' case where we just want to stop GUP from touching
  it?

Driver writers have often been confused about this, and who can blame them?

It also interacts badly with the eternal edgecase known as hugetlb - which
sets VMA_DONTEXPAND_BIT but doesn't also want to be treated like a
'special' flag.

Another issue is that we cannot make sensible assumptions about flag
use. It's not possible to assume VMA_IO_BIT means iommu because drivers
abuse it and mlock abuses it.

Special is also an overloaded term in mm. VDSO and VVAR mappings are also
called 'special' but they're special in a... special way.

Sometimes things are called special that are a subset of
VMA_SPECIAL_FLAGS (VMA_PFNMAP_BIT and VMA_MIXEDMAP_BIT for instance when it
comes to zapping or vm_normal_folio()).

There's a specific kind of special for THP too, which considers
PFN map, mixed map 'special' but DAX not.

It's all rather a mess.

This series brings some order to things by both limiting what drivers can
do with VMA flags and switching to using predicates that describe
behaviour, not arbitrary flags.

It establishes the invariant that only kernel-owned mappings may set
VMA_IO_BIT or clear VMA_MAYWRITE_BIT in an mmap hook, enforcing this by
validating VMA state after every mmap and mmap_prepare hook.

It updates usbmon and sg to mmap_prepare in order to do so, adding a new
mmap action for mapping discontiguous kernel pages, and has hfi1 and the
ALSA PCM status page map their pages eagerly instead.

It also establishes the invariant that VMA_MIXEDMAP_BIT be set when mapping
kernel memory, something that is usually the case but happens not to be for
some users - specifically defio, cmt_speech, uprobes and the bpf arena, all
of which are updated to do the right thing.

It replaces VM_SPECIAL and arbitrary flag tests with predicates that say
what is actually being tested:

   vma_is_kernel_owned()   Does a driver or kernel code manage a VMA's
                           life cycle?

  vma_is_fixed_mapping()   Is the VMA not permitted to be expanded or
                           merged?

     vma_is_persistent()   Do bytes written to the VMA stay written, and
                           bytes read stay the same unless userland changes
                           them?

         vma_can_merge()   Can the VMA be merged with a compatible
                           neighbour?

           vma_can_gup()   Can GUP obtain pages from the VMA, i.e. is it
                           neither a PFN map nor memory-mapped I/O?

Remaining raw VMA_IO_BIT, VMA_PFNMAP_BIT and VMA_MIXEDMAP_BIT tests
scattered across mm are also converted to predicates where it makes sense
to do so.

And also the opportunity is taken to eliminate THP's vma_is_special_huge()
which was an existing source of confusion.

Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
---
v3:
* Fixed up bug in patch 1 as reported by Mike - have to delay setting
  map->vma_flags until after action prepare, though map->vm_file needs to
  be set before for correct reference count management.
* Updated 4/40 to add a symmetric vm_end check as well as vm_start in case
  of a dangerously insane driver, as per Sashiko.
* Updated 8/40 to check if a driver did something REALLY stupid like having
  a NULL discontig_kernel_page_ops ptr, as per Sashiko.
* Updated 15/40 to trivially synchronise userland test comments.
* Updated 17/40 to correctly duplicate code to the userland VMA tests as
  per Sashiko.

v2:
* Rebased on mm-unstable.
* Introduced new patch to fix various mmap_prepare and file interactions
  that weren't quite right as per Sashiko. None impact anything upstream
  yet so it doesn't need to be a fix.
* Restore vma->vm_start if an mmap hook has moved it before tearing the
  VMA down, so we unmap the range we established rather than the one the
  hook invented, as per Sashiko.
* Reject a discontiguous kernel page batch of zero pages rather than
  looping forever, and bound batches by the pages remaining in the VMA,
  as per Sashiko.
* Add the missing map_kernel_discontig member to the userland VMA tests'
  copy of struct mmap_action as per Sashiko.
* Set VM_DONTEXPAND on hfi1's RCV_HDRQ, RCV_EGRBUF and RTAIL mappings, as
  dma_mmap_coherent() doesn't on the IOMMU-DMA path, as per Sashiko.
* Recompute vma->vm_page_prot in snd_pcm_mmap_status() after clearing
  VM_WRITE, as vm_insert_page() uses it immediately rather than at fault
  time, as per Sashiko.
* munlock_vma_folio() now tests VMA_LOCKED_MASK so an unmap racing the
  mlock walk still munlocks folios the walk has already counted, as per
  Sashiko.
https://lore.kernel.org/r/20260914-b4-mmap-prepare-vma-flag-sanify-v2-0-7d9781ed5361@kernel.org

v1:
https://lore.kernel.org/r/20260908-b4-mmap-prepare-vma-flag-sanify-v1-0-dacf19cce22b@kernel.org

---
Lorenzo Stoakes (ARM) (40):
      mm/vma: fix mmap_prepare file handling, remove file_doesnt_need_get
      mm/vma: predicate setting mmap_prepare VMA fields on new vma alloc
      mm/vma: introduce and use vma_[flags_]can_merge()
      mm: consistently validate VMA state after mmap[_prepare] hooks
      mm/vma: ensure mmap_prepare doesn't set actions on a mergeable vma
      mm: make map_kernel_pages_[prepare,complete] internal and unexported
      mm/vma: tidy up map kernel pages enum values
      mm: add mmap action for discontiguous kernel page mapping
      docs: filesystems: update mmap_prepare docs for discontig kernel pgs
      drivers/usb/mon: update to use mmap_prepare + map kernel pages
      infiniband: update hfi1 to use remap_vmalloc_range()
      selinux: reject writable opens of policy file, drop mmap shared/write check
      ALSA: pcm: use vm_insert_page() to map PCM status page
      bpf: arena: mark arena_map_mmap() mappings VM_MIXEDMAP
      mm/vma: add vma[_flags]_is_kernel_owned() predicates
      mm/vma: only allow mmap to clear VMA_MAYWRITE_BIT if kernel-owned
      mm/vma: add and use vma_[flags]_is_fixed_mapping
      scsi: sg: convert mmap hook to mmap_prepare and rework
      fbdev: defio: assert FBINFO_VIRTFB, drop VM_IO, add VM_MIXEDMAP
      HSI: cmt_speech: convert mmap hook to mmap_prepare, refactor
      mm/gup: error out early on !VMA_MAYREAD_BIT VMAs
      uprobes: remove VM_IO, set VM_MIXEDMAP for mapped kernel pages
      mm/mlock: clear VMA_LOCKED_MASK over mmap callback
      mm/mlock: eliminate weird VMA_IO_BIT abuse and simplify
      mm/vma: enforce that only kernel-owned mappings may set VMA_IO_BIT
      mm: remove VMA_IO_BIT check in vma[_flags]_is_kernel_owned()
      mm: remove hugetlb_inline.h
      mm: rename is_vm_hugetlb_page() to vma_is_hugetlb()
      mm: drop some redundant checks around hugetlb VMAs
      mm/madvise: update is_valid_guard_vma() to use vma_can_merge()
      mm/vma: introduce vma[_flags]_is_persistent()
      mm/uffd: use predicates for userfaultfd checks
      mm/madvise: use predicates for madvise(..., MADV_DOFORK)
      mm: eliminate VMA_SPECIAL_FLAGS usage when hugetlb explicitly tested
      mm: eliminate VMA_SPECIAL_FLAGS check in lru_gen_look_around()
      mm: avoid use of VMA_SPECIAL_FLAGS in migrate_vma_setup()
      mm: eliminate VM_SPECIAL, VMA_SPECIAL_FLAGS
      fuse: dax: do not set VM_MIXEDMAP
      mm/huge_memory: remove vma_is_special_huge()
      mm/vma: introduce and use vma[_flags]_can_gup()

 Documentation/filesystems/mmap_prepare.rst |  81 +++++++++
 arch/arm64/kvm/mmu.c                       |   4 +-
 arch/powerpc/mm/book3s64/radix_tlb.c       |   6 +-
 arch/powerpc/mm/nohash/e500_hugetlbpage.c  |   2 +-
 arch/powerpc/mm/nohash/tlb.c               |   2 +-
 arch/riscv/kvm/mmu.c                       |   2 +-
 arch/riscv/mm/tlbflush.c                   |   2 +-
 arch/s390/mm/gmap_helpers.c                |   6 +-
 arch/sparc/mm/init_64.c                    |   2 +-
 arch/x86/kernel/uprobes.c                  |   2 +-
 drivers/gpu/drm/drm_gpusvm.c               |   5 +-
 drivers/hsi/clients/cmt_speech.c           |  33 +---
 drivers/infiniband/hw/hfi1/file_ops.c      |  83 +++------
 drivers/scsi/sg.c                          | 115 ++++++-------
 drivers/usb/mon/mon_bin.c                  |  82 +++++----
 drivers/video/fbdev/core/fb_defio.c        |   6 +-
 drivers/video/fbdev/ssd1307fb.c            |   2 +
 fs/coredump.c                              |   6 +-
 fs/fuse/dax.c                              |   2 +-
 fs/hugetlbfs/inode.c                       |   2 +-
 fs/proc/task_mmu.c                         |   8 +-
 include/asm-generic/tlb.h                  |   4 +-
 include/linux/hugetlb.h                    |   5 +-
 include/linux/hugetlb_inline.h             |  28 ---
 include/linux/mm.h                         | 266 +++++++++++++++++++++++++++--
 include/linux/mm_types.h                   |  50 +++++-
 include/linux/pagemap.h                    |   1 -
 include/linux/rmap.h                       |   2 +-
 include/linux/userfaultfd_k.h              |   1 -
 kernel/bpf/arena.c                         |   3 +-
 kernel/events/core.c                       |   2 +-
 kernel/events/uprobes.c                    |   4 +-
 kernel/sched/fair.c                        |   3 +-
 mm/folio.c                                 |   2 +-
 mm/gup.c                                   |  15 +-
 mm/hmm.c                                   |   3 +-
 mm/huge_memory.c                           |  31 ++--
 mm/hugetlb.c                               |  14 +-
 mm/internal.h                              |  81 +++++----
 mm/ksm.c                                   |   4 +-
 mm/madvise.c                               |  28 +--
 mm/memory.c                                | 142 ++++++++++++---
 mm/mempolicy.c                             |   5 +-
 mm/migrate_device.c                        |  12 +-
 mm/mlock.c                                 |  51 +++---
 mm/mmap.c                                  |   2 +-
 mm/mmu_gather.c                            |   2 +-
 mm/mprotect.c                              |   5 +-
 mm/mremap.c                                |  11 +-
 mm/page_vma_mapped.c                       |   4 +-
 mm/pagewalk.c                              |   2 +-
 mm/rmap.c                                  |   4 +-
 mm/swapfile.c                              |   2 +-
 mm/userfaultfd.c                           |  45 +++--
 mm/util.c                                  |  29 +++-
 mm/vma.c                                   | 246 +++++++++++++++++++-------
 mm/vma.h                                   |  31 +++-
 mm/vma_internal.h                          |   1 -
 mm/vmscan.c                                |   9 +-
 security/selinux/selinuxfs.c               |  11 +-
 sound/core/pcm_native.c                    |  38 ++---
 tools/testing/vma/include/dup.h            |  80 +++++++--
 tools/testing/vma/include/stubs.h          |   2 +-
 tools/testing/vma/tests/merge.c            |  10 +-
 64 files changed, 1169 insertions(+), 575 deletions(-)
---
base-commit: 6b41451631cabf9ea3b384c2a099088e1598f963
change-id: 20260721-b4-mmap-prepare-vma-flag-sanify-2100425df5aa

Best regards,
-- 
Lorenzo Stoakes (ARM) <ljs@kernel.org>
Re: [PATCH v3 00/40] mm: make VMA flag semantics explicit, eliminate VM_SPECIAL
Posted by Lorenzo Stoakes (ARM) 1 day, 15 hours ago
I know everybody is very busy but I'd appreciate if people could take the time
to have a look at this if possible!

I asked an LLM to look at review replies per week in mm (see below), and
clearly 7.3 is an insane cycle, which I understand.

(And dropping the THP M, remarkably, has not resulted in a drop in review
workload for me).

However, I'd also ask people to perhaps 'give a little back' to those who
are doing a lot of review also :) review can be thankless at the best of
times, but having your own work sit there unreviewed for weeks while you
are working so hard to review others' work is a little much.

Thanks!

┌───────────────────┬──────┬──────┬──────┬──────┬──────┬──────┬───────┐
│      person       │ 6.17 │ 6.18 │ 6.19 │ 7.0  │ 7.1  │ 7.2  │  7.3  │
├───────────────────┼──────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ David Hildenbrand │ 84.8 │ 70.8 │ 44.5 │ 74.7 │ 80.3 │ 85.7 │ 107.0 │
├───────────────────┼──────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ Lorenzo Stoakes   │ 50.9 │ 25.1 │ 16.0 │ 45.4 │ 35.2 │ 40.1 │ 52.1  │
├───────────────────┼──────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ Mike Rapoport     │ 11.8 │ 17.7 │ 12.8 │ 17.8 │ 30.1 │ 21.0 │ 24.1  │
├───────────────────┼──────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ Zi Yan            │ 17.3 │ 12.8 │ 15.0 │ 14.4 │ 10.2 │ 24.0 │ 28.7  │
├───────────────────┼──────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ Vlastimil Babka   │ 15.6 │ 19.3 │ 12.5 │ 23.8 │ 17.8 │ 25.2 │ 14.7  │
├───────────────────┼──────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ SeongJae Park     │ 16.2 │ 7.7  │ 13.3 │ 23.4 │ 22.8 │ 17.0 │ 19.2  │
├───────────────────┼──────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ Shakeel Butt      │ 13.0 │ 5.3  │ 11.7 │ 9.6  │ 7.2  │ 8.2  │ 14.5  │
├───────────────────┼──────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ Harry Yoo         │ 10.6 │ 15.0 │ 11.7 │ 15.0 │ 10.9 │ 11.3 │ 6.8   │
├───────────────────┼──────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ Baolin Wang       │ 6.0  │ 3.0  │ 4.1  │ 7.1  │ 6.7  │ 6.4  │ 17.4  │
├───────────────────┼──────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ Kiryl Shutsemau   │ 5.6  │ 3.3  │ 2.2  │ 3.0  │ 0.9  │ 2.9  │ 16.2  │
└───────────────────┴──────┴──────┴──────┴──────┴──────┴──────┴───────┘
Re: [PATCH v3 00/40] mm: make VMA flag semantics explicit, eliminate VM_SPECIAL
Posted by Andrew Morton 1 week ago
On Thu, 17 Sep 2026 17:22:09 +0100 "Lorenzo Stoakes (ARM)" <ljs@kernel.org> wrote:

> The VM_SPECIAL / VMA_SPECIAL_FLAGS mask conflates several unrelated
> properties:
> 
> * Is this kernel-owned, whether MMIO, kernel-allocated pages, or ordinary
>   pages a driver maps itself?
> * Can it be expanded or merged?
> * Is this a 'weird' case like mlock where migration might race and we
>   'have' to set invalid flags to notify?
> * Is it another 'weird' case where we just want to stop GUP from touching
>   it?
> 
> ...
> 
> This series brings some order to things by both limiting what drivers can
> do with VMA flags and switching to using predicates that describe
> behaviour, not arbitrary flags.

Thanks, I've updated mm.git's mm-unstable branch to this version.

> v3:
> * Fixed up bug in patch 1 as reported by Mike - have to delay setting
>   map->vma_flags until after action prepare, though map->vm_file needs to
>   be set before for correct reference count management.
> * Updated 4/40 to add a symmetric vm_end check as well as vm_start in case
>   of a dangerously insane driver, as per Sashiko.
> * Updated 8/40 to check if a driver did something REALLY stupid like having
>   a NULL discontig_kernel_page_ops ptr, as per Sashiko.
> * Updated 15/40 to trivially synchronise userland test comments.
> * Updated 17/40 to correctly duplicate code to the userland VMA tests as
>   per Sashiko.

Here's how v3 altered mm.git:


 mm/internal.h                   |    4 ++-
 mm/memory.c                     |    2 -
 mm/vma.c                        |   34 +++++++++++++++++-------------
 mm/vma.h                        |    3 +-
 tools/testing/vma/include/dup.h |   24 ++++++++++++++++-----
 5 files changed, 45 insertions(+), 22 deletions(-)

--- a/mm/internal.h~b
+++ a/mm/internal.h
@@ -244,6 +244,7 @@ static inline void vma_close(struct vm_a
 static inline int mmap_file(struct file *file, struct vm_area_struct *vma)
 {
 	const unsigned long prev_start = vma->vm_start;
+	const unsigned long prev_end = vma->vm_end;
 	const vma_flags_t prev_flags = vma->flags;
 	int err;
 
@@ -263,9 +264,10 @@ static inline int mmap_file(struct file
 	if (unlikely(err))
 		return err;
 
-	err = mmap_hook_validate(prev_start, &prev_flags, vma);
+	err = mmap_hook_validate(prev_start, prev_end, &prev_flags, vma);
 	if (unlikely(err)) {
 		vma->vm_start = prev_start;
+		vma->vm_end = prev_end;
 		vma_close(vma);
 	}
 
--- a/mm/memory.c~b
+++ a/mm/memory.c
@@ -2653,7 +2653,7 @@ int map_discontig_kernel_pages_prepare(s
 		action->map_kernel_discontig.ops;
 
 	/* At minimum need to be able to get pages. */
-	if (WARN_ON_ONCE(!ops->get))
+	if (WARN_ON_ONCE(!ops || !ops->get))
 		return -EINVAL;
 
 	__map_kernel_pages_prepare(desc);
--- a/mm/vma.c~b
+++ a/mm/vma.c
@@ -2797,15 +2797,15 @@ static int mmap_validate_vma_flags(const
 }
 
 /* Check to ensure a driver hasn't done something crazy. */
-static int mmap_validate(unsigned long prev_start,
-			 unsigned long curr_start,
+static int mmap_validate(unsigned long prev_start, unsigned long prev_end,
+			 unsigned long curr_start, unsigned long curr_end,
 			 const vma_flags_t *prev_flags,
 			 const vma_flags_t *curr_flags)
 {
 	bool was_maywrite, is_maywrite;
 
-	/* Drivers cannot alter the address of the VMA. */
-	if (WARN_ON_ONCE(prev_start != curr_start))
+	/* Drivers cannot alter the range of the VMA. */
+	if (WARN_ON_ONCE(prev_start != curr_start || prev_end != curr_end))
 		return -EINVAL;
 
 	was_maywrite = vma_flags_test(prev_flags, VMA_MAYWRITE_BIT);
@@ -2843,7 +2843,8 @@ int mmap_prepare_validate(const struct v
 	    WARN_ON_ONCE(desc->action.type != MMAP_NOTHING))
 		return -EINVAL;
 
-	return mmap_validate(prev_desc->start, desc->start,
+	return mmap_validate(prev_desc->start, prev_desc->end,
+			     desc->start, desc->end,
 			     &prev_desc->vma_flags, &desc->vma_flags);
 }
 
@@ -2851,19 +2852,22 @@ int mmap_prepare_validate(const struct v
  * mmap_hook_validate() - Ensure the driver hasn't violated invariants in
  * its f_op->mmap hook.
  * @prev_start: The start of the mapping prior to the mmap hook.
+ * @prev_end: The end of the mapping prior to the mmap hook.
  * @prev_flags: The VMA flags set for the VMA prior to the mmap hook.
  * @vma: The VMA after the hook has been applied.
  *
  * Returns: 0 on success, otherwise an error.
  */
-int mmap_hook_validate(unsigned long prev_start,
+int mmap_hook_validate(unsigned long prev_start, unsigned long prev_end,
 		       const vma_flags_t *prev_flags,
 		       const struct vm_area_struct *vma)
 {
 	const unsigned long start = vma->vm_start;
+	const unsigned long end = vma->vm_end;
 	const vma_flags_t *flags = &vma->flags;
 
-	return mmap_validate(prev_start, start, prev_flags, flags);
+	return mmap_validate(prev_start, prev_end, start, end, prev_flags,
+			     flags);
 }
 
 static int call_action_prepare(struct mmap_state *map,
@@ -2900,15 +2904,9 @@ static int call_mmap_prepare(struct mmap
 	if (err)
 		return err;
 
-	/* Update fields permitted to be changed. */
-	map->pgoff = desc->pgoff;
+	/* Update first so file refcount tracked correctly. */
 	if (desc->vm_file != map->vm_file)
 		map->vm_file = desc->vm_file;
-	map->vma_flags = desc->vma_flags;
-	map->page_prot = desc->page_prot;
-	/* User-defined fields. */
-	map->vm_ops = desc->vm_ops;
-	map->vm_private_data = desc->private_data;
 
 	/* It's invalid for mmap_prepare hooks to clear vm_ops. */
 	if (!desc->vm_ops)
@@ -2923,6 +2921,14 @@ static int call_mmap_prepare(struct mmap
 	if (err)
 		return err;
 
+	/* Update fields permitted to be changed. */
+	map->pgoff = desc->pgoff;
+	map->vma_flags = desc->vma_flags;
+	map->page_prot = desc->page_prot;
+	/* User-defined fields. */
+	map->vm_ops = desc->vm_ops;
+	map->vm_private_data = desc->private_data;
+
 	/*
 	 * MAP_PRIVATE-/dev/zero mappings are an ancient way of getting
 	 * anonymous mappings. Rather than allowing these mappings to be odd
--- a/mm/vma.h~b
+++ a/mm/vma.h
@@ -786,7 +786,7 @@ void vm_area_free(struct vm_area_struct
 int mmap_prepare_validate(const struct vm_area_desc *prev_desc,
 			  const struct vm_area_desc *desc);
 
-int mmap_hook_validate(unsigned long prev_start,
+int mmap_hook_validate(unsigned long prev_start, unsigned long prev_end,
 		       const vma_flags_t *prev_flags,
 		       const struct vm_area_struct *vma);
 
@@ -851,6 +851,7 @@ static inline int mmap_prepare_validate(
 }
 
 static inline int mmap_hook_validate(unsigned long prev_start,
+				     unsigned long prev_end,
 				     const vma_flags_t *prev_flags,
 				     const struct vm_area_struct *vma)
 {
--- a/tools/testing/vma/include/dup.h~b
+++ a/tools/testing/vma/include/dup.h
@@ -1667,22 +1667,36 @@ static inline bool vma_is_kernel_owned(c
 	return vma_flags_is_kernel_owned(&vma->flags);
 }
 
+static inline bool vma_flags_is_fixed_mapping(const vma_flags_t *flags)
+{
+	/*
+	 * VMA_PFNMAP_BIT should imply VMA_DONTEXPAND_BIT, but some callers set
+	 * only the former.
+	 */
+	return vma_flags_test_any(flags, VMA_PFNMAP_BIT, VMA_DONTEXPAND_BIT);
+}
+
+static inline bool vma_is_fixed_mapping(const struct vm_area_struct *vma)
+{
+	return vma_flags_is_fixed_mapping(&vma->flags);
+}
+
 static inline bool vma_flags_can_merge(const vma_flags_t *flags)
 {
 	/*
-	 * VMA merging assumes that the properties of a VMA completely describe
-	 * the properties of that VMA.
+	 * VMA merging assumes that a VMA's flags and fields completely describe
+	 * its state.
 	 *
 	 * However, kernel-owned mappings may have established state upon mapping
 	 * not embodied in any attribute of the VMA.
 	 *
-	 * Additionally, PFN maps encode the source PFN of the range in
-	 * vma->vm_pgoff, which may otherwise cause spurious merges.
+	 * Additionally, private (CoW) PFN maps encode the source PFN of the
+	 * range in vma->vm_pgoff, which may otherwise cause spurious merges.
 	 */
 	if (vma_flags_is_kernel_owned(flags))
 		return false;
 	/* VMA explicitly marked as being unmergeable. */
-	if (vma_flags_test(flags, VMA_DONTEXPAND_BIT))
+	if (vma_flags_is_fixed_mapping(flags))
 		return false;
 
 	return true;
_