Gather all the VMA flags whose presence implies that page tables must be
copied on fork into a single bitmap - VM_COPY_ON_FORK - and use this
rather than specifying individual flags in vma_needs_copy().
We also add VM_MAYBE_GUARD to this list, as it being set on a VMA implies
that there may be metadata contained in the page tables (that is - guard
markers) which would will not and cannot be propagated upon fork.
This was already being done manually previously in vma_needs_copy(), but
this makes it very explicit, alongside VM_PFNMAP, VM_MIXEDMAP and
VM_UFFD_WP all of which imply the same.
Note that VM_STICKY flags ought generally to be marked VM_COPY_ON_FORK too
- because equally a flag being VM_STICKY indicates that the VMA contains
metadat that is not propagated by being faulted in - i.e. that the VMA
metadata does not fully describe the VMA alone, and thus we must propagate
whatever metadata there is on a fork.
However, for maximum flexibility, we do not make this necessarily the case
here.
Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
---
include/linux/mm.h | 26 ++++++++++++++++++++++++++
mm/memory.c | 18 ++++--------------
tools/testing/vma/vma_internal.h | 26 ++++++++++++++++++++++++++
3 files changed, 56 insertions(+), 14 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index fea113d1d723..af2904aeb163 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -555,6 +555,32 @@ extern unsigned int kobjsize(const void *objp);
*/
#define VM_IGNORE_MERGE (VM_SOFTDIRTY | VM_STICKY)
+/*
+ * Flags which should result in page tables being copied on fork. These are
+ * flags which indicate that the VMA maps page tables which cannot be
+ * reconsistuted upon page fault, so necessitate page table copying upon
+ *
+ * VM_PFNMAP / VM_MIXEDMAP - These contain kernel-mapped data which cannot be
+ * reasonably reconstructed on page fault.
+ *
+ * VM_UFFD_WP - Encodes metadata about an installed uffd
+ * write protect handler, which cannot be
+ * reconstructed on page fault.
+ *
+ * We always copy pgtables when dst_vma has uffd-wp
+ * enabled even if it's file-backed
+ * (e.g. shmem). Because when uffd-wp is enabled,
+ * pgtable contains uffd-wp protection information,
+ * that's something we can't retrieve from page cache,
+ * and skip copying will lose those info.
+ *
+ * VM_MAYBE_GUARD - Could contain page guard region markers which
+ * by design are a property of the page tables
+ * only and thus cannot be reconstructed on page
+ * fault.
+ */
+#define VM_COPY_ON_FORK (VM_PFNMAP | VM_MIXEDMAP | VM_UFFD_WP | VM_MAYBE_GUARD)
+
/*
* mapping from the currently active vm_flags protection bits (the
* low four bits) to a page protection mask..
diff --git a/mm/memory.c b/mm/memory.c
index a520720702f0..732414852570 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1463,25 +1463,15 @@ copy_p4d_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
static bool
vma_needs_copy(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)
{
+ if (src_vma->vm_flags & VM_COPY_ON_FORK)
+ return true;
/*
- * Always copy pgtables when dst_vma has uffd-wp enabled even if it's
- * file-backed (e.g. shmem). Because when uffd-wp is enabled, pgtable
- * contains uffd-wp protection information, that's something we can't
- * retrieve from page cache, and skip copying will lose those info.
+ * The presence of an anon_vma indicates an anonymous VMA has page
+ * tables which naturally cannot be reconstituted on page fault.
*/
- if (userfaultfd_wp(dst_vma))
- return true;
-
- if (src_vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP))
- return true;
-
if (src_vma->anon_vma)
return true;
- /* Guard regions have modified page tables that require copying. */
- if (src_vma->vm_flags & VM_MAYBE_GUARD)
- return true;
-
/*
* Don't copy ptes where a page fault will fill them correctly. Fork
* becomes much lighter when there are big shared or private readonly
diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index 73c2025777e6..233819a9e7ee 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -145,6 +145,32 @@ extern unsigned long dac_mmap_min_addr;
*/
#define VM_IGNORE_MERGE (VM_SOFTDIRTY | VM_STICKY)
+/*
+ * Flags which should result in page tables being copied on fork. These are
+ * flags which indicate that the VMA maps page tables which cannot be
+ * reconsistuted upon page fault, so necessitate page table copying upon
+ *
+ * VM_PFNMAP / VM_MIXEDMAP - These contain kernel-mapped data which cannot be
+ * reasonably reconstructed on page fault.
+ *
+ * VM_UFFD_WP - Encodes metadata about an installed uffd
+ * write protect handler, which cannot be
+ * reconstructed on page fault.
+ *
+ * We always copy pgtables when dst_vma has uffd-wp
+ * enabled even if it's file-backed
+ * (e.g. shmem). Because when uffd-wp is enabled,
+ * pgtable contains uffd-wp protection information,
+ * that's something we can't retrieve from page cache,
+ * and skip copying will lose those info.
+ *
+ * VM_MAYBE_GUARD - Could contain page guard region markers which
+ * by design are a property of the page tables
+ * only and thus cannot be reconstructed on page
+ * fault.
+ */
+#define VM_COPY_ON_FORK (VM_PFNMAP | VM_MIXEDMAP | VM_UFFD_WP | VM_MAYBE_GUARD)
+
#define FIRST_USER_ADDRESS 0UL
#define USER_PGTABLES_CEILING 0UL
--
2.51.2On Tue, 18 Nov 2025 10:17:47 +0000 Lorenzo Stoakes <lorenzo.stoakes@oracle.com> wrote:
> Gather all the VMA flags whose presence implies that page tables must be
> copied on fork into a single bitmap - VM_COPY_ON_FORK - and use this
> rather than specifying individual flags in vma_needs_copy().
>
> We also add VM_MAYBE_GUARD to this list, as it being set on a VMA implies
> that there may be metadata contained in the page tables (that is - guard
> markers) which would will not and cannot be propagated upon fork.
>
> This was already being done manually previously in vma_needs_copy(), but
> this makes it very explicit, alongside VM_PFNMAP, VM_MIXEDMAP and
> VM_UFFD_WP all of which imply the same.
>
> Note that VM_STICKY flags ought generally to be marked VM_COPY_ON_FORK too
> - because equally a flag being VM_STICKY indicates that the VMA contains
> metadat that is not propagated by being faulted in - i.e. that the VMA
> metadata does not fully describe the VMA alone, and thus we must propagate
> whatever metadata there is on a fork.
>
> However, for maximum flexibility, we do not make this necessarily the case
> here.
>
Hi Lorenzo, one more from the review automation:
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index fea113d1d723c..af2904aeb1631 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -555,6 +555,32 @@ extern unsigned int kobjsize(const void *objp);
> */
> #define VM_IGNORE_MERGE (VM_SOFTDIRTY | VM_STICKY)
>
> +/*
> + * Flags which should result in page tables being copied on fork. These are
> + * flags which indicate that the VMA maps page tables which cannot be
> + * reconsistuted upon page fault, so necessitate page table copying upon
> + *
> + * VM_PFNMAP / VM_MIXEDMAP - These contain kernel-mapped data which cannot be
> + * reasonably reconstructed on page fault.
> + *
> + * VM_UFFD_WP - Encodes metadata about an installed uffd
> + * write protect handler, which cannot be
> + * reconstructed on page fault.
> + *
> + * We always copy pgtables when dst_vma has uffd-wp
^^^^^^^
The comment says "dst_vma" but the new code in vma_needs_copy() checks
src_vma->vm_flags. Is this intentional?
The old code checked userfaultfd_wp(dst_vma), which tests whether the
child VMA has VM_UFFD_WP set. After dup_userfaultfd() clears VM_UFFD_WP
from the child when !UFFD_FEATURE_EVENT_FORK, the old code would skip
page table copying since the child no longer needs uffd-wp protection.
The new code checks src_vma->vm_flags & VM_COPY_ON_FORK, which includes
VM_UFFD_WP. Since the parent VMA still has VM_UFFD_WP set, this will now
force page table copying even when the child has had VM_UFFD_WP cleared.
For file-backed VMAs (e.g., shmem) with uffd-wp but !UFFD_FEATURE_EVENT_FORK,
this appears to cause unnecessary page table copying where the old code
would have allowed the child to fault pages in fresh.
The call chain is:
dup_mmap()
-> vm_area_dup(mpnt) // child gets same flags
-> dup_userfaultfd(tmp, &uf) // may clear VM_UFFD_WP from child
-> copy_page_range(tmp, mpnt)
-> vma_needs_copy(dst_vma=tmp, src_vma=mpnt)
> + * enabled even if it's file-backed
> + * (e.g. shmem). Because when uffd-wp is enabled,
> + * pgtable contains uffd-wp protection information,
> + * that's something we can't retrieve from page cache,
> + * and skip copying will lose those info.
> + *
> + * VM_MAYBE_GUARD - Could contain page guard region markers which
> + * by design are a property of the page tables
> + * only and thus cannot be reconstructed on page
> + * fault.
> + */
> +#define VM_COPY_ON_FORK (VM_PFNMAP | VM_MIXEDMAP | VM_UFFD_WP | VM_MAYBE_GUARD)
[ ... ]
> diff --git a/mm/memory.c b/mm/memory.c
> index d1728d0538d64..27bc457b32c2e 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -1463,25 +1463,15 @@ copy_p4d_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
> static bool
> vma_needs_copy(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)
> {
> + if (src_vma->vm_flags & VM_COPY_ON_FORK)
^^^^^^^^
Was the change from checking dst_vma to src_vma for VM_UFFD_WP intentional?
The old code was:
if (userfaultfd_wp(dst_vma))
return true;
which expanded to:
if (dst_vma->vm_flags & VM_UFFD_WP)
return true;
On Tue, Jan 13, 2026 at 03:12:55PM -0800, Chris Mason wrote:
> On Tue, 18 Nov 2025 10:17:47 +0000 Lorenzo Stoakes <lorenzo.stoakes@oracle.com> wrote:
>
> > Gather all the VMA flags whose presence implies that page tables must be
> > copied on fork into a single bitmap - VM_COPY_ON_FORK - and use this
> > rather than specifying individual flags in vma_needs_copy().
> >
> > We also add VM_MAYBE_GUARD to this list, as it being set on a VMA implies
> > that there may be metadata contained in the page tables (that is - guard
> > markers) which would will not and cannot be propagated upon fork.
> >
> > This was already being done manually previously in vma_needs_copy(), but
> > this makes it very explicit, alongside VM_PFNMAP, VM_MIXEDMAP and
> > VM_UFFD_WP all of which imply the same.
> >
> > Note that VM_STICKY flags ought generally to be marked VM_COPY_ON_FORK too
> > - because equally a flag being VM_STICKY indicates that the VMA contains
> > metadat that is not propagated by being faulted in - i.e. that the VMA
> > metadata does not fully describe the VMA alone, and thus we must propagate
> > whatever metadata there is on a fork.
> >
> > However, for maximum flexibility, we do not make this necessarily the case
> > here.
> >
>
> Hi Lorenzo, one more from the review automation:
>
> > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > index fea113d1d723c..af2904aeb1631 100644
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -555,6 +555,32 @@ extern unsigned int kobjsize(const void *objp);
> > */
> > #define VM_IGNORE_MERGE (VM_SOFTDIRTY | VM_STICKY)
> >
> > +/*
> > + * Flags which should result in page tables being copied on fork. These are
> > + * flags which indicate that the VMA maps page tables which cannot be
> > + * reconsistuted upon page fault, so necessitate page table copying upon
> > + *
> > + * VM_PFNMAP / VM_MIXEDMAP - These contain kernel-mapped data which cannot be
> > + * reasonably reconstructed on page fault.
> > + *
> > + * VM_UFFD_WP - Encodes metadata about an installed uffd
> > + * write protect handler, which cannot be
> > + * reconstructed on page fault.
> > + *
> > + * We always copy pgtables when dst_vma has uffd-wp
> ^^^^^^^
> The comment says "dst_vma" but the new code in vma_needs_copy() checks
> src_vma->vm_flags. Is this intentional?
Hmm, looking deeper into it, because I assumed some basic level of sanity
here, but UFFD is terrible so that's never valid thinking I guess:
We invoke vma_needs_copy() from one place - copy_page_range().
And copy_page_range() from dup_mmap(), i.e. on fork.
There:
for_each_vma(vmi, mpnt) {
...
tmp = vm_area_dup(mpnt);
So in the dubious nomenclature of the fork code, 'tmp' is dst_vma and 'mpnt' is
src_vma.
vm_area_dup() calls vm_area_init_from() (delightfully flipping param order)
which calls vm_flags_init(), which copies src_vma's VMA flags to dst_vma.
Now dup_userfaultfd() is also invoked, which, if !UFFD_FEATURE_EVENT_FORK
in the original uffd context's features list, does userfaultfd_reset_ctx().
This invokes userfaultfd_set_ctx() with vma (equal to tmp, i.e. dst_vma
eventually) and vm_flags == 0, which then invokes userfaultfd_set_vm_flags
to vma->vm_flags & ~__VM_UFFD_FLAGS | vm_flags, where vm_flags is 0 so:
vma->vm_flags & ~(VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR)
That is, clearing VM_UFFD_WP on the the destination VMA, but not the
source.
(I really hate that any of this got merged, it's total insanity.)
>
> The old code checked userfaultfd_wp(dst_vma), which tests whether the
> child VMA has VM_UFFD_WP set. After dup_userfaultfd() clears VM_UFFD_WP
> from the child when !UFFD_FEATURE_EVENT_FORK, the old code would skip
> page table copying since the child no longer needs uffd-wp protection.
>
> The new code checks src_vma->vm_flags & VM_COPY_ON_FORK, which includes
> VM_UFFD_WP. Since the parent VMA still has VM_UFFD_WP set, this will now
> force page table copying even when the child has had VM_UFFD_WP cleared.
>
> For file-backed VMAs (e.g., shmem) with uffd-wp but !UFFD_FEATURE_EVENT_FORK,
> this appears to cause unnecessary page table copying where the old code
> would have allowed the child to fault pages in fresh.
>
> The call chain is:
>
> dup_mmap()
> -> vm_area_dup(mpnt) // child gets same flags
> -> dup_userfaultfd(tmp, &uf) // may clear VM_UFFD_WP from child
Yup as above, this abomination is in the source code.
> -> copy_page_range(tmp, mpnt)
> -> vma_needs_copy(dst_vma=tmp, src_vma=mpnt)
>
> > + * enabled even if it's file-backed
> > + * (e.g. shmem). Because when uffd-wp is enabled,
> > + * pgtable contains uffd-wp protection information,
> > + * that's something we can't retrieve from page cache,
> > + * and skip copying will lose those info.
> > + *
> > + * VM_MAYBE_GUARD - Could contain page guard region markers which
> > + * by design are a property of the page tables
> > + * only and thus cannot be reconstructed on page
> > + * fault.
> > + */
> > +#define VM_COPY_ON_FORK (VM_PFNMAP | VM_MIXEDMAP | VM_UFFD_WP | VM_MAYBE_GUARD)
>
> [ ... ]
>
> > diff --git a/mm/memory.c b/mm/memory.c
> > index d1728d0538d64..27bc457b32c2e 100644
> > --- a/mm/memory.c
> > +++ b/mm/memory.c
> > @@ -1463,25 +1463,15 @@ copy_p4d_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,
> > static bool
> > vma_needs_copy(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)
> > {
> > + if (src_vma->vm_flags & VM_COPY_ON_FORK)
> ^^^^^^^^
> Was the change from checking dst_vma to src_vma for VM_UFFD_WP intentional?
>
> The old code was:
>
> if (userfaultfd_wp(dst_vma))
> return true;
>
> which expanded to:
>
> if (dst_vma->vm_flags & VM_UFFD_WP)
> return true;
>
I'll send a fix.
On Wed, Jan 14, 2026 at 10:27:24AM +0000, Lorenzo Stoakes wrote: > I'll send a fix. Actually it's clear that assert is just wrong, so I'll drop it. we don't assert elsewhere, this seems to be a 'nice to have', and I'd rather not bother with a 'mmap or VMA lock' check here as it doesn't seem worth it. Will send shortly.
On 18.11.25 11:17, Lorenzo Stoakes wrote: > Gather all the VMA flags whose presence implies that page tables must be > copied on fork into a single bitmap - VM_COPY_ON_FORK - and use this > rather than specifying individual flags in vma_needs_copy(). > > We also add VM_MAYBE_GUARD to this list, as it being set on a VMA implies > that there may be metadata contained in the page tables (that is - guard > markers) which would will not and cannot be propagated upon fork. > > This was already being done manually previously in vma_needs_copy(), but > this makes it very explicit, alongside VM_PFNMAP, VM_MIXEDMAP and > VM_UFFD_WP all of which imply the same. > > Note that VM_STICKY flags ought generally to be marked VM_COPY_ON_FORK too > - because equally a flag being VM_STICKY indicates that the VMA contains > metadat that is not propagated by being faulted in - i.e. that the VMA > metadata does not fully describe the VMA alone, and thus we must propagate > whatever metadata there is on a fork. > > However, for maximum flexibility, we do not make this necessarily the case > here. > > Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com> > Reviewed-by: Pedro Falcato <pfalcato@suse.de> > Reviewed-by: Vlastimil Babka <vbabka@suse.cz> > --- Acked-by: David Hildenbrand (Red Hat) <david@kernel.org> -- Cheers David
© 2016 - 2026 Red Hat, Inc.