It is possible that the dup_mmap() call fails on allocating or setting
up a vma after the maple tree of the oldmm is copied.  Today, that
failure point is marked by inserting an XA_ZERO entry over the failure
point so that the exact location does not need to be communicated
through to exit_mmap().

However, a race exists in the tear down process because the dup_mmap()
drops the mmap lock before exit_mmap() can remove the partially set up
vma tree.  This means that other tasks may get to the mm tree and find
the invalid vma pointer (since it's an XA_ZERO entry), even though the
mm is marked as MMF_OOM_SKIP and MMF_UNSTABLE.

To remove the race fully, the tree must be cleaned up before dropping
the lock.  This is accomplished by extracting the vma cleanup in
exit_mmap() and changing the required functions to pass through the vma
search limit.  Any other tree modifications would require extra cycles
which should be spent on freeing memory.

This does run the risk of increasing the possibility of finding no vmas
(which is already possible!) in code this isn't careful.

The final three patches are to address the excessive argument lists
being passed between the functions.  Using the struct unmap_desc also
allows some special-case code to be removed in favour of the struct
setup differences.

RFC: https://lore.kernel.org/linux-mm/20250815191031.3769540-1-Liam.Howlett@oracle.com/
Changes since RFC:
- Change BUG_ON to WARN_ON_ONCE in tear_down_vmas() - Thanks Lorenzo
- Added sanity WARN_ON_ONCE() to free_pgtables() for ceiling and
  tree_max - Thanks Lorenzo
- Added comment to free_pgtables() in a vain attempt to inform users
  what's going on - Thanks Lorenzo
- Fixed free_pgtables() in testing code
- Added struct unmap_desc to describe the unmap event to reduce the
  argument list  - Thanks Lorenzo
- Rebased against mm-new

Liam R. Howlett (9):
  mm/mmap: Move exit_mmap() trace point
  mm/mmap: Abstract vma clean up from exit_mmap()
  mm/vma: Add limits to unmap_region() for vmas
  mm/memory: Add tree limit to free_pgtables()
  mm/vma: Add page table limit to unmap_region()
  mm: Change dup_mmap() recovery
  mm: Introduce unmap_desc struct to reduce function arguments
  mm/vma: Use unmap_desc in vms_clear_ptes() and exit_mmap()
  mm: Use unmap_desc struct for freeing page tables.

 include/linux/mm.h               |  3 -
 mm/internal.h                    |  8 ++-
 mm/memory.c                      | 69 +++++++++++++----------
 mm/mmap.c                        | 96 ++++++++++++++++++++++----------
 mm/vma.c                         | 55 +++++++++---------
 mm/vma.h                         | 48 +++++++++++++++-
 tools/testing/vma/vma_internal.h | 24 ++------
 7 files changed, 193 insertions(+), 110 deletions(-)

-- 
2.47.2