We have now introduced a mechanism that obviates the need for a reattempted
merge via the mmap_prepare() file hook, so eliminate this functionality
altogether.

The retry merge logic has been the cause of a great deal of complexity in
the past and required a great deal of careful manoeuvring of code to ensure
its continued and correct functionality.

It has also recently been involved in an issue surrounding maple tree
state, which again points to its problematic nature.

We make it much easier to reason about mmap() logic by eliminating this and
simply writing a VMA once. This also opens the doors to future optimisation
and improvement in the mmap() logic.

For any device or file system which encounters unwanted VMA fragmentation
as a result of this change (that is, having not implemented .mmap_prepare
hooks), the issue is easily resolvable by doing so.

Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
---
 mm/vma.c | 14 --------------
 1 file changed, 14 deletions(-)

diff --git a/mm/vma.c b/mm/vma.c
index 3f32e04bb6cc..3ff6cfbe3338 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -24,7 +24,6 @@ struct mmap_state {
 	void *vm_private_data;
 
 	unsigned long charged;
-	bool retry_merge;
 
 	struct vm_area_struct *prev;
 	struct vm_area_struct *next;
@@ -2417,8 +2416,6 @@ static int __mmap_new_file_vma(struct mmap_state *map,
 			!(map->flags & VM_MAYWRITE) &&
 			(vma->vm_flags & VM_MAYWRITE));
 
-	/* If the flags change (and are mergeable), let's retry later. */
-	map->retry_merge = vma->vm_flags != map->flags && !(vma->vm_flags & VM_SPECIAL);
 	map->flags = vma->vm_flags;
 
 	return 0;
@@ -2622,17 +2619,6 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr,
 	if (have_mmap_prepare)
 		set_vma_user_defined_fields(vma, &map);
 
-	/* If flags changed, we might be able to merge, so try again. */
-	if (map.retry_merge) {
-		struct vm_area_struct *merged;
-		VMG_MMAP_STATE(vmg, &map, vma);
-
-		vma_iter_config(map.vmi, map.addr, map.end);
-		merged = vma_merge_existing_range(&vmg);
-		if (merged)
-			vma = merged;
-	}
-
 	__mmap_complete(&map, vma);
 
 	return addr;
-- 
2.49.0

* Lorenzo Stoakes <lorenzo.stoakes@oracle.com> [250509 08:14]:
> We have now introduced a mechanism that obviates the need for a reattempted
> merge via the mmap_prepare() file hook, so eliminate this functionality
> altogether.
> 
> The retry merge logic has been the cause of a great deal of complexity in
> the past and required a great deal of careful manoeuvring of code to ensure
> its continued and correct functionality.
> 
> It has also recently been involved in an issue surrounding maple tree
> state, which again points to its problematic nature.
> 
> We make it much easier to reason about mmap() logic by eliminating this and
> simply writing a VMA once. This also opens the doors to future optimisation
> and improvement in the mmap() logic.
> 
> For any device or file system which encounters unwanted VMA fragmentation
> as a result of this change (that is, having not implemented .mmap_prepare
> hooks), the issue is easily resolvable by doing so.
> 
> Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
> Reviewed-by: David Hildenbrand <david@redhat.com>

I have a few tests for the vma test suite that would test this path.
I'll just let them go.

Reviewed-by: Liam R. Howlett <Liam.Howlett@oracle.com>

> ---
>  mm/vma.c | 14 --------------
>  1 file changed, 14 deletions(-)
> 
> diff --git a/mm/vma.c b/mm/vma.c
> index 3f32e04bb6cc..3ff6cfbe3338 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -24,7 +24,6 @@ struct mmap_state {
>  	void *vm_private_data;
>  
>  	unsigned long charged;
> -	bool retry_merge;
>  
>  	struct vm_area_struct *prev;
>  	struct vm_area_struct *next;
> @@ -2417,8 +2416,6 @@ static int __mmap_new_file_vma(struct mmap_state *map,
>  			!(map->flags & VM_MAYWRITE) &&
>  			(vma->vm_flags & VM_MAYWRITE));
>  
> -	/* If the flags change (and are mergeable), let's retry later. */
> -	map->retry_merge = vma->vm_flags != map->flags && !(vma->vm_flags & VM_SPECIAL);
>  	map->flags = vma->vm_flags;
>  
>  	return 0;
> @@ -2622,17 +2619,6 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr,
>  	if (have_mmap_prepare)
>  		set_vma_user_defined_fields(vma, &map);
>  
> -	/* If flags changed, we might be able to merge, so try again. */
> -	if (map.retry_merge) {
> -		struct vm_area_struct *merged;
> -		VMG_MMAP_STATE(vmg, &map, vma);
> -
> -		vma_iter_config(map.vmi, map.addr, map.end);
> -		merged = vma_merge_existing_range(&vmg);
> -		if (merged)
> -			vma = merged;
> -	}
> -
>  	__mmap_complete(&map, vma);
>  
>  	return addr;
> -- 
> 2.49.0
>

On Tue, May 13, 2025 at 6:25 AM Liam R. Howlett <Liam.Howlett@oracle.com> wrote:
>
> * Lorenzo Stoakes <lorenzo.stoakes@oracle.com> [250509 08:14]:
> > We have now introduced a mechanism that obviates the need for a reattempted
> > merge via the mmap_prepare() file hook, so eliminate this functionality
> > altogether.
> >
> > The retry merge logic has been the cause of a great deal of complexity in
> > the past and required a great deal of careful manoeuvring of code to ensure
> > its continued and correct functionality.
> >
> > It has also recently been involved in an issue surrounding maple tree
> > state, which again points to its problematic nature.
> >
> > We make it much easier to reason about mmap() logic by eliminating this and
> > simply writing a VMA once. This also opens the doors to future optimisation
> > and improvement in the mmap() logic.
> >
> > For any device or file system which encounters unwanted VMA fragmentation
> > as a result of this change (that is, having not implemented .mmap_prepare
> > hooks), the issue is easily resolvable by doing so.
> >
> > Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
> > Reviewed-by: David Hildenbrand <david@redhat.com>
>
> I have a few tests for the vma test suite that would test this path.
> I'll just let them go.
>
> Reviewed-by: Liam R. Howlett <Liam.Howlett@oracle.com>

Reviewed-by: Suren Baghdasaryan <surenb@google.com>

>
> > ---
> >  mm/vma.c | 14 --------------
> >  1 file changed, 14 deletions(-)
> >
> > diff --git a/mm/vma.c b/mm/vma.c
> > index 3f32e04bb6cc..3ff6cfbe3338 100644
> > --- a/mm/vma.c
> > +++ b/mm/vma.c
> > @@ -24,7 +24,6 @@ struct mmap_state {
> >       void *vm_private_data;
> >
> >       unsigned long charged;
> > -     bool retry_merge;
> >
> >       struct vm_area_struct *prev;
> >       struct vm_area_struct *next;
> > @@ -2417,8 +2416,6 @@ static int __mmap_new_file_vma(struct mmap_state *map,
> >                       !(map->flags & VM_MAYWRITE) &&
> >                       (vma->vm_flags & VM_MAYWRITE));
> >
> > -     /* If the flags change (and are mergeable), let's retry later. */
> > -     map->retry_merge = vma->vm_flags != map->flags && !(vma->vm_flags & VM_SPECIAL);

Could we have a WARN_ON() here please? I know you took care of in-tree
drivers but if an out-of-tree driver does this there will be no
indication that it has to change its ways. I know we don't care about
out-of-tree ones but they still exist, so maybe we can be nice here
and issue a warning?

> >       map->flags = vma->vm_flags;
> >
> >       return 0;
> > @@ -2622,17 +2619,6 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr,
> >       if (have_mmap_prepare)
> >               set_vma_user_defined_fields(vma, &map);
> >
> > -     /* If flags changed, we might be able to merge, so try again. */
> > -     if (map.retry_merge) {
> > -             struct vm_area_struct *merged;
> > -             VMG_MMAP_STATE(vmg, &map, vma);
> > -
> > -             vma_iter_config(map.vmi, map.addr, map.end);
> > -             merged = vma_merge_existing_range(&vmg);
> > -             if (merged)
> > -                     vma = merged;
> > -     }
> > -
> >       __mmap_complete(&map, vma);
> >
> >       return addr;
> > --
> > 2.49.0
> >

On 5/9/25 14:13, Lorenzo Stoakes wrote:
> We have now introduced a mechanism that obviates the need for a reattempted
> merge via the mmap_prepare() file hook, so eliminate this functionality
> altogether.
> 
> The retry merge logic has been the cause of a great deal of complexity in
> the past and required a great deal of careful manoeuvring of code to ensure
> its continued and correct functionality.
> 
> It has also recently been involved in an issue surrounding maple tree
> state, which again points to its problematic nature.
> 
> We make it much easier to reason about mmap() logic by eliminating this and
> simply writing a VMA once. This also opens the doors to future optimisation
> and improvement in the mmap() logic.
> 
> For any device or file system which encounters unwanted VMA fragmentation
> as a result of this change (that is, having not implemented .mmap_prepare
> hooks), the issue is easily resolvable by doing so.
> 
> Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
> Reviewed-by: David Hildenbrand <david@redhat.com>

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>