When KSM-by-default is established using prctl(PR_SET_MEMORY_MERGE), this
defaults all newly mapped VMAs to having VM_MERGEABLE set, and thus makes
them available to KSM for samepage merging. It also sets VM_MERGEABLE in
all existing VMAs.

However this causes an issue upon mapping of new VMAs - the initial flags
will never have VM_MERGEABLE set when attempting a merge with adjacent VMAs
(this is set later in the mmap() logic), and adjacent VMAs will ALWAYS have
VM_MERGEABLE set.

This renders literally all VMAs in the virtual address space unmergeable.

To avoid this, this series performs the check for PR_SET_MEMORY_MERGE far
earlier in the mmap() logic, prior to the merge being attempted.

However we run into a complexity with the depreciated .mmap() callback - if
a driver hooks this, it might change flags thus adjusting KSM merge
eligibility.

This isn't a problem for brk(), where the VMA must be anonymous. However
for mmap() we are conservative - if the VMA is anonymous then we can always
proceed, however if not, we permit only shmem mappings and drivers which
implement .mmap_prepare().

If we can't be sure of the driver changing things, then we maintain the
same behaviour of performing the KSM check later in the mmap() logic (and
thus losing VMA mergeability).

Since the .mmap_prepare() hook is invoked prior to the KSM check, this
means we can always perform the KSM check early if it is present. Over time
as drivers are converted, we can do away with the later check altogether.

A great many use-cases for this logic will use anonymous or shmem memory at
any rate, as KSM is not supported for the page cache, and the driver
outliers in question are MAP_PRIVATE mappings of these files.

So this change should already cover the majority of actual KSM use-cases.

v2:
* Removed unnecessary ret local variable in ksm_vma_flags() as per David.
* added Stefan Roesch in cc and added Fixes tag as per Andrew, David.
* Propagated tags (thanks everyone!)
* Removed unnecessary !CONFIG_KSM ksm_add_vma() stub from
  include/linux/ksm.h.
* Added missing !CONFIG_KSM ksm_vma_flags() stub in
  include/linux/ksm.h.
* After discussion with David, I've decided to defer removing the
  VM_SPECIAL case for KSM, we can address this in a follow-up series.
* Expanded 3/4 commit message to reference KSM eligibility vs. merging and
  referenced future plans to permit KSM for VM_SPECIAL VMAs.

v1:
https://lore.kernel.org/all/cover.1747431920.git.lorenzo.stoakes@oracle.com/

Lorenzo Stoakes (4):
  mm: ksm: have KSM VMA checks not require a VMA pointer
  mm: ksm: refer to special VMAs via VM_SPECIAL in ksm_compatible()
  mm: prevent KSM from completely breaking VMA merging
  tools/testing/selftests: add VMA merge tests for KSM merge

 include/linux/fs.h                 |  7 ++-
 include/linux/ksm.h                |  8 +--
 mm/ksm.c                           | 49 ++++++++++++-------
 mm/vma.c                           | 49 ++++++++++++++++++-
 tools/testing/selftests/mm/merge.c | 78 ++++++++++++++++++++++++++++++
 5 files changed, 167 insertions(+), 24 deletions(-)

--
2.49.0

And in the annals of 'hit send then immediately notice mistake' we have
entry number 1,379 :P

This subject should read [PATCH v2 0/4] obviously, like the patches that
reply to it in-thread.

Apologies for that!

Other than that, series should be fine :>)

Cheers, Lorenzo

On Wed, May 21, 2025 at 07:20:27PM +0100, Lorenzo Stoakes wrote:
> When KSM-by-default is established using prctl(PR_SET_MEMORY_MERGE), this
> defaults all newly mapped VMAs to having VM_MERGEABLE set, and thus makes
> them available to KSM for samepage merging. It also sets VM_MERGEABLE in
> all existing VMAs.
>
> However this causes an issue upon mapping of new VMAs - the initial flags
> will never have VM_MERGEABLE set when attempting a merge with adjacent VMAs
> (this is set later in the mmap() logic), and adjacent VMAs will ALWAYS have
> VM_MERGEABLE set.
>
> This renders literally all VMAs in the virtual address space unmergeable.
>
> To avoid this, this series performs the check for PR_SET_MEMORY_MERGE far
> earlier in the mmap() logic, prior to the merge being attempted.
>
> However we run into a complexity with the depreciated .mmap() callback - if
> a driver hooks this, it might change flags thus adjusting KSM merge
> eligibility.
>
> This isn't a problem for brk(), where the VMA must be anonymous. However
> for mmap() we are conservative - if the VMA is anonymous then we can always
> proceed, however if not, we permit only shmem mappings and drivers which
> implement .mmap_prepare().
>
> If we can't be sure of the driver changing things, then we maintain the
> same behaviour of performing the KSM check later in the mmap() logic (and
> thus losing VMA mergeability).
>
> Since the .mmap_prepare() hook is invoked prior to the KSM check, this
> means we can always perform the KSM check early if it is present. Over time
> as drivers are converted, we can do away with the later check altogether.
>
> A great many use-cases for this logic will use anonymous or shmem memory at
> any rate, as KSM is not supported for the page cache, and the driver
> outliers in question are MAP_PRIVATE mappings of these files.
>
> So this change should already cover the majority of actual KSM use-cases.
>
> v2:
> * Removed unnecessary ret local variable in ksm_vma_flags() as per David.
> * added Stefan Roesch in cc and added Fixes tag as per Andrew, David.
> * Propagated tags (thanks everyone!)
> * Removed unnecessary !CONFIG_KSM ksm_add_vma() stub from
>   include/linux/ksm.h.
> * Added missing !CONFIG_KSM ksm_vma_flags() stub in
>   include/linux/ksm.h.
> * After discussion with David, I've decided to defer removing the
>   VM_SPECIAL case for KSM, we can address this in a follow-up series.
> * Expanded 3/4 commit message to reference KSM eligibility vs. merging and
>   referenced future plans to permit KSM for VM_SPECIAL VMAs.
>
> v1:
> https://lore.kernel.org/all/cover.1747431920.git.lorenzo.stoakes@oracle.com/
>
> Lorenzo Stoakes (4):
>   mm: ksm: have KSM VMA checks not require a VMA pointer
>   mm: ksm: refer to special VMAs via VM_SPECIAL in ksm_compatible()
>   mm: prevent KSM from completely breaking VMA merging
>   tools/testing/selftests: add VMA merge tests for KSM merge
>
>  include/linux/fs.h                 |  7 ++-
>  include/linux/ksm.h                |  8 +--
>  mm/ksm.c                           | 49 ++++++++++++-------
>  mm/vma.c                           | 49 ++++++++++++++++++-
>  tools/testing/selftests/mm/merge.c | 78 ++++++++++++++++++++++++++++++
>  5 files changed, 167 insertions(+), 24 deletions(-)
>
> --
> 2.49.0
>