[RFC PATCH v2 00/11] add shmem mTHP collapse support

Baolin Wang posted 11 patches 4 days, 4 hours ago
include/linux/shmem_fs.h                      |   4 +-
mm/khugepaged.c                               | 174 ++++++++++++----
mm/shmem.c                                    |  10 +-
.../selftests/mm/folio_split_race_test.c      |   2 +-
tools/testing/selftests/mm/khugepaged.c       | 195 +++++++++++++-----
.../testing/selftests/mm/prctl_thp_disable.c  |   2 +-
tools/testing/selftests/mm/run_vmtests.sh     |   4 +
tools/testing/selftests/mm/soft-dirty.c       |   2 +-
.../selftests/mm/split_huge_page_test.c       | 139 +------------
tools/testing/selftests/mm/uffd-common.c      |   4 +-
tools/testing/selftests/mm/vm_util.c          | 184 ++++++++++++++++-
tools/testing/selftests/mm/vm_util.h          |   8 +-
12 files changed, 492 insertions(+), 236 deletions(-)
[RFC PATCH v2 00/11] add shmem mTHP collapse support
Posted by Baolin Wang 4 days, 4 hours ago
(Note: this patchset is not targeting v7.2, but posted for early feedback.)

This is a follow-up patchset for mTHP collapse to support shmem mTHP collapse,
which is based on Nico's patchset[1].

The shmem mTHP collapse strategy follows the anonymous mTHP collapse approach:
track present pages via a bitmap while scanning PMD ranges for collapse candidates,
then use the bitmap after the scan completes to determine the most efficient
mTHP order to collapse to. Built on the basic framework added for anonymous
mTHP collapse, the shmem mTHP collapse implementation is straightforward
(Thanks for Nico's work).

In addition, I have added some anon/shmem mTHP collapse selftests, and now all
khugepaged test cases can pass.

Note: I have not yet enabled large order collapse for file folios (file folios
currently only support PMD-sized large folio collapse). Although file large order
collapse would be more straightforward to implement after shmem mTHP collapse
support is added (requiring some changes to file_thp_enabled()), I think this
still need some discussion on whether it is necessary to support other large
orders collapse for file folios.

Comments are welcome. Thanks.

Changes from RFC v1:
https://lore.kernel.org/all/cover.1755677674.git.baolin.wang@linux.alibaba.com/
 - Rebase on the new code, and update to use the new functions.
 - Add more test cases.

[1] https://lore.kernel.org/all/20260605161422.213817-1-npache@redhat.com/

Baolin Wang (11):
  mm: khugepaged: add max_ptes_none check in collapse_file()
  mm: khugepaged: generalize collapse_file() for shmem mTHP support
  mm: khugepaged: add an order check for PMD-sized THP statistics
  mm: khugepaged: add shmem mTHP collapse support
  mm: shmem: run khugepaged for all shmem mTHP orders
  mm: khugepaged: allow khugepaged to check all shmem mTHP-sized orders
  mm: khugepaged: skip large folios that don't need to be collapsed
  selftests: mm: extend the check_huge() to support mTHP check
  selftests: mm: move gather_after_split_folio_orders() into vm_util.c
    file
  selftests: mm: implement the mTHP-sized hugepage check helpers
  selftests: mm: add mTHP collapse test cases

 include/linux/shmem_fs.h                      |   4 +-
 mm/khugepaged.c                               | 174 ++++++++++++----
 mm/shmem.c                                    |  10 +-
 .../selftests/mm/folio_split_race_test.c      |   2 +-
 tools/testing/selftests/mm/khugepaged.c       | 195 +++++++++++++-----
 .../testing/selftests/mm/prctl_thp_disable.c  |   2 +-
 tools/testing/selftests/mm/run_vmtests.sh     |   4 +
 tools/testing/selftests/mm/soft-dirty.c       |   2 +-
 .../selftests/mm/split_huge_page_test.c       | 139 +------------
 tools/testing/selftests/mm/uffd-common.c      |   4 +-
 tools/testing/selftests/mm/vm_util.c          | 184 ++++++++++++++++-
 tools/testing/selftests/mm/vm_util.h          |   8 +-
 12 files changed, 492 insertions(+), 236 deletions(-)

-- 
2.47.3
Re: [RFC PATCH v2 00/11] add shmem mTHP collapse support
Posted by Nico Pache 3 days, 22 hours ago
On Wed, Jun 10, 2026 at 4:29 AM Baolin Wang
<baolin.wang@linux.alibaba.com> wrote:
>
> (Note: this patchset is not targeting v7.2, but posted for early feedback.)
>
> This is a follow-up patchset for mTHP collapse to support shmem mTHP collapse,
> which is based on Nico's patchset[1].
>
> The shmem mTHP collapse strategy follows the anonymous mTHP collapse approach:
> track present pages via a bitmap while scanning PMD ranges for collapse candidates,
> then use the bitmap after the scan completes to determine the most efficient
> mTHP order to collapse to. Built on the basic framework added for anonymous
> mTHP collapse, the shmem mTHP collapse implementation is straightforward
> (Thanks for Nico's work).

As promised I will review this series :)

>
> In addition, I have added some anon/shmem mTHP collapse selftests, and now all
> khugepaged test cases can pass.

Thank you for doing that!!

I was just working on adding the anon mTHP selftests. Any chance we
can separate out those bits and just send a series for adding anon
mTHP selftests without the file-related changes? Then your series adds
the bits you need here for shmem collapse? Thank you for doing all the
heavy lifting there to generalize those functions :)

Cheers,
-- Nico

>
> Note: I have not yet enabled large order collapse for file folios (file folios
> currently only support PMD-sized large folio collapse). Although file large order
> collapse would be more straightforward to implement after shmem mTHP collapse
> support is added (requiring some changes to file_thp_enabled()), I think this
> still need some discussion on whether it is necessary to support other large
> orders collapse for file folios.
>
> Comments are welcome. Thanks.
>
> Changes from RFC v1:
> https://lore.kernel.org/all/cover.1755677674.git.baolin.wang@linux.alibaba.com/
>  - Rebase on the new code, and update to use the new functions.
>  - Add more test cases.
>
> [1] https://lore.kernel.org/all/20260605161422.213817-1-npache@redhat.com/
>
> Baolin Wang (11):
>   mm: khugepaged: add max_ptes_none check in collapse_file()
>   mm: khugepaged: generalize collapse_file() for shmem mTHP support
>   mm: khugepaged: add an order check for PMD-sized THP statistics
>   mm: khugepaged: add shmem mTHP collapse support
>   mm: shmem: run khugepaged for all shmem mTHP orders
>   mm: khugepaged: allow khugepaged to check all shmem mTHP-sized orders
>   mm: khugepaged: skip large folios that don't need to be collapsed
>   selftests: mm: extend the check_huge() to support mTHP check
>   selftests: mm: move gather_after_split_folio_orders() into vm_util.c
>     file
>   selftests: mm: implement the mTHP-sized hugepage check helpers
>   selftests: mm: add mTHP collapse test cases
>
>  include/linux/shmem_fs.h                      |   4 +-
>  mm/khugepaged.c                               | 174 ++++++++++++----
>  mm/shmem.c                                    |  10 +-
>  .../selftests/mm/folio_split_race_test.c      |   2 +-
>  tools/testing/selftests/mm/khugepaged.c       | 195 +++++++++++++-----
>  .../testing/selftests/mm/prctl_thp_disable.c  |   2 +-
>  tools/testing/selftests/mm/run_vmtests.sh     |   4 +
>  tools/testing/selftests/mm/soft-dirty.c       |   2 +-
>  .../selftests/mm/split_huge_page_test.c       | 139 +------------
>  tools/testing/selftests/mm/uffd-common.c      |   4 +-
>  tools/testing/selftests/mm/vm_util.c          | 184 ++++++++++++++++-
>  tools/testing/selftests/mm/vm_util.h          |   8 +-
>  12 files changed, 492 insertions(+), 236 deletions(-)
>
> --
> 2.47.3
>
Re: [RFC PATCH v2 00/11] add shmem mTHP collapse support
Posted by Baolin Wang 3 days, 13 hours ago

On 6/11/26 12:28 AM, Nico Pache wrote:
> On Wed, Jun 10, 2026 at 4:29 AM Baolin Wang
> <baolin.wang@linux.alibaba.com> wrote:
>>
>> (Note: this patchset is not targeting v7.2, but posted for early feedback.)
>>
>> This is a follow-up patchset for mTHP collapse to support shmem mTHP collapse,
>> which is based on Nico's patchset[1].
>>
>> The shmem mTHP collapse strategy follows the anonymous mTHP collapse approach:
>> track present pages via a bitmap while scanning PMD ranges for collapse candidates,
>> then use the bitmap after the scan completes to determine the most efficient
>> mTHP order to collapse to. Built on the basic framework added for anonymous
>> mTHP collapse, the shmem mTHP collapse implementation is straightforward
>> (Thanks for Nico's work).
> 
> As promised I will review this series :)

Great. Thanks.

>> In addition, I have added some anon/shmem mTHP collapse selftests, and now all
>> khugepaged test cases can pass.
> 
> Thank you for doing that!!
> 
> I was just working on adding the anon mTHP selftests. Any chance we
> can separate out those bits and just send a series for adding anon
> mTHP selftests without the file-related changes? Then your series adds
> the bits you need here for shmem collapse? Thank you for doing all the
> heavy lifting there to generalize those functions :)
It's easy to separate out the anon mTHP selftests, but I'd like to get 
some feedback first. Once everyone feels the selftests are in reasonable 
shape, I'd appreciate help with review and testing (and even adding more 
test cases on top of them).
Re: [RFC PATCH v2 00/11] add shmem mTHP collapse support
Posted by Lorenzo Stoakes 3 days, 8 hours ago
On Wed, Jun 10, 2026 at 06:29:08PM +0800, Baolin Wang wrote:
> (Note: this patchset is not targeting v7.2, but posted for early feedback.)

Hi Baolin,

Speaking broadly, and to repeat what I said to Nico - I'd like to
re-emphasise my _strong_ desire for us to pay down technical debt in THP
code before adding any further major features.

Really I'd have liked to see us do this _before_ khugepaged mTHP, but I
didn't want to put Nico through more than he already endured with that
series :)

So I'd really like to see some action on that before we embark on any
further significant changes.

This is no comment on your changes, just something that we really need to
address in THP generally.

Will try to give some detailed commentary on the actual changes when I have
time also :)

Cheers, Lorenzo

>
> This is a follow-up patchset for mTHP collapse to support shmem mTHP collapse,
> which is based on Nico's patchset[1].
>
> The shmem mTHP collapse strategy follows the anonymous mTHP collapse approach:
> track present pages via a bitmap while scanning PMD ranges for collapse candidates,
> then use the bitmap after the scan completes to determine the most efficient
> mTHP order to collapse to. Built on the basic framework added for anonymous
> mTHP collapse, the shmem mTHP collapse implementation is straightforward
> (Thanks for Nico's work).
>
> In addition, I have added some anon/shmem mTHP collapse selftests, and now all
> khugepaged test cases can pass.
>
> Note: I have not yet enabled large order collapse for file folios (file folios
> currently only support PMD-sized large folio collapse). Although file large order
> collapse would be more straightforward to implement after shmem mTHP collapse
> support is added (requiring some changes to file_thp_enabled()), I think this
> still need some discussion on whether it is necessary to support other large
> orders collapse for file folios.
>
> Comments are welcome. Thanks.
>
> Changes from RFC v1:
> https://lore.kernel.org/all/cover.1755677674.git.baolin.wang@linux.alibaba.com/
>  - Rebase on the new code, and update to use the new functions.
>  - Add more test cases.
>
> [1] https://lore.kernel.org/all/20260605161422.213817-1-npache@redhat.com/
>
> Baolin Wang (11):
>   mm: khugepaged: add max_ptes_none check in collapse_file()
>   mm: khugepaged: generalize collapse_file() for shmem mTHP support
>   mm: khugepaged: add an order check for PMD-sized THP statistics
>   mm: khugepaged: add shmem mTHP collapse support
>   mm: shmem: run khugepaged for all shmem mTHP orders
>   mm: khugepaged: allow khugepaged to check all shmem mTHP-sized orders
>   mm: khugepaged: skip large folios that don't need to be collapsed
>   selftests: mm: extend the check_huge() to support mTHP check
>   selftests: mm: move gather_after_split_folio_orders() into vm_util.c
>     file
>   selftests: mm: implement the mTHP-sized hugepage check helpers
>   selftests: mm: add mTHP collapse test cases
>
>  include/linux/shmem_fs.h                      |   4 +-
>  mm/khugepaged.c                               | 174 ++++++++++++----
>  mm/shmem.c                                    |  10 +-
>  .../selftests/mm/folio_split_race_test.c      |   2 +-
>  tools/testing/selftests/mm/khugepaged.c       | 195 +++++++++++++-----
>  .../testing/selftests/mm/prctl_thp_disable.c  |   2 +-
>  tools/testing/selftests/mm/run_vmtests.sh     |   4 +
>  tools/testing/selftests/mm/soft-dirty.c       |   2 +-
>  .../selftests/mm/split_huge_page_test.c       | 139 +------------
>  tools/testing/selftests/mm/uffd-common.c      |   4 +-
>  tools/testing/selftests/mm/vm_util.c          | 184 ++++++++++++++++-
>  tools/testing/selftests/mm/vm_util.h          |   8 +-
>  12 files changed, 492 insertions(+), 236 deletions(-)
>
> --
> 2.47.3
>
Re: [RFC PATCH v2 00/11] add shmem mTHP collapse support
Posted by Baolin Wang 3 days, 2 hours ago
Hi Lorenzo,

On 6/11/26 2:18 PM, Lorenzo Stoakes wrote:
> On Wed, Jun 10, 2026 at 06:29:08PM +0800, Baolin Wang wrote:
>> (Note: this patchset is not targeting v7.2, but posted for early feedback.)
> 
> Hi Baolin,
> 
> Speaking broadly, and to repeat what I said to Nico - I'd like to
> re-emphasise my _strong_ desire for us to pay down technical debt in THP
> code before adding any further major features.
> 
> Really I'd have liked to see us do this _before_ khugepaged mTHP, but I
> didn't want to put Nico through more than he already endured with that
> series :)
> 
> So I'd really like to see some action on that before we embark on any
> further significant changes.
> 
> This is no comment on your changes, just something that we really need to
> address in THP generally.
> 
> Will try to give some detailed commentary on the actual changes when I have
> time also :)

I fully understand the current state of THP maintenance, which is why 
this is an RFC. I’m not expecting it to be merged into mainline quickly 
(but thanks to Lance for the comments).

Also, as Nico noted, it might be worth reviewing the mTHP collapse 
selftests in this series firstly? as they can help validate the 
anonymous mTHP collapse path for future changes.
Re: [RFC PATCH v2 00/11] add shmem mTHP collapse support
Posted by Lorenzo Stoakes 3 days, 2 hours ago
On Thu, Jun 11, 2026 at 07:46:55PM +0800, Baolin Wang wrote:
> Hi Lorenzo,
>
> On 6/11/26 2:18 PM, Lorenzo Stoakes wrote:
> > On Wed, Jun 10, 2026 at 06:29:08PM +0800, Baolin Wang wrote:
> > > (Note: this patchset is not targeting v7.2, but posted for early feedback.)
> >
> > Hi Baolin,
> >
> > Speaking broadly, and to repeat what I said to Nico - I'd like to
> > re-emphasise my _strong_ desire for us to pay down technical debt in THP
> > code before adding any further major features.
> >
> > Really I'd have liked to see us do this _before_ khugepaged mTHP, but I
> > didn't want to put Nico through more than he already endured with that
> > series :)
> >
> > So I'd really like to see some action on that before we embark on any
> > further significant changes.
> >
> > This is no comment on your changes, just something that we really need to
> > address in THP generally.
> >
> > Will try to give some detailed commentary on the actual changes when I have
> > time also :)
>
> I fully understand the current state of THP maintenance, which is why this
> is an RFC. I’m not expecting it to be merged into mainline quickly (but
> thanks to Lance for the comments).

Ack, well maybe you'd be surprised if I didn't mention it ;)

And I will try to have a look through on basis of the technical changes!

>
> Also, as Nico noted, it might be worth reviewing the mTHP collapse selftests
> in this series firstly? as they can help validate the anonymous mTHP
> collapse path for future changes.

Yeah that sounds like a plan and thank you for writing them :) more testing is
welcome and very much appreciated.

Cheers, Lorenzo