According to Documentation/mm/transhuge.rst, MADV_COLLAPSE is expected to
ignore any THP or mTHP interface settings.

However, after commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled()
into shmem_allowable_huge_orders()"), performing MADV_COLLAPSE on shmem
will depend on /sys/.../hugepages-2048kB/shmem_enabled being set to "inherit"
(although that is the default), which can cause MADV_COLLAPSE to fail
unexpectedly.

Fix this by returning the result of shmem_huge_global_enabled() directly
when MADV_COLLAPSE is requested, allowing PMD-order collapse while ignoring
shmem THP/mTHP settings.

Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
Reported-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Closes: https://lore.kernel.org/all/20260908125105.1510704-7-kirill@shutemov.name/
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
---
Hi Kiryl,

I did not update the selftests, as I hope this fix will be merged first.
After this fix has landed, I can send a patch to update the selftests.
---
 mm/shmem.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/shmem.c b/mm/shmem.c
index c92ed17dbc4a..b572c60f2af8 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2049,8 +2049,11 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,
 
 	global_orders = shmem_huge_global_enabled(inode, index, write_end,
 						  shmem_huge_force, vma, vm_flags);
-	/* Tmpfs huge pages allocation */
-	if (!vma || !vma_is_anon_shmem(vma))
+	/*
+	 * Tmpfs huge pages allocation or forced collapse ignores
+	 * sysfs configs.
+	 */
+	if (!vma || !vma_is_anon_shmem(vma) || shmem_huge_force)
 		return global_orders;
 
 	/*
-- 
2.47.3

On Fri, Sep 11, 2026 at 12:04:33PM +0800, Baolin Wang wrote:
>According to Documentation/mm/transhuge.rst, MADV_COLLAPSE is expected to
>ignore any THP or mTHP interface settings.
>
>However, after commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled()
>into shmem_allowable_huge_orders()"), performing MADV_COLLAPSE on shmem
>will depend on /sys/.../hugepages-2048kB/shmem_enabled being set to "inherit"
>(although that is the default), which can cause MADV_COLLAPSE to fail
>unexpectedly.
>
>Fix this by returning the result of shmem_huge_global_enabled() directly
>when MADV_COLLAPSE is requested, allowing PMD-order collapse while ignoring
>shmem THP/mTHP settings.
>
>Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
>Reported-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
>Closes: https://lore.kernel.org/all/20260908125105.1510704-7-kirill@shutemov.name/
>Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
>---

With the Fixes tag corrected as Kiryl suggested, LGTM!

Reviewed-by: Lance Yang <lance.yang@linux.dev>

On Fri, Sep 11, 2026 at 12:04:33PM +0800, Baolin Wang wrote:
> According to Documentation/mm/transhuge.rst, MADV_COLLAPSE is expected to
> ignore any THP or mTHP interface settings.
> 
> However, after commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled()
> into shmem_allowable_huge_orders()"), performing MADV_COLLAPSE on shmem
> will depend on /sys/.../hugepages-2048kB/shmem_enabled being set to "inherit"
> (although that is the default), which can cause MADV_COLLAPSE to fail
> unexpectedly.
> 
> Fix this by returning the result of shmem_huge_global_enabled() directly
> when MADV_COLLAPSE is requested, allowing PMD-order collapse while ignoring
> shmem THP/mTHP settings.
> 
> Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")

Hm. I don't think it is the right Fixes tag. "No functional changes" in
its commit message seems to be true.

This might be a better fit:

Fixes: 26c7d8413aaf ("mm: thp: support "THPeligible" semantics for mTHP with anonymous shmem")

> Reported-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
> Closes: https://lore.kernel.org/all/20260908125105.1510704-7-kirill@shutemov.name/
> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>

Thanks!

Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>


-- 
  Kiryl Shutsemau / Kirill A. Shutemov

On 9/11/26 7:37 PM, Kiryl Shutsemau wrote:
> On Fri, Sep 11, 2026 at 12:04:33PM +0800, Baolin Wang wrote:
>> According to Documentation/mm/transhuge.rst, MADV_COLLAPSE is expected to
>> ignore any THP or mTHP interface settings.
>>
>> However, after commit 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled()
>> into shmem_allowable_huge_orders()"), performing MADV_COLLAPSE on shmem
>> will depend on /sys/.../hugepages-2048kB/shmem_enabled being set to "inherit"
>> (although that is the default), which can cause MADV_COLLAPSE to fail
>> unexpectedly.
>>
>> Fix this by returning the result of shmem_huge_global_enabled() directly
>> when MADV_COLLAPSE is requested, allowing PMD-order collapse while ignoring
>> shmem THP/mTHP settings.
>>
>> Fixes: 6beeab870e70 ("mm: shmem: move shmem_huge_global_enabled() into shmem_allowable_huge_orders()")
> 
> Hm. I don't think it is the right Fixes tag. "No functional changes" in
> its commit message seems to be true.
> 
> This might be a better fit:
> 
> Fixes: 26c7d8413aaf ("mm: thp: support "THPeligible" semantics for mTHP with anonymous shmem")

Ah. You are right. Will update in v2.

>> Reported-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
>> Closes: https://lore.kernel.org/all/20260908125105.1510704-7-kirill@shutemov.name/
>> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
> 
> Thanks!
> 
> Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org>

Thanks.