This series fixes a few bugs on HugeTLB folio allocation failure paths.
Some of these issues were pointed out by Sashiko while I was working on
[1], and others were revealed while fixing the original issues.
1. subpool usage leak on allocation failure (Sashiko pointed this out
in [2])
When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
hugetlb cgroup charging failure) and gbl_chg == 1, the error path skips
restoring the page to the subpool, leaking the subpool's used_hpages
counter. Fix this by calling hugepage_subpool_put_pages()
unconditionally if map_chg is true.
2. Folio refcount mismatch on memcg charge failure leading to
VM_BUG_ON_FOLIO() (Sashiko pointed this out in [3])
The error path in alloc_hugetlb_folio() calls free_huge_folio() directly
on a folio with a refcount of 1 (set via folio_ref_unfreeze() earlier).
This triggers VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) if
CONFIG_DEBUG_VM is enabled, and can corrupt allocator state otherwise.
Fix this by using folio_put() instead of free_huge_folio() to correctly
drop the refcount before freeing.
3. Returning -ENOMEM on memcg charge failure causes infinite loop
alloc_hugetlb_folio() propagates -ENOMEM on charge failure, which maps
to VM_FAULT_OOM. Because HugeTLB physical allocations are high-order and
use __GFP_RETRY_MAYFAIL, the OOM killer is bypassed. Returning
VM_FAULT_OOM leaks to the #PF handler, which cannot make progress and
retries the faulting instruction indefinitely. Fix this by returning
-ENOSPC instead of -ENOMEM on charge failure, which maps to
VM_FAULT_SIGBUS, terminating the process cleanly.
4. vma reservation leak on mem_cgroup_charge_hugetlb() failure
When mem_cgroup_charge_hugetlb() fails, the error path historically
bypassed vma_end_reservation(). Since the reservation had already been
committed via vma_commit_reservation(), this left the reservation map in
an inconsistent state, leaking resv_huge_pages when the process
exited. Fix this by moving mem_cgroup_charge_hugetlb() earlier in
alloc_hugetlb_folio(), before vma_commit_reservation() is called.
Cleaned up patches, updated documentation, and fixed Sashiko's comments on
v3:
1. subpool cleanup in alloc_hugetlb_folio() was incomplete - I folded the
fix in.
2. More issues (UAF, and another accounting issue) reported [5] - I added
two more patches.
Keeping this section since I think people didn't have much time to review
v3.
1. subpool page tracking
The subpool now tracks every page allocated via
the subpool through used_hpages, regardless of whether min_hpages or
max_hpages is requested by the user.
When min_hpages is set, rsv_hpages is initialized to be equal to
min_hpages. As reserved pages are used, the count is transferred from
rsv_hpages to used_hpages. When reserved pages run out, as long as
max_hpages is not requested or not yet met, used_hpages continues to be
incremented. The following invariant holds true while used_hpages <=
min_hpages:
used_hpages + rsv_hpages = min_hpages
when used_hpages exceeds min_hpages, rsv_hpages must always be 0.
With this, pages can and must always be returned to the subpool if there's
some reservation or allocation failure after getting a page from the
subpool. One thing I missed in v2 was in processing the returned page. In
v2, the number of pages returned was compared to min_hpages, ignoring
used_hpages if it exists. v3 limits the number of pages returned to
min_hpages - used_hpages, upholding the invariant above.
2. (Re-)introduction of the try-commit-cancel protocol for memcg charging.
In v2, I took the shortcut where I moved mem_cgroup_charge_hugetlb() up
above committing vma reservations, and jumped to subpool cleanup when the
charging failed. With that, Sashiko pointed out that there are still issues
with subpool tracking.
In [4] I thought the issue was that gbl_chg == 0 (I flipped the condition
in [4], omg) doesn't always mean a reservation was actually used, but after
digging more, that doesn't seem to be true, since if gbl_chg == 0,
restore_reserve is set on the folio, and free_huge_folio() would have done
a h->resv_huge_pages++.
In my opinion, calling free_huge_folio() in the middle of the cleanup does
too much, and I think the cleaner approach would be to do all the charging
symmetrically, so that the subpool cleanup will also be symmetric
and (hopefully) easier to understand.
+ I would like reviews on all the non-WIP patches. Those are the ones that
I'd like merged first.
+ There are reproducers (not meant to be merged) to help prove
effectiveness of fixes.
+ There are WIP patches for subpool-related refactoring. Those helped me
check that the subpool stuff is working and is posted to show how it
works. I think the subpool-related refactoring and tests would be good to
merge, but perhaps in another patch series? The refactoring was heavily
AI-driven and should be considered RFC. Would like to have your feedback
on that refactoring too.
+ libhugetlbfs tests pass
+ ./tools/testing/selftests/mm/ksft_hugetlb.sh passes
+ Reproducers all pass
v1: https://lore.kernel.org/r/20260707-hugetlb-alloc-failure-fixes-v1-0-5bbd3a4b836d@google.com
v2: https://lore.kernel.org/r/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b@google.com
v3: https://lore.kernel.org/r/20260720-hugetlb-alloc-failure-fixes-v3-0-7d2a169aa9ee@google.com
The series [1] has some changes that are dependent on these fixes, so I'll wait
for your reviews before continuing on [1].
Thank you!
[1] https://lore.kernel.org/all/20260702-hugetlb-open-up-v4-0-d53cefcccf34@google.com/T/
[2] https://sashiko.dev/#/patchset/20260518-hugetlb-open-up-v3-0-e14b302477f8%40google.com?part=5
[3] https://sashiko.dev/#/patchset/20260702-hugetlb-open-up-v4-0-d53cefcccf34%40google.com?part=6
[4] https://lore.kernel.org/all/CAEvNRgGN0HSJ2iLSDD2haSKOxifa-uhkO9Hwossh0+Q_d9fzOw@mail.gmail.com/
[5] https://sashiko.dev/#/patchset/20260720-hugetlb-alloc-failure-fixes-v3-0-7d2a169aa9ee%40google.com?part=8
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
Ackerley Tng (16):
mm: hugetlb: Track used_hpages when getting/putting pages from subpool
mm: hugetlb: Return -ENOSPC on memcg charge failure
mm: hugetlb: Use try-commit-cancel protocol for memcg charge of folios
mm: hugetlb: Remove unused mem_cgroup_charge_hugetlb function
mm: hugetlb: Fix subpool usage leak on allocation failure
mm: hugetlb: Rename local variables for clarity in hugetlb_reserve_pages()
mm: hugetlb: Fix Use-After-Free in unlock_or_release_subpool()
fs: hugetlbfs: Fix global reservation leak in hugetlbfs_fill_super()
WIP: mm: hugetlb: Move subpool functions to hugetlb_subpool.c
WIP: fs: hugetlbfs: Refactor subpool getters and integrate with hugetlb_subpool API
WIP: mm: hugetlb: Make struct hugepage_subpool private to hugetlb_subpool.c
WIP: tools: testing: Add unit tests for HugeTLB subpool functions
WIP: Reproducer for allocation failure due to cgroup v2 memory limits
WIP: Reproducer for subpool usage leak
WIP: Reproducer for false restoration on shared HugeTLB mappings
WIP: Reproducer for out_put_pages subpool reserve leakage
Documentation/mm/hugetlbfs_reserv.rst | 17 +-
.../translations/zh_CN/mm/hugetlbfs_reserv.rst | 11 +-
cgroup_v2_allocation_failure.c | 169 +++++++++
fs/hugetlbfs/inode.c | 31 +-
hugetlb_reserve_pages_out_put_pages.c | 49 +++
hugetlb_reserve_pages_out_put_pages.sh | 153 ++++++++
include/linux/hugetlb.h | 17 +-
include/linux/memcontrol.h | 30 +-
mm/Makefile | 2 +-
mm/hugetlb.c | 260 +++-----------
mm/hugetlb_subpool.c | 230 ++++++++++++
mm/hugetlb_subpool.h | 21 ++
mm/memcontrol.c | 116 +++++-
subpool_leak_max_size.sh | 72 ++++
subpool_shared_leak.c | 43 +++
subpool_shared_leak.sh | 87 +++++
tools/testing/hugetlb_subpool/.gitignore | 1 +
tools/testing/hugetlb_subpool/Makefile | 18 +
tools/testing/hugetlb_subpool/test_subpool.c | 400 +++++++++++++++++++++
19 files changed, 1445 insertions(+), 282 deletions(-)
---
base-commit: 4539944e515183668109bdf4d0c3d7d228383d88
change-id: 20260706-hugetlb-alloc-failure-fixes-c7f775eca29f
Best regards,
--
Ackerley Tng <ackerleytng@google.com>