Documentation/mm/hugetlbfs_reserv.rst | 17 +-- .../translations/zh_CN/mm/hugetlbfs_reserv.rst | 11 +- fs/hugetlbfs/inode.c | 8 +- include/linux/hugetlb.h | 4 +- mm/hugetlb.c | 142 +++++++++++---------- 5 files changed, 85 insertions(+), 97 deletions(-)
HugeTLB subpools currently only track used pages (used_hpages) when a
maximum size limit (max_hpages) is configured on the mount.
This breaks minimum size (min_hpages) guarantees. The subpool guarantee
requires that the sum of used pages and remaining subpool reserves
satisfies the configured minimum:
used_hpages + rsv_hpages >= min_hpages
where used_hpages includes both allocated folios and active reservations.
Therefore, when pages or reservations are released, the maximum number of
reservations the subpool can absorb is:
limit = max(0, min_hpages - used_hpages)
When a mount specifies a minimum size guarantee without a maximum limit,
used_hpages is untracked and remains zero. The subpool assumes limit ==
min_hpages, falsely restoring reservations to the subpool even when
existing usage already satisfies the guarantee.
This series addresses three problem areas resulting from untracked usage
and asymmetric error handling:
1. False restoration and premature subpool freeing (Patch 1):
+ Global reservations (resv_huge_pages) are not decremented when
reservations are released, depleting available huge pages host-wide.
+ On unmount, subpool_is_free() checks (rsv_hpages == min_hpages).
Because of the false reservation restoration, this evaluates to true,
freeing the subpool while folios are still active in the page cache
and causing a use-after-free when they are later released.
(See Patch 1 for a detailed step-by-step trace.)
2. Reservation rollback races and underflow (Patches 2 & 4):
+ On hugetlb_acct_memory() failure in hugetlb_reserve_pages(), the
error path manually calculates rollbacks using local variables.
This is race-prone when concurrent threads interact with the
subpool, risking reservation leaks or counter underflows.
+ On region_add() failure, prematurely releasing global reservations
forces error cleanup to attempt fresh allocations that can fail
under memory pressure.
3. Subpool usage leak on allocation failure (Patch 3):
+ In alloc_hugetlb_folio(), when folio allocation fails and
gbl_chg == 1, the cleanup path skips hugepage_subpool_put_pages(),
permanently leaking used_hpages.
Always tracking used_hpages establishes a single source of truth within the
subpool. Allocation and reservation error paths can now handle failures
symmetrically: return the full page count to the subpool via
hugepage_subpool_put_pages(), and rely on its return value to reconcile
global reservations with hugetlb_acct_memory().
This series is a subset of patches from [1] and replaces [1].
[1] https://lore.kernel.org/all/20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com/
Tested:
+ Reproducers (see below) pass
+ tools/testing/selftests/mm/ksft_hugetlb.sh passes
+ libhugetlbfs tests pass
Changes in v3:
+ Rebased on 7.3-rc3.
+ Picked up Reviewed-bys.
+ Rename resv_get_accted to resv_get_accounted as Joshua requested.
+ Now Cc-ing stable on patch 4.
+ Moved trace of a false reservation restoration from cover letter into
patch 1.
v2: https://lore.kernel.org/r/20260909-hugetlb-subpool-always-track-used-v2-0-30c5d83b572a@google.com
v1: https://lore.kernel.org/r/20260902-hugetlb-subpool-always-track-used-v1-0-de1cd14bd713@google.com
Get reproducers from:
https://github.com/googleprodkernel/linux-cc/commits/hugetlb-subpool-always-track-used-with-reproducers-v3
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
Ackerley Tng (4):
mm: hugetlb: Track used_hpages when getting/putting pages from subpool
mm: hugetlb: Fix out_put_pages subpool reserve calculation
mm: hugetlb: Fix subpool usage leak on allocation failure
mm: hugetlb: Avoid re-allocating global reservations on region add failure
Documentation/mm/hugetlbfs_reserv.rst | 17 +--
.../translations/zh_CN/mm/hugetlbfs_reserv.rst | 11 +-
fs/hugetlbfs/inode.c | 8 +-
include/linux/hugetlb.h | 4 +-
mm/hugetlb.c | 142 +++++++++++----------
5 files changed, 85 insertions(+), 97 deletions(-)
---
base-commit: fd73f4a6659897191fa0d40695fe370925dd3780
change-id: 20260902-hugetlb-subpool-always-track-used-2624840f4c08
Best regards,
--
Ackerley Tng <ackerleytng@google.com>
On Wed, 16 Sep 2026 16:39:00 -0700 Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@kernel.org> wrote:
> HugeTLB subpools currently only track used pages (used_hpages) when a
> maximum size limit (max_hpages) is configured on the mount.
>
> ...
>
> This series addresses three problem areas resulting from untracked usage
> and asymmetric error handling:
These are rather serious-sounding problems.
I'm seeing Fixes: for 10-year-old patches, for one last year and one
this year. The fourth patch doesn't identify what to fix.
So if downstream people (-stable maintainers, others) follow our
recommendations, some kernels will get two of these patches, other
kernel versions will get three and some lucky kernels might get all
four. Are you confident that the patches can be split apart in this
fashion and still produce a good result? After all, we're sending
downstream some patch combinations which we didn't test.
Also, can you speculate about why these things took so long to
discover?
Ho hum, complicated. Let's at least slap a Fixes: on [4/4]. My
trillion-dollar sidekick suggests a9e443086489 ("hugetlb: disable
region_add file_region coalescing").
Our other little friend is talking to us:
https://sashiko.dev/#/patchset/20260916-hugetlb-subpool-always-track-used-v3-0-38aae9b5ccdd@google.com
I'd like to queue this up promptly because it fixes things. But let's
seek additional review input and argue with Sashiko before taking that
step.
On Wed, 16 Sep 2026 20:13:07 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:
> So if downstream people (-stable maintainers, others) follow our
> recommendations, some kernels will get two of these patches, other
> kernel versions will get three and some lucky kernels might get all
> four. Are you confident that the patches can be split apart in this
> fashion and still produce a good result?
Two things I noticed while testing this series on v7.3-rc3 (x86_64
QEMU, one CPU):
1. It overlaps with two fixes already queued in mm.git, so I've added
their authors to Cc:
- 3/4 rewrites the same out_subpool_put: block as Zhao Li's
"mm/hugetlb: fix max-only subpool accounting on
alloc_hugetlb_folio failure" (mm-hotfixes-unstable), and has the
same Fixes: tag.
- 2/4 rewrites the same out_put_pages: block as Jinmeng Zhou's
"mm/hugetlb: fix subpool minimum reservation rollback"
(mm-unstable).
3/4 doesn't apply to mm-hotfixes-unstable, and 2/4-4/4 don't apply
to mm-unstable. If I read them right, the queued fixes only handle
mounts with size=, while 2/4 and 3/4 also cover min_size mounts, so
they would probably replace them.
2. On splitting: 1/4 applies cleanly on top of both queued fixes, but
in my tests it made min_size-only mounts worse on its own. As far
as I can tell, that's because it starts tracking used_hpages on
those mounts, while the two error paths only release it after 2/4
and 3/4.
HugePages_Rsvd, expected value in parentheses. Q = the two queued
fixes, wrap = 18446744073709551615:
rc3 +Q +1/4 +Q+1/4 +1/4..4/4
min_size=4M only:
SIGBUS faults [1], no files (2) 2 2 0 0 2
min_size=8M only:
failed mmap [2], umounted (0) 0 0 3 3 0
size=8M,min_size=4M:
SIGBUS faults [1], no files (2) 0 0 0 0 2
size=10M,min_size=8M:
failed mmap [2], umounted (0) wrap 0 wrap 0 0
With 1/4 alone, the min_size-only mount loses its reservation, and
in the failed-mmap case three huge pages stay reserved after umount,
presumably because the subpool is never freed.
So it looks to me like 1/4 shouldn't go anywhere without 2/4 and 3/4.
I haven't tested older stable trees. With all four patches applied,
all the cases above give the expected values, and so does the
partial-truncate case from the 1/4 changelog (Rsvd drops to 0 after
the truncate instead of staying at 1).
I'm happy to rerun these tests on a rebased v4.
[1] https://lore.kernel.org/r/20260923065714.20781-1-kmehltretter@gmail.com/
[2] the scenario from Jinmeng's changelog:
https://lore.kernel.org/20260907132055.26696-1-zhoujinmeng@bytedance.com
Thanks,
Karl
Karl Mehltretter <kmehltretter@gmail.com> writes: > On Wed, 16 Sep 2026 20:13:07 -0700 Andrew Morton <akpm@linux-foundation.org> wrote: >> So if downstream people (-stable maintainers, others) follow our >> recommendations, some kernels will get two of these patches, other >> kernel versions will get three and some lucky kernels might get all >> four. Are you confident that the patches can be split apart in this >> fashion and still produce a good result? > > Two things I noticed while testing this series on v7.3-rc3 (x86_64 > QEMU, one CPU): > Thank you so much for testing! > 1. It overlaps with two fixes already queued in mm.git, so I've added > their authors to Cc: > > - 3/4 rewrites the same out_subpool_put: block as Zhao Li's > "mm/hugetlb: fix max-only subpool accounting on > alloc_hugetlb_folio failure" (mm-hotfixes-unstable), and has the > same Fixes: tag. Andrew also brought this up on v2 of this series [3]. > - 2/4 rewrites the same out_put_pages: block as Jinmeng Zhou's > "mm/hugetlb: fix subpool minimum reservation rollback" > (mm-unstable). > Just took a quick look, this fix doesn't address the case where there might be another CPU that updates subpool state. In that race, the global state will be messed up. > 3/4 doesn't apply to mm-hotfixes-unstable, and 2/4-4/4 don't apply > to mm-unstable. If I read them right, the queued fixes only handle > mounts with size=, while 2/4 and 3/4 also cover min_size mounts, so > they would probably replace them. > Yes, this set of fixes should be a superset of what's fixed in the other patches. My vote is to switch over to this set. Would it be okay to drop the other fixes from mm-*unstable? > 2. On splitting: 1/4 applies cleanly on top of both queued fixes, but > in my tests it made min_size-only mounts worse on its own. As far > as I can tell, that's because it starts tracking used_hpages on > those mounts, while the two error paths only release it after 2/4 > and 3/4. > > HugePages_Rsvd, expected value in parentheses. Q = the two queued > fixes, wrap = 18446744073709551615: > > rc3 +Q +1/4 +Q+1/4 +1/4..4/4 > min_size=4M only: > SIGBUS faults [1], no files (2) 2 2 0 0 2 > min_size=8M only: > failed mmap [2], umounted (0) 0 0 3 3 0 > size=8M,min_size=4M: > SIGBUS faults [1], no files (2) 0 0 0 0 2 > size=10M,min_size=8M: > failed mmap [2], umounted (0) wrap 0 wrap 0 0 > > With 1/4 alone, the min_size-only mount loses its reservation, and > in the failed-mmap case three huge pages stay reserved after umount, > presumably because the subpool is never freed. > > So it looks to me like 1/4 shouldn't go anywhere without 2/4 and 3/4. Thanks for testing this! IIUC from your table, 1/4 introduces regressions. How do we handle this, should I squash patches 1/2/3 together, or is there some way to use the Fixes: tags to require those 3 to go together? > I haven't tested older stable trees. With all four patches applied, > all the cases above give the expected values, and so does the > partial-truncate case from the 1/4 changelog (Rsvd drops to 0 after > the truncate instead of staying at 1). > > I'm happy to rerun these tests on a rebased v4. > > [1] https://lore.kernel.org/r/20260923065714.20781-1-kmehltretter@gmail.com/ > [2] the scenario from Jinmeng's changelog: > https://lore.kernel.org/20260907132055.26696-1-zhoujinmeng@bytedance.com > > Thanks, > Karl [3] https://lore.kernel.org/all/CAEvNRgHbzY30n1hz2vv7BiBMMhci+NEv_ubP1LjQNmBuu6Kjgw@mail.gmail.com/
Andrew Morton <akpm@linux-foundation.org> writes: > > [...snip...] > Missed out replying to this! > So if downstream people (-stable maintainers, others) follow our > recommendations, some kernels will get two of these patches, other > kernel versions will get three and some lucky kernels might get all > four. Are you confident that the patches can be split apart in this > fashion and still produce a good result? Would it be better to add Fixes tags such that if one patch is applied, all 4 are applied? How do I set that up? Patch 1 is kind of a fundamental fix in subpool accounting, which allows other fixes/simplifications in different places. > After all, we're sending > downstream some patch combinations which we didn't test. > I think patches 2, 3, 4 would kind of depend on 1. Without 1, the others would probably do the wrong thing. Or would it be better to just squash all into 1 patch? Each patch does fix a different part, and also I didn't want too big a patch, which would make it hard to review. In terms of bisecting, even with patch 1, the bugs fixed in patches 2, 3 and 4 would still be buggy. > > [...snip...] >
Andrew Morton <akpm@linux-foundation.org> writes:
> On Wed, 16 Sep 2026 16:39:00 -0700 Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@kernel.org> wrote:
>
>> HugeTLB subpools currently only track used pages (used_hpages) when a
>> maximum size limit (max_hpages) is configured on the mount.
>>
>> ...
>>
>> This series addresses three problem areas resulting from untracked usage
>> and asymmetric error handling:
>
> These are rather serious-sounding problems.
>
> I'm seeing Fixes: for 10-year-old patches, for one last year and one
> this year. The fourth patch doesn't identify what to fix.
>
> So if downstream people (-stable maintainers, others) follow our
> recommendations, some kernels will get two of these patches, other
> kernel versions will get three and some lucky kernels might get all
> four. Are you confident that the patches can be split apart in this
> fashion and still produce a good result? After all, we're sending
> downstream some patch combinations which we didn't test.
>
> Also, can you speculate about why these things took so long to
> discover?
>
I wouldn't have noticed these if Sashiko didn't point out these
pre-existing issues while I was trying to refactor alloc_hugetlb_folio()
in the earlier patch series.
Here are my guesses:
> 1. False restoration and premature subpool freeing (Patch 1):
> + Global reservations (resv_huge_pages) are not decremented when
> reservations are released, depleting available huge pages host-wide.
This requires min_size to be requested and NOT max_size, and it needs
an awkward partial truncation of the file.
> + On unmount, subpool_is_free() checks (rsv_hpages == min_hpages).
> Because of the false reservation restoration, this evaluates to true,
> freeing the subpool while folios are still active in the page cache
> and causing a use-after-free when they are later released.
> (See Patch 1 for a detailed step-by-step trace.)
Unmounting masks the issue, because when the subpool is freed,
spool->rsv_hpages gets returned to the global hstate, restoring the
global state. The use-after-free doesn't actually happen unless a folio
outlives the subpool (not common), since free_huge_folio() is the one
that uses after the subpool is freed.
> 2. Reservation rollback races and underflow (Patches 2 & 4):
> + On hugetlb_acct_memory() failure in hugetlb_reserve_pages(), the
> error path manually calculates rollbacks using local variables.
> This is race-prone when concurrent threads interact with the
> subpool, risking reservation leaks or counter underflows.
I didn't reproduce the race here that could mess up global state, and
races probably hid this. I guess the messing up could go both ways so
perhaps on a machine with heavy load, future races offset the old error?
> + On region_add() failure, prematurely releasing global reservations
> forces error cleanup to attempt fresh allocations that can fail
> under memory pressure.
I didn't write a reproducer for this one, but it's an edge case (memory
pressure causing huge page allocations to fail) of an edge case (OOM on
trying to allocate an entry for the resv_map). It might occur together
if the host is completely out of memory, but I think in that case people
worry more about the system OOM than fixing a HugeTLB bug.
> 3. Subpool usage leak on allocation failure (Patch 3):
> + In alloc_hugetlb_folio(), when folio allocation fails and
> gbl_chg == 1, the cleanup path skips hugepage_subpool_put_pages(),
> permanently leaking used_hpages.
This is a failure path, which probably isn't executed that often. The
reproducer created a very small mount and was specially written to
trigger this edge case.
Overall, I was also second guessing myself: perhaps it was never the
intent to manage reservations by tracking used_hpages throughout, but in
09a95e29cb30 ("mm/hugetlb: optimize minimum size (min_size) accounting")
Mike noted that min size accounting "does not take into account used
pages within the filesystem", and the patch added a check for
used_hpages. Based on that, and the _original_ code documentation below:
long used_hpages; /* Used count against maximum, includes */
/* both allocated and reserved pages. */
I think the original intent was for used_hpages to track both allocated
and reserved pages, and patch 1 restores the original intent, and also
simplifies the logic in many places.
> Ho hum, complicated. Let's at least slap a Fixes: on [4/4]. My
> trillion-dollar sidekick suggests a9e443086489 ("hugetlb: disable
> region_add file_region coalescing").
>
Your trillion-dollar sidekick got the right patch but the wrong hash :)
Fixes: 0db9d74ed884 ("hugetlb: disable region_add file_region coalescing")
I'll also reply to the last patch with the above. If there are no
further revisions required, could you please add the above tag in for
me?
> Our other little friend is talking to us:
> https://sashiko.dev/#/patchset/20260916-hugetlb-subpool-always-track-used-v3-0-38aae9b5ccdd@google.com
>
> I'd like to queue this up promptly because it fixes things. But let's
> seek additional review input and argue with Sashiko before taking that
> step.
I'll copy Sashiko's comments onto this patch series in replies. :)
On Thu, 17 Sep 2026 20:05:33 -0700 Ackerley Tng <ackerleytng@google.com> wrote:
> > Ho hum, complicated. Let's at least slap a Fixes: on [4/4]. My
> > trillion-dollar sidekick suggests a9e443086489 ("hugetlb: disable
> > region_add file_region coalescing").
> >
>
> Your trillion-dollar sidekick got the right patch but the wrong hash :)
>
> Fixes: 0db9d74ed884 ("hugetlb: disable region_add file_region coalescing")
Well, I only feed it $100/mo.
> I'll also reply to the last patch with the above. If there are no
> further revisions required, could you please add the above tag in for
> me?
Shall try to remember to to that. Whack me if you see the
"added-to-mm" email and I forgot.
> > Our other little friend is talking to us:
> > https://sashiko.dev/#/patchset/20260916-hugetlb-subpool-always-track-used-v3-0-38aae9b5ccdd@google.com
> >
> > I'd like to queue this up promptly because it fixes things. But let's
> > seek additional review input and argue with Sashiko before taking that
> > step.
>
> I'll copy Sashiko's comments onto this patch series in replies. :)
Great, thanks.
On Wed, 16 Sep 2026 20:13:07 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:
> On Wed, 16 Sep 2026 16:39:00 -0700 Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@kernel.org> wrote:
>
> > HugeTLB subpools currently only track used pages (used_hpages) when a
> > maximum size limit (max_hpages) is configured on the mount.
> >
> > ...
> >
> > This series addresses three problem areas resulting from untracked usage
> > and asymmetric error handling:
>
> These are rather serious-sounding problems.
>
> I'm seeing Fixes: for 10-year-old patches, for one last year and one
> this year. The fourth patch doesn't identify what to fix.
>
> So if downstream people (-stable maintainers, others) follow our
> recommendations, some kernels will get two of these patches, other
> kernel versions will get three and some lucky kernels might get all
> four. Are you confident that the patches can be split apart in this
> fashion and still produce a good result? After all, we're sending
> downstream some patch combinations which we didn't test.
>
> Also, can you speculate about why these things took so long to
> discover?
Hi Andrew,
I'll be brave and admit that I knew there were hugetlb accounting issues
for a while because I was seeing them at some ~rare~ frequency, but
I couldn't figure out exactly what was happening and didn't report it.
In my defense this was before LLMs were doing a good job of finding
bugs in code ; -)
So I'm very grateful that Ackerley took the time to find what was
happening and sent the fixes.
> Ho hum, complicated. Let's at least slap a Fixes: on [4/4]. My
> trillion-dollar sidekick suggests a9e443086489 ("hugetlb: disable
> region_add file_region coalescing").
I just reviewed the 4th part, I'm not sure if there are any
userspace-visible changes for this. Maybe just an additional retry for
allocating from the global subpool?
> Our other little friend is talking to us:
> https://sashiko.dev/#/patchset/20260916-hugetlb-subpool-always-track-used-v3-0-38aae9b5ccdd@google.com
>
> I'd like to queue this up promptly because it fixes things. But let's
> seek additional review input and argue with Sashiko before taking that
> step.
Agreed, I left my reviewed tag on 2 & 3 but it seems like Sashiko is
raising some concerns. Ackerley, any thoughts on those?
Thank you as always, Andrew!
Joshua
© 2016 - 2026 Red Hat, Inc.