The interleave node selectors and bulk allocators take copies of
nodemasks and node weights (for weighted interleave) in the fault path.
Both of these copies can be entirely eliminated.

For node weights, use SRCU to pin the weights in place.  This eliminates
a copy and a kmalloc from the bulk allocator path.

For nodemasks, we can operate directly on pol->nodes as long as we bounds
check the walk.  A concurrent rebind can shrink the mask, or tear the read
of it so the mask appears empty.

 - The interleave node selectors fall back to numa_node_id() when that
   happens, which is what they already did when a copy came back empty.

 - The bulk allocator simply returns what it managed to allocate.

The node count and weight totals are read separately from the nodemask
walk that consumes them - creating a time-of-check / time-of-use race.
Just clamp the walk to a single pass (number of nodes), and clamp each
bulk allocation chunk to the space left in the request.

The cost is distribution accuracy during a rebind.  The copies never
corrected for that either - they only kept the code from dividing by
zero and overrunning the allocation request.

Gregory Price (2):
  mm/mempolicy: use SRCU for the weighted interleave state
  mm/mempolicy: stop copying the nodemask in the interleave paths

 mm/mempolicy.c | 156 ++++++++++++++++++++++++++-----------------------
 1 file changed, 83 insertions(+), 73 deletions(-)

-- 
2.55.0

On Fri, 28 Aug 2026 21:59:41 -0400 Gregory Price <gourry@gourry.net> wrote:

> The interleave node selectors and bulk allocators take copies of
> nodemasks and node weights (for weighted interleave) in the fault path.
> Both of these copies can be entirely eliminated.
> 
> For node weights, use SRCU to pin the weights in place.  This eliminates
> a copy and a kmalloc from the bulk allocator path.
> 
> For nodemasks, we can operate directly on pol->nodes as long as we bounds
> check the walk.  A concurrent rebind can shrink the mask, or tear the read
> of it so the mask appears empty.
> 
>  - The interleave node selectors fall back to numa_node_id() when that
>    happens, which is what they already did when a copy came back empty.
> 
>  - The bulk allocator simply returns what it managed to allocate.
> 
> The node count and weight totals are read separately from the nodemask
> walk that consumes them - creating a time-of-check / time-of-use race.
> Just clamp the walk to a single pass (number of nodes), and clamp each
> bulk allocation chunk to the space left in the request.
> 
> The cost is distribution accuracy during a rebind.  The copies never
> corrected for that either - they only kept the code from dividing by
> zero and overrunning the allocation request.

Not very well, it seems.  Sashiko thinks there's a div-by-zero in
alloc_pages_bulk_interleave().

	https://sashiko.dev/#/patchset/20260829015943.1258774-1-gourry@gourry.net

> Gregory Price (2):
>   mm/mempolicy: use SRCU for the weighted interleave state
>   mm/mempolicy: stop copying the nodemask in the interleave paths

Looks nice, thanks - I'll queue it for testing.

On Sat, Aug 29, 2026 at 04:18:50PM -0700, Andrew Morton wrote:
> On Fri, 28 Aug 2026 21:59:41 -0400 Gregory Price <gourry@gourry.net> wrote:
> 
> > The interleave node selectors and bulk allocators take copies of
> > nodemasks and node weights (for weighted interleave) in the fault path.
> > Both of these copies can be entirely eliminated.
> > 
> > For node weights, use SRCU to pin the weights in place.  This eliminates
> > a copy and a kmalloc from the bulk allocator path.
> > 
> > For nodemasks, we can operate directly on pol->nodes as long as we bounds
> > check the walk.  A concurrent rebind can shrink the mask, or tear the read
> > of it so the mask appears empty.
> > 
> >  - The interleave node selectors fall back to numa_node_id() when that
> >    happens, which is what they already did when a copy came back empty.
> > 
> >  - The bulk allocator simply returns what it managed to allocate.
> > 
> > The node count and weight totals are read separately from the nodemask
> > walk that consumes them - creating a time-of-check / time-of-use race.
> > Just clamp the walk to a single pass (number of nodes), and clamp each
> > bulk allocation chunk to the space left in the request.
> > 
> > The cost is distribution accuracy during a rebind.  The copies never
> > corrected for that either - they only kept the code from dividing by
> > zero and overrunning the allocation request.
> 
> Not very well, it seems.  Sashiko thinks there's a div-by-zero in
> alloc_pages_bulk_interleave().
> 
> 	https://sashiko.dev/#/patchset/20260829015943.1258774-1-gourry@gourry.net
> 

That's what this was for :]

https://lore.kernel.org/linux-mm/20260828193111.1023497-1-gourry@gourry.net/

~Gregory