The interleave node selectors and bulk allocators take copies of
nodemasks and node weights (for weighted interleave) in the fault path.
Both of these copies can be entirely eliminated.

For node weights, use SRCU to pin the weights in place.  This eliminates
a copy and a kmalloc from the bulk allocator path.

For nodemasks, we can operate directly on pol->nodes as long as we bounds
check the walk.  A concurrent rebind can shrink the mask, or tear the read
of it so the mask appears empty.

 - The interleave node selectors fall back to numa_node_id() when that
   happens, which is what they already did when a copy came back empty.

 - The bulk allocator simply returns what it managed to allocate.

The node count and weight totals are read separately from the nodemask
walk that consumes them - creating a time-of-check / time-of-use race.
Just clamp the walk to a single pass (number of nodes), and clamp each
bulk allocation chunk to the space left in the request.

The cost is distribution accuracy during a rebind.  The copies never
corrected for that either - they only kept the code from dividing by
zero and overrunning the allocation request.

Gregory Price (2):
  mm/mempolicy: use SRCU for the weighted interleave state
  mm/mempolicy: stop copying the nodemask in the interleave paths

 mm/mempolicy.c | 163 +++++++++++++++++++++++++++----------------------
 1 file changed, 90 insertions(+), 73 deletions(-)

---

v2: Comments and commit message [David]

-- 
2.55.0

On Thu, 17 Sep 2026 20:12:01 -0400 Gregory Price <gourry@gourry.net> wrote:

> The interleave node selectors and bulk allocators take copies of
> nodemasks and node weights (for weighted interleave) in the fault path.
> Both of these copies can be entirely eliminated.
> 
> For node weights, use SRCU to pin the weights in place.  This eliminates
> a copy and a kmalloc from the bulk allocator path.
> 
> For nodemasks, we can operate directly on pol->nodes as long as we bounds
> check the walk.  A concurrent rebind can shrink the mask, or tear the read
> of it so the mask appears empty.
> 
>  - The interleave node selectors fall back to numa_node_id() when that
>    happens, which is what they already did when a copy came back empty.
> 
>  - The bulk allocator simply returns what it managed to allocate.
> 
> The node count and weight totals are read separately from the nodemask
> walk that consumes them - creating a time-of-check / time-of-use race.
> Just clamp the walk to a single pass (number of nodes), and clamp each
> bulk allocation chunk to the space left in the request.
> 
> The cost is distribution accuracy during a rebind.  The copies never
> corrected for that either - they only kept the code from dividing by
> zero and overrunning the allocation request.

Thanks, I updated mm-unstable to this version.

> v2: Comments and commit message [David]

Here's how v2 altered mm.git:


 mm/mempolicy.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

--- a/mm/mempolicy.c~b
+++ a/mm/mempolicy.c
@@ -2231,8 +2231,13 @@ static unsigned int weighted_interleave_
 
 	/*
 	 * The target was calculated in a separate loop, and a concurrent
-	 * rebind can change the total number of nodes.  Clamp this loop to
-	 * a single pass (nnodes) to keep the walk bounded by node count.
+	 * rebind can change the contents of pol->nodes as we calculate.
+	 * Access is safe, in the worst case we suddenly perceive an empty
+	 * nodemask and return numa_node_id() below - otherwise we may
+	 * simply cause a skew in allocations.
+	 *
+	 * Clamp this loop to a single pass (nnodes) to keep the walk
+	 * bounded by node count.
 	 */
 	while (target && nnodes-- && nid < MAX_NUMNODES) {
 		/* detect system default usage */
@@ -2266,6 +2271,8 @@ static unsigned int interleave_nid(struc
 		return numa_node_id();
 	target = ilx % nnodes;
 	nid = first_node(pol->nodes);
+
+	/* A concurrent cpuset rebind may cause us to see an empty nodemask */
 	for (i = 0; i < target && nid < MAX_NUMNODES; i++)
 		nid = next_node_in(nid, pol->nodes);
 
_