From nobody Thu Apr 9 05:05:12 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1CEB363C58 for ; Wed, 11 Mar 2026 08:26:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773217584; cv=none; b=m7fuB3gPLg2ewvTzzhvqFzCWcKvCZpilPY+aWV5g+7X9Lmgi2YmeUu7YvBWvM1EiMlshUZpku0AJngiZ7Ipx1GEvqLEY/tzYVmNg3C7ldpEf7dtdwQ/cPnnqjw6B4Od57V7Kx0ljuiP+i2hLWIs2w+rF2HMbFfqkDOz2wTFUgVQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773217584; c=relaxed/simple; bh=J6j7UsI+cNPPl/AmQOY+S8xOMkbllA7iRFJ8y3xLuF8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=bdlONeoeaW+uA0ApO3DwOyPEBAzgnmb8+yTATuJy6PrkDKQ2iwSI+goij+vwstivfaZB3x3wf0/O6dXPXQ63vh+9ra4W5Z3+HcavGpb9MtQj7+Um7P7YnwTbd/JAQL1Oo4N58ph9djvb8sKihCLQFwSTVr6e+wOG3K8oCLtM5fo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=vKUnPNoF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="vKUnPNoF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9F73DC2BC9E; Wed, 11 Mar 2026 08:26:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1773217584; bh=J6j7UsI+cNPPl/AmQOY+S8xOMkbllA7iRFJ8y3xLuF8=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=vKUnPNoFC8EPzF3wz00RcDYXtZzZprk9CE0wuKaf9wTG9GwD0V0HDSBMxA93asiCT pVo8hdXNt8ncLsW3azyEEIAU8XS8HWJxiAaLFH+QrKa4nc+kdELSjyZ95E3VPjBPLG o1SrCbc5KTESoC3d9eVReitG6emHwhhP5sgoGbsCyd8pqQ+5M5tqBB3RFjG5bOJAte Qm1ntGa9wEv0CEgoy6e/FATIh0g3HeNF6D6ySTEhDrrrcqjxgpYqy/EwGfI1B0Hy/v vJ+0QPzJZ4uivz0+rpvh8rrsHnewZmHWjcuJtHVZjNq122TqSQzAWHtfqMckchQttb Ii4ceXDNYpa0g== From: "Vlastimil Babka (SUSE)" Date: Wed, 11 Mar 2026 09:25:55 +0100 Subject: [PATCH 1/3] slab: decouple pointer to barn from kmem_cache_node Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260311-b4-slab-memoryless-barns-v1-1-70ab850be4ce@kernel.org> References: <20260311-b4-slab-memoryless-barns-v1-0-70ab850be4ce@kernel.org> In-Reply-To: <20260311-b4-slab-memoryless-barns-v1-0-70ab850be4ce@kernel.org> To: Ming Lei , Harry Yoo Cc: Hao Li , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Vlastimil Babka (SUSE)" X-Mailer: b4 0.14.3 The pointer to barn currently exists in struct kmem_cache_node. That struct is instantiated for every NUMA node with memory, but we want to have a barn for every online node (including memoryless). Thus decouple the two structures. In struct kmem_cache we have an array for kmem_cache_node pointers that appears to be sized MAX_NUMNODES but the actual size calculation in kmem_cache_init() uses nr_node_ids. Therefore we can't just add another array of barn pointers. Instead change the array to newly introduced struct kmem_cache_per_node_ptrs holding both kmem_cache_node and barn pointer. Adjust barn accessor and allocation/initialization code accordingly. For now no functional change intended, barns are created 1:1 together with kmem_cache_nodes. Signed-off-by: Vlastimil Babka (SUSE) Reviewed-by: Harry Yoo --- mm/slab.h | 7 +++- mm/slub.c | 128 +++++++++++++++++++++++++++++++++++-----------------------= ---- 2 files changed, 78 insertions(+), 57 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index e9ab292acd22..c735e6b4dddb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -191,6 +191,11 @@ struct kmem_cache_order_objects { unsigned int x; }; =20 +struct kmem_cache_per_node_ptrs { + struct node_barn *barn; + struct kmem_cache_node *node; +}; + /* * Slab cache management. */ @@ -247,7 +252,7 @@ struct kmem_cache { struct kmem_cache_stats __percpu *cpu_stats; #endif =20 - struct kmem_cache_node *node[MAX_NUMNODES]; + struct kmem_cache_per_node_ptrs per_node[MAX_NUMNODES]; }; =20 /* diff --git a/mm/slub.c b/mm/slub.c index 20cb4f3b636d..609a183f8533 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -59,7 +59,7 @@ * 0. cpu_hotplug_lock * 1. slab_mutex (Global Mutex) * 2a. kmem_cache->cpu_sheaves->lock (Local trylock) - * 2b. node->barn->lock (Spinlock) + * 2b. barn->lock (Spinlock) * 2c. node->list_lock (Spinlock) * 3. slab_lock(slab) (Only on some arches) * 4. object_map_lock (Only for debugging) @@ -136,7 +136,7 @@ * or spare sheaf can handle the allocation or free, there is no other * overhead. * - * node->barn->lock (spinlock) + * barn->lock (spinlock) * * This lock protects the operations on per-NUMA-node barn. It can quick= ly * serve an empty or full sheaf if available, and avoid more expensive r= efill @@ -436,26 +436,24 @@ struct kmem_cache_node { atomic_long_t total_objects; struct list_head full; #endif - struct node_barn *barn; }; =20 static inline struct kmem_cache_node *get_node(struct kmem_cache *s, int n= ode) { - return s->node[node]; + return s->per_node[node].node; +} + +static inline struct node_barn *get_barn_node(struct kmem_cache *s, int no= de) +{ + return s->per_node[node].barn; } =20 /* - * Get the barn of the current cpu's closest memory node. It may not exist= on - * systems with memoryless nodes but without CONFIG_HAVE_MEMORYLESS_NODES + * Get the barn of the current cpu's memory node. It may be a memoryless n= ode. */ static inline struct node_barn *get_barn(struct kmem_cache *s) { - struct kmem_cache_node *n =3D get_node(s, numa_mem_id()); - - if (!n) - return NULL; - - return n->barn; + return get_barn_node(s, numa_node_id()); } =20 /* @@ -5791,7 +5789,6 @@ bool free_to_pcs(struct kmem_cache *s, void *object, = bool allow_spin) =20 static void rcu_free_sheaf(struct rcu_head *head) { - struct kmem_cache_node *n; struct slab_sheaf *sheaf; struct node_barn *barn =3D NULL; struct kmem_cache *s; @@ -5814,12 +5811,10 @@ static void rcu_free_sheaf(struct rcu_head *head) if (__rcu_free_sheaf_prepare(s, sheaf)) goto flush; =20 - n =3D get_node(s, sheaf->node); - if (!n) + barn =3D get_barn_node(s, sheaf->node); + if (!barn) goto flush; =20 - barn =3D n->barn; - /* due to slab_free_hook() */ if (unlikely(sheaf->size =3D=3D 0)) goto empty; @@ -7430,7 +7425,7 @@ static inline int calculate_order(unsigned int size) } =20 static void -init_kmem_cache_node(struct kmem_cache_node *n, struct node_barn *barn) +init_kmem_cache_node(struct kmem_cache_node *n) { n->nr_partial =3D 0; spin_lock_init(&n->list_lock); @@ -7440,9 +7435,6 @@ init_kmem_cache_node(struct kmem_cache_node *n, struc= t node_barn *barn) atomic_long_set(&n->total_objects, 0); INIT_LIST_HEAD(&n->full); #endif - n->barn =3D barn; - if (barn) - barn_init(barn); } =20 #ifdef CONFIG_SLUB_STATS @@ -7537,8 +7529,8 @@ static void early_kmem_cache_node_alloc(int node) n =3D kasan_slab_alloc(kmem_cache_node, n, GFP_KERNEL, false); slab->freelist =3D get_freepointer(kmem_cache_node, n); slab->inuse =3D 1; - kmem_cache_node->node[node] =3D n; - init_kmem_cache_node(n, NULL); + kmem_cache_node->per_node[node].node =3D n; + init_kmem_cache_node(n); inc_slabs_node(kmem_cache_node, node, slab->objects); =20 /* @@ -7553,15 +7545,20 @@ static void free_kmem_cache_nodes(struct kmem_cache= *s) int node; struct kmem_cache_node *n; =20 - for_each_kmem_cache_node(s, node, n) { - if (n->barn) { - WARN_ON(n->barn->nr_full); - WARN_ON(n->barn->nr_empty); - kfree(n->barn); - n->barn =3D NULL; - } + for_each_node(node) { + struct node_barn *barn =3D get_barn_node(s, node); =20 - s->node[node] =3D NULL; + if (!barn) + continue; + + WARN_ON(barn->nr_full); + WARN_ON(barn->nr_empty); + kfree(barn); + s->per_node[node].barn =3D NULL; + } + + for_each_kmem_cache_node(s, node, n) { + s->per_node[node].node =3D NULL; kmem_cache_free(kmem_cache_node, n); } } @@ -7582,31 +7579,36 @@ static int init_kmem_cache_nodes(struct kmem_cache = *s) =20 for_each_node_mask(node, slab_nodes) { struct kmem_cache_node *n; - struct node_barn *barn =3D NULL; =20 if (slab_state =3D=3D DOWN) { early_kmem_cache_node_alloc(node); continue; } =20 - if (cache_has_sheaves(s)) { - barn =3D kmalloc_node(sizeof(*barn), GFP_KERNEL, node); - - if (!barn) - return 0; - } - n =3D kmem_cache_alloc_node(kmem_cache_node, GFP_KERNEL, node); - if (!n) { - kfree(barn); + if (!n) return 0; - } =20 - init_kmem_cache_node(n, barn); + init_kmem_cache_node(n); + s->per_node[node].node =3D n; + } + + if (slab_state =3D=3D DOWN || !cache_has_sheaves(s)) + return 1; + + for_each_node_mask(node, slab_nodes) { + struct node_barn *barn; + + barn =3D kmalloc_node(sizeof(*barn), GFP_KERNEL, node); + + if (!barn) + return 0; =20 - s->node[node] =3D n; + barn_init(barn); + s->per_node[node].barn =3D barn; } + return 1; } =20 @@ -7895,10 +7897,15 @@ int __kmem_cache_shutdown(struct kmem_cache *s) if (cache_has_sheaves(s)) rcu_barrier(); =20 + for_each_node(node) { + struct node_barn *barn =3D get_barn_node(s, node); + + if (barn) + barn_shrink(s, barn); + } + /* Attempt to free all objects */ for_each_kmem_cache_node(s, node, n) { - if (n->barn) - barn_shrink(s, n->barn); free_partial(s, n); if (n->nr_partial || node_nr_slabs(n)) return 1; @@ -8108,14 +8115,18 @@ static int __kmem_cache_do_shrink(struct kmem_cache= *s) unsigned long flags; int ret =3D 0; =20 + for_each_node(node) { + struct node_barn *barn =3D get_barn_node(s, node); + + if (barn) + barn_shrink(s, barn); + } + for_each_kmem_cache_node(s, node, n) { INIT_LIST_HEAD(&discard); for (i =3D 0; i < SHRINK_PROMOTE_MAX; i++) INIT_LIST_HEAD(promote + i); =20 - if (n->barn) - barn_shrink(s, n->barn); - spin_lock_irqsave(&n->list_lock, flags); =20 /* @@ -8204,7 +8215,8 @@ static int slab_mem_going_online_callback(int nid) if (get_node(s, nid)) continue; =20 - if (cache_has_sheaves(s)) { + if (cache_has_sheaves(s) && !get_barn_node(s, nid)) { + barn =3D kmalloc_node(sizeof(*barn), GFP_KERNEL, nid); =20 if (!barn) { @@ -8225,13 +8237,17 @@ static int slab_mem_going_online_callback(int nid) goto out; } =20 - init_kmem_cache_node(n, barn); + init_kmem_cache_node(n); + s->per_node[nid].node =3D n; =20 - s->node[nid] =3D n; + if (barn) { + barn_init(barn); + s->per_node[nid].barn =3D barn; + } } /* * Any cache created after this point will also have kmem_cache_node - * initialized for the new node. + * and barn initialized for the new node. */ node_set(nid, slab_nodes); out: @@ -8323,7 +8339,7 @@ static void __init bootstrap_cache_sheaves(struct kme= m_cache *s) } =20 barn_init(barn); - get_node(s, node)->barn =3D barn; + s->per_node[node].barn =3D barn; } =20 for_each_possible_cpu(cpu) { @@ -8394,8 +8410,8 @@ void __init kmem_cache_init(void) slab_state =3D PARTIAL; =20 create_boot_cache(kmem_cache, "kmem_cache", - offsetof(struct kmem_cache, node) + - nr_node_ids * sizeof(struct kmem_cache_node *), + offsetof(struct kmem_cache, per_node) + + nr_node_ids * sizeof(struct kmem_cache_per_node_ptrs), SLAB_HWCACHE_ALIGN | SLAB_NO_OBJ_EXT, 0, 0); =20 kmem_cache =3D bootstrap(&boot_kmem_cache); --=20 2.53.0