From nobody Tue Feb 10 03:38:30 2026 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 009D03161A6 for ; Thu, 23 Oct 2025 13:53:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761227624; cv=none; b=gy1+6GtWOegiohKEox5gNS5s83qE3YFzmYhQ7ZkVPNMKYepicpstLPqZyIRXkoAPwIh11hHDr8VW+tYafH5uLVk/x8+GrDGmylsYjBf/vO02oquTzvoGdIgETTx8LUU239apWctE6Xkm1PHdJx4X2B5AwAoZ5VMFWNUO2bRyY9w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761227624; c=relaxed/simple; bh=Tx3alZCX/iWjxCho3p1HiJERNNhEkf5M0K5IdItTQjY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=uMB/yHUvn2Av2fAxuhmM+LQ5E6KBxF80fPzUwOutJpwReiqOuX1QTOWKmM7PN7a8AHbCwG1cCB6bd8Eg3glYsX//8TRCHkcqax0Py+we/IEY6QDgjSLTMHRhNbl002Ufi7mqWQoTV7q4RQ8ZQNKjlhc3Iz00lEflxpV345ZX78I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id AFC0F1F7CE; Thu, 23 Oct 2025 13:53:01 +0000 (UTC) Authentication-Results: smtp-out2.suse.de; none Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id BF88613B0F; Thu, 23 Oct 2025 13:52:54 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id oPB8LjYz+mjvQQAAD6G6ig (envelope-from ); Thu, 23 Oct 2025 13:52:54 +0000 From: Vlastimil Babka Date: Thu, 23 Oct 2025 15:52:39 +0200 Subject: [PATCH RFC 17/19] slab: refill sheaves from all nodes Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251023-sheaves-for-all-v1-17-6ffa2c9941c0@suse.cz> References: <20251023-sheaves-for-all-v1-0-6ffa2c9941c0@suse.cz> In-Reply-To: <20251023-sheaves-for-all-v1-0-6ffa2c9941c0@suse.cz> To: Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , Harry Yoo Cc: Uladzislau Rezki , "Liam R. Howlett" , Suren Baghdasaryan , Sebastian Andrzej Siewior , Alexei Starovoitov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, bpf@vger.kernel.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.14.3 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: AFC0F1F7CE X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Spam-Flag: NO X-Spam-Score: -4.00 X-Spam-Level: X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-4.00 / 50.00]; REPLY(-4.00)[] __refill_objects() currently only attempts to get partial slabs from the local node and then allocates new slab(s). Expand it to trying also other nodes while observing the remote node defrag ratio, similarly to get_any_partial(). This will prevent allocating new slabs on a node while other nodes have many free slabs. It does mean sheaves will contain non-local objects in that case. Allocations that care about specific node will still be served appropriately, but might get a slowpath allocation. Like get_any_partial() we do observe cpuset_zone_allowed(), although we might be refilling a sheaf that will be then used from a different allocation context. We can also use the resulting refill_objects() in __kmem_cache_alloc_bulk() for non-debug caches. This means kmem_cache_alloc_bulk() will get better performance when sheaves are exhausted. kmem_cache_alloc_bulk() cannot indicate a preferred node so it's compatible with sheaves refill in preferring the local node. Signed-off-by: Vlastimil Babka --- mm/slub.c | 130 ++++++++++++++++++++++++++++++++++++++++++++++++----------= ---- 1 file changed, 102 insertions(+), 28 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index d55afa9b277f..4e003493ba60 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2505,8 +2505,8 @@ static void free_empty_sheaf(struct kmem_cache *s, st= ruct slab_sheaf *sheaf) } =20 static unsigned int -__refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int m= in, - unsigned int max); +refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min, + unsigned int max); =20 static int refill_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf, gfp_t gfp) @@ -2517,8 +2517,8 @@ static int refill_sheaf(struct kmem_cache *s, struct = slab_sheaf *sheaf, if (!to_fill) return 0; =20 - filled =3D __refill_objects(s, &sheaf->objects[sheaf->size], gfp, - to_fill, to_fill); + filled =3D refill_objects(s, &sheaf->objects[sheaf->size], gfp, to_fill, + to_fill); =20 sheaf->size +=3D filled; =20 @@ -6423,25 +6423,21 @@ void kmem_cache_free_bulk(struct kmem_cache *s, siz= e_t size, void **p) EXPORT_SYMBOL(kmem_cache_free_bulk); =20 static unsigned int -__refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int m= in, - unsigned int max) +__refill_objects_node(struct kmem_cache *s, void **p, gfp_t gfp, unsigned = int min, + unsigned int max, struct kmem_cache_node *n) { struct slab *slab, *slab2; struct partial_context pc; unsigned int refilled =3D 0; unsigned long flags; void *object; - int node; =20 pc.flags =3D gfp; pc.min_objects =3D min; pc.max_objects =3D max; =20 - node =3D numa_mem_id(); - - /* TODO: consider also other nodes? */ - if (!get_partial_node_bulk(s, get_node(s, node), &pc)) - goto new_slab; + if (!get_partial_node_bulk(s, n, &pc)) + return 0; =20 list_for_each_entry_safe(slab, slab2, &pc.slabs, slab_list) { =20 @@ -6480,8 +6476,6 @@ __refill_objects(struct kmem_cache *s, void **p, gfp_= t gfp, unsigned int min, } =20 if (unlikely(!list_empty(&pc.slabs))) { - struct kmem_cache_node *n =3D get_node(s, node); - spin_lock_irqsave(&n->list_lock, flags); =20 list_for_each_entry_safe(slab, slab2, &pc.slabs, slab_list) { @@ -6503,13 +6497,91 @@ __refill_objects(struct kmem_cache *s, void **p, gf= p_t gfp, unsigned int min, } } =20 + return refilled; +} =20 - if (likely(refilled >=3D min)) - goto out; +#ifdef CONFIG_NUMA +static unsigned int +__refill_objects_any(struct kmem_cache *s, void **p, gfp_t gfp, unsigned i= nt min, + unsigned int max, int local_node) +{ + struct zonelist *zonelist; + struct zoneref *z; + struct zone *zone; + enum zone_type highest_zoneidx =3D gfp_zone(gfp); + unsigned int cpuset_mems_cookie; + unsigned int refilled =3D 0; + + /* see get_any_partial() for the defrag ratio description */ + if (!s->remote_node_defrag_ratio || + get_cycles() % 1024 > s->remote_node_defrag_ratio) + return 0; + + do { + cpuset_mems_cookie =3D read_mems_allowed_begin(); + zonelist =3D node_zonelist(mempolicy_slab_node(), gfp); + for_each_zone_zonelist(zone, z, zonelist, highest_zoneidx) { + struct kmem_cache_node *n; + unsigned int r; + + n =3D get_node(s, zone_to_nid(zone)); + + if (!n || !cpuset_zone_allowed(zone, gfp) || + n->nr_partial <=3D s->min_partial) + continue; + + r =3D __refill_objects_node(s, p, gfp, min, max, n); + refilled +=3D r; + + if (r >=3D min) { + /* + * Don't check read_mems_allowed_retry() here - + * if mems_allowed was updated in parallel, that + * was a harmless race between allocation and + * the cpuset update + */ + return refilled; + } + p +=3D r; + min -=3D r; + max -=3D r; + } + } while (read_mems_allowed_retry(cpuset_mems_cookie)); + + return refilled; +} +#else +static inline unsigned int +__refill_objects_any(struct kmem_cache *s, void **p, gfp_t gfp, unsigned i= nt min, + unsigned int max, int local_node) +{ + return 0; +} +#endif + +static unsigned int +refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min, + unsigned int max) +{ + int local_node =3D numa_mem_id(); + unsigned int refilled; + unsigned long flags; + struct slab *slab; + void *object; + + refilled =3D __refill_objects_node(s, p, gfp, min, max, + get_node(s, local_node)); + if (refilled >=3D min) + return refilled; + + refilled +=3D __refill_objects_any(s, p + refilled, gfp, min - refilled, + max - refilled, local_node); + if (refilled >=3D min) + return refilled; =20 new_slab: =20 - slab =3D new_slab(s, pc.flags, node); + slab =3D new_slab(s, gfp, local_node); if (!slab) goto out; =20 @@ -6541,8 +6613,8 @@ __refill_objects(struct kmem_cache *s, void **p, gfp_= t gfp, unsigned int min, =20 if (refilled < min) goto new_slab; -out: =20 +out: return refilled; } =20 @@ -6552,18 +6624,20 @@ int __kmem_cache_alloc_bulk(struct kmem_cache *s, g= fp_t flags, size_t size, { int i; =20 - /* - * TODO: this might be more efficient (if necessary) by reusing - * __refill_objects() - */ - for (i =3D 0; i < size; i++) { + if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) { + for (i =3D 0; i < size; i++) { =20 - p[i] =3D ___slab_alloc(s, flags, NUMA_NO_NODE, _RET_IP_, - s->object_size); - if (unlikely(!p[i])) - goto error; + p[i] =3D ___slab_alloc(s, flags, NUMA_NO_NODE, _RET_IP_, + s->object_size); + if (unlikely(!p[i])) + goto error; =20 - maybe_wipe_obj_freeptr(s, p[i]); + maybe_wipe_obj_freeptr(s, p[i]); + } + } else { + i =3D refill_objects(s, p, flags, size, size); + if (i < size) + goto error; } =20 return i; --=20 2.51.1