From nobody Tue Feb 10 04:17:45 2026 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 722A52676E8 for ; Fri, 14 Feb 2025 16:27:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739550473; cv=none; b=aJIAPn2l0+zbFpFt/Jk0CfCmRM6QWUFWwOIfa8M/M7umWUklMO+2VigRf8qXmUbg5bj/rH3FF0GzlIeVKmCXpZ5JdRaaJ48eXrcwgfY7iXeNrjKMtuu1oPyP0PqzZTCoKvA/LyfYKbxpByPNU28UaE+mRts8dp4u8FdRAMQZVjw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739550473; c=relaxed/simple; bh=mRPFid06NgIdjdVbiBZAXkBK2kT2FnvoPAgernzWSi4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=mWy7bkw9BkejCqsIQVYfleU1PmvQUI3nXswj1384x+TlM3cTeC/aqTQ5UI+Uq1meZf1tieRKGNIXd4JbQzBprmMrD6bxM0Cr3jIQNqwJ3TXubtoxvEW5A/Lw4Ajm/YvCRpfCDe8YI4l2rINIqodsu7kW4dqYsDZ5uJM0JU3KoS4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=fail smtp.mailfrom=suse.cz; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id D0C8B1F391; Fri, 14 Feb 2025 16:27:42 +0000 (UTC) Authentication-Results: smtp-out2.suse.de; none Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id B7FB813AF0; Fri, 14 Feb 2025 16:27:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id +NqlLP5ur2eHSAAAD6G6ig (envelope-from ); Fri, 14 Feb 2025 16:27:42 +0000 From: Vlastimil Babka Date: Fri, 14 Feb 2025 17:27:41 +0100 Subject: [PATCH RFC v2 05/10] slab: switch percpu sheaves locking to localtry_lock Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250214-slub-percpu-caches-v2-5-88592ee0966a@suse.cz> References: <20250214-slub-percpu-caches-v2-0-88592ee0966a@suse.cz> In-Reply-To: <20250214-slub-percpu-caches-v2-0-88592ee0966a@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes Cc: Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 50.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[] X-Spam-Flag: NO X-Spam-Score: -4.00 X-Rspamd-Queue-Id: D0C8B1F391 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Spam-Level: Instead of local_lock_irqsave(), use localtry_trylock() when potential callers include irq context, and localtry_lock() otherwise (such as when we already know the gfp flags allow blocking). This should reduce the locking (due to irq disabling/enabling) overhead. Failing to use percpu sheaves in an irq due to preempting an already locked user of sheaves should be rare so it's a favorable tradeoff. Signed-off-by: Vlastimil Babka Reviewed-by: Harry Yoo Reviewed-by: Suren Baghdasaryan --- mm/slub.c | 122 ++++++++++++++++++++++++++++++++++++++--------------------= ---- 1 file changed, 76 insertions(+), 46 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 40175747212fefb27137309b27571abe8d0966e2..3d7345e7e938d53950ed0d6abe8= eb0e93cf8f5b1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -450,7 +450,7 @@ struct slab_sheaf { }; =20 struct slub_percpu_sheaves { - local_lock_t lock; + localtry_lock_t lock; struct slab_sheaf *main; /* never NULL when unlocked */ struct slab_sheaf *spare; /* empty or full, may be NULL */ struct slab_sheaf *rcu_free; @@ -2529,16 +2529,19 @@ static struct slab_sheaf *alloc_full_sheaf(struct k= mem_cache *s, gfp_t gfp) =20 static void __kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void= **p); =20 -static void sheaf_flush_main(struct kmem_cache *s) +/* returns true if at least partially flushed */ +static bool sheaf_flush_main(struct kmem_cache *s) { struct slub_percpu_sheaves *pcs; unsigned int batch, remaining; void *objects[PCS_BATCH_MAX]; struct slab_sheaf *sheaf; - unsigned long flags; + bool ret =3D false; =20 next_batch: - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + return ret; + pcs =3D this_cpu_ptr(s->cpu_sheaves); sheaf =3D pcs->main; =20 @@ -2549,14 +2552,18 @@ static void sheaf_flush_main(struct kmem_cache *s) =20 remaining =3D sheaf->size; =20 - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 __kmem_cache_free_bulk(s, batch, &objects[0]); =20 stat_add(s, SHEAF_FLUSH_MAIN, batch); =20 + ret =3D true; + if (remaining) goto next_batch; + + return ret; } =20 static void sheaf_flush(struct kmem_cache *s, struct slab_sheaf *sheaf) @@ -2593,6 +2600,8 @@ static void rcu_free_sheaf_nobarn(struct rcu_head *he= ad) * Caller needs to make sure migration is disabled in order to fully flush * single cpu's sheaves * + * must not be called from an irq + * * flushing operations are rare so let's keep it simple and flush to slabs * directly, skipping the barn */ @@ -2600,9 +2609,8 @@ static void pcs_flush_all(struct kmem_cache *s) { struct slub_percpu_sheaves *pcs; struct slab_sheaf *spare, *rcu_free; - unsigned long flags; =20 - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + localtry_lock(&s->cpu_sheaves->lock); pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 spare =3D pcs->spare; @@ -2611,7 +2619,7 @@ static void pcs_flush_all(struct kmem_cache *s) rcu_free =3D pcs->rcu_free; pcs->rcu_free =3D NULL; =20 - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 if (spare) { sheaf_flush(s, spare); @@ -4554,10 +4562,11 @@ static __fastpath_inline void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp) { struct slub_percpu_sheaves *pcs; - unsigned long flags; void *object; =20 - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + return NULL; + pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 if (unlikely(pcs->main->size =3D=3D 0)) { @@ -4590,7 +4599,7 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp) } } =20 - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 if (!can_alloc) return NULL; @@ -4612,7 +4621,11 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp) if (!full) return NULL; =20 - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + /* + * we can reach here only when gfpflags_allow_blocking + * so this must not be an irq + */ + localtry_lock(&s->cpu_sheaves->lock); pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 /* @@ -4646,7 +4659,7 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp) do_alloc: object =3D pcs->main->objects[--pcs->main->size]; =20 - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 stat(s, ALLOC_PCS); =20 @@ -4658,12 +4671,13 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache = *s, size_t size, void **p) { struct slub_percpu_sheaves *pcs; struct slab_sheaf *main; - unsigned long flags; unsigned int allocated =3D 0; unsigned int batch; =20 next_batch: - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + return allocated; + pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 if (unlikely(pcs->main->size =3D=3D 0)) { @@ -4683,7 +4697,7 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s= , size_t size, void **p) goto do_alloc; } =20 - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 /* * Once full sheaves in barn are depleted, let the bulk @@ -4701,7 +4715,7 @@ unsigned int alloc_from_pcs_bulk(struct kmem_cache *s= , size_t size, void **p) main->size -=3D batch; memcpy(p, main->objects + main->size, batch * sizeof(void *)); =20 - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 stat_add(s, ALLOC_PCS, batch); =20 @@ -5121,13 +5135,14 @@ static void __slab_free(struct kmem_cache *s, struc= t slab *slab, * The object is expected to have passed slab_free_hook() already. */ static __fastpath_inline -void free_to_pcs(struct kmem_cache *s, void *object) +bool free_to_pcs(struct kmem_cache *s, void *object) { struct slub_percpu_sheaves *pcs; - unsigned long flags; =20 restart: - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + return false; + pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 if (unlikely(pcs->main->size =3D=3D s->sheaf_capacity)) { @@ -5162,7 +5177,7 @@ void free_to_pcs(struct kmem_cache *s, void *object) struct slab_sheaf *to_flush =3D pcs->spare; =20 pcs->spare =3D NULL; - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 sheaf_flush(s, to_flush); empty =3D to_flush; @@ -5170,17 +5185,27 @@ void free_to_pcs(struct kmem_cache *s, void *object) } =20 alloc_empty: - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 empty =3D alloc_empty_sheaf(s, GFP_NOWAIT); =20 if (!empty) { - sheaf_flush_main(s); - goto restart; + if (sheaf_flush_main(s)) + goto restart; + else + return false; } =20 got_empty: - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) { + struct node_barn *barn; + + barn =3D get_node(s, numa_mem_id())->barn; + + barn_put_empty_sheaf(barn, empty, true); + return false; + } + pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 /* @@ -5209,9 +5234,11 @@ void free_to_pcs(struct kmem_cache *s, void *object) do_free: pcs->main->objects[pcs->main->size++] =3D object; =20 - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 stat(s, FREE_PCS); + + return true; } =20 static void __rcu_free_sheaf_prepare(struct kmem_cache *s, @@ -5270,9 +5297,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *o= bj) { struct slub_percpu_sheaves *pcs; struct slab_sheaf *rcu_sheaf; - unsigned long flags; =20 - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + goto fail; + pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 if (unlikely(!pcs->rcu_free)) { @@ -5286,16 +5314,16 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *= obj) goto do_free; } =20 - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 empty =3D alloc_empty_sheaf(s, GFP_NOWAIT); =20 - if (!empty) { - stat(s, FREE_RCU_SHEAF_FAIL); - return false; - } + if (!empty) + goto fail; + + if (!localtry_trylock(&s->cpu_sheaves->lock)) + goto fail; =20 - local_lock_irqsave(&s->cpu_sheaves->lock, flags); pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 if (unlikely(pcs->rcu_free)) @@ -5311,19 +5339,22 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *= obj) rcu_sheaf->objects[rcu_sheaf->size++] =3D obj; =20 if (likely(rcu_sheaf->size < s->sheaf_capacity)) { - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); stat(s, FREE_RCU_SHEAF); return true; } =20 pcs->rcu_free =3D NULL; - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf); =20 stat(s, FREE_RCU_SHEAF); - return true; + +fail: + stat(s, FREE_RCU_SHEAF_FAIL); + return false; } =20 /* @@ -5335,7 +5366,6 @@ static void free_to_pcs_bulk(struct kmem_cache *s, si= ze_t size, void **p) { struct slub_percpu_sheaves *pcs; struct slab_sheaf *main; - unsigned long flags; unsigned int batch, i =3D 0; bool init; =20 @@ -5358,7 +5388,9 @@ static void free_to_pcs_bulk(struct kmem_cache *s, si= ze_t size, void **p) } =20 next_batch: - local_lock_irqsave(&s->cpu_sheaves->lock, flags); + if (!localtry_trylock(&s->cpu_sheaves->lock)) + goto fallback; + pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 if (unlikely(pcs->main->size =3D=3D s->sheaf_capacity)) { @@ -5389,13 +5421,13 @@ static void free_to_pcs_bulk(struct kmem_cache *s, = size_t size, void **p) } =20 no_empty: - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 /* * if we depleted all empty sheaves in the barn or there are too * many full sheaves, free the rest to slab pages */ - +fallback: __kmem_cache_free_bulk(s, size, p); return; } @@ -5407,7 +5439,7 @@ static void free_to_pcs_bulk(struct kmem_cache *s, si= ze_t size, void **p) memcpy(main->objects + main->size, p, batch * sizeof(void *)); main->size +=3D batch; =20 - local_unlock_irqrestore(&s->cpu_sheaves->lock, flags); + localtry_unlock(&s->cpu_sheaves->lock); =20 stat_add(s, FREE_PCS, batch); =20 @@ -5507,9 +5539,7 @@ void slab_free(struct kmem_cache *s, struct slab *sla= b, void *object, if (unlikely(!slab_free_hook(s, object, slab_want_init_on_free(s), false)= )) return; =20 - if (s->cpu_sheaves) - free_to_pcs(s, object); - else + if (!s->cpu_sheaves || !free_to_pcs(s, object)) do_slab_free(s, slab, object, object, 1, addr); } =20 @@ -6288,7 +6318,7 @@ static int init_percpu_sheaves(struct kmem_cache *s) =20 pcs =3D per_cpu_ptr(s->cpu_sheaves, cpu); =20 - local_lock_init(&pcs->lock); + localtry_lock_init(&pcs->lock); =20 nid =3D cpu_to_mem(cpu); =20 --=20 2.48.1