From nobody Tue Feb 10 22:18:12 2026 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 890BC322C7D for ; Thu, 23 Oct 2025 13:53:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761227618; cv=none; b=j1q/1DaZhvDSVJ4JLnoE4fwwLtPnrnpOfxjray4YJw/oZloFMYmAjlqi+tmzu90HsB4Qolnl1X3EjfyvkL8W0cicgfFU5bTMmUPXXlu+zkM0LnXw0Ouf7FLhLctV9xZYusFC30BgKy13FUtKB/YGMw9ka5AshVkusUkS4QGLONQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761227618; c=relaxed/simple; bh=vx1wW9DLHRJV5gzusVwZw+FGAKUmsrqzn78ZmvdiWmA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=FheBqrzmScxD1/dwOZweTK/zWZVKKJoqxH/WOKMEqHfB10hFBuxbLmj/lL8XJzDOSUZdNg76doQ8Dedal/lT11HaY7L6ECy1vNDKqO29637Tqnp0dmeo5pazG32lVcTjrIN1C3PY354O4M4pk7rm+Gx1EDQ7SgWPaXm9EJN7EHE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id AB7B41F7CD; Thu, 23 Oct 2025 13:53:01 +0000 (UTC) Authentication-Results: smtp-out2.suse.de; none Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 73D6613B0C; Thu, 23 Oct 2025 13:52:54 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id AKv3GzYz+mjvQQAAD6G6ig (envelope-from ); Thu, 23 Oct 2025 13:52:54 +0000 From: Vlastimil Babka Date: Thu, 23 Oct 2025 15:52:36 +0200 Subject: [PATCH RFC 14/19] slab: simplify kmalloc_nolock() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251023-sheaves-for-all-v1-14-6ffa2c9941c0@suse.cz> References: <20251023-sheaves-for-all-v1-0-6ffa2c9941c0@suse.cz> In-Reply-To: <20251023-sheaves-for-all-v1-0-6ffa2c9941c0@suse.cz> To: Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , Harry Yoo Cc: Uladzislau Rezki , "Liam R. Howlett" , Suren Baghdasaryan , Sebastian Andrzej Siewior , Alexei Starovoitov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, bpf@vger.kernel.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.14.3 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Queue-Id: AB7B41F7CD X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Spam-Flag: NO X-Spam-Score: -4.00 X-Spam-Level: X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-4.00 / 50.00]; REPLY(-4.00)[] The kmalloc_nolock() implementation has several complications and restrictions due to SLUB's cpu slab locking, lockless fastpath and PREEMPT_RT differences. With cpu slab usage removed, we can simplify things: - the local_lock_cpu_slab() macros became unused, remove them - we no longer need to set up lockdep classes on PREEMPT_RT - we no longer need to annotate ___slab_alloc as NOKPROBE_SYMBOL since there's no lockless cpu freelist manipulation anymore - __slab_alloc_node() can be called from kmalloc_nolock_noprof() unconditionally Note that we still need __CMPXCHG_DOUBLE, because while it was removed we don't use cmpxchg16b on cpu freelist anymore, we still use it on slab freelist, and the alternative is slab_lock() which can be interrupted by a nmi. Clarify the comment to mention it specifically. Signed-off-by: Vlastimil Babka --- mm/slab.h | 1 - mm/slub.c | 100 ++++------------------------------------------------------= ---- 2 files changed, 6 insertions(+), 95 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index b2663cc594f3..7dde0b56a7b0 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -208,7 +208,6 @@ struct kmem_cache_order_objects { */ struct kmem_cache { struct kmem_cache_cpu __percpu *cpu_slab; - struct lock_class_key lock_key; struct slub_percpu_sheaves __percpu *cpu_sheaves; /* Used for retrieving partial slabs, etc. */ slab_flags_t flags; diff --git a/mm/slub.c b/mm/slub.c index 6f5ca26bbb00..6dd7fd153391 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3679,29 +3679,12 @@ static inline unsigned int init_tid(int cpu) =20 static void init_kmem_cache_cpus(struct kmem_cache *s) { -#ifdef CONFIG_PREEMPT_RT - /* - * Register lockdep key for non-boot kmem caches to avoid - * WARN_ON_ONCE(static_obj(key))) in lockdep_register_key() - */ - bool finegrain_lockdep =3D !init_section_contains(s, 1); -#else - /* - * Don't bother with different lockdep classes for each - * kmem_cache, since we only use local_trylock_irqsave(). - */ - bool finegrain_lockdep =3D false; -#endif int cpu; struct kmem_cache_cpu *c; =20 - if (finegrain_lockdep) - lockdep_register_key(&s->lock_key); for_each_possible_cpu(cpu) { c =3D per_cpu_ptr(s->cpu_slab, cpu); local_trylock_init(&c->lock); - if (finegrain_lockdep) - lockdep_set_class(&c->lock, &s->lock_key); c->tid =3D init_tid(cpu); } } @@ -3792,47 +3775,6 @@ static void deactivate_slab(struct kmem_cache *s, st= ruct slab *slab, } } =20 -/* - * ___slab_alloc()'s caller is supposed to check if kmem_cache::kmem_cache= _cpu::lock - * can be acquired without a deadlock before invoking the function. - * - * Without LOCKDEP we trust the code to be correct. kmalloc_nolock() is - * using local_lock_is_locked() properly before calling local_lock_cpu_sla= b(), - * and kmalloc() is not used in an unsupported context. - * - * With LOCKDEP, on PREEMPT_RT lockdep does its checking in local_lock_irq= save(). - * On !PREEMPT_RT we use trylock to avoid false positives in NMI, but - * lockdep_assert() will catch a bug in case: - * #1 - * kmalloc() -> ___slab_alloc() -> irqsave -> NMI -> bpf -> kmalloc_nolock= () - * or - * #2 - * kmalloc() -> ___slab_alloc() -> irqsave -> tracepoint/kprobe -> bpf -> = kmalloc_nolock() - * - * On PREEMPT_RT an invocation is not possible from IRQ-off or preempt - * disabled context. The lock will always be acquired and if needed it - * block and sleep until the lock is available. - * #1 is possible in !PREEMPT_RT only. - * #2 is possible in both with a twist that irqsave is replaced with rt_sp= inlock: - * kmalloc() -> ___slab_alloc() -> rt_spin_lock(kmem_cache_A) -> - * tracepoint/kprobe -> bpf -> kmalloc_nolock() -> rt_spin_lock(kmem_ca= che_B) - * - * local_lock_is_locked() prevents the case kmem_cache_A =3D=3D kmem_cache= _B - */ -#if defined(CONFIG_PREEMPT_RT) || !defined(CONFIG_LOCKDEP) -#define local_lock_cpu_slab(s, flags) \ - local_lock_irqsave(&(s)->cpu_slab->lock, flags) -#else -#define local_lock_cpu_slab(s, flags) \ - do { \ - bool __l =3D local_trylock_irqsave(&(s)->cpu_slab->lock, flags); \ - lockdep_assert(__l); \ - } while (0) -#endif - -#define local_unlock_cpu_slab(s, flags) \ - local_unlock_irqrestore(&(s)->cpu_slab->lock, flags) - static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu = *c) { unsigned long flags; @@ -4320,19 +4262,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp= _t gfpflags, int node, =20 return freelist; } -/* - * We disallow kprobes in ___slab_alloc() to prevent reentrance - * - * kmalloc() -> ___slab_alloc() -> local_lock_cpu_slab() protected part of - * ___slab_alloc() manipulating c->freelist -> kprobe -> bpf -> - * kmalloc_nolock() or kfree_nolock() -> __update_cpu_freelist_fast() - * manipulating c->freelist without lock. - * - * This does not prevent kprobe in functions called from ___slab_alloc() s= uch as - * local_lock_irqsave() itself, and that is fine, we only need to protect = the - * c->freelist manipulation in ___slab_alloc() itself. - */ -NOKPROBE_SYMBOL(___slab_alloc); =20 static __always_inline void *__slab_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node, unsigned long addr, size_t orig_size) @@ -5201,10 +5130,11 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_= flags, int node) if (!(s->flags & __CMPXCHG_DOUBLE) && !kmem_cache_debug(s)) /* * kmalloc_nolock() is not supported on architectures that - * don't implement cmpxchg16b, but debug caches don't use - * per-cpu slab and per-cpu partial slabs. They rely on - * kmem_cache_node->list_lock, so kmalloc_nolock() can - * attempt to allocate from debug caches by + * don't implement cmpxchg16b and thus need slab_lock() + * which could be preempted by a nmi. + * But debug caches don't use that and only rely on + * kmem_cache_node->list_lock, so kmalloc_nolock() can attempt + * to allocate from debug caches by * spin_trylock_irqsave(&n->list_lock, ...) */ return NULL; @@ -5214,27 +5144,13 @@ void *kmalloc_nolock_noprof(size_t size, gfp_t gfp_= flags, int node) if (ret) goto success; =20 - ret =3D ERR_PTR(-EBUSY); - /* * Do not call slab_alloc_node(), since trylock mode isn't * compatible with slab_pre_alloc_hook/should_failslab and * kfence_alloc. Hence call __slab_alloc_node() (at most twice) * and slab_post_alloc_hook() directly. - * - * In !PREEMPT_RT ___slab_alloc() manipulates (freelist,tid) pair - * in irq saved region. It assumes that the same cpu will not - * __update_cpu_freelist_fast() into the same (freelist,tid) pair. - * Therefore use in_nmi() to check whether particular bucket is in - * irq protected section. - * - * If in_nmi() && local_lock_is_locked(s->cpu_slab) then it means that - * this cpu was interrupted somewhere inside ___slab_alloc() after - * it did local_lock_irqsave(&s->cpu_slab->lock, flags). - * In this case fast path with __update_cpu_freelist_fast() is not safe. */ - if (!in_nmi() || !local_lock_is_locked(&s->cpu_slab->lock)) - ret =3D __slab_alloc_node(s, alloc_gfp, node, _RET_IP_, size); + ret =3D __slab_alloc_node(s, alloc_gfp, node, _RET_IP_, size); =20 if (PTR_ERR(ret) =3D=3D -EBUSY) { if (can_retry) { @@ -7250,10 +7166,6 @@ void __kmem_cache_release(struct kmem_cache *s) { cache_random_seq_destroy(s); pcs_destroy(s); -#ifdef CONFIG_PREEMPT_RT - if (s->cpu_slab) - lockdep_unregister_key(&s->lock_key); -#endif free_percpu(s->cpu_slab); free_kmem_cache_nodes(s); } --=20 2.51.1