From nobody Sat Jul 25 02:36:11 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46EE54195B5; Mon, 20 Jul 2026 12:45:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551536; cv=none; b=R75PLQqeAwqcZNcJDlNyOG5zgzqT4ACHsJK0Z+V/8YLW969CVWYNGihMqT1vHm1nfllSpZabWHyszKsvz2J81fmaxUyHrVd5yO2qgdTqHDKKXkgCPkRcfJmhV6Cim3GQoAYE/sIi1Sge8gxWAfSaOzTcCfyhcxm9zmrkCAG91eA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551536; c=relaxed/simple; bh=dOo2XI1WhT00YISDEkjFw3/Ye8gqUMdrblWBIBxJN9A=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=NkRoox4+WnICeb+KZpCAbMtUpbWdx1IXQwKU4MzmREPwlk8vQNjEV22CYUo1Da7ztftmW5I/S09volRYfwpgIimDJumBVx4hxbzWEl//VLyMu9ZoPtUZ/7Vryj/UrUZeuMvbXdwaiePB1cS/xuwgKnNo4cjn73BMOfpCtWg7deE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=M6xQMojN; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="M6xQMojN" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 865B11F00AC4; Mon, 20 Jul 2026 12:45:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784551533; bh=nLn2u9Hh8FCrQb3s4mbnvG4RbIr4QffGlO3KDcKz9h0=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=M6xQMojN18hBukXlBKZ0LNGDGz4ICdaN1aqiI+JJOnzHvU1ThqI+NPQuEsyv3/PQ6 him/X3OlnRpbj9wH7jSFcPBm+28ZyRgVlO6o/UqtXKvbbVch+RUAE1Jcbpk6A8NrdK 4Y9mFxfHbbvxXliSDI45KOMBxCRdJx7c73+PKupFWHB3U3I5bzTN0NM1MVe510X1Jq pBzkca8wcyJRni72Z9g+AyeIgnDfs3qUiH7CaeeZq0EzipXfgLdSvQHW5p++H3p9XY sMfAZbU5reUrnQBl1LnZGtloY5xFwJ80MVqdeNeTjeJgSuTiYyFgTNRVxwjnIQVjT2 UrY0FyzE4k3ng== From: "Harry Yoo (Oracle)" Date: Mon, 20 Jul 2026 21:44:30 +0900 Subject: [PATCH slab/for-next v4 1/8] mm/slab, slub_kunit: register kprobe to trigger _nolock APIs Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-kfree_rcu_nolock-v4-1-964e03c41a4e@kernel.org> References: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> In-Reply-To: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> To: Vlastimil Babka , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , Alexei Starovoitov , Andrii Nakryiko , Puranjay Mohan , Amery Hung , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Pedro Falcato , Suren Baghdasaryan , Shengming Hu Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, rcu@vger.kernel.org, bpf@vger.kernel.org, "Harry Yoo (Oracle)" X-Mailer: b4 0.14.3 Since kmalloc_nolock() always fails in NMI and hardirq contexts on PREEMPT_RT, slub_kunit cannot properly test _nolock() APIs. Register a kprobe pre-handler to invoke kmalloc_nolock() and kfree_nolock() in the middle of the slab allocator. However, do not register the handler on UP kernels because that use case is not well supported [1] in the kernel. To attach the pre-handler while s->cpu_sheaves->lock or n->list_lock is held, add a wrapper function for lockdep_assert_held() that calls a no-op function slab_attach_kprobe_locked() on debug builds. The function is optimized away when neither CONFIG_PROVE_LOCKING nor CONFIG_DEBUG_VM is selected and register_kprobe() fails. The function calls barrier() to prevent the compiler from optimizing away its callsites. Otherwise, the compiler may consider the function does not have any side effect and remove callsites. Compared to using plain kprobe, this has two advantages: 1) it avoids hardcoding function names in the test, and 2) it can trigger those APIs in the middle of a function, where the lock is expected to be held as annotated with lockdep. While it was proposed [2] to use kunit function redirection to test this, it is currently infeasible as some lock helpers don't have symbols. Factor out the nested loop that calls kmalloc and friends to test_kmalloc_kfree(), and call them in test_kmalloc_kfree_nolock_{perf,kprobe}(), each being an independent test case. During the refactoring, drop alloc_fail handling as it doesn't provide much benefits. Link: https://lore.kernel.org/linux-mm/20260427-nolock-api-fix-v2-0-a6b83a9= 2d9a4@kernel.org [1] Link: https://lore.kernel.org/linux-mm/6edebc2b-5f5a-4b9c-9a4c-564310acee1b= @kernel.org [2] Acked-by: Vlastimil Babka (SUSE) Signed-off-by: Harry Yoo (Oracle) Reviewed-by: Shengming Hu Suggested-by: Alexei Starovoitov Suggested-by: Vlastimil Babka (SUSE) --- lib/tests/slub_kunit.c | 167 +++++++++++++++++++++++++++++++++++----------= ---- mm/slub.c | 36 ++++++++--- 2 files changed, 148 insertions(+), 55 deletions(-) diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c index fa6d31dbca16..a391467c1aa6 100644 --- a/lib/tests/slub_kunit.c +++ b/lib/tests/slub_kunit.c @@ -8,6 +8,7 @@ #include #include #include +#include #include "../mm/slab.h" =20 static struct kunit_resource resource; @@ -292,7 +293,7 @@ static void test_krealloc_redzone_zeroing(struct kunit = *test) kmem_cache_destroy(s); } =20 -#ifdef CONFIG_PERF_EVENTS +#if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CON= FIG_SMP)) #define NR_ITERATIONS 1000 #define NR_OBJECTS 1000 static void *objects[NR_OBJECTS]; @@ -302,26 +303,40 @@ struct test_nolock_context { int callback_count; int alloc_ok; int alloc_fail; +#ifdef CONFIG_PERF_EVENTS struct perf_event *event; +#endif +#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP) + struct kprobe kprobe; +#endif }; =20 -static struct perf_event_attr hw_attr =3D { - .type =3D PERF_TYPE_HARDWARE, - .config =3D PERF_COUNT_HW_CPU_CYCLES, - .size =3D sizeof(struct perf_event_attr), - .pinned =3D 1, - .disabled =3D 1, - .freq =3D 1, - .sample_freq =3D 100000, -}; +static void test_kmalloc_kfree(void) +{ + int i, j; + + for (i =3D 0; i < NR_ITERATIONS; i++) { + for (j =3D 0; j < NR_OBJECTS; j++) { + gfp_t gfp =3D (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT; =20 -static void overflow_handler_test_kmalloc_kfree_nolock(struct perf_event *= event, - struct perf_sample_data *data, - struct pt_regs *regs) + objects[j] =3D kmalloc_obj(*objects[j], gfp); + if (!objects[j]) { + j--; + while (j >=3D 0) + kfree(objects[j--]); + return; + } + } + + for (j =3D 0; j < NR_OBJECTS; j++) + kfree(objects[j]); + } +} + +static void test_nolock(struct test_nolock_context *ctx) { void *objp; gfp_t gfp; - struct test_nolock_context *ctx =3D event->overflow_handler_context; =20 /* __GFP_ACCOUNT to test kmalloc_nolock() in alloc_slab_obj_exts() */ gfp =3D (ctx->callback_count % 2) ? 0 : __GFP_ACCOUNT; @@ -335,47 +350,104 @@ static void overflow_handler_test_kmalloc_kfree_nolo= ck(struct perf_event *event, kfree_nolock(objp); ctx->callback_count++; } +#endif =20 -static void test_kmalloc_kfree_nolock(struct kunit *test) +#ifdef CONFIG_PERF_EVENTS +static struct perf_event_attr hw_attr =3D { + .type =3D PERF_TYPE_HARDWARE, + .config =3D PERF_COUNT_HW_CPU_CYCLES, + .size =3D sizeof(struct perf_event_attr), + .pinned =3D 1, + .disabled =3D 1, + .freq =3D 1, + .sample_freq =3D 100000, +}; + +static void overflow_handler_test_nolock(struct perf_event *event, + struct perf_sample_data *data, + struct pt_regs *regs) +{ + struct test_nolock_context *ctx =3D event->overflow_handler_context; + + test_nolock(ctx); +} + +static bool enable_perf_events(struct test_nolock_context *ctx) { - int i, j; - struct test_nolock_context ctx =3D { .test =3D test }; struct perf_event *event; - bool alloc_fail =3D false; =20 event =3D perf_event_create_kernel_counter(&hw_attr, -1, current, - overflow_handler_test_kmalloc_kfree_nolock, - &ctx); + overflow_handler_test_nolock, + ctx); + if (IS_ERR(event)) - kunit_skip(test, "Failed to create perf event"); - ctx.event =3D event; - perf_event_enable(ctx.event); - for (i =3D 0; i < NR_ITERATIONS; i++) { - for (j =3D 0; j < NR_OBJECTS; j++) { - gfp_t gfp =3D (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT; + return false; =20 - objects[j] =3D kmalloc(64, gfp); - if (!objects[j]) { - j--; - while (j >=3D 0) - kfree(objects[j--]); - alloc_fail =3D true; - goto cleanup; - } - } - for (j =3D 0; j < NR_OBJECTS; j++) - kfree(objects[j]); - } + ctx->event =3D event; + perf_event_enable(ctx->event); + return true; +} =20 -cleanup: - perf_event_disable(ctx.event); - perf_event_release_kernel(ctx.event); +static void disable_perf_events(struct test_nolock_context *ctx) +{ + kunit_info(ctx->test, "HW perf events: callback_count: %d, alloc_ok: %d, = alloc_fail: %d\n", + ctx->callback_count, ctx->alloc_ok, ctx->alloc_fail); + + perf_event_disable(ctx->event); + perf_event_release_kernel(ctx->event); +} + +static void test_kmalloc_kfree_nolock_perf(struct kunit *test) +{ + struct test_nolock_context ctx =3D { .test =3D test }; =20 - kunit_info(test, "callback_count: %d, alloc_ok: %d, alloc_fail: %d\n", - ctx.callback_count, ctx.alloc_ok, ctx.alloc_fail); + if (!enable_perf_events(&ctx)) + kunit_skip(test, "Failed to enable perf event, skipping"); =20 - if (alloc_fail) - kunit_skip(test, "Allocation failed"); + test_kmalloc_kfree(); + + disable_perf_events(&ctx); + KUNIT_EXPECT_EQ(test, 0, slab_errors); +} +#endif + +#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP) +static int slab_kprobe_pre_handler(struct kprobe *p, struct pt_regs *regs) +{ + struct test_nolock_context *ctx; + + ctx =3D container_of(p, struct test_nolock_context, kprobe); + test_nolock(ctx); + return 0; +} + +static bool register_slab_kprobes(struct test_nolock_context *ctx) +{ + ctx->kprobe.symbol_name =3D "slab_attach_kprobe_locked"; + ctx->kprobe.pre_handler =3D slab_kprobe_pre_handler; + + if (register_kprobe(&ctx->kprobe)) + return false; + return true; +} + +static void unregister_slab_kprobes(struct test_nolock_context *ctx) +{ + kunit_info(ctx->test, "kprobes: callback_count: %d, alloc_ok: %d, alloc_f= ail: %d\n", + ctx->callback_count, ctx->alloc_ok, ctx->alloc_fail); + unregister_kprobe(&ctx->kprobe); +} + +static void test_kmalloc_kfree_nolock_kprobe(struct kunit *test) +{ + struct test_nolock_context ctx =3D { .test =3D test }; + + if (!register_slab_kprobes(&ctx)) + kunit_skip(test, "Failed to register kprobe, skipping"); + + test_kmalloc_kfree(); + + unregister_slab_kprobes(&ctx); KUNIT_EXPECT_EQ(test, 0, slab_errors); } #endif @@ -405,7 +477,10 @@ static struct kunit_case test_cases[] =3D { KUNIT_CASE(test_leak_destroy), KUNIT_CASE(test_krealloc_redzone_zeroing), #ifdef CONFIG_PERF_EVENTS - KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock), + KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_perf), +#endif +#if defined(CONFIG_KPROBES) && defined(CONFIG_SMP) + KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_kprobe), #endif {} }; diff --git a/mm/slub.c b/mm/slub.c index da9efb040040..e32a68677537 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -908,6 +908,24 @@ static inline unsigned int obj_exts_offset_in_object(s= truct kmem_cache *s) } #endif =20 +/* + * A no-op function used to attach kprobe handlers in slub_kunit tests. + * The barrier is needed to prevent the compiler from optimizing out calls= ites. + */ +#if defined(CONFIG_DEBUG_VM) || defined(CONFIG_PROVE_LOCKING) +static noinline void slab_attach_kprobe_locked(void) +{ + barrier(); +} +#else +static inline void slab_attach_kprobe_locked(void) { } +#endif + +#define slab_lockdep_assert_held(lock) do { \ + lockdep_assert_held(lock); \ + slab_attach_kprobe_locked(); \ +} while (0) + #ifdef CONFIG_SLUB_DEBUG =20 /* @@ -1665,7 +1683,7 @@ static void add_full(struct kmem_cache *s, if (!(s->flags & SLAB_STORE_USER)) return; =20 - lockdep_assert_held(&n->list_lock); + slab_lockdep_assert_held(&n->list_lock); list_add(&slab->slab_list, &n->full); } =20 @@ -1674,7 +1692,7 @@ static void remove_full(struct kmem_cache *s, struct = kmem_cache_node *n, struct if (!(s->flags & SLAB_STORE_USER)) return; =20 - lockdep_assert_held(&n->list_lock); + slab_lockdep_assert_held(&n->list_lock); list_del(&slab->slab_list); } =20 @@ -2868,7 +2886,7 @@ static unsigned int __sheaf_flush_main_batch(struct k= mem_cache *s) void *objects[PCS_BATCH_MAX]; struct slab_sheaf *sheaf; =20 - lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); + slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); =20 pcs =3D this_cpu_ptr(s->cpu_sheaves); sheaf =3D pcs->main; @@ -3547,7 +3565,7 @@ __add_partial(struct kmem_cache_node *n, struct slab = *slab, enum add_mode mode) static inline void add_partial(struct kmem_cache_node *n, struct slab *slab, enum add_mode mode) { - lockdep_assert_held(&n->list_lock); + slab_lockdep_assert_held(&n->list_lock); __add_partial(n, slab, mode); } =20 @@ -3561,7 +3579,7 @@ static inline void clear_node_partial_state(struct km= em_cache_node *n, static inline void remove_partial(struct kmem_cache_node *n, struct slab *slab) { - lockdep_assert_held(&n->list_lock); + slab_lockdep_assert_held(&n->list_lock); list_del(&slab->slab_list); clear_node_partial_state(n, slab); } @@ -3577,7 +3595,7 @@ static void *alloc_single_from_partial(struct kmem_ca= che *s, { void *object; =20 - lockdep_assert_held(&n->list_lock); + slab_lockdep_assert_held(&n->list_lock); =20 #ifdef CONFIG_SLUB_DEBUG if (s->flags & SLAB_CONSISTENCY_CHECKS) { @@ -4642,7 +4660,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct= slub_percpu_sheaves *pcs, struct node_barn *barn; bool allow_spin; =20 - lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); + slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); =20 /* Bootstrap or debug cache, back off */ if (unlikely(!cache_has_sheaves(s))) { @@ -5765,7 +5783,7 @@ static void __pcs_install_empty_sheaf(struct kmem_cac= he *s, struct slub_percpu_sheaves *pcs, struct slab_sheaf *empty, struct node_barn *barn) { - lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); + slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); =20 /* This is what we expect to find if nobody interrupted us. */ if (likely(!pcs->spare)) { @@ -5816,7 +5834,7 @@ __pcs_replace_full_main(struct kmem_cache *s, struct = slub_percpu_sheaves *pcs, bool put_fail; =20 restart: - lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); + slab_lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock)); =20 /* Bootstrap or debug cache, back off */ if (unlikely(!cache_has_sheaves(s))) { --=20 2.53.0 From nobody Sat Jul 25 02:36:11 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D92A419317; Mon, 20 Jul 2026 12:45:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551540; cv=none; b=c0/+TnLEeJQpNVmNr2Y3IXTy+LmvOBmui37dG4fSt/kiHQD3JwQodPe1nEQACiVeUzl8vw5Z6pxVWXNdywujpHfHrc/oXvxLx1uJl3kAlgGr38v/5yX9FQZzUSjM+o6bjLZ4AdQ1jFKPl41QdUZ+zBgAQbfG76JLgsjT7l4kYIo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551540; c=relaxed/simple; bh=qZAN/3UpeKtSuciPkQqFy2NNocauB7H5gDD7EAnl7MA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Rj5h/CtL+Yq9tUXMlGzEaUSwVekXInU4VunXIRtNG8fovmAByCB5hmKfYJSucambHAlt2uOvgG6xwd8JdP/vrbhXX+IFCQYopOYkbvMtnJk8H5CG+RLKqTgAYy3x/Qcm/fmv8z8ejDOWFfqg/3chS7jlUtHNB214a+vWKIWPjBg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QzdowtEn; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QzdowtEn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 969B01F00A3A; Mon, 20 Jul 2026 12:45:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784551539; bh=M7imT+FTw9hwkIa++YEpGORHLE52uWQJDNq1DlQ+ILo=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=QzdowtEnOJWwFNbLbV2yvYoyji13SiHT3RbhpTWm01Vb0CDHB5GanlqnnhmLU6vjs IJXAcAqXGPaRQtTnXRM86CdRaGkoeYL67caTz/cQ9bf9MZaBPdnjInjxYoJ9iJojxP UgL2qymEknwnABdFsw/Qd2KPeYvlsNTC6T/Vk0cL85DgTHJqiVx3tnZeWg4vhZJUwG NiB/PJe1cADxfsP6BVJEYuvxjevtnJODGAcFU/K+PTgq4fDfGVrTONKgUfm1BZn8d0 MBKdMb++oT+5U3U9rae8fsFDKfAEPB0U2X69dMIfXZycK26wj9STpnS/sNYSy+B0u+ +F7NhIDhZQwhA== From: "Harry Yoo (Oracle)" Date: Mon, 20 Jul 2026 21:44:31 +0900 Subject: [PATCH slab/for-next v4 2/8] mm/slab: handle the !allow_spin case in kfree_rcu_sheaf() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-kfree_rcu_nolock-v4-2-964e03c41a4e@kernel.org> References: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> In-Reply-To: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> To: Vlastimil Babka , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , Alexei Starovoitov , Andrii Nakryiko , Puranjay Mohan , Amery Hung , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Pedro Falcato , Suren Baghdasaryan , Shengming Hu Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, rcu@vger.kernel.org, bpf@vger.kernel.org, "Harry Yoo (Oracle)" X-Mailer: b4 0.14.3 Teach kfree_rcu_sheaf() how to handle the !allow_spin case. Try to get an empty sheaf from pcs->spare or the barn even when spinning is not allowed. Unlike __pcs_replace_full_main(), try harder to allocate an empty sheaf because the fallback path will be more expensive than kfree_nolock(). Now that slab has internal alloc_flags to describe context, introduce free_flags analogously and convert free_flags to alloc_flags when allocating memory in the free path. alloc_empty_sheaf() now strips __GFP_RECLAIM when SLAB_ALLOC_NOLOCK is specified. When trylock fails or the kernel observes non-NULL pcs->rcu_free after lock acquisition, free the sheaf instead of putting it to the barn. This is rare and not worth complicating the code. Since call_rcu() cannot be called in an unknown context, kfree_rcu_sheaf() fails when the rcu sheaf becomes full. Link: https://lore.kernel.org/linux-mm/872bd673-3d45-4111-8a41-31185db3ece5= @kernel.org Signed-off-by: Harry Yoo (Oracle) Reviewed-by: Vlastimil Babka (SUSE) Suggested-by: Alexei Starovoitov Suggested-by: Vlastimil Babka (SUSE) --- mm/slab.h | 18 +++++++++++++++++- mm/slab_common.c | 2 +- mm/slub.c | 36 ++++++++++++++++++++++++++++-------- 3 files changed, 46 insertions(+), 10 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 281a65233795..85ef2ebc9812 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -23,11 +23,27 @@ #define SLAB_ALLOC_NEW_SLAB 0x02 /* a flag for alloc_slab_obj_exts() */ #define SLAB_ALLOC_NO_RECURSE 0x04 /* prevent kmalloc() recursion */ =20 +#define SLAB_FREE_DEFAULT 0x00 /* no flags */ +#define SLAB_FREE_NOLOCK 0x01 /* spinning not allowed */ + +static inline unsigned int to_alloc_flags(unsigned int free_flags) +{ + if (free_flags & SLAB_FREE_NOLOCK) + return SLAB_ALLOC_NOLOCK; + else + return SLAB_ALLOC_DEFAULT; +} + static inline bool alloc_flags_allow_spinning(const unsigned int alloc_fla= gs) { return !(alloc_flags & SLAB_ALLOC_NOLOCK); } =20 +static inline bool free_flags_allow_spinning(const unsigned int free_flags) +{ + return !(free_flags & SLAB_FREE_NOLOCK); +} + void *__kmalloc_flags_noprof(DECL_TOKEN_PARAMS(size, token), gfp_t flags, unsigned int alloc_flags, int node) __assume_kmalloc_alignment __alloc_size(1); @@ -429,7 +445,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache = *s) return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT)); } =20 -bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj); +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_= flags); void flush_all_rcu_sheaves(void); void flush_rcu_sheaves_on_cache(struct kmem_cache *s); =20 diff --git a/mm/slab_common.c b/mm/slab_common.c index b6426d7ceec9..e07b4e6d6679 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1605,7 +1605,7 @@ static bool kfree_rcu_sheaf(void *obj) =20 s =3D slab->slab_cache; if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) =3D=3D numa_mem_id(= ))) - return __kfree_rcu_sheaf(s, obj); + return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT); =20 return false; } diff --git a/mm/slub.c b/mm/slub.c index e32a68677537..0c350274fbff 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2814,10 +2814,14 @@ static inline struct slab_sheaf *alloc_empty_sheaf(= struct kmem_cache *s, =20 gfp &=3D ~OBJCGS_CLEAR_MASK; =20 + if (alloc_flags & SLAB_ALLOC_NOLOCK) + gfp &=3D ~__GFP_RECLAIM; + return __alloc_empty_sheaf(s, gfp, alloc_flags, s->sheaf_capacity); } =20 -static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *shea= f) +static void __free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sh= eaf, + bool allow_spin) { /* * If the sheaf was created with SLAB_ALLOC_NO_RECURSE flag then its @@ -2829,11 +2833,20 @@ static void free_empty_sheaf(struct kmem_cache *s, = struct slab_sheaf *sheaf) mark_obj_codetag_empty(sheaf); =20 VM_WARN_ON_ONCE(sheaf->size > 0); - kfree(sheaf); + + if (likely(allow_spin)) + kfree(sheaf); + else + kfree_nolock(sheaf); =20 stat(s, SHEAF_FREE); } =20 +static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *shea= f) +{ + __free_empty_sheaf(s, sheaf, /* allow_spin =3D */ true); +} + static unsigned int refill_objects(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min, unsigned int max); @@ -6044,10 +6057,11 @@ static void rcu_free_sheaf(struct rcu_head *head) */ static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG); =20 -bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj) +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_= flags) { struct slub_percpu_sheaves *pcs; struct slab_sheaf *rcu_sheaf; + bool allow_spin =3D free_flags_allow_spinning(free_flags); =20 if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT))) return false; @@ -6060,9 +6074,9 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *ob= j) pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 if (unlikely(!pcs->rcu_free)) { - struct slab_sheaf *empty; struct node_barn *barn; + unsigned int alloc_flags =3D to_alloc_flags(free_flags); =20 /* Bootstrap or debug cache, fall back */ if (unlikely(!cache_has_sheaves(s))) { @@ -6082,7 +6096,7 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *ob= j) goto fail; } =20 - empty =3D barn_get_empty_sheaf(barn, true); + empty =3D barn_get_empty_sheaf(barn, allow_spin); =20 if (empty) { pcs->rcu_free =3D empty; @@ -6091,20 +6105,20 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *= obj) =20 local_unlock(&s->cpu_sheaves->lock); =20 - empty =3D alloc_empty_sheaf(s, GFP_NOWAIT, SLAB_ALLOC_DEFAULT); + empty =3D alloc_empty_sheaf(s, GFP_NOWAIT, alloc_flags); =20 if (!empty) goto fail; =20 if (!local_trylock(&s->cpu_sheaves->lock)) { - barn_put_empty_sheaf(barn, empty); + __free_empty_sheaf(s, empty, allow_spin); goto fail; } =20 pcs =3D this_cpu_ptr(s->cpu_sheaves); =20 if (unlikely(pcs->rcu_free)) - barn_put_empty_sheaf(barn, empty); + __free_empty_sheaf(s, empty, allow_spin); else pcs->rcu_free =3D empty; } @@ -6122,6 +6136,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *o= bj) if (likely(rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf =3D NULL; } else { + if (unlikely(!allow_spin)) { + /* call_rcu() cannot be called in an unknown context */ + rcu_sheaf->size--; + local_unlock(&s->cpu_sheaves->lock); + goto fail; + } pcs->rcu_free =3D NULL; rcu_sheaf->node =3D numa_node_id(); } --=20 2.53.0 From nobody Sat Jul 25 02:36:11 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B91AA3D7A16; Mon, 20 Jul 2026 12:45:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551546; cv=none; b=ddWnwzX+L6WIjCnUPuvIw0tM233oObCsrI7iS1jlJnvOGmuoLFXdZE4tzYF5U/10UETphnA+MuhnEj7D38g1CIyUand4QS5hvcLXQDQJofJIR9zFqKPG0TB0+GwAih098Xgd4exROzSSLWMxSYyrNkNpcl4m10TU/1AT5+5uePU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551546; c=relaxed/simple; bh=gBca40z1NbOcdl7POOTl1q4QE43EIoZSZD2kATW63Kw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=hO72aXjVyuRQNjd9vdGnSH16TU0wJkKfdhvoxUnUZK90g/+12O+DlOx9YRXRcLXEb5YLLAN5LH6uy0kZNFLiexjasrBcqKxgWKd/hremURDqtzhL4SEL+nKMEP/rEaiEK9p/3TLs1DxfQYp1lcJZq+MdtGCkVWoo17I8b9p7Mx8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JSe/I62a; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JSe/I62a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CD0951F000E9; Mon, 20 Jul 2026 12:45:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784551545; bh=gEdEptG0SQeRDcDet0HzsJ7ij1w4V3/phd6b9zQynDM=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=JSe/I62a0Ly9w+RpYYJBwxSnl/fcB1i1i+ovbkac+K7zljbU1l40+93Xs9ETK/nQY 0ye6HHrTCwU0wr71HfK6CPqwYFw5hQcIAa8Hk2ae518qQl1QP8z41Q+RLWh1ObU537 AkAWqNhBPp5ZLV4OZqBzUvCG6IrpXb1HT8FxcEjKLsSKB1Gl3H8hKaMMhFtK7fJTIT m2XTAQcrOLOAMCRT4Rtzt9qqMI23AtvrfVJFv1nbI5R9WwrA8WwX+rc1Ar2VkE2OhC jCQXx9oE+gwDRDAQXiqlXkuhvihklWv0jIwGHvHR0E27/VlTC1rb7m2/ZdGQs2P9hS KYY9IVpgfF00Q== From: "Harry Yoo (Oracle)" Date: Mon, 20 Jul 2026 21:44:32 +0900 Subject: [PATCH slab/for-next v4 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-kfree_rcu_nolock-v4-3-964e03c41a4e@kernel.org> References: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> In-Reply-To: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> To: Vlastimil Babka , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , Alexei Starovoitov , Andrii Nakryiko , Puranjay Mohan , Amery Hung , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Pedro Falcato , Suren Baghdasaryan , Shengming Hu Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, rcu@vger.kernel.org, bpf@vger.kernel.org, "Harry Yoo (Oracle)" X-Mailer: b4 0.14.3 call_rcu() disables IRQs with local_irq_save() to protect its per-cpu data structures. Therefore, if IRQs are not disabled, they cannot be corrupted by reentrance into call_rcu(). So fall back to the deferred path only when !allow_spin && irqs_disabled(). The RCU subsystem does not guarantee this contractually, and this optimization relies on RCU's implementation details. Ideally, it should be removed once call_rcu_nolock() is supported by the RCU subsystem. Link: https://lore.kernel.org/linux-mm/CAADnVQKRVD5ZSnEKbZZU7w86gHbGHUug2pv= zpgZTngNS+fg4rw@mail.gmail.com Suggested-by: Alexei Starovoitov Signed-off-by: Harry Yoo (Oracle) Suggested-by: Vlastimil Babka (SUSE) --- mm/slub.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 0c350274fbff..680fe64aff72 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -6136,8 +6136,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *o= bj, unsigned int free_flags) if (likely(rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf =3D NULL; } else { - if (unlikely(!allow_spin)) { - /* call_rcu() cannot be called in an unknown context */ + /* + * With !allow_spin, we might have interrupted call_rcu()'s + * IRQ-disabled critical section. If IRQs are not disabled, + * we know that's not the case. + */ + if (unlikely(!allow_spin && irqs_disabled())) { rcu_sheaf->size--; local_unlock(&s->cpu_sheaves->lock); goto fail; --=20 2.53.0 From nobody Sat Jul 25 02:36:11 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E4B1419FCC; Mon, 20 Jul 2026 12:45:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551553; cv=none; b=WhlbRFVHEROFMkOH7ucUTOR2BSVBCkvzyZOJwretpgB+5ZOpKXXSOinSX7RArxOCHbOLL26Ey45xIpXgcQ/bdK2+c50bRUudAHKHk+XInwp2lvnqEYNZi9IUbJsNqmDOnPrSOs2pfUsjjbXG+jQ+4xar4nmS/40UuPlZc98NJ34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551553; c=relaxed/simple; bh=8DPvhzfaG4acHM+kXrphkoecMpwVSfDPYU60IuVu2wM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=rWTg7fx2X6XgKizEw8FihN4T7+y0HmmmHU6j8RjYkQb4WE3Xbxl07YCgfP1gZvuNAYSKp4E0gYGAYzQ4xMAtGnmABLKPVXNBs3KP3ghBqxMPIy7078KM3K344FFWr2tSKcyI/P5Pd42PPnMsT9Y4jbWk8DihSG8h6rF13zG4ds8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZFj+X6YN; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZFj+X6YN" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0909D1F00A3E; Mon, 20 Jul 2026 12:45:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784551551; bh=Ra4Fb4kdhCfFUfu58piVrVDiMeFaEqKjymoRqiTdUjg=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=ZFj+X6YNmvcSASRoE+QeBprhXOaKzVYC2OqVlhXlhv6dQ1ZN0TmnjT540pnjBbT7y PfPqWq/JnZ3IbEMSUhjw9rYzuEufryOAzk46N7JDsG4kGg2pfUjhwWbR9Qa810QNb1 CiqLX9XqfQvaM9+/uhOEOAFBiAYI9zcrb9czKW0Ny4phm+IaWp8AAKuGLZvBpZm5nE du4DwRjr+IbDnjrcXk0Os6YR7QEc/l5IZqXl+dEeEMZ29iMWXchp/wcPn6hQ0j4P3X qyYO7jdNW5gsa7+kXPYp9JvJyZ9Oq1dEk88i2vCWyVmlZtrvlkGEJxBW9FOsuTr5LZ vvFsJtpotgL8g== From: "Harry Yoo (Oracle)" Date: Mon, 20 Jul 2026 21:44:33 +0900 Subject: [PATCH slab/for-next v4 4/8] mm/slab: extend deferred free mechanism to handle rcu sheaves Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-kfree_rcu_nolock-v4-4-964e03c41a4e@kernel.org> References: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> In-Reply-To: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> To: Vlastimil Babka , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , Alexei Starovoitov , Andrii Nakryiko , Puranjay Mohan , Amery Hung , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Pedro Falcato , Suren Baghdasaryan , Shengming Hu Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, rcu@vger.kernel.org, bpf@vger.kernel.org, "Harry Yoo (Oracle)" X-Mailer: b4 0.14.3 __kfree_rcu_sheaf() cannot invoke call_rcu() when spinning is not allowed and IRQs are disabled. To relax the limitation, extend the deferred free fallback so that a full rcu sheaf can be submitted to call_rcu() via the existing IRQ work. Since the deferred mechanism does more than deferred freeing of objects, rename the struct to deferred_percpu_work and adjust names accordingly. When a sheaf is queued on an IRQ work, it is detached from pcs->rcu_free but call_rcu() is not invoked until the irq_work runs. To keep the kvfree_rcu barrier's promise, call irq_work_sync() on each CPU before calling rcu_barrier(). In the meantime, remove the TODO item as apparently there is no simple and effective way to achieve that. This is because, unlike sheaves, kfree_rcu() batches objects from different caches together. Suggested-by: Alexei Starovoitov Reviewed-by: Pedro Falcato Reviewed-by: Vlastimil Babka (SUSE) Signed-off-by: Harry Yoo (Oracle) Suggested-by: Vlastimil Babka (SUSE) --- mm/slab.h | 2 +- mm/slab_common.c | 7 ++--- mm/slub.c | 85 +++++++++++++++++++++++++++++++++-------------------= ---- 3 files changed, 53 insertions(+), 41 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 85ef2ebc9812..fdd293b3efa5 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -760,7 +760,7 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *o= bject, struct slab *slab) void __check_heap_object(const void *ptr, unsigned long n, const struct slab *slab, bool to_user); =20 -void defer_free_barrier(void); +void deferred_work_barrier(void); =20 static inline bool slub_debug_orig_size(struct kmem_cache *s) { diff --git a/mm/slab_common.c b/mm/slab_common.c index e07b4e6d6679..9c9ae1384f47 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -551,7 +551,7 @@ void kmem_cache_destroy(struct kmem_cache *s) } =20 /* Wait for deferred work from kmalloc/kfree_nolock() */ - defer_free_barrier(); + deferred_work_barrier(); =20 cpus_read_lock(); mutex_lock(&slab_mutex); @@ -2113,13 +2113,10 @@ void kvfree_rcu_barrier_on_cache(struct kmem_cache = *s) cpus_read_lock(); flush_rcu_sheaves_on_cache(s); cpus_read_unlock(); + deferred_work_barrier(); rcu_barrier(); } =20 - /* - * TODO: Introduce a version of __kvfree_rcu_barrier() that works - * on a specific slab cache. - */ __kvfree_rcu_barrier(); } EXPORT_SYMBOL_GPL(kvfree_rcu_barrier_on_cache); diff --git a/mm/slub.c b/mm/slub.c index 680fe64aff72..8afa6b47b1f2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -418,6 +418,8 @@ struct slab_sheaf { union { struct rcu_head rcu_head; struct list_head barn_list; + /* only used to defer call_rcu() in unknown context */ + struct llist_node llnode; /* only used for prefilled sheafs */ struct { unsigned int capacity; @@ -4077,6 +4079,20 @@ static void flush_all(struct kmem_cache *s) cpus_read_unlock(); } =20 +struct deferred_percpu_work { + struct llist_head objects; + struct llist_head rcu_sheaves; + struct irq_work work; +}; + +static void deferred_percpu_work_fn(struct irq_work *work); + +static DEFINE_PER_CPU(struct deferred_percpu_work, deferred_percpu_work) = =3D { + .objects =3D LLIST_HEAD_INIT(objects), + .rcu_sheaves =3D LLIST_HEAD_INIT(rcu_sheaves), + .work =3D IRQ_WORK_INIT(deferred_percpu_work_fn), +}; + static void flush_rcu_sheaf(struct work_struct *w) { struct slub_percpu_sheaves *pcs; @@ -4148,6 +4164,7 @@ void flush_all_rcu_sheaves(void) mutex_unlock(&slab_mutex); cpus_read_unlock(); =20 + deferred_work_barrier(); rcu_barrier(); } =20 @@ -6136,16 +6153,6 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *o= bj, unsigned int free_flags) if (likely(rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf =3D NULL; } else { - /* - * With !allow_spin, we might have interrupted call_rcu()'s - * IRQ-disabled critical section. If IRQs are not disabled, - * we know that's not the case. - */ - if (unlikely(!allow_spin && irqs_disabled())) { - rcu_sheaf->size--; - local_unlock(&s->cpu_sheaves->lock); - goto fail; - } pcs->rcu_free =3D NULL; rcu_sheaf->node =3D numa_node_id(); } @@ -6154,8 +6161,22 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *o= bj, unsigned int free_flags) * we flush before local_unlock to make sure a racing * flush_all_rcu_sheaves() doesn't miss this sheaf */ - if (rcu_sheaf) - call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf); + if (rcu_sheaf) { + /* + * With !allow_spin, we might have interrupted call_rcu()'s + * IRQ-disabled critical section. If IRQs are not disabled, + * we know that's not the case. + */ + if (unlikely(!allow_spin && irqs_disabled())) { + struct deferred_percpu_work *dpw; + + dpw =3D this_cpu_ptr(&deferred_percpu_work); + if (llist_add(&rcu_sheaf->llnode, &dpw->rcu_sheaves)) + irq_work_queue(&dpw->work); + } else { + call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf); + } + } =20 local_unlock(&s->cpu_sheaves->lock); =20 @@ -6354,31 +6375,21 @@ static void free_to_pcs_bulk(struct kmem_cache *s, = size_t size, void **p) } } =20 -struct defer_free { - struct llist_head objects; - struct irq_work work; -}; - -static void free_deferred_objects(struct irq_work *work); - -static DEFINE_PER_CPU(struct defer_free, defer_free_objects) =3D { - .objects =3D LLIST_HEAD_INIT(objects), - .work =3D IRQ_WORK_INIT(free_deferred_objects), -}; - /* * In PREEMPT_RT irq_work runs in per-cpu kthread, so it's safe * to take sleeping spin_locks from __slab_free(). * In !PREEMPT_RT irq_work will run after local_unlock_irqrestore(). */ -static void free_deferred_objects(struct irq_work *work) +static void deferred_percpu_work_fn(struct irq_work *work) { - struct defer_free *df =3D container_of(work, struct defer_free, work); - struct llist_head *objs =3D &df->objects; + struct deferred_percpu_work *dpw; + struct llist_head *objs, *rcu_sheaves; struct llist_node *llnode, *pos, *t; + struct slab_sheaf *sheaf, *next; =20 - if (llist_empty(objs)) - return; + dpw =3D container_of(work, struct deferred_percpu_work, work); + rcu_sheaves =3D &dpw->rcu_sheaves; + objs =3D &dpw->objects; =20 llnode =3D llist_del_all(objs); llist_for_each_safe(pos, t, llnode) { @@ -6402,27 +6413,31 @@ static void free_deferred_objects(struct irq_work *= work) __slab_free(s, slab, x, x, 1, _THIS_IP_); stat(s, FREE_SLOWPATH); } + + llnode =3D llist_del_all(rcu_sheaves); + llist_for_each_entry_safe(sheaf, next, llnode, llnode) + call_rcu(&sheaf->rcu_head, rcu_free_sheaf); } =20 static void defer_free(struct kmem_cache *s, void *head) { - struct defer_free *df; + struct deferred_percpu_work *dpw; =20 guard(preempt)(); =20 head =3D kasan_reset_tag(head); =20 - df =3D this_cpu_ptr(&defer_free_objects); - if (llist_add(head + s->offset, &df->objects)) - irq_work_queue(&df->work); + dpw =3D this_cpu_ptr(&deferred_percpu_work); + if (llist_add(head + s->offset, &dpw->objects)) + irq_work_queue(&dpw->work); } =20 -void defer_free_barrier(void) +void deferred_work_barrier(void) { int cpu; =20 for_each_possible_cpu(cpu) - irq_work_sync(&per_cpu_ptr(&defer_free_objects, cpu)->work); + irq_work_sync(&per_cpu_ptr(&deferred_percpu_work, cpu)->work); } =20 static __fastpath_inline --=20 2.53.0 From nobody Sat Jul 25 02:36:11 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CAA141A76D; Mon, 20 Jul 2026 12:45:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551559; cv=none; b=L9DogLHOKrBl/TljbrAYuU3GlV1bYAIy7N8lUAjLFddbz6wumzUjkbChy9nXfDBUJ6NitC+9/eBfxstP3TyaRJDkTE84ioKci1ZKU4ymKcLgZyh/mgyhGmmiWwdp6JcpvRFk6KbN/aa0J1JNDEU8/NQ8V5iE4FkVk+UxbTJX4rM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551559; c=relaxed/simple; bh=cFwkADh+XAG2oI8Bdc9dWbLEIl5vVV9vjBqbY6QkSLE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=a0eZmDLAdBQEHy0LXkSqMmUAJ+6n9aBz2hOsfdLsakgnjQq9cCn4Qv2yj2IMsPIpVx1sXV8bhaj5jEuV4GB1skzk3E7F1Pdx/CGBmQRnWS36wb7/Zr3i94XyEq9/3q76caui5nNzKz6Me2ROllOryqIObReErlO2x+9eU7NHNCY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=j93iNg0a; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="j93iNg0a" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 472DE1F00A3D; Mon, 20 Jul 2026 12:45:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784551558; bh=bu4nv7yTgUcNSgTheBamJjAfAc2vPVO37Yi1zmMWM20=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=j93iNg0aWP450n0a3vmlN7SaoaGxWSkb3BUzwdA2rGcitaFkj5+dSMRAEPgJh7aJl 6ZLAxACbMa3fYYF/mJ/whZO48aCz5FNPlHdLCruy3FguwoIInFxB5BiAJgHZ9q7LUb ufENV/Ga+9Xxe1KeECbeaPU2oA/D0QbA/FPqGaqW5PFGCs4lqTYh6+0lqI7H+HL2pc utgb4ysvYjoWYv23sM/f9OL7I6EfsyQVKEgQnkZ2/YsHRQZJafAIFGKKd6wKdaEObW 4v41gtUlBdI91V2Ku/P1rQLoKXieIFD6k6jS7mV0AINyqNZXXCmLY5VCwGDo5+lmgH tT1ZjcfeH+Pbg== From: "Harry Yoo (Oracle)" Date: Mon, 20 Jul 2026 21:44:34 +0900 Subject: [PATCH slab/for-next v4 5/8] mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-kfree_rcu_nolock-v4-5-964e03c41a4e@kernel.org> References: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> In-Reply-To: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> To: Vlastimil Babka , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , Alexei Starovoitov , Andrii Nakryiko , Puranjay Mohan , Amery Hung , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Pedro Falcato , Suren Baghdasaryan , Shengming Hu Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, rcu@vger.kernel.org, bpf@vger.kernel.org, "Harry Yoo (Oracle)" X-Mailer: b4 0.14.3 As suggested by Vlastimil Babka [1], kfree_rcu_sheaf() can be used on PREEMPT_RT if we always assume spinning is not allowed on PREEMPT_RT. This is because local_trylock and spinlock_t are safe to use with trylock and unlock as long as the kernel does not spin and the context is not NMI and not hardirq. Now that __kfree_rcu_sheaf() knows how to handle SLAB_FREE_NOLOCK, relax the limitation and try the sheaves path on PREEMPT_RT as well. Keep the lockdep map on non RT kernels. However, do not use the lockdep map on PREEMPT_RT to avoid suppressing valid lockdep warnings. As pointed by Vlastimil Babka [2], on PREEMPT_RT it is unnecessary to defer call_rcu() under IRQ-disabled section or raw spinlock. However, let us avoid adding more complexity as the scenario is not supposed to be common on PREEMPT_RT, with a hope that call_rcu_nolock() will be soon supported in RCU. Link: https://lore.kernel.org/linux-mm/6811cc17-8ee4-48c8-8cbf-6bf4d9f98162= @kernel.org [1] Link: https://lore.kernel.org/linux-mm/40591888-3a87-433e-b3d2-cda1cab543be= @kernel.org [2] Suggested-by: Vlastimil Babka (SUSE) Signed-off-by: Harry Yoo (Oracle) Reviewed-by: Vlastimil Babka (SUSE) Suggested-by: Alexei Starovoitov --- mm/slab_common.c | 12 ++++++++++-- mm/slub.c | 17 ++++++++++------- 2 files changed, 20 insertions(+), 9 deletions(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 9c9ae1384f47..8c631bf97cd5 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1595,6 +1595,14 @@ static bool kfree_rcu_sheaf(void *obj) { struct kmem_cache *s; struct slab *slab; + unsigned int free_flags =3D SLAB_FREE_DEFAULT; + + /* + * It is not safe to spin on PREEMPT_RT because the kernel might be + * holding a raw spinlock and slab acquires sleeping locks. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + free_flags =3D SLAB_FREE_NOLOCK; =20 if (is_vmalloc_addr(obj)) return false; @@ -1605,7 +1613,7 @@ static bool kfree_rcu_sheaf(void *obj) =20 s =3D slab->slab_cache; if (likely(!IS_ENABLED(CONFIG_NUMA) || slab_nid(slab) =3D=3D numa_mem_id(= ))) - return __kfree_rcu_sheaf(s, obj, SLAB_FREE_DEFAULT); + return __kfree_rcu_sheaf(s, obj, free_flags); =20 return false; } @@ -1954,7 +1962,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr) if (!head) might_sleep(); =20 - if (!IS_ENABLED(CONFIG_PREEMPT_RT) && kfree_rcu_sheaf(ptr)) + if (kfree_rcu_sheaf(ptr)) return; =20 // Queue the object but don't yet schedule the batch. diff --git a/mm/slub.c b/mm/slub.c index 8afa6b47b1f2..deac315d0f23 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -6065,12 +6065,13 @@ static void rcu_free_sheaf(struct rcu_head *head) * kvfree_call_rcu() can be called while holding a raw_spinlock_t. Since * __kfree_rcu_sheaf() may acquire a spinlock_t (sleeping lock on PREEMPT_= RT), * this would violate lock nesting rules. Therefore, kvfree_call_rcu() avo= ids - * this problem by bypassing the sheaves layer entirely on PREEMPT_RT. + * this problem by passing allow_spin =3D false on PREEMPT_RT. * * However, lockdep still complains that it is invalid to acquire spinlock= _t * while holding raw_spinlock_t, even on !PREEMPT_RT where spinlock_t is a * spinning lock. Tell lockdep that acquiring spinlock_t is valid here - * by temporarily raising the wait-type to LD_WAIT_CONFIG. + * by temporarily raising the wait-type to LD_WAIT_CONFIG. Skip the lockde= p map + * on PREEMPT_RT to avoid suppressing valid lockdep warnings. */ static DEFINE_WAIT_OVERRIDE_MAP(kfree_rcu_sheaf_map, LD_WAIT_CONFIG); =20 @@ -6080,10 +6081,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *= obj, unsigned int free_flags) struct slab_sheaf *rcu_sheaf; bool allow_spin =3D free_flags_allow_spinning(free_flags); =20 - if (WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT))) - return false; + VM_WARN_ON_ONCE(IS_ENABLED(CONFIG_PREEMPT_RT) && allow_spin); =20 - lock_map_acquire_try(&kfree_rcu_sheaf_map); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + lock_map_acquire_try(&kfree_rcu_sheaf_map); =20 if (!local_trylock(&s->cpu_sheaves->lock)) goto fail; @@ -6181,12 +6182,14 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *= obj, unsigned int free_flags) local_unlock(&s->cpu_sheaves->lock); =20 stat(s, FREE_RCU_SHEAF); - lock_map_release(&kfree_rcu_sheaf_map); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + lock_map_release(&kfree_rcu_sheaf_map); return true; =20 fail: stat(s, FREE_RCU_SHEAF_FAIL); - lock_map_release(&kfree_rcu_sheaf_map); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + lock_map_release(&kfree_rcu_sheaf_map); return false; } =20 --=20 2.53.0 From nobody Sat Jul 25 02:36:11 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61F4A41A780; Mon, 20 Jul 2026 12:46:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551565; cv=none; b=ErHSj71H2/uYMfEV3NiuOlUWR55cei/UqXWBpT90FXCaGHJ1Dt2tF5E+1t7avl7X/0esGepVrQkYcT6ft2yNg+OXafY8qzhQuEPLDcOmQ5lhGgKKg/S1Nin/HQLEzioY9Pkx8fcqpGhq4eEzizCUhx/0U409W4pfqhhG1XeqZuQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551565; c=relaxed/simple; bh=Va+rLO7sraHhM+J+c5tWzPYqL9emUWtSdO32Hoba948=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=cC/Eu9/yJUejuIXgjZ8d9YXftyLeJN3+ruL5cvvpZajc+dFof6o5WMrljF5WycCDk8+BG2YBGHv38N0vnh0JZ7xPvDcdaW1CoeVKBKa1pNz5E3fiqB989XycCHQ64DcXI93Bf4PvoUPYLmtpoTy+0OFF8J0B3RYeCEsj49AsDb8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=EEWY2lrj; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="EEWY2lrj" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 724DC1F00A3E; Mon, 20 Jul 2026 12:45:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784551564; bh=U2r5dY/zdCbyFplpp0JiAH+XUKzLBzbYJJk8npBgjXI=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=EEWY2lrjXo3Hkbn+9x22TcF418OoNqFOV8MhNW/Adjsg41DzPs4zfk7zQbcepY8Z5 ngFGVemytDm9/OwBDC6YJMXWt6fDzXM4Vq4ko/SKYPdywr/8aZ0KeYgwYXBUTeGsAE aZ+atV6a2ZISJakGiYzAapPRTfhoWER0VND6hhHXy103AByrKZLppXX60wCkKzkQ8m EsyXXkqjXpd0jPI7476cvxFG2r9Ytaz7jDKrDWIN1s2A53VHsfakTdVQxtwPy/QCuD KYRUWZ1WNGHFlgKHLDIPIcFbG8Vy+H3AfWw9M7/Xm+aiSDtwOH9L98sd157mjlgXRI 4boNrLUarPrmQ== From: "Harry Yoo (Oracle)" Date: Mon, 20 Jul 2026 21:44:35 +0900 Subject: [PATCH slab/for-next v4 6/8] mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-kfree_rcu_nolock-v4-6-964e03c41a4e@kernel.org> References: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> In-Reply-To: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> To: Vlastimil Babka , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , Alexei Starovoitov , Andrii Nakryiko , Puranjay Mohan , Amery Hung , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Pedro Falcato , Suren Baghdasaryan , Shengming Hu Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, rcu@vger.kernel.org, bpf@vger.kernel.org, "Harry Yoo (Oracle)" X-Mailer: b4 0.14.3 rcu_head is overkill for kvfree_rcu() because the callback function is always either kfree(), vfree(), or free_large_kmalloc(), and thus there is no need for a function pointer. kvfree_rcu batching reuses the field to store the start address of an object, however, this is not strictly needed because we can calculate the start address in the slowpath. For the purpose of kvfree_rcu batching, it is sufficient to implement a linked list using a single pointer. Introduce a new struct called kvfree_rcu_head (the name was suggested by Vlastimil Babka), which is similar to rcu_head but is only a single pointer to build a linked list, without a function pointer, when CONFIG_KVFREE_RCU_BATCHED=3Dy. When kvfree_rcu is not batched, kvfree_rcu_head is the same size as rcu_head. Note that shrinking struct kvfree_rcu_head on CONFIG_KVFREE_RCU_BATCHED=3Dn kernels would inevitably require additional complexity and also some sort of batching (which defeats the purpose of the config option) because it cannot fall back to call_rcu(). For now there are no user-visible changes to the API. k[v]free_rcu() simply casts rcu_head to kvfree_rcu_head. While this does not affect the API, it allows kfree_rcu_nolock() to reuse kvfree_rcu batching as a fallback when trylock or sheaf allocation fails. Stop storing the object pointer in rcu_head.func and instead calculate the object's start address in kvfree_rcu_list(). Factor out the existing logic to calculate the start address from kvfree_rcu_cb() to kvmalloc_obj_start_addr(). Signed-off-by: Harry Yoo (Oracle) Reviewed-by: Vlastimil Babka (SUSE) Suggested-by: Alexei Starovoitov Suggested-by: Vlastimil Babka (SUSE) --- include/linux/rcupdate.h | 10 ++++++---- include/linux/types.h | 8 ++++++++ include/trace/events/rcu.h | 2 +- mm/slab.h | 28 ++++++++++++++++++++++++++++ mm/slab_common.c | 17 ++++++++--------- mm/slub.c | 40 +++++++++------------------------------- 6 files changed, 60 insertions(+), 45 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 5e95acc33989..ef5bb6981133 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -1098,19 +1098,21 @@ static inline void rcu_read_unlock_migrate(void) /* * In mm/slab_common.c, no suitable header to include here. */ -void kvfree_call_rcu(struct rcu_head *head, void *ptr); +void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr); =20 /* * The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See t= he * comment of kfree_rcu() for details. */ -#define kvfree_rcu_arg_2(ptr, rhf) \ +#define kvfree_rcu_arg_2(ptr, kvrhf) \ do { \ typeof (ptr) ___p =3D (ptr); \ + struct kvfree_rcu_head *___head; \ \ if (___p) { \ - BUILD_BUG_ON(offsetof(typeof(*(ptr)), rhf) >=3D 4096); \ - kvfree_call_rcu(&((___p)->rhf), (void *) (___p)); \ + BUILD_BUG_ON(offsetof(typeof(*(ptr)), kvrhf) >=3D 4096); \ + ___head =3D (struct kvfree_rcu_head *) &(___p)->kvrhf; \ + kvfree_call_rcu(___head, (void *) (___p)); \ } \ } while (0) =20 diff --git a/include/linux/types.h b/include/linux/types.h index 93166b0b0617..8b0976669d66 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -255,6 +255,14 @@ struct callback_head { } __attribute__((aligned(sizeof(void *)))); #define rcu_head callback_head =20 +#ifdef CONFIG_KVFREE_RCU_BATCHED +struct kvfree_rcu_head { + struct kvfree_rcu_head *next; +}; +#else +#define kvfree_rcu_head rcu_head +#endif + typedef void (*rcu_callback_t)(struct rcu_head *head); typedef void (*call_rcu_func_t)(struct rcu_head *head, rcu_callback_t func= ); =20 diff --git a/include/trace/events/rcu.h b/include/trace/events/rcu.h index 5fbdabe3faea..a74027126a2f 100644 --- a/include/trace/events/rcu.h +++ b/include/trace/events/rcu.h @@ -625,7 +625,7 @@ TRACE_EVENT_RCU(rcu_invoke_callback, */ TRACE_EVENT_RCU(rcu_invoke_kvfree_callback, =20 - TP_PROTO(const char *rcuname, struct rcu_head *rhp, unsigned long offset), + TP_PROTO(const char *rcuname, struct kvfree_rcu_head *rhp, unsigned long = offset), =20 TP_ARGS(rcuname, rhp, offset), =20 diff --git a/mm/slab.h b/mm/slab.h index fdd293b3efa5..1efeaceb7db3 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -351,6 +351,33 @@ static inline int objs_per_slab(const struct kmem_cach= e *cache, return slab->objects; } =20 +/* kvfree_rcu_head offset can be only less than page size */ +static inline void *kvmalloc_obj_start_addr(void *head) +{ + void *obj =3D head; + + if (unlikely(is_vmalloc_addr(obj))) { + obj =3D (void *) PAGE_ALIGN_DOWN((unsigned long)obj); + } else { + struct page *page =3D virt_to_page(obj); + struct slab *slab =3D page_slab(page); + + if (!slab) { + obj =3D (void *) PAGE_ALIGN_DOWN((unsigned long)obj); + } else if (is_kfence_address(obj)) { + obj =3D kfence_object_start(obj); + } else { + struct kmem_cache *s =3D slab->slab_cache; + unsigned int idx =3D __obj_to_index(s, slab_address(slab), obj); + + obj =3D slab_address(slab) + s->size * idx; + obj =3D fixup_red_left(s, obj); + } + } + + return obj; +} + /* * State of the slab allocator. * @@ -761,6 +788,7 @@ void __check_heap_object(const void *ptr, unsigned long= n, const struct slab *slab, bool to_user); =20 void deferred_work_barrier(void); +void defer_kfree_rcu(struct kvfree_rcu_head *head); =20 static inline bool slub_debug_orig_size(struct kmem_cache *s) { diff --git a/mm/slab_common.c b/mm/slab_common.c index 8c631bf97cd5..c1c32909fa52 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1346,7 +1346,7 @@ struct kvfree_rcu_bulk_data { =20 struct kfree_rcu_cpu_work { struct rcu_work rcu_work; - struct rcu_head *head_free; + struct kvfree_rcu_head *head_free; struct rcu_gp_oldstate head_free_gp_snap; struct list_head bulk_head_free[FREE_N_CHANNELS]; struct kfree_rcu_cpu *krcp; @@ -1382,7 +1382,7 @@ struct kfree_rcu_cpu_work { struct kfree_rcu_cpu { // Objects queued on a linked list // through their rcu_head structures. - struct rcu_head *head; + struct kvfree_rcu_head *head; unsigned long head_gp_snap; atomic_t head_count; =20 @@ -1523,12 +1523,12 @@ kvfree_rcu_bulk(struct kfree_rcu_cpu *krcp, } =20 static void -kvfree_rcu_list(struct rcu_head *head) +kvfree_rcu_list(struct kvfree_rcu_head *head) { - struct rcu_head *next; + struct kvfree_rcu_head *next; =20 for (; head; head =3D next) { - void *ptr =3D (void *) head->func; + void *ptr =3D kvmalloc_obj_start_addr(head); unsigned long offset =3D (void *) head - ptr; =20 next =3D head->next; @@ -1552,7 +1552,7 @@ static void kfree_rcu_work(struct work_struct *work) unsigned long flags; struct kvfree_rcu_bulk_data *bnode, *n; struct list_head bulk_head[FREE_N_CHANNELS]; - struct rcu_head *head; + struct kvfree_rcu_head *head; struct kfree_rcu_cpu *krcp; struct kfree_rcu_cpu_work *krwp; struct rcu_gp_oldstate head_gp_snap; @@ -1683,7 +1683,7 @@ kvfree_rcu_drain_ready(struct kfree_rcu_cpu *krcp) { struct list_head bulk_ready[FREE_N_CHANNELS]; struct kvfree_rcu_bulk_data *bnode, *n; - struct rcu_head *head_ready =3D NULL; + struct kvfree_rcu_head *head_ready =3D NULL; unsigned long flags; int i; =20 @@ -1946,7 +1946,7 @@ void __init kfree_rcu_scheduler_running(void) * be free'd in workqueue context. This allows us to: batch requests toget= her to * reduce the number of grace periods during heavy kfree_rcu()/kvfree_rcu(= ) load. */ -void kvfree_call_rcu(struct rcu_head *head, void *ptr) +void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr) { unsigned long flags; struct kfree_rcu_cpu *krcp; @@ -1984,7 +1984,6 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr) // Inline if kvfree_rcu(one_arg) call. goto unlock_return; =20 - head->func =3D ptr; head->next =3D krcp->head; WRITE_ONCE(krcp->head, head); atomic_inc(&krcp->head_count); diff --git a/mm/slub.c b/mm/slub.c index deac315d0f23..d0976396d703 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -6699,43 +6699,21 @@ static void free_large_kmalloc(struct page *page, v= oid *object) */ void kvfree_rcu_cb(struct rcu_head *head) { - void *obj =3D head; - struct page *page; - struct slab *slab; - struct kmem_cache *s; - void *slab_addr; + void *obj; + + obj =3D kvmalloc_obj_start_addr(head); =20 if (is_vmalloc_addr(obj)) { - obj =3D (void *) PAGE_ALIGN_DOWN((unsigned long)obj); vfree(obj); - return; - } - - page =3D virt_to_page(obj); - slab =3D page_slab(page); - if (!slab) { - /* - * rcu_head offset can be only less than page size so no need to - * consider allocation order - */ - obj =3D (void *) PAGE_ALIGN_DOWN((unsigned long)obj); - free_large_kmalloc(page, obj); - return; - } - - s =3D slab->slab_cache; - slab_addr =3D slab_address(slab); - - if (is_kfence_address(obj)) { - obj =3D kfence_object_start(obj); } else { - unsigned int idx =3D __obj_to_index(s, slab_addr, obj); + struct page *page =3D virt_to_page(obj); + struct slab *slab =3D page_slab(page); =20 - obj =3D slab_addr + s->size * idx; - obj =3D fixup_red_left(s, obj); + if (slab) + slab_free(slab->slab_cache, slab, obj, _RET_IP_); + else + free_large_kmalloc(page, obj); } - - slab_free(s, slab, obj, _RET_IP_); } =20 /** --=20 2.53.0 From nobody Sat Jul 25 02:36:11 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79E8E4195C3; Mon, 20 Jul 2026 12:46:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551571; cv=none; b=Zju4FOPQuPO8lAjfGUszzZFCZhchYkWJxTlBH61QliphHXF/ehud9xXR8KjuUHuZBc9qjU0hB8SZrBorJAl43Ocwu6IeVyVNreXNW+tSUhvGdRYBKcy8IfLVW1OZKtpNsXGoVjIa85/2boj8BweSSPvHTTM4wOSLTe6sUqydDrE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551571; c=relaxed/simple; bh=Td9yzK+lW3SpaynxjYepq1jC6b/ZmqRlIDj4vYJb8pc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Xslr+hafU1rg1Cetr1zEembw99bsKeKpO9p/mIeJitnUsZUoyJZU29DGcJe9kPmJL9kBl/IHrZMfn7JEtx/Zd9Ij4tp7bU9ASdvQSYHzHdXn9+mZlPElJkTfgvYsD9uXsKj8XsduMq7YBamXA+zTNmCBgkZxfkzN2/AHPKkncrQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RX13nvMZ; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RX13nvMZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6A73C1F000E9; Mon, 20 Jul 2026 12:46:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784551570; bh=tR+P/h2nByeigiwGZHMwdsEAW1PKLC/u8oNgU53PQWs=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=RX13nvMZggBA2+prEs/70znjmQ6xnGVlj01nl2N3late5GXTZ5ldIXKVfpeO6FGOE uq6jE4Bg7233yFmRNr5Ym/MJa6QAdPWAdEZjhnXlyPZdkCs4lvAA5gAD23Z0RnM6/m Nlz8B5WZYRksPbohtdp2ggK6YFe047r9lQmhgkI4eN3xS/o90+1/2tvU/Y7gw5AuUO ZhBgS6EfD+AmWhLkvMDkDJJqnJB7WkKhoDv5sRz4JEg7AeP6AdUdYCtgXynth9HjFw XwJQcb9Gr2ThOYi4o5aQDyO7CfHbABRCDCMUhkifAUdxdiTCcQuJ4zi6m/ruUZ2Egj OLZYyrMO5zOzA== From: "Harry Yoo (Oracle)" Date: Mon, 20 Jul 2026 21:44:36 +0900 Subject: [PATCH slab/for-next v4 7/8] mm/slab: introduce kfree_rcu_nolock() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-kfree_rcu_nolock-v4-7-964e03c41a4e@kernel.org> References: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> In-Reply-To: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> To: Vlastimil Babka , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , Alexei Starovoitov , Andrii Nakryiko , Puranjay Mohan , Amery Hung , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Pedro Falcato , Suren Baghdasaryan , Shengming Hu Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, rcu@vger.kernel.org, bpf@vger.kernel.org, "Harry Yoo (Oracle)" X-Mailer: b4 0.14.3 Currently, k[v]free_rcu() cannot be called in unknown context since it could lead to a deadlock when called in the middle of k[v]free_rcu(). Make users' lives easier by introducing kfree_rcu_nolock() variant, now that kfree_rcu_sheaf() is available on PREEMPT_RT and __kfree_rcu_sheaf() handles unknown context. kfree_rcu_nolock() falls back to the kvfree_rcu batching when sheaves path fails. In most cases, the sheaves path is expected to succeed and it's unnecessary to add complexity to the existing kvfree_rcu batching by teaching it how to handle unknown context. Since defer_kfree_rcu() can be called on caches without sheaves, move deferred_work_barrier() and rcu_barrier() outside the branch in kvfree_rcu_barrier_on_cache(). Signed-off-by: Harry Yoo (Oracle) Reviewed-by: Vlastimil Babka (SUSE) Suggested-by: Alexei Starovoitov Suggested-by: Vlastimil Babka (SUSE) --- include/linux/rcupdate.h | 19 +++++++++++++++++++ mm/slab_common.c | 28 ++++++++++++++++++++++++++-- mm/slub.c | 24 +++++++++++++++++++++++- 3 files changed, 68 insertions(+), 3 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index ef5bb6981133..ce0fd401352e 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -1099,6 +1099,7 @@ static inline void rcu_read_unlock_migrate(void) * In mm/slab_common.c, no suitable header to include here. */ void kvfree_call_rcu(struct kvfree_rcu_head *head, void *ptr); +void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr); =20 /* * The BUILD_BUG_ON() makes sure the rcu_head offset can be handled. See t= he @@ -1124,6 +1125,24 @@ do { \ kvfree_call_rcu(NULL, (void *) (___p)); \ } while (0) =20 +/** + * kfree_rcu_nolock() - a version of kfree_rcu() that can be called in any= context. + * @ptr: pointer to kfree for double-argument invocations. + * @kvrhf: the name of the struct kvfree_rcu_head within the type of @ptr. + * + * Objects that are not allocated via kmalloc_nolock() are not supported. + * kfree_rcu_nolock() supports 2-arg variant only. + */ +#define kfree_rcu_nolock(ptr, kvrhf) \ +do { \ + typeof (ptr) ___p =3D (ptr); \ + \ + if (___p) { \ + BUILD_BUG_ON(offsetof(typeof(*(ptr)), kvrhf) >=3D 4096); \ + kfree_call_rcu_nolock(&((___p)->kvrhf), (void *) (___p)); \ + } \ +} while (0) + /* * Place this after a lock-acquisition primitive to guarantee that * an UNLOCK+LOCK pair acts as a full barrier. This guarantee applies diff --git a/mm/slab_common.c b/mm/slab_common.c index c1c32909fa52..09c457645e03 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1263,6 +1263,29 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc); EXPORT_TRACEPOINT_SYMBOL(kfree); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free); =20 +void kfree_call_rcu_nolock(struct kvfree_rcu_head *head, void *ptr) +{ + struct slab *slab; + struct kmem_cache *s; + + VM_WARN_ON_ONCE(is_vmalloc_addr(ptr) || !virt_to_slab(ptr)); + + slab =3D virt_to_slab(ptr); + s =3D slab->slab_cache; + + if (unlikely(IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) !=3D numa_mem_id()= )) + goto fallback; + + if (unlikely(!__kfree_rcu_sheaf(s, ptr, SLAB_FREE_NOLOCK))) + goto fallback; + + return; + +fallback: + defer_kfree_rcu(head); +} +EXPORT_SYMBOL_GPL(kfree_call_rcu_nolock); + #ifndef CONFIG_KVFREE_RCU_BATCHED =20 void kvfree_call_rcu(struct rcu_head *head, void *ptr) @@ -2120,10 +2143,11 @@ void kvfree_rcu_barrier_on_cache(struct kmem_cache = *s) cpus_read_lock(); flush_rcu_sheaves_on_cache(s); cpus_read_unlock(); - deferred_work_barrier(); - rcu_barrier(); } =20 + /* kfree_rcu_nolock() might have deferred frees even without sheaves */ + deferred_work_barrier(); + rcu_barrier(); __kvfree_rcu_barrier(); } EXPORT_SYMBOL_GPL(kvfree_rcu_barrier_on_cache); diff --git a/mm/slub.c b/mm/slub.c index d0976396d703..7114451dd5d0 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4081,6 +4081,7 @@ static void flush_all(struct kmem_cache *s) =20 struct deferred_percpu_work { struct llist_head objects; + struct llist_head objects_by_rcu; struct llist_head rcu_sheaves; struct irq_work work; }; @@ -4089,6 +4090,7 @@ static void deferred_percpu_work_fn(struct irq_work *= work); =20 static DEFINE_PER_CPU(struct deferred_percpu_work, deferred_percpu_work) = =3D { .objects =3D LLIST_HEAD_INIT(objects), + .objects_by_rcu =3D LLIST_HEAD_INIT(objects_by_rcu), .rcu_sheaves =3D LLIST_HEAD_INIT(rcu_sheaves), .work =3D IRQ_WORK_INIT(deferred_percpu_work_fn), }; @@ -6386,13 +6388,14 @@ static void free_to_pcs_bulk(struct kmem_cache *s, = size_t size, void **p) static void deferred_percpu_work_fn(struct irq_work *work) { struct deferred_percpu_work *dpw; - struct llist_head *objs, *rcu_sheaves; + struct llist_head *objs, *objs_by_rcu, *rcu_sheaves; struct llist_node *llnode, *pos, *t; struct slab_sheaf *sheaf, *next; =20 dpw =3D container_of(work, struct deferred_percpu_work, work); rcu_sheaves =3D &dpw->rcu_sheaves; objs =3D &dpw->objects; + objs_by_rcu =3D &dpw->objects_by_rcu; =20 llnode =3D llist_del_all(objs); llist_for_each_safe(pos, t, llnode) { @@ -6417,6 +6420,14 @@ static void deferred_percpu_work_fn(struct irq_work = *work) stat(s, FREE_SLOWPATH); } =20 + llnode =3D llist_del_all(objs_by_rcu); + llist_for_each_safe(pos, t, llnode) { + void *head =3D pos; + void *objp =3D kvmalloc_obj_start_addr(head); + + kvfree_call_rcu(head, objp); + } + llnode =3D llist_del_all(rcu_sheaves); llist_for_each_entry_safe(sheaf, next, llnode, llnode) call_rcu(&sheaf->rcu_head, rcu_free_sheaf); @@ -6435,6 +6446,17 @@ static void defer_free(struct kmem_cache *s, void *h= ead) irq_work_queue(&dpw->work); } =20 +void defer_kfree_rcu(struct kvfree_rcu_head *head) +{ + struct deferred_percpu_work *dpw; + + guard(preempt)(); + + dpw =3D this_cpu_ptr(&deferred_percpu_work); + if (llist_add((struct llist_node *)head, &dpw->objects_by_rcu)) + irq_work_queue(&dpw->work); +} + void deferred_work_barrier(void) { int cpu; --=20 2.53.0 From nobody Sat Jul 25 02:36:11 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4106141737D; Mon, 20 Jul 2026 12:46:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551577; cv=none; b=Nbr4B2bukTQZxeB1S5t7RaHQSmdStufBp7dF1gBpWh10ZJxcY3TWv+lUnGEVcfzP4f73NMQX2uILSCKxMSapFT3W57SrzGKqfjm67/0hrI66x7V6B1nPTXuPxrdpZq9nYMLYlaqV4//sz97eILTztMzUg0PzeQaPEMoiqeu+p8k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784551577; c=relaxed/simple; bh=rTl/DxJagCa+9Hy+ONCJKy/uvlP3BJsWT2MiI42DyvM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=lNVqYu6haL+DqKKqXPyKJKSbo8XsS+IAYe84kL7pYATg2DmB7ncPxXap10p9XG+paCa1T5ZO1ipB8CLlYK3QVVB2p5qjAFmX4dv6fXTPeh60HSVfm+3pN6Ak0PDIkf/4lT6a87Bpsk7y+mj1I3c5HB0yHtA6Jtufd6y/l8CjSCs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HDvhW/pe; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HDvhW/pe" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 87BE51F00A3D; Mon, 20 Jul 2026 12:46:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784551576; bh=8Mf6ICl8QV0PyvQov1wiSUifOBTOQefLFAPMuDN3x10=; h=From:Date:Subject:References:In-Reply-To:To:Cc; b=HDvhW/peMepVeAoQL2dW+MaaHo1MGlQYb4TFy3R2tW14fKkWa7VZuh/s7cn77dzVq sSUYzoDD0SiPfWqY6iHqc1QPDZJF4mfo7RDsW0xED6Ocm1iSrM006ijnC32MukKMJg +M4A9DHP5vFdtw8YzVwC+FgASQzKKZ/7g8LmG1g7O925c6BlyXO/vrp/UVfT2tnXnB f88Yo0i+2f8bKWy6EHEkO8A/eC2WpRLdVtXDEkY4qe1PwbwR5P2G2Y25t+7v1ZHSRf BgXSa4nahdSdUJJPlQDjNhmC9dsQ5Q1yMC0NEV9C+gyKdIxG/GsiVQFZ1uaKW0ckTc vj+VGSJbrqYpA== From: "Harry Yoo (Oracle)" Date: Mon, 20 Jul 2026 21:44:37 +0900 Subject: [PATCH slab/for-next v4 8/8] slub_kunit: extend the test for kfree_rcu_nolock() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-kfree_rcu_nolock-v4-8-964e03c41a4e@kernel.org> References: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> In-Reply-To: <20260720-kfree_rcu_nolock-v4-0-964e03c41a4e@kernel.org> To: Vlastimil Babka , Andrew Morton , Hao Li , Christoph Lameter , David Rientjes , Roman Gushchin , Alexei Starovoitov , Andrii Nakryiko , Puranjay Mohan , Amery Hung , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Pedro Falcato , Suren Baghdasaryan , Shengming Hu Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev, rcu@vger.kernel.org, bpf@vger.kernel.org, "Harry Yoo (Oracle)" X-Mailer: b4 0.14.3 When slub_kunit is not built-in, call kfree_rcu() and kfree_rcu_nolock() to test kfree_rcu_nolock() in slub_kunit. Rename the test case as the test covers more _nolock() APIs. Acked-by: Vlastimil Babka (SUSE) Signed-off-by: Harry Yoo (Oracle) Suggested-by: Alexei Starovoitov Suggested-by: Vlastimil Babka (SUSE) --- lib/tests/slub_kunit.c | 45 +++++++++++++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 16 deletions(-) diff --git a/lib/tests/slub_kunit.c b/lib/tests/slub_kunit.c index a391467c1aa6..e3b63f0338d5 100644 --- a/lib/tests/slub_kunit.c +++ b/lib/tests/slub_kunit.c @@ -162,7 +162,10 @@ static void test_kmalloc_redzone_access(struct kunit *= test) } =20 struct test_kfree_rcu_struct { - struct rcu_head rcu; + union { + struct rcu_head rcu; + struct kvfree_rcu_head kvrcu; + }; }; =20 static void test_kfree_rcu(struct kunit *test) @@ -296,7 +299,7 @@ static void test_krealloc_redzone_zeroing(struct kunit = *test) #if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CON= FIG_SMP)) #define NR_ITERATIONS 1000 #define NR_OBJECTS 1000 -static void *objects[NR_OBJECTS]; +static struct test_kfree_rcu_struct *objects[NR_OBJECTS]; =20 struct test_nolock_context { struct kunit *test; @@ -311,13 +314,14 @@ struct test_nolock_context { #endif }; =20 -static void test_kmalloc_kfree(void) +static void test_kmalloc_and_friends(void) { int i, j; + bool can_use_kfree_rcu =3D !IS_BUILTIN(CONFIG_SLUB_KUNIT_TEST); =20 for (i =3D 0; i < NR_ITERATIONS; i++) { for (j =3D 0; j < NR_OBJECTS; j++) { - gfp_t gfp =3D (i % 2) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT; + gfp_t gfp =3D (i & 1) ? GFP_KERNEL : GFP_KERNEL_ACCOUNT; =20 objects[j] =3D kmalloc_obj(*objects[j], gfp); if (!objects[j]) { @@ -328,26 +332,35 @@ static void test_kmalloc_kfree(void) } } =20 - for (j =3D 0; j < NR_OBJECTS; j++) - kfree(objects[j]); + for (j =3D 0; j < NR_OBJECTS; j++) { + if (can_use_kfree_rcu && (i & 2)) + kfree_rcu(objects[j], rcu); + else + kfree(objects[j]); + } } } =20 static void test_nolock(struct test_nolock_context *ctx) { - void *objp; + struct test_kfree_rcu_struct *objp; gfp_t gfp; + bool can_use_kfree_rcu =3D !IS_BUILTIN(CONFIG_SLUB_KUNIT_TEST); =20 /* __GFP_ACCOUNT to test kmalloc_nolock() in alloc_slab_obj_exts() */ - gfp =3D (ctx->callback_count % 2) ? 0 : __GFP_ACCOUNT; - objp =3D kmalloc_nolock(64, gfp, NUMA_NO_NODE); + gfp =3D (ctx->callback_count & 1) ? 0 : __GFP_ACCOUNT; + objp =3D kmalloc_nolock(sizeof(*objp), gfp, NUMA_NO_NODE); =20 if (objp) ctx->alloc_ok++; else ctx->alloc_fail++; =20 - kfree_nolock(objp); + if (can_use_kfree_rcu && (ctx->callback_count & 2)) + kfree_rcu_nolock(objp, kvrcu); + else + kfree_nolock(objp); + ctx->callback_count++; } #endif @@ -397,14 +410,14 @@ static void disable_perf_events(struct test_nolock_co= ntext *ctx) perf_event_release_kernel(ctx->event); } =20 -static void test_kmalloc_kfree_nolock_perf(struct kunit *test) +static void test_kmalloc_nolock_and_friends_perf(struct kunit *test) { struct test_nolock_context ctx =3D { .test =3D test }; =20 if (!enable_perf_events(&ctx)) kunit_skip(test, "Failed to enable perf event, skipping"); =20 - test_kmalloc_kfree(); + test_kmalloc_and_friends(); =20 disable_perf_events(&ctx); KUNIT_EXPECT_EQ(test, 0, slab_errors); @@ -438,14 +451,14 @@ static void unregister_slab_kprobes(struct test_noloc= k_context *ctx) unregister_kprobe(&ctx->kprobe); } =20 -static void test_kmalloc_kfree_nolock_kprobe(struct kunit *test) +static void test_kmalloc_nolock_and_friends_kprobe(struct kunit *test) { struct test_nolock_context ctx =3D { .test =3D test }; =20 if (!register_slab_kprobes(&ctx)) kunit_skip(test, "Failed to register kprobe, skipping"); =20 - test_kmalloc_kfree(); + test_kmalloc_and_friends(); =20 unregister_slab_kprobes(&ctx); KUNIT_EXPECT_EQ(test, 0, slab_errors); @@ -477,10 +490,10 @@ static struct kunit_case test_cases[] =3D { KUNIT_CASE(test_leak_destroy), KUNIT_CASE(test_krealloc_redzone_zeroing), #ifdef CONFIG_PERF_EVENTS - KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_perf), + KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf), #endif #if defined(CONFIG_KPROBES) && defined(CONFIG_SMP) - KUNIT_CASE_SLOW(test_kmalloc_kfree_nolock_kprobe), + KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_kprobe), #endif {} }; --=20 2.53.0