From nobody Tue Feb 10 14:43:57 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 434FB1F5842; Mon, 29 Dec 2025 19:11:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; cv=none; b=h81eeKbZWLQS1BjxYLqxnIK9Sy1EB9VrrHNgXBdZ82Dfwm8eXDYSpNZJjNu0ZvWyHwk/H34OWg6cCOQjEJjRAhxPZWVRGAB1YA6YZBmGjwcuBOpbhNCbOtf5/Pya0ral1WiOMCr2Ylt236VmS+6Qy0/6VbyG1ZeUJQCXz2TR7dU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; c=relaxed/simple; bh=B9PEC0JrYy+6c6bPjig947rtmpAQCJFpOF8b/Oky+bs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=c8wQrJEYgaq455XyaC0q/7Hy3Rr7AfLiGOJeN5eqZsHLeYDpRr7OP6NK/AgypSzzqP/J9/zhzgZ8C4Hd4lrXa7kfN3YhF68dKMMGftIc/4YSYOxgOnWBS5zP8hSPjIKvj5beJuGLjSd+XpdWg7OVU43FJGoXzWur89d+tg9vuDI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DnppIv7h; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DnppIv7h" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B2586C4CEF7; Mon, 29 Dec 2025 19:11:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1767035466; bh=B9PEC0JrYy+6c6bPjig947rtmpAQCJFpOF8b/Oky+bs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DnppIv7hY9by29ceeSrlX43MnjgAPmbCiUkPlowDkeUCe0sXVJ6j6PEi1vNmt21/1 UU9JjI8opMDmAOA8HOOa/Ap1RmjN1YjgGqVR/gfLiE+CdTsWDHdmLJ+S1CCfz7bK1O eVFttIRcoTx6fcSn7d/2wgj1Ie4hiMDb0P1P9djEAbP0xl3dd7Wah/Tdlsl3m68fpt WfpDr5laevvxA9SRGZ91uPmSN7reK4dSk65BCajM/TODmsmMzyXev8T7vtFVtH9kcS FvWXC0BJ2WRbqqdAgGUVnK4vFh5eBLO7hRn2JNvZ/SPfiNRcV6m7orsGce03AOQECR OZBadJcVZRfJw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 6B9C2CE0C97; Mon, 29 Dec 2025 11:11:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , kernel test robot , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v4 1/9] rcu: Re-implement RCU Tasks Trace in terms of SRCU-fast Date: Mon, 29 Dec 2025 11:10:56 -0800 Message-Id: <20251229191104.693447-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit saves more than 500 lines of RCU code by re-implementing RCU Tasks Trace in terms of SRCU-fast. Follow-up work will remove more code that does not cause problems by its presence, but that is no longer required. This variant places smp_mb() in rcu_read_{,un}lock_trace(), and in the same place that srcu_read_{,un}lock() would put them. These smp_mb() calls will be removed on common-case architectures in a later commit. In the meantime, it serves to enforce ordering between the underlying srcu_read_{,un}lock_fast() markers and the intervening critical section, even on architectures that permit attaching tracepoints on regions of code not watched by RCU. Such architectures defeat SRCU-fast's use of implicit single-instruction, interrupts-disabled, and atomic-operation RCU read-side critical sections, which have no effect when RCU is not watching. The aforementioned later commit will insert these smp_mb() calls only on architectures that have not used noinstr to prevent attaching tracepoints to code where RCU is not watching. [ paulmck: Apply kernel test robot, Boqun Feng, and Zqiang feedback. ] [ paulmck: Split out Tiny SRCU fixes per Andrii Nakryiko feedback. ] Signed-off-by: Paul E. McKenney Tested-by: kernel test robot Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Reviewed-by: Joel Fernandes --- include/linux/rcupdate_trace.h | 107 ++++-- include/linux/sched.h | 1 + kernel/rcu/tasks.h | 621 +-------------------------------- 3 files changed, 99 insertions(+), 630 deletions(-) diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index e6c44eb428ab6..3f46cbe670003 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -12,28 +12,28 @@ #include #include =20 -extern struct lockdep_map rcu_trace_lock_map; +#ifdef CONFIG_TASKS_TRACE_RCU +extern struct srcu_struct rcu_tasks_trace_srcu_struct; +#endif // #ifdef CONFIG_TASKS_TRACE_RCU =20 -#ifdef CONFIG_DEBUG_LOCK_ALLOC +#if defined(CONFIG_DEBUG_LOCK_ALLOC) && defined(CONFIG_TASKS_TRACE_RCU) =20 static inline int rcu_read_lock_trace_held(void) { - return lock_is_held(&rcu_trace_lock_map); + return srcu_read_lock_held(&rcu_tasks_trace_srcu_struct); } =20 -#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ +#else // #if defined(CONFIG_DEBUG_LOCK_ALLOC) && defined(CONFIG_TASKS_TRAC= E_RCU) =20 static inline int rcu_read_lock_trace_held(void) { return 1; } =20 -#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ +#endif // #else // #if defined(CONFIG_DEBUG_LOCK_ALLOC) && defined(CONFIG_= TASKS_TRACE_RCU) =20 #ifdef CONFIG_TASKS_TRACE_RCU =20 -void rcu_read_unlock_trace_special(struct task_struct *t); - /** * rcu_read_lock_trace - mark beginning of RCU-trace read-side critical se= ction * @@ -50,12 +50,14 @@ static inline void rcu_read_lock_trace(void) { struct task_struct *t =3D current; =20 - WRITE_ONCE(t->trc_reader_nesting, READ_ONCE(t->trc_reader_nesting) + 1); - barrier(); - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) && - t->trc_reader_special.b.need_mb) - smp_mb(); // Pairs with update-side barriers - rcu_lock_acquire(&rcu_trace_lock_map); + if (t->trc_reader_nesting++) { + // In case we interrupted a Tasks Trace RCU reader. + rcu_try_lock_acquire(&rcu_tasks_trace_srcu_struct.dep_map); + return; + } + barrier(); // nesting before scp to protect against interrupt handler. + t->trc_reader_scp =3D srcu_read_lock_fast(&rcu_tasks_trace_srcu_struct); + smp_mb(); // Placeholder for more selective ordering } =20 /** @@ -69,26 +71,75 @@ static inline void rcu_read_lock_trace(void) */ static inline void rcu_read_unlock_trace(void) { - int nesting; + struct srcu_ctr __percpu *scp; struct task_struct *t =3D current; =20 - rcu_lock_release(&rcu_trace_lock_map); - nesting =3D READ_ONCE(t->trc_reader_nesting) - 1; - barrier(); // Critical section before disabling. - // Disable IPI-based setting of .need_qs. - WRITE_ONCE(t->trc_reader_nesting, INT_MIN + nesting); - if (likely(!READ_ONCE(t->trc_reader_special.s)) || nesting) { - WRITE_ONCE(t->trc_reader_nesting, nesting); - return; // We assume shallow reader nesting. - } - WARN_ON_ONCE(nesting !=3D 0); - rcu_read_unlock_trace_special(t); + smp_mb(); // Placeholder for more selective ordering + scp =3D t->trc_reader_scp; + barrier(); // scp before nesting to protect against interrupt handler. + if (!--t->trc_reader_nesting) + srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp); + else + srcu_lock_release(&rcu_tasks_trace_srcu_struct.dep_map); +} + +/** + * call_rcu_tasks_trace() - Queue a callback trace task-based grace period + * @rhp: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a trace rcu-tasks + * grace period elapses, in other words after all currently executing + * trace rcu-tasks read-side critical sections have completed. These + * read-side critical sections are delimited by calls to rcu_read_lock_tra= ce() + * and rcu_read_unlock_trace(). + * + * See the description of call_rcu() for more detailed information on + * memory ordering guarantees. + */ +static inline void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback= _t func) +{ + call_srcu(&rcu_tasks_trace_srcu_struct, rhp, func); +} + +/** + * synchronize_rcu_tasks_trace - wait for a trace rcu-tasks grace period + * + * Control will return to the caller some time after a trace rcu-tasks + * grace period has elapsed, in other words after all currently executing + * trace rcu-tasks read-side critical sections have elapsed. These read-si= de + * critical sections are delimited by calls to rcu_read_lock_trace() + * and rcu_read_unlock_trace(). + * + * This is a very specialized primitive, intended only for a few uses in + * tracing and other situations requiring manipulation of function preambl= es + * and profiling hooks. The synchronize_rcu_tasks_trace() function is not + * (yet) intended for heavy use from multiple CPUs. + * + * See the description of synchronize_rcu() for more detailed information + * on memory ordering guarantees. + */ +static inline void synchronize_rcu_tasks_trace(void) +{ + synchronize_srcu(&rcu_tasks_trace_srcu_struct); } =20 -void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func); -void synchronize_rcu_tasks_trace(void); -void rcu_barrier_tasks_trace(void); +/** + * rcu_barrier_tasks_trace - Wait for in-flight call_rcu_tasks_trace() cal= lbacks. + * + * Note that rcu_barrier_tasks_trace() is not obligated to actually wait, + * for example, if there are no pending callbacks. + */ +static inline void rcu_barrier_tasks_trace(void) +{ + srcu_barrier(&rcu_tasks_trace_srcu_struct); +} + +// Placeholders to enable stepwise transition. +void rcu_tasks_trace_get_gp_data(int *flags, unsigned long *gp_seq); +void __init rcu_tasks_trace_suppress_unused(void); struct task_struct *get_rcu_tasks_trace_gp_kthread(void); + #else /* * The BPF JIT forms these addresses even when it doesn't call these diff --git a/include/linux/sched.h b/include/linux/sched.h index d395f2810facb..fe39d422b37d7 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -945,6 +945,7 @@ struct task_struct { =20 #ifdef CONFIG_TASKS_TRACE_RCU int trc_reader_nesting; + struct srcu_ctr __percpu *trc_reader_scp; int trc_ipi_to_cpu; union rcu_special trc_reader_special; struct list_head trc_holdout_list; diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 2dc044fd126eb..1fe789c99f361 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -718,7 +718,6 @@ static void __init rcu_tasks_bootup_oddness(void) #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ } =20 - /* Dump out rcutorture-relevant state common to all RCU-tasks flavors. */ static void show_rcu_tasks_generic_gp_kthread(struct rcu_tasks *rtp, char = *s) { @@ -803,7 +802,7 @@ static void rcu_tasks_torture_stats_print_generic(struc= t rcu_tasks *rtp, char *t =20 static void exit_tasks_rcu_finish_trace(struct task_struct *t); =20 -#if defined(CONFIG_TASKS_RCU) || defined(CONFIG_TASKS_TRACE_RCU) +#if defined(CONFIG_TASKS_RCU) =20 //////////////////////////////////////////////////////////////////////// // @@ -898,7 +897,7 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) rtp->postgp_func(rtp); } =20 -#endif /* #if defined(CONFIG_TASKS_RCU) || defined(CONFIG_TASKS_TRACE_RCU)= */ +#endif /* #if defined(CONFIG_TASKS_RCU) */ =20 #ifdef CONFIG_TASKS_RCU =20 @@ -1453,94 +1452,27 @@ EXPORT_SYMBOL_GPL(rcu_tasks_rude_get_gp_data); // // Tracing variant of Tasks RCU. This variant is designed to be used // to protect tracing hooks, including those of BPF. This variant -// therefore: -// -// 1. Has explicit read-side markers to allow finite grace periods -// in the face of in-kernel loops for PREEMPT=3Dn builds. -// -// 2. Protects code in the idle loop, exception entry/exit, and -// CPU-hotplug code paths, similar to the capabilities of SRCU. -// -// 3. Avoids expensive read-side instructions, having overhead similar -// to that of Preemptible RCU. -// -// There are of course downsides. For example, the grace-period code -// can send IPIs to CPUs, even when those CPUs are in the idle loop or -// in nohz_full userspace. If needed, these downsides can be at least -// partially remedied. -// -// Perhaps most important, this variant of RCU does not affect the vanilla -// flavors, rcu_preempt and rcu_sched. The fact that RCU Tasks Trace -// readers can operate from idle, offline, and exception entry/exit in no -// way allows rcu_preempt and rcu_sched readers to also do so. -// -// The implementation uses rcu_tasks_wait_gp(), which relies on function -// pointers in the rcu_tasks structure. The rcu_spawn_tasks_trace_kthread= () -// function sets these function pointers up so that rcu_tasks_wait_gp() -// invokes these functions in this order: -// -// rcu_tasks_trace_pregp_step(): -// Disables CPU hotplug, adds all currently executing tasks to the -// holdout list, then checks the state of all tasks that blocked -// or were preempted within their current RCU Tasks Trace read-side -// critical section, adding them to the holdout list if appropriate. -// Finally, this function re-enables CPU hotplug. -// The ->pertask_func() pointer is NULL, so there is no per-task processin= g. -// rcu_tasks_trace_postscan(): -// Invokes synchronize_rcu() to wait for late-stage exiting tasks -// to finish exiting. -// check_all_holdout_tasks_trace(), repeatedly until holdout list is empty: -// Scans the holdout list, attempting to identify a quiescent state -// for each task on the list. If there is a quiescent state, the -// corresponding task is removed from the holdout list. Once this -// list is empty, the grace period has completed. -// rcu_tasks_trace_postgp(): -// Provides the needed full memory barrier and does debug checks. -// -// The exit_tasks_rcu_finish_trace() synchronizes with exiting tasks. -// -// Pre-grace-period update-side code is ordered before the grace period -// via the ->cbs_lock and barriers in rcu_tasks_kthread(). Pre-grace-peri= od -// read-side code is ordered before the grace period by atomic operations -// on .b.need_qs flag of each task involved in this process, or by schedul= er -// context-switch ordering (for locked-down non-running readers). - -// The lockdep state must be outside of #ifdef to be useful. -#ifdef CONFIG_DEBUG_LOCK_ALLOC -static struct lock_class_key rcu_lock_trace_key; -struct lockdep_map rcu_trace_lock_map =3D - STATIC_LOCKDEP_MAP_INIT("rcu_read_lock_trace", &rcu_lock_trace_key); -EXPORT_SYMBOL_GPL(rcu_trace_lock_map); -#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ +// is implemented via a straightforward mapping onto SRCU-fast. =20 #ifdef CONFIG_TASKS_TRACE_RCU =20 -// Record outstanding IPIs to each CPU. No point in sending two... -static DEFINE_PER_CPU(bool, trc_ipi_to_cpu); - -// The number of detections of task quiescent state relying on -// heavyweight readers executing explicit memory barriers. -static unsigned long n_heavy_reader_attempts; -static unsigned long n_heavy_reader_updates; -static unsigned long n_heavy_reader_ofl_updates; -static unsigned long n_trc_holdouts; +DEFINE_SRCU_FAST(rcu_tasks_trace_srcu_struct); +EXPORT_SYMBOL_GPL(rcu_tasks_trace_srcu_struct); =20 -void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func); -DEFINE_RCU_TASKS(rcu_tasks_trace, rcu_tasks_wait_gp, call_rcu_tasks_trace, - "RCU Tasks Trace"); - -/* Load from ->trc_reader_special.b.need_qs with proper ordering. */ -static u8 rcu_ld_need_qs(struct task_struct *t) +// Placeholder to suppress build errors through transition period. +void __init rcu_tasks_trace_suppress_unused(void) { - smp_mb(); // Enforce full grace-period ordering. - return smp_load_acquire(&t->trc_reader_special.b.need_qs); -} - -/* Store to ->trc_reader_special.b.need_qs with proper ordering. */ -static void rcu_st_need_qs(struct task_struct *t, u8 v) -{ - smp_store_release(&t->trc_reader_special.b.need_qs, v); - smp_mb(); // Enforce full grace-period ordering. +#ifndef CONFIG_TINY_RCU + show_rcu_tasks_generic_gp_kthread(NULL, NULL); +#endif // #ifndef CONFIG_TINY_RCU + rcu_spawn_tasks_kthread_generic(NULL); + synchronize_rcu_tasks_generic(NULL); + call_rcu_tasks_generic(NULL, NULL, NULL); + call_rcu_tasks_iw_wakeup(NULL); + cblist_init_generic(NULL); +#ifndef CONFIG_TINY_RCU + rcu_tasks_torture_stats_print_generic(NULL, NULL, NULL, NULL); +#endif // #ifndef CONFIG_TINY_RCU } =20 /* @@ -1555,321 +1487,12 @@ u8 rcu_trc_cmpxchg_need_qs(struct task_struct *t, = u8 old, u8 new) } EXPORT_SYMBOL_GPL(rcu_trc_cmpxchg_need_qs); =20 -/* - * If we are the last reader, signal the grace-period kthread. - * Also remove from the per-CPU list of blocked tasks. - */ -void rcu_read_unlock_trace_special(struct task_struct *t) -{ - unsigned long flags; - struct rcu_tasks_percpu *rtpcp; - union rcu_special trs; - - // Open-coded full-word version of rcu_ld_need_qs(). - smp_mb(); // Enforce full grace-period ordering. - trs =3D smp_load_acquire(&t->trc_reader_special); - - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) && t->trc_reader_special.b= .need_mb) - smp_mb(); // Pairs with update-side barriers. - // Update .need_qs before ->trc_reader_nesting for irq/NMI handlers. - if (trs.b.need_qs =3D=3D (TRC_NEED_QS_CHECKED | TRC_NEED_QS)) { - u8 result =3D rcu_trc_cmpxchg_need_qs(t, TRC_NEED_QS_CHECKED | TRC_NEED_= QS, - TRC_NEED_QS_CHECKED); - - WARN_ONCE(result !=3D trs.b.need_qs, "%s: result =3D %d", __func__, resu= lt); - } - if (trs.b.blocked) { - rtpcp =3D per_cpu_ptr(rcu_tasks_trace.rtpcpu, t->trc_blkd_cpu); - raw_spin_lock_irqsave_rcu_node(rtpcp, flags); - list_del_init(&t->trc_blkd_node); - WRITE_ONCE(t->trc_reader_special.b.blocked, false); - raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); - } - WRITE_ONCE(t->trc_reader_nesting, 0); -} -EXPORT_SYMBOL_GPL(rcu_read_unlock_trace_special); - /* Add a newly blocked reader task to its CPU's list. */ void rcu_tasks_trace_qs_blkd(struct task_struct *t) { - unsigned long flags; - struct rcu_tasks_percpu *rtpcp; - - local_irq_save(flags); - rtpcp =3D this_cpu_ptr(rcu_tasks_trace.rtpcpu); - raw_spin_lock_rcu_node(rtpcp); // irqs already disabled - t->trc_blkd_cpu =3D smp_processor_id(); - if (!rtpcp->rtp_blkd_tasks.next) - INIT_LIST_HEAD(&rtpcp->rtp_blkd_tasks); - list_add(&t->trc_blkd_node, &rtpcp->rtp_blkd_tasks); - WRITE_ONCE(t->trc_reader_special.b.blocked, true); - raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); } EXPORT_SYMBOL_GPL(rcu_tasks_trace_qs_blkd); =20 -/* Add a task to the holdout list, if it is not already on the list. */ -static void trc_add_holdout(struct task_struct *t, struct list_head *bhp) -{ - if (list_empty(&t->trc_holdout_list)) { - get_task_struct(t); - list_add(&t->trc_holdout_list, bhp); - n_trc_holdouts++; - } -} - -/* Remove a task from the holdout list, if it is in fact present. */ -static void trc_del_holdout(struct task_struct *t) -{ - if (!list_empty(&t->trc_holdout_list)) { - list_del_init(&t->trc_holdout_list); - put_task_struct(t); - n_trc_holdouts--; - } -} - -/* IPI handler to check task state. */ -static void trc_read_check_handler(void *t_in) -{ - int nesting; - struct task_struct *t =3D current; - struct task_struct *texp =3D t_in; - - // If the task is no longer running on this CPU, leave. - if (unlikely(texp !=3D t)) - goto reset_ipi; // Already on holdout list, so will check later. - - // If the task is not in a read-side critical section, and - // if this is the last reader, awaken the grace-period kthread. - nesting =3D READ_ONCE(t->trc_reader_nesting); - if (likely(!nesting)) { - rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS_CHECKED); - goto reset_ipi; - } - // If we are racing with an rcu_read_unlock_trace(), try again later. - if (unlikely(nesting < 0)) - goto reset_ipi; - - // Get here if the task is in a read-side critical section. - // Set its state so that it will update state for the grace-period - // kthread upon exit from that critical section. - rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS | TRC_NEED_QS_CHECKED); - -reset_ipi: - // Allow future IPIs to be sent on CPU and for task. - // Also order this IPI handler against any later manipulations of - // the intended task. - smp_store_release(per_cpu_ptr(&trc_ipi_to_cpu, smp_processor_id()), false= ); // ^^^ - smp_store_release(&texp->trc_ipi_to_cpu, -1); // ^^^ -} - -/* Callback function for scheduler to check locked-down task. */ -static int trc_inspect_reader(struct task_struct *t, void *bhp_in) -{ - struct list_head *bhp =3D bhp_in; - int cpu =3D task_cpu(t); - int nesting; - bool ofl =3D cpu_is_offline(cpu); - - if (task_curr(t) && !ofl) { - // If no chance of heavyweight readers, do it the hard way. - if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) - return -EINVAL; - - // If heavyweight readers are enabled on the remote task, - // we can inspect its state despite its currently running. - // However, we cannot safely change its state. - n_heavy_reader_attempts++; - // Check for "running" idle tasks on offline CPUs. - if (!rcu_watching_zero_in_eqs(cpu, &t->trc_reader_nesting)) - return -EINVAL; // No quiescent state, do it the hard way. - n_heavy_reader_updates++; - nesting =3D 0; - } else { - // The task is not running, so C-language access is safe. - nesting =3D t->trc_reader_nesting; - WARN_ON_ONCE(ofl && task_curr(t) && (t !=3D idle_task(task_cpu(t)))); - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) && ofl) - n_heavy_reader_ofl_updates++; - } - - // If not exiting a read-side critical section, mark as checked - // so that the grace-period kthread will remove it from the - // holdout list. - if (!nesting) { - rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS_CHECKED); - return 0; // In QS, so done. - } - if (nesting < 0) - return -EINVAL; // Reader transitioning, try again later. - - // The task is in a read-side critical section, so set up its - // state so that it will update state upon exit from that critical - // section. - if (!rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS | TRC_NEED_QS_CHECKED)) - trc_add_holdout(t, bhp); - return 0; -} - -/* Attempt to extract the state for the specified task. */ -static void trc_wait_for_one_reader(struct task_struct *t, - struct list_head *bhp) -{ - int cpu; - - // If a previous IPI is still in flight, let it complete. - if (smp_load_acquire(&t->trc_ipi_to_cpu) !=3D -1) // Order IPI - return; - - // The current task had better be in a quiescent state. - if (t =3D=3D current) { - rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS_CHECKED); - WARN_ON_ONCE(READ_ONCE(t->trc_reader_nesting)); - return; - } - - // Attempt to nail down the task for inspection. - get_task_struct(t); - if (!task_call_func(t, trc_inspect_reader, bhp)) { - put_task_struct(t); - return; - } - put_task_struct(t); - - // If this task is not yet on the holdout list, then we are in - // an RCU read-side critical section. Otherwise, the invocation of - // trc_add_holdout() that added it to the list did the necessary - // get_task_struct(). Either way, the task cannot be freed out - // from under this code. - - // If currently running, send an IPI, either way, add to list. - trc_add_holdout(t, bhp); - if (task_curr(t) && - time_after(jiffies + 1, rcu_tasks_trace.gp_start + rcu_task_ipi_delay= )) { - // The task is currently running, so try IPIing it. - cpu =3D task_cpu(t); - - // If there is already an IPI outstanding, let it happen. - if (per_cpu(trc_ipi_to_cpu, cpu) || t->trc_ipi_to_cpu >=3D 0) - return; - - per_cpu(trc_ipi_to_cpu, cpu) =3D true; - t->trc_ipi_to_cpu =3D cpu; - rcu_tasks_trace.n_ipis++; - if (smp_call_function_single(cpu, trc_read_check_handler, t, 0)) { - // Just in case there is some other reason for - // failure than the target CPU being offline. - WARN_ONCE(1, "%s(): smp_call_function_single() failed for CPU: %d\n", - __func__, cpu); - rcu_tasks_trace.n_ipis_fails++; - per_cpu(trc_ipi_to_cpu, cpu) =3D false; - t->trc_ipi_to_cpu =3D -1; - } - } -} - -/* - * Initialize for first-round processing for the specified task. - * Return false if task is NULL or already taken care of, true otherwise. - */ -static bool rcu_tasks_trace_pertask_prep(struct task_struct *t, bool notse= lf) -{ - // During early boot when there is only the one boot CPU, there - // is no idle task for the other CPUs. Also, the grace-period - // kthread is always in a quiescent state. In addition, just return - // if this task is already on the list. - if (unlikely(t =3D=3D NULL) || (t =3D=3D current && notself) || !list_emp= ty(&t->trc_holdout_list)) - return false; - - rcu_st_need_qs(t, 0); - t->trc_ipi_to_cpu =3D -1; - return true; -} - -/* Do first-round processing for the specified task. */ -static void rcu_tasks_trace_pertask(struct task_struct *t, struct list_hea= d *hop) -{ - if (rcu_tasks_trace_pertask_prep(t, true)) - trc_wait_for_one_reader(t, hop); -} - -/* Initialize for a new RCU-tasks-trace grace period. */ -static void rcu_tasks_trace_pregp_step(struct list_head *hop) -{ - LIST_HEAD(blkd_tasks); - int cpu; - unsigned long flags; - struct rcu_tasks_percpu *rtpcp; - struct task_struct *t; - - // There shouldn't be any old IPIs, but... - for_each_possible_cpu(cpu) - WARN_ON_ONCE(per_cpu(trc_ipi_to_cpu, cpu)); - - // Disable CPU hotplug across the CPU scan for the benefit of - // any IPIs that might be needed. This also waits for all readers - // in CPU-hotplug code paths. - cpus_read_lock(); - - // These rcu_tasks_trace_pertask_prep() calls are serialized to - // allow safe access to the hop list. - for_each_online_cpu(cpu) { - rcu_read_lock(); - // Note that cpu_curr_snapshot() picks up the target - // CPU's current task while its runqueue is locked with - // an smp_mb__after_spinlock(). This ensures that either - // the grace-period kthread will see that task's read-side - // critical section or the task will see the updater's pre-GP - // accesses. The trailing smp_mb() in cpu_curr_snapshot() - // does not currently play a role other than simplify - // that function's ordering semantics. If these simplified - // ordering semantics continue to be redundant, that smp_mb() - // might be removed. - t =3D cpu_curr_snapshot(cpu); - if (rcu_tasks_trace_pertask_prep(t, true)) - trc_add_holdout(t, hop); - rcu_read_unlock(); - cond_resched_tasks_rcu_qs(); - } - - // Only after all running tasks have been accounted for is it - // safe to take care of the tasks that have blocked within their - // current RCU tasks trace read-side critical section. - for_each_possible_cpu(cpu) { - rtpcp =3D per_cpu_ptr(rcu_tasks_trace.rtpcpu, cpu); - raw_spin_lock_irqsave_rcu_node(rtpcp, flags); - list_splice_init(&rtpcp->rtp_blkd_tasks, &blkd_tasks); - while (!list_empty(&blkd_tasks)) { - rcu_read_lock(); - t =3D list_first_entry(&blkd_tasks, struct task_struct, trc_blkd_node); - list_del_init(&t->trc_blkd_node); - list_add(&t->trc_blkd_node, &rtpcp->rtp_blkd_tasks); - raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); - rcu_tasks_trace_pertask(t, hop); - rcu_read_unlock(); - raw_spin_lock_irqsave_rcu_node(rtpcp, flags); - } - raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); - cond_resched_tasks_rcu_qs(); - } - - // Re-enable CPU hotplug now that the holdout list is populated. - cpus_read_unlock(); -} - -/* - * Do intermediate processing between task and holdout scans. - */ -static void rcu_tasks_trace_postscan(struct list_head *hop) -{ - // Wait for late-stage exiting tasks to finish exiting. - // These might have passed the call to exit_tasks_rcu_finish(). - - // If you remove the following line, update rcu_trace_implies_rcu_gp()!!! - synchronize_rcu(); - // Any tasks that exit after this point will set - // TRC_NEED_QS_CHECKED in ->trc_reader_special.b.need_qs. -} - /* Communicate task state back to the RCU tasks trace stall warning reques= t. */ struct trc_stall_chk_rdr { int nesting; @@ -1877,241 +1500,39 @@ struct trc_stall_chk_rdr { u8 needqs; }; =20 -static int trc_check_slow_task(struct task_struct *t, void *arg) -{ - struct trc_stall_chk_rdr *trc_rdrp =3D arg; - - if (task_curr(t) && cpu_online(task_cpu(t))) - return false; // It is running, so decline to inspect it. - trc_rdrp->nesting =3D READ_ONCE(t->trc_reader_nesting); - trc_rdrp->ipi_to_cpu =3D READ_ONCE(t->trc_ipi_to_cpu); - trc_rdrp->needqs =3D rcu_ld_need_qs(t); - return true; -} - -/* Show the state of a task stalling the current RCU tasks trace GP. */ -static void show_stalled_task_trace(struct task_struct *t, bool *firstrepo= rt) -{ - int cpu; - struct trc_stall_chk_rdr trc_rdr; - bool is_idle_tsk =3D is_idle_task(t); - - if (*firstreport) { - pr_err("INFO: rcu_tasks_trace detected stalls on tasks:\n"); - *firstreport =3D false; - } - cpu =3D task_cpu(t); - if (!task_call_func(t, trc_check_slow_task, &trc_rdr)) - pr_alert("P%d: %c%c\n", - t->pid, - ".I"[t->trc_ipi_to_cpu >=3D 0], - ".i"[is_idle_tsk]); - else - pr_alert("P%d: %c%c%c%c nesting: %d%c%c cpu: %d%s\n", - t->pid, - ".I"[trc_rdr.ipi_to_cpu >=3D 0], - ".i"[is_idle_tsk], - ".N"[cpu >=3D 0 && tick_nohz_full_cpu(cpu)], - ".B"[!!data_race(t->trc_reader_special.b.blocked)], - trc_rdr.nesting, - " !CN"[trc_rdr.needqs & 0x3], - " ?"[trc_rdr.needqs > 0x3], - cpu, cpu_online(cpu) ? "" : "(offline)"); - sched_show_task(t); -} - -/* List stalled IPIs for RCU tasks trace. */ -static void show_stalled_ipi_trace(void) -{ - int cpu; - - for_each_possible_cpu(cpu) - if (per_cpu(trc_ipi_to_cpu, cpu)) - pr_alert("\tIPI outstanding to CPU %d\n", cpu); -} - -/* Do one scan of the holdout list. */ -static void check_all_holdout_tasks_trace(struct list_head *hop, - bool needreport, bool *firstreport) -{ - struct task_struct *g, *t; - - // Disable CPU hotplug across the holdout list scan for IPIs. - cpus_read_lock(); - - list_for_each_entry_safe(t, g, hop, trc_holdout_list) { - // If safe and needed, try to check the current task. - if (READ_ONCE(t->trc_ipi_to_cpu) =3D=3D -1 && - !(rcu_ld_need_qs(t) & TRC_NEED_QS_CHECKED)) - trc_wait_for_one_reader(t, hop); - - // If check succeeded, remove this task from the list. - if (smp_load_acquire(&t->trc_ipi_to_cpu) =3D=3D -1 && - rcu_ld_need_qs(t) =3D=3D TRC_NEED_QS_CHECKED) - trc_del_holdout(t); - else if (needreport) - show_stalled_task_trace(t, firstreport); - cond_resched_tasks_rcu_qs(); - } - - // Re-enable CPU hotplug now that the holdout list scan has completed. - cpus_read_unlock(); - - if (needreport) { - if (*firstreport) - pr_err("INFO: rcu_tasks_trace detected stalls? (Late IPI?)\n"); - show_stalled_ipi_trace(); - } -} - -static void rcu_tasks_trace_empty_fn(void *unused) -{ -} - -/* Wait for grace period to complete and provide ordering. */ -static void rcu_tasks_trace_postgp(struct rcu_tasks *rtp) -{ - int cpu; - - // Wait for any lingering IPI handlers to complete. Note that - // if a CPU has gone offline or transitioned to userspace in the - // meantime, all IPI handlers should have been drained beforehand. - // Yes, this assumes that CPUs process IPIs in order. If that ever - // changes, there will need to be a recheck and/or timed wait. - for_each_online_cpu(cpu) - if (WARN_ON_ONCE(smp_load_acquire(per_cpu_ptr(&trc_ipi_to_cpu, cpu)))) - smp_call_function_single(cpu, rcu_tasks_trace_empty_fn, NULL, 1); - - smp_mb(); // Caller's code must be ordered after wakeup. - // Pairs with pretty much every ordering primitive. -} - /* Report any needed quiescent state for this exiting task. */ static void exit_tasks_rcu_finish_trace(struct task_struct *t) { - union rcu_special trs =3D READ_ONCE(t->trc_reader_special); - - rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS_CHECKED); - WARN_ON_ONCE(READ_ONCE(t->trc_reader_nesting)); - if (WARN_ON_ONCE(rcu_ld_need_qs(t) & TRC_NEED_QS || trs.b.blocked)) - rcu_read_unlock_trace_special(t); - else - WRITE_ONCE(t->trc_reader_nesting, 0); -} - -/** - * call_rcu_tasks_trace() - Queue a callback trace task-based grace period - * @rhp: structure to be used for queueing the RCU updates. - * @func: actual callback function to be invoked after the grace period - * - * The callback function will be invoked some time after a trace rcu-tasks - * grace period elapses, in other words after all currently executing - * trace rcu-tasks read-side critical sections have completed. These - * read-side critical sections are delimited by calls to rcu_read_lock_tra= ce() - * and rcu_read_unlock_trace(). - * - * See the description of call_rcu() for more detailed information on - * memory ordering guarantees. - */ -void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func) -{ - call_rcu_tasks_generic(rhp, func, &rcu_tasks_trace); } -EXPORT_SYMBOL_GPL(call_rcu_tasks_trace); - -/** - * synchronize_rcu_tasks_trace - wait for a trace rcu-tasks grace period - * - * Control will return to the caller some time after a trace rcu-tasks - * grace period has elapsed, in other words after all currently executing - * trace rcu-tasks read-side critical sections have elapsed. These read-si= de - * critical sections are delimited by calls to rcu_read_lock_trace() - * and rcu_read_unlock_trace(). - * - * This is a very specialized primitive, intended only for a few uses in - * tracing and other situations requiring manipulation of function preambl= es - * and profiling hooks. The synchronize_rcu_tasks_trace() function is not - * (yet) intended for heavy use from multiple CPUs. - * - * See the description of synchronize_rcu() for more detailed information - * on memory ordering guarantees. - */ -void synchronize_rcu_tasks_trace(void) -{ - RCU_LOCKDEP_WARN(lock_is_held(&rcu_trace_lock_map), "Illegal synchronize_= rcu_tasks_trace() in RCU Tasks Trace read-side critical section"); - synchronize_rcu_tasks_generic(&rcu_tasks_trace); -} -EXPORT_SYMBOL_GPL(synchronize_rcu_tasks_trace); - -/** - * rcu_barrier_tasks_trace - Wait for in-flight call_rcu_tasks_trace() cal= lbacks. - * - * Although the current implementation is guaranteed to wait, it is not - * obligated to, for example, if there are no pending callbacks. - */ -void rcu_barrier_tasks_trace(void) -{ - rcu_barrier_tasks_generic(&rcu_tasks_trace); -} -EXPORT_SYMBOL_GPL(rcu_barrier_tasks_trace); =20 int rcu_tasks_trace_lazy_ms =3D -1; module_param(rcu_tasks_trace_lazy_ms, int, 0444); =20 static int __init rcu_spawn_tasks_trace_kthread(void) { - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) { - rcu_tasks_trace.gp_sleep =3D HZ / 10; - rcu_tasks_trace.init_fract =3D HZ / 10; - } else { - rcu_tasks_trace.gp_sleep =3D HZ / 200; - if (rcu_tasks_trace.gp_sleep <=3D 0) - rcu_tasks_trace.gp_sleep =3D 1; - rcu_tasks_trace.init_fract =3D HZ / 200; - if (rcu_tasks_trace.init_fract <=3D 0) - rcu_tasks_trace.init_fract =3D 1; - } - if (rcu_tasks_trace_lazy_ms >=3D 0) - rcu_tasks_trace.lazy_jiffies =3D msecs_to_jiffies(rcu_tasks_trace_lazy_m= s); - rcu_tasks_trace.pregp_func =3D rcu_tasks_trace_pregp_step; - rcu_tasks_trace.postscan_func =3D rcu_tasks_trace_postscan; - rcu_tasks_trace.holdouts_func =3D check_all_holdout_tasks_trace; - rcu_tasks_trace.postgp_func =3D rcu_tasks_trace_postgp; - rcu_spawn_tasks_kthread_generic(&rcu_tasks_trace); return 0; } =20 #if !defined(CONFIG_TINY_RCU) void show_rcu_tasks_trace_gp_kthread(void) { - char buf[64]; - - snprintf(buf, sizeof(buf), "N%lu h:%lu/%lu/%lu", - data_race(n_trc_holdouts), - data_race(n_heavy_reader_ofl_updates), - data_race(n_heavy_reader_updates), - data_race(n_heavy_reader_attempts)); - show_rcu_tasks_generic_gp_kthread(&rcu_tasks_trace, buf); } EXPORT_SYMBOL_GPL(show_rcu_tasks_trace_gp_kthread); =20 void rcu_tasks_trace_torture_stats_print(char *tt, char *tf) { - rcu_tasks_torture_stats_print_generic(&rcu_tasks_trace, tt, tf, ""); } EXPORT_SYMBOL_GPL(rcu_tasks_trace_torture_stats_print); #endif // !defined(CONFIG_TINY_RCU) =20 struct task_struct *get_rcu_tasks_trace_gp_kthread(void) { - return rcu_tasks_trace.kthread_ptr; + return NULL; } EXPORT_SYMBOL_GPL(get_rcu_tasks_trace_gp_kthread); =20 void rcu_tasks_trace_get_gp_data(int *flags, unsigned long *gp_seq) { - *flags =3D 0; - *gp_seq =3D rcu_seq_current(&rcu_tasks_trace.tasks_gp_seq); } EXPORT_SYMBOL_GPL(rcu_tasks_trace_get_gp_data); =20 @@ -2251,10 +1672,6 @@ void __init tasks_cblist_init_generic(void) #ifdef CONFIG_TASKS_RUDE_RCU cblist_init_generic(&rcu_tasks_rude); #endif - -#ifdef CONFIG_TASKS_TRACE_RCU - cblist_init_generic(&rcu_tasks_trace); -#endif } =20 static int __init rcu_init_tasks_generic(void) --=20 2.40.1 From nobody Tue Feb 10 14:43:57 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52DE323A98E; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; cv=none; b=IVxkeKzu1YAIQC8bIRsaJGYa8CpEA4gCoMaNxVNyHgRsCwTlb82zTCZjUvyJ6docCZwjxPCiUDwSow7nHblX3RS7cpXChamZ/UW5AFsU/mDNhMb3QiNrDapRP4AanVkIk5ug3qbaZtwv//mz8p6O0GuCl+KIcdiZNDKmhWH1Rkc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; c=relaxed/simple; bh=H1OYPAFLY33QGyp3X+2tKdJZfWJ6CMrZ3iYMWsERzxo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FmLNxCvlX+yuW+Qa6pPC/+GdJgYty9TfgaYyp6h6moKVM4Objhwr2iu39f9xjLa+ndv+LE1r3ELvOueON0A4eJRsLJ/BIn7NH3HZybnyUGTYXSOp7wDDuXO5DiBlh6wFaLuRrb7aBhQYBmJhHgPk+xCmsMpIR0hXBo/4Rusq8t8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rlO2+I/4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rlO2+I/4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CC06FC16AAE; Mon, 29 Dec 2025 19:11:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1767035466; bh=H1OYPAFLY33QGyp3X+2tKdJZfWJ6CMrZ3iYMWsERzxo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rlO2+I/4vlAsAG6ulFZ+cjRLNfVRd5kvKMWEbsXkY3HBLrKPyW6hWC1j7F2H7vin5 KNLfChK0Oo+S3bH/4/IprHwAUN1nkZ7mI/j0PY/2cZ3/388R/kY/abNo0QQKcfc2RG BRWSg8PY6FbDuVshHdl+Cr/IWppsK/UmQuHCaskaGupp2rpHR8naM63U+5em65i6Cn MAuN5Y6EgOrsQWGuu6JSrsY+SKs98B25B475RS+1oIm/T8xW0Yw2qtCyCFY5cRBN8r Fj/zR/hq5nj1RYi9AlgRuaZ0nmC3ZSYt19lbeUiKNnIXIFwAQ0BRwyUTQHgcGO+ZAQ Vj0UUO17WbE5w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 6E51ACE0CAF; Mon, 29 Dec 2025 11:11:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Frederic Weisbecker , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v4 2/9] context_tracking: Remove rcu_task_trace_heavyweight_{enter,exit}() Date: Mon, 29 Dec 2025 11:10:57 -0800 Message-Id: <20251229191104.693447-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Because SRCU-fast does not use IPIs for its grace periods, there is no need for real-time workloads to switch to an IPI-free mode, and there is in turn no need for either rcu_task_trace_heavyweight_enter() or rcu_task_trace_heavyweight_exit(). This commit therefore removes them. Signed-off-by: Paul E. McKenney Cc: Frederic Weisbecker Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Reviewed-by: Joel Fernandes --- kernel/context_tracking.c | 20 -------------------- 1 file changed, 20 deletions(-) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index fb5be6e9b423f..a743e7ffa6c00 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -54,24 +54,6 @@ static __always_inline void rcu_task_enter(void) #endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ } =20 -/* Turn on heavyweight RCU tasks trace readers on kernel exit. */ -static __always_inline void rcu_task_trace_heavyweight_enter(void) -{ -#ifdef CONFIG_TASKS_TRACE_RCU - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) - current->trc_reader_special.b.need_mb =3D true; -#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ -} - -/* Turn off heavyweight RCU tasks trace readers on kernel entry. */ -static __always_inline void rcu_task_trace_heavyweight_exit(void) -{ -#ifdef CONFIG_TASKS_TRACE_RCU - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) - current->trc_reader_special.b.need_mb =3D false; -#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ -} - /* * Record entry into an extended quiescent state. This is only to be * called when not already in an extended quiescent state, that is, @@ -85,7 +67,6 @@ static noinstr void ct_kernel_exit_state(int offset) * critical sections, and we also must force ordering with the * next idle sojourn. */ - rcu_task_trace_heavyweight_enter(); // Before CT state update! // RCU is still watching. Better not be in extended quiescent state! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !rcu_is_watching_curr_cp= u()); (void)ct_state_inc(offset); @@ -108,7 +89,6 @@ static noinstr void ct_kernel_enter_state(int offset) */ seq =3D ct_state_inc(offset); // RCU is now watching. Better not be in an extended quiescent state! - rcu_task_trace_heavyweight_exit(); // After CT state update! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & CT_RCU_WATCHING)= ); } =20 --=20 2.40.1 From nobody Tue Feb 10 14:43:57 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52D632236F0; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; cv=none; b=TcKCvUq/HfBCx+6/1rErq/3K67nHMHixPU3jMJFqmsimgesdW69fcqD1UWjitoxT//pxdzhYx29LW4jrHnmUB5HDKZ3Z8u9Z63RcVx5j3xJnaN8ppagnB/uLD82Wt4ZF5gsgQYfBHr66S2P3haaTVx4CEDdh3JxWDfB0w2LRssw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; c=relaxed/simple; bh=1YRkuHtj9Xo0mLgw+Xr1XrwtDVKisJQwW30B052OFV8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DEVZ7FP405eAo/JLrY26c8UTLgBMqGu1XwgZCA7pEhaLq5zUK8S0Od7UNeg7JprLiCRt5JjjfHBT+TJxR6zKYr2pNOegtX7pu0jTfo/glBveg/Xd3YZZsbqMxgvwOd9Nsaygp2lZ02x2f7lyVbOOH21t6+kLhdMgGzB9FYoOZGU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Gtj4IqWK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Gtj4IqWK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D5752C19422; Mon, 29 Dec 2025 19:11:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1767035466; bh=1YRkuHtj9Xo0mLgw+Xr1XrwtDVKisJQwW30B052OFV8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Gtj4IqWKy0HxYPI0pYHWsXSYzhBeLIkeI4quei1OTAro8CSciCm8dUgED6RhZ0IKH TvjaV4guWt0+MbGwvM1gDnLMlFQkE3JT05OYpNyU5xHTcUDp7wk0VXgHrbnYfva14J KotNDnEEzl+zhgSrrJ0pKS3djfwblKLd7pb+8OYPFHdaF5hZJpnSFEhMP5S16xZYcc VNfC0E4yiIZ5nVbeLc5zgBN+vc6Zt6IbVlbxd6AnbAUzT9c8Q0d0t63GmBLuaywxS0 uRIa062tjLRJakWeiXEAtnJTAU9gspaAVi2QI9bs0NojOkPGQP00QdssNLV97R35bQ +Ju9CcCMruHaA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 70B51CE0E29; Mon, 29 Dec 2025 11:11:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v4 3/9] rcu: Clean up after the SRCU-fastification of RCU Tasks Trace Date: Mon, 29 Dec 2025 11:10:58 -0800 Message-Id: <20251229191104.693447-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that RCU Tasks Trace has been re-implemented in terms of SRCU-fast, the ->trc_ipi_to_cpu, ->trc_blkd_cpu, ->trc_blkd_node, ->trc_holdout_list, and ->trc_reader_special task_struct fields are no longer used. In addition, the rcu_tasks_trace_qs(), rcu_tasks_trace_qs_blkd(), exit_tasks_rcu_finish_trace(), and rcu_spawn_tasks_trace_kthread(), show_rcu_tasks_trace_gp_kthread(), rcu_tasks_trace_get_gp_data(), rcu_tasks_trace_torture_stats_print(), and get_rcu_tasks_trace_gp_kthread() functions and all the other functions that they invoke are no longer used. Also, the TRC_NEED_QS and TRC_NEED_QS_CHECKED CPP macros are no longer used. Neither are the rcu_tasks_trace_lazy_ms and rcu_task_ipi_delay rcupdate module parameters and the TASKS_TRACE_RCU_READ_MB Kconfig option. This commit therefore removes all of them. [ paulmck: Apply Alexei Starovoitov feedback. ] Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Reviewed-by: Joel Fernandes --- .../admin-guide/kernel-parameters.txt | 15 ---- include/linux/rcupdate.h | 31 +------- include/linux/rcupdate_trace.h | 2 - include/linux/sched.h | 5 -- init/init_task.c | 3 - kernel/fork.c | 3 - kernel/rcu/Kconfig | 18 ----- kernel/rcu/rcu.h | 9 --- kernel/rcu/rcuscale.c | 7 -- kernel/rcu/rcutorture.c | 2 - kernel/rcu/tasks.h | 79 +------------------ .../selftests/rcutorture/configs/rcu/TRACE01 | 1 - .../selftests/rcutorture/configs/rcu/TRACE02 | 1 - 13 files changed, 2 insertions(+), 174 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index a8d0afde7f85a..1b8e5cadbecbc 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6249,13 +6249,6 @@ Kernel parameters dynamically) adjusted. This parameter is intended for use in testing. =20 - rcupdate.rcu_task_ipi_delay=3D [KNL] - Set time in jiffies during which RCU tasks will - avoid sending IPIs, starting with the beginning - of a given grace period. Setting a large - number avoids disturbing real-time workloads, - but lengthens grace periods. - rcupdate.rcu_task_lazy_lim=3D [KNL] Number of callbacks on a given CPU that will cancel laziness on that CPU. Use -1 to disable @@ -6299,14 +6292,6 @@ Kernel parameters of zero will disable batching. Batching is always disabled for synchronize_rcu_tasks(). =20 - rcupdate.rcu_tasks_trace_lazy_ms=3D [KNL] - Set timeout in milliseconds RCU Tasks - Trace asynchronous callback batching for - call_rcu_tasks_trace(). A negative value - will take the default. A value of zero will - disable batching. Batching is always disabled - for synchronize_rcu_tasks_trace(). - rcupdate.rcu_self_test=3D [KNL] Run the RCU early boot self tests =20 diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index c5b30054cd018..bd5a420cf09a0 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -175,36 +175,7 @@ void rcu_tasks_torture_stats_print(char *tt, char *tf); # define synchronize_rcu_tasks synchronize_rcu # endif =20 -# ifdef CONFIG_TASKS_TRACE_RCU -// Bits for ->trc_reader_special.b.need_qs field. -#define TRC_NEED_QS 0x1 // Task needs a quiescent state. -#define TRC_NEED_QS_CHECKED 0x2 // Task has been checked for needing quie= scent state. - -u8 rcu_trc_cmpxchg_need_qs(struct task_struct *t, u8 old, u8 new); -void rcu_tasks_trace_qs_blkd(struct task_struct *t); - -# define rcu_tasks_trace_qs(t) \ - do { \ - int ___rttq_nesting =3D READ_ONCE((t)->trc_reader_nesting); \ - \ - if (unlikely(READ_ONCE((t)->trc_reader_special.b.need_qs) =3D=3D TRC_NEE= D_QS) && \ - likely(!___rttq_nesting)) { \ - rcu_trc_cmpxchg_need_qs((t), TRC_NEED_QS, TRC_NEED_QS_CHECKED); \ - } else if (___rttq_nesting && ___rttq_nesting !=3D INT_MIN && \ - !READ_ONCE((t)->trc_reader_special.b.blocked)) { \ - rcu_tasks_trace_qs_blkd(t); \ - } \ - } while (0) -void rcu_tasks_trace_torture_stats_print(char *tt, char *tf); -# else -# define rcu_tasks_trace_qs(t) do { } while (0) -# endif - -#define rcu_tasks_qs(t, preempt) \ -do { \ - rcu_tasks_classic_qs((t), (preempt)); \ - rcu_tasks_trace_qs(t); \ -} while (0) +#define rcu_tasks_qs(t, preempt) rcu_tasks_classic_qs((t), (preempt)) =20 # ifdef CONFIG_TASKS_RUDE_RCU void synchronize_rcu_tasks_rude(void); diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index 3f46cbe670003..0bd47f12ecd17 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -136,9 +136,7 @@ static inline void rcu_barrier_tasks_trace(void) } =20 // Placeholders to enable stepwise transition. -void rcu_tasks_trace_get_gp_data(int *flags, unsigned long *gp_seq); void __init rcu_tasks_trace_suppress_unused(void); -struct task_struct *get_rcu_tasks_trace_gp_kthread(void); =20 #else /* diff --git a/include/linux/sched.h b/include/linux/sched.h index fe39d422b37d7..56156643ccac8 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -946,11 +946,6 @@ struct task_struct { #ifdef CONFIG_TASKS_TRACE_RCU int trc_reader_nesting; struct srcu_ctr __percpu *trc_reader_scp; - int trc_ipi_to_cpu; - union rcu_special trc_reader_special; - struct list_head trc_holdout_list; - struct list_head trc_blkd_node; - int trc_blkd_cpu; #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ =20 struct sched_info sched_info; diff --git a/init/init_task.c b/init/init_task.c index 49b13d7c3985d..db92c404d59a8 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -195,9 +195,6 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = =3D { #endif #ifdef CONFIG_TASKS_TRACE_RCU .trc_reader_nesting =3D 0, - .trc_reader_special.s =3D 0, - .trc_holdout_list =3D LIST_HEAD_INIT(init_task.trc_holdout_list), - .trc_blkd_node =3D LIST_HEAD_INIT(init_task.trc_blkd_node), #endif #ifdef CONFIG_CPUSETS .mems_allowed_seq =3D SEQCNT_SPINLOCK_ZERO(init_task.mems_allowed_seq, diff --git a/kernel/fork.c b/kernel/fork.c index b1f3915d5f8ec..d7ed107cbb47d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1828,9 +1828,6 @@ static inline void rcu_copy_process(struct task_struc= t *p) #endif /* #ifdef CONFIG_TASKS_RCU */ #ifdef CONFIG_TASKS_TRACE_RCU p->trc_reader_nesting =3D 0; - p->trc_reader_special.s =3D 0; - INIT_LIST_HEAD(&p->trc_holdout_list); - INIT_LIST_HEAD(&p->trc_blkd_node); #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ } =20 diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 4d9b21f69eaae..8d5a1ecb7d56c 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -313,24 +313,6 @@ config RCU_NOCB_CPU_CB_BOOST Say Y here if you want to set RT priority for offloading kthreads. Say N here if you are building a !PREEMPT_RT kernel and are unsure. =20 -config TASKS_TRACE_RCU_READ_MB - bool "Tasks Trace RCU readers use memory barriers in user and idle" - depends on RCU_EXPERT && TASKS_TRACE_RCU - default PREEMPT_RT || NR_CPUS < 8 - help - Use this option to further reduce the number of IPIs sent - to CPUs executing in userspace or idle during tasks trace - RCU grace periods. Given that a reasonable setting of - the rcupdate.rcu_task_ipi_delay kernel boot parameter - eliminates such IPIs for many workloads, proper setting - of this Kconfig option is important mostly for aggressive - real-time installations and for battery-powered devices, - hence the default chosen above. - - Say Y here if you hate IPIs. - Say N here if you hate read-side memory barriers. - Take the default if you are unsure. - config RCU_LAZY bool "RCU callback lazy invocation functionality" depends on RCU_NOCB_CPU diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 9cf01832a6c3d..dc5d614b372c1 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -544,10 +544,6 @@ struct task_struct *get_rcu_tasks_rude_gp_kthread(void= ); void rcu_tasks_rude_get_gp_data(int *flags, unsigned long *gp_seq); #endif // # ifdef CONFIG_TASKS_RUDE_RCU =20 -#ifdef CONFIG_TASKS_TRACE_RCU -void rcu_tasks_trace_get_gp_data(int *flags, unsigned long *gp_seq); -#endif - #ifdef CONFIG_TASKS_RCU_GENERIC void tasks_cblist_init_generic(void); #else /* #ifdef CONFIG_TASKS_RCU_GENERIC */ @@ -673,11 +669,6 @@ void show_rcu_tasks_rude_gp_kthread(void); #else static inline void show_rcu_tasks_rude_gp_kthread(void) {} #endif -#if !defined(CONFIG_TINY_RCU) && defined(CONFIG_TASKS_TRACE_RCU) -void show_rcu_tasks_trace_gp_kthread(void); -#else -static inline void show_rcu_tasks_trace_gp_kthread(void) {} -#endif =20 #ifdef CONFIG_TINY_RCU static inline bool rcu_cpu_beenfullyonline(int cpu) { return true; } diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 7484d8ad5767b..1c50f89fbd6f7 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -400,11 +400,6 @@ static void tasks_trace_scale_read_unlock(int idx) rcu_read_unlock_trace(); } =20 -static void rcu_tasks_trace_scale_stats(void) -{ - rcu_tasks_trace_torture_stats_print(scale_type, SCALE_FLAG); -} - static struct rcu_scale_ops tasks_tracing_ops =3D { .ptype =3D RCU_TASKS_FLAVOR, .init =3D rcu_sync_scale_init, @@ -416,8 +411,6 @@ static struct rcu_scale_ops tasks_tracing_ops =3D { .gp_barrier =3D rcu_barrier_tasks_trace, .sync =3D synchronize_rcu_tasks_trace, .exp_sync =3D synchronize_rcu_tasks_trace, - .rso_gp_kthread =3D get_rcu_tasks_trace_gp_kthread, - .stats =3D IS_ENABLED(CONFIG_TINY_RCU) ? NULL : rcu_tasks_trace_scale_st= ats, .name =3D "tasks-tracing" }; =20 diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 07e51974b06bc..78a6ebe77d35d 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -1180,8 +1180,6 @@ static struct rcu_torture_ops tasks_tracing_ops =3D { .exp_sync =3D synchronize_rcu_tasks_trace, .call =3D call_rcu_tasks_trace, .cb_barrier =3D rcu_barrier_tasks_trace, - .gp_kthread_dbg =3D show_rcu_tasks_trace_gp_kthread, - .get_gp_data =3D rcu_tasks_trace_get_gp_data, .cbflood_max =3D 50000, .irq_capable =3D 1, .slow_gps =3D 1, diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 1fe789c99f361..1249b47f0a8da 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -161,11 +161,6 @@ static void tasks_rcu_exit_srcu_stall(struct timer_lis= t *unused); static DEFINE_TIMER(tasks_rcu_exit_srcu_stall_timer, tasks_rcu_exit_srcu_s= tall); #endif =20 -/* Avoid IPIing CPUs early in the grace period. */ -#define RCU_TASK_IPI_DELAY (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) ? H= Z / 2 : 0) -static int rcu_task_ipi_delay __read_mostly =3D RCU_TASK_IPI_DELAY; -module_param(rcu_task_ipi_delay, int, 0644); - /* Control stall timeouts. Disable with <=3D 0, otherwise jiffies till st= all. */ #define RCU_TASK_BOOT_STALL_TIMEOUT (HZ * 30) #define RCU_TASK_STALL_TIMEOUT (HZ * 60 * 10) @@ -800,8 +795,6 @@ static void rcu_tasks_torture_stats_print_generic(struc= t rcu_tasks *rtp, char *t =20 #endif // #ifndef CONFIG_TINY_RCU =20 -static void exit_tasks_rcu_finish_trace(struct task_struct *t); - #if defined(CONFIG_TASKS_RCU) =20 //////////////////////////////////////////////////////////////////////// @@ -1321,13 +1314,11 @@ void exit_tasks_rcu_finish(void) raw_spin_lock_irqsave_rcu_node(rtpcp, flags); list_del_init(&t->rcu_tasks_exit_list); raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); - - exit_tasks_rcu_finish_trace(t); } =20 #else /* #ifdef CONFIG_TASKS_RCU */ void exit_tasks_rcu_start(void) { } -void exit_tasks_rcu_finish(void) { exit_tasks_rcu_finish_trace(current); } +void exit_tasks_rcu_finish(void) { } #endif /* #else #ifdef CONFIG_TASKS_RCU */ =20 #ifdef CONFIG_TASKS_RUDE_RCU @@ -1475,69 +1466,6 @@ void __init rcu_tasks_trace_suppress_unused(void) #endif // #ifndef CONFIG_TINY_RCU } =20 -/* - * Do a cmpxchg() on ->trc_reader_special.b.need_qs, allowing for - * the four-byte operand-size restriction of some platforms. - * - * Returns the old value, which is often ignored. - */ -u8 rcu_trc_cmpxchg_need_qs(struct task_struct *t, u8 old, u8 new) -{ - return cmpxchg(&t->trc_reader_special.b.need_qs, old, new); -} -EXPORT_SYMBOL_GPL(rcu_trc_cmpxchg_need_qs); - -/* Add a newly blocked reader task to its CPU's list. */ -void rcu_tasks_trace_qs_blkd(struct task_struct *t) -{ -} -EXPORT_SYMBOL_GPL(rcu_tasks_trace_qs_blkd); - -/* Communicate task state back to the RCU tasks trace stall warning reques= t. */ -struct trc_stall_chk_rdr { - int nesting; - int ipi_to_cpu; - u8 needqs; -}; - -/* Report any needed quiescent state for this exiting task. */ -static void exit_tasks_rcu_finish_trace(struct task_struct *t) -{ -} - -int rcu_tasks_trace_lazy_ms =3D -1; -module_param(rcu_tasks_trace_lazy_ms, int, 0444); - -static int __init rcu_spawn_tasks_trace_kthread(void) -{ - return 0; -} - -#if !defined(CONFIG_TINY_RCU) -void show_rcu_tasks_trace_gp_kthread(void) -{ -} -EXPORT_SYMBOL_GPL(show_rcu_tasks_trace_gp_kthread); - -void rcu_tasks_trace_torture_stats_print(char *tt, char *tf) -{ -} -EXPORT_SYMBOL_GPL(rcu_tasks_trace_torture_stats_print); -#endif // !defined(CONFIG_TINY_RCU) - -struct task_struct *get_rcu_tasks_trace_gp_kthread(void) -{ - return NULL; -} -EXPORT_SYMBOL_GPL(get_rcu_tasks_trace_gp_kthread); - -void rcu_tasks_trace_get_gp_data(int *flags, unsigned long *gp_seq) -{ -} -EXPORT_SYMBOL_GPL(rcu_tasks_trace_get_gp_data); - -#else /* #ifdef CONFIG_TASKS_TRACE_RCU */ -static void exit_tasks_rcu_finish_trace(struct task_struct *t) { } #endif /* #else #ifdef CONFIG_TASKS_TRACE_RCU */ =20 #ifndef CONFIG_TINY_RCU @@ -1545,7 +1473,6 @@ void show_rcu_tasks_gp_kthreads(void) { show_rcu_tasks_classic_gp_kthread(); show_rcu_tasks_rude_gp_kthread(); - show_rcu_tasks_trace_gp_kthread(); } #endif /* #ifndef CONFIG_TINY_RCU */ =20 @@ -1684,10 +1611,6 @@ static int __init rcu_init_tasks_generic(void) rcu_spawn_tasks_rude_kthread(); #endif =20 -#ifdef CONFIG_TASKS_TRACE_RCU - rcu_spawn_tasks_trace_kthread(); -#endif - // Run the self-tests. rcu_tasks_initiate_self_tests(); =20 diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TRACE01 b/tools= /testing/selftests/rcutorture/configs/rcu/TRACE01 index 85b407467454a..18efab346381a 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/TRACE01 +++ b/tools/testing/selftests/rcutorture/configs/rcu/TRACE01 @@ -10,5 +10,4 @@ CONFIG_PROVE_LOCKING=3Dn #CHECK#CONFIG_PROVE_RCU=3Dn CONFIG_FORCE_TASKS_TRACE_RCU=3Dy #CHECK#CONFIG_TASKS_TRACE_RCU=3Dy -CONFIG_TASKS_TRACE_RCU_READ_MB=3Dy CONFIG_RCU_EXPERT=3Dy diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TRACE02 b/tools= /testing/selftests/rcutorture/configs/rcu/TRACE02 index 9003c56cd7648..8da390e828297 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/TRACE02 +++ b/tools/testing/selftests/rcutorture/configs/rcu/TRACE02 @@ -9,6 +9,5 @@ CONFIG_PROVE_LOCKING=3Dy #CHECK#CONFIG_PROVE_RCU=3Dy CONFIG_FORCE_TASKS_TRACE_RCU=3Dy #CHECK#CONFIG_TASKS_TRACE_RCU=3Dy -CONFIG_TASKS_TRACE_RCU_READ_MB=3Dn CONFIG_RCU_EXPERT=3Dy CONFIG_DEBUG_OBJECTS=3Dy --=20 2.40.1 From nobody Tue Feb 10 14:43:57 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52E4823AB88; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; cv=none; b=qhiBegbwvJmO9qKYC8lk3PXHghg5cHDaQHygJrLukJRLAJBfgmjUg8VZrIVIok/vg/dNBHKyEIqvRerfCXCNHb9INcxlJqkEhRghGWrdkJAkfiL5GDBsFWEQT8LTmhreLLN7o6Yj4uqq3r/RGYoFfLu/oE+nepLXMHljKl6gT1w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; c=relaxed/simple; bh=GZhDAA9ezGZPzDeUVwifsh1ekjSX48vKF1A/9IzvzwI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OOiTakowj8gV2IE8MCgYRJIhR7MdSXQpcDrj5YX6x1E9TCdVXQhySC/a4cb5oc79bO6IbT5B5KU7gjTtUa2lNkGeDO1CoP5D3yNKoTJalbaT9Jd6HY1WqJHUlmlDoE/SJkY/l7igIoVtfq4syJgcsimZPNapOqqfk/6uz/giyEQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=eZoEfqpC; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="eZoEfqpC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D1443C2BC86; Mon, 29 Dec 2025 19:11:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1767035466; bh=GZhDAA9ezGZPzDeUVwifsh1ekjSX48vKF1A/9IzvzwI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eZoEfqpC7Dob5fwQik8Ed/0klyOSPTCosLIQ3e8gzMO0L+4m6169dwsf+ebP5sNsd g9vThapp7kyXwAb6qP7t16p/xDJhQkCh47vguTo1bd2r/RGqJTrgzRSs9G918ENwvx xxgaWFVnEMuEw3fkInGfRIC+DJi1vxWimUttKCuCBKTvLEe62RS4j1BIBp5mpLCaZT vYDBRr6re8XfPsBgtwwj2DohYjSUISiPPbMuZQxBCaILoij4W160JQfKUY9p9WOje8 i3zSYUpsfVoGv4RbT3XpDKi3ckfgRepCuXXKH28W7B+RJeNxd9uvHCNqdvViYDOckk dVfg9vVUKnNvQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7331CCE1078; Mon, 29 Dec 2025 11:11:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v4 4/9] rcu: Move rcu_tasks_trace_srcu_struct out of #ifdef CONFIG_TASKS_RCU_GENERIC Date: Mon, 29 Dec 2025 11:10:59 -0800 Message-Id: <20251229191104.693447-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Moving the rcu_tasks_trace_srcu_struct structure instance out from under the CONFIG_TASKS_RCU_GENERIC Kconfig option permits the CONFIG_TASKS_TRACE_RCU Kconfig option to stop enabling this CONFIG_TASKS_RCU_GENERIC Kconfig option. This commit also therefore makes it so. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Reviewed-by: Joel Fernandes --- kernel/rcu/Kconfig | 2 +- kernel/rcu/tasks.h | 42 +++++++++++++----------------------------- 2 files changed, 14 insertions(+), 30 deletions(-) diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 8d5a1ecb7d56c..c381a31301168 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -82,7 +82,7 @@ config NEED_SRCU_NMI_SAFE def_bool HAVE_NMI && !ARCH_HAS_NMI_SAFE_THIS_CPU_OPS && !TINY_SRCU =20 config TASKS_RCU_GENERIC - def_bool TASKS_RCU || TASKS_RUDE_RCU || TASKS_TRACE_RCU + def_bool TASKS_RCU || TASKS_RUDE_RCU help This option enables generic infrastructure code supporting task-based RCU implementations. Not for manual selection. diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 1249b47f0a8da..76f952196a292 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1439,35 +1439,6 @@ EXPORT_SYMBOL_GPL(rcu_tasks_rude_get_gp_data); =20 #endif /* #ifdef CONFIG_TASKS_RUDE_RCU */ =20 -//////////////////////////////////////////////////////////////////////// -// -// Tracing variant of Tasks RCU. This variant is designed to be used -// to protect tracing hooks, including those of BPF. This variant -// is implemented via a straightforward mapping onto SRCU-fast. - -#ifdef CONFIG_TASKS_TRACE_RCU - -DEFINE_SRCU_FAST(rcu_tasks_trace_srcu_struct); -EXPORT_SYMBOL_GPL(rcu_tasks_trace_srcu_struct); - -// Placeholder to suppress build errors through transition period. -void __init rcu_tasks_trace_suppress_unused(void) -{ -#ifndef CONFIG_TINY_RCU - show_rcu_tasks_generic_gp_kthread(NULL, NULL); -#endif // #ifndef CONFIG_TINY_RCU - rcu_spawn_tasks_kthread_generic(NULL); - synchronize_rcu_tasks_generic(NULL); - call_rcu_tasks_generic(NULL, NULL, NULL); - call_rcu_tasks_iw_wakeup(NULL); - cblist_init_generic(NULL); -#ifndef CONFIG_TINY_RCU - rcu_tasks_torture_stats_print_generic(NULL, NULL, NULL, NULL); -#endif // #ifndef CONFIG_TINY_RCU -} - -#endif /* #else #ifdef CONFIG_TASKS_TRACE_RCU */ - #ifndef CONFIG_TINY_RCU void show_rcu_tasks_gp_kthreads(void) { @@ -1621,3 +1592,16 @@ core_initcall(rcu_init_tasks_generic); #else /* #ifdef CONFIG_TASKS_RCU_GENERIC */ static inline void rcu_tasks_bootup_oddness(void) {} #endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */ + +#ifdef CONFIG_TASKS_TRACE_RCU + +//////////////////////////////////////////////////////////////////////// +// +// Tracing variant of Tasks RCU. This variant is designed to be used +// to protect tracing hooks, including those of BPF. This variant +// is implemented via a straightforward mapping onto SRCU-fast. + +DEFINE_SRCU_FAST(rcu_tasks_trace_srcu_struct); +EXPORT_SYMBOL_GPL(rcu_tasks_trace_srcu_struct); + +#endif /* #else #ifdef CONFIG_TASKS_TRACE_RCU */ --=20 2.40.1 From nobody Tue Feb 10 14:43:57 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 922E327F00A; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; cv=none; b=ouChiIVwAxZr/D0GKKSmoPWYTkc2EfSH8PDon5twxxyVHjbsIKW0diVtqjcHuZ98tZhGvwKuC7/2re4vOQzIA8uxKODvW0QPIY+9zxqXjCLnJTCHP7Tk0DChX+pfO6aqbsVQfAld4AcOwQCndntaLRXOTuzfIskkiDk3wa535PY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; c=relaxed/simple; bh=XUhX9y4AUAsfs0RX5l5s4LfdmkuCqgRoPEpYS58H5Dc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QWw9DzMSFz38+KdCHc8aFhYqPBxoBGESH2Tun/SaGBzLjLJpagkmvhr++j6TDDEaFkh/j2UtapKdFrLsHGNC0Lk5DFG1rhlRAi2RJYlhekHS7v0mSW0cxhmz+W6uZUxz5rr3rLO9bqkE5oMW4EMD+XSxr4znncT3eJyDemfu9Og= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=l0qAfahU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="l0qAfahU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DF686C19423; Mon, 29 Dec 2025 19:11:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1767035466; bh=XUhX9y4AUAsfs0RX5l5s4LfdmkuCqgRoPEpYS58H5Dc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l0qAfahUtbXzmwgG+XQyijQM0r4r6JUP9lQKh6mw0X3MdCOypBw35vsG11bNpxWmO MnkAegtgNiXIBzMOgWZnZiBqVLv+WUvGBUBCK1JLi4I68fnIGPGAlxteRIaFmpAU5o sDGvXMzma7w4Uv73EwqU8ZdB+zViV7n0HlWl+q2W8aBe3LJEl0XNonIQX9VukYFuvY XQK17K8MCWFsawHctH4Qdh08z2qramzau4HP9gBdBglkWJlsg2SIsoK+fTldwTEghE 0FpsR6jq+09lsQ3OC2t8DdZP55zMPa2pgEmjEyDRG4oea8vV+Z33/FuCv7YeSOvR6X l5hYgMuU8m/ZA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 757A2CE1079; Mon, 29 Dec 2025 11:11:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v4 5/9] rcu: Add noinstr-fast rcu_read_{,un}lock_tasks_trace() APIs Date: Mon, 29 Dec 2025 11:11:00 -0800 Message-Id: <20251229191104.693447-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When expressing RCU Tasks Trace in terms of SRCU-fast, it was necessary to keep a nesting count and per-CPU srcu_ctr structure pointer in the task_struct structure, which is slow to access. But an alternative is to instead make rcu_read_lock_tasks_trace() and rcu_read_unlock_tasks_trace(), which match the underlying SRCU-fast semantics, avoiding the task_struct accesses. When all callers have switched to the new API, the previous rcu_read_lock_trace() and rcu_read_unlock_trace() APIs will be removed. The rcu_read_{,un}lock_{,tasks_}trace() functions need to use smp_mb() only if invoked where RCU is not watching, that is, from locations where a call to rcu_is_watching() would return false. In architectures that define the ARCH_WANTS_NO_INSTR Kconfig option, use of noinstr and friends ensures that tracing happens only where RCU is watching, so those architectures can dispense entirely with the read-side calls to smp_mb(). Other architectures include these read-side calls by default, but in many installations there might be either larger than average tolerance for risk, prohibition of removing tracing on a running system, or careful review and approval of removing of tracing. Such installations can build their kernels with CONFIG_TASKS_TRACE_RCU_NO_MB=3Dy to avoid those read-side calls to smp_mb(), thus accepting responsibility for run-time removal of tracing from code regions that RCU is not watching. Those wishing to disable read-side memory barriers for an entire architecture can select this TASKS_TRACE_RCU_NO_MB Kconfig option, hence the polarity. [ paulmck: Apply Peter Zijlstra feedback. ] Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Reviewed-by: Joel Fernandes --- include/linux/rcupdate_trace.h | 65 +++++++++++++++++++++++++++++----- kernel/rcu/Kconfig | 23 ++++++++++++ 2 files changed, 80 insertions(+), 8 deletions(-) diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index 0bd47f12ecd17..f47ba9c074601 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -34,6 +34,53 @@ static inline int rcu_read_lock_trace_held(void) =20 #ifdef CONFIG_TASKS_TRACE_RCU =20 +/** + * rcu_read_lock_tasks_trace - mark beginning of RCU-trace read-side criti= cal section + * + * When synchronize_rcu_tasks_trace() is invoked by one task, then that + * task is guaranteed to block until all other tasks exit their read-side + * critical sections. Similarly, if call_rcu_trace() is invoked on one + * task while other tasks are within RCU read-side critical sections, + * invocation of the corresponding RCU callback is deferred until after + * the all the other tasks exit their critical sections. + * + * For more details, please see the documentation for + * srcu_read_lock_fast(). For a description of how implicit RCU + * readers provide the needed ordering for architectures defining the + * ARCH_WANTS_NO_INSTR Kconfig option (and thus promising never to trace + * code where RCU is not watching), please see the __srcu_read_lock_fast() + * (non-kerneldoc) header comment. Otherwise, the smp_mb() below provided + * the needed ordering. + */ +static inline struct srcu_ctr __percpu *rcu_read_lock_tasks_trace(void) +{ + struct srcu_ctr __percpu *ret =3D __srcu_read_lock_fast(&rcu_tasks_trace_= srcu_struct); + + rcu_try_lock_acquire(&rcu_tasks_trace_srcu_struct.dep_map); + if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB)) + smp_mb(); // Provide ordering on noinstr-incomplete architectures. + return ret; +} + +/** + * rcu_read_unlock_tasks_trace - mark end of RCU-trace read-side critical = section + * @scp: return value from corresponding rcu_read_lock_tasks_trace(). + * + * Pairs with the preceding call to rcu_read_lock_tasks_trace() that + * returned the value passed in via scp. + * + * For more details, please see the documentation for rcu_read_unlock(). + * For memory-ordering information, please see the header comment for the + * rcu_read_lock_tasks_trace() function. + */ +static inline void rcu_read_unlock_tasks_trace(struct srcu_ctr __percpu *s= cp) +{ + if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB)) + smp_mb(); // Provide ordering on noinstr-incomplete architectures. + __srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp); + srcu_lock_release(&rcu_tasks_trace_srcu_struct.dep_map); +} + /** * rcu_read_lock_trace - mark beginning of RCU-trace read-side critical se= ction * @@ -50,14 +97,15 @@ static inline void rcu_read_lock_trace(void) { struct task_struct *t =3D current; =20 + rcu_try_lock_acquire(&rcu_tasks_trace_srcu_struct.dep_map); if (t->trc_reader_nesting++) { // In case we interrupted a Tasks Trace RCU reader. - rcu_try_lock_acquire(&rcu_tasks_trace_srcu_struct.dep_map); return; } barrier(); // nesting before scp to protect against interrupt handler. - t->trc_reader_scp =3D srcu_read_lock_fast(&rcu_tasks_trace_srcu_struct); - smp_mb(); // Placeholder for more selective ordering + t->trc_reader_scp =3D __srcu_read_lock_fast(&rcu_tasks_trace_srcu_struct); + if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB)) + smp_mb(); // Placeholder for more selective ordering } =20 /** @@ -74,13 +122,14 @@ static inline void rcu_read_unlock_trace(void) struct srcu_ctr __percpu *scp; struct task_struct *t =3D current; =20 - smp_mb(); // Placeholder for more selective ordering scp =3D t->trc_reader_scp; barrier(); // scp before nesting to protect against interrupt handler. - if (!--t->trc_reader_nesting) - srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp); - else - srcu_lock_release(&rcu_tasks_trace_srcu_struct.dep_map); + if (!--t->trc_reader_nesting) { + if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB)) + smp_mb(); // Placeholder for more selective ordering + __srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp); + } + srcu_lock_release(&rcu_tasks_trace_srcu_struct.dep_map); } =20 /** diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index c381a31301168..762299291e09b 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -142,6 +142,29 @@ config TASKS_TRACE_RCU default n select IRQ_WORK =20 +config TASKS_TRACE_RCU_NO_MB + bool "Override RCU Tasks Trace inclusion of read-side memory barriers" + depends on RCU_EXPERT && TASKS_TRACE_RCU + default ARCH_WANTS_NO_INSTR + help + This option prevents the use of read-side memory barriers in + rcu_read_lock_tasks_trace() and rcu_read_unlock_tasks_trace() + even in kernels built with CONFIG_ARCH_WANTS_NO_INSTR=3Dn, that is, + in kernels that do not have noinstr set up in entry/exit code. + By setting this option, you are promising to carefully review + use of ftrace, BPF, and friends to ensure that no tracing + operation is attached to a function that runs in that portion + of the entry/exit code that RCU does not watch, that is, + where rcu_is_watching() returns false. Alternatively, you + might choose to never remove traces except by rebooting. + + Those wishing to disable read-side memory barriers for an entire + architecture can select this Kconfig option, hence the polarity. + + Say Y here if you need speed and will review use of tracing. + Say N here for certain esoteric testing of RCU itself. + Take the default if you are unsure. + config RCU_STALL_COMMON def_bool TREE_RCU help --=20 2.40.1 From nobody Tue Feb 10 14:43:57 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9368F27F4CA; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; cv=none; b=Enhf8i/KazqImsL/mV7ma0QP/wk65lEhrgdxK7zvwiO+wXIQ3R1YQrT+yX0axK+u8WqWtkNR7m5gxAF/FJ04Gao6KTP4EEXpKWORPX2G4SkKUq8M/Uc05DdV0s6F6/ViAGCs+wAQj56MWX0A7AH1s0+vULau4De0PaED0FOuXIM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; c=relaxed/simple; bh=Ax7FBvNwaT/XyPQJp8bumqr3pkAZvIxwCUWKEB+5I40=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pd6Hv6kHT9+HnPI5oAg3X9BDdNBvLHZgiRXOzemEcp3p0oeFgXnKRlReFb7DD6bxA6qADmht3m7Pn0XKH2EdyxtzZd6CQIj29adfBjAx7dQcc2r6eQ2phbjAstaYCbRRx+J+tA5PM/D4yso07zbCUY/ipyA6CW6jL8hrb1HibTo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kBtkOgI7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kBtkOgI7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1F65EC2BC87; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1767035467; bh=Ax7FBvNwaT/XyPQJp8bumqr3pkAZvIxwCUWKEB+5I40=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kBtkOgI7FntPLyVBT9mVPpSLqozf4X28y1qnhCHDQWIy4bcYWvqujwWFRqcU29j4V F93hqWB5o/+Yg0qLpVwjdTZGTuLyRbYHBBOWlu7dhExKrj9yhSNZv36K6lVNGTRwfp QeFNldkeTSacIPbBJZ972g88LKSYkaZMTGHFx7iTPqRyiPNE7U4Yud6jNtxROzt9P4 yABjw3hbMSAwS4QhPWzbEEVBcKUOK9WX1wGVaDeSJq6ezABE5w5yT9qqgXigW2Rawm V/1Q9Jj4OLSwi7sb/PBPzNsdtQvThcxnWzDfEW+ufDIqJ74qlwgqIXnfDX8cpHpZF4 UeFcJk8NU9Wgw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 77A79CE10F4; Mon, 29 Dec 2025 11:11:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v4 6/9] rcu: Update Requirements.rst for RCU Tasks Trace Date: Mon, 29 Dec 2025 11:11:01 -0800 Message-Id: <20251229191104.693447-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit updates the documentation to declare that RCU Tasks Trace is implemented as a thin wrapper around SRCU-fast. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Reviewed-by: Joel Fernandes --- .../RCU/Design/Requirements/Requirements.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Docum= entation/RCU/Design/Requirements/Requirements.rst index ba417a08b93df..b5cdbba3ec2e7 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2780,12 +2780,12 @@ Tasks Trace RCU ~~~~~~~~~~~~~~~ =20 Some forms of tracing need to sleep in readers, but cannot tolerate -SRCU's read-side overhead, which includes a full memory barrier in both -srcu_read_lock() and srcu_read_unlock(). This need is handled by a -Tasks Trace RCU that uses scheduler locking and IPIs to synchronize with -readers. Real-time systems that cannot tolerate IPIs may build their -kernels with ``CONFIG_TASKS_TRACE_RCU_READ_MB=3Dy``, which avoids the IPIs= at -the expense of adding full memory barriers to the read-side primitives. +SRCU's read-side overhead, which includes a full memory barrier in +both srcu_read_lock() and srcu_read_unlock(). This need is handled by +a Tasks Trace RCU API implemented as thin wrappers around SRCU-fast, +which avoids the read-side memory barriers, at least for architectures +that apply noinstr to kernel entry/exit code (or that build with +``CONFIG_TASKS_TRACE_RCU_NO_MB=3Dy``. =20 The tasks-trace-RCU API is also reasonably compact, consisting of rcu_read_lock_trace(), rcu_read_unlock_trace(), --=20 2.40.1 From nobody Tue Feb 10 14:43:57 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 905C627E077; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; cv=none; b=P6sdDrw+bST3oNZWqj7AmowpNuXuTJZ3aYR43jq21e5NmivuP+pZiEhG5dQr5tR0jgjen1rbtPrrAMyHCsVgzBeFgN+h2JGx9nzstvX5QVQjMvr96GbEiJgOwvhmeVesyQ1XC+RiJW7Qbw1JVUEGo8JRjBM7HZohgX1gL6lWfmM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; c=relaxed/simple; bh=am3a4f/tTp1G67zvZ8p5t5twCf+9bpA6UUdHIvr9SDs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=irQM3RkGi414DD3/BSUJMb5Q403ro7o6sSVPjNRhRULFuPsMq712fZ8un5fezvFU2OG1XfmBt9F1QApVlXGzkgV+YtlFgHAPyZW3V0yc9mX2PSCag2bYJ85tZ1raT5WKalmZ22h/w/a18+rg8xTALi4ixLUFoAmmexaASnlnC04= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=G6gWU0gq; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="G6gWU0gq" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E622C2BCAF; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1767035467; bh=am3a4f/tTp1G67zvZ8p5t5twCf+9bpA6UUdHIvr9SDs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G6gWU0gqaXHs8iS6OF0/ijZeI/dlu1SZzULEc1e2hs+9YERcIhFpGckuIrroFy+xC bZC+bANsXxraTtvqxlgbtXWx4mXfURFyUzl4dftonsFzToBNiz2unHSKaeK6n0tOTh 8lm6tgM4bFd9LTe8RTWDMsdcC2hmGuYwBZfbn+kuegj1Zl5Iy54/GrfNWX2IzjQw2W ri/ei+KI8JV5UTEWdfsB4wIyhar3xAcn9crea6U7mLfqHbsx49Z7fMlp6sWHJsqIXE d15RuyM+zaTcQtWy0aB3/xSO7uOCtHc3TCrpys60Dxgd7rdSs2fQCgsiWoauSB0xp5 bbybjIdaZ8c6Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7A13ACE1109; Mon, 29 Dec 2025 11:11:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Joe Perches , Andy Whitcroft , Dwaipayan Ray , Lukas Bulwahn , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v4 7/9] checkpatch: Deprecate rcu_read_{,un}lock_trace() Date: Mon, 29 Dec 2025 11:11:02 -0800 Message-Id: <20251229191104.693447-7-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Uses of rcu_read_lock_trace() and rcu_read_unlock_trace() are better served by the new rcu_read_lock_tasks_trace() and rcu_read_unlock_tasks_trace() APIs. Therefore, mark the old APIs as deprecated. Signed-off-by: Paul E. McKenney Acked-by: Joe Perches Cc: Andy Whitcroft Cc: Dwaipayan Ray Cc: Lukas Bulwahn Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Reviewed-by: Joel Fernandes --- scripts/checkpatch.pl | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl index c0250244cf7a3..362a8d1cd3278 100755 --- a/scripts/checkpatch.pl +++ b/scripts/checkpatch.pl @@ -863,7 +863,9 @@ our %deprecated_apis =3D ( #These should be enough to drive away new IDR users "DEFINE_IDR" =3D> "DEFINE_XARRAY", "idr_init" =3D> "xa_init", - "idr_init_base" =3D> "xa_init_flags" + "idr_init_base" =3D> "xa_init_flags", + "rcu_read_lock_trace" =3D> "rcu_read_lock_tasks_trace", + "rcu_read_unlock_trace" =3D> "rcu_read_unlock_tasks_trace", ); =20 #Create a search pattern for all these strings to speed up a loop below --=20 2.40.1 From nobody Tue Feb 10 14:43:57 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 845D6274671; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; cv=none; b=sf1uCP3G5QtAc3M5th/BLP+rlcbjJbfN2B2c3GAahZ6sTafn+7TyDHZEx6mJtfTu6yMS3iaVvGf5YFg/CNKDH/A6WhOOEPM6qNRe0KWm9N/doTp+u14y3dbAhj//ASJTmdg/kXILl1GYURBdiqnRYCLCYYAg6vdrVTY4/sB5YbE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; c=relaxed/simple; bh=cPP2uxUbUwhnZIVIHEGQUBtpGHQ7pekEysBIW2q62FU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UnCIAbt8GIb1X+Ipo4ZjceV2+P2PMifdN0jmoIRixO/c4TrUkFJG0vrTE/0agxuxoHNQiYrX8euUqaxlikqa0dDtjF26MDqG3Hk+KsxDwPps1FvCctf9O+F6shRMn/b+3hvriAD5mTSbbLB9Z3bZ3bVeuVFiqtA0JgwMIcFyUzo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OKdSoCCc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OKdSoCCc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 39187C2BCB6; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1767035467; bh=cPP2uxUbUwhnZIVIHEGQUBtpGHQ7pekEysBIW2q62FU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OKdSoCCcJcjXZntNLz69qz37Ix9/0AH64kd0KU1xOw7Tkiz4tB/c3qnX9Phn38aQK d4PqagTAzkhGdiSnOlTdp4/8DqxapZBqXb0tFBBnsvHxg2RC1c5xc/6PSXpovlu8LJ 7SiWuPvULjo4X9PK9Q75MD3lMY3cIynD5gC+SgRHG6Fs5/SaX+GjDSCJnwqQ5Xf9XZ e/f1H2eEhpRZATufiUbOkRAXMQG96/cXQoCPysxf6OZbM+ZR8iMl6BaamUtjE2P+PY QXJn1vDkmChMZLcU7w1i3v6his49CdGu6o//MwU8e5luiENCXkRgcOsU5PcsIjppMU zfJzTfhQ/RQpg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7C3DCCE1148; Mon, 29 Dec 2025 11:11:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v4 8/9] srcu: Create an rcu_tasks_trace_expedite_current() function Date: Mon, 29 Dec 2025 11:11:03 -0800 Message-Id: <20251229191104.693447-8-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates an rcu_tasks_trace_expedite_current() function that expedites the current (and possibly the next) RCU Tasks Trace grace period. If the current RCU Tasks Trace grace period is already waiting, that wait will complete before the expediting takes effect. If there is no RCU Tasks Trace grace period in flight, this function might well create one. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Reviewed-by: Joel Fernandes --- include/linux/rcupdate_trace.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index f47ba9c074601..cee89e51e45cb 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -184,6 +184,20 @@ static inline void rcu_barrier_tasks_trace(void) srcu_barrier(&rcu_tasks_trace_srcu_struct); } =20 +/** + * rcu_tasks_trace_expedite_current - Expedite the current Tasks Trace RCU= grace period + * + * Cause the current Tasks Trace RCU grace period to become expedited. + * The grace period following the current one might also be expedited. + * If there is no current grace period, one might be created. If the + * current grace period is currently sleeping, that sleep will complete + * before expediting will take effect. + */ +static inline void rcu_tasks_trace_expedite_current(void) +{ + srcu_expedite_current(&rcu_tasks_trace_srcu_struct); +} + // Placeholders to enable stepwise transition. void __init rcu_tasks_trace_suppress_unused(void); =20 --=20 2.40.1 From nobody Tue Feb 10 14:43:57 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 853B3274B4D; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; cv=none; b=SN70VcjPXu+WTucKqOsbgGIXazbhoJ1R2sIDHb4Ptc++x2ADDo8w/VMF8Uh3l9EwHSwLWMA9/tNhCmvJlWd0IxQdmw5QAXlV2QaxsRinDccAvPiOInPqpNYUvN4+WaHfGAX3WHOMxml5BQRunEZ1Voxam43m9/BmkRhllcouxm4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767035467; c=relaxed/simple; bh=RCb48nJYGD9FDhOCx8rn3xYPgrAldJAxldDGmLx+cXw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RWVHDm5ZEDuD2jq9YWHtbeTXlL6izCNyYtxZZTotXzK18xyvvGod7aw3D4thRobjeI0iWuyxVULTLs/NraDap65yxtf0N/yLTwn9M32ql2QO5iJunvB+Ph8CttH7aaoX5aKPVKNbrAJUUWg3zknz4E6u1QTUR72QW/iScBUzcio= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GPud9yx8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GPud9yx8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3AED9C116D0; Mon, 29 Dec 2025 19:11:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1767035467; bh=RCb48nJYGD9FDhOCx8rn3xYPgrAldJAxldDGmLx+cXw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GPud9yx8fkrJj1bXBaOEtWEXx0zNuJBAsu1KSTeBA4eyzJ87sk7zKkPbbljl2BCWg hzJ1mZSmjuyXZ7CN+mZLSoKHod4Vj9uuG80RK0O8m2aj3IcutlWyF9PHA3+o7DuyNE dEBdIkjmrSF/MUoyk4iturHrQhCcWcmfMg95Tzyhihsxr+HJgAG6lpyRmCTgy/565j wTyDTRG7tbsXKfL8JoRIxlP/j0lCVgLIMA59x5LFmNs9405nkQRKlF4/QblKz9o0HU 7SX1J9jPchzMrJ8G2rbuOfnWLXZvef27IPMjQalNhR13FJ1fqt/iLTBwQ4/D+o5dbP ptww06FsZUElw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7E802CE11A4; Mon, 29 Dec 2025 11:11:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v4 9/9] rcutorture: Test rcu_tasks_trace_expedite_current() Date: Mon, 29 Dec 2025 11:11:04 -0800 Message-Id: <20251229191104.693447-9-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds a ->exp_current member to the tasks_tracing_ops structure to test the rcu_tasks_trace_expedite_current() function. [ paulmck: Apply kernel test robot feedback. ] Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Reviewed-by: Joel Fernandes --- kernel/rcu/rcutorture.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 78a6ebe77d35d..d00b043823aef 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -1178,6 +1178,7 @@ static struct rcu_torture_ops tasks_tracing_ops =3D { .deferred_free =3D rcu_tasks_tracing_torture_deferred_free, .sync =3D synchronize_rcu_tasks_trace, .exp_sync =3D synchronize_rcu_tasks_trace, + .exp_current =3D rcu_tasks_trace_expedite_current, .call =3D call_rcu_tasks_trace, .cb_barrier =3D rcu_barrier_tasks_trace, .cbflood_max =3D 50000, --=20 2.40.1