From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B48B124BBE4; Tue, 23 Sep 2025 14:22:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637323; cv=none; b=WJUM6OfGb+o6PVNHc8mU2SSPocUIizKGu+J9GlmZ0ARd+Mzwl64xavdKu0v6Y9zYtaGWpkAKDbtPbnofvll8J8ji89csd7uw2Z5jdkJDWQojhSsQGGurVOOz0oio9kkHcdbYg0r0HAe1RkEUEs+FpyOFAu1Tt6UbXscg1yurLg0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637323; c=relaxed/simple; bh=V5+JDdN+fLIW0rZe8ms8PxR0P3GOXR+NvoTDuCl4DmA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=efb2Vmf0Nb/fzxdphY1Tw5i8muQbXqW+aqmv6fmJ3v+VINEN3ME5Md4ruLnrONjAH0r+oJcOM0pfPjREpLtfka+3huK5JgBvTudYjbfOfgCrxVRbect4CsFgd5Y5+MN74q0icZcQ6To0mpR3ADLuJT5S3lnAwsi/ohTYIsRUBTA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NY/2XjLy; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NY/2XjLy" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7CBCBC113D0; Tue, 23 Sep 2025 14:21:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637323; bh=V5+JDdN+fLIW0rZe8ms8PxR0P3GOXR+NvoTDuCl4DmA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NY/2XjLyjP8v898z/RuaoHTMLwqR4jOkB1FblDIzwNiyQlYmD1fS/6r5GF1fqodn5 74tTiF3nYt/EFfnDSrpeH7NiTtJHt+pOciWMfhF8MbuFpMKIRLsLP0mcO5INDRrfcB sYUqH3wqC+Ie3MUV1q4BD3gFjGUg9MMlqN0tLr78Qg+5EmWMyaPKJBoVHoFN1qyI4x spX/1W3zVaFSnk1xdvAG7blYB2imYeYvF+6VUVJ4dSWuoVXNca5WfAVHINg9G4K6k6 VDcskmA+zGBWV/orPn9bHTKIZsqrCAhumemDpAhtFWRj63NbxzdO8r2cGBlA8Jr1jd 6l0dgHduAu90Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id B5696CE0E5A; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , kernel test robot , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 01/34] rcu: Re-implement RCU Tasks Trace in terms of SRCU-fast Date: Tue, 23 Sep 2025 07:20:03 -0700 Message-Id: <20250923142036.112290-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit saves more than 500 lines of RCU code by re-implementing RCU Tasks Trace in terms of SRCU-fast. Follow-up work will remove more code that does not cause problems by its presence, but that is no longer required. This variant places smp_mb() in rcu_read_{,un}lock_trace(), which will be removed on common-case architectures in a later commit. [ paulmck: Apply kernel test robot, Boqun Feng, and Zqiang feedback. ] Signed-off-by: Paul E. McKenney Tested-by: kernel test robot Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/rcupdate_trace.h | 107 ++++-- include/linux/sched.h | 1 + kernel/rcu/srcutiny.c | 13 +- kernel/rcu/tasks.h | 617 +-------------------------------- 4 files changed, 104 insertions(+), 634 deletions(-) diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index e6c44eb428ab63..3f46cbe6700038 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -12,28 +12,28 @@ #include #include =20 -extern struct lockdep_map rcu_trace_lock_map; +#ifdef CONFIG_TASKS_TRACE_RCU +extern struct srcu_struct rcu_tasks_trace_srcu_struct; +#endif // #ifdef CONFIG_TASKS_TRACE_RCU =20 -#ifdef CONFIG_DEBUG_LOCK_ALLOC +#if defined(CONFIG_DEBUG_LOCK_ALLOC) && defined(CONFIG_TASKS_TRACE_RCU) =20 static inline int rcu_read_lock_trace_held(void) { - return lock_is_held(&rcu_trace_lock_map); + return srcu_read_lock_held(&rcu_tasks_trace_srcu_struct); } =20 -#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ +#else // #if defined(CONFIG_DEBUG_LOCK_ALLOC) && defined(CONFIG_TASKS_TRAC= E_RCU) =20 static inline int rcu_read_lock_trace_held(void) { return 1; } =20 -#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ +#endif // #else // #if defined(CONFIG_DEBUG_LOCK_ALLOC) && defined(CONFIG_= TASKS_TRACE_RCU) =20 #ifdef CONFIG_TASKS_TRACE_RCU =20 -void rcu_read_unlock_trace_special(struct task_struct *t); - /** * rcu_read_lock_trace - mark beginning of RCU-trace read-side critical se= ction * @@ -50,12 +50,14 @@ static inline void rcu_read_lock_trace(void) { struct task_struct *t =3D current; =20 - WRITE_ONCE(t->trc_reader_nesting, READ_ONCE(t->trc_reader_nesting) + 1); - barrier(); - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) && - t->trc_reader_special.b.need_mb) - smp_mb(); // Pairs with update-side barriers - rcu_lock_acquire(&rcu_trace_lock_map); + if (t->trc_reader_nesting++) { + // In case we interrupted a Tasks Trace RCU reader. + rcu_try_lock_acquire(&rcu_tasks_trace_srcu_struct.dep_map); + return; + } + barrier(); // nesting before scp to protect against interrupt handler. + t->trc_reader_scp =3D srcu_read_lock_fast(&rcu_tasks_trace_srcu_struct); + smp_mb(); // Placeholder for more selective ordering } =20 /** @@ -69,26 +71,75 @@ static inline void rcu_read_lock_trace(void) */ static inline void rcu_read_unlock_trace(void) { - int nesting; + struct srcu_ctr __percpu *scp; struct task_struct *t =3D current; =20 - rcu_lock_release(&rcu_trace_lock_map); - nesting =3D READ_ONCE(t->trc_reader_nesting) - 1; - barrier(); // Critical section before disabling. - // Disable IPI-based setting of .need_qs. - WRITE_ONCE(t->trc_reader_nesting, INT_MIN + nesting); - if (likely(!READ_ONCE(t->trc_reader_special.s)) || nesting) { - WRITE_ONCE(t->trc_reader_nesting, nesting); - return; // We assume shallow reader nesting. - } - WARN_ON_ONCE(nesting !=3D 0); - rcu_read_unlock_trace_special(t); + smp_mb(); // Placeholder for more selective ordering + scp =3D t->trc_reader_scp; + barrier(); // scp before nesting to protect against interrupt handler. + if (!--t->trc_reader_nesting) + srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp); + else + srcu_lock_release(&rcu_tasks_trace_srcu_struct.dep_map); +} + +/** + * call_rcu_tasks_trace() - Queue a callback trace task-based grace period + * @rhp: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a trace rcu-tasks + * grace period elapses, in other words after all currently executing + * trace rcu-tasks read-side critical sections have completed. These + * read-side critical sections are delimited by calls to rcu_read_lock_tra= ce() + * and rcu_read_unlock_trace(). + * + * See the description of call_rcu() for more detailed information on + * memory ordering guarantees. + */ +static inline void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback= _t func) +{ + call_srcu(&rcu_tasks_trace_srcu_struct, rhp, func); +} + +/** + * synchronize_rcu_tasks_trace - wait for a trace rcu-tasks grace period + * + * Control will return to the caller some time after a trace rcu-tasks + * grace period has elapsed, in other words after all currently executing + * trace rcu-tasks read-side critical sections have elapsed. These read-si= de + * critical sections are delimited by calls to rcu_read_lock_trace() + * and rcu_read_unlock_trace(). + * + * This is a very specialized primitive, intended only for a few uses in + * tracing and other situations requiring manipulation of function preambl= es + * and profiling hooks. The synchronize_rcu_tasks_trace() function is not + * (yet) intended for heavy use from multiple CPUs. + * + * See the description of synchronize_rcu() for more detailed information + * on memory ordering guarantees. + */ +static inline void synchronize_rcu_tasks_trace(void) +{ + synchronize_srcu(&rcu_tasks_trace_srcu_struct); } =20 -void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func); -void synchronize_rcu_tasks_trace(void); -void rcu_barrier_tasks_trace(void); +/** + * rcu_barrier_tasks_trace - Wait for in-flight call_rcu_tasks_trace() cal= lbacks. + * + * Note that rcu_barrier_tasks_trace() is not obligated to actually wait, + * for example, if there are no pending callbacks. + */ +static inline void rcu_barrier_tasks_trace(void) +{ + srcu_barrier(&rcu_tasks_trace_srcu_struct); +} + +// Placeholders to enable stepwise transition. +void rcu_tasks_trace_get_gp_data(int *flags, unsigned long *gp_seq); +void __init rcu_tasks_trace_suppress_unused(void); struct task_struct *get_rcu_tasks_trace_gp_kthread(void); + #else /* * The BPF JIT forms these addresses even when it doesn't call these diff --git a/include/linux/sched.h b/include/linux/sched.h index 2b272382673d62..89d3646155525f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -939,6 +939,7 @@ struct task_struct { =20 #ifdef CONFIG_TASKS_TRACE_RCU int trc_reader_nesting; + struct srcu_ctr __percpu *trc_reader_scp; int trc_ipi_to_cpu; union rcu_special trc_reader_special; struct list_head trc_holdout_list; diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c index e3b64a5e0ec7e1..3450c3751ef7ad 100644 --- a/kernel/rcu/srcutiny.c +++ b/kernel/rcu/srcutiny.c @@ -106,15 +106,15 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int = idx) newval =3D READ_ONCE(ssp->srcu_lock_nesting[idx]) - 1; WRITE_ONCE(ssp->srcu_lock_nesting[idx], newval); preempt_enable(); - if (!newval && READ_ONCE(ssp->srcu_gp_waiting) && in_task()) + if (!newval && READ_ONCE(ssp->srcu_gp_waiting) && in_task() && !irqs_disa= bled()) swake_up_one(&ssp->srcu_wq); } EXPORT_SYMBOL_GPL(__srcu_read_unlock); =20 /* * Workqueue handler to drive one grace period and invoke any callbacks - * that become ready as a result. Single-CPU and !PREEMPTION operation - * means that we get away with murder on synchronization. ;-) + * that become ready as a result. Single-CPU operation and preemption + * disabling mean that we get away with murder on synchronization. ;-) */ void srcu_drive_gp(struct work_struct *wp) { @@ -141,7 +141,12 @@ void srcu_drive_gp(struct work_struct *wp) WRITE_ONCE(ssp->srcu_idx, ssp->srcu_idx + 1); WRITE_ONCE(ssp->srcu_gp_waiting, true); /* srcu_read_unlock() wakes! */ preempt_enable(); - swait_event_exclusive(ssp->srcu_wq, !READ_ONCE(ssp->srcu_lock_nesting[idx= ])); + do { + // Deadlock issues prevent __srcu_read_unlock() from + // doing an unconditional wakeup, so polling is required. + swait_event_timeout_exclusive(ssp->srcu_wq, + !READ_ONCE(ssp->srcu_lock_nesting[idx]), HZ / 10); + } while (READ_ONCE(ssp->srcu_lock_nesting[idx])); preempt_disable(); // Needed for PREEMPT_LAZY WRITE_ONCE(ssp->srcu_gp_waiting, false); /* srcu_read_unlock() cheap. */ WRITE_ONCE(ssp->srcu_idx, ssp->srcu_idx + 1); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 2dc044fd126eb0..418fa242cf0288 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -718,7 +718,6 @@ static void __init rcu_tasks_bootup_oddness(void) #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ } =20 - /* Dump out rcutorture-relevant state common to all RCU-tasks flavors. */ static void show_rcu_tasks_generic_gp_kthread(struct rcu_tasks *rtp, char = *s) { @@ -803,7 +802,7 @@ static void rcu_tasks_torture_stats_print_generic(struc= t rcu_tasks *rtp, char *t =20 static void exit_tasks_rcu_finish_trace(struct task_struct *t); =20 -#if defined(CONFIG_TASKS_RCU) || defined(CONFIG_TASKS_TRACE_RCU) +#if defined(CONFIG_TASKS_RCU) =20 //////////////////////////////////////////////////////////////////////// // @@ -898,7 +897,7 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) rtp->postgp_func(rtp); } =20 -#endif /* #if defined(CONFIG_TASKS_RCU) || defined(CONFIG_TASKS_TRACE_RCU)= */ +#endif /* #if defined(CONFIG_TASKS_RCU) */ =20 #ifdef CONFIG_TASKS_RCU =20 @@ -1453,94 +1452,23 @@ EXPORT_SYMBOL_GPL(rcu_tasks_rude_get_gp_data); // // Tracing variant of Tasks RCU. This variant is designed to be used // to protect tracing hooks, including those of BPF. This variant -// therefore: -// -// 1. Has explicit read-side markers to allow finite grace periods -// in the face of in-kernel loops for PREEMPT=3Dn builds. -// -// 2. Protects code in the idle loop, exception entry/exit, and -// CPU-hotplug code paths, similar to the capabilities of SRCU. -// -// 3. Avoids expensive read-side instructions, having overhead similar -// to that of Preemptible RCU. -// -// There are of course downsides. For example, the grace-period code -// can send IPIs to CPUs, even when those CPUs are in the idle loop or -// in nohz_full userspace. If needed, these downsides can be at least -// partially remedied. -// -// Perhaps most important, this variant of RCU does not affect the vanilla -// flavors, rcu_preempt and rcu_sched. The fact that RCU Tasks Trace -// readers can operate from idle, offline, and exception entry/exit in no -// way allows rcu_preempt and rcu_sched readers to also do so. -// -// The implementation uses rcu_tasks_wait_gp(), which relies on function -// pointers in the rcu_tasks structure. The rcu_spawn_tasks_trace_kthread= () -// function sets these function pointers up so that rcu_tasks_wait_gp() -// invokes these functions in this order: -// -// rcu_tasks_trace_pregp_step(): -// Disables CPU hotplug, adds all currently executing tasks to the -// holdout list, then checks the state of all tasks that blocked -// or were preempted within their current RCU Tasks Trace read-side -// critical section, adding them to the holdout list if appropriate. -// Finally, this function re-enables CPU hotplug. -// The ->pertask_func() pointer is NULL, so there is no per-task processin= g. -// rcu_tasks_trace_postscan(): -// Invokes synchronize_rcu() to wait for late-stage exiting tasks -// to finish exiting. -// check_all_holdout_tasks_trace(), repeatedly until holdout list is empty: -// Scans the holdout list, attempting to identify a quiescent state -// for each task on the list. If there is a quiescent state, the -// corresponding task is removed from the holdout list. Once this -// list is empty, the grace period has completed. -// rcu_tasks_trace_postgp(): -// Provides the needed full memory barrier and does debug checks. -// -// The exit_tasks_rcu_finish_trace() synchronizes with exiting tasks. -// -// Pre-grace-period update-side code is ordered before the grace period -// via the ->cbs_lock and barriers in rcu_tasks_kthread(). Pre-grace-peri= od -// read-side code is ordered before the grace period by atomic operations -// on .b.need_qs flag of each task involved in this process, or by schedul= er -// context-switch ordering (for locked-down non-running readers). - -// The lockdep state must be outside of #ifdef to be useful. -#ifdef CONFIG_DEBUG_LOCK_ALLOC -static struct lock_class_key rcu_lock_trace_key; -struct lockdep_map rcu_trace_lock_map =3D - STATIC_LOCKDEP_MAP_INIT("rcu_read_lock_trace", &rcu_lock_trace_key); -EXPORT_SYMBOL_GPL(rcu_trace_lock_map); -#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ +// is implemented via a straightforward mapping onto SRCU-fast. =20 #ifdef CONFIG_TASKS_TRACE_RCU =20 -// Record outstanding IPIs to each CPU. No point in sending two... -static DEFINE_PER_CPU(bool, trc_ipi_to_cpu); - -// The number of detections of task quiescent state relying on -// heavyweight readers executing explicit memory barriers. -static unsigned long n_heavy_reader_attempts; -static unsigned long n_heavy_reader_updates; -static unsigned long n_heavy_reader_ofl_updates; -static unsigned long n_trc_holdouts; +DEFINE_SRCU(rcu_tasks_trace_srcu_struct); +EXPORT_SYMBOL_GPL(rcu_tasks_trace_srcu_struct); =20 -void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func); -DEFINE_RCU_TASKS(rcu_tasks_trace, rcu_tasks_wait_gp, call_rcu_tasks_trace, - "RCU Tasks Trace"); - -/* Load from ->trc_reader_special.b.need_qs with proper ordering. */ -static u8 rcu_ld_need_qs(struct task_struct *t) +// Placeholder to suppress build errors through transition period. +void __init rcu_tasks_trace_suppress_unused(void) { - smp_mb(); // Enforce full grace-period ordering. - return smp_load_acquire(&t->trc_reader_special.b.need_qs); -} - -/* Store to ->trc_reader_special.b.need_qs with proper ordering. */ -static void rcu_st_need_qs(struct task_struct *t, u8 v) -{ - smp_store_release(&t->trc_reader_special.b.need_qs, v); - smp_mb(); // Enforce full grace-period ordering. + show_rcu_tasks_generic_gp_kthread(NULL, NULL); + rcu_spawn_tasks_kthread_generic(NULL); + synchronize_rcu_tasks_generic(NULL); + call_rcu_tasks_generic(NULL, NULL, NULL); + call_rcu_tasks_iw_wakeup(NULL); + cblist_init_generic(NULL); + rcu_tasks_torture_stats_print_generic(NULL, NULL, NULL, NULL); } =20 /* @@ -1555,321 +1483,12 @@ u8 rcu_trc_cmpxchg_need_qs(struct task_struct *t, = u8 old, u8 new) } EXPORT_SYMBOL_GPL(rcu_trc_cmpxchg_need_qs); =20 -/* - * If we are the last reader, signal the grace-period kthread. - * Also remove from the per-CPU list of blocked tasks. - */ -void rcu_read_unlock_trace_special(struct task_struct *t) -{ - unsigned long flags; - struct rcu_tasks_percpu *rtpcp; - union rcu_special trs; - - // Open-coded full-word version of rcu_ld_need_qs(). - smp_mb(); // Enforce full grace-period ordering. - trs =3D smp_load_acquire(&t->trc_reader_special); - - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) && t->trc_reader_special.b= .need_mb) - smp_mb(); // Pairs with update-side barriers. - // Update .need_qs before ->trc_reader_nesting for irq/NMI handlers. - if (trs.b.need_qs =3D=3D (TRC_NEED_QS_CHECKED | TRC_NEED_QS)) { - u8 result =3D rcu_trc_cmpxchg_need_qs(t, TRC_NEED_QS_CHECKED | TRC_NEED_= QS, - TRC_NEED_QS_CHECKED); - - WARN_ONCE(result !=3D trs.b.need_qs, "%s: result =3D %d", __func__, resu= lt); - } - if (trs.b.blocked) { - rtpcp =3D per_cpu_ptr(rcu_tasks_trace.rtpcpu, t->trc_blkd_cpu); - raw_spin_lock_irqsave_rcu_node(rtpcp, flags); - list_del_init(&t->trc_blkd_node); - WRITE_ONCE(t->trc_reader_special.b.blocked, false); - raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); - } - WRITE_ONCE(t->trc_reader_nesting, 0); -} -EXPORT_SYMBOL_GPL(rcu_read_unlock_trace_special); - /* Add a newly blocked reader task to its CPU's list. */ void rcu_tasks_trace_qs_blkd(struct task_struct *t) { - unsigned long flags; - struct rcu_tasks_percpu *rtpcp; - - local_irq_save(flags); - rtpcp =3D this_cpu_ptr(rcu_tasks_trace.rtpcpu); - raw_spin_lock_rcu_node(rtpcp); // irqs already disabled - t->trc_blkd_cpu =3D smp_processor_id(); - if (!rtpcp->rtp_blkd_tasks.next) - INIT_LIST_HEAD(&rtpcp->rtp_blkd_tasks); - list_add(&t->trc_blkd_node, &rtpcp->rtp_blkd_tasks); - WRITE_ONCE(t->trc_reader_special.b.blocked, true); - raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); } EXPORT_SYMBOL_GPL(rcu_tasks_trace_qs_blkd); =20 -/* Add a task to the holdout list, if it is not already on the list. */ -static void trc_add_holdout(struct task_struct *t, struct list_head *bhp) -{ - if (list_empty(&t->trc_holdout_list)) { - get_task_struct(t); - list_add(&t->trc_holdout_list, bhp); - n_trc_holdouts++; - } -} - -/* Remove a task from the holdout list, if it is in fact present. */ -static void trc_del_holdout(struct task_struct *t) -{ - if (!list_empty(&t->trc_holdout_list)) { - list_del_init(&t->trc_holdout_list); - put_task_struct(t); - n_trc_holdouts--; - } -} - -/* IPI handler to check task state. */ -static void trc_read_check_handler(void *t_in) -{ - int nesting; - struct task_struct *t =3D current; - struct task_struct *texp =3D t_in; - - // If the task is no longer running on this CPU, leave. - if (unlikely(texp !=3D t)) - goto reset_ipi; // Already on holdout list, so will check later. - - // If the task is not in a read-side critical section, and - // if this is the last reader, awaken the grace-period kthread. - nesting =3D READ_ONCE(t->trc_reader_nesting); - if (likely(!nesting)) { - rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS_CHECKED); - goto reset_ipi; - } - // If we are racing with an rcu_read_unlock_trace(), try again later. - if (unlikely(nesting < 0)) - goto reset_ipi; - - // Get here if the task is in a read-side critical section. - // Set its state so that it will update state for the grace-period - // kthread upon exit from that critical section. - rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS | TRC_NEED_QS_CHECKED); - -reset_ipi: - // Allow future IPIs to be sent on CPU and for task. - // Also order this IPI handler against any later manipulations of - // the intended task. - smp_store_release(per_cpu_ptr(&trc_ipi_to_cpu, smp_processor_id()), false= ); // ^^^ - smp_store_release(&texp->trc_ipi_to_cpu, -1); // ^^^ -} - -/* Callback function for scheduler to check locked-down task. */ -static int trc_inspect_reader(struct task_struct *t, void *bhp_in) -{ - struct list_head *bhp =3D bhp_in; - int cpu =3D task_cpu(t); - int nesting; - bool ofl =3D cpu_is_offline(cpu); - - if (task_curr(t) && !ofl) { - // If no chance of heavyweight readers, do it the hard way. - if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) - return -EINVAL; - - // If heavyweight readers are enabled on the remote task, - // we can inspect its state despite its currently running. - // However, we cannot safely change its state. - n_heavy_reader_attempts++; - // Check for "running" idle tasks on offline CPUs. - if (!rcu_watching_zero_in_eqs(cpu, &t->trc_reader_nesting)) - return -EINVAL; // No quiescent state, do it the hard way. - n_heavy_reader_updates++; - nesting =3D 0; - } else { - // The task is not running, so C-language access is safe. - nesting =3D t->trc_reader_nesting; - WARN_ON_ONCE(ofl && task_curr(t) && (t !=3D idle_task(task_cpu(t)))); - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) && ofl) - n_heavy_reader_ofl_updates++; - } - - // If not exiting a read-side critical section, mark as checked - // so that the grace-period kthread will remove it from the - // holdout list. - if (!nesting) { - rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS_CHECKED); - return 0; // In QS, so done. - } - if (nesting < 0) - return -EINVAL; // Reader transitioning, try again later. - - // The task is in a read-side critical section, so set up its - // state so that it will update state upon exit from that critical - // section. - if (!rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS | TRC_NEED_QS_CHECKED)) - trc_add_holdout(t, bhp); - return 0; -} - -/* Attempt to extract the state for the specified task. */ -static void trc_wait_for_one_reader(struct task_struct *t, - struct list_head *bhp) -{ - int cpu; - - // If a previous IPI is still in flight, let it complete. - if (smp_load_acquire(&t->trc_ipi_to_cpu) !=3D -1) // Order IPI - return; - - // The current task had better be in a quiescent state. - if (t =3D=3D current) { - rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS_CHECKED); - WARN_ON_ONCE(READ_ONCE(t->trc_reader_nesting)); - return; - } - - // Attempt to nail down the task for inspection. - get_task_struct(t); - if (!task_call_func(t, trc_inspect_reader, bhp)) { - put_task_struct(t); - return; - } - put_task_struct(t); - - // If this task is not yet on the holdout list, then we are in - // an RCU read-side critical section. Otherwise, the invocation of - // trc_add_holdout() that added it to the list did the necessary - // get_task_struct(). Either way, the task cannot be freed out - // from under this code. - - // If currently running, send an IPI, either way, add to list. - trc_add_holdout(t, bhp); - if (task_curr(t) && - time_after(jiffies + 1, rcu_tasks_trace.gp_start + rcu_task_ipi_delay= )) { - // The task is currently running, so try IPIing it. - cpu =3D task_cpu(t); - - // If there is already an IPI outstanding, let it happen. - if (per_cpu(trc_ipi_to_cpu, cpu) || t->trc_ipi_to_cpu >=3D 0) - return; - - per_cpu(trc_ipi_to_cpu, cpu) =3D true; - t->trc_ipi_to_cpu =3D cpu; - rcu_tasks_trace.n_ipis++; - if (smp_call_function_single(cpu, trc_read_check_handler, t, 0)) { - // Just in case there is some other reason for - // failure than the target CPU being offline. - WARN_ONCE(1, "%s(): smp_call_function_single() failed for CPU: %d\n", - __func__, cpu); - rcu_tasks_trace.n_ipis_fails++; - per_cpu(trc_ipi_to_cpu, cpu) =3D false; - t->trc_ipi_to_cpu =3D -1; - } - } -} - -/* - * Initialize for first-round processing for the specified task. - * Return false if task is NULL or already taken care of, true otherwise. - */ -static bool rcu_tasks_trace_pertask_prep(struct task_struct *t, bool notse= lf) -{ - // During early boot when there is only the one boot CPU, there - // is no idle task for the other CPUs. Also, the grace-period - // kthread is always in a quiescent state. In addition, just return - // if this task is already on the list. - if (unlikely(t =3D=3D NULL) || (t =3D=3D current && notself) || !list_emp= ty(&t->trc_holdout_list)) - return false; - - rcu_st_need_qs(t, 0); - t->trc_ipi_to_cpu =3D -1; - return true; -} - -/* Do first-round processing for the specified task. */ -static void rcu_tasks_trace_pertask(struct task_struct *t, struct list_hea= d *hop) -{ - if (rcu_tasks_trace_pertask_prep(t, true)) - trc_wait_for_one_reader(t, hop); -} - -/* Initialize for a new RCU-tasks-trace grace period. */ -static void rcu_tasks_trace_pregp_step(struct list_head *hop) -{ - LIST_HEAD(blkd_tasks); - int cpu; - unsigned long flags; - struct rcu_tasks_percpu *rtpcp; - struct task_struct *t; - - // There shouldn't be any old IPIs, but... - for_each_possible_cpu(cpu) - WARN_ON_ONCE(per_cpu(trc_ipi_to_cpu, cpu)); - - // Disable CPU hotplug across the CPU scan for the benefit of - // any IPIs that might be needed. This also waits for all readers - // in CPU-hotplug code paths. - cpus_read_lock(); - - // These rcu_tasks_trace_pertask_prep() calls are serialized to - // allow safe access to the hop list. - for_each_online_cpu(cpu) { - rcu_read_lock(); - // Note that cpu_curr_snapshot() picks up the target - // CPU's current task while its runqueue is locked with - // an smp_mb__after_spinlock(). This ensures that either - // the grace-period kthread will see that task's read-side - // critical section or the task will see the updater's pre-GP - // accesses. The trailing smp_mb() in cpu_curr_snapshot() - // does not currently play a role other than simplify - // that function's ordering semantics. If these simplified - // ordering semantics continue to be redundant, that smp_mb() - // might be removed. - t =3D cpu_curr_snapshot(cpu); - if (rcu_tasks_trace_pertask_prep(t, true)) - trc_add_holdout(t, hop); - rcu_read_unlock(); - cond_resched_tasks_rcu_qs(); - } - - // Only after all running tasks have been accounted for is it - // safe to take care of the tasks that have blocked within their - // current RCU tasks trace read-side critical section. - for_each_possible_cpu(cpu) { - rtpcp =3D per_cpu_ptr(rcu_tasks_trace.rtpcpu, cpu); - raw_spin_lock_irqsave_rcu_node(rtpcp, flags); - list_splice_init(&rtpcp->rtp_blkd_tasks, &blkd_tasks); - while (!list_empty(&blkd_tasks)) { - rcu_read_lock(); - t =3D list_first_entry(&blkd_tasks, struct task_struct, trc_blkd_node); - list_del_init(&t->trc_blkd_node); - list_add(&t->trc_blkd_node, &rtpcp->rtp_blkd_tasks); - raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); - rcu_tasks_trace_pertask(t, hop); - rcu_read_unlock(); - raw_spin_lock_irqsave_rcu_node(rtpcp, flags); - } - raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); - cond_resched_tasks_rcu_qs(); - } - - // Re-enable CPU hotplug now that the holdout list is populated. - cpus_read_unlock(); -} - -/* - * Do intermediate processing between task and holdout scans. - */ -static void rcu_tasks_trace_postscan(struct list_head *hop) -{ - // Wait for late-stage exiting tasks to finish exiting. - // These might have passed the call to exit_tasks_rcu_finish(). - - // If you remove the following line, update rcu_trace_implies_rcu_gp()!!! - synchronize_rcu(); - // Any tasks that exit after this point will set - // TRC_NEED_QS_CHECKED in ->trc_reader_special.b.need_qs. -} - /* Communicate task state back to the RCU tasks trace stall warning reques= t. */ struct trc_stall_chk_rdr { int nesting; @@ -1877,241 +1496,39 @@ struct trc_stall_chk_rdr { u8 needqs; }; =20 -static int trc_check_slow_task(struct task_struct *t, void *arg) -{ - struct trc_stall_chk_rdr *trc_rdrp =3D arg; - - if (task_curr(t) && cpu_online(task_cpu(t))) - return false; // It is running, so decline to inspect it. - trc_rdrp->nesting =3D READ_ONCE(t->trc_reader_nesting); - trc_rdrp->ipi_to_cpu =3D READ_ONCE(t->trc_ipi_to_cpu); - trc_rdrp->needqs =3D rcu_ld_need_qs(t); - return true; -} - -/* Show the state of a task stalling the current RCU tasks trace GP. */ -static void show_stalled_task_trace(struct task_struct *t, bool *firstrepo= rt) -{ - int cpu; - struct trc_stall_chk_rdr trc_rdr; - bool is_idle_tsk =3D is_idle_task(t); - - if (*firstreport) { - pr_err("INFO: rcu_tasks_trace detected stalls on tasks:\n"); - *firstreport =3D false; - } - cpu =3D task_cpu(t); - if (!task_call_func(t, trc_check_slow_task, &trc_rdr)) - pr_alert("P%d: %c%c\n", - t->pid, - ".I"[t->trc_ipi_to_cpu >=3D 0], - ".i"[is_idle_tsk]); - else - pr_alert("P%d: %c%c%c%c nesting: %d%c%c cpu: %d%s\n", - t->pid, - ".I"[trc_rdr.ipi_to_cpu >=3D 0], - ".i"[is_idle_tsk], - ".N"[cpu >=3D 0 && tick_nohz_full_cpu(cpu)], - ".B"[!!data_race(t->trc_reader_special.b.blocked)], - trc_rdr.nesting, - " !CN"[trc_rdr.needqs & 0x3], - " ?"[trc_rdr.needqs > 0x3], - cpu, cpu_online(cpu) ? "" : "(offline)"); - sched_show_task(t); -} - -/* List stalled IPIs for RCU tasks trace. */ -static void show_stalled_ipi_trace(void) -{ - int cpu; - - for_each_possible_cpu(cpu) - if (per_cpu(trc_ipi_to_cpu, cpu)) - pr_alert("\tIPI outstanding to CPU %d\n", cpu); -} - -/* Do one scan of the holdout list. */ -static void check_all_holdout_tasks_trace(struct list_head *hop, - bool needreport, bool *firstreport) -{ - struct task_struct *g, *t; - - // Disable CPU hotplug across the holdout list scan for IPIs. - cpus_read_lock(); - - list_for_each_entry_safe(t, g, hop, trc_holdout_list) { - // If safe and needed, try to check the current task. - if (READ_ONCE(t->trc_ipi_to_cpu) =3D=3D -1 && - !(rcu_ld_need_qs(t) & TRC_NEED_QS_CHECKED)) - trc_wait_for_one_reader(t, hop); - - // If check succeeded, remove this task from the list. - if (smp_load_acquire(&t->trc_ipi_to_cpu) =3D=3D -1 && - rcu_ld_need_qs(t) =3D=3D TRC_NEED_QS_CHECKED) - trc_del_holdout(t); - else if (needreport) - show_stalled_task_trace(t, firstreport); - cond_resched_tasks_rcu_qs(); - } - - // Re-enable CPU hotplug now that the holdout list scan has completed. - cpus_read_unlock(); - - if (needreport) { - if (*firstreport) - pr_err("INFO: rcu_tasks_trace detected stalls? (Late IPI?)\n"); - show_stalled_ipi_trace(); - } -} - -static void rcu_tasks_trace_empty_fn(void *unused) -{ -} - -/* Wait for grace period to complete and provide ordering. */ -static void rcu_tasks_trace_postgp(struct rcu_tasks *rtp) -{ - int cpu; - - // Wait for any lingering IPI handlers to complete. Note that - // if a CPU has gone offline or transitioned to userspace in the - // meantime, all IPI handlers should have been drained beforehand. - // Yes, this assumes that CPUs process IPIs in order. If that ever - // changes, there will need to be a recheck and/or timed wait. - for_each_online_cpu(cpu) - if (WARN_ON_ONCE(smp_load_acquire(per_cpu_ptr(&trc_ipi_to_cpu, cpu)))) - smp_call_function_single(cpu, rcu_tasks_trace_empty_fn, NULL, 1); - - smp_mb(); // Caller's code must be ordered after wakeup. - // Pairs with pretty much every ordering primitive. -} - /* Report any needed quiescent state for this exiting task. */ static void exit_tasks_rcu_finish_trace(struct task_struct *t) { - union rcu_special trs =3D READ_ONCE(t->trc_reader_special); - - rcu_trc_cmpxchg_need_qs(t, 0, TRC_NEED_QS_CHECKED); - WARN_ON_ONCE(READ_ONCE(t->trc_reader_nesting)); - if (WARN_ON_ONCE(rcu_ld_need_qs(t) & TRC_NEED_QS || trs.b.blocked)) - rcu_read_unlock_trace_special(t); - else - WRITE_ONCE(t->trc_reader_nesting, 0); -} - -/** - * call_rcu_tasks_trace() - Queue a callback trace task-based grace period - * @rhp: structure to be used for queueing the RCU updates. - * @func: actual callback function to be invoked after the grace period - * - * The callback function will be invoked some time after a trace rcu-tasks - * grace period elapses, in other words after all currently executing - * trace rcu-tasks read-side critical sections have completed. These - * read-side critical sections are delimited by calls to rcu_read_lock_tra= ce() - * and rcu_read_unlock_trace(). - * - * See the description of call_rcu() for more detailed information on - * memory ordering guarantees. - */ -void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func) -{ - call_rcu_tasks_generic(rhp, func, &rcu_tasks_trace); -} -EXPORT_SYMBOL_GPL(call_rcu_tasks_trace); - -/** - * synchronize_rcu_tasks_trace - wait for a trace rcu-tasks grace period - * - * Control will return to the caller some time after a trace rcu-tasks - * grace period has elapsed, in other words after all currently executing - * trace rcu-tasks read-side critical sections have elapsed. These read-si= de - * critical sections are delimited by calls to rcu_read_lock_trace() - * and rcu_read_unlock_trace(). - * - * This is a very specialized primitive, intended only for a few uses in - * tracing and other situations requiring manipulation of function preambl= es - * and profiling hooks. The synchronize_rcu_tasks_trace() function is not - * (yet) intended for heavy use from multiple CPUs. - * - * See the description of synchronize_rcu() for more detailed information - * on memory ordering guarantees. - */ -void synchronize_rcu_tasks_trace(void) -{ - RCU_LOCKDEP_WARN(lock_is_held(&rcu_trace_lock_map), "Illegal synchronize_= rcu_tasks_trace() in RCU Tasks Trace read-side critical section"); - synchronize_rcu_tasks_generic(&rcu_tasks_trace); } -EXPORT_SYMBOL_GPL(synchronize_rcu_tasks_trace); - -/** - * rcu_barrier_tasks_trace - Wait for in-flight call_rcu_tasks_trace() cal= lbacks. - * - * Although the current implementation is guaranteed to wait, it is not - * obligated to, for example, if there are no pending callbacks. - */ -void rcu_barrier_tasks_trace(void) -{ - rcu_barrier_tasks_generic(&rcu_tasks_trace); -} -EXPORT_SYMBOL_GPL(rcu_barrier_tasks_trace); =20 int rcu_tasks_trace_lazy_ms =3D -1; module_param(rcu_tasks_trace_lazy_ms, int, 0444); =20 static int __init rcu_spawn_tasks_trace_kthread(void) { - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) { - rcu_tasks_trace.gp_sleep =3D HZ / 10; - rcu_tasks_trace.init_fract =3D HZ / 10; - } else { - rcu_tasks_trace.gp_sleep =3D HZ / 200; - if (rcu_tasks_trace.gp_sleep <=3D 0) - rcu_tasks_trace.gp_sleep =3D 1; - rcu_tasks_trace.init_fract =3D HZ / 200; - if (rcu_tasks_trace.init_fract <=3D 0) - rcu_tasks_trace.init_fract =3D 1; - } - if (rcu_tasks_trace_lazy_ms >=3D 0) - rcu_tasks_trace.lazy_jiffies =3D msecs_to_jiffies(rcu_tasks_trace_lazy_m= s); - rcu_tasks_trace.pregp_func =3D rcu_tasks_trace_pregp_step; - rcu_tasks_trace.postscan_func =3D rcu_tasks_trace_postscan; - rcu_tasks_trace.holdouts_func =3D check_all_holdout_tasks_trace; - rcu_tasks_trace.postgp_func =3D rcu_tasks_trace_postgp; - rcu_spawn_tasks_kthread_generic(&rcu_tasks_trace); return 0; } =20 #if !defined(CONFIG_TINY_RCU) void show_rcu_tasks_trace_gp_kthread(void) { - char buf[64]; - - snprintf(buf, sizeof(buf), "N%lu h:%lu/%lu/%lu", - data_race(n_trc_holdouts), - data_race(n_heavy_reader_ofl_updates), - data_race(n_heavy_reader_updates), - data_race(n_heavy_reader_attempts)); - show_rcu_tasks_generic_gp_kthread(&rcu_tasks_trace, buf); } EXPORT_SYMBOL_GPL(show_rcu_tasks_trace_gp_kthread); =20 void rcu_tasks_trace_torture_stats_print(char *tt, char *tf) { - rcu_tasks_torture_stats_print_generic(&rcu_tasks_trace, tt, tf, ""); } EXPORT_SYMBOL_GPL(rcu_tasks_trace_torture_stats_print); #endif // !defined(CONFIG_TINY_RCU) =20 struct task_struct *get_rcu_tasks_trace_gp_kthread(void) { - return rcu_tasks_trace.kthread_ptr; + return NULL; } EXPORT_SYMBOL_GPL(get_rcu_tasks_trace_gp_kthread); =20 void rcu_tasks_trace_get_gp_data(int *flags, unsigned long *gp_seq) { - *flags =3D 0; - *gp_seq =3D rcu_seq_current(&rcu_tasks_trace.tasks_gp_seq); } EXPORT_SYMBOL_GPL(rcu_tasks_trace_get_gp_data); =20 @@ -2251,10 +1668,6 @@ void __init tasks_cblist_init_generic(void) #ifdef CONFIG_TASKS_RUDE_RCU cblist_init_generic(&rcu_tasks_rude); #endif - -#ifdef CONFIG_TASKS_TRACE_RCU - cblist_init_generic(&rcu_tasks_trace); -#endif } =20 static int __init rcu_init_tasks_generic(void) --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C1FD230BDB; Tue, 23 Sep 2025 14:21:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637296; cv=none; b=b5GKXQ2KP1kDxCewVvqsBt1Eyy8FnKtPFhpC7DvRjHUhVk1ZTcOFYUQ/mTlJ3g+JhjsoXoz6+YCUe0MoZm3zPZs3bNcDJT/7PnsauNmpS+l6Zm0tFE0va0Qln50Y8oUuHckyEQAqqJRutLv6iyDl0G503j8kK4FUTlFPXyKISkk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637296; c=relaxed/simple; bh=FCc3TPZVw9grK2mrF1tFguXdehf44jNav5ZIKxFutYw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Hl/NLF4v5h5EvjZVtdfbxGTwHk5+nqQbFGFbA4LG2kwg2+yaKar5Q2ZsbNmYX7XmNV0/Do7EbrJ9f23C/+fwlx0RhMnwvuae7x/BLA1TOA58JAkQHTlO4b2r+/k4PjbmWUkXtMb//MrLBlh6B7n/AB6oUQ2zzQ4bg6VemwGJeaA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XOWiwj2f; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XOWiwj2f" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DC6E6C16AAE; Tue, 23 Sep 2025 14:21:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637296; bh=FCc3TPZVw9grK2mrF1tFguXdehf44jNav5ZIKxFutYw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XOWiwj2fy5+SAHmDwfpQz0qEuhN9OGAyPIICL4OsBCybJUXFQQbzFkOGjlpT2A5mo vrXHUNQk/pFYoiVjV3NTbClHy1YLbx+YLiC9lrfKkmJgIFMBN+chEnBi0l+Ax1846t QsbSyB9Q2UVHKiUHagwX0dgovj/iM3r4bhq52wDV/ENp5L45bVIhYYddLt6OQHmZD7 BzxP9S8ecYbyuVBz7Qz5O24iOqRowwOBwJXFpFkf1//tCQDxeKWlhVNLPZL9Fqr7QE be98X9WOx9A2Aoc1J22w2rJHHIGgQmWEgycf/aubjgUOlG+bwVZJS+dqTiNbUxsFqt nm+rh38NmISPg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id B91F0CE0EDC; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 02/34] rcu: Remove unused ->trc_ipi_to_cpu and ->trc_blkd_cpu from task_struct Date: Tue, 23 Sep 2025 07:20:04 -0700 Message-Id: <20250923142036.112290-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that RCU Tasks Trace has been re-implemented in terms of SRCU-fast, the ->trc_ipi_to_cpu and ->trc_blkd_cpu task_struct fields are no longer used. This commit therefore removes them. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/sched.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 89d3646155525f..47bd062b21c2bc 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -940,11 +940,9 @@ struct task_struct { #ifdef CONFIG_TASKS_TRACE_RCU int trc_reader_nesting; struct srcu_ctr __percpu *trc_reader_scp; - int trc_ipi_to_cpu; union rcu_special trc_reader_special; struct list_head trc_holdout_list; struct list_head trc_blkd_node; - int trc_blkd_cpu; #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ =20 struct sched_info sched_info; --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A926A22A7F1; Tue, 23 Sep 2025 14:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; cv=none; b=r/YLGEYnHofrgsQLyM5TlIMYKOjFSToIAiZjFgqNFRoiAhhkvcjY2LOPvh8Qk9VsL6lNpOBg7WHA4kyOLWD2/W6eiLULVi4+SyxR+P0A8TxvbmoARpgirn9o0VIhylZUnUet6b0oT281EqYSDyFtbegmwgHYA1w13vbvTlq4vlU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; c=relaxed/simple; bh=xsMVIazrHCxbo8kiV/02R0kxVkmtf66xKO9RsbF5hME=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=b2r2fHYTi+IBDAOn2OHdzkzNx9Y8F9y1vQJabXU8fty4cJrI1r9bZF0Q348uZ11OodDPfHH9qt3iZTzoEICVEwf8wke04aGl3D15L9fk1hQXgK6EQ46F6dHofrMuI5F43HOZb8MZrrMnz5KjdUlDB5ipXar+M+7Gc9P5CM5vGVw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kA0ivHQN; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kA0ivHQN" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CFCCBC116D0; Tue, 23 Sep 2025 14:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637302; bh=xsMVIazrHCxbo8kiV/02R0kxVkmtf66xKO9RsbF5hME=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kA0ivHQNN+qep6DSP67RPLXjz/jEtlKdCPLlxaD6viNJUK68p6t45r78aSrTs97pH R6NU/YnAMBNjhb0C3vydjuGb+/Du7zOFF68ywSOs8XllVQs9bDNHbw4dJRxREMIYf5 vCzfMBeexuz+5uagaJT8Pobh05vp7XRiuF+MKzrsh7iRSRai4Elx7trxZeyzKU/hKm g7eD7FplcufHfRwvOhn+TXvSqYp1/tvsbANIFIaE7qdARjdGn6yvtYg2Eq0YugrkJn WIXu1KqwDkRwBQ6J/WooW2xSh16bD+YQrAT5qx9TC+D7l/c12UTu2ruyN0gh/CGOJh DKG4daNytfSsw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id BC08ECE0F21; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 03/34] rcu: Remove ->trc_blkd_node from task_struct Date: Tue, 23 Sep 2025 07:20:05 -0700 Message-Id: <20250923142036.112290-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that RCU Tasks Trace has been re-implemented in terms of SRCU-fast, the ->trc_blkd_node task_struct field is only initialized, and never actually used. This commit therefore removes it. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/sched.h | 1 - init/init_task.c | 1 - kernel/fork.c | 1 - 3 files changed, 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 47bd062b21c2bc..3f840cfa941132 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -942,7 +942,6 @@ struct task_struct { struct srcu_ctr __percpu *trc_reader_scp; union rcu_special trc_reader_special; struct list_head trc_holdout_list; - struct list_head trc_blkd_node; #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ =20 struct sched_info sched_info; diff --git a/init/init_task.c b/init/init_task.c index e557f622bd9061..dd6226251689c8 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -167,7 +167,6 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = =3D { .trc_reader_nesting =3D 0, .trc_reader_special.s =3D 0, .trc_holdout_list =3D LIST_HEAD_INIT(init_task.trc_holdout_list), - .trc_blkd_node =3D LIST_HEAD_INIT(init_task.trc_blkd_node), #endif #ifdef CONFIG_CPUSETS .mems_allowed_seq =3D SEQCNT_SPINLOCK_ZERO(init_task.mems_allowed_seq, diff --git a/kernel/fork.c b/kernel/fork.c index af673856499dca..69d34c123b9e5f 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1782,7 +1782,6 @@ static inline void rcu_copy_process(struct task_struc= t *p) p->trc_reader_nesting =3D 0; p->trc_reader_special.s =3D 0; INIT_LIST_HEAD(&p->trc_holdout_list); - INIT_LIST_HEAD(&p->trc_blkd_node); #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ } =20 --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA5EF327A1A; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637314; cv=none; b=egAdWiGu3ZTaWBX5qDDph2Pb8Mkin4vwRPAb4bFuR/efCDBeBV/H6MhEJvRHKZbznahiRD0xTEl+ZRK9VbT6ivMmibbYkl8IwHe/QSIk4eAJ38W2CS/ffutFgW5YgxL5V6KZYIQKSX3YGa7stmCj9ERmZsro7227GeUGqNxef8U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637314; c=relaxed/simple; bh=ZhOe7rNnjZ2YNl4vA1vdnXfrcjpjHHIwSM1yRejM3bY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fGCjID4lnv/6tnIaUiN/tbacgC4Uu88OQemF0T1qCFKMnv8BEsOs8OdV3HBa5rdFsNV8pundXN65AnIs2zP/IOdecA453hoGWBgfkrfAzyuuJFIr/q6PHQ+mBpmeVWgZ5byZPmiLLLSGzRalyUhikl4m7f4by+OYtOxO/fzf3ew= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WMOeIG9B; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WMOeIG9B" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 66C6EC113CF; Tue, 23 Sep 2025 14:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637314; bh=ZhOe7rNnjZ2YNl4vA1vdnXfrcjpjHHIwSM1yRejM3bY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WMOeIG9BRilsBzGHPtTMcJFpJE5CgdGJ4eJlXzk/4XarRfrzuUysdwecFf/GJE2bj BrZe5HZQZ9rwFIQH0DQeJnUa3u61WRuVf+OZdC/omqUFPanI6nFiyCqBpjEoNeF/nV Fc7E5K44JJUHad/7D5sxkY2voarGVL+9MXYlbrFP6xCXgLjC/8ABRNqw4It+yPteHj PKnO0MDlq9lsNaWl6qWY4+VoSyAubeAe8YHp2rk5fPyDqtiop3+0phMIiIlEATHorz l8lxGfB26AH96uKkp1R+wyRpN4E8IGrSWs+/pNLyZ17MS9sV8QL9C5d8NRsbDSJ8EF 3mSZu8ZUtQcHg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id BEC00CE0F71; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 04/34] rcu: Remove ->trc_holdout_list from task_struct Date: Tue, 23 Sep 2025 07:20:06 -0700 Message-Id: <20250923142036.112290-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that RCU Tasks Trace has been re-implemented in terms of SRCU-fast, the ->trc_holdout_list task_struct field is only initialized, and never actually used. This commit therefore removes it. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/sched.h | 1 - init/init_task.c | 1 - kernel/fork.c | 1 - 3 files changed, 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 3f840cfa941132..e62c99e44a5502 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -941,7 +941,6 @@ struct task_struct { int trc_reader_nesting; struct srcu_ctr __percpu *trc_reader_scp; union rcu_special trc_reader_special; - struct list_head trc_holdout_list; #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ =20 struct sched_info sched_info; diff --git a/init/init_task.c b/init/init_task.c index dd6226251689c8..91c37f66ec8e9a 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -166,7 +166,6 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = =3D { #ifdef CONFIG_TASKS_TRACE_RCU .trc_reader_nesting =3D 0, .trc_reader_special.s =3D 0, - .trc_holdout_list =3D LIST_HEAD_INIT(init_task.trc_holdout_list), #endif #ifdef CONFIG_CPUSETS .mems_allowed_seq =3D SEQCNT_SPINLOCK_ZERO(init_task.mems_allowed_seq, diff --git a/kernel/fork.c b/kernel/fork.c index 69d34c123b9e5f..12388e895d2955 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1781,7 +1781,6 @@ static inline void rcu_copy_process(struct task_struc= t *p) #ifdef CONFIG_TASKS_TRACE_RCU p->trc_reader_nesting =3D 0; p->trc_reader_special.s =3D 0; - INIT_LIST_HEAD(&p->trc_holdout_list); #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ } =20 --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1F092652AF; Tue, 23 Sep 2025 14:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; cv=none; b=trCmbb8Z0owHqSvdRtHRj7JN+EmbgJeWoP4w5C3biXoWRHRYyw8EOyNQ7hs0do+tkKNUjhHxgpJZQZsNBHYA7Qv4k+oy1OSp2Q/OQHWclKwUU/p0ArKrw7bGVTgu+1Ocw+v2oatcYsQevakBrXGbc9QLMvL2/1brHE//o9/k++Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; c=relaxed/simple; bh=ywaNuK6aAoHek5VbjiDHoWBoyJGmii7APxujsObI1Hw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=R59ll+h1qupbJrgWNLD8m4vZhp972Vp0xtWwnSIiuOMrpnjGePqxDI75Ca0aEb5VLAJf5rKICaJLVhDBotpKJvfMArNGvcLq5KRyieh6jiT6cRoVXottxyOF/CcIDcepBf5WKlkUSqjcuNggjWQaCYCu1QFNUF3NV97pn2sUC00= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mAWfk/tc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mAWfk/tc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AEBE7C113CF; Tue, 23 Sep 2025 14:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637302; bh=ywaNuK6aAoHek5VbjiDHoWBoyJGmii7APxujsObI1Hw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mAWfk/tcqGU2ZN9qQKIuvRLU0ExfB40CI1/8bxmGjEBXZFJggmySHlRiFHU3UMbQ1 HTrQNF2gBijNApZ/q+bF9A9X6EzRGIrEIplMIpbI1upvoE8Pa2FdHtsgsRKUXCF17w AydEhEu0CxFHj+dRc0wid4MxrP7QQtNStIwtOwlfQSBuwnlOheWmyyyuaZMUVq7b8j ROkH2hzMd7O+TtC34LIMEyEuK5CYQpvefsVW2hUqw4tqw6Cte09XaMvG/ryqJf/CuE dMzV6gO6vnysPvpkgo5UF5vJI1UeG14K570Jo7BSKwdHdHaFB2kjpBWLtX4XhpOV4T TeThG3uKEmisA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C1435CE11D4; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 05/34] rcu: Remove rcu_tasks_trace_qs() and the functions that it calls Date: Tue, 23 Sep 2025 07:20:07 -0700 Message-Id: <20250923142036.112290-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Because SRCU-fast does not need quiescent-state helpers, this commit removes rcu_tasks_trace_qs() and those things that only it uses, including rcu_trc_cmpxchg_need_qs(), TRC_NEED_QS, and TRC_NEED_QS_CHECKED. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/rcupdate.h | 26 +------------------------- kernel/rcu/tasks.h | 12 ------------ 2 files changed, 1 insertion(+), 37 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 120536f4c6eb1d..879525c5764a0c 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -176,35 +176,11 @@ void rcu_tasks_torture_stats_print(char *tt, char *tf= ); # endif =20 # ifdef CONFIG_TASKS_TRACE_RCU -// Bits for ->trc_reader_special.b.need_qs field. -#define TRC_NEED_QS 0x1 // Task needs a quiescent state. -#define TRC_NEED_QS_CHECKED 0x2 // Task has been checked for needing quie= scent state. - -u8 rcu_trc_cmpxchg_need_qs(struct task_struct *t, u8 old, u8 new); void rcu_tasks_trace_qs_blkd(struct task_struct *t); - -# define rcu_tasks_trace_qs(t) \ - do { \ - int ___rttq_nesting =3D READ_ONCE((t)->trc_reader_nesting); \ - \ - if (unlikely(READ_ONCE((t)->trc_reader_special.b.need_qs) =3D=3D TRC_NEE= D_QS) && \ - likely(!___rttq_nesting)) { \ - rcu_trc_cmpxchg_need_qs((t), TRC_NEED_QS, TRC_NEED_QS_CHECKED); \ - } else if (___rttq_nesting && ___rttq_nesting !=3D INT_MIN && \ - !READ_ONCE((t)->trc_reader_special.b.blocked)) { \ - rcu_tasks_trace_qs_blkd(t); \ - } \ - } while (0) void rcu_tasks_trace_torture_stats_print(char *tt, char *tf); -# else -# define rcu_tasks_trace_qs(t) do { } while (0) # endif =20 -#define rcu_tasks_qs(t, preempt) \ -do { \ - rcu_tasks_classic_qs((t), (preempt)); \ - rcu_tasks_trace_qs(t); \ -} while (0) +#define rcu_tasks_qs(t, preempt) do { rcu_tasks_classic_qs((t), (preempt))= ; } while (0) =20 # ifdef CONFIG_TASKS_RUDE_RCU void synchronize_rcu_tasks_rude(void); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 418fa242cf0288..967c43b1937bae 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1471,18 +1471,6 @@ void __init rcu_tasks_trace_suppress_unused(void) rcu_tasks_torture_stats_print_generic(NULL, NULL, NULL, NULL); } =20 -/* - * Do a cmpxchg() on ->trc_reader_special.b.need_qs, allowing for - * the four-byte operand-size restriction of some platforms. - * - * Returns the old value, which is often ignored. - */ -u8 rcu_trc_cmpxchg_need_qs(struct task_struct *t, u8 old, u8 new) -{ - return cmpxchg(&t->trc_reader_special.b.need_qs, old, new); -} -EXPORT_SYMBOL_GPL(rcu_trc_cmpxchg_need_qs); - /* Add a newly blocked reader task to its CPU's list. */ void rcu_tasks_trace_qs_blkd(struct task_struct *t) { --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E5D92327A20; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637315; cv=none; b=hHApeDu4Ko6eL6RJ853mGlwt/i2B7jImbHtmbXjiAGxnUljfa9aVQ6fqlbfP7W/XLvxW8zBMlrb9UXNFBBW15Udl2njN6BpJI0sHoQE70ome8iqEExoBzL/jtWVAa8LPs5r1mgjb/BhTE5e3qK37odRQ/ceedQUTWIttIoBxiAw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637315; c=relaxed/simple; bh=gORzw/K5miRE/rNReHV1tjj65oo9eOSliJxhQHUaXtc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tHi9ls/20ybVNKqlYW+1a/mML9siYBMz9FpZRpRd9QrFZkzOsg42vhiPUDZfh5q2WyY5oGkUoCU5y6CQngN9AZAa6AghPBMfxYtRkMZ85WtNVmj8m/HyiVnNnyo6NCYhLGphjUFdUz3hq7O9ge4FyXVk7See3l3Ofrc7KnSFz2g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QwSootcX; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QwSootcX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 722A2C19421; Tue, 23 Sep 2025 14:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637314; bh=gORzw/K5miRE/rNReHV1tjj65oo9eOSliJxhQHUaXtc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QwSootcX+hMz03OhUwZFvSEPmGOuAvoPBUqXbKeY+1HBzAKWcbk3vf8SYzkTIVI58 63B6udTAqRiPYP9xcnZSZvS5lyIp7dnsTk088VwtyzlihPrg/XW02yAiXYFewnAIps 67TKJO3GP3d6Mo5OwLcnF/L/gy2sGg9CUBG4RbpIXCE1aCXMvOZw1qLcBLudvqxaua 8PfH5Yy0ARtAZZPdon28prIgi+9JdstTRmWAuqGb5fzyKxhIEuUh/ipTzhMUM/j0uY 0VQDe2j0VN1yQJK42PG66b3aRnBVQMwblXjBwm0NKO74FHFIvMBBFN7Z0ujPuUAjqS NBwKcqUvCjOdQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C413ACE127E; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Frederic Weisbecker , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 06/34] context_tracking: Remove rcu_task_trace_heavyweight_{enter,exit}() Date: Tue, 23 Sep 2025 07:20:08 -0700 Message-Id: <20250923142036.112290-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Because SRCU-fast does not use IPIs for its grace periods, there is no need for real-time workloads to switch to an IPI-free mode, and there is in turn no need for either rcu_task_trace_heavyweight_enter() or rcu_task_trace_heavyweight_exit(). This commit therefore removes them. Signed-off-by: Paul E. McKenney Cc: Frederic Weisbecker Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- kernel/context_tracking.c | 20 -------------------- 1 file changed, 20 deletions(-) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index fb5be6e9b423f2..a743e7ffa6c00f 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -54,24 +54,6 @@ static __always_inline void rcu_task_enter(void) #endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ } =20 -/* Turn on heavyweight RCU tasks trace readers on kernel exit. */ -static __always_inline void rcu_task_trace_heavyweight_enter(void) -{ -#ifdef CONFIG_TASKS_TRACE_RCU - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) - current->trc_reader_special.b.need_mb =3D true; -#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ -} - -/* Turn off heavyweight RCU tasks trace readers on kernel entry. */ -static __always_inline void rcu_task_trace_heavyweight_exit(void) -{ -#ifdef CONFIG_TASKS_TRACE_RCU - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) - current->trc_reader_special.b.need_mb =3D false; -#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ -} - /* * Record entry into an extended quiescent state. This is only to be * called when not already in an extended quiescent state, that is, @@ -85,7 +67,6 @@ static noinstr void ct_kernel_exit_state(int offset) * critical sections, and we also must force ordering with the * next idle sojourn. */ - rcu_task_trace_heavyweight_enter(); // Before CT state update! // RCU is still watching. Better not be in extended quiescent state! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !rcu_is_watching_curr_cp= u()); (void)ct_state_inc(offset); @@ -108,7 +89,6 @@ static noinstr void ct_kernel_enter_state(int offset) */ seq =3D ct_state_inc(offset); // RCU is now watching. Better not be in an extended quiescent state! - rcu_task_trace_heavyweight_exit(); // After CT state update! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & CT_RCU_WATCHING)= ); } =20 --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55B6D328997; Tue, 23 Sep 2025 14:21:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; cv=none; b=aJK+FEH3W7s5/EJRZ7l4n+tgSKdE5uG4MpHiSy2cdZgXdI/rNNspxiMB7Wz+DTCJQS3Kr79aD02CR3/MHnQs2AyTTTwivgn6UhmOhmJhGzuY5wyhCslpz5L3HCh5UAO/7IIrKfsnE1jSb0zjOgs4PNN9NQYshe13udXgtXKagPc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; c=relaxed/simple; bh=vhENAcGeT3O7mnVVnjo6mRyfATWkOODJWG3NzL/dFHc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jJrfoE4/AnJ90JyjAHgpGo4LylV9URCZ8d7mYTACq6iQXVRAPEoBEiJ4xexlQ1R/n0VuVkmidamyKqxu1gebX6fSw1RLIllk809f55fFag3fG14Q7Iog0vwwe7U7EUOgtfKpurCtEgBAiMArM1ZsfrtFMyhQ8zUTm1TvOCyVcvo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MZ4jcmzv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MZ4jcmzv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id ADC26C2BCB1; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637316; bh=vhENAcGeT3O7mnVVnjo6mRyfATWkOODJWG3NzL/dFHc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MZ4jcmzvu5yPLh3dT7MSC3uGwk/CYGXuTJQfC2gJ6RgKoSUYZTDPYMvzwgvAMotSu kvtFmDOyUr+ktFKfRDn8v4xuabNCvyGzshX3SnNDz2a/TK/R7SzKaNeoXK7+5dETKp 3Z3jTUVW8e/QWvdV3DRz//yCmavtBj/xFk1EdPfJ3GBfs545H4o889SSO+bpjSOUfY ysSy1e74b0+0opa43vAvxuXquMo+cUL5cforS6IpZfXDp6dKjViGAIUgBoaFALeaKb wj+ACOeiuTEVP3mVN4dkLNg1aOQt7lLorqflLFoT+I6BeY9FL2sMcJDUlrPfxTKTIo bIu26RmNhQKCA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C68A3CE12A5; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 07/34] rcu: Remove ->trc_reader_special from task_struct Date: Tue, 23 Sep 2025 07:20:09 -0700 Message-Id: <20250923142036.112290-7-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that RCU Tasks Trace has been re-implemented in terms of SRCU-fast, and no-longer-needed quiescent-state functions have been removed, the ->trc_reader_special task_struct field is only initialized, and never actually used. This commit therefore removes it. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/sched.h | 1 - init/init_task.c | 1 - kernel/fork.c | 1 - 3 files changed, 3 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index e62c99e44a5502..577fafd22a0e6f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -940,7 +940,6 @@ struct task_struct { #ifdef CONFIG_TASKS_TRACE_RCU int trc_reader_nesting; struct srcu_ctr __percpu *trc_reader_scp; - union rcu_special trc_reader_special; #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ =20 struct sched_info sched_info; diff --git a/init/init_task.c b/init/init_task.c index 91c37f66ec8e9a..0c075f3cc8fc5a 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -165,7 +165,6 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = =3D { #endif #ifdef CONFIG_TASKS_TRACE_RCU .trc_reader_nesting =3D 0, - .trc_reader_special.s =3D 0, #endif #ifdef CONFIG_CPUSETS .mems_allowed_seq =3D SEQCNT_SPINLOCK_ZERO(init_task.mems_allowed_seq, diff --git a/kernel/fork.c b/kernel/fork.c index 12388e895d2955..5686d50b62cfaf 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1780,7 +1780,6 @@ static inline void rcu_copy_process(struct task_struc= t *p) #endif /* #ifdef CONFIG_TASKS_RCU */ #ifdef CONFIG_TASKS_TRACE_RCU p->trc_reader_nesting =3D 0; - p->trc_reader_special.s =3D 0; #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ } =20 --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 558A1328996; Tue, 23 Sep 2025 14:21:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; cv=none; b=qG2Stg0fj1PNNguqlVB+s3zfj3X+lWvDzqy5PNGfKTJyGy/Q8LM/KahSrhEJbrK4/MfDaXBEP+y6QY6fm0Kpw+pi4hBhxaMPHvESPncfkL/p5d3KbwT0UKC5PiYNdNgN9ng50WzvJ0NMXNNOHStK/IJnoHt2UsMyWpTWcGE8WBU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; c=relaxed/simple; bh=UWhSG3+L9kGDuQv4d75Qlf1/uqGMYN1a+rwGs+aTo7s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Q8PBK816CZiM6fA3ban7Dq9NO52S7rr1xUB5rk2LPTjo4FaCKdYsVVG4mQBeMbNHMcN+xos6r6gchMJZjEuuFDMJglzq+ZDwoxD7uGT+4DxReSIRih2Aas9QCKvzGaGVBqyspMGmEG5yyOXteedplpTSu3khJu2ujYwoz45hlQM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bWV3n/IF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bWV3n/IF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A398EC2BCB0; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637316; bh=UWhSG3+L9kGDuQv4d75Qlf1/uqGMYN1a+rwGs+aTo7s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bWV3n/IFCRYk0ertJZgeB+IZS5IrwdockOH6qZT87qpRdq+RFjThQ47I6aTDKvG/z zcc+cXvbCx9AS+UW1tZntqbYm4R8TIxQfnmChHnXESyW6ZlFlbof3a2iycgdncSh/f bWTaMyT951D2nTS7HQms1VptnQV0ElehZErjpCQ+ZY+TWGjLiSBvDjHgNqKVQSDV3J lOxmy7voQYa+bgWz0vBUmYF9hFLizOyHwASI7/sgfpGQMp4PV/7QJYqxVShSR9o1YN LZPBnnr9M7vj91px5G48xSwdxAeNvtzokQfgt8j5VcukZNpuwOUopSM79goMuhY6YT d5EiY5XJNW2dg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C93FECE12C3; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 08/34] rcu: Remove now-empty RCU Tasks Trace functions and calls to them Date: Tue, 23 Sep 2025 07:20:10 -0700 Message-Id: <20250923142036.112290-8-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit removes the now-empty functions rcu_tasks_trace_qs_blkd(), exit_tasks_rcu_finish_trace(), and rcu_spawn_tasks_trace_kthread(), along with all calls to them. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/rcupdate.h | 1 - kernel/rcu/tasks.h | 28 +--------------------------- 2 files changed, 1 insertion(+), 28 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 879525c5764a0c..7611730e34bc33 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -176,7 +176,6 @@ void rcu_tasks_torture_stats_print(char *tt, char *tf); # endif =20 # ifdef CONFIG_TASKS_TRACE_RCU -void rcu_tasks_trace_qs_blkd(struct task_struct *t); void rcu_tasks_trace_torture_stats_print(char *tt, char *tf); # endif =20 diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 967c43b1937bae..25dc49ebad251d 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -800,8 +800,6 @@ static void rcu_tasks_torture_stats_print_generic(struc= t rcu_tasks *rtp, char *t =20 #endif // #ifndef CONFIG_TINY_RCU =20 -static void exit_tasks_rcu_finish_trace(struct task_struct *t); - #if defined(CONFIG_TASKS_RCU) =20 //////////////////////////////////////////////////////////////////////// @@ -1321,13 +1319,11 @@ void exit_tasks_rcu_finish(void) raw_spin_lock_irqsave_rcu_node(rtpcp, flags); list_del_init(&t->rcu_tasks_exit_list); raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); - - exit_tasks_rcu_finish_trace(t); } =20 #else /* #ifdef CONFIG_TASKS_RCU */ void exit_tasks_rcu_start(void) { } -void exit_tasks_rcu_finish(void) { exit_tasks_rcu_finish_trace(current); } +void exit_tasks_rcu_finish(void) { } #endif /* #else #ifdef CONFIG_TASKS_RCU */ =20 #ifdef CONFIG_TASKS_RUDE_RCU @@ -1471,12 +1467,6 @@ void __init rcu_tasks_trace_suppress_unused(void) rcu_tasks_torture_stats_print_generic(NULL, NULL, NULL, NULL); } =20 -/* Add a newly blocked reader task to its CPU's list. */ -void rcu_tasks_trace_qs_blkd(struct task_struct *t) -{ -} -EXPORT_SYMBOL_GPL(rcu_tasks_trace_qs_blkd); - /* Communicate task state back to the RCU tasks trace stall warning reques= t. */ struct trc_stall_chk_rdr { int nesting; @@ -1484,19 +1474,9 @@ struct trc_stall_chk_rdr { u8 needqs; }; =20 -/* Report any needed quiescent state for this exiting task. */ -static void exit_tasks_rcu_finish_trace(struct task_struct *t) -{ -} - int rcu_tasks_trace_lazy_ms =3D -1; module_param(rcu_tasks_trace_lazy_ms, int, 0444); =20 -static int __init rcu_spawn_tasks_trace_kthread(void) -{ - return 0; -} - #if !defined(CONFIG_TINY_RCU) void show_rcu_tasks_trace_gp_kthread(void) { @@ -1520,8 +1500,6 @@ void rcu_tasks_trace_get_gp_data(int *flags, unsigned= long *gp_seq) } EXPORT_SYMBOL_GPL(rcu_tasks_trace_get_gp_data); =20 -#else /* #ifdef CONFIG_TASKS_TRACE_RCU */ -static void exit_tasks_rcu_finish_trace(struct task_struct *t) { } #endif /* #else #ifdef CONFIG_TASKS_TRACE_RCU */ =20 #ifndef CONFIG_TINY_RCU @@ -1668,10 +1646,6 @@ static int __init rcu_init_tasks_generic(void) rcu_spawn_tasks_rude_kthread(); #endif =20 -#ifdef CONFIG_TASKS_TRACE_RCU - rcu_spawn_tasks_trace_kthread(); -#endif - // Run the self-tests. rcu_tasks_initiate_self_tests(); =20 --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6B2123A98E; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637314; cv=none; b=trfyfwJpc+3XYddGjOpiANMfvjcTNh7z/87Up3822qjwerJYkeuvDtIoLsHhm7JITBiLDDAMxeD2cQWKgR3cCOvMFQNRNiVp+3FdpGd38Ut/Cl/hTCsue2MHM8lUV6Ju1XOJk3b/qW372Y2KtuFxbd1M3m6HU67gkKc3WTIkIUg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637314; c=relaxed/simple; bh=FM7Dhz9Lrv6Pm3hueBcDMymUaOI0qleJ0d7VFIxSBSU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=op8e436IPL30GcLsmHZmM6A37MVi/3+jQ0lbJ3djCXyfdT087pLPkn9+gG+nsQGLkWVgyGLwobprO8cBGJjF0zUAz9+l8c8hc7zQZeh9DupKthbOgSkxCK8uTd1WGceFDwRdEQ1RNdmqAm2ul3XpBC5Pu1UkfU/2qXw7rPLq0X4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=EcSg25hh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="EcSg25hh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 70B9CC116D0; Tue, 23 Sep 2025 14:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637314; bh=FM7Dhz9Lrv6Pm3hueBcDMymUaOI0qleJ0d7VFIxSBSU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EcSg25hhBf6d8CkJJ7um9/Xmz9fJUx/RrHo97Sv7dyPX6Jzm88CzbzeL9xWYQ3XXt GMx5KVyQUptFLPWFcvaXBiGQsa35TghWpzIdw95uyVmD2q/3WNn0dJxA/ChFAYbjxx q84ostRBrhSBDI+FRMkIY42l4jW/tKRQk0Q51QkASM+3zXs/RdFob9QRCofGsrtv/E +Vy2BsqqxpGBnj7rqunChvIbfCJ10Odu3ODrsG2wiFdm4q6HA2mhcChOow2mtSSChf O/UTi2Ego0gOV+AKR2Mn4FovtzhQUsyymNajbrJ43fGqY7MJXQTeqhpw2oRbj3BKSz p0FFQU3Cx0eqw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id CC131CE130C; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 09/34] rcu: Remove unused rcu_tasks_trace_lazy_ms and trc_stall_chk_rdr struct Date: Tue, 23 Sep 2025 07:20:11 -0700 Message-Id: <20250923142036.112290-9-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The rcupdate.rcu_tasks_trace_lazy_ms and trc_stall_chk_rdr structure are no longer used, so this commit removes them. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- Documentation/admin-guide/kernel-parameters.txt | 8 -------- kernel/rcu/tasks.h | 10 ---------- 2 files changed, 18 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 747a55abf4946b..54d31a5e46e1cf 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6128,14 +6128,6 @@ of zero will disable batching. Batching is always disabled for synchronize_rcu_tasks(). =20 - rcupdate.rcu_tasks_trace_lazy_ms=3D [KNL] - Set timeout in milliseconds RCU Tasks - Trace asynchronous callback batching for - call_rcu_tasks_trace(). A negative value - will take the default. A value of zero will - disable batching. Batching is always disabled - for synchronize_rcu_tasks_trace(). - rcupdate.rcu_self_test=3D [KNL] Run the RCU early boot self tests =20 diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 25dc49ebad251d..50f5c483e0e15a 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1467,16 +1467,6 @@ void __init rcu_tasks_trace_suppress_unused(void) rcu_tasks_torture_stats_print_generic(NULL, NULL, NULL, NULL); } =20 -/* Communicate task state back to the RCU tasks trace stall warning reques= t. */ -struct trc_stall_chk_rdr { - int nesting; - int ipi_to_cpu; - u8 needqs; -}; - -int rcu_tasks_trace_lazy_ms =3D -1; -module_param(rcu_tasks_trace_lazy_ms, int, 0444); - #if !defined(CONFIG_TINY_RCU) void show_rcu_tasks_trace_gp_kthread(void) { --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 492BB328563; Tue, 23 Sep 2025 14:21:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637315; cv=none; b=mIlwIfzMj6956eXG1XA1ov9jNCv70+ezbXGVMP8jQ2CVbt2H/DGigrUqOA0IoksbL+ev7yE1ZR+EfqrU1Rd//zkTpqjLQH/4LFqaAOvsuDSV3eh0/nbZX/xrDAjLUmaJ8t2Z5glhG9o6xWipHn+lir+CNxFfFt6Vgl9pHbayq7M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637315; c=relaxed/simple; bh=zPxFwXtYgPLDmkgZ+C7PtVRQ3LVvyGVKpWPncscteiI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OF8WpTptipjvjYRRB9QldIhTFxEbfTzLOrL8462IIlhUdObV5vX8Wnu3P16sEebupmwBHn1LC+qsJXh2zQ4bGxt7dOBA7JcvlWrAt4/KxKvvaDxZoeTZcZui+l5LioBvJYgT3oWnh1Xu+4tiCP4QnbvKmeV0c/pEZEXN+mfksec= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CFgL6W0X; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CFgL6W0X" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 77AA0C19423; Tue, 23 Sep 2025 14:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637314; bh=zPxFwXtYgPLDmkgZ+C7PtVRQ3LVvyGVKpWPncscteiI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CFgL6W0X9TD7itXUCA79zfHAMgsDv/F5ZjscgtnZe2i+axXkbeDke8n6lkPlH2Rvb J0dOPDolpyRr9ODVcAPyii5ZUSIc3NecjkdxrjpKoGK9PxkUjTga/rNCyKfXNDccwt euFWwYsRXqJ1RrE0ktNztksZjKsMjQgKCnRdz5HOUc+u6iu7XRQJ/CClDr2nwFoNNJ OBC7zbx4oJ/pADdOHTZd7sdREWWx+Zm3SKQF01WdNLmvvLAcffNc2cZt5aKK8UFirW nFf89dNlRCTzjudGzHVB+l1kHVwWL/q/qI8++OL4Je1wo8Kg+LcSyI/a0LJqvnZrjA THcqpEPX/A9aw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id CF24CCE1385; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 10/34] rcu: Remove now-empty show_rcu_tasks_trace_gp_kthread() function Date: Tue, 23 Sep 2025 07:20:12 -0700 Message-Id: <20250923142036.112290-10-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The show_rcu_tasks_trace_gp_kthread() is used by rcutorture to diagnose bugs involving the RCU Tasks Trace grace-period kthread, which now no longer exists. This commit therefore removes this function. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- kernel/rcu/rcu.h | 5 ----- kernel/rcu/rcutorture.c | 1 - kernel/rcu/tasks.h | 6 ------ 3 files changed, 12 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 9cf01832a6c3d1..b397572b4a5331 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -673,11 +673,6 @@ void show_rcu_tasks_rude_gp_kthread(void); #else static inline void show_rcu_tasks_rude_gp_kthread(void) {} #endif -#if !defined(CONFIG_TINY_RCU) && defined(CONFIG_TASKS_TRACE_RCU) -void show_rcu_tasks_trace_gp_kthread(void); -#else -static inline void show_rcu_tasks_trace_gp_kthread(void) {} -#endif =20 #ifdef CONFIG_TINY_RCU static inline bool rcu_cpu_beenfullyonline(int cpu) { return true; } diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 72619e5e8549dc..5120e1ce811413 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -1130,7 +1130,6 @@ static struct rcu_torture_ops tasks_tracing_ops =3D { .exp_sync =3D synchronize_rcu_tasks_trace, .call =3D call_rcu_tasks_trace, .cb_barrier =3D rcu_barrier_tasks_trace, - .gp_kthread_dbg =3D show_rcu_tasks_trace_gp_kthread, .get_gp_data =3D rcu_tasks_trace_get_gp_data, .cbflood_max =3D 50000, .irq_capable =3D 1, diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 50f5c483e0e15a..6e2530fe23e620 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1468,11 +1468,6 @@ void __init rcu_tasks_trace_suppress_unused(void) } =20 #if !defined(CONFIG_TINY_RCU) -void show_rcu_tasks_trace_gp_kthread(void) -{ -} -EXPORT_SYMBOL_GPL(show_rcu_tasks_trace_gp_kthread); - void rcu_tasks_trace_torture_stats_print(char *tt, char *tf) { } @@ -1497,7 +1492,6 @@ void show_rcu_tasks_gp_kthreads(void) { show_rcu_tasks_classic_gp_kthread(); show_rcu_tasks_rude_gp_kthread(); - show_rcu_tasks_trace_gp_kthread(); } #endif /* #ifndef CONFIG_TINY_RCU */ =20 --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B2B921B9CD; Tue, 23 Sep 2025 14:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; cv=none; b=c53iYkyfVQIB9swAlS9UnDL8wZOV3o7TjKeMSCstUCy+0hHDcWcG6+5h85zPKao2aezm9/upktXVufo5bFtw973Lv8GZRhb9qz+0qfHuvrsjWD+MaGU7lQlm0cDYR++vZzgajbH5YXpO1AH2MPaywKe5Njwd93XBJtvu8ZSBwvM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; c=relaxed/simple; bh=XNmsZMoAIYYD/q9UvCMskFAVtHocTahtlj2a9AJbvFU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ByV1J3V3u3zqJFZnqCQ+UPNYJ9G8nmNUwlO2rhh/9yzxT2R/6p29RWO5RjI6EcsDcI1uQDfV+SgzYnYX59F0wxLRsxQVGKR6+HFa5cGyL75MH3WPPb2PADk5rTZH2WErGX0vWOufjU/ohLF9JaAT7W3blMF8r/z/9BfFyQLC9j8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RXkTPm51; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RXkTPm51" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AC177C113D0; Tue, 23 Sep 2025 14:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637302; bh=XNmsZMoAIYYD/q9UvCMskFAVtHocTahtlj2a9AJbvFU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RXkTPm51Q0u1KO26bxn9O6W+5DJZubvIwqRQ81/rkxtkRS+yDPKVm6MNJOZMRSjkQ DJ1yUtBNMjAMADSTaeqC4uLQGfvUrI4N+dxKSODKGAP/1bwn1hlVhBbVohMfZxb4Cs 6dSfaZeVhZCLC3ByJiGz7AmwyhNtAkWfalQiV4Dl7tD6sDLh8sVUJjQw/eBD9wylOJ kuP+hkoXPkgSltxQo0SqGziXOs2NEUwXmduLuzM6TxLr4znjf9SdeuqT2y5sNsVGv2 lj2aAQFWHuqBjovwJpajFzWG7ZX1vZNO0qaOS1iXt01AFGcxQRzN0MOGbzvchEnrdo CJNevCpDCrOZA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D1F50CE1389; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 11/34] rcu: Remove now-empty rcu_tasks_trace_get_gp_data() function Date: Tue, 23 Sep 2025 07:20:13 -0700 Message-Id: <20250923142036.112290-11-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The rcu_tasks_trace_get_gp_data() is used by rcutorture to diagnose bugs involving the RCU Tasks Trace grace-period kthread, which now no longer exists. This commit therefore removes this function. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/rcupdate_trace.h | 1 - kernel/rcu/rcu.h | 4 ---- kernel/rcu/rcutorture.c | 1 - kernel/rcu/tasks.h | 5 ----- 4 files changed, 11 deletions(-) diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index 3f46cbe6700038..ffd4dcd6339ae4 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -136,7 +136,6 @@ static inline void rcu_barrier_tasks_trace(void) } =20 // Placeholders to enable stepwise transition. -void rcu_tasks_trace_get_gp_data(int *flags, unsigned long *gp_seq); void __init rcu_tasks_trace_suppress_unused(void); struct task_struct *get_rcu_tasks_trace_gp_kthread(void); =20 diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index b397572b4a5331..dc5d614b372c1e 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -544,10 +544,6 @@ struct task_struct *get_rcu_tasks_rude_gp_kthread(void= ); void rcu_tasks_rude_get_gp_data(int *flags, unsigned long *gp_seq); #endif // # ifdef CONFIG_TASKS_RUDE_RCU =20 -#ifdef CONFIG_TASKS_TRACE_RCU -void rcu_tasks_trace_get_gp_data(int *flags, unsigned long *gp_seq); -#endif - #ifdef CONFIG_TASKS_RCU_GENERIC void tasks_cblist_init_generic(void); #else /* #ifdef CONFIG_TASKS_RCU_GENERIC */ diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 5120e1ce811413..485fa822b6a753 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -1130,7 +1130,6 @@ static struct rcu_torture_ops tasks_tracing_ops =3D { .exp_sync =3D synchronize_rcu_tasks_trace, .call =3D call_rcu_tasks_trace, .cb_barrier =3D rcu_barrier_tasks_trace, - .get_gp_data =3D rcu_tasks_trace_get_gp_data, .cbflood_max =3D 50000, .irq_capable =3D 1, .slow_gps =3D 1, diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 6e2530fe23e620..7a8c1fd9addb7a 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1480,11 +1480,6 @@ struct task_struct *get_rcu_tasks_trace_gp_kthread(v= oid) } EXPORT_SYMBOL_GPL(get_rcu_tasks_trace_gp_kthread); =20 -void rcu_tasks_trace_get_gp_data(int *flags, unsigned long *gp_seq) -{ -} -EXPORT_SYMBOL_GPL(rcu_tasks_trace_get_gp_data); - #endif /* #else #ifdef CONFIG_TASKS_TRACE_RCU */ =20 #ifndef CONFIG_TINY_RCU --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7729238D54; Tue, 23 Sep 2025 14:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; cv=none; b=ItX1H+TFISFt6p9lz5LQLzmf/QZmMHa5bkF1R2slOho1Z2H/9fOUQGNd2oHSvhgZhZxDI2hpQ3D9fShj9kB4+WuGXt3r5h8G/GuhfqjXMatzzWuvkYJgC8ldnB9cVZXvG/odOt8k8cHLwfZXm+7qz8le973/VJKlsCXNbU5t6kA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; c=relaxed/simple; bh=3ftSpkUjjOHmFcE+Uq9SLwvx1RopJ6G++mEEFYkpkZ4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aSifj+/6KmKpa3S2VVI8P5Ql+DeWrqhFF1kBb2+ZyIao41ZmT9xi8/fFqMrVttf2BBCup4bl6L5MsFB1d1bcTnH1pNBEkNiW4sVYfOkic22R7HXDua5E3Qi1JK//vEVpJ2MkS34MWNAC6B0/CLOucfLPNWcdTmPqaqgcvK3qZkY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=t0vt1Tl2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="t0vt1Tl2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8722EC4CEF5; Tue, 23 Sep 2025 14:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637302; bh=3ftSpkUjjOHmFcE+Uq9SLwvx1RopJ6G++mEEFYkpkZ4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=t0vt1Tl2+DF2UKvcLFZ3Cps9QQb7cz+d6KFaVOugaMeZpOZDu8lApEwjdzz8fG7Sd cR7eAooKdz2zexKu3vqe8kiFrs0PILqSnZhoNgYUpWtr9hO8FqYoWS1nmD4TwXVExG 7+YHfmW7nO24xd7WypYTmQonhxXZxssetbQYtKAsVwWOZbAPSdcY4OTM2I6jYHgBI6 HvkLCXRhQEERuHBb0leKu4yG0ssBVIRkENRG+fS1VNq+ngd8kGxTYMKUIZaUn6JVog w/qjmNJFBBQ0vQ2NpvZrWrcwNRqdVu+mIwBuRkKvoTiJ06ePdQ4XRVszJLD+P898QN sHIK36k/gzmwQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D4817CE13DF; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 12/34] rcu: Remove now-empty rcu_tasks_trace_torture_stats_print() function Date: Tue, 23 Sep 2025 07:20:14 -0700 Message-Id: <20250923142036.112290-12-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The rcu_tasks_trace_torture_stats_print() is used by rcuscale to diagnose bugs involving the RCU Tasks Trace grace-period kthread, which now no longer exists. This commit therefore removes this function. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/rcupdate.h | 4 ---- kernel/rcu/rcuscale.c | 6 ------ kernel/rcu/tasks.h | 7 ------- 3 files changed, 17 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 7611730e34bc33..f3a7478bab2e26 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -175,10 +175,6 @@ void rcu_tasks_torture_stats_print(char *tt, char *tf); # define synchronize_rcu_tasks synchronize_rcu # endif =20 -# ifdef CONFIG_TASKS_TRACE_RCU -void rcu_tasks_trace_torture_stats_print(char *tt, char *tf); -# endif - #define rcu_tasks_qs(t, preempt) do { rcu_tasks_classic_qs((t), (preempt))= ; } while (0) =20 # ifdef CONFIG_TASKS_RUDE_RCU diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index b521d04559927a..16ba9050dab66b 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -400,11 +400,6 @@ static void tasks_trace_scale_read_unlock(int idx) rcu_read_unlock_trace(); } =20 -static void rcu_tasks_trace_scale_stats(void) -{ - rcu_tasks_trace_torture_stats_print(scale_type, SCALE_FLAG); -} - static struct rcu_scale_ops tasks_tracing_ops =3D { .ptype =3D RCU_TASKS_FLAVOR, .init =3D rcu_sync_scale_init, @@ -417,7 +412,6 @@ static struct rcu_scale_ops tasks_tracing_ops =3D { .sync =3D synchronize_rcu_tasks_trace, .exp_sync =3D synchronize_rcu_tasks_trace, .rso_gp_kthread =3D get_rcu_tasks_trace_gp_kthread, - .stats =3D IS_ENABLED(CONFIG_TINY_RCU) ? NULL : rcu_tasks_trace_scale_st= ats, .name =3D "tasks-tracing" }; =20 diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 7a8c1fd9addb7a..4fb61b3c78283d 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1467,13 +1467,6 @@ void __init rcu_tasks_trace_suppress_unused(void) rcu_tasks_torture_stats_print_generic(NULL, NULL, NULL, NULL); } =20 -#if !defined(CONFIG_TINY_RCU) -void rcu_tasks_trace_torture_stats_print(char *tt, char *tf) -{ -} -EXPORT_SYMBOL_GPL(rcu_tasks_trace_torture_stats_print); -#endif // !defined(CONFIG_TINY_RCU) - struct task_struct *get_rcu_tasks_trace_gp_kthread(void) { return NULL; --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55822328995; Tue, 23 Sep 2025 14:21:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; cv=none; b=RkYgyi07e4mLtMHImiXzuhD5asQZu8aZb6ou0Fyhl4w2gx8dQx45VqXsubPmr9RnaUbOGoIOxy1x5NsoAGsfup5ysU4+BSD2eZD345c7+C+qH0OA7DBtZ3wByTpMN8xiIM/WsBtDO3u0skpBRWpmlxfq6rcvVEaF6/QoQND5iYg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; c=relaxed/simple; bh=dnfn0DMyZqHCyXyDBX7gMCl/aLGTGuiQmNJ7YTCoJDk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oDpH6GrVeLyjPnXM8vtGGZDme181ZFh6hNTw9kZje6dmCcCcjqefrpcP1KvLrWvIqqedbWNzWbuuOzpFJojravS8/Aa/foYKD0JBxxVlS4/rPPXMR4Fmb3wyKw3U7B8ssRHnutS0wBsrvlcj5vp4jVfufCQjzllYigrx4JrWowY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ParMViib; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ParMViib" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A49DAC2BCB2; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637316; bh=dnfn0DMyZqHCyXyDBX7gMCl/aLGTGuiQmNJ7YTCoJDk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ParMViibAmTiqgmB/IaxzQprSDe/SSYbMrxjZuNxLy6Idjo9dZ7kQRw5NzLellG2E hdKH2/297Ey+6gSGkBjDXb4/ZDFRBrQzOY9vXUK2yxKxg+3gPPSMIo2x8xbxFuQ/5v r0fp5fj4bGXlHXxTf7x14SxjGtQlO7ghIBG6/YSqqC8Ogg1UE3CBQNuPi+t83qCLgE tkxOWoIDWawKIiXakty6VpJhY74sIQ6ZwH1dJ84lgAFTg/Bs87rdGGqOuo39B8shU0 mckXAWK0ZLywViwmRrPaYRUZ8xDOFZyypZDu+y7phQ29h+HW+aOFoArhKjmm5GNt7J Rzili71TsYQQg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D7442CE1418; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 13/34] rcu: Remove now-empty get_rcu_tasks_trace_gp_kthread() function Date: Tue, 23 Sep 2025 07:20:15 -0700 Message-Id: <20250923142036.112290-13-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The get_rcu_tasks_trace_gp_kthread() is used by rcuscale to diagnose bugs involving the RCU Tasks Trace grace-period kthread, which now no longer exists. This commit therefore removes this function. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/rcupdate_trace.h | 1 - kernel/rcu/rcuscale.c | 1 - kernel/rcu/tasks.h | 6 ------ 3 files changed, 8 deletions(-) diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index ffd4dcd6339ae4..0bd47f12ecd17b 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -137,7 +137,6 @@ static inline void rcu_barrier_tasks_trace(void) =20 // Placeholders to enable stepwise transition. void __init rcu_tasks_trace_suppress_unused(void); -struct task_struct *get_rcu_tasks_trace_gp_kthread(void); =20 #else /* diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 16ba9050dab66b..17d038e26b65de 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -411,7 +411,6 @@ static struct rcu_scale_ops tasks_tracing_ops =3D { .gp_barrier =3D rcu_barrier_tasks_trace, .sync =3D synchronize_rcu_tasks_trace, .exp_sync =3D synchronize_rcu_tasks_trace, - .rso_gp_kthread =3D get_rcu_tasks_trace_gp_kthread, .name =3D "tasks-tracing" }; =20 diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 4fb61b3c78283d..e70609c85ece5d 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1467,12 +1467,6 @@ void __init rcu_tasks_trace_suppress_unused(void) rcu_tasks_torture_stats_print_generic(NULL, NULL, NULL, NULL); } =20 -struct task_struct *get_rcu_tasks_trace_gp_kthread(void) -{ - return NULL; -} -EXPORT_SYMBOL_GPL(get_rcu_tasks_trace_gp_kthread); - #endif /* #else #ifdef CONFIG_TASKS_TRACE_RCU */ =20 #ifndef CONFIG_TINY_RCU --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2B5F267B02; Tue, 23 Sep 2025 14:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637303; cv=none; b=qxKshdvlg7CqLwJ8JxN8TOKRGlbEegYgItzmfSaR/8xHEQ/owY2nXbQxc3zJtSc06LFSXwkyQvO+RMPBduo/2igqkIHt9Ea4a+rExswRafhiBERWZRrZsw8WNMFLpgfA3ZZLuMMx6T/UmToe2cQLVvP1aP1rwpG2FnREWFyWRNU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637303; c=relaxed/simple; bh=egf5O4384GKLj9kiAAW/t23M4cxF514rK1fhEiB3mLc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=p1Ck8vq5biFk/JSsjBWFFnQoTldtcXVEq2uDoa6AUn2BEoLFMmJw1rTkv/6NGkaKQYlURzHxFGG783NEyub24za4sAAWr+1llzsx1yZWeywPWi7uCczyCpLBXw5MsTK8MPG63oeI17Q70wFhlErQukN3WvFaBDF0fsvM4EzE6go= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DJL4QXit; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DJL4QXit" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DE43DC19421; Tue, 23 Sep 2025 14:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637302; bh=egf5O4384GKLj9kiAAW/t23M4cxF514rK1fhEiB3mLc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DJL4QXit1l6Kem2Ql0r3Nj2CJr1/6HBINjQfcxUij3p95z5OeQPdF8L4ApeXNXmwt oihJA1dQTbk4l1fBCHQnpjaXYW4A7RC0BF/JnJexJ81rP3iivrs8zKxAHyCtw0I/mb IaXv2HmiQ7uToiHvAaPRd4fOgRbiq+scr7Rz1OWXYq0aoBjaCrdG9LELiMXhB8Wd9t SCjsAjP+zhtRDA14e9SC9W9wQcFfniq+vUrUSLGl+TMzgoUf0O40V8PwN5yQi4lGVI oqIIdvHezL9Kf4l2dpLBR2zHw2N66DFUUc3IDCWmpkAldbUcDWKaPmwAd0b/Wu8Ak7 T5ALFd/ruQh4Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D9E99CE1427; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 14/34] rcu: Move rcu_tasks_trace_srcu_struct out of #ifdef CONFIG_TASKS_RCU_GENERIC Date: Tue, 23 Sep 2025 07:20:16 -0700 Message-Id: <20250923142036.112290-14-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Moving rcu_tasks_trace_srcu_struct out the CONFIG_TASKS_RCU_GENERIC removed, and also permits the CONFIG_TASKS_TRACE_RCU Kconfig option to stop enabling the CONFIG_TASKS_RCU_GENERIC Kconfig option. This commit therefore makes it so. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- kernel/rcu/Kconfig | 2 +- kernel/rcu/tasks.h | 38 +++++++++++++------------------------- 2 files changed, 14 insertions(+), 26 deletions(-) diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index ceaf6594f634cd..eeaa1cd47c6c4c 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -82,7 +82,7 @@ config NEED_SRCU_NMI_SAFE def_bool HAVE_NMI && !ARCH_HAS_NMI_SAFE_THIS_CPU_OPS && !TINY_SRCU =20 config TASKS_RCU_GENERIC - def_bool TASKS_RCU || TASKS_RUDE_RCU || TASKS_TRACE_RCU + def_bool TASKS_RCU || TASKS_RUDE_RCU help This option enables generic infrastructure code supporting task-based RCU implementations. Not for manual selection. diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index e70609c85ece5d..a3c16e65e16812 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1444,31 +1444,6 @@ EXPORT_SYMBOL_GPL(rcu_tasks_rude_get_gp_data); =20 #endif /* #ifdef CONFIG_TASKS_RUDE_RCU */ =20 -//////////////////////////////////////////////////////////////////////// -// -// Tracing variant of Tasks RCU. This variant is designed to be used -// to protect tracing hooks, including those of BPF. This variant -// is implemented via a straightforward mapping onto SRCU-fast. - -#ifdef CONFIG_TASKS_TRACE_RCU - -DEFINE_SRCU(rcu_tasks_trace_srcu_struct); -EXPORT_SYMBOL_GPL(rcu_tasks_trace_srcu_struct); - -// Placeholder to suppress build errors through transition period. -void __init rcu_tasks_trace_suppress_unused(void) -{ - show_rcu_tasks_generic_gp_kthread(NULL, NULL); - rcu_spawn_tasks_kthread_generic(NULL); - synchronize_rcu_tasks_generic(NULL); - call_rcu_tasks_generic(NULL, NULL, NULL); - call_rcu_tasks_iw_wakeup(NULL); - cblist_init_generic(NULL); - rcu_tasks_torture_stats_print_generic(NULL, NULL, NULL, NULL); -} - -#endif /* #else #ifdef CONFIG_TASKS_TRACE_RCU */ - #ifndef CONFIG_TINY_RCU void show_rcu_tasks_gp_kthreads(void) { @@ -1622,3 +1597,16 @@ core_initcall(rcu_init_tasks_generic); #else /* #ifdef CONFIG_TASKS_RCU_GENERIC */ static inline void rcu_tasks_bootup_oddness(void) {} #endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */ + +#ifdef CONFIG_TASKS_TRACE_RCU + +//////////////////////////////////////////////////////////////////////// +// +// Tracing variant of Tasks RCU. This variant is designed to be used +// to protect tracing hooks, including those of BPF. This variant +// is implemented via a straightforward mapping onto SRCU-fast. + +DEFINE_SRCU(rcu_tasks_trace_srcu_struct); +EXPORT_SYMBOL_GPL(rcu_tasks_trace_srcu_struct); + +#endif /* #else #ifdef CONFIG_TASKS_TRACE_RCU */ --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 80CD83294FD; Tue, 23 Sep 2025 14:21:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; cv=none; b=XgKPSTAi7NXbaLnmTrGzie8g9MZqrexuYaEoj09Ejvyn0VVtJhWWvfS4snCzPmY38QSbbf4qoTrvDyUDoq/xKz83s2sZrECgftET0HHg2h1dqLYnGS5CrgDyqt+pU/Aiorajq/RZupN2gCnbFdc75SGqwdgxB48uawMYG9GSjsI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; c=relaxed/simple; bh=IOkf+dURDETQ7kOjDcCR/fZHWRdctmeYkirYqvh5ji4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=levz+FXWX91ant2d6rhzn+v9pQ9GupuSrU+hL5zo1KXtN80Shu25LZHnHPGERBviDo6Fpq2WT/xMvpYavxXwFnzzODZ7ilTngBnnKSpVoKtb3NoqDxxk9V+ABhng58GfvmlTutYki82BCNFTFBhFGW8d8NH/iX8kEqUMHaZvxnk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NGZCLjZG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NGZCLjZG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A6A47C113D0; Tue, 23 Sep 2025 14:21:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637316; bh=IOkf+dURDETQ7kOjDcCR/fZHWRdctmeYkirYqvh5ji4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NGZCLjZGPRJdy7hRXo0SkUkOrj3qXYCsSsoPE0c8CMJjLywd3xkRtQscNzjTItTAn X8iGFhF5ZvoSk9g33xQiY+1ZIXcEX9tw46jezZZCiTWTTqcUVrtOrW60DGLNwzJQ3L 4DrKU2l2P6CZYgv/Qif18Fc7Y6rqZKG+mA3mHoUdN7ulXss/aLhhoMCZXnSvOyj7/f l/Q3igSoQNK/w6hgFpIYzl12hGlkUrfKKKzitQvNHhAOrPX6hIv6PDogMH+6MJ3tU3 jfiZ7ldroLqvomvb1EMMbHYiXe8SaAMUXNv0vi9MxeDSAE3jxS4g0gcEE/42fyQjwa B3MI5Xp7XNblQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id DCC45CE1437; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 15/34] rcu: Add noinstr-fast rcu_read_{,un}lock_tasks_trace() APIs Date: Tue, 23 Sep 2025 07:20:17 -0700 Message-Id: <20250923142036.112290-15-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When expressing RCU Tasks Trace in terms of SRCU-fast, it was necessary to keep a nesting count and per-CPU srcu_ctr structure pointer in the task_struct structure, which is slow to access. But an alternative is to instead make rcu_read_lock_tasks_trace() and rcu_read_unlock_tasks_trace(), which match the underlying SRCU-fast semantics, avoiding the task_struct accesses. When all callers have switched to the new API, the previous rcu_read_lock_trace() and rcu_read_unlock_trace() APIs will be removed. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/rcupdate_trace.h | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index 0bd47f12ecd17b..b87151e6b23881 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -34,6 +34,43 @@ static inline int rcu_read_lock_trace_held(void) =20 #ifdef CONFIG_TASKS_TRACE_RCU =20 +/** + * rcu_read_lock_tasks_trace - mark beginning of RCU-trace read-side criti= cal section + * + * When synchronize_rcu_tasks_trace() is invoked by one task, then that + * task is guaranteed to block until all other tasks exit their read-side + * critical sections. Similarly, if call_rcu_trace() is invoked on one + * task while other tasks are within RCU read-side critical sections, + * invocation of the corresponding RCU callback is deferred until after + * the all the other tasks exit their critical sections. + * + * For more details, please see the documentation for srcu_read_lock_fast(= ). + */ +static inline struct srcu_ctr __percpu *rcu_read_lock_tasks_trace(void) +{ + struct srcu_ctr __percpu *ret =3D srcu_read_lock_fast(&rcu_tasks_trace_sr= cu_struct); + + if (IS_ENABLED(CONFIG_ARCH_WANTS_NO_INSTR)) + smp_mb(); + return ret; +} + +/** + * rcu_read_unlock_tasks_trace - mark end of RCU-trace read-side critical = section + * @scp: return value from corresponding rcu_read_lock_tasks_trace(). + * + * Pairs with the preceding call to rcu_read_lock_tasks_trace() that + * returned the value passed in via scp. + * + * For more details, please see the documentation for rcu_read_unlock(). + */ +static inline void rcu_read_unlock_tasks_trace(struct srcu_ctr __percpu *s= cp) +{ + if (!IS_ENABLED(CONFIG_ARCH_WANTS_NO_INSTR)) + smp_mb(); + srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp); +} + /** * rcu_read_lock_trace - mark beginning of RCU-trace read-side critical se= ction * --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EE6632951D; Tue, 23 Sep 2025 14:22:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637324; cv=none; b=lTMqZDnvSxqpEWsHuoxabftxRytPOn+M+ByPuNp2IOrzYV33rirnW7CJSCQ6pvpNld8M5HBSHzQVwgpAL6LaZ9Nbx7lSYagKEZRha/l5a8Kr/CYXVo3KpYLDc6hH9CP3mf7BnNnbvFtFaY42LWv7NC4lfpo1krB82nkKRMv5zYI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637324; c=relaxed/simple; bh=/tPpaG65gLbvD7rU4GOXgMQxivskQwWOEjt2Xk9cxII=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=vCAeDdFLoJSU9rO7oIIM4A4xu8xgTQd3F9bnSuNgcXyuGaX7cjn60AN3S9QJR/88NIgXUhHu8Dy2su/pzIEPOU+CMnlYIWRR+tlBO+EuZQgklJlrfCZZNiE7glse9GFiLzS0gpGUqqtwjN2rnkPLYmsWhkVWaRQLNECH7gfDoW0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OX4sPfBT; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OX4sPfBT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8C42CC113CF; Tue, 23 Sep 2025 14:22:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637324; bh=/tPpaG65gLbvD7rU4GOXgMQxivskQwWOEjt2Xk9cxII=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OX4sPfBTV+8CTfM3Idwl5WTLtfiK3ifO9ACtJgyVxrM7e2gOYrUIx3tS530p53tCo eHbOhfLYa0wD4/XLppz7b8P/IO4ZmcDXmPdjr6dXnykEBmegczWtE3f7WmZy94Hsf5 yjGF1ERZsszXvrweaN0nrai25TX026nu7REBV57ZV9hRT5Nb2VanXW7Nrp4OAuI98E Cny0yfPgiuHskPSoJNJ4kQ6YrVTxUuCx1UmZbxZLxLkf4bTVD6lM0EMCMCzErlUUA1 NKTxfRm+05U+96+v1DGmk+IJPcdSnZDjkwAPGe9ezY7Pj+3twSy60TdP1ahVq1CeN0 9kzS/QTm1YKpA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id DF8EECE14C5; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 16/34] rcu: Remove now-unused rcu_task_ipi_delay and TASKS_TRACE_RCU_READ_MB Date: Tue, 23 Sep 2025 07:20:18 -0700 Message-Id: <20250923142036.112290-16-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that RCU Tasks Trace being defined in terms of SRCU-fast, the rcupdate.rcu_task_ipi_delay kernel boot parameter and the TASKS_TRACE_RCU_READ_MB Kconfig option are no longer used. This commit therefore removes them. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- .../admin-guide/kernel-parameters.txt | 7 ------- kernel/rcu/Kconfig | 18 ------------------ kernel/rcu/tasks.h | 5 ----- .../selftests/rcutorture/configs/rcu/TRACE01 | 1 - .../selftests/rcutorture/configs/rcu/TRACE02 | 1 - 5 files changed, 32 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 54d31a5e46e1cf..40fc198bed4db9 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6078,13 +6078,6 @@ dynamically) adjusted. This parameter is intended for use in testing. =20 - rcupdate.rcu_task_ipi_delay=3D [KNL] - Set time in jiffies during which RCU tasks will - avoid sending IPIs, starting with the beginning - of a given grace period. Setting a large - number avoids disturbing real-time workloads, - but lengthens grace periods. - rcupdate.rcu_task_lazy_lim=3D [KNL] Number of callbacks on a given CPU that will cancel laziness on that CPU. Use -1 to disable diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index eeaa1cd47c6c4c..73a6cc364628b5 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -313,24 +313,6 @@ config RCU_NOCB_CPU_CB_BOOST Say Y here if you want to set RT priority for offloading kthreads. Say N here if you are building a !PREEMPT_RT kernel and are unsure. =20 -config TASKS_TRACE_RCU_READ_MB - bool "Tasks Trace RCU readers use memory barriers in user and idle" - depends on RCU_EXPERT && TASKS_TRACE_RCU - default PREEMPT_RT || NR_CPUS < 8 - help - Use this option to further reduce the number of IPIs sent - to CPUs executing in userspace or idle during tasks trace - RCU grace periods. Given that a reasonable setting of - the rcupdate.rcu_task_ipi_delay kernel boot parameter - eliminates such IPIs for many workloads, proper setting - of this Kconfig option is important mostly for aggressive - real-time installations and for battery-powered devices, - hence the default chosen above. - - Say Y here if you hate IPIs. - Say N here if you hate read-side memory barriers. - Take the default if you are unsure. - config RCU_LAZY bool "RCU callback lazy invocation functionality" depends on RCU_NOCB_CPU diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index a3c16e65e16812..833e180db744f2 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -161,11 +161,6 @@ static void tasks_rcu_exit_srcu_stall(struct timer_lis= t *unused); static DEFINE_TIMER(tasks_rcu_exit_srcu_stall_timer, tasks_rcu_exit_srcu_s= tall); #endif =20 -/* Avoid IPIing CPUs early in the grace period. */ -#define RCU_TASK_IPI_DELAY (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) ? H= Z / 2 : 0) -static int rcu_task_ipi_delay __read_mostly =3D RCU_TASK_IPI_DELAY; -module_param(rcu_task_ipi_delay, int, 0644); - /* Control stall timeouts. Disable with <=3D 0, otherwise jiffies till st= all. */ #define RCU_TASK_BOOT_STALL_TIMEOUT (HZ * 30) #define RCU_TASK_STALL_TIMEOUT (HZ * 60 * 10) diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TRACE01 b/tools= /testing/selftests/rcutorture/configs/rcu/TRACE01 index 85b407467454a2..18efab346381a4 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/TRACE01 +++ b/tools/testing/selftests/rcutorture/configs/rcu/TRACE01 @@ -10,5 +10,4 @@ CONFIG_PROVE_LOCKING=3Dn #CHECK#CONFIG_PROVE_RCU=3Dn CONFIG_FORCE_TASKS_TRACE_RCU=3Dy #CHECK#CONFIG_TASKS_TRACE_RCU=3Dy -CONFIG_TASKS_TRACE_RCU_READ_MB=3Dy CONFIG_RCU_EXPERT=3Dy diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TRACE02 b/tools= /testing/selftests/rcutorture/configs/rcu/TRACE02 index 9003c56cd76484..8da390e8282977 100644 --- a/tools/testing/selftests/rcutorture/configs/rcu/TRACE02 +++ b/tools/testing/selftests/rcutorture/configs/rcu/TRACE02 @@ -9,6 +9,5 @@ CONFIG_PROVE_LOCKING=3Dy #CHECK#CONFIG_PROVE_RCU=3Dy CONFIG_FORCE_TASKS_TRACE_RCU=3Dy #CHECK#CONFIG_TASKS_TRACE_RCU=3Dy -CONFIG_TASKS_TRACE_RCU_READ_MB=3Dn CONFIG_RCU_EXPERT=3Dy CONFIG_DEBUG_OBJECTS=3Dy --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25B9B22DFA4; Tue, 23 Sep 2025 14:21:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637296; cv=none; b=skpTZbqZLUIUosztpggXtByvG4WDnjVEUyvRNnRfp53kp6Tzlhq538uUIDzOl1UROTb5HFo3CMdEuCFmdHhBioYm7LWhWtoKGdejvac6m9hHWPCJ9KSq702gb9bLZWzRCb2rmrkyNLUlrqx1lFLHT5dJ2SZVLMnYnC1zE6IKyNY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637296; c=relaxed/simple; bh=qnaBMWA94ufanssrOAqRkiBTs8H62VLm1Z5ULAg5fZ0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RO1pf4xdgenkjpSqiWyfv2uNztmBIjN/uERwwDJEkRuL3UjRCHpsZkYkd3sfTEFbCH8WsL27ly+3mUo2525i6LD/NunJGJIxng/WGVSBq8+aiw981hZYN/YwAlaWvWgbF3WlQnE6IGXav4/d8pQjW9bEsItuhNaC0EOchsYgpLU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZmSJei00; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZmSJei00" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CD978C113D0; Tue, 23 Sep 2025 14:21:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637295; bh=qnaBMWA94ufanssrOAqRkiBTs8H62VLm1Z5ULAg5fZ0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZmSJei00IJME6ljBHGfRq6vFRO1zny7h5HcG9np//bWh5BsdA0YjbsvAIeab/hDb7 ywF7Q8uD63Idrf3xx+GQnJo27ng1MxN0jQk52GWlOBa8wIvEyJ0pnvHshIvYkM2DeT 97zwSrCgE8aKM4zNHOF5/dW+2v6Qs4yXye7lX8rkzVNkgTQ7myA2SwgEJ9fKQqllF0 oS4mW1D/g05OWWBax9Uf/CzPiyMJMvjqjCNjEBQcbmARsynA6vMJ+tp0M+lqDbdtZ5 uZ+eTts8OkH74l4R0FUcbTYy2X19MbwszvsLKoTEe1WyZqy70JNE5Fzh//6pHw+tOR f5t3B/QJEhRSQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E2384CE14F7; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 17/34] srcu: Create a DEFINE_SRCU_FAST() Date: Tue, 23 Sep 2025 07:20:19 -0700 Message-Id: <20250923142036.112290-17-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates a DEFINE_SRCU_FAST() that is similar to DEFINE_SRCU(), but which creates an srcu_struct that is usable only by readers initiated by srcu_read_lock_fast() and friends. Note that DEFINE_SRCU_FAST() is usable only by built-in code, not from loadable modules. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcutiny.h | 1 + include/linux/srcutree.h | 20 +++++++++++++++----- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 51ce25f07930ee..00e5f05288d5e7 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -45,6 +45,7 @@ void srcu_drive_gp(struct work_struct *wp); */ #define DEFINE_SRCU(name) \ struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name) +#define DEFINE_SRCU_FAST(name) DEFINE_SRCU(name) #define DEFINE_STATIC_SRCU(name) \ static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name) =20 diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 42098e0fa0b7dd..1adc58d2ab6425 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -189,23 +189,33 @@ struct srcu_struct { * init_srcu_struct(&my_srcu); * * See include/linux/percpu-defs.h for the rules on per-CPU variables. + * + * DEFINE_SRCU_FAST() creates an srcu_struct and associated structures + * whose readers must be of the SRCU-fast variety. DEFINE_SRCU_FAST() + * cannot be used within modules due to there being no place to + * put the desired ->srcu_reader_flavor value. If in-module use of + * DEFINE_SRCU_FAST() becomes necessary, the srcu_struct structure will + * need to grow in order to store this value. */ #ifdef MODULE -# define __DEFINE_SRCU(name, is_static) \ +# define __DEFINE_SRCU(name, fast, is_static) \ static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s= rcu_usage); \ is_static struct srcu_struct name =3D __SRCU_STRUCT_INIT_MODULE(name, nam= e##_srcu_usage); \ extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ __section("___srcu_struct_ptrs") =3D &name #else -# define __DEFINE_SRCU(name, is_static) \ - static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ +# define __DEFINE_SRCU(name, fast, is_static) \ + static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data) =3D { \ + .srcu_reader_flavor =3D fast, \ + }; \ static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s= rcu_usage); \ is_static struct srcu_struct name =3D \ __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data) +#define DEFINE_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST,= /* not static */) #endif -#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) -#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static) +#define DEFINE_SRCU(name) __DEFINE_SRCU(name, 0, /* not static */) +#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, 0, static) =20 int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B52C7327A17; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637314; cv=none; b=IpUIQz5AhLI040GhK+kcYMITGhCLbK6Ycx+G10RfPebjhBYaI97jfwEfW9U58SAn5SNV8arpCXPJnEZuvGy9BszT2unvZaUv2w3fzG4DxkETshzOikyIcj39BICQuJ/0xZDJeAEsngk9Umt6aPIgUbWeMDsnI+Z/asKXZu5cQC4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637314; c=relaxed/simple; bh=BRVx7fQjFG64Qiug5K5umKC5WWCM8o4bRG4z4HvlEEQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bcgK5FImLL0dhGpk3GNwtwqiszMebHGQK1UaQ8yJoule3zp1Eri0D4KOzKMUJXFePXgAvLfidkjkbUqxRtcfjpVX0d25yrZloPxtoj2ELSkNvdfhiVD7rT7dpkhKqRDthN/qBa1nCTgN5NsRxMrIIr42PtB418ELXBDPL0kBpww= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OApka8aa; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OApka8aa" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 68E3CC113D0; Tue, 23 Sep 2025 14:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637314; bh=BRVx7fQjFG64Qiug5K5umKC5WWCM8o4bRG4z4HvlEEQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OApka8aalsFuqOpnme+RbDp7WChtnhdTjYyBgRE2zvyTWJ6IZiPLjICSoWiQARijg H96sIbe7E7FRvaoYXORN1vX3CSy+J/13+drt/c/9TYi8ydnd6XYCW6S3JtaUm/mZWN S4unj1Zg1c7tDTNxoRIjk1WCnEjImBHz77zizHFd2hz0N1aaLrrMXd00u4PfhzYZD1 k8SkCjaI6FiBDvo8utS/PhWuXivizSoC+TZhbY95q2qB4hrzFi7XUQtfc/k37lb3Ro rleizDmIKCoMF/NNt4jp7c66ayD0WghxHeyYeHBnXz247jqMZYpuSEI8loD6+n+V2m IAOKHf3GFuxHg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E5447CE1538; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 18/34] rcu: Use smp_mb() only when necessary in RCU Tasks Trace readers Date: Tue, 23 Sep 2025 07:20:20 -0700 Message-Id: <20250923142036.112290-18-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The rcu_read_{,un}lock_{,tasks_}trace() functions need to use smp_mb() only if invoked where RCU is not watching, that is, from locations where a call to rcu_is_watching() would return false. In architectures that define the ARCH_WANTS_NO_INSTR Kconfig option, use of noinstr and friends ensures that tracing happens only where RCU is watching, so those architectures can dispense entirely with the read-side calls to smp_mb(). Other architectures include these read-side calls by default, but in many installations there might be either larger than average tolerance for risk, prohibition of removing tracing on a running system, or careful review and approval of removing of tracing. Such installations can build their kernels with CONFIG_TASKS_TRACE_RCU_NO_MB=3Dy to avoid those read-side calls to smp_mb(), thus accepting responsibility for run-time removal of tracing from code regions that RCU is not watching. Those wishing to disable read-side memory barriers for an entire architecture can select this TASKS_TRACE_RCU_NO_MB Kconfig option, hence the polarity. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/rcupdate_trace.h | 32 ++++++++++++++++++-------------- kernel/rcu/Kconfig | 23 +++++++++++++++++++++++ kernel/rcu/tasks.h | 7 ++++++- 3 files changed, 47 insertions(+), 15 deletions(-) diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index b87151e6b23881..7f7977fb56aca5 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -48,10 +48,11 @@ static inline int rcu_read_lock_trace_held(void) */ static inline struct srcu_ctr __percpu *rcu_read_lock_tasks_trace(void) { - struct srcu_ctr __percpu *ret =3D srcu_read_lock_fast(&rcu_tasks_trace_sr= cu_struct); + struct srcu_ctr __percpu *ret =3D __srcu_read_lock_fast(&rcu_tasks_trace_= srcu_struct); =20 - if (IS_ENABLED(CONFIG_ARCH_WANTS_NO_INSTR)) - smp_mb(); + rcu_try_lock_acquire(&rcu_tasks_trace_srcu_struct.dep_map); + if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB)) + smp_mb(); // Provide ordering on noinstr-incomplete architectures. return ret; } =20 @@ -66,9 +67,10 @@ static inline struct srcu_ctr __percpu *rcu_read_lock_ta= sks_trace(void) */ static inline void rcu_read_unlock_tasks_trace(struct srcu_ctr __percpu *s= cp) { - if (!IS_ENABLED(CONFIG_ARCH_WANTS_NO_INSTR)) - smp_mb(); - srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp); + if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB)) + smp_mb(); // Provide ordering on noinstr-incomplete architectures. + __srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp); + srcu_lock_release(&rcu_tasks_trace_srcu_struct.dep_map); } =20 /** @@ -87,14 +89,15 @@ static inline void rcu_read_lock_trace(void) { struct task_struct *t =3D current; =20 + rcu_try_lock_acquire(&rcu_tasks_trace_srcu_struct.dep_map); if (t->trc_reader_nesting++) { // In case we interrupted a Tasks Trace RCU reader. - rcu_try_lock_acquire(&rcu_tasks_trace_srcu_struct.dep_map); return; } barrier(); // nesting before scp to protect against interrupt handler. - t->trc_reader_scp =3D srcu_read_lock_fast(&rcu_tasks_trace_srcu_struct); - smp_mb(); // Placeholder for more selective ordering + t->trc_reader_scp =3D __srcu_read_lock_fast(&rcu_tasks_trace_srcu_struct); + if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB)) + smp_mb(); // Placeholder for more selective ordering } =20 /** @@ -111,13 +114,14 @@ static inline void rcu_read_unlock_trace(void) struct srcu_ctr __percpu *scp; struct task_struct *t =3D current; =20 - smp_mb(); // Placeholder for more selective ordering scp =3D t->trc_reader_scp; barrier(); // scp before nesting to protect against interrupt handler. - if (!--t->trc_reader_nesting) - srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp); - else - srcu_lock_release(&rcu_tasks_trace_srcu_struct.dep_map); + if (!--t->trc_reader_nesting) { + if (!IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB)) + smp_mb(); // Placeholder for more selective ordering + __srcu_read_unlock_fast(&rcu_tasks_trace_srcu_struct, scp); + } + srcu_lock_release(&rcu_tasks_trace_srcu_struct.dep_map); } =20 /** diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 73a6cc364628b5..6a319e2926589f 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -142,6 +142,29 @@ config TASKS_TRACE_RCU default n select IRQ_WORK =20 +config TASKS_TRACE_RCU_NO_MB + bool "Override RCU Tasks Trace inclusion of read-side memory barriers" + depends on RCU_EXPERT && TASKS_TRACE_RCU + default ARCH_WANTS_NO_INSTR + help + This option prevents the use of read-side memory barriers in + rcu_read_lock_tasks_trace() and rcu_read_unlock_tasks_trace() + even in kernels built with CONFIG_ARCH_WANTS_NO_INSTR=3Dn, that is, + in kernels that do not have noinstr set up in entry/exit code. + By setting this option, you are promising to carefully review + use of ftrace, BPF, and friends to ensure that no tracing + operation is attached to a function that runs in that portion + of the entry/exit code that RCU does not watch, that is, + where rcu_is_watching() returns false. Alternatively, you + might choose to never remove traces except by rebooting. + + Those wishing to disable read-side memory barriers for an entire + architecture can select this Kconfig option, hence the polarity. + + Say Y here if you need speed and will review use of tracing. + Say N here for certain esoteric testing of RCU itself. + Take the default if you are unsure. + config RCU_STALL_COMMON def_bool TREE_RCU help diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 833e180db744f2..bf1226834c9423 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1600,8 +1600,13 @@ static inline void rcu_tasks_bootup_oddness(void) {} // Tracing variant of Tasks RCU. This variant is designed to be used // to protect tracing hooks, including those of BPF. This variant // is implemented via a straightforward mapping onto SRCU-fast. +// DEFINE_SRCU_FAST() is required because rcu_read_lock_trace() must +// use __srcu_read_lock_fast() in order to bypass the rcu_is_watching() +// checks in kernels built with CONFIG_TASKS_TRACE_RCU_NO_MB=3Dn, which al= so +// bypasses the srcu_check_read_flavor_force() that would otherwise mark +// rcu_tasks_trace_srcu_struct as needing SRCU-fast readers. =20 -DEFINE_SRCU(rcu_tasks_trace_srcu_struct); +DEFINE_SRCU_FAST(rcu_tasks_trace_srcu_struct); EXPORT_SYMBOL_GPL(rcu_tasks_trace_srcu_struct); =20 #endif /* #else #ifdef CONFIG_TASKS_TRACE_RCU */ --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D74332897F; Tue, 23 Sep 2025 14:21:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; cv=none; b=Tq8JtYvCU3sbdLg+ImMevpqfEXXOyZaJaDzADfYTkzbQZghcXoQ6CZgsqGkzarPuSDVjYXCcesrLqnTYwHXxmCcq6+rZGDOnvxe8sCe+odQU8lwXwwZy+4uhjn65Og53bVcljKexOwAgSQg1eDkJ5ZNBl9FvXL418A/P/6V1NDY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; c=relaxed/simple; bh=1LD05Jje7JY4+QRieuy7/gMogSMrNyvAzrT45QnCY+Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=A0ncqXQK4571eA0yGC/aDfSAr7mMIMUYv7P3DZljR8YpNX4EkHde2gbkqFoCbQNbLQJFOaGyswIUmELWGd+NuPiDrN9Qh3SouRUCFbxldGJQ9mMshfZ2neTxnB5rkziBORrj+Usk2apH9nX10DnBBdGuihsoHOSfa5sqB15NOhs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SrV3QjWB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SrV3QjWB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A2312C19424; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637316; bh=1LD05Jje7JY4+QRieuy7/gMogSMrNyvAzrT45QnCY+Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SrV3QjWBTs00EdQF3t4xT8BmwIulEwEyj+2C86PmYn1Z/ImvxTZ9aSTdntkKAR/sF E+PZebkgvIiG6KvcvdL17qNv1mLHqjr7+xzsIUjKUVt6BCgNKOJkLvUmaZT1tt6Wm7 jinoFsHeH7f8/J88yNEFbgbO0u00qyTDdah9gv+TNb7ibNg36TRGJtDWZAAW2lo66V vOcLnNF2KtWnLTLqjsRKGw3I8onscKGVutCJ9/syMd7CqXzDGaPay3jgh4d77u6cQE 2byaePensyVGX56JAqm/JmJRZqazyOUXlG0c3/YIOTgxrRDsuGcCNBztksy3CwmT7d b5BGIqpjr41Qw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E7D2DCE158D; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 19/34] rcu: Update Requirements.rst for RCU Tasks Trace Date: Tue, 23 Sep 2025 07:20:21 -0700 Message-Id: <20250923142036.112290-19-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit updates the documentation to declare that RCU Tasks Trace is implemented as a thin wrapper around SRCU-fast. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- .../RCU/Design/Requirements/Requirements.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Docum= entation/RCU/Design/Requirements/Requirements.rst index f24b3c0b9b0dc6..4a116d7a564edc 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2779,12 +2779,12 @@ Tasks Trace RCU ~~~~~~~~~~~~~~~ =20 Some forms of tracing need to sleep in readers, but cannot tolerate -SRCU's read-side overhead, which includes a full memory barrier in both -srcu_read_lock() and srcu_read_unlock(). This need is handled by a -Tasks Trace RCU that uses scheduler locking and IPIs to synchronize with -readers. Real-time systems that cannot tolerate IPIs may build their -kernels with ``CONFIG_TASKS_TRACE_RCU_READ_MB=3Dy``, which avoids the IPIs= at -the expense of adding full memory barriers to the read-side primitives. +SRCU's read-side overhead, which includes a full memory barrier in +both srcu_read_lock() and srcu_read_unlock(). This need is handled by +a Tasks Trace RCU API implemented as thin wrappers around SRCU-fast, +which avoids the read-side memory barriers, at least for architectures +that apply noinstr to kernel entry/exit code (or that build with +``CONFIG_TASKS_TRACE_RCU_NO_MB=3Dy``. =20 The tasks-trace-RCU API is also reasonably compact, consisting of rcu_read_lock_trace(), rcu_read_unlock_trace(), --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB6BC267729; Tue, 23 Sep 2025 14:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637303; cv=none; b=bbZfao31XNzXZRIdPDHATIfbw5xlKttgemj3Vzd7kEhFTahf/sqwBWKHVlM/a2AEnVjvXGKRfDuyyd0Q2ersE0OteohuRfQHW6ddAOk8s8EsmurOqQocWnxhz1MJU6Wd0M2bTtRUcWgmeQMhQqZuu/l4cjPpYcu8IulukfIXczc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637303; c=relaxed/simple; bh=nj4NUdLqqgo6c9WwY9DR0wHNaXFdGzVpfYVSNHOxwxY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WAINGsYjokvMLCnf1cjRAK3KoPEGniz+YJ6+CMXrR2R2nOoZgGvhQ8v/Y2IN62pX4ZSDNmpbacgSY5+1uT+ow8BgGGI+s3pye0L0xo8tbCH/rglVFTseTFy6x0O3h9EUt2kucms2Qs/GyRSSqf3sJzRUb/dQtfcMfTK9J1MjQ/0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RWqkTeZI; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RWqkTeZI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB22DC19422; Tue, 23 Sep 2025 14:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637302; bh=nj4NUdLqqgo6c9WwY9DR0wHNaXFdGzVpfYVSNHOxwxY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RWqkTeZI6PpwDPOCkx4bznJ/JVPtzc/R2z1wVVn4YiU+L6I/UtskUUrc8GwuZw3ws iu6lkC0drEptZoDCc0x3rZBN8ABjiEXwzi4dQoKhcdKjMvkywdg3Ury/+TkbwwYSIX 7bVQN5y5mFZ737HQXxbjLUSdSgy2HfPQs/ULPb5WXwpHH2eF0d9JmBxdqS+UcvkRN3 IIitf5CVSaErbmh6WU13s4YX6l+5EV4shXSjFsKojQgyXcNsnezmOm/ojj79358Qwq keabPrYtr6jeZ5kxfH0c04nDQpGjuk8ZD9Jv3ItJncFYmmg8oCoFMu3Ree931kA47+ Z2hlwUms8aiRA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EAD94CE15CE; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andy Whitcroft , Joe Perches , Dwaipayan Ray , Lukas Bulwahn , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 20/34] checkpatch: Deprecate rcu_read_{,un}lock_trace() Date: Tue, 23 Sep 2025 07:20:22 -0700 Message-Id: <20250923142036.112290-20-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Uses of rcu_read_lock_trace() and rcu_read_unlock_trace() are better served by the new rcu_read_lock_tasks_trace() and rcu_read_unlock_tasks_trace() APIs. Therefore, mark the old APIs as deprecated. Signed-off-by: Paul E. McKenney Cc: Andy Whitcroft Cc: Joe Perches Cc: Dwaipayan Ray Cc: Lukas Bulwahn Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: Acked-by: Joe Perches --- scripts/checkpatch.pl | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl index e722dd6fa8ef3d..3bb7d35a5cfcba 100755 --- a/scripts/checkpatch.pl +++ b/scripts/checkpatch.pl @@ -860,6 +860,8 @@ our %deprecated_apis =3D ( "kunmap" =3D> "kunmap_local", "kmap_atomic" =3D> "kmap_local_page", "kunmap_atomic" =3D> "kunmap_local", + "rcu_read_lock_trace" =3D> "rcu_read_lock_tasks_trace", + "rcu_read_unlock_trace" =3D> "rcu_read_unlock_tasks_trace", ); =20 #Create a search pattern for all these strings to speed up a loop below --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2FCDB328980; Tue, 23 Sep 2025 14:21:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; cv=none; b=cCOGp7nxJxeECjEFWpZcrveINr1V4A1zFMBiU7aOYmd2Gm+C1LdrhAijYuLDpbwjNaVhlF7xCpJoVf7SXWLMvUmSVnKOCKxArEPeXiSFGipvb2xk74qRztPkFT8qWXvsS42K6oYzh03UoagOlJf5enXjTSytALCxjhWpoR3F4ME= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; c=relaxed/simple; bh=5+6cap8BHuj7LD+F6jHOG4xWIftMKuTGrdQG1g1VuNA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JoF6jZ6ZtFi0lkOXMBUj4FYCiLtcfHuoD216t7pQJN5q3wrM/AHZRqLj0xDIKcOdNNNSl86Y6uxiaF8T5FITq4lfqjlljDnAMY7IWjhbJU5dsqxD9wELKGj5KnkvCSEgdWLfi8l1kKcOlEomoYddAFFCW2mWNu3R7QC7kYfv/k8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=H2jd3S49; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="H2jd3S49" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5AF92C19425; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637315; bh=5+6cap8BHuj7LD+F6jHOG4xWIftMKuTGrdQG1g1VuNA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H2jd3S491QmBDmq9L8to24wB3YThSK2EAkQSyg4uPIF0zDugc6D9vS9MioJZDH4oP mGyzNGJEeTcAr2m/Q897xpXiH//1kZZ7bDJ6OZ5/R+UYwmDGzOAlaW1Ks42FE57sBk g93tBLJssCeeHcXUfQHqXziHJ5Fg3dLPSDdLwSiuPQRq4lDy618j8mAT6u8OoGSKEb RzMRlMUMWT4IpIG2aNwt1WWPYZFiazBP7psKDck+Rxdwv/BmL4cmsHIYb3tKclkIKw d7ZiHHOgfCGmo/4ih77MVs7/sez1nQwoFOEfQMlHWSV8y3X0tseQF84U7e0VzpQoea 0KU5l0jkAyhqg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id ED820CE161F; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Leon Hwang , Alexei Starovoitov Subject: [PATCH 21/34] rcu: Mark diagnostic functions as notrace Date: Tue, 23 Sep 2025 07:20:23 -0700 Message-Id: <20250923142036.112290-21-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The rcu_lockdep_current_cpu_online(), rcu_read_lock_sched_held(), rcu_read_lock_held(), rcu_read_lock_bh_held(), rcu_read_lock_any_held() are used by tracing-related code paths, so putting traces on them is unlikely to make anyone happy. This commit therefore marks them all "notrace". Reported-by: Leon Hwang Reported-by: Alexei Starovoitov Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- kernel/rcu/update.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 31690ffa452afe..8ddd07fed36334 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4021,7 +4021,7 @@ bool rcu_cpu_online(int cpu) * RCU on an offline processor during initial boot, hence the check for * rcu_scheduler_fully_active. */ -bool rcu_lockdep_current_cpu_online(void) +bool notrace rcu_lockdep_current_cpu_online(void) { struct rcu_data *rdp; bool ret =3D false; diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index c912b594ba987f..dfeba9b3539508 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -117,7 +117,7 @@ static bool rcu_read_lock_held_common(bool *ret) return false; } =20 -int rcu_read_lock_sched_held(void) +int notrace rcu_read_lock_sched_held(void) { bool ret; =20 @@ -342,7 +342,7 @@ EXPORT_SYMBOL_GPL(debug_lockdep_rcu_enabled); * Note that rcu_read_lock() is disallowed if the CPU is either idle or * offline from an RCU perspective, so check for those as well. */ -int rcu_read_lock_held(void) +int notrace rcu_read_lock_held(void) { bool ret; =20 @@ -367,7 +367,7 @@ EXPORT_SYMBOL_GPL(rcu_read_lock_held); * Note that rcu_read_lock_bh() is disallowed if the CPU is either idle or * offline from an RCU perspective, so check for those as well. */ -int rcu_read_lock_bh_held(void) +int notrace rcu_read_lock_bh_held(void) { bool ret; =20 @@ -377,7 +377,7 @@ int rcu_read_lock_bh_held(void) } EXPORT_SYMBOL_GPL(rcu_read_lock_bh_held); =20 -int rcu_read_lock_any_held(void) +int notrace rcu_read_lock_any_held(void) { bool ret; =20 --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA66D327A1B; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637314; cv=none; b=qoTgD9ze3ANwzmdtwxQ/yniqg9A1M6ncF2H+zzoviQ4wHudu8Js665FzJacqxxMMLvZLUCh+jR+eAJCxvvEXU7pio5zUS5cELrF+KoayAzJmwA0ZpKkphTmh5LSWA7kTevQEIhIdm/dS+cLC+3xW6yYfzhZblwAxCxL4LnyZiwc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637314; c=relaxed/simple; bh=sjVFcWrDJfqVEdr4Iyjd1Hv55BK3Tses2HzyitawB4I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=amyDrEc2JduQY8xwCEdIlRGUaJlYrPGEvik9m2w4b9/KkjJPRw2+qt+WLzCmX58Eyb/QczzR+xFQOW5lG4XuV/LoNeSyEuTLFLxXta9/qV9twfSzN+DxUS6Pl3Gpry/+LwmWcu7kgUKlh7WmSlAw6znfKaTjvuhsIV+vZCqWE4g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=snVY875E; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="snVY875E" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 635DBC4CEF5; Tue, 23 Sep 2025 14:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637314; bh=sjVFcWrDJfqVEdr4Iyjd1Hv55BK3Tses2HzyitawB4I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=snVY875Eb2Lb7LfZ7EomZznXyL9uZZZMayd9fE1PcbfFAJRgS3F6fK9Db+v1KuNtD b5SlywBoINrzsyOtweljjXLRz5DJfGU9W5txp5LkPd2FWb7waGdc5GWlaRkh6cUu6a 5uCR4UAjXdgc7Jtg8qqQi1N2iFpzApbk6lCOHXIraDY9kGUX/afkQySElwUd3AMGHs vaVf8br7rc4TZrC1n2Lt6MYDuV95tDSBEz1Dgjr9gebinlykCd/972KlYP251J0Zx4 VJwgP9S5mKuxT+r0FfgB2xtTNWQJ3Zxh1PWOn4OExQxw5sIzFeHKMr9ImaBShTW9P7 Ex1EdAQtnRZWw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id F07D1CE167D; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 22/34] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast Date: Tue, 23 Sep 2025 07:20:24 -0700 Message-Id: <20250923142036.112290-22-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current use of guard(preempt_notrace)() within __DECLARE_TRACE() to protect invocation of __DO_TRACE_CALL() means that BPF programs attached to tracepoints are non-preemptible. This is unhelpful in real-time systems, whose users apparently wish to use BPF while also achieving low latencies. (Who knew?) One option would be to use preemptible RCU, but this introduces many opportunities for infinite recursion, which many consider to be counterproductive, especially given the relatively small stacks provided by the Linux kernel. These opportunities could be shut down by sufficiently energetic duplication of code, but this sort of thing is considered impolite in some circles. Therefore, use the shiny new SRCU-fast API, which provides somewhat faster readers than those of preemptible RCU, at least on my laptop, where task_struct access is more expensive than access to per-CPU variables. And SRCU fast provides way faster readers than does SRCU, courtesy of being able to avoid the read-side use of smp_mb(). Also, it is quite straightforward to create srcu_read_{,un}lock_fast_notrace() functions. While in the area, SRCU now supports early boot call_srcu(). Therefore, remove the checks that used to avoid such use from rcu_free_old_probes() before this commit was applied: e53244e2c893 ("tracepoint: Remove SRCU protection") The current commit can be thought of as an approximate revert of that commit, with some compensating additions of preemption disabling pointed out by Steven Rostedt (thank you, Steven!). This preemption disabling uses guard(preempt_notrace)(), and while in the area a couple of other use cases were also converted to guards. However, Yonghong Song points out that BPF expects non-sleepable BPF programs to remain on the same CPU, which means that migration must be disabled whenever preemption remains enabled. In addition, non-RT kernels have performance expectations on BPF that would be violated by allowing the BPF programs to be preempted. Therefore, continue to disable preemption in non-RT kernels, and protect the BPF program with both SRCU and migration disabling for RT kernels, and even then only if preemption is not already disabled. [ paulmck: Apply kernel test robot and Yonghong Song feedback. ] Link: https://lore.kernel.org/all/20250613152218.1924093-1-bigeasy@linutron= ix.de/ Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/tracepoint.h | 45 ++++++++++++++++++++++-------------- include/trace/perf.h | 4 ++-- include/trace/trace_events.h | 4 ++-- kernel/tracepoint.c | 21 ++++++++++++++++- 4 files changed, 52 insertions(+), 22 deletions(-) diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 826ce3f8e1f851..9f8b19cd303acc 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -33,6 +33,8 @@ struct trace_eval_map { =20 #define TRACEPOINT_DEFAULT_PRIO 10 =20 +extern struct srcu_struct tracepoint_srcu; + extern int tracepoint_probe_register(struct tracepoint *tp, void *probe, void *data); extern int @@ -115,7 +117,10 @@ void for_each_tracepoint_in_module(struct module *mod, static inline void tracepoint_synchronize_unregister(void) { synchronize_rcu_tasks_trace(); - synchronize_rcu(); + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + synchronize_srcu(&tracepoint_srcu); + else + synchronize_rcu(); } static inline bool tracepoint_is_faultable(struct tracepoint *tp) { @@ -266,23 +271,29 @@ static inline struct tracepoint *tracepoint_ptr_deref= (tracepoint_ptr_t *p) return static_branch_unlikely(&__tracepoint_##name.key);\ } =20 -#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \ +#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \ __DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_pro= to)) \ - static inline void __do_trace_##name(proto) \ - { \ - if (cond) { \ - guard(preempt_notrace)(); \ - __DO_TRACE_CALL(name, TP_ARGS(args)); \ - } \ - } \ - static inline void trace_##name(proto) \ - { \ - if (static_branch_unlikely(&__tracepoint_##name.key)) \ - __do_trace_##name(args); \ - if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \ - WARN_ONCE(!rcu_is_watching(), \ - "RCU not watching for tracepoint"); \ - } \ + static inline void __do_trace_##name(proto) \ + { \ + if (cond) { \ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && preemptible()) { \ + guard(srcu_fast_notrace)(&tracepoint_srcu); \ + guard(migrate)(); \ + __DO_TRACE_CALL(name, TP_ARGS(args)); \ + } else { \ + guard(preempt_notrace)(); \ + __DO_TRACE_CALL(name, TP_ARGS(args)); \ + } \ + } \ + } \ + static inline void trace_##name(proto) \ + { \ + if (static_branch_unlikely(&__tracepoint_##name.key)) \ + __do_trace_##name(args); \ + if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \ + WARN_ONCE(!rcu_is_watching(), \ + "RCU not watching for tracepoint"); \ + } \ } =20 #define __DECLARE_TRACE_SYSCALL(name, proto, args, data_proto) \ diff --git a/include/trace/perf.h b/include/trace/perf.h index a1754b73a8f55b..348ad1d9b5566e 100644 --- a/include/trace/perf.h +++ b/include/trace/perf.h @@ -71,6 +71,7 @@ perf_trace_##call(void *__data, proto) \ u64 __count __attribute__((unused)); \ struct task_struct *__task __attribute__((unused)); \ \ + guard(preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ } =20 @@ -85,9 +86,8 @@ perf_trace_##call(void *__data, proto) \ struct task_struct *__task __attribute__((unused)); \ \ might_fault(); \ - preempt_disable_notrace(); \ + guard(preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ - preempt_enable_notrace(); \ } =20 /* diff --git a/include/trace/trace_events.h b/include/trace/trace_events.h index 4f22136fd4656c..fbc07d353be6b6 100644 --- a/include/trace/trace_events.h +++ b/include/trace/trace_events.h @@ -436,6 +436,7 @@ __DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args)= , PARAMS(tstruct), \ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \ { \ + guard(preempt_notrace)(); \ do_trace_event_raw_event_##call(__data, args); \ } =20 @@ -447,9 +448,8 @@ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \ { \ might_fault(); \ - preempt_disable_notrace(); \ + guard(preempt_notrace)(); \ do_trace_event_raw_event_##call(__data, args); \ - preempt_enable_notrace(); \ } =20 /* diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 62719d2941c900..21bb6779821476 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -25,6 +25,9 @@ enum tp_func_state { extern tracepoint_ptr_t __start___tracepoints_ptrs[]; extern tracepoint_ptr_t __stop___tracepoints_ptrs[]; =20 +DEFINE_SRCU_FAST(tracepoint_srcu); +EXPORT_SYMBOL_GPL(tracepoint_srcu); + enum tp_transition_sync { TP_TRANSITION_SYNC_1_0_1, TP_TRANSITION_SYNC_N_2_1, @@ -34,6 +37,7 @@ enum tp_transition_sync { =20 struct tp_transition_snapshot { unsigned long rcu; + unsigned long srcu_gp; bool ongoing; }; =20 @@ -46,6 +50,7 @@ static void tp_rcu_get_state(enum tp_transition_sync sync) =20 /* Keep the latest get_state snapshot. */ snapshot->rcu =3D get_state_synchronize_rcu(); + snapshot->srcu_gp =3D start_poll_synchronize_srcu(&tracepoint_srcu); snapshot->ongoing =3D true; } =20 @@ -56,6 +61,8 @@ static void tp_rcu_cond_sync(enum tp_transition_sync sync) if (!snapshot->ongoing) return; cond_synchronize_rcu(snapshot->rcu); + if (!poll_state_synchronize_srcu(&tracepoint_srcu, snapshot->srcu_gp)) + synchronize_srcu(&tracepoint_srcu); snapshot->ongoing =3D false; } =20 @@ -101,17 +108,29 @@ static inline void *allocate_probes(int count) return p =3D=3D NULL ? NULL : p->probes; } =20 -static void rcu_free_old_probes(struct rcu_head *head) +static void srcu_free_old_probes(struct rcu_head *head) { kfree(container_of(head, struct tp_probes, rcu)); } =20 +static void rcu_free_old_probes(struct rcu_head *head) +{ + call_srcu(&tracepoint_srcu, head, srcu_free_old_probes); +} + static inline void release_probes(struct tracepoint *tp, struct tracepoint= _func *old) { if (old) { struct tp_probes *tp_probes =3D container_of(old, struct tp_probes, probes[0]); =20 + /* + * Tracepoint probes are protected by either RCU or + * Tasks Trace RCU and also by SRCU. By calling the SRCU + * callback in the [Tasks Trace] RCU callback we cover + * both cases. So let us chain the SRCU and [Tasks Trace] + * RCU callbacks to wait for both grace periods. + */ if (tracepoint_is_faultable(tp)) call_rcu_tasks_trace(&tp_probes->rcu, rcu_free_old_probes); else --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5847F23D7E0; Tue, 23 Sep 2025 14:21:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637296; cv=none; b=Ca/2YM0qoh70e1JR7EIi21Wi0EJSF3RvUENuc3pnjrMECzgtFulnE8aJbWGfLOcsXofoAnDrXUoZ4fYYiMuG+h6r/eP2mCLRIMsT9coPBn0StxnrGN1Zje9THVbGAnOt95dVm3ovxm07PCyp67JKrfBsH1xRZzOhsGVxgGWyCbQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637296; c=relaxed/simple; bh=atbo6CDG84sqK8ypx2s+xqkRme+N3DbN/SaGf0JqKIM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LuZ6dp2cYH6pW7N9KHpaaHr03Cu45oijUclQ2rtUZhmtbX9qlm0+ObspNQ6dxOCvPTRJEOXmnVzE/7umU6BaZjkEnKHb1T9FctgK+qmVDXLkzAdJtqsnr/X1p/5b4CjRmRl7yCgdBf2Wlq3ZVEgy9cJXYBnIz65Cltf8UK4deJE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UsM+vCG2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UsM+vCG2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D4B9BC113CF; Tue, 23 Sep 2025 14:21:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637296; bh=atbo6CDG84sqK8ypx2s+xqkRme+N3DbN/SaGf0JqKIM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UsM+vCG2YgxCBd7N9SeiDxI9C3U8QUsa8rp12QFNo2A+cFAaz0CxahxaiWO+pzsSD osTWw31z+Y1T5MTtCjC2JAkSp7xawRlVXIY/2ri/VRFiRYPBlppgSy5ATDAwwIOnJK QSPYkU79nbBcHOKVHsq7F7l4jW5VH3chd0Dr2LniwcJEEA1Lmo0Dp02gu71NUZCIQA W1rJBZod7zGZYixDEKHtM5f6Vkw7fyKgSxCa4s9x7JpzZ5vCk3x8vz7FA2gkHiuQkn Ax/stwJlfRHQu5FRdPoE5qTtn7woEEDswNqqO59/9TnIjicyHJwh7BmmAqLLUvhNi0 SMKiJkHviLRBw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id F3620CE1688; Tue, 23 Sep 2025 07:20:37 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 23/34] srcu: Create an srcu_expedite_current() function Date: Tue, 23 Sep 2025 07:20:25 -0700 Message-Id: <20250923142036.112290-23-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates an srcu_expedite_current() function that expedites the current (and possibly the next) SRCU grace period for the specified srcu_struct structure. This functionality will be inherited by RCU Tasks Trace courtesy of its mapping to SRCU fast. If the current SRCU grace period is already waiting, that wait will complete before the expediting takes effect. If there is no SRCU grace period in flight, this function might well create one. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/srcutiny.h | 1 + include/linux/srcutree.h | 8 ++++++ kernel/rcu/srcutree.c | 58 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 00e5f05288d5e7..941e8210479607 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -104,6 +104,7 @@ static inline void srcu_barrier(struct srcu_struct *ssp) synchronize_srcu(ssp); } =20 +static inline void srcu_expedite_current(struct srcu_struct *ssp) { } #define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) #define srcu_check_read_flavor_force(ssp, read_flavor) do { } while (0) =20 diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 1adc58d2ab6425..4a5d1c0e7db361 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -42,6 +42,8 @@ struct srcu_data { struct timer_list delay_work; /* Delay for CB invoking */ struct work_struct work; /* Context for CB invoking. */ struct rcu_head srcu_barrier_head; /* For srcu_barrier() use. */ + struct rcu_head srcu_ec_head; /* For srcu_expedite_current() use. */ + int srcu_ec_state; /* State for srcu_expedite_current(). */ struct srcu_node *mynode; /* Leaf srcu_node. */ unsigned long grpmask; /* Mask for leaf srcu_node */ /* ->srcu_data_have_cbs[]. */ @@ -135,6 +137,11 @@ struct srcu_struct { #define SRCU_STATE_SCAN1 1 #define SRCU_STATE_SCAN2 2 =20 +/* Values for srcu_expedite_current() state (->srcu_ec_state). */ +#define SRCU_EC_IDLE 0 +#define SRCU_EC_PENDING 1 +#define SRCU_EC_REPOST 2 + /* * Values for initializing gp sequence fields. Higher values allow wrap ar= ounds to * occur earlier. @@ -220,6 +227,7 @@ struct srcu_struct { int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); +void srcu_expedite_current(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); =20 // Converts a per-CPU pointer to an ->srcu_ctrs[] array element to that diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 1ff94b76d91f15..f2f11492e6936e 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1688,6 +1688,64 @@ void srcu_barrier(struct srcu_struct *ssp) } EXPORT_SYMBOL_GPL(srcu_barrier); =20 +/* Callback for srcu_expedite_current() usage. */ +static void srcu_expedite_current_cb(struct rcu_head *rhp) +{ + unsigned long flags; + bool needcb =3D false; + struct srcu_data *sdp =3D container_of(rhp, struct srcu_data, srcu_ec_hea= d); + + spin_lock_irqsave_sdp_contention(sdp, &flags); + if (sdp->srcu_ec_state =3D=3D SRCU_EC_IDLE) { + WARN_ON_ONCE(1); + } else if (sdp->srcu_ec_state =3D=3D SRCU_EC_PENDING) { + sdp->srcu_ec_state =3D SRCU_EC_IDLE; + } else { + WARN_ON_ONCE(sdp->srcu_ec_state !=3D SRCU_EC_REPOST); + sdp->srcu_ec_state =3D SRCU_EC_PENDING; + needcb =3D true; + } + spin_unlock_irqrestore_rcu_node(sdp, flags); + // If needed, requeue ourselves as an expedited SRCU callback. + if (needcb) + __call_srcu(sdp->ssp, &sdp->srcu_ec_head, srcu_expedite_current_cb, fals= e); +} + +/** + * srcu_expedite_current - Expedite the current SRCU grace period + * @ssp: srcu_struct to expedite. + * + * Cause the current SRCU grace period to become expedited. The grace + * period following the current one might also be expedited. If there is + * no current grace period, one might be created. If the current grace + * period is currently sleeping, that sleep will complete before expediting + * will take effect. + */ +void srcu_expedite_current(struct srcu_struct *ssp) +{ + unsigned long flags; + bool needcb =3D false; + struct srcu_data *sdp; + + preempt_disable(); + sdp =3D this_cpu_ptr(ssp->sda); + spin_lock_irqsave_sdp_contention(sdp, &flags); + if (sdp->srcu_ec_state =3D=3D SRCU_EC_IDLE) { + sdp->srcu_ec_state =3D SRCU_EC_PENDING; + needcb =3D true; + } else if (sdp->srcu_ec_state =3D=3D SRCU_EC_PENDING) { + sdp->srcu_ec_state =3D SRCU_EC_REPOST; + } else { + WARN_ON_ONCE(sdp->srcu_ec_state !=3D SRCU_EC_REPOST); + } + spin_unlock_irqrestore_rcu_node(sdp, flags); + // If needed, queue an expedited SRCU callback. + if (needcb) + __call_srcu(ssp, &sdp->srcu_ec_head, srcu_expedite_current_cb, false); + preempt_enable(); +} +EXPORT_SYMBOL_GPL(srcu_expedite_current); + /** * srcu_batches_completed - return batches completed. * @ssp: srcu_struct on which to report batch completion. --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1E7225A322; Tue, 23 Sep 2025 14:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; cv=none; b=koxUWOh0Fe4zFTbHyme44WDg2DVLxwVV+g1P61ZoA88y99e0SRjYpSkYzRIeLh0L8bATXbOFkLbfGooUWX0dYo7pEyIB15JDAJS8CcHfI/sehfaou2n0bH5BsKekK1aHbR5NKfZn//Sf5npwlV3QV9joHQx/4PktVcou2NcCq7A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; c=relaxed/simple; bh=H2Rpde2Lnzmi1rrKiEk9cEd+o18Lx4S73RaemLuL/J0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=huttp4icZAQHfrnumOMQHdYRZgpOhhYQTWthnjKB0ryuG3doAlpOj7gCc+zBRDDNZXwZKB/3y9nwa9knr0iYtTVhpqMQQed2FnT/nyIlmtmBcZ1scQ1hEkJpv/oRgzJI6vHfiTAeafvZzM871PnpdRLD85ehIZucZAWh5MrOc5w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BZNk3e/Y; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BZNk3e/Y" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AD378C116C6; Tue, 23 Sep 2025 14:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637302; bh=H2Rpde2Lnzmi1rrKiEk9cEd+o18Lx4S73RaemLuL/J0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BZNk3e/YqvMbTZTSLqyol9n8i8uj4qcUOyZq6crGHGyjTnhSeORX279zXHZUxwx2d VytjYuVy0eVDp4acvPYq6HYIGkPoyBsMiOJAflZ93fsaMb+WgSppR9yr6aKHuCt4IV 2SydQI98sh6uP4Y2zSJAekqCLLwjWPF+a6aCBee7YbUFtsgzbQC8Efpn2D106j+vv3 WC34d12o7agsARRC23Gxm7RpJJWBT538LlyWMSpNUYDrr7kZ+GvuJVQndI7/Azvt+8 k77SWufuZ9tKOrHro8uvj6WEa/XD/jueiKVaGiNtmEZ/FasjzYaP+zamjTtWFjLYTP pt8xBHD3sHEuA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 01FB3CE168B; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 24/34] rcutorture: Test srcu_expedite_current() Date: Tue, 23 Sep 2025 07:20:26 -0700 Message-Id: <20250923142036.112290-24-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds a ->exp_current member to the rcu_torture_ops structure to test the srcu_expedite_current() function. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- kernel/rcu/rcutorture.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 485fa822b6a753..64803d09fc733a 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -389,6 +389,7 @@ struct rcu_torture_ops { void (*deferred_free)(struct rcu_torture *p); void (*sync)(void); void (*exp_sync)(void); + void (*exp_current)(void); unsigned long (*get_gp_state_exp)(void); unsigned long (*start_gp_poll_exp)(void); void (*start_gp_poll_exp_full)(struct rcu_gp_oldstate *rgosp); @@ -857,6 +858,11 @@ static void srcu_torture_synchronize_expedited(void) synchronize_srcu_expedited(srcu_ctlp); } =20 +static void srcu_torture_expedite_current(void) +{ + srcu_expedite_current(srcu_ctlp); +} + static struct rcu_torture_ops srcu_ops =3D { .ttype =3D SRCU_FLAVOR, .init =3D rcu_sync_torture_init, @@ -871,6 +877,7 @@ static struct rcu_torture_ops srcu_ops =3D { .deferred_free =3D srcu_torture_deferred_free, .sync =3D srcu_torture_synchronize, .exp_sync =3D srcu_torture_synchronize_expedited, + .exp_current =3D srcu_torture_expedite_current, .same_gp_state =3D same_state_synchronize_srcu, .get_comp_state =3D get_completed_synchronize_srcu, .get_gp_state =3D srcu_torture_get_gp_state, @@ -919,6 +926,7 @@ static struct rcu_torture_ops srcud_ops =3D { .deferred_free =3D srcu_torture_deferred_free, .sync =3D srcu_torture_synchronize, .exp_sync =3D srcu_torture_synchronize_expedited, + .exp_current =3D srcu_torture_expedite_current, .same_gp_state =3D same_state_synchronize_srcu, .get_comp_state =3D get_completed_synchronize_srcu, .get_gp_state =3D srcu_torture_get_gp_state, @@ -1698,6 +1706,8 @@ rcu_torture_writer(void *arg) ulo[i] =3D cur_ops->get_comp_state(); gp_snap =3D cur_ops->start_gp_poll(); rcu_torture_writer_state =3D RTWS_POLL_WAIT; + if (cur_ops->exp_current && !torture_random(&rand) % 0xff) + cur_ops->exp_current(); while (!cur_ops->poll_gp_state(gp_snap)) { gp_snap1 =3D cur_ops->get_gp_state(); for (i =3D 0; i < ulo_size; i++) @@ -1718,6 +1728,8 @@ rcu_torture_writer(void *arg) cur_ops->get_comp_state_full(&rgo[i]); cur_ops->start_gp_poll_full(&gp_snap_full); rcu_torture_writer_state =3D RTWS_POLL_WAIT_FULL; + if (cur_ops->exp_current && !torture_random(&rand) % 0xff) + cur_ops->exp_current(); while (!cur_ops->poll_gp_state_full(&gp_snap_full)) { cur_ops->get_gp_state_full(&gp_snap1_full); for (i =3D 0; i < rgo_size; i++) --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BF57327A28; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637315; cv=none; b=mkj8ds2pNHmRG1sEBtUxWvJkzy+EpD6lwJ1VxtJiO2IrMC5FkOvNyHFZurqN6Mb4Ei2+sCiIzxBmUVj/wGJJ9LADjpoebzaaMcN0ClQFOY1NkZ6qxzF2uGlMpQFL592mqS/G7zmVzG4hxPJ08cj0l9XVB0MuXKn8r78Z3bzVTJo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637315; c=relaxed/simple; bh=NiuHAgAJpC2TRweVL4XTZuEsRPP/Cvyb13DBOTQHKg8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UqgRiVf2CPw8S8ujgXKqvNBuEGgZR7/1imXdiDi5IeAp/S1BdPh5vGz0sRh/39aw4OSmAZTZNuVxlpIf9YupiNH598PlrNSP3Oynh7JlCJQFmIZfhBkvTHWRECPqRw9lsqB4GUJ66TR8zeEp1nQ+nHcBAO21KC1w4D67DleGsh0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Ra0kNFvw; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ra0kNFvw" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6F9B0C116C6; Tue, 23 Sep 2025 14:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637314; bh=NiuHAgAJpC2TRweVL4XTZuEsRPP/Cvyb13DBOTQHKg8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ra0kNFvw9LCjAKbmIOhztBvXixxP6dWztSS9jYD3y72qb9CiNmGz1U5bB/DWZcJAS kQTSsSWkUSqgRGrnUPc323f94AuU4F/hBQwd/Cc0QXgE+56xvNxXIfUCHqUNqKertl 5GRBm/f18z0cJwH+flPGrtO6WS7sG8Pgl84IS+RAOAlTtPTmXvBPtACGq5APvrTAck r5pFGq77/gW0Q4vo6L0tzT+US/sZqkCwIhNEhzxt5ksOddCzKufs5V4lFE0RtVYu3v ekJR19lyktl+qU78SY63MwbiiH87VrWucn+I/DthLSHnOBeCpvYYsyYQBFCL1XwYjB OZDO0hnMUE3Gw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 04BE4CE168F; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 25/34] srcu: Create an rcu_tasks_trace_expedite_current() function Date: Tue, 23 Sep 2025 07:20:27 -0700 Message-Id: <20250923142036.112290-25-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates an rcu_tasks_trace_expedite_current() function that expedites the current (and possibly the next) RCU Tasks Trace grace period. If the current RCU Tasks Trace grace period is already waiting, that wait will complete before the expediting takes effect. If there is no RCU Tasks Trace grace period in flight, this function might well create one. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/rcupdate_trace.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index 7f7977fb56aca5..70231f95cd5e71 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -176,6 +176,20 @@ static inline void rcu_barrier_tasks_trace(void) srcu_barrier(&rcu_tasks_trace_srcu_struct); } =20 +/** + * rcu_tasks_trace_expedite_current - Expedite the current Tasks Trace RCU= grace period + * + * Cause the current Tasks Trace RCU grace period to become expedited. + * The grace period following the current one might also be expedited. + * If there is no current grace period, one might be created. If the + * current grace period is currently sleeping, that sleep will complete + * before expediting will take effect. + */ +static inline void rcu_tasks_trace_expedite_current(void) +{ + srcu_expedite_current(&rcu_tasks_trace_srcu_struct); +} + // Placeholders to enable stepwise transition. void __init rcu_tasks_trace_suppress_unused(void); =20 --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDDAE23D7E0; Tue, 23 Sep 2025 14:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; cv=none; b=Lof9QmSBuljSHODx94l+kMCNvkTAtjVcmKFHkCDG/7ztQOq0cqsJmfKHSTsbgEGn+jmXpsAiR4mRS95wdRfxcwVuVa93KJDG9NkO8v5gIMkKPKt1cxPgzfJ6Da+6wkc4vNVoZ0su206dhm+hQq1bpyVC3DdrNrYkL381LsJyWs8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; c=relaxed/simple; bh=voJ4b6wMLOYakegASPWhkeR8X8Tsr/P1l+3ILgsYDF4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pMu6JdsXFsICf41Gbb1qt5ok9dLs0GN8dAEHVSruv3nAY2ZT7ktq5XLSL0g/PN4+oeSDaprn0djXC9PBBI9Z2+fy3SAQojZzfQiKxlWB3vjNCOt4RDfACtTXdpcvQHwaHGg/M3Qj0nQ6V/+edJDN7/sQBDkm0EiS/1LWJM+3HTA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tCS9U0Lh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tCS9U0Lh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E2B28C19423; Tue, 23 Sep 2025 14:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637302; bh=voJ4b6wMLOYakegASPWhkeR8X8Tsr/P1l+3ILgsYDF4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tCS9U0Lhi+elUXnefbj5rnsXjSzaHvGHocOzcXCFwTfuRQNVAlrl8woqLjEHd/zPc VXM0CwQQtVbkB7B0sghWV07t6XXCkZFinn8xqMSaQK5v0qicAEQLY0uPcd6jmqdlvj ou02Om4Jfpd2jyy1doGS3UpNHF4sFWxIxhOWkbH/Rok58Lq/koO8eMVJAf+z6MEnzG WmwPjB09dHeXSlQbSXgkBatd6lgm3Yrr7p/CHkkIKP9vpFpGcWSdRjSYNqiLgbg3SA wjGhIcH83gk8QA1Humouh2j8Hxj4j6fhnoPYzaXn55/FpYQNTlX/KZvHpqIPfGpvft hjjk68iBHuhAw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 075D1CE16CE; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 26/34] rcutorture: Test rcu_tasks_trace_expedite_current() Date: Tue, 23 Sep 2025 07:20:28 -0700 Message-Id: <20250923142036.112290-26-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds a ->exp_current member to the tasks_tracing_ops structure to test the rcu_tasks_trace_expedite_current() function. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- kernel/rcu/rcutorture.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 64803d09fc733a..2e3806b996a80a 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -1136,6 +1136,7 @@ static struct rcu_torture_ops tasks_tracing_ops =3D { .deferred_free =3D rcu_tasks_tracing_torture_deferred_free, .sync =3D synchronize_rcu_tasks_trace, .exp_sync =3D synchronize_rcu_tasks_trace, + .exp_current =3D rcu_tasks_trace_expedite_current, .call =3D call_rcu_tasks_trace, .cb_barrier =3D rcu_barrier_tasks_trace, .cbflood_max =3D 50000, --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D939F266B46; Tue, 23 Sep 2025 14:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; cv=none; b=Y7x/ub5EHZFTBBlJDea8Gvz45Xha8OVumtBM0RbPymNK0BfbN2NMd/jlDY54K9xuV3NEvw8U1/qyws4Z2FUAW9smtsRNuTGF8XYrO83kdHcHQW4XhO4v1lnfXbU3xjtXGu5q3qbIfsFkHmTx41IP8JTR24dCLg6r3vP7jDA7Hc0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; c=relaxed/simple; bh=Ko1cw61hU1PAAj+U3rTa7B+e/zegi9QjEUcXug5eMKI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WGGQ3y4oANLWypWuZ/U/5yc9gQqSWArZJWff86+TgGMtEy1SQ5TFEhtw5wZhZXpfWYmoBz8WoFlh6Zie7GtPfn/jplFVNqm2QSitj2phZdsZ5aDTK4enfekaDq2N91gIHDzgfFPybImbUk7RrX2W+XW30RkCFHSks4qp8sjO1p0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NyaO48o5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NyaO48o5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6B4AC4AF09; Tue, 23 Sep 2025 14:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637302; bh=Ko1cw61hU1PAAj+U3rTa7B+e/zegi9QjEUcXug5eMKI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NyaO48o57k4+Agc6ginPdjeD+MQkDVMSHNF5EBoDbN2LWLdOFpicNnOhl3z2XmnbZ iyNcNLewqEpPhAIop/a4WEmikqLPghjJe0N7IWk7uinvUCWx73uUTGTwu40OcB1bOX nB9wNhK744xCd1DcblpVxN6OLfyX2bREK1ZBUCWxHsiDS88K9SP381Zv4ly+uxWl5Z Xd4uJZ6Npx2ag43HqKGmq4NxUTHJn2csgmS/Vy0PPcqoM0Ib+1Ugq0+TX00RFKECIi 5zr1oT2G9YKT2p/0lbUzm5s3KjgjSSCf6yvKWpv/pnLazKNYilkila9pbUA3/zpVTZ P46muwfMqjFRg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0A17BCE172F; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 27/34] srcu: Make DEFINE_SRCU_FAST() available to modules Date: Tue, 23 Sep 2025 07:20:29 -0700 Message-Id: <20250923142036.112290-27-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit makes DEFINE_SRCU_FAST() available to modules. The trick is that DEFINE_SRCU_FAST() does not allocate the per-CPU data structure when used within a module (by design), which means that it is not possible to compile-time-initialize any of the fields in this structure, including in particular the ->srcu_reader_flavor field. This commit therefore creates an ->srcu_reader_flavor field in the srcu_struct structure, adds arguments to the DEFINE_SRCU()-related macros to initialize this new field, and extending the checks in the __srcu_check_read_flavor() function to include this new field. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/notifier.h | 2 +- include/linux/srcutiny.h | 10 ++++++---- include/linux/srcutree.h | 24 +++++++++++++----------- kernel/rcu/srcutree.c | 3 +++ 4 files changed, 23 insertions(+), 16 deletions(-) diff --git a/include/linux/notifier.h b/include/linux/notifier.h index b42e6473496882..01b6c9d9956f90 100644 --- a/include/linux/notifier.h +++ b/include/linux/notifier.h @@ -109,7 +109,7 @@ extern void srcu_init_notifier_head(struct srcu_notifie= r_head *nh); .mutex =3D __MUTEX_INITIALIZER(name.mutex), \ .head =3D NULL, \ .srcuu =3D __SRCU_USAGE_INIT(name.srcuu), \ - .srcu =3D __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu), \ + .srcu =3D __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu, 0), \ } =20 #define ATOMIC_NOTIFIER_HEAD(name) \ diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 941e8210479607..b5f62259d56fb1 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -31,7 +31,7 @@ struct srcu_struct { =20 void srcu_drive_gp(struct work_struct *wp); =20 -#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored) \ +#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored, ____ignored) \ { \ .srcu_wq =3D __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \ .srcu_cb_tail =3D &name.srcu_cb_head, \ @@ -44,10 +44,12 @@ void srcu_drive_gp(struct work_struct *wp); * Tree SRCU, which needs some per-CPU data. */ #define DEFINE_SRCU(name) \ - struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name) -#define DEFINE_SRCU_FAST(name) DEFINE_SRCU(name) + struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, name) #define DEFINE_STATIC_SRCU(name) \ - static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name) + static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) +#define DEFINE_SRCU_FAST(name) DEFINE_SRCU(name) +#define DEFINE_STATIC_SRCU_FAST(name) \ + static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) =20 // Dummy structure for srcu_notifier_head. struct srcu_usage { }; diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 4a5d1c0e7db361..05400f70baa40a 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -104,6 +104,7 @@ struct srcu_usage { struct srcu_struct { struct srcu_ctr __percpu *srcu_ctrp; struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ + u8 srcu_reader_flavor; struct lockdep_map dep_map; struct srcu_usage *srcu_sup; /* Update-side data. */ }; @@ -162,20 +163,21 @@ struct srcu_struct { .work =3D __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ } =20 -#define __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ +#define __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ .srcu_sup =3D &usage_name, \ + .srcu_reader_flavor =3D fast, \ __SRCU_DEP_MAP_INIT(name) =20 -#define __SRCU_STRUCT_INIT_MODULE(name, usage_name) \ +#define __SRCU_STRUCT_INIT_MODULE(name, usage_name, fast) \ { \ - __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ + __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ } =20 -#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name) \ +#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name, fast) \ { \ .sda =3D &pcpu_name, \ .srcu_ctrp =3D &pcpu_name.srcu_ctrs[0], \ - __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ + __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ } =20 /* @@ -207,22 +209,22 @@ struct srcu_struct { #ifdef MODULE # define __DEFINE_SRCU(name, fast, is_static) \ static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s= rcu_usage); \ - is_static struct srcu_struct name =3D __SRCU_STRUCT_INIT_MODULE(name, nam= e##_srcu_usage); \ + is_static struct srcu_struct name =3D __SRCU_STRUCT_INIT_MODULE(name, nam= e##_srcu_usage, \ + fast); \ extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ __section("___srcu_struct_ptrs") =3D &name #else # define __DEFINE_SRCU(name, fast, is_static) \ - static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data) =3D { \ - .srcu_reader_flavor =3D fast, \ - }; \ + static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s= rcu_usage); \ is_static struct srcu_struct name =3D \ - __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data) -#define DEFINE_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST,= /* not static */) + __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data, fast) #endif #define DEFINE_SRCU(name) __DEFINE_SRCU(name, 0, /* not static */) #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, 0, static) +#define DEFINE_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST,= /* not static */) +#define DEFINE_STATIC_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR= _FAST, static) =20 int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index f2f11492e6936e..7c8b39d4dd3162 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -734,6 +734,9 @@ void __srcu_check_read_flavor(struct srcu_struct *ssp, = int read_flavor) =20 sdp =3D raw_cpu_ptr(ssp->sda); old_read_flavor =3D READ_ONCE(sdp->srcu_reader_flavor); + WARN_ON_ONCE(ssp->srcu_reader_flavor && read_flavor !=3D ssp->srcu_reader= _flavor); + WARN_ON_ONCE(old_read_flavor && ssp->srcu_reader_flavor && + old_read_flavor !=3D ssp->srcu_reader_flavor); if (!old_read_flavor) { old_read_flavor =3D cmpxchg(&sdp->srcu_reader_flavor, 0, read_flavor); if (!old_read_flavor) --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 485222C187; Tue, 23 Sep 2025 14:21:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637296; cv=none; b=WtI62079Sn0jz/55q/6FmC5Tsrg8NnijTarOD3ekRgWpPUk7hpaEqEiFhDuU8lw+uSGXGGplvy7EsHXG4aO70Or1teJXzEZ2rB7o/xgHYXNdHLb9IIGKEF58mdQK4GXP+q1kRAdjzjKNV7PQ4gB9Dj/gWGdm+f6r3vkxUF0oss4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637296; c=relaxed/simple; bh=4c507eqy0EGwafCH6pxIpf/X1KtAub6CGBV9RmYtNM0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=a+tBG6m3uSj67Sf+lX0SwcB2KKXKxVfhcvTujKjNOlyUpK4EskE12hL2pJulhRoWnBhCeoDb1fnZgSy4eQBkwk/DYzedT+2Yd4KzXvnfynkI9CDD/Mb3vHzX24iKiK+FNPSbnxg7DHZ+ViCTUfDackk5SQtmBlBuxnCCykJTR3U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GXVcsyO0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GXVcsyO0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB8D5C4CEF5; Tue, 23 Sep 2025 14:21:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637295; bh=4c507eqy0EGwafCH6pxIpf/X1KtAub6CGBV9RmYtNM0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GXVcsyO0I+bJsahDDW5Mxc+Xl69wA3fjUGhiSQoEI/cPVHJ0W3WB3r8FegxcwE1Jp BygEFEtZZ0IecTwd2MDGPAuItQ4h9b5HN/zxw2JRU/wzeaqTeSId5Dz69hwZCj3U1K sWFEjbETExrm/55EsP3BuRsfgpGyE4umrlf4WvgeAan7xIwpMEa1Qdx+p38BEGmJA0 AnT4bzq3YP6dK/DurTGso0sn9ah67mqD/G/Fd2DYFRn0q/rKkp5aT60XHyEDR0yF7s bDNsaqfzj84SrFRk23fpZ7vNgY4sgQFynMp5AEbeMCjHwmqVDAORNz0UHsxms/5jmr huZQ0KyOXYAIg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0CE8ECE1763; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 28/34] srcu: Make SRCU-fast available to heap srcu_struct structures Date: Tue, 23 Sep 2025 07:20:30 -0700 Message-Id: <20250923142036.112290-28-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit defines a new init_srcu_struct_fast() function that marks the specified srcu_struct structure for use by srcu_read_lock_fast() and friends. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 16 ++++++++++++++-- include/linux/srcutiny.h | 4 ++++ kernel/rcu/srcutree.c | 33 +++++++++++++++++++++++++++++++-- 3 files changed, 49 insertions(+), 4 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index ada65b58bc4c5f..26de47820c58cd 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -25,8 +25,10 @@ struct srcu_struct; =20 #ifdef CONFIG_DEBUG_LOCK_ALLOC =20 -int __init_srcu_struct(struct srcu_struct *ssp, const char *name, - struct lock_class_key *key); +int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct l= ock_class_key *key); +#ifndef CONFIG_TINY_SRCU +int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, str= uct lock_class_key *key); +#endif // #ifndef CONFIG_TINY_SRCU =20 #define init_srcu_struct(ssp) \ ({ \ @@ -35,10 +37,20 @@ int __init_srcu_struct(struct srcu_struct *ssp, const c= har *name, __init_srcu_struct((ssp), #ssp, &__srcu_key); \ }) =20 +#define init_srcu_struct_fast(ssp) \ +({ \ + static struct lock_class_key __srcu_key; \ + \ + __init_srcu_struct_fast((ssp), #ssp, &__srcu_key); \ +}) + #define __SRCU_DEP_MAP_INIT(srcu_name) .dep_map =3D { .name =3D #srcu_name= }, #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 int init_srcu_struct(struct srcu_struct *ssp); +#ifndef CONFIG_TINY_SRCU +int init_srcu_struct_fast(struct srcu_struct *ssp); +#endif // #ifndef CONFIG_TINY_SRCU =20 #define __SRCU_DEP_MAP_INIT(srcu_name) #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index b5f62259d56fb1..92e6ab53398fc0 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -54,6 +54,10 @@ void srcu_drive_gp(struct work_struct *wp); // Dummy structure for srcu_notifier_head. struct srcu_usage { }; #define __SRCU_USAGE_INIT(name) { } +#define __init_srcu_struct_fast __init_srcu_struct +#ifndef CONFIG_DEBUG_LOCK_ALLOC +#define init_srcu_struct_fast init_srcu_struct +#endif // #ifndef CONFIG_DEBUG_LOCK_ALLOC =20 void synchronize_srcu(struct srcu_struct *ssp); =20 diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 7c8b39d4dd3162..177f9ec1f926e1 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -286,16 +286,29 @@ static int init_srcu_struct_fields(struct srcu_struct= *ssp, bool is_static) =20 #ifdef CONFIG_DEBUG_LOCK_ALLOC =20 -int __init_srcu_struct(struct srcu_struct *ssp, const char *name, - struct lock_class_key *key) +static int +__init_srcu_struct_common(struct srcu_struct *ssp, const char *name, struc= t lock_class_key *key) { /* Don't re-initialize a lock while it is held. */ debug_check_no_locks_freed((void *)ssp, sizeof(*ssp)); lockdep_init_map(&ssp->dep_map, name, key, 0); return init_srcu_struct_fields(ssp, false); } + +int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct l= ock_class_key *key) +{ + ssp->srcu_reader_flavor =3D 0; + return __init_srcu_struct_common(ssp, name, key); +} EXPORT_SYMBOL_GPL(__init_srcu_struct); =20 +int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, str= uct lock_class_key *key) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST; + return __init_srcu_struct_common(ssp, name, key); +} +EXPORT_SYMBOL_GPL(__init_srcu_struct_fast); + #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /** @@ -308,10 +321,26 @@ EXPORT_SYMBOL_GPL(__init_srcu_struct); */ int init_srcu_struct(struct srcu_struct *ssp) { + ssp->srcu_reader_flavor =3D 0; return init_srcu_struct_fields(ssp, false); } EXPORT_SYMBOL_GPL(init_srcu_struct); =20 +/** + * init_srcu_struct_fast - initialize a fast-reader sleep-RCU structure + * @ssp: structure to initialize. + * + * Must invoke this on a given srcu_struct before passing that srcu_struct + * to any other function. Each srcu_struct represents a separate domain + * of SRCU protection. + */ +int init_srcu_struct_fast(struct srcu_struct *ssp) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST; + return init_srcu_struct_fields(ssp, false); +} +EXPORT_SYMBOL_GPL(init_srcu_struct_fast); + #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /* --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0B45251791; Tue, 23 Sep 2025 14:22:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637325; cv=none; b=S+xRzUO2IFpAvWvIIiAh9g5++zS5HUGbPCKBi6BtgBCFKMPh+aS4yUGeksWy/f8QqNRchoZnbcoltgy64GwK/jVdGHtLcBiFbOUAV/Y+/ezMxFWtHa0zL1pPp2fIvIJuwLTVfztrbBFamEIDuH1uKJ5unytuCVWYc3pRTNNnREg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637325; c=relaxed/simple; bh=/9ihPRofiSX+oBfCmJpcNKAf4WsfpDFWiCi2HWwyMDs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=q2cllku6hJ5MMxZHoaMeksYPSU7xy6ZKmRqJD05xXUzwzB+e68vhQEb06EKRZ3TmvaBDIftRB5EdSCWaw7xIs5xaN9Fk1R6lYTSBuJ0clocWWcbJwvFsEws4oFJv9hYVa2wsa1Wy4kpGVqPwY9DUgpHIQ8ZSYht4+dQWnP2DB60= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kI1FhNaC; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kI1FhNaC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D79E1C116C6; Tue, 23 Sep 2025 14:22:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637324; bh=/9ihPRofiSX+oBfCmJpcNKAf4WsfpDFWiCi2HWwyMDs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kI1FhNaCOg3rQqc/Vkz+IMTFsm+YnYuQHpsjLc4gN0jB/p40ijLyyJdRORo3YXiRD /q0hrBoBddaB/VoZIMTYubFS0r+eSqVHjDV7c61wtZrmGMDiR0c82rKQ4kIaQxpAUJ s70HwmmQ0kcDEsaismPnRf7SszVk0skEmreJWyGFVzGksAp8oOCKrWtTplv9zOWsXF /VbkkBR5KETt9iv9Ti6eN8IKQxhxJ0CGODpx8ivLkc+PSOm3xbd6+oF5Tw14XFJyyd qUddy2TAKMCN5+U1RAGATW8bBep++ISLDIFhP5HrM1Y56l8Wu5hirTBTqPMOM9CNf+ FWo+SIfflWgFg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0F626CE1775; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 29/34] srcu: Make grace-period determination use ssp->srcu_reader_flavor Date: Tue, 23 Sep 2025 07:20:31 -0700 Message-Id: <20250923142036.112290-29-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit causes the srcu_readers_unlock_idx() function to take the srcu_struct structure's ->srcu_reader_flavor field into account. This ensures that structures defined via DEFINE_SRCU_FAST( or initialized via init_srcu_struct_fast() have their grace periods use synchronize_srcu() or synchronize_srcu_expedited() instead of smp_mb(), even before the first SRCU reader has been entered. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- kernel/rcu/srcutree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 177f9ec1f926e1..8a871976c37dfe 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -490,7 +490,7 @@ static bool srcu_readers_lock_idx(struct srcu_struct *s= sp, int idx, bool gp, uns static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int = idx, unsigned long *rdm) { int cpu; - unsigned long mask =3D 0; + unsigned long mask =3D ssp->srcu_reader_flavor; unsigned long sum =3D 0; =20 for_each_possible_cpu(cpu) { --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D9417266B56; Tue, 23 Sep 2025 14:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; cv=none; b=CYKfGaCuVTcoYsK7ZzK5QFL+UJEk0M8VvZRGT+78sD68qlv4R1Xf2Pi5zmaKKiqFw7zdOZ3VQjwsw+sGA4vjQivkXy4Vo96gq5XA8d8IQJepXcQG/lI6EHGR3ea+c0baXPtLOq4n2FJ3ynMbVf5YH4a5WtCR/h1Aw5w6e5/Lk78= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637302; c=relaxed/simple; bh=M6lQK/a51z9FSyLcsJB4DdYne3vYn3DuAPGWx5z897o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dLQBVnZ35QgA5G/iDidtX0//q2UkJj2HTgT7wjosHjHzUsagMh38TxrbwpdQFtHf9aw04LtMX0dBZNkiMterO3WOMncQF2g3hjEzi6LBm8lMCIcx9zXlc+Kg6oaO4US4u82fFX1dvNwM9LPtkmiKFNTX/38msV613j6MSJeb2x8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=d0ANnOgz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="d0ANnOgz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E08EDC16AAE; Tue, 23 Sep 2025 14:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637302; bh=M6lQK/a51z9FSyLcsJB4DdYne3vYn3DuAPGWx5z897o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=d0ANnOgzC+ASozW2HZ75QzJ1W5a4HSPXIKWUZjUpnaOdlCuk9bJ5g+J2autwFuIsA Lk5l4kLzfaFObvlQffxspHyFT8OSZTsdk71pQQ0Kh/f6q5DB6JC0a0F+SPXWMoIPFv qDilZ26eN5uljN+xOeTh8Fqbo7FEaDiLLFVG3GfCaN2zv7y3jHYTB9JjTeYwbvTrPR V7jKus+x/COwFC2jldC0ttsDgtNEpRJ0KQwTGb1gPSdpZADYfYX9UbbN25urxAuCl1 sSCk51Kec9EYFhg7d7kAKAbtIrGx6jjnA/xpg/DvUQzMJvBQwK1w8sLIWV1bDFO1bc cLokgYfQa1FUA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 12166CE17C2; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 30/34] rcutorture: Exercise DEFINE_STATIC_SRCU_FAST() and init_srcu_struct_fast() Date: Tue, 23 Sep 2025 07:20:32 -0700 Message-Id: <20250923142036.112290-30-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit updates the initialization for the "srcu" and "srcud" torture types to use DEFINE_STATIC_SRCU_FAST() and init_srcu_struct_fast(), respectively, when reader_flavor is equal to SRCU_READ_FLAVOR_FAST. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- kernel/rcu/rcutorture.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 2e3806b996a80a..e859a1ca7dc841 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -692,10 +692,18 @@ static struct rcu_torture_ops rcu_busted_ops =3D { */ =20 DEFINE_STATIC_SRCU(srcu_ctl); +DEFINE_STATIC_SRCU_FAST(srcu_ctlf); static struct srcu_struct srcu_ctld; static struct srcu_struct *srcu_ctlp =3D &srcu_ctl; static struct rcu_torture_ops srcud_ops; =20 +static void srcu_torture_init(void) +{ + rcu_sync_torture_init(); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) + srcu_ctlp =3D &srcu_ctlf; +} + static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) { srcutorture_get_gp_data(srcu_ctlp, flags, gp_seq); @@ -865,7 +873,7 @@ static void srcu_torture_expedite_current(void) =20 static struct rcu_torture_ops srcu_ops =3D { .ttype =3D SRCU_FLAVOR, - .init =3D rcu_sync_torture_init, + .init =3D srcu_torture_init, .readlock =3D srcu_torture_read_lock, .read_delay =3D srcu_read_delay, .readunlock =3D srcu_torture_read_unlock, @@ -897,10 +905,13 @@ static struct rcu_torture_ops srcu_ops =3D { .name =3D "srcu" }; =20 -static void srcu_torture_init(void) +static void srcud_torture_init(void) { rcu_sync_torture_init(); - WARN_ON(init_srcu_struct(&srcu_ctld)); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) + WARN_ON(init_srcu_struct_fast(&srcu_ctld)); + else + WARN_ON(init_srcu_struct(&srcu_ctld)); srcu_ctlp =3D &srcu_ctld; } =20 @@ -913,7 +924,7 @@ static void srcu_torture_cleanup(void) /* As above, but dynamically allocated. */ static struct rcu_torture_ops srcud_ops =3D { .ttype =3D SRCU_FLAVOR, - .init =3D srcu_torture_init, + .init =3D srcud_torture_init, .cleanup =3D srcu_torture_cleanup, .readlock =3D srcu_torture_read_lock, .read_delay =3D srcu_read_delay, --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7AD363294E5; Tue, 23 Sep 2025 14:21:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; cv=none; b=L8IKeQsmXEuKjZiyj+Y4Ne8oC2BO294x25NXGyCKttta1aJlLK0ZUB05k1lKJG/OOnZ0hS8nUqOgR9hUPMD7R/Phwp3dri2hizbifxGVgXColLF/qnh11O8SC8y9418f2xPFM6WPq8d9ALrLF2xSBb/RijVw3U0+je4q0O+OBAY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637316; c=relaxed/simple; bh=XhR9e3hqlUudx27Vubeb4TW2DWMcOEhmyHuKz9F2A/k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tZYsHo0Mllii9bq9lohQ7rFGaYidCvkvSMkUTHZjIhBIZ3gypEfyAlqPrgG4mK3YVMZsiiFCwdAgCyWcKqg2lVRbYuqyVlxLohi2OCtrpl0WURAJ+bLrvRXAUlkdDaFvid8S8koNCQslTez+/5oFwQQ1013Zpwa29tCUv0ER3UA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JmsQRlS2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JmsQRlS2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A43A4C4CEF5; Tue, 23 Sep 2025 14:21:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637316; bh=XhR9e3hqlUudx27Vubeb4TW2DWMcOEhmyHuKz9F2A/k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JmsQRlS22/evgVeHiGU3FgC2zBql2UQd7beaQa45pTf4fhMxEWXk4mBBBSnu2r6vh 7iObu+6ntuaGMaArKy6lZejiXFWVIaYwaqLF99i+iPX6mHFn+3C+U4Z1TSjetR1nTz uf7E4CSj3c3P8mq1X9HZsviftFFKBmcxm9FAynSAvyquQA6qPxxL0gRAydHjvIWcUx jK8KeW7MRsyTx7FbpVs6XIJZJ1nh2uAabf07eB4o4NibebZfnyGDoZz2Q3GUg3VHQj x6PUgkvGrgixV/AQUsS+jHI/c6zyXD8YCEtj8ziLxcQr967mLLOS1lFw8AfQ6V3fZ5 uLP65GSme0ndQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 14981CE18B1; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 31/34] refscale: Exercise DEFINE_STATIC_SRCU_FAST() and init_srcu_struct_fast() Date: Tue, 23 Sep 2025 07:20:33 -0700 Message-Id: <20250923142036.112290-31-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit updates the initialization for the "srcu-fast" scale type to use DEFINE_STATIC_SRCU_FAST() when reader_flavor is equal to SRCU_READ_FLAVOR_FAST. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- kernel/rcu/refscale.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/refscale.c b/kernel/rcu/refscale.c index 19841704d8f579..ece77f6d055b85 100644 --- a/kernel/rcu/refscale.c +++ b/kernel/rcu/refscale.c @@ -184,6 +184,7 @@ static const struct ref_scale_ops rcu_ops =3D { =20 // Definitions for SRCU ref scale testing. DEFINE_STATIC_SRCU(srcu_refctl_scale); +DEFINE_STATIC_SRCU_FAST(srcu_fast_refctl_scale); static struct srcu_struct *srcu_ctlp =3D &srcu_refctl_scale; =20 static void srcu_ref_scale_read_section(const int nloops) @@ -216,6 +217,12 @@ static const struct ref_scale_ops srcu_ops =3D { .name =3D "srcu" }; =20 +static bool srcu_fast_sync_scale_init(void) +{ + srcu_ctlp =3D &srcu_fast_refctl_scale; + return true; +} + static void srcu_fast_ref_scale_read_section(const int nloops) { int i; @@ -240,7 +247,7 @@ static void srcu_fast_ref_scale_delay_section(const int= nloops, const int udl, c } =20 static const struct ref_scale_ops srcu_fast_ops =3D { - .init =3D rcu_sync_scale_init, + .init =3D srcu_fast_sync_scale_init, .readsection =3D srcu_fast_ref_scale_read_section, .delaysection =3D srcu_fast_ref_scale_delay_section, .name =3D "srcu-fast" --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 583ED23D7DE; Tue, 23 Sep 2025 14:21:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637296; cv=none; b=HzAmTYB9DX/Sn/o0uu+kzMIbQjF9FloKLaAhg5ovnFxhwy6R7Bl2Es5VOqeryumRnfBteaXktjJhEEOelYrhpwo+awbXlBsKZTrf7J4ssF7ooEX7MYcQUDO7LgAvXCzw1fWDMjzo47znkZ9l/sB15qYTKXIZ7QLOg1V/jVmC5Zk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637296; c=relaxed/simple; bh=tGrdv4Ck+rLWm+HN+jcwDBzUoYqwe2sdQlV4IgJA9RM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FhRULcqwEIeMUZYHxrpA+Au2W/7nYJMHvbe+5PDCteEykshL/HTvkNjfblXNjadB1RkgFQw2rdtmsqbCfiYDgtJoUqwfIHx8Jl60qHoUBQHgc4RNBiHPpG1QVtT5IyuDnWnXTE6VTuTZ4I8pSxActrhJ9BbApkmqkoNkJg0xJFU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lUibeIJt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lUibeIJt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DA6D6C116C6; Tue, 23 Sep 2025 14:21:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637296; bh=tGrdv4Ck+rLWm+HN+jcwDBzUoYqwe2sdQlV4IgJA9RM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lUibeIJt/wqdPZLe9AI0KGHncnoL+JeQiFLJdbp6I7qDqXV2wT/VS1Qvr6tP3zxWz Mc3hE8M31wH9b3yJGiD8zuKDfYHU6c60zJkYKvdhDsBlCLGw0LY3EOXdZ/bqcdCXtR qduLK80KKrp4h//haoid6++l/+qiTgDngdcGg67TPT3TllpfS3OLEWsk+vauqCEkK1 3htGb1D33khli0dTd9WwWYuVa0RovDagtASHQhn1bLAhr7JoX4k3qFkFaklKwTHFQV 510w3O5LLND1GqAE3Q5onwfTnxY52JKTwbZCyDf6PFcPVual1nNC6UCm51dCMQtNfJ NsUHidnEDRbcA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 172BDCE18F1; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 32/34] srcu: Require special srcu_struct define/init for SRCU-fast readers Date: Tue, 23 Sep 2025 07:20:34 -0700 Message-Id: <20250923142036.112290-32-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds CONFIG_PROVE_RCU=3Dy checking to enforce the new rule that srcu_struct structures passed to srcu_read_lock_fast() and other SRCU-fast read-side markers be either initialized with init_srcu_struct_fast() on the one hand or defined using either DEFINE_SRCU_FAST() or DEFINE_STATIC_SRCU_FAST(). This will enable removal of the non-debug read-side checks from srcu_read_lock_fast() and friends, which on my laptop provides a 25% speedup (which admittedly amounts to about half a nanosecond, but when tracing fastpaths...) Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 34 ++++++++++++++++++++++------------ kernel/rcu/srcutree.c | 1 + 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 26de47820c58cd..2982b5a6930fa6 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -271,17 +271,26 @@ static inline int srcu_read_lock(struct srcu_struct *= ssp) __acquires(ssp) * @ssp: srcu_struct in which to register the new reader. * * Enter an SRCU read-side critical section, but for a light-weight - * smp_mb()-free reader. See srcu_read_lock() for more information. - * - * If srcu_read_lock_fast() is ever used on an srcu_struct structure, - * then none of the other flavors may be used, whether before, during, - * or after. Note that grace-period auto-expediting is disabled for _fast - * srcu_struct structures because auto-expedited grace periods invoke - * synchronize_rcu_expedited(), IPIs and all. - * - * Note that srcu_read_lock_fast() can be invoked only from those contexts - * where RCU is watching, that is, from contexts where it would be legal - * to invoke rcu_read_lock(). Otherwise, lockdep will complain. + * smp_mb()-free reader. See srcu_read_lock() for more information. This + * function is NMI-safe, in a manner similar to srcu_read_lock_nmisafe(). + * + * For srcu_read_lock_fast() to be used on an srcu_struct structure, + * that structure must have been defined using either DEFINE_SRCU_FAST() + * or DEFINE_STATIC_SRCU_FAST() on the one hand or initialized with + * init_srcu_struct_fast() on the other. Such an srcu_struct structure + * cannot be passed to any non-fast variant of srcu_read_{,un}lock() or + * srcu_{down,up}_read(). In kernels built with CONFIG_PROVE_RCU=3Dy, + * __srcu_check_read_flavor() will complain bitterly if you ignore this + * restriction. + * + * Grace-period auto-expediting is disabled for SRCU-fast srcu_struct + * structures because SRCU-fast expedited grace periods invoke + * synchronize_rcu_expedited(), IPIs and all. If you need expedited + * SRCU-fast grace periods, use synchronize_srcu_expedited(). + * + * The srcu_read_lock_fast() function can be invoked only from those + * contexts where RCU is watching, that is, from contexts where it would + * be legal to invoke rcu_read_lock(). Otherwise, lockdep will complain. */ static inline struct srcu_ctr __percpu *srcu_read_lock_fast(struct srcu_st= ruct *ssp) __acquires(ssp) { @@ -317,7 +326,8 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast_notrace(struct srcu_ * srcu_down_read() for more information. * * The same srcu_struct may be used concurrently by srcu_down_read_fast() - * and srcu_read_lock_fast(). + * and srcu_read_lock_fast(). However, the same definition/initialization + * requirements called out for srcu_read_lock_safe() apply. */ static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_st= ruct *ssp) __acquires(ssp) { diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 8a871976c37dfe..204edd2801c21b 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -766,6 +766,7 @@ void __srcu_check_read_flavor(struct srcu_struct *ssp, = int read_flavor) WARN_ON_ONCE(ssp->srcu_reader_flavor && read_flavor !=3D ssp->srcu_reader= _flavor); WARN_ON_ONCE(old_read_flavor && ssp->srcu_reader_flavor && old_read_flavor !=3D ssp->srcu_reader_flavor); + WARN_ON_ONCE(read_flavor =3D=3D SRCU_READ_FLAVOR_FAST && !ssp->srcu_reade= r_flavor); if (!old_read_flavor) { old_read_flavor =3D cmpxchg(&sdp->srcu_reader_flavor, 0, read_flavor); if (!old_read_flavor) --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D218832CF9E; Tue, 23 Sep 2025 14:22:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637324; cv=none; b=HqJeE9Sj5N6kXSv4FJTJBg6/WaOHwrvqqCFwjx8nObJAm2OA9g9LDCvel5DLuRyT7QjK1OpFUprFxKR86Odh69JkvMGqvpA2GmZTC43HcVrWY+x8Gwsy36tS7ENrUmg516Uc2LJpKjgbEehwnr5/rjDNNcYjX6NSntj0+t6Ueng= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637324; c=relaxed/simple; bh=h25HXfqFwZPbzPp+0/q7PleMKwW/TXZPNmjI5yN4W8Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Mh31SHTaJj0sk7ccMOdSZ3VFGI3Uq45ZkTCQ7b+DX8994f5IVeMuwL/6EX2cFiJNN25qoDJBPMkTLQZ6QuM16vYqGirCtjo19yWG4oWfXbmPrd+ExED4kvGzXaCVFt8uHi9zl1pB4rlvH3+uFOoaf/MXIZ5rJ6dy26AxcYx7PlM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=I8ngf+eF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="I8ngf+eF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C4351C116D0; Tue, 23 Sep 2025 14:22:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637324; bh=h25HXfqFwZPbzPp+0/q7PleMKwW/TXZPNmjI5yN4W8Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=I8ngf+eFlR3x+0cony+Sq9zeidtSqj5t9+bQ//qxHP+dC+8FYJX2+JcRNxVygs653 2hqS8/d1rQQlBUYHFg74WXTiMgS7P1fgybJCx7r8wV91s9e4aG68K1pYCr7bcRysBI zHqx7REWOzLWOec41FeYWefKqDLd4XsgVNzAEAhXRNjx4coU1sgC/CKdB2Qd2ra/QZ ORIhwvzKdiaH/2soQJAXYsRHd3RauQYegT0uPXFVFmZkg7biuMGxfkwfxRPlXRWOew XMYmZvSYiCx5LxHnxKjUrFAeMHkbvWVRpKXQVhAP0SZafMVGaP3FzTCBpWuNgirieo BFLDC5+JKusBA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 19E9CCE18FE; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 33/34] srcu: Make SRCU-fast readers enforce use of SRCU-fast definition/init Date: Tue, 23 Sep 2025 07:20:35 -0700 Message-Id: <20250923142036.112290-33-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit makes CONFIG_PROVE_RCU=3Dy kernels enforce the new rule that srcu_struct structures that are passed to srcu_read_lock_fast() and other SRCU-fast read-side markers be either initialized with init_srcu_struct_fast() on the one hand or defined with DEFINE_SRCU_FAST() or DEFINE_STATIC_SRCU_FAST() on the other. This eliminates the read-side test that was formerly included in srcu_read_lock_fast() and friends, speeding these primitives up by about 25% (admittedly only about half of a nanosecond, but when tracing on fastpaths...) Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 6 +++--- include/linux/srcutiny.h | 1 - include/linux/srcutree.h | 16 +--------------- kernel/rcu/tasks.h | 5 ----- 4 files changed, 4 insertions(+), 24 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 2982b5a6930fa6..41e27c1d917d3e 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -297,7 +297,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast(struct srcu_struct * struct srcu_ctr __percpu *retval; =20 RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast()."); - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); retval =3D __srcu_read_lock_fast(ssp); rcu_try_lock_acquire(&ssp->dep_map); return retval; @@ -312,7 +312,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast_notrace(struct srcu_ { struct srcu_ctr __percpu *retval; =20 - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); retval =3D __srcu_read_lock_fast(ssp); return retval; } @@ -333,7 +333,7 @@ static inline struct srcu_ctr __percpu *srcu_down_read_= fast(struct srcu_struct * { WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read= _fast()."); - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); return __srcu_read_lock_fast(ssp); } =20 diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 92e6ab53398fc0..1ecc3393fb26be 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -112,7 +112,6 @@ static inline void srcu_barrier(struct srcu_struct *ssp) =20 static inline void srcu_expedite_current(struct srcu_struct *ssp) { } #define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) -#define srcu_check_read_flavor_force(ssp, read_flavor) do { } while (0) =20 /* Defined here to avoid size increase for non-torture kernels. */ static inline void srcu_torture_stats_print(struct srcu_struct *ssp, diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 05400f70baa40a..fd24ec146af614 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -311,21 +311,7 @@ __srcu_read_unlock_fast(struct srcu_struct *ssp, struc= t srcu_ctr __percpu *scp) =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); =20 -// Record reader usage even for CONFIG_PROVE_RCU=3Dn kernels. This is -// needed only for flavors that require grace-period smp_mb() calls to be -// promoted to synchronize_rcu(). -static inline void srcu_check_read_flavor_force(struct srcu_struct *ssp, i= nt read_flavor) -{ - struct srcu_data *sdp =3D raw_cpu_ptr(ssp->sda); - - if (likely(READ_ONCE(sdp->srcu_reader_flavor) & read_flavor)) - return; - - // Note that the cmpxchg() in __srcu_check_read_flavor() is fully ordered. - __srcu_check_read_flavor(ssp, read_flavor); -} - -// Record non-_lite() usage only for CONFIG_PROVE_RCU=3Dy kernels. +// Record SRCU-reader usage type only for CONFIG_PROVE_RCU=3Dy kernels. static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int rea= d_flavor) { if (IS_ENABLED(CONFIG_PROVE_RCU)) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index bf1226834c9423..76f952196a2921 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1600,11 +1600,6 @@ static inline void rcu_tasks_bootup_oddness(void) {} // Tracing variant of Tasks RCU. This variant is designed to be used // to protect tracing hooks, including those of BPF. This variant // is implemented via a straightforward mapping onto SRCU-fast. -// DEFINE_SRCU_FAST() is required because rcu_read_lock_trace() must -// use __srcu_read_lock_fast() in order to bypass the rcu_is_watching() -// checks in kernels built with CONFIG_TASKS_TRACE_RCU_NO_MB=3Dn, which al= so -// bypasses the srcu_check_read_flavor_force() that would otherwise mark -// rcu_tasks_trace_srcu_struct as needing SRCU-fast readers. =20 DEFINE_SRCU_FAST(rcu_tasks_trace_srcu_struct); EXPORT_SYMBOL_GPL(rcu_tasks_trace_srcu_struct); --=20 2.40.1 From nobody Thu Oct 2 02:05:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BED6327A27; Tue, 23 Sep 2025 14:21:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637315; cv=none; b=Q/iXrYmQPKXdHfuakgF/0Wd5MA5Ycfhkl1vQWdGGHQYEUVqRR9Is6Mq0u+A7bpY4MdGYglIEW+X2HiqBVhNl8mCsGIhC3q8NUJ7hPQQ8WvwLL3ggR+dcDwuoUb2xPfjgnHH9SYAGa/Aa5tT3dqWlVeNMRXP8OucgbJbop37oxfo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758637315; c=relaxed/simple; bh=81BLq8+oPnPePeqC2bm5XK2/WvvBNdRPQuuI5LrZL9M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Hmsxpy3+bjgV1VQ0CtRW7xILhe1i3Z9IKYOfjxXkevIJ8FEPBluYCs7J1E2/BcdEu2PcmAJyw2xib5bRj9S11ZtG4yv/5Dljr3AmeI5BzViIhkz8GIye6l4jyQUsS5X26zNgEi8x4rj9DKJUvXaLLsYGowwACV1Hjjwk0pH1W5o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=n+8huKaP; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="n+8huKaP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 77750C16AAE; Tue, 23 Sep 2025 14:21:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758637314; bh=81BLq8+oPnPePeqC2bm5XK2/WvvBNdRPQuuI5LrZL9M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=n+8huKaP7fycL9AnrUIrxXI+dZAqhQLq2JfxdF1ZFU3VDn7qlsEJHNQTIzYE6qjgF RJnjc4lHq72JFyK5kD7tmSINoelcZGTUiyy3SKIvLYxElpx+UA/TIE8R/ltT5PgBjT SZho6F34eMNnw+XdidceYBYsKFa2seev/id1oUmgPc/zgHLLYMH2Q8uIYVurHwEarJ Uz5kVFAkv4+CtyD90x9OsZbUSFWQWezhIev0XGQKGsGEncEqBhWo0WVHHdQHRq8gMU wEeUpZ/RNxvsG0GhS8r2Xl2xFZCHDsMEXNTIP6BxYg+LDkcciRC5zywT9i7gQ7clxv 3HeH1nEQzipZA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 1CADDCE1919; Tue, 23 Sep 2025 07:20:38 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 34/34] doc: Update for SRCU-fast definitions and initialization Date: Tue, 23 Sep 2025 07:20:36 -0700 Message-Id: <20250923142036.112290-34-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> References: <580ea2de-799a-4ddc-bde9-c16f3fb1e6e7@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit documents the DEFINE_SRCU_FAST(), DEFINE_STATIC_SRCU_FAST(), and init_srcu_struct_fast() API members. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- .../RCU/Design/Requirements/Requirements.rst | 33 ++++++++++--------- Documentation/RCU/checklist.rst | 12 ++++--- Documentation/RCU/whatisRCU.rst | 3 ++ 3 files changed, 27 insertions(+), 21 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Docum= entation/RCU/Design/Requirements/Requirements.rst index 4a116d7a564edc..b5cdbba3ec2e71 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2637,15 +2637,16 @@ synchronize_srcu() for some other domain ``ss1``, a= nd if an that was held across as ``ss``-domain synchronize_srcu(), deadlock would again be possible. Such a deadlock cycle could extend across an arbitrarily large number of different SRCU domains. Again, with great -power comes great responsibility. +power comes great responsibility, though lockdep is now able to detect +this sort of deadlock. =20 -Unlike the other RCU flavors, SRCU read-side critical sections can run -on idle and even offline CPUs. This ability requires that -srcu_read_lock() and srcu_read_unlock() contain memory barriers, -which means that SRCU readers will run a bit slower than would RCU -readers. It also motivates the smp_mb__after_srcu_read_unlock() API, -which, in combination with srcu_read_unlock(), guarantees a full -memory barrier. +Unlike the other RCU flavors, SRCU read-side critical sections can run on +idle and even offline CPUs, with the exception of srcu_read_lock_fast() +and friends. This ability requires that srcu_read_lock() and +srcu_read_unlock() contain memory barriers, which means that SRCU +readers will run a bit slower than would RCU readers. It also motivates +the smp_mb__after_srcu_read_unlock() API, which, in combination with +srcu_read_unlock(), guarantees a full memory barrier. =20 Also unlike other RCU flavors, synchronize_srcu() may **not** be invoked from CPU-hotplug notifiers, due to the fact that SRCU grace @@ -2681,15 +2682,15 @@ run some tests first. SRCU just might need a few ad= justment to deal with that sort of load. Of course, your mileage may vary based on the speed of your CPUs and the size of your memory. =20 -The `SRCU -API `__ +The `SRCU API +`__ includes srcu_read_lock(), srcu_read_unlock(), -srcu_dereference(), srcu_dereference_check(), -synchronize_srcu(), synchronize_srcu_expedited(), -call_srcu(), srcu_barrier(), and srcu_read_lock_held(). It -also includes DEFINE_SRCU(), DEFINE_STATIC_SRCU(), and -init_srcu_struct() APIs for defining and initializing -``srcu_struct`` structures. +srcu_dereference(), srcu_dereference_check(), synchronize_srcu(), +synchronize_srcu_expedited(), call_srcu(), srcu_barrier(), +and srcu_read_lock_held(). It also includes DEFINE_SRCU(), +DEFINE_STATIC_SRCU(), DEFINE_SRCU_FAST(), DEFINE_STATIC_SRCU_FAST(), +init_srcu_struct(), and init_srcu_struct_fast() APIs for defining and +initializing ``srcu_struct`` structures. =20 More recently, the SRCU API has added polling interfaces: =20 diff --git a/Documentation/RCU/checklist.rst b/Documentation/RCU/checklist.= rst index c9bfb2b218e525..4b30f701225fdb 100644 --- a/Documentation/RCU/checklist.rst +++ b/Documentation/RCU/checklist.rst @@ -417,11 +417,13 @@ over a rather long period of time, but improvements a= re always welcome! you should be using RCU rather than SRCU, because RCU is almost always faster and easier to use than is SRCU. =20 - Also unlike other forms of RCU, explicit initialization and - cleanup is required either at build time via DEFINE_SRCU() - or DEFINE_STATIC_SRCU() or at runtime via init_srcu_struct() - and cleanup_srcu_struct(). These last two are passed a - "struct srcu_struct" that defines the scope of a given + Also unlike other forms of RCU, explicit initialization + and cleanup is required either at build time via + DEFINE_SRCU(), DEFINE_STATIC_SRCU(), DEFINE_SRCU_FAST(), + or DEFINE_STATIC_SRCU_FAST() or at runtime via either + init_srcu_struct() or init_srcu_struct_fast() and + cleanup_srcu_struct(). These last three are passed a + `struct srcu_struct` that defines the scope of a given SRCU domain. Once initialized, the srcu_struct is passed to srcu_read_lock(), srcu_read_unlock() synchronize_srcu(), synchronize_srcu_expedited(), and call_srcu(). A given diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.= rst index cf0b0ac9f4636a..a1582bd653d115 100644 --- a/Documentation/RCU/whatisRCU.rst +++ b/Documentation/RCU/whatisRCU.rst @@ -1227,7 +1227,10 @@ SRCU: Initialization/cleanup/ordering:: =20 DEFINE_SRCU DEFINE_STATIC_SRCU + DEFINE_SRCU_FAST // for srcu_read_lock_fast() and friends + DEFINE_STATIC_SRCU_FAST // for srcu_read_lock_fast() and friends init_srcu_struct + init_srcu_struct_fast cleanup_srcu_struct smp_mb__after_srcu_read_unlock =20 --=20 2.40.1