From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EEC734845D; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=qfExtaMREh0Bb39tePWRLr0Lwpu5BEboaIDyUtYQpGTrJGaNoj8bGFLKfYq1xH48mw0VWynPhrMTqIXANEY8P60qMRtcIy+r9tEOz836xYpnhx2vtnz1rdul/81KLlDe7cHz25MaaxndNRobLg9Xq1daIAr04+h+HcqfeR7e4nM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=ZFNuHVX2di4q2gqCJGVL5/EBAk3Kdb3wjGuQ316eoLk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tpSfv29Ve7+jQ2h7vuG0KeHvJ6QBsbUUZMtl4JawFv4MoF0qidK5vLDoOVyy2inKJrdLP3+Zvm2rcLi+3YQ624XxtHTNs9YNaxeEPTYyiUbmAT9jEXmH31KA03lzKVZ8bENMgE4ELzCtM4wAWeoscf6r0AbEsWVrQi1VN6y/rCw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=K1syEvtK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="K1syEvtK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 634CFC4CEF5; Wed, 5 Nov 2025 20:32:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374739; bh=ZFNuHVX2di4q2gqCJGVL5/EBAk3Kdb3wjGuQ316eoLk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=K1syEvtKZACwITkLFfQVm+DUEDVzwY4gE7UCqsGOqpAb+E76WWOJC37XNDXqiKFkb +TOowTp85JZHbH31OTRXH2eam0N9Gssg3CGsOdRGes3lY1udJsKZrMDKMwS0RniOIF GS9l62kay1PiyOEY5651pYWq/lvppHqL7ADCOvwb1CBll/SymAET+DjOowSP10lao4 28Hup0ZDwPyeqIskkqodvDYrHUGQ1E0btu/ekzrqjy4n1JMukkIZmWPFUyS+oU7myi YBYW5fcG3pfCOHOr15jKZkUuA68b5zj9Y80v5lOPiMqLbvjWG35ZLa4rMXaL0r8A+s lFVnjnrjZMKTw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 86460CE0B94; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , kernel test robot , Zqiang Subject: [PATCH v2 01/16] srcu: Permit Tiny SRCU srcu_read_unlock() with interrupts disabled Date: Wed, 5 Nov 2025 12:32:01 -0800 Message-Id: <20251105203216.2701005-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current Tiny SRCU implementation of srcu_read_unlock() awakens the grace-period processing when exiting the outermost SRCU read-side critical section. However, not all Linux-kernel configurations and contexts permit swake_up_one() to be invoked while interrupts are disabled, and this can result in indefinitely extended SRCU grace periods. This commit therefore only invokes swake_up_one() when interrupts are enabled, and introduces polling to the grace-period workqueue handler. Reported-by: kernel test robot Reported-by: Zqiang Closes: https://lore.kernel.org/oe-lkp/202508261642.b15eefbb-lkp@intel.com Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutiny.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c index e3b64a5e0ec7..3450c3751ef7 100644 --- a/kernel/rcu/srcutiny.c +++ b/kernel/rcu/srcutiny.c @@ -106,15 +106,15 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int = idx) newval =3D READ_ONCE(ssp->srcu_lock_nesting[idx]) - 1; WRITE_ONCE(ssp->srcu_lock_nesting[idx], newval); preempt_enable(); - if (!newval && READ_ONCE(ssp->srcu_gp_waiting) && in_task()) + if (!newval && READ_ONCE(ssp->srcu_gp_waiting) && in_task() && !irqs_disa= bled()) swake_up_one(&ssp->srcu_wq); } EXPORT_SYMBOL_GPL(__srcu_read_unlock); =20 /* * Workqueue handler to drive one grace period and invoke any callbacks - * that become ready as a result. Single-CPU and !PREEMPTION operation - * means that we get away with murder on synchronization. ;-) + * that become ready as a result. Single-CPU operation and preemption + * disabling mean that we get away with murder on synchronization. ;-) */ void srcu_drive_gp(struct work_struct *wp) { @@ -141,7 +141,12 @@ void srcu_drive_gp(struct work_struct *wp) WRITE_ONCE(ssp->srcu_idx, ssp->srcu_idx + 1); WRITE_ONCE(ssp->srcu_gp_waiting, true); /* srcu_read_unlock() wakes! */ preempt_enable(); - swait_event_exclusive(ssp->srcu_wq, !READ_ONCE(ssp->srcu_lock_nesting[idx= ])); + do { + // Deadlock issues prevent __srcu_read_unlock() from + // doing an unconditional wakeup, so polling is required. + swait_event_timeout_exclusive(ssp->srcu_wq, + !READ_ONCE(ssp->srcu_lock_nesting[idx]), HZ / 10); + } while (READ_ONCE(ssp->srcu_lock_nesting[idx])); preempt_disable(); // Needed for PREEMPT_LAZY WRITE_ONCE(ssp->srcu_gp_waiting, false); /* srcu_read_unlock() cheap. */ WRITE_ONCE(ssp->srcu_idx, ssp->srcu_idx + 1); --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EFB3348866; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=FV0chTl8KpOOa22SSY3JYx/5rEKCpbKjeQGG5lufuO2/Z+6VWDK/l7a1+6lMYPW4WqLx8ED5yB5O83RWRgteJaO+tqB75gsCL8y99Si+bqCuigJG4VU36eoco0+I73X3zv0FQsNS3C8KQAcKW6Lzrn60dWZZTDqB3szWdHLfShs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=nCGLsD7q2HOJPiSlnIStGPVemKa7l7hTEsNDv1pJevc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lKd8gW1mUpxOJmvJJyqwypCA7z0emFkWPpP0bfRqRp/l0b0UZDhGOC9GFlH8+Wok2oQHQYT0Rr4uwOXMEAQPmOAwDU3i5MvSOLy3ZXsleuGi0lb9/8qZ8FcLu62k26A5o7iQXn0D0uJcbyGqASAY+ZgL0WR7IiOjEX4/MZxt4jw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tgXsm317; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tgXsm317" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 687B6C116B1; Wed, 5 Nov 2025 20:32:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374739; bh=nCGLsD7q2HOJPiSlnIStGPVemKa7l7hTEsNDv1pJevc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tgXsm317lR2XzjMkYSzfpo5MO7ATEjAdlQMkgCQfnGcj9lUzXRL3NIDiuBiRQztz1 BNgNO1zXUIeZ+wgVGZdpFT5u06l1Y4Ri/VahF7yocq0i9pJQ23R4Csq9ElnRlFPIZP DsWiHnhHNkfpczCll8kt7q7jOjlkO1UWRg5k99cpu6iEV2C2KYPQweNmsyn9xpDhRR E5egXlNcWCfrBWBlRXT5jP9HJ2GSUFrbCl1A+2semKjsGQMSxXiNQuALgD5UBEy1yM XQnVKO1PCK5pwhE9Npv68IcDwf12KnMqT7PBVrehHaWKj4uHi/wP8mCICUa4brFsks 4yyhvozZRo5kQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8952ECE0BA5; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v2 02/16] srcu: Create an srcu_expedite_current() function Date: Wed, 5 Nov 2025 12:32:02 -0800 Message-Id: <20251105203216.2701005-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates an srcu_expedite_current() function that expedites the current (and possibly the next) SRCU grace period for the specified srcu_struct structure. This functionality will be inherited by RCU Tasks Trace courtesy of its mapping to SRCU fast. If the current SRCU grace period is already waiting, that wait will complete before the expediting takes effect. If there is no SRCU grace period in flight, this function might well create one. [ paulmck: Apply Zqiang feedback for PREEMPT_RT use. ] Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/srcutiny.h | 1 + include/linux/srcutree.h | 8 ++++++ kernel/rcu/srcutree.c | 58 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 51ce25f07930..3bfbd44cb1b3 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -103,6 +103,7 @@ static inline void srcu_barrier(struct srcu_struct *ssp) synchronize_srcu(ssp); } =20 +static inline void srcu_expedite_current(struct srcu_struct *ssp) { } #define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) #define srcu_check_read_flavor_force(ssp, read_flavor) do { } while (0) =20 diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 42098e0fa0b7..93ad18acd6d0 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -42,6 +42,8 @@ struct srcu_data { struct timer_list delay_work; /* Delay for CB invoking */ struct work_struct work; /* Context for CB invoking. */ struct rcu_head srcu_barrier_head; /* For srcu_barrier() use. */ + struct rcu_head srcu_ec_head; /* For srcu_expedite_current() use. */ + int srcu_ec_state; /* State for srcu_expedite_current(). */ struct srcu_node *mynode; /* Leaf srcu_node. */ unsigned long grpmask; /* Mask for leaf srcu_node */ /* ->srcu_data_have_cbs[]. */ @@ -135,6 +137,11 @@ struct srcu_struct { #define SRCU_STATE_SCAN1 1 #define SRCU_STATE_SCAN2 2 =20 +/* Values for srcu_expedite_current() state (->srcu_ec_state). */ +#define SRCU_EC_IDLE 0 +#define SRCU_EC_PENDING 1 +#define SRCU_EC_REPOST 2 + /* * Values for initializing gp sequence fields. Higher values allow wrap ar= ounds to * occur earlier. @@ -210,6 +217,7 @@ struct srcu_struct { int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); +void srcu_expedite_current(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); =20 // Converts a per-CPU pointer to an ->srcu_ctrs[] array element to that diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 1ff94b76d91f..38b440b0b0c8 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1688,6 +1688,64 @@ void srcu_barrier(struct srcu_struct *ssp) } EXPORT_SYMBOL_GPL(srcu_barrier); =20 +/* Callback for srcu_expedite_current() usage. */ +static void srcu_expedite_current_cb(struct rcu_head *rhp) +{ + unsigned long flags; + bool needcb =3D false; + struct srcu_data *sdp =3D container_of(rhp, struct srcu_data, srcu_ec_hea= d); + + spin_lock_irqsave_sdp_contention(sdp, &flags); + if (sdp->srcu_ec_state =3D=3D SRCU_EC_IDLE) { + WARN_ON_ONCE(1); + } else if (sdp->srcu_ec_state =3D=3D SRCU_EC_PENDING) { + sdp->srcu_ec_state =3D SRCU_EC_IDLE; + } else { + WARN_ON_ONCE(sdp->srcu_ec_state !=3D SRCU_EC_REPOST); + sdp->srcu_ec_state =3D SRCU_EC_PENDING; + needcb =3D true; + } + spin_unlock_irqrestore_rcu_node(sdp, flags); + // If needed, requeue ourselves as an expedited SRCU callback. + if (needcb) + __call_srcu(sdp->ssp, &sdp->srcu_ec_head, srcu_expedite_current_cb, fals= e); +} + +/** + * srcu_expedite_current - Expedite the current SRCU grace period + * @ssp: srcu_struct to expedite. + * + * Cause the current SRCU grace period to become expedited. The grace + * period following the current one might also be expedited. If there is + * no current grace period, one might be created. If the current grace + * period is currently sleeping, that sleep will complete before expediting + * will take effect. + */ +void srcu_expedite_current(struct srcu_struct *ssp) +{ + unsigned long flags; + bool needcb =3D false; + struct srcu_data *sdp; + + migrate_disable(); + sdp =3D this_cpu_ptr(ssp->sda); + spin_lock_irqsave_sdp_contention(sdp, &flags); + if (sdp->srcu_ec_state =3D=3D SRCU_EC_IDLE) { + sdp->srcu_ec_state =3D SRCU_EC_PENDING; + needcb =3D true; + } else if (sdp->srcu_ec_state =3D=3D SRCU_EC_PENDING) { + sdp->srcu_ec_state =3D SRCU_EC_REPOST; + } else { + WARN_ON_ONCE(sdp->srcu_ec_state !=3D SRCU_EC_REPOST); + } + spin_unlock_irqrestore_rcu_node(sdp, flags); + // If needed, queue an expedited SRCU callback. + if (needcb) + __call_srcu(ssp, &sdp->srcu_ec_head, srcu_expedite_current_cb, false); + migrate_enable(); +} +EXPORT_SYMBOL_GPL(srcu_expedite_current); + /** * srcu_batches_completed - return batches completed. * @ssp: srcu_struct on which to report batch completion. --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F6B934A76E; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=GrWwxS6+gg8uKEs6uhbsAZznEN+FLp0Huqv42VR0PcCByRhyqhX6qyiFtUC8TjrCgmCn1ZSjcANzmIdVW2kIGCsUrNA8aybl0MWNsBPOcOKZmbao1cLvzyNB05Ba6pnKYlpRcajV1VMsXUdfcZrnMEOAFj+kY0+RmRPV3fyeL5k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=w01VFRMtubctWz2T4Nx3dEUt4UuqBNsYAd2a1T0fPq0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cj5d2I+jko+iTVWi4dy2I9vXRAy5fibR9P1bQrT/65rzxJCr/Z6lnrW9as4Pud8AIaGeY3n5SCrWecqoLzBNwZT3TzjIgjIr7Z9ljPcnwDDfxStEaEwL8/k8CUkk2GdRpXW0RLryGXQjNoHPfyoDtYr+tri6lHgysGt1ymWX7SA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=uqZW3TYI; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="uqZW3TYI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 70307C116D0; Wed, 5 Nov 2025 20:32:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374739; bh=w01VFRMtubctWz2T4Nx3dEUt4UuqBNsYAd2a1T0fPq0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uqZW3TYIk8mUbww2X56rdRiYQ1kQmld9u1n6GbTpTg6SYFV34Wd2W5yPxzvyVZxzV tMfGdrOOvPSBgdqOpL5Jh4U0dRuOSMnOVV++80+mLxvhtqTjIDg5R39GcNLDi9G6x4 tU8Tj0JAf0HTrbnwdJm6+egaa2GOdbByhh0GudUB34l1JcqH6CEcAKiBFev1Xi33p6 Ze/LQqdtS5TV68EINoj92NayL9VyT++OvLb0QTcVkPKPgqggqqiwi3xuzU35fbYyhc DycbK0ufC5na2iAcjOe7mEEH07v7fWcBDjstD51YK6I9oIOnjWaFgKKyl6w4Fn2evj 2/PhVn5BqG6eA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8C405CE0CA7; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v2 03/16] rcutorture: Test srcu_expedite_current() Date: Wed, 5 Nov 2025 12:32:03 -0800 Message-Id: <20251105203216.2701005-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds a ->exp_current member to the rcu_torture_ops structure to test the srcu_expedite_current() function. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- kernel/rcu/rcutorture.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 318bea62ed3e..ea8077c08fac 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -389,6 +389,7 @@ struct rcu_torture_ops { void (*deferred_free)(struct rcu_torture *p); void (*sync)(void); void (*exp_sync)(void); + void (*exp_current)(void); unsigned long (*get_gp_state_exp)(void); unsigned long (*start_gp_poll_exp)(void); void (*start_gp_poll_exp_full)(struct rcu_gp_oldstate *rgosp); @@ -857,6 +858,11 @@ static void srcu_torture_synchronize_expedited(void) synchronize_srcu_expedited(srcu_ctlp); } =20 +static void srcu_torture_expedite_current(void) +{ + srcu_expedite_current(srcu_ctlp); +} + static struct rcu_torture_ops srcu_ops =3D { .ttype =3D SRCU_FLAVOR, .init =3D rcu_sync_torture_init, @@ -871,6 +877,7 @@ static struct rcu_torture_ops srcu_ops =3D { .deferred_free =3D srcu_torture_deferred_free, .sync =3D srcu_torture_synchronize, .exp_sync =3D srcu_torture_synchronize_expedited, + .exp_current =3D srcu_torture_expedite_current, .same_gp_state =3D same_state_synchronize_srcu, .get_comp_state =3D get_completed_synchronize_srcu, .get_gp_state =3D srcu_torture_get_gp_state, @@ -919,6 +926,7 @@ static struct rcu_torture_ops srcud_ops =3D { .deferred_free =3D srcu_torture_deferred_free, .sync =3D srcu_torture_synchronize, .exp_sync =3D srcu_torture_synchronize_expedited, + .exp_current =3D srcu_torture_expedite_current, .same_gp_state =3D same_state_synchronize_srcu, .get_comp_state =3D get_completed_synchronize_srcu, .get_gp_state =3D srcu_torture_get_gp_state, @@ -1700,6 +1708,8 @@ rcu_torture_writer(void *arg) ulo[i] =3D cur_ops->get_comp_state(); gp_snap =3D cur_ops->start_gp_poll(); rcu_torture_writer_state =3D RTWS_POLL_WAIT; + if (cur_ops->exp_current && !torture_random(&rand) % 0xff) + cur_ops->exp_current(); while (!cur_ops->poll_gp_state(gp_snap)) { gp_snap1 =3D cur_ops->get_gp_state(); for (i =3D 0; i < ulo_size; i++) @@ -1720,6 +1730,8 @@ rcu_torture_writer(void *arg) cur_ops->get_comp_state_full(&rgo[i]); cur_ops->start_gp_poll_full(&gp_snap_full); rcu_torture_writer_state =3D RTWS_POLL_WAIT_FULL; + if (cur_ops->exp_current && !torture_random(&rand) % 0xff) + cur_ops->exp_current(); while (!cur_ops->poll_gp_state_full(&gp_snap_full)) { cur_ops->get_gp_state_full(&gp_snap1_full); for (i =3D 0; i < rgo_size; i++) --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C3AD34846A; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=ZTGOZKrPdX7rClF1EI9HBzTVQHIQHls8kCW33YCrRFWFTgxuvUwLVV0Y3IIivGBkLa0LFByxpN3MUp01WhklyQzbzBCw2HTL4tRoprr355sbMXif09xTpiAil4V9U/8ud+SLw8SMiuuR9Ok4YDtEicnQzsyKJFA3pfKKb5Io/fw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=40G+Sy5PfqcQaYHAZAnhVUifnZEIaEN9k49sIAuQdQs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AR8r39+1OqKmXAZRL/8VtVWI2r2RnO8PN2+1ZwlnVCl0wpP+UA6kwkAqrNYKol5O1uycLXs9A2+L2eAP0L6uFJMRmORr4fl0LwU+0cHbi3Q5Z5PW3B6kV1ozw/xz1/JC+2TwcdCgSURnYnX2IFrvZCvR3JCOg0vhAo3FwW+gnOM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aT/DKKZP; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aT/DKKZP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7002FC116C6; Wed, 5 Nov 2025 20:32:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374739; bh=40G+Sy5PfqcQaYHAZAnhVUifnZEIaEN9k49sIAuQdQs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aT/DKKZPCfStRBiUYt6D+ZHwNYOvui4TWAJWTQwuY7EbiAWNjH5HENdoQgbasUEbX hjwKWe5SRlZhAcEm0oI8WzHm3N+MvVAWzndho6I8VzPQlQGCkaeabLCWiIFrmtUzTs gdYX28hGCaefekpCYeS4aL3pFfS7qwe7iOmATiMfU1Z0f+BZHdqlK1ZvTrckOtsEID AdAkxm8+uhHRpnksZR1eXGlRaEm3o6fbimrVXFDbU+v/8Nj6w3ibR02nAob3Uy/Vmn PhHBJH7uBMPIGd7XYC2jFiXxEAFzpvEyyV04QfNIRyiUOhV5OoyHe87z2JU6X6L22x ip6kB5FM9DxwA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8F05ECE0D1C; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 04/16] srcu: Create a DEFINE_SRCU_FAST() Date: Wed, 5 Nov 2025 12:32:04 -0800 Message-Id: <20251105203216.2701005-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates DEFINE_SRCU_FAST() and DEFINE_STATIC_SRCU_FAST() macros that are similar to DEFINE_SRCU() and DEFINE_STATIC_SRCU(), but which create srcu_struct structures that are usable only by readers initiated by srcu_read_lock_fast() and friends. This commit does make DEFINE_SRCU_FAST() available to modules, in which case the per-CPU srcu_data structures are not created at compile time, but rather at module-load time. This means that the >srcu_reader_flavor field of the srcu_data structure is not available. Therefore, this commit instead creates an ->srcu_reader_flavor field in the srcu_struct structure, adds arguments to the DEFINE_SRCU()-related macros to initialize this new field, and extends the checks in the __srcu_check_read_flavor() function to include this new field. This commit also allows dynamically allocated srcu_struct structure to be marked for SRCU-fast readers. It does so by defining a new init_srcu_struct_fast() function that marks the specified srcu_struct structure for use by srcu_read_lock_fast() and friends. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/notifier.h | 2 +- include/linux/srcu.h | 16 ++++++++++++++-- include/linux/srcutiny.h | 13 ++++++++++--- include/linux/srcutree.h | 30 +++++++++++++++++++----------- kernel/rcu/srcutree.c | 36 ++++++++++++++++++++++++++++++++++-- 5 files changed, 78 insertions(+), 19 deletions(-) diff --git a/include/linux/notifier.h b/include/linux/notifier.h index b42e64734968..01b6c9d9956f 100644 --- a/include/linux/notifier.h +++ b/include/linux/notifier.h @@ -109,7 +109,7 @@ extern void srcu_init_notifier_head(struct srcu_notifie= r_head *nh); .mutex =3D __MUTEX_INITIALIZER(name.mutex), \ .head =3D NULL, \ .srcuu =3D __SRCU_USAGE_INIT(name.srcuu), \ - .srcu =3D __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu), \ + .srcu =3D __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu, 0), \ } =20 #define ATOMIC_NOTIFIER_HEAD(name) \ diff --git a/include/linux/srcu.h b/include/linux/srcu.h index ada65b58bc4c..26de47820c58 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -25,8 +25,10 @@ struct srcu_struct; =20 #ifdef CONFIG_DEBUG_LOCK_ALLOC =20 -int __init_srcu_struct(struct srcu_struct *ssp, const char *name, - struct lock_class_key *key); +int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct l= ock_class_key *key); +#ifndef CONFIG_TINY_SRCU +int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, str= uct lock_class_key *key); +#endif // #ifndef CONFIG_TINY_SRCU =20 #define init_srcu_struct(ssp) \ ({ \ @@ -35,10 +37,20 @@ int __init_srcu_struct(struct srcu_struct *ssp, const c= har *name, __init_srcu_struct((ssp), #ssp, &__srcu_key); \ }) =20 +#define init_srcu_struct_fast(ssp) \ +({ \ + static struct lock_class_key __srcu_key; \ + \ + __init_srcu_struct_fast((ssp), #ssp, &__srcu_key); \ +}) + #define __SRCU_DEP_MAP_INIT(srcu_name) .dep_map =3D { .name =3D #srcu_name= }, #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 int init_srcu_struct(struct srcu_struct *ssp); +#ifndef CONFIG_TINY_SRCU +int init_srcu_struct_fast(struct srcu_struct *ssp); +#endif // #ifndef CONFIG_TINY_SRCU =20 #define __SRCU_DEP_MAP_INIT(srcu_name) #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 3bfbd44cb1b3..92e6ab53398f 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -31,7 +31,7 @@ struct srcu_struct { =20 void srcu_drive_gp(struct work_struct *wp); =20 -#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored) \ +#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored, ____ignored) \ { \ .srcu_wq =3D __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \ .srcu_cb_tail =3D &name.srcu_cb_head, \ @@ -44,13 +44,20 @@ void srcu_drive_gp(struct work_struct *wp); * Tree SRCU, which needs some per-CPU data. */ #define DEFINE_SRCU(name) \ - struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name) + struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, name) #define DEFINE_STATIC_SRCU(name) \ - static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name) + static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) +#define DEFINE_SRCU_FAST(name) DEFINE_SRCU(name) +#define DEFINE_STATIC_SRCU_FAST(name) \ + static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) =20 // Dummy structure for srcu_notifier_head. struct srcu_usage { }; #define __SRCU_USAGE_INIT(name) { } +#define __init_srcu_struct_fast __init_srcu_struct +#ifndef CONFIG_DEBUG_LOCK_ALLOC +#define init_srcu_struct_fast init_srcu_struct +#endif // #ifndef CONFIG_DEBUG_LOCK_ALLOC =20 void synchronize_srcu(struct srcu_struct *ssp); =20 diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 93ad18acd6d0..7ff4a11bc5a3 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -104,6 +104,7 @@ struct srcu_usage { struct srcu_struct { struct srcu_ctr __percpu *srcu_ctrp; struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ + u8 srcu_reader_flavor; struct lockdep_map dep_map; struct srcu_usage *srcu_sup; /* Update-side data. */ }; @@ -162,20 +163,21 @@ struct srcu_struct { .work =3D __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ } =20 -#define __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ +#define __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ .srcu_sup =3D &usage_name, \ + .srcu_reader_flavor =3D fast, \ __SRCU_DEP_MAP_INIT(name) =20 -#define __SRCU_STRUCT_INIT_MODULE(name, usage_name) \ +#define __SRCU_STRUCT_INIT_MODULE(name, usage_name, fast) \ { \ - __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ + __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ } =20 -#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name) \ +#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name, fast) \ { \ .sda =3D &pcpu_name, \ .srcu_ctrp =3D &pcpu_name.srcu_ctrs[0], \ - __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ + __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ } =20 /* @@ -196,23 +198,29 @@ struct srcu_struct { * init_srcu_struct(&my_srcu); * * See include/linux/percpu-defs.h for the rules on per-CPU variables. + * + * DEFINE_SRCU_FAST() creates an srcu_struct and associated structures + * whose readers must be of the SRCU-fast variety. */ #ifdef MODULE -# define __DEFINE_SRCU(name, is_static) \ +# define __DEFINE_SRCU(name, fast, is_static) \ static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s= rcu_usage); \ - is_static struct srcu_struct name =3D __SRCU_STRUCT_INIT_MODULE(name, nam= e##_srcu_usage); \ + is_static struct srcu_struct name =3D __SRCU_STRUCT_INIT_MODULE(name, nam= e##_srcu_usage, \ + fast); \ extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ __section("___srcu_struct_ptrs") =3D &name #else -# define __DEFINE_SRCU(name, is_static) \ +# define __DEFINE_SRCU(name, fast, is_static) \ static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s= rcu_usage); \ is_static struct srcu_struct name =3D \ - __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data) + __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data, fast) #endif -#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) -#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static) +#define DEFINE_SRCU(name) __DEFINE_SRCU(name, 0, /* not static */) +#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, 0, static) +#define DEFINE_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST,= /* not static */) +#define DEFINE_STATIC_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR= _FAST, static) =20 int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 38b440b0b0c8..9869a13b8763 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -286,16 +286,29 @@ static int init_srcu_struct_fields(struct srcu_struct= *ssp, bool is_static) =20 #ifdef CONFIG_DEBUG_LOCK_ALLOC =20 -int __init_srcu_struct(struct srcu_struct *ssp, const char *name, - struct lock_class_key *key) +static int +__init_srcu_struct_common(struct srcu_struct *ssp, const char *name, struc= t lock_class_key *key) { /* Don't re-initialize a lock while it is held. */ debug_check_no_locks_freed((void *)ssp, sizeof(*ssp)); lockdep_init_map(&ssp->dep_map, name, key, 0); return init_srcu_struct_fields(ssp, false); } + +int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct l= ock_class_key *key) +{ + ssp->srcu_reader_flavor =3D 0; + return __init_srcu_struct_common(ssp, name, key); +} EXPORT_SYMBOL_GPL(__init_srcu_struct); =20 +int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, str= uct lock_class_key *key) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST; + return __init_srcu_struct_common(ssp, name, key); +} +EXPORT_SYMBOL_GPL(__init_srcu_struct_fast); + #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /** @@ -308,10 +321,26 @@ EXPORT_SYMBOL_GPL(__init_srcu_struct); */ int init_srcu_struct(struct srcu_struct *ssp) { + ssp->srcu_reader_flavor =3D 0; return init_srcu_struct_fields(ssp, false); } EXPORT_SYMBOL_GPL(init_srcu_struct); =20 +/** + * init_srcu_struct_fast - initialize a fast-reader sleep-RCU structure + * @ssp: structure to initialize. + * + * Must invoke this on a given srcu_struct before passing that srcu_struct + * to any other function. Each srcu_struct represents a separate domain + * of SRCU protection. + */ +int init_srcu_struct_fast(struct srcu_struct *ssp) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST; + return init_srcu_struct_fields(ssp, false); +} +EXPORT_SYMBOL_GPL(init_srcu_struct_fast); + #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /* @@ -734,6 +763,9 @@ void __srcu_check_read_flavor(struct srcu_struct *ssp, = int read_flavor) =20 sdp =3D raw_cpu_ptr(ssp->sda); old_read_flavor =3D READ_ONCE(sdp->srcu_reader_flavor); + WARN_ON_ONCE(ssp->srcu_reader_flavor && read_flavor !=3D ssp->srcu_reader= _flavor); + WARN_ON_ONCE(old_read_flavor && ssp->srcu_reader_flavor && + old_read_flavor !=3D ssp->srcu_reader_flavor); if (!old_read_flavor) { old_read_flavor =3D cmpxchg(&sdp->srcu_reader_flavor, 0, read_flavor); if (!old_read_flavor) --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F4F734887C; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=pmZVsYVldsczmSGlWIM0Zo2BYg/Lq5FJnNhP0e5pitvwPNwL/ghf5AV/8i+fAhrF5WoNC6PhA6b7RfkVelyzvlLlRE6yKDzbrrr/j8ZrEVbwfhMdvRgCnyby+n5dMSarHbnS01kpVl654KxUGvce5hsD/pZdOakU5AQLiMZABM4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=Q0AKERJDbUSfsvDf4x7Le3CWRCQuSmTY0wgl6K/Nz6M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nhZwOT06pMixNBSd93XRVo/XNL+4KCrt/0vt8T1ctJpwoCDvXOSZTRWcSKAM4mWYfylAGxk1AjSrFJg0CZz4wD9VPN0Bp3atiw1XxcP4v5Ux2Y4Mc7ntjyIQUSEcyjCZaDFdVazXvsFhQKQ+6BRa/O78lVDVa2AgO5wn7SMVYFo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=a0zkxm8m; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="a0zkxm8m" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 84E11C19421; Wed, 5 Nov 2025 20:32:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374739; bh=Q0AKERJDbUSfsvDf4x7Le3CWRCQuSmTY0wgl6K/Nz6M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=a0zkxm8mFO7vLeGUlhKNDvzrqqCJ0EWgw4UvfOtFP5952B63y1fRdQgZS32OnmQR0 O7EX8EbBUF6VtkPbhTj4J7M29E3fWzNM8z/CCkQGDDjRji/D7xUzKRJjlRsklWNjhc s33zlrIlL1d9heXeSugnO7FO7hwDXcBoXzxaiFj6CgoewRfxYandFy8y483vOAONwB pANzmEdIGy7GbXTtZa45tKpWwo9GoVIKILZc5/PCTrgYFqUe3FVil29mmRdvqcsvHO u8191369DjGEm8BLIiLxNW96O/Z8XQv4ZEhgiLV5+MpqS08sJtMM7W36n9EI2eqq1Y zj0NoOWyv63bQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 916D9CE0EBB; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 05/16] srcu: Make grace-period determination use ssp->srcu_reader_flavor Date: Wed, 5 Nov 2025 12:32:05 -0800 Message-Id: <20251105203216.2701005-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit causes the srcu_readers_unlock_idx() function to take the srcu_struct structure's ->srcu_reader_flavor field into account. This ensures that structures defined via DEFINE_SRCU_FAST( or initialized via init_srcu_struct_fast() have their grace periods use synchronize_srcu() or synchronize_srcu_expedited() instead of smp_mb(), even before the first SRCU reader has been entered. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- kernel/rcu/srcutree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 9869a13b8763..c29203b23d1a 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -490,7 +490,7 @@ static bool srcu_readers_lock_idx(struct srcu_struct *s= sp, int idx, bool gp, uns static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int = idx, unsigned long *rdm) { int cpu; - unsigned long mask =3D 0; + unsigned long mask =3D ssp->srcu_reader_flavor; unsigned long sum =3D 0; =20 for_each_possible_cpu(cpu) { --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2D2C34AAEF; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=PKrsktQbnrbQh4PboS/YexacQCxKD7f+KpiX+ASsQLIYJDmAbRaiF1FeAjLvF9eEyah+YPeCpkBpOKcLpr0x5py7rCiYw4d846rwuE5WI/lU3g34VGY01u1bBMp8Rm11XM++iCiNlUjuxRL7Qnn4B3z1FhBd93azQ6ABtFC0GFw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=0XIP1WBNxnsSzGwsfasxIItOHexh0+D3s4eF/W8IWy8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=N/usEIsugx/ZbTJ1uODaiBBz7oXR9aadU862lyPz+k8YU6zLNXzxuOud0oBQV6+xSNjKRKxOqx18ZuPcS9HFZWVjJL85kWA+7K4WBU9GwcvArNuLL73jE0y6Wj6zy5KMqKlIHZlnzxcfynjFnyJ0PRsDuXXctTxCemwYnv9aweg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=C3RF5cLi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="C3RF5cLi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A583C4AF09; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374740; bh=0XIP1WBNxnsSzGwsfasxIItOHexh0+D3s4eF/W8IWy8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C3RF5cLiNom7AgfajrPIU5BflJ0Zp11zbIbQ//L/TVIiBqI4DpidS1LZauPkjWjpa mQHaht3IEV/N6AidOafD3Bu96+qmfwTmaJK88s5Iht/o+KoUZs0kXU8wNw02tdNM27 eckR1/hscyYBB+8ubFvbTRq9ACNOKT+xRKQqtg+WBFnSby2QQ56N9RwCnqLkG4HTpc PLFrRzYxK9GZzeVH6JFUmU6QmsBq0WKDwmRdOebfyYzr4Ez/azdRdA6PSg+L1QDY9L Zsz5Rs7RaK21W5MoyAZ0MSe4pmXXjurHcSERpE5BjO1Bo5JLm+Eaz+5xN+QgOXsZtB IsaT3UH2c/xLQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 93EFECE0F4C; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 06/16] rcutorture: Exercise DEFINE_STATIC_SRCU_FAST() and init_srcu_struct_fast() Date: Wed, 5 Nov 2025 12:32:06 -0800 Message-Id: <20251105203216.2701005-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit updates the initialization for the "srcu" and "srcud" torture types to use DEFINE_STATIC_SRCU_FAST() and init_srcu_struct_fast(), respectively, when reader_flavor is equal to SRCU_READ_FLAVOR_FAST. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- kernel/rcu/rcutorture.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index ea8077c08fac..8022d32be351 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -692,10 +692,18 @@ static struct rcu_torture_ops rcu_busted_ops =3D { */ =20 DEFINE_STATIC_SRCU(srcu_ctl); +DEFINE_STATIC_SRCU_FAST(srcu_ctlf); static struct srcu_struct srcu_ctld; static struct srcu_struct *srcu_ctlp =3D &srcu_ctl; static struct rcu_torture_ops srcud_ops; =20 +static void srcu_torture_init(void) +{ + rcu_sync_torture_init(); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) + srcu_ctlp =3D &srcu_ctlf; +} + static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) { srcutorture_get_gp_data(srcu_ctlp, flags, gp_seq); @@ -865,7 +873,7 @@ static void srcu_torture_expedite_current(void) =20 static struct rcu_torture_ops srcu_ops =3D { .ttype =3D SRCU_FLAVOR, - .init =3D rcu_sync_torture_init, + .init =3D srcu_torture_init, .readlock =3D srcu_torture_read_lock, .read_delay =3D srcu_read_delay, .readunlock =3D srcu_torture_read_unlock, @@ -897,10 +905,13 @@ static struct rcu_torture_ops srcu_ops =3D { .name =3D "srcu" }; =20 -static void srcu_torture_init(void) +static void srcud_torture_init(void) { rcu_sync_torture_init(); - WARN_ON(init_srcu_struct(&srcu_ctld)); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) + WARN_ON(init_srcu_struct_fast(&srcu_ctld)); + else + WARN_ON(init_srcu_struct(&srcu_ctld)); srcu_ctlp =3D &srcu_ctld; } =20 @@ -913,7 +924,7 @@ static void srcu_torture_cleanup(void) /* As above, but dynamically allocated. */ static struct rcu_torture_ops srcud_ops =3D { .ttype =3D SRCU_FLAVOR, - .init =3D srcu_torture_init, + .init =3D srcud_torture_init, .cleanup =3D srcu_torture_cleanup, .readlock =3D srcu_torture_read_lock, .read_delay =3D srcu_read_delay, --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9D6934B192; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=BL1heNAumVis6F3dGqFcA+9GQjNgYMX+SJFMiJifJZIulkU3diAhD/Tfqwh1KdXkoeT7LQJo817ELPJKc4P4DdLlLyPMWkfQLaeSjDXGc2oh7vvaYqZ5w+9RcNciW/dRdLZyMxmj5BpIk+HT6xcrhGyf+KXvjJtbWC5IVvaX4gE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=Iq5yf1ytJm34oKtGaJvxP71I3WkFsasL56D7JZi7uYs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Prt5z5+9Nv91+hv9BlPlOqSDGoMrayRoifO355VuQ3WFuoRFwZfO97FDl43hFtARq8WTEn7q9up07dw9hriBfudzcC9dCY5bDZncaFvswBw5MuEws0PFsbKqyXTOVihNPZ8SsMbPmDiUmeR+fZskolKbuWahvwqKBcSLt45bdNs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=NnbpbU9z; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="NnbpbU9z" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A1D2C19422; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374740; bh=Iq5yf1ytJm34oKtGaJvxP71I3WkFsasL56D7JZi7uYs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NnbpbU9zgonUCD+g1k8V6B3Vb7Tnq9+ZT5Z9D35pM0/ezUfJDQNR1RCLMoLhGfvfP NLbdPna1nhaTon32nFH8XHdIJgd3vYMJno1JUHJN40jmrmwWN+uC0Gd80f5hL8WoCf 9HjTfd675e55lSK+DztPZcGieXgA87N5FrZBmh+qc82Bq/WZjYiqF6hdZqu0Sva16j IASuuDBwu5ogPk9DMBYXXR0WRxs5HkbwB/Mm/Fgo2pG6Tx/dL5bcR5W3Ymv3n92aEf o3EfFzdYy0R6iRzLQ02LYWeb/oYbOlS6kKMMrFqqzZ0DPsxEU7GWWhRXU4T0nhP+jG NQIhCJ1TjNH+A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 9676FCE0F56; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 07/16] srcu: Require special srcu_struct define/init for SRCU-fast readers Date: Wed, 5 Nov 2025 12:32:07 -0800 Message-Id: <20251105203216.2701005-7-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds CONFIG_PROVE_RCU=3Dy checking to enforce the new rule that srcu_struct structures passed to srcu_read_lock_fast() and other SRCU-fast read-side markers be either initialized with init_srcu_struct_fast() on the one hand or defined using either DEFINE_SRCU_FAST() or DEFINE_STATIC_SRCU_FAST(). This will enable removal of the non-debug read-side checks from srcu_read_lock_fast() and friends, which on my laptop provides a 25% speedup (which admittedly amounts to about half a nanosecond, but when tracing fastpaths...) Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 34 ++++++++++++++++++++++------------ kernel/rcu/srcutree.c | 1 + 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 26de47820c58..2982b5a6930f 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -271,17 +271,26 @@ static inline int srcu_read_lock(struct srcu_struct *= ssp) __acquires(ssp) * @ssp: srcu_struct in which to register the new reader. * * Enter an SRCU read-side critical section, but for a light-weight - * smp_mb()-free reader. See srcu_read_lock() for more information. - * - * If srcu_read_lock_fast() is ever used on an srcu_struct structure, - * then none of the other flavors may be used, whether before, during, - * or after. Note that grace-period auto-expediting is disabled for _fast - * srcu_struct structures because auto-expedited grace periods invoke - * synchronize_rcu_expedited(), IPIs and all. - * - * Note that srcu_read_lock_fast() can be invoked only from those contexts - * where RCU is watching, that is, from contexts where it would be legal - * to invoke rcu_read_lock(). Otherwise, lockdep will complain. + * smp_mb()-free reader. See srcu_read_lock() for more information. This + * function is NMI-safe, in a manner similar to srcu_read_lock_nmisafe(). + * + * For srcu_read_lock_fast() to be used on an srcu_struct structure, + * that structure must have been defined using either DEFINE_SRCU_FAST() + * or DEFINE_STATIC_SRCU_FAST() on the one hand or initialized with + * init_srcu_struct_fast() on the other. Such an srcu_struct structure + * cannot be passed to any non-fast variant of srcu_read_{,un}lock() or + * srcu_{down,up}_read(). In kernels built with CONFIG_PROVE_RCU=3Dy, + * __srcu_check_read_flavor() will complain bitterly if you ignore this + * restriction. + * + * Grace-period auto-expediting is disabled for SRCU-fast srcu_struct + * structures because SRCU-fast expedited grace periods invoke + * synchronize_rcu_expedited(), IPIs and all. If you need expedited + * SRCU-fast grace periods, use synchronize_srcu_expedited(). + * + * The srcu_read_lock_fast() function can be invoked only from those + * contexts where RCU is watching, that is, from contexts where it would + * be legal to invoke rcu_read_lock(). Otherwise, lockdep will complain. */ static inline struct srcu_ctr __percpu *srcu_read_lock_fast(struct srcu_st= ruct *ssp) __acquires(ssp) { @@ -317,7 +326,8 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast_notrace(struct srcu_ * srcu_down_read() for more information. * * The same srcu_struct may be used concurrently by srcu_down_read_fast() - * and srcu_read_lock_fast(). + * and srcu_read_lock_fast(). However, the same definition/initialization + * requirements called out for srcu_read_lock_safe() apply. */ static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_st= ruct *ssp) __acquires(ssp) { diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index c29203b23d1a..2f8aa280911e 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -766,6 +766,7 @@ void __srcu_check_read_flavor(struct srcu_struct *ssp, = int read_flavor) WARN_ON_ONCE(ssp->srcu_reader_flavor && read_flavor !=3D ssp->srcu_reader= _flavor); WARN_ON_ONCE(old_read_flavor && ssp->srcu_reader_flavor && old_read_flavor !=3D ssp->srcu_reader_flavor); + WARN_ON_ONCE(read_flavor =3D=3D SRCU_READ_FLAVOR_FAST && !ssp->srcu_reade= r_flavor); if (!old_read_flavor) { old_read_flavor =3D cmpxchg(&sdp->srcu_reader_flavor, 0, read_flavor); if (!old_read_flavor) --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C672734AB1C; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=RD7PJ3LLlsWJr6dG0YKz7Mf1P5XBppJVYxAUNfURCGxISf/q8KYYSeKFdAp++HmxR4BvlDnjwP/RZLKlsknL9lljIKllWoIB8j4XmwiiSjjNGX4j6S6y9q/UeOdbmnvEv6I0jIzo/nQd0DlRjxGjsfWDem5KRdYNpylaU6601p0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=MZmvN2dS91ATVvwCaKa51PWDnZiCJGVXmJf2Q3doHC0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hln0L4p7r1bqWk/0D64Ki07vEvAQkYxTVvC92Cm8DbHGz0A7zAVWedAJlvYAl+Esvr70cvREgj86k70w8vQ5HGcC722JSCHgimo1ECAg+AumoKELxkyJ1CzmNCWXAZ0pFmHPW8KrlcHKUUPevfcuRbYXB+fyrrBLcRlLeG7OnkI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IQgi5fxR; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IQgi5fxR" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5ADDFC4AF0C; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374740; bh=MZmvN2dS91ATVvwCaKa51PWDnZiCJGVXmJf2Q3doHC0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IQgi5fxR5PnrnjE9tykMd/2rpgOQ3qY2iIK4XVLA2X5khMkAPigAY2KTHP4pDomK6 I0tUH7i9ex4SxbivNVt/0mK2uGBWmD6weNBXpMFi3qESqGpjgrv7cPaYXcWqPHKs2o QYo55dC6FmT5uVB50KueuWZJCZcm3JriQ8FxecR0LOso0bdDMsIZ/nD450AtJPV5VM +cdGmurTFNDxR2u9hJWqCCoMi0cY1aC7oKCsFKmTE04XbHf2GnghHNgCTp5Tjm6b7G nGl5lDwYvi97C7/tCNIO+b9emJYX+x9qTUHj/0NwNfngPxadTDMLV3rkBpNrtMMJ9g /wIhNv5IbIsgQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 9950ACE0F65; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 08/16] srcu: Make SRCU-fast readers enforce use of SRCU-fast definition/init Date: Wed, 5 Nov 2025 12:32:08 -0800 Message-Id: <20251105203216.2701005-8-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit makes CONFIG_PROVE_RCU=3Dy kernels enforce the new rule that srcu_struct structures that are passed to srcu_read_lock_fast() and other SRCU-fast read-side markers be either initialized with init_srcu_struct_fast() on the one hand or defined with DEFINE_SRCU_FAST() or DEFINE_STATIC_SRCU_FAST() on the other. This eliminates the read-side test that was formerly included in srcu_read_lock_fast() and friends, speeding these primitives up by about 25% (admittedly only about half of a nanosecond, but when tracing on fastpaths...) Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 6 +++--- include/linux/srcutiny.h | 1 - include/linux/srcutree.h | 16 +--------------- 3 files changed, 4 insertions(+), 19 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 2982b5a6930f..41e27c1d917d 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -297,7 +297,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast(struct srcu_struct * struct srcu_ctr __percpu *retval; =20 RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast()."); - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); retval =3D __srcu_read_lock_fast(ssp); rcu_try_lock_acquire(&ssp->dep_map); return retval; @@ -312,7 +312,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast_notrace(struct srcu_ { struct srcu_ctr __percpu *retval; =20 - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); retval =3D __srcu_read_lock_fast(ssp); return retval; } @@ -333,7 +333,7 @@ static inline struct srcu_ctr __percpu *srcu_down_read_= fast(struct srcu_struct * { WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read= _fast()."); - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); return __srcu_read_lock_fast(ssp); } =20 diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 92e6ab53398f..1ecc3393fb26 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -112,7 +112,6 @@ static inline void srcu_barrier(struct srcu_struct *ssp) =20 static inline void srcu_expedite_current(struct srcu_struct *ssp) { } #define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) -#define srcu_check_read_flavor_force(ssp, read_flavor) do { } while (0) =20 /* Defined here to avoid size increase for non-torture kernels. */ static inline void srcu_torture_stats_print(struct srcu_struct *ssp, diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 7ff4a11bc5a3..6080a9094618 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -307,21 +307,7 @@ __srcu_read_unlock_fast(struct srcu_struct *ssp, struc= t srcu_ctr __percpu *scp) =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); =20 -// Record reader usage even for CONFIG_PROVE_RCU=3Dn kernels. This is -// needed only for flavors that require grace-period smp_mb() calls to be -// promoted to synchronize_rcu(). -static inline void srcu_check_read_flavor_force(struct srcu_struct *ssp, i= nt read_flavor) -{ - struct srcu_data *sdp =3D raw_cpu_ptr(ssp->sda); - - if (likely(READ_ONCE(sdp->srcu_reader_flavor) & read_flavor)) - return; - - // Note that the cmpxchg() in __srcu_check_read_flavor() is fully ordered. - __srcu_check_read_flavor(ssp, read_flavor); -} - -// Record non-_lite() usage only for CONFIG_PROVE_RCU=3Dy kernels. +// Record SRCU-reader usage type only for CONFIG_PROVE_RCU=3Dy kernels. static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int rea= d_flavor) { if (IS_ENABLED(CONFIG_PROVE_RCU)) --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C687834AB1E; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=IR4lNCnnqsWPijltbuCFGC4XK6o5alE/TTqncfXUXKS1WUyWqHNaHrFcSZKB+2E0kjWyrnDXgE4qW4wdg7eqaHdxZVNIzVffk9YCr58PGUJxDf3w1uwfJdbS3DHvxBhl/YzKDpz4EBm/thui/UDK78iMqu8Cc7bxAWtXuKDihgc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=7uPrv4irYTmEwJ2B3VG3qdNupsgZm8tMakjud6mO6xs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=J72t8dIp5WLhcE8Fyp0+L+6lCJEZtK+BvndWEKkIqvmZlM1lJU6Mxv85bgWX275lSKCO6maWK2ojfijuVB2cfZmK7KTcHFqrxLWc5foFdwg4EZlv3c3LCP8l+EQGydexP35+20dcrXj00t5jhKRMGL9T9uPdDYUkUzVaMRIAWoI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bTzmmkrY; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bTzmmkrY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A087C16AAE; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374740; bh=7uPrv4irYTmEwJ2B3VG3qdNupsgZm8tMakjud6mO6xs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bTzmmkrY46z2jaGX+VV+L4GYI1IHCnoNC6T68gI2f3l014ApTF6Y754EFT82w3oOL 8yjuA0T7uZw0bhDMPQzOIMnUN0LTzcvRtQ5povhIzd/rBBB6eRzjImNiRIO+4nI8fP EqOAsZ0VwPlMN4xgosnwynjFdBjAoCnxgryWJ4ibxgP4uRNWantt7KFQyE+RrNElYd wX5QyczHdhsvFT0bSoY7rkBDQgL7w8cZ9LNMqY46ebC8L9MAjkbxrQXe2aNFMOW3Up s/SHi1G66EFVR0zFcn0osygCw7qzrunRf8+U5Gjx7xtIfZviBgbWLsFNdNgl/vH4Kw JtVe58v8wWneA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 9BE01CE0F7D; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 09/16] doc: Update for SRCU-fast definitions and initialization Date: Wed, 5 Nov 2025 12:32:09 -0800 Message-Id: <20251105203216.2701005-9-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit documents the DEFINE_SRCU_FAST(), DEFINE_STATIC_SRCU_FAST(), and init_srcu_struct_fast() API members. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- .../RCU/Design/Requirements/Requirements.rst | 33 ++++++++++--------- Documentation/RCU/checklist.rst | 12 ++++--- Documentation/RCU/whatisRCU.rst | 3 ++ 3 files changed, 27 insertions(+), 21 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Docum= entation/RCU/Design/Requirements/Requirements.rst index f24b3c0b9b0d..ba417a08b93d 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2637,15 +2637,16 @@ synchronize_srcu() for some other domain ``ss1``, a= nd if an that was held across as ``ss``-domain synchronize_srcu(), deadlock would again be possible. Such a deadlock cycle could extend across an arbitrarily large number of different SRCU domains. Again, with great -power comes great responsibility. +power comes great responsibility, though lockdep is now able to detect +this sort of deadlock. =20 -Unlike the other RCU flavors, SRCU read-side critical sections can run -on idle and even offline CPUs. This ability requires that -srcu_read_lock() and srcu_read_unlock() contain memory barriers, -which means that SRCU readers will run a bit slower than would RCU -readers. It also motivates the smp_mb__after_srcu_read_unlock() API, -which, in combination with srcu_read_unlock(), guarantees a full -memory barrier. +Unlike the other RCU flavors, SRCU read-side critical sections can run on +idle and even offline CPUs, with the exception of srcu_read_lock_fast() +and friends. This ability requires that srcu_read_lock() and +srcu_read_unlock() contain memory barriers, which means that SRCU +readers will run a bit slower than would RCU readers. It also motivates +the smp_mb__after_srcu_read_unlock() API, which, in combination with +srcu_read_unlock(), guarantees a full memory barrier. =20 Also unlike other RCU flavors, synchronize_srcu() may **not** be invoked from CPU-hotplug notifiers, due to the fact that SRCU grace @@ -2681,15 +2682,15 @@ run some tests first. SRCU just might need a few ad= justment to deal with that sort of load. Of course, your mileage may vary based on the speed of your CPUs and the size of your memory. =20 -The `SRCU -API `__ +The `SRCU API +`__ includes srcu_read_lock(), srcu_read_unlock(), -srcu_dereference(), srcu_dereference_check(), -synchronize_srcu(), synchronize_srcu_expedited(), -call_srcu(), srcu_barrier(), and srcu_read_lock_held(). It -also includes DEFINE_SRCU(), DEFINE_STATIC_SRCU(), and -init_srcu_struct() APIs for defining and initializing -``srcu_struct`` structures. +srcu_dereference(), srcu_dereference_check(), synchronize_srcu(), +synchronize_srcu_expedited(), call_srcu(), srcu_barrier(), +and srcu_read_lock_held(). It also includes DEFINE_SRCU(), +DEFINE_STATIC_SRCU(), DEFINE_SRCU_FAST(), DEFINE_STATIC_SRCU_FAST(), +init_srcu_struct(), and init_srcu_struct_fast() APIs for defining and +initializing ``srcu_struct`` structures. =20 More recently, the SRCU API has added polling interfaces: =20 diff --git a/Documentation/RCU/checklist.rst b/Documentation/RCU/checklist.= rst index c9bfb2b218e5..4b30f701225f 100644 --- a/Documentation/RCU/checklist.rst +++ b/Documentation/RCU/checklist.rst @@ -417,11 +417,13 @@ over a rather long period of time, but improvements a= re always welcome! you should be using RCU rather than SRCU, because RCU is almost always faster and easier to use than is SRCU. =20 - Also unlike other forms of RCU, explicit initialization and - cleanup is required either at build time via DEFINE_SRCU() - or DEFINE_STATIC_SRCU() or at runtime via init_srcu_struct() - and cleanup_srcu_struct(). These last two are passed a - "struct srcu_struct" that defines the scope of a given + Also unlike other forms of RCU, explicit initialization + and cleanup is required either at build time via + DEFINE_SRCU(), DEFINE_STATIC_SRCU(), DEFINE_SRCU_FAST(), + or DEFINE_STATIC_SRCU_FAST() or at runtime via either + init_srcu_struct() or init_srcu_struct_fast() and + cleanup_srcu_struct(). These last three are passed a + `struct srcu_struct` that defines the scope of a given SRCU domain. Once initialized, the srcu_struct is passed to srcu_read_lock(), srcu_read_unlock() synchronize_srcu(), synchronize_srcu_expedited(), and call_srcu(). A given diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.= rst index cf0b0ac9f463..a1582bd653d1 100644 --- a/Documentation/RCU/whatisRCU.rst +++ b/Documentation/RCU/whatisRCU.rst @@ -1227,7 +1227,10 @@ SRCU: Initialization/cleanup/ordering:: =20 DEFINE_SRCU DEFINE_STATIC_SRCU + DEFINE_SRCU_FAST // for srcu_read_lock_fast() and friends + DEFINE_STATIC_SRCU_FAST // for srcu_read_lock_fast() and friends init_srcu_struct + init_srcu_struct_fast cleanup_srcu_struct smp_mb__after_srcu_read_unlock =20 --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B63BC34AB15; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=Q8ntBHSOJiEzXuFZA0udDarUoZt4Cq9pWmVgXsgiGx2fvF8S1Wu4zI+PR/EGf7rgPFMOPrALBbkmJb3ZVoCjR+lYVAyC7jU/rSz4xdpuJ8t1WitgceBEYMBaQkUVQuLDSiV0Cw0GNd5VqAfueo3L6ov8cdISXH03FK5aKtriduY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=MgE4NzahsQRutiZm/q0L791+QKh3LZkPzHRxKVcKPeY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eITdbNE+SHBxzcFxJsWqVUmYivRnCmirYjSTHO0gF0tbOcl/YdoBGzvgf6TPrnpaNm5aXG7OtquwrUKOfpWA+0x30KHCTvxIAslPnTruKBDH/FQxsotFD9OkXolHixYspuyU+DnfbSNXPGLQogeonCYD9M0kBHITNjjjrsih0IM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Bdm74BJn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Bdm74BJn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 75232C19424; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374740; bh=MgE4NzahsQRutiZm/q0L791+QKh3LZkPzHRxKVcKPeY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Bdm74BJnJcdCtzT7ZQrLeyqf2ipXfd4FAsujMDHxo6fpT59s7FOPa/zQGXDArhxpc h0zlbqYdmqFmrS4BW9ywPa/Rz2b1dPnkhQNpUBi9JTAM2iDv4s2rkFcnPYyG1gLRvy akPdpnEO+EilxC5Dr3kC7HsH3REOE94LiaLqzrnKdicGELIxIvzZm1VA6ONOk5ZAko V5+0WzSnEwB+KyH/4oiWc+QK2fBlEmtFPH6KxfinPKGhsZvN+0tuKbdKLr2E32FW41 3SMQzvBt9Fjovu8U7M1fJtRWVzYaAMj4dpLdTxtTmOHTZxXd+4kM24059ZJcXkMiHw TXcqR/BlDQ2rA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 9E837CE0F7E; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 10/16] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast Date: Wed, 5 Nov 2025 12:32:10 -0800 Message-Id: <20251105203216.2701005-10-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current use of guard(preempt_notrace)() within __DECLARE_TRACE() to protect invocation of __DO_TRACE_CALL() means that BPF programs attached to tracepoints are non-preemptible. This is unhelpful in real-time systems, whose users apparently wish to use BPF while also achieving low latencies. (Who knew?) One option would be to use preemptible RCU, but this introduces many opportunities for infinite recursion, which many consider to be counterproductive, especially given the relatively small stacks provided by the Linux kernel. These opportunities could be shut down by sufficiently energetic duplication of code, but this sort of thing is considered impolite in some circles. Therefore, use the shiny new SRCU-fast API, which provides somewhat faster readers than those of preemptible RCU, at least on my laptop, where task_struct access is more expensive than access to per-CPU variables. And SRCU fast provides way faster readers than does SRCU, courtesy of being able to avoid the read-side use of smp_mb(). Also, it is quite straightforward to create srcu_read_{,un}lock_fast_notrace() functions. While in the area, SRCU now supports early boot call_srcu(). Therefore, remove the checks that used to avoid such use from rcu_free_old_probes() before this commit was applied: e53244e2c893 ("tracepoint: Remove SRCU protection") The current commit can be thought of as an approximate revert of that commit, with some compensating additions of preemption disabling pointed out by Steven Rostedt (thank you, Steven!). This preemption disabling uses guard(preempt_notrace)(), and while in the area a couple of other use cases were also converted to guards. However, Yonghong Song points out that BPF expects non-sleepable BPF programs to remain on the same CPU, which means that migration must be disabled whenever preemption remains enabled. In addition, non-RT kernels have performance expectations on BPF that would be violated by allowing the BPF programs to be preempted. Therefore, continue to disable preemption in non-RT kernels, and protect the BPF program with both SRCU and migration disabling for RT kernels, and even then only if preemption is not already disabled. [ paulmck: Apply kernel test robot and Yonghong Song feedback. ] Link: https://lore.kernel.org/all/20250613152218.1924093-1-bigeasy@linutron= ix.de/ Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/tracepoint.h | 45 ++++++++++++++++++++++-------------- include/trace/perf.h | 4 ++-- include/trace/trace_events.h | 4 ++-- kernel/tracepoint.c | 21 ++++++++++++++++- 4 files changed, 52 insertions(+), 22 deletions(-) diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 826ce3f8e1f8..9f8b19cd303a 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -33,6 +33,8 @@ struct trace_eval_map { =20 #define TRACEPOINT_DEFAULT_PRIO 10 =20 +extern struct srcu_struct tracepoint_srcu; + extern int tracepoint_probe_register(struct tracepoint *tp, void *probe, void *data); extern int @@ -115,7 +117,10 @@ void for_each_tracepoint_in_module(struct module *mod, static inline void tracepoint_synchronize_unregister(void) { synchronize_rcu_tasks_trace(); - synchronize_rcu(); + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + synchronize_srcu(&tracepoint_srcu); + else + synchronize_rcu(); } static inline bool tracepoint_is_faultable(struct tracepoint *tp) { @@ -266,23 +271,29 @@ static inline struct tracepoint *tracepoint_ptr_deref= (tracepoint_ptr_t *p) return static_branch_unlikely(&__tracepoint_##name.key);\ } =20 -#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \ +#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \ __DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_pro= to)) \ - static inline void __do_trace_##name(proto) \ - { \ - if (cond) { \ - guard(preempt_notrace)(); \ - __DO_TRACE_CALL(name, TP_ARGS(args)); \ - } \ - } \ - static inline void trace_##name(proto) \ - { \ - if (static_branch_unlikely(&__tracepoint_##name.key)) \ - __do_trace_##name(args); \ - if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \ - WARN_ONCE(!rcu_is_watching(), \ - "RCU not watching for tracepoint"); \ - } \ + static inline void __do_trace_##name(proto) \ + { \ + if (cond) { \ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && preemptible()) { \ + guard(srcu_fast_notrace)(&tracepoint_srcu); \ + guard(migrate)(); \ + __DO_TRACE_CALL(name, TP_ARGS(args)); \ + } else { \ + guard(preempt_notrace)(); \ + __DO_TRACE_CALL(name, TP_ARGS(args)); \ + } \ + } \ + } \ + static inline void trace_##name(proto) \ + { \ + if (static_branch_unlikely(&__tracepoint_##name.key)) \ + __do_trace_##name(args); \ + if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \ + WARN_ONCE(!rcu_is_watching(), \ + "RCU not watching for tracepoint"); \ + } \ } =20 #define __DECLARE_TRACE_SYSCALL(name, proto, args, data_proto) \ diff --git a/include/trace/perf.h b/include/trace/perf.h index a1754b73a8f5..348ad1d9b556 100644 --- a/include/trace/perf.h +++ b/include/trace/perf.h @@ -71,6 +71,7 @@ perf_trace_##call(void *__data, proto) \ u64 __count __attribute__((unused)); \ struct task_struct *__task __attribute__((unused)); \ \ + guard(preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ } =20 @@ -85,9 +86,8 @@ perf_trace_##call(void *__data, proto) \ struct task_struct *__task __attribute__((unused)); \ \ might_fault(); \ - preempt_disable_notrace(); \ + guard(preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ - preempt_enable_notrace(); \ } =20 /* diff --git a/include/trace/trace_events.h b/include/trace/trace_events.h index 4f22136fd465..fbc07d353be6 100644 --- a/include/trace/trace_events.h +++ b/include/trace/trace_events.h @@ -436,6 +436,7 @@ __DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args)= , PARAMS(tstruct), \ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \ { \ + guard(preempt_notrace)(); \ do_trace_event_raw_event_##call(__data, args); \ } =20 @@ -447,9 +448,8 @@ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \ { \ might_fault(); \ - preempt_disable_notrace(); \ + guard(preempt_notrace)(); \ do_trace_event_raw_event_##call(__data, args); \ - preempt_enable_notrace(); \ } =20 /* diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 62719d2941c9..21bb67798214 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -25,6 +25,9 @@ enum tp_func_state { extern tracepoint_ptr_t __start___tracepoints_ptrs[]; extern tracepoint_ptr_t __stop___tracepoints_ptrs[]; =20 +DEFINE_SRCU_FAST(tracepoint_srcu); +EXPORT_SYMBOL_GPL(tracepoint_srcu); + enum tp_transition_sync { TP_TRANSITION_SYNC_1_0_1, TP_TRANSITION_SYNC_N_2_1, @@ -34,6 +37,7 @@ enum tp_transition_sync { =20 struct tp_transition_snapshot { unsigned long rcu; + unsigned long srcu_gp; bool ongoing; }; =20 @@ -46,6 +50,7 @@ static void tp_rcu_get_state(enum tp_transition_sync sync) =20 /* Keep the latest get_state snapshot. */ snapshot->rcu =3D get_state_synchronize_rcu(); + snapshot->srcu_gp =3D start_poll_synchronize_srcu(&tracepoint_srcu); snapshot->ongoing =3D true; } =20 @@ -56,6 +61,8 @@ static void tp_rcu_cond_sync(enum tp_transition_sync sync) if (!snapshot->ongoing) return; cond_synchronize_rcu(snapshot->rcu); + if (!poll_state_synchronize_srcu(&tracepoint_srcu, snapshot->srcu_gp)) + synchronize_srcu(&tracepoint_srcu); snapshot->ongoing =3D false; } =20 @@ -101,17 +108,29 @@ static inline void *allocate_probes(int count) return p =3D=3D NULL ? NULL : p->probes; } =20 -static void rcu_free_old_probes(struct rcu_head *head) +static void srcu_free_old_probes(struct rcu_head *head) { kfree(container_of(head, struct tp_probes, rcu)); } =20 +static void rcu_free_old_probes(struct rcu_head *head) +{ + call_srcu(&tracepoint_srcu, head, srcu_free_old_probes); +} + static inline void release_probes(struct tracepoint *tp, struct tracepoint= _func *old) { if (old) { struct tp_probes *tp_probes =3D container_of(old, struct tp_probes, probes[0]); =20 + /* + * Tracepoint probes are protected by either RCU or + * Tasks Trace RCU and also by SRCU. By calling the SRCU + * callback in the [Tasks Trace] RCU callback we cover + * both cases. So let us chain the SRCU and [Tasks Trace] + * RCU callbacks to wait for both grace periods. + */ if (tracepoint_is_faultable(tp)) call_rcu_tasks_trace(&tp_probes->rcu, rcu_free_old_probes); else --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9FFB34B1A3; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; cv=none; b=lNu2xDmxh7aU4nxd7B0uV/gA5xuKJ0X/ZjfqzAGlYccXNEyoV4Chi24CcycURCXEXbBuEXbeo4yv35d/euS8aSR0cUFnfVA2GE5X5Zsz/rG/AzFbUJlA3qf9DK28aIp5HBLwpvy/8B1PPSawSc9wjZVXpIj5kQTa0Q5lSaRGYsY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374740; c=relaxed/simple; bh=BK4yD7eyeQ0mHbL7KbeZ8Mx12t+bQxDXJ6lt6IBpXBE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=F388LORCc04P/wM2WCcqKJuOz+b4gO5Sy275UCg5xpkC9yS253R4Ujshif5V/uLDdu6Xim/edyo+2S6Mn/Xf0sj1rYzRiZ/Cai7cuBTvmjvtrzUpBXzRe+389XzrMbs+W+20XVve6Loq5ebZe1GkmGjc0Jt007l+9n1qarIPo5k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=vAvkoYk2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="vAvkoYk2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7263BC116C6; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374740; bh=BK4yD7eyeQ0mHbL7KbeZ8Mx12t+bQxDXJ6lt6IBpXBE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vAvkoYk2CcJVzgorkbzUvypit15MwLpUqJl/7tfiunIa12HGN5nZgFfkphZhk2KEu jfODpm1TqXr6qQzQ0AuTlMwkQ/uqj6GE7uXaoJvAjRWbbNzY45GHVkczETgoEbVdPV 1rucAZpM7ChQ6Wuo5gZBLApOaupy3wsh7yvylFbeFcTNnfN3co3HtEffizhBD3qcAt VEszxKC9IO1QKa51+tjuYUF2z89qyGbilox7Do0yjyR3F8bXjeisshIWj1bXXUvJL2 e26umtlPglL9G16iTKEo7nJBX2vpuGgaGvChJwZ94JXy19utTaxin9FqV0hLlNN5c2 ba2/5UlwhCnOA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A133ACE0F8F; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Leon Hwang , Alexei Starovoitov Subject: [PATCH v2 11/16] rcu: Mark diagnostic functions as notrace Date: Wed, 5 Nov 2025 12:32:11 -0800 Message-Id: <20251105203216.2701005-11-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The rcu_lockdep_current_cpu_online(), rcu_read_lock_sched_held(), rcu_read_lock_held(), rcu_read_lock_bh_held(), rcu_read_lock_any_held() are used by tracing-related code paths, so putting traces on them is unlikely to make anyone happy. This commit therefore marks them all "notrace". Reported-by: Leon Hwang Reported-by: Alexei Starovoitov Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- kernel/rcu/update.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 31690ffa452a..8ddd07fed363 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4021,7 +4021,7 @@ bool rcu_cpu_online(int cpu) * RCU on an offline processor during initial boot, hence the check for * rcu_scheduler_fully_active. */ -bool rcu_lockdep_current_cpu_online(void) +bool notrace rcu_lockdep_current_cpu_online(void) { struct rcu_data *rdp; bool ret =3D false; diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index c912b594ba98..dfeba9b35395 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -117,7 +117,7 @@ static bool rcu_read_lock_held_common(bool *ret) return false; } =20 -int rcu_read_lock_sched_held(void) +int notrace rcu_read_lock_sched_held(void) { bool ret; =20 @@ -342,7 +342,7 @@ EXPORT_SYMBOL_GPL(debug_lockdep_rcu_enabled); * Note that rcu_read_lock() is disallowed if the CPU is either idle or * offline from an RCU perspective, so check for those as well. */ -int rcu_read_lock_held(void) +int notrace rcu_read_lock_held(void) { bool ret; =20 @@ -367,7 +367,7 @@ EXPORT_SYMBOL_GPL(rcu_read_lock_held); * Note that rcu_read_lock_bh() is disallowed if the CPU is either idle or * offline from an RCU perspective, so check for those as well. */ -int rcu_read_lock_bh_held(void) +int notrace rcu_read_lock_bh_held(void) { bool ret; =20 @@ -377,7 +377,7 @@ int rcu_read_lock_bh_held(void) } EXPORT_SYMBOL_GPL(rcu_read_lock_bh_held); =20 -int rcu_read_lock_any_held(void) +int notrace rcu_read_lock_any_held(void) { bool ret; =20 --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46E4D34B661; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374741; cv=none; b=N+ivOAElpW7nOw1WyanHPx//1KMhLSa20o8wylXB+Zq2Ia+yVn/pGIfwBYzlbFTB60Am6bgiz/vhGR7mErwlK9Y1m1+XVFX9KkNpxhSnhmUBlr9/QKNk3mPDQ9arWKDw+G+RW2ynJ1cvINV0RUKi/duvQgQihpG0bCIJ9lX2y6o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374741; c=relaxed/simple; bh=SxM5obABZJXPt/ymairqXuBar3neaMVGlr0dx2kNeSw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SdEmfr+2r2Z9S3G+CBgNRwOVghXJsVmgIjBojwkDjvLVfSmnyH6fUN/31cNtCBojEaIOPFD1Fj+0yZeRdJc4NSE96RwUPlbXuTaua1PbWvLppmtd0KroLZ5r9LhZxfBORAVYvu4akGmJuzn3/uRjHlGpMujDciJ7G2+OfQlyhkk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rQqWGwL2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rQqWGwL2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7C961C2BC9E; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374740; bh=SxM5obABZJXPt/ymairqXuBar3neaMVGlr0dx2kNeSw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rQqWGwL2o9drFtCx5CpwBGO7KB4JNdBRaUaq84ybIBhIli6XjwarMA25lLOonojmp hoMPAou4EpsZfVBlw6m0o2ir8YNHMNhJCqWjSaj+uCwe9f/lNG8OtBjpWo2YEijvuA 8uHihImeTRkCBPpjlRKCVuOX/YZJcH0DNCyviZr3JwGITXptP45s2qXZEhRBrbfy5j 0P0V4tkauD/K8qEim1Uld8pS+uDyHl+1TT6SFf1TcSj5ZF70TmW+OXc9GuXjfCRXgP Tk5emLbiN3J4KK+H2i5s5M3RdsLZLLyUDmC9tOmBe23MITswn59ephbHAtLYKvtzUD WS/Z0yMLdAGXQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A3B00CE0F98; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 12/16] srcu: Add SRCU_READ_FLAVOR_FAST_UPDOWN CPP macro Date: Wed, 5 Nov 2025 12:32:12 -0800 Message-Id: <20251105203216.2701005-12-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds the SRCU_READ_FLAVOR_FAST_UPDOWN=3D0x8 macro and adjusts rcutorture to make use of it. In this commit, both SRCU_READ_FLAVOR_FAST=3D0x4 and the new SRCU_READ_FLAVOR_FAST_UPDOWN test SRCU-fast. When the SRCU-fast-updown is added, the new SRCU_READ_FLAVOR_FAST_UPDOWN macro will test it when passed to the rcutorture.reader_flavor module parameter. The old SRCU_READ_FLAVOR_FAST macro's value changed from 0x8 to 0x4. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 16 +++++++++------- kernel/rcu/rcutorture.c | 24 ++++++++++++++++++------ 2 files changed, 27 insertions(+), 13 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 41e27c1d917d..1dd6812aabe7 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -56,13 +56,15 @@ int init_srcu_struct_fast(struct srcu_struct *ssp); #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /* Values for SRCU Tree srcu_data ->srcu_reader_flavor, but also used by r= cutorture. */ -#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). -#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). -// 0x4 // SRCU-lite is no longer with us. -#define SRCU_READ_FLAVOR_FAST 0x8 // srcu_read_lock_fast(). -#define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR= _NMI | \ - SRCU_READ_FLAVOR_FAST) // All of the above. -#define SRCU_READ_FLAVOR_SLOWGP SRCU_READ_FLAVOR_FAST +#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). +#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). +// 0x4 // SRCU-lite is no longer with us. +#define SRCU_READ_FLAVOR_FAST 0x4 // srcu_read_lock_fast(). +#define SRCU_READ_FLAVOR_FAST_UPDOWN 0x8 // srcu_read_lock_fast(). +#define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_= NMI | \ + SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN) + // All of the above. +#define SRCU_READ_FLAVOR_SLOWGP (SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR= _FAST_UPDOWN) // Flavors requiring synchronize_rcu() // instead of smp_mb(). void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp); diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 8022d32be351..70ee838cab55 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -702,6 +702,8 @@ static void srcu_torture_init(void) rcu_sync_torture_init(); if (reader_flavor & SRCU_READ_FLAVOR_FAST) srcu_ctlp =3D &srcu_ctlf; + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) + srcu_ctlp =3D &srcu_ctlf; } =20 static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) @@ -728,6 +730,12 @@ static int srcu_torture_read_lock(void) ret +=3D idx << 1; } if (reader_flavor & SRCU_READ_FLAVOR_FAST) { + scp =3D srcu_read_lock_fast(srcu_ctlp); + idx =3D __srcu_ptr_to_ctr(srcu_ctlp, scp); + WARN_ON_ONCE(idx & ~0x1); + ret +=3D idx << 2; + } + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) { scp =3D srcu_read_lock_fast(srcu_ctlp); idx =3D __srcu_ptr_to_ctr(srcu_ctlp, scp); WARN_ON_ONCE(idx & ~0x1); @@ -758,8 +766,10 @@ srcu_read_delay(struct torture_random_state *rrsp, str= uct rt_read_seg *rtrsp) static void srcu_torture_read_unlock(int idx) { WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor= && (idx & ~0x1))); - if (reader_flavor & SRCU_READ_FLAVOR_FAST) + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8= ) >> 3)); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) + srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8= ) >> 2)); if (reader_flavor & SRCU_READ_FLAVOR_NMI) srcu_read_unlock_nmisafe(srcu_ctlp, (idx & 0x2) >> 1); if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) || !(reader_flavor & SRCU_R= EAD_FLAVOR_ALL)) @@ -793,7 +803,7 @@ static int srcu_torture_down_read(void) WARN_ON_ONCE(idx & ~0x1); return idx; } - if (reader_flavor & SRCU_READ_FLAVOR_FAST) { + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) { scp =3D srcu_down_read_fast(srcu_ctlp); idx =3D __srcu_ptr_to_ctr(srcu_ctlp, scp); WARN_ON_ONCE(idx & ~0x1); @@ -806,7 +816,7 @@ static int srcu_torture_down_read(void) static void srcu_torture_up_read(int idx) { WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor= && (idx & ~0x1))); - if (reader_flavor & SRCU_READ_FLAVOR_FAST) + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) srcu_up_read_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >>= 3)); else if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) || !(reader_flavor & SRCU_READ_FLAVOR_ALL)) @@ -901,14 +911,16 @@ static struct rcu_torture_ops srcu_ops =3D { .no_pi_lock =3D IS_ENABLED(CONFIG_TINY_SRCU), .debug_objects =3D 1, .have_up_down =3D IS_ENABLED(CONFIG_TINY_SRCU) - ? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST, + ? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST_UPDOWN, .name =3D "srcu" }; =20 static void srcud_torture_init(void) { rcu_sync_torture_init(); - if (reader_flavor & SRCU_READ_FLAVOR_FAST) + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) + WARN_ON(init_srcu_struct_fast(&srcu_ctld)); + else if (reader_flavor & SRCU_READ_FLAVOR_FAST) WARN_ON(init_srcu_struct_fast(&srcu_ctld)); else WARN_ON(init_srcu_struct(&srcu_ctld)); @@ -953,7 +965,7 @@ static struct rcu_torture_ops srcud_ops =3D { .no_pi_lock =3D IS_ENABLED(CONFIG_TINY_SRCU), .debug_objects =3D 1, .have_up_down =3D IS_ENABLED(CONFIG_TINY_SRCU) - ? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST, + ? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST_UPDOWN, .name =3D "srcud" }; =20 --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A9EF34B663; Wed, 5 Nov 2025 20:32:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374741; cv=none; b=nPODPF1WJ3gcMpcA8nT2ghIAVoYRuXC+BoO2wlrNp371PIM9DeN1up1qDwtSRQ6Ba1ImothKHj0ue0lP7OayCWpiaGYvj2GO1pUIS6N79erl1/WCbNvXjvNccUKeAayvwZuyjm1w211xdPfZXWcNQGZxS1p2LNaEk3ENEWhVsWo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374741; c=relaxed/simple; bh=e8020D4IGboPZ/X1LsVXzWAnxHPKUMzBbRHImWEKJYU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=T6iq2q8knb2o/Od3My9B/E2eGhG0hijAZhYTP9FyDve4cE6HlBaONzlEFxNE4ITSaY571Wc8MOts3WPzm7Hq3PfzkQBHmMpShMYflRcIqo4isNc87OpA08AJccQS57XbHZfZtT0bUaKHQRFr1CP1gDTTBOSZ0H0IjiHZTA5VuJQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MqLfDi+e; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MqLfDi+e" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 81675C2BCAF; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374740; bh=e8020D4IGboPZ/X1LsVXzWAnxHPKUMzBbRHImWEKJYU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MqLfDi+e330hGasXldyhEXJIPyW/8d6hgx/5J+cRGnuEkiAlLsWN+EV8gi2d26n2E 8KOTjOKpK9F7cul/tUu1XmALrc7+DHpcLz0LwuZZs+3Q+3D0vp6vtqjD2Ic5PZo21N ZZDxvVJTPMkpNvgNOu3F5WxAqFr56cZUsRIRzsyKzWpLhnarRv8PtgO4/S2g5o/s7O zFhHCQjlokyxnuXitSj7Lu7/I1I86xaS/o1ZVHQXGYcXU5/7qq/4Qscm0rk3Nt+qdt pIoLSfqq4Ygd+hLq17E43KIhcJ4fKxYhs99Io611Qam4nKZW4qOxAa5WoG26HFEDaO 91zNY3muwaP4A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A627FCE0FEA; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH v2 13/16] torture: Permit negative kvm.sh --kconfig numberic arguments Date: Wed, 5 Nov 2025 12:32:13 -0800 Message-Id: <20251105203216.2701005-13-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit loosens the kvm.sh script's regular expressions to permit negative-valued Kconfig options, for example: --kconfig CONFIG_CMDLINE_LOG_WRAP_IDEAL_LEN=3D-1 Signed-off-by: Paul E. McKenney --- tools/testing/selftests/rcutorture/bin/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/rcutorture/bin/kvm.sh b/tools/testing/= selftests/rcutorture/bin/kvm.sh index 617cba339d28..fff15821c44c 100755 --- a/tools/testing/selftests/rcutorture/bin/kvm.sh +++ b/tools/testing/selftests/rcutorture/bin/kvm.sh @@ -199,7 +199,7 @@ do fi ;; --kconfig|--kconfigs) - checkarg --kconfig "(Kconfig options)" $# "$2" '^\(#CHECK#\)\?CONFIG_[A-= Z0-9_]\+=3D\([ynm]\|[0-9]\+\|"[^"]*"\)\( \+\(#CHECK#\)\?CONFIG_[A-Z0-9_]\+= =3D\([ynm]\|[0-9]\+\|"[^"]*"\)\)* *$' '^error$' + checkarg --kconfig "(Kconfig options)" $# "$2" '^\(#CHECK#\)\?CONFIG_[A-= Z0-9_]\+=3D\([ynm]\|-\?[0-9]\+\|"[^"]*"\)\( \+\(#CHECK#\)\?CONFIG_[A-Z0-9_]= \+=3D\([ynm]\|-\?[0-9]\+\|"[^"]*"\)\)* *$' '^error$' TORTURE_KCONFIG_ARG=3D"`echo "$TORTURE_KCONFIG_ARG $2" | sed -e 's/^ *//= ' -e 's/ *$//'`" shift ;; --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46F2534B662; Wed, 5 Nov 2025 20:32:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374741; cv=none; b=AVJ7nZFFv/0a7XFXJhd9p3Vq26B0vdZcE2EowQLNpRoylgJ6UhYY02dMYKUqkCtXrQka2oun04XslJcICe2jtZEpwR39NI6obnzWV66JGmhoex9kGPwDhABRF++UrpxVpW+BB55Z/xkYPUFnPQGT7+AO2HD85DdkaTSDKYxH+70= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374741; c=relaxed/simple; bh=RBKV/fx25yjgLjjokyJhRKWQbPIobFaP4EPJRNJ/XNo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oW0TrBBH6IOW63ENfIfv9SXCrruiTqGSkWnTjq8wtD6HSKLFSuLzSlip/6OF9goJ8CzFEoz7TPbqnJ865mtU2ZGfE3OEEXVzTw+jtIQKYjEqKYyW9j4V3H8uQagMqDe+4hycMzabbFG8iQe07D3ZWene0N/fBJtn33nrvBj11A8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mnlDKyih; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mnlDKyih" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8D440C4CEF5; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374740; bh=RBKV/fx25yjgLjjokyJhRKWQbPIobFaP4EPJRNJ/XNo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mnlDKyih3F51XskOWk5bMPXa5calzArTtSyoxQUAId6xGxS3O6X8bWmK5HBAzlJra mCnwVxAexHDJvBB0OCyAdpFN/ISlW1iUYLNVcgKvVI8cU6NYTgLQbB+BZc53lz1/2b ZAph6HsJFXwohLgkiK1QtlV/JUuc/rvJsFkO+N/dpPmodZxYBda+GN/fB9REShX1Tt BgOSPDtIX/YCdIyUhbObUt0iZ24nVZOFgEnDuXu9LdNb4CvMwHFM1T+cijzGr20Id9 HGdOb5YTe4LkGvvuViVJUMWwY3JuNVqK0MFPIvGoHdZq7DCRI7hShbvmD07AXcGRbP 4pISFTZcokn3g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A8DCACE0FF8; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH v2 14/16] srcu: Create an SRCU-fast-updown API Date: Wed, 5 Nov 2025 12:32:14 -0800 Message-Id: <20251105203216.2701005-14-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates an SRCU-fast-updown API, including DEFINE_SRCU_FAST_UPDOWN(), DEFINE_STATIC_SRCU_FAST_UPDOWN(), __init_srcu_struct_fast_updown(), init_srcu_struct_fast_updown(), srcu_read_lock_fast_updown(), srcu_read_unlock_fast_updown(), __srcu_read_lock_fast_updown(), and __srcu_read_unlock_fast_updown(). These are initially identical to their SRCU-fast counterparts, but both SRCU-fast and SRCU-fast-updown will be optimized in different directions by later commits. SRCU-fast will lack any sort of srcu_down_read() and srcu_up_read() APIs, which will enable extremely efficient NMI safety. For its part, SRCU-fast-updown will not be NMI safe, which will enable reasonably efficient implementations of srcu_down_read_fast() and srcu_up_read_fast(). This commit also adds rcutorture tests for the new APIs. This (annoyingly) needs to be in the same commit for bisectability. With this commit, the 0x8 value tests SRCU-fast-updown. However, most SRCU-fast testing will be via the RCU Tasks Trace wrappers. [ paulmck: Apply s/0x8/0x4/ missing change per Boqun Feng feedback. ] [ paulmck: Apply Akira Yokosawa feedback. ] Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/srcu.h | 77 +++++++++++++++++++++++++++++++++++++--- include/linux/srcutiny.h | 16 +++++++++ include/linux/srcutree.h | 55 ++++++++++++++++++++++++++-- kernel/rcu/rcutorture.c | 12 ++++--- kernel/rcu/srcutree.c | 39 +++++++++++++++++--- 5 files changed, 183 insertions(+), 16 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 1dd6812aabe7..344ad51c8f6c 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -28,6 +28,8 @@ struct srcu_struct; int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct l= ock_class_key *key); #ifndef CONFIG_TINY_SRCU int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, str= uct lock_class_key *key); +int __init_srcu_struct_fast_updown(struct srcu_struct *ssp, const char *na= me, + struct lock_class_key *key); #endif // #ifndef CONFIG_TINY_SRCU =20 #define init_srcu_struct(ssp) \ @@ -44,12 +46,20 @@ int __init_srcu_struct_fast(struct srcu_struct *ssp, co= nst char *name, struct lo __init_srcu_struct_fast((ssp), #ssp, &__srcu_key); \ }) =20 +#define init_srcu_struct_fast_updown(ssp) \ +({ \ + static struct lock_class_key __srcu_key; \ + \ + __init_srcu_struct_fast_updown((ssp), #ssp, &__srcu_key); \ +}) + #define __SRCU_DEP_MAP_INIT(srcu_name) .dep_map =3D { .name =3D #srcu_name= }, #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 int init_srcu_struct(struct srcu_struct *ssp); #ifndef CONFIG_TINY_SRCU int init_srcu_struct_fast(struct srcu_struct *ssp); +int init_srcu_struct_fast_updown(struct srcu_struct *ssp); #endif // #ifndef CONFIG_TINY_SRCU =20 #define __SRCU_DEP_MAP_INIT(srcu_name) @@ -305,6 +315,46 @@ static inline struct srcu_ctr __percpu *srcu_read_lock= _fast(struct srcu_struct * return retval; } =20 +/** + * srcu_read_lock_fast_updown - register a new reader for an SRCU-fast-upd= own structure. + * @ssp: srcu_struct in which to register the new reader. + * + * Enter an SRCU read-side critical section, but for a light-weight + * smp_mb()-free reader. See srcu_read_lock() for more information. + * This function is compatible with srcu_down_read_fast(), but is not + * NMI-safe. + * + * For srcu_read_lock_fast_updown() to be used on an srcu_struct + * structure, that structure must have been defined using either + * DEFINE_SRCU_FAST_UPDOWN() or DEFINE_STATIC_SRCU_FAST_UPDOWN() on the one + * hand or initialized with init_srcu_struct_fast_updown() on the other. + * Such an srcu_struct structure cannot be passed to any non-fast-updown + * variant of srcu_read_{,un}lock() or srcu_{down,up}_read(). In kernels + * built with CONFIG_PROVE_RCU=3Dy, __srcu_check_read_flavor() will compla= in + * bitterly if you ignore this * restriction. + * + * Grace-period auto-expediting is disabled for SRCU-fast-updown + * srcu_struct structures because SRCU-fast-updown expedited grace periods + * invoke synchronize_rcu_expedited(), IPIs and all. If you need expedited + * SRCU-fast-updown grace periods, use synchronize_srcu_expedited(). + * + * The srcu_read_lock_fast_updown() function can be invoked only from + * those contexts where RCU is watching, that is, from contexts where + * it would be legal to invoke rcu_read_lock(). Otherwise, lockdep will + * complain. + */ +static inline struct srcu_ctr __percpu *srcu_read_lock_fast_updown(struct = srcu_struct *ssp) +__acquires(ssp) +{ + struct srcu_ctr __percpu *retval; + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast_updown()."); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN); + retval =3D __srcu_read_lock_fast_updown(ssp); + rcu_try_lock_acquire(&ssp->dep_map); + return retval; +} + /* * Used by tracing, cannot be traced and cannot call lockdep. * See srcu_read_lock_fast() for more information. @@ -335,8 +385,8 @@ static inline struct srcu_ctr __percpu *srcu_down_read_= fast(struct srcu_struct * { WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read= _fast()."); - srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); - return __srcu_read_lock_fast(ssp); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN); + return __srcu_read_lock_fast_updown(ssp); } =20 /** @@ -432,6 +482,23 @@ static inline void srcu_read_unlock_fast(struct srcu_s= truct *ssp, struct srcu_ct RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 +/** + * srcu_read_unlock_fast_updown - unregister a old reader from an SRCU-fas= t-updown structure. + * @ssp: srcu_struct in which to unregister the old reader. + * @scp: return value from corresponding srcu_read_lock_fast_updown(). + * + * Exit an SRCU-fast-updown read-side critical section. + */ +static inline void +srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __pe= rcpu *scp) __releases(ssp) +{ + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN); + srcu_lock_release(&ssp->dep_map); + __srcu_read_unlock_fast_updown(ssp, scp); + RCU_LOCKDEP_WARN(!rcu_is_watching(), + "RCU must be watching srcu_read_unlock_fast_updown()."); +} + /* * Used by tracing, cannot be traced and cannot call lockdep. * See srcu_read_unlock_fast() for more information. @@ -455,9 +522,9 @@ static inline void srcu_up_read_fast(struct srcu_struct= *ssp, struct srcu_ctr __ __releases(ssp) { WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); - srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); - __srcu_read_unlock_fast(ssp, scp); - RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_up_read_f= ast()."); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN); + __srcu_read_unlock_fast_updown(ssp, scp); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_up_read_f= ast_updown()."); } =20 /** diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 1ecc3393fb26..e0698024667a 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -50,13 +50,18 @@ void srcu_drive_gp(struct work_struct *wp); #define DEFINE_SRCU_FAST(name) DEFINE_SRCU(name) #define DEFINE_STATIC_SRCU_FAST(name) \ static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) +#define DEFINE_SRCU_FAST_UPDOWN(name) DEFINE_SRCU(name) +#define DEFINE_STATIC_SRCU_FAST_UPDOWN(name) \ + static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) =20 // Dummy structure for srcu_notifier_head. struct srcu_usage { }; #define __SRCU_USAGE_INIT(name) { } #define __init_srcu_struct_fast __init_srcu_struct +#define __init_srcu_struct_fast_updown __init_srcu_struct #ifndef CONFIG_DEBUG_LOCK_ALLOC #define init_srcu_struct_fast init_srcu_struct +#define init_srcu_struct_fast_updown init_srcu_struct #endif // #ifndef CONFIG_DEBUG_LOCK_ALLOC =20 void synchronize_srcu(struct srcu_struct *ssp); @@ -100,6 +105,17 @@ static inline void __srcu_read_unlock_fast(struct srcu= _struct *ssp, struct srcu_ __srcu_read_unlock(ssp, __srcu_ptr_to_ctr(ssp, scp)); } =20 +static inline struct srcu_ctr __percpu *__srcu_read_lock_fast_updown(struc= t srcu_struct *ssp) +{ + return __srcu_ctr_to_ptr(ssp, __srcu_read_lock(ssp)); +} + +static inline +void __srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_c= tr __percpu *scp) +{ + __srcu_read_unlock(ssp, __srcu_ptr_to_ctr(ssp, scp)); +} + static inline void synchronize_srcu_expedited(struct srcu_struct *ssp) { synchronize_srcu(ssp); diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 6080a9094618..d6f978b50472 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -199,8 +199,15 @@ struct srcu_struct { * * See include/linux/percpu-defs.h for the rules on per-CPU variables. * - * DEFINE_SRCU_FAST() creates an srcu_struct and associated structures - * whose readers must be of the SRCU-fast variety. + * DEFINE_SRCU_FAST() and DEFINE_STATIC_SRCU_FAST create an srcu_struct + * and associated structures whose readers must be of the SRCU-fast variet= y. + * DEFINE_SRCU_FAST_UPDOWN() and DEFINE_STATIC_SRCU_FAST_UPDOWN() create + * an srcu_struct and associated structures whose readers must be of the + * SRCU-fast-updown variety. The key point (aside from error checking) wi= th + * both varieties is that the grace periods must use synchronize_rcu() + * instead of smp_mb(), and given that the first (for example) + * srcu_read_lock_fast() might race with the first synchronize_srcu(), + * this different must be specified at initialization time. */ #ifdef MODULE # define __DEFINE_SRCU(name, fast, is_static) \ @@ -221,6 +228,10 @@ struct srcu_struct { #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, 0, static) #define DEFINE_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST,= /* not static */) #define DEFINE_STATIC_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR= _FAST, static) +#define DEFINE_SRCU_FAST_UPDOWN(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR= _FAST_UPDOWN, \ + /* not static */) +#define DEFINE_STATIC_SRCU_FAST_UPDOWN(name) \ + __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST_UPDOWN, static) =20 int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); @@ -305,6 +316,46 @@ __srcu_read_unlock_fast(struct srcu_struct *ssp, struc= t srcu_ctr __percpu *scp) atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RC= U reader. } =20 +/* + * Counts the new reader in the appropriate per-CPU element of the + * srcu_struct. Returns a pointer that must be passed to the matching + * srcu_read_unlock_fast_updown(). This type of reader is compatible + * with srcu_down_read_fast() and srcu_up_read_fast(). + * + * See the __srcu_read_lock_fast() comment for more details. + */ +static inline +struct srcu_ctr __percpu notrace *__srcu_read_lock_fast_updown(struct srcu= _struct *ssp) +{ + struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); + + if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader. + else + atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); // Y, and implicit RCU = reader. + barrier(); /* Avoid leaking the critical section. */ + return scp; +} + +/* + * Removes the count for the old reader from the appropriate + * per-CPU element of the srcu_struct. Note that this may well be a + * different CPU than that which was incremented by the corresponding + * srcu_read_lock_fast(), but it must be within the same task. + * + * Please see the __srcu_read_lock_fast() function's header comment for + * information on implicit RCU readers and NMI safety. + */ +static inline void notrace +__srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __= percpu *scp) +{ + barrier(); /* Avoid leaking the critical section. */ + if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + this_cpu_inc(scp->srcu_unlocks.counter); // Z, and implicit RCU reader. + else + atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RC= U reader. +} + void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); =20 // Record SRCU-reader usage type only for CONFIG_PROVE_RCU=3Dy kernels. diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 70ee838cab55..a52e21634b28 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -693,6 +693,7 @@ static struct rcu_torture_ops rcu_busted_ops =3D { =20 DEFINE_STATIC_SRCU(srcu_ctl); DEFINE_STATIC_SRCU_FAST(srcu_ctlf); +DEFINE_STATIC_SRCU_FAST_UPDOWN(srcu_ctlfud); static struct srcu_struct srcu_ctld; static struct srcu_struct *srcu_ctlp =3D &srcu_ctl; static struct rcu_torture_ops srcud_ops; @@ -703,7 +704,7 @@ static void srcu_torture_init(void) if (reader_flavor & SRCU_READ_FLAVOR_FAST) srcu_ctlp =3D &srcu_ctlf; if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) - srcu_ctlp =3D &srcu_ctlf; + srcu_ctlp =3D &srcu_ctlfud; } =20 static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) @@ -736,7 +737,7 @@ static int srcu_torture_read_lock(void) ret +=3D idx << 2; } if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) { - scp =3D srcu_read_lock_fast(srcu_ctlp); + scp =3D srcu_read_lock_fast_updown(srcu_ctlp); idx =3D __srcu_ptr_to_ctr(srcu_ctlp, scp); WARN_ON_ONCE(idx & ~0x1); ret +=3D idx << 3; @@ -767,9 +768,10 @@ static void srcu_torture_read_unlock(int idx) { WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor= && (idx & ~0x1))); if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) - srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8= ) >> 3)); + srcu_read_unlock_fast_updown(srcu_ctlp, + __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3)); if (reader_flavor & SRCU_READ_FLAVOR_FAST) - srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8= ) >> 2)); + srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x4= ) >> 2)); if (reader_flavor & SRCU_READ_FLAVOR_NMI) srcu_read_unlock_nmisafe(srcu_ctlp, (idx & 0x2) >> 1); if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) || !(reader_flavor & SRCU_R= EAD_FLAVOR_ALL)) @@ -919,7 +921,7 @@ static void srcud_torture_init(void) { rcu_sync_torture_init(); if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) - WARN_ON(init_srcu_struct_fast(&srcu_ctld)); + WARN_ON(init_srcu_struct_fast_updown(&srcu_ctld)); else if (reader_flavor & SRCU_READ_FLAVOR_FAST) WARN_ON(init_srcu_struct_fast(&srcu_ctld)); else diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 2f8aa280911e..ea3f128de06f 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -309,13 +309,24 @@ int __init_srcu_struct_fast(struct srcu_struct *ssp, = const char *name, struct lo } EXPORT_SYMBOL_GPL(__init_srcu_struct_fast); =20 +int __init_srcu_struct_fast_updown(struct srcu_struct *ssp, const char *na= me, + struct lock_class_key *key) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST_UPDOWN; + return __init_srcu_struct_common(ssp, name, key); +} +EXPORT_SYMBOL_GPL(__init_srcu_struct_fast_updown); + #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /** * init_srcu_struct - initialize a sleep-RCU structure * @ssp: structure to initialize. * - * Must invoke this on a given srcu_struct before passing that srcu_struct + * Use this in place of DEFINE_SRCU() and DEFINE_STATIC_SRCU() + * for non-static srcu_struct structures that are to be passed to + * srcu_read_lock(), srcu_read_lock_nmisafe(), and friends. It is necessa= ry + * to invoke this on a given srcu_struct before passing that srcu_struct * to any other function. Each srcu_struct represents a separate domain * of SRCU protection. */ @@ -330,9 +341,11 @@ EXPORT_SYMBOL_GPL(init_srcu_struct); * init_srcu_struct_fast - initialize a fast-reader sleep-RCU structure * @ssp: structure to initialize. * - * Must invoke this on a given srcu_struct before passing that srcu_struct - * to any other function. Each srcu_struct represents a separate domain - * of SRCU protection. + * Use this in place of DEFINE_SRCU_FAST() and DEFINE_STATIC_SRCU_FAST() + * for non-static srcu_struct structures that are to be passed to + * srcu_read_lock_fast() and friends. It is necessary to invoke this on a + * given srcu_struct before passing that srcu_struct to any other function. + * Each srcu_struct represents a separate domain of SRCU protection. */ int init_srcu_struct_fast(struct srcu_struct *ssp) { @@ -341,6 +354,24 @@ int init_srcu_struct_fast(struct srcu_struct *ssp) } EXPORT_SYMBOL_GPL(init_srcu_struct_fast); =20 +/** + * init_srcu_struct_fast_updown - initialize a fast-reader up/down sleep-R= CU structure + * @ssp: structure to initialize. + * + * Use this function in place of DEFINE_SRCU_FAST_UPDOWN() and + * DEFINE_STATIC_SRCU_FAST_UPDOWN() for non-static srcu_struct + * structures that are to be passed to srcu_read_lock_fast_updown(), + * srcu_down_read_fast(), and friends. It is necessary to invoke this on a + * given srcu_struct before passing that srcu_struct to any other function. + * Each srcu_struct represents a separate domain of SRCU protection. + */ +int init_srcu_struct_fast_updown(struct srcu_struct *ssp) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST_UPDOWN; + return init_srcu_struct_fields(ssp, false); +} +EXPORT_SYMBOL_GPL(init_srcu_struct_fast_updown); + #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /* --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 083EA34B420; Wed, 5 Nov 2025 20:32:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374741; cv=none; b=dHUJNPZ59I1VAXkZnrwAE3YTz89Dv+3JDJsly8zkXTh4beHu+Ob9tinPmvbEkQo/Je+M7/iHsZITAKOkOLEjl6s4XfWD72zm0EPhBjApUTs0v/De7La3d/mylUURDl3z3XFV7jUAaQDLIcVoMYdxagmGFEPIeMrY8KEkvTpgEho= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374741; c=relaxed/simple; bh=3SQyEsKa4vegmpwnk5NwDoYKeDgsfEpb4R+BPinWNcE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZRRMqxjkF6dV3moqpj/fn5u5irXz4G/aKsFuCjsnxF3U6c9EvI9Tvcyupr0cweMi+eWH5zDNS/IKDFNBi/CyFbJgMSnhpggtAHlWJKCCzBfjRw8m+9xTV1DNXBrUkSLflABypfrnbmwN0fFr0EotH75yAidII4P5we+tbwbZRa4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Vhydh/tQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Vhydh/tQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 959E7C116B1; Wed, 5 Nov 2025 20:32:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374740; bh=3SQyEsKa4vegmpwnk5NwDoYKeDgsfEpb4R+BPinWNcE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Vhydh/tQI9xr/YlHbbUKT9zHxmVTJ6t+ba3rjA8qgpeoIQrB56U/+bcCd2s6qrqcY 7smrOg/+KvBe9txgBp5nzq9uiPHthKUSp0Y1FkseYDmMJOaxtXvDhS/LIjP/Dq48Zs xQvCyBF+NMHrht2k7PfBH6lKyKYti6Mmw+z35Vj6jfcB3/fLg4xSxULoEOThhIROGP I8QQDYUlBgyVOLdMvCkVsZEx8TjNNee1O+JZEu0ff56JeVlfzMVKOIlXY+RP1Buuqp NMYt36OPnMYbB4RhUS49kGrl7D1D8q+lG8vmgaGqdKSUIHMYrafKfB3XDy8sMNbpSb nzlTTu9qmXFsw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id AB7C6CE0FF9; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Catalin Marinas , Will Deacon , Mark Rutland , Mathieu Desnoyers , Sebastian Andrzej Siewior , linux-arm-kernel@lists.infradead.org, bpf@vger.kernel.org Subject: [PATCH v2 15/16] srcu: Optimize SRCU-fast-updown for arm64 Date: Wed, 5 Nov 2025 12:32:15 -0800 Message-Id: <20251105203216.2701005-15-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some arm64 platforms have slow per-CPU atomic operations, for example, the Neoverse V2. This commit therefore moves SRCU-fast from per-CPU atomic operations to interrupt-disabled non-read-modify-write-atomic atomic_read()/atomic_set() operations. This works because SRCU-fast-updown is not invoked from read-side primitives, which means that if srcu_read_unlock_fast() NMI handlers. This means that srcu_read_lock_fast_updown() and srcu_read_unlock_fast_updown() can exclude themselves and each other This reduces the overhead of calls to srcu_read_lock_fast_updown() and srcu_read_unlock_fast_updown() from about 100ns to about 12ns on an ARM Neoverse V2. Although this is not excellent compared to about 2ns on x86, it sure beats 100ns. This command was used to measure the overhead: tools/testing/selftests/rcutorture/bin/kvm.sh --torture refscale --allcpus = --duration 5 --configs NOPREEMPT --kconfig "CONFIG_NR_CPUS=3D64 CONFIG_TASK= S_TRACE_RCU=3Dy" --bootargs "refscale.loops=3D100000 refscale.guest_os_dela= y=3D5 refscale.nreaders=3D64 refscale.holdoff=3D30 torture.disable_onoff_at= _boot refscale.scale_type=3Dsrcu-fast-updown refscale.verbose_batched=3D8 t= orture.verbose_sleep_frequency=3D8 torture.verbose_sleep_duration=3D8 refsc= ale.nruns=3D100" --trust-make Signed-off-by: Paul E. McKenney Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: Cc: --- include/linux/srcutree.h | 51 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 48 insertions(+), 3 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index d6f978b50472..0e06f87e1d7c 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -253,6 +253,34 @@ static inline struct srcu_ctr __percpu *__srcu_ctr_to_= ptr(struct srcu_struct *ss return &ssp->sda->srcu_ctrs[idx]; } =20 +/* + * Non-atomic manipulation of SRCU lock counters. + */ +static inline struct srcu_ctr __percpu notrace *__srcu_read_lock_fast_na(s= truct srcu_struct *ssp) +{ + atomic_long_t *scnp; + struct srcu_ctr __percpu *scp; + + lockdep_assert_preemption_disabled(); + scp =3D READ_ONCE(ssp->srcu_ctrp); + scnp =3D raw_cpu_ptr(&scp->srcu_locks); + atomic_long_set(scnp, atomic_long_read(scnp) + 1); + return scp; +} + +/* + * Non-atomic manipulation of SRCU unlock counters. + */ +static inline void notrace +__srcu_read_unlock_fast_na(struct srcu_struct *ssp, struct srcu_ctr __perc= pu *scp) +{ + atomic_long_t *scnp; + + lockdep_assert_preemption_disabled(); + scnp =3D raw_cpu_ptr(&scp->srcu_unlocks); + atomic_long_set(scnp, atomic_long_read(scnp) + 1); +} + /* * Counts the new reader in the appropriate per-CPU element of the * srcu_struct. Returns a pointer that must be passed to the matching @@ -327,8 +355,18 @@ __srcu_read_unlock_fast(struct srcu_struct *ssp, struc= t srcu_ctr __percpu *scp) static inline struct srcu_ctr __percpu notrace *__srcu_read_lock_fast_updown(struct srcu= _struct *ssp) { - struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); + struct srcu_ctr __percpu *scp; =20 + if (IS_ENABLED(CONFIG_ARM64) && IS_ENABLED(CONFIG_ARM64_USE_LSE_PERCPU_AT= OMICS)) { + unsigned long flags; + + local_irq_save(flags); + scp =3D __srcu_read_lock_fast_na(ssp); + local_irq_restore(flags); /* Avoids leaking the critical section. */ + return scp; + } + + scp =3D READ_ONCE(ssp->srcu_ctrp); if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader. else @@ -350,10 +388,17 @@ static inline void notrace __srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __= percpu *scp) { barrier(); /* Avoid leaking the critical section. */ - if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + if (IS_ENABLED(CONFIG_ARM64)) { + unsigned long flags; + + local_irq_save(flags); + __srcu_read_unlock_fast_na(ssp, scp); + local_irq_restore(flags); + } else if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) { this_cpu_inc(scp->srcu_unlocks.counter); // Z, and implicit RCU reader. - else + } else { atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RC= U reader. + } } =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); --=20 2.40.1 From nobody Fri Dec 19 14:24:09 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC39334CFDC; Wed, 5 Nov 2025 20:32:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374742; cv=none; b=IEO4XDWsd8acGIPB41SGm9V7x9KzQaiafpF9gK5Zf97w6jdh+L03t8LvLBo/5oLsd/Bt7CwTjxnuBzFO6n33wNUY+vUneXLZy4cinI3SYuZqxTSl2/PB7wHNtLH4lj0VAMq1BtP0hBqfX7w+G3OQKjmZslayXb1D20XgSZxs5mg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762374742; c=relaxed/simple; bh=aJyIfG6g9+3VDV5coLmO7dbxbwXJBSKpYVjzcCHpIds=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fzLEoFOMbecjSm+eV9diXkuattIGxdb+32UkcI9dhjUXKEpORUZE+HdZA+25HYb+Hdy9kfPSAng/aMZBUWAVOw0PKm0RKNz+HuzWfCOQDKkobox/c6a9fX26ZDduOVjWU3RkxAvukHQ3o28utBD92SFMgMLpkm6AxsdI1ZZE2B4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=G2bZSqg6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="G2bZSqg6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 70EC2C19421; Wed, 5 Nov 2025 20:32:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762374741; bh=aJyIfG6g9+3VDV5coLmO7dbxbwXJBSKpYVjzcCHpIds=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G2bZSqg6l9PPM5qYpntafEwkVG4YOoE+G410qYshznWVMhYd2xKmxOqDJ3IvVF5nd bno1xlTnI1yGqkKdeV+M++pHH9/KrALffQNRjmXFGpkG2UNFvIMU+QVi1YScTuMvxy fj1Egd8MJq8GY8NzAp65ByzFF8ADdy9R7kjFbwTmi7aZ3L1w7Ovu/nW18GHuFppaSN 7xc4WDcLuOZs7GET0LsfAnwPaOeHlkrtBt9RSw4qgNTMTekjb2S3pUWlhJRu0VI/vC rgPOGBM6CkgflQe5xn4x1syEFgqhm5eAkY/dPmI2j030zMyJH7CuD1yOV7eVPZj7Gp dr30vTLRW5MIA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id ADB1BCE1063; Wed, 5 Nov 2025 12:32:18 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH v2 16/16] rcutorture: Make srcu{,d}_torture_init() announce the SRCU type Date: Wed, 5 Nov 2025 12:32:16 -0800 Message-Id: <20251105203216.2701005-16-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit causes rcutorture's srcu_torture_init() and srcud_torture_init() functions to announce on the console log which variant of SRCU is being tortured, for example: "torture: srcud_torture_init fast SRCU". [ paulmck: Apply feedback from kernel test robot. ] Signed-off-by: Paul E. McKenney --- kernel/rcu/rcutorture.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index a52e21634b28..11906c927321 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -701,10 +701,18 @@ static struct rcu_torture_ops srcud_ops; static void srcu_torture_init(void) { rcu_sync_torture_init(); - if (reader_flavor & SRCU_READ_FLAVOR_FAST) + if (!reader_flavor || (reader_flavor & SRCU_READ_FLAVOR_NORMAL)) + VERBOSE_TOROUT_STRING("srcu_torture_init normal SRCU"); + if (reader_flavor & SRCU_READ_FLAVOR_NMI) + VERBOSE_TOROUT_STRING("srcu_torture_init NMI-safe SRCU"); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) { srcu_ctlp =3D &srcu_ctlf; - if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) + VERBOSE_TOROUT_STRING("srcu_torture_init fast SRCU"); + } + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) { srcu_ctlp =3D &srcu_ctlfud; + VERBOSE_TOROUT_STRING("srcu_torture_init fast-up/down SRCU"); + } } =20 static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) @@ -920,12 +928,21 @@ static struct rcu_torture_ops srcu_ops =3D { static void srcud_torture_init(void) { rcu_sync_torture_init(); - if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) - WARN_ON(init_srcu_struct_fast_updown(&srcu_ctld)); - else if (reader_flavor & SRCU_READ_FLAVOR_FAST) + if (!reader_flavor || (reader_flavor & SRCU_READ_FLAVOR_NORMAL)) { + WARN_ON(init_srcu_struct(&srcu_ctld)); + VERBOSE_TOROUT_STRING("srcud_torture_init normal SRCU"); + } else if (reader_flavor & SRCU_READ_FLAVOR_NMI) { + WARN_ON(init_srcu_struct(&srcu_ctld)); + VERBOSE_TOROUT_STRING("srcud_torture_init NMI-safe SRCU"); + } else if (reader_flavor & SRCU_READ_FLAVOR_FAST) { WARN_ON(init_srcu_struct_fast(&srcu_ctld)); - else + VERBOSE_TOROUT_STRING("srcud_torture_init fast SRCU"); + } else if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) { + WARN_ON(init_srcu_struct_fast_updown(&srcu_ctld)); + VERBOSE_TOROUT_STRING("srcud_torture_init fast-up/down SRCU"); + } else { WARN_ON(init_srcu_struct(&srcu_ctld)); + } srcu_ctlp =3D &srcu_ctld; } =20 --=20 2.40.1