From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51AFE1F7580; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119878; cv=none; b=Ro3GM1ceuZyTvOYDmVMGKqDHz4hxH8E1elRTnuRpsZFWrV4SbmqC06dAHUlRr9x+2MO9zkLJdKhwmfupBXqhY/aRaEKWvDWOJ0hcNR2pBEoCb2OVcHO4VTiiixRUj7VZX4QeWciWxcvZuPG0Pap4ayQy32jkLX0MIZZAvu0ibTc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119878; c=relaxed/simple; bh=ZFNuHVX2di4q2gqCJGVL5/EBAk3Kdb3wjGuQ316eoLk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VglbR57XrUDFxka6XnodcI/LS/sdot+qzPvZLtFrB14l6xdSo3nPzqvbef/sFRPO9GgFV15w2Kf80jONakhbifwTPXPVXHMcB2U1XRP4Bd8TnyI8Y5IAIAS+2WW1SqllKCpibh0KeXJpahy/oSJ5Mlit0Go1Zkv6X2qnjQurJSU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=k5l8mNeB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="k5l8mNeB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DCD38C4CEF7; Sun, 2 Nov 2025 21:44:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119877; bh=ZFNuHVX2di4q2gqCJGVL5/EBAk3Kdb3wjGuQ316eoLk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=k5l8mNeB8i76/k6rRYPk85GRs3k5wyWNC5EF40Lq2BSQaLaROn69Vn9/Dk5FdJrX1 BdArVSYGgKwzvw7tJc7DGBt9MhIf6mtxKdzWDOhkkotLRcI1EzySt1Q6M+asABprf6 Bbtd5AxHH0HzcKJtHdB/okNInxXBF9Voz8w0r7o9HQHp00AQ1aXVZjn7UZxn0LwrGY GBIGI2Zo2t7QIcXMQ1QWlQEGscnxyh9KZMBvtLtriBzVH+SoRxFTCH4eBoswTTacth B8eH6xvGfRAomw9AwIxMk821+y4es6vfFES/SyG+rLizWKnPVBqS0wHk392seGEwiu R3icKwEXve95w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 78BE0CE0F4C; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , kernel test robot , Zqiang Subject: [PATCH 01/19] srcu: Permit Tiny SRCU srcu_read_unlock() with interrupts disabled Date: Sun, 2 Nov 2025 13:44:18 -0800 Message-Id: <20251102214436.3905633-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current Tiny SRCU implementation of srcu_read_unlock() awakens the grace-period processing when exiting the outermost SRCU read-side critical section. However, not all Linux-kernel configurations and contexts permit swake_up_one() to be invoked while interrupts are disabled, and this can result in indefinitely extended SRCU grace periods. This commit therefore only invokes swake_up_one() when interrupts are enabled, and introduces polling to the grace-period workqueue handler. Reported-by: kernel test robot Reported-by: Zqiang Closes: https://lore.kernel.org/oe-lkp/202508261642.b15eefbb-lkp@intel.com Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutiny.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c index e3b64a5e0ec7..3450c3751ef7 100644 --- a/kernel/rcu/srcutiny.c +++ b/kernel/rcu/srcutiny.c @@ -106,15 +106,15 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int = idx) newval =3D READ_ONCE(ssp->srcu_lock_nesting[idx]) - 1; WRITE_ONCE(ssp->srcu_lock_nesting[idx], newval); preempt_enable(); - if (!newval && READ_ONCE(ssp->srcu_gp_waiting) && in_task()) + if (!newval && READ_ONCE(ssp->srcu_gp_waiting) && in_task() && !irqs_disa= bled()) swake_up_one(&ssp->srcu_wq); } EXPORT_SYMBOL_GPL(__srcu_read_unlock); =20 /* * Workqueue handler to drive one grace period and invoke any callbacks - * that become ready as a result. Single-CPU and !PREEMPTION operation - * means that we get away with murder on synchronization. ;-) + * that become ready as a result. Single-CPU operation and preemption + * disabling mean that we get away with murder on synchronization. ;-) */ void srcu_drive_gp(struct work_struct *wp) { @@ -141,7 +141,12 @@ void srcu_drive_gp(struct work_struct *wp) WRITE_ONCE(ssp->srcu_idx, ssp->srcu_idx + 1); WRITE_ONCE(ssp->srcu_gp_waiting, true); /* srcu_read_unlock() wakes! */ preempt_enable(); - swait_event_exclusive(ssp->srcu_wq, !READ_ONCE(ssp->srcu_lock_nesting[idx= ])); + do { + // Deadlock issues prevent __srcu_read_unlock() from + // doing an unconditional wakeup, so polling is required. + swait_event_timeout_exclusive(ssp->srcu_wq, + !READ_ONCE(ssp->srcu_lock_nesting[idx]), HZ / 10); + } while (READ_ONCE(ssp->srcu_lock_nesting[idx])); preempt_disable(); // Needed for PREEMPT_LAZY WRITE_ONCE(ssp->srcu_gp_waiting, false); /* srcu_read_unlock() cheap. */ WRITE_ONCE(ssp->srcu_idx, ssp->srcu_idx + 1); --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77484253B5C; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; cv=none; b=F++xHgP+ZVncwpbjvHtMX/Z64jElmquaJgyiwtgBJqtqBQtdfCJts5P+Cx6XfkL7k2O5ST81QU/M7IwIhH319BYbCfTC6g47HgSLXXuO/Exeb25nz0ZAfAqhy6TSRlcz9P54Z2V53pvOvhmtKQDWooqMicoQPr1xDkR1gd0NHb4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; c=relaxed/simple; bh=nCGLsD7q2HOJPiSlnIStGPVemKa7l7hTEsNDv1pJevc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=T6/KDLwGk6BAQjtvK73Kif/PNT6LHsXEzbVq/Nuv9YqhFZXabT5oaUbgpRhD1YqMNYTj+cXgdXUqMwZBRmMm9H1QfPa2ikH/stnO2/mbhTjW/9ojckU79GzqNmc4Muplr7b0n0S8RqHSqi2+6Tf57mGVIu533EkEt4mgjJpM/Jo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Y59alB+C; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Y59alB+C" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 050D9C19421; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119879; bh=nCGLsD7q2HOJPiSlnIStGPVemKa7l7hTEsNDv1pJevc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y59alB+CHMcEw0H2sjInl49oEyk3z0Ql/twC/hBSNaLA4pedhttOz/dPuiwUAkXpg C3OnLaa9OZA+B9CjBl7H5M4FN7CHJFEY+OZN6QULo4MH5yXBJluZoNz6DMuvRkH3RB 2PUV5OW/MTnXEEXrIKNjGwEI3zY1CdLzmtJblSeKsw4bGs2q240URRZO7uHI6UPYsj T/DUuPWVYGE8mH+rISnFt65i78JSbUq21or5iP3cGexS4AqXlBeNqHGdxLUbjAoC+t LwL7Ib/gZlVij6UYfulGK0BTY6+bVC4X58oqiQ7zb5U+Tfx21QDd8pKBNgGLQALKnS xfxW94zd4fvnQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7BD65CE0F65; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 02/19] srcu: Create an srcu_expedite_current() function Date: Sun, 2 Nov 2025 13:44:19 -0800 Message-Id: <20251102214436.3905633-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates an srcu_expedite_current() function that expedites the current (and possibly the next) SRCU grace period for the specified srcu_struct structure. This functionality will be inherited by RCU Tasks Trace courtesy of its mapping to SRCU fast. If the current SRCU grace period is already waiting, that wait will complete before the expediting takes effect. If there is no SRCU grace period in flight, this function might well create one. [ paulmck: Apply Zqiang feedback for PREEMPT_RT use. ] Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/srcutiny.h | 1 + include/linux/srcutree.h | 8 ++++++ kernel/rcu/srcutree.c | 58 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 51ce25f07930..3bfbd44cb1b3 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -103,6 +103,7 @@ static inline void srcu_barrier(struct srcu_struct *ssp) synchronize_srcu(ssp); } =20 +static inline void srcu_expedite_current(struct srcu_struct *ssp) { } #define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) #define srcu_check_read_flavor_force(ssp, read_flavor) do { } while (0) =20 diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 42098e0fa0b7..93ad18acd6d0 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -42,6 +42,8 @@ struct srcu_data { struct timer_list delay_work; /* Delay for CB invoking */ struct work_struct work; /* Context for CB invoking. */ struct rcu_head srcu_barrier_head; /* For srcu_barrier() use. */ + struct rcu_head srcu_ec_head; /* For srcu_expedite_current() use. */ + int srcu_ec_state; /* State for srcu_expedite_current(). */ struct srcu_node *mynode; /* Leaf srcu_node. */ unsigned long grpmask; /* Mask for leaf srcu_node */ /* ->srcu_data_have_cbs[]. */ @@ -135,6 +137,11 @@ struct srcu_struct { #define SRCU_STATE_SCAN1 1 #define SRCU_STATE_SCAN2 2 =20 +/* Values for srcu_expedite_current() state (->srcu_ec_state). */ +#define SRCU_EC_IDLE 0 +#define SRCU_EC_PENDING 1 +#define SRCU_EC_REPOST 2 + /* * Values for initializing gp sequence fields. Higher values allow wrap ar= ounds to * occur earlier. @@ -210,6 +217,7 @@ struct srcu_struct { int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); +void srcu_expedite_current(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); =20 // Converts a per-CPU pointer to an ->srcu_ctrs[] array element to that diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 1ff94b76d91f..38b440b0b0c8 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1688,6 +1688,64 @@ void srcu_barrier(struct srcu_struct *ssp) } EXPORT_SYMBOL_GPL(srcu_barrier); =20 +/* Callback for srcu_expedite_current() usage. */ +static void srcu_expedite_current_cb(struct rcu_head *rhp) +{ + unsigned long flags; + bool needcb =3D false; + struct srcu_data *sdp =3D container_of(rhp, struct srcu_data, srcu_ec_hea= d); + + spin_lock_irqsave_sdp_contention(sdp, &flags); + if (sdp->srcu_ec_state =3D=3D SRCU_EC_IDLE) { + WARN_ON_ONCE(1); + } else if (sdp->srcu_ec_state =3D=3D SRCU_EC_PENDING) { + sdp->srcu_ec_state =3D SRCU_EC_IDLE; + } else { + WARN_ON_ONCE(sdp->srcu_ec_state !=3D SRCU_EC_REPOST); + sdp->srcu_ec_state =3D SRCU_EC_PENDING; + needcb =3D true; + } + spin_unlock_irqrestore_rcu_node(sdp, flags); + // If needed, requeue ourselves as an expedited SRCU callback. + if (needcb) + __call_srcu(sdp->ssp, &sdp->srcu_ec_head, srcu_expedite_current_cb, fals= e); +} + +/** + * srcu_expedite_current - Expedite the current SRCU grace period + * @ssp: srcu_struct to expedite. + * + * Cause the current SRCU grace period to become expedited. The grace + * period following the current one might also be expedited. If there is + * no current grace period, one might be created. If the current grace + * period is currently sleeping, that sleep will complete before expediting + * will take effect. + */ +void srcu_expedite_current(struct srcu_struct *ssp) +{ + unsigned long flags; + bool needcb =3D false; + struct srcu_data *sdp; + + migrate_disable(); + sdp =3D this_cpu_ptr(ssp->sda); + spin_lock_irqsave_sdp_contention(sdp, &flags); + if (sdp->srcu_ec_state =3D=3D SRCU_EC_IDLE) { + sdp->srcu_ec_state =3D SRCU_EC_PENDING; + needcb =3D true; + } else if (sdp->srcu_ec_state =3D=3D SRCU_EC_PENDING) { + sdp->srcu_ec_state =3D SRCU_EC_REPOST; + } else { + WARN_ON_ONCE(sdp->srcu_ec_state !=3D SRCU_EC_REPOST); + } + spin_unlock_irqrestore_rcu_node(sdp, flags); + // If needed, queue an expedited SRCU callback. + if (needcb) + __call_srcu(ssp, &sdp->srcu_ec_head, srcu_expedite_current_cb, false); + migrate_enable(); +} +EXPORT_SYMBOL_GPL(srcu_expedite_current); + /** * srcu_batches_completed - return batches completed. * @ssp: srcu_struct on which to report batch completion. --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C0A1225A34; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119878; cv=none; b=shkL1Tq/OCLgFTqfqjmPGN26v7fgDiPePAWE0gAbHDIQAmkXgO5uFMhrgYRJRK9tZj80xZb8gOwUi1cntfEHvDL3CCNqXHtFtPF2r0S4I9AfV/cvdXkMB1VnaUAdeMem2jw7gsQedzOdWtaSaL/Vay14ogUsWg0Jx2kqkgFmyNE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119878; c=relaxed/simple; bh=hiuAkXJp5N4WN5xE4QgqmeyHlsM9HdyJ3JawANTRqe0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RJkQwHXuM+wbTWgXeeZ/B9lJLbqiFsUlxp/DTJ2kA/iZGwm4yXP2JNX2I2FR+xyyzASB3UQMmP8PhVhTV1X9KsCoSrnNX8BECw7AGUnUSWDdQJ0vnk9bq6Jd5kVcvqZy2RFLT0qepxeFiakvCy12P5h1hh57/9L+L3GNBhiW0bs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hqT2BBNn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hqT2BBNn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 09921C116D0; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119878; bh=hiuAkXJp5N4WN5xE4QgqmeyHlsM9HdyJ3JawANTRqe0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hqT2BBNnZtQaTEXe1+q17zK9ihLD/gCJayiiANA5mS06b+dP/ameTRlDG/OJFBqZS rG4huuSiqvu5BF2GpaKYTghJ3uFJFAGbS7z2yQMsRUUm74bMG64czrt9KwAGebq/jm 6kttfwvp0m6OjEyTWtV3zixS5ZMxlCS5vDQ8JlE+vqxA907o4HTmn2Qtsx454gT9rW Vp71oRKXHzUeEqKI3sXbAZv/ELfRr9f+gsvA03B2cj4SdUGD1KS8QT4VJBhKpcRCOq +J24BFQO5abwIsIKNnYR0Br5ZaCI5Q6flc5nNaaciafZ20tSR8FULDl0IztNP72hBG h0+GX21ORfP9g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7EA35CE0F8F; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 03/19] rcutorture: Test srcu_expedite_current() Date: Sun, 2 Nov 2025 13:44:20 -0800 Message-Id: <20251102214436.3905633-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds a ->exp_current member to the rcu_torture_ops structure to test the srcu_expedite_current() function. Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- kernel/rcu/rcutorture.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 72619e5e8549..aa1f8240a276 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -389,6 +389,7 @@ struct rcu_torture_ops { void (*deferred_free)(struct rcu_torture *p); void (*sync)(void); void (*exp_sync)(void); + void (*exp_current)(void); unsigned long (*get_gp_state_exp)(void); unsigned long (*start_gp_poll_exp)(void); void (*start_gp_poll_exp_full)(struct rcu_gp_oldstate *rgosp); @@ -857,6 +858,11 @@ static void srcu_torture_synchronize_expedited(void) synchronize_srcu_expedited(srcu_ctlp); } =20 +static void srcu_torture_expedite_current(void) +{ + srcu_expedite_current(srcu_ctlp); +} + static struct rcu_torture_ops srcu_ops =3D { .ttype =3D SRCU_FLAVOR, .init =3D rcu_sync_torture_init, @@ -871,6 +877,7 @@ static struct rcu_torture_ops srcu_ops =3D { .deferred_free =3D srcu_torture_deferred_free, .sync =3D srcu_torture_synchronize, .exp_sync =3D srcu_torture_synchronize_expedited, + .exp_current =3D srcu_torture_expedite_current, .same_gp_state =3D same_state_synchronize_srcu, .get_comp_state =3D get_completed_synchronize_srcu, .get_gp_state =3D srcu_torture_get_gp_state, @@ -919,6 +926,7 @@ static struct rcu_torture_ops srcud_ops =3D { .deferred_free =3D srcu_torture_deferred_free, .sync =3D srcu_torture_synchronize, .exp_sync =3D srcu_torture_synchronize_expedited, + .exp_current =3D srcu_torture_expedite_current, .same_gp_state =3D same_state_synchronize_srcu, .get_comp_state =3D get_completed_synchronize_srcu, .get_gp_state =3D srcu_torture_get_gp_state, @@ -1700,6 +1708,8 @@ rcu_torture_writer(void *arg) ulo[i] =3D cur_ops->get_comp_state(); gp_snap =3D cur_ops->start_gp_poll(); rcu_torture_writer_state =3D RTWS_POLL_WAIT; + if (cur_ops->exp_current && !torture_random(&rand) % 0xff) + cur_ops->exp_current(); while (!cur_ops->poll_gp_state(gp_snap)) { gp_snap1 =3D cur_ops->get_gp_state(); for (i =3D 0; i < ulo_size; i++) @@ -1720,6 +1730,8 @@ rcu_torture_writer(void *arg) cur_ops->get_comp_state_full(&rgo[i]); cur_ops->start_gp_poll_full(&gp_snap_full); rcu_torture_writer_state =3D RTWS_POLL_WAIT_FULL; + if (cur_ops->exp_current && !torture_random(&rand) % 0xff) + cur_ops->exp_current(); while (!cur_ops->poll_gp_state_full(&gp_snap_full)) { cur_ops->get_gp_state_full(&gp_snap1_full); for (i =3D 0; i < rgo_size; i++) --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C103226D00; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119878; cv=none; b=h6L0yQlggCuTgif3oGNe/lFPWK1KrDmt+IbjEyLqtQtsM2H2UMOjt9mmYm8TNqUPTgC+GTu4KispaE3TKDuXhM92CQPQPEUw3ftQsLOxX0l1hhX47vZyc0BD+93pMgDHERXuuQbbKajpFU/VDgsbqPLyyTSnYljx9xqzo9JU6LU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119878; c=relaxed/simple; bh=40G+Sy5PfqcQaYHAZAnhVUifnZEIaEN9k49sIAuQdQs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dtsNmevQWIhl4sl3sbMZwqTdhw8oF/gDZrFEwgPCYdnz1Rbl/p6BupWSmPgt7wY7uFMu5Dj78vcScNrRyaEmS0mQGth8Tb8tQJb2HhTG0XldUlDy9vJ/TF7HtK9MICzs2YB+R1girUen8GebpXZQGVBvez+hJvnwul2d9H4mRGo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WntKR6ZG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WntKR6ZG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 06C60C116C6; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119878; bh=40G+Sy5PfqcQaYHAZAnhVUifnZEIaEN9k49sIAuQdQs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WntKR6ZGxq5MnWOVl40JNOS7Ywnydtz4ld6sqKZhBGe1WNDyF7Qsaf17w+2kvbwmE DwsZYI+AyZJfYnOe7JoCHEsWTyG7r07sYGtK+6ga9I8KR7y7cQx7XKO1lbFaGWhKpW JVY4BE1yU9SirccJh43M6p5JWFWDawOMpHkKu+lONBBwPsNweElGCY1YHQ+xdw+XNd 9WCUx4RpyWW2VYcN41J91ZGLqAHtX6gFRpQbHAbnCGXvhXd1cojTX7wflQdVEHhaTQ nZYbyplF/c1ePXkRGjDha0dwTBm5dItWnalh1uMCrxt5qK29mNrurX3E5AxkDagjk3 mctoquJuHKWrw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 81846CE0FF8; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 04/19] srcu: Create a DEFINE_SRCU_FAST() Date: Sun, 2 Nov 2025 13:44:21 -0800 Message-Id: <20251102214436.3905633-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates DEFINE_SRCU_FAST() and DEFINE_STATIC_SRCU_FAST() macros that are similar to DEFINE_SRCU() and DEFINE_STATIC_SRCU(), but which create srcu_struct structures that are usable only by readers initiated by srcu_read_lock_fast() and friends. This commit does make DEFINE_SRCU_FAST() available to modules, in which case the per-CPU srcu_data structures are not created at compile time, but rather at module-load time. This means that the >srcu_reader_flavor field of the srcu_data structure is not available. Therefore, this commit instead creates an ->srcu_reader_flavor field in the srcu_struct structure, adds arguments to the DEFINE_SRCU()-related macros to initialize this new field, and extends the checks in the __srcu_check_read_flavor() function to include this new field. This commit also allows dynamically allocated srcu_struct structure to be marked for SRCU-fast readers. It does so by defining a new init_srcu_struct_fast() function that marks the specified srcu_struct structure for use by srcu_read_lock_fast() and friends. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/notifier.h | 2 +- include/linux/srcu.h | 16 ++++++++++++++-- include/linux/srcutiny.h | 13 ++++++++++--- include/linux/srcutree.h | 30 +++++++++++++++++++----------- kernel/rcu/srcutree.c | 36 ++++++++++++++++++++++++++++++++++-- 5 files changed, 78 insertions(+), 19 deletions(-) diff --git a/include/linux/notifier.h b/include/linux/notifier.h index b42e64734968..01b6c9d9956f 100644 --- a/include/linux/notifier.h +++ b/include/linux/notifier.h @@ -109,7 +109,7 @@ extern void srcu_init_notifier_head(struct srcu_notifie= r_head *nh); .mutex =3D __MUTEX_INITIALIZER(name.mutex), \ .head =3D NULL, \ .srcuu =3D __SRCU_USAGE_INIT(name.srcuu), \ - .srcu =3D __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu), \ + .srcu =3D __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu, 0), \ } =20 #define ATOMIC_NOTIFIER_HEAD(name) \ diff --git a/include/linux/srcu.h b/include/linux/srcu.h index ada65b58bc4c..26de47820c58 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -25,8 +25,10 @@ struct srcu_struct; =20 #ifdef CONFIG_DEBUG_LOCK_ALLOC =20 -int __init_srcu_struct(struct srcu_struct *ssp, const char *name, - struct lock_class_key *key); +int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct l= ock_class_key *key); +#ifndef CONFIG_TINY_SRCU +int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, str= uct lock_class_key *key); +#endif // #ifndef CONFIG_TINY_SRCU =20 #define init_srcu_struct(ssp) \ ({ \ @@ -35,10 +37,20 @@ int __init_srcu_struct(struct srcu_struct *ssp, const c= har *name, __init_srcu_struct((ssp), #ssp, &__srcu_key); \ }) =20 +#define init_srcu_struct_fast(ssp) \ +({ \ + static struct lock_class_key __srcu_key; \ + \ + __init_srcu_struct_fast((ssp), #ssp, &__srcu_key); \ +}) + #define __SRCU_DEP_MAP_INIT(srcu_name) .dep_map =3D { .name =3D #srcu_name= }, #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 int init_srcu_struct(struct srcu_struct *ssp); +#ifndef CONFIG_TINY_SRCU +int init_srcu_struct_fast(struct srcu_struct *ssp); +#endif // #ifndef CONFIG_TINY_SRCU =20 #define __SRCU_DEP_MAP_INIT(srcu_name) #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 3bfbd44cb1b3..92e6ab53398f 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -31,7 +31,7 @@ struct srcu_struct { =20 void srcu_drive_gp(struct work_struct *wp); =20 -#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored) \ +#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored, ____ignored) \ { \ .srcu_wq =3D __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \ .srcu_cb_tail =3D &name.srcu_cb_head, \ @@ -44,13 +44,20 @@ void srcu_drive_gp(struct work_struct *wp); * Tree SRCU, which needs some per-CPU data. */ #define DEFINE_SRCU(name) \ - struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name) + struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, name) #define DEFINE_STATIC_SRCU(name) \ - static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name) + static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) +#define DEFINE_SRCU_FAST(name) DEFINE_SRCU(name) +#define DEFINE_STATIC_SRCU_FAST(name) \ + static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) =20 // Dummy structure for srcu_notifier_head. struct srcu_usage { }; #define __SRCU_USAGE_INIT(name) { } +#define __init_srcu_struct_fast __init_srcu_struct +#ifndef CONFIG_DEBUG_LOCK_ALLOC +#define init_srcu_struct_fast init_srcu_struct +#endif // #ifndef CONFIG_DEBUG_LOCK_ALLOC =20 void synchronize_srcu(struct srcu_struct *ssp); =20 diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 93ad18acd6d0..7ff4a11bc5a3 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -104,6 +104,7 @@ struct srcu_usage { struct srcu_struct { struct srcu_ctr __percpu *srcu_ctrp; struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ + u8 srcu_reader_flavor; struct lockdep_map dep_map; struct srcu_usage *srcu_sup; /* Update-side data. */ }; @@ -162,20 +163,21 @@ struct srcu_struct { .work =3D __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ } =20 -#define __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ +#define __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ .srcu_sup =3D &usage_name, \ + .srcu_reader_flavor =3D fast, \ __SRCU_DEP_MAP_INIT(name) =20 -#define __SRCU_STRUCT_INIT_MODULE(name, usage_name) \ +#define __SRCU_STRUCT_INIT_MODULE(name, usage_name, fast) \ { \ - __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ + __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ } =20 -#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name) \ +#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name, fast) \ { \ .sda =3D &pcpu_name, \ .srcu_ctrp =3D &pcpu_name.srcu_ctrs[0], \ - __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ + __SRCU_STRUCT_INIT_COMMON(name, usage_name, fast) \ } =20 /* @@ -196,23 +198,29 @@ struct srcu_struct { * init_srcu_struct(&my_srcu); * * See include/linux/percpu-defs.h for the rules on per-CPU variables. + * + * DEFINE_SRCU_FAST() creates an srcu_struct and associated structures + * whose readers must be of the SRCU-fast variety. */ #ifdef MODULE -# define __DEFINE_SRCU(name, is_static) \ +# define __DEFINE_SRCU(name, fast, is_static) \ static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s= rcu_usage); \ - is_static struct srcu_struct name =3D __SRCU_STRUCT_INIT_MODULE(name, nam= e##_srcu_usage); \ + is_static struct srcu_struct name =3D __SRCU_STRUCT_INIT_MODULE(name, nam= e##_srcu_usage, \ + fast); \ extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ __section("___srcu_struct_ptrs") =3D &name #else -# define __DEFINE_SRCU(name, is_static) \ +# define __DEFINE_SRCU(name, fast, is_static) \ static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ static struct srcu_usage name##_srcu_usage =3D __SRCU_USAGE_INIT(name##_s= rcu_usage); \ is_static struct srcu_struct name =3D \ - __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data) + __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data, fast) #endif -#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) -#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static) +#define DEFINE_SRCU(name) __DEFINE_SRCU(name, 0, /* not static */) +#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, 0, static) +#define DEFINE_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST,= /* not static */) +#define DEFINE_STATIC_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR= _FAST, static) =20 int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 38b440b0b0c8..9869a13b8763 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -286,16 +286,29 @@ static int init_srcu_struct_fields(struct srcu_struct= *ssp, bool is_static) =20 #ifdef CONFIG_DEBUG_LOCK_ALLOC =20 -int __init_srcu_struct(struct srcu_struct *ssp, const char *name, - struct lock_class_key *key) +static int +__init_srcu_struct_common(struct srcu_struct *ssp, const char *name, struc= t lock_class_key *key) { /* Don't re-initialize a lock while it is held. */ debug_check_no_locks_freed((void *)ssp, sizeof(*ssp)); lockdep_init_map(&ssp->dep_map, name, key, 0); return init_srcu_struct_fields(ssp, false); } + +int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct l= ock_class_key *key) +{ + ssp->srcu_reader_flavor =3D 0; + return __init_srcu_struct_common(ssp, name, key); +} EXPORT_SYMBOL_GPL(__init_srcu_struct); =20 +int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, str= uct lock_class_key *key) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST; + return __init_srcu_struct_common(ssp, name, key); +} +EXPORT_SYMBOL_GPL(__init_srcu_struct_fast); + #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /** @@ -308,10 +321,26 @@ EXPORT_SYMBOL_GPL(__init_srcu_struct); */ int init_srcu_struct(struct srcu_struct *ssp) { + ssp->srcu_reader_flavor =3D 0; return init_srcu_struct_fields(ssp, false); } EXPORT_SYMBOL_GPL(init_srcu_struct); =20 +/** + * init_srcu_struct_fast - initialize a fast-reader sleep-RCU structure + * @ssp: structure to initialize. + * + * Must invoke this on a given srcu_struct before passing that srcu_struct + * to any other function. Each srcu_struct represents a separate domain + * of SRCU protection. + */ +int init_srcu_struct_fast(struct srcu_struct *ssp) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST; + return init_srcu_struct_fields(ssp, false); +} +EXPORT_SYMBOL_GPL(init_srcu_struct_fast); + #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /* @@ -734,6 +763,9 @@ void __srcu_check_read_flavor(struct srcu_struct *ssp, = int read_flavor) =20 sdp =3D raw_cpu_ptr(ssp->sda); old_read_flavor =3D READ_ONCE(sdp->srcu_reader_flavor); + WARN_ON_ONCE(ssp->srcu_reader_flavor && read_flavor !=3D ssp->srcu_reader= _flavor); + WARN_ON_ONCE(old_read_flavor && ssp->srcu_reader_flavor && + old_read_flavor !=3D ssp->srcu_reader_flavor); if (!old_read_flavor) { old_read_flavor =3D cmpxchg(&sdp->srcu_reader_flavor, 0, read_flavor); if (!old_read_flavor) --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 932B2153BED; Mon, 3 Nov 2025 07:19:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762154352; cv=none; b=WbgU0fBj7qvCM9zZLRGz4dIyG8nMWi7fQpSgCvxbkIa5KTVq5qt/Ns/3aujIGR9xhSJB0OHeesTphRC8ShjbUQVtc7JN+YbhKowu0MMn0uvOBSyVSt0xFF1tMGIMFNFFoRPLwGl3c8DjSXTOMt/8U5dH+hQzsXCI3yLnoPozeKY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762154352; c=relaxed/simple; bh=Q0AKERJDbUSfsvDf4x7Le3CWRCQuSmTY0wgl6K/Nz6M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=osSOK/cWXFSH6GA1HXq8yobNLc7HHjwWl6+CDCi8HQbq+Xxwvk8Phb7299HBKZrHRd4gQheDu5YY6H8r5/6iUxUDl49SikGgN/jDwdYJf4FWgJflLPMp+ssp5Q2XOELgka8rTdu0XZF8mV8N9O58rbezJWOPDu5GrekSPYxrT9E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jnBM3d/E; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jnBM3d/E" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 382EBC4AF0B; Mon, 3 Nov 2025 07:19:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762154352; bh=Q0AKERJDbUSfsvDf4x7Le3CWRCQuSmTY0wgl6K/Nz6M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jnBM3d/E+ybQzkcBC9iT6IUwRUfd3ufuyVxc8Fl/nEFWDPKw5MGD2U9mumDTjbuL+ xSsCTxgqlHeH+1tAy98A17kJgirQ/bMgYUTzWldFY+dzypYu3+IpNHADmPW87XwZ0u EzZ/wA8a1h4HRdVcoIp/lRW6vNwHgiJXu5ru/ojh/3gp63fDNK/d7v6ml0pg37vOeP HLIGWXyg2ZmxfN69zi2K65i1OoskCrL+6PY2Qv9NJlcMJyJM7XhzbCPWgV/eKGHzj0 GJprz6KUBoUPCyPDbk0c5qsmpT8dKZk5k4NGhFe09KomwZmYkUzoM/sbdaQE/082Z+ BiLd3EwBVRNMA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 84200CE0FF9; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 05/19] srcu: Make grace-period determination use ssp->srcu_reader_flavor Date: Sun, 2 Nov 2025 13:44:22 -0800 Message-Id: <20251102214436.3905633-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit causes the srcu_readers_unlock_idx() function to take the srcu_struct structure's ->srcu_reader_flavor field into account. This ensures that structures defined via DEFINE_SRCU_FAST( or initialized via init_srcu_struct_fast() have their grace periods use synchronize_srcu() or synchronize_srcu_expedited() instead of smp_mb(), even before the first SRCU reader has been entered. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- kernel/rcu/srcutree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 9869a13b8763..c29203b23d1a 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -490,7 +490,7 @@ static bool srcu_readers_lock_idx(struct srcu_struct *s= sp, int idx, bool gp, uns static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int = idx, unsigned long *rdm) { int cpu; - unsigned long mask =3D 0; + unsigned long mask =3D ssp->srcu_reader_flavor; unsigned long sum =3D 0; =20 for_each_possible_cpu(cpu) { --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADA76256C88; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; cv=none; b=qxi8gYwdVF5Olw+TN3yLP87AXGX500iARmegHlFgeXUMu8vmVWxTYSUYp60tA1cgdwGzIycGv7HKPbQBmlTj+RQh09DGJOIICBO+7eJMwmTlkt5oMfkmWdig/Rke+jT/guNI5McdFDubuaGcCdxQ8QFxURgOa5CvZfV/hE1zEMY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; c=relaxed/simple; bh=Cu8i0gH4rjThNUjBWI39PqXKh0kLjosZp3RPH4rFPo8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=T+zKkB5xtJxZylpZitU394cVXhudAmHV+/wHNn4lFIhOTgZQTtQC8wwbDU3LPP5GNLSMnKXHmdJnAHj0x2Xoa0+5ctMp/ufB4YwMhUYrwmWiDBhX7L/oRS7GSPcMORcW9vSXUpPSg5CxiiL7/t4fit8VGm7cQpDbRklGQtYdgMc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Wod+oXgk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Wod+oXgk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7B8B8C116C6; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119879; bh=Cu8i0gH4rjThNUjBWI39PqXKh0kLjosZp3RPH4rFPo8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Wod+oXgk0zjxiOSXGCu3xJx6QIIT9WScaDOq1gpQDsVgWE7G0WhyBx9gP/FYUHXgN hInxHTm+FfXs5UuLdtMBRqV5VjC2FMb0BnKqbeNvRzYjH8RzAkdhiURvdDD5O+PX4l tPitJiuuI7LJeKmds0C2bux9TP849xCmWp0p46AYVom3r2L6BuaGKx2VrSWCaHPOYa 4+2IUIZPgi112nOyqTpqrA1n/fWZV44Y9PzI+T/hPRSVONQ7PlbC76u9OT3M7/Uowd eE6Ig5CRrP5Wz8U9uJ51vBuYCOqIrDdW9OHezQuYxiyJrLaw97Hwayk0Vw5ceMWxfQ eySEbKhACEy/Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 87105CE1104; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 06/19] rcutorture: Exercise DEFINE_STATIC_SRCU_FAST() and init_srcu_struct_fast() Date: Sun, 2 Nov 2025 13:44:23 -0800 Message-Id: <20251102214436.3905633-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit updates the initialization for the "srcu" and "srcud" torture types to use DEFINE_STATIC_SRCU_FAST() and init_srcu_struct_fast(), respectively, when reader_flavor is equal to SRCU_READ_FLAVOR_FAST. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- kernel/rcu/rcutorture.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index aa1f8240a276..e30311551a62 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -692,10 +692,18 @@ static struct rcu_torture_ops rcu_busted_ops =3D { */ =20 DEFINE_STATIC_SRCU(srcu_ctl); +DEFINE_STATIC_SRCU_FAST(srcu_ctlf); static struct srcu_struct srcu_ctld; static struct srcu_struct *srcu_ctlp =3D &srcu_ctl; static struct rcu_torture_ops srcud_ops; =20 +static void srcu_torture_init(void) +{ + rcu_sync_torture_init(); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) + srcu_ctlp =3D &srcu_ctlf; +} + static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) { srcutorture_get_gp_data(srcu_ctlp, flags, gp_seq); @@ -865,7 +873,7 @@ static void srcu_torture_expedite_current(void) =20 static struct rcu_torture_ops srcu_ops =3D { .ttype =3D SRCU_FLAVOR, - .init =3D rcu_sync_torture_init, + .init =3D srcu_torture_init, .readlock =3D srcu_torture_read_lock, .read_delay =3D srcu_read_delay, .readunlock =3D srcu_torture_read_unlock, @@ -897,10 +905,13 @@ static struct rcu_torture_ops srcu_ops =3D { .name =3D "srcu" }; =20 -static void srcu_torture_init(void) +static void srcud_torture_init(void) { rcu_sync_torture_init(); - WARN_ON(init_srcu_struct(&srcu_ctld)); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) + WARN_ON(init_srcu_struct_fast(&srcu_ctld)); + else + WARN_ON(init_srcu_struct(&srcu_ctld)); srcu_ctlp =3D &srcu_ctld; } =20 @@ -913,7 +924,7 @@ static void srcu_torture_cleanup(void) /* As above, but dynamically allocated. */ static struct rcu_torture_ops srcud_ops =3D { .ttype =3D SRCU_FLAVOR, - .init =3D srcu_torture_init, + .init =3D srcud_torture_init, .cleanup =3D srcu_torture_cleanup, .readlock =3D srcu_torture_read_lock, .read_delay =3D srcu_read_delay, --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3C6E245006; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119878; cv=none; b=QzKZGOnfbg5o5t20ysHoSYZf5sSHTpM87KvofRs3XSK3xmcxNMZ2FqqvlqfTj2l2g1Keck++EgAyvB8l51M0J99GyiOFGeFhTaYLsCAdtBo0CThgjN+4iiRljS59G0bBTQp0v/UqbM9BwrM/YoxcSt8eFS8brka0y9pyf306r4I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119878; c=relaxed/simple; bh=Iq5yf1ytJm34oKtGaJvxP71I3WkFsasL56D7JZi7uYs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Yx1shKROY2r0TAgU6sO9fImK04oFF7O5HH8rQ0o506Un2o9mQlQ0RsZCHUZm8t+dA3WRJ8oIT9GNdZ+dAWZyVXGwVev7+H5sEeg47HN4/rsk3MUCbJig5G3o63fAcdoCKrzFp7Bj1gO47AGvn1P0QLsdQAaHrrB75fa904+O9P4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RwZSP50U; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RwZSP50U" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 891A8C4CEF7; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119878; bh=Iq5yf1ytJm34oKtGaJvxP71I3WkFsasL56D7JZi7uYs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RwZSP50USITCF3AAeNOymMIl24QeLYxtEqVByYrCdXcdFh/yrE+8eOvH7rW/20Xf7 f5Ev8DK7t+GRVfnYiPElgARZzz8MPyxOC2RNbsPAZR+As1Zod2heyq1jg1G92oLYdw SK72e6y77QJRQcol74ylrw5JoH7lRi705w/AAHKMZPfzk43mP8ftoR/jZQ5A0pob1t POzq9ZLfnGfPOX6ZW2HPS6qPmO2nsW7Y5vD6MqvdzRjS9AVU1vj0QDz/eapfDx+Evl 5giqwpIFMJjpTTo039DTuLiSgtDi3t4hWNpHxLk1klOBnrhPtLR81s19yHN/63Pmf6 34LuphfdpSiTw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 89A58CE122D; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 07/19] srcu: Require special srcu_struct define/init for SRCU-fast readers Date: Sun, 2 Nov 2025 13:44:24 -0800 Message-Id: <20251102214436.3905633-7-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds CONFIG_PROVE_RCU=3Dy checking to enforce the new rule that srcu_struct structures passed to srcu_read_lock_fast() and other SRCU-fast read-side markers be either initialized with init_srcu_struct_fast() on the one hand or defined using either DEFINE_SRCU_FAST() or DEFINE_STATIC_SRCU_FAST(). This will enable removal of the non-debug read-side checks from srcu_read_lock_fast() and friends, which on my laptop provides a 25% speedup (which admittedly amounts to about half a nanosecond, but when tracing fastpaths...) Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 34 ++++++++++++++++++++++------------ kernel/rcu/srcutree.c | 1 + 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 26de47820c58..2982b5a6930f 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -271,17 +271,26 @@ static inline int srcu_read_lock(struct srcu_struct *= ssp) __acquires(ssp) * @ssp: srcu_struct in which to register the new reader. * * Enter an SRCU read-side critical section, but for a light-weight - * smp_mb()-free reader. See srcu_read_lock() for more information. - * - * If srcu_read_lock_fast() is ever used on an srcu_struct structure, - * then none of the other flavors may be used, whether before, during, - * or after. Note that grace-period auto-expediting is disabled for _fast - * srcu_struct structures because auto-expedited grace periods invoke - * synchronize_rcu_expedited(), IPIs and all. - * - * Note that srcu_read_lock_fast() can be invoked only from those contexts - * where RCU is watching, that is, from contexts where it would be legal - * to invoke rcu_read_lock(). Otherwise, lockdep will complain. + * smp_mb()-free reader. See srcu_read_lock() for more information. This + * function is NMI-safe, in a manner similar to srcu_read_lock_nmisafe(). + * + * For srcu_read_lock_fast() to be used on an srcu_struct structure, + * that structure must have been defined using either DEFINE_SRCU_FAST() + * or DEFINE_STATIC_SRCU_FAST() on the one hand or initialized with + * init_srcu_struct_fast() on the other. Such an srcu_struct structure + * cannot be passed to any non-fast variant of srcu_read_{,un}lock() or + * srcu_{down,up}_read(). In kernels built with CONFIG_PROVE_RCU=3Dy, + * __srcu_check_read_flavor() will complain bitterly if you ignore this + * restriction. + * + * Grace-period auto-expediting is disabled for SRCU-fast srcu_struct + * structures because SRCU-fast expedited grace periods invoke + * synchronize_rcu_expedited(), IPIs and all. If you need expedited + * SRCU-fast grace periods, use synchronize_srcu_expedited(). + * + * The srcu_read_lock_fast() function can be invoked only from those + * contexts where RCU is watching, that is, from contexts where it would + * be legal to invoke rcu_read_lock(). Otherwise, lockdep will complain. */ static inline struct srcu_ctr __percpu *srcu_read_lock_fast(struct srcu_st= ruct *ssp) __acquires(ssp) { @@ -317,7 +326,8 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast_notrace(struct srcu_ * srcu_down_read() for more information. * * The same srcu_struct may be used concurrently by srcu_down_read_fast() - * and srcu_read_lock_fast(). + * and srcu_read_lock_fast(). However, the same definition/initialization + * requirements called out for srcu_read_lock_safe() apply. */ static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_st= ruct *ssp) __acquires(ssp) { diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index c29203b23d1a..2f8aa280911e 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -766,6 +766,7 @@ void __srcu_check_read_flavor(struct srcu_struct *ssp, = int read_flavor) WARN_ON_ONCE(ssp->srcu_reader_flavor && read_flavor !=3D ssp->srcu_reader= _flavor); WARN_ON_ONCE(old_read_flavor && ssp->srcu_reader_flavor && old_read_flavor !=3D ssp->srcu_reader_flavor); + WARN_ON_ONCE(read_flavor =3D=3D SRCU_READ_FLAVOR_FAST && !ssp->srcu_reade= r_flavor); if (!old_read_flavor) { old_read_flavor =3D cmpxchg(&sdp->srcu_reader_flavor, 0, read_flavor); if (!old_read_flavor) --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D362D244693; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119878; cv=none; b=LksKexR9M90QpqLpssD/Sr9IqjiX1q9c2+IMs9Q1h6LAzaHtJJgGq8p2X9N/ymtper+x02IG7iYah/tkGlHeKmWPOmTH/InVF6/Dkbo/kMVaK0CqinqCTFwuZh1uD2sLdgf4EVU2IawvQlhM45VGU0axVZzBeUoiH3Mr6Z+h2k4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119878; c=relaxed/simple; bh=MZmvN2dS91ATVvwCaKa51PWDnZiCJGVXmJf2Q3doHC0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VQAf88UpUTRHP+Mn7Xiso7/sTdwjdzP7wa1dK9wGpsNHQAsg0DxsDqjQRzTgT3ndY6MoT3qbO3MViQzYlyyMRd11O7h+TWjTokBNB4eyWVt21/diZQOuDSDE2iImW4pkPpOhs/KH7ttdVkKLLimjOtu5ZeheWLY9l2izUSy/CcE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WUhFNUhU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WUhFNUhU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 93CD1C113D0; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119878; bh=MZmvN2dS91ATVvwCaKa51PWDnZiCJGVXmJf2Q3doHC0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WUhFNUhU6PsLXIhM6mDZX4XJV0ggyvdfkkIxVBcQisnQN/djlHGan60y62tWZK9Fh bVPQX0b0nwRWEca9g6/EJj67emEjOCykJD/kYDNhl5DflBLsy4T4VNiP/gzZtYRwUS rQYkOTC51E4KprOlDRVWJ25bVKLq24neMKZBKHNZDWpcFP3hq6nWb1or96vFAiqL4p F8V+CRaxViLW0X/9oCqx0peu8Jwd5ANqDBaxnqq7/QLydMstPcs6qjMsHks81M4wae sWrqhP59bg5Z0juENowamugvp+rPeHOY3GD8qGw+Aevvpmq+dR5b06jZyF8CRJXZ+j fa8PYAZ3KyLwQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8C4C4CE1257; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 08/19] srcu: Make SRCU-fast readers enforce use of SRCU-fast definition/init Date: Sun, 2 Nov 2025 13:44:25 -0800 Message-Id: <20251102214436.3905633-8-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit makes CONFIG_PROVE_RCU=3Dy kernels enforce the new rule that srcu_struct structures that are passed to srcu_read_lock_fast() and other SRCU-fast read-side markers be either initialized with init_srcu_struct_fast() on the one hand or defined with DEFINE_SRCU_FAST() or DEFINE_STATIC_SRCU_FAST() on the other. This eliminates the read-side test that was formerly included in srcu_read_lock_fast() and friends, speeding these primitives up by about 25% (admittedly only about half of a nanosecond, but when tracing on fastpaths...) Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 6 +++--- include/linux/srcutiny.h | 1 - include/linux/srcutree.h | 16 +--------------- 3 files changed, 4 insertions(+), 19 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 2982b5a6930f..41e27c1d917d 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -297,7 +297,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast(struct srcu_struct * struct srcu_ctr __percpu *retval; =20 RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast()."); - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); retval =3D __srcu_read_lock_fast(ssp); rcu_try_lock_acquire(&ssp->dep_map); return retval; @@ -312,7 +312,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast_notrace(struct srcu_ { struct srcu_ctr __percpu *retval; =20 - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); retval =3D __srcu_read_lock_fast(ssp); return retval; } @@ -333,7 +333,7 @@ static inline struct srcu_ctr __percpu *srcu_down_read_= fast(struct srcu_struct * { WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read= _fast()."); - srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); return __srcu_read_lock_fast(ssp); } =20 diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 92e6ab53398f..1ecc3393fb26 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -112,7 +112,6 @@ static inline void srcu_barrier(struct srcu_struct *ssp) =20 static inline void srcu_expedite_current(struct srcu_struct *ssp) { } #define srcu_check_read_flavor(ssp, read_flavor) do { } while (0) -#define srcu_check_read_flavor_force(ssp, read_flavor) do { } while (0) =20 /* Defined here to avoid size increase for non-torture kernels. */ static inline void srcu_torture_stats_print(struct srcu_struct *ssp, diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 7ff4a11bc5a3..6080a9094618 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -307,21 +307,7 @@ __srcu_read_unlock_fast(struct srcu_struct *ssp, struc= t srcu_ctr __percpu *scp) =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); =20 -// Record reader usage even for CONFIG_PROVE_RCU=3Dn kernels. This is -// needed only for flavors that require grace-period smp_mb() calls to be -// promoted to synchronize_rcu(). -static inline void srcu_check_read_flavor_force(struct srcu_struct *ssp, i= nt read_flavor) -{ - struct srcu_data *sdp =3D raw_cpu_ptr(ssp->sda); - - if (likely(READ_ONCE(sdp->srcu_reader_flavor) & read_flavor)) - return; - - // Note that the cmpxchg() in __srcu_check_read_flavor() is fully ordered. - __srcu_check_read_flavor(ssp, read_flavor); -} - -// Record non-_lite() usage only for CONFIG_PROVE_RCU=3Dy kernels. +// Record SRCU-reader usage type only for CONFIG_PROVE_RCU=3Dy kernels. static inline void srcu_check_read_flavor(struct srcu_struct *ssp, int rea= d_flavor) { if (IS_ENABLED(CONFIG_PROVE_RCU)) --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77405253957; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; cv=none; b=Zzp7DbZVuoBS1fRSTNVSsYpCS4Gg1XDC93aUK1K4rei6Hi0y0DWFaSkj/OLeCzJbhE/aDTsnlU6xdYNF4zLXi1R//rfY6RWxso31kiyoAHxZwNGDjCK/erLeUlJPx2rTO2gPZx7KaZEj5esshfRN1mYABGL6ZIdkms6aKP+4+mE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; c=relaxed/simple; bh=7uPrv4irYTmEwJ2B3VG3qdNupsgZm8tMakjud6mO6xs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PHilXZ9UmbKm4XhzyheU8JzgE02A3eiXdYGn7aU9IFyNVkVNqdOUs/XmMOH4NMM4C5eGPwxiJPGTnvbqG4WmvMVWl4dMCHBt7QMCRSqVRc5mD8bWYc+azk3JkTScJ6LCGbLWN/GuEF8hP16aPqREe75mjtSJDkhL8gz4IIx+jI8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qhUfumK2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qhUfumK2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A2697C116B1; Sun, 2 Nov 2025 21:44:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119878; bh=7uPrv4irYTmEwJ2B3VG3qdNupsgZm8tMakjud6mO6xs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qhUfumK2Q0jsURsB6qQleXPeuWhfqHw87RyLsYPgjb38axzSGzX83DeBqMpcB0PUP NsUP6ZL5Jv/4ExFMSEqulzBQyxx/JbL21O/Z6T0jMEA+DIMlkZ+SU8Kan2TVEFPW9h eP6VRjPNTUDIrv0IXIUFdI6HLYY0FqxBwVjldOK7cIUxf/C3V4Y8oZjHpjuUvl+JEQ mGYou++w5hXnzJombP1kVshtuWnYCNRS+UKpoxOOBVUJrau03zp9A6jX1/R83peNYD O6sn0riM0To7GnfGMUjy+e0Of8mJLPRd2pwQUhTEgdW4Ag04AOpShGS4C3ULcENhxx wBBFRvFgJJl9Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8F271CE12A7; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 09/19] doc: Update for SRCU-fast definitions and initialization Date: Sun, 2 Nov 2025 13:44:26 -0800 Message-Id: <20251102214436.3905633-9-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit documents the DEFINE_SRCU_FAST(), DEFINE_STATIC_SRCU_FAST(), and init_srcu_struct_fast() API members. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- .../RCU/Design/Requirements/Requirements.rst | 33 ++++++++++--------- Documentation/RCU/checklist.rst | 12 ++++--- Documentation/RCU/whatisRCU.rst | 3 ++ 3 files changed, 27 insertions(+), 21 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Docum= entation/RCU/Design/Requirements/Requirements.rst index f24b3c0b9b0d..ba417a08b93d 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2637,15 +2637,16 @@ synchronize_srcu() for some other domain ``ss1``, a= nd if an that was held across as ``ss``-domain synchronize_srcu(), deadlock would again be possible. Such a deadlock cycle could extend across an arbitrarily large number of different SRCU domains. Again, with great -power comes great responsibility. +power comes great responsibility, though lockdep is now able to detect +this sort of deadlock. =20 -Unlike the other RCU flavors, SRCU read-side critical sections can run -on idle and even offline CPUs. This ability requires that -srcu_read_lock() and srcu_read_unlock() contain memory barriers, -which means that SRCU readers will run a bit slower than would RCU -readers. It also motivates the smp_mb__after_srcu_read_unlock() API, -which, in combination with srcu_read_unlock(), guarantees a full -memory barrier. +Unlike the other RCU flavors, SRCU read-side critical sections can run on +idle and even offline CPUs, with the exception of srcu_read_lock_fast() +and friends. This ability requires that srcu_read_lock() and +srcu_read_unlock() contain memory barriers, which means that SRCU +readers will run a bit slower than would RCU readers. It also motivates +the smp_mb__after_srcu_read_unlock() API, which, in combination with +srcu_read_unlock(), guarantees a full memory barrier. =20 Also unlike other RCU flavors, synchronize_srcu() may **not** be invoked from CPU-hotplug notifiers, due to the fact that SRCU grace @@ -2681,15 +2682,15 @@ run some tests first. SRCU just might need a few ad= justment to deal with that sort of load. Of course, your mileage may vary based on the speed of your CPUs and the size of your memory. =20 -The `SRCU -API `__ +The `SRCU API +`__ includes srcu_read_lock(), srcu_read_unlock(), -srcu_dereference(), srcu_dereference_check(), -synchronize_srcu(), synchronize_srcu_expedited(), -call_srcu(), srcu_barrier(), and srcu_read_lock_held(). It -also includes DEFINE_SRCU(), DEFINE_STATIC_SRCU(), and -init_srcu_struct() APIs for defining and initializing -``srcu_struct`` structures. +srcu_dereference(), srcu_dereference_check(), synchronize_srcu(), +synchronize_srcu_expedited(), call_srcu(), srcu_barrier(), +and srcu_read_lock_held(). It also includes DEFINE_SRCU(), +DEFINE_STATIC_SRCU(), DEFINE_SRCU_FAST(), DEFINE_STATIC_SRCU_FAST(), +init_srcu_struct(), and init_srcu_struct_fast() APIs for defining and +initializing ``srcu_struct`` structures. =20 More recently, the SRCU API has added polling interfaces: =20 diff --git a/Documentation/RCU/checklist.rst b/Documentation/RCU/checklist.= rst index c9bfb2b218e5..4b30f701225f 100644 --- a/Documentation/RCU/checklist.rst +++ b/Documentation/RCU/checklist.rst @@ -417,11 +417,13 @@ over a rather long period of time, but improvements a= re always welcome! you should be using RCU rather than SRCU, because RCU is almost always faster and easier to use than is SRCU. =20 - Also unlike other forms of RCU, explicit initialization and - cleanup is required either at build time via DEFINE_SRCU() - or DEFINE_STATIC_SRCU() or at runtime via init_srcu_struct() - and cleanup_srcu_struct(). These last two are passed a - "struct srcu_struct" that defines the scope of a given + Also unlike other forms of RCU, explicit initialization + and cleanup is required either at build time via + DEFINE_SRCU(), DEFINE_STATIC_SRCU(), DEFINE_SRCU_FAST(), + or DEFINE_STATIC_SRCU_FAST() or at runtime via either + init_srcu_struct() or init_srcu_struct_fast() and + cleanup_srcu_struct(). These last three are passed a + `struct srcu_struct` that defines the scope of a given SRCU domain. Once initialized, the srcu_struct is passed to srcu_read_lock(), srcu_read_unlock() synchronize_srcu(), synchronize_srcu_expedited(), and call_srcu(). A given diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.= rst index cf0b0ac9f463..a1582bd653d1 100644 --- a/Documentation/RCU/whatisRCU.rst +++ b/Documentation/RCU/whatisRCU.rst @@ -1227,7 +1227,10 @@ SRCU: Initialization/cleanup/ordering:: =20 DEFINE_SRCU DEFINE_STATIC_SRCU + DEFINE_SRCU_FAST // for srcu_read_lock_fast() and friends + DEFINE_STATIC_SRCU_FAST // for srcu_read_lock_fast() and friends init_srcu_struct + init_srcu_struct_fast cleanup_srcu_struct smp_mb__after_srcu_read_unlock =20 --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98C95255E40; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; cv=none; b=dCO0PeQEQX1M2NhkqMITf6A0PTKNA13PydjEKSP45fW/ILg6+x8Qc6gGHgb37JFcXW6Bmm+3DdbKUUWL8tCn2VFRpuLL/HIw5f7Uu/AzzWzV/7scqN+kkc+JWtun1IMkQC4XFKLNtrOz+KvRbGNz6bqhY1A+C//lgOzACCA/ma0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; c=relaxed/simple; bh=MgE4NzahsQRutiZm/q0L791+QKh3LZkPzHRxKVcKPeY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XOZM+eFasmUTLDQKGU3ca0UkpzcwWzvw6AbbDL/Pj641/ivZjxP52sDUUQTemr+TRtMT8TuueBynMeqDpIIrWYglIZlVO55A+QRYlRDYyKwcKZwJkyT0CZznscM0js41i8yoMsTZvhhnqM19lQTt0Dnf7UhTjc1pMcckTp8dEi0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=q2fmiFzn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="q2fmiFzn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 393D5C4AF09; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119879; bh=MgE4NzahsQRutiZm/q0L791+QKh3LZkPzHRxKVcKPeY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q2fmiFznrE4om5FpTb8TbWCxZvYGYzDipx/N79XcWd0h7NRy7zrvvWCmqQr3rSbRM jswk/X2CiwN5FreqovTF+InRmoXOCWxhSvUj8QQhME1lYhTatr1bBR7HKpKsaJP03D WEmfG2l4ebxKlfsKozgvr/tRKbk3YSoFnJ2I4O08t4sRaSl7JvUa3Q0yKNMZl45TvK P2z32WbXF6+qu2NpEXCaZN0J8oPxyxv1BNehj53tgu2ZAU4EUuXiicqbeqysBHPstM PzqiIS4U3F7/TffpW0ZGpO9pZz1XyI2SjJd2qUgkOVv9HDnoK0Ewne6slDRuUL4h2l hQPyhZHcw6ggw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 91C47CE13CA; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 10/19] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast Date: Sun, 2 Nov 2025 13:44:27 -0800 Message-Id: <20251102214436.3905633-10-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current use of guard(preempt_notrace)() within __DECLARE_TRACE() to protect invocation of __DO_TRACE_CALL() means that BPF programs attached to tracepoints are non-preemptible. This is unhelpful in real-time systems, whose users apparently wish to use BPF while also achieving low latencies. (Who knew?) One option would be to use preemptible RCU, but this introduces many opportunities for infinite recursion, which many consider to be counterproductive, especially given the relatively small stacks provided by the Linux kernel. These opportunities could be shut down by sufficiently energetic duplication of code, but this sort of thing is considered impolite in some circles. Therefore, use the shiny new SRCU-fast API, which provides somewhat faster readers than those of preemptible RCU, at least on my laptop, where task_struct access is more expensive than access to per-CPU variables. And SRCU fast provides way faster readers than does SRCU, courtesy of being able to avoid the read-side use of smp_mb(). Also, it is quite straightforward to create srcu_read_{,un}lock_fast_notrace() functions. While in the area, SRCU now supports early boot call_srcu(). Therefore, remove the checks that used to avoid such use from rcu_free_old_probes() before this commit was applied: e53244e2c893 ("tracepoint: Remove SRCU protection") The current commit can be thought of as an approximate revert of that commit, with some compensating additions of preemption disabling pointed out by Steven Rostedt (thank you, Steven!). This preemption disabling uses guard(preempt_notrace)(), and while in the area a couple of other use cases were also converted to guards. However, Yonghong Song points out that BPF expects non-sleepable BPF programs to remain on the same CPU, which means that migration must be disabled whenever preemption remains enabled. In addition, non-RT kernels have performance expectations on BPF that would be violated by allowing the BPF programs to be preempted. Therefore, continue to disable preemption in non-RT kernels, and protect the BPF program with both SRCU and migration disabling for RT kernels, and even then only if preemption is not already disabled. [ paulmck: Apply kernel test robot and Yonghong Song feedback. ] Link: https://lore.kernel.org/all/20250613152218.1924093-1-bigeasy@linutron= ix.de/ Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/tracepoint.h | 45 ++++++++++++++++++++++-------------- include/trace/perf.h | 4 ++-- include/trace/trace_events.h | 4 ++-- kernel/tracepoint.c | 21 ++++++++++++++++- 4 files changed, 52 insertions(+), 22 deletions(-) diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 826ce3f8e1f8..9f8b19cd303a 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -33,6 +33,8 @@ struct trace_eval_map { =20 #define TRACEPOINT_DEFAULT_PRIO 10 =20 +extern struct srcu_struct tracepoint_srcu; + extern int tracepoint_probe_register(struct tracepoint *tp, void *probe, void *data); extern int @@ -115,7 +117,10 @@ void for_each_tracepoint_in_module(struct module *mod, static inline void tracepoint_synchronize_unregister(void) { synchronize_rcu_tasks_trace(); - synchronize_rcu(); + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + synchronize_srcu(&tracepoint_srcu); + else + synchronize_rcu(); } static inline bool tracepoint_is_faultable(struct tracepoint *tp) { @@ -266,23 +271,29 @@ static inline struct tracepoint *tracepoint_ptr_deref= (tracepoint_ptr_t *p) return static_branch_unlikely(&__tracepoint_##name.key);\ } =20 -#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \ +#define __DECLARE_TRACE(name, proto, args, cond, data_proto) \ __DECLARE_TRACE_COMMON(name, PARAMS(proto), PARAMS(args), PARAMS(data_pro= to)) \ - static inline void __do_trace_##name(proto) \ - { \ - if (cond) { \ - guard(preempt_notrace)(); \ - __DO_TRACE_CALL(name, TP_ARGS(args)); \ - } \ - } \ - static inline void trace_##name(proto) \ - { \ - if (static_branch_unlikely(&__tracepoint_##name.key)) \ - __do_trace_##name(args); \ - if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \ - WARN_ONCE(!rcu_is_watching(), \ - "RCU not watching for tracepoint"); \ - } \ + static inline void __do_trace_##name(proto) \ + { \ + if (cond) { \ + if (IS_ENABLED(CONFIG_PREEMPT_RT) && preemptible()) { \ + guard(srcu_fast_notrace)(&tracepoint_srcu); \ + guard(migrate)(); \ + __DO_TRACE_CALL(name, TP_ARGS(args)); \ + } else { \ + guard(preempt_notrace)(); \ + __DO_TRACE_CALL(name, TP_ARGS(args)); \ + } \ + } \ + } \ + static inline void trace_##name(proto) \ + { \ + if (static_branch_unlikely(&__tracepoint_##name.key)) \ + __do_trace_##name(args); \ + if (IS_ENABLED(CONFIG_LOCKDEP) && (cond)) { \ + WARN_ONCE(!rcu_is_watching(), \ + "RCU not watching for tracepoint"); \ + } \ } =20 #define __DECLARE_TRACE_SYSCALL(name, proto, args, data_proto) \ diff --git a/include/trace/perf.h b/include/trace/perf.h index a1754b73a8f5..348ad1d9b556 100644 --- a/include/trace/perf.h +++ b/include/trace/perf.h @@ -71,6 +71,7 @@ perf_trace_##call(void *__data, proto) \ u64 __count __attribute__((unused)); \ struct task_struct *__task __attribute__((unused)); \ \ + guard(preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ } =20 @@ -85,9 +86,8 @@ perf_trace_##call(void *__data, proto) \ struct task_struct *__task __attribute__((unused)); \ \ might_fault(); \ - preempt_disable_notrace(); \ + guard(preempt_notrace)(); \ do_perf_trace_##call(__data, args); \ - preempt_enable_notrace(); \ } =20 /* diff --git a/include/trace/trace_events.h b/include/trace/trace_events.h index 4f22136fd465..fbc07d353be6 100644 --- a/include/trace/trace_events.h +++ b/include/trace/trace_events.h @@ -436,6 +436,7 @@ __DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args)= , PARAMS(tstruct), \ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \ { \ + guard(preempt_notrace)(); \ do_trace_event_raw_event_##call(__data, args); \ } =20 @@ -447,9 +448,8 @@ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \ { \ might_fault(); \ - preempt_disable_notrace(); \ + guard(preempt_notrace)(); \ do_trace_event_raw_event_##call(__data, args); \ - preempt_enable_notrace(); \ } =20 /* diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 62719d2941c9..21bb67798214 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -25,6 +25,9 @@ enum tp_func_state { extern tracepoint_ptr_t __start___tracepoints_ptrs[]; extern tracepoint_ptr_t __stop___tracepoints_ptrs[]; =20 +DEFINE_SRCU_FAST(tracepoint_srcu); +EXPORT_SYMBOL_GPL(tracepoint_srcu); + enum tp_transition_sync { TP_TRANSITION_SYNC_1_0_1, TP_TRANSITION_SYNC_N_2_1, @@ -34,6 +37,7 @@ enum tp_transition_sync { =20 struct tp_transition_snapshot { unsigned long rcu; + unsigned long srcu_gp; bool ongoing; }; =20 @@ -46,6 +50,7 @@ static void tp_rcu_get_state(enum tp_transition_sync sync) =20 /* Keep the latest get_state snapshot. */ snapshot->rcu =3D get_state_synchronize_rcu(); + snapshot->srcu_gp =3D start_poll_synchronize_srcu(&tracepoint_srcu); snapshot->ongoing =3D true; } =20 @@ -56,6 +61,8 @@ static void tp_rcu_cond_sync(enum tp_transition_sync sync) if (!snapshot->ongoing) return; cond_synchronize_rcu(snapshot->rcu); + if (!poll_state_synchronize_srcu(&tracepoint_srcu, snapshot->srcu_gp)) + synchronize_srcu(&tracepoint_srcu); snapshot->ongoing =3D false; } =20 @@ -101,17 +108,29 @@ static inline void *allocate_probes(int count) return p =3D=3D NULL ? NULL : p->probes; } =20 -static void rcu_free_old_probes(struct rcu_head *head) +static void srcu_free_old_probes(struct rcu_head *head) { kfree(container_of(head, struct tp_probes, rcu)); } =20 +static void rcu_free_old_probes(struct rcu_head *head) +{ + call_srcu(&tracepoint_srcu, head, srcu_free_old_probes); +} + static inline void release_probes(struct tracepoint *tp, struct tracepoint= _func *old) { if (old) { struct tp_probes *tp_probes =3D container_of(old, struct tp_probes, probes[0]); =20 + /* + * Tracepoint probes are protected by either RCU or + * Tasks Trace RCU and also by SRCU. By calling the SRCU + * callback in the [Tasks Trace] RCU callback we cover + * both cases. So let us chain the SRCU and [Tasks Trace] + * RCU callbacks to wait for both grace periods. + */ if (tracepoint_is_faultable(tp)) call_rcu_tasks_trace(&tp_probes->rcu, rcu_free_old_probes); else --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5937F24E4A1; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; cv=none; b=uAme7bwPnmBojJ+6hdGWZbCnQKyvVRPp+h7nOeF80eiC5U70z/QpnO+rijWD1ZkqBY3yP0eQpHNBIS5g5k3Ha/TRhYUWUfiKeTDRFGjy0BZEWE+jhy12/vlbZmvAhwwMBnjxR+FozfNGcrxvap+3O7lC5Y7l959JDGSkFgxxPwM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; c=relaxed/simple; bh=BK4yD7eyeQ0mHbL7KbeZ8Mx12t+bQxDXJ6lt6IBpXBE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gks+wKfDjIGRyTs+/OZ54PsGdh8XfH5s7oL2dNFrlgoBLIfRu4KL+SB2eNTmLYSohlua/v3HmkMO03/6PHZZT3AoHVY+RA3aiwdVrcPDVQBlb45RF+m5fdahPZdBTjrJbK1iHNvF0kP1lrSrlekq5agGenju15/cdIbqnVoZiY8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=b0S5LSTA; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="b0S5LSTA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 39DB6C19424; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119879; bh=BK4yD7eyeQ0mHbL7KbeZ8Mx12t+bQxDXJ6lt6IBpXBE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b0S5LSTAGO2TwFeFUsEWC5bltZQV08z5eZBbq0JSE1PLO53LHTTobxVp2hvPWZG1A EvMxeHiI++c8lmiIVA222nmLMyu+DGL2csoE8G3XLzEZe54T9zt+h0K5t6sJQAo2Vb bQBtNju+CnCSWA8v9cK6jE+2I2K2hsV6bxxoyUkAH6yqDwAvlc8cH8CcM/WiYJYsUq y8bhT+eL9rw7zr06cLZhMqEshXeMkqoqVf+8887PzsKmEhrnvLbzr7Er32+827AzqZ aRZtQznE0AIKx8bRbPNeTuvaMgKPUqYheztex5D4GXqTp60T7lzS99/U/bnjyj78OV XpsxPSkKHaDGQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 94712CE15AA; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Leon Hwang , Alexei Starovoitov Subject: [PATCH 11/19] rcu: Mark diagnostic functions as notrace Date: Sun, 2 Nov 2025 13:44:28 -0800 Message-Id: <20251102214436.3905633-11-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The rcu_lockdep_current_cpu_online(), rcu_read_lock_sched_held(), rcu_read_lock_held(), rcu_read_lock_bh_held(), rcu_read_lock_any_held() are used by tracing-related code paths, so putting traces on them is unlikely to make anyone happy. This commit therefore marks them all "notrace". Reported-by: Leon Hwang Reported-by: Alexei Starovoitov Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- kernel/rcu/update.c | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 31690ffa452a..8ddd07fed363 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4021,7 +4021,7 @@ bool rcu_cpu_online(int cpu) * RCU on an offline processor during initial boot, hence the check for * rcu_scheduler_fully_active. */ -bool rcu_lockdep_current_cpu_online(void) +bool notrace rcu_lockdep_current_cpu_online(void) { struct rcu_data *rdp; bool ret =3D false; diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index c912b594ba98..dfeba9b35395 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -117,7 +117,7 @@ static bool rcu_read_lock_held_common(bool *ret) return false; } =20 -int rcu_read_lock_sched_held(void) +int notrace rcu_read_lock_sched_held(void) { bool ret; =20 @@ -342,7 +342,7 @@ EXPORT_SYMBOL_GPL(debug_lockdep_rcu_enabled); * Note that rcu_read_lock() is disallowed if the CPU is either idle or * offline from an RCU perspective, so check for those as well. */ -int rcu_read_lock_held(void) +int notrace rcu_read_lock_held(void) { bool ret; =20 @@ -367,7 +367,7 @@ EXPORT_SYMBOL_GPL(rcu_read_lock_held); * Note that rcu_read_lock_bh() is disallowed if the CPU is either idle or * offline from an RCU perspective, so check for those as well. */ -int rcu_read_lock_bh_held(void) +int notrace rcu_read_lock_bh_held(void) { bool ret; =20 @@ -377,7 +377,7 @@ int rcu_read_lock_bh_held(void) } EXPORT_SYMBOL_GPL(rcu_read_lock_bh_held); =20 -int rcu_read_lock_any_held(void) +int notrace rcu_read_lock_any_held(void) { bool ret; =20 --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5929A24E01D; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; cv=none; b=U0quv92Kge+5odlygCt+179ZD2+qalgY04KZoWXm+DQtdT4rkdXQuoam+kY/VxUa9Z8N6SkvdaIStC1FLcKGzMVSeHZRBgORpzKClpwyy2SQno9lhDWchQ3Qi915SRxBGyDBS3kxfqWTjPOY8f3Nnao7mhM/xd2JMwx5IdL+BBQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; c=relaxed/simple; bh=bdRaLOeGRlaNeOitkkHmPDSe++KDC1F2xKdPiknDIIg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=sJltZwqr/Br24WD2WuhstTGGdMGftLtCrlVoKVGzcO4qGGoTlwJuC7SH2dOqfMFT5HciXC6ZwgSDoNAB+q16mZvzigO45Hu2uHDfNgFwTLKi25auxtO8s1LIurl1eVL77gogeu62q7WQV3hvj0MI3z6RbL56v/e3EjSQDP6IhfY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aGROng2V; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aGROng2V" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 38E93C16AAE; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119879; bh=bdRaLOeGRlaNeOitkkHmPDSe++KDC1F2xKdPiknDIIg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aGROng2Vrwr7izBe1JMo0rTtk/LECrRXPoseahtv9nErOznfGE5Xvlp0yFprbemi8 AAZWbrKf76/4O4Y1sXXQZIHyp+RzljtpYC6tOVU4pPgmJ8ieX5A/AYc6zk2tcjswFw nSqzJeyiuZ/WAyIFG83ufr32gH9gIpKLxXgpdta4VAJFmP4q2vJ/bQHzZtp7c+/7Kv vqkVD2tfKFW9eO3puAEvnhHfDWf70CsL4ROx/OsQ5XlaGe00mkoS9khBx95BsrCWSk hKl7y4wT4E3zLofx6Q8HfKaBfrIXr25kD8dpCZ8Z/espWEkNgRCMaPG2U21b0pifu+ ibfESU8CBhx2A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 96CDECE15C3; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH 12/19] rcutorture: Permit kvm-again.sh to re-use the build directory Date: Sun, 2 Nov 2025 13:44:29 -0800 Message-Id: <20251102214436.3905633-12-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds "inplace" and "inplace-force" values to the kvm-again.sh "--link" argument, which causes the run's output to be placed into the build directory. This could be used to save build time if the machine went down partway into a run, but it can also be used to do a large number of builds, and run the resulting kernels concurrently even if the builds are based on different commits. A later commit will add this latter capability to kvm-series.sh in order to produce large speedups for branch-checking operations. Signed-off-by: Paul E. McKenney --- .../selftests/rcutorture/bin/kvm-again.sh | 56 +++++++++++++------ 1 file changed, 40 insertions(+), 16 deletions(-) diff --git a/tools/testing/selftests/rcutorture/bin/kvm-again.sh b/tools/te= sting/selftests/rcutorture/bin/kvm-again.sh index 88ca4e368489..b5239b52cb5d 100755 --- a/tools/testing/selftests/rcutorture/bin/kvm-again.sh +++ b/tools/testing/selftests/rcutorture/bin/kvm-again.sh @@ -31,7 +31,7 @@ fi if ! cp "$oldrun/scenarios" $T/scenarios.oldrun then # Later on, can reconstitute this from console.log files. - echo Prior run batches file does not exist: $oldrun/batches + echo Prior run scenarios file does not exist: $oldrun/scenarios exit 1 fi =20 @@ -68,7 +68,7 @@ usage () { echo " --datestamp string" echo " --dryrun" echo " --duration minutes | s | h | d" - echo " --link hard|soft|copy" + echo " --link hard|soft|copy|inplace|inplace-force" echo " --remote" echo " --rundir /new/res/path" echo "Command line: $scriptname $args" @@ -121,7 +121,7 @@ do shift ;; --link) - checkarg --link "hard|soft|copy" "$#" "$2" 'hard\|soft\|copy' '^--' + checkarg --link "hard|soft|copy|inplace|inplace-force" "$#" "$2" 'hard\|= soft\|copy\|inplace\|inplace-force' '^--' case "$2" in copy) arg_link=3D"cp -R" @@ -132,6 +132,14 @@ do soft) arg_link=3D"cp -Rs" ;; + inplace) + arg_link=3D"inplace" + rundir=3D"$oldrun" + ;; + inplace-force) + arg_link=3D"inplace-force" + rundir=3D"$oldrun" + ;; esac shift ;; @@ -172,21 +180,37 @@ fi =20 echo ---- Re-run results directory: $rundir =20 -# Copy old run directory tree over and adjust. -mkdir -p "`dirname "$rundir"`" -if ! $arg_link "$oldrun" "$rundir" -then - echo "Cannot copy from $oldrun to $rundir." - usage -fi -rm -f "$rundir"/*/{console.log,console.log.diags,qemu_pid,qemu-pid,qemu-re= tval,Warnings,kvm-test-1-run.sh.out,kvm-test-1-run-qemu.sh.out,vmlinux} "$r= undir"/log -touch "$rundir/log" -echo $scriptname $args | tee -a "$rundir/log" -echo $oldrun > "$rundir/re-run" -if ! test -d "$rundir/../../bin" +if test "$oldrun" !=3D "$rundir" then - $arg_link "$oldrun/../../bin" "$rundir/../.." + # Copy old run directory tree over and adjust. + mkdir -p "`dirname "$rundir"`" + if ! $arg_link "$oldrun" "$rundir" + then + echo "Cannot copy from $oldrun to $rundir." + usage + fi + rm -f "$rundir"/*/{console.log,console.log.diags,qemu_pid,qemu-pid,qemu-r= etval,Warnings,kvm-test-1-run.sh.out,kvm-test-1-run-qemu.sh.out,vmlinux} "$= rundir"/log + touch "$rundir/log" + echo $scriptname $args | tee -a "$rundir/log" + echo $oldrun > "$rundir/re-run" + if ! test -d "$rundir/../../bin" + then + $arg_link "$oldrun/../../bin" "$rundir/../.." + fi +else + # Check for a run having already happened. + find "$rundir" -name console.log -print > $T/oldrun-console.log + if test -s $T/oldrun-console.log + then + echo Run already took place in $rundir + if test "$arg_link" =3D inplace + then + usage + fi + fi fi + +# Find runs to be done based on their qemu-cmd files. for i in $rundir/*/qemu-cmd do cp "$i" $T --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98861255240; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; cv=none; b=T29BZ9RUSN6+iGO1S3xY3yxvKzNCOc0gMz5c2YboB19ZT/QR7HmnLj7jOiGMUJnkQULdYSYaDht9CkVu2SKKftjy5MFCf6VwBsYLV/7kYUJL8mmWrVa8x6+/w1ApCR4NOlU7zBdf8K1NvLBKmoDanRFxBAjaYkWdvU6S1/wl9LE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; c=relaxed/simple; bh=vGDS0HEqwbT4kBxbF1tBRuTN6wYIuGTYd7f7W76C1sk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=EwVHLEWqacqMXanCM83Dn1rXv2Ou+n33XahtHcW1DD7aK2L3Bn7DSHWec6ogQtVk9JiJ2vN54/RVnwimG+lxOztUFDAAjtQem1O+usGVr4rm+DAj8ElwL/iC6Yj3bey6BxkCUwMQ5SNQM0SPTiDEAtfgsC/pD+uTKnCqhAiMwUw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZikXbao/; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZikXbao/" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 55BFDC2BC86; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119879; bh=vGDS0HEqwbT4kBxbF1tBRuTN6wYIuGTYd7f7W76C1sk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZikXbao/h8H5u2wC+awuGBqqRwgAfFKn9GmA/PGbLovW1bQ9SYoYqCzhFevZ82bqE uzXC7R6fLnGBaWzkvWtcyG2xb+zvlu5R2we2YbgIKFWaGa7aY2VMKvN9I/tkBbo4Cs hangwbMfWKyTlzPSPMAwvg2FoUsxaY1yBenUKqlDaaW9/k8ku/QSnRcGJZr3LdIbFv zpA4I6Ig1RMKuKsm+w9THJnNwU3OZ46gOE1h5bG7fFI7hCAlKZnb1gaeDAXfi0LogV ax18uiYs5SybAmm04IpaNRWiAcoLKJu3YTZa6X2wdD2dky9AtBCC9Hp6HYpXsLQdCm KKf7BqrS4cJHw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 99F12CE1605; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH 13/19] srcu: Add SRCU_READ_FLAVOR_FAST_UPDOWN CPP macro Date: Sun, 2 Nov 2025 13:44:30 -0800 Message-Id: <20251102214436.3905633-13-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds the SRCU_READ_FLAVOR_FAST_UPDOWN=3D0x8 macro and adjusts rcutorture to make use of it. In this commit, both SRCU_READ_FLAVOR_FAST=3D0x4 and the new SRCU_READ_FLAVOR_FAST_UPDOWN test SRCU-fast. When the SRCU-fast-updown is added, the new SRCU_READ_FLAVOR_FAST_UPDOWN macro will test it when passed to the rcutorture.reader_flavor module parameter. The old SRCU_READ_FLAVOR_FAST macro's value changed from 0x8 to 0x4. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 16 +++++++++------- kernel/rcu/rcutorture.c | 24 ++++++++++++++++++------ 2 files changed, 27 insertions(+), 13 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 41e27c1d917d..1dd6812aabe7 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -56,13 +56,15 @@ int init_srcu_struct_fast(struct srcu_struct *ssp); #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /* Values for SRCU Tree srcu_data ->srcu_reader_flavor, but also used by r= cutorture. */ -#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). -#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). -// 0x4 // SRCU-lite is no longer with us. -#define SRCU_READ_FLAVOR_FAST 0x8 // srcu_read_lock_fast(). -#define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR= _NMI | \ - SRCU_READ_FLAVOR_FAST) // All of the above. -#define SRCU_READ_FLAVOR_SLOWGP SRCU_READ_FLAVOR_FAST +#define SRCU_READ_FLAVOR_NORMAL 0x1 // srcu_read_lock(). +#define SRCU_READ_FLAVOR_NMI 0x2 // srcu_read_lock_nmisafe(). +// 0x4 // SRCU-lite is no longer with us. +#define SRCU_READ_FLAVOR_FAST 0x4 // srcu_read_lock_fast(). +#define SRCU_READ_FLAVOR_FAST_UPDOWN 0x8 // srcu_read_lock_fast(). +#define SRCU_READ_FLAVOR_ALL (SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_= NMI | \ + SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN) + // All of the above. +#define SRCU_READ_FLAVOR_SLOWGP (SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR= _FAST_UPDOWN) // Flavors requiring synchronize_rcu() // instead of smp_mb(). void __srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp); diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index e30311551a62..587b28258b6e 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -702,6 +702,8 @@ static void srcu_torture_init(void) rcu_sync_torture_init(); if (reader_flavor & SRCU_READ_FLAVOR_FAST) srcu_ctlp =3D &srcu_ctlf; + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) + srcu_ctlp =3D &srcu_ctlf; } =20 static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) @@ -728,6 +730,12 @@ static int srcu_torture_read_lock(void) ret +=3D idx << 1; } if (reader_flavor & SRCU_READ_FLAVOR_FAST) { + scp =3D srcu_read_lock_fast(srcu_ctlp); + idx =3D __srcu_ptr_to_ctr(srcu_ctlp, scp); + WARN_ON_ONCE(idx & ~0x1); + ret +=3D idx << 2; + } + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) { scp =3D srcu_read_lock_fast(srcu_ctlp); idx =3D __srcu_ptr_to_ctr(srcu_ctlp, scp); WARN_ON_ONCE(idx & ~0x1); @@ -758,8 +766,10 @@ srcu_read_delay(struct torture_random_state *rrsp, str= uct rt_read_seg *rtrsp) static void srcu_torture_read_unlock(int idx) { WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor= && (idx & ~0x1))); - if (reader_flavor & SRCU_READ_FLAVOR_FAST) + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8= ) >> 3)); + if (reader_flavor & SRCU_READ_FLAVOR_FAST) + srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8= ) >> 2)); if (reader_flavor & SRCU_READ_FLAVOR_NMI) srcu_read_unlock_nmisafe(srcu_ctlp, (idx & 0x2) >> 1); if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) || !(reader_flavor & SRCU_R= EAD_FLAVOR_ALL)) @@ -793,7 +803,7 @@ static int srcu_torture_down_read(void) WARN_ON_ONCE(idx & ~0x1); return idx; } - if (reader_flavor & SRCU_READ_FLAVOR_FAST) { + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) { scp =3D srcu_down_read_fast(srcu_ctlp); idx =3D __srcu_ptr_to_ctr(srcu_ctlp, scp); WARN_ON_ONCE(idx & ~0x1); @@ -806,7 +816,7 @@ static int srcu_torture_down_read(void) static void srcu_torture_up_read(int idx) { WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor= && (idx & ~0x1))); - if (reader_flavor & SRCU_READ_FLAVOR_FAST) + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) srcu_up_read_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >>= 3)); else if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) || !(reader_flavor & SRCU_READ_FLAVOR_ALL)) @@ -901,14 +911,16 @@ static struct rcu_torture_ops srcu_ops =3D { .no_pi_lock =3D IS_ENABLED(CONFIG_TINY_SRCU), .debug_objects =3D 1, .have_up_down =3D IS_ENABLED(CONFIG_TINY_SRCU) - ? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST, + ? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST_UPDOWN, .name =3D "srcu" }; =20 static void srcud_torture_init(void) { rcu_sync_torture_init(); - if (reader_flavor & SRCU_READ_FLAVOR_FAST) + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) + WARN_ON(init_srcu_struct_fast(&srcu_ctld)); + else if (reader_flavor & SRCU_READ_FLAVOR_FAST) WARN_ON(init_srcu_struct_fast(&srcu_ctld)); else WARN_ON(init_srcu_struct(&srcu_ctld)); @@ -953,7 +965,7 @@ static struct rcu_torture_ops srcud_ops =3D { .no_pi_lock =3D IS_ENABLED(CONFIG_TINY_SRCU), .debug_objects =3D 1, .have_up_down =3D IS_ENABLED(CONFIG_TINY_SRCU) - ? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST, + ? 0 : SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_FAST_UPDOWN, .name =3D "srcud" }; =20 --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D4D125485F; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; cv=none; b=SUyCr4lB2cTnVus6ZP6qMRpRNZmVhb/Bgle1vDiaa9Kw3qWbokvgXaoeAL1HGRlWhRUd86Tny4gpCOhdGyfjuIOETIuKWGgz3F208khU70wCut3CoKVb04d+5AfPBDgHbGj4yyZyfBxnIq8bEeORgGPBczZ0UWu8KTaIqWJdcjo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; c=relaxed/simple; bh=e8020D4IGboPZ/X1LsVXzWAnxHPKUMzBbRHImWEKJYU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=sVQZlPQgEjIQmhRyZzTniYxdDiQAhfIo2ws0p+LJoT6obNfO33EmOu13CVYB4e7FRX7VPFkejlTOpajc7iijXcIVPSlDt7mR/SzGiTDa+iAtsDCpPbfmpAK/Tdf82RBMLeQY8maHDwObTQf5jSglxyYBXym8nhol6gpAWYzYmdk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FPKlm9gG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FPKlm9gG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5FC32C4CEF7; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119879; bh=e8020D4IGboPZ/X1LsVXzWAnxHPKUMzBbRHImWEKJYU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FPKlm9gGdVL73JjwFhtHs3O/QVW70mnFFUYGUGjp1IXg7qe1EeNFuIxv8nqP5jmpR opUhk7j4SuNXb0IQzalKbg5t5slXXAe9RiGhmOIDj3uf/8VJ6QLWf5uMQHKFmm5FMq YlrblJuXvTbcPoPexwdsakPnLFhpAaZleQZz4V11QACH2Czyfp3ejBR/7qqXRQA7mc hQRHuxS3VDI9/QnDnuRT8pA1mvd9dupLqExIeK8pAPvht6JkUicw4BgMQ+7G3KZEXh oaBVY975NBJ1exoDU/WWnKGKTgcN0H9UU+eVz4QW9o1UTOabq4QR6fWegKiHU1P+E1 v5hOoqo+ASr+w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 9C626CE160B; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH 14/19] torture: Permit negative kvm.sh --kconfig numberic arguments Date: Sun, 2 Nov 2025 13:44:31 -0800 Message-Id: <20251102214436.3905633-14-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit loosens the kvm.sh script's regular expressions to permit negative-valued Kconfig options, for example: --kconfig CONFIG_CMDLINE_LOG_WRAP_IDEAL_LEN=3D-1 Signed-off-by: Paul E. McKenney --- tools/testing/selftests/rcutorture/bin/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/rcutorture/bin/kvm.sh b/tools/testing/= selftests/rcutorture/bin/kvm.sh index 617cba339d28..fff15821c44c 100755 --- a/tools/testing/selftests/rcutorture/bin/kvm.sh +++ b/tools/testing/selftests/rcutorture/bin/kvm.sh @@ -199,7 +199,7 @@ do fi ;; --kconfig|--kconfigs) - checkarg --kconfig "(Kconfig options)" $# "$2" '^\(#CHECK#\)\?CONFIG_[A-= Z0-9_]\+=3D\([ynm]\|[0-9]\+\|"[^"]*"\)\( \+\(#CHECK#\)\?CONFIG_[A-Z0-9_]\+= =3D\([ynm]\|[0-9]\+\|"[^"]*"\)\)* *$' '^error$' + checkarg --kconfig "(Kconfig options)" $# "$2" '^\(#CHECK#\)\?CONFIG_[A-= Z0-9_]\+=3D\([ynm]\|-\?[0-9]\+\|"[^"]*"\)\( \+\(#CHECK#\)\?CONFIG_[A-Z0-9_]= \+=3D\([ynm]\|-\?[0-9]\+\|"[^"]*"\)\)* *$' '^error$' TORTURE_KCONFIG_ARG=3D"`echo "$TORTURE_KCONFIG_ARG $2" | sed -e 's/^ *//= ' -e 's/ *$//'`" shift ;; --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADB39256C89; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; cv=none; b=ZQRX8c8SZYE7UtZz3TDBKHCKwtOvKbLnb/m4z98eyzvi0018Sq7xbut3IYX7vpimHtfjJTvserzx6W6PAuCyL4zP9C2Fbxf7NddiccE+9KxHa4lFJngO0RpgbRWdQXew+Z456jj/x3Eb/E/vQ4TN/490gTkk/Wf7Js7wpDqzeRg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119879; c=relaxed/simple; bh=7l0RKrGot9+VxpVYsZ6++nqGzBBkeqDPhqDWexz8syM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=UYtUoowCn58J7+q2tFHZPtXwhrD16hFB9kXfoHeGB75m/BKMCPtPDeCtLWxvXMYLL2AhzcUufuttamC+sm8W8W3BFrkNWislbzZ9pZl89NAklg4BZ2ahvStbH+s0dpNeWAzdJ287LdUirO01rAjT21tJU5BKO+GmuReKxnb5cWE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Or5spg9F; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Or5spg9F" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7BAA8C116D0; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119879; bh=7l0RKrGot9+VxpVYsZ6++nqGzBBkeqDPhqDWexz8syM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Or5spg9FXTtnkljSyHz35kGCS5pd13dn6JO2z1fl5fpSR+B7I7cz1b8eX1dGoOrX0 h7anRIbbexF2wq8fEAnok2ej415ircL1pLgOP0eSHWWBHcQI85U/E8nA+50PAPL1/F pH9denxBl38dz7xNhY2yfmuxZTCcWqxBR5Ci7BEtOMvPXT61Z38P8AxdrdcI1lsF6o VZJX065pjwY2H9sGrsB7aKLpNqUqVTKWGuBIP1Qf+3py6mw6OIGOx+FsSBcmLXPh/A jec+QsivF+oaI/GKFEwZwhZbW2NR3fft5K+t1uMBsnhxHaYMj30byaGuIPdzYC7+Z1 qHN3mAOO/zt/w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 9FAACCE167B; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 15/19] srcu: Create an SRCU-fast-updown API Date: Sun, 2 Nov 2025 13:44:32 -0800 Message-Id: <20251102214436.3905633-15-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit creates an SRCU-fast-updown API, including DEFINE_SRCU_FAST_UPDOWN(), DEFINE_STATIC_SRCU_FAST_UPDOWN(), __init_srcu_struct_fast_updown(), init_srcu_struct_fast_updown(), srcu_read_lock_fast_updown(), srcu_read_unlock_fast_updown(), __srcu_read_lock_fast_updown(), and __srcu_read_unlock_fast_updown(). These are initially identical to their SRCU-fast counterparts, but both SRCU-fast and SRCU-fast-updown will be optimized in different directions by later commits. SRCU-fast will lack any sort of srcu_down_read() and srcu_up_read() APIs, which will enable extremely efficient NMI safety. For its part, SRCU-fast-updown will not be NMI safe, which will enable reasonably efficient implementations of srcu_down_read_fast() and srcu_up_read_fast(). Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- include/linux/srcu.h | 77 +++++++++++++++++++++++++++++++++++++--- include/linux/srcutiny.h | 16 +++++++++ include/linux/srcutree.h | 55 ++++++++++++++++++++++++++-- kernel/rcu/srcutree.c | 39 +++++++++++++++++--- 4 files changed, 176 insertions(+), 11 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 1dd6812aabe7..1fbf475eae5e 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -28,6 +28,8 @@ struct srcu_struct; int __init_srcu_struct(struct srcu_struct *ssp, const char *name, struct l= ock_class_key *key); #ifndef CONFIG_TINY_SRCU int __init_srcu_struct_fast(struct srcu_struct *ssp, const char *name, str= uct lock_class_key *key); +int __init_srcu_struct_fast_updown(struct srcu_struct *ssp, const char *na= me, + struct lock_class_key *key); #endif // #ifndef CONFIG_TINY_SRCU =20 #define init_srcu_struct(ssp) \ @@ -44,12 +46,20 @@ int __init_srcu_struct_fast(struct srcu_struct *ssp, co= nst char *name, struct lo __init_srcu_struct_fast((ssp), #ssp, &__srcu_key); \ }) =20 +#define init_srcu_struct_fast_updown(ssp) \ +({ \ + static struct lock_class_key __srcu_key; \ + \ + __init_srcu_struct_fast_updown((ssp), #ssp, &__srcu_key); \ +}) + #define __SRCU_DEP_MAP_INIT(srcu_name) .dep_map =3D { .name =3D #srcu_name= }, #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 int init_srcu_struct(struct srcu_struct *ssp); #ifndef CONFIG_TINY_SRCU int init_srcu_struct_fast(struct srcu_struct *ssp); +int init_srcu_struct_fast_updown(struct srcu_struct *ssp); #endif // #ifndef CONFIG_TINY_SRCU =20 #define __SRCU_DEP_MAP_INIT(srcu_name) @@ -305,6 +315,46 @@ static inline struct srcu_ctr __percpu *srcu_read_lock= _fast(struct srcu_struct * return retval; } =20 +/** + * srcu_read_lock_fast_updown - register a new reader for an SRCU-fast-upd= own structure. + * @ssp: srcu_struct in which to register the new reader. + * + * Enter an SRCU read-side critical section, but for a light-weight + * smp_mb()-free reader. See srcu_read_lock() for more information. + * This function is compatible with srcu_down_read_fast(), but is not + * NMI-safe. + * + * For srcu_read_lock_fast_updown() to be used on an srcu_struct + * structure, that structure must have been defined using either + * DEFINE_SRCU_FAST_UPDOWN() or DEFINE_STATIC_SRCU_FAST_UPDOWN() on the one + * hand or initialized with init_srcu_struct_fast_updown() on the other. + * Such an srcu_struct structure cannot be passed to any non-fast-updown + * variant of srcu_read_{,un}lock() or srcu_{down,up}_read(). In kernels + * built with CONFIG_PROVE_RCU=3Dy, () will complain bitterly if you ignore + * this * restriction. + * + * Grace-period auto-expediting is disabled for SRCU-fast-updown + * srcu_struct structures because SRCU-fast-updown expedited grace periods + * invoke synchronize_rcu_expedited(), IPIs and all. If you need expedited + * SRCU-fast-updown grace periods, use synchronize_srcu_expedited(). + * + * The srcu_read_lock_fast_updown() function can be invoked only from + those contexts where RCU is watching, that is, from contexts where + it would be legal to invoke rcu_read_lock(). Otherwise, lockdep will + complain. + */ +static inline struct srcu_ctr __percpu *srcu_read_lock_fast_updown(struct = srcu_struct *ssp) +__acquires(ssp) +{ + struct srcu_ctr __percpu *retval; + + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast_updown()."); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN); + retval =3D __srcu_read_lock_fast_updown(ssp); + rcu_try_lock_acquire(&ssp->dep_map); + return retval; +} + /* * Used by tracing, cannot be traced and cannot call lockdep. * See srcu_read_lock_fast() for more information. @@ -335,8 +385,8 @@ static inline struct srcu_ctr __percpu *srcu_down_read_= fast(struct srcu_struct * { WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read= _fast()."); - srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); - return __srcu_read_lock_fast(ssp); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN); + return __srcu_read_lock_fast_updown(ssp); } =20 /** @@ -432,6 +482,23 @@ static inline void srcu_read_unlock_fast(struct srcu_s= truct *ssp, struct srcu_ct RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 +/** + * srcu_read_unlock_fast_updown - unregister a old reader from an SRCU-fas= t-updown structure. + * @ssp: srcu_struct in which to unregister the old reader. + * @scp: return value from corresponding srcu_read_lock_fast_updown(). + * + * Exit an SRCU-fast-updown read-side critical section. + */ +static inline void +srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __pe= rcpu *scp) __releases(ssp) +{ + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN); + srcu_lock_release(&ssp->dep_map); + __srcu_read_unlock_fast_updown(ssp, scp); + RCU_LOCKDEP_WARN(!rcu_is_watching(), + "RCU must be watching srcu_read_unlock_fast_updown()."); +} + /* * Used by tracing, cannot be traced and cannot call lockdep. * See srcu_read_unlock_fast() for more information. @@ -455,9 +522,9 @@ static inline void srcu_up_read_fast(struct srcu_struct= *ssp, struct srcu_ctr __ __releases(ssp) { WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); - srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); - __srcu_read_unlock_fast(ssp, scp); - RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_up_read_f= ast()."); + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST_UPDOWN); + __srcu_read_unlock_fast_updown(ssp, scp); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_up_read_f= ast_updown()."); } =20 /** diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 1ecc3393fb26..e0698024667a 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -50,13 +50,18 @@ void srcu_drive_gp(struct work_struct *wp); #define DEFINE_SRCU_FAST(name) DEFINE_SRCU(name) #define DEFINE_STATIC_SRCU_FAST(name) \ static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) +#define DEFINE_SRCU_FAST_UPDOWN(name) DEFINE_SRCU(name) +#define DEFINE_STATIC_SRCU_FAST_UPDOWN(name) \ + static struct srcu_struct name =3D __SRCU_STRUCT_INIT(name, name, name, n= ame) =20 // Dummy structure for srcu_notifier_head. struct srcu_usage { }; #define __SRCU_USAGE_INIT(name) { } #define __init_srcu_struct_fast __init_srcu_struct +#define __init_srcu_struct_fast_updown __init_srcu_struct #ifndef CONFIG_DEBUG_LOCK_ALLOC #define init_srcu_struct_fast init_srcu_struct +#define init_srcu_struct_fast_updown init_srcu_struct #endif // #ifndef CONFIG_DEBUG_LOCK_ALLOC =20 void synchronize_srcu(struct srcu_struct *ssp); @@ -100,6 +105,17 @@ static inline void __srcu_read_unlock_fast(struct srcu= _struct *ssp, struct srcu_ __srcu_read_unlock(ssp, __srcu_ptr_to_ctr(ssp, scp)); } =20 +static inline struct srcu_ctr __percpu *__srcu_read_lock_fast_updown(struc= t srcu_struct *ssp) +{ + return __srcu_ctr_to_ptr(ssp, __srcu_read_lock(ssp)); +} + +static inline +void __srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_c= tr __percpu *scp) +{ + __srcu_read_unlock(ssp, __srcu_ptr_to_ctr(ssp, scp)); +} + static inline void synchronize_srcu_expedited(struct srcu_struct *ssp) { synchronize_srcu(ssp); diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 6080a9094618..d6f978b50472 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -199,8 +199,15 @@ struct srcu_struct { * * See include/linux/percpu-defs.h for the rules on per-CPU variables. * - * DEFINE_SRCU_FAST() creates an srcu_struct and associated structures - * whose readers must be of the SRCU-fast variety. + * DEFINE_SRCU_FAST() and DEFINE_STATIC_SRCU_FAST create an srcu_struct + * and associated structures whose readers must be of the SRCU-fast variet= y. + * DEFINE_SRCU_FAST_UPDOWN() and DEFINE_STATIC_SRCU_FAST_UPDOWN() create + * an srcu_struct and associated structures whose readers must be of the + * SRCU-fast-updown variety. The key point (aside from error checking) wi= th + * both varieties is that the grace periods must use synchronize_rcu() + * instead of smp_mb(), and given that the first (for example) + * srcu_read_lock_fast() might race with the first synchronize_srcu(), + * this different must be specified at initialization time. */ #ifdef MODULE # define __DEFINE_SRCU(name, fast, is_static) \ @@ -221,6 +228,10 @@ struct srcu_struct { #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, 0, static) #define DEFINE_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST,= /* not static */) #define DEFINE_STATIC_SRCU_FAST(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR= _FAST, static) +#define DEFINE_SRCU_FAST_UPDOWN(name) __DEFINE_SRCU(name, SRCU_READ_FLAVOR= _FAST_UPDOWN, \ + /* not static */) +#define DEFINE_STATIC_SRCU_FAST_UPDOWN(name) \ + __DEFINE_SRCU(name, SRCU_READ_FLAVOR_FAST_UPDOWN, static) =20 int __srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp); void synchronize_srcu_expedited(struct srcu_struct *ssp); @@ -305,6 +316,46 @@ __srcu_read_unlock_fast(struct srcu_struct *ssp, struc= t srcu_ctr __percpu *scp) atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RC= U reader. } =20 +/* + * Counts the new reader in the appropriate per-CPU element of the + * srcu_struct. Returns a pointer that must be passed to the matching + * srcu_read_unlock_fast_updown(). This type of reader is compatible + * with srcu_down_read_fast() and srcu_up_read_fast(). + * + * See the __srcu_read_lock_fast() comment for more details. + */ +static inline +struct srcu_ctr __percpu notrace *__srcu_read_lock_fast_updown(struct srcu= _struct *ssp) +{ + struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); + + if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader. + else + atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); // Y, and implicit RCU = reader. + barrier(); /* Avoid leaking the critical section. */ + return scp; +} + +/* + * Removes the count for the old reader from the appropriate + * per-CPU element of the srcu_struct. Note that this may well be a + * different CPU than that which was incremented by the corresponding + * srcu_read_lock_fast(), but it must be within the same task. + * + * Please see the __srcu_read_lock_fast() function's header comment for + * information on implicit RCU readers and NMI safety. + */ +static inline void notrace +__srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __= percpu *scp) +{ + barrier(); /* Avoid leaking the critical section. */ + if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + this_cpu_inc(scp->srcu_unlocks.counter); // Z, and implicit RCU reader. + else + atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RC= U reader. +} + void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); =20 // Record SRCU-reader usage type only for CONFIG_PROVE_RCU=3Dy kernels. diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 2f8aa280911e..ea3f128de06f 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -309,13 +309,24 @@ int __init_srcu_struct_fast(struct srcu_struct *ssp, = const char *name, struct lo } EXPORT_SYMBOL_GPL(__init_srcu_struct_fast); =20 +int __init_srcu_struct_fast_updown(struct srcu_struct *ssp, const char *na= me, + struct lock_class_key *key) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST_UPDOWN; + return __init_srcu_struct_common(ssp, name, key); +} +EXPORT_SYMBOL_GPL(__init_srcu_struct_fast_updown); + #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /** * init_srcu_struct - initialize a sleep-RCU structure * @ssp: structure to initialize. * - * Must invoke this on a given srcu_struct before passing that srcu_struct + * Use this in place of DEFINE_SRCU() and DEFINE_STATIC_SRCU() + * for non-static srcu_struct structures that are to be passed to + * srcu_read_lock(), srcu_read_lock_nmisafe(), and friends. It is necessa= ry + * to invoke this on a given srcu_struct before passing that srcu_struct * to any other function. Each srcu_struct represents a separate domain * of SRCU protection. */ @@ -330,9 +341,11 @@ EXPORT_SYMBOL_GPL(init_srcu_struct); * init_srcu_struct_fast - initialize a fast-reader sleep-RCU structure * @ssp: structure to initialize. * - * Must invoke this on a given srcu_struct before passing that srcu_struct - * to any other function. Each srcu_struct represents a separate domain - * of SRCU protection. + * Use this in place of DEFINE_SRCU_FAST() and DEFINE_STATIC_SRCU_FAST() + * for non-static srcu_struct structures that are to be passed to + * srcu_read_lock_fast() and friends. It is necessary to invoke this on a + * given srcu_struct before passing that srcu_struct to any other function. + * Each srcu_struct represents a separate domain of SRCU protection. */ int init_srcu_struct_fast(struct srcu_struct *ssp) { @@ -341,6 +354,24 @@ int init_srcu_struct_fast(struct srcu_struct *ssp) } EXPORT_SYMBOL_GPL(init_srcu_struct_fast); =20 +/** + * init_srcu_struct_fast_updown - initialize a fast-reader up/down sleep-R= CU structure + * @ssp: structure to initialize. + * + * Use this function in place of DEFINE_SRCU_FAST_UPDOWN() and + * DEFINE_STATIC_SRCU_FAST_UPDOWN() for non-static srcu_struct + * structures that are to be passed to srcu_read_lock_fast_updown(), + * srcu_down_read_fast(), and friends. It is necessary to invoke this on a + * given srcu_struct before passing that srcu_struct to any other function. + * Each srcu_struct represents a separate domain of SRCU protection. + */ +int init_srcu_struct_fast_updown(struct srcu_struct *ssp) +{ + ssp->srcu_reader_flavor =3D SRCU_READ_FLAVOR_FAST_UPDOWN; + return init_srcu_struct_fields(ssp, false); +} +EXPORT_SYMBOL_GPL(init_srcu_struct_fast_updown); + #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 /* --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED15C260580; Sun, 2 Nov 2025 21:44:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119881; cv=none; b=bfwGEBBzwX++YqKpP2nN5eu8qjONv+IGb2LAfxZ21s3AlrZ1yh1+1H4YdqiigZhrqkuPvbVPFQvW7y6PIaYO8gLH5KqVsWuvKhzAlU3mmpmxA3KUY2RQIDKFCYFe027fXsjgEFI1unAImeVaniijC1fLNEYUI5eBwSbHdIQ8joA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119881; c=relaxed/simple; bh=zFl0SXTsIUbs2MDRVmkN9CUYpYD2KuyK1hQqp11YVqM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=n+YCXyF9tEfPrZshJsosRSzY3Imfn+2YM7bZb8zTuJ1QJwTRqietTd5VQ75/TKUG8XG8paQfN/XYoN+PQTJJn4lH4RAvip7IZl0NmI9dvpnd4hz9oL5N3l3M6EO+ZcP0lBFWECEm/aEKN1Jbx2x3Po0Xk/62xzV1Vjv/9QcWGEg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=D82gWrNG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="D82gWrNG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A41B1C4CEF7; Sun, 2 Nov 2025 21:44:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119880; bh=zFl0SXTsIUbs2MDRVmkN9CUYpYD2KuyK1hQqp11YVqM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=D82gWrNGxstlUCU9VV2MvFrp7nU7tasVHLoYKTIF33XrY/YeGkGAHH+BJlWdZ2AVh BLFibrsVZftE10OqB8eCRi4Ns8Wy1eFKCs4TYIW8qYSwyeb0HA+Iukccp/2V6r6o35 zXncR6fnfLR9Dt5rEaB3n/LMiWZb+A7lu+kGMGAaMRB+GIUc93oK9Vi/O3SqtwZtYb USdDEfjqg8FZfhkYaYu1pdiYCB9PWQSRELF/lR3hehXO4UURlvAFcQeYXUw1R6jO3E QzQq+0m321s7Oy5gXczjbYUhJVZ4uDnr/y+ehrT7msJpHfFlsChiM7TWLLgz0PnUiT DYWGTepITycXw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A2DF0CE1699; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Andrii Nakryiko , Alexei Starovoitov , Peter Zijlstra , bpf@vger.kernel.org Subject: [PATCH 16/19] rcutorture: Test SRCU-fast separately from SRCU-fast-updown Date: Sun, 2 Nov 2025 13:44:33 -0800 Message-Id: <20251102214436.3905633-16-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit causes rcutorture to use the 0x4 value of the reader_flavor module parameter to test SRCU-fast. The 0x8 value tests SRCU-fast-updown. However, most SRCU-fast testing will be via the RCU Tasks Trace wrappers. [ paulmck: Apply s/0x8/0x4/ missing change per Boqun Feng feedback. ] Signed-off-by: Paul E. McKenney Cc: Andrii Nakryiko Cc: Alexei Starovoitov Cc: Peter Zijlstra Cc: --- kernel/rcu/rcutorture.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 587b28258b6e..4a9b6866c0cc 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -693,6 +693,7 @@ static struct rcu_torture_ops rcu_busted_ops =3D { =20 DEFINE_STATIC_SRCU(srcu_ctl); DEFINE_STATIC_SRCU_FAST(srcu_ctlf); +DEFINE_STATIC_SRCU_FAST_UPDOWN(srcu_ctlfud); static struct srcu_struct srcu_ctld; static struct srcu_struct *srcu_ctlp =3D &srcu_ctl; static struct rcu_torture_ops srcud_ops; @@ -703,7 +704,7 @@ static void srcu_torture_init(void) if (reader_flavor & SRCU_READ_FLAVOR_FAST) srcu_ctlp =3D &srcu_ctlf; if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) - srcu_ctlp =3D &srcu_ctlf; + srcu_ctlp =3D &srcu_ctlfud; } =20 static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) @@ -736,7 +737,7 @@ static int srcu_torture_read_lock(void) ret +=3D idx << 2; } if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) { - scp =3D srcu_read_lock_fast(srcu_ctlp); + scp =3D srcu_read_lock_fast_updown(srcu_ctlp); idx =3D __srcu_ptr_to_ctr(srcu_ctlp, scp); WARN_ON_ONCE(idx & ~0x1); ret +=3D idx << 3; @@ -767,9 +768,10 @@ static void srcu_torture_read_unlock(int idx) { WARN_ON_ONCE((reader_flavor && (idx & ~reader_flavor)) || (!reader_flavor= && (idx & ~0x1))); if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) - srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8= ) >> 3)); + srcu_read_unlock_fast_updown(srcu_ctlp, + __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8) >> 3)); if (reader_flavor & SRCU_READ_FLAVOR_FAST) - srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x8= ) >> 2)); + srcu_read_unlock_fast(srcu_ctlp, __srcu_ctr_to_ptr(srcu_ctlp, (idx & 0x4= ) >> 2)); if (reader_flavor & SRCU_READ_FLAVOR_NMI) srcu_read_unlock_nmisafe(srcu_ctlp, (idx & 0x2) >> 1); if ((reader_flavor & SRCU_READ_FLAVOR_NORMAL) || !(reader_flavor & SRCU_R= EAD_FLAVOR_ALL)) @@ -919,7 +921,7 @@ static void srcud_torture_init(void) { rcu_sync_torture_init(); if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) - WARN_ON(init_srcu_struct_fast(&srcu_ctld)); + WARN_ON(init_srcu_struct_fast_updown(&srcu_ctld)); else if (reader_flavor & SRCU_READ_FLAVOR_FAST) WARN_ON(init_srcu_struct_fast(&srcu_ctld)); else --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB07D25949A; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119880; cv=none; b=A343iKnfkZUN1SvVl800zdC5P84VrzkB3zevoJt017aUBrGRLv4F4841bmYSc9zYMYGkWQ3b3E0S9dokaR8TLPKhv61Vt20Gw0GM8HIczMJLY2CF/mVjzX5r+8wAz/Iimg74SS7UfwLM7j6VISvk70pOawz9giO7CLSAL/zkRLw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119880; c=relaxed/simple; bh=R4kFwOOFYwrXB688uqk8xRjU0CSAKW/oggKTVNMYKqU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=i8+P1i9c6CINaOli3CpIrH8gi072+6nrQKnOQxZlDMvh8kxs+6TLUnBHOMqC4/3mtYX2+dOnLz+n5HeHyqL7RqENwCRmZyI5ZiLU4JsNWnNzdRXJaLLayjqZZCNB/QhzfqqHRxWCjmI/r6Cw+DSfMKaNrI9n+0z2oz3loX+EdKQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HoNbtz9G; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HoNbtz9G" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BF40DC113D0; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119879; bh=R4kFwOOFYwrXB688uqk8xRjU0CSAKW/oggKTVNMYKqU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HoNbtz9GmnNxGwEWhNeSECkfCd8sjgQCadJgzAA8UCnfQpFu6H3nb5u7Mu09hr3/H oDptY4r0jGPfbbBmnOWcIHnRO1ND4GZKuFryKo1F7WcjLo0XxfZhj14m4j8ja1+clz Kj4hMfmB7dfSCRkBumoIUpiiehB85RLARaOBJKGtvIgumrCsbHtZgZE9wvsyQxjkI0 IfFxwva1a3PCHz8tWgLhuesiBr5xEXhVSSpSzNBrfnacmGtLPk45VI3oUECeKwQw7y +DdGvbjaNmNsAwmZicXIeDmgfOx9HWlX5sdGAu8hSCrgjGLdgsej49rRGAo973lskN 2qr32duqsG/Xw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A58B2CE16C7; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Catalin Marinas , Will Deacon , Mark Rutland , Mathieu Desnoyers , Sebastian Andrzej Siewior , linux-arm-kernel@lists.infradead.org, bpf@vger.kernel.org Subject: [PATCH 17/19] srcu: Optimize SRCU-fast-updown for arm64 Date: Sun, 2 Nov 2025 13:44:34 -0800 Message-Id: <20251102214436.3905633-17-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some arm64 platforms have slow per-CPU atomic operations, for example, the Neoverse V2. This commit therefore moves SRCU-fast from per-CPU atomic operations to interrupt-disabled non-read-modify-write-atomic atomic_read()/atomic_set() operations. This works because SRCU-fast-updown is not invoked from read-side primitives, which means that if srcu_read_unlock_fast() NMI handlers. This means that srcu_read_lock_fast_updown() and srcu_read_unlock_fast_updown() can exclude themselves and each other This reduces the overhead of calls to srcu_read_lock_fast_updown() and srcu_read_unlock_fast_updown() from about 100ns to about 12ns on an ARM Neoverse V2. Although this is not excellent compared to about 2ns on x86, it sure beats 100ns. This command was used to measure the overhead: tools/testing/selftests/rcutorture/bin/kvm.sh --torture refscale --allcpus = --duration 5 --configs NOPREEMPT --kconfig "CONFIG_NR_CPUS=3D64 CONFIG_TASK= S_TRACE_RCU=3Dy" --bootargs "refscale.loops=3D100000 refscale.guest_os_dela= y=3D5 refscale.nreaders=3D64 refscale.holdoff=3D30 torture.disable_onoff_at= _boot refscale.scale_type=3Dsrcu-fast-updown refscale.verbose_batched=3D8 t= orture.verbose_sleep_frequency=3D8 torture.verbose_sleep_duration=3D8 refsc= ale.nruns=3D100" --trust-make Signed-off-by: Paul E. McKenney Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: Cc: --- include/linux/srcutree.h | 56 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 5 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index d6f978b50472..70560dc4636c 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -253,6 +253,34 @@ static inline struct srcu_ctr __percpu *__srcu_ctr_to_= ptr(struct srcu_struct *ss return &ssp->sda->srcu_ctrs[idx]; } =20 +/* + * Non-atomic manipulation of SRCU lock counters. + */ +static inline struct srcu_ctr __percpu notrace *__srcu_read_lock_fast_na(s= truct srcu_struct *ssp) +{ + atomic_long_t *scnp; + struct srcu_ctr __percpu *scp; + + lockdep_assert_preemption_disabled(); + scp =3D READ_ONCE(ssp->srcu_ctrp); + scnp =3D raw_cpu_ptr(&scp->srcu_locks); + atomic_long_set(scnp, atomic_long_read(scnp) + 1); + return scp; +} + +/* + * Non-atomic manipulation of SRCU unlock counters. + */ +static inline void notrace +__srcu_read_unlock_fast_na(struct srcu_struct *ssp, struct srcu_ctr __perc= pu *scp) +{ + atomic_long_t *scnp; + + lockdep_assert_preemption_disabled(); + scnp =3D raw_cpu_ptr(&scp->srcu_unlocks); + atomic_long_set(scnp, atomic_long_read(scnp) + 1); +} + /* * Counts the new reader in the appropriate per-CPU element of the * srcu_struct. Returns a pointer that must be passed to the matching @@ -327,12 +355,23 @@ __srcu_read_unlock_fast(struct srcu_struct *ssp, stru= ct srcu_ctr __percpu *scp) static inline struct srcu_ctr __percpu notrace *__srcu_read_lock_fast_updown(struct srcu= _struct *ssp) { - struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); + struct srcu_ctr __percpu *scp; =20 - if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + if (IS_ENABLED(CONFIG_ARM64) && IS_ENABLED(CONFIG_ARM64_USE_LSE_PERCPU_AT= OMICS)) { + unsigned long flags; + + local_irq_save(flags); + scp =3D __srcu_read_lock_fast_na(ssp); + local_irq_restore(flags); /* Avoids leaking the critical section. */ + return scp; + } + + scp =3D READ_ONCE(ssp->srcu_ctrp); + if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) { this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader. - else + } else { atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); // Y, and implicit RCU = reader. + } barrier(); /* Avoid leaking the critical section. */ return scp; } @@ -350,10 +389,17 @@ static inline void notrace __srcu_read_unlock_fast_updown(struct srcu_struct *ssp, struct srcu_ctr __= percpu *scp) { barrier(); /* Avoid leaking the critical section. */ - if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + if (IS_ENABLED(CONFIG_ARM64)) { + unsigned long flags; + + local_irq_save(flags); + __srcu_read_unlock_fast_na(ssp, scp); + local_irq_restore(flags); + } else if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) { this_cpu_inc(scp->srcu_unlocks.counter); // Z, and implicit RCU reader. - else + } else { atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RC= U reader. + } } =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EDF37260583; Sun, 2 Nov 2025 21:44:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119881; cv=none; b=dVJwu2IEFJ7Dnx3zwphPwX/dDifK7+aEY2RsrmgTuXmS0brN1bDDlC+Xj+RAC0472EOczunc6qY2g/m94KcM+6zs5De8TIwaQcXbvLcDEZoyMsT0RYgu3qkdWpZMm2jgOVHsF5olU929sOCH1RI76BPL7lU2ZOZsJyeR8M38CeI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119881; c=relaxed/simple; bh=rZXvfrA3YGviZHzZMYqMHXTK7X68+gQX9VG+uQe++ZM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RRRwlqtkgC7po2gfv6dunZDiwoDy9+pK4Vfjwy/J9YQ1eLxA4mYffKuumSdhnm0yyEhuidd5Pq/ASLpnjcoKmlh5XW1AyyM8y46WYHGwjFaakRUBAwHvoUI/r2mGSbvyqz2v4fIhXMONhB2dCnsnwmkKlvx0IHo+fx3Um+xdxhQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=H2fw3VdO; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="H2fw3VdO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BCFE0C116D0; Sun, 2 Nov 2025 21:44:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119880; bh=rZXvfrA3YGviZHzZMYqMHXTK7X68+gQX9VG+uQe++ZM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H2fw3VdOZzJ3lx/k31T7AdEtUPtF3EaPp5nk7XbSiaHqrmOfOqhvyEhs7IS1orJyz K4Ifyqtsaqq9VwDUE+J62RCtf8KIsNtBMon+Urt/UsuAGPu6oQL07JSUSfR0z4IVfa zuAnUw/MN7lpnQrE4hJrISU6GI1E0p6Wws1bJ6cQzHBydGHEIPenfBFrxw+8VYE27S +ZOdO4fgXBpFKbEFfe9buSD06zCCP5EtskfVsQJP36/SogSeT8Qk/vB4EYj087TvwC g6alcrj5qE9DIXs0YVAA2ovQ3a5BPcAi/0Dmthy7QH9JSqYOeHuPM0rz5ZA4f687H4 forJg0cQ0ovfQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A80C6CE16CA; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH 18/19] rcutorture: Make srcu{,d}_torture_init() announce the SRCU type Date: Sun, 2 Nov 2025 13:44:35 -0800 Message-Id: <20251102214436.3905633-18-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit causes rcutorture's srcu_torture_init() and srcud_torture_init() functions to announce on the console log which variant of SRCU is being tortured, for example: "torture: srcud_torture_init fast SRCU". Signed-off-by: Paul E. McKenney --- kernel/rcu/rcutorture.c | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 4a9b6866c0cc..820c5e5b106b 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -701,10 +701,20 @@ static struct rcu_torture_ops srcud_ops; static void srcu_torture_init(void) { rcu_sync_torture_init(); - if (reader_flavor & SRCU_READ_FLAVOR_FAST) + if (!reader_flavor || (reader_flavor & SRCU_READ_FLAVOR_NORMAL)) { + VERBOSE_TOROUT_STRING("srcu_torture_init normal SRCU"); + } + if (reader_flavor & SRCU_READ_FLAVOR_NMI) { + VERBOSE_TOROUT_STRING("srcu_torture_init NMI-safe SRCU"); + } + if (reader_flavor & SRCU_READ_FLAVOR_FAST) { srcu_ctlp =3D &srcu_ctlf; - if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) + VERBOSE_TOROUT_STRING("srcu_torture_init fast SRCU"); + } + if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) { srcu_ctlp =3D &srcu_ctlfud; + VERBOSE_TOROUT_STRING("srcu_torture_init fast-up/down SRCU"); + } } =20 static void srcu_get_gp_data(int *flags, unsigned long *gp_seq) @@ -920,12 +930,21 @@ static struct rcu_torture_ops srcu_ops =3D { static void srcud_torture_init(void) { rcu_sync_torture_init(); - if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) - WARN_ON(init_srcu_struct_fast_updown(&srcu_ctld)); - else if (reader_flavor & SRCU_READ_FLAVOR_FAST) + if (!reader_flavor || (reader_flavor & SRCU_READ_FLAVOR_NORMAL)) { + WARN_ON(init_srcu_struct(&srcu_ctld)); + VERBOSE_TOROUT_STRING("srcud_torture_init normal SRCU"); + } else if (reader_flavor & SRCU_READ_FLAVOR_NMI) { + WARN_ON(init_srcu_struct(&srcu_ctld)); + VERBOSE_TOROUT_STRING("srcud_torture_init NMI-safe SRCU"); + } else if (reader_flavor & SRCU_READ_FLAVOR_FAST) { WARN_ON(init_srcu_struct_fast(&srcu_ctld)); - else + VERBOSE_TOROUT_STRING("srcud_torture_init fast SRCU"); + } else if (reader_flavor & SRCU_READ_FLAVOR_FAST_UPDOWN) { + WARN_ON(init_srcu_struct_fast_updown(&srcu_ctld)); + VERBOSE_TOROUT_STRING("srcud_torture_init fast-up/down SRCU"); + } else { WARN_ON(init_srcu_struct(&srcu_ctld)); + } srcu_ctlp =3D &srcu_ctld; } =20 @@ -2440,6 +2459,9 @@ static DEFINE_TORTURE_RANDOM_PERCPU(rcu_torture_timer= _rand); */ static void rcu_torture_timer(struct timer_list *unused) { + WARN_ON_ONCE(!in_serving_softirq()); + WARN_ON_ONCE(in_hardirq()); + WARN_ON_ONCE(in_nmi()); atomic_long_inc(&n_rcu_torture_timers); (void)rcu_torture_one_read(this_cpu_ptr(&rcu_torture_timer_rand), -1); =20 --=20 2.40.1 From nobody Mon Feb 9 04:45:38 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 58E2425D1E9; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119880; cv=none; b=Bhzu9wXM5CPmKOCrYdu4RwVDPMxrF8j39S7iEcoIeruWPIVbfySKu8Cd9X5Ik8NeragMp0TCJ+JxL3ittvaiWLQlISE/gqndSzv2w5YIB55YXQzgFTtTHG0hJQ7E0YxNDpDXekllg+kWvO8j8Fr1jcq1qM8329MYYTL6zDbNw8I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762119880; c=relaxed/simple; bh=/aBUUqkdfQzZ+qxvU0KljxF2xX83n5VIIQG+jKSVdFg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CYQgeO9N5nv7EiIFd5WAHNTBoMg6789MEPwbQMYh25PmpnQ8fFcL3oCCozptPVES4EqG/h/gvFnqe9KkZvSZSvi8w7vB6wheuLP5uNgo1Gpp4MlXk19oJ1l99QFMYw5/ssnKGDrScZS9DDri3Ymfi7Fxid/x0r+iogo8u1vxThM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=b1EMTg8l; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="b1EMTg8l" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D540BC19421; Sun, 2 Nov 2025 21:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1762119879; bh=/aBUUqkdfQzZ+qxvU0KljxF2xX83n5VIIQG+jKSVdFg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b1EMTg8lMaI3kCLR83XoezsSTL6mrs8DEEBygsr+3BXTpFSs3kLPOVCML1iT8EATe yp6jClEyIv+rwQmjWwSBVJ9HRudvPCrEznxW2GU59oRlNVAl+jh/bEr+a9xt+qYrfn XQLngiIwZpC8d6xcvz+vQomUMCAVJAWT9MfbsxXLr4AKPv4Ow1YcE6xLaoW9g10aRv LDNFcEStfXlPpZYU7hgqmhY0riU3dQt19CNZbFXKEuoenI+t+7um0bmbbeDQ2GdR1j t2L+urzjZUpI7Q576QcRrTU4m6o86+kgSwn/YAxexaecZX4XFb5y0lkBzfKqwYZUmE IAUiGqdnalnbw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id AA4BCCE16DB; Sun, 2 Nov 2025 13:44:37 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH 19/19] rcutorture: Remove redundant rcutorture_one_extend() from rcu_torture_one_read() Date: Sun, 2 Nov 2025 13:44:36 -0800 Message-Id: <20251102214436.3905633-19-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> References: <082fb8ba-91b8-448e-a472-195eb7b282fd@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit removes a harmless but potentially confusing invocation of rcutorture_one_extend() within rcu_torture_one_read(). The immediately preceding call to rcu_torture_one_read_start() already does this cleanup, and the other call to rcu_torture_one_read_start() already relies on this. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcutorture.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 820c5e5b106b..423c4bddd17c 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -2440,10 +2440,8 @@ static bool rcu_torture_one_read(struct torture_rand= om_state *trsp, long myid) newstate =3D rcutorture_extend_mask(rtors.readstate, trsp); WARN_ON_ONCE(newstate & RCUTORTURE_RDR_UPDOWN); rcutorture_one_extend(&rtors.readstate, newstate, trsp, rtors.rtrsp++); - if (!rcu_torture_one_read_start(&rtors, trsp, myid)) { - rcutorture_one_extend(&rtors.readstate, 0, trsp, rtors.rtrsp); + if (!rcu_torture_one_read_start(&rtors, trsp, myid)) return false; - } rtors.rtrsp =3D rcutorture_loop_extend(&rtors.readstate, trsp, rtors.rtrs= p); rcu_torture_one_read_end(&rtors, trsp); return true; --=20 2.40.1