From nobody Thu Oct 2 09:21:04 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3BEE2882CE; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191207; cv=none; b=WEvUMNfjmRJU52//fXsBrKMRq+PRSSA9pVwKsQ5kBhAZX6yBKIvZTdx/LfXDYNrXIcH/mw9PC0jQHERUeLzOGu1BkbKG0IUnYOozk9vJP6WsA0Ye4U4YIkJir3RXvpDNAvBFEYIEpfrA7V6Nu3fFHkqGJjWrHILo0uKPuRta9ys= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191207; c=relaxed/simple; bh=pvHoXIq/XZZkE26WZ9PeQrCmEPRe5uY95ng191EMoZw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ixZ+B6FVSmVZ8Eb2tVqjBOnhtDvGjG0Zu+iZhGSPtNbbAV1iJlRiwNsR5cxQOptLFTeEVVDMiIXi6G3+rexpzU2wdc07CeZ0MKits4xEQlEFwqx7QMm9ZI1mdLB9yT5hQruwr0DNmdG5Ns/8hgPCbjzCz5h48DAUiX4WlvGQmpI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jrccNUEn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jrccNUEn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 67114C4CEE7; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758191207; bh=pvHoXIq/XZZkE26WZ9PeQrCmEPRe5uY95ng191EMoZw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jrccNUEnjRHl9F8GDoKa1N7ClgGg/k52t4ZIFLCJsgVvQodJ7XqcwB1tCju/s6bBP syJQj+LS13u3dJXauOPTERhSy3GtobNeAzPHwOPhrkwiaWcxW/O32zSgyRz8vqK33s XGID4k2J13SxEaLCZl11tNOmhoHFOvz9jeeb046Uobepzy7K6JPaPc6UlTk+uixdd2 Rb5jHDozj3OhMMr1LH4G+PPEkQhT92vb1tlUr3C9un8FgwCkFNMHNqIfbek3aoWSoG QdSYCJY/FQxfsCxQZTubHz4tPTIfChWF8bMXPfOtBo4tqxiVF2wgKDoQL9gBI1Aw6L 5mjxG66EZn0mQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E31D0CE0B32; Thu, 18 Sep 2025 03:26:46 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Joel Fernandes , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 1/6] srcu: Move rcu_is_watching() checks to srcu_read_{,un}lock_fast() Date: Thu, 18 Sep 2025 03:26:41 -0700 Message-Id: <20250918102646.2592821-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> References: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The rcu_is_watching() warnings are currently in the SRCU-tree implementations of __srcu_read_lock_fast() and __srcu_read_unlock_fast(). However, this makes it difficult to create _notrace variants of srcu_read_lock_fast() and srcu_read_unlock_fast(). This commit therefore moves these checks to srcu_read_lock_fast(), srcu_read_unlock_fast(), srcu_down_read_fast(), and srcu_up_read_fast(). Signed-off-by: Paul E. McKenney Reviewed-by: Joel Fernandes Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 4 ++++ include/linux/srcutree.h | 2 -- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index f179700fecafb8..478c73d067f7d3 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -275,6 +275,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast(struct srcu_struct * { struct srcu_ctr __percpu *retval; =20 + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast()."); srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); retval =3D __srcu_read_lock_fast(ssp); rcu_try_lock_acquire(&ssp->dep_map); @@ -295,6 +296,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast(struct srcu_struct * static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_st= ruct *ssp) __acquires(ssp) { WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read= _fast()."); srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); return __srcu_read_lock_fast(ssp); } @@ -389,6 +391,7 @@ static inline void srcu_read_unlock_fast(struct srcu_st= ruct *ssp, struct srcu_ct srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); srcu_lock_release(&ssp->dep_map); __srcu_read_unlock_fast(ssp, scp); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 /** @@ -405,6 +408,7 @@ static inline void srcu_up_read_fast(struct srcu_struct= *ssp, struct srcu_ctr __ WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); __srcu_read_unlock_fast(ssp, scp); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_up_read_f= ast()."); } =20 /** diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index bf44d8d1e69eab..043b5a67ef71eb 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -244,7 +244,6 @@ static inline struct srcu_ctr __percpu *__srcu_read_loc= k_fast(struct srcu_struct { struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); =20 - RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast()."); if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) this_cpu_inc(scp->srcu_locks.counter); /* Y */ else @@ -275,7 +274,6 @@ static inline void __srcu_read_unlock_fast(struct srcu_= struct *ssp, struct srcu_ this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ else atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); /* Z */ - RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); --=20 2.40.1 From nobody Thu Oct 2 09:21:04 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3B6E287242; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191207; cv=none; b=r7QoC4lHtLTQ2FMmYd1JGqVyqPUpywX0U5BNlPKsdSWnrO+3+XPfLkcrMw8lygHco8FU+dxU8niQpcUxfTgCR+g4oxKUvqayMRL283MGR6hDT0QQbid98UJK74F7EfEl41NCJLF9VhEllGdV7V/e9+ORrqxwAAfHQ+BJUlgF3tY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191207; c=relaxed/simple; bh=jDxGNAMMOCyH7qVFbWf8yWfHDDI/sC4nlnOnRxPII+E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=krN2GeiaXk6UG5rbrmu7jMs3kzwDLrZTkr44EtW3sW2rWwRLHcW4Y2JDf3Xf9fdRq82LOt9Fj1is7ZdFm5RnJstyLa7xb5jLRGHZ5mSXlaIBiY3oiX0JX7XX5ThfkH26KMmP6KkR1wilZ+2gLy+0qdAyPrdOrytfDWUlAmIUXrA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UTytjL4A; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UTytjL4A" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 704C0C4CEEB; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758191207; bh=jDxGNAMMOCyH7qVFbWf8yWfHDDI/sC4nlnOnRxPII+E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UTytjL4A4BgUDBzqbbX7my/OQWfUASU6stKhOT01kx3j1TDlEH3jwA5Y/UsmZZSy5 hHAaO5iQt+mYr6KJJGMX/B7JnimnZsZG1KB4A3hnJptRQwNCkENgSsxOmKF+7fUp8E 1/qfsIfm5ko9e+3dBAiXB52lcsg7tIsV5ZhVgBjUv14NV+xgdzPFMpX4A8Qk5B4WHQ DtyzKd+n6Yzr9yd+jCz6EyvEOojjRzsU2ue1LPSw8CwxuI38bhImvGsCImv7ONJkHr mw7lDU7VA2RsodEUyPJW3tFiQUL44JvVqy0Ug0ns3b7MVxGPuyxdpy1d8jqZFd3IlZ 8OArLEanKS3xw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E6B9FCE0D66; Thu, 18 Sep 2025 03:26:46 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Joel Fernandes , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 2/6] srcu: Add srcu_read_lock_fast_notrace() and srcu_read_unlock_fast_notrace() Date: Thu, 18 Sep 2025 03:26:42 -0700 Message-Id: <20250918102646.2592821-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> References: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds no-trace variants of the srcu_read_lock_fast() and srcu_read_unlock_fast() functions for tracing use. [ paulmck: Apply notrace feedback from Joel Fernandes, Steven Rostedt, and = Mathieu Desnoyers. ] [ paulmck: Apply excess-notrace feedback from Boqun Feng. ] Link: https://lore.kernel.org/all/20250721162433.10454-1-paulmck@kernel.org Signed-off-by: Paul E. McKenney Reviewed-by: Joel Fernandes Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 25 +++++++++++++++++++++++++ include/linux/srcutree.h | 5 +++-- 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 478c73d067f7d3..7a692bf8f99b96 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -282,6 +282,20 @@ static inline struct srcu_ctr __percpu *srcu_read_lock= _fast(struct srcu_struct * return retval; } =20 +/* + * Used by tracing, cannot be traced and cannot call lockdep. + * See srcu_read_lock_fast() for more information. + */ +static inline struct srcu_ctr __percpu *srcu_read_lock_fast_notrace(struct= srcu_struct *ssp) + __acquires(ssp) +{ + struct srcu_ctr __percpu *retval; + + srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + retval =3D __srcu_read_lock_fast(ssp); + return retval; +} + /** * srcu_down_read_fast - register a new reader for an SRCU-protected struc= ture. * @ssp: srcu_struct in which to register the new reader. @@ -394,6 +408,17 @@ static inline void srcu_read_unlock_fast(struct srcu_s= truct *ssp, struct srcu_ct RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 +/* + * Used by tracing, cannot be traced and cannot call lockdep. + * See srcu_read_unlock_fast() for more information. + */ +static inline void srcu_read_unlock_fast_notrace(struct srcu_struct *ssp, + struct srcu_ctr __percpu *scp) __releases(ssp) +{ + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); + __srcu_read_unlock_fast(ssp, scp); +} + /** * srcu_up_read_fast - unregister a old reader from an SRCU-protected stru= cture. * @ssp: srcu_struct in which to unregister the old reader. diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 043b5a67ef71eb..4d2fee4d38289f 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -240,7 +240,7 @@ static inline struct srcu_ctr __percpu *__srcu_ctr_to_p= tr(struct srcu_struct *ss * on architectures that support NMIs but do not supply NMI-safe * implementations of this_cpu_inc(). */ -static inline struct srcu_ctr __percpu *__srcu_read_lock_fast(struct srcu_= struct *ssp) +static inline struct srcu_ctr __percpu notrace *__srcu_read_lock_fast(stru= ct srcu_struct *ssp) { struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); =20 @@ -267,7 +267,8 @@ static inline struct srcu_ctr __percpu *__srcu_read_loc= k_fast(struct srcu_struct * on architectures that support NMIs but do not supply NMI-safe * implementations of this_cpu_inc(). */ -static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct= srcu_ctr __percpu *scp) +static inline void notrace +__srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu = *scp) { barrier(); /* Avoid leaking the critical section. */ if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) --=20 2.40.1 From nobody Thu Oct 2 09:21:04 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3C5E28FA9A; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191207; cv=none; b=qKIkvolQdtPYIPZqvLnTGFQML8rtUnRV9hzlGKn4Kc+b1SdTopdc4nz+mBzBW7ClOugX4lAOg3GCI2H/LyOEpJr0rj7xUOOxRw6VGYrtZ2UlnefCY1PaWrrlUYQaP9zzC0w01ay4qUH9dBYnROFhsjdvJJ91llaxnwZH4+zlSS4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191207; c=relaxed/simple; bh=Nsb0nSy5MOec7iftbI+KS9MGNfFWdbMuV/dAsboyXJE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CwzfvyPUj1OBJrTkzhS10HONzFCFAUvjgv2YXuKY4DgirpUCOLb4K0OBMFjCtK7w7Nqb73x58XN8jSCsxm8hNG7vtPIeqAl7MLsbRcWsuXrf09IPm3qu48/sP08/72PLzTyjaG6zCdZievLfFbT2PhUKyacweqfcgQgSzTlRyoo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Erl13Lw2; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Erl13Lw2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7B653C4CEF7; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758191207; bh=Nsb0nSy5MOec7iftbI+KS9MGNfFWdbMuV/dAsboyXJE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Erl13Lw2D6g5crg6B62xZXtL7Z29gdMGd3A21xt2dIu4H2icsiP8qe0DdcSAbLFpl Knh3ppD4mhTAnv4TnMSL/6rGnnzm4P/x+iGfM0nUbHJt1jULB1Svjje7NLSZF0tczH 8h4UI04/FszYESsKRhiDP903aYg8/O8noOyzBvaDZEoGE3RZ6L3x8tC2XNoWApwoMs viq7SC78c+5j0uXwP4C+NgpDOdPXbM4yRUyWNhULXxJJG6Xjp4IjL9fzwJHyN6cxR5 R6eGFRrWa3kmVypGzxKd9vKj9w787/6FEPsDE5YFSp3nWfUtqVu0M1sxNnyBCfW7YT IP/0NOhYkM7zA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E9851CE0EFF; Thu, 18 Sep 2025 03:26:46 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Joel Fernandes , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 3/6] srcu: Add guards for notrace variants of SRCU-fast readers Date: Thu, 18 Sep 2025 03:26:43 -0700 Message-Id: <20250918102646.2592821-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> References: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds the usual scoped_guard(srcu_fast_notrace, &my_srcu) and guard(srcu_fast_notrace)(&my_srcu). Signed-off-by: Paul E. McKenney Reviewed-by: Joel Fernandes Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 7a692bf8f99b96..ada65b58bc4c5f 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -515,4 +515,9 @@ DEFINE_LOCK_GUARD_1(srcu_fast, struct srcu_struct, srcu_read_unlock_fast(_T->lock, _T->scp), struct srcu_ctr __percpu *scp) =20 +DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, + _T->scp =3D srcu_read_lock_fast_notrace(_T->lock), + srcu_read_unlock_fast_notrace(_T->lock, _T->scp), + struct srcu_ctr __percpu *scp) + #endif --=20 2.40.1 From nobody Thu Oct 2 09:21:04 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3AFE2773F0; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191207; cv=none; b=mpkJvcDObtpwP0UWGOPryLJKk7W7mP0Hfpbg4xJQv2ZLDMmPiS1qBlHlO7U3euze2VI8ApDQQV2CGjkjM+YaOeEDaaw3IGQOKLwpEuwcm8UUskTehSL3fYvQvhdGA5G9I3UhnUdYNtVw+8Ot32xVdncFKCO8+bTNJpYRZ6IGzqE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191207; c=relaxed/simple; bh=v7iUSj7Qthedvqy0Qh8t0G4dno7vMWETYPwH31KRU0M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=csdbJypnU2bb+QZHwdlTtosXHPbnpSlqjVgvj66dFfb8Cq+fmykzzKEO/faN36jz2oqA3utera0YDs8Jn46A/dYr74sXERwR1irWmDYtyH56QGe2a3TK8vSbOmanp1LQ6xfrbt48lMZ/vohq3zqmP9DJ0ld0YjcOKLnB2uKX8tw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pvBA8nrq; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pvBA8nrq" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 81D85C4CEFA; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758191207; bh=v7iUSj7Qthedvqy0Qh8t0G4dno7vMWETYPwH31KRU0M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pvBA8nrquF9CBjPb4mvzgD8sMOShsLDvoIFoeV9Nkk+U0MigU8AFrqo5WfaFyPM/y 1hbFl9S+CLbEsaTZ5dan/3dqUHbqIfE1J4+iVLTNtlUI6Ts4AvRO62QxaTkRU+rLUH EJ8JPv34HGX8QT/CwLXvYKbc2Jz5YQBKFZquCx+Qg+LoXDK+ZNZGJ+XDyQxvyixIYJ ObFeLPxkG0GpZ0PerYnT5TV2pMpBRktnEcsh21xjpTbAoqtIFN0AxDmUhZzqoUYjt3 8H7Et6OowIgPCu6/V/sIB2T2xlt8ayIb0PVI+yPNaP+Mi8N+rJa+DvOJ1eduqKYbnH aiQoOc1jJ/jDQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EC4BCCE0F82; Thu, 18 Sep 2025 03:26:46 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 4/6] srcu: Document __srcu_read_{,un}lock_fast() implicit RCU readers Date: Thu, 18 Sep 2025 03:26:44 -0700 Message-Id: <20250918102646.2592821-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> References: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit documents the implicit RCU readers that are implied by the this_cpu_inc() and atomic_long_inc() operations in __srcu_read_lock_fast() and __srcu_read_unlock_fast(). While in the area, fix the documentation of the memory pairing of atomic_long_inc() in __srcu_read_lock_fast(). [ paulmck: Apply Joel Fernandes feedback. ] Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcutree.h | 42 ++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 4d2fee4d38289f..42098e0fa0b7dd 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -232,9 +232,27 @@ static inline struct srcu_ctr __percpu *__srcu_ctr_to_= ptr(struct srcu_struct *ss * srcu_read_unlock_fast(). * * Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side - * critical sections either because they disables interrupts, because they - * are a single instruction, or because they are a read-modify-write atomic - * operation, depending on the whims of the architecture. + * critical sections either because they disables interrupts, because + * they are a single instruction, or because they are read-modify-write + * atomic operations, depending on the whims of the architecture. + * This matters because the SRCU-fast grace-period mechanism uses either + * synchronize_rcu() or synchronize_rcu_expedited(), that is, RCU, + * *not* SRCU, in order to eliminate the need for the read-side smp_mb() + * invocations that are used by srcu_read_lock() and srcu_read_unlock(). + * The __srcu_read_unlock_fast() function also relies on this same RCU + * (again, *not* SRCU) trick to eliminate the need for smp_mb(). + * + * The key point behind this RCU trick is that if any part of a given + * RCU reader precedes the beginning of a given RCU grace period, then + * the entirety of that RCU reader and everything preceding it happens + * before the end of that same RCU grace period. Similarly, if any part + * of a given RCU reader follows the end of a given RCU grace period, + * then the entirety of that RCU reader and everything following it + * happens after the beginning of that same RCU grace period. Therefore, + * the operations labeled Y in __srcu_read_lock_fast() and those labeled Z + * in __srcu_read_unlock_fast() are ordered against the corresponding SRCU + * read-side critical section from the viewpoint of the SRCU grace period. + * This is all the ordering that is required, hence no calls to smp_mb(). * * This means that __srcu_read_lock_fast() is not all that fast * on architectures that support NMIs but do not supply NMI-safe @@ -245,9 +263,9 @@ static inline struct srcu_ctr __percpu notrace *__srcu_= read_lock_fast(struct src struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); =20 if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) - this_cpu_inc(scp->srcu_locks.counter); /* Y */ + this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader. else - atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); /* Z */ + atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); // Y, and implicit RCU = reader. barrier(); /* Avoid leaking the critical section. */ return scp; } @@ -258,23 +276,17 @@ static inline struct srcu_ctr __percpu notrace *__src= u_read_lock_fast(struct src * different CPU than that which was incremented by the corresponding * srcu_read_lock_fast(), but it must be within the same task. * - * Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side - * critical sections either because they disables interrupts, because they - * are a single instruction, or because they are a read-modify-write atomic - * operation, depending on the whims of the architecture. - * - * This means that __srcu_read_unlock_fast() is not all that fast - * on architectures that support NMIs but do not supply NMI-safe - * implementations of this_cpu_inc(). + * Please see the __srcu_read_lock_fast() function's header comment for + * information on implicit RCU readers and NMI safety. */ static inline void notrace __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu = *scp) { barrier(); /* Avoid leaking the critical section. */ if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) - this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ + this_cpu_inc(scp->srcu_unlocks.counter); // Z, and implicit RCU reader. else - atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); /* Z */ + atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RC= U reader. } =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); --=20 2.40.1 From nobody Thu Oct 2 09:21:04 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAF46299A81; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191208; cv=none; b=mhqx0Wo92vqp28MMVeqwFmX9Ksk+b36oSLfu5VzseFIXQqNPxJE3fGsL1CoTNpu/0GVBPQcnnVVDj6vNAZqAciTG9WTivhyR3ST261Dwju0DLbnbJAqMEeM0ci+MS06OLWXXt0F8a8yz6I1iF4neWuJHKcdj+vXhAkGuoQjJPAg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191208; c=relaxed/simple; bh=bUr/c79awvu8OWGK8+lJnU//Ku79FE8Xqz9RETWtyqk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NzWs6STjft1s9W837mlvxbxTd+bqEeh6CuIiFS/pnH7qR6aqZkPIZyG9ddhGAB7b+XjGgTlfZd6/wys94utGYFmCAcpb5hi567XxwGWeUnR1+c6Tm5SG+Gup2KU+qnFj+f4VUH4meAGmOSC7MtSl3IG9sRkUbtp+TqT7FWZbiDQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=o3u2Gg5t; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="o3u2Gg5t" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8AA0AC4CEFE; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758191207; bh=bUr/c79awvu8OWGK8+lJnU//Ku79FE8Xqz9RETWtyqk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o3u2Gg5thkUtaN/NWfHNKAS3ogUHOVnaqYP5Dvo0z71KexcdHXAUFxOWauAF0z4V6 cRdy/Y1sDwcGugIX3dO6q1thgjiQSQDV3HZ5J8gIM2IZ+TMqrW7z/FJkR2wr8o0XfO 3+ko/bI6kwWiP+ggqp5UCJc4HD3yo+PdfZtb+Jw+Gw/MQ4yQJXXr6fXt9HxDMu3Hy/ 0Dc1d4gbvfQi4yCpUXo8PUb7cexVHOaA2nzvkO+AR1agmlSXs5ZIFFrtx+Q85CIrUy jwvEg6x3gpfqc5glLN/1JwtAHt5hHpAgvGmsfOceK6pTND+Np4vAG7F1hT+78GZdEI BLByvkTUTZzhw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EED18CE10B6; Thu, 18 Sep 2025 03:26:46 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v2 5/6] srcu: Document srcu_flip() memory-barrier D relation to SRCU-fast Date: Thu, 18 Sep 2025 03:26:45 -0700 Message-Id: <20250918102646.2592821-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> References: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The smp_mb() memory barrier at the end of srcu_flip() has a comment, but that comment does not make it clear that this memory barrier is an optimization, as opposed to being needed for correctness. This commit therefore adds this information and points out that it is omitted for SRCU-fast, where a much heavier weight synchronize_srcu() would be required. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- kernel/rcu/srcutree.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index c5e8ebc493d5ee..1ff94b76d91f15 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1168,6 +1168,16 @@ static void srcu_flip(struct srcu_struct *ssp) * counter update. Note that both this memory barrier and the * one in srcu_readers_active_idx_check() provide the guarantee * for __srcu_read_lock(). + * + * Note that this is a performance optimization, in which we spend + * an otherwise unnecessary smp_mb() in order to reduce the number + * of full per-CPU-variable scans in srcu_readers_lock_idx() and + * srcu_readers_unlock_idx(). But this performance optimization + * is not so optimal for SRCU-fast, where we would be spending + * not smp_mb(), but rather synchronize_rcu(). At the same time, + * the overhead of the smp_mb() is in the noise, so there is no + * point in omitting it in the SRCU-fast case. So the same code + * is executed either way. */ smp_mb(); /* D */ /* Pairs with C. */ } --=20 2.40.1 From nobody Thu Oct 2 09:21:04 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6FE8D2F5A19; Thu, 18 Sep 2025 10:26:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191208; cv=none; b=CNe5ntobzOuefElkR0W0YKPXT1lOYV9po6Qq+qH7Vlp6oS5mbmG6HbErH5uoAhh8V0nf7mYFC/kA1pg7jWf5DMnhHNurYzmXpvk+LstemdiYTLChvKTZhhsCFbRXPUr6iHJypyYx3oaDBNvGhO5Uqwj3KjlpB4Mp9GvoTkc1qgY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758191208; c=relaxed/simple; bh=2O4JbyeyNOFGCtQWseGjahocWyWME/uonyjbz+Ok46w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bVUBtAN/dEeHVllGazfAZsPUQce4bhIZ345QbCF7QKIBTwF0XAw+9n32h7YTVtc6misfokhJlV5SRMYInAoFaLlJ8YR2qNvNb3LrVGTptX/ybU8Tbsz2SKL+iZlEyGJ6rhiqlGtbGhYxYKkqDZoaR2kkYmhbi+8+68ntXGK4GtY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VwImpqaX; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VwImpqaX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F2EF7C4CEFB; Thu, 18 Sep 2025 10:26:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758191208; bh=2O4JbyeyNOFGCtQWseGjahocWyWME/uonyjbz+Ok46w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VwImpqaXeSqK7RNRa7Xs5c/p1sbAcyEksae+NZZqwBK7/ed5JP33ZZDjt9u7OU8C5 hnGl9vaK4zdFvuwyolTM3Yi1uxg7HhaLuKZCDxAuHx+aZiOxEAvhTo8bCy6XLtZ/1p wXxvvoO7qREp+kc9Y+VBWmXGN2nY6C4pQt9NtiF1+cf90CCVx1Ve9BVJv78JHRvGDZ Nc23trUlBcASrsMUBZtCcEc4fZkWc/SgGeP7No+6RsH8Ez4MP9EBh+D/giw5TCncTD VpDFl0S3ymeJ7XIdfXuoQIZcgFid2wewM9EvOTGqapryy7bcGUAhcm9Emf8Q4tRpRG 6IZ7t3bySBCvA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id F169CCE10DB; Thu, 18 Sep 2025 03:26:46 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zqiang , "Paul E . McKenney" Subject: [PATCH v2 6/6] srcu/tiny: Remove preempt_disable/enable() in srcu_gp_start_if_needed() Date: Thu, 18 Sep 2025 03:26:46 -0700 Message-Id: <20250918102646.2592821-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> References: <89b6f92e-2aa6-4869-ad4f-47bb3fbadfbb@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Zqiang Currently, the srcu_gp_start_if_needed() is always be invoked in preempt disable's critical section, this commit therefore remove redundant preempt_disable/enable() in srcu_gp_start_if_needed() and adds a call to lockdep_assert_preemption_disabled() in order to enable lockdep to diagnose mistaken invocations of this function from preempts-enabled code. Fixes: 65b4a59557f6 ("srcu: Make Tiny SRCU explicitly disable preemption") Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutiny.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c index 6e9fe2ce1075d5..e3b64a5e0ec7e1 100644 --- a/kernel/rcu/srcutiny.c +++ b/kernel/rcu/srcutiny.c @@ -176,10 +176,9 @@ static void srcu_gp_start_if_needed(struct srcu_struct= *ssp) { unsigned long cookie; =20 - preempt_disable(); // Needed for PREEMPT_LAZY + lockdep_assert_preemption_disabled(); // Needed for PREEMPT_LAZY cookie =3D get_state_synchronize_srcu(ssp); if (ULONG_CMP_GE(READ_ONCE(ssp->srcu_idx_max), cookie)) { - preempt_enable(); return; } WRITE_ONCE(ssp->srcu_idx_max, cookie); @@ -189,7 +188,6 @@ static void srcu_gp_start_if_needed(struct srcu_struct = *ssp) else if (list_empty(&ssp->srcu_work.entry)) list_add(&ssp->srcu_work.entry, &srcu_boot_list); } - preempt_enable(); } =20 /* --=20 2.40.1