From nobody Sat Oct 4 12:46:56 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DC629478; Sat, 16 Aug 2025 00:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302761; cv=none; b=uYgD/aD07IRKnEmIscgZRwXKwnnojmHz00fFF+j66g4RCgE2IaWiCkYqPSmYuDZK77HIGUYZzaw4z6EuBGM7CZs8KrpL3GMEZRW4o5eMq4YmV64jao4Phm2kVM0JCddH2xj2D/8ye+sj8zNG8hatmINJMdTgcgJzT3Kalro9cCU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302761; c=relaxed/simple; bh=pvHoXIq/XZZkE26WZ9PeQrCmEPRe5uY95ng191EMoZw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bfl30zfYhyt8N1qsHhKBaJ1xaq0yTgPxkONAZA+bAMZuH/N+OEmadJPJUiBWzjCLXM5AQwA8RHGmogNDAZ7ETuvAUxYwdnOdE29hr47I6Lay6iBb94VBESS4CMpf70at2hiN23vEULbjE1hWOApCzWDlUJrMUI2ehRUQqR6RaHk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=F9OiDTq1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="F9OiDTq1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 46135C4CEF4; Sat, 16 Aug 2025 00:06:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755302761; bh=pvHoXIq/XZZkE26WZ9PeQrCmEPRe5uY95ng191EMoZw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F9OiDTq1hkz7FglPboA2M+TXIoQPEBTp3KbyJC2a0njYKMcI35pqvyrM11VApJkTh zxExpCsvCB8V/v8B5buo8zqa5BMvZ1jLP0ihcgVBguLtQIqbOChfqIu/EqoEu+uVhv D9jODUyNbVdzxZ3GCtZeoDqCHy+6MR+El8kQ3hySVgMxkLrKR+5fSvPlTQcMFl+1rY hU2+7T1YmkO2J8xfyoaD/lKwiWqbsQsTSpV3LoKkrIWFXJSBr/tL3laUI1534J3RF1 U+XqN/uAa1LaMEskZv3metAc6p+XTCIA9xFX5UIBshR1+nIz0/OGl/E9bMfJ57EXmD ZpTR3+yVzHVNg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 830C7CE0B2D; Fri, 15 Aug 2025 17:06:00 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Joel Fernandes , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v6 1/6] srcu: Move rcu_is_watching() checks to srcu_read_{,un}lock_fast() Date: Fri, 15 Aug 2025 17:05:54 -0700 Message-Id: <20250816000559.2622626-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The rcu_is_watching() warnings are currently in the SRCU-tree implementations of __srcu_read_lock_fast() and __srcu_read_unlock_fast(). However, this makes it difficult to create _notrace variants of srcu_read_lock_fast() and srcu_read_unlock_fast(). This commit therefore moves these checks to srcu_read_lock_fast(), srcu_read_unlock_fast(), srcu_down_read_fast(), and srcu_up_read_fast(). Signed-off-by: Paul E. McKenney Reviewed-by: Joel Fernandes Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 4 ++++ include/linux/srcutree.h | 2 -- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index f179700fecafb8..478c73d067f7d3 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -275,6 +275,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast(struct srcu_struct * { struct srcu_ctr __percpu *retval; =20 + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast()."); srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); retval =3D __srcu_read_lock_fast(ssp); rcu_try_lock_acquire(&ssp->dep_map); @@ -295,6 +296,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast(struct srcu_struct * static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_st= ruct *ssp) __acquires(ssp) { WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read= _fast()."); srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); return __srcu_read_lock_fast(ssp); } @@ -389,6 +391,7 @@ static inline void srcu_read_unlock_fast(struct srcu_st= ruct *ssp, struct srcu_ct srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); srcu_lock_release(&ssp->dep_map); __srcu_read_unlock_fast(ssp, scp); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 /** @@ -405,6 +408,7 @@ static inline void srcu_up_read_fast(struct srcu_struct= *ssp, struct srcu_ctr __ WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); __srcu_read_unlock_fast(ssp, scp); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_up_read_f= ast()."); } =20 /** diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index bf44d8d1e69eab..043b5a67ef71eb 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -244,7 +244,6 @@ static inline struct srcu_ctr __percpu *__srcu_read_loc= k_fast(struct srcu_struct { struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); =20 - RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast()."); if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) this_cpu_inc(scp->srcu_locks.counter); /* Y */ else @@ -275,7 +274,6 @@ static inline void __srcu_read_unlock_fast(struct srcu_= struct *ssp, struct srcu_ this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ else atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); /* Z */ - RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); --=20 2.40.1 From nobody Sat Oct 4 12:46:56 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DC2F9463; Sat, 16 Aug 2025 00:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302761; cv=none; b=LzLj3Bq/nuXtDfpzWylGiPJMqMe2Uf4mprvRCZ+Bj1RvgcPdx0UL5E+uf1s8kPvzYGwu8OfrkRl4NOhYoCj1vRclknrBxA9iv0Uqjpiadgq0Gb21yqVYbu9QXRQ6U9ZggchOBxDEhXuhnyayB/89Kp4iUu7K3gtEeVtHtcruq/U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302761; c=relaxed/simple; bh=jDxGNAMMOCyH7qVFbWf8yWfHDDI/sC4nlnOnRxPII+E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OISDeGLMNJegO8Tu5R3AISD2h3BV7BABCkU8LScWzVcrHsfq2mEaaaAy2xXZ7t/n21ZUNY7yoChxcPhTP58yPg7uAYMRFrv4JWvVN8Vm9hRFPhSu3rpPDJkiy4qp6O1dAdu5ABG/OdyTyLHffBN5kXl3qmYSafnX/6eukfYwALM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=i/iPnTid; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="i/iPnTid" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 421DDC4CEEB; Sat, 16 Aug 2025 00:06:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755302761; bh=jDxGNAMMOCyH7qVFbWf8yWfHDDI/sC4nlnOnRxPII+E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i/iPnTide+3z6ikOSLFc3JJCE58bRR5yyK0/DMdOySITFv0bIy+YUG1u6S9B/yIrw nERIyqchad7TboqwWypPYlDaFafIkG46Txbv3QRD363z0LTAIRwGeI558zX01jlMsM jx3omBm1pAOzmaRdX9Ms6bDhyERbo3hS1vi3bgzFlJnJE1NxPuPeoCyfH4p0yaegI/ D0rOGmllMnDwBZVEd427ie0SR3X5NQBalmcqddjS8ihiABnHqRIauz6HEK+zpa0f4P IZOwCIBvBonsez3ksL0he6SHBNjm+2oQxAYz9XCC1ckCsC1TGjtoslroLQW2aCxq44 jH4Eg4QUI9UuQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 86331CE0B30; Fri, 15 Aug 2025 17:06:00 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Joel Fernandes , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v6 2/6] srcu: Add srcu_read_lock_fast_notrace() and srcu_read_unlock_fast_notrace() Date: Fri, 15 Aug 2025 17:05:55 -0700 Message-Id: <20250816000559.2622626-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds no-trace variants of the srcu_read_lock_fast() and srcu_read_unlock_fast() functions for tracing use. [ paulmck: Apply notrace feedback from Joel Fernandes, Steven Rostedt, and = Mathieu Desnoyers. ] [ paulmck: Apply excess-notrace feedback from Boqun Feng. ] Link: https://lore.kernel.org/all/20250721162433.10454-1-paulmck@kernel.org Signed-off-by: Paul E. McKenney Reviewed-by: Joel Fernandes Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 25 +++++++++++++++++++++++++ include/linux/srcutree.h | 5 +++-- 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 478c73d067f7d3..7a692bf8f99b96 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -282,6 +282,20 @@ static inline struct srcu_ctr __percpu *srcu_read_lock= _fast(struct srcu_struct * return retval; } =20 +/* + * Used by tracing, cannot be traced and cannot call lockdep. + * See srcu_read_lock_fast() for more information. + */ +static inline struct srcu_ctr __percpu *srcu_read_lock_fast_notrace(struct= srcu_struct *ssp) + __acquires(ssp) +{ + struct srcu_ctr __percpu *retval; + + srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + retval =3D __srcu_read_lock_fast(ssp); + return retval; +} + /** * srcu_down_read_fast - register a new reader for an SRCU-protected struc= ture. * @ssp: srcu_struct in which to register the new reader. @@ -394,6 +408,17 @@ static inline void srcu_read_unlock_fast(struct srcu_s= truct *ssp, struct srcu_ct RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 +/* + * Used by tracing, cannot be traced and cannot call lockdep. + * See srcu_read_unlock_fast() for more information. + */ +static inline void srcu_read_unlock_fast_notrace(struct srcu_struct *ssp, + struct srcu_ctr __percpu *scp) __releases(ssp) +{ + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); + __srcu_read_unlock_fast(ssp, scp); +} + /** * srcu_up_read_fast - unregister a old reader from an SRCU-protected stru= cture. * @ssp: srcu_struct in which to unregister the old reader. diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 043b5a67ef71eb..4d2fee4d38289f 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -240,7 +240,7 @@ static inline struct srcu_ctr __percpu *__srcu_ctr_to_p= tr(struct srcu_struct *ss * on architectures that support NMIs but do not supply NMI-safe * implementations of this_cpu_inc(). */ -static inline struct srcu_ctr __percpu *__srcu_read_lock_fast(struct srcu_= struct *ssp) +static inline struct srcu_ctr __percpu notrace *__srcu_read_lock_fast(stru= ct srcu_struct *ssp) { struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); =20 @@ -267,7 +267,8 @@ static inline struct srcu_ctr __percpu *__srcu_read_loc= k_fast(struct srcu_struct * on architectures that support NMIs but do not supply NMI-safe * implementations of this_cpu_inc(). */ -static inline void __srcu_read_unlock_fast(struct srcu_struct *ssp, struct= srcu_ctr __percpu *scp) +static inline void notrace +__srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu = *scp) { barrier(); /* Avoid leaking the critical section. */ if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) --=20 2.40.1 From nobody Sat Oct 4 12:46:56 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E7CF944F; Sat, 16 Aug 2025 00:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302761; cv=none; b=maAlBPbO9jxZljx6eXzchogf2W29SZkKK0PhbYSR3IZa69e9gELDGkrp1v79BlT9JjcOjUxr9ZB+CpGmbT7fchg4UPN20s9ni33qfL2BG0O6ZFvic91xzY+QG+3MVqbBXdJob5dOWCeZ/8UmsaDbx5TpAw+ajuxNkWndYyeSu/4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302761; c=relaxed/simple; bh=Nsb0nSy5MOec7iftbI+KS9MGNfFWdbMuV/dAsboyXJE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FZ81bPxDa8mZugENC/HyO8eKMH9wq/GJ8tUOYKSQQ44qvyl7My+tSy1Qmf5heAg19hp64/o8lFa5MQjbKV/o339uPvX/OHhbBhLelWFIttZODhVISo2MOnuXl5v9jNoBA4aD1xsOXLuwi46XANcSIULGpvDG/bm2MtEeU+UjRyw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RrHVkK0e; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RrHVkK0e" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 48C0AC4CEF6; Sat, 16 Aug 2025 00:06:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755302761; bh=Nsb0nSy5MOec7iftbI+KS9MGNfFWdbMuV/dAsboyXJE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RrHVkK0eULQgxNHH/Gd+LKc/qRhrcPLxwa5IKJe2uApV+ykhHK41+YvBEGKAJmAAz O6KdrN+WM3/zieqPAfoqqyL2kH4FO0eJp9L1uMUnQYiLz0KBGXkzcBsrfY+LcHHKuf ZkvCNQYv7AtIWQhpPT8zHY6E+YBkXuOboR1b5UwfZNGa5TV0WF0cfaxeRA1JYlfx7D bmBA7B4pyMj9cZkFeeYseTZvDM9wj9KnC8nkIo9KlAEXlxepJyaaJhqOUg1weyBCTX We+nvxy8UziZTjOG+JXMIgJn9RHI5BOCGRgg+9TxreB6rKbZofyVXg2/y/v3v8UNiS Zq5TCalh9hM6w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 88ED0CE0B31; Fri, 15 Aug 2025 17:06:00 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Joel Fernandes , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v6 3/6] srcu: Add guards for notrace variants of SRCU-fast readers Date: Fri, 15 Aug 2025 17:05:56 -0700 Message-Id: <20250816000559.2622626-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds the usual scoped_guard(srcu_fast_notrace, &my_srcu) and guard(srcu_fast_notrace)(&my_srcu). Signed-off-by: Paul E. McKenney Reviewed-by: Joel Fernandes Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcu.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 7a692bf8f99b96..ada65b58bc4c5f 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -515,4 +515,9 @@ DEFINE_LOCK_GUARD_1(srcu_fast, struct srcu_struct, srcu_read_unlock_fast(_T->lock, _T->scp), struct srcu_ctr __percpu *scp) =20 +DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, + _T->scp =3D srcu_read_lock_fast_notrace(_T->lock), + srcu_read_unlock_fast_notrace(_T->lock, _T->scp), + struct srcu_ctr __percpu *scp) + #endif --=20 2.40.1 From nobody Sat Oct 4 12:46:56 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DC8EAD24; Sat, 16 Aug 2025 00:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302761; cv=none; b=IU4vvbewKHSnr+5GPEW+/YS7U9HAbzzrsFYbwGif8p3WxnsSUxU/KfcWif/X8sjN2cSuD1BBBHLcww4CWW1uZxLNSWuWzrR+4knb8tpfOqvvbltvHMfxMzKs9J5Uj1rlyJtPP1t4Gt4vMCq7lL3H4SguhLOhxCRawGCE7cuA8qo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302761; c=relaxed/simple; bh=Psrmv3HmCJPePj1DKWtPeo1mfeO4KSEh8kCtaDMB2GQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iTfoXCR+941T352i9eqfrgqB9N6+HdbbHiQuX5j517Fgh1F1os42PteFDc+wFh7EdRnEoirLCuufpqkfWrh34ECA+BUmrWhf0/KfaZFjy+pCKzgkyFzp6aVVXq66MoHqoCeoMNQHF2CwCo1j2ePVhHGI4bSJE5CyWPPcteKwphs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DgNbBXrv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DgNbBXrv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 54FEAC4CEF5; Sat, 16 Aug 2025 00:06:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755302761; bh=Psrmv3HmCJPePj1DKWtPeo1mfeO4KSEh8kCtaDMB2GQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DgNbBXrvfy69vBlpGhBpJ1qdV1wRgolCGYc2T6MqhVrwzNwnrLfU0WL3brk1kHYYv dEJr2KLXYiJeNTMwFftqZ7b0RUaWAYQBvJ+w/YQZiP6D+lNU1B2809sQibGN/b/5Vx XyLgqXwaqi30MLB5hvOWwkejBwqPc3tsEHl12YMTeyrYb5Lqmwg7rdB8jRuG+5qIA8 fp5tC8dZB+4GLrEyWY/ZHYJ8Bvm00i8BvKyuQ5LJ3tJaLyB9ZRPCV5NXfaYGazUwEF Z30RatPah7laH12cdp4cOY/B7GjT3x7gWU1ISjNoZm9xVRcgMT4LzpRoA5qIcK91mN nt8vBCLoNlMzg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8B7DACE0B6D; Fri, 15 Aug 2025 17:06:00 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v6 4/6] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast Date: Fri, 15 Aug 2025 17:05:57 -0700 Message-Id: <20250816000559.2622626-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current use of guard(preempt_notrace)() within __DECLARE_TRACE() to protect invocation of __DO_TRACE_CALL() means that BPF programs attached to tracepoints are non-preemptible. This is unhelpful in real-time systems, whose users apparently wish to use BPF while also achieving low latencies. (Who knew?) One option would be to use preemptible RCU, but this introduces many opportunities for infinite recursion, which many consider to be counterproductive, especially given the relatively small stacks provided by the Linux kernel. These opportunities could be shut down by sufficiently energetic duplication of code, but this sort of thing is considered impolite in some circles. Therefore, use the shiny new SRCU-fast API, which provides somewhat faster readers than those of preemptible RCU, at least on my laptop, where task_struct access is more expensive than access to per-CPU variables. And SRCU fast provides way faster readers than does SRCU, courtesy of being able to avoid the read-side use of smp_mb(). Also, it is quite straightforward to create srcu_read_{,un}lock_fast_notrace() functions. While in the area, SRCU now supports early boot call_srcu(). Therefore, remove the checks that used to avoid such use from rcu_free_old_probes() before this commit was applied: e53244e2c893 ("tracepoint: Remove SRCU protection") The current commit can be thought of as an approximate revert of that commit, with some compensating additions of preemption disabling pointed out by Steven Rostedt (thank you, Steven!). [ Apply kernel test robot feedback. ] Link: https://lore.kernel.org/all/20250613152218.1924093-1-bigeasy@linutron= ix.de/ Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/tracepoint.h | 6 ++++-- include/trace/perf.h | 2 ++ include/trace/trace_events.h | 2 ++ kernel/tracepoint.c | 21 ++++++++++++++++++++- 4 files changed, 28 insertions(+), 3 deletions(-) diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 826ce3f8e1f851..a22c1ab88560b8 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -33,6 +33,8 @@ struct trace_eval_map { =20 #define TRACEPOINT_DEFAULT_PRIO 10 =20 +extern struct srcu_struct tracepoint_srcu; + extern int tracepoint_probe_register(struct tracepoint *tp, void *probe, void *data); extern int @@ -115,7 +117,7 @@ void for_each_tracepoint_in_module(struct module *mod, static inline void tracepoint_synchronize_unregister(void) { synchronize_rcu_tasks_trace(); - synchronize_rcu(); + synchronize_srcu(&tracepoint_srcu); } static inline bool tracepoint_is_faultable(struct tracepoint *tp) { @@ -271,7 +273,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(t= racepoint_ptr_t *p) static inline void __do_trace_##name(proto) \ { \ if (cond) { \ - guard(preempt_notrace)(); \ + guard(srcu_fast_notrace)(&tracepoint_srcu); \ __DO_TRACE_CALL(name, TP_ARGS(args)); \ } \ } \ diff --git a/include/trace/perf.h b/include/trace/perf.h index a1754b73a8f55b..1b7925a859665f 100644 --- a/include/trace/perf.h +++ b/include/trace/perf.h @@ -71,7 +71,9 @@ perf_trace_##call(void *__data, proto) \ u64 __count __attribute__((unused)); \ struct task_struct *__task __attribute__((unused)); \ \ + preempt_disable_notrace(); \ do_perf_trace_##call(__data, args); \ + preempt_enable_notrace(); \ } =20 #undef DECLARE_EVENT_SYSCALL_CLASS diff --git a/include/trace/trace_events.h b/include/trace/trace_events.h index 4f22136fd4656c..0504a423ca2539 100644 --- a/include/trace/trace_events.h +++ b/include/trace/trace_events.h @@ -436,7 +436,9 @@ __DECLARE_EVENT_CLASS(call, PARAMS(proto), PARAMS(args)= , PARAMS(tstruct), \ static notrace void \ trace_event_raw_event_##call(void *__data, proto) \ { \ + preempt_disable_notrace(); \ do_trace_event_raw_event_##call(__data, args); \ + preempt_enable_notrace(); \ } =20 #undef DECLARE_EVENT_SYSCALL_CLASS diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 62719d2941c900..e19973015cbd73 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -25,6 +25,9 @@ enum tp_func_state { extern tracepoint_ptr_t __start___tracepoints_ptrs[]; extern tracepoint_ptr_t __stop___tracepoints_ptrs[]; =20 +DEFINE_SRCU(tracepoint_srcu); +EXPORT_SYMBOL_GPL(tracepoint_srcu); + enum tp_transition_sync { TP_TRANSITION_SYNC_1_0_1, TP_TRANSITION_SYNC_N_2_1, @@ -34,6 +37,7 @@ enum tp_transition_sync { =20 struct tp_transition_snapshot { unsigned long rcu; + unsigned long srcu_gp; bool ongoing; }; =20 @@ -46,6 +50,7 @@ static void tp_rcu_get_state(enum tp_transition_sync sync) =20 /* Keep the latest get_state snapshot. */ snapshot->rcu =3D get_state_synchronize_rcu(); + snapshot->srcu_gp =3D start_poll_synchronize_srcu(&tracepoint_srcu); snapshot->ongoing =3D true; } =20 @@ -56,6 +61,8 @@ static void tp_rcu_cond_sync(enum tp_transition_sync sync) if (!snapshot->ongoing) return; cond_synchronize_rcu(snapshot->rcu); + if (!poll_state_synchronize_srcu(&tracepoint_srcu, snapshot->srcu_gp)) + synchronize_srcu(&tracepoint_srcu); snapshot->ongoing =3D false; } =20 @@ -101,17 +108,29 @@ static inline void *allocate_probes(int count) return p =3D=3D NULL ? NULL : p->probes; } =20 -static void rcu_free_old_probes(struct rcu_head *head) +static void srcu_free_old_probes(struct rcu_head *head) { kfree(container_of(head, struct tp_probes, rcu)); } =20 +static void rcu_free_old_probes(struct rcu_head *head) +{ + call_srcu(&tracepoint_srcu, head, srcu_free_old_probes); +} + static inline void release_probes(struct tracepoint *tp, struct tracepoint= _func *old) { if (old) { struct tp_probes *tp_probes =3D container_of(old, struct tp_probes, probes[0]); =20 + /* + * Tracepoint probes are protected by either RCU or + * Tasks Trace RCU and also by SRCU. By calling the SRCU + * callback in the [Tasks Trace] RCU callback we cover + * both cases. So let us chain the SRCU and [Tasks Trace] + * RCU callbacks to wait for both grace periods. + */ if (tracepoint_is_faultable(tp)) call_rcu_tasks_trace(&tp_probes->rcu, rcu_free_old_probes); else --=20 2.40.1 From nobody Sat Oct 4 12:46:56 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECCC779F5; Sat, 16 Aug 2025 00:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302762; cv=none; b=FiBdzeNHxnLqcHPcgUOv7MqpipR7gSb2et3cnCzG1UKOdYeAXETDwlg6bGxcc+LTHOZAjuS7yKYPo8bjfRktLRbN+ClU8y1dz4KLQINUbIC+1iTvcX/0v3uFBy1/FsKcuYbFozTxqkj/c4sjY/N/CDMmK2gO7Hz3NeDVgz/+rlo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302762; c=relaxed/simple; bh=v7iUSj7Qthedvqy0Qh8t0G4dno7vMWETYPwH31KRU0M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aiG7oPWybtj/JnTaJAG3aV8HgWCWd41Rne9OEmnWEP41rkuULKU30IsFJPRJ8blb/nZy7Vtf5twd4kBfwD5WqtwTEe+A3fABUAaV7k/7723UZqRD3dtke2SV6tLeoZC/PHyVgl8ow26tvjegV8vXF9W2k+wCL1He+acym1/2Stc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lHfKxtdc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lHfKxtdc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5C89CC4CEF8; Sat, 16 Aug 2025 00:06:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755302761; bh=v7iUSj7Qthedvqy0Qh8t0G4dno7vMWETYPwH31KRU0M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lHfKxtdcT8kD7vpQqZz6fl72BuJCuP9eNgkgJSgxu7u7y2nuwspO63Ix8I7wdhdSd UYODa4nnakkDoXO5RZZ94QON3pNzsqR6nqWb9QVFNvqgtNv699h8XgNh6+rWzVh2V4 pl5N83NlWSLrznOtUhc2RHXlKE3PAYwa0bFKEnJQoNh27B+Wt+QzB/IsYBqu6WxGb5 Tm68V3VfVH4MLwUlLfZ7b4hAvB4PLZNuAmg3z2E9FdouFQYinrkqvJ6kLaqmaBjkH8 UziAhlArKvEFGq1/hxnNGRzXdRgJ471POXkE1irkot8APO4XGMZwCYHL0iiJuyiP6F r7DLWs48W8u+w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8E02FCE0EA2; Fri, 15 Aug 2025 17:06:00 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v6 5/6] srcu: Document __srcu_read_{,un}lock_fast() implicit RCU readers Date: Fri, 15 Aug 2025 17:05:58 -0700 Message-Id: <20250816000559.2622626-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit documents the implicit RCU readers that are implied by the this_cpu_inc() and atomic_long_inc() operations in __srcu_read_lock_fast() and __srcu_read_unlock_fast(). While in the area, fix the documentation of the memory pairing of atomic_long_inc() in __srcu_read_lock_fast(). [ paulmck: Apply Joel Fernandes feedback. ] Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- include/linux/srcutree.h | 42 ++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 4d2fee4d38289f..42098e0fa0b7dd 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -232,9 +232,27 @@ static inline struct srcu_ctr __percpu *__srcu_ctr_to_= ptr(struct srcu_struct *ss * srcu_read_unlock_fast(). * * Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side - * critical sections either because they disables interrupts, because they - * are a single instruction, or because they are a read-modify-write atomic - * operation, depending on the whims of the architecture. + * critical sections either because they disables interrupts, because + * they are a single instruction, or because they are read-modify-write + * atomic operations, depending on the whims of the architecture. + * This matters because the SRCU-fast grace-period mechanism uses either + * synchronize_rcu() or synchronize_rcu_expedited(), that is, RCU, + * *not* SRCU, in order to eliminate the need for the read-side smp_mb() + * invocations that are used by srcu_read_lock() and srcu_read_unlock(). + * The __srcu_read_unlock_fast() function also relies on this same RCU + * (again, *not* SRCU) trick to eliminate the need for smp_mb(). + * + * The key point behind this RCU trick is that if any part of a given + * RCU reader precedes the beginning of a given RCU grace period, then + * the entirety of that RCU reader and everything preceding it happens + * before the end of that same RCU grace period. Similarly, if any part + * of a given RCU reader follows the end of a given RCU grace period, + * then the entirety of that RCU reader and everything following it + * happens after the beginning of that same RCU grace period. Therefore, + * the operations labeled Y in __srcu_read_lock_fast() and those labeled Z + * in __srcu_read_unlock_fast() are ordered against the corresponding SRCU + * read-side critical section from the viewpoint of the SRCU grace period. + * This is all the ordering that is required, hence no calls to smp_mb(). * * This means that __srcu_read_lock_fast() is not all that fast * on architectures that support NMIs but do not supply NMI-safe @@ -245,9 +263,9 @@ static inline struct srcu_ctr __percpu notrace *__srcu_= read_lock_fast(struct src struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); =20 if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) - this_cpu_inc(scp->srcu_locks.counter); /* Y */ + this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader. else - atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); /* Z */ + atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); // Y, and implicit RCU = reader. barrier(); /* Avoid leaking the critical section. */ return scp; } @@ -258,23 +276,17 @@ static inline struct srcu_ctr __percpu notrace *__src= u_read_lock_fast(struct src * different CPU than that which was incremented by the corresponding * srcu_read_lock_fast(), but it must be within the same task. * - * Note that both this_cpu_inc() and atomic_long_inc() are RCU read-side - * critical sections either because they disables interrupts, because they - * are a single instruction, or because they are a read-modify-write atomic - * operation, depending on the whims of the architecture. - * - * This means that __srcu_read_unlock_fast() is not all that fast - * on architectures that support NMIs but do not supply NMI-safe - * implementations of this_cpu_inc(). + * Please see the __srcu_read_lock_fast() function's header comment for + * information on implicit RCU readers and NMI safety. */ static inline void notrace __srcu_read_unlock_fast(struct srcu_struct *ssp, struct srcu_ctr __percpu = *scp) { barrier(); /* Avoid leaking the critical section. */ if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) - this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ + this_cpu_inc(scp->srcu_unlocks.counter); // Z, and implicit RCU reader. else - atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); /* Z */ + atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RC= U reader. } =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); --=20 2.40.1 From nobody Sat Oct 4 12:46:56 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9119C2AE7F; Sat, 16 Aug 2025 00:06:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302762; cv=none; b=FmC3Sx2VRD9SMS/mc/9rHQ12jydmsDNzY7oDnqhlHYjX9xDRRICaDVJQ3e3FCTcDsDIKLaUs4GOmT5yk6bNVAvY04vHB54Wx+/ndpOZm9jwQfhheo2RdhXQFMaoJIcc485+NdWwDcxK+B57z7lJ4AaMthwdtzCvfFO1KLi7t04I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755302762; c=relaxed/simple; bh=bUr/c79awvu8OWGK8+lJnU//Ku79FE8Xqz9RETWtyqk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uhn8U5BrFtCqC/IDJ9amJGtzPlPx6yQyv8k3+Ja7pggNa/o7ABc8U+oBrPVgXEcz7wLvxMCIrB8X0iH4JdIIQF6L9fu/dNftPe2Ena34AnO51zDw4RvaUvndx/jIb8qsXE64HsXC4+Bip4gPCiI0xHLyJN6U3wxl5uXXJmObnhM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KXLAdGS1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KXLAdGS1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 18012C4CEF4; Sat, 16 Aug 2025 00:06:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755302762; bh=bUr/c79awvu8OWGK8+lJnU//Ku79FE8Xqz9RETWtyqk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KXLAdGS1dy2hxu45WdWoBAx+PzL/7iHNKWZd7EHjDUSbd9p4sU1ixyFCS1tEuoY9P F/bQyO5wpjnkt6HDQT3kzdhkBm/z0kr9jlV03Hz8LYUefSwmfR2wiKBvy5FvUnARB4 fHz+pYb7Ri9Mx4ZULWA0HUxb6nEeawgjCc2WOzI5BAhd8jMtQ4beO5lJbyKkbEFb8d Fh6jyDXTJBiqn4O+CCy7V7YEQrd5ORxkQ9MTkj5gCQBiLBPbCusmfXQbDcAG1B65U8 yBIxOZbeYXv/R0Q4qW+pD13/m6wyS5LrGjZ+YR2DO3HAk9MKyRjAY7fkWfwz8MNcz6 tBgacMYFCjCLg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 90731CE0EA9; Fri, 15 Aug 2025 17:06:00 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior , bpf@vger.kernel.org Subject: [PATCH v6 6/6] srcu: Document srcu_flip() memory-barrier D relation to SRCU-fast Date: Fri, 15 Aug 2025 17:05:59 -0700 Message-Id: <20250816000559.2622626-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The smp_mb() memory barrier at the end of srcu_flip() has a comment, but that comment does not make it clear that this memory barrier is an optimization, as opposed to being needed for correctness. This commit therefore adds this information and points out that it is omitted for SRCU-fast, where a much heavier weight synchronize_srcu() would be required. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: --- kernel/rcu/srcutree.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index c5e8ebc493d5ee..1ff94b76d91f15 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1168,6 +1168,16 @@ static void srcu_flip(struct srcu_struct *ssp) * counter update. Note that both this memory barrier and the * one in srcu_readers_active_idx_check() provide the guarantee * for __srcu_read_lock(). + * + * Note that this is a performance optimization, in which we spend + * an otherwise unnecessary smp_mb() in order to reduce the number + * of full per-CPU-variable scans in srcu_readers_lock_idx() and + * srcu_readers_unlock_idx(). But this performance optimization + * is not so optimal for SRCU-fast, where we would be spending + * not smp_mb(), but rather synchronize_rcu(). At the same time, + * the overhead of the smp_mb() is in the noise, so there is no + * point in omitting it in the SRCU-fast case. So the same code + * is executed either way. */ smp_mb(); /* D */ /* Pairs with C. */ } --=20 2.40.1