From nobody Mon Oct 6 12:06:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC0722DECC5; Mon, 21 Jul 2025 16:24:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753115075; cv=none; b=QGNHtr2YT/SKQMmt0/IkA1G/PvOPabDf+OnhTCBm/8z2L56guw8MfWmkqjOr86fwueCp1JUQUwvut6PbA7JnRF7f1srwssDVKj8kQTMCm54Co7jLy2M0QcF1ikuIYjVYETldD22djjhd8AqTqlPeJxl7scGfNaFiM2Fx22GGnOs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753115075; c=relaxed/simple; bh=qadST8dMScf+fp2b57F+1Eh6ZSs22RC1OPSOHO2JUcw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oxvR+s9MikCR43E2v68pMQ/Ay30gxNbzwgly/E8jBQdSgBym6wSyb2UIFhXpjwWMafe2DEszqHe9ZtKlrUYSmsdQgX1Bj+6gijKrskXF16Aia0GRw0Q9y1kfcFuW/Z/TYYqay+3czi2f6rRBnrR5IJ0TAfJvIirU9EiqE8siOAE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OE+sEf4d; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OE+sEf4d" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 57082C4CEF4; Mon, 21 Jul 2025 16:24:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753115075; bh=qadST8dMScf+fp2b57F+1Eh6ZSs22RC1OPSOHO2JUcw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OE+sEf4dz2CjS15tpmH1Chr9a5LEU4bN7btuUvx173JE7aG+RRSfwWDayURCat0fG 4OdfaZAI6Qy1sEKW3+KwdfnueR8s4uYpTWSrC5PfKlIVzdfLpbRFlmXysRcvuRimCl O747OWigeo52ChnMuM6Khd7bSxcSjmeXbswy4yzY6bA+AZpTAh+HzdaeQa9KkRVvE4 zrjFo0QHmTTSXCIUlRfgjYVtdoJpKUcIBo7pXqq+osbGpc5zlA/FR0qysxCDSU1DuB otM9MoXmNbInWq7FzwzErndd6htnPPdpDi1DqB/5wbhTTv6rhasSsV4TAArTet65wR 8Iei1/W5y2evA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0D0C3CE0C41; Mon, 21 Jul 2025 09:24:35 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior Subject: [PATCH v3 1/4] srcu: Move rcu_is_watching() checks to srcu_read_{,un}lock_fast() Date: Mon, 21 Jul 2025 09:24:30 -0700 Message-Id: <20250721162433.10454-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> References: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The rcu_is_watching() warnings are currently in the SRCU-tree implementations of __srcu_read_lock_fast() and __srcu_read_unlock_fast(). However, this makes it difficult to create _notrace variants of srcu_read_lock_fast() and srcu_read_unlock_fast(). This commit therefore moves these checks to srcu_read_lock_fast(), srcu_read_unlock_fast(), srcu_down_read_fast(), and srcu_up_read_fast(). Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Reviewed-by: Joel Fernandes --- include/linux/srcu.h | 4 ++++ include/linux/srcutree.h | 2 -- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index f179700fecafb..478c73d067f7d 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -275,6 +275,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast(struct srcu_struct * { struct srcu_ctr __percpu *retval; =20 + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast()."); srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); retval =3D __srcu_read_lock_fast(ssp); rcu_try_lock_acquire(&ssp->dep_map); @@ -295,6 +296,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_= fast(struct srcu_struct * static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_st= ruct *ssp) __acquires(ssp) { WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_down_read= _fast()."); srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); return __srcu_read_lock_fast(ssp); } @@ -389,6 +391,7 @@ static inline void srcu_read_unlock_fast(struct srcu_st= ruct *ssp, struct srcu_ct srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); srcu_lock_release(&ssp->dep_map); __srcu_read_unlock_fast(ssp, scp); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 /** @@ -405,6 +408,7 @@ static inline void srcu_up_read_fast(struct srcu_struct= *ssp, struct srcu_ctr __ WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi()); srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); __srcu_read_unlock_fast(ssp, scp); + RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_up_read_f= ast()."); } =20 /** diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index bf44d8d1e69ea..043b5a67ef71e 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -244,7 +244,6 @@ static inline struct srcu_ctr __percpu *__srcu_read_loc= k_fast(struct srcu_struct { struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); =20 - RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_lock= _fast()."); if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) this_cpu_inc(scp->srcu_locks.counter); /* Y */ else @@ -275,7 +274,6 @@ static inline void __srcu_read_unlock_fast(struct srcu_= struct *ssp, struct srcu_ this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ else atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); /* Z */ - RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); --=20 2.40.1 From nobody Mon Oct 6 12:06:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BBFD42DEA6E; Mon, 21 Jul 2025 16:24:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753115075; cv=none; b=DZOb2deT3twohjL5Sghg/YyiRw5qFpxBVxrj4znBTy45faDnRtjkg1OGPSeWqyEdtFQ9jm/o/9ixCFma93Ha6MmFtWSiWfndDeftQ0DWb4YaBpxlHn5SWWKF5VRnMGnOQaGAMvAUcl/AHtT1zLqV3QSaHGI1FT2QaXkJ0AmRYjU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753115075; c=relaxed/simple; bh=9QmLJANZefPN4fX6BWptBQk5yBPaFOqHuPr+mZ0Kvo4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=a2JdjR/9tVR2UKvTqDwa1MUv3CdGdqCx8ADSTUuFh66FHj7RuN0DGJ8HVxombKP323QnNYotjf6xZXpNXU6FSKMy+7hwQPLXsxwXGQRK1j0hquSYd3L/LSfYhfgWK0UG3TKU/XWXuzXoPVpbJ4ySnDc6rkFHwzFQ444/xlDklBM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Jq6F2f/g; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Jq6F2f/g" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 60846C4CEF6; Mon, 21 Jul 2025 16:24:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753115075; bh=9QmLJANZefPN4fX6BWptBQk5yBPaFOqHuPr+mZ0Kvo4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Jq6F2f/gSimRuTQNMi+gVsJIc3/EGJEmNdIFuBtjzbjK/LZ2WRuMxw0386rPM7yEw 2iVFcGg2JjFuDmEbs9QgkBpm1h2Cgf+i5G/+wFqPqdqq78AEMakRQdo2HBBw0URc+i an0do7TW+UaUUPo1QLwXTWkeSms7HyZ15Q9m0r2ENds/PIeu31K6CRy+GBka4y3TSC WnEInAflcdOBifcUei305Wy8l5cNODIa3tP61DRl1whiW8Fe/l26qPNCIImVH4cw08 K3T0hwgedPYbBjkvd+ySphv5dK6k8eCEDk7sw/bent6hAy418gLI6T9XiSqd/EHBd7 hw2SnhaEyWXtw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0F0CACE0CB3; Mon, 21 Jul 2025 09:24:35 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior Subject: [PATCH v3 2/4] srcu: Add srcu_read_lock_fast_notrace() and srcu_read_unlock_fast_notrace() Date: Mon, 21 Jul 2025 09:24:31 -0700 Message-Id: <20250721162433.10454-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> References: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds no-trace variants of the srcu_read_lock_fast() and srcu_read_unlock_fast() functions for tracing use. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Reviewed-by: Joel Fernandes --- include/linux/srcu.h | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 478c73d067f7d..7a692bf8f99b9 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -282,6 +282,20 @@ static inline struct srcu_ctr __percpu *srcu_read_lock= _fast(struct srcu_struct * return retval; } =20 +/* + * Used by tracing, cannot be traced and cannot call lockdep. + * See srcu_read_lock_fast() for more information. + */ +static inline struct srcu_ctr __percpu *srcu_read_lock_fast_notrace(struct= srcu_struct *ssp) + __acquires(ssp) +{ + struct srcu_ctr __percpu *retval; + + srcu_check_read_flavor_force(ssp, SRCU_READ_FLAVOR_FAST); + retval =3D __srcu_read_lock_fast(ssp); + return retval; +} + /** * srcu_down_read_fast - register a new reader for an SRCU-protected struc= ture. * @ssp: srcu_struct in which to register the new reader. @@ -394,6 +408,17 @@ static inline void srcu_read_unlock_fast(struct srcu_s= truct *ssp, struct srcu_ct RCU_LOCKDEP_WARN(!rcu_is_watching(), "RCU must be watching srcu_read_unlo= ck_fast()."); } =20 +/* + * Used by tracing, cannot be traced and cannot call lockdep. + * See srcu_read_unlock_fast() for more information. + */ +static inline void srcu_read_unlock_fast_notrace(struct srcu_struct *ssp, + struct srcu_ctr __percpu *scp) __releases(ssp) +{ + srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_FAST); + __srcu_read_unlock_fast(ssp, scp); +} + /** * srcu_up_read_fast - unregister a old reader from an SRCU-protected stru= cture. * @ssp: srcu_struct in which to unregister the old reader. --=20 2.40.1 From nobody Mon Oct 6 12:06:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACCB5283FC3; Mon, 21 Jul 2025 16:24:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753115075; cv=none; b=G2YvRATo8mqqQiswukoAqD8VjO1GSuAOES5Xly52EeVP9UPn6z3uVHUjErTrJ4QE0N3/wMg9vuNJ6crxScpaJkPPuUmt0ujGOo1t4jBVR0f6OJqbWEbZqx5UlqoQ02vYVShk9t4QmmuyckqDHT4XPVhmpJ+xP+EME1ZiwyMdOWQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753115075; c=relaxed/simple; bh=BlaBOIkXVR5xoqsab95nUX5wZC2ogd2y9wiA1pKiHvc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZNKPtOZ4k++z3D49DPgK1Gs24lMjapod5hyD/fjOq4SGUBbGqgGlcZPp67bIOqf++IhK/Ifp7x3umV6e9yUajrwHxSSs0/3c6TAkiUWRqfVbS2kPpkOQ/EysbESKgFlzihzGuiDln0xOGxf9wCO9nVGpGxEae+V1xE2LD15A5sA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mMU+GxNy; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mMU+GxNy" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7B7DFC4CEF9; Mon, 21 Jul 2025 16:24:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753115075; bh=BlaBOIkXVR5xoqsab95nUX5wZC2ogd2y9wiA1pKiHvc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mMU+GxNyXXdWdUG9kGhuq7GFpbXjwvHwvJ0zmAmXfIERL6R8AhFxEHSTeyE1t+jhV UkCBtHksg3x2KISLZCgWfZPrXVOD54O37RxDtvIndgcxUATz6A7DqCg+oRhuGYX99L zE0XMUmQEknfwjBY+G9Hl989Ye1laOLfkWghQrvPWqfaiuUo+ni90ynyUmwIj5rWse JCI4V/RIT1r5+VuxGmaTg/TRl2kZonvl0fAKAyAxNetfOO8t3tpCs7OeOQEn5ocfMi /3GKZufahpP101Uh8ayWclRyupvS+9DqSqqTsIYdHZWlH90i2MK3GcY+p7Np1T+Lqh OSJg+kHJ3vpsA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 11B9CCE0D3A; Mon, 21 Jul 2025 09:24:35 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior Subject: [PATCH v3 3/4] srcu: Add guards for notrace variants of SRCU-fast readers Date: Mon, 21 Jul 2025 09:24:32 -0700 Message-Id: <20250721162433.10454-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> References: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This adds the usual scoped_guard(srcu_fast_notrace, &my_srcu) and guard(srcu_fast_notrace)(&my_srcu). Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Reviewed-by: Joel Fernandes --- include/linux/srcu.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 7a692bf8f99b9..ada65b58bc4c5 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -515,4 +515,9 @@ DEFINE_LOCK_GUARD_1(srcu_fast, struct srcu_struct, srcu_read_unlock_fast(_T->lock, _T->scp), struct srcu_ctr __percpu *scp) =20 +DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, + _T->scp =3D srcu_read_lock_fast_notrace(_T->lock), + srcu_read_unlock_fast_notrace(_T->lock, _T->scp), + struct srcu_ctr __percpu *scp) + #endif --=20 2.40.1 From nobody Mon Oct 6 12:06:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D09E12DFA37; Mon, 21 Jul 2025 16:24:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753115075; cv=none; b=Z595DzcRd+xUTs36RAhPn1CzRCPHB4jVa1XrKNm6gkkMOGdPNgruOkwydapVpOrq6ztlm8SvIU/DIZy+oV0WcjOe0WbwPo7OrZe0fNgBD/+ecWUw8qjn3cnZErEe+3TrZ2FPE5KNxoOqna9CMhV/J2vXCHrEuJ6AdACD/emzz6M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753115075; c=relaxed/simple; bh=Fw6xgVyZPHlGcRlgHk3BS90lx5Y6xVMXcZ2s8SWzVHc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jLCFguneoGLRG7ZiBlrxcfQMnYh/RmHacRJD7nCBOEEfOXCYPIsU+gRtXl6/eRvXMcWNA4KwK5WBAft4i0BM0mV8G+Z+I8Gae7kuSDcATD9/x3DjvJnKmVK1V/yP6WL2xzKjAI8galm+9TKJ35S9JvsFAGE7GfHnwoXik/wpduU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jvl7MVrU; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jvl7MVrU" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 76BEAC4CEF8; Mon, 21 Jul 2025 16:24:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753115075; bh=Fw6xgVyZPHlGcRlgHk3BS90lx5Y6xVMXcZ2s8SWzVHc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jvl7MVrU2WNaAl1YUCU/7NxkhZmO73XfXhU7zbb/kTTrRJYo31lmYM8cFVXwqz4Vw mlmmo/d9gtf2tL6TasVkm38z729F/WOgIvRmsoQDqDfx05bSg5ArqR8mqlDDoOvpmg 4VWl+9eWNWZnAJIaQr7TRCWeo5KIXIlpB1tlc8O/KFGzbMItP/NQQP+Fm2JWjz2frp Gnlkj15JCd2lb4Sbv1zOpRdBGqLy7N+FYUEJf06IgtmoJ/h61Uv4nNiHC5mlrvHk5c JW+Ds+ushoHayp8mHVwN6bzCF+EnF/B+/m9WQayqkcsl3+W5T9UALoS5Nro31WZ/gw heHYwdOhKIhjw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 1474ECE0E09; Mon, 21 Jul 2025 09:24:35 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Mathieu Desnoyers , Sebastian Andrzej Siewior Subject: [PATCH v3 4/4] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast Date: Mon, 21 Jul 2025 09:24:33 -0700 Message-Id: <20250721162433.10454-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> References: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current use of guard(preempt_notrace)() within __DECLARE_TRACE() to protect invocation of __DO_TRACE_CALL() means that BPF programs attached to tracepoints are non-preemptible. This is unhelpful in real-time systems, whose users apparently wish to use BPF while also achieving low latencies. (Who knew?) One option would be to use preemptible RCU, but this introduces many opportunities for infinite recursion, which many consider to be counterproductive, especially given the relatively small stacks provided by the Linux kernel. These opportunities could be shut down by sufficiently energetic duplication of code, but this sort of thing is considered impolite in some circles. Therefore, use the shiny new SRCU-fast API, which provides somewhat faster readers than those of preemptible RCU, at least on my laptop, where task_struct access is more expensive than access to per-CPU variables. And SRCU fast provides way faster readers than does SRCU, courtesy of being able to avoid the read-side use of smp_mb(). Also, it is quite straightforward to create srcu_read_{,un}lock_fast_notrace() functions. While in the area, SRCU now supports early boot call_srcu(). Therefore, remove the checks that used to avoid such use from rcu_free_old_probes() before this commit was applied: e53244e2c893 ("tracepoint: Remove SRCU protection") The current commit can be thought of as an approximate revert of that commit. Link: https://lore.kernel.org/all/20250613152218.1924093-1-bigeasy@linutron= ix.de/ Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior --- include/linux/tracepoint.h | 6 ++++-- kernel/tracepoint.c | 21 ++++++++++++++++++++- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 826ce3f8e1f85..a22c1ab88560b 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -33,6 +33,8 @@ struct trace_eval_map { =20 #define TRACEPOINT_DEFAULT_PRIO 10 =20 +extern struct srcu_struct tracepoint_srcu; + extern int tracepoint_probe_register(struct tracepoint *tp, void *probe, void *data); extern int @@ -115,7 +117,7 @@ void for_each_tracepoint_in_module(struct module *mod, static inline void tracepoint_synchronize_unregister(void) { synchronize_rcu_tasks_trace(); - synchronize_rcu(); + synchronize_srcu(&tracepoint_srcu); } static inline bool tracepoint_is_faultable(struct tracepoint *tp) { @@ -271,7 +273,7 @@ static inline struct tracepoint *tracepoint_ptr_deref(t= racepoint_ptr_t *p) static inline void __do_trace_##name(proto) \ { \ if (cond) { \ - guard(preempt_notrace)(); \ + guard(srcu_fast_notrace)(&tracepoint_srcu); \ __DO_TRACE_CALL(name, TP_ARGS(args)); \ } \ } \ diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 62719d2941c90..e19973015cbd7 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -25,6 +25,9 @@ enum tp_func_state { extern tracepoint_ptr_t __start___tracepoints_ptrs[]; extern tracepoint_ptr_t __stop___tracepoints_ptrs[]; =20 +DEFINE_SRCU(tracepoint_srcu); +EXPORT_SYMBOL_GPL(tracepoint_srcu); + enum tp_transition_sync { TP_TRANSITION_SYNC_1_0_1, TP_TRANSITION_SYNC_N_2_1, @@ -34,6 +37,7 @@ enum tp_transition_sync { =20 struct tp_transition_snapshot { unsigned long rcu; + unsigned long srcu_gp; bool ongoing; }; =20 @@ -46,6 +50,7 @@ static void tp_rcu_get_state(enum tp_transition_sync sync) =20 /* Keep the latest get_state snapshot. */ snapshot->rcu =3D get_state_synchronize_rcu(); + snapshot->srcu_gp =3D start_poll_synchronize_srcu(&tracepoint_srcu); snapshot->ongoing =3D true; } =20 @@ -56,6 +61,8 @@ static void tp_rcu_cond_sync(enum tp_transition_sync sync) if (!snapshot->ongoing) return; cond_synchronize_rcu(snapshot->rcu); + if (!poll_state_synchronize_srcu(&tracepoint_srcu, snapshot->srcu_gp)) + synchronize_srcu(&tracepoint_srcu); snapshot->ongoing =3D false; } =20 @@ -101,17 +108,29 @@ static inline void *allocate_probes(int count) return p =3D=3D NULL ? NULL : p->probes; } =20 -static void rcu_free_old_probes(struct rcu_head *head) +static void srcu_free_old_probes(struct rcu_head *head) { kfree(container_of(head, struct tp_probes, rcu)); } =20 +static void rcu_free_old_probes(struct rcu_head *head) +{ + call_srcu(&tracepoint_srcu, head, srcu_free_old_probes); +} + static inline void release_probes(struct tracepoint *tp, struct tracepoint= _func *old) { if (old) { struct tp_probes *tp_probes =3D container_of(old, struct tp_probes, probes[0]); =20 + /* + * Tracepoint probes are protected by either RCU or + * Tasks Trace RCU and also by SRCU. By calling the SRCU + * callback in the [Tasks Trace] RCU callback we cover + * both cases. So let us chain the SRCU and [Tasks Trace] + * RCU callbacks to wait for both grace periods. + */ if (tracepoint_is_faultable(tp)) call_rcu_tasks_trace(&tp_probes->rcu, rcu_free_old_probes); else --=20 2.40.1 From nobody Mon Oct 6 12:06:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BEBC286880; Tue, 22 Jul 2025 22:16:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753222593; cv=none; b=HJeH0F2UXaml6Kc3ajeyjJROnUswvZAF2nrzCVNzRUAw1UdCTyMB5d1dDv6bO36NU3Fh3YxknAUQkmKSNc38tVCSDEUseuY8MFDaoEbNiix71TfKP6l16kIrq4FJGtlV0f/qowaxKiy/w5yezrHwW2eiCoDOrugIbUNqIe252Uo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753222593; c=relaxed/simple; bh=XP+sVWW4AN1gL3AcsGgjJ89qOVHp/Q606n0jAmQCKsw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=catmzsmIjA8NidKnWXR5OxS2PT4R6vD+OkRW9yhtYAq+wCwnsg3LgJqV2/XbB+6FX1DUqYMed+zMHuXJnkxAi5c3CzULrBYpFUmHFUvNa43gH2rbrXSP1QsMtJkXPXNWEP9aHsDvx3gakA6Q2KEA609/KPydC+7voGS+5jnk3cM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aGBtmpHa; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aGBtmpHa" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 37957C4CEEB; Tue, 22 Jul 2025 22:16:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753222592; bh=XP+sVWW4AN1gL3AcsGgjJ89qOVHp/Q606n0jAmQCKsw=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=aGBtmpHaQSKXa/vldcrNntKFVYSfSiWQvcLpMUIvi85TIyRCG9z8Qv07Uv6af4NEZ yR2K4t5GKVJ5dOi9/sJxMyAuERQmfAGMU9Ek/nhz0QiSFnAVXAXSQx8jGzlt9yEVyL y5+CT+b9PF4Ni7ETQcvWbAfpEfYbYG4P37hD1i6agKbOSL+1odZXI5In4ga0NqWX1B ExVwnhwTYCerfMNRpRO1CY0UqVn/c+7ocJ/cm7sffQvCDwJON7vSxl/ojUXM0J6cF6 7JeQLGMxBgRgkURQ3uYpKMJmSKmq5RUemPm7GH5W1lyEcgC8NoxZNz2dA7ppzPtZwQ x4/ZWaIYshCIA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id CD651CE0CF0; Tue, 22 Jul 2025 15:16:31 -0700 (PDT) Date: Tue, 22 Jul 2025 15:16:31 -0700 From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, bpf@vger.kernel.org Subject: [PATCH v3 5/4] srcu: Document __srcu_read_{,un}lock_fast() implicit RCU readers Message-ID: Reply-To: paulmck@kernel.org References: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit documents the implicit RCU readers that are implied by the this_cpu_inc() and atomic_long_inc() operations in __srcu_read_lock_fast() and __srcu_read_unlock_fast(). While in the area, fix the documentation of the memory pairing of atomic_long_inc() in __srcu_read_lock_fast(). Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 043b5a67ef71e..78e1a7b845ba9 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -245,9 +245,9 @@ static inline struct srcu_ctr __percpu *__srcu_read_loc= k_fast(struct srcu_struct struct srcu_ctr __percpu *scp =3D READ_ONCE(ssp->srcu_ctrp); =20 if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) - this_cpu_inc(scp->srcu_locks.counter); /* Y */ + this_cpu_inc(scp->srcu_locks.counter); // Y, and implicit RCU reader. else - atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); /* Z */ + atomic_long_inc(raw_cpu_ptr(&scp->srcu_locks)); // Y, and implicit RCU = reader. barrier(); /* Avoid leaking the critical section. */ return scp; } @@ -271,9 +271,9 @@ static inline void __srcu_read_unlock_fast(struct srcu_= struct *ssp, struct srcu_ { barrier(); /* Avoid leaking the critical section. */ if (!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) - this_cpu_inc(scp->srcu_unlocks.counter); /* Z */ + this_cpu_inc(scp->srcu_unlocks.counter); // Z, and implicit RCU reader. else - atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); /* Z */ + atomic_long_inc(raw_cpu_ptr(&scp->srcu_unlocks)); // Z, and implicit RC= U reader. } =20 void __srcu_check_read_flavor(struct srcu_struct *ssp, int read_flavor); From nobody Mon Oct 6 12:06:20 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0FD38284B5D; Tue, 22 Jul 2025 22:17:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753222677; cv=none; b=ewZDZllDMMzW6JhiQUie52qRr/HOKJX5LQtt6S/vUOejR1j7QAelqMFZwg77LXtqK1T/4X1ZCr7TkYeVS9zeE0psB40xnryjWhQGQUy7Nca/zQZvtmg30J0XxlJc4DIttap+4PJCXcl1FqMaOifFoUYeLdgjYpT/cLokhtr4dRE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753222677; c=relaxed/simple; bh=zemiTL/QDNWqSJxFQuFc22tZnd8puCgIlJWLNjceOvQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=IrTDUR7RafcyBLlNLpVHd745hb+E2Od7mxTQNj9oaS+TWwmX6IRnq0ER1b7yRfwE9gXQZ99Z1i8WU/Jbgljs1SM8BTxre3jqKIxG8P5NvWQQJk/ua31T8w6OswZCGR10t18Nh+XQl6h1aWgrpWXqIldOZ9p0aEj4ITfV3AYPGos= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=C6mRlixQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="C6mRlixQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E58CC4CEEB; Tue, 22 Jul 2025 22:17:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1753222676; bh=zemiTL/QDNWqSJxFQuFc22tZnd8puCgIlJWLNjceOvQ=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=C6mRlixQsArvQp5SFM7L9c6Ps8ntfC3C09109/p7fsMBzK4YSEyelxcSb+Dm+mUwB C1G7JtMFWJUB6f8h4nHauMkWB8WT5rPbeQrN4tq6/BHNKowSBsoWNBpXxXCT+BmnY0 jqc76JoB/GiagC4yyHzin3sYuW6miQo0mQoNykZ+xSG8d89mroa+D2Zdx81VTGCCQO 5YvaLDYE5cK7fUl0cpMarEMrL/8y87vg0o5geQ/8k+ydtV1Mswc7SoGtGL+Oz/xw6M s0yiI9EgyfXW1lRWNMiIOg/PSdBvrJYPTJQCSB8tUKrqMcIs/CajDBrSj3lYiJL1cy 3LEEPW7dNxr2w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 59409CE0CF0; Tue, 22 Jul 2025 15:17:56 -0700 (PDT) Date: Tue, 22 Jul 2025 15:17:56 -0700 From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, bpf@vger.kernel.org Subject: [PATCH v3 6/4] srcu: Document srcu_flip() memory-barrier D relation to SRCU-fast Message-ID: <9d53dfbd-3b6a-4110-a65e-dadb8fc5066b@paulmck-laptop> Reply-To: paulmck@kernel.org References: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <7387f0c2-75bc-420d-aa7e-3a9ac72d369c@paulmck-laptop> Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The smp_mb() memory barrier at the end of srcu_flip() has a comment, but that comment does not make it clear that this memory barrier is an optimization, as opposed to being needed for correctness. This commit therefore adds this information and points out that it is omitted for SRCU-fast, where a much heavier weight synchronize_srcu() would be required. Signed-off-by: Paul E. McKenney Cc: Mathieu Desnoyers Cc: Steven Rostedt Cc: Sebastian Andrzej Siewior Cc: diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index c5e8ebc493d5e..1ff94b76d91f1 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1168,6 +1168,16 @@ static void srcu_flip(struct srcu_struct *ssp) * counter update. Note that both this memory barrier and the * one in srcu_readers_active_idx_check() provide the guarantee * for __srcu_read_lock(). + * + * Note that this is a performance optimization, in which we spend + * an otherwise unnecessary smp_mb() in order to reduce the number + * of full per-CPU-variable scans in srcu_readers_lock_idx() and + * srcu_readers_unlock_idx(). But this performance optimization + * is not so optimal for SRCU-fast, where we would be spending + * not smp_mb(), but rather synchronize_rcu(). At the same time, + * the overhead of the smp_mb() is in the noise, so there is no + * point in omitting it in the SRCU-fast case. So the same code + * is executed either way. */ smp_mb(); /* D */ /* Pairs with C. */ }