From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2241EC433EF for ; Tue, 19 Apr 2022 00:03:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237309AbiDSAG3 (ORCPT ); Mon, 18 Apr 2022 20:06:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236888AbiDSAGI (ORCPT ); Mon, 18 Apr 2022 20:06:08 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B9AF13F97; Mon, 18 Apr 2022 17:03:27 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E0A6AB81128; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A09EDC385A8; Tue, 19 Apr 2022 00:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326604; bh=y1siE5tC9sWo5al2WNPETNUZe9pTnoQcxs2YYb0TG44=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KYMb70gCftj/SsILItuVHxmEewLDfc6FeqyqAXtBKC8MvCtkwglw+Xzurk3oZs7nn fKDQnG4PFKzHayisRHaTsqF6f82yllXla7fY5TkFtSkHwQu+6M88eaLQYU8mvhWwbf WOULLvIe7At2sgO3iUWvK8+8BXh7FFK0DRa9ojlZzhT5M2Zysp0zDFRqtCPcpyvZtY f0mybrPKp1Qhg8IR13wfTSjg+0VE6JdJCqRQWosQBPRVKn4xJAYkPNiDgZ5ChJyNSb YxC/y31NghQpOFfLyLIlShOuDDc0jI8GHPeXvjSAwoVX/Ker43nvNMlGxvFsryp8+T WEFwz+B9mm3Vg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 4D08B5C04BD; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 01/21] srcu: Tighten cleanup_srcu_struct() GP checks Date: Mon, 18 Apr 2022 17:03:02 -0700 Message-Id: <20220419000322.3948903-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, cleanup_srcu_struct() checks for a grace period in progress, but it does not check for a grace period that has not yet started but which might start at any time. Such a situation could result in a use-after-free bug, so this commit adds a check for a grace period that is needed but not yet started to cleanup_srcu_struct(). Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 6833d8887181..d30e4db04506 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -382,9 +382,11 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) return; /* Forgot srcu_barrier(), so just leak it! */ } if (WARN_ON(rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)) !=3D SRCU_STATE_ID= LE) || + WARN_ON(rcu_seq_current(&ssp->srcu_gp_seq) !=3D ssp->srcu_gp_seq_need= ed) || WARN_ON(srcu_readers_active(ssp))) { - pr_info("%s: Active srcu_struct %p state: %d\n", - __func__, ssp, rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq))); + pr_info("%s: Active srcu_struct %p read state: %d gp state: %lu/%lu\n", + __func__, ssp, rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)), + rcu_seq_current(&ssp->srcu_gp_seq), ssp->srcu_gp_seq_needed); return; /* Caller forgot to stop doing call_srcu()? */ } free_percpu(ssp->sda); --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDCCCC433EF for ; Tue, 19 Apr 2022 00:03:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237076AbiDSAGe (ORCPT ); Mon, 18 Apr 2022 20:06:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236902AbiDSAGI (ORCPT ); Mon, 18 Apr 2022 20:06:08 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DAB723159; Mon, 18 Apr 2022 17:03:27 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 01DC4B81141; Tue, 19 Apr 2022 00:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9AE32C385A9; Tue, 19 Apr 2022 00:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326604; bh=R6yVW5CkBqUs8J0Ergawy2eOvj41C2+kBPhfDbE3hE4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hkMpzHDdAA+7YDs/T/Qxc2I6UxKpodNNxSXtwLiIghxU/AIQrR6Ze9HLlBI6ZMGpN 7Pi/SGf6aW8KJi9DpDx9eKBTF5RcI10YRi6n3J2kN/POvrGCAoc8wfrmXjyHsim35f arDWx/p8lH5AHDSxInfBRZd2g0JzTNASDDHJpObN7SuWKNAfHrqFn1M/jRuVFVVWhA XoDGYveepKAbeqiP884rkT2mfNmYwIK7fxz/gqft3CDMZMAlsECJucBriFGmv/RBuh /chylE04iSy1qjvXA/4+vCzGARUCsrosKip7Hc0GUAzsUR1ZJD3dF/ypfjRWkH4Kmd M2mvH3AXXSaTQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 501265C04C6; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 02/21] srcu: Fix s/is/if/ typo in srcu_node comment Date: Mon, 18 Apr 2022 17:03:03 -0700 Message-Id: <20220419000322.3948903-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit fixed a typo in the srcu_node structure's ->srcu_have_cbs comment. While in the area, redo a couple of comments to take advantage of 100-character line lengths. Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index cb1f4351e8ba..4025840ba9a3 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -47,11 +47,9 @@ struct srcu_data { */ struct srcu_node { spinlock_t __private lock; - unsigned long srcu_have_cbs[4]; /* GP seq for children */ - /* having CBs, but only */ - /* is > ->srcu_gq_seq. */ - unsigned long srcu_data_have_cbs[4]; /* Which srcu_data structs */ - /* have CBs for given GP? */ + unsigned long srcu_have_cbs[4]; /* GP seq for children having CBs, but o= nly */ + /* if greater than ->srcu_gq_seq. */ + unsigned long srcu_data_have_cbs[4]; /* Which srcu_data structs have CBs = for given GP? */ unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */ struct srcu_node *srcu_parent; /* Next up in tree. */ int grplo; /* Least CPU for node. */ --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA4F2C4332F for ; Tue, 19 Apr 2022 00:03:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236913AbiDSAGI (ORCPT ); Mon, 18 Apr 2022 20:06:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52692 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229920AbiDSAGG (ORCPT ); Mon, 18 Apr 2022 20:06:06 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCE2213F97; Mon, 18 Apr 2022 17:03:25 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 52C5261297; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A39B9C385AB; Tue, 19 Apr 2022 00:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326604; bh=ZZ4EmnWyc2TdVDNKqE6NcBxGvAGwflG6QiNpPkNuJC4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ha8RHPGuMI1zI4mVn5mllWh3XNLkKtKmVxZkmbdYEj3racbDox6/QJSbdtbphKH7w J5yRCFmzYQ4cQuerZfTFTZ6p1q6oSCH8ILXrvuVSzRobUd4iJLLBPJfKj2z+wqAIR7 OCjjGDYwqjHsjqyzlDQJsk3ahHbXuPVy7jfH2cNGWOkd5cOg2qV9lauDnC3oguxf+/ qhdamWDMhyrYRbbT1p5GnNg/mRepxLEJ4o4TIlcx8kCLGQhnf1M1YI011qEr4MoLjr l+iCpEnPPsfPOaIe8CLojQKBVEys3MbM/lHaxeAL02c6JflakzOzcltqQyycCp21rA NghOTjSwwCSDA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 51EFD5C0848; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 03/21] srcu: Make srcu_funnel_gp_start() cache ->mynode in snp_leaf Date: Mon, 18 Apr 2022 17:03:04 -0700 Message-Id: <20220419000322.3948903-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, the srcu_funnel_gp_start() walks its local variable snp up the tree and reloads sdp->mynode whenever it is necessary to check whether it is still at the leaf srcu_node level. This works, but is a bit more obtuse than absolutely necessary. In addition, upcoming commits will dynamically size srcu_struct structures, in which case sdp->mynode will no longer necessarily be a constant, and this commit helps prepare for that dynamic sizing. Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index d30e4db04506..7d13e35e5d27 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -632,20 +632,21 @@ static void srcu_funnel_gp_start(struct srcu_struct *= ssp, struct srcu_data *sdp, { unsigned long flags; int idx =3D rcu_seq_ctr(s) % ARRAY_SIZE(sdp->mynode->srcu_have_cbs); - struct srcu_node *snp =3D sdp->mynode; + struct srcu_node *snp; + struct srcu_node *snp_leaf =3D sdp->mynode; unsigned long snp_seq; =20 /* Each pass through the loop does one level of the srcu_node tree. */ - for (; snp !=3D NULL; snp =3D snp->srcu_parent) { - if (rcu_seq_done(&ssp->srcu_gp_seq, s) && snp !=3D sdp->mynode) + for (snp =3D snp_leaf; snp !=3D NULL; snp =3D snp->srcu_parent) { + if (rcu_seq_done(&ssp->srcu_gp_seq, s) && snp !=3D snp_leaf) return; /* GP already done and CBs recorded. */ spin_lock_irqsave_rcu_node(snp, flags); if (ULONG_CMP_GE(snp->srcu_have_cbs[idx], s)) { snp_seq =3D snp->srcu_have_cbs[idx]; - if (snp =3D=3D sdp->mynode && snp_seq =3D=3D s) + if (snp =3D=3D snp_leaf && snp_seq =3D=3D s) snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; spin_unlock_irqrestore_rcu_node(snp, flags); - if (snp =3D=3D sdp->mynode && snp_seq !=3D s) { + if (snp =3D=3D snp_leaf && snp_seq !=3D s) { srcu_schedule_cbs_sdp(sdp, do_norm ? SRCU_INTERVAL : 0); @@ -656,7 +657,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ss= p, struct srcu_data *sdp, return; } snp->srcu_have_cbs[idx] =3D s; - if (snp =3D=3D sdp->mynode) + if (snp =3D=3D snp_leaf) snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; if (!do_norm && ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, s)) WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA692C433F5 for ; Tue, 19 Apr 2022 00:03:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236857AbiDSAGN (ORCPT ); Mon, 18 Apr 2022 20:06:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52728 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236810AbiDSAGH (ORCPT ); Mon, 18 Apr 2022 20:06:07 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9A6F21816; Mon, 18 Apr 2022 17:03:25 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3D4D0612C4; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 968BEC385A1; Tue, 19 Apr 2022 00:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326604; bh=/HCo85xQM6HRnnblLcN7u6Q2a90k5H0ufJjNTjySAKw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KIQ23jKW5KJdsUEkqSW23ZJq/oUnDzogK/vbHINtol3iMl6fdx/IU6+S0K3vZOxlx yF1fT4/7Mlx/KHXU9vQeqr/VANEiyQoXse7HyqghHVph/kwvJ1tPlGLU4IrBPVe5Qn qA8HLVCGyvOM9kDe+dOYGpkcJQUqPvw/LsHbx9rGXC++pkfyic4Yx51NLn0dhBJy7E cBIiqRPnPOAoQj+UAUpN6e5nC4bSXsp80LoarHQEzsLnZQrCm+N5dGRPpH/lH6wmbE 8EZNwxeRhqO4P/z2TMfT92mMYnr6/cacS9UnOFHFjsPoU9DbSnIjibL97FlVrhsoDz fxmoIt3gh5+9A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 5415C5C0A0D; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 04/21] srcu: Make Tree SRCU able to operate without snp_node array Date: Mon, 18 Apr 2022 17:03:05 -0700 Message-Id: <20220419000322.3948903-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit makes Tree SRCU able to operate without an snp_node array, that is, when the srcu_data structures' ->mynode pointers are NULL. This can result in high contention on the srcu_struct structure's ->lock, but only when there are lots of call_srcu(), synchronize_srcu(), and synchronize_srcu_expedited() calls. Note that when there is no snp_node array, all SRCU callbacks use CPU 0's callback queue. This is optimal in the common case of low update-side load because it removes the need to search each CPU for the single callback that made the grace period happen. Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 14 ++- kernel/rcu/srcutree.c | 203 +++++++++++++++++++++------------------ 2 files changed, 124 insertions(+), 93 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 4025840ba9a3..8d1da136a93a 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -63,8 +63,9 @@ struct srcu_struct { struct srcu_node node[NUM_RCU_NODES]; /* Combining tree. */ struct srcu_node *level[RCU_NUM_LVLS + 1]; /* First node at each level. */ + int srcu_size_state; /* Small-to-big transition state. */ struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ - spinlock_t __private lock; /* Protect counters */ + spinlock_t __private lock; /* Protect counters and size state. */ struct mutex srcu_gp_mutex; /* Serialize GP work. */ unsigned int srcu_idx; /* Current rdr array element. */ unsigned long srcu_gp_seq; /* Grace-period seq #. */ @@ -83,6 +84,17 @@ struct srcu_struct { struct lockdep_map dep_map; }; =20 +/* Values for size state variable (->srcu_size_state). */ +#define SRCU_SIZE_SMALL 0 +#define SRCU_SIZE_ALLOC 1 +#define SRCU_SIZE_WAIT_BARRIER 2 +#define SRCU_SIZE_WAIT_CALL 3 +#define SRCU_SIZE_WAIT_CBS1 4 +#define SRCU_SIZE_WAIT_CBS2 5 +#define SRCU_SIZE_WAIT_CBS3 6 +#define SRCU_SIZE_WAIT_CBS4 7 +#define SRCU_SIZE_BIG 8 + /* Values for state variable (bottom bits of ->srcu_gp_seq). */ #define SRCU_STATE_IDLE 0 #define SRCU_STATE_SCAN1 1 diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 7d13e35e5d27..e23696edd43b 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -152,16 +152,17 @@ static void init_srcu_struct_nodes(struct srcu_struct= *ssp) sdp->ssp =3D ssp; sdp->grpmask =3D 1 << (cpu - sdp->mynode->grplo); } + smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_WAIT_BARRIER); } =20 /* * Initialize non-compile-time initialized fields, including the - * associated srcu_node and srcu_data structures. The is_static - * parameter is passed through to init_srcu_struct_nodes(), and - * also tells us that ->sda has already been wired up to srcu_data. + * associated srcu_node and srcu_data structures. The is_static parameter + * tells us that ->sda has already been wired up to srcu_data. */ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) { + ssp->srcu_size_state =3D SRCU_SIZE_SMALL; mutex_init(&ssp->srcu_cb_mutex); mutex_init(&ssp->srcu_gp_mutex); ssp->srcu_idx =3D 0; @@ -175,6 +176,7 @@ static int init_srcu_struct_fields(struct srcu_struct *= ssp, bool is_static) if (!ssp->sda) return -ENOMEM; init_srcu_struct_nodes(ssp); + ssp->srcu_size_state =3D SRCU_SIZE_BIG; ssp->srcu_gp_seq_needed_exp =3D 0; ssp->srcu_last_gp_end =3D ktime_get_mono_fast_ns(); smp_store_release(&ssp->srcu_gp_seq_needed, 0); /* Init done. */ @@ -391,6 +393,7 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) } free_percpu(ssp->sda); ssp->sda =3D NULL; + ssp->srcu_size_state =3D SRCU_SIZE_SMALL; } EXPORT_SYMBOL_GPL(cleanup_srcu_struct); =20 @@ -439,6 +442,10 @@ static void srcu_gp_start(struct srcu_struct *ssp) struct srcu_data *sdp =3D this_cpu_ptr(ssp->sda); int state; =20 + if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) + sdp =3D per_cpu_ptr(ssp->sda, 0); + else + sdp =3D this_cpu_ptr(ssp->sda); lockdep_assert_held(&ACCESS_PRIVATE(ssp, lock)); WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); spin_lock_rcu_node(sdp); /* Interrupts already disabled. */ @@ -539,38 +546,40 @@ static void srcu_gp_end(struct srcu_struct *ssp) /* A new grace period can start at this point. But only one. */ =20 /* Initiate callback invocation as needed. */ - idx =3D rcu_seq_ctr(gpseq) % ARRAY_SIZE(snp->srcu_have_cbs); - srcu_for_each_node_breadth_first(ssp, snp) { - spin_lock_irq_rcu_node(snp); - cbs =3D false; - last_lvl =3D snp >=3D ssp->level[rcu_num_lvls - 1]; - if (last_lvl) - cbs =3D snp->srcu_have_cbs[idx] =3D=3D gpseq; - snp->srcu_have_cbs[idx] =3D gpseq; - rcu_seq_set_state(&snp->srcu_have_cbs[idx], 1); - if (ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, gpseq)) - WRITE_ONCE(snp->srcu_gp_seq_needed_exp, gpseq); - mask =3D snp->srcu_data_have_cbs[idx]; - snp->srcu_data_have_cbs[idx] =3D 0; - spin_unlock_irq_rcu_node(snp); - if (cbs) - srcu_schedule_cbs_snp(ssp, snp, mask, cbdelay); - - /* Occasionally prevent srcu_data counter wrap. */ - if (!(gpseq & counter_wrap_check) && last_lvl) - for (cpu =3D snp->grplo; cpu <=3D snp->grphi; cpu++) { - sdp =3D per_cpu_ptr(ssp->sda, cpu); - spin_lock_irqsave_rcu_node(sdp, flags); - if (ULONG_CMP_GE(gpseq, - sdp->srcu_gp_seq_needed + 100)) - sdp->srcu_gp_seq_needed =3D gpseq; - if (ULONG_CMP_GE(gpseq, - sdp->srcu_gp_seq_needed_exp + 100)) - sdp->srcu_gp_seq_needed_exp =3D gpseq; - spin_unlock_irqrestore_rcu_node(sdp, flags); - } + if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) { + srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, 0), cbdelay); + } else { + idx =3D rcu_seq_ctr(gpseq) % ARRAY_SIZE(snp->srcu_have_cbs); + srcu_for_each_node_breadth_first(ssp, snp) { + spin_lock_irq_rcu_node(snp); + cbs =3D false; + last_lvl =3D snp >=3D ssp->level[rcu_num_lvls - 1]; + if (last_lvl) + cbs =3D snp->srcu_have_cbs[idx] =3D=3D gpseq; + snp->srcu_have_cbs[idx] =3D gpseq; + rcu_seq_set_state(&snp->srcu_have_cbs[idx], 1); + if (ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, gpseq)) + WRITE_ONCE(snp->srcu_gp_seq_needed_exp, gpseq); + mask =3D snp->srcu_data_have_cbs[idx]; + snp->srcu_data_have_cbs[idx] =3D 0; + spin_unlock_irq_rcu_node(snp); + if (cbs) + srcu_schedule_cbs_snp(ssp, snp, mask, cbdelay); + } } =20 + /* Occasionally prevent srcu_data counter wrap. */ + if (!(gpseq & counter_wrap_check)) + for_each_possible_cpu(cpu) { + sdp =3D per_cpu_ptr(ssp->sda, cpu); + spin_lock_irqsave_rcu_node(sdp, flags); + if (ULONG_CMP_GE(gpseq, sdp->srcu_gp_seq_needed + 100)) + sdp->srcu_gp_seq_needed =3D gpseq; + if (ULONG_CMP_GE(gpseq, sdp->srcu_gp_seq_needed_exp + 100)) + sdp->srcu_gp_seq_needed_exp =3D gpseq; + spin_unlock_irqrestore_rcu_node(sdp, flags); + } + /* Callback initiation done, allow grace periods after next. */ mutex_unlock(&ssp->srcu_cb_mutex); =20 @@ -599,18 +608,19 @@ static void srcu_funnel_exp_start(struct srcu_struct = *ssp, struct srcu_node *snp { unsigned long flags; =20 - for (; snp !=3D NULL; snp =3D snp->srcu_parent) { - if (rcu_seq_done(&ssp->srcu_gp_seq, s) || - ULONG_CMP_GE(READ_ONCE(snp->srcu_gp_seq_needed_exp), s)) - return; - spin_lock_irqsave_rcu_node(snp, flags); - if (ULONG_CMP_GE(snp->srcu_gp_seq_needed_exp, s)) { + if (snp) + for (; snp !=3D NULL; snp =3D snp->srcu_parent) { + if (rcu_seq_done(&ssp->srcu_gp_seq, s) || + ULONG_CMP_GE(READ_ONCE(snp->srcu_gp_seq_needed_exp), s)) + return; + spin_lock_irqsave_rcu_node(snp, flags); + if (ULONG_CMP_GE(snp->srcu_gp_seq_needed_exp, s)) { + spin_unlock_irqrestore_rcu_node(snp, flags); + return; + } + WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); spin_unlock_irqrestore_rcu_node(snp, flags); - return; } - WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); - spin_unlock_irqrestore_rcu_node(snp, flags); - } spin_lock_irqsave_rcu_node(ssp, flags); if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, s)) WRITE_ONCE(ssp->srcu_gp_seq_needed_exp, s); @@ -633,36 +643,37 @@ static void srcu_funnel_gp_start(struct srcu_struct *= ssp, struct srcu_data *sdp, unsigned long flags; int idx =3D rcu_seq_ctr(s) % ARRAY_SIZE(sdp->mynode->srcu_have_cbs); struct srcu_node *snp; - struct srcu_node *snp_leaf =3D sdp->mynode; + struct srcu_node *snp_leaf =3D smp_load_acquire(&sdp->mynode); unsigned long snp_seq; =20 - /* Each pass through the loop does one level of the srcu_node tree. */ - for (snp =3D snp_leaf; snp !=3D NULL; snp =3D snp->srcu_parent) { - if (rcu_seq_done(&ssp->srcu_gp_seq, s) && snp !=3D snp_leaf) - return; /* GP already done and CBs recorded. */ - spin_lock_irqsave_rcu_node(snp, flags); - if (ULONG_CMP_GE(snp->srcu_have_cbs[idx], s)) { - snp_seq =3D snp->srcu_have_cbs[idx]; - if (snp =3D=3D snp_leaf && snp_seq =3D=3D s) - snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; - spin_unlock_irqrestore_rcu_node(snp, flags); - if (snp =3D=3D snp_leaf && snp_seq !=3D s) { - srcu_schedule_cbs_sdp(sdp, do_norm - ? SRCU_INTERVAL - : 0); + if (snp_leaf) + /* Each pass through the loop does one level of the srcu_node tree. */ + for (snp =3D snp_leaf; snp !=3D NULL; snp =3D snp->srcu_parent) { + if (rcu_seq_done(&ssp->srcu_gp_seq, s) && snp !=3D snp_leaf) + return; /* GP already done and CBs recorded. */ + spin_lock_irqsave_rcu_node(snp, flags); + if (ULONG_CMP_GE(snp->srcu_have_cbs[idx], s)) { + snp_seq =3D snp->srcu_have_cbs[idx]; + if (snp =3D=3D snp_leaf && snp_seq =3D=3D s) + snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; + spin_unlock_irqrestore_rcu_node(snp, flags); + if (snp =3D=3D snp_leaf && snp_seq !=3D s) { + srcu_schedule_cbs_sdp(sdp, do_norm + ? SRCU_INTERVAL + : 0); + return; + } + if (!do_norm) + srcu_funnel_exp_start(ssp, snp, s); return; } - if (!do_norm) - srcu_funnel_exp_start(ssp, snp, s); - return; + snp->srcu_have_cbs[idx] =3D s; + if (snp =3D=3D snp_leaf) + snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; + if (!do_norm && ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, s)) + WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); + spin_unlock_irqrestore_rcu_node(snp, flags); } - snp->srcu_have_cbs[idx] =3D s; - if (snp =3D=3D snp_leaf) - snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; - if (!do_norm && ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, s)) - WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); - spin_unlock_irqrestore_rcu_node(snp, flags); - } =20 /* Top of tree, must ensure the grace period will be started. */ spin_lock_irqsave_rcu_node(ssp, flags); @@ -820,7 +831,10 @@ static unsigned long srcu_gp_start_if_needed(struct sr= cu_struct *ssp, =20 check_init_srcu_struct(ssp); idx =3D srcu_read_lock(ssp); - sdp =3D raw_cpu_ptr(ssp->sda); + if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_CALL) + sdp =3D per_cpu_ptr(ssp->sda, 0); + else + sdp =3D raw_cpu_ptr(ssp->sda); spin_lock_irqsave_rcu_node(sdp, flags); if (rhp) rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp); @@ -840,7 +854,7 @@ static unsigned long srcu_gp_start_if_needed(struct src= u_struct *ssp, if (needgp) srcu_funnel_gp_start(ssp, sdp, s, do_norm); else if (needexp) - srcu_funnel_exp_start(ssp, sdp->mynode, s); + srcu_funnel_exp_start(ssp, smp_load_acquire(&sdp->mynode), s); srcu_read_unlock(ssp, idx); return s; } @@ -1100,6 +1114,28 @@ static void srcu_barrier_cb(struct rcu_head *rhp) complete(&ssp->srcu_barrier_completion); } =20 +/* + * Enqueue an srcu_barrier() callback on the specified srcu_data + * structure's ->cblist. but only if that ->cblist already has at least o= ne + * callback enqueued. Note that if a CPU already has callbacks enqueue, + * it must have already registered the need for a future grace period, + * so all we need do is enqueue a callback that will use the same grace + * period as the last callback already in the queue. + */ +static void srcu_barrier_one_cpu(struct srcu_struct *ssp, struct srcu_data= *sdp) +{ + spin_lock_irq_rcu_node(sdp); + atomic_inc(&ssp->srcu_barrier_cpu_cnt); + sdp->srcu_barrier_head.func =3D srcu_barrier_cb; + debug_rcu_head_queue(&sdp->srcu_barrier_head); + if (!rcu_segcblist_entrain(&sdp->srcu_cblist, + &sdp->srcu_barrier_head)) { + debug_rcu_head_unqueue(&sdp->srcu_barrier_head); + atomic_dec(&ssp->srcu_barrier_cpu_cnt); + } + spin_unlock_irq_rcu_node(sdp); +} + /** * srcu_barrier - Wait until all in-flight call_srcu() callbacks complete. * @ssp: srcu_struct on which to wait for in-flight callbacks. @@ -1107,7 +1143,6 @@ static void srcu_barrier_cb(struct rcu_head *rhp) void srcu_barrier(struct srcu_struct *ssp) { int cpu; - struct srcu_data *sdp; unsigned long s =3D rcu_seq_snap(&ssp->srcu_barrier_seq); =20 check_init_srcu_struct(ssp); @@ -1123,27 +1158,11 @@ void srcu_barrier(struct srcu_struct *ssp) /* Initial count prevents reaching zero until all CBs are posted. */ atomic_set(&ssp->srcu_barrier_cpu_cnt, 1); =20 - /* - * Each pass through this loop enqueues a callback, but only - * on CPUs already having callbacks enqueued. Note that if - * a CPU already has callbacks enqueue, it must have already - * registered the need for a future grace period, so all we - * need do is enqueue a callback that will use the same - * grace period as the last callback already in the queue. - */ - for_each_possible_cpu(cpu) { - sdp =3D per_cpu_ptr(ssp->sda, cpu); - spin_lock_irq_rcu_node(sdp); - atomic_inc(&ssp->srcu_barrier_cpu_cnt); - sdp->srcu_barrier_head.func =3D srcu_barrier_cb; - debug_rcu_head_queue(&sdp->srcu_barrier_head); - if (!rcu_segcblist_entrain(&sdp->srcu_cblist, - &sdp->srcu_barrier_head)) { - debug_rcu_head_unqueue(&sdp->srcu_barrier_head); - atomic_dec(&ssp->srcu_barrier_cpu_cnt); - } - spin_unlock_irq_rcu_node(sdp); - } + if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) + srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, 0)); + else + for_each_possible_cpu(cpu) + srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, cpu)); =20 /* Remove the initial count, at which point reaching zero can happen. */ if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt)) --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FB40C433EF for ; Tue, 19 Apr 2022 00:04:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237345AbiDSAGs (ORCPT ); Mon, 18 Apr 2022 20:06:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236926AbiDSAGI (ORCPT ); Mon, 18 Apr 2022 20:06:08 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AD2323156; Mon, 18 Apr 2022 17:03:27 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E755DB81135; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9399AC385A7; Tue, 19 Apr 2022 00:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326604; bh=47r/UxjiVgvyJfx/ws2RJoYn9FjCCOZxKBBpxPOtox0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nG1nO9aD+TjNi82ye7YidV0G5vkjszXl7L5NOjlsvqVsmZATnPD7YU5ig4A7wUekB rIy0wAZXVJghdS3Qz/AzMU72OvvjW6455PVssBQ/hmXkgDKI390k6LkZLbeiyR7Le8 uLQuUDroZArEo16S1clOAU7OSuUZZgeiIR7t9aZQBoRI/AbJ9AoTaa7Tbt06PK1Vcf JU/K2nhB1G3L+5uIa+OS8htZi0yPPqUg/H22RpL/B0ao8MVUkPkcd/hJl1096mo085 4I4907Y5K5WTDzDTPMeimzEtpl6fOdWlGw1J8oR5v0uCJ71BaD+50PDdKWIpb+TYwc gWr9x6g+pmqBg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 560505C0A23; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 05/21] srcu: Dynamically allocate srcu_node array Date: Mon, 18 Apr 2022 17:03:06 -0700 Message-Id: <20220419000322.3948903-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit shrinks the srcu_struct structure by converting its ->node field from a fixed-size compile-time array to a pointer to a dynamically allocated array. In kernels built with large values of NR_CPUS that boot on systems with smaller numbers of CPUs, this can save significant memory. [ paulmck: Apply kernel test robot feedback. ] Reported-by: A cast of thousands Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 2 +- kernel/rcu/srcutree.c | 65 ++++++++++++++++++++++++++++++---------- 2 files changed, 51 insertions(+), 16 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 8d1da136a93a..8501b6b45941 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -60,7 +60,7 @@ struct srcu_node { * Per-SRCU-domain structure, similar in function to rcu_state. */ struct srcu_struct { - struct srcu_node node[NUM_RCU_NODES]; /* Combining tree. */ + struct srcu_node *node; /* Combining tree. */ struct srcu_node *level[RCU_NUM_LVLS + 1]; /* First node at each level. */ int srcu_size_state; /* Small-to-big transition state. */ diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index e23696edd43b..e98cc218e42b 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -24,6 +24,7 @@ #include #include #include +#include #include =20 #include "rcu.h" @@ -75,12 +76,42 @@ do { \ spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags) \ =20 /* - * Initialize SRCU combining tree. Note that statically allocated + * Initialize SRCU per-CPU data. Note that statically allocated * srcu_struct structures might already have srcu_read_lock() and * srcu_read_unlock() running against them. So if the is_static parameter * is set, don't initialize ->srcu_lock_count[] and ->srcu_unlock_count[]. */ -static void init_srcu_struct_nodes(struct srcu_struct *ssp) +static void init_srcu_struct_data(struct srcu_struct *ssp) +{ + int cpu; + struct srcu_data *sdp; + + /* + * Initialize the per-CPU srcu_data array, which feeds into the + * leaves of the srcu_node tree. + */ + WARN_ON_ONCE(ARRAY_SIZE(sdp->srcu_lock_count) !=3D + ARRAY_SIZE(sdp->srcu_unlock_count)); + for_each_possible_cpu(cpu) { + sdp =3D per_cpu_ptr(ssp->sda, cpu); + spin_lock_init(&ACCESS_PRIVATE(sdp, lock)); + rcu_segcblist_init(&sdp->srcu_cblist); + sdp->srcu_cblist_invoking =3D false; + sdp->srcu_gp_seq_needed =3D ssp->srcu_gp_seq; + sdp->srcu_gp_seq_needed_exp =3D ssp->srcu_gp_seq; + sdp->mynode =3D NULL; + sdp->cpu =3D cpu; + INIT_WORK(&sdp->work, srcu_invoke_callbacks); + timer_setup(&sdp->delay_work, srcu_delay_timer, 0); + sdp->ssp =3D ssp; + } +} + +/* + * Allocated and initialize SRCU combining tree. Returns @true if + * allocation succeeded and @false otherwise. + */ +static bool init_srcu_struct_nodes(struct srcu_struct *ssp) { int cpu; int i; @@ -92,6 +123,9 @@ static void init_srcu_struct_nodes(struct srcu_struct *s= sp) =20 /* Initialize geometry if it has not already been initialized. */ rcu_init_geometry(); + ssp->node =3D kcalloc(rcu_num_nodes, sizeof(*ssp->node), GFP_ATOMIC); + if (!ssp->node) + return false; =20 /* Work out the overall tree geometry. */ ssp->level[0] =3D &ssp->node[0]; @@ -129,30 +163,20 @@ static void init_srcu_struct_nodes(struct srcu_struct= *ssp) * Initialize the per-CPU srcu_data array, which feeds into the * leaves of the srcu_node tree. */ - WARN_ON_ONCE(ARRAY_SIZE(sdp->srcu_lock_count) !=3D - ARRAY_SIZE(sdp->srcu_unlock_count)); level =3D rcu_num_lvls - 1; snp_first =3D ssp->level[level]; for_each_possible_cpu(cpu) { sdp =3D per_cpu_ptr(ssp->sda, cpu); - spin_lock_init(&ACCESS_PRIVATE(sdp, lock)); - rcu_segcblist_init(&sdp->srcu_cblist); - sdp->srcu_cblist_invoking =3D false; - sdp->srcu_gp_seq_needed =3D ssp->srcu_gp_seq; - sdp->srcu_gp_seq_needed_exp =3D ssp->srcu_gp_seq; sdp->mynode =3D &snp_first[cpu / levelspread[level]]; for (snp =3D sdp->mynode; snp !=3D NULL; snp =3D snp->srcu_parent) { if (snp->grplo < 0) snp->grplo =3D cpu; snp->grphi =3D cpu; } - sdp->cpu =3D cpu; - INIT_WORK(&sdp->work, srcu_invoke_callbacks); - timer_setup(&sdp->delay_work, srcu_delay_timer, 0); - sdp->ssp =3D ssp; sdp->grpmask =3D 1 << (cpu - sdp->mynode->grplo); } smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_WAIT_BARRIER); + return true; } =20 /* @@ -163,6 +187,7 @@ static void init_srcu_struct_nodes(struct srcu_struct *= ssp) static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) { ssp->srcu_size_state =3D SRCU_SIZE_SMALL; + ssp->node =3D NULL; mutex_init(&ssp->srcu_cb_mutex); mutex_init(&ssp->srcu_gp_mutex); ssp->srcu_idx =3D 0; @@ -175,8 +200,16 @@ static int init_srcu_struct_fields(struct srcu_struct = *ssp, bool is_static) ssp->sda =3D alloc_percpu(struct srcu_data); if (!ssp->sda) return -ENOMEM; - init_srcu_struct_nodes(ssp); - ssp->srcu_size_state =3D SRCU_SIZE_BIG; + init_srcu_struct_data(ssp); + if (!init_srcu_struct_nodes(ssp)) { + if (!is_static) { + free_percpu(ssp->sda); + ssp->sda =3D NULL; + return -ENOMEM; + } + } else { + ssp->srcu_size_state =3D SRCU_SIZE_BIG; + } ssp->srcu_gp_seq_needed_exp =3D 0; ssp->srcu_last_gp_end =3D ktime_get_mono_fast_ns(); smp_store_release(&ssp->srcu_gp_seq_needed, 0); /* Init done. */ @@ -393,6 +426,8 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) } free_percpu(ssp->sda); ssp->sda =3D NULL; + kfree(ssp->node); + ssp->node =3D NULL; ssp->srcu_size_state =3D SRCU_SIZE_SMALL; } EXPORT_SYMBOL_GPL(cleanup_srcu_struct); --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4584C4332F for ; Tue, 19 Apr 2022 00:03:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237260AbiDSAG0 (ORCPT ); Mon, 18 Apr 2022 20:06:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236891AbiDSAGI (ORCPT ); Mon, 18 Apr 2022 20:06:08 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AB6A23143; Mon, 18 Apr 2022 17:03:27 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 11B4D612C6; Tue, 19 Apr 2022 00:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EDC98C385B1; Tue, 19 Apr 2022 00:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=2XAXZJ/M66oAiuzE8T3AY4xy86cvdJePw2OQPWXK+IQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OSrS2cqngvWALkLX5H8LTSfabjuqNIUZu3pLzaRndMucty2ZdP88wywHFBLsp1BJO NpkPBMPKM1y2aomWWkAoPCK0fG2lGqB9ScwWodg9Lrb7llhahhV1rVkIEKxLZirUEC qYrlkbcnIYzaIXXrmfBYJwp75IdOWdWfVVApxk/L2Y5uQNWMQhfUTEOiLv+r8Yi/gz bwI2weiBxMRvDt7YNdsqR9D2k4WCqoyW/84q83+rhlzMt3H9xZ/J+tEm1mTseoGJ7S rejOqKvKHCIp3RQNyWIVLDyZig95Un/tH5eXPb3I5KgIjUlaJUe52U+weyOK6yGSET aUOk6GMzZYN1Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 57F665C0B86; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 06/21] srcu: Add size-state transitioning code Date: Mon, 18 Apr 2022 17:03:07 -0700 Message-Id: <20220419000322.3948903-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is just dead code at the moment, and will be used once the state-transition code is activated. Because srcu_barrier() must be aware of transition before call_srcu(), the state machine waits for an SRCU grace period before callbacks are queued to the non-CPU-0 queues. This requres that portions of srcu_barrier() be enclosed in an SRCU read-side critical section. Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index e98cc218e42b..00f9aed6f7b9 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -562,6 +562,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) unsigned long mask; struct srcu_data *sdp; struct srcu_node *snp; + int ss_state; =20 /* Prevent more than one additional grace period. */ mutex_lock(&ssp->srcu_cb_mutex); @@ -629,6 +630,15 @@ static void srcu_gp_end(struct srcu_struct *ssp) } else { spin_unlock_irq_rcu_node(ssp); } + + /* Transition to big if needed. */ + ss_state =3D smp_load_acquire(&ssp->srcu_size_state); + if (ss_state !=3D SRCU_SIZE_SMALL && ss_state !=3D SRCU_SIZE_BIG) { + if (ss_state =3D=3D SRCU_SIZE_ALLOC) + init_srcu_struct_nodes(ssp); + else + smp_store_release(&ssp->srcu_size_state, ss_state + 1); + } } =20 /* @@ -1178,6 +1188,7 @@ static void srcu_barrier_one_cpu(struct srcu_struct *= ssp, struct srcu_data *sdp) void srcu_barrier(struct srcu_struct *ssp) { int cpu; + int idx; unsigned long s =3D rcu_seq_snap(&ssp->srcu_barrier_seq); =20 check_init_srcu_struct(ssp); @@ -1193,11 +1204,13 @@ void srcu_barrier(struct srcu_struct *ssp) /* Initial count prevents reaching zero until all CBs are posted. */ atomic_set(&ssp->srcu_barrier_cpu_cnt, 1); =20 + idx =3D srcu_read_lock(ssp); if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, 0)); else for_each_possible_cpu(cpu) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, cpu)); + srcu_read_unlock(ssp, idx); =20 /* Remove the initial count, at which point reaching zero can happen. */ if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt)) --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 966FDC433F5 for ; Tue, 19 Apr 2022 00:03:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237176AbiDSAGV (ORCPT ); Mon, 18 Apr 2022 20:06:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236861AbiDSAGI (ORCPT ); Mon, 18 Apr 2022 20:06:08 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F56022520; Mon, 18 Apr 2022 17:03:26 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E57E2612CC; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ECB0BC385B4; Tue, 19 Apr 2022 00:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=TExMxQ+s8VL+lQFQF6UBa6tCSz4HgmmMt09Ph4H2mgc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WP4zQvdAs6tW8skCp2dzVy51tOq+QW5m977wUJbWpQdp56UVNsxkZSiYcjzzxtqov KXDPM7lL+H2w1aYt8ok8LFu21RAjStpe/KodLqT86Q67GN8Yh1PC9Och3OlWOQEAJ0 6zokfgPMRPrDsUsQsAUst99VWPZ34KyF3diJXgMI/tWD2XbQki5erTLa8D7+pPmaPU DPtAEMRvaJf6ecbRBIpv3uwZdVbnoR3qQsG4x2gG9YgNmisJge4SXDttHX2CYL7niN H/Ub1jj0JgmuXdXunYVMudmKIl4Bv/m+UVHDbgQhER3AbPPeqSOAe3uLr3pztj1CNf c+YJQ5hatZDXQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 59BD25C0DFD; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 07/21] srcu: Make rcutorture dump the SRCU size state Date: Mon, 18 Apr 2022 17:03:08 -0700 Message-Id: <20220419000322.3948903-7-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit adds the numeric and string version of ->srcu_size_state to the Tree-SRCU-specific portion of the rcutorture output. [ paulmck: Apply feedback from kernel test robot and Dan Carpenter. ] [ quic_neeraju: Apply feedback from Jiapeng Chong. ] Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 00f9aed6f7b9..0874e3be6a5a 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1407,15 +1407,33 @@ void srcutorture_get_gp_data(enum rcutorture_type t= est_type, } EXPORT_SYMBOL_GPL(srcutorture_get_gp_data); =20 +static const char * const srcu_size_state_name[] =3D { + "SRCU_SIZE_SMALL", + "SRCU_SIZE_ALLOC", + "SRCU_SIZE_WAIT_BARRIER", + "SRCU_SIZE_WAIT_CALL", + "SRCU_SIZE_WAIT_CBS1", + "SRCU_SIZE_WAIT_CBS2", + "SRCU_SIZE_WAIT_CBS3", + "SRCU_SIZE_WAIT_CBS4", + "SRCU_SIZE_BIG", + "SRCU_SIZE_???", +}; + void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) { int cpu; int idx; unsigned long s0 =3D 0, s1 =3D 0; + int ss_state =3D READ_ONCE(ssp->srcu_size_state); + int ss_state_idx =3D ss_state; =20 idx =3D ssp->srcu_idx & 0x1; - pr_alert("%s%s Tree SRCU g%ld per-CPU(idx=3D%d):", - tt, tf, rcu_seq_current(&ssp->srcu_gp_seq), idx); + if (ss_state < 0 || ss_state >=3D ARRAY_SIZE(srcu_size_state_name)) + ss_state_idx =3D ARRAY_SIZE(srcu_size_state_name) - 1; + pr_alert("%s%s Tree SRCU g%ld state %d (%s) per-CPU(idx=3D%d):", + tt, tf, rcu_seq_current(&ssp->srcu_gp_seq), ss_state, + srcu_size_state_name[ss_state_idx], idx); for_each_possible_cpu(cpu) { unsigned long l0, l1; unsigned long u0, u1; --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB408C433EF for ; Tue, 19 Apr 2022 00:03:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237119AbiDSAGR (ORCPT ); Mon, 18 Apr 2022 20:06:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236816AbiDSAGH (ORCPT ); Mon, 18 Apr 2022 20:06:07 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55B2F220F8; Mon, 18 Apr 2022 17:03:26 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D8DC06123D; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EC447C385B3; Tue, 19 Apr 2022 00:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=ujAyuo2Re7IiLXwCepAV8QiIbPdUJSCn6Diz0hsqCNM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AV0vI7xIYHF0tgRnGAl6JjQxa7Y2KKctEbDxKMBJx83p4cePocEWqio+4xmY4T6P+ MqN69opFiefUgRsSTFCrdMkxRGY+axXe7VVG6LTMbpu0mKqN/fT4wiA9jNO+TvypyF 7iIvn2+9qxdZ31k7PL3dlrUXNy+esLOuYLY897kGFESzxAuOwj9dTLyWsUNKzeglsV 4TUC4YkNaExhdwkIAJPKLhR2d8rrUuRer0HzOHcIdJARAXvS89vmpHOqoCfcBmBqFe Ij/zXN4vr8gC/7Pvps0ZyQfrVLcRfspT8xsb+fZd0lQTuRJsc9WQ3XRcCTNU/7PA5n CxMfi90UHq4nA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 5B7A05C121E; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 08/21] srcu: Compute snp_seq earlier in srcu_funnel_gp_start() Date: Mon, 18 Apr 2022 17:03:09 -0700 Message-Id: <20220419000322.3948903-8-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, srcu_funnel_gp_start() tests snp->srcu_have_cbs[idx] and then separately assigns it to the snp_seq local variable. This commit does the assignment earlier to simplify the code a bit. While in the area, this commit also takes advantage of the 100-character line limit to put the call to srcu_schedule_cbs_sdp() on a single line. Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 0874e3be6a5a..fec608f69962 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -697,15 +697,13 @@ static void srcu_funnel_gp_start(struct srcu_struct *= ssp, struct srcu_data *sdp, if (rcu_seq_done(&ssp->srcu_gp_seq, s) && snp !=3D snp_leaf) return; /* GP already done and CBs recorded. */ spin_lock_irqsave_rcu_node(snp, flags); - if (ULONG_CMP_GE(snp->srcu_have_cbs[idx], s)) { - snp_seq =3D snp->srcu_have_cbs[idx]; + snp_seq =3D snp->srcu_have_cbs[idx]; + if (ULONG_CMP_GE(snp_seq, s)) { if (snp =3D=3D snp_leaf && snp_seq =3D=3D s) snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; spin_unlock_irqrestore_rcu_node(snp, flags); if (snp =3D=3D snp_leaf && snp_seq !=3D s) { - srcu_schedule_cbs_sdp(sdp, do_norm - ? SRCU_INTERVAL - : 0); + srcu_schedule_cbs_sdp(sdp, do_norm ? SRCU_INTERVAL : 0); return; } if (!do_norm) --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B71A8C433EF for ; Tue, 19 Apr 2022 00:04:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237793AbiDSAHA (ORCPT ); Mon, 18 Apr 2022 20:07:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52790 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236930AbiDSAGJ (ORCPT ); Mon, 18 Apr 2022 20:06:09 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 096292316D; Mon, 18 Apr 2022 17:03:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A8E6AB81144; Tue, 19 Apr 2022 00:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F0CC9C385B7; Tue, 19 Apr 2022 00:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=iyJn1yotRTIEBfPDSDKM1gy8KIt7R7cfA59bZDZBO2c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=euaXErJ69GFH6SlFDJuLr/ePnr2+HVSW6yPG5axeuTidfP0r4h446YoAGVF+1aaFK v1TwbP6g2eZEUJ7g3uI8kn6tQ9qDCvifV8SwrwgwblmqimBV8xSvzKADlweP8kn8pE FdjUWroN4Y0ijSlueKAaBvnoekeFZnUn4ZVUywtaDTWKQkoyomaRQVJnkLaO/N7yH3 7kVhatnB2aCpLoG5OTtofT6JrRP8513O8lD7+MwEhIVLL0CUSBfFQGvlRGfCEV8JZi TQy5t5a3g3+ODBEbCezLyF+7854E+6Z5fY/8F0geBH7c12WfgZV+uNLgtj0PxOHPLl uJ23XKnhP/1+g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 5D4CB5C12AB; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 09/21] srcu: Use invalid initial value for srcu_node GP sequence numbers Date: Mon, 18 Apr 2022 17:03:10 -0700 Message-Id: <20220419000322.3948903-9-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, tree SRCU relies on the srcu_node structures being initialized at the same time that the srcu_struct itself is initialized, and thus use the initial grace-period sequence number as the initial value for the srcu_node structure's ->srcu_have_cbs[] and ->srcu_gp_seq_needed_exp fields. Although this has a high probability of also working when the srcu_node array is allocated and initialized at some random later time, it would be better to avoid leaving such things to chance. This commit therefore initializes these fields with 0x2, which is a recognizable invalid value. It then adds the required checks for this invalid value in order to avoid confusion on long-running kernels (especially those on 32-bit systems) that allocate and initialize srcu_node arrays late in life. Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index fec608f69962..155c430c6a73 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -107,6 +107,18 @@ static void init_srcu_struct_data(struct srcu_struct *= ssp) } } =20 +/* Invalid seq state, used during snp node initialization */ +#define SRCU_SNP_INIT_SEQ 0x2 + +/* + * Check whether sequence number corresponding to snp node, + * is invalid. + */ +static inline bool srcu_invl_snp_seq(unsigned long s) +{ + return rcu_seq_state(s) =3D=3D SRCU_SNP_INIT_SEQ; +} + /* * Allocated and initialize SRCU combining tree. Returns @true if * allocation succeeded and @false otherwise. @@ -139,10 +151,10 @@ static bool init_srcu_struct_nodes(struct srcu_struct= *ssp) WARN_ON_ONCE(ARRAY_SIZE(snp->srcu_have_cbs) !=3D ARRAY_SIZE(snp->srcu_data_have_cbs)); for (i =3D 0; i < ARRAY_SIZE(snp->srcu_have_cbs); i++) { - snp->srcu_have_cbs[i] =3D 0; + snp->srcu_have_cbs[i] =3D SRCU_SNP_INIT_SEQ; snp->srcu_data_have_cbs[i] =3D 0; } - snp->srcu_gp_seq_needed_exp =3D 0; + snp->srcu_gp_seq_needed_exp =3D SRCU_SNP_INIT_SEQ; snp->grplo =3D -1; snp->grphi =3D -1; if (snp =3D=3D &ssp->node[0]) { @@ -386,8 +398,7 @@ static bool srcu_readers_active(struct srcu_struct *ssp) */ static unsigned long srcu_get_delay(struct srcu_struct *ssp) { - if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_gp_seq), - READ_ONCE(ssp->srcu_gp_seq_needed_exp))) + if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_gp_seq), READ_ONCE(ssp->srcu_gp_seq_= needed_exp))) return 0; return SRCU_INTERVAL; } @@ -561,6 +572,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) int idx; unsigned long mask; struct srcu_data *sdp; + unsigned long sgsne; struct srcu_node *snp; int ss_state; =20 @@ -594,7 +606,8 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbs =3D snp->srcu_have_cbs[idx] =3D=3D gpseq; snp->srcu_have_cbs[idx] =3D gpseq; rcu_seq_set_state(&snp->srcu_have_cbs[idx], 1); - if (ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, gpseq)) + sgsne =3D snp->srcu_gp_seq_needed_exp; + if (srcu_invl_snp_seq(sgsne) || ULONG_CMP_LT(sgsne, gpseq)) WRITE_ONCE(snp->srcu_gp_seq_needed_exp, gpseq); mask =3D snp->srcu_data_have_cbs[idx]; snp->srcu_data_have_cbs[idx] =3D 0; @@ -652,14 +665,17 @@ static void srcu_funnel_exp_start(struct srcu_struct = *ssp, struct srcu_node *snp unsigned long s) { unsigned long flags; + unsigned long sgsne; =20 if (snp) for (; snp !=3D NULL; snp =3D snp->srcu_parent) { + sgsne =3D READ_ONCE(snp->srcu_gp_seq_needed_exp); if (rcu_seq_done(&ssp->srcu_gp_seq, s) || - ULONG_CMP_GE(READ_ONCE(snp->srcu_gp_seq_needed_exp), s)) + (!srcu_invl_snp_seq(sgsne) && ULONG_CMP_GE(sgsne, s))) return; spin_lock_irqsave_rcu_node(snp, flags); - if (ULONG_CMP_GE(snp->srcu_gp_seq_needed_exp, s)) { + sgsne =3D snp->srcu_gp_seq_needed_exp; + if (!srcu_invl_snp_seq(sgsne) && ULONG_CMP_GE(sgsne, s)) { spin_unlock_irqrestore_rcu_node(snp, flags); return; } @@ -687,6 +703,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ss= p, struct srcu_data *sdp, { unsigned long flags; int idx =3D rcu_seq_ctr(s) % ARRAY_SIZE(sdp->mynode->srcu_have_cbs); + unsigned long sgsne; struct srcu_node *snp; struct srcu_node *snp_leaf =3D smp_load_acquire(&sdp->mynode); unsigned long snp_seq; @@ -698,7 +715,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ss= p, struct srcu_data *sdp, return; /* GP already done and CBs recorded. */ spin_lock_irqsave_rcu_node(snp, flags); snp_seq =3D snp->srcu_have_cbs[idx]; - if (ULONG_CMP_GE(snp_seq, s)) { + if (!srcu_invl_snp_seq(snp_seq) && ULONG_CMP_GE(snp_seq, s)) { if (snp =3D=3D snp_leaf && snp_seq =3D=3D s) snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; spin_unlock_irqrestore_rcu_node(snp, flags); @@ -713,7 +730,8 @@ static void srcu_funnel_gp_start(struct srcu_struct *ss= p, struct srcu_data *sdp, snp->srcu_have_cbs[idx] =3D s; if (snp =3D=3D snp_leaf) snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; - if (!do_norm && ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, s)) + sgsne =3D snp->srcu_gp_seq_needed_exp; + if (!do_norm && (srcu_invl_snp_seq(sgsne) || ULONG_CMP_LT(sgsne, s))) WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); spin_unlock_irqrestore_rcu_node(snp, flags); } --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C4A1C4332F for ; Tue, 19 Apr 2022 00:05:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239029AbiDSAHk (ORCPT ); Mon, 18 Apr 2022 20:07:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236828AbiDSAGH (ORCPT ); Mon, 18 Apr 2022 20:06:07 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F33D22505; Mon, 18 Apr 2022 17:03:26 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DFFB2612CA; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F0B38C385B5; Tue, 19 Apr 2022 00:03:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=IXnjpPJm5PvwKH+ze2uBeBae2XPGQCkrZNv5KgoR3iE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sE7mPbknUKd2kgvgS4LIp9J7AQpXwU12+h9xsTt3Ii/k34v1PqRuI/Xu4yfLMgekZ 5xYTJ/aSuFSJeuvwZ0gpWPApd48IfTprpi6vT4evKuNeUNwnQ/yD0JlttT2fxwDeqr N0OBsXoW3pFOU9CA6uSMbhLB0CSFcWsESFijAVMhpdfk3VOpFVaww/dlTsRvpjpgQv ih5KcMI//fGqfVzPnNfiThQOujBaBT21662y+PFnb7YTisfhOTlkTVFALBjX9dvpcT KSUsii5rLCDEqUgGAtI5jdhynWmsv7IG/2ldVWYA8gAlAatvwNG9kSdrylC+Vgpd8j ZmdNkFlfMwnBQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 5F2335C1308; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Neeraj Upadhyay , "Paul E . McKenney" Subject: [PATCH rcu 10/21] srcu: Ensure snp nodes tree is fully initialized before traversal Date: Mon, 18 Apr 2022 17:03:11 -0700 Message-Id: <20220419000322.3948903-10-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Neeraj Upadhyay For configurations where snp node tree is not initialized at init time (added in subsequent commits), srcu_funnel_gp_start() and srcu_funnel_exp_start() can potential traverse and observe the snp nodes' transient (uninitialized) states. This can potentially happen, when init_srcu_struct_nodes() initialization of sdp->mynode races with srcu_funnel_gp_start() and srcu_funnel_exp_start() Consider the case below where srcu_funnel_gp_start() observes sdp->mynode to be not NULL and uses an uninitialized sdp->grpmask P1 P2 init_srcu_struct_nodes() void srcu_funnel_gp_start(...) { for_each_possible_cpu(cpu) { ... sdp->mynode =3D &snp_first[...]; for (snp =3D sdp->mynode;...) struct srcu_node *snp_leaf =3D smp_load_acquire(&sdp->mynode) ... if (snp_leaf) { for (snp =3D snp_leaf; ...) ... if (snp =3D=3D snp_leaf) snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; sdp->grpmask =3D 1 << (cpu - sdp->mynode->grplo); } } Similarly, init_srcu_struct_nodes() and srcu_funnel_exp_start() can race, where srcu_funnel_exp_start() could observe state of snp lock before spin_lock_init(). P1 P2 init_srcu_struct_nodes() void srcu_funnel_exp_start(...) { srcu_for_each_node_breadth_first(ssp, snp) { for (; ...) { spin_lock_...(snp, ) spin_lock_init(&ACCESS_PRIVATE(snp, lock)); ... } for_each_possible_cpu(cpu) { ... sdp->mynode =3D &snp_first[...]; To avoid these issues, ensure that snp node tree initialization is complete i.e. after SRCU_SIZE_WAIT_BARRIER srcu_size_state is reached, before traversing the tree. Given that srcu_funnel_gp_start() and srcu_funnel_exp_start() are called within SRCU read side critical sections, this check is safe, in the sense that all callbacks are enqueued on CPU0 srcu_cblist until SRCU_SIZE_WAIT_CALL is entered, and these read side critical sections (containing srcu_funnel_gp_start() and srcu_funnel_exp_start()) need to complete, before SRCU_SIZE_WAIT_CALL is reached. Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 155c430c6a73..2e7ed67646db 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -705,9 +705,15 @@ static void srcu_funnel_gp_start(struct srcu_struct *s= sp, struct srcu_data *sdp, int idx =3D rcu_seq_ctr(s) % ARRAY_SIZE(sdp->mynode->srcu_have_cbs); unsigned long sgsne; struct srcu_node *snp; - struct srcu_node *snp_leaf =3D smp_load_acquire(&sdp->mynode); + struct srcu_node *snp_leaf; unsigned long snp_seq; =20 + /* Ensure that snp node tree is fully initialized before traversing it */ + if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) + snp_leaf =3D NULL; + else + snp_leaf =3D sdp->mynode; + if (snp_leaf) /* Each pass through the loop does one level of the srcu_node tree. */ for (snp =3D snp_leaf; snp !=3D NULL; snp =3D snp->srcu_parent) { @@ -889,10 +895,13 @@ static unsigned long srcu_gp_start_if_needed(struct s= rcu_struct *ssp, bool needgp =3D false; unsigned long s; struct srcu_data *sdp; + struct srcu_node *sdp_mynode; + int ss_state; =20 check_init_srcu_struct(ssp); idx =3D srcu_read_lock(ssp); - if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_CALL) + ss_state =3D smp_load_acquire(&ssp->srcu_size_state); + if (ss_state < SRCU_SIZE_WAIT_CALL) sdp =3D per_cpu_ptr(ssp->sda, 0); else sdp =3D raw_cpu_ptr(ssp->sda); @@ -912,10 +921,17 @@ static unsigned long srcu_gp_start_if_needed(struct s= rcu_struct *ssp, needexp =3D true; } spin_unlock_irqrestore_rcu_node(sdp, flags); + + /* Ensure that snp node tree is fully initialized before traversing it */ + if (ss_state < SRCU_SIZE_WAIT_BARRIER) + sdp_mynode =3D NULL; + else + sdp_mynode =3D sdp->mynode; + if (needgp) srcu_funnel_gp_start(ssp, sdp, s, do_norm); else if (needexp) - srcu_funnel_exp_start(ssp, smp_load_acquire(&sdp->mynode), s); + srcu_funnel_exp_start(ssp, sdp_mynode, s); srcu_read_unlock(ssp, idx); return s; } --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C99FCC433EF for ; Tue, 19 Apr 2022 00:04:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237351AbiDSAG4 (ORCPT ); Mon, 18 Apr 2022 20:06:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236941AbiDSAGJ (ORCPT ); Mon, 18 Apr 2022 20:06:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B24B24587; Mon, 18 Apr 2022 17:03:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7748F612CD; Tue, 19 Apr 2022 00:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 077CFC385B8; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=Psh5an79Q919sHkZgWjtHptebmVck+c5jxmqmY+1jEA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OoVe53oh1QG0zEnTs+2z83AKFY/CKJUBnttE74hz5WBbB4NMI9xfYGEXJaTOvzgxN 04lt9i+xg/wCS9BLvPhnR47CSmjHXCSEJNgJw359HbCeWROH0Iz1oCjCzXb09jiTsv 8P7EQPPsc0Vk4gHgmITtWkC4iW43eGycKZkbO6ldcuLFb3fhFjNAos3UZAutpTo3DD Y6dVq4C1G9yTNiX2mcPpRYANY1I13IrAG9C6YnRIa8brfVi1gvT/deIZK9YqB1DvN0 DyMzPmMDl8fLUuDBXzkgLu8eWxZKL+kEYeyRVpY9AvZdYJlIsU7IdiOyKowQcAomRU k33O7K+hja5qQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 60E685C1355; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 11/21] srcu: Add boot-time control over srcu_node array allocation Date: Mon, 18 Apr 2022 17:03:12 -0700 Message-Id: <20220419000322.3948903-11-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit adds an srcu_tree.convert_to_big kernel parameter that either refuses to convert at all (0), converts immediately at init_srcu_struct() time (1), or lets rcutorture convert it (2). An addition contention-based dynamic conversion choice will be added, along with documentation. [ paulmck: Apply callback-scanning feedback from Neeraj Upadhyay. ] Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- .../admin-guide/kernel-parameters.txt | 13 +++++ kernel/rcu/srcutree.c | 48 ++++++++++++------- 2 files changed, 45 insertions(+), 16 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 3f1cc5e317ed..1f1fcac7777d 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5608,6 +5608,19 @@ off: Disable mitigation and remove performance impact to RDRAND and RDSEED =20 + srcutree.convert_to_big [KNL] + Specifies under what conditions an SRCU tree + srcu_struct structure will be converted to big + form, that is, with an rcu_node tree: + + 0: Never. + 1: At init_srcu_struct() time. + 2: When rcutorture decides to. + + Either way, the srcu_node tree will be sized based + on the actual runtime number of CPUs (nr_cpu_ids) + instead of the compile-time CONFIG_NR_CPUS. + srcutree.counter_wrap_check [KNL] Specifies how frequently to check for grace-period sequence counter wrap for the diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 2e7ed67646db..9dd048989027 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -39,6 +39,15 @@ module_param(exp_holdoff, ulong, 0444); static ulong counter_wrap_check =3D (ULONG_MAX >> 2); module_param(counter_wrap_check, ulong, 0444); =20 +/* + * Control conversion to SRCU_SIZE_BIG: + * 0: Don't convert at all (default). + * 1: Convert at init_srcu_struct() time. + * 2: Convert when rcutorture invokes srcu_torture_stats_print(). + */ +static int convert_to_big; +module_param(convert_to_big, int, 0444); + /* Early-boot callback-management, so early that no lock is required! */ static LIST_HEAD(srcu_boot_list); static bool __read_mostly srcu_init_done; @@ -123,7 +132,7 @@ static inline bool srcu_invl_snp_seq(unsigned long s) * Allocated and initialize SRCU combining tree. Returns @true if * allocation succeeded and @false otherwise. */ -static bool init_srcu_struct_nodes(struct srcu_struct *ssp) +static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flag= s) { int cpu; int i; @@ -135,7 +144,7 @@ static bool init_srcu_struct_nodes(struct srcu_struct *= ssp) =20 /* Initialize geometry if it has not already been initialized. */ rcu_init_geometry(); - ssp->node =3D kcalloc(rcu_num_nodes, sizeof(*ssp->node), GFP_ATOMIC); + ssp->node =3D kcalloc(rcu_num_nodes, sizeof(*ssp->node), gfp_flags); if (!ssp->node) return false; =20 @@ -213,17 +222,19 @@ static int init_srcu_struct_fields(struct srcu_struct= *ssp, bool is_static) if (!ssp->sda) return -ENOMEM; init_srcu_struct_data(ssp); - if (!init_srcu_struct_nodes(ssp)) { - if (!is_static) { - free_percpu(ssp->sda); - ssp->sda =3D NULL; - return -ENOMEM; - } - } else { - ssp->srcu_size_state =3D SRCU_SIZE_BIG; - } ssp->srcu_gp_seq_needed_exp =3D 0; ssp->srcu_last_gp_end =3D ktime_get_mono_fast_ns(); + if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 1) { + if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { + if (!is_static) { + free_percpu(ssp->sda); + ssp->sda =3D NULL; + return -ENOMEM; + } + } else { + WRITE_ONCE(ssp->srcu_size_state, SRCU_SIZE_BIG); + } + } smp_store_release(&ssp->srcu_gp_seq_needed, 0); /* Init done. */ return 0; } @@ -594,7 +605,8 @@ static void srcu_gp_end(struct srcu_struct *ssp) /* A new grace period can start at this point. But only one. */ =20 /* Initiate callback invocation as needed. */ - if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) { + ss_state =3D smp_load_acquire(&ssp->srcu_size_state); + if (ss_state < SRCU_SIZE_WAIT_BARRIER) { srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, 0), cbdelay); } else { idx =3D rcu_seq_ctr(gpseq) % ARRAY_SIZE(snp->srcu_have_cbs); @@ -603,13 +615,16 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbs =3D false; last_lvl =3D snp >=3D ssp->level[rcu_num_lvls - 1]; if (last_lvl) - cbs =3D snp->srcu_have_cbs[idx] =3D=3D gpseq; + cbs =3D ss_state < SRCU_SIZE_BIG || snp->srcu_have_cbs[idx] =3D=3D gps= eq; snp->srcu_have_cbs[idx] =3D gpseq; rcu_seq_set_state(&snp->srcu_have_cbs[idx], 1); sgsne =3D snp->srcu_gp_seq_needed_exp; if (srcu_invl_snp_seq(sgsne) || ULONG_CMP_LT(sgsne, gpseq)) WRITE_ONCE(snp->srcu_gp_seq_needed_exp, gpseq); - mask =3D snp->srcu_data_have_cbs[idx]; + if (ss_state < SRCU_SIZE_BIG) + mask =3D ~0; + else + mask =3D snp->srcu_data_have_cbs[idx]; snp->srcu_data_have_cbs[idx] =3D 0; spin_unlock_irq_rcu_node(snp); if (cbs) @@ -645,10 +660,9 @@ static void srcu_gp_end(struct srcu_struct *ssp) } =20 /* Transition to big if needed. */ - ss_state =3D smp_load_acquire(&ssp->srcu_size_state); if (ss_state !=3D SRCU_SIZE_SMALL && ss_state !=3D SRCU_SIZE_BIG) { if (ss_state =3D=3D SRCU_SIZE_ALLOC) - init_srcu_struct_nodes(ssp); + init_srcu_struct_nodes(ssp, GFP_KERNEL); else smp_store_release(&ssp->srcu_size_state, ss_state + 1); } @@ -1494,6 +1508,8 @@ void srcu_torture_stats_print(struct srcu_struct *ssp= , char *tt, char *tf) s1 +=3D c1; } pr_cont(" T(%ld,%ld)\n", s0, s1); + if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 2) + WRITE_ONCE(ssp->srcu_size_state, SRCU_SIZE_ALLOC); } EXPORT_SYMBOL_GPL(srcu_torture_stats_print); =20 --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E965DC433F5 for ; Tue, 19 Apr 2022 00:05:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238321AbiDSAHa (ORCPT ); Mon, 18 Apr 2022 20:07:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236892AbiDSAGI (ORCPT ); Mon, 18 Apr 2022 20:06:08 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5590323158; Mon, 18 Apr 2022 17:03:27 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 18B93612CF; Tue, 19 Apr 2022 00:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0C379C385B9; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=g34Kr64bP04pVIhScsqxQ1L3lPX5LIieDgKQrcvhsG0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tQrwVQdTyviGb8h+us4HSkrVKTnEVhx8x/MCZStonKhk23aEBRyAiz1EhRvAw/5Gt 6PtwfgJCti971cuCVhl7BP8OLOesQiI50Xw6SvlqKftQFJdNHdskdMc1+P8du12JFy gx6+mWLSmfxEgeGf//T9+8YdHLTRSna3EIvvBcoFB2OCuhsu6ysx+c8ZgNLbaRtf7B /Ha+BwFkudpw1LOeXX26MrDYFAflxe7dMLZwSN06A2+pC6M1LMK1XMLBAysED2qviu qhkwKtEcuRS5DyZNzcYp3vWW2+DyDqALMRKMKHcXqdJrTjBsNAiIEKVWb4BbhNEDL6 /UMhGE606Xk0Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 629A65C1369; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Alexander Aring , "Paul E . McKenney" Subject: [PATCH rcu 12/21] srcu: Use export for srcu_struct defined by DEFINE_STATIC_SRCU() Date: Mon, 18 Apr 2022 17:03:13 -0700 Message-Id: <20220419000322.3948903-12-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alexander Aring If an srcu_struct structure defined by tree SRCU's DEFINE_STATIC_SRCU() is used by a module, sparse will give the following diagnostic: sparse: symbol '__srcu_struct_nodes_srcu' was not declared. Should it be st= atic? The problem is that a within-module DEFINE_STATIC_SRCU() must define a non-static srcu_struct because it is exported by referencing it in a special '__section("___srcu_struct_ptrs")'. This reference is needed so that module load and unloading can invoke init_srcu_struct() and cleanup_srcu_struct(), respectively. Unfortunately, sparse is unaware of '__section("___srcu_struct_ptrs")', resulting in the above false-positive diagnostic. To avoid this false positive, this commit therefore creates a prototype of the srcu_struct with an "extern" keyword. Signed-off-by: Alexander Aring Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 8501b6b45941..44e998643f48 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -131,6 +131,7 @@ struct srcu_struct { #ifdef MODULE # define __DEFINE_SRCU(name, is_static) \ is_static struct srcu_struct name; \ + extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ __section("___srcu_struct_ptrs") =3D &name #else --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19D93C433EF for ; Tue, 19 Apr 2022 00:03:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237398AbiDSAGg (ORCPT ); Mon, 18 Apr 2022 20:06:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236855AbiDSAGJ (ORCPT ); Mon, 18 Apr 2022 20:06:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B0572317E; Mon, 18 Apr 2022 17:03:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 20FB1612D0; Tue, 19 Apr 2022 00:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 13CF9C385BA; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=yD8/2f4i74rv4I75fc7QN5JRO3/GVs7/E4TdpFpPISc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=R8q1Cv0Vkp9q/MSz5wqfJReVoVMUwgvmUz05/uY8RS055LvWx/W+JI1bSz3Is/ZF3 IUxNeKNZ+Mhn0pvn4N3ww4zCDyaRswoa5ikzO8JKIMYKKnssJEVK62xfx9Fz8zvuoz 9Tdzzv4U7sIanHJZPGP83cSVCq9WSvhhTR0b+s9M9FTWDjwG1drc51ncTX8wh794Ho 3HKIUl5pjft6Iltsrdko7douITCeRHpiscJ3Clug/FV35HF0ga/bnFWXx0JbpqlAdq lz4PGDv0xOOXhLNxtxrb6bsJlAHxUGp+C0mHhAP7yyvFYLtnhzYZUnlvn2PWsjxq6o DHcOvA7ZRgF4Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 644845C23F4; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 13/21] srcu: Avoid NULL dereference in srcu_torture_stats_print() Date: Mon, 18 Apr 2022 17:03:14 -0700 Message-Id: <20220419000322.3948903-13-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" You really shouldn't invoke srcu_torture_stats_print() after invoking cleanup_srcu_struct(), but there is really no reason to get a compiler-obfuscated per-CPU-variable NULL pointer dereference as the diagnostic. This commit therefore checks for NULL ->sda and makes a more polite console-message complaint in that case. Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 62 ++++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 28 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 9dd048989027..b7138dbe1a2d 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1477,37 +1477,43 @@ void srcu_torture_stats_print(struct srcu_struct *s= sp, char *tt, char *tf) idx =3D ssp->srcu_idx & 0x1; if (ss_state < 0 || ss_state >=3D ARRAY_SIZE(srcu_size_state_name)) ss_state_idx =3D ARRAY_SIZE(srcu_size_state_name) - 1; - pr_alert("%s%s Tree SRCU g%ld state %d (%s) per-CPU(idx=3D%d):", + pr_alert("%s%s Tree SRCU g%ld state %d (%s)", tt, tf, rcu_seq_current(&ssp->srcu_gp_seq), ss_state, - srcu_size_state_name[ss_state_idx], idx); - for_each_possible_cpu(cpu) { - unsigned long l0, l1; - unsigned long u0, u1; - long c0, c1; - struct srcu_data *sdp; - - sdp =3D per_cpu_ptr(ssp->sda, cpu); - u0 =3D data_race(sdp->srcu_unlock_count[!idx]); - u1 =3D data_race(sdp->srcu_unlock_count[idx]); + srcu_size_state_name[ss_state_idx]); + if (!ssp->sda) { + // Called after cleanup_srcu_struct(), perhaps. + pr_cont(" No per-CPU srcu_data structures (->sda =3D=3D NULL).\n"); + } else { + pr_cont(" per-CPU(idx=3D%d):", idx); + for_each_possible_cpu(cpu) { + unsigned long l0, l1; + unsigned long u0, u1; + long c0, c1; + struct srcu_data *sdp; =20 - /* - * Make sure that a lock is always counted if the corresponding - * unlock is counted. - */ - smp_rmb(); - - l0 =3D data_race(sdp->srcu_lock_count[!idx]); - l1 =3D data_race(sdp->srcu_lock_count[idx]); - - c0 =3D l0 - u0; - c1 =3D l1 - u1; - pr_cont(" %d(%ld,%ld %c)", - cpu, c0, c1, - "C."[rcu_segcblist_empty(&sdp->srcu_cblist)]); - s0 +=3D c0; - s1 +=3D c1; + sdp =3D per_cpu_ptr(ssp->sda, cpu); + u0 =3D data_race(sdp->srcu_unlock_count[!idx]); + u1 =3D data_race(sdp->srcu_unlock_count[idx]); + + /* + * Make sure that a lock is always counted if the corresponding + * unlock is counted. + */ + smp_rmb(); + + l0 =3D data_race(sdp->srcu_lock_count[!idx]); + l1 =3D data_race(sdp->srcu_lock_count[idx]); + + c0 =3D l0 - u0; + c1 =3D l1 - u1; + pr_cont(" %d(%ld,%ld %c)", + cpu, c0, c1, + "C."[rcu_segcblist_empty(&sdp->srcu_cblist)]); + s0 +=3D c0; + s1 +=3D c1; + } + pr_cont(" T(%ld,%ld)\n", s0, s1); } - pr_cont(" T(%ld,%ld)\n", s0, s1); if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 2) WRITE_ONCE(ssp->srcu_size_state, SRCU_SIZE_ALLOC); } --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F590C433EF for ; Tue, 19 Apr 2022 00:04:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237452AbiDSAHH (ORCPT ); Mon, 18 Apr 2022 20:07:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236959AbiDSAGJ (ORCPT ); Mon, 18 Apr 2022 20:06:09 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 997C729811; Mon, 18 Apr 2022 17:03:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 2124AB81147; Tue, 19 Apr 2022 00:03:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 10929C385BB; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=lyjONtqSbnzzy2uBjKzwL8KfGkjOoSlgn7pyA+mj6/U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Mkgdn4M5oB9OoSqAu55ZEyQz393wg6gC6DZAwCRubHlDFFbt87VBtGr8Vko7bWh4u Iui26/tjpkfUxiXQ+x5dt56JOjUN5+DoBVbxyH7TTp2+rwrmg30EiIDE4d/wqfnkyi ZPnYEwaDXU2QvosmHPtStrD4cp15M/624W+rdifuxnDuV3WgSMy283vrOS4xtiXL06 EZy+myupAat6x0JQHbB06wAanOWOUqVj8dnu5xD/za3V0xL/RtLp7vQMWoxIbpwvYb rkuA6hgP+VaGPTqsttKtSrFoXfE/vTxhw93j2+ueym18o18bM0lvpsXY/m1Ub/fZgM his/QY6is5SBw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 660AC5C2B13; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 14/21] srcu: Prevent cleanup_srcu_struct() from freeing non-dynamic ->sda Date: Mon, 18 Apr 2022 17:03:15 -0700 Message-Id: <20220419000322.3948903-14-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When an srcu_struct structure is created (but not in a kernel module) by DEFINE_SRCU() and friends, the per-CPU srcu_data structure is statically allocated. In all other cases, that structure is obtained from alloc_percpu(), in which case cleanup_srcu_struct() must invoke free_percpu() on the resulting ->sda pointer in the srcu_struct pointer. Which it does. Except that it also invokes free_percpu() on the ->sda pointer referencing the statically allocated per-CPU srcu_data structures. Which free_percpu() is surprisingly OK with. This commit nevertheless stops cleanup_srcu_struct() from freeing statically allocated per-CPU srcu_data structures. Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 1 + kernel/rcu/srcutree.c | 9 ++++++--- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 44e998643f48..44bd204498a1 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -73,6 +73,7 @@ struct srcu_struct { unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */ unsigned long srcu_last_gp_end; /* Last GP end timestamp (ns) */ struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ + bool sda_is_static; /* May ->sda be passed to free_percpu()? */ unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ struct mutex srcu_barrier_mutex; /* Serialize barrier ops. */ struct completion srcu_barrier_completion; diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index b7138dbe1a2d..7209fd95dde9 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -217,6 +217,7 @@ static int init_srcu_struct_fields(struct srcu_struct *= ssp, bool is_static) mutex_init(&ssp->srcu_barrier_mutex); atomic_set(&ssp->srcu_barrier_cpu_cnt, 0); INIT_DELAYED_WORK(&ssp->work, process_srcu); + ssp->sda_is_static =3D is_static; if (!is_static) ssp->sda =3D alloc_percpu(struct srcu_data); if (!ssp->sda) @@ -226,7 +227,7 @@ static int init_srcu_struct_fields(struct srcu_struct *= ssp, bool is_static) ssp->srcu_last_gp_end =3D ktime_get_mono_fast_ns(); if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 1) { if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { - if (!is_static) { + if (!ssp->sda_is_static) { free_percpu(ssp->sda); ssp->sda =3D NULL; return -ENOMEM; @@ -446,8 +447,10 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) rcu_seq_current(&ssp->srcu_gp_seq), ssp->srcu_gp_seq_needed); return; /* Caller forgot to stop doing call_srcu()? */ } - free_percpu(ssp->sda); - ssp->sda =3D NULL; + if (!ssp->sda_is_static) { + free_percpu(ssp->sda); + ssp->sda =3D NULL; + } kfree(ssp->node); ssp->node =3D NULL; ssp->srcu_size_state =3D SRCU_SIZE_SMALL; --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D2EAC433F5 for ; Tue, 19 Apr 2022 00:04:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236195AbiDSAGx (ORCPT ); Mon, 18 Apr 2022 20:06:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236928AbiDSAGJ (ORCPT ); Mon, 18 Apr 2022 20:06:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C220252B7; Mon, 18 Apr 2022 17:03:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 83B53612DB; Tue, 19 Apr 2022 00:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1EA40C385BF; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=dq5EhGRV2Ky6K2MIwwoj/ydAY0Nn8LeiyWw4AjAjknk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kdPxWn0X8WovhPqBfABGi2JA5idnO5voDpqxlPRx/LYFSXMhr8uJ0igVjL5Olfo3n 9YXvZyZp32vewI01Dyz53JtD3GqYT9kHWXSlxh25qmeWSrnPKO8ISqn9wPe7JkziSP RbwRRJGedYWe+k/FNluGQPItvRPOUXnrDFtYhh1K9Nj3rt02h2aSqvmqBlXZ78xsD5 CE0iQScP2CqwzR4p6o2wzCkWv/QRY8JZNbeJInDLzSaROdWzf0rgSrecuPGSHmehyE 27CtMiAPHe+IUYa7Ssp5mcfCbyaizditBRh/cXDAYoP+odjZgH6Q+pcceD5Tv7EjXm kB5yiNQGl1t+w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 67C505C2DEA; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 15/21] srcu: Explain srcu_funnel_gp_start() call to list_add() is safe Date: Mon, 18 Apr 2022 17:03:16 -0700 Message-Id: <20220419000322.3948903-15-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit adds a comment explaining why an unprotected call to list_add() from srcu_funnel_gp_start() can be safe. TL;DR: It is only called during very early boot when we don't have no steeking concurrency! Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 7209fd95dde9..64993a172cff 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -776,6 +776,12 @@ static void srcu_funnel_gp_start(struct srcu_struct *s= sp, struct srcu_data *sdp, rcu_seq_state(ssp->srcu_gp_seq) =3D=3D SRCU_STATE_IDLE) { WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); srcu_gp_start(ssp); + + // And how can that list_add() in the "else" clause + // possibly be safe for concurrent execution? Well, + // it isn't. And it does not have to be. After all, it + // can only be executed during early boot when there is only + // the one boot CPU running with interrupts still disabled. if (likely(srcu_init_done)) queue_delayed_work(rcu_gp_wq, &ssp->work, srcu_get_delay(ssp)); --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25E65C433F5 for ; Tue, 19 Apr 2022 00:04:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237447AbiDSAGl (ORCPT ); Mon, 18 Apr 2022 20:06:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236929AbiDSAGJ (ORCPT ); Mon, 18 Apr 2022 20:06:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C33B25EB9; Mon, 18 Apr 2022 17:03:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3DDD9612D2; Tue, 19 Apr 2022 00:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 49B1BC385C6; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=nNJzJ7ht+CbKYymIkOO+SwWfnhHVpzPdSX+sR/dQxpU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eGOX6Wc1yC7t7X1I1bHokg/oMTHfD1+ZF8aL7evt0quDKJ3mMegJ+f5+92g4fin12 ziRIE8uB49OCyf997rjFW1pwQNTu7BNYv0XqaeltJF0SGSeEjy0wAE0CwpQ7D5Z6PC AIZ1A4ITiS0VA/uzMN2C9xKneCI308C99ci1ykxGU2owmOm5hzP5XKHdDBEwskEhmN t+v2E2mX4Mn/CrI6RKkbpNVppMomRd0vc5w6KQ2tTpgKSQNKgdvwuDC2FvAjQ4CIW+ /g1Swb/v92kQ3FUpYjwiqs3cTIASCcgPgO/ZLMzCLUFnKNHQLPcZSh0EbY8o2SMtIQ 0D1YvYaTgyj1A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 697FC5C2DF5; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 16/21] srcu: Create concurrency-safe helper for initiating size transition Date: Mon, 18 Apr 2022 17:03:17 -0700 Message-Id: <20220419000322.3948903-16-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Once there are contention-initiated size transitions, it will be possible for rcutorture to initiate a transition at the same time as a contention-initiated transition. This commit therefore creates a concurrency-safe helper function named srcu_transition_to_big() to safely initiate size transitions. Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 64993a172cff..c9460374d437 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -272,6 +272,25 @@ EXPORT_SYMBOL_GPL(init_srcu_struct); =20 #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 +/* + * Initiate an idempotent transition to SRCU_SIZE_BIG. + */ +static void srcu_transition_to_big(struct srcu_struct *ssp) +{ + unsigned long flags; + + /* Double-checked locking on ->srcu_size-state. */ + if (smp_load_acquire(&ssp->srcu_size_state) !=3D SRCU_SIZE_SMALL) + return; + spin_lock_irqsave_rcu_node(ssp, flags); + if (smp_load_acquire(&ssp->srcu_size_state) !=3D SRCU_SIZE_SMALL) { + spin_unlock_irqrestore_rcu_node(ssp, flags); + return; + } + smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_ALLOC); + spin_unlock_irqrestore_rcu_node(ssp, flags); +} + /* * First-use initialization of statically allocated srcu_struct * structure. Wiring up the combining tree is more than can be @@ -1523,8 +1542,8 @@ void srcu_torture_stats_print(struct srcu_struct *ssp= , char *tt, char *tf) } pr_cont(" T(%ld,%ld)\n", s0, s1); } - if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 2) - WRITE_ONCE(ssp->srcu_size_state, SRCU_SIZE_ALLOC); + if (convert_to_big =3D=3D 2) + srcu_transition_to_big(ssp); } EXPORT_SYMBOL_GPL(srcu_torture_stats_print); =20 --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB451C433F5 for ; Tue, 19 Apr 2022 00:05:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238768AbiDSAHf (ORCPT ); Mon, 18 Apr 2022 20:07:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237002AbiDSAGL (ORCPT ); Mon, 18 Apr 2022 20:06:11 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2A30222A3; Mon, 18 Apr 2022 17:03:29 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 646DCB8114E; Tue, 19 Apr 2022 00:03:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 49DDCC385C9; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=IS66TlnWS5Eiy9CLr4zipmghBlyHKD0nvzdhCVNl4zg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XDUiUrLxtNPSKbwDSrdkq1Gpg6/5lB5BBITk/Ujts31NHBO+WnJ0XKixOjRLGlz3e prlOnwtJtZJwNBL6jJGlncOiEzIEQM9JiQy49puwro/elWMdKZ6vfCcyfaFMJZSWZX tbcJdDpLezYjP13Mf1IU33y/CjqGfhrjSYM2adA3O03RsbhWv3M6DsQ1mQJAOS/ne0 l9aJ7KI8sf7RsOzTAom5xS/BJ58yKpFd6VKsaeejHpxdyXneLagGzEkJsj2EG1JrvI KJye454HfbDwdBz8OEqMLvOdeLvgiag/vFi6N/ijYDSEiFrAhLgpWR+l+qPfJIaVLm /fMV2CSakMPGA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 6B5955C30D2; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 17/21] srcu: Add contention-triggered addition of srcu_node tree Date: Mon, 18 Apr 2022 17:03:18 -0700 Message-Id: <20220419000322.3948903-17-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit instruments the acquisitions of the srcu_struct structure's ->lock, enabling the initiation of a transition from SRCU_SIZE_SMALL to SRCU_SIZE_BIG when sufficient contention is experienced. The instrumentation counts the number of trylock failures within the confines of a single jiffy. If that number exceeds the value specified by the srcutree.small_contention_lim kernel boot parameter (which defaults to 100), and if the value specified by the srcutree.convert_to_big kernel boot parameter has the 0x10 bit set (defaults to 0), then a transition will be automatically initiated. By default, there will never be any transitions, so that none of the srcu_struct structures ever gains an srcu_node array. The useful values for srcutree.convert_to_big are: 0x00: Never convert. 0x01: Always convert at init_srcu_struct() time. 0x02: Convert when rcutorture prints its first round of statistics. 0x03: Decide conversion approach at boot given system size. 0x10: Convert if contention is encountered. 0x12: Convert if contention is encountered or when rcutorture prints its first round of statistics, whichever comes first. The value 0x11 acts the same as 0x01 because the conversion happens before there is any chance of contention. [ paulmck: Apply "static" feedback from kernel test robot. ] Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- .../admin-guide/kernel-parameters.txt | 9 ++ include/linux/srcutree.h | 2 + kernel/rcu/srcutree.c | 107 ++++++++++++++---- 3 files changed, 94 insertions(+), 24 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 1f1fcac7777d..177e688768c0 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5616,6 +5616,7 @@ 0: Never. 1: At init_srcu_struct() time. 2: When rcutorture decides to. + 0x1X: Above plus if high contention. =20 Either way, the srcu_node tree will be sized based on the actual runtime number of CPUs (nr_cpu_ids) @@ -5638,6 +5639,14 @@ expediting. Set to zero to disable automatic expediting. =20 + srcutree.small_contention_lim [KNL] + Specifies the number of update-side contention + events per jiffy will be tolerated before + initiating a conversion of an srcu_struct + structure to big form. Note that the value of + srcutree.convert_to_big must have the 0x10 bit + set for contention-based conversions to occur. + ssbd=3D [ARM64,HW] Speculative Store Bypass Disable control =20 diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 44bd204498a1..1b9ff4ed37e4 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -72,6 +72,8 @@ struct srcu_struct { unsigned long srcu_gp_seq_needed; /* Latest gp_seq needed. */ unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */ unsigned long srcu_last_gp_end; /* Last GP end timestamp (ns) */ + unsigned long srcu_size_jiffies; /* Current contention-measurement interv= al. */ + unsigned long srcu_n_lock_retries; /* Contention events in current interv= al. */ struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ bool sda_is_static; /* May ->sda be passed to free_percpu()? */ unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index c9460374d437..0bc6a0a3edee 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -41,13 +41,29 @@ module_param(counter_wrap_check, ulong, 0444); =20 /* * Control conversion to SRCU_SIZE_BIG: - * 0: Don't convert at all (default). - * 1: Convert at init_srcu_struct() time. - * 2: Convert when rcutorture invokes srcu_torture_stats_print(). + * 0: Don't convert at all (default). + * 1: Convert at init_srcu_struct() time. + * 2: Convert when rcutorture invokes srcu_torture_stats_print(). + * 3: Decide at boot time based on system shape. + * 0x1x: Convert when excessive contention encountered. */ -static int convert_to_big; +#define SRCU_SIZING_NONE 0 +#define SRCU_SIZING_INIT 1 +#define SRCU_SIZING_TORTURE 2 +#define SRCU_SIZING_AUTO 3 +#define SRCU_SIZING_CONTEND 0x10 +#define SRCU_SIZING_IS(x) ((convert_to_big & ~SRCU_SIZING_CONTEND) =3D=3D = x) +#define SRCU_SIZING_IS_NONE() (SRCU_SIZING_IS(SRCU_SIZING_NONE)) +#define SRCU_SIZING_IS_INIT() (SRCU_SIZING_IS(SRCU_SIZING_INIT)) +#define SRCU_SIZING_IS_TORTURE() (SRCU_SIZING_IS(SRCU_SIZING_TORTURE)) +#define SRCU_SIZING_IS_CONTEND() (convert_to_big & SRCU_SIZING_CONTEND) +static int convert_to_big =3D SRCU_SIZING_NONE; module_param(convert_to_big, int, 0444); =20 +/* Contention events per jiffy to initiate transition to big. */ +static int small_contention_lim __read_mostly =3D 100; +module_param(small_contention_lim, int, 0444); + /* Early-boot callback-management, so early that no lock is required! */ static LIST_HEAD(srcu_boot_list); static bool __read_mostly srcu_init_done; @@ -58,31 +74,40 @@ static void process_srcu(struct work_struct *work); static void srcu_delay_timer(struct timer_list *t); =20 /* Wrappers for lock acquisition and release, see raw_spin_lock_rcu_node()= . */ -#define spin_lock_rcu_node(p) \ -do { \ - spin_lock(&ACCESS_PRIVATE(p, lock)); \ - smp_mb__after_unlock_lock(); \ +#define spin_lock_rcu_node(p) \ +do { \ + spin_lock(&ACCESS_PRIVATE(p, lock)); \ + smp_mb__after_unlock_lock(); \ } while (0) =20 #define spin_unlock_rcu_node(p) spin_unlock(&ACCESS_PRIVATE(p, lock)) =20 -#define spin_lock_irq_rcu_node(p) \ -do { \ - spin_lock_irq(&ACCESS_PRIVATE(p, lock)); \ - smp_mb__after_unlock_lock(); \ +#define spin_lock_irq_rcu_node(p) \ +do { \ + spin_lock_irq(&ACCESS_PRIVATE(p, lock)); \ + smp_mb__after_unlock_lock(); \ } while (0) =20 -#define spin_unlock_irq_rcu_node(p) \ +#define spin_unlock_irq_rcu_node(p) \ spin_unlock_irq(&ACCESS_PRIVATE(p, lock)) =20 -#define spin_lock_irqsave_rcu_node(p, flags) \ -do { \ - spin_lock_irqsave(&ACCESS_PRIVATE(p, lock), flags); \ - smp_mb__after_unlock_lock(); \ +#define spin_lock_irqsave_rcu_node(p, flags) \ +do { \ + spin_lock_irqsave(&ACCESS_PRIVATE(p, lock), flags); \ + smp_mb__after_unlock_lock(); \ } while (0) =20 -#define spin_unlock_irqrestore_rcu_node(p, flags) \ - spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags) \ +#define spin_trylock_irqsave_rcu_node(p, flags) \ +({ \ + bool ___locked =3D spin_trylock_irqsave(&ACCESS_PRIVATE(p, lock), flags);= \ + \ + if (___locked) \ + smp_mb__after_unlock_lock(); \ + ___locked; \ +}) + +#define spin_unlock_irqrestore_rcu_node(p, flags) \ + spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags) \ =20 /* * Initialize SRCU per-CPU data. Note that statically allocated @@ -225,7 +250,7 @@ static int init_srcu_struct_fields(struct srcu_struct *= ssp, bool is_static) init_srcu_struct_data(ssp); ssp->srcu_gp_seq_needed_exp =3D 0; ssp->srcu_last_gp_end =3D ktime_get_mono_fast_ns(); - if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 1) { + if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && SRCU_SIZING= _IS_INIT()) { if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { if (!ssp->sda_is_static) { free_percpu(ssp->sda); @@ -272,6 +297,15 @@ EXPORT_SYMBOL_GPL(init_srcu_struct); =20 #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 +/* + * Initiate a transition to SRCU_SIZE_BIG with lock held. + */ +static void __srcu_transition_to_big(struct srcu_struct *ssp) +{ + lockdep_assert_held(&ACCESS_PRIVATE(ssp, lock)); + smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_ALLOC); +} + /* * Initiate an idempotent transition to SRCU_SIZE_BIG. */ @@ -287,10 +321,35 @@ static void srcu_transition_to_big(struct srcu_struct= *ssp) spin_unlock_irqrestore_rcu_node(ssp, flags); return; } - smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_ALLOC); + __srcu_transition_to_big(ssp); spin_unlock_irqrestore_rcu_node(ssp, flags); } =20 +/* + * Acquire the specified srcu_struct structure's ->lock, but check for + * excessive contention, which results in initiation of a transition + * to SRCU_SIZE_BIG. But only if the srcutree.convert_to_big module + * parameter permits this. + */ +static void spin_lock_irqsave_ssp_contention(struct srcu_struct *ssp, unsi= gned long *flags) +{ + unsigned long j; + + if (spin_trylock_irqsave_rcu_node(ssp, *flags)) + return; + spin_lock_irqsave_rcu_node(ssp, *flags); + if (!SRCU_SIZING_IS_CONTEND() || ssp->srcu_size_state) + return; + j =3D jiffies; + if (ssp->srcu_size_jiffies !=3D j) { + ssp->srcu_size_jiffies =3D j; + ssp->srcu_n_lock_retries =3D 0; + } + if (++ssp->srcu_n_lock_retries <=3D small_contention_lim) + return; + __srcu_transition_to_big(ssp); +} + /* * First-use initialization of statically allocated srcu_struct * structure. Wiring up the combining tree is more than can be @@ -718,7 +777,7 @@ static void srcu_funnel_exp_start(struct srcu_struct *s= sp, struct srcu_node *snp WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); spin_unlock_irqrestore_rcu_node(snp, flags); } - spin_lock_irqsave_rcu_node(ssp, flags); + spin_lock_irqsave_ssp_contention(ssp, &flags); if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, s)) WRITE_ONCE(ssp->srcu_gp_seq_needed_exp, s); spin_unlock_irqrestore_rcu_node(ssp, flags); @@ -779,7 +838,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ss= p, struct srcu_data *sdp, } =20 /* Top of tree, must ensure the grace period will be started. */ - spin_lock_irqsave_rcu_node(ssp, flags); + spin_lock_irqsave_ssp_contention(ssp, &flags); if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed, s)) { /* * Record need for grace period s. Pair with load @@ -1542,7 +1601,7 @@ void srcu_torture_stats_print(struct srcu_struct *ssp= , char *tt, char *tf) } pr_cont(" T(%ld,%ld)\n", s0, s1); } - if (convert_to_big =3D=3D 2) + if (SRCU_SIZING_IS_TORTURE()) srcu_transition_to_big(ssp); } EXPORT_SYMBOL_GPL(srcu_torture_stats_print); --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C16C8C433EF for ; Tue, 19 Apr 2022 00:04:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236929AbiDSAHF (ORCPT ); Mon, 18 Apr 2022 20:07:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236870AbiDSAGJ (ORCPT ); Mon, 18 Apr 2022 20:06:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B176237FA; Mon, 18 Apr 2022 17:03:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A4726612DC; Tue, 19 Apr 2022 00:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4BEF1C385C7; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=YwNURKj9mCIVjxikZL9NORCNyWNTo9uPcdOZhbmsazc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RxdI+F6nO1OTvalSor++1mZLjYnYU8Y+aoppUZZ8k3QCqMWlamU/36mJsF+OZGNBT 86yeE7C3ZvDHvpaj+g00OR0hgH3ecm1bhH7UsBlPDRubzRgTS3lsq+RxR2/RIfna81 YkzC9dnjPQKUPrv6t1R6kGnkf+1qWhayp0KqEv/OvRFxFsbOGQgF53cP7oaXl3mIL3 E4R51gi4VREdCVH25xDY76DlJIpllyVqDGH5W8h9+IdyooPU8OP40d0xuiKLIyBLSz 4EAdjiGIH7Giw8Q3UaOSG5gMWXzIkKUqDZGUv95f1d9rW6u7kgPYTqHjdHyj6+LSZa 8kPVNIsHbcDww== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 6D2B45C30DF; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 18/21] srcu: Automatically determine size-transition strategy at boot Date: Mon, 18 Apr 2022 17:03:19 -0700 Message-Id: <20220419000322.3948903-18-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit adds a srcutree.convert_to_big option of zero that causes SRCU to decide at boot whether to wait for contention (small systems) or immediately expand to large (large systems). A new srcutree.big_cpu_lim (defaulting to 128) defines how many CPUs constitute a large system. Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- .../admin-guide/kernel-parameters.txt | 9 ++++++++ kernel/rcu/srcutree.c | 23 ++++++++++++++++--- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 177e688768c0..057e30613d00 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5608,6 +5608,14 @@ off: Disable mitigation and remove performance impact to RDRAND and RDSEED =20 + srcutree.big_cpu_lim [KNL] + Specifies the number of CPUs constituting a + large system, such that srcu_struct structures + should immediately allocate an srcu_node array. + This kernel-boot parameter defaults to 128, but + takes effect only when srcutree.convert_to_big + is equal to zero. + srcutree.convert_to_big [KNL] Specifies under what conditions an SRCU tree srcu_struct structure will be converted to big @@ -5616,6 +5624,7 @@ 0: Never. 1: At init_srcu_struct() time. 2: When rcutorture decides to. + 3: Decide at boot time (default). 0x1X: Above plus if high contention. =20 Either way, the srcu_node tree will be sized based diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 0bc6a0a3edee..b9dec26245e0 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -41,10 +41,10 @@ module_param(counter_wrap_check, ulong, 0444); =20 /* * Control conversion to SRCU_SIZE_BIG: - * 0: Don't convert at all (default). + * 0: Don't convert at all. * 1: Convert at init_srcu_struct() time. * 2: Convert when rcutorture invokes srcu_torture_stats_print(). - * 3: Decide at boot time based on system shape. + * 3: Decide at boot time based on system shape (default). * 0x1x: Convert when excessive contention encountered. */ #define SRCU_SIZING_NONE 0 @@ -57,9 +57,13 @@ module_param(counter_wrap_check, ulong, 0444); #define SRCU_SIZING_IS_INIT() (SRCU_SIZING_IS(SRCU_SIZING_INIT)) #define SRCU_SIZING_IS_TORTURE() (SRCU_SIZING_IS(SRCU_SIZING_TORTURE)) #define SRCU_SIZING_IS_CONTEND() (convert_to_big & SRCU_SIZING_CONTEND) -static int convert_to_big =3D SRCU_SIZING_NONE; +static int convert_to_big =3D SRCU_SIZING_AUTO; module_param(convert_to_big, int, 0444); =20 +/* Number of CPUs to trigger init_srcu_struct()-time transition to big. */ +static int big_cpu_lim __read_mostly =3D 128; +module_param(big_cpu_lim, int, 0444); + /* Contention events per jiffy to initiate transition to big. */ static int small_contention_lim __read_mostly =3D 100; module_param(small_contention_lim, int, 0444); @@ -1619,6 +1623,17 @@ void __init srcu_init(void) { struct srcu_struct *ssp; =20 + /* Decide on srcu_struct-size strategy. */ + if (SRCU_SIZING_IS(SRCU_SIZING_AUTO)) { + if (nr_cpu_ids >=3D big_cpu_lim) { + convert_to_big =3D SRCU_SIZING_INIT; // Don't bother waiting for conten= tion. + pr_info("%s: Setting srcu_struct sizes to big.\n", __func__); + } else { + convert_to_big =3D SRCU_SIZING_NONE | SRCU_SIZING_CONTEND; + pr_info("%s: Setting srcu_struct sizes based on contention.\n", __func_= _); + } + } + /* * Once that is set, call_srcu() can follow the normal path and * queue delayed work. This must follow RCU workqueues creation @@ -1629,6 +1644,8 @@ void __init srcu_init(void) ssp =3D list_first_entry(&srcu_boot_list, struct srcu_struct, work.work.entry); list_del_init(&ssp->work.work.entry); + if (SRCU_SIZING_IS(SRCU_SIZING_INIT) && ssp->srcu_size_state =3D=3D SRCU= _SIZE_SMALL) + ssp->srcu_size_state =3D SRCU_SIZE_ALLOC; queue_work(rcu_gp_wq, &ssp->work.work); } } --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 385CCC433EF for ; Tue, 19 Apr 2022 00:04:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237543AbiDSAHY (ORCPT ); Mon, 18 Apr 2022 20:07:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236978AbiDSAGK (ORCPT ); Mon, 18 Apr 2022 20:06:10 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88D3513F97; Mon, 18 Apr 2022 17:03:29 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 62C6FB8114D; Tue, 19 Apr 2022 00:03:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 49B42C385C8; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=+VktB+rpP/FYBGfEw3GwNqdjH8hwEOL7Xhn+Gfa9jsg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UUzfG8mrpV2g9yQ6tQ6arKwCvcEBkkkBxBG8iyPo8sCeFge/2ZcW9LYuAeCmUYTRW NokxM8JVd60S0Lg6cMgWebTv2AX1RKsAknyNVYkWIzhlZUzAs88s410yS79yHBOVYW ZGn4qHTiNuhX743jmRGvzcsYGfLNPJqz7uxQT7Pr4dxfZ0D02Ey7E3fmT8cxbsRk25 VsWfL17b68KCAZwi1hHtizNfOr/Zdj2ewBxjzIGlmTRuedgssSf1SHalehPAJZUdhR 14WHBUuJ1FH2ZIeAe3wjK8ExOT0qvSkWLV7PAIUY3/V5CCqaTULlviY6vtRHBNJymu aX+vmwRnsHA2g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 6EEE85C30E1; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay Subject: [PATCH rcu 19/21] srcu: Add contention check to call_srcu() srcu_data ->lock acquisition Date: Mon, 18 Apr 2022 17:03:20 -0700 Message-Id: <20220419000322.3948903-19-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit increases the sensitivity of contention detection by adding checks to the acquisition of the srcu_data structure's lock on the call_srcu() code path. Co-developed-by: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 45 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index b9dec26245e0..862008c147b0 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -330,18 +330,13 @@ static void srcu_transition_to_big(struct srcu_struct= *ssp) } =20 /* - * Acquire the specified srcu_struct structure's ->lock, but check for - * excessive contention, which results in initiation of a transition - * to SRCU_SIZE_BIG. But only if the srcutree.convert_to_big module - * parameter permits this. + * Check to see if the just-encountered contention event justifies + * a transition to SRCU_SIZE_BIG. */ -static void spin_lock_irqsave_ssp_contention(struct srcu_struct *ssp, unsi= gned long *flags) +static void spin_lock_irqsave_check_contention(struct srcu_struct *ssp) { unsigned long j; =20 - if (spin_trylock_irqsave_rcu_node(ssp, *flags)) - return; - spin_lock_irqsave_rcu_node(ssp, *flags); if (!SRCU_SIZING_IS_CONTEND() || ssp->srcu_size_state) return; j =3D jiffies; @@ -354,6 +349,38 @@ static void spin_lock_irqsave_ssp_contention(struct sr= cu_struct *ssp, unsigned l __srcu_transition_to_big(ssp); } =20 +/* + * Acquire the specified srcu_data structure's ->lock, but check for + * excessive contention, which results in initiation of a transition + * to SRCU_SIZE_BIG. But only if the srcutree.convert_to_big module + * parameter permits this. + */ +static void spin_lock_irqsave_sdp_contention(struct srcu_data *sdp, unsign= ed long *flags) +{ + struct srcu_struct *ssp =3D sdp->ssp; + + if (spin_trylock_irqsave_rcu_node(sdp, *flags)) + return; + spin_lock_irqsave_rcu_node(ssp, *flags); + spin_lock_irqsave_check_contention(ssp); + spin_unlock_irqrestore_rcu_node(ssp, *flags); + spin_lock_irqsave_rcu_node(sdp, *flags); +} + +/* + * Acquire the specified srcu_struct structure's ->lock, but check for + * excessive contention, which results in initiation of a transition + * to SRCU_SIZE_BIG. But only if the srcutree.convert_to_big module + * parameter permits this. + */ +static void spin_lock_irqsave_ssp_contention(struct srcu_struct *ssp, unsi= gned long *flags) +{ + if (spin_trylock_irqsave_rcu_node(ssp, *flags)) + return; + spin_lock_irqsave_rcu_node(ssp, *flags); + spin_lock_irqsave_check_contention(ssp); +} + /* * First-use initialization of statically allocated srcu_struct * structure. Wiring up the combining tree is more than can be @@ -1010,7 +1037,7 @@ static unsigned long srcu_gp_start_if_needed(struct s= rcu_struct *ssp, sdp =3D per_cpu_ptr(ssp->sda, 0); else sdp =3D raw_cpu_ptr(ssp->sda); - spin_lock_irqsave_rcu_node(sdp, flags); + spin_lock_irqsave_sdp_contention(sdp, &flags); if (rhp) rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp); rcu_segcblist_advance(&sdp->srcu_cblist, --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15EFAC433F5 for ; Tue, 19 Apr 2022 00:04:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238007AbiDSAHQ (ORCPT ); Mon, 18 Apr 2022 20:07:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236933AbiDSAGJ (ORCPT ); Mon, 18 Apr 2022 20:06:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B29F2496B; Mon, 18 Apr 2022 17:03:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 45917612D3; Tue, 19 Apr 2022 00:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 55FC8C385CA; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=OYXi6oMxVVELQzNdo5jYt/IiIWzJKIlnGTPP/z8oSgM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BLD46zBfqV7QiuOTvLwhMxluOSVA9Nx7TDtqZARGcq8qpjqXyjeCvIaVoBvQnp7KX 0/QccrXhKAnM+FKnjSZAzfrzBkQNqF28uuH2tK/Vt2FB5p8WlqeKkeUvPXee/abFoC hLiXlMfCXtIZsKEzoQxHkm9/8Z/9a6/0q9ll709k6za1CzSCJA/aHLrIErJ9s2hsmq 8blodPsacCRJ+UFvAausD28U2IJsIXiXmvfegbCKfVpdAfLQ6e2I2KksIXqmqVnJru NtAtAUr5TIgzEWRdU+SONh0juvCKGL2tWrfHFyOaSqMyEDMIb5Qeu6C6IvH4CJtyNg dgwPbzDbbZWDA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 710145C30E3; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay , Song Liu , kernel test robot Subject: [PATCH rcu 20/21] srcu: Prevent expedited GPs and blocking readers from consuming CPU Date: Mon, 18 Apr 2022 17:03:21 -0700 Message-Id: <20220419000322.3948903-20-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If an SRCU reader blocks while a synchronize_srcu_expedited() waits for that same reader, then that grace period will spawn an endless series of workqueue handlers, consuming a full CPU. This quickly gets pointless because consuming more CPU isn't going to make that reader get done faster, especially if it is blocked waiting for an external event. This commit therefore spawns at most one pair of back-to-back workqueue handlers per expedited grace period phase, instead inserting increasing delays as that grace period phase grows older, but capped at 10 jiffies. In any case, if there have been at least 100 back-to-back workqueue handlers within a single jiffy, regardless of grace period or grace-period phase, then a one-jiffy delay is inserted. [ paulmck: Apply feedback from kernel test robot. ] Cc: Neeraj Upadhyay Reported-by: Song Liu Tested-by: kernel test robot Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 4 ++++ kernel/rcu/srcutree.c | 44 ++++++++++++++++++++++++++++++++++------ 2 files changed, 42 insertions(+), 6 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 1b9ff4ed37e4..e3014319d1ad 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -71,9 +71,11 @@ struct srcu_struct { unsigned long srcu_gp_seq; /* Grace-period seq #. */ unsigned long srcu_gp_seq_needed; /* Latest gp_seq needed. */ unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */ + unsigned long srcu_gp_start; /* Last GP start timestamp (jiffies) */ unsigned long srcu_last_gp_end; /* Last GP end timestamp (ns) */ unsigned long srcu_size_jiffies; /* Current contention-measurement interv= al. */ unsigned long srcu_n_lock_retries; /* Contention events in current interv= al. */ + unsigned long srcu_n_exp_nodelay; /* # expedited no-delays in current GP = phase. */ struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ bool sda_is_static; /* May ->sda be passed to free_percpu()? */ unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ @@ -83,6 +85,8 @@ struct srcu_struct { atomic_t srcu_barrier_cpu_cnt; /* # CPUs not yet posting a */ /* callback for the barrier */ /* operation. */ + unsigned long reschedule_jiffies; + unsigned long reschedule_count; struct delayed_work work; struct lockdep_map dep_map; }; diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 862008c147b0..6dd44e759f12 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -511,7 +511,10 @@ static bool srcu_readers_active(struct srcu_struct *ss= p) return sum; } =20 -#define SRCU_INTERVAL 1 +#define SRCU_INTERVAL 1 // Base delay if no expedited GPs pending. +#define SRCU_MAX_INTERVAL 10 // Maximum incremental delay from slow reader= s. +#define SRCU_MAX_NODELAY_PHASE 1 // Maximum per-GP-phase consecutive no-de= lay instances. +#define SRCU_MAX_NODELAY 100 // Maximum consecutive no-delay instances. =20 /* * Return grace-period delay, zero if there are expedited grace @@ -519,9 +522,18 @@ static bool srcu_readers_active(struct srcu_struct *ss= p) */ static unsigned long srcu_get_delay(struct srcu_struct *ssp) { + unsigned long jbase =3D SRCU_INTERVAL; + if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_gp_seq), READ_ONCE(ssp->srcu_gp_seq_= needed_exp))) - return 0; - return SRCU_INTERVAL; + jbase =3D 0; + if (rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq))) + jbase +=3D jiffies - READ_ONCE(ssp->srcu_gp_start); + if (!jbase) { + WRITE_ONCE(ssp->srcu_n_exp_nodelay, READ_ONCE(ssp->srcu_n_exp_nodelay) += 1); + if (READ_ONCE(ssp->srcu_n_exp_nodelay) > SRCU_MAX_NODELAY_PHASE) + jbase =3D 1; + } + return jbase > SRCU_MAX_INTERVAL ? SRCU_MAX_INTERVAL : jbase; } =20 /** @@ -623,6 +635,8 @@ static void srcu_gp_start(struct srcu_struct *ssp) (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, rcu_seq_snap(&ssp->srcu_gp_seq)); spin_unlock_rcu_node(sdp); /* Interrupts remain disabled. */ + WRITE_ONCE(ssp->srcu_gp_start, jiffies); + WRITE_ONCE(ssp->srcu_n_exp_nodelay, 0); smp_mb(); /* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ rcu_seq_start(&ssp->srcu_gp_seq); state =3D rcu_seq_state(ssp->srcu_gp_seq); @@ -706,7 +720,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) spin_lock_irq_rcu_node(ssp); idx =3D rcu_seq_state(ssp->srcu_gp_seq); WARN_ON_ONCE(idx !=3D SRCU_STATE_SCAN2); - cbdelay =3D srcu_get_delay(ssp); + cbdelay =3D !!srcu_get_delay(ssp); WRITE_ONCE(ssp->srcu_last_gp_end, ktime_get_mono_fast_ns()); rcu_seq_end(&ssp->srcu_gp_seq); gpseq =3D rcu_seq_current(&ssp->srcu_gp_seq); @@ -893,7 +907,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ss= p, struct srcu_data *sdp, // the one boot CPU running with interrupts still disabled. if (likely(srcu_init_done)) queue_delayed_work(rcu_gp_wq, &ssp->work, - srcu_get_delay(ssp)); + !!srcu_get_delay(ssp)); else if (list_empty(&ssp->work.work.entry)) list_add(&ssp->work.work.entry, &srcu_boot_list); } @@ -1448,6 +1462,7 @@ static void srcu_advance_state(struct srcu_struct *ss= p) srcu_flip(ssp); spin_lock_irq_rcu_node(ssp); rcu_seq_set_state(&ssp->srcu_gp_seq, SRCU_STATE_SCAN2); + ssp->srcu_n_exp_nodelay =3D 0; spin_unlock_irq_rcu_node(ssp); } =20 @@ -1462,6 +1477,7 @@ static void srcu_advance_state(struct srcu_struct *ss= p) mutex_unlock(&ssp->srcu_gp_mutex); return; /* readers present, retry later. */ } + ssp->srcu_n_exp_nodelay =3D 0; srcu_gp_end(ssp); /* Releases ->srcu_gp_mutex. */ } } @@ -1552,12 +1568,28 @@ static void srcu_reschedule(struct srcu_struct *ssp= , unsigned long delay) */ static void process_srcu(struct work_struct *work) { + unsigned long curdelay; + unsigned long j; struct srcu_struct *ssp; =20 ssp =3D container_of(work, struct srcu_struct, work.work); =20 srcu_advance_state(ssp); - srcu_reschedule(ssp, srcu_get_delay(ssp)); + curdelay =3D srcu_get_delay(ssp); + if (curdelay) { + WRITE_ONCE(ssp->reschedule_count, 0); + } else { + j =3D jiffies; + if (READ_ONCE(ssp->reschedule_jiffies) =3D=3D j) { + WRITE_ONCE(ssp->reschedule_count, READ_ONCE(ssp->reschedule_count) + 1); + if (READ_ONCE(ssp->reschedule_count) > SRCU_MAX_NODELAY) + curdelay =3D 1; + } else { + WRITE_ONCE(ssp->reschedule_count, 1); + WRITE_ONCE(ssp->reschedule_jiffies, j); + } + } + srcu_reschedule(ssp, curdelay); } =20 void srcutorture_get_gp_data(enum rcutorture_type test_type, --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:52 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53230C433EF for ; Tue, 19 Apr 2022 00:05:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237679AbiDSAH1 (ORCPT ); Mon, 18 Apr 2022 20:07:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229920AbiDSAGK (ORCPT ); Mon, 18 Apr 2022 20:06:10 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A06C1FA7E; Mon, 18 Apr 2022 17:03:29 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 61FA1B81142; Tue, 19 Apr 2022 00:03:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5655AC385CB; Tue, 19 Apr 2022 00:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326605; bh=UYudwu1QrF7TbcsMmA2KTg1pYLeL053zgVc8k9emey8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F0KjUbQ6RmtdKOR9u22u/p1b8WtaU3OW9MgNbw68SjOFVxyrgJ14FEVxnqaJTDdkm IeFMPr7qYL1SN4hcRLnBT0kKRKSVltLRibOhKu9HYwf+HVI1NIl3jk9eKOTV0GpsYO 6wZoWy6NeAsN4tYzLXbYhcKyqSYcc63GaANK/+DGz6DuSAgdVIUgs+E5ljAzbuwnQ4 9N/2I9aIRJkAxZtRV+vp/1NQqOg3E6jeLSWcSbYvB261tkxR0Uettttmrzzc5nnwl2 CFx/iFR1vwiGQ93Nvhcd6ZjPeBHvJnNiIAyvlxEVni1mgYf4QnDRIkbtMKpY4AIICA HxMheCjm7gRDA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 72BA55C30EA; Mon, 18 Apr 2022 17:03:24 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Lukas Bulwahn , "Paul E . McKenney" Subject: [PATCH rcu 21/21] srcu: Drop needless initialization of sdp in srcu_gp_start() Date: Mon, 18 Apr 2022 17:03:22 -0700 Message-Id: <20220419000322.3948903-21-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> References: <20220419000315.GA3948789@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Lukas Bulwahn Commit 9c7ef4c30f12 ("srcu: Make Tree SRCU able to operate without snp_node array") initializes the local variable sdp differently depending on the srcu's state in srcu_gp_start(). Either way, this initialization overwrites the value used when sdp is defined. This commit therefore drops this pointless definition-time initialization. Although there is no functional change, compiler code generation may be affected. Signed-off-by: Lukas Bulwahn Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 6dd44e759f12..50ba70f019de 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -620,7 +620,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock); */ static void srcu_gp_start(struct srcu_struct *ssp) { - struct srcu_data *sdp =3D this_cpu_ptr(ssp->sda); + struct srcu_data *sdp; int state; =20 if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) --=20 2.31.1.189.g2e36527f23