From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D90CC433FE for ; Fri, 4 Feb 2022 23:39:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378286AbiBDXjK (ORCPT ); Fri, 4 Feb 2022 18:39:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378231AbiBDXjF (ORCPT ); Fri, 4 Feb 2022 18:39:05 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1ED2BE025B2D; Fri, 4 Feb 2022 15:39:05 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id AA3A561C36; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 15A85C004E1; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=4LJKR8LrRJnPaBrRYneeP2Xiv81UOQNEztaGmdaYkWg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CrzsfZAU6BrQKiS8paJXVOS+vA8vuIFNCndI6SQEWTMRK35sU2o4Iv1IOqMYp1DNE EHlAIlGcpJYO5oZO+4gv51pGE2AA5+jI/yqnS2urARjckX3XiMBzOCcRdC7oW384s3 /3T+UXNUGMnTD0ywBZ5ZN2BJxS6GoT79fkq7Fy/yrOHLverSXhREW/C3S2vNL9g2cZ 54OmKpSZitYdNPqseO9A8v8uqoG2qeZUvtqWtJJBW+9vn6Gq+7qN1n/EJug33iYA0C DH9xPy5u6D6LdVP8mnxiVmQli9LDIQOnpOcO5b6OBubqM4+swyPVj85VsJp/Vch1Ym mqe2robaYreAA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E77755C0418; Fri, 4 Feb 2022 15:39:03 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 01/19] srcu: Tighten cleanup_srcu_struct() GP checks Date: Fri, 4 Feb 2022 15:38:44 -0800 Message-Id: <20220204233902.1902-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, cleanup_srcu_struct() checks for a grace period in progress, but it does not check for a grace period that has not yet started but which might start at any time. Such a situation could result in a use-after-free bug, so this commit adds a check for a grace period that is needed but not yet started to cleanup_srcu_struct(). Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 6833d88871816..d30e4db04506a 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -382,9 +382,11 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) return; /* Forgot srcu_barrier(), so just leak it! */ } if (WARN_ON(rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)) !=3D SRCU_STATE_ID= LE) || + WARN_ON(rcu_seq_current(&ssp->srcu_gp_seq) !=3D ssp->srcu_gp_seq_need= ed) || WARN_ON(srcu_readers_active(ssp))) { - pr_info("%s: Active srcu_struct %p state: %d\n", - __func__, ssp, rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq))); + pr_info("%s: Active srcu_struct %p read state: %d gp state: %lu/%lu\n", + __func__, ssp, rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)), + rcu_seq_current(&ssp->srcu_gp_seq), ssp->srcu_gp_seq_needed); return; /* Caller forgot to stop doing call_srcu()? */ } free_percpu(ssp->sda); --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25064C433EF for ; Fri, 4 Feb 2022 23:39:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378266AbiBDXjH (ORCPT ); Fri, 4 Feb 2022 18:39:07 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:47868 "EHLO dfw.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378223AbiBDXjF (ORCPT ); Fri, 4 Feb 2022 18:39:05 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BB21B61C12; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 245D5C340ED; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=GiLrQ3LZHqRBis8jo4OGf4RQr8OnWDgZfbmOreYAyMo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i0lG4kL5LGHXvjH7vnxUId9ZbQvzm8X/+B9WefPgJWzq0Ug2D97o7gHKwFEqtnYHQ f8VOtFwA9jRK0gYUicubYeiZpge1GQzcnzTheDJdR7+nzHOadvqYhbvD2gBhwnNijV c1fUaAYivxO0UiwxJAlDe5sal6hXtBINJDsR3spMQhH3L3/+11Ye6A/VpCdnmc21Yu cwva3hZm3zKvz/r1F7PFg+6OAtdIkRB9Rv+UzzBO2aVfdOSKiQdWgyzK4xkeYqQ6bv fOwn/hT2rYpfNluiYeuy3d3QzRmzq1RkFt/uKbNw2RwZGlml46WrX1eAWsoMzr2Mz/ DEtDr24/y6kAw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E98DD5C0829; Fri, 4 Feb 2022 15:39:03 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 02/19] srcu: Fix s/is/if/ typo in srcu_node comment Date: Fri, 4 Feb 2022 15:38:45 -0800 Message-Id: <20220204233902.1902-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit fixed a typo in the srcu_node structure's ->srcu_have_cbs comment. While in the area, redo a couple of comments to take advantage of 100-character line lengths. Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index cb1f4351e8baa..4025840ba9a38 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -47,11 +47,9 @@ struct srcu_data { */ struct srcu_node { spinlock_t __private lock; - unsigned long srcu_have_cbs[4]; /* GP seq for children */ - /* having CBs, but only */ - /* is > ->srcu_gq_seq. */ - unsigned long srcu_data_have_cbs[4]; /* Which srcu_data structs */ - /* have CBs for given GP? */ + unsigned long srcu_have_cbs[4]; /* GP seq for children having CBs, but o= nly */ + /* if greater than ->srcu_gq_seq. */ + unsigned long srcu_data_have_cbs[4]; /* Which srcu_data structs have CBs = for given GP? */ unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */ struct srcu_node *srcu_parent; /* Next up in tree. */ int grplo; /* Least CPU for node. */ --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA823C433F5 for ; Fri, 4 Feb 2022 23:39:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378281AbiBDXjN (ORCPT ); Fri, 4 Feb 2022 18:39:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378232AbiBDXjF (ORCPT ); Fri, 4 Feb 2022 18:39:05 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22BE7E025B2E; Fri, 4 Feb 2022 15:39:05 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B44F761C9C; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1C04EC340F0; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=GaED8m6moSZlPrHrLk8hrW3KCvJ5yaGPsNW+YksLZx0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HpIdZnCZJNa73OinJU/3Iy56uJGqrXp8ZqUVTIPqZW2UKLRfCudhWojiaCRrZHUC2 JGVW8by03zoCJa6vtIlgiejEw5MBnl4Ezb9sS/kYGmbCs4IaUG33uAu3N9hKNPOrTc EwWgRQ0CD3tozPzFefyjIhEp3VMWexcdNQEo0wWe8POh+PAZH/s3dB+k3X/6A6V+Yw PMUcahYkqi3LBsk/kIRrzIi63hjK9QEihjhjHMt5G4KsMNE+ZzfnyCqA2Gctzuevag ARpUyqT2YBUbHW/nDmHilUPaju8s7nXAG9K+U5Y1+tPkxkpVq3/pxP7NA5NrYZ95hS jKRsMt4i8UAGg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EB8B35C08B7; Fri, 4 Feb 2022 15:39:03 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 03/19] srcu: Make srcu_funnel_gp_start() cache ->mynode in snp_leaf Date: Fri, 4 Feb 2022 15:38:46 -0800 Message-Id: <20220204233902.1902-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, the srcu_funnel_gp_start() walks its local variable snp up the tree and reloads sdp->mynode whenever it is necessary to check whether it is still at the leaf srcu_node level. This works, but is a bit more obtuse than absolutely necessary. In addition, upcoming commits will dynamically size srcu_struct structures, in which case sdp->mynode will no longer necessarily be a constant, and this commit helps prepare for that dynamic sizing. Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index d30e4db04506a..7d13e35e5d277 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -632,20 +632,21 @@ static void srcu_funnel_gp_start(struct srcu_struct *= ssp, struct srcu_data *sdp, { unsigned long flags; int idx =3D rcu_seq_ctr(s) % ARRAY_SIZE(sdp->mynode->srcu_have_cbs); - struct srcu_node *snp =3D sdp->mynode; + struct srcu_node *snp; + struct srcu_node *snp_leaf =3D sdp->mynode; unsigned long snp_seq; =20 /* Each pass through the loop does one level of the srcu_node tree. */ - for (; snp !=3D NULL; snp =3D snp->srcu_parent) { - if (rcu_seq_done(&ssp->srcu_gp_seq, s) && snp !=3D sdp->mynode) + for (snp =3D snp_leaf; snp !=3D NULL; snp =3D snp->srcu_parent) { + if (rcu_seq_done(&ssp->srcu_gp_seq, s) && snp !=3D snp_leaf) return; /* GP already done and CBs recorded. */ spin_lock_irqsave_rcu_node(snp, flags); if (ULONG_CMP_GE(snp->srcu_have_cbs[idx], s)) { snp_seq =3D snp->srcu_have_cbs[idx]; - if (snp =3D=3D sdp->mynode && snp_seq =3D=3D s) + if (snp =3D=3D snp_leaf && snp_seq =3D=3D s) snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; spin_unlock_irqrestore_rcu_node(snp, flags); - if (snp =3D=3D sdp->mynode && snp_seq !=3D s) { + if (snp =3D=3D snp_leaf && snp_seq !=3D s) { srcu_schedule_cbs_sdp(sdp, do_norm ? SRCU_INTERVAL : 0); @@ -656,7 +657,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ss= p, struct srcu_data *sdp, return; } snp->srcu_have_cbs[idx] =3D s; - if (snp =3D=3D sdp->mynode) + if (snp =3D=3D snp_leaf) snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; if (!do_norm && ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, s)) WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 612EFC433F5 for ; Fri, 4 Feb 2022 23:39:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378509AbiBDXjn (ORCPT ); Fri, 4 Feb 2022 18:39:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378259AbiBDXjH (ORCPT ); Fri, 4 Feb 2022 18:39:07 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC101E025B38; Fri, 4 Feb 2022 15:39:06 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 609EEB83907; Fri, 4 Feb 2022 23:39:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2C6A9C340EF; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=221MY6qzRj/ZNQOo8A/J8gIIMZFreoPNXKFxD592S9c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jYLUlJ/hVFOcHTJMUyYKrPMXjifFCArym2p/WIl6HUltxLWc1jDx7KP8fqG9uYao0 qAV3EQmZchihYALL5wXdBG1Q9LAn6Lo242o2jEDIvRGd5RS2eBIo8xJWtJEfO0Y1w9 iUPWYhokK178NcmyWOKoK6kK6gVpAr+ll3544kqBM/si1cRDQUqeEjo2VVQRvSosG8 Jhw8qDaf8V8fzOLXPyZ3YGSDe4jNFhzQw7PnmiUbYGHXlqYYX/U6Yk52JBov23W9YD vy5MERi1LmqmZIckYPo9QqV0Qk7aWDemNHhhLZijBE8imaXr/y7/72NwpoFj8PQ6nR Q1H/XjWuBbp/Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EDB1E5C08DD; Fri, 4 Feb 2022 15:39:03 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 04/19] srcu: Dynamically allocate srcu_node array Date: Fri, 4 Feb 2022 15:38:47 -0800 Message-Id: <20220204233902.1902-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit shrinks the srcu_struct structure by converting its ->node field from a fixed-size compile-time array to a pointer to a dynamically allocated array. In kernels built with large values of NR_CPUS that boot on systems with smaller numbers of CPUs, this can save significant memory. [ paulmck: Apply kernel test robot feedback. ] Reported-by: A cast of thousands Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 2 +- kernel/rcu/srcutree.c | 58 ++++++++++++++++++++++++++++++---------- 2 files changed, 45 insertions(+), 15 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 4025840ba9a38..f7190058fb8ab 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -60,7 +60,7 @@ struct srcu_node { * Per-SRCU-domain structure, similar in function to rcu_state. */ struct srcu_struct { - struct srcu_node node[NUM_RCU_NODES]; /* Combining tree. */ + struct srcu_node *node; /* Combining tree. */ struct srcu_node *level[RCU_NUM_LVLS + 1]; /* First node at each level. */ struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 7d13e35e5d277..eaf00c354a631 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -24,6 +24,7 @@ #include #include #include +#include #include =20 #include "rcu.h" @@ -75,12 +76,44 @@ do { \ spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags) \ =20 /* - * Initialize SRCU combining tree. Note that statically allocated + * Initialize SRCU per-CPU data. Note that statically allocated * srcu_struct structures might already have srcu_read_lock() and * srcu_read_unlock() running against them. So if the is_static parameter * is set, don't initialize ->srcu_lock_count[] and ->srcu_unlock_count[]. + * + * Returns @true if allocation succeeded and @false otherwise. + */ +static void init_srcu_struct_data(struct srcu_struct *ssp) +{ + int cpu; + struct srcu_data *sdp; + + /* + * Initialize the per-CPU srcu_data array, which feeds into the + * leaves of the srcu_node tree. + */ + WARN_ON_ONCE(ARRAY_SIZE(sdp->srcu_lock_count) !=3D + ARRAY_SIZE(sdp->srcu_unlock_count)); + for_each_possible_cpu(cpu) { + sdp =3D per_cpu_ptr(ssp->sda, cpu); + spin_lock_init(&ACCESS_PRIVATE(sdp, lock)); + rcu_segcblist_init(&sdp->srcu_cblist); + sdp->srcu_cblist_invoking =3D false; + sdp->srcu_gp_seq_needed =3D ssp->srcu_gp_seq; + sdp->srcu_gp_seq_needed_exp =3D ssp->srcu_gp_seq; + sdp->mynode =3D NULL; + sdp->cpu =3D cpu; + INIT_WORK(&sdp->work, srcu_invoke_callbacks); + timer_setup(&sdp->delay_work, srcu_delay_timer, 0); + sdp->ssp =3D ssp; + } +} + +/* + * Allocated and initialize SRCU combining tree. Returns @true if + * allocation succeeded and @false otherwise. */ -static void init_srcu_struct_nodes(struct srcu_struct *ssp) +static bool init_srcu_struct_nodes(struct srcu_struct *ssp) { int cpu; int i; @@ -92,6 +125,9 @@ static void init_srcu_struct_nodes(struct srcu_struct *s= sp) =20 /* Initialize geometry if it has not already been initialized. */ rcu_init_geometry(); + ssp->node =3D kcalloc(rcu_num_nodes, sizeof(*ssp->node), GFP_ATOMIC); + if (!ssp->node) + return false; =20 /* Work out the overall tree geometry. */ ssp->level[0] =3D &ssp->node[0]; @@ -129,29 +165,19 @@ static void init_srcu_struct_nodes(struct srcu_struct= *ssp) * Initialize the per-CPU srcu_data array, which feeds into the * leaves of the srcu_node tree. */ - WARN_ON_ONCE(ARRAY_SIZE(sdp->srcu_lock_count) !=3D - ARRAY_SIZE(sdp->srcu_unlock_count)); level =3D rcu_num_lvls - 1; snp_first =3D ssp->level[level]; for_each_possible_cpu(cpu) { sdp =3D per_cpu_ptr(ssp->sda, cpu); - spin_lock_init(&ACCESS_PRIVATE(sdp, lock)); - rcu_segcblist_init(&sdp->srcu_cblist); - sdp->srcu_cblist_invoking =3D false; - sdp->srcu_gp_seq_needed =3D ssp->srcu_gp_seq; - sdp->srcu_gp_seq_needed_exp =3D ssp->srcu_gp_seq; sdp->mynode =3D &snp_first[cpu / levelspread[level]]; for (snp =3D sdp->mynode; snp !=3D NULL; snp =3D snp->srcu_parent) { if (snp->grplo < 0) snp->grplo =3D cpu; snp->grphi =3D cpu; } - sdp->cpu =3D cpu; - INIT_WORK(&sdp->work, srcu_invoke_callbacks); - timer_setup(&sdp->delay_work, srcu_delay_timer, 0); - sdp->ssp =3D ssp; sdp->grpmask =3D 1 << (cpu - sdp->mynode->grplo); } + return true; } =20 /* @@ -162,6 +188,7 @@ static void init_srcu_struct_nodes(struct srcu_struct *= ssp) */ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) { + ssp->node =3D NULL; mutex_init(&ssp->srcu_cb_mutex); mutex_init(&ssp->srcu_gp_mutex); ssp->srcu_idx =3D 0; @@ -174,7 +201,8 @@ static int init_srcu_struct_fields(struct srcu_struct *= ssp, bool is_static) ssp->sda =3D alloc_percpu(struct srcu_data); if (!ssp->sda) return -ENOMEM; - init_srcu_struct_nodes(ssp); + init_srcu_struct_data(ssp); + WARN_ON_ONCE(!init_srcu_struct_nodes(ssp)); ssp->srcu_gp_seq_needed_exp =3D 0; ssp->srcu_last_gp_end =3D ktime_get_mono_fast_ns(); smp_store_release(&ssp->srcu_gp_seq_needed, 0); /* Init done. */ @@ -391,6 +419,8 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) } free_percpu(ssp->sda); ssp->sda =3D NULL; + kfree(ssp->node); + ssp->node =3D NULL; } EXPORT_SYMBOL_GPL(cleanup_srcu_struct); =20 --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10016C433EF for ; Fri, 4 Feb 2022 23:39:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378429AbiBDXjb (ORCPT ); Fri, 4 Feb 2022 18:39:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378246AbiBDXjG (ORCPT ); Fri, 4 Feb 2022 18:39:06 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F152E025B30; Fri, 4 Feb 2022 15:39:05 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D8FE761CA4; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33D37C340F2; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=tOAzBg36M9bT8CIyEo/qjiTNeDJct5k9/98cm9hBKSY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E9XgVwiAuZoCdSlN9nZ4ZT32iRUIIz0WYLRiBBSgSBJinWtgmEPcXkYqfUO6varJz sxpKOXo6jH0FlFESOelIe+Tqhu7F/75c8AmM98JmyWX8drWtDhim3RaTyxZu7zAeJo 5UXQy1odzJpeRnxytZzDY38+kkgQVSAUJpwwYr5B8z0qS2h4kEV6MDISLignZLyZbJ eDMry1OGsv8oeiC/Z7lL3gDIm2fWVOCkgOK1tC+plFMwxiT82tLvr50Hm+/O+IImy5 KtoXk5cxG9e3f9RJq+T0csjgQV8TRV4c4uslnoR4F1EYDqNNlod46pNotTUH0yRbvz Fr95mT1JeT6Og== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id EFA4B5C0992; Fri, 4 Feb 2022 15:39:03 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 05/19] srcu: Make Tree SRCU able to operate without snp_node array Date: Fri, 4 Feb 2022 15:38:48 -0800 Message-Id: <20220204233902.1902-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit makes Tree SRCU able to operate without an snp_node array, that is, when the srcu_data structures' ->mynode pointers are NULL. This can result in high contention on the srcu_struct structure's ->lock, but only when there are lots of call_srcu(), synchronize_srcu(), and synchronize_srcu_expedited() calls. Note that when there is no snp_node array, all SRCU callbacks use CPU 0's callback queue. This is optimal in the common case of low update-side load because it removes the need to search each CPU for the single callback that made the grace period happen. Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 14 ++- kernel/rcu/srcutree.c | 202 +++++++++++++++++++++------------------ 2 files changed, 123 insertions(+), 93 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index f7190058fb8ab..8501b6b459411 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -63,8 +63,9 @@ struct srcu_struct { struct srcu_node *node; /* Combining tree. */ struct srcu_node *level[RCU_NUM_LVLS + 1]; /* First node at each level. */ + int srcu_size_state; /* Small-to-big transition state. */ struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ - spinlock_t __private lock; /* Protect counters */ + spinlock_t __private lock; /* Protect counters and size state. */ struct mutex srcu_gp_mutex; /* Serialize GP work. */ unsigned int srcu_idx; /* Current rdr array element. */ unsigned long srcu_gp_seq; /* Grace-period seq #. */ @@ -83,6 +84,17 @@ struct srcu_struct { struct lockdep_map dep_map; }; =20 +/* Values for size state variable (->srcu_size_state). */ +#define SRCU_SIZE_SMALL 0 +#define SRCU_SIZE_ALLOC 1 +#define SRCU_SIZE_WAIT_BARRIER 2 +#define SRCU_SIZE_WAIT_CALL 3 +#define SRCU_SIZE_WAIT_CBS1 4 +#define SRCU_SIZE_WAIT_CBS2 5 +#define SRCU_SIZE_WAIT_CBS3 6 +#define SRCU_SIZE_WAIT_CBS4 7 +#define SRCU_SIZE_BIG 8 + /* Values for state variable (bottom bits of ->srcu_gp_seq). */ #define SRCU_STATE_IDLE 0 #define SRCU_STATE_SCAN1 1 diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index eaf00c354a631..2bbe8a5d9ae86 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -177,14 +177,14 @@ static bool init_srcu_struct_nodes(struct srcu_struct= *ssp) } sdp->grpmask =3D 1 << (cpu - sdp->mynode->grplo); } + smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_WAIT_BARRIER); return true; } =20 /* * Initialize non-compile-time initialized fields, including the - * associated srcu_node and srcu_data structures. The is_static - * parameter is passed through to init_srcu_struct_nodes(), and - * also tells us that ->sda has already been wired up to srcu_data. + * associated srcu_node and srcu_data structures. The is_static parameter + * tells us that ->sda has already been wired up to srcu_data. */ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) { @@ -421,6 +421,7 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) ssp->sda =3D NULL; kfree(ssp->node); ssp->node =3D NULL; + ssp->srcu_size_state =3D SRCU_SIZE_SMALL; } EXPORT_SYMBOL_GPL(cleanup_srcu_struct); =20 @@ -469,6 +470,10 @@ static void srcu_gp_start(struct srcu_struct *ssp) struct srcu_data *sdp =3D this_cpu_ptr(ssp->sda); int state; =20 + if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) + sdp =3D per_cpu_ptr(ssp->sda, 0); + else + sdp =3D this_cpu_ptr(ssp->sda); lockdep_assert_held(&ACCESS_PRIVATE(ssp, lock)); WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); spin_lock_rcu_node(sdp); /* Interrupts already disabled. */ @@ -569,38 +574,40 @@ static void srcu_gp_end(struct srcu_struct *ssp) /* A new grace period can start at this point. But only one. */ =20 /* Initiate callback invocation as needed. */ - idx =3D rcu_seq_ctr(gpseq) % ARRAY_SIZE(snp->srcu_have_cbs); - srcu_for_each_node_breadth_first(ssp, snp) { - spin_lock_irq_rcu_node(snp); - cbs =3D false; - last_lvl =3D snp >=3D ssp->level[rcu_num_lvls - 1]; - if (last_lvl) - cbs =3D snp->srcu_have_cbs[idx] =3D=3D gpseq; - snp->srcu_have_cbs[idx] =3D gpseq; - rcu_seq_set_state(&snp->srcu_have_cbs[idx], 1); - if (ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, gpseq)) - WRITE_ONCE(snp->srcu_gp_seq_needed_exp, gpseq); - mask =3D snp->srcu_data_have_cbs[idx]; - snp->srcu_data_have_cbs[idx] =3D 0; - spin_unlock_irq_rcu_node(snp); - if (cbs) - srcu_schedule_cbs_snp(ssp, snp, mask, cbdelay); - - /* Occasionally prevent srcu_data counter wrap. */ - if (!(gpseq & counter_wrap_check) && last_lvl) - for (cpu =3D snp->grplo; cpu <=3D snp->grphi; cpu++) { - sdp =3D per_cpu_ptr(ssp->sda, cpu); - spin_lock_irqsave_rcu_node(sdp, flags); - if (ULONG_CMP_GE(gpseq, - sdp->srcu_gp_seq_needed + 100)) - sdp->srcu_gp_seq_needed =3D gpseq; - if (ULONG_CMP_GE(gpseq, - sdp->srcu_gp_seq_needed_exp + 100)) - sdp->srcu_gp_seq_needed_exp =3D gpseq; - spin_unlock_irqrestore_rcu_node(sdp, flags); - } + if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) { + srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, 0), cbdelay); + } else { + idx =3D rcu_seq_ctr(gpseq) % ARRAY_SIZE(snp->srcu_have_cbs); + srcu_for_each_node_breadth_first(ssp, snp) { + spin_lock_irq_rcu_node(snp); + cbs =3D false; + last_lvl =3D snp >=3D ssp->level[rcu_num_lvls - 1]; + if (last_lvl) + cbs =3D snp->srcu_have_cbs[idx] =3D=3D gpseq; + snp->srcu_have_cbs[idx] =3D gpseq; + rcu_seq_set_state(&snp->srcu_have_cbs[idx], 1); + if (ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, gpseq)) + WRITE_ONCE(snp->srcu_gp_seq_needed_exp, gpseq); + mask =3D snp->srcu_data_have_cbs[idx]; + snp->srcu_data_have_cbs[idx] =3D 0; + spin_unlock_irq_rcu_node(snp); + if (cbs) + srcu_schedule_cbs_snp(ssp, snp, mask, cbdelay); + } } =20 + /* Occasionally prevent srcu_data counter wrap. */ + if (!(gpseq & counter_wrap_check)) + for_each_possible_cpu(cpu) { + sdp =3D per_cpu_ptr(ssp->sda, cpu); + spin_lock_irqsave_rcu_node(sdp, flags); + if (ULONG_CMP_GE(gpseq, sdp->srcu_gp_seq_needed + 100)) + sdp->srcu_gp_seq_needed =3D gpseq; + if (ULONG_CMP_GE(gpseq, sdp->srcu_gp_seq_needed_exp + 100)) + sdp->srcu_gp_seq_needed_exp =3D gpseq; + spin_unlock_irqrestore_rcu_node(sdp, flags); + } + /* Callback initiation done, allow grace periods after next. */ mutex_unlock(&ssp->srcu_cb_mutex); =20 @@ -629,18 +636,19 @@ static void srcu_funnel_exp_start(struct srcu_struct = *ssp, struct srcu_node *snp { unsigned long flags; =20 - for (; snp !=3D NULL; snp =3D snp->srcu_parent) { - if (rcu_seq_done(&ssp->srcu_gp_seq, s) || - ULONG_CMP_GE(READ_ONCE(snp->srcu_gp_seq_needed_exp), s)) - return; - spin_lock_irqsave_rcu_node(snp, flags); - if (ULONG_CMP_GE(snp->srcu_gp_seq_needed_exp, s)) { + if (snp) + for (; snp !=3D NULL; snp =3D snp->srcu_parent) { + if (rcu_seq_done(&ssp->srcu_gp_seq, s) || + ULONG_CMP_GE(READ_ONCE(snp->srcu_gp_seq_needed_exp), s)) + return; + spin_lock_irqsave_rcu_node(snp, flags); + if (ULONG_CMP_GE(snp->srcu_gp_seq_needed_exp, s)) { + spin_unlock_irqrestore_rcu_node(snp, flags); + return; + } + WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); spin_unlock_irqrestore_rcu_node(snp, flags); - return; } - WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); - spin_unlock_irqrestore_rcu_node(snp, flags); - } spin_lock_irqsave_rcu_node(ssp, flags); if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, s)) WRITE_ONCE(ssp->srcu_gp_seq_needed_exp, s); @@ -663,36 +671,37 @@ static void srcu_funnel_gp_start(struct srcu_struct *= ssp, struct srcu_data *sdp, unsigned long flags; int idx =3D rcu_seq_ctr(s) % ARRAY_SIZE(sdp->mynode->srcu_have_cbs); struct srcu_node *snp; - struct srcu_node *snp_leaf =3D sdp->mynode; + struct srcu_node *snp_leaf =3D smp_load_acquire(&sdp->mynode); unsigned long snp_seq; =20 - /* Each pass through the loop does one level of the srcu_node tree. */ - for (snp =3D snp_leaf; snp !=3D NULL; snp =3D snp->srcu_parent) { - if (rcu_seq_done(&ssp->srcu_gp_seq, s) && snp !=3D snp_leaf) - return; /* GP already done and CBs recorded. */ - spin_lock_irqsave_rcu_node(snp, flags); - if (ULONG_CMP_GE(snp->srcu_have_cbs[idx], s)) { - snp_seq =3D snp->srcu_have_cbs[idx]; - if (snp =3D=3D snp_leaf && snp_seq =3D=3D s) - snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; - spin_unlock_irqrestore_rcu_node(snp, flags); - if (snp =3D=3D snp_leaf && snp_seq !=3D s) { - srcu_schedule_cbs_sdp(sdp, do_norm - ? SRCU_INTERVAL - : 0); + if (snp_leaf) + /* Each pass through the loop does one level of the srcu_node tree. */ + for (snp =3D snp_leaf; snp !=3D NULL; snp =3D snp->srcu_parent) { + if (rcu_seq_done(&ssp->srcu_gp_seq, s) && snp !=3D snp_leaf) + return; /* GP already done and CBs recorded. */ + spin_lock_irqsave_rcu_node(snp, flags); + if (ULONG_CMP_GE(snp->srcu_have_cbs[idx], s)) { + snp_seq =3D snp->srcu_have_cbs[idx]; + if (snp =3D=3D snp_leaf && snp_seq =3D=3D s) + snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; + spin_unlock_irqrestore_rcu_node(snp, flags); + if (snp =3D=3D snp_leaf && snp_seq !=3D s) { + srcu_schedule_cbs_sdp(sdp, do_norm + ? SRCU_INTERVAL + : 0); + return; + } + if (!do_norm) + srcu_funnel_exp_start(ssp, snp, s); return; } - if (!do_norm) - srcu_funnel_exp_start(ssp, snp, s); - return; + snp->srcu_have_cbs[idx] =3D s; + if (snp =3D=3D snp_leaf) + snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; + if (!do_norm && ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, s)) + WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); + spin_unlock_irqrestore_rcu_node(snp, flags); } - snp->srcu_have_cbs[idx] =3D s; - if (snp =3D=3D snp_leaf) - snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; - if (!do_norm && ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, s)) - WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); - spin_unlock_irqrestore_rcu_node(snp, flags); - } =20 /* Top of tree, must ensure the grace period will be started. */ spin_lock_irqsave_rcu_node(ssp, flags); @@ -850,7 +859,11 @@ static unsigned long srcu_gp_start_if_needed(struct sr= cu_struct *ssp, =20 check_init_srcu_struct(ssp); idx =3D srcu_read_lock(ssp); - sdp =3D raw_cpu_ptr(ssp->sda); + if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_CALL) { + sdp =3D per_cpu_ptr(ssp->sda, 0); + } else { + sdp =3D raw_cpu_ptr(ssp->sda); + } spin_lock_irqsave_rcu_node(sdp, flags); if (rhp) rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp); @@ -870,7 +883,7 @@ static unsigned long srcu_gp_start_if_needed(struct src= u_struct *ssp, if (needgp) srcu_funnel_gp_start(ssp, sdp, s, do_norm); else if (needexp) - srcu_funnel_exp_start(ssp, sdp->mynode, s); + srcu_funnel_exp_start(ssp, smp_load_acquire(&sdp->mynode), s); srcu_read_unlock(ssp, idx); return s; } @@ -1130,6 +1143,28 @@ static void srcu_barrier_cb(struct rcu_head *rhp) complete(&ssp->srcu_barrier_completion); } =20 +/* + * Enqueue an srcu_barrier() callback on the specified srcu_data + * structure's ->cblist. but only if that ->cblist already has at least o= ne + * callback enqueued. Note that if a CPU already has callbacks enqueue, + * it must have already registered the need for a future grace period, + * so all we need do is enqueue a callback that will use the same grace + * period as the last callback already in the queue. + */ +static void srcu_barrier_one_cpu(struct srcu_struct *ssp, struct srcu_data= *sdp) +{ + spin_lock_irq_rcu_node(sdp); + atomic_inc(&ssp->srcu_barrier_cpu_cnt); + sdp->srcu_barrier_head.func =3D srcu_barrier_cb; + debug_rcu_head_queue(&sdp->srcu_barrier_head); + if (!rcu_segcblist_entrain(&sdp->srcu_cblist, + &sdp->srcu_barrier_head)) { + debug_rcu_head_unqueue(&sdp->srcu_barrier_head); + atomic_dec(&ssp->srcu_barrier_cpu_cnt); + } + spin_unlock_irq_rcu_node(sdp); +} + /** * srcu_barrier - Wait until all in-flight call_srcu() callbacks complete. * @ssp: srcu_struct on which to wait for in-flight callbacks. @@ -1137,7 +1172,6 @@ static void srcu_barrier_cb(struct rcu_head *rhp) void srcu_barrier(struct srcu_struct *ssp) { int cpu; - struct srcu_data *sdp; unsigned long s =3D rcu_seq_snap(&ssp->srcu_barrier_seq); =20 check_init_srcu_struct(ssp); @@ -1153,27 +1187,11 @@ void srcu_barrier(struct srcu_struct *ssp) /* Initial count prevents reaching zero until all CBs are posted. */ atomic_set(&ssp->srcu_barrier_cpu_cnt, 1); =20 - /* - * Each pass through this loop enqueues a callback, but only - * on CPUs already having callbacks enqueued. Note that if - * a CPU already has callbacks enqueue, it must have already - * registered the need for a future grace period, so all we - * need do is enqueue a callback that will use the same - * grace period as the last callback already in the queue. - */ - for_each_possible_cpu(cpu) { - sdp =3D per_cpu_ptr(ssp->sda, cpu); - spin_lock_irq_rcu_node(sdp); - atomic_inc(&ssp->srcu_barrier_cpu_cnt); - sdp->srcu_barrier_head.func =3D srcu_barrier_cb; - debug_rcu_head_queue(&sdp->srcu_barrier_head); - if (!rcu_segcblist_entrain(&sdp->srcu_cblist, - &sdp->srcu_barrier_head)) { - debug_rcu_head_unqueue(&sdp->srcu_barrier_head); - atomic_dec(&ssp->srcu_barrier_cpu_cnt); - } - spin_unlock_irq_rcu_node(sdp); - } + if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) + srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, 0)); + else + for_each_possible_cpu(cpu) + srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, cpu)); =20 /* Remove the initial count, at which point reaching zero can happen. */ if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt)) --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DE4FC43217 for ; Fri, 4 Feb 2022 23:39:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378327AbiBDXjR (ORCPT ); Fri, 4 Feb 2022 18:39:17 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:47876 "EHLO dfw.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378227AbiBDXjF (ORCPT ); Fri, 4 Feb 2022 18:39:05 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4363861CA5; Fri, 4 Feb 2022 23:39:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5C816C340F7; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=obmVj21+RsUzmU/ireCGRzvyfnAIEu2kyxPtUNQcgrE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fiafyUABsd2JMhYSjdMT4N3MAImrf2mwmsXwQ277A6JkUM+N3rIBoI0tUj59mbncL ycf+7vfcNr2yFLNKocKX8q9loV2Vibgl/UuXPP9P1aVaO8wEL/37zDoJbENm/j1eqE rjgMfFI6EerYy6slvhcyajTUgb2IyVPRDoV4KOw/FE4k84tPqILnNizJuTBwDxEqEn lguY+eNABrLSG0jmLgqjDbYiz6sXg2QeI50IESgE9DZP788EFx8DNJoBO507HqXvxK AP0BguYIaxseZpKs+FBtj6YSuzU9AadMxvxakGd48x36YUrqp5y1vA0Z3qWhnfgUbM tXUWbXKCO2juw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id F1BF35C0A0A; Fri, 4 Feb 2022 15:39:03 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 06/19] srcu: Add size-state transitioning code Date: Fri, 4 Feb 2022 15:38:49 -0800 Message-Id: <20220204233902.1902-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This is just dead code at the moment, but it serves to prevent spurious compiler warnings about init_srcu_struct_nodes() being unused. This function will once again be used once the state-transition code is activated. Because srcu_barrier() must be aware of transition before call_srcu(), the state machine waits for an SRCU grace period before callbacks are queued to the non-CPU-0 queues. This requres that portions of srcu_barrier() be enclosed in an SRCU read-side critical section. Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 2bbe8a5d9ae86..fce041b648ee3 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -202,7 +202,6 @@ static int init_srcu_struct_fields(struct srcu_struct *= ssp, bool is_static) if (!ssp->sda) return -ENOMEM; init_srcu_struct_data(ssp); - WARN_ON_ONCE(!init_srcu_struct_nodes(ssp)); ssp->srcu_gp_seq_needed_exp =3D 0; ssp->srcu_last_gp_end =3D ktime_get_mono_fast_ns(); smp_store_release(&ssp->srcu_gp_seq_needed, 0); /* Init done. */ @@ -555,6 +554,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) unsigned long mask; struct srcu_data *sdp; struct srcu_node *snp; + int ss_state; =20 /* Prevent more than one additional grace period. */ mutex_lock(&ssp->srcu_cb_mutex); @@ -622,6 +622,15 @@ static void srcu_gp_end(struct srcu_struct *ssp) } else { spin_unlock_irq_rcu_node(ssp); } + + /* Transition to big if needed. */ + ss_state =3D smp_load_acquire(&ssp->srcu_size_state); + if (ss_state && ss_state !=3D SRCU_SIZE_BIG) { + if (ss_state =3D=3D SRCU_SIZE_ALLOC) + init_srcu_struct_nodes(ssp); + else + smp_store_release(&ssp->srcu_size_state, ss_state + 1); + } } =20 /* @@ -1172,6 +1181,7 @@ static void srcu_barrier_one_cpu(struct srcu_struct *= ssp, struct srcu_data *sdp) void srcu_barrier(struct srcu_struct *ssp) { int cpu; + int idx; unsigned long s =3D rcu_seq_snap(&ssp->srcu_barrier_seq); =20 check_init_srcu_struct(ssp); @@ -1187,11 +1197,13 @@ void srcu_barrier(struct srcu_struct *ssp) /* Initial count prevents reaching zero until all CBs are posted. */ atomic_set(&ssp->srcu_barrier_cpu_cnt, 1); =20 + idx =3D srcu_read_lock(ssp); if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, 0)); else for_each_possible_cpu(cpu) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, cpu)); + srcu_read_unlock(ssp, idx); =20 /* Remove the initial count, at which point reaching zero can happen. */ if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt)) --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01F63C433EF for ; Fri, 4 Feb 2022 23:39:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378410AbiBDXj3 (ORCPT ); Fri, 4 Feb 2022 18:39:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378247AbiBDXjG (ORCPT ); Fri, 4 Feb 2022 18:39:06 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03D0CE025B34; Fri, 4 Feb 2022 15:39:06 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 47E0761CAB; Fri, 4 Feb 2022 23:39:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5DD1CC340F9; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=vMfcdUf1wVddckJ+Hs5AKckEdG/I81gYVH5ZKNn9rPc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Du9GrBbcTixFYdS8ZTNv8eGpml/GYOZkQ78syOhoE+k9dZGyoiOH4fZLIVD/sKd2y Z7T7MJ1ZgLfxiPtZF41d2TwVGNYnJDJ3Oc7X7ZHs39cIfFGlksGXJDmQM61AZazQ9e SkHJrLBHoZ44LXSahR/vlco0K3WgQ22ANDyi1/31Gr0y+6eUI29PQgctO09im7neTg tgXj1DkuXVtEs5cgjyoUMcbSXytKwbBFS/YmkaUa9Yf6ZjT0PL60Q2szsK/OSnqZhW vYJiIFb3hxvTyAprfcgBG6TUNEmcAVcj+tZ23L+7DcptS1Sr3hDQDPXlsObZzMWJdI 59dccLjtUrIVg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id F37795C0A21; Fri, 4 Feb 2022 15:39:03 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 07/19] srcu: Make rcutorture dump the SRCU size state Date: Fri, 4 Feb 2022 15:38:50 -0800 Message-Id: <20220204233902.1902-7-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit adds the numeric and string version of ->srcu_size_state to the Tree-SRCU-specific portion of the rcutorture output. [ paulmck: Apply feedback from kernel test robot and Dan Carpenter. ] Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index fce041b648ee3..550991cc213d3 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1400,15 +1400,34 @@ void srcutorture_get_gp_data(enum rcutorture_type t= est_type, } EXPORT_SYMBOL_GPL(srcutorture_get_gp_data); =20 +const char * const srcu_size_state_name[] =3D +{ + "SRCU_SIZE_SMALL", + "SRCU_SIZE_ALLOC", + "SRCU_SIZE_WAIT_BARRIER", + "SRCU_SIZE_WAIT_CALL", + "SRCU_SIZE_WAIT_CBS1", + "SRCU_SIZE_WAIT_CBS2", + "SRCU_SIZE_WAIT_CBS3", + "SRCU_SIZE_WAIT_CBS4", + "SRCU_SIZE_BIG", + "SRCU_SIZE_???", +}; + void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) { int cpu; int idx; unsigned long s0 =3D 0, s1 =3D 0; + int ss_state =3D READ_ONCE(ssp->srcu_size_state); + int ss_state_idx =3D ss_state; =20 idx =3D ssp->srcu_idx & 0x1; - pr_alert("%s%s Tree SRCU g%ld per-CPU(idx=3D%d):", - tt, tf, rcu_seq_current(&ssp->srcu_gp_seq), idx); + if (ss_state < 0 || ss_state >=3D ARRAY_SIZE(srcu_size_state_name)) + ss_state_idx =3D ARRAY_SIZE(srcu_size_state_name) - 1; + pr_alert("%s%s Tree SRCU g%ld state %d (%s) per-CPU(idx=3D%d):", + tt, tf, rcu_seq_current(&ssp->srcu_gp_seq), ss_state, + srcu_size_state_name[ss_state_idx], idx); for_each_possible_cpu(cpu) { unsigned long l0, l1; unsigned long u0, u1; --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53C9BC433EF for ; Fri, 4 Feb 2022 23:39:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378494AbiBDXjk (ORCPT ); Fri, 4 Feb 2022 18:39:40 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:39388 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378244AbiBDXjH (ORCPT ); Fri, 4 Feb 2022 18:39:07 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E157FB83975; Fri, 4 Feb 2022 23:39:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 613EBC340FA; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=nCSQipMnQYydZIPhraT+nyb+OMvwqOwj8Ix07rY+kdY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=V4kMvhyl60zgrjHY7bYaZtCXznXyENURYnxdfo4Er+bbyIqpdDzOkXxfeI4JpwQPW KJl4ENf/51ixYVgXIZTplwD/9BHycOcQyaDOuC5dB/4XNsiTZQ+WI7Oaper4Txa5IM k3ey1mAC8ScItixP4B8b2+qzEHeFt7bUD+Go3mycKbZFIm3UFhmdIzJbeBy+JvIdvr YkXLYLP9TcXFOyBg/rXxLyisuEuIh5Bne1e9EbUYDhwnxwX3Ms3lhnmeoYGFA3/Kli Mrro8Ve7XfLNlHo8R1t5LuzxPuPh9trnzqVq2v+ZAXCsS2Z1H92A+J6o3a4buiforZ ivT/3RDF/96FQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0166B5C0A6A; Fri, 4 Feb 2022 15:39:03 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 08/19] srcu: Add boot-time control over srcu_node array allocation Date: Fri, 4 Feb 2022 15:38:51 -0800 Message-Id: <20220204233902.1902-8-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit adds an srcu_tree.convert_to_big kernel parameter that either refuses to convert at all (0), converts immediately at init_srcu_struct() time (1), or lets rcutorture convert it (2). An addition contention-based dynamic convertion choice will be added, along with documentation. Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 550991cc213d3..d1f1ff930acf5 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -39,6 +39,15 @@ module_param(exp_holdoff, ulong, 0444); static ulong counter_wrap_check =3D (ULONG_MAX >> 2); module_param(counter_wrap_check, ulong, 0444); =20 +/* + * Control conversion to SRCU_SIZE_BIG: + * 0: Don't convert at all (default). + * 1: Convert at init_srcu_struct() time. + * 2: Convert when rcutorture invokes srcu_torture_stats_print(). + */ +static int convert_to_big; +module_param(convert_to_big, int, 0444); + /* Early-boot callback-management, so early that no lock is required! */ static LIST_HEAD(srcu_boot_list); static bool __read_mostly srcu_init_done; @@ -113,7 +122,7 @@ static void init_srcu_struct_data(struct srcu_struct *s= sp) * Allocated and initialize SRCU combining tree. Returns @true if * allocation succeeded and @false otherwise. */ -static bool init_srcu_struct_nodes(struct srcu_struct *ssp) +static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flag= s) { int cpu; int i; @@ -125,7 +134,7 @@ static bool init_srcu_struct_nodes(struct srcu_struct *= ssp) =20 /* Initialize geometry if it has not already been initialized. */ rcu_init_geometry(); - ssp->node =3D kcalloc(rcu_num_nodes, sizeof(*ssp->node), GFP_ATOMIC); + ssp->node =3D kcalloc(rcu_num_nodes, sizeof(*ssp->node), gfp_flags); if (!ssp->node) return false; =20 @@ -204,6 +213,16 @@ static int init_srcu_struct_fields(struct srcu_struct = *ssp, bool is_static) init_srcu_struct_data(ssp); ssp->srcu_gp_seq_needed_exp =3D 0; ssp->srcu_last_gp_end =3D ktime_get_mono_fast_ns(); + if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 1) { + if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { + if (!is_static) { + free_percpu(ssp->sda); + ssp->sda =3D NULL; + } + return -ENOMEM; + } + WRITE_ONCE(ssp->srcu_size_state, SRCU_SIZE_BIG); + } smp_store_release(&ssp->srcu_gp_seq_needed, 0); /* Init done. */ return 0; } @@ -627,7 +646,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) ss_state =3D smp_load_acquire(&ssp->srcu_size_state); if (ss_state && ss_state !=3D SRCU_SIZE_BIG) { if (ss_state =3D=3D SRCU_SIZE_ALLOC) - init_srcu_struct_nodes(ssp); + init_srcu_struct_nodes(ssp, GFP_KERNEL); else smp_store_release(&ssp->srcu_size_state, ss_state + 1); } @@ -1456,6 +1475,8 @@ void srcu_torture_stats_print(struct srcu_struct *ssp= , char *tt, char *tf) s1 +=3D c1; } pr_cont(" T(%ld,%ld)\n", s0, s1); + if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 2) + WRITE_ONCE(ssp->srcu_size_state, SRCU_SIZE_ALLOC); } EXPORT_SYMBOL_GPL(srcu_torture_stats_print); =20 --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 489DCC433EF for ; Fri, 4 Feb 2022 23:39:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378334AbiBDXjd (ORCPT ); Fri, 4 Feb 2022 18:39:33 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:47886 "EHLO dfw.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378233AbiBDXjF (ORCPT ); Fri, 4 Feb 2022 18:39:05 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 51AB261CAC; Fri, 4 Feb 2022 23:39:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 64BA7C340FB; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=mqSCacukpi9DpKBNP6/4XacTjUfOTHmgmq5c2Vhl5ZY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HBmISx4QN//uuEdE2VRV5qNtIRRsOl3IlYaNRvJs1cxaJkRfLBlr/4TvnHTxHxu89 O+fKSxbW2cmNpCg2irPWVfgMOwgwhwzSVzNXF4BJ1KlZ9TswEAfqAAmKSmLjYLdRDM HDjh+gf7S8oADOw0MZEYePQn5ExQPOGdocKzjfY2AutOlXNfsAjZStwpVCpmHs1aKh Gtgi5wySQCAkxu2jPF1p/Mj4L8Wn8KApm1NrMICOoSSWxisMl+ZhOuWl6uIFUQ8O8d Mzc+ZkWK6OlrztM1PJJ0vTwJuhWZ/l1p5MDxvzysWNBMEp0QtMb66MkGC+efyad1c/ yA5qZCa4m4n5A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 032305C0AEF; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Alexander Aring , "Paul E . McKenney" Subject: [PATCH rcu 09/19] srcu: Use export for srcu_struct defined by DEFINE_STATIC_SRCU() Date: Fri, 4 Feb 2022 15:38:52 -0800 Message-Id: <20220204233902.1902-9-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alexander Aring If an srcu_struct structure defined by tree SRCU's DEFINE_STATIC_SRCU() is used by a module, sparse will give the following diagnostic: sparse: symbol '__srcu_struct_nodes_srcu' was not declared. Should it be st= atic? The problem is that a within-module DEFINE_STATIC_SRCU() must define a non-static srcu_struct because it is exported by referencing it in a special '__section("___srcu_struct_ptrs")'. This reference is needed so that module load and unloading can invoke init_srcu_struct() and cleanup_srcu_struct(), respectively. Unfortunately, sparse is unaware of '__section("___srcu_struct_ptrs")', resulting in the above false-positive diagnostic. To avoid this false positive, this commit therefore creates a prototype of the srcu_struct with an "extern" keyword. Signed-off-by: Alexander Aring Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 8501b6b459411..44e998643f483 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -131,6 +131,7 @@ struct srcu_struct { #ifdef MODULE # define __DEFINE_SRCU(name, is_static) \ is_static struct srcu_struct name; \ + extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ __section("___srcu_struct_ptrs") =3D &name #else --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92EACC433F5 for ; Fri, 4 Feb 2022 23:39:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378346AbiBDXjV (ORCPT ); Fri, 4 Feb 2022 18:39:21 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:47898 "EHLO dfw.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378234AbiBDXjG (ORCPT ); Fri, 4 Feb 2022 18:39:06 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5B4BF61CAA; Fri, 4 Feb 2022 23:39:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 74553C340F8; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=pFRvJEGkCMqV8E4Iw4z3sZVDAdjH8/YPUIbC9N1Jb60=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kEh5G3viP0ymeb00w5uxfJRn8WqEk0daLHPUYdN3xB1um/Aq+jnPjLew2Ll1aRHUa wpIiHFWKCR/25DdfD9x6BCmW+aA5qlmrbW2e0PIH0jI6PVqmQTANgM1va1Zmw8NWEA ZTkBOA4aXCTBI2BQv11ggzw/7Pi063/NceP+F8tfDqM9Ov6PTLKSE1bZjL3VGQnHVP HziYI69YkL6oAPB1w6/wBQF371seS0BiklxpnZMjVU3nP0NBE4KvxqRtjlkU8I71So jF65D0ZRbCY48rE4F7ERPhw2X/DhVDnr18uauGBxnxlQMJyFiHPLlpJA3i62l72g9j XLELH21C9vg0w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0511B5C0BCB; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 10/19] srcu: Compute snp_seq earlier in srcu_funnel_gp_start() Date: Fri, 4 Feb 2022 15:38:53 -0800 Message-Id: <20220204233902.1902-10-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, srcu_funnel_gp_start() tests snp->srcu_have_cbs[idx] and then separately assigns it to the snp_seq local variable. This commit does the assignment earlier to simplify the code a bit. While in the area, this commit also takes advantage of the 100-character line limit to put the call to srcu_schedule_cbs_sdp() on a single line. Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index d1f1ff930acf5..8f55967b58a74 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -708,15 +708,13 @@ static void srcu_funnel_gp_start(struct srcu_struct *= ssp, struct srcu_data *sdp, if (rcu_seq_done(&ssp->srcu_gp_seq, s) && snp !=3D snp_leaf) return; /* GP already done and CBs recorded. */ spin_lock_irqsave_rcu_node(snp, flags); - if (ULONG_CMP_GE(snp->srcu_have_cbs[idx], s)) { - snp_seq =3D snp->srcu_have_cbs[idx]; + snp_seq =3D snp->srcu_have_cbs[idx]; + if (ULONG_CMP_GE(snp_seq, s)) { if (snp =3D=3D snp_leaf && snp_seq =3D=3D s) snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; spin_unlock_irqrestore_rcu_node(snp, flags); if (snp =3D=3D snp_leaf && snp_seq !=3D s) { - srcu_schedule_cbs_sdp(sdp, do_norm - ? SRCU_INTERVAL - : 0); + srcu_schedule_cbs_sdp(sdp, do_norm ? SRCU_INTERVAL : 0); return; } if (!do_norm) --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A136AC433EF for ; Fri, 4 Feb 2022 23:39:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378538AbiBDXjt (ORCPT ); Fri, 4 Feb 2022 18:39:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378223AbiBDXjI (ORCPT ); Fri, 4 Feb 2022 18:39:08 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 79936E025B3B; Fri, 4 Feb 2022 15:39:07 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 20F62B83977; Fri, 4 Feb 2022 23:39:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7566EC36AE5; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=lU9KtK+rLNOMgb4Dp6EanwjYkdwCh9D21wVszXVcZ0g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G9vQOuQV/hPVAKCM+QM4Ue/aNi2hZxtOs+ra0ra8/8FiUR2TlvVcno4pn9WO6RkUD 98gWGGVtL972aeGl9/WNG+tAxwFP3uXwPeIRiIt0Lk7pLZaR/j5yhQ4AncpTOEEvro I9+ke7F/mrPRu9QO1Ek1MR00OYtKnhlSFOPWMrKUZcVn/hz2eeOunpDqaAsQf5YrUo HFq8CBDQgOwQGPSkMeDhck6ur5ffjYpj3yVfBs6tWpmYlRf6/Xo3a9pqVFpeJWs1xk uezZxUwoHE149z1Sh8yvubgGIPcvXGms528GEiZVTByY34hO0M6n3e9lM9FhXviTxz DaElMQ2ChKvqQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 06DE05C0DB8; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 11/19] srcu: Use invalid initial value for srcu_node GP sequence numbers Date: Fri, 4 Feb 2022 15:38:54 -0800 Message-Id: <20220204233902.1902-11-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, tree SRCU relies on the srcu_node structures being initialized at the same time that the srcu_struct itself is initialized, and thus use the initial grace-period sequence number as the initial value for the srcu_node structure's ->srcu_have_cbs[] and ->srcu_gp_seq_needed_exp fields. Although this has a high probability of also working when the srcu_node array is allocated and initialized at some random later time, it would be better to avoid leaving such things to chance. This commit therefore initializes these fields with 0x1, which is a recognizable invalid value. It then adds the required checks for this invalid value in order to avoid confusion on long-running kernels (especially those on 32-bit systems) that allocate and initialize srcu_node arrays late in life. Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 8f55967b58a74..4201815744d85 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -150,10 +150,10 @@ static bool init_srcu_struct_nodes(struct srcu_struct= *ssp, gfp_t gfp_flags) WARN_ON_ONCE(ARRAY_SIZE(snp->srcu_have_cbs) !=3D ARRAY_SIZE(snp->srcu_data_have_cbs)); for (i =3D 0; i < ARRAY_SIZE(snp->srcu_have_cbs); i++) { - snp->srcu_have_cbs[i] =3D 0; + snp->srcu_have_cbs[i] =3D 0x1; snp->srcu_data_have_cbs[i] =3D 0; } - snp->srcu_gp_seq_needed_exp =3D 0; + snp->srcu_gp_seq_needed_exp =3D 0x1; snp->grplo =3D -1; snp->grphi =3D -1; if (snp =3D=3D &ssp->node[0]) { @@ -397,8 +397,7 @@ static bool srcu_readers_active(struct srcu_struct *ssp) */ static unsigned long srcu_get_delay(struct srcu_struct *ssp) { - if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_gp_seq), - READ_ONCE(ssp->srcu_gp_seq_needed_exp))) + if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_gp_seq), READ_ONCE(ssp->srcu_gp_seq_= needed_exp))) return 0; return SRCU_INTERVAL; } @@ -572,6 +571,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) int idx; unsigned long mask; struct srcu_data *sdp; + unsigned long sgsne; struct srcu_node *snp; int ss_state; =20 @@ -605,7 +605,8 @@ static void srcu_gp_end(struct srcu_struct *ssp) cbs =3D snp->srcu_have_cbs[idx] =3D=3D gpseq; snp->srcu_have_cbs[idx] =3D gpseq; rcu_seq_set_state(&snp->srcu_have_cbs[idx], 1); - if (ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, gpseq)) + sgsne =3D snp->srcu_gp_seq_needed_exp; + if (rcu_seq_state(sgsne) || ULONG_CMP_LT(sgsne, gpseq)) WRITE_ONCE(snp->srcu_gp_seq_needed_exp, gpseq); mask =3D snp->srcu_data_have_cbs[idx]; snp->srcu_data_have_cbs[idx] =3D 0; @@ -663,14 +664,17 @@ static void srcu_funnel_exp_start(struct srcu_struct = *ssp, struct srcu_node *snp unsigned long s) { unsigned long flags; + unsigned long sgsne; =20 if (snp) for (; snp !=3D NULL; snp =3D snp->srcu_parent) { + sgsne =3D READ_ONCE(snp->srcu_gp_seq_needed_exp); if (rcu_seq_done(&ssp->srcu_gp_seq, s) || - ULONG_CMP_GE(READ_ONCE(snp->srcu_gp_seq_needed_exp), s)) + (!rcu_seq_state(sgsne) && ULONG_CMP_GE(sgsne, s))) return; spin_lock_irqsave_rcu_node(snp, flags); - if (ULONG_CMP_GE(snp->srcu_gp_seq_needed_exp, s)) { + sgsne =3D snp->srcu_gp_seq_needed_exp; + if (!rcu_seq_state(sgsne) && ULONG_CMP_GE(sgsne, s)) { spin_unlock_irqrestore_rcu_node(snp, flags); return; } @@ -698,6 +702,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ss= p, struct srcu_data *sdp, { unsigned long flags; int idx =3D rcu_seq_ctr(s) % ARRAY_SIZE(sdp->mynode->srcu_have_cbs); + unsigned long sgsne; struct srcu_node *snp; struct srcu_node *snp_leaf =3D smp_load_acquire(&sdp->mynode); unsigned long snp_seq; @@ -724,7 +729,8 @@ static void srcu_funnel_gp_start(struct srcu_struct *ss= p, struct srcu_data *sdp, snp->srcu_have_cbs[idx] =3D s; if (snp =3D=3D snp_leaf) snp->srcu_data_have_cbs[idx] |=3D sdp->grpmask; - if (!do_norm && ULONG_CMP_LT(snp->srcu_gp_seq_needed_exp, s)) + sgsne =3D snp->srcu_gp_seq_needed_exp; + if (!do_norm && (rcu_seq_state(sgsne) || ULONG_CMP_LT(sgsne, s))) WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); spin_unlock_irqrestore_rcu_node(snp, flags); } --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B359C433EF for ; Fri, 4 Feb 2022 23:39:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378560AbiBDXjw (ORCPT ); Fri, 4 Feb 2022 18:39:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378269AbiBDXjI (ORCPT ); Fri, 4 Feb 2022 18:39:08 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B651E025B34; Fri, 4 Feb 2022 15:39:07 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 14F00B83961; Fri, 4 Feb 2022 23:39:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7763CC340FF; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=Q0Hs9+xdt5kC0I4NW3DFLJXE6gilkL4WWu77Q0NgULo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qPQYS/2yHEy5lqMaWQ3v73ycqTdQ4l/7422evQyMBa8ez3GmlR4LdHOjXQluVzmyj cCMn44aX+64HG1a9xsi3wJvmFYpSnBzwcxKdGtDwF2prNpcjXkQebXgyAOR4UCCNGj BER/+BePZGePl5csnYBc7HpIZzKNNsEkgQ7Dpp5kvLi3pNiFqe/wyK7h8D0zlLfQZg HWI5BdbxjItvCoD0H75Zr/fR+d6hn0XAsx17dLpBjdXCwesud0L9VXbaP0etrdDAGu ulMyr8AajqQv6s0CoOJDjLHuONTz/OyjEf3sRwHxhfWOjnAZ2TVrkAmhINXdJnwJiI VRQ9P6PzWm6bA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 08DAA5C0E5E; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 12/19] srcu: Avoid NULL dereference in srcu_torture_stats_print() Date: Fri, 4 Feb 2022 15:38:55 -0800 Message-Id: <20220204233902.1902-12-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" You really shouldn't invoke srcu_torture_stats_print() after invoking cleanup_srcu_struct(), but there is really no reason to get a compiler-obfuscated per-CPU-variable NULL pointer dereference as the diagnostic. This commit therefore checks for NULL ->sda and makes a more polite console-message complaint in that case. Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 62 ++++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 28 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 4201815744d85..767487ad5440a 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1448,37 +1448,43 @@ void srcu_torture_stats_print(struct srcu_struct *s= sp, char *tt, char *tf) idx =3D ssp->srcu_idx & 0x1; if (ss_state < 0 || ss_state >=3D ARRAY_SIZE(srcu_size_state_name)) ss_state_idx =3D ARRAY_SIZE(srcu_size_state_name) - 1; - pr_alert("%s%s Tree SRCU g%ld state %d (%s) per-CPU(idx=3D%d):", + pr_alert("%s%s Tree SRCU g%ld state %d (%s)", tt, tf, rcu_seq_current(&ssp->srcu_gp_seq), ss_state, - srcu_size_state_name[ss_state_idx], idx); - for_each_possible_cpu(cpu) { - unsigned long l0, l1; - unsigned long u0, u1; - long c0, c1; - struct srcu_data *sdp; - - sdp =3D per_cpu_ptr(ssp->sda, cpu); - u0 =3D data_race(sdp->srcu_unlock_count[!idx]); - u1 =3D data_race(sdp->srcu_unlock_count[idx]); + srcu_size_state_name[ss_state_idx]); + if (!ssp->sda) { + // Called after cleanup_srcu_struct(), perhaps. + pr_cont(" No per-CPU srcu_data structures (->sda =3D=3D NULL).\n"); + } else { + pr_cont(" per-CPU(idx=3D%d):", idx); + for_each_possible_cpu(cpu) { + unsigned long l0, l1; + unsigned long u0, u1; + long c0, c1; + struct srcu_data *sdp; =20 - /* - * Make sure that a lock is always counted if the corresponding - * unlock is counted. - */ - smp_rmb(); - - l0 =3D data_race(sdp->srcu_lock_count[!idx]); - l1 =3D data_race(sdp->srcu_lock_count[idx]); - - c0 =3D l0 - u0; - c1 =3D l1 - u1; - pr_cont(" %d(%ld,%ld %c)", - cpu, c0, c1, - "C."[rcu_segcblist_empty(&sdp->srcu_cblist)]); - s0 +=3D c0; - s1 +=3D c1; + sdp =3D per_cpu_ptr(ssp->sda, cpu); + u0 =3D data_race(sdp->srcu_unlock_count[!idx]); + u1 =3D data_race(sdp->srcu_unlock_count[idx]); + + /* + * Make sure that a lock is always counted if the corresponding + * unlock is counted. + */ + smp_rmb(); + + l0 =3D data_race(sdp->srcu_lock_count[!idx]); + l1 =3D data_race(sdp->srcu_lock_count[idx]); + + c0 =3D l0 - u0; + c1 =3D l1 - u1; + pr_cont(" %d(%ld,%ld %c)", + cpu, c0, c1, + "C."[rcu_segcblist_empty(&sdp->srcu_cblist)]); + s0 +=3D c0; + s1 +=3D c1; + } + pr_cont(" T(%ld,%ld)\n", s0, s1); } - pr_cont(" T(%ld,%ld)\n", s0, s1); if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 2) WRITE_ONCE(ssp->srcu_size_state, SRCU_SIZE_ALLOC); } --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA46AC433F5 for ; Fri, 4 Feb 2022 23:39:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378375AbiBDXjZ (ORCPT ); Fri, 4 Feb 2022 18:39:25 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:47904 "EHLO dfw.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378241AbiBDXjG (ORCPT ); Fri, 4 Feb 2022 18:39:06 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6F7F661CAE; Fri, 4 Feb 2022 23:39:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8A862C34101; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=N5jnqfLu0rIDFCwoYEN18JsUpJw3bRQcJZW9jv7EzbM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Mx1x6Z3tsckbbkDcQLaxbGSdko9wjpCJpgpqS3fUnp3WDOwvzTGqFmCDQfDXfvosh 00zS8VtHbzV5TU/QPm/2Agnevzv9x4E6B0Et9IPIONnV387Tdmq2ZFbU6dKZyyuyid wmwa0df42hHC73YchWBFUH65y5IKVDLfgbAQOeDTtCOsQ2uMYq1mJFXE6DutRIzDmq 8hwohOugZIk7uXKEzeDcPOrFOlyYyUg69sSNXHe6uoWIWFUySH7ibuVJE7j3/eU5Fp 1hIQ/mFaaxzSxLE56qPamJXJot1A7/71y367TkwseQXWfZQMrn9Wl9I7Ow9tzwdyOi AU0INPor/pZVA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0A8F85C0F91; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 13/19] srcu: Prevent cleanup_srcu_struct() from freeing non-dynamic ->sda Date: Fri, 4 Feb 2022 15:38:56 -0800 Message-Id: <20220204233902.1902-13-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When an srcu_struct structure is created (but not in a kernel module) by DEFINE_SRCU() and friends, the per-CPU srcu_data structure is statically allocated. In all other cases, that structure is obtained from alloc_percpu(), in which case cleanup_srcu_struct() must invoke free_percpu() on the resulting ->sda pointer in the srcu_struct pointer. Which it does. Except that it also invokes free_percpu() on the ->sda pointer referencing the statically allocated per-CPU srcu_data structures. Which free_percpu() is surprisingly OK with. This commit nevertheless stops cleanup_srcu_struct() from freeing statically allocated per-CPU srcu_data structures. Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 1 + kernel/rcu/srcutree.c | 13 +++++++++---- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 44e998643f483..44bd204498a11 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -73,6 +73,7 @@ struct srcu_struct { unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */ unsigned long srcu_last_gp_end; /* Last GP end timestamp (ns) */ struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ + bool sda_is_static; /* May ->sda be passed to free_percpu()? */ unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ struct mutex srcu_barrier_mutex; /* Serialize barrier ops. */ struct completion srcu_barrier_completion; diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 767487ad5440a..39dc3015dfeba 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -206,8 +206,11 @@ static int init_srcu_struct_fields(struct srcu_struct = *ssp, bool is_static) mutex_init(&ssp->srcu_barrier_mutex); atomic_set(&ssp->srcu_barrier_cpu_cnt, 0); INIT_DELAYED_WORK(&ssp->work, process_srcu); - if (!is_static) + ssp->sda_is_static =3D false; + if (!is_static) { ssp->sda =3D alloc_percpu(struct srcu_data); + ssp->sda_is_static =3D true; + } if (!ssp->sda) return -ENOMEM; init_srcu_struct_data(ssp); @@ -215,7 +218,7 @@ static int init_srcu_struct_fields(struct srcu_struct *= ssp, bool is_static) ssp->srcu_last_gp_end =3D ktime_get_mono_fast_ns(); if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 1) { if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { - if (!is_static) { + if (ssp->sda_is_static) { free_percpu(ssp->sda); ssp->sda =3D NULL; } @@ -434,8 +437,10 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) rcu_seq_current(&ssp->srcu_gp_seq), ssp->srcu_gp_seq_needed); return; /* Caller forgot to stop doing call_srcu()? */ } - free_percpu(ssp->sda); - ssp->sda =3D NULL; + if (ssp->sda_is_static) { + free_percpu(ssp->sda); + ssp->sda =3D NULL; + } kfree(ssp->node); ssp->node =3D NULL; ssp->srcu_size_state =3D SRCU_SIZE_SMALL; --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5931C433EF for ; Fri, 4 Feb 2022 23:39:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378534AbiBDXjq (ORCPT ); Fri, 4 Feb 2022 18:39:46 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:39394 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378252AbiBDXjH (ORCPT ); Fri, 4 Feb 2022 18:39:07 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 25D41B83978; Fri, 4 Feb 2022 23:39:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7F668C340FE; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=YN9frrNfF3TNUCGWsBeNYaG+rDwqFVcfd14+/Fo20Oc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PjK+j/73QG6IlGAiD5JlVVRmH6wiDABPzi4FaSPGtxoTDAclRBs7ZcklPNAFHfV+6 OM0HoFpgK8/MZtbXYBytOM1pgCniSPJGvVX0ZPNhK+CqCwZpG3FccpYesNhuhyQNY9 RsXmvKCE1wALd8AvBTu18ExVHxaQXH33Sf1FQRtNRY4gO2JLZ4lsKE4dWdDzaFkE/s Wa7WixFfyWFK6co9bcZ8C4qZaeEGLlhXkXVnCPKFR73T9SsC7tpwwxwu+T2G+9mIHh LRN82d8P/7fwwmjw3UYst6MxbtNGPe8Qd2zzXMXgPPkQvpscehT/rmteaCMLYvX2ag 848KAyZG8nVGg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0C9B75C0FCA; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 14/19] srcu: Explain srcu_funnel_gp_start() call to list_add() is safe Date: Fri, 4 Feb 2022 15:38:57 -0800 Message-Id: <20220204233902.1902-14-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit adds a comment explaining why an unprotected call to list_add() from srcu_funnel_gp_start() can be safe. TL;DR: It is only called during very early boot when we don't have no steeking concurrency! Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 39dc3015dfeba..d16104405c8f9 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -757,6 +757,12 @@ static void srcu_funnel_gp_start(struct srcu_struct *s= sp, struct srcu_data *sdp, rcu_seq_state(ssp->srcu_gp_seq) =3D=3D SRCU_STATE_IDLE) { WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); srcu_gp_start(ssp); + + // And how can that list_add() in the "else" clause + // possibly be safe for concurrent execution? Well, + // it isn't. And it does not have to be. After all, it + // can only be executed during early boot when there is only + // the one boot CPU running with interrupts still disabled. if (likely(srcu_init_done)) queue_delayed_work(rcu_gp_wq, &ssp->work, srcu_get_delay(ssp)); --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AE48C433F5 for ; Fri, 4 Feb 2022 23:39:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378582AbiBDXj4 (ORCPT ); Fri, 4 Feb 2022 18:39:56 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:39422 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378267AbiBDXjI (ORCPT ); Fri, 4 Feb 2022 18:39:08 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 929A3B83976; Fri, 4 Feb 2022 23:39:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8CFF6C34104; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=B3ldLUHX3CEZ1V3crg0anclvyymU/4MkEs5XFCophWg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sqWN18zcs/lhFJxDxrLLN0UUifOkhaNMImXVXjJDP5G6zvNlMwgMAN9qrYoSdfdu8 xQlCTInpa6eYhzEKPi0f3lx0OrRRoKisPEU8eT6akFIDeTiEU/o7nVhO9FZm1AVbCv W/+Jg9FhPlALnAiyktKBHlEOK8gCx3OEi/F8GQczqQ5iHNcRsz4gtTwYY6Iv47tWXQ jzYTySAQ7qOpEb+YcFF0R1zD4y13VhoEF7O+3CUTLioQr6kSc2nA5NB+S68eLIrJIY ddg/veNR55yb4+DKy5f6pDBC7G6lsPem0DsPzYsI2caOAArfOpgrJaKyptQwp3GyQx jHkg9DlPJ1TeQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0E7935C10CE; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 15/19] srcu: Create concurrency-safe helper for initiating size transition Date: Fri, 4 Feb 2022 15:38:58 -0800 Message-Id: <20220204233902.1902-15-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Once there are contention-initiated size transitions, it will be possible for rcutorture to initiate a transition at the same time as a contention-initiated transition. This commit therefore creates a concurrency-safe helper function named srcu_transition_to_big() to safely initiate size transitions. Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index d16104405c8f9..5eee113531cdc 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -262,6 +262,25 @@ EXPORT_SYMBOL_GPL(init_srcu_struct); =20 #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 +/* + * Initiate an idempotent transition to SRCU_SIZE_BIG. + */ +static void srcu_transition_to_big(struct srcu_struct *ssp) +{ + unsigned long flags; + + /* Double-checked locking on ->srcu_size-state. */ + if (smp_load_acquire(&ssp->srcu_size_state) !=3D SRCU_SIZE_SMALL) + return; + spin_lock_irqsave_rcu_node(ssp, flags); + if (smp_load_acquire(&ssp->srcu_size_state) !=3D SRCU_SIZE_SMALL) { + spin_unlock_irqrestore_rcu_node(ssp, flags); + return; + } + smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_ALLOC); + spin_unlock_irqrestore_rcu_node(ssp, flags); +} + /* * First-use initialization of statically allocated srcu_struct * structure. Wiring up the combining tree is more than can be @@ -1496,8 +1515,8 @@ void srcu_torture_stats_print(struct srcu_struct *ssp= , char *tt, char *tf) } pr_cont(" T(%ld,%ld)\n", s0, s1); } - if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 2) - WRITE_ONCE(ssp->srcu_size_state, SRCU_SIZE_ALLOC); + if (convert_to_big =3D=3D 2) + srcu_transition_to_big(ssp); } EXPORT_SYMBOL_GPL(srcu_torture_stats_print); =20 --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEA7CC433FE for ; Fri, 4 Feb 2022 23:39:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378389AbiBDXj6 (ORCPT ); Fri, 4 Feb 2022 18:39:58 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:39420 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378268AbiBDXjI (ORCPT ); Fri, 4 Feb 2022 18:39:08 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 71B4FB8397A; Fri, 4 Feb 2022 23:39:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9CBB5C34108; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=K5zozA+k1BZU8VUVsoGu3r4tnm7mfejnZ1gaOcBE8jM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=arP/LcPgNLHad0SZ1XPZxqcXZ1vvS6L6AjzTUfJDCBQBPToAflyzs8TuDQ0a++dqa f+nxNCwzDYvyTLIlIyULhqDoxbQ7nkJVfroLSiWBc9op2BLgdQ1OvdgYq3OhMDrq+R blrrEVVpK+lW/3j83sGjmK7scLclXs86+qr9Wno1m18JFHcNr/0nP7UyU6FEmhGjld w+vMZQjaKwO2kJ0SYHxbQiL32H0L0JWdwr6IDd3QTjB733wruMqkyxFQWHgFxpicKT gJXbZQLdgJ41Z+DxNDiCUdhkT82UWSfRfOvca5+9z75FC5RXrNJnvEdqczRT5zEwKR OlZthcxKMdzyw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 1078E5C1184; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 16/19] srcu: Add contention-triggered addition of srcu_node tree Date: Fri, 4 Feb 2022 15:38:59 -0800 Message-Id: <20220204233902.1902-16-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit instruments the acquisitions of the srcu_struct structure's ->lock, enabling the initiation of a transition from SRCU_SIZE_SMALL to SRCU_SIZE_BIG when sufficient contention is experienced. The instrumentation counts the number of trylock failures within the confines of a single jiffy. If that number exceeds the value specified by the srcutree.small_contention_lim kernel boot parameter (which defaults to 100), and if the value specified by the srcutree.convert_to_big kernel boot parameter has the 0x10 bit set (defaults to 0), then a transition will be automatically initiated. By default, there will never be any transitions, so that none of the srcu_struct structures ever gains an srcu_node array. The useful values for srcutree.convert_to_big are: 0x00: Decide conversion approach at boot given system size. 0x01: Never convert. 0x02: Always convert at init_srcu_struct() time. 0x03: Convert when rcutorture prints its first round of statistics. 0x11: Convert if contention is encountered. 0x12: Convert if contention is encountered or when rcutorture prints its first round of statistics, whichever comes first. The value 0x12 acts the same as 0x02 because the convertion happens before there is any chance of contention. [ paulmck: Apply "static" feedback from kernel test robot. ] Signed-off-by: Paul E. McKenney --- .../admin-guide/kernel-parameters.txt | 22 ++++ include/linux/srcutree.h | 2 + kernel/rcu/srcutree.c | 107 ++++++++++++++---- 3 files changed, 107 insertions(+), 24 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index f5a27f067db9e..d7b64263fefbc 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5525,6 +5525,20 @@ off: Disable mitigation and remove performance impact to RDRAND and RDSEED =20 + srcutree.convert_to_big [KNL] + Specifies under what conditions an SRCU tree + srcu_struct structure will be converted to big + form, that is, with an rcu_node tree: + + 1: Never. + 2: At init_srcu_struct() time. + 3: When rcutorture decides to. + 0x1X: Above plus if high contention. + + Either way, the srcu_node tree will be sized based + on the actual runtime number of CPUs (nr_cpu_ids) + instead of the compile-time CONFIG_NR_CPUS. + srcutree.counter_wrap_check [KNL] Specifies how frequently to check for grace-period sequence counter wrap for the @@ -5542,6 +5556,14 @@ expediting. Set to zero to disable automatic expediting. =20 + srcutree.small_contention_lim [KNL] + Specifies the number of update-side contention + events per jiffy will be tolerated before + initiating a conversion of an srcu_struct + structure to big form. Note that the value of + srcutree.convert_to_big must have the 0x10 bit + set for contention-based conversions to occur. + ssbd=3D [ARM64,HW] Speculative Store Bypass Disable control =20 diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 44bd204498a11..1b9ff4ed37e45 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -72,6 +72,8 @@ struct srcu_struct { unsigned long srcu_gp_seq_needed; /* Latest gp_seq needed. */ unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */ unsigned long srcu_last_gp_end; /* Last GP end timestamp (ns) */ + unsigned long srcu_size_jiffies; /* Current contention-measurement interv= al. */ + unsigned long srcu_n_lock_retries; /* Contention events in current interv= al. */ struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ bool sda_is_static; /* May ->sda be passed to free_percpu()? */ unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 5eee113531cdc..1d72c7e3c33b5 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -41,13 +41,29 @@ module_param(counter_wrap_check, ulong, 0444); =20 /* * Control conversion to SRCU_SIZE_BIG: - * 0: Don't convert at all (default). - * 1: Convert at init_srcu_struct() time. - * 2: Convert when rcutorture invokes srcu_torture_stats_print(). + * 0: Decide at boot time based on system shape. + * 1: Don't convert at all (default). + * 2: Convert at init_srcu_struct() time. + * 3: Convert when rcutorture invokes srcu_torture_stats_print(). + * 0x1x: Convert when excessive contention encountered. */ -static int convert_to_big; +#define SRCU_SIZING_AUTO 0 +#define SRCU_SIZING_NONE 1 +#define SRCU_SIZING_INIT 2 +#define SRCU_SIZING_TORTURE 3 +#define SRCU_SIZING_CONTEND 0x10 +#define SRCU_SIZING_IS(x) ((convert_to_big & ~SRCU_SIZING_CONTEND) =3D=3D = x) +#define SRCU_SIZING_IS_NONE() (SRCU_SIZING_IS(SRCU_SIZING_NONE)) +#define SRCU_SIZING_IS_INIT() (SRCU_SIZING_IS(SRCU_SIZING_INIT)) +#define SRCU_SIZING_IS_TORTURE() (SRCU_SIZING_IS(SRCU_SIZING_TORTURE)) +#define SRCU_SIZING_IS_CONTEND() (convert_to_big & SRCU_SIZING_CONTEND) +static int convert_to_big =3D SRCU_SIZING_NONE; module_param(convert_to_big, int, 0444); =20 +/* Contention events per jiffy to initiate transition to big. */ +static int small_contention_lim __read_mostly =3D 100; +module_param(small_contention_lim, int, 0444); + /* Early-boot callback-management, so early that no lock is required! */ static LIST_HEAD(srcu_boot_list); static bool __read_mostly srcu_init_done; @@ -58,31 +74,40 @@ static void process_srcu(struct work_struct *work); static void srcu_delay_timer(struct timer_list *t); =20 /* Wrappers for lock acquisition and release, see raw_spin_lock_rcu_node()= . */ -#define spin_lock_rcu_node(p) \ -do { \ - spin_lock(&ACCESS_PRIVATE(p, lock)); \ - smp_mb__after_unlock_lock(); \ +#define spin_lock_rcu_node(p) \ +do { \ + spin_lock(&ACCESS_PRIVATE(p, lock)); \ + smp_mb__after_unlock_lock(); \ } while (0) =20 #define spin_unlock_rcu_node(p) spin_unlock(&ACCESS_PRIVATE(p, lock)) =20 -#define spin_lock_irq_rcu_node(p) \ -do { \ - spin_lock_irq(&ACCESS_PRIVATE(p, lock)); \ - smp_mb__after_unlock_lock(); \ +#define spin_lock_irq_rcu_node(p) \ +do { \ + spin_lock_irq(&ACCESS_PRIVATE(p, lock)); \ + smp_mb__after_unlock_lock(); \ } while (0) =20 -#define spin_unlock_irq_rcu_node(p) \ +#define spin_unlock_irq_rcu_node(p) \ spin_unlock_irq(&ACCESS_PRIVATE(p, lock)) =20 -#define spin_lock_irqsave_rcu_node(p, flags) \ -do { \ - spin_lock_irqsave(&ACCESS_PRIVATE(p, lock), flags); \ - smp_mb__after_unlock_lock(); \ +#define spin_lock_irqsave_rcu_node(p, flags) \ +do { \ + spin_lock_irqsave(&ACCESS_PRIVATE(p, lock), flags); \ + smp_mb__after_unlock_lock(); \ } while (0) =20 -#define spin_unlock_irqrestore_rcu_node(p, flags) \ - spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags) \ +#define spin_trylock_irqsave_rcu_node(p, flags) \ +({ \ + bool ___locked =3D spin_trylock_irqsave(&ACCESS_PRIVATE(p, lock), flags);= \ + \ + if (___locked) \ + smp_mb__after_unlock_lock(); \ + ___locked; \ +}) + +#define spin_unlock_irqrestore_rcu_node(p, flags) \ + spin_unlock_irqrestore(&ACCESS_PRIVATE(p, lock), flags) \ =20 /* * Initialize SRCU per-CPU data. Note that statically allocated @@ -216,7 +241,7 @@ static int init_srcu_struct_fields(struct srcu_struct *= ssp, bool is_static) init_srcu_struct_data(ssp); ssp->srcu_gp_seq_needed_exp =3D 0; ssp->srcu_last_gp_end =3D ktime_get_mono_fast_ns(); - if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && convert_to_= big =3D=3D 1) { + if (READ_ONCE(ssp->srcu_size_state) =3D=3D SRCU_SIZE_SMALL && SRCU_SIZING= _IS_INIT()) { if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { if (ssp->sda_is_static) { free_percpu(ssp->sda); @@ -262,6 +287,15 @@ EXPORT_SYMBOL_GPL(init_srcu_struct); =20 #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ =20 +/* + * Initiate a transition to SRCU_SIZE_BIG with lock held. + */ +static void __srcu_transition_to_big(struct srcu_struct *ssp) +{ + lockdep_assert_held(&ACCESS_PRIVATE(ssp, lock)); + smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_ALLOC); +} + /* * Initiate an idempotent transition to SRCU_SIZE_BIG. */ @@ -277,10 +311,35 @@ static void srcu_transition_to_big(struct srcu_struct= *ssp) spin_unlock_irqrestore_rcu_node(ssp, flags); return; } - smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_ALLOC); + __srcu_transition_to_big(ssp); spin_unlock_irqrestore_rcu_node(ssp, flags); } =20 +/* + * Acquire the specified srcu_struct structure's ->lock, but check for + * excessive contention, which results in initiation of a transition + * to SRCU_SIZE_BIG. But only if the srcutree.convert_to_big module + * parameter permits this. + */ +static void spin_lock_irqsave_ssp_contention(struct srcu_struct *ssp, unsi= gned long *flags) +{ + unsigned long j; + + if (spin_trylock_irqsave_rcu_node(ssp, *flags)) + return; + spin_lock_irqsave_rcu_node(ssp, *flags); + if (!SRCU_SIZING_IS_CONTEND() || ssp->srcu_size_state) + return; + j =3D jiffies; + if (ssp->srcu_size_jiffies !=3D j) { + ssp->srcu_size_jiffies =3D j; + ssp->srcu_n_lock_retries =3D 0; + } + if (++ssp->srcu_n_lock_retries <=3D small_contention_lim) + return; + __srcu_transition_to_big(ssp); +} + /* * First-use initialization of statically allocated srcu_struct * structure. Wiring up the combining tree is more than can be @@ -705,7 +764,7 @@ static void srcu_funnel_exp_start(struct srcu_struct *s= sp, struct srcu_node *snp WRITE_ONCE(snp->srcu_gp_seq_needed_exp, s); spin_unlock_irqrestore_rcu_node(snp, flags); } - spin_lock_irqsave_rcu_node(ssp, flags); + spin_lock_irqsave_ssp_contention(ssp, &flags); if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, s)) WRITE_ONCE(ssp->srcu_gp_seq_needed_exp, s); spin_unlock_irqrestore_rcu_node(ssp, flags); @@ -760,7 +819,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ss= p, struct srcu_data *sdp, } =20 /* Top of tree, must ensure the grace period will be started. */ - spin_lock_irqsave_rcu_node(ssp, flags); + spin_lock_irqsave_ssp_contention(ssp, &flags); if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed, s)) { /* * Record need for grace period s. Pair with load @@ -1515,7 +1574,7 @@ void srcu_torture_stats_print(struct srcu_struct *ssp= , char *tt, char *tf) } pr_cont(" T(%ld,%ld)\n", s0, s1); } - if (convert_to_big =3D=3D 2) + if (SRCU_SIZING_IS_TORTURE()) srcu_transition_to_big(ssp); } EXPORT_SYMBOL_GPL(srcu_torture_stats_print); --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1CD0C433EF for ; Fri, 4 Feb 2022 23:39:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378358AbiBDXjX (ORCPT ); Fri, 4 Feb 2022 18:39:23 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:47900 "EHLO dfw.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378235AbiBDXjG (ORCPT ); Fri, 4 Feb 2022 18:39:06 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 73FF761CAF; Fri, 4 Feb 2022 23:39:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9C936C34107; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=fIwHLIzRVHxcHsAxuneAYdfjPuEOFZXTqbxd4AMS7xs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=L4/BtovDZUIEjpWVSFR5ZCHl1jn7TXZvk8AogxvhS70GdwNHAQeXwRgrPmsLEjzmM qNgKrdPTuicRul4l6laB8ANcoOgXi16jlPWXlRiiXxa4wdhWkmedhm7qL7SKKTXJSE S2TRjfvQiR5tOzXtR6rA7HegO7quW6x9bhxrQiGEg33Jrr21T6lqqRlb2Wrti0UmTu x0OI6rML3+NvSuIC6NvXuJn6gWkzJT9cCcC2fdoR2QDeAacT1y9PAqscsRy/xvdVpX ihSKwxqwd4vnO4SJ1ihg4pK1sGmIVbnEMKUNRHvLwnQx8zsO6dnEbE97QUECr3Vl/0 ls/XlK88bmUgQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 124125C1234; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Jiapeng Chong , Abaci Robot , "Paul E . McKenney" Subject: [PATCH rcu 17/19] srcu: Make srcu_size_state_name static Date: Fri, 4 Feb 2022 15:39:00 -0800 Message-Id: <20220204233902.1902-17-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Jiapeng Chong This symbol is not used outside of srcutree.c, so this commit marks it stat= ic. Doing so fixes the following sparse warning: kernel/rcu/srcutree.c:1426:12: warning: symbol 'srcu_size_state_name' was not declared. Should it be static? Reported-by: Abaci Robot Signed-off-by: Jiapeng Chong Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 1d72c7e3c33b5..f13e84ecbca6a 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1512,7 +1512,7 @@ void srcutorture_get_gp_data(enum rcutorture_type tes= t_type, } EXPORT_SYMBOL_GPL(srcutorture_get_gp_data); =20 -const char * const srcu_size_state_name[] =3D +static const char * const srcu_size_state_name[] =3D { "SRCU_SIZE_SMALL", "SRCU_SIZE_ALLOC", --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B39F5C433EF for ; Fri, 4 Feb 2022 23:39:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378391AbiBDXj1 (ORCPT ); Fri, 4 Feb 2022 18:39:27 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:47902 "EHLO dfw.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378238AbiBDXjG (ORCPT ); Fri, 4 Feb 2022 18:39:06 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7DDC561CAD; Fri, 4 Feb 2022 23:39:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9DBAFC3410B; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=x6pO0fVHUYqRxZjocoXSG7Zfk3O9t9S3+6VG14PmyDQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RT8Ix1XY0Dhl+tMwdgoyhOHoJwg5lQFAXNynILCQKTTo+BwkaMdea0ku1mYbtpKpI 4L1VSbN/8D7GNH9Ca4xqgDvNBXLSqhchM649udlwpsWOFxUyVpMIROLXrwhMJFI3QS UZilcqCrQHcSSA0R8TS7gRapyu3/SWCuJMbmE33hNUrvOQSZJZ3aFBIiGfYKbWFUrM JaxH+UoajrcsrUyYx2AG6ORymSQUVXbZcSWiT96K1WemTtze7MujmPrTrt7mkDu0u3 UTl5TigFQphw+NvKJKDO8eqIKPizaHutC9dMo1M4X07XPXVg+IdmOIzGMHcXC2pX1m k6mwkXMqA8Urg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 140265C1273; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 18/19] srcu: Automatically determine size-transition strategy at boot Date: Fri, 4 Feb 2022 15:39:01 -0800 Message-Id: <20220204233902.1902-18-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit adds a srcutree.convert_to_big option of zero that causes SRCU to decide at boot whether to wait for contention (small systems) or immediately expand to large (large systems). A new srcutree.big_cpu_lim (defaulting to 128) defines how many CPUs constitute a large system. Signed-off-by: Paul E. McKenney --- .../admin-guide/kernel-parameters.txt | 9 +++++++++ kernel/rcu/srcutree.c | 19 ++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index d7b64263fefbc..ad2388a431944 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5525,11 +5525,20 @@ off: Disable mitigation and remove performance impact to RDRAND and RDSEED =20 + srcutree.big_cpu_lim [KNL] + Specifies the number of CPUs constituting a + large system, such that srcu_struct structures + should immediately allocate an srcu_node array. + This kernel-boot parameter defaults to 128, but + takes effect only when srcutree.convert_to_big + is equal to zero. + srcutree.convert_to_big [KNL] Specifies under what conditions an SRCU tree srcu_struct structure will be converted to big form, that is, with an rcu_node tree: =20 + 0: Decide at boot time (default). 1: Never. 2: At init_srcu_struct() time. 3: When rcutorture decides to. diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index f13e84ecbca6a..c3968e091d019 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -57,9 +57,13 @@ module_param(counter_wrap_check, ulong, 0444); #define SRCU_SIZING_IS_INIT() (SRCU_SIZING_IS(SRCU_SIZING_INIT)) #define SRCU_SIZING_IS_TORTURE() (SRCU_SIZING_IS(SRCU_SIZING_TORTURE)) #define SRCU_SIZING_IS_CONTEND() (convert_to_big & SRCU_SIZING_CONTEND) -static int convert_to_big =3D SRCU_SIZING_NONE; +static int convert_to_big =3D SRCU_SIZING_AUTO; module_param(convert_to_big, int, 0444); =20 +/* Number of CPUs to trigger init_srcu_struct()-time transition to big. */ +static int big_cpu_lim __read_mostly =3D 128; +module_param(big_cpu_lim, int, 0444); + /* Contention events per jiffy to initiate transition to big. */ static int small_contention_lim __read_mostly =3D 100; module_param(small_contention_lim, int, 0444); @@ -1592,6 +1596,17 @@ void __init srcu_init(void) { struct srcu_struct *ssp; =20 + /* Decide on srcu_struct-size strategy. */ + if (SRCU_SIZING_IS(SRCU_SIZING_AUTO)) { + if (nr_cpu_ids >=3D big_cpu_lim) { + convert_to_big =3D SRCU_SIZING_INIT; // Don't bother waiting for conten= tion. + pr_info("%s: Setting srcu_struct sizes to big.\n", __func__); + } else { + convert_to_big =3D SRCU_SIZING_NONE | SRCU_SIZING_CONTEND; + pr_info("%s: Setting srcu_struct sizes based on contention.\n", __func_= _); + } + } + /* * Once that is set, call_srcu() can follow the normal path and * queue delayed work. This must follow RCU workqueues creation @@ -1602,6 +1617,8 @@ void __init srcu_init(void) ssp =3D list_first_entry(&srcu_boot_list, struct srcu_struct, work.work.entry); list_del_init(&ssp->work.work.entry); + if (SRCU_SIZING_IS(SRCU_SIZING_INIT) && !ssp->srcu_size_state) + ssp->srcu_size_state =3D SRCU_SIZE_ALLOC; queue_work(rcu_gp_wq, &ssp->work.work); } } --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87245C433EF for ; Fri, 4 Feb 2022 23:39:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378466AbiBDXjg (ORCPT ); Fri, 4 Feb 2022 18:39:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378255AbiBDXjH (ORCPT ); Fri, 4 Feb 2022 18:39:07 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85B92E025B35; Fri, 4 Feb 2022 15:39:06 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E6A8361C36; Fri, 4 Feb 2022 23:39:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A006CC3410A; Fri, 4 Feb 2022 23:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017944; bh=ibhe7XGDHOnx160ajeMSJLi117GE5aB+Eytzu80AjNY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qsVivvSMsTx3VvTxpYwe6Krv037OLf58VOh6Nv9AN/x7gnf24Zn1ujSSNVXNhJyUx zy273elWuiXOz203bRtbmZFBM3P03QzCnOVwI9X8J36zMtmTFhsg4/UqY4xa0xpyLm wXTyPawfbuRAjWf252okzpQCyzDBvbUekWH1X9TSO2m4C60D8q1po0ojku4FX9ailn Q+j7Un8FH08t+aw8UOaCCWecDrbfYkk0j+91vIfUSlBaoX37SUt9NfpqWquTz1zJ/N 8oljxwJMBW/48KyvIXNPCre8lCDJdZh2tPcO1US1h7afZB9rGG7X4CBFrhdtEjx2OF xHLdO7/DkI+LA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 15F3F5C14AE; Fri, 4 Feb 2022 15:39:04 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 19/19] srcu: Add contention check to call_srcu() srcu_data ->lock acquisition Date: Fri, 4 Feb 2022 15:39:02 -0800 Message-Id: <20220204233902.1902-19-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> References: <20220204233858.GA1469@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit increases the sensitivity of contention detection by adding checks to the acquisition of the srcu_data structure's lock on the call_srcu() code path. Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 45 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index c3968e091d019..31a7a9e2445da 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -320,18 +320,13 @@ static void srcu_transition_to_big(struct srcu_struct= *ssp) } =20 /* - * Acquire the specified srcu_struct structure's ->lock, but check for - * excessive contention, which results in initiation of a transition - * to SRCU_SIZE_BIG. But only if the srcutree.convert_to_big module - * parameter permits this. + * Check to see if the just-encountered contention event justifies + * a transition to SRCU_SIZE_BIG. */ -static void spin_lock_irqsave_ssp_contention(struct srcu_struct *ssp, unsi= gned long *flags) +static void spin_lock_irqsave_check_contention(struct srcu_struct *ssp) { unsigned long j; =20 - if (spin_trylock_irqsave_rcu_node(ssp, *flags)) - return; - spin_lock_irqsave_rcu_node(ssp, *flags); if (!SRCU_SIZING_IS_CONTEND() || ssp->srcu_size_state) return; j =3D jiffies; @@ -344,6 +339,38 @@ static void spin_lock_irqsave_ssp_contention(struct sr= cu_struct *ssp, unsigned l __srcu_transition_to_big(ssp); } =20 +/* + * Acquire the specified srcu_data structure's ->lock, but check for + * excessive contention, which results in initiation of a transition + * to SRCU_SIZE_BIG. But only if the srcutree.convert_to_big module + * parameter permits this. + */ +static void spin_lock_irqsave_sdp_contention(struct srcu_data *sdp, unsign= ed long *flags) +{ + struct srcu_struct *ssp =3D sdp->ssp; + + if (spin_trylock_irqsave_rcu_node(sdp, *flags)) + return; + spin_lock_irqsave_rcu_node(ssp, *flags); + spin_lock_irqsave_check_contention(ssp); + spin_unlock_irqrestore_rcu_node(ssp, *flags); + spin_lock_irqsave_rcu_node(sdp, *flags); +} + +/* + * Acquire the specified srcu_struct structure's ->lock, but check for + * excessive contention, which results in initiation of a transition + * to SRCU_SIZE_BIG. But only if the srcutree.convert_to_big module + * parameter permits this. + */ +static void spin_lock_irqsave_ssp_contention(struct srcu_struct *ssp, unsi= gned long *flags) +{ + if (spin_trylock_irqsave_rcu_node(ssp, *flags)) + return; + spin_lock_irqsave_rcu_node(ssp, *flags); + spin_lock_irqsave_check_contention(ssp); +} + /* * First-use initialization of statically allocated srcu_struct * structure. Wiring up the combining tree is more than can be @@ -989,7 +1016,7 @@ static unsigned long srcu_gp_start_if_needed(struct sr= cu_struct *ssp, } else { sdp =3D raw_cpu_ptr(ssp->sda); } - spin_lock_irqsave_rcu_node(sdp, flags); + spin_lock_irqsave_sdp_contention(sdp, &flags); if (rhp) rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp); rcu_segcblist_advance(&sdp->srcu_cblist, --=20 2.31.1.189.g2e36527f23