From nobody Mon Jun 29 17:36:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84C1BC433F5 for ; Fri, 4 Feb 2022 23:08:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377920AbiBDXIQ (ORCPT ); Fri, 4 Feb 2022 18:08:16 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:53354 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377895AbiBDXIJ (ORCPT ); Fri, 4 Feb 2022 18:08:09 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 26BF7B83965; Fri, 4 Feb 2022 23:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CC13BC004E1; Fri, 4 Feb 2022 23:08:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644016086; bh=CiCcujZjYhOJvEnjNIIZ+YVdOcFjULHe8Yb/A8k1ttI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=H56nWpBosw7NEj7C7wBlD+SpSrCJXvErjGPL4VGUbUO8i6mfjDb+4MXxmr4QaKp9P mXZkW06xLnj8joGWYjiIfxl2WiQp3CLeXqkKFKczakIGRSy8bdqRq5Oien5QE7can5 PCX68Bs1SWxjq07WXda2tljvQPMH5lcZwde3FYtyKeDuGpC/iUZvHU6WCfVHGfSMA5 sqKCuObsSsAwLalz9e6NOdzWj6gmwGazZ3/F5C0RXhPovi5W7GSdZzQDv3z+cjAPxu 8sr2Uat7mmxPcuJrCt1O/FoHNo4v/FfJt5nQpc//EAXgconBjx7OkZaXDq8qhzMIEK 7haNnh7qRWvoQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 950B15C0418; Fri, 4 Feb 2022 15:08:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay , Frederic Weisbecker Subject: [PATCH rcu 1/9] MAINTAINERS: Add Frederic and Neeraj to their RCU files Date: Fri, 4 Feb 2022 15:07:57 -0800 Message-Id: <20220204230805.4193767-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> References: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Adding Frederic as an RCU maintainer for kernel/rcu/tree_nocb.h given his work with offloading and de-offloading callbacks from CPUs. Also adding Neeraj for kernel/rcu/tasks.h given his focused work on RCU Tasks Trace. As in I am reasonably certain that each understands the full contents of the corresponding file. Cc: Neeraj Upadhyay Cc: Frederic Weisbecker Signed-off-by: Paul E. McKenney Acked-by: Frederic Weisbecker --- MAINTAINERS | 2 ++ 1 file changed, 2 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index ea3e6c9143848..5ad49de532b1a 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -16298,6 +16298,8 @@ F: tools/testing/selftests/resctrl/ =20 READ-COPY UPDATE (RCU) M: "Paul E. McKenney" +M: Frederic Weisbecker (kernel/rcu/tree_nocb.h) +M: Neeraj Upadhyay (kernel/rcu/tasks.h) M: Josh Triplett R: Steven Rostedt R: Mathieu Desnoyers --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 320B0C433EF for ; Fri, 4 Feb 2022 23:08:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355975AbiBDXI0 (ORCPT ); Fri, 4 Feb 2022 18:08:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234736AbiBDXII (ORCPT ); Fri, 4 Feb 2022 18:08:08 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E19FBDF99CDC; Fri, 4 Feb 2022 15:08:07 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 812D961B9C; Fri, 4 Feb 2022 23:08:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D30CDC340EF; Fri, 4 Feb 2022 23:08:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644016086; bh=oGrMHn07x9XfkVQxwaEWPiDvFUjbNzIN/CwooO7F/ew=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QvcWvQ5U8uCqJRP4iTgCQNvSdUEH+kJiQdkYsoBOsDZ4WjPJ+x/+PJUjTLpkuhmAF hAqz4amEGFBL0MD+pYhLVKGjXEAVL8x/IO8/3mgsYaz4bq+JjMu7UiEYRgy1ROe8xo NrdWFtzIamgg/3Ygkwt5A+iv3ytCwJdUOLpFsg1nglJBgBGso6bY2wI3+IbBMh0Kc1 4hcLDSvLifEcTAyjOWwpPIhc0bt03O0TF6BKh7XiUilIHArVTO9Lxd9zwAe5OwVT/m GM9kEowtj5TqTk3PauqhXrqSpzW1rqmLzXwqtzD9bwJML//JIGh5TwfHufDqyrPE94 y5J6z/1ALTlpA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 98CC65C0829; Fri, 4 Feb 2022 15:08:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Uladzislau Rezki (Sony)" , "Paul E . McKenney" Subject: [PATCH rcu 2/9] rcu: Fix description of kvfree_rcu() Date: Fri, 4 Feb 2022 15:07:58 -0800 Message-Id: <20220204230805.4193767-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> References: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: "Uladzislau Rezki (Sony)" The kvfree_rcu() header comment's description of the "ptr" parameter is unclear, therefore rephrase it to make it clear that it is a pointer to the memory to eventually be passed to kvfree(). Reported-by: Steven Rostedt Signed-off-by: Uladzislau Rezki (Sony) Signed-off-by: Paul E. McKenney --- include/linux/rcupdate.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 88b42eb464068..9d7df8d36af07 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -924,7 +924,7 @@ static inline notrace void rcu_read_unlock_sched_notrac= e(void) * * kvfree_rcu(ptr); * - * where @ptr is a pointer to kvfree(). + * where @ptr is the pointer to be freed by kvfree(). * * Please note, head-less way of freeing is permitted to * use from a context that has to follow might_sleep() --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1C9FC433F5 for ; Fri, 4 Feb 2022 23:08:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377925AbiBDXIS (ORCPT ); Fri, 4 Feb 2022 18:08:18 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:53364 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377899AbiBDXIJ (ORCPT ); Fri, 4 Feb 2022 18:08:09 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 58822B83963; Fri, 4 Feb 2022 23:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB0D4C36AE2; Fri, 4 Feb 2022 23:08:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644016087; bh=unSHV5pCcmnfZnpIr4QRTxITUK7+vXyWYLgKVbprAK4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q2nuwxUMINEcQUq4ReBOV8RjIQ80vmJviSaSEUvbIaTGN/AXFJbW48cDHofBjQUuP 3A862b7eab49xMCt7FxriB2e7IzDvaprXTLfeAOislr0S39ARNV4r5B4W+1GYSjwWQ 6gm1snjr6aJY/Mv7scINfGqTD6n9diIGPxMTAzFqeY6mmsGTzZK8D93QQlZ0vb97D1 BP53RWIvvduh6OFE6+8VNyGf2YM3mSozr1emNK2ucxy7VRObULIuU0TjKdAepxmZAN 509KmyELWzfvDVSg0h0zsVdm83PUCmtwLktTOTXizM90UkWDhSWewxd11YekSUjw4i BDSKvoAxgtabA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 9AC7B5C08B7; Fri, 4 Feb 2022 15:08:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, David Woodhouse , "Paul E . McKenney" Subject: [PATCH rcu 3/9] rcu: Add mutex for rcu boost kthread spawning and affinity setting Date: Fri, 4 Feb 2022 15:07:59 -0800 Message-Id: <20220204230805.4193767-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> References: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: David Woodhouse As we handle parallel CPU bringup, we will need to take care to avoid spawning multiple boost threads, or race conditions when setting their affinity. Spotted by Paul McKenney. Signed-off-by: David Woodhouse Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 1 + kernel/rcu/tree.h | 3 +++ kernel/rcu/tree_plugin.h | 10 ++++++++-- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index a4c25a6283b0b..d1d1a8c51223b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4570,6 +4570,7 @@ static void __init rcu_init_one(void) init_waitqueue_head(&rnp->exp_wq[2]); init_waitqueue_head(&rnp->exp_wq[3]); spin_lock_init(&rnp->exp_lock); + mutex_init(&rnp->boost_kthread_mutex); } } =20 diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 486fc901bd085..3b8b60de07c38 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -110,6 +110,9 @@ struct rcu_node { /* side effect, not as a lock. */ unsigned long boost_time; /* When to start boosting (jiffies). */ + struct mutex boost_kthread_mutex; + /* Exclusion for thread spawning and affinity */ + /* manipulation. */ struct task_struct *boost_kthread_task; /* kthread that takes care of priority */ /* boosting for this rcu_node structure. */ diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index c5b45c2f68a15..07845dcd33c5e 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1172,15 +1172,16 @@ static void rcu_spawn_one_boost_kthread(struct rcu_= node *rnp) struct sched_param sp; struct task_struct *t; =20 + mutex_lock(&rnp->boost_kthread_mutex); if (rnp->boost_kthread_task || !rcu_scheduler_fully_active) - return; + goto out; =20 rcu_state.boost =3D 1; =20 t =3D kthread_create(rcu_boost_kthread, (void *)rnp, "rcub/%d", rnp_index); if (WARN_ON_ONCE(IS_ERR(t))) - return; + goto out; =20 raw_spin_lock_irqsave_rcu_node(rnp, flags); rnp->boost_kthread_task =3D t; @@ -1188,6 +1189,9 @@ static void rcu_spawn_one_boost_kthread(struct rcu_no= de *rnp) sp.sched_priority =3D kthread_prio; sched_setscheduler_nocheck(t, SCHED_FIFO, &sp); wake_up_process(t); /* get to TASK_INTERRUPTIBLE quickly. */ + + out: + mutex_unlock(&rnp->boost_kthread_mutex); } =20 /* @@ -1210,6 +1214,7 @@ static void rcu_boost_kthread_setaffinity(struct rcu_= node *rnp, int outgoingcpu) return; if (!zalloc_cpumask_var(&cm, GFP_KERNEL)) return; + mutex_lock(&rnp->boost_kthread_mutex); for_each_leaf_node_possible_cpu(rnp, cpu) if ((mask & leaf_node_cpu_bit(rnp, cpu)) && cpu !=3D outgoingcpu) @@ -1218,6 +1223,7 @@ static void rcu_boost_kthread_setaffinity(struct rcu_= node *rnp, int outgoingcpu) if (cpumask_weight(cm) =3D=3D 0) cpumask_copy(cm, housekeeping_cpumask(HK_FLAG_RCU)); set_cpus_allowed_ptr(t, cm); + mutex_unlock(&rnp->boost_kthread_mutex); free_cpumask_var(cm); } =20 --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1969C433F5 for ; Fri, 4 Feb 2022 23:08:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377977AbiBDXI3 (ORCPT ); Fri, 4 Feb 2022 18:08:29 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:53370 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377901AbiBDXIJ (ORCPT ); Fri, 4 Feb 2022 18:08:09 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7B7EDB83969; Fri, 4 Feb 2022 23:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1673CC340F3; Fri, 4 Feb 2022 23:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644016087; bh=mPWgAdkyrX/RwbFRVrRX5RqVqjHoL8oAY4muj7gz8PI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BuVPuasbQiMjDpJkz7nIdLTaoS7fpqwBtnINOH+qHg7MH2ZkT/ZZIhnrHFWYXLnVk i+IlXLompFRpHDXhQekytUVjeEJVuEx4abBXRPl5+SjjuX/CshH6G1Ab2ci4+yIOQj W5And4cME3M2AXQ8to0cDcym77q2L2hA/nWoUgcw/W8BoYBeHXzdT0qCWmBdkPuPuq ix1SJ+CcsM+oEeRFM93vGtrTQXhzLcqy2TC5HMnxBebTZfQfroS65xhuh0BZMA0KHn 1BKOXzKYxoC/TuuXnvjHIZIUOttrA8QIL6CTqUdQ4Ac5prF8pUA5/cy2q28265NkNn qyQ89g2KsjaqQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 9CBEA5C08DD; Fri, 4 Feb 2022 15:08:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 4/9] rcu: Inline __call_rcu() into call_rcu() Date: Fri, 4 Feb 2022 15:08:00 -0800 Message-Id: <20220204230805.4193767-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> References: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Because __call_rcu() is invoked only by call_rcu(), this commit inlines the former into the latter. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 91 ++++++++++++++++++++++------------------------- 1 file changed, 42 insertions(+), 49 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d1d1a8c51223b..f1bb7ccc00847 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2995,9 +2995,47 @@ static void check_cb_ovld(struct rcu_data *rdp) raw_spin_unlock_rcu_node(rnp); } =20 -/* Helper function for call_rcu() and friends. */ -static void -__call_rcu(struct rcu_head *head, rcu_callback_t func) +/** + * call_rcu() - Queue an RCU callback for invocation after a grace period. + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. However, the callback function + * might well execute concurrently with RCU read-side critical sections + * that started after call_rcu() was invoked. + * + * RCU read-side critical sections are delimited by rcu_read_lock() + * and rcu_read_unlock(), and may be nested. In addition, but only in + * v5.0 and later, regions of code across which interrupts, preemption, + * or softirqs have been disabled also serve as RCU read-side critical + * sections. This includes hardware interrupt handlers, softirq handlers, + * and NMI handlers. + * + * Note that all CPUs must agree that the grace period extended beyond + * all pre-existing RCU read-side critical section. On systems with more + * than one CPU, this means that when "func()" is invoked, each CPU is + * guaranteed to have executed a full memory barrier since the end of its + * last RCU read-side critical section whose beginning preceded the call + * to call_rcu(). It also means that each CPU executing an RCU read-side + * critical section that continues beyond the start of "func()" must have + * executed a memory barrier after the call_rcu() but before the beginning + * of that RCU read-side critical section. Note that these guarantees + * include CPUs that are offline, idle, or executing in user mode, as + * well as CPUs that are executing in the kernel. + * + * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the + * resulting RCU callback function "func()", then both CPU A and CPU B are + * guaranteed to execute a full memory barrier during the time interval + * between the call to call_rcu() and the invocation of "func()" -- even + * if CPU A and CPU B are the same CPU (but again only if the system has + * more than one CPU). + * + * Implementation of these memory-ordering guarantees is described here: + * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. + */ +void call_rcu(struct rcu_head *head, rcu_callback_t func) { static atomic_t doublefrees; unsigned long flags; @@ -3011,7 +3049,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) /* * Probable double call_rcu(), so leak the callback. * Use rcu:rcu_callback trace event to find the previous - * time callback was passed to __call_rcu(). + * time callback was passed to call_rcu(). */ if (atomic_inc_return(&doublefrees) < 4) { pr_err("%s(): Double-freed CB %p->%pS()!!! ", __func__, head, head->fu= nc); @@ -3060,51 +3098,6 @@ __call_rcu(struct rcu_head *head, rcu_callback_t fun= c) local_irq_restore(flags); } } - -/** - * call_rcu() - Queue an RCU callback for invocation after a grace period. - * @head: structure to be used for queueing the RCU updates. - * @func: actual callback function to be invoked after the grace period - * - * The callback function will be invoked some time after a full grace - * period elapses, in other words after all pre-existing RCU read-side - * critical sections have completed. However, the callback function - * might well execute concurrently with RCU read-side critical sections - * that started after call_rcu() was invoked. - * - * RCU read-side critical sections are delimited by rcu_read_lock() - * and rcu_read_unlock(), and may be nested. In addition, but only in - * v5.0 and later, regions of code across which interrupts, preemption, - * or softirqs have been disabled also serve as RCU read-side critical - * sections. This includes hardware interrupt handlers, softirq handlers, - * and NMI handlers. - * - * Note that all CPUs must agree that the grace period extended beyond - * all pre-existing RCU read-side critical section. On systems with more - * than one CPU, this means that when "func()" is invoked, each CPU is - * guaranteed to have executed a full memory barrier since the end of its - * last RCU read-side critical section whose beginning preceded the call - * to call_rcu(). It also means that each CPU executing an RCU read-side - * critical section that continues beyond the start of "func()" must have - * executed a memory barrier after the call_rcu() but before the beginning - * of that RCU read-side critical section. Note that these guarantees - * include CPUs that are offline, idle, or executing in user mode, as - * well as CPUs that are executing in the kernel. - * - * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the - * resulting RCU callback function "func()", then both CPU A and CPU B are - * guaranteed to execute a full memory barrier during the time interval - * between the call to call_rcu() and the invocation of "func()" -- even - * if CPU A and CPU B are the same CPU (but again only if the system has - * more than one CPU). - * - * Implementation of these memory-ordering guarantees is described here: - * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. - */ -void call_rcu(struct rcu_head *head, rcu_callback_t func) -{ - __call_rcu(head, func); -} EXPORT_SYMBOL_GPL(call_rcu); =20 =20 --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A41CC433F5 for ; Fri, 4 Feb 2022 23:08:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377941AbiBDXIV (ORCPT ); Fri, 4 Feb 2022 18:08:21 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:53366 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377900AbiBDXIJ (ORCPT ); Fri, 4 Feb 2022 18:08:09 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 6230FB83967; Fri, 4 Feb 2022 23:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 09C6EC340F1; Fri, 4 Feb 2022 23:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644016087; bh=F0tWzgWyTYIaOE59H/lt7xyxwXQQkMt7UW6k3PjcRfI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KwMVc5kHGt9LR4D9VYGVwl6A0quOy2YKoBtx1/u+nLM9XeDt3jyxmRqfu2bo4niXn w+WPAwrBt9D6gcQwGgUyMjLorhqF0sym7cctQiIA3VHSbWHB9j3AWCbVORY9FCDc1S O4uthPcIySi1ixNzcltPZvye5kP3m/Yu/OuWZDlTUZ7TxpI3w1ReRy24yNS0zEq8MV Awyj4r/rUMTRyetyWqeYjaIN39BYEnyvlRrHjV3bukPm58ZqiPAyjhlvTfatDlq/Ap O2WDPi46M2UZH12XJR2/25ymYx9bPCC2BZxdQwO74nP17mr4O32JfgGcGgqYe/T0nm +cIqXasP51y0A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 9E9F05C0992; Fri, 4 Feb 2022 15:08:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Zqiang , Marco Elver , "Paul E . McKenney" Subject: [PATCH rcu 5/9] kasan: Record work creation stack trace with interrupts enabled Date: Fri, 4 Feb 2022 15:08:01 -0800 Message-Id: <20220204230805.4193767-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> References: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Zqiang Recording the work creation stack trace for KASAN reports in call_rcu() is expensive, due to unwinding the stack, but also due to acquiring depot_lock inside stackdepot (which may be contended). Because calling kasan_record_aux_stack_noalloc() does not require interrupts to already be disabled, this may unnecessarily extend the time with interrupts disabled. Therefore, move calling kasan_record_aux_stack() before the section with interrupts disabled. Acked-by: Marco Elver Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index f1bb7ccc00847..ca8d7dd026eeb 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3060,8 +3060,8 @@ void call_rcu(struct rcu_head *head, rcu_callback_t f= unc) } head->func =3D func; head->next =3D NULL; - local_irq_save(flags); kasan_record_aux_stack_noalloc(head); + local_irq_save(flags); rdp =3D this_cpu_ptr(&rcu_data); =20 /* Add the callback to our list. */ --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73252C43219 for ; Fri, 4 Feb 2022 23:08:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377911AbiBDXIN (ORCPT ); Fri, 4 Feb 2022 18:08:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377896AbiBDXII (ORCPT ); Fri, 4 Feb 2022 18:08:08 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 816E9DF99CE4; Fri, 4 Feb 2022 15:08:08 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2302B61BB9; Fri, 4 Feb 2022 23:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2425CC340F6; Fri, 4 Feb 2022 23:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644016087; bh=HtrXU6XavCULFLTGswh4umGI+avoB5uYHWhzGpWLZXU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BmI+L8KAmubNxgJ795HEDlDRj4rXhcbDQ3pLpxzVFKHEQop3Tq3tK53DOjvQibsVW jX/Aa+BSVnoFknjOBOHpO0I5wBlMBnBkayQTucETiEW/GAxW2ErFznEl37dTe5U5gn hCbbno3nQ9t6uzfaXhbtzWgPFH0nzAP9A00BjIyArkXpHDGwIu2vVeO1LeBpIMA9KO o8FHQRQ56yqKBsK4e6MiHvrcM8sFZuo2UKHTz2BfH3yvfOYrvWJJU5pYae4T40kLir YYrZaeNZJ+al2h+LbDN1sG9Kjlp+OgxL4672PeLUhR3I1wstEaNPpWedSXxHcOacdj xp97qdjgnz2nQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A05CC5C0A0A; Fri, 4 Feb 2022 15:08:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Zhouyi Zhou , Frederic Weisbecker Subject: [PATCH rcu 6/9] rcu: Mark writes to the rcu_segcblist structure's ->flags field Date: Fri, 4 Feb 2022 15:08:02 -0800 Message-Id: <20220204230805.4193767-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> References: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" KCSAN reports data races between the rcu_segcblist_clear_flags() and rcu_segcblist_set_flags() functions, though misreporting the latter as a call to rcu_segcblist_is_enabled() from call_rcu(). This commit converts the updates of this field to WRITE_ONCE(), relying on the resulting unmarked reads to continue to detect buggy concurrent writes to this field. Reported-by: Zhouyi Zhou Signed-off-by: Paul E. McKenney Cc: Frederic Weisbecker --- kernel/rcu/rcu_segcblist.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h index e373fbe44da5e..431cee212467d 100644 --- a/kernel/rcu/rcu_segcblist.h +++ b/kernel/rcu/rcu_segcblist.h @@ -56,13 +56,13 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcb= list *rsclp) static inline void rcu_segcblist_set_flags(struct rcu_segcblist *rsclp, int flags) { - rsclp->flags |=3D flags; + WRITE_ONCE(rsclp->flags, rsclp->flags | flags); } =20 static inline void rcu_segcblist_clear_flags(struct rcu_segcblist *rsclp, int flags) { - rsclp->flags &=3D ~flags; + WRITE_ONCE(rsclp->flags, rsclp->flags & ~flags); } =20 static inline bool rcu_segcblist_test_flags(struct rcu_segcblist *rsclp, --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1DC4C433EF for ; Fri, 4 Feb 2022 23:08:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378013AbiBDXIm (ORCPT ); Fri, 4 Feb 2022 18:08:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377898AbiBDXII (ORCPT ); Fri, 4 Feb 2022 18:08:08 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82252DF99CE5; Fri, 4 Feb 2022 15:08:08 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1EE1C61BB6; Fri, 4 Feb 2022 23:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 218BBC340F5; Fri, 4 Feb 2022 23:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644016087; bh=B7FvPqeozGPDV9tkcqYeb//UtgSVBahiXAeDSDUUo24=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HQYQ6V+U0M9A0ZYoUl1E6hknxd/84B6yAj5N6tmmlFgkFkw516jAFK6PzAtcc6cIC XlQPgMF0rTfzhNgVZMFPBt2cGYRc5ABd+nB8O6QWD5Ko+Tfku0XqBLYRvyEkYlb5FW poraXgASREN6c7pKHn3vDhGGKfLiHhnyEAY4O+aOHzJLvzIkgYCJXnXJsW8iAgwPB2 MZLqc9XIFLAhDL01TGADSOCTlHTP/xWRs0/crxnJ0Lrlt6DOjEpG21T8PxjWU4ni8k 2FDdEfdyGJI0pynGLHMrry4NF2WaH1bnfw/GQIVNJ0pRZJKsNpAElwOTdhHoLyzzKd 65qBiw666a1cw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A26005C0A21; Fri, 4 Feb 2022 15:08:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Ingo Molnar , "Paul E . McKenney" Subject: [PATCH rcu 7/9] rcu: Uninline multi-use function: finish_rcuwait() Date: Fri, 4 Feb 2022 15:08:03 -0800 Message-Id: <20220204230805.4193767-7-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> References: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Ingo Molnar This is a rarely used function, so uninlining its 3 instructions is probably a win or a wash - but the main motivation is to make independent of task_struct details. Signed-off-by: Ingo Molnar Signed-off-by: Paul E. McKenney --- include/linux/rcuwait.h | 6 +----- kernel/rcu/update.c | 7 +++++++ 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/include/linux/rcuwait.h b/include/linux/rcuwait.h index 61c56cca95c43..8052d34da7826 100644 --- a/include/linux/rcuwait.h +++ b/include/linux/rcuwait.h @@ -47,11 +47,7 @@ static inline void prepare_to_rcuwait(struct rcuwait *w) rcu_assign_pointer(w->task, current); } =20 -static inline void finish_rcuwait(struct rcuwait *w) -{ - rcu_assign_pointer(w->task, NULL); - __set_current_state(TASK_RUNNING); -} +extern void finish_rcuwait(struct rcuwait *w); =20 #define rcuwait_wait_event(w, condition, state) \ ({ \ diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index 156892c22bb56..df6197f0d5ce8 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -407,6 +407,13 @@ void __wait_rcu_gp(bool checktiny, int n, call_rcu_fun= c_t *crcu_array, } EXPORT_SYMBOL_GPL(__wait_rcu_gp); =20 +void finish_rcuwait(struct rcuwait *w) +{ + rcu_assign_pointer(w->task, NULL); + __set_current_state(TASK_RUNNING); +} +EXPORT_SYMBOL_GPL(finish_rcuwait); + #ifdef CONFIG_DEBUG_OBJECTS_RCU_HEAD void init_rcu_head(struct rcu_head *head) { --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65B72C433F5 for ; Fri, 4 Feb 2022 23:08:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377997AbiBDXIe (ORCPT ); Fri, 4 Feb 2022 18:08:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241532AbiBDXIK (ORCPT ); Fri, 4 Feb 2022 18:08:10 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B469FDF99CE8; Fri, 4 Feb 2022 15:08:08 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 50A5561BBB; Fri, 4 Feb 2022 23:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 39906C340FA; Fri, 4 Feb 2022 23:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644016087; bh=qsBtBAeY/gtg63/z/Rr+5Yet5jpG77AmKZjY5rm9uxI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UbjNU04Akt+g/rq/voKRWHq7twSiZMlDEb2jea7U00UleobVCXNurpdX2oL5LGNWn b5owZ3tf5IICDfeWjILkN3OgIfOgutTKlfz8hV4LRIX1JnMVGnKLV5EV8rbdNxJdhj gxPbrC/CakS+3nCbUjnYv5tuDnZBZiP1oBgjcdd7kymx/CD15qU5ilzqFGYNx+224e 6p4+IS10oVataMtjqwdK/BuSIpc6zL8SvTIwkNhTldx3URTetr/Bg+xzQgxR9HILGD bwM6uqEcdlICdgv52Wj7DaJw4csP0NW2ymobfrVLkcucSNqNanfGapuIJB47x+UmPW MVKe1hJJsB0QA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A41635C0A6A; Fri, 4 Feb 2022 15:08:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Ingo Molnar , "Paul E . McKenney" Subject: [PATCH rcu 8/9] rcu: Remove __read_mostly annotations from rcu_scheduler_active externs Date: Fri, 4 Feb 2022 15:08:04 -0800 Message-Id: <20220204230805.4193767-8-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> References: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Ingo Molnar Remove the __read_mostly attributes from the rcu_scheduler_active extern declarations, because these attributes are ignored for prototypes and we'd have to include the full header to gain this functionally pointless attribute defined. Signed-off-by: Ingo Molnar Signed-off-by: Paul E. McKenney --- include/linux/rcupdate.h | 2 +- include/linux/rcutree.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 9d7df8d36af07..e7c39c200e2b0 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -84,7 +84,7 @@ static inline int rcu_preempt_depth(void) =20 /* Internal to kernel */ void rcu_init(void); -extern int rcu_scheduler_active __read_mostly; +extern int rcu_scheduler_active; void rcu_sched_clock_irq(int user); void rcu_report_dead(unsigned int cpu); void rcutree_migrate_callbacks(int cpu); diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 53209d6694001..76665db179fa1 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -62,7 +62,7 @@ static inline void rcu_irq_exit_check_preempt(void) { } void exit_rcu(void); =20 void rcu_scheduler_starting(void); -extern int rcu_scheduler_active __read_mostly; +extern int rcu_scheduler_active; void rcu_end_inkernel_boot(void); bool rcu_inkernel_boot_has_ended(void); bool rcu_is_watching(void); --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:49 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B4A0C433F5 for ; Fri, 4 Feb 2022 23:08:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378005AbiBDXIi (ORCPT ); Fri, 4 Feb 2022 18:08:38 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:58142 "EHLO dfw.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377870AbiBDXII (ORCPT ); Fri, 4 Feb 2022 18:08:08 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4D00561B5F; Fri, 4 Feb 2022 23:08:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33539C340F8; Fri, 4 Feb 2022 23:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644016087; bh=gjHD7UwRzDJPLtTblEcUscAE/WgIgNPhdoq37BeV0NY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BN+G5FV0a9W/VdPvUvOzcZe7dfcK1uFGKsNxkaSeO7pEg+w43hgMYVFdc2W4Gpw2C wyxnVT7TEk9dnADEqhQnDv2TmGZdp2VlqSEuHL70J4u6Zw+v4PhRLoskRpLB+bUth/ ePZQe96G1r5lRfq3bxqoD9bLLSsQGlr10BBfw4zOvrLpV98L5ENQF3f0NcOOZXbEtL zlYJsidT/A//cMB+Q+0K+KA/S367nQeoxBOlGS4W/rNiecKwIdxHRECrUfuidql6/4 WsgV0iYdY0/xyT8Zh3niT/dHa9d22HIk1U/AEIXXNJPp9zf/MB3VR03dATvK6XAvx3 myr4cOGHT3SwQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id A5CCF5C0AEF; Fri, 4 Feb 2022 15:08:06 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Yury Norov , "Paul E . McKenney" Subject: [PATCH rcu 9/9] rcu: Replace cpumask_weight with cpumask_empty where appropriate Date: Fri, 4 Feb 2022 15:08:05 -0800 Message-Id: <20220204230805.4193767-9-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> References: <20220204230751.GA4193671@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Yury Norov In some places, RCU code calls cpumask_weight() to check if any bit of a given cpumask is set. We can do it more efficiently with cpumask_empty() because cpumask_empty() stops traversing the cpumask as soon as it finds first set bit, while cpumask_weight() counts all bits unconditionally. Signed-off-by: Yury Norov Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 4 ++-- kernel/rcu/tree_plugin.h | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index eeafb546a7a09..f83c7b1d61103 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1169,7 +1169,7 @@ void __init rcu_init_nohz(void) struct rcu_data *rdp; =20 #if defined(CONFIG_NO_HZ_FULL) - if (tick_nohz_full_running && cpumask_weight(tick_nohz_full_mask)) + if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask)) need_rcu_nocb_mask =3D true; #endif /* #if defined(CONFIG_NO_HZ_FULL) */ =20 @@ -1348,7 +1348,7 @@ static void __init rcu_organize_nocb_kthreads(void) */ void rcu_bind_current_to_nocb(void) { - if (cpumask_available(rcu_nocb_mask) && cpumask_weight(rcu_nocb_mask)) + if (cpumask_available(rcu_nocb_mask) && !cpumask_empty(rcu_nocb_mask)) WARN_ON(sched_setaffinity(current->pid, rcu_nocb_mask)); } EXPORT_SYMBOL_GPL(rcu_bind_current_to_nocb); diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 07845dcd33c5e..efd0c87d2ffae 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1220,7 +1220,7 @@ static void rcu_boost_kthread_setaffinity(struct rcu_= node *rnp, int outgoingcpu) cpu !=3D outgoingcpu) cpumask_set_cpu(cpu, cm); cpumask_and(cm, cm, housekeeping_cpumask(HK_FLAG_RCU)); - if (cpumask_weight(cm) =3D=3D 0) + if (cpumask_empty(cm)) cpumask_copy(cm, housekeeping_cpumask(HK_FLAG_RCU)); set_cpus_allowed_ptr(t, cm); mutex_unlock(&rnp->boost_kthread_mutex); --=20 2.31.1.189.g2e36527f23