From nobody Mon Jun 29 17:37:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FCFBC433EF for ; Fri, 4 Feb 2022 22:55:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355352AbiBDWzY (ORCPT ); Fri, 4 Feb 2022 17:55:24 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:51522 "EHLO dfw.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355156AbiBDWzK (ORCPT ); Fri, 4 Feb 2022 17:55:10 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BC5DB61AE3; Fri, 4 Feb 2022 22:55:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2D1E2C004E1; Fri, 4 Feb 2022 22:55:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644015309; bh=9dVTciEfhGptcn3wHTdf5MGi0l2fBamxpAPctlU28m4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Jg5LmfB9vKPgT9F4sry4V3UojfJsUFlVavHImbPoT8kVvfzencBqKK92T9XSSKi1i fZvs5q8fMtiSUzwquxLE1//pzIQtGMBKIUYGjxbcMAq6tnlFVWC2fI8b3qkKBd7OYh c5LWh5Uv3iV6d7MYZm14tdaaz1xqaTpyzJAjuEF/GXvuyMTGbjz+vO5TaPXAwWXZB4 B7rZAUaAiPt/LRn6jT71YhJFu+wvT74ml2ri/huqdjqDGaCF9cH5fw3a4VIoxTlGtS os1HAdJyBZDDYCXFFHzTz0kdzKV/pfw4Ax57O1GeslqnRhjlJEIwuuhQDaKrCh0daq iXDf10VKCgDtA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0CC845C0418; Fri, 4 Feb 2022 14:55:09 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Neeraj Upadhyay , Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 1/3] rcu/exp: Fix check for idle context in rcu_exp_handler Date: Fri, 4 Feb 2022 14:55:05 -0800 Message-Id: <20220204225507.4193113-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204225409.GA4193020@paulmck-ThinkPad-P17-Gen-1> References: <20220204225409.GA4193020@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Neeraj Upadhyay For PREEMPT_RCU, the rcu_exp_handler() function checks whether the current CPU is in idle, by calling rcu_dynticks_curr_cpu_in_eqs(). However, rcu_exp_handler() is called in IPI handler context. So, it should be checking the idle context using rcu_is_cpu_rrupt_from_idle(). Fix this by using rcu_is_cpu_rrupt_from_idle() instead of rcu_dynticks_curr_cpu_in_eqs(). Non-preempt configuration already uses the correct check. Reviewed-by: Frederic Weisbecker Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_exp.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 237a79989abae..1568c8ef185b2 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -656,7 +656,7 @@ static void rcu_exp_handler(void *unused) */ if (!depth) { if (!(preempt_count() & (PREEMPT_MASK | SOFTIRQ_MASK)) || - rcu_dynticks_curr_cpu_in_eqs()) { + rcu_is_cpu_rrupt_from_idle()) { rcu_report_exp_rdp(rdp); } else { WRITE_ONCE(rdp->cpu_no_qs.b.exp, true); --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:37:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59415C433F5 for ; Fri, 4 Feb 2022 22:55:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355466AbiBDWzM (ORCPT ); Fri, 4 Feb 2022 17:55:12 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:51526 "EHLO dfw.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355228AbiBDWzK (ORCPT ); Fri, 4 Feb 2022 17:55:10 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D876861AE4; Fri, 4 Feb 2022 22:55:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3E127C340ED; Fri, 4 Feb 2022 22:55:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644015309; bh=P+ojzxqenLejeCItnmyBFPt2VGf6j96fThZ8UVIBRYw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DHKmctannmKHUrJzHzmcO/yPgn8p9ryi7B1Cs5nyx66lpnHGpzex8Xr5W/Nh2Wi+U jQWn5RHlkeLf3FsrA4RrhVBqVwO6gHGwLu71Q6J22z4bg+NM9SAq83aNSsXdUPUi5Q NXjsIffO2T14PbS7gTqkM91g7yr63CjtUoHeIzB5cKexQ9fI3AjadQRUNrgctJruwJ V2lx0k8r31B8hCIB+tLzjajKdLk30pZ/zofAqrVAZScOOtt/aVEymNKuiyzLfgVzVd cvTSsZNLTGUHe9Szdr4QYbTqzEo5MXmG44VkgDvf2Yi8SjuBbm/EAUYNHjJ+CSIzCV NS+BacmVFB4KA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 0EF305C0829; Fri, 4 Feb 2022 14:55:09 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 2/3] rcu: Mark ->expmask access in synchronize_rcu_expedited_wait() Date: Fri, 4 Feb 2022 14:55:06 -0800 Message-Id: <20220204225507.4193113-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204225409.GA4193020@paulmck-ThinkPad-P17-Gen-1> References: <20220204225409.GA4193020@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This commit adds a READ_ONCE() to an access to the rcu_node structure's ->expmask field to prevent compiler mischief. Detected by KCSAN. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_exp.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 1568c8ef185b2..60197ea24ceb9 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -502,7 +502,8 @@ static void synchronize_rcu_expedited_wait(void) if (synchronize_rcu_expedited_wait_once(1)) return; rcu_for_each_leaf_node(rnp) { - for_each_leaf_node_cpu_mask(rnp, cpu, rnp->expmask) { + mask =3D READ_ONCE(rnp->expmask); + for_each_leaf_node_cpu_mask(rnp, cpu, mask) { rdp =3D per_cpu_ptr(&rcu_data, cpu); if (rdp->rcu_forced_tick_exp) continue; --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:37:21 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C0B7C433EF for ; Fri, 4 Feb 2022 22:55:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355735AbiBDWz1 (ORCPT ); Fri, 4 Feb 2022 17:55:27 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:46782 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355287AbiBDWzL (ORCPT ); Fri, 4 Feb 2022 17:55:11 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 76F34B83960; Fri, 4 Feb 2022 22:55:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 44340C340F0; Fri, 4 Feb 2022 22:55:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644015309; bh=a6SxQDd5Q2UNK1R/nHemOtR3mPTlyCL9AXpjqkoI1uA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eQ5ZMo6pt9WmIQAmgSW1KTmUxCdabjoxv3BD4OMbNJYW5zRQlL+EHLkTqXcxT8IFq CRRp3I+A45lPcj0gIJGR8tmujr6Vrg/RlNHL6XtCe65ftue2B5+JFXNnGmEqRsHdAP Iaumqvp0Pf86A5OtpwvZUpRO2GDbaJygPuJU/F8a3uJ7BmU6kPRfJ9VjB3flLmi7U+ fyscc4gpXx5rwed7JLauQafTqhSn/3n4xrgorYOTrnQlcrEXymHX8IabiqzjifJUQY adF2EFZ7NvbcpefyCW+2z4ye3MUPva4Mc0Ao+SDZohG5ddf0zU1VVSnWdT/wOhhJAu ARFiYXukyIhiA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 1110A5C08B7; Fri, 4 Feb 2022 14:55:09 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Mukesh Ojha , Tejun Heo Subject: [PATCH rcu 3/3] rcu: Allow expedited RCU grace periods on incoming CPUs Date: Fri, 4 Feb 2022 14:55:07 -0800 Message-Id: <20220204225507.4193113-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204225409.GA4193020@paulmck-ThinkPad-P17-Gen-1> References: <20220204225409.GA4193020@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Although it is usually safe to invoke synchronize_rcu_expedited() from a preemption-enabled CPU-hotplug notifier, if it is invoked from a notifier between CPUHP_AP_RCUTREE_ONLINE and CPUHP_AP_ACTIVE, its attempts to invoke a workqueue handler will hang due to RCU waiting on a CPU that the scheduler is not paying attention to. This commit therefore expands use of the existing workqueue-independent synchronize_rcu_expedited() from early boot to also include CPUs that are being hotplugged. Link: https://lore.kernel.org/lkml/7359f994-8aaf-3cea-f5cf-c0d3929689d6@qui= cinc.com/ Reported-by: Mukesh Ojha Cc: Tejun Heo Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_exp.h | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 60197ea24ceb9..1a45667402260 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -816,7 +816,7 @@ static int rcu_print_task_exp_stall(struct rcu_node *rn= p) */ void synchronize_rcu_expedited(void) { - bool boottime =3D (rcu_scheduler_active =3D=3D RCU_SCHEDULER_INIT); + bool no_wq; struct rcu_exp_work rew; struct rcu_node *rnp; unsigned long s; @@ -841,9 +841,15 @@ void synchronize_rcu_expedited(void) if (exp_funnel_lock(s)) return; /* Someone else did our work for us. */ =20 + /* Don't use workqueue during boot or from an incoming CPU. */ + preempt_disable(); + no_wq =3D rcu_scheduler_active =3D=3D RCU_SCHEDULER_INIT || + !cpumask_test_cpu(smp_processor_id(), cpu_active_mask); + preempt_enable(); + /* Ensure that load happens before action based on it. */ - if (unlikely(boottime)) { - /* Direct call during scheduler init and early_initcalls(). */ + if (unlikely(no_wq)) { + /* Direct call for scheduler init, early_initcall()s, and incoming CPUs.= */ rcu_exp_sel_wait_wake(s); } else { /* Marshall arguments & schedule the expedited grace period. */ @@ -861,7 +867,7 @@ void synchronize_rcu_expedited(void) /* Let the next expedited grace period start. */ mutex_unlock(&rcu_state.exp_mutex); =20 - if (likely(!boottime)) + if (likely(!no_wq)) destroy_work_on_stack(&rew.rew_work); } EXPORT_SYMBOL_GPL(synchronize_rcu_expedited); --=20 2.31.1.189.g2e36527f23