From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91478C433FE for ; Fri, 4 Feb 2022 23:24:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378129AbiBDXYO (ORCPT ); Fri, 4 Feb 2022 18:24:14 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:45774 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231625AbiBDXYL (ORCPT ); Fri, 4 Feb 2022 18:24:11 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 9C4DCCE24BD; Fri, 4 Feb 2022 23:24:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E5082C004E1; Fri, 4 Feb 2022 23:24:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017047; bh=om82ARS4XbnxrZfOOyNn+KdAReXpStBVl0i5dOwAfCc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eH9L3qfUwFpU1qxP9tcvZ3+1JxgGPh2mh8iXH8s0bpgBkH1tdPxemeEil2VwEVOjC HdGLUveCUhPyU2x4av3UXkvEtNC+XVRwJH59rftrQxBYhMhNwyiO1YMLsS1Z6iJPwv ts9IzhUNJxm/d3QROQOqEL2dxfDK3cJSAa11DrnC17j2CxiE1sRwoR23GELmAIBenN CI2xbR+R6iLc+boGgk2+xrrxGuFahMB0jDG77hJvp9lA+lGBAVtXp5HuMUb2cODoot HGAKbNWurGw9NKub3JMNzT1zxeZuxT3FDflZ7GMhL5X2+zBVUuKj3GNFUn/AibDYdO 3UuL85/84bhrw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id BDB145C0418; Fri, 4 Feb 2022 15:24:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 01/10] rcu: Mark accesses to boost_starttime Date: Fri, 4 Feb 2022 15:23:57 -0800 Message-Id: <20220204232406.814-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> References: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The boost_starttime shared variable has conflicting unmarked C-language accesses, which are dangerous at best. This commit therefore adds appropriate marking. This was found by KCSAN. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcutorture.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 422f7e4cc08de..829ae0b7d3c04 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -997,7 +997,7 @@ static int rcu_torture_boost(void *arg) goto checkwait; =20 /* Wait for the next test interval. */ - oldstarttime =3D boost_starttime; + oldstarttime =3D READ_ONCE(boost_starttime); while (time_before(jiffies, oldstarttime)) { schedule_timeout_interruptible(oldstarttime - jiffies); if (stutter_wait("rcu_torture_boost")) @@ -1041,10 +1041,11 @@ static int rcu_torture_boost(void *arg) * interval. Besides, we are running at RT priority, * so delays should be relatively rare. */ - while (oldstarttime =3D=3D boost_starttime && !kthread_should_stop()) { + while (oldstarttime =3D=3D READ_ONCE(boost_starttime) && !kthread_should= _stop()) { if (mutex_trylock(&boost_mutex)) { if (oldstarttime =3D=3D boost_starttime) { - boost_starttime =3D jiffies + test_boost_interval * HZ; + WRITE_ONCE(boost_starttime, + jiffies + test_boost_interval * HZ); n_rcu_torture_boosts++; } mutex_unlock(&boost_mutex); --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4DABC433EF for ; Fri, 4 Feb 2022 23:24:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378145AbiBDXYU (ORCPT ); Fri, 4 Feb 2022 18:24:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378110AbiBDXYL (ORCPT ); Fri, 4 Feb 2022 18:24:11 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1722DFC13F7; Fri, 4 Feb 2022 15:24:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4D938B83971; Fri, 4 Feb 2022 23:24:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F2871C340ED; Fri, 4 Feb 2022 23:24:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017048; bh=xnhPzJ/bmGOHPfjmOwYiKQqr+09QnV28RTS3sYV19ns=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JrysNh470JefdMAGjbfzB129wKp9vEiypUnHUgLKNUqqKoMkUtiDXg6Hbr/Vs+2Hn t/xK4m+odPRNoohWnMd5v05VtySKS11E+kwCDTpo3ICuqaoqIMevwcgeunFP/i5Adl oHSoZW0No9Z2pWp7BkbOLiAH4FPsE8HSPQzat74e2ZP4rUc5yHzxOq9BD9wI2l/3pd +7NF8ffahkZCG/f6L6NI0XGyjQsUvUk1nAZZyedOJcG1i79LQmSqYn9dflPydIj5Qy zO7SFvByANgwlEdtpyAsHu59ui84OGyOVbYe/KUDODnaTroJU/AhyN+wpM/njJ7WRG d8B8qXW5f0Ipw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C1AE55C0829; Fri, 4 Feb 2022 15:24:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Neeraj Upadhyay , David Woodhouse , "Paul E . McKenney" Subject: [PATCH rcu 02/10] rcu/nocb: Handle concurrent nocb kthreads creation Date: Fri, 4 Feb 2022 15:23:58 -0800 Message-Id: <20220204232406.814-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> References: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Neeraj Upadhyay When multiple CPUs in the same nocb gp/cb group concurrently come online, they might try to concurrently create the same rcuog kthread. Fix this by using nocb gp CPU's spawn mutex to provide mutual exclusion for the rcuog kthread creation code. [ paulmck: Whitespace fixes per kernel test robot feedback. ] Acked-by: David Woodhouse Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.h | 2 ++ kernel/rcu/tree_nocb.h | 7 ++++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 486fc901bd085..24dd4b0d805f1 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -203,6 +203,8 @@ struct rcu_data { int nocb_defer_wakeup; /* Defer wakeup of nocb_kthread. */ struct timer_list nocb_timer; /* Enforce finite deferral. */ unsigned long nocb_gp_adv_time; /* Last call_rcu() CB adv (jiffies). */ + struct mutex nocb_gp_kthread_mutex; /* Exclusion for nocb gp kthread */ + /* spawning */ =20 /* The following fields are used by call_rcu, hence own cacheline. */ raw_spinlock_t nocb_bypass_lock ____cacheline_internodealigned_in_smp; diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index eeafb546a7a09..1e40519d1a05a 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1226,6 +1226,7 @@ static void __init rcu_boot_init_nocb_percpu_data(str= uct rcu_data *rdp) raw_spin_lock_init(&rdp->nocb_gp_lock); timer_setup(&rdp->nocb_timer, do_nocb_deferred_wakeup_timer, 0); rcu_cblist_init(&rdp->nocb_bypass); + mutex_init(&rdp->nocb_gp_kthread_mutex); } =20 /* @@ -1248,13 +1249,17 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu) =20 /* If we didn't spawn the GP kthread first, reorganize! */ rdp_gp =3D rdp->nocb_gp_rdp; + mutex_lock(&rdp_gp->nocb_gp_kthread_mutex); if (!rdp_gp->nocb_gp_kthread) { t =3D kthread_run(rcu_nocb_gp_kthread, rdp_gp, "rcuog/%d", rdp_gp->cpu); - if (WARN_ONCE(IS_ERR(t), "%s: Could not start rcuo GP kthread, OOM is no= w expected behavior\n", __func__)) + if (WARN_ONCE(IS_ERR(t), "%s: Could not start rcuo GP kthread, OOM is no= w expected behavior\n", __func__)) { + mutex_unlock(&rdp_gp->nocb_gp_kthread_mutex); return; + } WRITE_ONCE(rdp_gp->nocb_gp_kthread, t); } + mutex_unlock(&rdp_gp->nocb_gp_kthread_mutex); =20 /* Spawn the kthread for this CPU. */ t =3D kthread_run(rcu_nocb_cb_kthread, rdp, --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBD3CC433F5 for ; Fri, 4 Feb 2022 23:24:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378151AbiBDXYR (ORCPT ); Fri, 4 Feb 2022 18:24:17 -0500 Received: from sin.source.kernel.org ([145.40.73.55]:45782 "EHLO sin.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378103AbiBDXYL (ORCPT ); Fri, 4 Feb 2022 18:24:11 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id B08A8CE24C1; Fri, 4 Feb 2022 23:24:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 02851C340EF; Fri, 4 Feb 2022 23:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017048; bh=lXaKWM2nPsvGaA4M3xknAyaLNPy151HXLW17hO5rkR0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QnImF1IPtt2zc7pNIiSc2VDYAKsr5iB+XuE8GMXgIErTZEXiwud2gsJLyx82dKXSJ gr5xTuvTk3XW7wT32rpThhKRiTROaVolrxdCn2HnTeQn9gmHJpp7T5b4R9W4UXtzDo chgLeBoJvXe711StkHFdLSA7zfQ92UsTxAq7hhbIbu8ap5YHXXfUDvSwXccd+980Nn SVtPd+GXllM9hQQ1yEL36+SOyBAOoMmvRY4G8xzujEKG6p2ikTSS7s79+QmEvw9+5W 3lT9H2Wmntc1Wxfz0Bv7dOv8ueWgry79JhnhLpG/K6iKm/NIInF7PDsdkYHpF9ITm7 ILyofZD1mdihA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C3B355C08B7; Fri, 4 Feb 2022 15:24:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Neeraj Upadhyay , "Paul E . McKenney" Subject: [PATCH rcu 03/10] rcu: Remove unused rcu_state.boost Date: Fri, 4 Feb 2022 15:23:59 -0800 Message-Id: <20220204232406.814-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> References: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Neeraj Upadhyay Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.h | 5 ++--- kernel/rcu/tree_plugin.h | 2 -- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 24dd4b0d805f1..e9990945483f1 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -304,9 +304,8 @@ struct rcu_state { =20 /* The following fields are guarded by the root rcu_node's lock. */ =20 - u8 boost ____cacheline_internodealigned_in_smp; - /* Subject to priority boost. */ - unsigned long gp_seq; /* Grace-period sequence #. */ + unsigned long gp_seq ____cacheline_internodealigned_in_smp; + /* Grace-period sequence #. */ unsigned long gp_max; /* Maximum GP duration in */ /* jiffies. */ struct task_struct *gp_kthread; /* Task for grace periods. */ diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index c5b45c2f68a15..109429e70a642 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1175,8 +1175,6 @@ static void rcu_spawn_one_boost_kthread(struct rcu_no= de *rnp) if (rnp->boost_kthread_task || !rcu_scheduler_fully_active) return; =20 - rcu_state.boost =3D 1; - t =3D kthread_create(rcu_boost_kthread, (void *)rnp, "rcub/%d", rnp_index); if (WARN_ON_ONCE(IS_ERR(t))) --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C0C5C4332F for ; Fri, 4 Feb 2022 23:24:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378117AbiBDXYL (ORCPT ); Fri, 4 Feb 2022 18:24:11 -0500 Received: from ams.source.kernel.org ([145.40.68.75]:60958 "EHLO ams.source.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346197AbiBDXYK (ORCPT ); Fri, 4 Feb 2022 18:24:10 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 553CFB83973; Fri, 4 Feb 2022 23:24:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07821C340F0; Fri, 4 Feb 2022 23:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017048; bh=EnUP4evSm9pHbY5I/gW2mI3yvHr8E8M9zkokamOt2mo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LkD/mAKjCco9ffJfgdLcTqIeY8837S5n7V0mSJLhZ5dsqX6t4FeLKLlgU8O/XBU5p n8c0AiSdU13zdV5Hz6XX9z51F30/yMvu5CZUS1z0N4M+lDvHP+L9Yv0L+1wnoqLABI /e7Lq+VRyGbyMbfXg80LR366FfwpyV++HAZivjWsQ91LyPXB9sa0r4FZJjvpwgkpnH m0gb5rdpkuQqvLfpV8dN2/TzE2UbWn54dtBK2bR3D6L7DHWe8xCgJYF+jps5LBsC2r F52z5iD4RB7gs8TD0AXhZdnXH2cK4cW2NdHweTjegJiCqNoyFNu6diGEGcg15yurTH SHmH7tjNEhB7g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C58AB5C08DD; Fri, 4 Feb 2022 15:24:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Zqiang , Sebastian Andrzej Siewior , "Paul E . McKenney" Subject: [PATCH rcu 04/10] rcu: Create per-cpu rcuc kthreads only when rcutree.use_softirq=0 Date: Fri, 4 Feb 2022 15:24:00 -0800 Message-Id: <20220204232406.814-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> References: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Zqiang The per-CPU "rcuc" kthreads are used only by kernels booted with rcutree.use_softirq=3D0, but they are nevertheless unconditionally created by kernels built with CONFIG_RCU_BOOST=3Dy. This results in "rcuc" kthreads being created that are never actually used. This commit therefore refrains from creating these kthreads unless the kernel is actually booted with rcutree.use_softirq=3D0. Acked-by: Sebastian Andrzej Siewior Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index a4c25a6283b0b..4e5e37e5ee3c9 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2894,7 +2894,7 @@ static int __init rcu_spawn_core_kthreads(void) =20 for_each_possible_cpu(cpu) per_cpu(rcu_data.rcu_cpu_has_work, cpu) =3D 0; - if (!IS_ENABLED(CONFIG_RCU_BOOST) && use_softirq) + if (use_softirq) return 0; WARN_ONCE(smpboot_register_percpu_thread(&rcu_cpu_thread_spec), "%s: Could not start rcuc kthread, OOM is now expected behavior\n", __= func__); --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F2A0C433F5 for ; Fri, 4 Feb 2022 23:24:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378193AbiBDXY0 (ORCPT ); Fri, 4 Feb 2022 18:24:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346197AbiBDXYM (ORCPT ); Fri, 4 Feb 2022 18:24:12 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BB06DFC13FC; Fri, 4 Feb 2022 15:24:11 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id BB447CE24C2; Fri, 4 Feb 2022 23:24:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 09A1CC36AE2; Fri, 4 Feb 2022 23:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017048; bh=2W4MLM7m8I6vA/YIMpHFfSjn+OnxWplMJIEA37I40k8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=flUuBFwvI4fLaQDspE1Sab5DOwLJiY5hHXQnfrf7d7Nxfx6oAY5dMvW2eY6p23hZk 7ed3S1kFjkaU/Qom8jYy0zwgtg7u/anfmWCE3Yy7WdSSN4dak2RjV4CK45U+X2pbDs 5G2lkbVPr/TXXccd/pFZ1GJC3OyV3MLZEvnlCeY1zixz1oB1GpSRoeJyEQDvxYi3a0 QScKepFutglBWAvqtbMEYeJdQdaNSoVXfPA8PSxpQWH/TrokdwK0tNknzCaj1MBxaQ fKyhy0xEaSLYabrVehm/BfRw5t7DfWJtD6rK5+q6sEFmCxvb071jEkyvDYTYs9OQTF zD59ojt8CWZlw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C76395C0992; Fri, 4 Feb 2022 15:24:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Alison Chaiken , "Paul E . McKenney" Subject: [PATCH rcu 05/10] rcu: Move kthread_prio bounds-check to a separate function Date: Fri, 4 Feb 2022 15:24:01 -0800 Message-Id: <20220204232406.814-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> References: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alison Chaiken Move the bounds-check of the kthread_prio cmdline parameter to a new function in order to faciliate a different callsite. Signed-off-by: Alison Chaiken Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 39 +++++++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 4e5e37e5ee3c9..5bf0312f66760 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4440,26 +4440,10 @@ static int rcu_pm_notify(struct notifier_block *sel= f, static int __init rcu_spawn_gp_kthread(void) { unsigned long flags; - int kthread_prio_in =3D kthread_prio; struct rcu_node *rnp; struct sched_param sp; struct task_struct *t; =20 - /* Force priority into range. */ - if (IS_ENABLED(CONFIG_RCU_BOOST) && kthread_prio < 2 - && IS_BUILTIN(CONFIG_RCU_TORTURE_TEST)) - kthread_prio =3D 2; - else if (IS_ENABLED(CONFIG_RCU_BOOST) && kthread_prio < 1) - kthread_prio =3D 1; - else if (kthread_prio < 0) - kthread_prio =3D 0; - else if (kthread_prio > 99) - kthread_prio =3D 99; - - if (kthread_prio !=3D kthread_prio_in) - pr_alert("rcu_spawn_gp_kthread(): Limited prio to %d from %d\n", - kthread_prio, kthread_prio_in); - rcu_scheduler_fully_active =3D 1; t =3D kthread_create(rcu_gp_kthread, NULL, "%s", rcu_state.name); if (WARN_ONCE(IS_ERR(t), "%s: Could not start grace-period kthread, OOM i= s now expected behavior\n", __func__)) @@ -4584,6 +4568,28 @@ static void __init rcu_init_one(void) } } =20 +/* + * Force priority from the kernel command-line into range. + */ +static void __init sanitize_kthread_prio(void) +{ + int kthread_prio_in =3D kthread_prio; + + if (IS_ENABLED(CONFIG_RCU_BOOST) && kthread_prio < 2 + && IS_BUILTIN(CONFIG_RCU_TORTURE_TEST)) + kthread_prio =3D 2; + else if (IS_ENABLED(CONFIG_RCU_BOOST) && kthread_prio < 1) + kthread_prio =3D 1; + else if (kthread_prio < 0) + kthread_prio =3D 0; + else if (kthread_prio > 99) + kthread_prio =3D 99; + + if (kthread_prio !=3D kthread_prio_in) + pr_alert("%s: Limited prio to %d from %d\n", + __func__, kthread_prio, kthread_prio_in); +} + /* * Compute the rcu_node tree geometry from kernel parameters. This cannot * replace the definitions in tree.h because those are needed to size @@ -4744,6 +4750,7 @@ void __init rcu_init(void) =20 kfree_rcu_batch_init(); rcu_bootup_announce(); + sanitize_kthread_prio(); rcu_init_geometry(); rcu_init_one(); if (dump_tree) --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C23C6C433EF for ; Fri, 4 Feb 2022 23:24:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378228AbiBDXYc (ORCPT ); Fri, 4 Feb 2022 18:24:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378126AbiBDXYN (ORCPT ); Fri, 4 Feb 2022 18:24:13 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2053DFC13F7; Fri, 4 Feb 2022 15:24:12 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 2A081CE24BF; Fri, 4 Feb 2022 23:24:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 30028C340F6; Fri, 4 Feb 2022 23:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017048; bh=LeUpGVYNzDBxeiIoq3od/ZlEdAcMblILIHqxQOTVUVk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Feo7pgOZuZYf3v3lCGnTWyIC3IpqVgkWMiJCIPT9ll+FR3QukYD39DYeA3ewKE4cP pH7uxc62Lw/aFNXvcZzWxlAthCTGQ7sqreBiUyzbSw/lCRjUPCeEScw0x7+eItzUTf f13gxb/rnc2wqB9qWw4384rru6v7YE+oMOY1Er5WxUDl2aMZ0eHinIzf/McaxjCCMC pijg3fFVZ/WnHNOifM9AmSOkdnmr/4H1Qg00Eu4Xa5lzFGVFEnt6cwoqAR7asqAUzQ iHNxFx2EMk3u9PaAjVfjBl3QUbgsXhgmt2oKfwfi11xnAKkMh0ThNnxxcrkWWaLyQ/ XtQXjWoO4NPHQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id C94C05C0A0A; Fri, 4 Feb 2022 15:24:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Alison Chaiken , "Paul E . McKenney" Subject: [PATCH rcu 06/10] rcu: Make priority of grace-period thread consistent Date: Fri, 4 Feb 2022 15:24:02 -0800 Message-Id: <20220204232406.814-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> References: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alison Chaiken The priority of RCU grace period threads is set to kthread_prio when they are launched from rcu_spawn_gp_kthread(). The same is not true of rcu_spawn_one_nocb_kthread(). Accordingly, add priority elevation to rcu_spawn_one_nocb_kthread(). Signed-off-by: Alison Chaiken Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 1e40519d1a05a..ea889cbfc3b95 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1239,6 +1239,7 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu) struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); struct rcu_data *rdp_gp; struct task_struct *t; + struct sched_param sp; =20 if (!rcu_scheduler_fully_active || !rcu_nocb_is_setup) return; @@ -1248,6 +1249,7 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu) return; =20 /* If we didn't spawn the GP kthread first, reorganize! */ + sp.sched_priority =3D kthread_prio; rdp_gp =3D rdp->nocb_gp_rdp; mutex_lock(&rdp_gp->nocb_gp_kthread_mutex); if (!rdp_gp->nocb_gp_kthread) { @@ -1258,6 +1260,8 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu) return; } WRITE_ONCE(rdp_gp->nocb_gp_kthread, t); + if (kthread_prio) + sched_setscheduler_nocheck(t, SCHED_FIFO, &sp); } mutex_unlock(&rdp_gp->nocb_gp_kthread_mutex); =20 --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75930C433F5 for ; Fri, 4 Feb 2022 23:24:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378215AbiBDXY3 (ORCPT ); Fri, 4 Feb 2022 18:24:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378119AbiBDXYM (ORCPT ); Fri, 4 Feb 2022 18:24:12 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11D0EDFC13F7; Fri, 4 Feb 2022 15:24:12 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C11E9B83976; Fri, 4 Feb 2022 23:24:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 31E16C340F7; Fri, 4 Feb 2022 23:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017048; bh=VXP8VoUHOvneEifMGVYTGH44aYLtxxiEIdVb15Xe2Eg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eLojOQH9n0ljY6gSgQseyW6vKL294dJgpjiOU4Vxt3ueIcnQvB1/GnTx6VGT0EQC2 un7xVli6wtNGgs8tprBm1dB4y9opDKU51K7ZPFkLQwiUaMJ7oGUd+fP2k8MsV2CKMp WRBiF5KLV9SVyKsh/ipxKGnKQnOPvwSoHwFzFShpEz6/njukgYdUNM+Dk8bjOhcvrn G633QHVzGajPXPnoyU+15tGf9bY5awneHhqyr5kfm18T743u4vmBsR0GS6hnRhYhL6 z9zi60UVvtPEicPT8aQtf8gqtIK2nMPp7SI9POjjYfx91Zf7mcJyIcrV+ocaWlAKtu n9ZR3J7p4oQsg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id CB71D5C0A21; Fri, 4 Feb 2022 15:24:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Alison Chaiken , "Paul E . McKenney" Subject: [PATCH rcu 07/10] rcu: Elevate priority of offloaded callback threads Date: Fri, 4 Feb 2022 15:24:03 -0800 Message-Id: <20220204232406.814-7-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> References: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alison Chaiken When CONFIG_PREEMPT_RT=3Dy, the rcutree.kthread_prio command-line parameter signals initialization code to boost the priority of rcuc callbacks to the designated value. With the additional CONFIG_RCU_NOCB_CPU=3Dy configuration and an additional rcu_nocbs command-line parameter, the callbacks on the listed cores are offloaded to new rcuop kthreads that are not pinned to the cores whose post-grace-period work is performed. While the rcuop kthreads perform the same function as the rcuc kthreads they offload, the kthread_prio parameter only boosts the priority of the rcuc kthreads. Fix this inconsistency by elevating rcuop kthreads to the same priority as the rcuc kthreads. Signed-off-by: Alison Chaiken Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 +- kernel/rcu/tree_nocb.h | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5bf0312f66760..9e4c5b281f003 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -153,7 +153,7 @@ static void sync_sched_exp_online_cleanup(int cpu); static void check_cb_ovld_locked(struct rcu_data *rdp, struct rcu_node *rn= p); static bool rcu_rdp_is_offloaded(struct rcu_data *rdp); =20 -/* rcuc/rcub kthread realtime priority */ +/* rcuc/rcub/rcuop kthread realtime priority */ static int kthread_prio =3D IS_ENABLED(CONFIG_RCU_BOOST) ? 1 : 0; module_param(kthread_prio, int, 0444); =20 diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index ea889cbfc3b95..547c41437c767 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1270,6 +1270,9 @@ static void rcu_spawn_cpu_nocb_kthread(int cpu) "rcuo%c/%d", rcu_state.abbr, cpu); if (WARN_ONCE(IS_ERR(t), "%s: Could not start rcuo CB kthread, OOM is now= expected behavior\n", __func__)) return; + + if (kthread_prio) + sched_setscheduler_nocheck(t, SCHED_FIFO, &sp); WRITE_ONCE(rdp->nocb_cb_kthread, t); WRITE_ONCE(rdp->nocb_gp_kthread, rdp_gp->nocb_gp_kthread); } --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FA82C433F5 for ; Fri, 4 Feb 2022 23:24:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378162AbiBDXYh (ORCPT ); Fri, 4 Feb 2022 18:24:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378125AbiBDXYN (ORCPT ); Fri, 4 Feb 2022 18:24:13 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF346DFC1801; Fri, 4 Feb 2022 15:24:12 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 0EE14CE24C4; Fri, 4 Feb 2022 23:24:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3BCB2C340F8; Fri, 4 Feb 2022 23:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017048; bh=h409ldMmqmMfyS/cakSSPSAQvvvTJuo4yTyqJxPa/So=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AU+5hKZHKnlUjDLvaWr9xDeQL6JY71dY3zhQ8bhk/aowznbMi1Ugjm8NRwQhq/zS3 PIcgo/GtqXCMKN3xB8vj/JvVvtitUDiy3JaAhtnl7sOOvInzfy+L1Vy40Yx0ExfCFU OFvBlQmHa6Po+MRQbTkHjPT3h5CsvrFGdCnrU5XHSbY4mPDreQojVIEaqlOQgZnNR5 R2p8vNg4/AXk9b/NJSpVsdP5PeTEOloB2P1COs07oYWhPN2nO2nSR2psoLZ3V4WtTo 6AwizYYZbp/USZCedFoEVeeKWHuHhuMXWGA5z3jdAb4QRmdBKVi9MhFqiDqnS4gWm8 cnK/jU+0mxgQQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id CD3945C0A6A; Fri, 4 Feb 2022 15:24:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Alison Chaiken , "Paul E . McKenney" Subject: [PATCH rcu 08/10] rcu: Update documentation regarding kthread_prio cmdline parameter Date: Fri, 4 Feb 2022 15:24:04 -0800 Message-Id: <20220204232406.814-8-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> References: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Alison Chaiken Inform readers that the priority of RCU no-callback threads will also be boosted. Signed-off-by: Alison Chaiken Signed-off-by: Paul E. McKenney --- Documentation/admin-guide/kernel-parameters.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index f5a27f067db9e..8e2e65122b993 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4504,6 +4504,8 @@ (the least-favored priority). Otherwise, when RCU_BOOST is not set, valid values are 0-99 and the default is zero (non-realtime operation). + When RCU_NOCB_CPU is set, also adjust the + priority of NOCB callback kthreads. =20 rcutree.rcu_nocb_gp_stride=3D [KNL] Set the number of NOCB callback kthreads in --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8001EC433F5 for ; Fri, 4 Feb 2022 23:24:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378270AbiBDXYl (ORCPT ); Fri, 4 Feb 2022 18:24:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378131AbiBDXYO (ORCPT ); Fri, 4 Feb 2022 18:24:14 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E85FFDFC13FC; Fri, 4 Feb 2022 15:24:12 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 31C38CE24C5; Fri, 4 Feb 2022 23:24:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3EFB6C340FA; Fri, 4 Feb 2022 23:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017048; bh=JV+QYVqegQ9o52kCXFiEB9n9SUhqCcXNcpjWqfXua+M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B6Qjk6ScEl/hJ3/TxQCYUyLdoPTGafyqax0qSGcvEQGAeErtwXX7A1Ca7j800prlo rWzlDkmo1osvzqQECSYmFrweK5t9xlSWykuGRGgHyrW2iOigJaUzYG7DPHLDeqZ6eW eUGlMCXUzBVlc59asHeZD+8l47SF4rZv14t0Cq/SF+r3GfIbkF98uqfH3OwQ0k/x05 orZxGKndEskRpSBryalm3EqW82dKNeFSHMWYxZD5p68201fDAJbZ5KjS4zlWM2/jR7 jD2puXGxQRqUzfUgrxXDq16BVjaDpTALRlmBaJhpxRVRhe4DQ7Iie0iixYWFsYclY6 68s+1HnzAuJWA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id CF1DC5C0AEF; Fri, 4 Feb 2022 15:24:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Tim Murray , Joel Fernandes , Neeraj Upadhyay , Uladzislau Rezki , Todd Kjos , Sandeep Patil , stable@vger.kernel.org Subject: [PATCH rcu 09/10] rcu: Don't deboost before reporting expedited quiescent state Date: Fri, 4 Feb 2022 15:24:05 -0800 Message-Id: <20220204232406.814-9-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> References: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently rcu_preempt_deferred_qs_irqrestore() releases rnp->boost_mtx before reporting the expedited quiescent state. Under heavy real-time load, this can result in this function being preempted before the quiescent state is reported, which can in turn prevent the expedited grace period from completing. Tim Murray reports that the resulting expedited grace periods can take hundreds of milliseconds and even more than one second, when they should normally complete in less than a millisecond. This was fine given that there were no particular response-time constraints for synchronize_rcu_expedited(), as it was designed for throughput rather than latency. However, some users now need sub-100-millisecond response-time constratints. This patch therefore follows Neeraj's suggestion (seconded by Tim and by Uladzislau Rezki) of simply reversing the two operations. Reported-by: Tim Murray Reported-by: Joel Fernandes Reported-by: Neeraj Upadhyay Reviewed-by: Neeraj Upadhyay Reviewed-by: Uladzislau Rezki (Sony) Tested-by: Tim Murray Cc: Todd Kjos Cc: Sandeep Patil Cc: # 5.4.x Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_plugin.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 109429e70a642..02ac057ba3f83 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -556,16 +556,16 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct= *t, unsigned long flags) raw_spin_unlock_irqrestore_rcu_node(rnp, flags); } =20 - /* Unboost if we were boosted. */ - if (IS_ENABLED(CONFIG_RCU_BOOST) && drop_boost_mutex) - rt_mutex_futex_unlock(&rnp->boost_mtx.rtmutex); - /* * If this was the last task on the expedited lists, * then we need to report up the rcu_node hierarchy. */ if (!empty_exp && empty_exp_now) rcu_report_exp_rnp(rnp, true); + + /* Unboost if we were boosted. */ + if (IS_ENABLED(CONFIG_RCU_BOOST) && drop_boost_mutex) + rt_mutex_futex_unlock(&rnp->boost_mtx.rtmutex); } else { local_irq_restore(flags); } --=20 2.31.1.189.g2e36527f23 From nobody Mon Jun 29 17:36:57 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA239C433EF for ; Fri, 4 Feb 2022 23:24:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378173AbiBDXYo (ORCPT ); Fri, 4 Feb 2022 18:24:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378132AbiBDXYO (ORCPT ); Fri, 4 Feb 2022 18:24:14 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14323DFC1802; Fri, 4 Feb 2022 15:24:13 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 631BFCE24C7; Fri, 4 Feb 2022 23:24:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 47F89C340F9; Fri, 4 Feb 2022 23:24:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644017048; bh=mMQC49huRwtBjDgnnc/hOD/p12a3Ew3u3sUoYjyRRDk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i1ksoy/EeSrokEKzsbaaIoEqUxGZYddBUlWaVRa/JzOfrP/ejrTGjeI/YSOZy3Hi9 26s2YQYLlR14Zb6b+bdStjwh2r6CIHLbvwLMGxfMP8af3u6ff1PSETCtRJp6youaJX 7/eRDhRzRHq9BLAqQRb3VwI+7UrxesZ1L6J/QEGNhStwnscxcSb4ptc7jaySy2KUFK NpiuJH8i988wUh+NLkBZ1IQG7DAnQ4akmGkD2eprcg61fb/G/i0Ok9xBEX6OPKDqEi CLBfAR37xbdjyk0+Jdfr3gZYuAPwPeqQYkhQmiRTyv/lcrnNWhZimoBXvmj/Iq89wZ lIydMp1zf61EQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D0ED45C0BCB; Fri, 4 Feb 2022 15:24:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Zqiang , Ammar Faizi , "Paul E . McKenney" Subject: [PATCH rcu 10/10] rcu: Add per-CPU rcuc task dumps to RCU CPU stall warnings Date: Fri, 4 Feb 2022 15:24:06 -0800 Message-Id: <20220204232406.814-10-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> References: <20220204232355.GA728@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Zqiang When the rcutree.use_softirq kernel boot parameter is set to zero, all RCU_SOFTIRQ processing is carried out by the per-CPU rcuc kthreads. If these kthreads are being starved, quiescent states will not be reported, which in turn means that the grace period will not end, which can in turn trigger RCU CPU stall warnings. This commit therefore dumps stack traces of stalled CPUs' rcuc kthreads, which can help identify what is preventing those kthreads from running. Suggested-by: Ammar Faizi Reviewed-by: Ammar Faizi Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 3 +++ kernel/rcu/tree.h | 1 + kernel/rcu/tree_plugin.h | 3 +++ kernel/rcu/tree_stall.h | 35 +++++++++++++++++++++++++++++++++++ 4 files changed, 42 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 9e4c5b281f003..bd9b2af247abd 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2850,10 +2850,12 @@ static void rcu_cpu_kthread(unsigned int cpu) { unsigned int *statusp =3D this_cpu_ptr(&rcu_data.rcu_cpu_kthread_status); char work, *workp =3D this_cpu_ptr(&rcu_data.rcu_cpu_has_work); + unsigned long *j =3D this_cpu_ptr(&rcu_data.rcuc_activity); int spincnt; =20 trace_rcu_utilization(TPS("Start CPU kthread@rcu_run")); for (spincnt =3D 0; spincnt < 10; spincnt++) { + WRITE_ONCE(*j, jiffies); local_bh_disable(); *statusp =3D RCU_KTHREAD_RUNNING; local_irq_disable(); @@ -2874,6 +2876,7 @@ static void rcu_cpu_kthread(unsigned int cpu) schedule_timeout_idle(2); trace_rcu_utilization(TPS("End CPU kthread@rcu_yield")); *statusp =3D RCU_KTHREAD_WAITING; + WRITE_ONCE(*j, jiffies); } =20 static struct smp_hotplug_thread rcu_cpu_thread_spec =3D { diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index e9990945483f1..b84cc5742c317 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -239,6 +239,7 @@ struct rcu_data { /* rcuc per-CPU kthread or NULL. */ unsigned int rcu_cpu_kthread_status; char rcu_cpu_has_work; + unsigned long rcuc_activity; =20 /* 7) Diagnostic data, including RCU CPU stall warnings. */ unsigned int softirq_snap; /* Snapshot of softirq activity. */ diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 02ac057ba3f83..8167cab1bffc8 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -996,12 +996,15 @@ dump_blkd_tasks(struct rcu_node *rnp, int ncheck) */ static void rcu_cpu_kthread_setup(unsigned int cpu) { + struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); #ifdef CONFIG_RCU_BOOST struct sched_param sp; =20 sp.sched_priority =3D kthread_prio; sched_setscheduler_nocheck(current, SCHED_FIFO, &sp); #endif /* #ifdef CONFIG_RCU_BOOST */ + + WRITE_ONCE(rdp->rcuc_activity, jiffies); } =20 #ifdef CONFIG_RCU_BOOST diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 21bebf7c9030b..0c5d8516516af 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -379,6 +379,15 @@ static bool rcu_is_gp_kthread_starving(unsigned long *= jp) return j > 2 * HZ; } =20 +static bool rcu_is_rcuc_kthread_starving(struct rcu_data *rdp, unsigned lo= ng *jp) +{ + unsigned long j =3D jiffies - READ_ONCE(rdp->rcuc_activity); + + if (jp) + *jp =3D j; + return j > 2 * HZ; +} + /* * Print out diagnostic information for the specified stalled CPU. * @@ -430,6 +439,29 @@ static void print_cpu_stall_info(int cpu) falsepositive ? " (false positive?)" : ""); } =20 +static void rcuc_kthread_dump(struct rcu_data *rdp) +{ + int cpu; + unsigned long j; + struct task_struct *rcuc; + + rcuc =3D rdp->rcu_cpu_kthread_task; + if (!rcuc) + return; + + cpu =3D task_cpu(rcuc); + if (cpu_is_offline(cpu) || idle_cpu(cpu)) + return; + + if (!rcu_is_rcuc_kthread_starving(rdp, &j)) + return; + + pr_err("%s kthread starved for %ld jiffies\n", rcuc->comm, j); + sched_show_task(rcuc); + if (!trigger_single_cpu_backtrace(cpu)) + dump_cpu_task(cpu); +} + /* Complain about starvation of grace-period kthread. */ static void rcu_check_gp_kthread_starvation(void) { @@ -601,6 +633,9 @@ static void print_cpu_stall(unsigned long gps) rcu_check_gp_kthread_expired_fqs_timer(); rcu_check_gp_kthread_starvation(); =20 + if (!use_softirq) + rcuc_kthread_dump(rdp); + rcu_dump_cpu_stacks(); =20 raw_spin_lock_irqsave_rcu_node(rnp, flags); --=20 2.31.1.189.g2e36527f23