From nobody Mon May 11 02:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30532C433EF for ; Tue, 19 Apr 2022 00:00:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236116AbiDSADH (ORCPT ); Mon, 18 Apr 2022 20:03:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236108AbiDSAC6 (ORCPT ); Mon, 18 Apr 2022 20:02:58 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C80211C34; Mon, 18 Apr 2022 17:00:17 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CF3BE6128A; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 30AA4C385A8; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326416; bh=B3FKOAVi0h12Jmquk99abkkXI6DRfWVVNn7fsz2s47Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eIyVIelsePL2hXcgK1LeA+D2oq1psbS3en+cIiWfV7CXR5CSKfcL1p4YR/RyUp8ia S8NKPlNqDslxqSV7Ze5lTntCIkums0iS3VJCgwe+BKyEEs9wD2xtXD7NusbV5suYKl LBRPZga+AsvMFW+fabpwlOd+621ojGTW/xp9GgJiNRbKb6hdsZv6i/YySjaJ3pg2LQ NDX6paN4TEYkdD8Q/VtTaTDYFcZYGHMJ3vhtzVDgqrmnrclZ15bSwyYHzCsD8lRXvo PXgW1s9rH/OZomfLNHgOg9o4UNnJ0PPM4NPY5u2H3MjsbqJiPdbvJvaCfwL5ysvr5e v4KpUy+s8l4lQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D5D605C04BD; Mon, 18 Apr 2022 17:00:15 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Padmanabha Srinivasaiah , "Paul E . McKenney" Subject: [PATCH rcu 1/9] rcu-tasks: Fix race in schedule and flush work Date: Mon, 18 Apr 2022 17:00:06 -0700 Message-Id: <20220419000014.3948512-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> References: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Padmanabha Srinivasaiah While booting secondary CPUs, cpus_read_[lock/unlock] is not keeping online cpumask stable. The transient online mask results in below calltrace. [ 0.324121] CPU1: Booted secondary processor 0x0000000001 [0x410fd083] [ 0.346652] Detected PIPT I-cache on CPU2 [ 0.347212] CPU2: Booted secondary processor 0x0000000002 [0x410fd083] [ 0.377255] Detected PIPT I-cache on CPU3 [ 0.377823] CPU3: Booted secondary processor 0x0000000003 [0x410fd083] [ 0.379040] ------------[ cut here ]------------ [ 0.383662] WARNING: CPU: 0 PID: 10 at kernel/workqueue.c:3084 __flush_w= ork+0x12c/0x138 [ 0.384850] Modules linked in: [ 0.385403] CPU: 0 PID: 10 Comm: rcu_tasks_rude_ Not tainted 5.17.0-rc3-= v8+ #13 [ 0.386473] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT) [ 0.387289] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE= =3D--) [ 0.388308] pc : __flush_work+0x12c/0x138 [ 0.388970] lr : __flush_work+0x80/0x138 [ 0.389620] sp : ffffffc00aaf3c60 [ 0.390139] x29: ffffffc00aaf3d20 x28: ffffffc009c16af0 x27: ffffff80f76= 1df48 [ 0.391316] x26: 0000000000000004 x25: 0000000000000003 x24: 00000000000= 00100 [ 0.392493] x23: ffffffffffffffff x22: ffffffc009c16b10 x21: ffffffc009c= 16b28 [ 0.393668] x20: ffffffc009e53861 x19: ffffff80f77fbf40 x18: 00000000d74= 4fcc9 [ 0.394842] x17: 000000000000000b x16: 00000000000001c2 x15: ffffffc009e= 57550 [ 0.396016] x14: 0000000000000000 x13: ffffffffffffffff x12: 00000001000= 00000 [ 0.397190] x11: 0000000000000462 x10: ffffff8040258008 x9 : 00000001000= 00000 [ 0.398364] x8 : 0000000000000000 x7 : ffffffc0093c8bf4 x6 : 00000000000= 00000 [ 0.399538] x5 : 0000000000000000 x4 : ffffffc00a976e40 x3 : ffffffc0081= 0444c [ 0.400711] x2 : 0000000000000004 x1 : 0000000000000000 x0 : 00000000000= 00000 [ 0.401886] Call trace: [ 0.402309] __flush_work+0x12c/0x138 [ 0.402941] schedule_on_each_cpu+0x228/0x278 [ 0.403693] rcu_tasks_rude_wait_gp+0x130/0x144 [ 0.404502] rcu_tasks_kthread+0x220/0x254 [ 0.405264] kthread+0x174/0x1ac [ 0.405837] ret_from_fork+0x10/0x20 [ 0.406456] irq event stamp: 102 [ 0.406966] hardirqs last enabled at (101): [] _raw_s= pin_unlock_irq+0x78/0xb4 [ 0.408304] hardirqs last disabled at (102): [] el1_db= g+0x24/0x5c [ 0.409410] softirqs last enabled at (54): [] local_b= h_enable+0xc/0x2c [ 0.410645] softirqs last disabled at (50): [] local_b= h_disable+0xc/0x2c [ 0.411890] ---[ end trace 0000000000000000 ]--- [ 0.413000] smp: Brought up 1 node, 4 CPUs [ 0.413762] SMP: Total of 4 processors activated. [ 0.414566] CPU features: detected: 32-bit EL0 Support [ 0.415414] CPU features: detected: 32-bit EL1 Support [ 0.416278] CPU features: detected: CRC32 instructions [ 0.447021] Callback from call_rcu_tasks_rude() invoked. [ 0.506693] Callback from call_rcu_tasks() invoked. This commit therefore fixes this issue by applying a single-CPU optimization to the RCU Tasks Rude grace-period process. The key point here is that the purpose of this RCU flavor is to force a schedule on each online CPU since some past event. But the rcu_tasks_rude_wait_gp() function runs in the context of the RCU Tasks Rude's grace-period kthread, so there must already have been a context switch on the current CPU since the call to either synchronize_rcu_tasks_rude() or call_rcu_tasks_rude(). So if there is only a single CPU online, RCU Tasks Rude's grace-period kthread does not need to anything at all. It turns out that the rcu_tasks_rude_wait_gp() function's call to schedule_on_each_cpu() causes problems during early boot. During that time, there is only one online CPU, namely the boot CPU. Therefore, applying this single-CPU optimization fixes early-boot instances of this problem. Link: https://lore.kernel.org/lkml/20220210184319.25009-1-treasure4paddy@gm= ail.com/T/ Suggested-by: Paul E. McKenney Signed-off-by: Padmanabha Srinivasaiah Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 99cf3a13954c..b43320b149d2 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -950,6 +950,9 @@ static void rcu_tasks_be_rude(struct work_struct *work) // Wait for one rude RCU-tasks grace period. static void rcu_tasks_rude_wait_gp(struct rcu_tasks *rtp) { + if (num_online_cpus() <=3D 1) + return; // Fastpath for only one CPU. + rtp->n_ipis +=3D cpumask_weight(cpu_online_mask); schedule_on_each_cpu(rcu_tasks_be_rude); } --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5A8BC433EF for ; Tue, 19 Apr 2022 00:00:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236509AbiDSADL (ORCPT ); Mon, 18 Apr 2022 20:03:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236098AbiDSAC6 (ORCPT ); Mon, 18 Apr 2022 20:02:58 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DA6011C38; Mon, 18 Apr 2022 17:00:17 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D38D861290; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3083CC385A7; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326416; bh=15WmndtzksixwsdyWx7zI/CoD8NXMZvP4L24gel6g5k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=K614iE0sBFKWi1b858MHkARE9dhOxftOh3hBjwl4nM91zm6YjAMJ69fOVJyMfYrwH R9d67UpJZpKN3ywRs7poOMxCNirzr+VF3M02RIH85Qb6mfCOfx6CHNjRVCwJ2ostEw O8C2YX+KF1KHDkwaOeJWtjhYEYkF202re59kEKNcUlpUMUOt5BTIkb9hhQ1ht5FYEz dkwa8bJtrZFcbeFleC2I0xhipUUCiea8dtnyBi7qobiU1DZqxap2/OSBXDJjVVJJ6+ QumrKNyKzxu1c35u4R+wnPM5qU0iI4+rF/vOV8/sm/RT1fqWn9aY8oSEoAH07orjPu F52wF73G8vtxw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id D970F5C04C6; Mon, 18 Apr 2022 17:00:15 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 2/9] rcu-tasks: Print pre-stall-warning informational messages Date: Mon, 18 Apr 2022 17:00:07 -0700 Message-Id: <20220419000014.3948512-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> References: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" RCU-tasks stall-warning messages are printed after the grace period is ten minutes old. Unfortunately, most of us will have rebooted the system in response to an apparently-hung command long before the ten minutes is up, and will thus see what looks to be a silent hang. This commit therefore adds pr_info() messages that are printed earlier. These should avoid being classified as errors, but should give impatient users a hint. These are controlled by new rcupdate.rcu_task_stall_info and rcupdate.rcu_task_stall_info_mult kernel-boot parameters. The former defines the initial delay in jiffies (defaulting to 10 seconds) and the latter defines the multiplier (defaulting to 3). Thus, by default, the first message will appear 10 seconds into the RCU-tasks grace period, the second 40 seconds in, and the third 160 seconds in. There would be a fourth at 640 seconds in, but the stall warning message appears 600 seconds in, and once a stall warning is printed for a given grace period, no further informational messages are printed. Signed-off-by: Paul E. McKenney --- .../admin-guide/kernel-parameters.txt | 30 ++++++++++++-- kernel/rcu/tasks.h | 40 ++++++++++++++++--- 2 files changed, 62 insertions(+), 8 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 3f1cc5e317ed..babc701d4864 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4955,10 +4955,34 @@ number avoids disturbing real-time workloads, but lengthens grace periods. =20 + rcupdate.rcu_task_stall_info=3D [KNL] + Set initial timeout in jiffies for RCU task stall + informational messages, which give some indication + of the problem for those not patient enough to + wait for ten minutes. Informational messages are + only printed prior to the stall-warning message + for a given grace period. Disable with a value + less than or equal to zero. Defaults to ten + seconds. A change in value does not take effect + until the beginning of the next grace period. + + rcupdate.rcu_task_stall_info_mult=3D [KNL] + Multiplier for time interval between successive + RCU task stall informational messages for a given + RCU tasks grace period. This value is clamped + to one through ten, inclusive. It defaults to + the value three, so that the first informational + message is printed 10 seconds into the grace + period, the second at 40 seconds, the third at + 160 seconds, and then the stall warning at 600 + seconds would prevent a fourth at 640 seconds. + rcupdate.rcu_task_stall_timeout=3D [KNL] - Set timeout in jiffies for RCU task stall warning - messages. Disable with a value less than or equal - to zero. + Set timeout in jiffies for RCU task stall + warning messages. Disable with a value less + than or equal to zero. Defaults to ten minutes. + A change in value does not take effect until + the beginning of the next grace period. =20 rcupdate.rcu_self_test=3D [KNL] Run the RCU early boot self tests diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index b43320b149d2..76799c81d4be 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -143,6 +143,11 @@ module_param(rcu_task_ipi_delay, int, 0644); #define RCU_TASK_STALL_TIMEOUT (HZ * 60 * 10) static int rcu_task_stall_timeout __read_mostly =3D RCU_TASK_STALL_TIMEOUT; module_param(rcu_task_stall_timeout, int, 0644); +#define RCU_TASK_STALL_INFO (HZ * 10) +static int rcu_task_stall_info __read_mostly =3D RCU_TASK_STALL_INFO; +module_param(rcu_task_stall_info, int, 0644); +static int rcu_task_stall_info_mult __read_mostly =3D 3; +module_param(rcu_task_stall_info_mult, int, 0444); =20 static int rcu_task_enqueue_lim __read_mostly =3D -1; module_param(rcu_task_enqueue_lim, int, 0444); @@ -548,8 +553,15 @@ static void __init rcu_spawn_tasks_kthread_generic(str= uct rcu_tasks *rtp) static void __init rcu_tasks_bootup_oddness(void) { #if defined(CONFIG_TASKS_RCU) || defined(CONFIG_TASKS_TRACE_RCU) + int rtsimc; + if (rcu_task_stall_timeout !=3D RCU_TASK_STALL_TIMEOUT) pr_info("\tTasks-RCU CPU stall warnings timeout set to %d (rcu_task_stal= l_timeout).\n", rcu_task_stall_timeout); + rtsimc =3D clamp(rcu_task_stall_info_mult, 1, 10); + if (rtsimc !=3D rcu_task_stall_info_mult) { + pr_info("\tTasks-RCU CPU stall info multiplier clamped to %d (rcu_task_s= tall_info_mult).\n", rtsimc); + rcu_task_stall_info_mult =3D rtsimc; + } #endif /* #ifdef CONFIG_TASKS_RCU */ #ifdef CONFIG_TASKS_RCU pr_info("\tTrampoline variant of Tasks RCU enabled.\n"); @@ -592,10 +604,15 @@ static void exit_tasks_rcu_finish_trace(struct task_s= truct *t); /* Wait for one RCU-tasks grace period. */ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) { - struct task_struct *g, *t; - unsigned long lastreport; - LIST_HEAD(holdouts); + struct task_struct *g; int fract; + LIST_HEAD(holdouts); + unsigned long j; + unsigned long lastinfo; + unsigned long lastreport; + bool reported =3D false; + int rtsi; + struct task_struct *t; =20 set_tasks_gp_state(rtp, RTGS_PRE_WAIT_GP); rtp->pregp_func(); @@ -621,6 +638,8 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) * is empty, we are done. */ lastreport =3D jiffies; + lastinfo =3D lastreport; + rtsi =3D READ_ONCE(rcu_task_stall_info); =20 // Start off with initial wait and slowly back off to 1 HZ wait. fract =3D rtp->init_fract; @@ -630,7 +649,7 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) bool needreport; int rtst; =20 - /* Slowly back off waiting for holdouts */ + // Slowly back off waiting for holdouts set_tasks_gp_state(rtp, RTGS_WAIT_SCAN_HOLDOUTS); schedule_timeout_idle(fract); =20 @@ -639,12 +658,23 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) =20 rtst =3D READ_ONCE(rcu_task_stall_timeout); needreport =3D rtst > 0 && time_after(jiffies, lastreport + rtst); - if (needreport) + if (needreport) { lastreport =3D jiffies; + reported =3D true; + } firstreport =3D true; WARN_ON(signal_pending(current)); set_tasks_gp_state(rtp, RTGS_SCAN_HOLDOUTS); rtp->holdouts_func(&holdouts, needreport, &firstreport); + + // Print pre-stall informational messages if needed. + j =3D jiffies; + if (rtsi > 0 && !reported && time_after(j, lastinfo + rtsi)) { + lastinfo =3D j; + rtsi =3D rtsi * rcu_task_stall_info_mult; + pr_info("%s: %s grace period %lu is %lu jiffies old.\n", + __func__, rtp->kname, rtp->tasks_gp_seq, j - rtp->gp_start); + } } =20 set_tasks_gp_state(rtp, RTGS_POST_GP); --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81246C4332F for ; Tue, 19 Apr 2022 00:00:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236379AbiDSADD (ORCPT ); Mon, 18 Apr 2022 20:03:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236116AbiDSAC6 (ORCPT ); Mon, 18 Apr 2022 20:02:58 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DC2111C3F; Mon, 18 Apr 2022 17:00:17 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D5BA96123D; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 39513C385A1; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326416; bh=3PHr6DaJm3KpAi0VoxU2Z4Urf23rCyOQHDJiKZI+ijM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sGet6K9vISgKGjn1gS6P1dcO4kA+em3X3NGknBPDrM8g6pTs/v4MrfdShT6NJtaEh a+MrX7faiqb7QF4jamJwhFlmvld3MxR5wcOSnDBJjcFFWi0JN31QI3chf0rrHF1FBA XW7TtAspSmObUrE388VxRCdqdDWAK9dORLe4nfGRJynRCyHvA4BJxzl6Av6Gk1UuZg eG3lY7scaX8nfyAZvqIA7IoYiX8N0yKx/reQoypmfmtP2erRQ0OIBujfRIvuzeLgBz YLbVyJEGJ2LcD2zRfi987OCcRNWsv3PSh5fW7vLd7Fs6NdSreK3ppt9offb2lwuNIX y60smg8R3tYCg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id DB7245C0848; Mon, 18 Apr 2022 17:00:15 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Sebastian Andrzej Siewior , "Paul E . McKenney" Subject: [PATCH rcu 3/9] rcu-tasks: Use rcuwait for the rcu_tasks_kthread() Date: Mon, 18 Apr 2022 17:00:08 -0700 Message-Id: <20220419000014.3948512-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> References: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sebastian Andrzej Siewior The waitqueue used by rcu_tasks_kthread() has always only one waiter. With a guaranteed only one waiter, this can be replaced with rcuwait which is smaller and simpler. With rcuwait based wake counterpart, the irqwork function (call_rcu_tasks_iw_wakeup()) can be invoked hardirq context because it is only a wake up and no sleeping locks are involved (unlike the wait_queue_head). As a side effect, this is also one piece of the puzzle to pass the RCU selftest at early boot on PREEMPT_RT. Replace wait_queue_head with rcuwait and let the irqwork run in hardirq context on PREEMPT_RT. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 76799c81d4be..4b91cb214ca7 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -46,7 +46,7 @@ struct rcu_tasks_percpu { =20 /** * struct rcu_tasks - Definition for a Tasks-RCU-like mechanism. - * @cbs_wq: Wait queue allowing new callback to get kthread's attention. + * @cbs_wait: RCU wait allowing a new callback to get kthread's attention. * @cbs_gbl_lock: Lock protecting callback list. * @kthread_ptr: This flavor's grace-period/callback-invocation kthread. * @gp_func: This flavor's grace-period-wait function. @@ -77,7 +77,7 @@ struct rcu_tasks_percpu { * @kname: This flavor's kthread name. */ struct rcu_tasks { - struct wait_queue_head cbs_wq; + struct rcuwait cbs_wait; raw_spinlock_t cbs_gbl_lock; int gp_state; int gp_sleep; @@ -113,11 +113,11 @@ static void call_rcu_tasks_iw_wakeup(struct irq_work = *iwp); #define DEFINE_RCU_TASKS(rt_name, gp, call, n) \ static DEFINE_PER_CPU(struct rcu_tasks_percpu, rt_name ## __percpu) =3D { = \ .lock =3D __RAW_SPIN_LOCK_UNLOCKED(rt_name ## __percpu.cbs_pcpu_lock), \ - .rtp_irq_work =3D IRQ_WORK_INIT(call_rcu_tasks_iw_wakeup), \ + .rtp_irq_work =3D IRQ_WORK_INIT_HARD(call_rcu_tasks_iw_wakeup), \ }; \ static struct rcu_tasks rt_name =3D \ { \ - .cbs_wq =3D __WAIT_QUEUE_HEAD_INITIALIZER(rt_name.cbs_wq), \ + .cbs_wait =3D __RCUWAIT_INITIALIZER(rt_name.wait), \ .cbs_gbl_lock =3D __RAW_SPIN_LOCK_UNLOCKED(rt_name.cbs_gbl_lock), \ .gp_func =3D gp, \ .call_func =3D call, \ @@ -266,7 +266,7 @@ static void call_rcu_tasks_iw_wakeup(struct irq_work *i= wp) struct rcu_tasks_percpu *rtpcp =3D container_of(iwp, struct rcu_tasks_per= cpu, rtp_irq_work); =20 rtp =3D rtpcp->rtpp; - wake_up(&rtp->cbs_wq); + rcuwait_wake_up(&rtp->cbs_wait); } =20 // Enqueue a callback for the specified flavor of Tasks RCU. @@ -514,7 +514,9 @@ static int __noreturn rcu_tasks_kthread(void *arg) set_tasks_gp_state(rtp, RTGS_WAIT_CBS); =20 /* If there were none, wait a bit and start over. */ - wait_event_idle(rtp->cbs_wq, (needgpcb =3D rcu_tasks_need_gpcb(rtp))); + rcuwait_wait_event(&rtp->cbs_wait, + (needgpcb =3D rcu_tasks_need_gpcb(rtp)), + TASK_IDLE); =20 if (needgpcb & 0x2) { // Wait for one grace period. --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D293EC433F5 for ; Tue, 19 Apr 2022 00:00:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236619AbiDSADY (ORCPT ); Mon, 18 Apr 2022 20:03:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236105AbiDSAC7 (ORCPT ); Mon, 18 Apr 2022 20:02:59 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BDCA713D2C; Mon, 18 Apr 2022 17:00:18 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 71E3FB81135; Tue, 19 Apr 2022 00:00:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 30AC7C385A9; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326416; bh=5RvY22qrA5+M1MxIQaC3zBsZAloMmunOEBx8nCIFweI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MzqPKgqdxdkO0t+B3BkeVP9Ie/wGz4j1e26YZJQVOnVOKWEqliHkqd/ffgpIbPmfE EXAnNnN8CRzvryxThjfMqy6fInMwUs6ikD8ZC37BUUv56dLRM79CUgplirDakvkON0 80boKDfCaJz/pTLjIChEpdTMSR10NElvwslgPPAoPHex+LxGhaNs0hQRNEZTqD/tJR 971UOCCs6JxZYHOnpdLr/WCmJSXsNhTr9cvO3aa25qaYUsk4F+OnBR/P2QaAcw2O0E 3/T0UPPCGMwh0nlzBa5M3AmVytdnaqiivCxkFxzeHTGWJ5fRPtvbACVNXTk3Z8NSMs 82VEroYcpbozw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id DD4E85C0A0D; Mon, 18 Apr 2022 17:00:15 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Martin KaFai Lau , Neil Spring Subject: [PATCH rcu 4/9] rcu-tasks: Make Tasks RCU account for userspace execution Date: Mon, 18 Apr 2022 17:00:09 -0700 Message-Id: <20220419000014.3948512-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> References: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The main Tasks RCU quiescent state is voluntary context switch. However, userspace execution is also a valid quiescent state, and is a valuable one for userspace applications that spin repeatedly executing light-weight non-sleeping system calls. Currently, such an application can delay a Tasks RCU grace period for many tens of seconds. This commit therefore enlists the aid of the scheduler-clock interrupt to provide a Tasks RCU quiescent state when it interrupted a task executing in userspace. [ paulmck: Apply feedback from kernel test robot. ] Cc: Martin KaFai Lau Cc: Neil Spring Signed-off-by: Paul E. McKenney --- include/linux/rcupdate.h | 1 + kernel/rcu/tree.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index e7c39c200e2b..1a32036c918c 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -196,6 +196,7 @@ void synchronize_rcu_tasks_rude(void); void exit_tasks_rcu_start(void); void exit_tasks_rcu_finish(void); #else /* #ifdef CONFIG_TASKS_RCU_GENERIC */ +#define rcu_tasks_classic_qs(t, preempt) do { } while (0) #define rcu_tasks_qs(t, preempt) do { } while (0) #define rcu_note_voluntary_context_switch(t) do { } while (0) #define call_rcu_tasks call_rcu diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index a4b8189455d5..8dbfb63f0391 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2624,6 +2624,8 @@ void rcu_sched_clock_irq(int user) rcu_flavor_sched_clock_irq(user); if (rcu_pending(user)) invoke_rcu_core(); + if (user) + rcu_tasks_classic_qs(current, false); lockdep_assert_irqs_disabled(); =20 trace_rcu_utilization(TPS("End scheduler-tick")); --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85126C433F5 for ; Tue, 19 Apr 2022 00:00:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236648AbiDSADa (ORCPT ); Mon, 18 Apr 2022 20:03:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236181AbiDSAC7 (ORCPT ); Mon, 18 Apr 2022 20:02:59 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D31DC13D54; Mon, 18 Apr 2022 17:00:18 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7F686B81136; Tue, 19 Apr 2022 00:00:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3D1D6C385AB; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326416; bh=nsc9VJDf4f5ASTnU9OeOCzg1hO7iWgQlVPIuzBYroJw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BZmWSSGFrRg4/kZ/0YNd1hYx4XjeAnFMLHBAdEknFyG/ehMZuMvfSThB4Mpdce4k4 8RU424Da2WIzitATEkKdisC3WVsi5cnuGHwLbRnshBsavj+ZojxfnqRawnrOKLD2X7 yiPuXXwC4nydQ60sXFDMTVYYymKUMwEYtJ9XBeA4GTGtu/zKV/WGP2VMh1BxzmkzZX rP0UCgMVjCv6i7GiKE5c7yTWJZP6cfZYI5LbfYq2cC1snGSSqS+NL+Aw5ht0QlVZta hMN9HPOnMZK0nIQNvEf1T09V/YrohPMqZV/I5GFT7+4+4+rsLVdqqFL6Tc9ABkdUiY 6p/jYwVz9MTlw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id DF4045C0A23; Mon, 18 Apr 2022 17:00:15 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Sebastian Andrzej Siewior , Martin KaFai Lau , Andrii Nakryiko , "Paul E . McKenney" Subject: [PATCH rcu 5/9] rcu-tasks: Use schedule_hrtimeout_range() to wait for grace periods Date: Mon, 18 Apr 2022 17:00:10 -0700 Message-Id: <20220419000014.3948512-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> References: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Sebastian Andrzej Siewior The synchronous RCU-tasks grace-period-wait primitives invoke schedule_timeout_idle() to give readers a chance to exit their read-side critical sections. Unfortunately, this fails during early boot on PREEMPT_RT because PREEMPT_RT relies solely on ksoftirqd to run timer handlers. Because ksoftirqd cannot operate until its kthreads are spawned, there is a brief period of time following scheduler initialization where PREEMPT_RT cannot run the timer handlers that schedule_timeout_idle() relies on, resulting in a hang. To avoid this boot-time hang, this commit replaces schedule_timeout_idle() with schedule_hrtimeout(), so that the timer expires in hardirq context. This is ensures that the timer fires even on PREEMPT_RT throughout the irqs-enabled portions of boot as well as during runtime. The timer is set to expire between fract and fract + HZ / 2 jiffies in order to align with any other timers that might expire during that time, thus reducing the number of wakeups. Note that RCU-tasks grace periods are infrequent, so the use of hrtimer should be fine. In contrast, in common-case code, user of hrtimer could result in performance issues. Cc: Martin KaFai Lau Cc: Andrii Nakryiko Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 4b91cb214ca7..71fe340ab82a 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -647,13 +647,16 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) fract =3D rtp->init_fract; =20 while (!list_empty(&holdouts)) { + ktime_t exp; bool firstreport; bool needreport; int rtst; =20 // Slowly back off waiting for holdouts set_tasks_gp_state(rtp, RTGS_WAIT_SCAN_HOLDOUTS); - schedule_timeout_idle(fract); + exp =3D jiffies_to_nsecs(fract); + __set_current_state(TASK_IDLE); + schedule_hrtimeout_range(&exp, jiffies_to_nsecs(HZ / 2), HRTIMER_MODE_RE= L_HARD); =20 if (fract < HZ) fract++; --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A149C433EF for ; Tue, 19 Apr 2022 00:00:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236561AbiDSADR (ORCPT ); Mon, 18 Apr 2022 20:03:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236152AbiDSAC6 (ORCPT ); Mon, 18 Apr 2022 20:02:58 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 172F913CDC; Mon, 18 Apr 2022 17:00:18 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 758FC612A0; Tue, 19 Apr 2022 00:00:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8EAE3C385B4; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326416; bh=d8BookNJh/O4qU39WT4SVwBgry0u2pt1lr7SQzvjnkA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NTmmCdR/7pN5hzrQK2g+6jsxLyKLU+7QwtwM+OzzNSX3KNYpZPdA3BqGL/NEypEhZ wI18v5qZESaM1SFEBTKsgIPmDOfkmjSoR5I1qFpbOO4BsNglKmgPo37SfTWbxjGiwi alMc4vTUdNB0mKBxmrPt2ygaONntkH9q2v5jsGP/ojmu51XbTR/tWEYGNPrAZN6S6b 3lhGhOKz/OtDxBOxAu5vwy8bj7huAGyZ4tsW7U87iTY2Wy8MK8LNYBRgKrYvlI5xxw sLfcSZGbyNRwL9UqboiohZOSN3O3JjVc/E+fxc14EP3tIXC9gENeQ21jfvFsN1Rh68 iPAx3FGXDSCvQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E12025C0B86; Mon, 18 Apr 2022 17:00:15 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 6/9] rcu-tasks: Restore use of timers for non-RT kernels Date: Mon, 18 Apr 2022 17:00:11 -0700 Message-Id: <20220419000014.3948512-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> References: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The use of hrtimers for RCU-tasks grace-period delays works well in general, but can result in excessive grace-period delays for some corner-case workloads. This commit therefore reverts to the use of timers for non-RT kernels to mitigate those grace-period delays. Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 71fe340ab82a..405614039515 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -654,9 +654,13 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) =20 // Slowly back off waiting for holdouts set_tasks_gp_state(rtp, RTGS_WAIT_SCAN_HOLDOUTS); - exp =3D jiffies_to_nsecs(fract); - __set_current_state(TASK_IDLE); - schedule_hrtimeout_range(&exp, jiffies_to_nsecs(HZ / 2), HRTIMER_MODE_RE= L_HARD); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) { + schedule_timeout_idle(fract); + } else { + exp =3D jiffies_to_nsecs(fract); + __set_current_state(TASK_IDLE); + schedule_hrtimeout_range(&exp, jiffies_to_nsecs(HZ / 2), HRTIMER_MODE_R= EL_HARD); + } =20 if (fract < HZ) fract++; --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE101C433EF for ; Tue, 19 Apr 2022 00:00:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236590AbiDSADV (ORCPT ); Mon, 18 Apr 2022 20:03:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236126AbiDSAC6 (ORCPT ); Mon, 18 Apr 2022 20:02:58 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1701013CC8; Mon, 18 Apr 2022 17:00:18 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 72B3B61297; Tue, 19 Apr 2022 00:00:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8DFF2C385B6; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326416; bh=xBCTSb1Zw5agh9IQQuLKEyNN49GdD9FsTkuxc17sgcg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kmxi/26iXVhEk9jWuQSEXWR1R26p63ZIJDTLiIMzRxUilfYLXMeLtDEOyCe0wIxFh JHU3Fp4DEodko8v5TqKbv4SKkR80D3f0JlLU+WCw9YOXZM/jijzmYyPYtDB4O401BG 4o2iLuXPKIQfu72gSxGJC+XJXTAvE7swIcDRf/FX3Nz9dz5fjzvBhUhGUGn04HRSqe Dy3Ivh4hWvpFaHmleMfRvFQ2P1rE6dvAwq1y1t/+2Q73jkZ/+weVjW2PBHk+C24ucH OP+JYjfuWLADk+EnUsZYqY32fgg9eCUVAGFp2+2VG2/XWpaGf3fkqSdL652ayQBJFg CBKrZFaOeQ65g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E2F6F5C0DFD; Mon, 18 Apr 2022 17:00:15 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 7/9] rcu-tasks: Make show_rcu_tasks_generic_gp_kthread() check all CPUs Date: Mon, 18 Apr 2022 17:00:12 -0700 Message-Id: <20220419000014.3948512-7-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> References: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Currently, the show_rcu_tasks_generic_gp_kthread() function only looks at CPU 0's callback lists. Although this is not fatal, it can confuse debugging efforts in cases where any of the Tasks RCU flavors are in per-CPU queueing mode. This commit therefore causes this function to scan all CPUs' callback queues. Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 405614039515..3aad0dfbfaf4 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -582,7 +582,17 @@ static void __init rcu_tasks_bootup_oddness(void) /* Dump out rcutorture-relevant state common to all RCU-tasks flavors. */ static void show_rcu_tasks_generic_gp_kthread(struct rcu_tasks *rtp, char = *s) { - struct rcu_tasks_percpu *rtpcp =3D per_cpu_ptr(rtp->rtpcpu, 0); // for_ea= ch... + int cpu; + bool havecbs =3D false; + + for_each_possible_cpu(cpu) { + struct rcu_tasks_percpu *rtpcp =3D per_cpu_ptr(rtp->rtpcpu, cpu); + + if (!data_race(rcu_segcblist_empty(&rtpcp->cblist))) { + havecbs =3D true; + break; + } + } pr_info("%s: %s(%d) since %lu g:%lu i:%lu/%lu %c%c %s\n", rtp->kname, tasks_gp_state_getname(rtp), data_race(rtp->gp_state), @@ -590,7 +600,7 @@ static void show_rcu_tasks_generic_gp_kthread(struct rc= u_tasks *rtp, char *s) data_race(rcu_seq_current(&rtp->tasks_gp_seq)), data_race(rtp->n_ipis_fails), data_race(rtp->n_ipis), ".k"[!!data_race(rtp->kthread_ptr)], - ".C"[!data_race(rcu_segcblist_empty(&rtpcp->cblist))], + ".C"[havecbs], s); } #endif // #ifndef CONFIG_TINY_RCU --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F6E0C433EF for ; Tue, 19 Apr 2022 00:00:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236184AbiDSADe (ORCPT ); Mon, 18 Apr 2022 20:03:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236277AbiDSADA (ORCPT ); Mon, 18 Apr 2022 20:03:00 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96A9F11C34; Mon, 18 Apr 2022 17:00:19 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 3A4A1B81106; Tue, 19 Apr 2022 00:00:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8DF45C385B5; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326416; bh=hro1DWmXh+gkHoQfRvRaQ76L7LJUd/OIij5/KjJOEbM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PONY76fFMgRSlbxeEpKhCEEWAgNfZrIZ0EiH98otJUgWntQxnrYMQWTUkErEDysSG NEGHtpJb8SCpy4tV/w6ZvF+RTDM/CxPqHKvug4L1dukt+nlRd8vC/cycVx20dA2tlQ HEL6MTbcwlXR+Ps2nRGWnnheMkr32MizHM0GpbaHrsWIKYycO+s7deXzHev9ara/VH lsiO4QlFEo8bzYcTE4IzolzYmfZkkfTsNmp4NU+poAP/HpPNePHZFTAI6YOyKbuoOP bdras8rBNjtZWMkOUghhWYpGQMbcoCRGC6B4ih/dRIFghjzePRxNTTRRokQm2UywoU xO6SysXc8UyEg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E4DDD5C121E; Mon, 18 Apr 2022 17:00:15 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Eric Dumazet , "Paul E . McKenney" Subject: [PATCH rcu 8/9] rcu-tasks: Handle sparse cpu_possible_mask Date: Mon, 18 Apr 2022 17:00:13 -0700 Message-Id: <20220419000014.3948512-8-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> References: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Eric Dumazet If the rcupdate.rcu_task_enqueue_lim kernel boot parameter is set to something greater than 1 and less than nr_cpu_ids, the code attempts to use a subset of the CPU's RCU Tasks callback lists. This works, but only if the cpu_possible_mask is contiguous. If there are "holes" in this mask, the callback-enqueue code might attempt to access a non-existent per-CPU ->rtcpu variable for a non-existent CPU. For example, if only CPUs 0, 4, 8, 12, 16 and so on are in cpu_possible_mask, specifying rcupdate.rcu_task_enqueue_lim=3D4 would cause the code to attempt to use callback queues for non-existent CPUs 1, 2, and 3. Because such systems have existed in the past and might still exist, the code needs to gracefully handle this situation. This commit therefore checks to see whether the desired CPU is present in cpu_possible_mask, and, if not, searches for the next CPU. This means that the systems administrator of a system with a sparse cpu_possible_mask will need to account for this sparsity when specifying the value of the rcupdate.rcu_task_enqueue_lim kernel boot parameter. For example, setting this parameter to the value 4 will use only CPUs 0 and 4, which CPU 4 getting three times the callback load of CPU 0. This commit assumes that bit (nr_cpu_ids - 1) is always set in cpu_possible_mask. Link: https://lore.kernel.org/lkml/CANn89iKaNEwyNZ=3DL_PQnkH0LP_XjLYrr_dpyR= KNNoDJaWKdrmg@mail.gmail.com/ Signed-off-by: Eric Dumazet Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 3aad0dfbfaf4..fd70d86eb7cd 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -273,7 +273,9 @@ static void call_rcu_tasks_iw_wakeup(struct irq_work *i= wp) static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t fu= nc, struct rcu_tasks *rtp) { + int chosen_cpu; unsigned long flags; + int ideal_cpu; unsigned long j; bool needadjust =3D false; bool needwake; @@ -283,8 +285,9 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp= , rcu_callback_t func, rhp->func =3D func; local_irq_save(flags); rcu_read_lock(); - rtpcp =3D per_cpu_ptr(rtp->rtpcpu, - smp_processor_id() >> READ_ONCE(rtp->percpu_enqueue_shift)); + ideal_cpu =3D smp_processor_id() >> READ_ONCE(rtp->percpu_enqueue_shift); + chosen_cpu =3D cpumask_next(ideal_cpu - 1, cpu_possible_mask); + rtpcp =3D per_cpu_ptr(rtp->rtpcpu, chosen_cpu); if (!raw_spin_trylock_rcu_node(rtpcp)) { // irqs already disabled. raw_spin_lock_rcu_node(rtpcp); // irqs already disabled. j =3D jiffies; --=20 2.31.1.189.g2e36527f23 From nobody Mon May 11 02:06:54 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3FFCC433F5 for ; Tue, 19 Apr 2022 00:01:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236739AbiDSADk (ORCPT ); Mon, 18 Apr 2022 20:03:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236195AbiDSAC7 (ORCPT ); Mon, 18 Apr 2022 20:02:59 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8492B11145; Mon, 18 Apr 2022 17:00:19 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 27747B81141; Tue, 19 Apr 2022 00:00:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 94F09C385B7; Tue, 19 Apr 2022 00:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650326416; bh=4EFYBPM+fGUTWUUGcdM/Li8/8Rj9SLDD6se7yA60e4k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J99x+TkUhjHDOZsEQ0dz7mFkh00X4WYXNS3ZJ6SD5xAMIOFF8DlJbaDR8N7xzy+7P RRhB6UUmJzwcWAO/0BqkWQw9CO6m5fCmW4EOFcMy0TC9AK9K+ZcxigVsQqicd4aLdn dY+shGly3v0HR7R4flST8Cpps60IA6ys1V2CLBG516vM8KtTVgOgJTJ2DT4eqxgjvV C4FZPbVgp027G20MH8VF0RiHkREB5IcZMIHGtseZ9ddKN/3ALnWmOvO133u0RP4U56 whDggywIOBgHlv1d+bA3EDTbd3Pn6pjcKXIZCP033DXcqx2MMcuIg37YHstBiiH0u3 AnV6hcqxRUajA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id E6AB25C12AB; Mon, 18 Apr 2022 17:00:15 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Eric Dumazet Subject: [PATCH rcu 9/9] rcu-tasks: Handle sparse cpu_possible_mask in rcu_tasks_invoke_cbs() Date: Mon, 18 Apr 2022 17:00:14 -0700 Message-Id: <20220419000014.3948512-9-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> References: <20220419000007.GA3945818@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If the cpu_possible_mask is sparse (for example, if bits are set only for CPUs 0, 4, 8, ...), then rcu_tasks_invoke_cbs() will access per-CPU data for a CPU not in cpu_possible_mask. It makes these accesses while doing a workqueue-based binary search for non-empty callback lists. Although this search must pass through CPUs not represented in cpu_possible_mask, it has no need to check the callback list for such CPUs. This commit therefore changes the rcu_tasks_invoke_cbs() function's binary search so as to only check callback lists for CPUs present in cpu_possible_mask. Reported-by: Eric Dumazet Signed-off-by: Paul E. McKenney --- kernel/rcu/tasks.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index fd70d86eb7cd..3925e32159b5 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -468,7 +468,7 @@ static void rcu_tasks_invoke_cbs(struct rcu_tasks *rtp,= struct rcu_tasks_percpu } } =20 - if (rcu_segcblist_empty(&rtpcp->cblist)) + if (rcu_segcblist_empty(&rtpcp->cblist) || !cpu_possible(cpu)) return; raw_spin_lock_irqsave_rcu_node(rtpcp, flags); rcu_segcblist_advance(&rtpcp->cblist, rcu_seq_current(&rtp->tasks_gp_seq)= ); --=20 2.31.1.189.g2e36527f23