From nobody Tue Jun 23 01:10:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9577AC433EF for ; Mon, 14 Mar 2022 13:38:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241419AbiCNNjM (ORCPT ); Mon, 14 Mar 2022 09:39:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241536AbiCNNjH (ORCPT ); Mon, 14 Mar 2022 09:39:07 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 111B6BC9A for ; Mon, 14 Mar 2022 06:37:57 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4CEA161185 for ; Mon, 14 Mar 2022 13:37:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AB0E7C340F6; Mon, 14 Mar 2022 13:37:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1647265075; bh=rr6lMXcZFN4o3VbDgDdxjHH2wTI1ygkpoMFDut5z9oY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=X3bh7TI4xgrruUgSrEmv/4AuzHcpyQSpJIa0FuZaWaMv4mcshalAfoD36sdz+ns5s X3BaROjRbrIlIPwohmsov8p8LfOeXy12c3KnYFe4+8MHwo6XlEY35FttA0T0gi2zRh OIDWSmaSnyPEPsecwG/BgcvriCgoHZdQFHKu+EKI0GGrDLgtLYNvZDwvFi0hQmvhbY lLtL6tKBTTdgNkbBUx1zqRidLlNJLjGzR9+NAoqlCmsd4JBHxrKpxYhCrLRFGixUNp FtMThnu4v3lF1tbd08zq7JvgzlD+8Z5AMpcg+d3t08o8sG+14JeKNm5Zcupk3jatEH 3DynZcEX2q6bw== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Peter Zijlstra , Marco Elver , Neeraj Upadhyay , Valentin Schneider , Boqun Feng , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 1/3] rcu: Fix expedited GP polling against UP/no-preempt environment Date: Mon, 14 Mar 2022 14:37:36 +0100 Message-Id: <20220314133738.269522-2-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220314133738.269522-1-frederic@kernel.org> References: <20220314133738.269522-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" synchronize_rcu_expedited() has an early return condition: if the current CPU is the only one online and the kernel doesn't run in preemption mode, the current assumed quiescent state is worth a grace period. However the expedited grace period polling caller of synchronize_rcu_expedited() takes a GP sequence snapshot and expects it to complete by the end of the synchronize_rcu_expedited() call. Yet if synchronize_rcu_expedited() relies on the above described UP/no-preempt early return, the grace period sequence won't move and may cause an expedited grace period polling stall. Fix this with treating polling differently while calling synchronize_rcu_expedited() and ignore the UP-no-preempt optimization in this case. Reported-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Peter Zijlstra Cc: Neeraj Upadhyay Cc: Valentin Schneider --- kernel/rcu/tree_exp.h | 57 ++++++++++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 22 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index d5f30085b0cf..3d8216ced93e 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -794,27 +794,14 @@ static int rcu_print_task_exp_stall(struct rcu_node *= rnp) =20 #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ =20 -/** - * synchronize_rcu_expedited - Brute-force RCU grace period - * - * Wait for an RCU grace period, but expedite it. The basic idea is to - * IPI all non-idle non-nohz online CPUs. The IPI handler checks whether - * the CPU is in an RCU critical section, and if so, it sets a flag that - * causes the outermost rcu_read_unlock() to report the quiescent state - * for RCU-preempt or asks the scheduler for help for RCU-sched. On the - * other hand, if the CPU is not in an RCU read-side critical section, - * the IPI handler reports the quiescent state immediately. - * - * Although this is a great improvement over previous expedited - * implementations, it is still unfriendly to real-time workloads, so is - * thus not recommended for any sort of common-case code. In fact, if - * you are using synchronize_rcu_expedited() in a loop, please restructure - * your code to batch your updates, and then use a single synchronize_rcu() - * instead. - * - * This has the same semantics as (but is more brutal than) synchronize_rc= u(). +/* + * Start and wait for an expedited grace period completion. + * If it happens to be called by polling functions (@polling =3D true), + * there is no possible early return in UP no-preempt mode because + * the callers are waiting for an actual given sequence snapshot to start + * and end. */ -void synchronize_rcu_expedited(void) +static void __synchronize_rcu_expedited(bool polling) { bool boottime =3D (rcu_scheduler_active =3D=3D RCU_SCHEDULER_INIT); struct rcu_exp_work rew; @@ -827,7 +814,7 @@ void synchronize_rcu_expedited(void) "Illegal synchronize_rcu_expedited() in RCU read-side critical section= "); =20 /* Is the state is such that the call is a grace period? */ - if (rcu_blocking_is_gp()) + if (rcu_blocking_is_gp() && !polling) return; =20 /* If expedited grace periods are prohibited, fall back to normal. */ @@ -863,6 +850,32 @@ void synchronize_rcu_expedited(void) =20 if (likely(!boottime)) destroy_work_on_stack(&rew.rew_work); + +} + +/** + * synchronize_rcu_expedited - Brute-force RCU grace period + * + * Wait for an RCU grace period, but expedite it. The basic idea is to + * IPI all non-idle non-nohz online CPUs. The IPI handler checks whether + * the CPU is in an RCU critical section, and if so, it sets a flag that + * causes the outermost rcu_read_unlock() to report the quiescent state + * for RCU-preempt or asks the scheduler for help for RCU-sched. On the + * other hand, if the CPU is not in an RCU read-side critical section, + * the IPI handler reports the quiescent state immediately. + * + * Although this is a great improvement over previous expedited + * implementations, it is still unfriendly to real-time workloads, so is + * thus not recommended for any sort of common-case code. In fact, if + * you are using synchronize_rcu_expedited() in a loop, please restructure + * your code to batch your updates, and then use a single synchronize_rcu() + * instead. + * + * This has the same semantics as (but is more brutal than) synchronize_rc= u(). + */ +void synchronize_rcu_expedited(void) +{ + __synchronize_rcu_expedited(false); } EXPORT_SYMBOL_GPL(synchronize_rcu_expedited); =20 @@ -903,7 +916,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *w= p) if (s & 0x1) return; while (!sync_exp_work_done(s)) - synchronize_rcu_expedited(); + __synchronize_rcu_expedited(true); raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); s =3D rnp->exp_seq_poll_rq; if (!(s & 0x1) && !sync_exp_work_done(s)) --=20 2.25.1 From nobody Tue Jun 23 01:10:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CB32C433EF for ; Mon, 14 Mar 2022 13:38:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241896AbiCNNjV (ORCPT ); Mon, 14 Mar 2022 09:39:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241863AbiCNNjO (ORCPT ); Mon, 14 Mar 2022 09:39:14 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 498FC1119 for ; Mon, 14 Mar 2022 06:38:00 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 64E8AB80DF4 for ; Mon, 14 Mar 2022 13:37:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 28375C340ED; Mon, 14 Mar 2022 13:37:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1647265078; bh=jk6umewZFxLd1Bf2B9rzTYEamkBtxaQMtyRXWqXjK5A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TKfpqy3ZScfBv4im+6QkVJ2j+hVA+h1uU3wriYMhuXUNZBxFRLWsePk0v1wZusbnX 0q0YFvXFwecwBCtTmJgWZOI/SsubvaBd5lpApSyErtV9tP4PGG7a4TDjz49QtWuZIG WzDJ1zmlXqWunGCdlDeo5jVbKjuOFv/52iSS+mI2uvuw6VFIvabFzglHiDiGoNyzoX tBnMMLWEETg3yI6TGrSEgXXTnl1ml10/7SXJfsSi27mTSvsbQ3Er0uODB5rSQrxEWr UHZgJEHjrJWjE7kdpAlYCPUkKLweEK2KRubcDMEJIObN61Qadhg8mnyvf934CI0NQS i1axDfgMDsBbg== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Valentin Schneider , Peter Zijlstra , Marco Elver , Neeraj Upadhyay , Frederic Weisbecker , Boqun Feng , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 2/3] preempt/dynamic: Introduce preempt mode accessors Date: Mon, 14 Mar 2022 14:37:37 +0100 Message-Id: <20220314133738.269522-3-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220314133738.269522-1-frederic@kernel.org> References: <20220314133738.269522-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Valentin Schneider CONFIG_PREEMPT{_NONE, _VOLUNTARY} designate either: o The build-time preemption model when !PREEMPT_DYNAMIC o The default boot-time preemption model when PREEMPT_DYNAMIC IOW, using those on PREEMPT_DYNAMIC kernels is meaningless - the actual model could have been set to something else by the "preempt=3Dfoo" cmdline parameter. Introduce a set of helpers to determine the actual preemption mode used by the live kernel. Suggested-by: Marco Elver Signed-off-by: Valentin Schneider Signed-off-by: Frederic Weisbecker Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Peter Zijlstra Cc: Neeraj Upadhyay Reported-by: Paul E. McKenney --- include/linux/sched.h | 16 ++++++++++++++++ kernel/sched/core.c | 11 +++++++++++ 2 files changed, 27 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 508b91d57470..d348e886e4d0 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2096,6 +2096,22 @@ static inline void cond_resched_rcu(void) #endif } =20 +#ifdef CONFIG_PREEMPT_DYNAMIC + +extern bool preempt_mode_none(void); +extern bool preempt_mode_voluntary(void); +extern bool preempt_mode_full(void); + +#else + +#define preempt_mode_none() IS_ENABLED(CONFIG_PREEMPT_NONE) +#define preempt_mode_voluntary() IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) +#define preempt_mode_full() IS_ENABLED(CONFIG_PREEMPT) + +#endif + +#define preempt_mode_rt() IS_ENABLED(CONFIG_PREEMPT_RT) + /* * Does a critical section need to be broken due to another * task waiting?: (technically does not depend on CONFIG_PREEMPTION, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 2e4ae00e52d1..6e5c8f0f874a 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6684,6 +6684,17 @@ static void __init preempt_dynamic_init(void) } } =20 +#define PREEMPT_MODE_ACCESSOR(mode) \ + bool preempt_mode_##mode(void) \ + { \ + WARN_ON_ONCE(preempt_dynamic_mode =3D=3D preempt_dynamic_undefined); \ + return preempt_dynamic_mode =3D=3D preempt_dynamic_##mode; \ + } + +PREEMPT_MODE_ACCESSOR(none) +PREEMPT_MODE_ACCESSOR(voluntary) +PREEMPT_MODE_ACCESSOR(full) + #else /* !CONFIG_PREEMPT_DYNAMIC */ =20 static inline void preempt_dynamic_init(void) { } --=20 2.25.1 From nobody Tue Jun 23 01:10:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15C2DC433FE for ; Mon, 14 Mar 2022 13:38:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241946AbiCNNjY (ORCPT ); Mon, 14 Mar 2022 09:39:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241865AbiCNNjO (ORCPT ); Mon, 14 Mar 2022 09:39:14 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 359ADBC2D for ; Mon, 14 Mar 2022 06:38:02 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 394676118A for ; Mon, 14 Mar 2022 13:38:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9AC3EC340F4; Mon, 14 Mar 2022 13:37:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1647265080; bh=nF/6Lmz1r80SPWKXpwqXTgYh6Kwp/g/JtR3krTnNLFA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kMt8EzGg6jjebnIXApidbamvN96dj89DXDxybsVMYWVpADdfqZbMgK0L+NMXH6ljc OnNS/oOBBYEsOrwFaa4cJ9hMH+TJioRwBbSCvL7CSdNyAVPS60mFZ2gykQhicQDPhr vRJ68rfOoJs1ItVfSLXQCSDfefnMzthgHhsuQtil8owAw2MFLchrEhDbzS5zC+0Efb qe1PBnzYW4TTqkEQzIASN/T6X5STNDdu9xRev2pnakFlHYzFBHRM5p/4dOuwxn7GA+ ZXh6mdpwVeS4V0sAkfw0MVr5rMDxnoIrYhoDbc+ZuddUqj5VW4H95khNFu39k2PcqK CcxeH9K2aE2Mw== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Peter Zijlstra , Marco Elver , Neeraj Upadhyay , Valentin Schneider , Boqun Feng , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 3/3] rcu: Fix preemption mode check on synchronize_rcu[_expedited]() Date: Mon, 14 Mar 2022 14:37:38 +0100 Message-Id: <20220314133738.269522-4-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220314133738.269522-1-frederic@kernel.org> References: <20220314133738.269522-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" An early check on synchronize_rcu[_expedited]() tries to determine if the current CPU is in UP mode on an SMP no-preempt kernel, in which case there is no need to start a grace period since the current assumed quiescent state is all we need. However the preemption mode doesn't take into account the boot selected preemption mode under CONFIG_PREEMPT_DYNAMIC=3Dy, missing a possible early return if the running flavour is "none" or "voluntary". Use the shiny new preempt mode accessors to fix this. Reported-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Peter Zijlstra Cc: Neeraj Upadhyay Cc: Valentin Schneider --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d2db709f83d9..8fe2f9843139 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3769,7 +3769,7 @@ static int rcu_blocking_is_gp(void) { int ret; =20 - if (IS_ENABLED(CONFIG_PREEMPTION)) + if (preempt_mode_full() || preempt_mode_rt()) return rcu_scheduler_active =3D=3D RCU_SCHEDULER_INACTIVE; might_sleep(); /* Check for RCU read-side critical section. */ preempt_disable(); --=20 2.25.1