From nobody Mon Jun 22 21:33:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE494C433EF for ; Wed, 16 Mar 2022 14:43:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356819AbiCPOoX (ORCPT ); Wed, 16 Mar 2022 10:44:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349287AbiCPOoT (ORCPT ); Wed, 16 Mar 2022 10:44:19 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D035321E12 for ; Wed, 16 Mar 2022 07:43:03 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6C0B16157E for ; Wed, 16 Mar 2022 14:43:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 439B7C340F0; Wed, 16 Mar 2022 14:43:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1647441782; bh=Btxl5/3u4gskbSG4b4/bNPaMjoqQkhs9Zqi5gM6zYuQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=A7L6ywOzJGQ2utyWu78gOXP+PxjR/PLmPS90laso5pyGHal9pMDd13p5qdC6LoQ2h q9B3FOVK2bTW1zoFOvRM/TRE0Ysw1B55RsYek8FgHzviftqmZZwpPvX1JamBsiBK01 sk50SEEKPX9IAgqvwcariiobAf0LVVbdB/8qso0YiZ/X5Cr7/A2HjIo7rkHyR+0fZo 65bRkAYATjY9upSLZrWbPrcMUdt0fXVktswB9C5MDE1rLD5GoCjN/RKSCJ8x1P8s7U hgAuSstLvXO94sLozcxaf7vVRna5wEFF/pn1Dy1b60lkMko1tXKrq0wa0Y27Ta1gNJ 7qNPPjn3QgHsw== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Uladzislau Rezki , Boqun Feng , Neeraj Upadhyay , Joel Fernandes Subject: [PATCH 1/4] rcu: Remove needless polling work requeue for further waiter Date: Wed, 16 Mar 2022 15:42:52 +0100 Message-Id: <20220316144255.336021-2-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220316144255.336021-1-frederic@kernel.org> References: <20220316144255.336021-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If another expedited polling site is waiting for the next grace period, there is no need to requeue the work because it is guaranteed to be either already queued or executing the actual polling upon completion. Signed-off-by: Frederic Weisbecker Cc: Neeraj Upadhyay Cc: Boqun Feng Cc: Uladzislau Rezki Cc: Joel Fernandes --- kernel/rcu/tree_exp.h | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index a6cb02a4d27c..b6fd857f34ba 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -919,9 +919,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *w= p) __synchronize_rcu_expedited(true); raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); s =3D rnp->exp_seq_poll_rq; - if (!(s & 0x1) && !sync_exp_work_done(s)) - queue_work(rcu_gp_wq, &rnp->exp_poll_wq); - else + if (!(s & 0x1) && sync_exp_work_done(s)) rnp->exp_seq_poll_rq |=3D 0x1; raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags); } --=20 2.25.1 From nobody Mon Jun 22 21:33:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63124C433F5 for ; Wed, 16 Mar 2022 14:43:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356842AbiCPOoa (ORCPT ); Wed, 16 Mar 2022 10:44:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50416 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356572AbiCPOoU (ORCPT ); Wed, 16 Mar 2022 10:44:20 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDCE85FF9 for ; Wed, 16 Mar 2022 07:43:05 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6D3D561583 for ; Wed, 16 Mar 2022 14:43:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 49447C340F1; Wed, 16 Mar 2022 14:43:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1647441784; bh=lHeihKp98S9nAlrutKy8Jq1rETYQC/dxIy6QH3t9vAA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=j2szZO8x3DJqWy2wda/Vbq6jPT3EaU6raecAs28R0PCkU6cFabavYkx1Xpw9fxc1v Q4M7qJDzdIwZUP94V8HzUdYENuyZZoOcYo6c8jejCV7QNgYvcw1SMAbbGf5zbxon0+ ydPeRkiMXOaQ0VjjwDAHYS3WdO3VGE9gUzuMX2ucPJ607JnaMHSDNnKyxiFC3QIS1J bNXRxZeMo9s6VbffV+GtjjBJmMUf87yWbR1wJZHpV/FAeyhq5MWeCEUZtyziw1vmAG +tN85AFFV2GXYPYu/I2OMex4k4k33ycpbaNo76kMvB7kq3BXqtJcvTH6hqrp/1pcxf KCzpv/Qfo2yrw== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Uladzislau Rezki , Boqun Feng , Neeraj Upadhyay , Joel Fernandes Subject: [PATCH 2/4] rcu: No need to reset the poll request flag before completion Date: Wed, 16 Mar 2022 15:42:53 +0100 Message-Id: <20220316144255.336021-3-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220316144255.336021-1-frederic@kernel.org> References: <20220316144255.336021-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The flag allowing to requeue the polling work is reset before the polling even starts. However there is no point in having two competing polling on the same grace period. Just reset the flag once we have completed the grace period only. Signed-off-by: Frederic Weisbecker Cc: Neeraj Upadhyay Cc: Boqun Feng Cc: Uladzislau Rezki Cc: Joel Fernandes --- kernel/rcu/tree_exp.h | 1 - 1 file changed, 1 deletion(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index b6fd857f34ba..763ec35546ed 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -911,7 +911,6 @@ static void sync_rcu_do_polled_gp(struct work_struct *w= p) =20 raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); s =3D rnp->exp_seq_poll_rq; - rnp->exp_seq_poll_rq |=3D 0x1; raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags); if (s & 0x1) return; --=20 2.25.1 From nobody Mon Jun 22 21:33:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2781FC433EF for ; Wed, 16 Mar 2022 14:43:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356833AbiCPOo1 (ORCPT ); Wed, 16 Mar 2022 10:44:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356818AbiCPOoX (ORCPT ); Wed, 16 Mar 2022 10:44:23 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B47D23BD3 for ; Wed, 16 Mar 2022 07:43:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 40A48B81BF9 for ; Wed, 16 Mar 2022 14:43:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4FC6FC340F0; Wed, 16 Mar 2022 14:43:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1647441786; bh=+MPNWmYaV22ZBHxQaIDswT0l2QT2UXXflRMR8tnJfho=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Lt/yHnmWLDYkpm1P5DbK9fKk0I5ZrK80pJrO0ZaZUHBMoTl+TjmIqQxDSmQGLCXg3 89pYhEsXau9n/j5BwolMB2sOidW3N9+ZNIB04iPEEYmHqYJMd92HOVFzsj/pS3OChH orrSJaVostvTj9KgsEdhj1f5HaYvZONauYzircQ51fzuGxYKJGhCUfhHK52ydZLoEh TXkAH/OfGRXOudwWFXp97dqbKJ4G1vXI4upUk62nPyQ4KmGhWQOVAYdHKbLawuxqWI uxPEwxmwhSgUnOZ8mEteez2DO9/B9yyGpl/LOWprGsF7I/7X3m7xD8oR4WZt006OFs WRhYH6jx0W5ug== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Uladzislau Rezki , Boqun Feng , Neeraj Upadhyay , Joel Fernandes Subject: [PATCH 3/4] rcu: Perform early sequence fetch for polling locklessly Date: Wed, 16 Mar 2022 15:42:54 +0100 Message-Id: <20220316144255.336021-4-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220316144255.336021-1-frederic@kernel.org> References: <20220316144255.336021-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The workqueue ordering guarantees that the work sees all the accesses of the task prior to its call to the corresponding queue_work(). Therefore the sequence to poll can be retrieved locklessly. The only downside is that it is then possible to miss the 0x1 flag set by a prior work. But this could already happen concurrently anyway after the exp_poll_lock is unlocked. In the worst case the slow path involving synchronize_rcu_expedited() takes care of the situation. Signed-off-by: Frederic Weisbecker Cc: Neeraj Upadhyay Cc: Boqun Feng Cc: Uladzislau Rezki Cc: Joel Fernandes --- kernel/rcu/tree_exp.h | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 763ec35546ed..c4a19c6a83cf 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -909,9 +909,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *w= p) struct rcu_node *rnp =3D container_of(wp, struct rcu_node, exp_poll_wq); unsigned long s; =20 - raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); - s =3D rnp->exp_seq_poll_rq; - raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags); + s =3D READ_ONCE(rnp->exp_seq_poll_rq); if (s & 0x1) return; while (!sync_exp_work_done(s)) @@ -919,7 +917,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *w= p) raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); s =3D rnp->exp_seq_poll_rq; if (!(s & 0x1) && sync_exp_work_done(s)) - rnp->exp_seq_poll_rq |=3D 0x1; + WRITE_ONCE(rnp->exp_seq_poll_rq, s | 0x1); raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags); } =20 @@ -949,7 +947,7 @@ unsigned long start_poll_synchronize_rcu_expedited(void) if (rcu_init_invoked()) raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); if ((rnp->exp_seq_poll_rq & 0x1) || ULONG_CMP_LT(rnp->exp_seq_poll_rq, s)= ) { - rnp->exp_seq_poll_rq =3D s; + WRITE_ONCE(rnp->exp_seq_poll_rq, s); if (rcu_init_invoked()) queue_work(rcu_gp_wq, &rnp->exp_poll_wq); } --=20 2.25.1 From nobody Mon Jun 22 21:33:27 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91DC8C433F5 for ; Wed, 16 Mar 2022 14:43:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356848AbiCPOod (ORCPT ); Wed, 16 Mar 2022 10:44:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349287AbiCPOoY (ORCPT ); Wed, 16 Mar 2022 10:44:24 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E056423BD9 for ; Wed, 16 Mar 2022 07:43:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7C8876157E for ; Wed, 16 Mar 2022 14:43:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5841BC340E9; Wed, 16 Mar 2022 14:43:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1647441788; bh=vCDDcFPUQSzH6TzPOyC9N+1Fa+Z5jBQqQSiXx5AHDfY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hTnEfioT1MiaIJDcbMDSF2ozD5jg1PGrdv+V5vQabLfC44BrAsVisMVOxAfF5qGLX WU4wQ/uVk92KAamwdJYA0oSlYTDXEZwUbpQgft6ptEPapS96wIa+xpsxtdT8FMhHZ0 XgCJf7UD27qhqlCpLf1XeKcTPXqEq8tQ4NKaTLgEU99g1mnZBA+KAyekKUp986HQA9 jAHrtssaz/DzHlR0S2w+aKK/7vhHGgCq268y1f+xHgQAb1mF6imcfQ52Fkpj7VGR0E FRnSshQyf04fqbdd25nYZZUc2+Vwc/Q0jpYvne8b1sNAGHTjcI8td01xcEwTMKgJJe oDYKtGQ3JX7EA== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Uladzislau Rezki , Boqun Feng , Neeraj Upadhyay , Joel Fernandes Subject: [PATCH 4/4] rcu: Name internal polling flag Date: Wed, 16 Mar 2022 15:42:55 +0100 Message-Id: <20220316144255.336021-5-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220316144255.336021-1-frederic@kernel.org> References: <20220316144255.336021-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Give a proper self-explanatory name to the expedited grace period internal polling flag. Signed-off-by: Frederic Weisbecker Cc: Neeraj Upadhyay Cc: Boqun Feng Cc: Uladzislau Rezki Cc: Joel Fernandes --- kernel/rcu/rcu.h | 5 +++++ kernel/rcu/tree.c | 2 +- kernel/rcu/tree_exp.h | 9 +++++---- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index eccbdbdaa02e..8a62bb416ba4 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -30,6 +30,11 @@ #define RCU_GET_STATE_USE_NORMAL 0x2 #define RCU_GET_STATE_BAD_FOR_NORMAL (RCU_GET_STATE_FROM_EXPEDITED | RCU_G= ET_STATE_USE_NORMAL) =20 +/* + * Low-order bit definitions for polled grace-period internals. + */ +#define RCU_EXP_SEQ_POLL_DONE 0x1 + /* * Return the counter portion of a sequence number previously returned * by rcu_seq_snap() or rcu_seq_current(). diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5da381a3cbe5..b3223b365f9f 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4679,7 +4679,7 @@ static void __init rcu_init_one(void) spin_lock_init(&rnp->exp_lock); mutex_init(&rnp->boost_kthread_mutex); raw_spin_lock_init(&rnp->exp_poll_lock); - rnp->exp_seq_poll_rq =3D 0x1; + rnp->exp_seq_poll_rq =3D RCU_EXP_SEQ_POLL_DONE; INIT_WORK(&rnp->exp_poll_wq, sync_rcu_do_polled_gp); } } diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index c4a19c6a83cf..7ccb909d6355 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -910,14 +910,14 @@ static void sync_rcu_do_polled_gp(struct work_struct = *wp) unsigned long s; =20 s =3D READ_ONCE(rnp->exp_seq_poll_rq); - if (s & 0x1) + if (s & RCU_EXP_SEQ_POLL_DONE) return; while (!sync_exp_work_done(s)) __synchronize_rcu_expedited(true); raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); s =3D rnp->exp_seq_poll_rq; - if (!(s & 0x1) && sync_exp_work_done(s)) - WRITE_ONCE(rnp->exp_seq_poll_rq, s | 0x1); + if (!(s & RCU_EXP_SEQ_POLL_DONE) && sync_exp_work_done(s)) + WRITE_ONCE(rnp->exp_seq_poll_rq, s | RCU_EXP_SEQ_POLL_DONE); raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags); } =20 @@ -946,7 +946,8 @@ unsigned long start_poll_synchronize_rcu_expedited(void) rnp =3D rdp->mynode; if (rcu_init_invoked()) raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); - if ((rnp->exp_seq_poll_rq & 0x1) || ULONG_CMP_LT(rnp->exp_seq_poll_rq, s)= ) { + if ((rnp->exp_seq_poll_rq & RCU_EXP_SEQ_POLL_DONE) || + ULONG_CMP_LT(rnp->exp_seq_poll_rq, s)) { WRITE_ONCE(rnp->exp_seq_poll_rq, s); if (rcu_init_invoked()) queue_work(rcu_gp_wq, &rnp->exp_poll_wq); --=20 2.25.1