From nobody Fri Apr 3 06:44:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 249A8C6FA89 for ; Thu, 15 Sep 2022 00:14:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229705AbiIOAOk (ORCPT ); Wed, 14 Sep 2022 20:14:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41174 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229936AbiIOAOg (ORCPT ); Wed, 14 Sep 2022 20:14:36 -0400 Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A12E5F6E for ; Wed, 14 Sep 2022 17:14:34 -0700 (PDT) Received: by mail-qk1-x72b.google.com with SMTP id 3so11757336qka.5 for ; Wed, 14 Sep 2022 17:14:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=A1vsCw7qsSbg6LG6s3lCEoHneJ7sDD1rTCb2gsMBhiI=; b=PVl0MtbuC8jP0UnDsRzD+DryiQcwfi3cdRxo1mYiLGB8fjmQS20KFseigIWbxllokR COcSb6OcEDsD5gucsUYep4hzLW6CW/pF7H6hf7a7LE9R+G6dJUVv+ekd/+7NxLMJCCVL 4QUu2IzMbm6D2kPkwWMzqte+24i+TJwQtpCTU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=A1vsCw7qsSbg6LG6s3lCEoHneJ7sDD1rTCb2gsMBhiI=; b=hMtEzw69nzRPd4tPoXlbywgmRPSzGKCGC3dd4zV2fP7c6xC1HLVdO2b5Mf6t4dD0lb uCGFrM5NQII/jlOatjCmPqw0U8P2zFzFM1Jxo+d10etP2luVqhgFd18zMuaBFkYeUlnx ZulMropoG7smLttNdipcHHvX3RvmyKNwUl4hEWC92BjerIMYY0uJD+e97lO9QjZN/3FA 8piCTXdV6D9f5GYql33RIzSdTmdvXMOEmNlFPgcAyvduyDaqV/R+S8f8w7RinhLmZ4gm PwnAsFDWppbGtz+FRJdTsjZabkj7nWcVczj4f0JGShCzcyLjHAJpVVL8Zdv6w4iY4qre 1GzA== X-Gm-Message-State: ACgBeo3ksSqJif8U85UBUp5ZjuaGecBcYfBSqCmkM73dOnUgK15dsAyS mT/BXITMwZ5poE6YIqJ21EDW8GLwP1QritWBgSM= X-Google-Smtp-Source: AA6agR7skx+qmUrnXxbC9eolRQX/E9WCYME4MPsQJ6mEMoqK9nsQdr33ypJE00CInZ5jxffolQVMlw== X-Received: by 2002:a05:620a:454e:b0:6ce:3c11:24ad with SMTP id u14-20020a05620a454e00b006ce3c1124admr13034044qkp.606.1663200873386; Wed, 14 Sep 2022 17:14:33 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id do11-20020a05620a2b0b00b006bbb07ebd83sm2872422qkb.108.2022.09.14.17.14.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Sep 2022 17:14:32 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Frederic Weisbecker , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , "Paul E. McKenney" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH rcu/next 1/3] rcu/tree: Use READ_ONCE() for lockless read of rnp->qsmask Date: Thu, 15 Sep 2022 00:14:17 +0000 Message-Id: <20220915001419.55617-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220915001419.55617-1-joel@joelfernandes.org> References: <20220915001419.55617-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The rnp->qsmask is locklessly accessed from rcutree_dying_cpu(). This may help avoid load tearing due to concurrent access, KCSAN issues, and preserve sanity of people reading the mask in tracing. Signed-off-by: Joel Fernandes (Google) Reviewed-by: Frederic Weisbecker --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 0ca21ac0f064..5ec97e3f7468 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2106,7 +2106,7 @@ int rcutree_dying_cpu(unsigned int cpu) if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) return 0; =20 - blkd =3D !!(rnp->qsmask & rdp->grpmask); + blkd =3D !!(READ_ONCE(rnp->qsmask) & rdp->grpmask); trace_rcu_grace_period(rcu_state.name, READ_ONCE(rnp->gp_seq), blkd ? TPS("cpuofl-bgp") : TPS("cpuofl")); return 0; --=20 2.37.2.789.g6183377224-goog From nobody Fri Apr 3 06:44:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E213ECAAD3 for ; Thu, 15 Sep 2022 00:14:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230032AbiIOAOw (ORCPT ); Wed, 14 Sep 2022 20:14:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229978AbiIOAOi (ORCPT ); Wed, 14 Sep 2022 20:14:38 -0400 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3346264E5 for ; Wed, 14 Sep 2022 17:14:35 -0700 (PDT) Received: by mail-qk1-x731.google.com with SMTP id q11so9098621qkc.12 for ; Wed, 14 Sep 2022 17:14:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=7NTlW4AVEaEifdUDNJKG5n5TSALhIocTrw3kCCpTxNA=; b=aufasvy68P6Ewg+Se2UzkhM+dsbLyQxC4Y8rMgj0/dn7wakhHYWGMOKsnUKG3S4ABp 6EucXs9y5wMo4i5DJQlt8fqTu/P8aaNCEAyBL7IzBTgXEuHN/qUGlR8gR2BWlMi8Mm7/ 9WKL7uRA0DCxmbHJoZcawp2vF9OiRs32DNSDU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=7NTlW4AVEaEifdUDNJKG5n5TSALhIocTrw3kCCpTxNA=; b=pEZ770tLeMcB4QypZ4HP4BhHiuWG34BHTNvOXVCXglVvVahQa7kDPJYMCJpcmhpwZ+ z0LzmIHC13KhGwb1ub7ntbcNuCbGSxZOyRZ6RnDSc/zR4g8xZbxF8d6ux5HiOAUWxRYG lMOMfvNDEVfLJfPFr010ZZYddSiWR4gaf2OWsZB4C7qDMAHI1GawsgpVH9L6MXfsMQpn y8roI/Ct8ZDv2AaYzxcuSBrpxXfXFLKXJkTIzZHkRapxO35cQCI1zZRYnfG38o90IF+6 B4rCnBrFK7hNWn4tJwI+Fa3Sk0GbE5hXRS4xznWKvCw4iIHUSAD7fZa4yjjUae238xCl 3VUA== X-Gm-Message-State: ACgBeo186fSBJOGqHICNRfyAdrvMSs1PmSxJy4psFmX20v6oPz5AuEdN Yky48A3yssj0aOfWYFcZzZncNtG8Uz9Uo9cc8eo= X-Google-Smtp-Source: AA6agR4dx+hoiCK3T5tFVcbkeQk/wU2na31F054HHTCbDwmPVtIe+kG4dyVZZowUw7+G+FYkfxNT4A== X-Received: by 2002:a05:620a:2491:b0:6bb:8982:8f66 with SMTP id i17-20020a05620a249100b006bb89828f66mr28190732qkn.665.1663200874404; Wed, 14 Sep 2022 17:14:34 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id do11-20020a05620a2b0b00b006bbb07ebd83sm2872422qkb.108.2022.09.14.17.14.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Sep 2022 17:14:33 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Frederic Weisbecker , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , "Paul E. McKenney" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH rcu/next 2/3] rcu: Move trace_rcu_callback() before bypassing Date: Thu, 15 Sep 2022 00:14:18 +0000 Message-Id: <20220915001419.55617-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220915001419.55617-1-joel@joelfernandes.org> References: <20220915001419.55617-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If any CB is queued into the bypass list, then trace_rcu_callback() does not show it. This makes it not clear when a callback was actually queued, as you only end up getting a trace_rcu_invoke_callback() trace. Fix it by moving trace_rcu_callback() before trace_rcu_nocb_try_bypass(). Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5ec97e3f7468..9fe581be8696 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2809,10 +2809,7 @@ void call_rcu(struct rcu_head *head, rcu_callback_t = func) } =20 check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) - return; // Enqueued onto ->nocb_bypass, so just leave. - // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. - rcu_segcblist_enqueue(&rdp->cblist, head); + if (__is_kvfree_rcu_offset((unsigned long)func)) trace_rcu_kvfree_callback(rcu_state.name, head, (unsigned long)func, @@ -2821,6 +2818,11 @@ void call_rcu(struct rcu_head *head, rcu_callback_t = func) trace_rcu_callback(rcu_state.name, head, rcu_segcblist_n_cbs(&rdp->cblist)); =20 + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + return; // Enqueued onto ->nocb_bypass, so just leave. + // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. + rcu_segcblist_enqueue(&rdp->cblist, head); + trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); =20 /* Go handle any RCU core processing required. */ --=20 2.37.2.789.g6183377224-goog From nobody Fri Apr 3 06:44:48 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52B27C6FA82 for ; Thu, 15 Sep 2022 00:14:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230017AbiIOAO4 (ORCPT ); Wed, 14 Sep 2022 20:14:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229984AbiIOAOi (ORCPT ); Wed, 14 Sep 2022 20:14:38 -0400 Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5FD92718 for ; Wed, 14 Sep 2022 17:14:36 -0700 (PDT) Received: by mail-qk1-x736.google.com with SMTP id c19so10235673qkm.7 for ; Wed, 14 Sep 2022 17:14:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=SqiD/rmTmUnZN+sxp6myZ+MDQ3Vwh3eOQpZ6rdR1rxE=; b=ZLcx7NHB52/qzkYWSR8ckkf1gareZ4GmzTiq/hCOgGnRBEHkz0NxGweHu1K3EVjRIg G3fMULXiX7AFK9DZkvhNqmCZECLF0CkEWQAp6MDA9wyeA6dFgX5uV1lxs9PA0qPeJ9Kz BW7OgVuE0HCV8Sui9C6J77DMSvj8TO6/fRzB8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=SqiD/rmTmUnZN+sxp6myZ+MDQ3Vwh3eOQpZ6rdR1rxE=; b=Jhjh66UXA28tCB9UVV2v0SdOXufiQ4ntHNeTzrXA/QbvgYoGcwU48XFfN/BAdK35lb nroROPHeqhxU52zADno63F29Ai2zxKBDXHNdx+BMN+LdogvPRZgfMHDfxbFQtAI3r+Eo qnBdoBuZxpwQOdcyVMyO+NQJcadrlCiexISLlejj3OvbmFSdwzBEg+erxkhipP1GL6tT thnYw3VzeLsH4uJXtfNp7Q2IE/AUlX+a1FfBpVxN7ekKLiIc5HEvU7r/9fD3ioMHMZGV 7x37/OZCr+tATUBbUYsNgpaAsN1/MQH6SsZa4FmCRLEdmD8Sx632YM2BBuv8NoqtNR9w fI9g== X-Gm-Message-State: ACgBeo270nlJYk0p9pc9w118dO9k+U5Ejj9tKHyIAinoj5AOuu/7a5K2 BVV1xeXbKh744GftIR65QXDiTMQGmkoNfUMUQs0= X-Google-Smtp-Source: AA6agR6Jaelol8J/6+LJ4V0dMBTzfQIqYm7WXyULloG4qJkDuOxHTYKXfQ9I2xuKCi40lhnAtDyzpw== X-Received: by 2002:a05:620a:318a:b0:6ce:732a:f92 with SMTP id bi10-20020a05620a318a00b006ce732a0f92mr6969342qkb.347.1663200875110; Wed, 14 Sep 2022 17:14:35 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id do11-20020a05620a2b0b00b006bbb07ebd83sm2872422qkb.108.2022.09.14.17.14.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Sep 2022 17:14:34 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org Cc: "Joel Fernandes (Google)" , Frederic Weisbecker , Josh Triplett , Lai Jiangshan , Mathieu Desnoyers , Neeraj Upadhyay , "Paul E. McKenney" , rcu@vger.kernel.org, Steven Rostedt Subject: [PATCH rcu/next 3/3] rcu: Fix late wakeup when flush of bypass cblist happens Date: Thu, 15 Sep 2022 00:14:19 +0000 Message-Id: <20220915001419.55617-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog In-Reply-To: <20220915001419.55617-1-joel@joelfernandes.org> References: <20220915001419.55617-1-joel@joelfernandes.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When the bypass cblist gets too big or its timeout has occurred, it is flushed into the main cblist. However, the bypass timer is still running and the behavior is that it would eventually expire and wake the GP thread. Since we are going to use the bypass cblist for lazy CBs, do the wakeup soon as the flush happens. Otherwise, the lazy-timer will go off much later and the now-non-lazy cblist CBs can get stranded for the duration of the timer. This is a good thing to do anyway, since it makes the behavior consistent w= ith behavior of other code paths where queueing something into the ->cblist mak= es the GP kthread in a non-sleeping state quickly. Signed-off-by: Joel Fernandes (Google) Reviewed-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 0a5f0ef41484..04c87f250e01 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -433,8 +433,9 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, s= truct rcu_head *rhp, if ((ncbs && j !=3D READ_ONCE(rdp->nocb_bypass_first)) || ncbs >=3D qhimark) { rcu_nocb_lock(rdp); + *was_alldone =3D !rcu_segcblist_pend_cbs(&rdp->cblist); + if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { - *was_alldone =3D !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); @@ -447,7 +448,12 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, = struct rcu_head *rhp, rcu_advance_cbs_nowake(rdp->mynode, rdp); rdp->nocb_gp_adv_time =3D j; } - rcu_nocb_unlock_irqrestore(rdp, flags); + + // The flush succeeded and we moved CBs into the regular list. + // Don't wait for the wake up timer as it may be too far ahead. + // Wake up the GP thread now instead, if the cblist was empty. + __call_rcu_nocb_wake(rdp, *was_alldone, flags); + return true; // Callback already enqueued. } =20 --=20 2.37.2.789.g6183377224-goog