From nobody Fri Dec 19 17:33:17 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C418C6FD1C for ; Wed, 22 Mar 2023 19:45:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230022AbjCVTpY (ORCPT ); Wed, 22 Mar 2023 15:45:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229838AbjCVTpS (ORCPT ); Wed, 22 Mar 2023 15:45:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6135E3E0BD; Wed, 22 Mar 2023 12:45:07 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EF741622B2; Wed, 22 Mar 2023 19:45:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8E6C7C4339E; Wed, 22 Mar 2023 19:45:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679514306; bh=a59gUDfTCHexacSrsbU3Hhq+A3tfJIUWFd5dyE5PoH0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KsIagmLyRTqKsfR7uIUg9MjUB6redqBnrKvV76N6o1Vx+jY7uIq9Y/alf5qVC/2ey ohoXznoXXIOPSCCOrKmSUnBTPG7XgSSOYnAU0cwsvTb1jWWtbM7mUf8bn8zfxUqq7d sUKfStasnjoGheU7ibY1W95SuRdThbFK9Ec/iAOHru2Z9iREVbwiQjm2+mgwBrHgpT Rv8lUWHtGFZv1KFuOaULMtS5KzIMTqZzlDzOsR6T4Fl82KuJ/xakJuFUd6056Ubqmc H7gSghF0E0NFS/fX2z1J5oKfP42YjQUcRkcAmriowRcVz/Mqs+RrElKEEgy3Ag1wLo 1ff65pFu9K4+A== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 1/4] rcu/nocb: Protect lazy shrinker against concurrent (de-)offloading Date: Wed, 22 Mar 2023 20:44:53 +0100 Message-Id: <20230322194456.2331527-2-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322194456.2331527-1-frederic@kernel.org> References: <20230322194456.2331527-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The shrinker may run concurrently with callbacks (de-)offloading. As such, calling rcu_nocb_lock() is very dangerous because it does a conditional locking. The worst outcome is that rcu_nocb_lock() doesn't lock but rcu_nocb_unlock() eventually unlocks, or the reverse, creating an imbalance. Fix this with protecting against (de-)offloading using the barrier mutex. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index f2280616f9d5..dd9b655ae533 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1336,13 +1336,25 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struc= t shrink_control *sc) unsigned long flags; unsigned long count =3D 0; =20 + /* + * Protect against concurrent (de-)offloading. Otherwise nocb locking + * may be ignored or imbalanced. + */ + mutex_lock(&rcu_state.barrier_mutex); + /* Snapshot count of all CPUs */ for_each_possible_cpu(cpu) { struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); - int _count =3D READ_ONCE(rdp->lazy_len); + int _count; + + if (!rcu_rdp_is_offloaded(rdp)) + continue; + + _count =3D READ_ONCE(rdp->lazy_len); =20 if (_count =3D=3D 0) continue; + rcu_nocb_lock_irqsave(rdp, flags); WRITE_ONCE(rdp->lazy_len, 0); rcu_nocb_unlock_irqrestore(rdp, flags); @@ -1352,6 +1364,9 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct = shrink_control *sc) if (sc->nr_to_scan <=3D 0) break; } + + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_STOP; } =20 --=20 2.34.1 From nobody Fri Dec 19 17:33:17 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB378C6FD1F for ; Wed, 22 Mar 2023 19:45:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230213AbjCVTp3 (ORCPT ); Wed, 22 Mar 2023 15:45:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229871AbjCVTpS (ORCPT ); Wed, 22 Mar 2023 15:45:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AEC273BC54; Wed, 22 Mar 2023 12:45:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 39A03622B3; Wed, 22 Mar 2023 19:45:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CF921C4339C; Wed, 22 Mar 2023 19:45:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679514308; bh=0Nr44nYCi7dzslDo1zOnGsqBigDrConQ2cePzusvGBA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q4ElfwI+077Zl/ofdd0x4Ay6VqneIR52LHDh8XjmdxboYiaUFydahOC+vJdDP3gVM qKcsiuWKsMku4MXcouUMFY1ScE5FU63BdMz/qvvDXmr/AZV7oPpzqxEvDSG7ZWC5VG V8oTfqnZjRS0wCjLYzNexf3mUk13See+COaCA70KUfzrfUHDjbZB0nODKhY9g9GEf8 VlO40NfZqTBD6abnXkfgTUW2DfAPV2va79D0yiYOm/MWe4O3X9NKiweMNPuOVmIVlW w9kbIexABoAg5VoldEaKqo16pOjfJbMT35K/KjLJDwpMnw+JO9srJ+pZ11tOc5oRAy k2dQYxj84yMGg== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 2/4] rcu/nocb: Fix shrinker race against callback enqueuer Date: Wed, 22 Mar 2023 20:44:54 +0100 Message-Id: <20230322194456.2331527-3-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322194456.2331527-1-frederic@kernel.org> References: <20230322194456.2331527-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The shrinker resets the lazy callbacks counter in order to trigger the pending lazy queue flush though the rcuog kthread. The counter reset is protected by the ->nocb_lock against concurrent accesses...except for one of them. Here is a list of existing synchronized readers/writer: 1) The first lazy enqueuer (incrementing ->lazy_len to 1) does so under ->nocb_lock and ->nocb_bypass_lock. 2) The further lazy enqueuers (incrementing ->lazy_len above 1) do so under ->nocb_bypass_lock _only_. 3) The lazy flush checks and resets to 0 under ->nocb_lock and ->nocb_bypass_lock. The shrinker protects its ->lazy_len reset against cases 1) and 3) but not against 2). As such, setting ->lazy_len to 0 under the ->nocb_lock may be cancelled right away by an overwrite from an enqueuer, leading rcuog to ignore the flush. To avoid that, use the proper bypass flush API which takes care of all those details. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index dd9b655ae533..cb57e8312231 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1356,7 +1356,7 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct = shrink_control *sc) continue; =20 rcu_nocb_lock_irqsave(rdp, flags); - WRITE_ONCE(rdp->lazy_len, 0); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); sc->nr_to_scan -=3D _count; --=20 2.34.1 From nobody Fri Dec 19 17:33:17 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 947FEC76196 for ; Wed, 22 Mar 2023 19:45:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230101AbjCVTp0 (ORCPT ); Wed, 22 Mar 2023 15:45:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229846AbjCVTpS (ORCPT ); Wed, 22 Mar 2023 15:45:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE7ED32CEB; Wed, 22 Mar 2023 12:45:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7BD9A622B1; Wed, 22 Mar 2023 19:45:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B7CDC4339B; Wed, 22 Mar 2023 19:45:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679514310; bh=mP8fCieTmDyHtxHWglNueIzXLMyc5wglCPH3QTQMCKk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z4SeTifBUVRdEuhbIjeDAFkhZN28kaeoNGLv044rupmYaop8EMvGXNg2ZnWWILzay 0/ndpYG1LRNCIps3ZYm+OjblNQ2Jx0Z3xyIAkpKvkR1m1tRxv8P7j1HqZY7LEGGRy1 X0ePCHGOVygrJy6zx3cr2agJePWejOCbV1uAqk+a8eH1knyQ1aUreHgnlv+jkpKuci KXqPE/9qDyKZR1ZiG2tgroIW0BCGzx+YdpgkmPLqfpoduP4LSHPPzwj0YANEhzCJeA z1vOPQ5UwH7MIwmZQr92YILDBs0gB2vgKDgl5LGO7A2ob8JooFjAcnOmAIp+xfTlVy KBFcXUO9lpmjA== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 3/4] rcu/nocb: Recheck lazy callbacks under the ->nocb_lock from shrinker Date: Wed, 22 Mar 2023 20:44:55 +0100 Message-Id: <20230322194456.2331527-4-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322194456.2331527-1-frederic@kernel.org> References: <20230322194456.2331527-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The ->lazy_len is only checked locklessly. Recheck again under the ->nocb_lock to avoid spending more time on flushing/waking if not necessary. The ->lazy_len can still increment concurrently (from 1 to infinity) but under the ->nocb_lock we at least know for sure if there are lazy callbacks at all (->lazy_len > 0). Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index cb57e8312231..a3dc7465b0b2 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1350,12 +1350,20 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struc= t shrink_control *sc) if (!rcu_rdp_is_offloaded(rdp)) continue; =20 + if (!READ_ONCE(rdp->lazy_len)) + continue; + + rcu_nocb_lock_irqsave(rdp, flags); + /* + * Recheck under the nocb lock. Since we are not holding the bypass + * lock we may still race with increments from the enqueuer but still + * we know for sure if there is at least one lazy callback. + */ _count =3D READ_ONCE(rdp->lazy_len); - - if (_count =3D=3D 0) + if (!_count) { + rcu_nocb_unlock_irqrestore(rdp, flags); continue; - - rcu_nocb_lock_irqsave(rdp, flags); + } WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); --=20 2.34.1 From nobody Fri Dec 19 17:33:17 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAEBBC761AF for ; Wed, 22 Mar 2023 19:45:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230272AbjCVTpb (ORCPT ); Wed, 22 Mar 2023 15:45:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229928AbjCVTpT (ORCPT ); Wed, 22 Mar 2023 15:45:19 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C81F11EFE2; Wed, 22 Mar 2023 12:45:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 78F8CB81DEA; Wed, 22 Mar 2023 19:45:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A46EC433AA; Wed, 22 Mar 2023 19:45:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679514313; bh=o/d+MExJk6dT2Q3smaaeBjF9BtLmW7hBw77NXSOsuZA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sMzpGXOBEC1RG/wSZamyrsu+iU5m/0UxfjbhgycVXYNRHFbH1PZzU36l2ZJ4mSV0j LkxFT6qscyXdSGYE7CmKa1TSgHrv8ZKVgvw5F65b2ML4Bc3nLmvr37dSZcgONG+5pJ mdw05b0W+QtvQbJBzNWZzZiXPt0XXlvpK3HV5gZA+FxyPMGB7CvKDCBH8iioZ4teNE d3i8DgCAfvDwfQLZEvkwaEIUyJ5yDb+kP0dxZBPQzjEi/l9KdYglF8ryipqJ9Imq7y zm+q6aVtWhZGneLo//5aGNwdBjFj+fGPKk8BA3LqKTtQbIbBhhQyejpRxL+Q7gZWZr Y82MOffgaHTBQ== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 4/4] rcu/nocb: Make shrinker to iterate only NOCB CPUs Date: Wed, 22 Mar 2023 20:44:56 +0100 Message-Id: <20230322194456.2331527-5-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322194456.2331527-1-frederic@kernel.org> References: <20230322194456.2331527-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating over the NOCB mask is enough for both counting and scanning. Just lock the mostly uncontended barrier mutex on counting as well in order to keep rcu_nocb_mask stable. Signed-off-by: Frederic Weisbecker Reviewed-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index a3dc7465b0b2..185c0c9a60d4 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1319,13 +1319,21 @@ lazy_rcu_shrink_count(struct shrinker *shrink, stru= ct shrink_control *sc) int cpu; unsigned long count =3D 0; =20 + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; + + /* Protect rcu_nocb_mask against concurrent (de-)offloading. */ + mutex_lock(&rcu_state.barrier_mutex); + /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); =20 count +=3D READ_ONCE(rdp->lazy_len); } =20 + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_EMPTY; } =20 @@ -1336,6 +1344,8 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct = shrink_control *sc) unsigned long flags; unsigned long count =3D 0; =20 + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; /* * Protect against concurrent (de-)offloading. Otherwise nocb locking * may be ignored or imbalanced. @@ -1343,7 +1353,7 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct = shrink_control *sc) mutex_lock(&rcu_state.barrier_mutex); =20 /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); int _count; =20 --=20 2.34.1