From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DB41C433F5 for ; Tue, 3 May 2022 10:01:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233979AbiECKEt (ORCPT ); Tue, 3 May 2022 06:04:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233933AbiECKEk (ORCPT ); Tue, 3 May 2022 06:04:40 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 995D460CC for ; Tue, 3 May 2022 03:01:08 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C681A61519 for ; Tue, 3 May 2022 10:01:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 23F23C385B3; Tue, 3 May 2022 10:01:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572067; bh=ns1vdA0/qrQYg4Q4rjwNdkKJ1jBqx0nuAlBJgu/kcNw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qgGdG5XmBbU8wwaFMrvp6wxXhQKeWEp5Xa8y95eouP6jxqTh5xPq32RuWZSM9LMQK 9VCmR1IZFNJ2muzPJmB8fCXmqCukeKYorhf/RlYChdt3CGAPoT06KJ+FJe19VaAo8m j4SHUTzpekdWVE1akkZmdefyZ7UH0uFvfRG39xIab/PN1Pe7dtuGT+JoaGZ8V5yRsJ pzLXZISM1/9HlwtbSS3ftwLnlY08rJPS6qaNyy+j2y9ScabVLfX7uPcaywedxYg9Ta pgGk/34BhoinRNM5eHpLhQPjrGWWUOxwtN243v12XvWPZ38eNt5nd71XJOM68Ye1B4 18PLocgvrkHhg== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 01/21] context_tracking: Remove unused context_tracking_in_user() Date: Tue, 3 May 2022 12:00:31 +0200 Message-Id: <20220503100051.2799723-2-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This function is not used and CT_WARN_ON() coupled with ct_state() is the preferred way to assert context tracking state values. Reported-by: Nicolas Saenz Julienne Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Signed-off-by: Frederic Weisbecker Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/context_tracking_state.h | 5 ----- 1 file changed, 5 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context= _tracking_state.h index 65a60d3313b0..02be3ac2040d 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -41,12 +41,7 @@ static inline bool context_tracking_enabled_this_cpu(voi= d) return context_tracking_enabled() && __this_cpu_read(context_tracking.act= ive); } =20 -static __always_inline bool context_tracking_in_user(void) -{ - return __this_cpu_read(context_tracking.state) =3D=3D CONTEXT_USER; -} #else -static inline bool context_tracking_in_user(void) { return false; } static inline bool context_tracking_enabled(void) { return false; } static inline bool context_tracking_enabled_cpu(int cpu) { return false; } static inline bool context_tracking_enabled_this_cpu(void) { return false;= } --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A75D0C433EF for ; Tue, 3 May 2022 10:01:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233988AbiECKE5 (ORCPT ); Tue, 3 May 2022 06:04:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233950AbiECKEo (ORCPT ); Tue, 3 May 2022 06:04:44 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 496A029837 for ; Tue, 3 May 2022 03:01:13 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id F0EFFB81A67 for ; Tue, 3 May 2022 10:01:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A5AA8C385B2; Tue, 3 May 2022 10:01:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572070; bh=gtCi5UK8fkkNrz2ICXApjgkShDwXqyXoI/epZZ4dS8M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B9YAV/C1zFgTdoJef/Yq8rIG5SW8UVwR83kS16qY/trXv7o6WYNYeZs2udfYe2XRp 8Wgo/7SHG4NnP58UZD75ueEXKJOrJeu/up/u2JvM6P8lqBaM64BFpPWoX28H56YqE8 rQB4XARxDg5Rb1GBS6zrwtNPP6bAx8BKPn8Hpk7wY4KfgNYr6FCZ8AMztJ58C1am/L tPH4S8viwJMHn7RwqPMkyqKP8r91eVd5JE+Eoz7uYz/G2PqrNrIJjM+tyo5GS4TS/H g6QpJtcAu5YWQL2J+W09A3dlYkRhgpUt6nFNV6Mhz8k1GSCopxGVFCxMctBgx5EDGR rEUWdxKOgOtaQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 02/21] rcu: Set rcu_idle_enter() as noinstr Date: Tue, 3 May 2022 12:00:32 +0200 Message-Id: <20220503100051.2799723-3-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This function enters into RCU idle mode, make sure that illegal uses of RCU are spotted there by objtools. Reported-by: Peter Zijlstra Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Signed-off-by: Frederic Weisbecker Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5c587e00811c..a5da4b87afde 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -659,7 +659,7 @@ static noinstr void rcu_eqs_enter(bool user) * If you add or remove a call to rcu_idle_enter(), be sure to test with * CONFIG_RCU_EQS_DEBUG=3Dy. */ -void rcu_idle_enter(void) +noinstr void rcu_idle_enter(void) { lockdep_assert_irqs_disabled(); rcu_eqs_enter(false); --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78896C433F5 for ; Tue, 3 May 2022 10:01:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233978AbiECKFB (ORCPT ); Tue, 3 May 2022 06:05:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50678 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233380AbiECKEr (ORCPT ); Tue, 3 May 2022 06:04:47 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45EA137A2D for ; Tue, 3 May 2022 03:01:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D220161519 for ; Tue, 3 May 2022 10:01:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33D8CC385A4; Tue, 3 May 2022 10:01:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572074; bh=1zuNr9aZbocCTXcOg+LUMS7leXk/IsG5o8dXhATCIOE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QHMecCv0A+1dlx7SuDLWJi02/MeKHB0hqnxlPOOQWHd/mJLFXdvnN1PupQwat4wix FWq4JWLW3vi+9mHfMpPbcBxc7nE+DIBbn7qLAWtCz0XceDTDkV3f/81nenyYn2uc2+ hno7OG9/VfSnFT1T6mzjC8bRID4fnAKe6J5nt9YKWjwLVhj0jyaUcfSE4S5WX5RuyS Ezqpl9cKBVSTYCJ0ysSoWsxUY2fZrM0bLiT5toQ8O4T9p3fFrwoldwiJ7gTtxcpr6r ZXlTSZHRx5lBMPbU1KnlXlhf6LYyqWwhYdpdXCZi4liAjlhbJ3apYuH+b3RQekXWoZ WaTrg+t5jc6RQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 03/21] rcu: Add a note about noinstr VS unsafe eqs functions Date: Tue, 3 May 2022 12:00:33 +0200 Message-Id: <20220503100051.2799723-4-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Some RCU functions enter or exit into/from RCU idle mode while using trace-able and lockdep-aware IRQs (un-)masking. This could be easily solved with using raw versions of local_irq_*() but we would then lose some precious debugging informations. Another possible way to solve this may consist in using rude RCU-tasks in lockdep and irqsoff tracing. In any case and until this get solved, those RCU functions can't get tagged as noinstr even though they should. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- kernel/rcu/tree.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index a5da4b87afde..2a6e7723fc4d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -838,6 +838,10 @@ void rcu_irq_exit_check_preempt(void) * * If you add or remove a call to rcu_irq_exit_irqson(), be sure to test * with CONFIG_RCU_EQS_DEBUG=3Dy. + * + * FIXME: This function should be noinstr but the below local_irq_restore(= ) is + * unsafe because it involves illegal RCU uses through tracing and lockdep. + * This must be fixed first. */ void rcu_irq_exit_irqson(void) { @@ -895,6 +899,10 @@ static void noinstr rcu_eqs_exit(bool user) * * If you add or remove a call to rcu_idle_exit(), be sure to test with * CONFIG_RCU_EQS_DEBUG=3Dy. + * + * FIXME: This function should be noinstr but the below local_irq_save() is + * unsafe because it involves illegal RCU uses through tracing and lockdep. + * This must be fixed first. */ void rcu_idle_exit(void) { @@ -1076,6 +1084,10 @@ noinstr void rcu_irq_enter(void) * * If you add or remove a call to rcu_irq_enter_irqson(), be sure to test * with CONFIG_RCU_EQS_DEBUG=3Dy. + * + * FIXME: This function should be noinstr but the below local_irq_save() is + * unsafe because it involves illegal RCU uses through tracing and lockdep. + * This must be fixed first. */ void rcu_irq_enter_irqson(void) { --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27809C433EF for ; Tue, 3 May 2022 10:01:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234074AbiECKFH (ORCPT ); Tue, 3 May 2022 06:05:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234005AbiECKE6 (ORCPT ); Tue, 3 May 2022 06:04:58 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4DB6937A83 for ; Tue, 3 May 2022 03:01:18 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6332A614A7 for ; Tue, 3 May 2022 10:01:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B6BBAC385B1; Tue, 3 May 2022 10:01:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572077; bh=uGn4/MomNIQOvt1kVzK6g2/4eo2NkPnsPiqsExJKXrY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fPa1gBwHddN2GTHFmzaXKJTfNGwuYoqQq2uQH2mQ+zjFGoiGJb6jUM/+g3HRamuBi W+N9XW4ooBhF1D8h6t/Lc84wDIgNxH0nkOlQ34GMzJ5qPF13PNWLfUcPiduwIsrle1 p4XLb0vO25tPdXKja3CzFckGrwu3gdBYk/AgrA8708vPyE+gzBQOzBWoUeplAU+dIL z0f8R2FI46i0Jg9z1XGzriwq8Rz9Z/4KrbcCixEhgmiBjflzN0OSAMkDEznWmBhMeT ChbWHP08l87/OkrjaZyMTTMsDm9a5ZDAd0Rr3+sg8p2W8zaDMvir4UEDzRzaGfwnl5 bxdzapIrFjKAQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 04/21] context_tracking: Add a note about noinstr VS unsafe context tracking functions Date: Tue, 3 May 2022 12:00:34 +0200 Message-Id: <20220503100051.2799723-5-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Some context tracking functions enter or exit into/from RCU idle mode while using trace-able and lockdep-aware IRQs (un-)masking. This could be easily solved with using raw versions of local_irq_*() but we would then lose some precious debugging informations. Another possible way to solve this may consist in using rude RCU-tasks in lockdep and irqsoff tracing. In any case and until this get solved, those functions can't get tagged as noinstr even though they should. Reported-by: Peter Zijlstra Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Signed-off-by: Frederic Weisbecker Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- kernel/context_tracking.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 36a98c48aedc..1f44b0461d14 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -103,6 +103,11 @@ void noinstr __context_tracking_enter(enum ctx_state s= tate) } EXPORT_SYMBOL_GPL(__context_tracking_enter); =20 +/* + * FIXME: This function should be noinstr but the below local_irq_restore(= ) is + * unsafe because it involves illegal RCU uses through tracing and lockdep= . This + * must be fixed first. + */ void context_tracking_enter(enum ctx_state state) { unsigned long flags; @@ -125,6 +130,10 @@ void context_tracking_enter(enum ctx_state state) NOKPROBE_SYMBOL(context_tracking_enter); EXPORT_SYMBOL_GPL(context_tracking_enter); =20 +/* + * FIXME: This function should be noinstr but it unsafely calls local_irq_= restore(), + * involving illegal RCU uses through tracing and lockdep. This must be fi= xed first. + */ void context_tracking_user_enter(void) { user_enter(); @@ -168,6 +177,11 @@ void noinstr __context_tracking_exit(enum ctx_state st= ate) } EXPORT_SYMBOL_GPL(__context_tracking_exit); =20 +/* + * FIXME: This function should be noinstr but the below local_irq_save() is + * unsafe because it involves illegal RCU uses through tracing and lockdep= . This + * must be fixed first. + */ void context_tracking_exit(enum ctx_state state) { unsigned long flags; @@ -182,6 +196,10 @@ void context_tracking_exit(enum ctx_state state) NOKPROBE_SYMBOL(context_tracking_exit); EXPORT_SYMBOL_GPL(context_tracking_exit); =20 +/* + * FIXME: This function should be noinstr but it unsafely calls local_irq_= save(), + * involving illegal RCU uses through tracing and lockdep. This must be fi= xed first. + */ void context_tracking_user_exit(void) { user_exit(); --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDA89C433EF for ; Tue, 3 May 2022 10:02:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234085AbiECKFr (ORCPT ); Tue, 3 May 2022 06:05:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234021AbiECKFD (ORCPT ); Tue, 3 May 2022 06:05:03 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD799387A9 for ; Tue, 3 May 2022 03:01:23 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 92A65B80EF2 for ; Tue, 3 May 2022 10:01:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 48D35C385A4; Tue, 3 May 2022 10:01:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572081; bh=5YGt+TwBa/VyBHNI6sHuOnprFZdkpVxNXEJJW70dIwI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F5hKH4ARv6WJ5JvVaWxD1aVwYKKFF4JKnUaPL1TF95wxtpiC/7AhTpOk2iOvx1I7p R+i3S89DpNXuMucLWQ+iRyF5Eq9+37J0rsIXXYXd0NTyu+wIWuMbW8UTgFKSqv8Neq VFLJ1xSgjGEuxq7SUAliakBZpgfRpBquJqdIwH9IH7sMGu301S7WYh9zuvffigQA7v Yy6+14+LzaIf1SyJuDuKBxblwOJoFIRW7RqfUmlBYPgBp2BaGkC8wlxtjDP/FXgZJ2 zg4FCQHPIznlQJtSKqBGWp4PdDY0Nj65XiPQgZRbsH8XIlDxlg7L035h66N65jKGW9 o4zqIjnBf1pnw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 05/21] context_tracking: Rename __context_tracking_enter/exit() to __ct_user_enter/exit() Date: Tue, 3 May 2022 12:00:35 +0200 Message-Id: <20220503100051.2799723-6-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The context tracking namespace is going to expand and some new functions will require even longer names. Start shrinking the context_tracking prefix to "ct" as is already the case for some existing macros, this will make the introduction of new functions easier. Acked-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/context_tracking.h | 12 ++++++------ kernel/context_tracking.c | 20 ++++++++++---------- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_track= ing.h index 7a14807c9d1a..773035124bad 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -14,8 +14,8 @@ extern void context_tracking_cpu_set(int cpu); =20 /* Called with interrupts disabled. */ -extern void __context_tracking_enter(enum ctx_state state); -extern void __context_tracking_exit(enum ctx_state state); +extern void __ct_user_enter(enum ctx_state state); +extern void __ct_user_exit(enum ctx_state state); =20 extern void context_tracking_enter(enum ctx_state state); extern void context_tracking_exit(enum ctx_state state); @@ -38,13 +38,13 @@ static inline void user_exit(void) static __always_inline void user_enter_irqoff(void) { if (context_tracking_enabled()) - __context_tracking_enter(CONTEXT_USER); + __ct_user_enter(CONTEXT_USER); =20 } static __always_inline void user_exit_irqoff(void) { if (context_tracking_enabled()) - __context_tracking_exit(CONTEXT_USER); + __ct_user_exit(CONTEXT_USER); } =20 static inline enum ctx_state exception_enter(void) @@ -74,7 +74,7 @@ static inline void exception_exit(enum ctx_state prev_ctx) static __always_inline bool context_tracking_guest_enter(void) { if (context_tracking_enabled()) - __context_tracking_enter(CONTEXT_GUEST); + __ct_user_enter(CONTEXT_GUEST); =20 return context_tracking_enabled_this_cpu(); } @@ -82,7 +82,7 @@ static __always_inline bool context_tracking_guest_enter(= void) static __always_inline void context_tracking_guest_exit(void) { if (context_tracking_enabled()) - __context_tracking_exit(CONTEXT_GUEST); + __ct_user_exit(CONTEXT_GUEST); } =20 /** diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 1f44b0461d14..554c2c9587eb 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -51,15 +51,15 @@ static __always_inline void context_tracking_recursion_= exit(void) } =20 /** - * context_tracking_enter - Inform the context tracking that the CPU is go= ing - * enter user or guest space mode. + * __ct_user_enter - Inform the context tracking that the CPU is going + * to enter user or guest space mode. * * This function must be called right before we switch from the kernel * to user or guest space, when it's guaranteed the remaining kernel * instructions to execute won't use any RCU read side critical section * because this function sets RCU in extended quiescent state. */ -void noinstr __context_tracking_enter(enum ctx_state state) +void noinstr __ct_user_enter(enum ctx_state state) { /* Kernel threads aren't supposed to go to userspace */ WARN_ON_ONCE(!current->mm); @@ -101,7 +101,7 @@ void noinstr __context_tracking_enter(enum ctx_state st= ate) } context_tracking_recursion_exit(); } -EXPORT_SYMBOL_GPL(__context_tracking_enter); +EXPORT_SYMBOL_GPL(__ct_user_enter); =20 /* * FIXME: This function should be noinstr but the below local_irq_restore(= ) is @@ -124,7 +124,7 @@ void context_tracking_enter(enum ctx_state state) return; =20 local_irq_save(flags); - __context_tracking_enter(state); + __ct_user_enter(state); local_irq_restore(flags); } NOKPROBE_SYMBOL(context_tracking_enter); @@ -141,8 +141,8 @@ void context_tracking_user_enter(void) NOKPROBE_SYMBOL(context_tracking_user_enter); =20 /** - * context_tracking_exit - Inform the context tracking that the CPU is - * exiting user or guest mode and entering the ker= nel. + * __ct_user_exit - Inform the context tracking that the CPU is + * exiting user or guest mode and entering the kernel. * * This function must be called after we entered the kernel from user or * guest space before any use of RCU read side critical section. This @@ -152,7 +152,7 @@ NOKPROBE_SYMBOL(context_tracking_user_enter); * This call supports re-entrancy. This way it can be called from any exce= ption * handler without needing to know if we came from userspace or not. */ -void noinstr __context_tracking_exit(enum ctx_state state) +void noinstr __ct_user_exit(enum ctx_state state) { if (!context_tracking_recursion_enter()) return; @@ -175,7 +175,7 @@ void noinstr __context_tracking_exit(enum ctx_state sta= te) } context_tracking_recursion_exit(); } -EXPORT_SYMBOL_GPL(__context_tracking_exit); +EXPORT_SYMBOL_GPL(__ct_user_exit); =20 /* * FIXME: This function should be noinstr but the below local_irq_save() is @@ -190,7 +190,7 @@ void context_tracking_exit(enum ctx_state state) return; =20 local_irq_save(flags); - __context_tracking_exit(state); + __ct_user_exit(state); local_irq_restore(flags); } NOKPROBE_SYMBOL(context_tracking_exit); --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 572E5C433EF for ; Tue, 3 May 2022 10:01:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234054AbiECKFN (ORCPT ); Tue, 3 May 2022 06:05:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234042AbiECKFE (ORCPT ); Tue, 3 May 2022 06:05:04 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 885E738BF1 for ; Tue, 3 May 2022 03:01:27 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 25394B81A67 for ; Tue, 3 May 2022 10:01:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9F02C385B3; Tue, 3 May 2022 10:01:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572084; bh=Ez/muF9+kjJrmlCOLg5f0MjmhahBJ1sWlDpnRqsrmKg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zb3Y9k5nQEaVXAytIyY438nQFUDiWu1//6acSnzwWTZlmg8CJlP6G6fweJfGsHayV eCf5KZ+OlcDJrPblb2PKH1WQv4dBeRSDiu6cajElG6bEaFI17aNoHDNv4r5FLThOmb ucDWsK0gow5W3hx4sYLU9/jFO9YGR7EWwbMXJR0gTjYjJ5wOQeECgIxpdJi5a/FBuu /GQp/2Wa7T4NHkKWIdwjp4ABbKW8jQ1MCq2WEfW7Pk2iEjnBxNqskS/oL8nb5bYx6a fDDhcp80z1OLOH9LCbG4eAMI7yD8BI+/g2x/q8XhbJxpKtKES0aiz78ruZnR0210V9 eQ/kP8tatos2Q== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 06/21] context_tracking: Rename context_tracking_user_enter/exit() to user_enter/exit_callable() Date: Tue, 3 May 2022 12:00:36 +0200 Message-Id: <20220503100051.2799723-7-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" context_tracking_user_enter() and context_tracking_user_exit() are ASM callable versions of user_enter() and user_exit() for architectures that didn't manage to check the context tracking static key from ASM. Change those function names to better reflect their purpose. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- arch/arm/kernel/entry-header.S | 8 ++++---- arch/csky/kernel/entry.S | 4 ++-- arch/riscv/kernel/entry.S | 6 +++--- include/linux/context_tracking.h | 4 ++-- kernel/context_tracking.c | 20 ++++++++++++++------ 5 files changed, 25 insertions(+), 17 deletions(-) diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 9a1dc142f782..0c14dcc43e58 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -370,10 +370,10 @@ ALT_UP_B(.L1_\@) #ifdef CONFIG_CONTEXT_TRACKING .if \save stmdb sp!, {r0-r3, ip, lr} - bl context_tracking_user_exit + bl user_exit_callable ldmia sp!, {r0-r3, ip, lr} .else - bl context_tracking_user_exit + bl user_exit_callable .endif #endif .endm @@ -382,10 +382,10 @@ ALT_UP_B(.L1_\@) #ifdef CONFIG_CONTEXT_TRACKING .if \save stmdb sp!, {r0-r3, ip, lr} - bl context_tracking_user_enter + bl user_enter_callable ldmia sp!, {r0-r3, ip, lr} .else - bl context_tracking_user_enter + bl user_enter_callable .endif #endif .endm diff --git a/arch/csky/kernel/entry.S b/arch/csky/kernel/entry.S index a4ababf25e24..bc734d17c16f 100644 --- a/arch/csky/kernel/entry.S +++ b/arch/csky/kernel/entry.S @@ -23,7 +23,7 @@ mfcr a0, epsr btsti a0, 31 bt 1f - jbsr context_tracking_user_exit + jbsr user_exit_callable ldw a0, (sp, LSAVE_A0) ldw a1, (sp, LSAVE_A1) ldw a2, (sp, LSAVE_A2) @@ -160,7 +160,7 @@ ret_from_exception: cmpnei r10, 0 bt exit_work #ifdef CONFIG_CONTEXT_TRACKING - jbsr context_tracking_user_enter + jbsr user_enter_callable #endif 1: #ifdef CONFIG_PREEMPTION diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index c8b9ce274b9a..ecd132af2edf 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -112,11 +112,11 @@ _save_context: #endif =20 #ifdef CONFIG_CONTEXT_TRACKING - /* If previous state is in user mode, call context_tracking_user_exit. */ + /* If previous state is in user mode, call user_exit_callable(). */ li a0, SR_PP and a0, s1, a0 bnez a0, skip_context_tracking - call context_tracking_user_exit + call user_exit_callable skip_context_tracking: #endif =20 @@ -256,7 +256,7 @@ resume_userspace: bnez s1, work_pending =20 #ifdef CONFIG_CONTEXT_TRACKING - call context_tracking_user_enter + call user_enter_callable #endif =20 /* Save unwound kernel stack pointer in thread_info */ diff --git a/include/linux/context_tracking.h b/include/linux/context_track= ing.h index 773035124bad..69532cd18f72 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -19,8 +19,8 @@ extern void __ct_user_exit(enum ctx_state state); =20 extern void context_tracking_enter(enum ctx_state state); extern void context_tracking_exit(enum ctx_state state); -extern void context_tracking_user_enter(void); -extern void context_tracking_user_exit(void); +extern void user_enter_callable(void); +extern void user_exit_callable(void); =20 static inline void user_enter(void) { diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 554c2c9587eb..e4e7aad6c2bc 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -130,15 +130,19 @@ void context_tracking_enter(enum ctx_state state) NOKPROBE_SYMBOL(context_tracking_enter); EXPORT_SYMBOL_GPL(context_tracking_enter); =20 -/* +/** + * user_enter_callable() - Unfortunate ASM callable version of user_enter(= ) for + * archs that didn't manage to check the context tracking + * static key from low level code. + * * FIXME: This function should be noinstr but it unsafely calls local_irq_= restore(), * involving illegal RCU uses through tracing and lockdep. This must be fi= xed first. */ -void context_tracking_user_enter(void) +void user_enter_callable(void) { user_enter(); } -NOKPROBE_SYMBOL(context_tracking_user_enter); +NOKPROBE_SYMBOL(user_enter_callable); =20 /** * __ct_user_exit - Inform the context tracking that the CPU is @@ -196,15 +200,19 @@ void context_tracking_exit(enum ctx_state state) NOKPROBE_SYMBOL(context_tracking_exit); EXPORT_SYMBOL_GPL(context_tracking_exit); =20 -/* +/** + * user_exit_callable() - Unfortunate ASM callable version of user_exit() = for + * archs that didn't manage to check the context tracking + * static key from low level code. + * * FIXME: This function should be noinstr but it unsafely calls local_irq_= save(), * involving illegal RCU uses through tracing and lockdep. This must be fi= xed first. */ -void context_tracking_user_exit(void) +void user_exit_callable(void) { user_exit(); } -NOKPROBE_SYMBOL(context_tracking_user_exit); +NOKPROBE_SYMBOL(user_exit_callable); =20 void __init context_tracking_cpu_set(int cpu) { --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC8EAC433F5 for ; Tue, 3 May 2022 10:01:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234024AbiECKFV (ORCPT ); Tue, 3 May 2022 06:05:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234051AbiECKFF (ORCPT ); Tue, 3 May 2022 06:05:05 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DAAFC38DA8 for ; Tue, 3 May 2022 03:01:29 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0863160EB1 for ; Tue, 3 May 2022 10:01:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 58716C385B0; Tue, 3 May 2022 10:01:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572088; bh=VbESeDYHIHGEyPhJM9sQpDnZacs3e6ZczXrBfGPfyQk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tqFnefO2XJ6qA7+I8aicUlxqX+QPZ5QJMJv7229QcnkBI/+7mKcRm9Ulj9BDz1Zqz lcg3MLp73UQcR+EcvCZsA7uAqPa7owg9kh8Ju3J0F3rMyla8DMWYivWl/Su3hA3zqG 2/WUKCT31ZI9SYlmvgz9qE12q2Pv7limDg2uCMj0IDmHZxaUSCr4H2plxtG6X33g3k 4YCT2fr7Ni7vQGK6r9939XicS600WULNbG9MtBXAdEkCuRjTn3SBuc/DufX5XgdEzG w/KV3Mf+kDcAg+gEeug/1qARmMvbBuS/muBxLT6Kr6dN27T8lUIgRmDJ5tt5OmO9Em 1ROc0763EDkUQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 07/21] context_tracking: Rename context_tracking_enter/exit() to ct_user_enter/exit() Date: Tue, 3 May 2022 12:00:37 +0200 Message-Id: <20220503100051.2799723-8-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" context_tracking_enter() and context_tracking_exit() have confusing names that don't explain the fact they are referring to user/guest state. Use more self-explanatory names and shrink to the new context tracking prefix instead. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/context_tracking.h | 13 +++++++------ kernel/context_tracking.c | 12 ++++++------ 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_track= ing.h index 69532cd18f72..7a5f04ae1758 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -17,21 +17,22 @@ extern void context_tracking_cpu_set(int cpu); extern void __ct_user_enter(enum ctx_state state); extern void __ct_user_exit(enum ctx_state state); =20 -extern void context_tracking_enter(enum ctx_state state); -extern void context_tracking_exit(enum ctx_state state); +extern void ct_user_enter(enum ctx_state state); +extern void ct_user_exit(enum ctx_state state); + extern void user_enter_callable(void); extern void user_exit_callable(void); =20 static inline void user_enter(void) { if (context_tracking_enabled()) - context_tracking_enter(CONTEXT_USER); + ct_user_enter(CONTEXT_USER); =20 } static inline void user_exit(void) { if (context_tracking_enabled()) - context_tracking_exit(CONTEXT_USER); + ct_user_exit(CONTEXT_USER); } =20 /* Called with interrupts disabled. */ @@ -57,7 +58,7 @@ static inline enum ctx_state exception_enter(void) =20 prev_ctx =3D this_cpu_read(context_tracking.state); if (prev_ctx !=3D CONTEXT_KERNEL) - context_tracking_exit(prev_ctx); + ct_user_exit(prev_ctx); =20 return prev_ctx; } @@ -67,7 +68,7 @@ static inline void exception_exit(enum ctx_state prev_ctx) if (!IS_ENABLED(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK) && context_tracking_enabled()) { if (prev_ctx !=3D CONTEXT_KERNEL) - context_tracking_enter(prev_ctx); + ct_user_enter(prev_ctx); } } =20 diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index e4e7aad6c2bc..bfbc5d846a62 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -108,7 +108,7 @@ EXPORT_SYMBOL_GPL(__ct_user_enter); * unsafe because it involves illegal RCU uses through tracing and lockdep= . This * must be fixed first. */ -void context_tracking_enter(enum ctx_state state) +void ct_user_enter(enum ctx_state state) { unsigned long flags; =20 @@ -127,8 +127,8 @@ void context_tracking_enter(enum ctx_state state) __ct_user_enter(state); local_irq_restore(flags); } -NOKPROBE_SYMBOL(context_tracking_enter); -EXPORT_SYMBOL_GPL(context_tracking_enter); +NOKPROBE_SYMBOL(ct_user_enter); +EXPORT_SYMBOL_GPL(ct_user_enter); =20 /** * user_enter_callable() - Unfortunate ASM callable version of user_enter(= ) for @@ -186,7 +186,7 @@ EXPORT_SYMBOL_GPL(__ct_user_exit); * unsafe because it involves illegal RCU uses through tracing and lockdep= . This * must be fixed first. */ -void context_tracking_exit(enum ctx_state state) +void ct_user_exit(enum ctx_state state) { unsigned long flags; =20 @@ -197,8 +197,8 @@ void context_tracking_exit(enum ctx_state state) __ct_user_exit(state); local_irq_restore(flags); } -NOKPROBE_SYMBOL(context_tracking_exit); -EXPORT_SYMBOL_GPL(context_tracking_exit); +NOKPROBE_SYMBOL(ct_user_exit); +EXPORT_SYMBOL_GPL(ct_user_exit); =20 /** * user_exit_callable() - Unfortunate ASM callable version of user_exit() = for --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A2BAC433EF for ; Tue, 3 May 2022 10:02:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234080AbiECKFn (ORCPT ); Tue, 3 May 2022 06:05:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234072AbiECKFH (ORCPT ); Tue, 3 May 2022 06:05:07 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B22B160CC for ; Tue, 3 May 2022 03:01:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 58C01B80EF2 for ; Tue, 3 May 2022 10:01:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DB1C7C385B5; Tue, 3 May 2022 10:01:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572092; bh=/+I4eKb6/i5JzPadanskIAapdZhrFOJ8ueIeiMxV+hE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=m4jn8ET9Uyu08NBjeeilRvjLQarZmcRhMpPv7wp8iVEEdSMre/QhbXNS7knUkQhRY AIU1qjL8YHM4jmChQrawzVyeYVcc6wkewkcbp1CkAMZL2uc74dVKvy6ayzy9ADlCm9 PSH2IFi5JS/5CsQHETECvlS/SHvbRSYU17yWPQUyhkRPyulC2jndgSjM7P++YSdlku clSHiZ6Mht0+kvBzVR8nSXc+xh/A5kd75ZGRUhxkCJI7ItbkItv2u0XkcWzU4xjgDE suXLMgju0Nhev6hQTW/yh/on4VA0Dlar+Sy2UzQRZ41GIh5a7fJZh4douZcvgOjhK3 9Sp+ZST3SSYsw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 08/21] context_tracking: Rename context_tracking_cpu_set() to ct_cpu_track_user() Date: Tue, 3 May 2022 12:00:38 +0200 Message-Id: <20220503100051.2799723-9-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" context_tracking_cpu_set() is called in order to tell a CPU to track user/kernel transitions. Since context tracking is going to expand in to also track transitions from/to idle/IRQ/NMIs, the scope of this function name becomes too broad and needs to be made more specific. Also shorten the prefix to align with the new namespace. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/context_tracking.h | 2 +- kernel/context_tracking.c | 4 ++-- kernel/time/tick-sched.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_track= ing.h index 7a5f04ae1758..63259fece7c7 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -11,7 +11,7 @@ =20 =20 #ifdef CONFIG_CONTEXT_TRACKING -extern void context_tracking_cpu_set(int cpu); +extern void ct_cpu_track_user(int cpu); =20 /* Called with interrupts disabled. */ extern void __ct_user_enter(enum ctx_state state); diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index bfbc5d846a62..f48eaad314ba 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -214,7 +214,7 @@ void user_exit_callable(void) } NOKPROBE_SYMBOL(user_exit_callable); =20 -void __init context_tracking_cpu_set(int cpu) +void __init ct_cpu_track_user(int cpu) { static __initdata bool initialized =3D false; =20 @@ -244,6 +244,6 @@ void __init context_tracking_init(void) int cpu; =20 for_each_possible_cpu(cpu) - context_tracking_cpu_set(cpu); + ct_cpu_track_user(cpu); } #endif diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 2d76c91b85de..06709885c590 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -571,7 +571,7 @@ void __init tick_nohz_init(void) } =20 for_each_cpu(cpu, tick_nohz_full_mask) - context_tracking_cpu_set(cpu); + ct_cpu_track_user(cpu); =20 ret =3D cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "kernel/nohz:predown", NULL, --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8687FC433F5 for ; Tue, 3 May 2022 10:02:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234062AbiECKF2 (ORCPT ); Tue, 3 May 2022 06:05:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234046AbiECKFJ (ORCPT ); Tue, 3 May 2022 06:05:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EDEFE38BC0 for ; Tue, 3 May 2022 03:01:36 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3D61060EB1 for ; Tue, 3 May 2022 10:01:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 67A90C385B0; Tue, 3 May 2022 10:01:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572095; bh=Ugut7cTgSuLA8lel62jKGHXm1jI39y+bNT3Ti3/+kKE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aPoNEfwNsqmcY9AEW5BQtwEWz9nyfkcziaZZ++2UeUfw5b0ULjKetLae+LSZiq5C6 youWT4GFQsJbvlBrpP3QaX1ei7bp61cS7c9DmSfc1ee7Nl6vX16ATbEkWqNTcrbo+y eWa2TBprdNMhQdeN3M7m0N7ANZbqnCzVa8Wnm8k0Mt2qmXXr/hmC+SwkEUilHIL8p1 2k7pTcHotg/YjIhJQZDGwInceNKMKdQLfkt3J7DZLy8EWhL6+Xrz8QRCfIOkWTW9t9 TsgIFcfSoMiXM0abTcY1jM4rAgzp3Kj2ZI27+kGYx5xe4i7m7U1JSdoE8W2qUW6Z6w 8duaYxZrDX01g== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 09/21] context_tracking: Split user tracking Kconfig Date: Tue, 3 May 2022 12:00:39 +0200 Message-Id: <20220503100051.2799723-10-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Context tracking is going to be used not only to track user transitions but also idle/IRQs/NMIs. The user tracking part will then become a separate feature. Prepare Kconfig for that. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- .../time/context-tracking/arch-support.txt | 6 ++--- arch/Kconfig | 4 ++-- arch/arm/Kconfig | 2 +- arch/arm/kernel/entry-common.S | 4 ++-- arch/arm/kernel/entry-header.S | 4 ++-- arch/arm64/Kconfig | 2 +- arch/csky/Kconfig | 2 +- arch/csky/kernel/entry.S | 4 ++-- arch/mips/Kconfig | 2 +- arch/powerpc/Kconfig | 2 +- arch/powerpc/include/asm/context_tracking.h | 2 +- arch/riscv/Kconfig | 2 +- arch/riscv/kernel/entry.S | 6 ++--- arch/sparc/Kconfig | 2 +- arch/sparc/kernel/rtrap_64.S | 2 +- arch/x86/Kconfig | 4 ++-- include/linux/context_tracking.h | 12 +++++----- include/linux/context_tracking_state.h | 4 ++-- init/Kconfig | 4 ++-- kernel/context_tracking.c | 6 ++++- kernel/sched/core.c | 2 +- kernel/time/Kconfig | 22 +++++++++++-------- 22 files changed, 54 insertions(+), 46 deletions(-) diff --git a/Documentation/features/time/context-tracking/arch-support.txt = b/Documentation/features/time/context-tracking/arch-support.txt index bb1c1801553e..59ec30c84541 100644 --- a/Documentation/features/time/context-tracking/arch-support.txt +++ b/Documentation/features/time/context-tracking/arch-support.txt @@ -1,7 +1,7 @@ # -# Feature name: context-tracking -# Kconfig: HAVE_CONTEXT_TRACKING -# description: arch supports context tracking for NO_HZ_FULL +# Feature name: user-context-tracking +# Kconfig: HAVE_CONTEXT_TRACKING_USER +# description: arch supports user context tracking for NO_HZ_FULL # ----------------------- | arch |status| diff --git a/arch/Kconfig b/arch/Kconfig index 1bf29ce754af..7a8cd4d67d9c 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -769,7 +769,7 @@ config HAVE_ARCH_WITHIN_STACK_FRAMES and similar) by implementing an inline arch_within_stack_frames(), which is used by CONFIG_HARDENED_USERCOPY. =20 -config HAVE_CONTEXT_TRACKING +config HAVE_CONTEXT_TRACKING_USER bool help Provide kernel/user boundaries probes necessary for subsystems @@ -780,7 +780,7 @@ config HAVE_CONTEXT_TRACKING protected inside rcu_irq_enter/rcu_irq_exit() but preemption or signal handling on irq exit still need to be protected. =20 -config HAVE_CONTEXT_TRACKING_OFFSTACK +config HAVE_CONTEXT_TRACKING_USER_OFFSTACK bool help Architecture neither relies on exception_enter()/exception_exit() diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 2e8091e2d8a8..64eb31946598 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -84,7 +84,7 @@ config ARM select HAVE_ARCH_TRANSPARENT_HUGEPAGE if ARM_LPAE select HAVE_ARM_SMCCC if CPU_V7 select HAVE_EBPF_JIT if !CPU_ENDIAN_BE32 - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_C_RECORDMCOUNT select HAVE_BUILDTIME_MCOUNT_SORT select HAVE_DEBUG_KMEMLEAK if !XIP_KERNEL diff --git a/arch/arm/kernel/entry-common.S b/arch/arm/kernel/entry-common.S index 90d40f4d56cf..393d3945a43f 100644 --- a/arch/arm/kernel/entry-common.S +++ b/arch/arm/kernel/entry-common.S @@ -28,7 +28,7 @@ #include "entry-header.S" =20 saved_psr .req r8 -#if defined(CONFIG_TRACE_IRQFLAGS) || defined(CONFIG_CONTEXT_TRACKING) +#if defined(CONFIG_TRACE_IRQFLAGS) || defined(CONFIG_CONTEXT_TRACKING_USER) saved_pc .req r9 #define TRACE(x...) x #else @@ -38,7 +38,7 @@ saved_pc .req lr =20 .section .entry.text,"ax",%progbits .align 5 -#if !(IS_ENABLED(CONFIG_TRACE_IRQFLAGS) || IS_ENABLED(CONFIG_CONTEXT_TRACK= ING) || \ +#if !(IS_ENABLED(CONFIG_TRACE_IRQFLAGS) || IS_ENABLED(CONFIG_CONTEXT_TRACK= ING_USER) || \ IS_ENABLED(CONFIG_DEBUG_RSEQ)) /* * This is the fast syscall return path. We do as little as possible here, diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 0c14dcc43e58..8a54ff24bed0 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -367,7 +367,7 @@ ALT_UP_B(.L1_\@) * between user and kernel mode. */ .macro ct_user_exit, save =3D 1 -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER .if \save stmdb sp!, {r0-r3, ip, lr} bl user_exit_callable @@ -379,7 +379,7 @@ ALT_UP_B(.L1_\@) .endm =20 .macro ct_user_enter, save =3D 1 -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER .if \save stmdb sp!, {r0-r3, ip, lr} bl user_enter_callable diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 57c4c995965f..6e8757b89a54 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -171,7 +171,7 @@ config ARM64 select HAVE_C_RECORDMCOUNT select HAVE_CMPXCHG_DOUBLE select HAVE_CMPXCHG_LOCAL - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_DEBUG_KMEMLEAK select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE diff --git a/arch/csky/Kconfig b/arch/csky/Kconfig index 75ef86605d69..171b59901214 100644 --- a/arch/csky/Kconfig +++ b/arch/csky/Kconfig @@ -42,7 +42,7 @@ config CSKY select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_SECCOMP_FILTER - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_VIRT_CPU_ACCOUNTING_GEN select HAVE_DEBUG_BUGVERBOSE select HAVE_DEBUG_KMEMLEAK diff --git a/arch/csky/kernel/entry.S b/arch/csky/kernel/entry.S index bc734d17c16f..547b4cd1b24b 100644 --- a/arch/csky/kernel/entry.S +++ b/arch/csky/kernel/entry.S @@ -19,7 +19,7 @@ .endm =20 .macro context_tracking -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER mfcr a0, epsr btsti a0, 31 bt 1f @@ -159,7 +159,7 @@ ret_from_exception: and r10, r9 cmpnei r10, 0 bt exit_work -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER jbsr user_enter_callable #endif 1: diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index de3b32a507d2..d0048cddd558 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -56,7 +56,7 @@ config MIPS select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE if CPU_SUPPORTS_HUGEPAGES select HAVE_ASM_MODVERSIONS - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_TIF_NOHZ select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAK diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 174edabb74fa..f2fddc09a345 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -197,7 +197,7 @@ config PPC select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select HAVE_ASM_MODVERSIONS - select HAVE_CONTEXT_TRACKING if PPC64 + select HAVE_CONTEXT_TRACKING_USER if PPC64 select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAK select HAVE_DEBUG_STACKOVERFLOW diff --git a/arch/powerpc/include/asm/context_tracking.h b/arch/powerpc/inc= lude/asm/context_tracking.h index f2682b28b050..4b63931c49e0 100644 --- a/arch/powerpc/include/asm/context_tracking.h +++ b/arch/powerpc/include/asm/context_tracking.h @@ -2,7 +2,7 @@ #ifndef _ASM_POWERPC_CONTEXT_TRACKING_H #define _ASM_POWERPC_CONTEXT_TRACKING_H =20 -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER #define SCHEDULE_USER bl schedule_user #else #define SCHEDULE_USER bl schedule diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 00fd9c548f26..08316de9fa9f 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -83,7 +83,7 @@ config RISCV select HAVE_ARCH_THREAD_STRUCT_WHITELIST select HAVE_ARCH_VMAP_STACK if MMU && 64BIT select HAVE_ASM_MODVERSIONS - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_DEBUG_KMEMLEAK select HAVE_DMA_CONTIGUOUS if MMU select HAVE_EBPF_JIT if MMU diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index ecd132af2edf..fba900eaa0b8 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -111,7 +111,7 @@ _save_context: call __trace_hardirqs_off #endif =20 -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER /* If previous state is in user mode, call user_exit_callable(). */ li a0, SR_PP and a0, s1, a0 @@ -176,7 +176,7 @@ handle_syscall: */ csrs CSR_STATUS, SR_IE #endif -#if defined(CONFIG_TRACE_IRQFLAGS) || defined(CONFIG_CONTEXT_TRACKING) +#if defined(CONFIG_TRACE_IRQFLAGS) || defined(CONFIG_CONTEXT_TRACKING_USER) /* Recover a0 - a7 for system calls */ REG_L a0, PT_A0(sp) REG_L a1, PT_A1(sp) @@ -255,7 +255,7 @@ resume_userspace: andi s1, s0, _TIF_WORK_MASK bnez s1, work_pending =20 -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER call user_enter_callable #endif =20 diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index 9200bc04701c..be50eae3ab0d 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -71,7 +71,7 @@ config SPARC64 select HAVE_DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD select HAVE_SYSCALL_TRACEPOINTS - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_TIF_NOHZ select HAVE_DEBUG_KMEMLEAK select IOMMU_HELPER diff --git a/arch/sparc/kernel/rtrap_64.S b/arch/sparc/kernel/rtrap_64.S index c5fd4b450d9b..eef102765a7e 100644 --- a/arch/sparc/kernel/rtrap_64.S +++ b/arch/sparc/kernel/rtrap_64.S @@ -15,7 +15,7 @@ #include #include =20 -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER # define SCHEDULE_USER schedule_user #else # define SCHEDULE_USER schedule diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b0142e01002e..368a1c5cbb08 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -185,8 +185,8 @@ config X86 select HAVE_ASM_MODVERSIONS select HAVE_CMPXCHG_DOUBLE select HAVE_CMPXCHG_LOCAL - select HAVE_CONTEXT_TRACKING if X86_64 - select HAVE_CONTEXT_TRACKING_OFFSTACK if HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER if X86_64 + select HAVE_CONTEXT_TRACKING_USER_OFFSTACK if HAVE_CONTEXT_TRACKING_USER select HAVE_C_RECORDMCOUNT select HAVE_OBJTOOL_MCOUNT if STACK_VALIDATION select HAVE_BUILDTIME_MCOUNT_SORT diff --git a/include/linux/context_tracking.h b/include/linux/context_track= ing.h index 63259fece7c7..e35ae66b4794 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -10,7 +10,7 @@ #include =20 =20 -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER extern void ct_cpu_track_user(int cpu); =20 /* Called with interrupts disabled. */ @@ -52,7 +52,7 @@ static inline enum ctx_state exception_enter(void) { enum ctx_state prev_ctx; =20 - if (IS_ENABLED(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK) || + if (IS_ENABLED(CONFIG_HAVE_CONTEXT_TRACKING_USER_OFFSTACK) || !context_tracking_enabled()) return 0; =20 @@ -65,7 +65,7 @@ static inline enum ctx_state exception_enter(void) =20 static inline void exception_exit(enum ctx_state prev_ctx) { - if (!IS_ENABLED(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK) && + if (!IS_ENABLED(CONFIG_HAVE_CONTEXT_TRACKING_USER_OFFSTACK) && context_tracking_enabled()) { if (prev_ctx !=3D CONTEXT_KERNEL) ct_user_enter(prev_ctx); @@ -109,14 +109,14 @@ static inline enum ctx_state ct_state(void) { return = CONTEXT_DISABLED; } static __always_inline bool context_tracking_guest_enter(void) { return fa= lse; } static inline void context_tracking_guest_exit(void) { } =20 -#endif /* !CONFIG_CONTEXT_TRACKING */ +#endif /* !CONFIG_CONTEXT_TRACKING_USER */ =20 #define CT_WARN_ON(cond) WARN_ON(context_tracking_enabled() && (cond)) =20 -#ifdef CONFIG_CONTEXT_TRACKING_FORCE +#ifdef CONFIG_CONTEXT_TRACKING_USER_FORCE extern void context_tracking_init(void); #else static inline void context_tracking_init(void) { } -#endif /* CONFIG_CONTEXT_TRACKING_FORCE */ +#endif /* CONFIG_CONTEXT_TRACKING_USER_FORCE */ =20 #endif diff --git a/include/linux/context_tracking_state.h b/include/linux/context= _tracking_state.h index 02be3ac2040d..913acfee5dbf 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -22,7 +22,7 @@ struct context_tracking { } state; }; =20 -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER extern struct static_key_false context_tracking_key; DECLARE_PER_CPU(struct context_tracking, context_tracking); =20 @@ -45,6 +45,6 @@ static inline bool context_tracking_enabled_this_cpu(void) static inline bool context_tracking_enabled(void) { return false; } static inline bool context_tracking_enabled_cpu(int cpu) { return false; } static inline bool context_tracking_enabled_this_cpu(void) { return false;= } -#endif /* CONFIG_CONTEXT_TRACKING */ +#endif /* CONFIG_CONTEXT_TRACKING_USER */ =20 #endif diff --git a/init/Kconfig b/init/Kconfig index ddcbefe535e9..3d88ecadf771 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -502,11 +502,11 @@ config VIRT_CPU_ACCOUNTING_NATIVE =20 config VIRT_CPU_ACCOUNTING_GEN bool "Full dynticks CPU time accounting" - depends on HAVE_CONTEXT_TRACKING + depends on HAVE_CONTEXT_TRACKING_USER depends on HAVE_VIRT_CPU_ACCOUNTING_GEN depends on GENERIC_CLOCKEVENTS select VIRT_CPU_ACCOUNTING - select CONTEXT_TRACKING + select CONTEXT_TRACKING_USER help Select this option to enable task and CPU time accounting on full dynticks systems. This accounting is implemented by watching every diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index f48eaad314ba..da067a36b326 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -22,6 +22,8 @@ #include #include =20 +#ifdef CONFIG_CONTEXT_TRACKING_USER + #define CREATE_TRACE_POINTS #include =20 @@ -238,7 +240,7 @@ void __init ct_cpu_track_user(int cpu) initialized =3D true; } =20 -#ifdef CONFIG_CONTEXT_TRACKING_FORCE +#ifdef CONFIG_CONTEXT_TRACKING_USER_FORCE void __init context_tracking_init(void) { int cpu; @@ -247,3 +249,5 @@ void __init context_tracking_init(void) ct_cpu_track_user(cpu); } #endif + +#endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 068c088e9584..f1a31cd1e726 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6483,7 +6483,7 @@ void __sched schedule_idle(void) } while (need_resched()); } =20 -#if defined(CONFIG_CONTEXT_TRACKING) && !defined(CONFIG_HAVE_CONTEXT_TRACK= ING_OFFSTACK) +#if defined(CONFIG_CONTEXT_TRACKING_USER) && !defined(CONFIG_HAVE_CONTEXT_= TRACKING_USER_OFFSTACK) asmlinkage __visible void __sched schedule_user(void) { /* diff --git a/kernel/time/Kconfig b/kernel/time/Kconfig index 27b7868b5c30..aad89cc96787 100644 --- a/kernel/time/Kconfig +++ b/kernel/time/Kconfig @@ -111,7 +111,7 @@ config NO_HZ_FULL # NO_HZ_COMMON dependency # We need at least one periodic CPU for timekeeping depends on SMP - depends on HAVE_CONTEXT_TRACKING + depends on HAVE_CONTEXT_TRACKING_USER # VIRT_CPU_ACCOUNTING_GEN dependency depends on HAVE_VIRT_CPU_ACCOUNTING_GEN select NO_HZ_COMMON @@ -140,28 +140,32 @@ endchoice config CONTEXT_TRACKING bool =20 -config CONTEXT_TRACKING_FORCE - bool "Force context tracking" - depends on CONTEXT_TRACKING +config CONTEXT_TRACKING_USER + select CONTEXT_TRACKING + bool + +config CONTEXT_TRACKING_USER_FORCE + bool "Force user context tracking" + depends on CONTEXT_TRACKING_USER default y if !NO_HZ_FULL help The major pre-requirement for full dynticks to work is to - support the context tracking subsystem. But there are also + support the user context tracking subsystem. But there are also other dependencies to provide in order to make the full dynticks working. =20 This option stands for testing when an arch implements the - context tracking backend but doesn't yet fulfill all the + user context tracking backend but doesn't yet fulfill all the requirements to make the full dynticks feature working. Without the full dynticks, there is no way to test the support - for context tracking and the subsystems that rely on it: RCU + for user context tracking and the subsystems that rely on it: RCU userspace extended quiescent state and tickless cputime accounting. This option copes with the absence of the full - dynticks subsystem by forcing the context tracking on all + dynticks subsystem by forcing the user context tracking on all CPUs in the system. =20 Say Y only if you're working on the development of an - architecture backend for the context tracking. + architecture backend for the user context tracking. =20 Say N otherwise, this option brings an overhead that you don't want in production. --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AEB6C433F5 for ; Tue, 3 May 2022 10:02:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234075AbiECKFj (ORCPT ); Tue, 3 May 2022 06:05:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233995AbiECKFM (ORCPT ); Tue, 3 May 2022 06:05:12 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3856837BC2 for ; Tue, 3 May 2022 03:01:40 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C755060EB1 for ; Tue, 3 May 2022 10:01:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2119AC385B2; Tue, 3 May 2022 10:01:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572099; bh=ozItTx5APvSOoE9WsEkdLt/utSEuuDRVqyc5MrUBHxM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cq106HeShm7GJ2Nx4CHSGjkO+4Ts8LbUORQKvgPREQtcHvVCqW/RkPzd9ckhx/Z1/ 5o1GBJntbuV17uRx6QS9BpOP2IXFYAjlD2bYQGLncjQSl8FVbwiR88uDGLwPedefX+ 1eJ9FWBOHP2VMiwR76YK6h8+WG3iXPvDWB+blY1g69rfPZH/NCwFSIdFnPHyQmLr6G L1UPY1LtyPU5IjqS4MVdPZI7FLzzoSm+zvgovgwOeWLyGMOfnHfD4UR+WTnzWGbJyd 5YQrIxt7c+zjOuls8FIYD/GzX2RxPVzwAs+OqHQeVRVaCxB9vCf7HBhJ2hPx5Ooyxv s2eiDl1mbw4aQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 10/21] context_tracking: Take idle eqs entrypoints over RCU Date: Tue, 3 May 2022 12:00:40 +0200 Message-Id: <20220503100051.2799723-11-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The RCU dynticks counter is going to be merged into the context tracking subsystem. Start with moving the idle extended quiescent states entrypoints to context tracking. For now those are dumb redirections to existing RCU calls. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- Documentation/RCU/stallwarn.rst | 4 ++-- arch/arm/mach-imx/cpuidle-imx6q.c | 5 +++-- drivers/acpi/processor_idle.c | 5 +++-- drivers/cpuidle/cpuidle.c | 9 +++++---- include/linux/context_tracking.h | 8 ++++++++ include/linux/rcupdate.h | 2 +- kernel/context_tracking.c | 12 ++++++++++++ kernel/locking/lockdep.c | 2 +- kernel/rcu/Kconfig | 2 ++ kernel/rcu/tree.c | 2 -- kernel/rcu/update.c | 2 +- kernel/sched/idle.c | 10 +++++----- kernel/sched/sched.h | 1 + 13 files changed, 44 insertions(+), 20 deletions(-) diff --git a/Documentation/RCU/stallwarn.rst b/Documentation/RCU/stallwarn.= rst index 1d863b04727c..bdd52b40f307 100644 --- a/Documentation/RCU/stallwarn.rst +++ b/Documentation/RCU/stallwarn.rst @@ -97,8 +97,8 @@ warnings: which will include additional debugging information. =20 - A low-level kernel issue that either fails to invoke one of the - variants of rcu_user_enter(), rcu_user_exit(), rcu_idle_enter(), - rcu_idle_exit(), rcu_irq_enter(), or rcu_irq_exit() on the one + variants of rcu_user_enter(), rcu_user_exit(), ct_idle_enter(), + ct_idle_exit(), rcu_irq_enter(), or rcu_irq_exit() on the one hand, or that invokes one of them too many times on the other. Historically, the most frequent issue has been an omission of either irq_enter() or irq_exit(), which in turn invoke diff --git a/arch/arm/mach-imx/cpuidle-imx6q.c b/arch/arm/mach-imx/cpuidle-= imx6q.c index 094337dc1bc7..d086cbae09c3 100644 --- a/arch/arm/mach-imx/cpuidle-imx6q.c +++ b/arch/arm/mach-imx/cpuidle-imx6q.c @@ -3,6 +3,7 @@ * Copyright (C) 2012 Freescale Semiconductor, Inc. */ =20 +#include #include #include #include @@ -24,9 +25,9 @@ static int imx6q_enter_wait(struct cpuidle_device *dev, imx6_set_lpm(WAIT_UNCLOCKED); raw_spin_unlock(&cpuidle_lock); =20 - rcu_idle_enter(); + ct_idle_enter(); cpu_do_idle(); - rcu_idle_exit(); + ct_idle_exit(); =20 raw_spin_lock(&cpuidle_lock); if (num_idle_cpus-- =3D=3D num_online_cpus()) diff --git a/drivers/acpi/processor_idle.c b/drivers/acpi/processor_idle.c index 32b20efff5f8..935f4113d5f6 100644 --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -22,6 +22,7 @@ #include #include #include +#include =20 /* * Include the apic definitions for x86 to have the APIC timer related def= ines @@ -648,11 +649,11 @@ static int acpi_idle_enter_bm(struct cpuidle_driver *= drv, raw_spin_unlock(&c3_lock); } =20 - rcu_idle_enter(); + ct_idle_enter(); =20 acpi_idle_do_entry(cx); =20 - rcu_idle_exit(); + ct_idle_exit(); =20 /* Re-enable bus master arbitration */ if (dis_bm) { diff --git a/drivers/cpuidle/cpuidle.c b/drivers/cpuidle/cpuidle.c index ef2ea1b12cd8..62dd956025f3 100644 --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -23,6 +23,7 @@ #include #include #include +#include #include =20 #include "cpuidle.h" @@ -150,12 +151,12 @@ static void enter_s2idle_proper(struct cpuidle_driver= *drv, */ stop_critical_timings(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - rcu_idle_enter(); + ct_idle_enter(); target_state->enter_s2idle(dev, drv, index); if (WARN_ON_ONCE(!irqs_disabled())) local_irq_disable(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - rcu_idle_exit(); + ct_idle_exit(); tick_unfreeze(); start_critical_timings(); =20 @@ -233,10 +234,10 @@ int cpuidle_enter_state(struct cpuidle_device *dev, s= truct cpuidle_driver *drv, =20 stop_critical_timings(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - rcu_idle_enter(); + ct_idle_enter(); entered_state =3D target_state->enter(dev, drv, index); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) - rcu_idle_exit(); + ct_idle_exit(); start_critical_timings(); =20 sched_clock_idle_wakeup_event(); diff --git a/include/linux/context_tracking.h b/include/linux/context_track= ing.h index e35ae66b4794..27afb75f2650 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -119,4 +119,12 @@ extern void context_tracking_init(void); static inline void context_tracking_init(void) { } #endif /* CONFIG_CONTEXT_TRACKING_USER_FORCE */ =20 +#ifdef CONFIG_CONTEXT_TRACKING +extern void ct_idle_enter(void); +extern void ct_idle_exit(void); +#else +static inline void ct_idle_enter(void) { } +static inline void ct_idle_exit(void) { } +#endif /* !CONFIG_CONTEXT_TRACKING */ + #endif diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 7f12daa4708b..d3b25e65f144 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -129,7 +129,7 @@ static inline void rcu_nocb_flush_deferred_wakeup(void)= { } * @a: Code that RCU needs to pay attention to. * * RCU read-side critical sections are forbidden in the inner idle loop, - * that is, between the rcu_idle_enter() and the rcu_idle_exit() -- RCU + * that is, between the ct_idle_enter() and the ct_idle_exit() -- RCU * will happily ignore any such read-side critical sections. However, * things like powertop need tracepoints in the inner idle loop. * diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index da067a36b326..987f0b2a3962 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -251,3 +251,15 @@ void __init context_tracking_init(void) #endif =20 #endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */ + +noinstr void ct_idle_enter(void) +{ + rcu_idle_enter(); +} +EXPORT_SYMBOL_GPL(ct_idle_enter); + +void ct_idle_exit(void) +{ + rcu_idle_exit(); +} +EXPORT_SYMBOL_GPL(ct_idle_exit); diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index c06cab6546ed..5f0dfe37234b 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -6546,7 +6546,7 @@ void lockdep_rcu_suspicious(const char *file, const i= nt line, const char *s) =20 /* * If a CPU is in the RCU-free window in idle (ie: in the section - * between rcu_idle_enter() and rcu_idle_exit(), then RCU + * between ct_idle_enter() and ct_idle_exit(), then RCU * considers that CPU to be in an "extended quiescent state", * which means that RCU will be completely ignoring that CPU. * Therefore, rcu_read_lock() and friends have absolutely no diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 27aab870ae4c..85f3b884fa09 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -8,6 +8,8 @@ menu "RCU Subsystem" config TREE_RCU bool default y if SMP + # Dynticks-idle tracking + select CONTEXT_TRACKING help This option selects the RCU implementation that is designed for very large SMP system with hundreds or diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 2a6e7723fc4d..92a288668e7c 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -664,7 +664,6 @@ noinstr void rcu_idle_enter(void) lockdep_assert_irqs_disabled(); rcu_eqs_enter(false); } -EXPORT_SYMBOL_GPL(rcu_idle_enter); =20 #ifdef CONFIG_NO_HZ_FULL =20 @@ -912,7 +911,6 @@ void rcu_idle_exit(void) rcu_eqs_exit(false); local_irq_restore(flags); } -EXPORT_SYMBOL_GPL(rcu_idle_exit); =20 #ifdef CONFIG_NO_HZ_FULL /** diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index 2e93acad1e31..738842c4886b 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -85,7 +85,7 @@ module_param(rcu_normal_after_boot, int, 0444); * and while lockdep is disabled. * * Note that if the CPU is in the idle loop from an RCU point of view (ie: - * that we are in the section between rcu_idle_enter() and rcu_idle_exit()) + * that we are in the section between ct_idle_enter() and ct_idle_exit()) * then rcu_read_lock_held() sets ``*ret`` to false even if the CPU did an * rcu_read_lock(). The reason for this is that RCU ignores CPUs that are * in such a section, considering these as in extended quiescent state, diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index 8f8b5020e76a..6de222b23b49 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -53,14 +53,14 @@ static noinline int __cpuidle cpu_idle_poll(void) { trace_cpu_idle(0, smp_processor_id()); stop_critical_timings(); - rcu_idle_enter(); + ct_idle_enter(); local_irq_enable(); =20 while (!tif_need_resched() && (cpu_idle_force_poll || tick_check_broadcast_expired())) cpu_relax(); =20 - rcu_idle_exit(); + ct_idle_exit(); start_critical_timings(); trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id()); =20 @@ -98,12 +98,12 @@ void __cpuidle default_idle_call(void) * * Trace IRQs enable here, then switch off RCU, and have * arch_cpu_idle() use raw_local_irq_enable(). Note that - * rcu_idle_enter() relies on lockdep IRQ state, so switch that + * ct_idle_enter() relies on lockdep IRQ state, so switch that * last -- this is very similar to the entry code. */ trace_hardirqs_on_prepare(); lockdep_hardirqs_on_prepare(_THIS_IP_); - rcu_idle_enter(); + ct_idle_enter(); lockdep_hardirqs_on(_THIS_IP_); =20 arch_cpu_idle(); @@ -116,7 +116,7 @@ void __cpuidle default_idle_call(void) */ raw_local_irq_disable(); lockdep_hardirqs_off(_THIS_IP_); - rcu_idle_exit(); + ct_idle_exit(); lockdep_hardirqs_on(_THIS_IP_); raw_local_irq_enable(); =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 58263f90c559..f398f2bf05f4 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70A92C433F5 for ; Tue, 3 May 2022 10:02:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234015AbiECKGA (ORCPT ); Tue, 3 May 2022 06:06:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51716 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234065AbiECKFT (ORCPT ); Tue, 3 May 2022 06:05:19 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 849A737A16 for ; Tue, 3 May 2022 03:01:45 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 063C8B81A67 for ; Tue, 3 May 2022 10:01:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A317FC385A9; Tue, 3 May 2022 10:01:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572102; bh=Ig/p5SHL/cvpwJMy088AefgEj1HUsUFoNHgebWeAc6k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=peq1R+NAbkGgPFyqo4szGRZJVUvCNzMQqoeKfPX9jCoLdCC4kyhiXyVYAgT3aZsLw 2wvB/gM70oRIoDard3upDC4USfABepz48Ttemmy5juc+1MGc8F4bZZ+20d6F2Pghgj 3Ay6ty3zckyBMVoaJXITo5Mp4wJbbeO0rbhfnTPkXD8j2WDtc3DbFZV7dY2Ay54Mci R88Tjs2IZT9ro4xsNc+h+/P7AEvwKAoLQyq4FSZI8cPu0YDWscVe1lehb6d9zC7OvI Q85sM8pHJro6p0H7/8UDjODaHJ6D6Lnj7v+BHuYzSDjQFissR+tWvZWKfeAvuow02T IoiK8ktv+8Slg== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 11/21] context_tracking: Take IRQ eqs entrypoints over RCU Date: Tue, 3 May 2022 12:00:41 +0200 Message-Id: <20220503100051.2799723-12-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The RCU dynticks counter is going to be merged into the context tracking subsystem. Prepare with moving the IRQ extended quiescent states entrypoints to context tracking. For now those are dumb redirection to existing RCU calls. Acked-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- .../RCU/Design/Requirements/Requirements.rst | 10 ++++---- Documentation/RCU/stallwarn.rst | 4 ++-- arch/Kconfig | 2 +- arch/arm64/kernel/entry-common.c | 6 ++--- arch/x86/mm/fault.c | 2 +- drivers/cpuidle/cpuidle-psci.c | 8 +++---- drivers/cpuidle/cpuidle-riscv-sbi.c | 8 +++---- include/linux/context_tracking_irq.h | 17 +++++++++++++ include/linux/context_tracking_state.h | 1 + include/linux/entry-common.h | 10 ++++---- include/linux/rcupdate.h | 5 ++-- include/linux/tracepoint.h | 4 ++-- kernel/context_tracking.c | 24 +++++++++++++++++-- kernel/cpu_pm.c | 8 +++---- kernel/entry/common.c | 12 +++++----- kernel/softirq.c | 4 ++-- kernel/trace/trace.c | 6 ++--- 17 files changed, 85 insertions(+), 46 deletions(-) create mode 100644 include/linux/context_tracking_irq.h diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Docum= entation/RCU/Design/Requirements/Requirements.rst index 04ed8bf27a0e..2a657fd2bca3 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -1844,10 +1844,10 @@ that meets this requirement. =20 Furthermore, NMI handlers can be interrupted by what appear to RCU to be normal interrupts. One way that this can happen is for code that -directly invokes rcu_irq_enter() and rcu_irq_exit() to be called +directly invokes ct_irq_enter() and ct_irq_exit() to be called from an NMI handler. This astonishing fact of life prompted the current -code structure, which has rcu_irq_enter() invoking -rcu_nmi_enter() and rcu_irq_exit() invoking rcu_nmi_exit(). +code structure, which has ct_irq_enter() invoking +rcu_nmi_enter() and ct_irq_exit() invoking rcu_nmi_exit(). And yes, I also learned of this requirement the hard way. =20 Loadable Modules @@ -2195,7 +2195,7 @@ scheduling-clock interrupt be enabled when RCU needs = it to be: sections, and RCU believes this CPU to be idle, no problem. This sort of thing is used by some architectures for light-weight exception handlers, which can then avoid the overhead of - rcu_irq_enter() and rcu_irq_exit() at exception entry and + ct_irq_enter() and ct_irq_exit() at exception entry and exit, respectively. Some go further and avoid the entireties of irq_enter() and irq_exit(). Just make very sure you are running some of your tests with @@ -2226,7 +2226,7 @@ scheduling-clock interrupt be enabled when RCU needs = it to be: +-----------------------------------------------------------------------+ | **Answer**: | +-----------------------------------------------------------------------+ -| One approach is to do ``rcu_irq_exit();rcu_irq_enter();`` every so | +| One approach is to do ``ct_irq_exit();ct_irq_enter();`` every so | | often. But given that long-running interrupt handlers can cause other | | problems, not least for response time, shouldn't you work to keep | | your interrupt handler's runtime within reasonable bounds? | diff --git a/Documentation/RCU/stallwarn.rst b/Documentation/RCU/stallwarn.= rst index bdd52b40f307..7858c3afa1f4 100644 --- a/Documentation/RCU/stallwarn.rst +++ b/Documentation/RCU/stallwarn.rst @@ -98,11 +98,11 @@ warnings: =20 - A low-level kernel issue that either fails to invoke one of the variants of rcu_user_enter(), rcu_user_exit(), ct_idle_enter(), - ct_idle_exit(), rcu_irq_enter(), or rcu_irq_exit() on the one + ct_idle_exit(), ct_irq_enter(), or ct_irq_exit() on the one hand, or that invokes one of them too many times on the other. Historically, the most frequent issue has been an omission of either irq_enter() or irq_exit(), which in turn invoke - rcu_irq_enter() or rcu_irq_exit(), respectively. Building your + ct_irq_enter() or ct_irq_exit(), respectively. Building your kernel with CONFIG_RCU_EQS_DEBUG=3Dy can help track down these types of issues, which sometimes arise in architecture-specific code. =20 diff --git a/arch/Kconfig b/arch/Kconfig index 7a8cd4d67d9c..888f9725b20b 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -777,7 +777,7 @@ config HAVE_CONTEXT_TRACKING_USER Syscalls need to be wrapped inside user_exit()-user_enter(), either optimized behind static key or through the slow path using TIF_NOHZ flag. Exceptions handlers must be wrapped as well. Irqs are already - protected inside rcu_irq_enter/rcu_irq_exit() but preemption or signal + protected inside ct_irq_enter/ct_irq_exit() but preemption or signal handling on irq exit still need to be protected. =20 config HAVE_CONTEXT_TRACKING_USER_OFFSTACK diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-com= mon.c index 878c65aa7206..a1490e8bf5d4 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -41,7 +41,7 @@ static __always_inline void __enter_from_kernel_mode(stru= ct pt_regs *regs) =20 if (!IS_ENABLED(CONFIG_TINY_RCU) && is_idle_task(current)) { lockdep_hardirqs_off(CALLER_ADDR0); - rcu_irq_enter(); + ct_irq_enter(); trace_hardirqs_off_finish(); =20 regs->exit_rcu =3D true; @@ -76,7 +76,7 @@ static __always_inline void __exit_to_kernel_mode(struct = pt_regs *regs) if (regs->exit_rcu) { trace_hardirqs_on_prepare(); lockdep_hardirqs_on_prepare(CALLER_ADDR0); - rcu_irq_exit(); + ct_irq_exit(); lockdep_hardirqs_on(CALLER_ADDR0); return; } @@ -84,7 +84,7 @@ static __always_inline void __exit_to_kernel_mode(struct = pt_regs *regs) trace_hardirqs_on(); } else { if (regs->exit_rcu) - rcu_irq_exit(); + ct_irq_exit(); } } =20 diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index d0074c6ed31a..b781785b1ff3 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1526,7 +1526,7 @@ DEFINE_IDTENTRY_RAW_ERRORCODE(exc_page_fault) =20 /* * Entry handling for valid #PF from kernel mode is slightly - * different: RCU is already watching and rcu_irq_enter() must not + * different: RCU is already watching and ct_irq_enter() must not * be invoked because a kernel fault on a user space address might * sleep. * diff --git a/drivers/cpuidle/cpuidle-psci.c b/drivers/cpuidle/cpuidle-psci.c index b51b5df08450..fe31b2d522b3 100644 --- a/drivers/cpuidle/cpuidle-psci.c +++ b/drivers/cpuidle/cpuidle-psci.c @@ -68,12 +68,12 @@ static int __psci_enter_domain_idle_state(struct cpuidl= e_device *dev, return -1; =20 /* Do runtime PM to manage a hierarchical CPU toplogy. */ - rcu_irq_enter_irqson(); + ct_irq_enter_irqson(); if (s2idle) dev_pm_genpd_suspend(pd_dev); else pm_runtime_put_sync_suspend(pd_dev); - rcu_irq_exit_irqson(); + ct_irq_exit_irqson(); =20 state =3D psci_get_domain_state(); if (!state) @@ -81,12 +81,12 @@ static int __psci_enter_domain_idle_state(struct cpuidl= e_device *dev, =20 ret =3D psci_cpu_suspend_enter(state) ? -1 : idx; =20 - rcu_irq_enter_irqson(); + ct_irq_enter_irqson(); if (s2idle) dev_pm_genpd_resume(pd_dev); else pm_runtime_get_sync(pd_dev); - rcu_irq_exit_irqson(); + ct_irq_exit_irqson(); =20 cpu_pm_exit(); =20 diff --git a/drivers/cpuidle/cpuidle-riscv-sbi.c b/drivers/cpuidle/cpuidle-= riscv-sbi.c index b459eda2cd37..759603181d53 100644 --- a/drivers/cpuidle/cpuidle-riscv-sbi.c +++ b/drivers/cpuidle/cpuidle-riscv-sbi.c @@ -115,12 +115,12 @@ static int __sbi_enter_domain_idle_state(struct cpuid= le_device *dev, return -1; =20 /* Do runtime PM to manage a hierarchical CPU toplogy. */ - rcu_irq_enter_irqson(); + ct_irq_enter_irqson(); if (s2idle) dev_pm_genpd_suspend(pd_dev); else pm_runtime_put_sync_suspend(pd_dev); - rcu_irq_exit_irqson(); + ct_irq_exit_irqson(); =20 if (sbi_is_domain_state_available()) state =3D sbi_get_domain_state(); @@ -129,12 +129,12 @@ static int __sbi_enter_domain_idle_state(struct cpuid= le_device *dev, =20 ret =3D sbi_suspend(state) ? -1 : idx; =20 - rcu_irq_enter_irqson(); + ct_irq_enter_irqson(); if (s2idle) dev_pm_genpd_resume(pd_dev); else pm_runtime_get_sync(pd_dev); - rcu_irq_exit_irqson(); + ct_irq_exit_irqson(); =20 cpu_pm_exit(); =20 diff --git a/include/linux/context_tracking_irq.h b/include/linux/context_t= racking_irq.h new file mode 100644 index 000000000000..60e3ed15a04e --- /dev/null +++ b/include/linux/context_tracking_irq.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_CONTEXT_TRACKING_IRQ_H +#define _LINUX_CONTEXT_TRACKING_IRQ_H + +#ifdef CONFIG_CONTEXT_TRACKING +void ct_irq_enter(void); +void ct_irq_exit(void); +void ct_irq_enter_irqson(void); +void ct_irq_exit_irqson(void); +#else +static inline void ct_irq_enter(void) { } +static inline void ct_irq_exit(void) { } +static inline void ct_irq_enter_irqson(void) { } +static inline void ct_irq_exit_irqson(void) { } +#endif + +#endif diff --git a/include/linux/context_tracking_state.h b/include/linux/context= _tracking_state.h index 913acfee5dbf..7b46b43b8c98 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -4,6 +4,7 @@ =20 #include #include +#include =20 struct context_tracking { /* diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h index ab78bd4c2eb0..2d4cf0b5fe1c 100644 --- a/include/linux/entry-common.h +++ b/include/linux/entry-common.h @@ -357,7 +357,7 @@ void irqentry_exit_to_user_mode(struct pt_regs *regs); /** * struct irqentry_state - Opaque object for exception state storage * @exit_rcu: Used exclusively in the irqentry_*() calls; signals whether = the - * exit path has to invoke rcu_irq_exit(). + * exit path has to invoke ct_irq_exit(). * @lockdep: Used exclusively in the irqentry_nmi_*() calls; ensures that * lockdep state is restored correctly on exit from nmi. * @@ -395,12 +395,12 @@ typedef struct irqentry_state { * * For kernel mode entries RCU handling is done conditional. If RCU is * watching then the only RCU requirement is to check whether the tick has - * to be restarted. If RCU is not watching then rcu_irq_enter() has to be - * invoked on entry and rcu_irq_exit() on exit. + * to be restarted. If RCU is not watching then ct_irq_enter() has to be + * invoked on entry and ct_irq_exit() on exit. * - * Avoiding the rcu_irq_enter/exit() calls is an optimization but also + * Avoiding the ct_irq_enter/exit() calls is an optimization but also * solves the problem of kernel mode pagefaults which can schedule, which - * is not possible after invoking rcu_irq_enter() without undoing it. + * is not possible after invoking ct_irq_enter() without undoing it. * * For user mode entries irqentry_enter_from_user_mode() is invoked to * establish the proper context for NOHZ_FULL. Otherwise scheduling on exit diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index d3b25e65f144..923e4fa04338 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -29,6 +29,7 @@ #include #include #include +#include =20 #define ULONG_CMP_GE(a, b) (ULONG_MAX / 2 >=3D (a) - (b)) #define ULONG_CMP_LT(a, b) (ULONG_MAX / 2 < (a) - (b)) @@ -144,9 +145,9 @@ static inline void rcu_nocb_flush_deferred_wakeup(void)= { } */ #define RCU_NONIDLE(a) \ do { \ - rcu_irq_enter_irqson(); \ + ct_irq_enter_irqson(); \ do { a; } while (0); \ - rcu_irq_exit_irqson(); \ + ct_irq_exit_irqson(); \ } while (0) =20 /* diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index 28031b15f878..55717a2eda08 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -200,13 +200,13 @@ static inline struct tracepoint *tracepoint_ptr_deref= (tracepoint_ptr_t *p) */ \ if (rcuidle) { \ __idx =3D srcu_read_lock_notrace(&tracepoint_srcu);\ - rcu_irq_enter_irqson(); \ + ct_irq_enter_irqson(); \ } \ \ __DO_TRACE_CALL(name, TP_ARGS(args)); \ \ if (rcuidle) { \ - rcu_irq_exit_irqson(); \ + ct_irq_exit_irqson(); \ srcu_read_unlock_notrace(&tracepoint_srcu, __idx);\ } \ \ diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 987f0b2a3962..a8f6d1580144 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -75,7 +75,7 @@ void noinstr __ct_user_enter(enum ctx_state state) * At this stage, only low level arch entry code remains and * then we'll run in userspace. We can assume there won't be * any RCU read-side critical section until the next call to - * user_exit() or rcu_irq_enter(). Let's remove RCU's dependency + * user_exit() or ct_irq_enter(). Let's remove RCU's dependency * on the tick. */ if (state =3D=3D CONTEXT_USER) { @@ -117,7 +117,7 @@ void ct_user_enter(enum ctx_state state) /* * Some contexts may involve an exception occuring in an irq, * leading to that nesting: - * rcu_irq_enter() rcu_user_exit() rcu_user_exit() rcu_irq_exit() + * ct_irq_enter() rcu_user_exit() rcu_user_exit() ct_irq_exit() * This would mess up the dyntick_nesting count though. And rcu_irq_*() * helpers are enough to protect RCU uses inside the exception. So * just return immediately if we detect we are in an IRQ. @@ -263,3 +263,23 @@ void ct_idle_exit(void) rcu_idle_exit(); } EXPORT_SYMBOL_GPL(ct_idle_exit); + +noinstr void ct_irq_enter(void) +{ + rcu_irq_enter(); +} + +noinstr void ct_irq_exit(void) +{ + rcu_irq_exit(); +} + +void ct_irq_enter_irqson(void) +{ + rcu_irq_enter_irqson(); +} + +void ct_irq_exit_irqson(void) +{ + rcu_irq_exit_irqson(); +} diff --git a/kernel/cpu_pm.c b/kernel/cpu_pm.c index 246efc74e3f3..ba4ba71facf9 100644 --- a/kernel/cpu_pm.c +++ b/kernel/cpu_pm.c @@ -35,11 +35,11 @@ static int cpu_pm_notify(enum cpu_pm_event event) * disfunctional in cpu idle. Copy RCU_NONIDLE code to let RCU know * this. */ - rcu_irq_enter_irqson(); + ct_irq_enter_irqson(); rcu_read_lock(); ret =3D raw_notifier_call_chain(&cpu_pm_notifier.chain, event, NULL); rcu_read_unlock(); - rcu_irq_exit_irqson(); + ct_irq_exit_irqson(); =20 return notifier_to_errno(ret); } @@ -49,11 +49,11 @@ static int cpu_pm_notify_robust(enum cpu_pm_event event= _up, enum cpu_pm_event ev unsigned long flags; int ret; =20 - rcu_irq_enter_irqson(); + ct_irq_enter_irqson(); raw_spin_lock_irqsave(&cpu_pm_notifier.lock, flags); ret =3D raw_notifier_call_chain_robust(&cpu_pm_notifier.chain, event_up, = event_down, NULL); raw_spin_unlock_irqrestore(&cpu_pm_notifier.lock, flags); - rcu_irq_exit_irqson(); + ct_irq_exit_irqson(); =20 return notifier_to_errno(ret); } diff --git a/kernel/entry/common.c b/kernel/entry/common.c index e57a224d6b79..9a89386696ba 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -321,7 +321,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs = *regs) } =20 /* - * If this entry hit the idle task invoke rcu_irq_enter() whether + * If this entry hit the idle task invoke ct_irq_enter() whether * RCU is watching or not. * * Interrupts can nest when the first interrupt invokes softirq @@ -332,12 +332,12 @@ noinstr irqentry_state_t irqentry_enter(struct pt_reg= s *regs) * not nested into another interrupt. * * Checking for rcu_is_watching() here would prevent the nesting - * interrupt to invoke rcu_irq_enter(). If that nested interrupt is + * interrupt to invoke ct_irq_enter(). If that nested interrupt is * the tick then rcu_flavor_sched_clock_irq() would wrongfully * assume that it is the first interrupt and eventually claim * quiescent state and end grace periods prematurely. * - * Unconditionally invoke rcu_irq_enter() so RCU state stays + * Unconditionally invoke ct_irq_enter() so RCU state stays * consistent. * * TINY_RCU does not support EQS, so let the compiler eliminate @@ -350,7 +350,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs = *regs) * as in irqentry_enter_from_user_mode(). */ lockdep_hardirqs_off(CALLER_ADDR0); - rcu_irq_enter(); + ct_irq_enter(); instrumentation_begin(); trace_hardirqs_off_finish(); instrumentation_end(); @@ -418,7 +418,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqent= ry_state_t state) trace_hardirqs_on_prepare(); lockdep_hardirqs_on_prepare(CALLER_ADDR0); instrumentation_end(); - rcu_irq_exit(); + ct_irq_exit(); lockdep_hardirqs_on(CALLER_ADDR0); return; } @@ -436,7 +436,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqent= ry_state_t state) * was not watching on entry. */ if (state.exit_rcu) - rcu_irq_exit(); + ct_irq_exit(); } } =20 diff --git a/kernel/softirq.c b/kernel/softirq.c index fac801815554..7db9db832b31 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -607,7 +607,7 @@ void irq_enter_rcu(void) */ void irq_enter(void) { - rcu_irq_enter(); + ct_irq_enter(); irq_enter_rcu(); } =20 @@ -659,7 +659,7 @@ void irq_exit_rcu(void) void irq_exit(void) { __irq_exit_rcu(); - rcu_irq_exit(); + ct_irq_exit(); /* must be last! */ lockdep_hardirq_exit(); } diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index f4de111fa18f..85e519cc0af9 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -3104,15 +3104,15 @@ void __trace_stack(struct trace_array *tr, unsigned= int trace_ctx, /* * When an NMI triggers, RCU is enabled via rcu_nmi_enter(), * but if the above rcu_is_watching() failed, then the NMI - * triggered someplace critical, and rcu_irq_enter() should + * triggered someplace critical, and ct_irq_enter() should * not be called from NMI. */ if (unlikely(in_nmi())) return; =20 - rcu_irq_enter_irqson(); + ct_irq_enter_irqson(); __ftrace_trace_stack(buffer, trace_ctx, skip, NULL); - rcu_irq_exit_irqson(); + ct_irq_exit_irqson(); } =20 /** --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35DFDC433F5 for ; Tue, 3 May 2022 10:02:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234040AbiECKGR (ORCPT ); Tue, 3 May 2022 06:06:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233660AbiECKFx (ORCPT ); Tue, 3 May 2022 06:05:53 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4103338BED for ; Tue, 3 May 2022 03:01:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 2C793CE09AF for ; Tue, 3 May 2022 10:01:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3481CC385B4; Tue, 3 May 2022 10:01:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572106; bh=WSZpGkaPSOhYt8eqQKRv1VU5U8lM7OEWU0wLB++eSuM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cFr9FS/wDvT7ffPUSU855BZQj1PEfPIuIBK71MkBqq832JJ/hL/x4SKRSKAmMPj4U UuMtUau732m1AprOT1O9BN+kFO2vCL0HG9UnRJxy/X6g9raY5bUDr5E0aC8yf7zKC8 FhdlgDZnpN1eyxA6y2lhxqURUeQ8JI7M/HlyXwcxvLGJ0C5xL1WQ3PiVgvOYI0iiRs EOjZtFARQWoaS0Z5opfevOXY6n3W1c47P9HXIL7b6npFI1/PEdGpjaS+srwCHrOzQM 6jZUc3eZGSIUQCscjs7GA4osDLRNOFaVNdVpxeMYX8AXI9FrWyrlydW+vA/9wtxkz7 w4g42xqMRU7ag== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 12/21] context_tracking: Take NMI eqs entrypoints over RCU Date: Tue, 3 May 2022 12:00:42 +0200 Message-Id: <20220503100051.2799723-13-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The RCU dynticks counter is going to be merged into the context tracking subsystem. Prepare with moving the NMI extended quiescent states entrypoints to context tracking. For now those are dumb redirection to existing RCU calls. Acked-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- Documentation/RCU/Design/Requirements/Requirements.rst | 2 +- arch/Kconfig | 2 +- arch/arm64/kernel/entry-common.c | 8 ++++---- include/linux/context_tracking_irq.h | 4 ++++ include/linux/hardirq.h | 4 ++-- kernel/context_tracking.c | 10 ++++++++++ kernel/entry/common.c | 4 ++-- kernel/extable.c | 4 ++-- kernel/trace/trace.c | 2 +- 9 files changed, 27 insertions(+), 13 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Docum= entation/RCU/Design/Requirements/Requirements.rst index 2a657fd2bca3..ac323f2af19b 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -1847,7 +1847,7 @@ normal interrupts. One way that this can happen is fo= r code that directly invokes ct_irq_enter() and ct_irq_exit() to be called from an NMI handler. This astonishing fact of life prompted the current code structure, which has ct_irq_enter() invoking -rcu_nmi_enter() and ct_irq_exit() invoking rcu_nmi_exit(). +ct_nmi_enter() and ct_irq_exit() invoking ct_nmi_exit(). And yes, I also learned of this requirement the hard way. =20 Loadable Modules diff --git a/arch/Kconfig b/arch/Kconfig index 888f9725b20b..a1150a190bf6 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -792,7 +792,7 @@ config HAVE_CONTEXT_TRACKING_USER_OFFSTACK =20 - Critical entry code isn't preemptible (or better yet: not interruptible). - - No use of RCU read side critical sections, unless rcu_nmi_enter() + - No use of RCU read side critical sections, unless ct_nmi_enter() got called. - No use of instrumentation, unless instrumentation_begin() got called. diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-com= mon.c index a1490e8bf5d4..b8fcdd777a8a 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -161,7 +161,7 @@ static void noinstr arm64_enter_nmi(struct pt_regs *reg= s) __nmi_enter(); lockdep_hardirqs_off(CALLER_ADDR0); lockdep_hardirq_enter(); - rcu_nmi_enter(); + ct_nmi_enter(); =20 trace_hardirqs_off_finish(); ftrace_nmi_enter(); @@ -182,7 +182,7 @@ static void noinstr arm64_exit_nmi(struct pt_regs *regs) lockdep_hardirqs_on_prepare(CALLER_ADDR0); } =20 - rcu_nmi_exit(); + ct_nmi_exit(); lockdep_hardirq_exit(); if (restore) lockdep_hardirqs_on(CALLER_ADDR0); @@ -199,7 +199,7 @@ static void noinstr arm64_enter_el1_dbg(struct pt_regs = *regs) regs->lockdep_hardirqs =3D lockdep_hardirqs_enabled(); =20 lockdep_hardirqs_off(CALLER_ADDR0); - rcu_nmi_enter(); + ct_nmi_enter(); =20 trace_hardirqs_off_finish(); } @@ -218,7 +218,7 @@ static void noinstr arm64_exit_el1_dbg(struct pt_regs *= regs) lockdep_hardirqs_on_prepare(CALLER_ADDR0); } =20 - rcu_nmi_exit(); + ct_nmi_exit(); if (restore) lockdep_hardirqs_on(CALLER_ADDR0); } diff --git a/include/linux/context_tracking_irq.h b/include/linux/context_t= racking_irq.h index 60e3ed15a04e..11043bf724b7 100644 --- a/include/linux/context_tracking_irq.h +++ b/include/linux/context_tracking_irq.h @@ -7,11 +7,15 @@ void ct_irq_enter(void); void ct_irq_exit(void); void ct_irq_enter_irqson(void); void ct_irq_exit_irqson(void); +void ct_nmi_enter(void); +void ct_nmi_exit(void); #else static inline void ct_irq_enter(void) { } static inline void ct_irq_exit(void) { } static inline void ct_irq_enter_irqson(void) { } static inline void ct_irq_exit_irqson(void) { } +static inline void ct_nmi_enter(void) { } +static inline void ct_nmi_exit(void) { } #endif =20 #endif diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h index 76878b357ffa..345cdbe9c1b7 100644 --- a/include/linux/hardirq.h +++ b/include/linux/hardirq.h @@ -124,7 +124,7 @@ extern void rcu_nmi_exit(void); do { \ __nmi_enter(); \ lockdep_hardirq_enter(); \ - rcu_nmi_enter(); \ + ct_nmi_enter(); \ instrumentation_begin(); \ ftrace_nmi_enter(); \ instrumentation_end(); \ @@ -143,7 +143,7 @@ extern void rcu_nmi_exit(void); instrumentation_begin(); \ ftrace_nmi_exit(); \ instrumentation_end(); \ - rcu_nmi_exit(); \ + ct_nmi_exit(); \ lockdep_hardirq_exit(); \ __nmi_exit(); \ } while (0) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index a8f6d1580144..7aa5b0cad19e 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -283,3 +283,13 @@ void ct_irq_exit_irqson(void) { rcu_irq_exit_irqson(); } + +noinstr void ct_nmi_enter(void) +{ + rcu_nmi_enter(); +} + +noinstr void ct_nmi_exit(void) +{ + rcu_nmi_exit(); +} diff --git a/kernel/entry/common.c b/kernel/entry/common.c index 9a89386696ba..256fb950a603 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -449,7 +449,7 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_r= egs *regs) __nmi_enter(); lockdep_hardirqs_off(CALLER_ADDR0); lockdep_hardirq_enter(); - rcu_nmi_enter(); + ct_nmi_enter(); =20 instrumentation_begin(); trace_hardirqs_off_finish(); @@ -469,7 +469,7 @@ void noinstr irqentry_nmi_exit(struct pt_regs *regs, ir= qentry_state_t irq_state) } instrumentation_end(); =20 - rcu_nmi_exit(); + ct_nmi_exit(); lockdep_hardirq_exit(); if (irq_state.lockdep) lockdep_hardirqs_on(CALLER_ADDR0); diff --git a/kernel/extable.c b/kernel/extable.c index bda5e9761541..71f482581cab 100644 --- a/kernel/extable.c +++ b/kernel/extable.c @@ -114,7 +114,7 @@ int kernel_text_address(unsigned long addr) =20 /* Treat this like an NMI as it can happen anywhere */ if (no_rcu) - rcu_nmi_enter(); + ct_nmi_enter(); =20 if (is_module_text_address(addr)) goto out; @@ -127,7 +127,7 @@ int kernel_text_address(unsigned long addr) ret =3D 0; out: if (no_rcu) - rcu_nmi_exit(); + ct_nmi_exit(); =20 return ret; } diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 85e519cc0af9..c096da6ea121 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -3102,7 +3102,7 @@ void __trace_stack(struct trace_array *tr, unsigned i= nt trace_ctx, } =20 /* - * When an NMI triggers, RCU is enabled via rcu_nmi_enter(), + * When an NMI triggers, RCU is enabled via ct_nmi_enter(), * but if the above rcu_is_watching() failed, then the NMI * triggered someplace critical, and ct_irq_enter() should * not be called from NMI. --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3B48C433F5 for ; Tue, 3 May 2022 10:02:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234110AbiECKGV (ORCPT ); Tue, 3 May 2022 06:06:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51570 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234102AbiECKF4 (ORCPT ); Tue, 3 May 2022 06:05:56 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3060387BA for ; Tue, 3 May 2022 03:01:52 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 0D0F8B81A67 for ; Tue, 3 May 2022 10:01:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B7D32C385B2; Tue, 3 May 2022 10:01:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572109; bh=AumO1WHwxqLU3UgtGjbcJH3pOM5KP1OVHSyP/XjQQ2Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rOgU7mAQCmmpaXIXlHrJmgC5D5u3sDRlg5gSkkSbO4Zf4tm2ubf/K0FEEdjF7cvcI QrU765v1VIqo2FmJ4cejUvqmx0jAGwL4Hz+nUtSvJnrGRTy3Do6W9oZqZMmZiHhX2V 7FDDhzigqY5CiFxge9eusgt9Usu4V/fG+xNwuLDrZJLqVw4Lwkyao04y4dQCbzXjXH d1+eDLnZD5RcSxsZVyO8Zx/UGxvwgbJ/awdKRIJPI/vKDXEEyn2btgu0nfpsB4uTGp 1FrZPpwPsNWwdIQNTS6rh6Rk90vJUWvWiRTY7R3pTfhZptSJeV9+UDj0+CJ5v0QyYk p9GEyRLseDvRQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 13/21] rcu/context-tracking: Remove rcu_irq_enter/exit() Date: Tue, 3 May 2022 12:00:43 +0200 Message-Id: <20220503100051.2799723-14-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Now rcu_irq_enter/exit() is an unecessary middle call between ct_irq_enter/exit() and nmi_irq_enter/exit(). Take this opportunity to remove the former functions and move the comments above them to the new entrypoints. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/rcutiny.h | 4 -- include/linux/rcutree.h | 4 -- kernel/context_tracking.c | 79 +++++++++++++++++++++++++++++++-- kernel/rcu/tree.c | 91 --------------------------------------- 4 files changed, 75 insertions(+), 103 deletions(-) diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index ab7e20dfb07b..5921d9ae7ab5 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -88,10 +88,6 @@ static inline void rcu_cpu_stall_reset(void) { } static inline int rcu_jiffies_till_stall_check(void) { return 21 * HZ; } static inline void rcu_idle_enter(void) { } static inline void rcu_idle_exit(void) { } -static inline void rcu_irq_enter(void) { } -static inline void rcu_irq_exit_irqson(void) { } -static inline void rcu_irq_enter_irqson(void) { } -static inline void rcu_irq_exit(void) { } static inline void rcu_irq_exit_check_preempt(void) { } #define rcu_is_idle_cpu(cpu) \ (is_idle_task(current) && !in_nmi() && !in_hardirq() && !in_serving_softi= rq()) diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 20dbaa9a3882..cafe3fbf4272 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -49,10 +49,6 @@ void cond_synchronize_rcu(unsigned long oldstate); =20 void rcu_idle_enter(void); void rcu_idle_exit(void); -void rcu_irq_enter(void); -void rcu_irq_exit(void); -void rcu_irq_enter_irqson(void); -void rcu_irq_exit_irqson(void); bool rcu_is_idle_cpu(int cpu); =20 #ifdef CONFIG_PROVE_RCU diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 7aa5b0cad19e..a9570365c7f3 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -264,24 +264,95 @@ void ct_idle_exit(void) } EXPORT_SYMBOL_GPL(ct_idle_exit); =20 +/** + * ct_irq_enter - inform RCU that current CPU is entering irq away from id= le + * + * Enter an interrupt handler, which might possibly result in exiting + * idle mode, in other words, entering the mode in which read-side critical + * sections can occur. The caller must have disabled interrupts. + * + * Note that the Linux kernel is fully capable of entering an interrupt + * handler that it never exits, for example when doing upcalls to user mod= e! + * This code assumes that the idle loop never does upcalls to user mode. + * If your architecture's idle loop does do upcalls to user mode (or does + * anything else that results in unbalanced calls to the irq_enter() and + * irq_exit() functions), RCU will give you what you deserve, good and har= d. + * But very infrequently and irreproducibly. + * + * Use things like work queues to work around this limitation. + * + * You have been warned. + * + * If you add or remove a call to ct_irq_enter(), be sure to test with + * CONFIG_RCU_EQS_DEBUG=3Dy. + */ noinstr void ct_irq_enter(void) { - rcu_irq_enter(); + lockdep_assert_irqs_disabled(); + ct_nmi_enter(); } =20 +/** + * ct_irq_exit - inform RCU that current CPU is exiting irq towards idle + * + * Exit from an interrupt handler, which might possibly result in entering + * idle mode, in other words, leaving the mode in which read-side critical + * sections can occur. The caller must have disabled interrupts. + * + * This code assumes that the idle loop never does anything that might + * result in unbalanced calls to irq_enter() and irq_exit(). If your + * architecture's idle loop violates this assumption, RCU will give you wh= at + * you deserve, good and hard. But very infrequently and irreproducibly. + * + * Use things like work queues to work around this limitation. + * + * You have been warned. + * + * If you add or remove a call to ct_irq_exit(), be sure to test with + * CONFIG_RCU_EQS_DEBUG=3Dy. + */ noinstr void ct_irq_exit(void) { - rcu_irq_exit(); + lockdep_assert_irqs_disabled(); + ct_nmi_exit(); } =20 +/* + * Wrapper for ct_irq_enter() where interrupts are enabled. + * + * If you add or remove a call to ct_irq_enter_irqson(), be sure to test + * with CONFIG_RCU_EQS_DEBUG=3Dy. + * + * FIXME: This function should be noinstr but the below local_irq_save() is + * unsafe because it involves illegal RCU uses through tracing and lockdep. + * This must be fixed first. + */ void ct_irq_enter_irqson(void) { - rcu_irq_enter_irqson(); + unsigned long flags; + + local_irq_save(flags); + ct_irq_enter(); + local_irq_restore(flags); } =20 +/* + * Wrapper for ct_irq_exit() where interrupts are enabled. + * + * If you add or remove a call to ct_irq_exit_irqson(), be sure to test + * with CONFIG_RCU_EQS_DEBUG=3Dy. + * + * FIXME: This function should be noinstr but the below local_irq_restore(= ) is + * unsafe because it involves illegal RCU uses through tracing and lockdep. + * This must be fixed first. + */ void ct_irq_exit_irqson(void) { - rcu_irq_exit_irqson(); + unsigned long flags; + + local_irq_save(flags); + ct_irq_exit(); + local_irq_restore(flags); } =20 noinstr void ct_nmi_enter(void) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 92a288668e7c..38c23229103f 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -789,31 +789,6 @@ noinstr void rcu_nmi_exit(void) rcu_dynticks_task_enter(); } =20 -/** - * rcu_irq_exit - inform RCU that current CPU is exiting irq towards idle - * - * Exit from an interrupt handler, which might possibly result in entering - * idle mode, in other words, leaving the mode in which read-side critical - * sections can occur. The caller must have disabled interrupts. - * - * This code assumes that the idle loop never does anything that might - * result in unbalanced calls to irq_enter() and irq_exit(). If your - * architecture's idle loop violates this assumption, RCU will give you wh= at - * you deserve, good and hard. But very infrequently and irreproducibly. - * - * Use things like work queues to work around this limitation. - * - * You have been warned. - * - * If you add or remove a call to rcu_irq_exit(), be sure to test with - * CONFIG_RCU_EQS_DEBUG=3Dy. - */ -void noinstr rcu_irq_exit(void) -{ - lockdep_assert_irqs_disabled(); - rcu_nmi_exit(); -} - #ifdef CONFIG_PROVE_RCU /** * rcu_irq_exit_check_preempt - Validate that scheduling is possible @@ -832,25 +807,6 @@ void rcu_irq_exit_check_preempt(void) } #endif /* #ifdef CONFIG_PROVE_RCU */ =20 -/* - * Wrapper for rcu_irq_exit() where interrupts are enabled. - * - * If you add or remove a call to rcu_irq_exit_irqson(), be sure to test - * with CONFIG_RCU_EQS_DEBUG=3Dy. - * - * FIXME: This function should be noinstr but the below local_irq_restore(= ) is - * unsafe because it involves illegal RCU uses through tracing and lockdep. - * This must be fixed first. - */ -void rcu_irq_exit_irqson(void) -{ - unsigned long flags; - - local_irq_save(flags); - rcu_irq_exit(); - local_irq_restore(flags); -} - /* * Exit an RCU extended quiescent state, which can be either the * idle loop or adaptive-tickless usermode execution. @@ -1049,53 +1005,6 @@ noinstr void rcu_nmi_enter(void) barrier(); } =20 -/** - * rcu_irq_enter - inform RCU that current CPU is entering irq away from i= dle - * - * Enter an interrupt handler, which might possibly result in exiting - * idle mode, in other words, entering the mode in which read-side critical - * sections can occur. The caller must have disabled interrupts. - * - * Note that the Linux kernel is fully capable of entering an interrupt - * handler that it never exits, for example when doing upcalls to user mod= e! - * This code assumes that the idle loop never does upcalls to user mode. - * If your architecture's idle loop does do upcalls to user mode (or does - * anything else that results in unbalanced calls to the irq_enter() and - * irq_exit() functions), RCU will give you what you deserve, good and har= d. - * But very infrequently and irreproducibly. - * - * Use things like work queues to work around this limitation. - * - * You have been warned. - * - * If you add or remove a call to rcu_irq_enter(), be sure to test with - * CONFIG_RCU_EQS_DEBUG=3Dy. - */ -noinstr void rcu_irq_enter(void) -{ - lockdep_assert_irqs_disabled(); - rcu_nmi_enter(); -} - -/* - * Wrapper for rcu_irq_enter() where interrupts are enabled. - * - * If you add or remove a call to rcu_irq_enter_irqson(), be sure to test - * with CONFIG_RCU_EQS_DEBUG=3Dy. - * - * FIXME: This function should be noinstr but the below local_irq_save() is - * unsafe because it involves illegal RCU uses through tracing and lockdep. - * This must be fixed first. - */ -void rcu_irq_enter_irqson(void) -{ - unsigned long flags; - - local_irq_save(flags); - rcu_irq_enter(); - local_irq_restore(flags); -} - /* * Check to see if any future non-offloaded RCU-related work will need * to be done by the current CPU, even if none need be done immediately, --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D80AC433EF for ; Tue, 3 May 2022 10:03:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234172AbiECKHN (ORCPT ); Tue, 3 May 2022 06:07:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234007AbiECKF5 (ORCPT ); Tue, 3 May 2022 06:05:57 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70E1838DB3 for ; Tue, 3 May 2022 03:01:54 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E95546152A for ; Tue, 3 May 2022 10:01:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4446BC385A4; Tue, 3 May 2022 10:01:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572113; bh=KzZMflvNTePfVDdtCeGLwMrCmi4gutfxLVwPBqOuL9Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Cke63GAuogAL7SGltXPJpAPosIIydpQwsva2lBP5kxiGee58UA/+5snApOeTgV223 Q+ayJMc48K8TzGXQPw7tE2qHO5JJY8y+VqZFppYXpJjwwAUQyEm/l5eCZI/QyHSx/8 8QjULxct0nQ/IF1MyrYuOtK95hvbV6tlLv4SLTxY/bJpyqJ0AupKjzTUrJ9AbSdyaX 6eVIO8VgZrWSRNRJfdpiCzPj+eH0RmUJOZbLPTAT6tw5QIjXiJtnXiHFL3RiN2TJPK 5Iv+sKtxOb/OMqo3d3rQvnJlwhpUOlvrXgo5ImiNAhjQ3YVIaBlCttx+Oy8mL0AjJH CRnAM/wKd1kyQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 14/21] rcu/context_tracking: Move dynticks counter to context tracking Date: Tue, 3 May 2022 12:00:44 +0200 Message-Id: <20220503100051.2799723-15-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In order to prepare for merging RCU dynticks counter into the context tracking state, move the rcu_data's dynticks field to the context tracking structure. It will later be mixed within the context tracking state itself. Acked-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/context_tracking_state.h | 27 ++++++++++++- kernel/context_tracking.c | 9 +++-- kernel/rcu/tree.c | 56 +++++++++++++------------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_stall.h | 4 +- 6 files changed, 61 insertions(+), 38 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context= _tracking_state.h index 7b46b43b8c98..014dc431521b 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -7,6 +7,7 @@ #include =20 struct context_tracking { +#ifdef CONFIG_CONTEXT_TRACKING_USER /* * When active is false, probes are unset in order * to minimize overhead: TIF flags are cleared @@ -21,12 +22,34 @@ struct context_tracking { CONTEXT_USER, CONTEXT_GUEST, } state; +#endif + atomic_t dynticks; /* Even value for idle, else odd. */ }; =20 -#ifdef CONFIG_CONTEXT_TRACKING_USER -extern struct static_key_false context_tracking_key; +#ifdef CONFIG_CONTEXT_TRACKING DECLARE_PER_CPU(struct context_tracking, context_tracking); =20 +static __always_inline int ct_dynticks(void) +{ + return atomic_read(this_cpu_ptr(&context_tracking.dynticks)); +} + +static __always_inline int ct_dynticks_cpu(int cpu) +{ + struct context_tracking *ct =3D per_cpu_ptr(&context_tracking, cpu); + return atomic_read(&ct->dynticks); +} + +static __always_inline int ct_dynticks_cpu_acquire(int cpu) +{ + struct context_tracking *ct =3D per_cpu_ptr(&context_tracking, cpu); + return atomic_read_acquire(&ct->state); +} +#endif + +#ifdef CONFIG_CONTEXT_TRACKING_USER +extern struct static_key_false context_tracking_key; + static __always_inline bool context_tracking_enabled(void) { return static_branch_unlikely(&context_tracking_key); diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index a9570365c7f3..20be30c24723 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -30,9 +30,6 @@ DEFINE_STATIC_KEY_FALSE(context_tracking_key); EXPORT_SYMBOL_GPL(context_tracking_key); =20 -DEFINE_PER_CPU(struct context_tracking, context_tracking); -EXPORT_SYMBOL_GPL(context_tracking); - static noinstr bool context_tracking_recursion_enter(void) { int recursion; @@ -252,6 +249,12 @@ void __init context_tracking_init(void) =20 #endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */ =20 +DEFINE_PER_CPU(struct context_tracking, context_tracking) =3D { + .dynticks =3D ATOMIC_INIT(1), +}; +EXPORT_SYMBOL_GPL(context_tracking); + + noinstr void ct_idle_enter(void) { rcu_idle_enter(); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 38c23229103f..7667186731e3 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -77,7 +77,6 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) =3D { .dynticks_nesting =3D 1, .dynticks_nmi_nesting =3D DYNTICK_IRQ_NONIDLE, - .dynticks =3D ATOMIC_INIT(1), #ifdef CONFIG_RCU_NOCB_CPU .cblist.flags =3D SEGCBLIST_RCU_CORE, #endif @@ -268,7 +267,7 @@ void rcu_softirq_qs(void) */ static noinline noinstr unsigned long rcu_dynticks_inc(int incby) { - return arch_atomic_add_return(incby, this_cpu_ptr(&rcu_data.dynticks)); + return arch_atomic_add_return(incby, this_cpu_ptr(&context_tracking.dynti= cks)); } =20 /* @@ -324,9 +323,7 @@ static noinstr void rcu_dynticks_eqs_exit(void) */ static void rcu_dynticks_eqs_online(void) { - struct rcu_data *rdp =3D this_cpu_ptr(&rcu_data); - - if (atomic_read(&rdp->dynticks) & 0x1) + if (ct_dynticks() & 0x1) return; rcu_dynticks_inc(1); } @@ -338,17 +335,17 @@ static void rcu_dynticks_eqs_online(void) */ static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void) { - return !(arch_atomic_read(this_cpu_ptr(&rcu_data.dynticks)) & 0x1); + return !(arch_atomic_read(this_cpu_ptr(&context_tracking.dynticks)) & 0x1= ); } =20 /* * Snapshot the ->dynticks counter with full ordering so as to allow * stable comparison of this counter with past and future snapshots. */ -static int rcu_dynticks_snap(struct rcu_data *rdp) +static int rcu_dynticks_snap(int cpu) { smp_mb(); // Fundamental RCU ordering guarantee. - return atomic_read_acquire(&rdp->dynticks); + return ct_dynticks_cpu_acquire(cpu); } =20 /* @@ -363,9 +360,7 @@ static bool rcu_dynticks_in_eqs(int snap) /* Return true if the specified CPU is currently idle from an RCU viewpoin= t. */ bool rcu_is_idle_cpu(int cpu) { - struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); - - return rcu_dynticks_in_eqs(rcu_dynticks_snap(rdp)); + return rcu_dynticks_in_eqs(rcu_dynticks_snap(cpu)); } =20 /* @@ -375,7 +370,7 @@ bool rcu_is_idle_cpu(int cpu) */ static bool rcu_dynticks_in_eqs_since(struct rcu_data *rdp, int snap) { - return snap !=3D rcu_dynticks_snap(rdp); + return snap !=3D rcu_dynticks_snap(rdp->cpu); } =20 /* @@ -384,11 +379,10 @@ static bool rcu_dynticks_in_eqs_since(struct rcu_data= *rdp, int snap) */ bool rcu_dynticks_zero_in_eqs(int cpu, int *vp) { - struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); int snap; =20 // If not quiescent, force back to earlier extended quiescent state. - snap =3D atomic_read(&rdp->dynticks) & ~0x1; + snap =3D ct_dynticks_cpu(cpu) & ~0x1; =20 smp_rmb(); // Order ->dynticks and *vp reads. if (READ_ONCE(*vp)) @@ -396,7 +390,7 @@ bool rcu_dynticks_zero_in_eqs(int cpu, int *vp) smp_rmb(); // Order *vp read and ->dynticks re-read. =20 // If still in the same extended quiescent state, we are good! - return snap =3D=3D atomic_read(&rdp->dynticks); + return snap =3D=3D ct_dynticks_cpu(cpu); } =20 /* @@ -620,6 +614,7 @@ EXPORT_SYMBOL_GPL(rcutorture_get_gp_data); static noinstr void rcu_eqs_enter(bool user) { struct rcu_data *rdp =3D this_cpu_ptr(&rcu_data); + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); =20 WARN_ON_ONCE(rdp->dynticks_nmi_nesting !=3D DYNTICK_IRQ_NONIDLE); WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); @@ -633,12 +628,12 @@ static noinstr void rcu_eqs_enter(bool user) =20 lockdep_assert_irqs_disabled(); instrumentation_begin(); - trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, atomic_read(&rd= p->dynticks)); + trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, ct_dynticks()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); rcu_preempt_deferred_qs(current); =20 // instrumentation for the noinstr rcu_dynticks_eqs_enter() - instrument_atomic_write(&rdp->dynticks, sizeof(rdp->dynticks)); + instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); =20 instrumentation_end(); WRITE_ONCE(rdp->dynticks_nesting, 0); /* Avoid irq-access tearing. */ @@ -740,7 +735,7 @@ noinstr void rcu_user_enter(void) * rcu_nmi_exit - inform RCU of exit from NMI context * * If we are returning from the outermost NMI handler that interrupted an - * RCU-idle period, update rdp->dynticks and rdp->dynticks_nmi_nesting + * RCU-idle period, update ct->dynticks and rdp->dynticks_nmi_nesting * to let the RCU grace-period handling know that the CPU is back to * being RCU-idle. * @@ -749,6 +744,7 @@ noinstr void rcu_user_enter(void) */ noinstr void rcu_nmi_exit(void) { + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); struct rcu_data *rdp =3D this_cpu_ptr(&rcu_data); =20 instrumentation_begin(); @@ -766,7 +762,7 @@ noinstr void rcu_nmi_exit(void) */ if (rdp->dynticks_nmi_nesting !=3D 1) { trace_rcu_dyntick(TPS("--=3D"), rdp->dynticks_nmi_nesting, rdp->dynticks= _nmi_nesting - 2, - atomic_read(&rdp->dynticks)); + ct_dynticks()); WRITE_ONCE(rdp->dynticks_nmi_nesting, /* No store tearing. */ rdp->dynticks_nmi_nesting - 2); instrumentation_end(); @@ -774,11 +770,11 @@ noinstr void rcu_nmi_exit(void) } =20 /* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */ - trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, atomic_r= ead(&rdp->dynticks)); + trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, ct_dynti= cks()); WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); /* Avoid store tearing. */ =20 // instrumentation for the noinstr rcu_dynticks_eqs_enter() - instrument_atomic_write(&rdp->dynticks, sizeof(rdp->dynticks)); + instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); instrumentation_end(); =20 // RCU is watching here ... @@ -817,6 +813,7 @@ void rcu_irq_exit_check_preempt(void) */ static void noinstr rcu_eqs_exit(bool user) { + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); struct rcu_data *rdp; long oldval; =20 @@ -836,9 +833,9 @@ static void noinstr rcu_eqs_exit(bool user) instrumentation_begin(); =20 // instrumentation for the noinstr rcu_dynticks_eqs_exit() - instrument_atomic_write(&rdp->dynticks, sizeof(rdp->dynticks)); + instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); =20 - trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, atomic_read(&rdp-= >dynticks)); + trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, ct_dynticks()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); WRITE_ONCE(rdp->dynticks_nesting, 1); WARN_ON_ONCE(rdp->dynticks_nmi_nesting); @@ -948,7 +945,7 @@ void __rcu_irq_enter_check_tick(void) /** * rcu_nmi_enter - inform RCU of entry to NMI context * - * If the CPU was idle from RCU's viewpoint, update rdp->dynticks and + * If the CPU was idle from RCU's viewpoint, update ct->dynticks and * rdp->dynticks_nmi_nesting to let the RCU grace-period handling know * that the CPU is active. This implementation permits nested NMIs, as * long as the nesting level does not overflow an int. (You will probably @@ -961,6 +958,7 @@ noinstr void rcu_nmi_enter(void) { long incby =3D 2; struct rcu_data *rdp =3D this_cpu_ptr(&rcu_data); + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); =20 /* Complain about underflow. */ WARN_ON_ONCE(rdp->dynticks_nmi_nesting < 0); @@ -984,9 +982,9 @@ noinstr void rcu_nmi_enter(void) =20 instrumentation_begin(); // instrumentation for the noinstr rcu_dynticks_curr_cpu_in_eqs() - instrument_atomic_read(&rdp->dynticks, sizeof(rdp->dynticks)); + instrument_atomic_read(&ct->dynticks, sizeof(ct->dynticks)); // instrumentation for the noinstr rcu_dynticks_eqs_exit() - instrument_atomic_write(&rdp->dynticks, sizeof(rdp->dynticks)); + instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); =20 incby =3D 1; } else if (!in_nmi()) { @@ -998,7 +996,7 @@ noinstr void rcu_nmi_enter(void) =20 trace_rcu_dyntick(incby =3D=3D 1 ? TPS("Endirq") : TPS("++=3D"), rdp->dynticks_nmi_nesting, - rdp->dynticks_nmi_nesting + incby, atomic_read(&rdp->dynticks)); + rdp->dynticks_nmi_nesting + incby, ct_dynticks()); instrumentation_end(); WRITE_ONCE(rdp->dynticks_nmi_nesting, /* Prevent store tearing. */ rdp->dynticks_nmi_nesting + incby); @@ -1142,7 +1140,7 @@ static void rcu_gpnum_ovf(struct rcu_node *rnp, struc= t rcu_data *rdp) */ static int dyntick_save_progress_counter(struct rcu_data *rdp) { - rdp->dynticks_snap =3D rcu_dynticks_snap(rdp); + rdp->dynticks_snap =3D rcu_dynticks_snap(rdp->cpu); if (rcu_dynticks_in_eqs(rdp->dynticks_snap)) { trace_rcu_fqs(rcu_state.name, rdp->gp_seq, rdp->cpu, TPS("dti")); rcu_gpnum_ovf(rdp->mynode, rdp); @@ -4237,7 +4235,7 @@ rcu_boot_init_percpu_data(int cpu) rdp->grpmask =3D leaf_node_cpu_bit(rdp->mynode, cpu); INIT_WORK(&rdp->strict_work, strict_work_handler); WARN_ON_ONCE(rdp->dynticks_nesting !=3D 1); - WARN_ON_ONCE(rcu_dynticks_in_eqs(rcu_dynticks_snap(rdp))); + WARN_ON_ONCE(rcu_dynticks_in_eqs(rcu_dynticks_snap(cpu))); rdp->barrier_seq_snap =3D rcu_state.barrier_sequence; rdp->rcu_ofl_gp_seq =3D rcu_state.gp_seq; rdp->rcu_ofl_gp_flags =3D RCU_GP_CLEANED; diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 996387962de3..55a6b2191d26 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -193,7 +193,6 @@ struct rcu_data { int dynticks_snap; /* Per-GP tracking for dynticks. */ long dynticks_nesting; /* Track process nesting level. */ long dynticks_nmi_nesting; /* Track irq/NMI nesting level. */ - atomic_t dynticks; /* Even value for idle, else odd. */ bool rcu_need_heavy_qs; /* GP old, so heavy quiescent state! */ bool rcu_urgent_qs; /* GP old need light quiescent state. */ bool rcu_forced_tick; /* Forced tick to provide QS. */ diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index f05a15b11fa0..0374bc15ae99 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -358,7 +358,7 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_= exp_work *rewp) !(rnp->qsmaskinitnext & mask)) { mask_ofl_test |=3D mask; } else { - snap =3D rcu_dynticks_snap(rdp); + snap =3D rcu_dynticks_snap(cpu); if (rcu_dynticks_in_eqs(snap)) mask_ofl_test |=3D mask; else diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 0a25a4ea6eef..05f5d7e820d0 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -463,7 +463,7 @@ static void print_cpu_stall_info(int cpu) } delta =3D rcu_seq_ctr(rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq); falsepositive =3D rcu_is_gp_kthread_starving(NULL) && - rcu_dynticks_in_eqs(rcu_dynticks_snap(rdp)); + rcu_dynticks_in_eqs(rcu_dynticks_snap(cpu)); rcuc_starved =3D rcu_is_rcuc_kthread_starving(rdp, &j); if (rcuc_starved) sprintf(buf, " rcuc=3D%ld jiffies(starved)", j); @@ -476,7 +476,7 @@ static void print_cpu_stall_info(int cpu) rdp->rcu_iw_pending ? (int)min(delta, 9UL) + '0' : "!."[!delta], ticks_value, ticks_title, - rcu_dynticks_snap(rdp) & 0xfff, + rcu_dynticks_snap(cpu) & 0xfff, rdp->dynticks_nesting, rdp->dynticks_nmi_nesting, rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EB4FC433EF for ; Tue, 3 May 2022 10:03:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229679AbiECKHL (ORCPT ); Tue, 3 May 2022 06:07:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234141AbiECKF5 (ORCPT ); Tue, 3 May 2022 06:05:57 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1939437A3E for ; Tue, 3 May 2022 03:01:58 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7444C61515 for ; Tue, 3 May 2022 10:01:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C4B22C385B1; Tue, 3 May 2022 10:01:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572116; bh=bwSxfE6puOK96r1ZbGP+6NPwUfRacKdQUrFeD3cw6R8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Joqu50azQ7DS7k70DtLXaFf+LEqte192foCA+MS6RFeZheh+37zqqCqaLaji7N9KM bbt78dFKGEgUy+w3Mj2tt0ZucCwV3SbMmMDa4yh6Bg3s4u8phpKxwPmAFzVPUuM1dT Hv8E5Lb+Se+1ZTp1WnVpGxyWLCzaUJDwEpAyfhICEy6NTJLe8ned7AWzPdga4eR/X9 8GYbqzROzWz5fdb9gedgx5S1AUxC88s+PolUnMtv4R+lCpG4E/lPXxqVcLO3YvlhS2 RLRulQLMHUuPmHreS4b8//LWMqd+yi31d6RVJP6Lf2Kw6nt26JVnSV37TghiHoYt1g IXVaeh+pgUltg== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 15/21] rcu/context_tracking: Move dynticks_nesting to context tracking Date: Tue, 3 May 2022 12:00:45 +0200 Message-Id: <20220503100051.2799723-16-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The RCU eqs tracking is going to be performed by the context tracking subsystem. The related nesting counters thus need to be moved to the context tracking structure. Acked-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/context_tracking_state.h | 12 ++++++++++ kernel/context_tracking.c | 1 + kernel/rcu/tree.c | 31 +++++++++++++------------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_stall.h | 2 +- 5 files changed, 30 insertions(+), 17 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context= _tracking_state.h index 014dc431521b..cd3a09fb0aea 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -24,6 +24,7 @@ struct context_tracking { } state; #endif atomic_t dynticks; /* Even value for idle, else odd. */ + long dynticks_nesting; /* Track process nesting level. */ }; =20 #ifdef CONFIG_CONTEXT_TRACKING @@ -45,6 +46,17 @@ static __always_inline int ct_dynticks_cpu_acquire(int c= pu) struct context_tracking *ct =3D per_cpu_ptr(&context_tracking, cpu); return atomic_read_acquire(&ct->state); } + +static __always_inline long ct_dynticks_nesting(void) +{ + return __this_cpu_read(context_tracking.dynticks_nesting); +} + +static __always_inline long ct_dynticks_nesting_cpu(int cpu) +{ + struct context_tracking *ct =3D per_cpu_ptr(&context_tracking, cpu); + return ct->dynticks_nesting; +} #endif =20 #ifdef CONFIG_CONTEXT_TRACKING_USER diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 20be30c24723..ae86fab4eab5 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -250,6 +250,7 @@ void __init context_tracking_init(void) #endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */ =20 DEFINE_PER_CPU(struct context_tracking, context_tracking) =3D { + .dynticks_nesting =3D 1, .dynticks =3D ATOMIC_INIT(1), }; EXPORT_SYMBOL_GPL(context_tracking); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 7667186731e3..826946b39943 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -75,7 +75,6 @@ /* Data structures. */ =20 static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) =3D { - .dynticks_nesting =3D 1, .dynticks_nmi_nesting =3D DYNTICK_IRQ_NONIDLE, #ifdef CONFIG_RCU_NOCB_CPU .cblist.flags =3D SEGCBLIST_RCU_CORE, @@ -436,7 +435,7 @@ static int rcu_is_cpu_rrupt_from_idle(void) lockdep_assert_irqs_disabled(); =20 /* Check for counter underflows */ - RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0, + RCU_LOCKDEP_WARN(ct_dynticks_nesting() < 0, "RCU dynticks_nesting counter underflow!"); RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) <=3D 0, "RCU dynticks_nmi_nesting counter underflow/zero!"); @@ -452,7 +451,7 @@ static int rcu_is_cpu_rrupt_from_idle(void) WARN_ON_ONCE(!nesting && !is_idle_task(current)); =20 /* Does CPU appear to be idle from an RCU standpoint? */ - return __this_cpu_read(rcu_data.dynticks_nesting) =3D=3D 0; + return ct_dynticks_nesting() =3D=3D 0; } =20 #define DEFAULT_RCU_BLIMIT (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) ? 1= 000 : 10) @@ -619,16 +618,16 @@ static noinstr void rcu_eqs_enter(bool user) WARN_ON_ONCE(rdp->dynticks_nmi_nesting !=3D DYNTICK_IRQ_NONIDLE); WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && - rdp->dynticks_nesting =3D=3D 0); - if (rdp->dynticks_nesting !=3D 1) { + ct_dynticks_nesting() =3D=3D 0); + if (ct_dynticks_nesting() !=3D 1) { // RCU will still be watching, so just do accounting and leave. - rdp->dynticks_nesting--; + ct->dynticks_nesting--; return; } =20 lockdep_assert_irqs_disabled(); instrumentation_begin(); - trace_rcu_dyntick(TPS("Start"), rdp->dynticks_nesting, 0, ct_dynticks()); + trace_rcu_dyntick(TPS("Start"), ct_dynticks_nesting(), 0, ct_dynticks()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); rcu_preempt_deferred_qs(current); =20 @@ -636,7 +635,7 @@ static noinstr void rcu_eqs_enter(bool user) instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); =20 instrumentation_end(); - WRITE_ONCE(rdp->dynticks_nesting, 0); /* Avoid irq-access tearing. */ + WRITE_ONCE(ct->dynticks_nesting, 0); /* Avoid irq-access tearing. */ // RCU is watching here ... rcu_dynticks_eqs_enter(); // ... but is no longer watching here. @@ -793,7 +792,7 @@ void rcu_irq_exit_check_preempt(void) { lockdep_assert_irqs_disabled(); =20 - RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) <=3D 0, + RCU_LOCKDEP_WARN(ct_dynticks_nesting() <=3D 0, "RCU dynticks_nesting counter underflow/zero!"); RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) !=3D DYNTICK_IRQ_NONIDLE, @@ -819,11 +818,11 @@ static void noinstr rcu_eqs_exit(bool user) =20 lockdep_assert_irqs_disabled(); rdp =3D this_cpu_ptr(&rcu_data); - oldval =3D rdp->dynticks_nesting; + oldval =3D ct_dynticks_nesting(); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && oldval < 0); if (oldval) { // RCU was already watching, so just do accounting and leave. - rdp->dynticks_nesting++; + ct->dynticks_nesting++; return; } rcu_dynticks_task_exit(); @@ -835,9 +834,9 @@ static void noinstr rcu_eqs_exit(bool user) // instrumentation for the noinstr rcu_dynticks_eqs_exit() instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); =20 - trace_rcu_dyntick(TPS("End"), rdp->dynticks_nesting, 1, ct_dynticks()); + trace_rcu_dyntick(TPS("End"), ct_dynticks_nesting(), 1, ct_dynticks()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); - WRITE_ONCE(rdp->dynticks_nesting, 1); + WRITE_ONCE(ct->dynticks_nesting, 1); WARN_ON_ONCE(rdp->dynticks_nmi_nesting); WRITE_ONCE(rdp->dynticks_nmi_nesting, DYNTICK_IRQ_NONIDLE); instrumentation_end(); @@ -4229,12 +4228,13 @@ static void rcu_init_new_rnp(struct rcu_node *rnp_l= eaf) static void __init rcu_boot_init_percpu_data(int cpu) { + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); =20 /* Set up local state, ensuring consistent view of global state. */ rdp->grpmask =3D leaf_node_cpu_bit(rdp->mynode, cpu); INIT_WORK(&rdp->strict_work, strict_work_handler); - WARN_ON_ONCE(rdp->dynticks_nesting !=3D 1); + WARN_ON_ONCE(ct->dynticks_nesting !=3D 1); WARN_ON_ONCE(rcu_dynticks_in_eqs(rcu_dynticks_snap(cpu))); rdp->barrier_seq_snap =3D rcu_state.barrier_sequence; rdp->rcu_ofl_gp_seq =3D rcu_state.gp_seq; @@ -4259,6 +4259,7 @@ rcu_boot_init_percpu_data(int cpu) int rcutree_prepare_cpu(unsigned int cpu) { unsigned long flags; + struct context_tracking *ct =3D per_cpu_ptr(&context_tracking, cpu); struct rcu_data *rdp =3D per_cpu_ptr(&rcu_data, cpu); struct rcu_node *rnp =3D rcu_get_root(); =20 @@ -4267,7 +4268,7 @@ int rcutree_prepare_cpu(unsigned int cpu) rdp->qlen_last_fqs_check =3D 0; rdp->n_force_qs_snap =3D READ_ONCE(rcu_state.n_force_qs); rdp->blimit =3D blimit; - rdp->dynticks_nesting =3D 1; /* CPU not up, no tearing. */ + ct->dynticks_nesting =3D 1; /* CPU not up, no tearing. */ raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */ =20 /* diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 55a6b2191d26..26400b511c5b 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -191,7 +191,6 @@ struct rcu_data { =20 /* 3) dynticks interface. */ int dynticks_snap; /* Per-GP tracking for dynticks. */ - long dynticks_nesting; /* Track process nesting level. */ long dynticks_nmi_nesting; /* Track irq/NMI nesting level. */ bool rcu_need_heavy_qs; /* GP old, so heavy quiescent state! */ bool rcu_urgent_qs; /* GP old need light quiescent state. */ diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 05f5d7e820d0..0235c03c8642 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -477,7 +477,7 @@ static void print_cpu_stall_info(int cpu) "!."[!delta], ticks_value, ticks_title, rcu_dynticks_snap(cpu) & 0xfff, - rdp->dynticks_nesting, rdp->dynticks_nmi_nesting, + ct_dynticks_nesting_cpu(cpu), rdp->dynticks_nmi_nesting, rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, rcuc_starved ? buf : "", --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C00BC433EF for ; Tue, 3 May 2022 10:03:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231876AbiECKHC (ORCPT ); Tue, 3 May 2022 06:07:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234088AbiECKGL (ORCPT ); Tue, 3 May 2022 06:06:11 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 761FD393C7 for ; Tue, 3 May 2022 03:02:01 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id F3E3160EB1 for ; Tue, 3 May 2022 10:02:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5320AC385B3; Tue, 3 May 2022 10:01:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572120; bh=IjWhJv0cZJOyDiXpsKZ3xyRacemptGlTDpAeI5xPX5s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b6SAaws+zO6LiHHUMGE/WjOvTHdj+GYnSfrCnXekq+2xW2JzPpnrRUCYj5F8vrPvm 8eF8CQ0exk6T5Px9VmTW0lugx/7GxvE7bQNX9ef1vNYnwJvvwjDr3g2EAlqrJNchmd U5sMjz85J6tQ3f+cWWMEHqfSL+twp5Or+hXFnv6I1hPI00M43htsdQMKrMzWcIbul0 lszR1UonQO6wjKqgxQSIBE1myE5xi1iPV2jMX8JHgfrh7Jc0niTIBsFjedO+djrZN4 A60+bF1wBhhuOd1zutfd7hK2wNGWyakfQqzgPZQz7z0DPs/cY9QgPXV548wiiyBLgk k+uR0C4tstAgg== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 16/21] rcu/context_tracking: Move dynticks_nmi_nesting to context tracking Date: Tue, 3 May 2022 12:00:46 +0200 Message-Id: <20220503100051.2799723-17-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The RCU eqs tracking is going to be performed by the context tracking subsystem. The related nesting counters thus need to be moved to the context tracking structure. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/context_tracking_state.h | 15 ++++++++ kernel/context_tracking.c | 1 + kernel/rcu/rcu.h | 4 --- kernel/rcu/tree.c | 48 +++++++++++--------------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_stall.h | 2 +- 6 files changed, 38 insertions(+), 33 deletions(-) diff --git a/include/linux/context_tracking_state.h b/include/linux/context= _tracking_state.h index cd3a09fb0aea..ded257130987 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -6,6 +6,9 @@ #include #include =20 +/* Offset to allow distinguishing irq vs. task-based idle entry/exit. */ +#define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) + struct context_tracking { #ifdef CONFIG_CONTEXT_TRACKING_USER /* @@ -25,6 +28,7 @@ struct context_tracking { #endif atomic_t dynticks; /* Even value for idle, else odd. */ long dynticks_nesting; /* Track process nesting level. */ + long dynticks_nmi_nesting; /* Track irq/NMI nesting level. */ }; =20 #ifdef CONFIG_CONTEXT_TRACKING @@ -57,6 +61,17 @@ static __always_inline long ct_dynticks_nesting_cpu(int = cpu) struct context_tracking *ct =3D per_cpu_ptr(&context_tracking, cpu); return ct->dynticks_nesting; } + +static __always_inline long ct_dynticks_nmi_nesting(void) +{ + return __this_cpu_read(context_tracking.dynticks_nmi_nesting); +} + +static __always_inline long ct_dynticks_nmi_nesting_cpu(int cpu) +{ + struct context_tracking *ct =3D per_cpu_ptr(&context_tracking, cpu); + return ct->dynticks_nmi_nesting; +} #endif =20 #ifdef CONFIG_CONTEXT_TRACKING_USER diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index ae86fab4eab5..14f04c521573 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -250,6 +250,7 @@ void __init context_tracking_init(void) #endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */ =20 DEFINE_PER_CPU(struct context_tracking, context_tracking) =3D { + .dynticks_nmi_nesting =3D DYNTICK_IRQ_NONIDLE, .dynticks_nesting =3D 1, .dynticks =3D ATOMIC_INIT(1), }; diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 15b96f990774..f86584fe90ba 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -12,10 +12,6 @@ =20 #include =20 -/* Offset to allow distinguishing irq vs. task-based idle entry/exit. */ -#define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) - - /* * Grace-period counter management. */ diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 826946b39943..7b1404db3c17 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -75,7 +75,6 @@ /* Data structures. */ =20 static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) =3D { - .dynticks_nmi_nesting =3D DYNTICK_IRQ_NONIDLE, #ifdef CONFIG_RCU_NOCB_CPU .cblist.flags =3D SEGCBLIST_RCU_CORE, #endif @@ -437,11 +436,11 @@ static int rcu_is_cpu_rrupt_from_idle(void) /* Check for counter underflows */ RCU_LOCKDEP_WARN(ct_dynticks_nesting() < 0, "RCU dynticks_nesting counter underflow!"); - RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) <=3D 0, + RCU_LOCKDEP_WARN(ct_dynticks_nmi_nesting() <=3D 0, "RCU dynticks_nmi_nesting counter underflow/zero!"); =20 /* Are we at first interrupt nesting level? */ - nesting =3D __this_cpu_read(rcu_data.dynticks_nmi_nesting); + nesting =3D ct_dynticks_nmi_nesting(); if (nesting > 1) return false; =20 @@ -612,11 +611,10 @@ EXPORT_SYMBOL_GPL(rcutorture_get_gp_data); */ static noinstr void rcu_eqs_enter(bool user) { - struct rcu_data *rdp =3D this_cpu_ptr(&rcu_data); struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); =20 - WARN_ON_ONCE(rdp->dynticks_nmi_nesting !=3D DYNTICK_IRQ_NONIDLE); - WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); + WARN_ON_ONCE(ct_dynticks_nmi_nesting() !=3D DYNTICK_IRQ_NONIDLE); + WRITE_ONCE(ct->dynticks_nmi_nesting, 0); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && ct_dynticks_nesting() =3D=3D 0); if (ct_dynticks_nesting() !=3D 1) { @@ -734,7 +732,7 @@ noinstr void rcu_user_enter(void) * rcu_nmi_exit - inform RCU of exit from NMI context * * If we are returning from the outermost NMI handler that interrupted an - * RCU-idle period, update ct->dynticks and rdp->dynticks_nmi_nesting + * RCU-idle period, update ct->dynticks and ct->dynticks_nmi_nesting * to let the RCU grace-period handling know that the CPU is back to * being RCU-idle. * @@ -744,7 +742,6 @@ noinstr void rcu_user_enter(void) noinstr void rcu_nmi_exit(void) { struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); - struct rcu_data *rdp =3D this_cpu_ptr(&rcu_data); =20 instrumentation_begin(); /* @@ -752,25 +749,25 @@ noinstr void rcu_nmi_exit(void) * (We are exiting an NMI handler, so RCU better be paying attention * to us!) */ - WARN_ON_ONCE(rdp->dynticks_nmi_nesting <=3D 0); + WARN_ON_ONCE(ct_dynticks_nmi_nesting() <=3D 0); WARN_ON_ONCE(rcu_dynticks_curr_cpu_in_eqs()); =20 /* * If the nesting level is not 1, the CPU wasn't RCU-idle, so * leave it in non-RCU-idle state. */ - if (rdp->dynticks_nmi_nesting !=3D 1) { - trace_rcu_dyntick(TPS("--=3D"), rdp->dynticks_nmi_nesting, rdp->dynticks= _nmi_nesting - 2, + if (ct_dynticks_nmi_nesting() !=3D 1) { + trace_rcu_dyntick(TPS("--=3D"), ct_dynticks_nmi_nesting(), ct_dynticks_n= mi_nesting() - 2, ct_dynticks()); - WRITE_ONCE(rdp->dynticks_nmi_nesting, /* No store tearing. */ - rdp->dynticks_nmi_nesting - 2); + WRITE_ONCE(ct->dynticks_nmi_nesting, /* No store tearing. */ + ct_dynticks_nmi_nesting() - 2); instrumentation_end(); return; } =20 /* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */ - trace_rcu_dyntick(TPS("Startirq"), rdp->dynticks_nmi_nesting, 0, ct_dynti= cks()); - WRITE_ONCE(rdp->dynticks_nmi_nesting, 0); /* Avoid store tearing. */ + trace_rcu_dyntick(TPS("Startirq"), ct_dynticks_nmi_nesting(), 0, ct_dynti= cks()); + WRITE_ONCE(ct->dynticks_nmi_nesting, 0); /* Avoid store tearing. */ =20 // instrumentation for the noinstr rcu_dynticks_eqs_enter() instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); @@ -794,7 +791,7 @@ void rcu_irq_exit_check_preempt(void) =20 RCU_LOCKDEP_WARN(ct_dynticks_nesting() <=3D 0, "RCU dynticks_nesting counter underflow/zero!"); - RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) !=3D + RCU_LOCKDEP_WARN(ct_dynticks_nmi_nesting() !=3D DYNTICK_IRQ_NONIDLE, "Bad RCU dynticks_nmi_nesting counter\n"); RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(), @@ -813,11 +810,9 @@ void rcu_irq_exit_check_preempt(void) static void noinstr rcu_eqs_exit(bool user) { struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); - struct rcu_data *rdp; long oldval; =20 lockdep_assert_irqs_disabled(); - rdp =3D this_cpu_ptr(&rcu_data); oldval =3D ct_dynticks_nesting(); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && oldval < 0); if (oldval) { @@ -837,8 +832,8 @@ static void noinstr rcu_eqs_exit(bool user) trace_rcu_dyntick(TPS("End"), ct_dynticks_nesting(), 1, ct_dynticks()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); WRITE_ONCE(ct->dynticks_nesting, 1); - WARN_ON_ONCE(rdp->dynticks_nmi_nesting); - WRITE_ONCE(rdp->dynticks_nmi_nesting, DYNTICK_IRQ_NONIDLE); + WARN_ON_ONCE(ct_dynticks_nmi_nesting()); + WRITE_ONCE(ct->dynticks_nmi_nesting, DYNTICK_IRQ_NONIDLE); instrumentation_end(); } =20 @@ -945,7 +940,7 @@ void __rcu_irq_enter_check_tick(void) * rcu_nmi_enter - inform RCU of entry to NMI context * * If the CPU was idle from RCU's viewpoint, update ct->dynticks and - * rdp->dynticks_nmi_nesting to let the RCU grace-period handling know + * ct->dynticks_nmi_nesting to let the RCU grace-period handling know * that the CPU is active. This implementation permits nested NMIs, as * long as the nesting level does not overflow an int. (You will probably * run out of stack space first.) @@ -956,11 +951,10 @@ void __rcu_irq_enter_check_tick(void) noinstr void rcu_nmi_enter(void) { long incby =3D 2; - struct rcu_data *rdp =3D this_cpu_ptr(&rcu_data); struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); =20 /* Complain about underflow. */ - WARN_ON_ONCE(rdp->dynticks_nmi_nesting < 0); + WARN_ON_ONCE(ct_dynticks_nmi_nesting() < 0); =20 /* * If idle from RCU viewpoint, atomically increment ->dynticks @@ -994,11 +988,11 @@ noinstr void rcu_nmi_enter(void) } =20 trace_rcu_dyntick(incby =3D=3D 1 ? TPS("Endirq") : TPS("++=3D"), - rdp->dynticks_nmi_nesting, - rdp->dynticks_nmi_nesting + incby, ct_dynticks()); + ct_dynticks_nmi_nesting(), + ct_dynticks_nmi_nesting() + incby, ct_dynticks()); instrumentation_end(); - WRITE_ONCE(rdp->dynticks_nmi_nesting, /* Prevent store tearing. */ - rdp->dynticks_nmi_nesting + incby); + WRITE_ONCE(ct->dynticks_nmi_nesting, /* Prevent store tearing. */ + ct_dynticks_nmi_nesting() + incby); barrier(); } =20 diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 26400b511c5b..11f3e0581782 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -191,7 +191,6 @@ struct rcu_data { =20 /* 3) dynticks interface. */ int dynticks_snap; /* Per-GP tracking for dynticks. */ - long dynticks_nmi_nesting; /* Track irq/NMI nesting level. */ bool rcu_need_heavy_qs; /* GP old, so heavy quiescent state! */ bool rcu_urgent_qs; /* GP old need light quiescent state. */ bool rcu_forced_tick; /* Forced tick to provide QS. */ diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 0235c03c8642..933445d826f1 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -477,7 +477,7 @@ static void print_cpu_stall_info(int cpu) "!."[!delta], ticks_value, ticks_title, rcu_dynticks_snap(cpu) & 0xfff, - ct_dynticks_nesting_cpu(cpu), rdp->dynticks_nmi_nesting, + ct_dynticks_nesting_cpu(cpu), ct_dynticks_nmi_nesting_cpu(cpu), rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, rcuc_starved ? buf : "", --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFB55C433EF for ; Tue, 3 May 2022 10:03:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234041AbiECKGx (ORCPT ); Tue, 3 May 2022 06:06:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53380 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234099AbiECKGM (ORCPT ); Tue, 3 May 2022 06:06:12 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0A8739699 for ; Tue, 3 May 2022 03:02:04 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8A4216150F for ; Tue, 3 May 2022 10:02:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D7FE6C385B2; Tue, 3 May 2022 10:02:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572124; bh=GUwBGopfqbKnbt7MNtk8uSBv/T4L7ECAAqia+WZIoIY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WQGXipaNRRbarkWzrRKldD4cVd3gKFGollLWsYALUhXlAyPiBEV6vUzeYKMU0rbPj qqTIITSIMTfJJrpYGEI7xpTWNXmfvXF2+/iRz7oGPVzCfu2V1p8b5AO/0KfZHivtmu PLFNheI5vJOC8uHAE4sm5rZN+HFvBlJFZPQyX0SR9Ff/C+4TqiqUDz4zDlyvUdkzov /ObgHq59HZuCmOtTViuLZXz1+FlhIOh5ZqWaBY8xYd6VUACcto5O4xVZCeiLDdG4eT Aeom6O2w5H5EGEnJoMpoVfhyacpms3gy3jSCxL5IAq8RAlxSj7PJG/KC/f8GGIX8/5 NB6PVQ2eeXb/w== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 17/21] rcu/context-tracking: Move deferred nocb resched to context tracking Date: Tue, 3 May 2022 12:00:47 +0200 Message-Id: <20220503100051.2799723-18-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" To prepare for migrating the RCU eqs accounting code to context tracking, split the last-resort deferred nocb resched from rcu_user_enter() and move it into a separate call from context tracking. Acked-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/rcutree.h | 6 ++++++ kernel/context_tracking.c | 8 ++++++++ kernel/rcu/tree.c | 15 ++------------- 3 files changed, 16 insertions(+), 13 deletions(-) diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index cafe3fbf4272..6294271f1615 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -76,4 +76,10 @@ int rcutree_dead_cpu(unsigned int cpu); int rcutree_dying_cpu(unsigned int cpu); void rcu_cpu_starting(unsigned int cpu); =20 +#if defined(CONFIG_NO_HZ_FULL) && (!defined(CONFIG_GENERIC_ENTRY) || !defi= ned(CONFIG_KVM_XFER_TO_GUEST_WORK)) +void rcu_irq_work_resched(void); +#else +static inline void rcu_irq_work_resched(void) { } +#endif + #endif /* __LINUX_RCUTREE_H */ diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 14f04c521573..4ca4aeadad06 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -60,6 +60,8 @@ static __always_inline void context_tracking_recursion_ex= it(void) */ void noinstr __ct_user_enter(enum ctx_state state) { + lockdep_assert_irqs_disabled(); + /* Kernel threads aren't supposed to go to userspace */ WARN_ON_ONCE(!current->mm); =20 @@ -81,6 +83,12 @@ void noinstr __ct_user_enter(enum ctx_state state) vtime_user_enter(current); instrumentation_end(); } + /* + * Other than generic entry implementation, we may be past the last + * rescheduling opportunity in the entry code. Trigger a self IPI + * that will fire and reschedule once we resume in user/guest mode. + */ + rcu_irq_work_resched(); rcu_user_enter(); } /* diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 7b1404db3c17..7d1a3e73ff25 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -681,7 +681,7 @@ static DEFINE_PER_CPU(struct irq_work, late_wakeup_work= ) =3D * last resort is to fire a local irq_work that will trigger a reschedule = once IRQs * get re-enabled again. */ -noinstr static void rcu_irq_work_resched(void) +noinstr void rcu_irq_work_resched(void) { struct rcu_data *rdp =3D this_cpu_ptr(&rcu_data); =20 @@ -697,10 +697,7 @@ noinstr static void rcu_irq_work_resched(void) } instrumentation_end(); } - -#else -static inline void rcu_irq_work_resched(void) { } -#endif +#endif /* #if !defined(CONFIG_GENERIC_ENTRY) || !defined(CONFIG_KVM_XFER_T= O_GUEST_WORK) */ =20 /** * rcu_user_enter - inform RCU that we are resuming userspace. @@ -715,14 +712,6 @@ static inline void rcu_irq_work_resched(void) { } */ noinstr void rcu_user_enter(void) { - lockdep_assert_irqs_disabled(); - - /* - * Other than generic entry implementation, we may be past the last - * rescheduling opportunity in the entry code. Trigger a self IPI - * that will fire and reschedule once we resume in user/guest mode. - */ - rcu_irq_work_resched(); rcu_eqs_enter(true); } =20 --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC36FC433F5 for ; Tue, 3 May 2022 10:03:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234251AbiECKG3 (ORCPT ); Tue, 3 May 2022 06:06:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234185AbiECKGN (ORCPT ); Tue, 3 May 2022 06:06:13 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EB223969F for ; Tue, 3 May 2022 03:02:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 341CBCE1E7C for ; Tue, 3 May 2022 10:02:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6A5A4C385A4; Tue, 3 May 2022 10:02:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572127; bh=0EUmFww5drZ15WTYnaK1XZTHt8CmyZosJNalWFJK5X0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jKBnw+lBOSd9P7mHihw65e5wBRmwiANlo79rgfwoMckjfCSgZD9WwA0l065ya9Onp Serq/iTjjM1pAQJPYI6OyDEDi+GKdM99Hv6sxkmp88v4WOWaO2CPsUlkRUCQ1r0wPO 5FVLdt7r2l1/xUg7QpFsY7FaHUq9eGO1k/7kRGSXD4ws6t2Af/ZmP+2P73rhrSxMnv g4xUiv/NUCYd/L/nvx47xDcsAxHmUUbxNxVIhdST1bCkdgcApoSsbWoy1DceN9YkGP FtvACdfqsgN2rwD1HXSwakCU/6NosEPAByfrXWVjBZ+G6nmOc+QtEywJilx1oOowto wqPlS6C2EEwLQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 18/21] rcu/context-tracking: Move RCU-dynticks internal functions to context_tracking Date: Tue, 3 May 2022 12:00:48 +0200 Message-Id: <20220503100051.2799723-19-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Move the core RCU eqs/dynticks functions to context tracking so that we can later merge all that code within context tracking. Acked-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/context_tracking.h | 20 ++ include/linux/rcutree.h | 3 + kernel/context_tracking.c | 342 +++++++++++++++++++++++++++++++ kernel/rcu/tree.c | 328 +---------------------------- kernel/rcu/tree.h | 5 - kernel/rcu/tree_plugin.h | 38 +--- 6 files changed, 370 insertions(+), 366 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_track= ing.h index 27afb75f2650..cad8aecb7321 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -122,6 +122,26 @@ static inline void context_tracking_init(void) { } #ifdef CONFIG_CONTEXT_TRACKING extern void ct_idle_enter(void); extern void ct_idle_exit(void); + +/* + * Is the current CPU in an extended quiescent state? + * + * No ordering, as we are sampling CPU-local information. + */ +static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void) +{ + return !(arch_atomic_read(this_cpu_ptr(&context_tracking.dynticks)) & 0x1= ); +} + +/* + * Increment the current CPU's context_tracking structure's ->dynticks fie= ld + * with ordering. Return the new value. + */ +static __always_inline unsigned long rcu_dynticks_inc(int incby) +{ + return arch_atomic_add_return(incby, this_cpu_ptr(&context_tracking.dynti= cks)); +} + #else static inline void ct_idle_enter(void) { } static inline void ct_idle_exit(void) { } diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index 6294271f1615..d9e8e6233c47 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -57,6 +57,9 @@ void rcu_irq_exit_check_preempt(void); static inline void rcu_irq_exit_check_preempt(void) { } #endif =20 +struct task_struct; +void rcu_preempt_deferred_qs(struct task_struct *t); + void exit_rcu(void); =20 void rcu_scheduler_starting(void); diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 4ca4aeadad06..279fb5875dd8 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -21,6 +21,348 @@ #include #include #include +#include + +#define TPS(x) tracepoint_string(x) + +/* Record the current task on dyntick-idle entry. */ +static __always_inline void rcu_dynticks_task_enter(void) +{ +#if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) + WRITE_ONCE(current->rcu_tasks_idle_cpu, smp_processor_id()); +#endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ +} + +/* Record no current task on dyntick-idle exit. */ +static __always_inline void rcu_dynticks_task_exit(void) +{ +#if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) + WRITE_ONCE(current->rcu_tasks_idle_cpu, -1); +#endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ +} + +/* Turn on heavyweight RCU tasks trace readers on idle/user entry. */ +static __always_inline void rcu_dynticks_task_trace_enter(void) +{ +#ifdef CONFIG_TASKS_TRACE_RCU + if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) + current->trc_reader_special.b.need_mb =3D true; +#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ +} + +/* Turn off heavyweight RCU tasks trace readers on idle/user exit. */ +static __always_inline void rcu_dynticks_task_trace_exit(void) +{ +#ifdef CONFIG_TASKS_TRACE_RCU + if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) + current->trc_reader_special.b.need_mb =3D false; +#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ +} + +/* + * Record entry into an extended quiescent state. This is only to be + * called when not already in an extended quiescent state, that is, + * RCU is watching prior to the call to this function and is no longer + * watching upon return. + */ +static noinstr void rcu_dynticks_eqs_enter(void) +{ + int seq; + + /* + * CPUs seeing atomic_add_return() must see prior RCU read-side + * critical sections, and we also must force ordering with the + * next idle sojourn. + */ + rcu_dynticks_task_trace_enter(); // Before ->dynticks update! + seq =3D rcu_dynticks_inc(1); + // RCU is no longer watching. Better be in extended quiescent state! + WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && (seq & 0x1)); +} + +/* + * Record exit from an extended quiescent state. This is only to be + * called from an extended quiescent state, that is, RCU is not watching + * prior to the call to this function and is watching upon return. + */ +static noinstr void rcu_dynticks_eqs_exit(void) +{ + int seq; + + /* + * CPUs seeing atomic_add_return() must see prior idle sojourns, + * and we also must force ordering with the next RCU read-side + * critical section. + */ + seq =3D rcu_dynticks_inc(1); + // RCU is now watching. Better not be in an extended quiescent state! + rcu_dynticks_task_trace_exit(); // After ->dynticks update! + WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & 0x1)); +} + +/* + * Enter an RCU extended quiescent state, which can be either the + * idle loop or adaptive-tickless usermode execution. + * + * We crowbar the ->dynticks_nmi_nesting field to zero to allow for + * the possibility of usermode upcalls having messed up our count + * of interrupt nesting level during the prior busy period. + */ +static noinstr void rcu_eqs_enter(bool user) +{ + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); + + WARN_ON_ONCE(ct_dynticks_nmi_nesting() !=3D DYNTICK_IRQ_NONIDLE); + WRITE_ONCE(ct->dynticks_nmi_nesting, 0); + WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && + ct_dynticks_nesting() =3D=3D 0); + if (ct_dynticks_nesting() !=3D 1) { + // RCU will still be watching, so just do accounting and leave. + ct->dynticks_nesting--; + return; + } + + lockdep_assert_irqs_disabled(); + instrumentation_begin(); + trace_rcu_dyntick(TPS("Start"), ct_dynticks_nesting(), 0, ct_dynticks()); + WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); + rcu_preempt_deferred_qs(current); + + // instrumentation for the noinstr rcu_dynticks_eqs_enter() + instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); + + instrumentation_end(); + WRITE_ONCE(ct->dynticks_nesting, 0); /* Avoid irq-access tearing. */ + // RCU is watching here ... + rcu_dynticks_eqs_enter(); + // ... but is no longer watching here. + rcu_dynticks_task_enter(); +} + +/** + * rcu_idle_enter - inform RCU that current CPU is entering idle + * + * Enter idle mode, in other words, -leave- the mode in which RCU + * read-side critical sections can occur. (Though RCU read-side + * critical sections can occur in irq handlers in idle, a possibility + * handled by irq_enter() and irq_exit().) + * + * If you add or remove a call to rcu_idle_enter(), be sure to test with + * CONFIG_RCU_EQS_DEBUG=3Dy. + */ +noinstr void rcu_idle_enter(void) +{ + lockdep_assert_irqs_disabled(); + rcu_eqs_enter(false); +} + +#ifdef CONFIG_NO_HZ_FULL +/** + * rcu_user_enter - inform RCU that we are resuming userspace. + * + * Enter RCU idle mode right before resuming userspace. No use of RCU + * is permitted between this call and rcu_user_exit(). This way the + * CPU doesn't need to maintain the tick for RCU maintenance purposes + * when the CPU runs in userspace. + * + * If you add or remove a call to rcu_user_enter(), be sure to test with + * CONFIG_RCU_EQS_DEBUG=3Dy. + */ +noinstr void rcu_user_enter(void) +{ + rcu_eqs_enter(true); +} +#endif /* CONFIG_NO_HZ_FULL */ + +/** + * rcu_nmi_exit - inform RCU of exit from NMI context + * + * If we are returning from the outermost NMI handler that interrupted an + * RCU-idle period, update ct->dynticks and ct->dynticks_nmi_nesting + * to let the RCU grace-period handling know that the CPU is back to + * being RCU-idle. + * + * If you add or remove a call to rcu_nmi_exit(), be sure to test + * with CONFIG_RCU_EQS_DEBUG=3Dy. + */ +noinstr void rcu_nmi_exit(void) +{ + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); + + instrumentation_begin(); + /* + * Check for ->dynticks_nmi_nesting underflow and bad ->dynticks. + * (We are exiting an NMI handler, so RCU better be paying attention + * to us!) + */ + WARN_ON_ONCE(ct_dynticks_nmi_nesting() <=3D 0); + WARN_ON_ONCE(rcu_dynticks_curr_cpu_in_eqs()); + + /* + * If the nesting level is not 1, the CPU wasn't RCU-idle, so + * leave it in non-RCU-idle state. + */ + if (ct_dynticks_nmi_nesting() !=3D 1) { + trace_rcu_dyntick(TPS("--=3D"), ct_dynticks_nmi_nesting(), ct_dynticks_n= mi_nesting() - 2, + ct_dynticks()); + WRITE_ONCE(ct->dynticks_nmi_nesting, /* No store tearing. */ + ct_dynticks_nmi_nesting() - 2); + instrumentation_end(); + return; + } + + /* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */ + trace_rcu_dyntick(TPS("Startirq"), ct_dynticks_nmi_nesting(), 0, ct_dynti= cks()); + WRITE_ONCE(ct->dynticks_nmi_nesting, 0); /* Avoid store tearing. */ + + // instrumentation for the noinstr rcu_dynticks_eqs_enter() + instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); + instrumentation_end(); + + // RCU is watching here ... + rcu_dynticks_eqs_enter(); + // ... but is no longer watching here. + + if (!in_nmi()) + rcu_dynticks_task_enter(); +} + +/* + * Exit an RCU extended quiescent state, which can be either the + * idle loop or adaptive-tickless usermode execution. + * + * We crowbar the ->dynticks_nmi_nesting field to DYNTICK_IRQ_NONIDLE to + * allow for the possibility of usermode upcalls messing up our count of + * interrupt nesting level during the busy period that is just now startin= g. + */ +static void noinstr rcu_eqs_exit(bool user) +{ + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); + long oldval; + + lockdep_assert_irqs_disabled(); + oldval =3D ct_dynticks_nesting(); + WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && oldval < 0); + if (oldval) { + // RCU was already watching, so just do accounting and leave. + ct->dynticks_nesting++; + return; + } + rcu_dynticks_task_exit(); + // RCU is not watching here ... + rcu_dynticks_eqs_exit(); + // ... but is watching here. + instrumentation_begin(); + + // instrumentation for the noinstr rcu_dynticks_eqs_exit() + instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); + + trace_rcu_dyntick(TPS("End"), ct_dynticks_nesting(), 1, ct_dynticks()); + WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); + WRITE_ONCE(ct->dynticks_nesting, 1); + WARN_ON_ONCE(ct_dynticks_nmi_nesting()); + WRITE_ONCE(ct->dynticks_nmi_nesting, DYNTICK_IRQ_NONIDLE); + instrumentation_end(); +} + +/** + * rcu_idle_exit - inform RCU that current CPU is leaving idle + * + * Exit idle mode, in other words, -enter- the mode in which RCU + * read-side critical sections can occur. + * + * If you add or remove a call to rcu_idle_exit(), be sure to test with + * CONFIG_RCU_EQS_DEBUG=3Dy. + * + * FIXME: This function should be noinstr but the below local_irq_save() is + * unsafe because it involves illegal RCU uses through tracing and lockdep. + * This must be fixed first. + */ +void rcu_idle_exit(void) +{ + unsigned long flags; + + local_irq_save(flags); + rcu_eqs_exit(false); + local_irq_restore(flags); +} +EXPORT_SYMBOL_GPL(rcu_idle_exit); + +#ifdef CONFIG_NO_HZ_FULL +/** + * rcu_user_exit - inform RCU that we are exiting userspace. + * + * Exit RCU idle mode while entering the kernel because it can + * run a RCU read side critical section anytime. + * + * If you add or remove a call to rcu_user_exit(), be sure to test with + * CONFIG_RCU_EQS_DEBUG=3Dy. + */ +void noinstr rcu_user_exit(void) +{ + rcu_eqs_exit(true); +} +#endif /* ifdef CONFIG_NO_HZ_FULL */ + +/** + * rcu_nmi_enter - inform RCU of entry to NMI context + * + * If the CPU was idle from RCU's viewpoint, update ct->dynticks and + * ct->dynticks_nmi_nesting to let the RCU grace-period handling know + * that the CPU is active. This implementation permits nested NMIs, as + * long as the nesting level does not overflow an int. (You will probably + * run out of stack space first.) + * + * If you add or remove a call to rcu_nmi_enter(), be sure to test + * with CONFIG_RCU_EQS_DEBUG=3Dy. + */ +noinstr void rcu_nmi_enter(void) +{ + long incby =3D 2; + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); + + /* Complain about underflow. */ + WARN_ON_ONCE(ct_dynticks_nmi_nesting() < 0); + + /* + * If idle from RCU viewpoint, atomically increment ->dynticks + * to mark non-idle and increment ->dynticks_nmi_nesting by one. + * Otherwise, increment ->dynticks_nmi_nesting by two. This means + * if ->dynticks_nmi_nesting is equal to one, we are guaranteed + * to be in the outermost NMI handler that interrupted an RCU-idle + * period (observation due to Andy Lutomirski). + */ + if (rcu_dynticks_curr_cpu_in_eqs()) { + + if (!in_nmi()) + rcu_dynticks_task_exit(); + + // RCU is not watching here ... + rcu_dynticks_eqs_exit(); + // ... but is watching here. + + instrumentation_begin(); + // instrumentation for the noinstr rcu_dynticks_curr_cpu_in_eqs() + instrument_atomic_read(&ct->dynticks, sizeof(ct->dynticks)); + // instrumentation for the noinstr rcu_dynticks_eqs_exit() + instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); + + incby =3D 1; + } else if (!in_nmi()) { + instrumentation_begin(); + rcu_irq_enter_check_tick(); + } else { + instrumentation_begin(); + } + + trace_rcu_dyntick(incby =3D=3D 1 ? TPS("Endirq") : TPS("++=3D"), + ct_dynticks_nmi_nesting(), + ct_dynticks_nmi_nesting() + incby, ct_dynticks()); + instrumentation_end(); + WRITE_ONCE(ct->dynticks_nmi_nesting, /* Prevent store tearing. */ + ct_dynticks_nmi_nesting() + incby); + barrier(); +} =20 #ifdef CONFIG_CONTEXT_TRACKING_USER =20 diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 7d1a3e73ff25..16d601512da7 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -62,6 +62,7 @@ #include #include #include +#include #include "../time/tick-internal.h" =20 #include "tree.h" @@ -259,56 +260,6 @@ void rcu_softirq_qs(void) rcu_tasks_qs(current, false); } =20 -/* - * Increment the current CPU's rcu_data structure's ->dynticks field - * with ordering. Return the new value. - */ -static noinline noinstr unsigned long rcu_dynticks_inc(int incby) -{ - return arch_atomic_add_return(incby, this_cpu_ptr(&context_tracking.dynti= cks)); -} - -/* - * Record entry into an extended quiescent state. This is only to be - * called when not already in an extended quiescent state, that is, - * RCU is watching prior to the call to this function and is no longer - * watching upon return. - */ -static noinstr void rcu_dynticks_eqs_enter(void) -{ - int seq; - - /* - * CPUs seeing atomic_add_return() must see prior RCU read-side - * critical sections, and we also must force ordering with the - * next idle sojourn. - */ - rcu_dynticks_task_trace_enter(); // Before ->dynticks update! - seq =3D rcu_dynticks_inc(1); - // RCU is no longer watching. Better be in extended quiescent state! - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && (seq & 0x1)); -} - -/* - * Record exit from an extended quiescent state. This is only to be - * called from an extended quiescent state, that is, RCU is not watching - * prior to the call to this function and is watching upon return. - */ -static noinstr void rcu_dynticks_eqs_exit(void) -{ - int seq; - - /* - * CPUs seeing atomic_add_return() must see prior idle sojourns, - * and we also must force ordering with the next RCU read-side - * critical section. - */ - seq =3D rcu_dynticks_inc(1); - // RCU is now watching. Better not be in an extended quiescent state! - rcu_dynticks_task_trace_exit(); // After ->dynticks update! - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & 0x1)); -} - /* * Reset the current CPU's ->dynticks counter to indicate that the * newly onlined CPU is no longer in an extended quiescent state. @@ -326,16 +277,6 @@ static void rcu_dynticks_eqs_online(void) rcu_dynticks_inc(1); } =20 -/* - * Is the current CPU in an extended quiescent state? - * - * No ordering, as we are sampling CPU-local information. - */ -static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void) -{ - return !(arch_atomic_read(this_cpu_ptr(&context_tracking.dynticks)) & 0x1= ); -} - /* * Snapshot the ->dynticks counter with full ordering so as to allow * stable comparison of this counter with past and future snapshots. @@ -601,65 +542,7 @@ void rcutorture_get_gp_data(enum rcutorture_type test_= type, int *flags, } EXPORT_SYMBOL_GPL(rcutorture_get_gp_data); =20 -/* - * Enter an RCU extended quiescent state, which can be either the - * idle loop or adaptive-tickless usermode execution. - * - * We crowbar the ->dynticks_nmi_nesting field to zero to allow for - * the possibility of usermode upcalls having messed up our count - * of interrupt nesting level during the prior busy period. - */ -static noinstr void rcu_eqs_enter(bool user) -{ - struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); - - WARN_ON_ONCE(ct_dynticks_nmi_nesting() !=3D DYNTICK_IRQ_NONIDLE); - WRITE_ONCE(ct->dynticks_nmi_nesting, 0); - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && - ct_dynticks_nesting() =3D=3D 0); - if (ct_dynticks_nesting() !=3D 1) { - // RCU will still be watching, so just do accounting and leave. - ct->dynticks_nesting--; - return; - } - - lockdep_assert_irqs_disabled(); - instrumentation_begin(); - trace_rcu_dyntick(TPS("Start"), ct_dynticks_nesting(), 0, ct_dynticks()); - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); - rcu_preempt_deferred_qs(current); - - // instrumentation for the noinstr rcu_dynticks_eqs_enter() - instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); - - instrumentation_end(); - WRITE_ONCE(ct->dynticks_nesting, 0); /* Avoid irq-access tearing. */ - // RCU is watching here ... - rcu_dynticks_eqs_enter(); - // ... but is no longer watching here. - rcu_dynticks_task_enter(); -} - -/** - * rcu_idle_enter - inform RCU that current CPU is entering idle - * - * Enter idle mode, in other words, -leave- the mode in which RCU - * read-side critical sections can occur. (Though RCU read-side - * critical sections can occur in irq handlers in idle, a possibility - * handled by irq_enter() and irq_exit().) - * - * If you add or remove a call to rcu_idle_enter(), be sure to test with - * CONFIG_RCU_EQS_DEBUG=3Dy. - */ -noinstr void rcu_idle_enter(void) -{ - lockdep_assert_irqs_disabled(); - rcu_eqs_enter(false); -} - -#ifdef CONFIG_NO_HZ_FULL - -#if !defined(CONFIG_GENERIC_ENTRY) || !defined(CONFIG_KVM_XFER_TO_GUEST_WO= RK) +#if defined(CONFIG_NO_HZ_FULL) && (!defined(CONFIG_GENERIC_ENTRY) || !defi= ned(CONFIG_KVM_XFER_TO_GUEST_WORK)) /* * An empty function that will trigger a reschedule on * IRQ tail once IRQs get re-enabled on userspace/guest resume. @@ -697,78 +580,7 @@ noinstr void rcu_irq_work_resched(void) } instrumentation_end(); } -#endif /* #if !defined(CONFIG_GENERIC_ENTRY) || !defined(CONFIG_KVM_XFER_T= O_GUEST_WORK) */ - -/** - * rcu_user_enter - inform RCU that we are resuming userspace. - * - * Enter RCU idle mode right before resuming userspace. No use of RCU - * is permitted between this call and rcu_user_exit(). This way the - * CPU doesn't need to maintain the tick for RCU maintenance purposes - * when the CPU runs in userspace. - * - * If you add or remove a call to rcu_user_enter(), be sure to test with - * CONFIG_RCU_EQS_DEBUG=3Dy. - */ -noinstr void rcu_user_enter(void) -{ - rcu_eqs_enter(true); -} - -#endif /* CONFIG_NO_HZ_FULL */ - -/** - * rcu_nmi_exit - inform RCU of exit from NMI context - * - * If we are returning from the outermost NMI handler that interrupted an - * RCU-idle period, update ct->dynticks and ct->dynticks_nmi_nesting - * to let the RCU grace-period handling know that the CPU is back to - * being RCU-idle. - * - * If you add or remove a call to rcu_nmi_exit(), be sure to test - * with CONFIG_RCU_EQS_DEBUG=3Dy. - */ -noinstr void rcu_nmi_exit(void) -{ - struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); - - instrumentation_begin(); - /* - * Check for ->dynticks_nmi_nesting underflow and bad ->dynticks. - * (We are exiting an NMI handler, so RCU better be paying attention - * to us!) - */ - WARN_ON_ONCE(ct_dynticks_nmi_nesting() <=3D 0); - WARN_ON_ONCE(rcu_dynticks_curr_cpu_in_eqs()); - - /* - * If the nesting level is not 1, the CPU wasn't RCU-idle, so - * leave it in non-RCU-idle state. - */ - if (ct_dynticks_nmi_nesting() !=3D 1) { - trace_rcu_dyntick(TPS("--=3D"), ct_dynticks_nmi_nesting(), ct_dynticks_n= mi_nesting() - 2, - ct_dynticks()); - WRITE_ONCE(ct->dynticks_nmi_nesting, /* No store tearing. */ - ct_dynticks_nmi_nesting() - 2); - instrumentation_end(); - return; - } - - /* This NMI interrupted an RCU-idle CPU, restore RCU-idleness. */ - trace_rcu_dyntick(TPS("Startirq"), ct_dynticks_nmi_nesting(), 0, ct_dynti= cks()); - WRITE_ONCE(ct->dynticks_nmi_nesting, 0); /* Avoid store tearing. */ - - // instrumentation for the noinstr rcu_dynticks_eqs_enter() - instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); - instrumentation_end(); - - // RCU is watching here ... - rcu_dynticks_eqs_enter(); - // ... but is no longer watching here. - - if (!in_nmi()) - rcu_dynticks_task_enter(); -} +#endif /* #if defined(CONFIG_NO_HZ_FULL) && (!defined(CONFIG_GENERIC_ENTRY= ) || !defined(CONFIG_KVM_XFER_TO_GUEST_WORK)) */ =20 #ifdef CONFIG_PROVE_RCU /** @@ -788,81 +600,7 @@ void rcu_irq_exit_check_preempt(void) } #endif /* #ifdef CONFIG_PROVE_RCU */ =20 -/* - * Exit an RCU extended quiescent state, which can be either the - * idle loop or adaptive-tickless usermode execution. - * - * We crowbar the ->dynticks_nmi_nesting field to DYNTICK_IRQ_NONIDLE to - * allow for the possibility of usermode upcalls messing up our count of - * interrupt nesting level during the busy period that is just now startin= g. - */ -static void noinstr rcu_eqs_exit(bool user) -{ - struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); - long oldval; - - lockdep_assert_irqs_disabled(); - oldval =3D ct_dynticks_nesting(); - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && oldval < 0); - if (oldval) { - // RCU was already watching, so just do accounting and leave. - ct->dynticks_nesting++; - return; - } - rcu_dynticks_task_exit(); - // RCU is not watching here ... - rcu_dynticks_eqs_exit(); - // ... but is watching here. - instrumentation_begin(); - - // instrumentation for the noinstr rcu_dynticks_eqs_exit() - instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); - - trace_rcu_dyntick(TPS("End"), ct_dynticks_nesting(), 1, ct_dynticks()); - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); - WRITE_ONCE(ct->dynticks_nesting, 1); - WARN_ON_ONCE(ct_dynticks_nmi_nesting()); - WRITE_ONCE(ct->dynticks_nmi_nesting, DYNTICK_IRQ_NONIDLE); - instrumentation_end(); -} - -/** - * rcu_idle_exit - inform RCU that current CPU is leaving idle - * - * Exit idle mode, in other words, -enter- the mode in which RCU - * read-side critical sections can occur. - * - * If you add or remove a call to rcu_idle_exit(), be sure to test with - * CONFIG_RCU_EQS_DEBUG=3Dy. - * - * FIXME: This function should be noinstr but the below local_irq_save() is - * unsafe because it involves illegal RCU uses through tracing and lockdep. - * This must be fixed first. - */ -void rcu_idle_exit(void) -{ - unsigned long flags; - - local_irq_save(flags); - rcu_eqs_exit(false); - local_irq_restore(flags); -} - #ifdef CONFIG_NO_HZ_FULL -/** - * rcu_user_exit - inform RCU that we are exiting userspace. - * - * Exit RCU idle mode while entering the kernel because it can - * run a RCU read side critical section anytime. - * - * If you add or remove a call to rcu_user_exit(), be sure to test with - * CONFIG_RCU_EQS_DEBUG=3Dy. - */ -void noinstr rcu_user_exit(void) -{ - rcu_eqs_exit(true); -} - /** * __rcu_irq_enter_check_tick - Enable scheduler tick on CPU if RCU needs = it. * @@ -925,66 +663,6 @@ void __rcu_irq_enter_check_tick(void) } #endif /* CONFIG_NO_HZ_FULL */ =20 -/** - * rcu_nmi_enter - inform RCU of entry to NMI context - * - * If the CPU was idle from RCU's viewpoint, update ct->dynticks and - * ct->dynticks_nmi_nesting to let the RCU grace-period handling know - * that the CPU is active. This implementation permits nested NMIs, as - * long as the nesting level does not overflow an int. (You will probably - * run out of stack space first.) - * - * If you add or remove a call to rcu_nmi_enter(), be sure to test - * with CONFIG_RCU_EQS_DEBUG=3Dy. - */ -noinstr void rcu_nmi_enter(void) -{ - long incby =3D 2; - struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); - - /* Complain about underflow. */ - WARN_ON_ONCE(ct_dynticks_nmi_nesting() < 0); - - /* - * If idle from RCU viewpoint, atomically increment ->dynticks - * to mark non-idle and increment ->dynticks_nmi_nesting by one. - * Otherwise, increment ->dynticks_nmi_nesting by two. This means - * if ->dynticks_nmi_nesting is equal to one, we are guaranteed - * to be in the outermost NMI handler that interrupted an RCU-idle - * period (observation due to Andy Lutomirski). - */ - if (rcu_dynticks_curr_cpu_in_eqs()) { - - if (!in_nmi()) - rcu_dynticks_task_exit(); - - // RCU is not watching here ... - rcu_dynticks_eqs_exit(); - // ... but is watching here. - - instrumentation_begin(); - // instrumentation for the noinstr rcu_dynticks_curr_cpu_in_eqs() - instrument_atomic_read(&ct->dynticks, sizeof(ct->dynticks)); - // instrumentation for the noinstr rcu_dynticks_eqs_exit() - instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); - - incby =3D 1; - } else if (!in_nmi()) { - instrumentation_begin(); - rcu_irq_enter_check_tick(); - } else { - instrumentation_begin(); - } - - trace_rcu_dyntick(incby =3D=3D 1 ? TPS("Endirq") : TPS("++=3D"), - ct_dynticks_nmi_nesting(), - ct_dynticks_nmi_nesting() + incby, ct_dynticks()); - instrumentation_end(); - WRITE_ONCE(ct->dynticks_nmi_nesting, /* Prevent store tearing. */ - ct_dynticks_nmi_nesting() + incby); - barrier(); -} - /* * Check to see if any future non-offloaded RCU-related work will need * to be done by the current CPU, even if none need be done immediately, diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 11f3e0581782..4ed7f1bdbb0f 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -435,7 +435,6 @@ static void rcu_cpu_kthread_setup(unsigned int cpu); static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp); static bool rcu_preempt_has_tasks(struct rcu_node *rnp); static bool rcu_preempt_need_deferred_qs(struct task_struct *t); -static void rcu_preempt_deferred_qs(struct task_struct *t); static void zero_cpu_stall_ticks(struct rcu_data *rdp); static struct swait_queue_head *rcu_nocb_gp_get(struct rcu_node *rnp); static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); @@ -475,10 +474,6 @@ do { \ =20 static void rcu_bind_gp_kthread(void); static bool rcu_nohz_full_cpu(void); -static void rcu_dynticks_task_enter(void); -static void rcu_dynticks_task_exit(void); -static void rcu_dynticks_task_trace_enter(void); -static void rcu_dynticks_task_trace_exit(void); =20 /* Forward declarations for tree_stall.h */ static void record_gp_stall_check_time(void); diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 440d9e02a26e..1bb1f571285d 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -595,7 +595,7 @@ static notrace bool rcu_preempt_need_deferred_qs(struct= task_struct *t) * evaluate safety in terms of interrupt, softirq, and preemption * disabling. */ -static notrace void rcu_preempt_deferred_qs(struct task_struct *t) +notrace void rcu_preempt_deferred_qs(struct task_struct *t) { unsigned long flags; =20 @@ -935,7 +935,7 @@ static bool rcu_preempt_need_deferred_qs(struct task_st= ruct *t) // period for a quiescent state from this CPU. Note that requests from // tasks are handled when removing the task from the blocked-tasks list // below. -static void rcu_preempt_deferred_qs(struct task_struct *t) +void rcu_preempt_deferred_qs(struct task_struct *t) { struct rcu_data *rdp =3D this_cpu_ptr(&rcu_data); =20 @@ -1290,37 +1290,3 @@ static void rcu_bind_gp_kthread(void) return; housekeeping_affine(current, HK_TYPE_RCU); } - -/* Record the current task on dyntick-idle entry. */ -static __always_inline void rcu_dynticks_task_enter(void) -{ -#if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) - WRITE_ONCE(current->rcu_tasks_idle_cpu, smp_processor_id()); -#endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ -} - -/* Record no current task on dyntick-idle exit. */ -static __always_inline void rcu_dynticks_task_exit(void) -{ -#if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) - WRITE_ONCE(current->rcu_tasks_idle_cpu, -1); -#endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ -} - -/* Turn on heavyweight RCU tasks trace readers on idle/user entry. */ -static __always_inline void rcu_dynticks_task_trace_enter(void) -{ -#ifdef CONFIG_TASKS_TRACE_RCU - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) - current->trc_reader_special.b.need_mb =3D true; -#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ -} - -/* Turn off heavyweight RCU tasks trace readers on idle/user exit. */ -static __always_inline void rcu_dynticks_task_trace_exit(void) -{ -#ifdef CONFIG_TASKS_TRACE_RCU - if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) - current->trc_reader_special.b.need_mb =3D false; -#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ -} --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6259C433FE for ; Tue, 3 May 2022 10:03:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234036AbiECKGe (ORCPT ); Tue, 3 May 2022 06:06:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234214AbiECKGP (ORCPT ); Tue, 3 May 2022 06:06:15 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1ECD3983C for ; Tue, 3 May 2022 03:02:13 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 411D1B81C96 for ; Tue, 3 May 2022 10:02:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EAD87C385BB; Tue, 3 May 2022 10:02:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572131; bh=twsdKeg1BiJ0kaSJbwYhbKiCBWwghWtn3Tq0Oq38oPw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hC8zC0ILZ0FEDCkkbrFrR4anpLA3qA3tLP1ED7SIUiy87VCfUb7LdRneQbdXT2P0w j7KfvIHkSI0X/IyutcTqZ6x9sonaXvkxBroSkioXQnKG+3GjzuKjPGaiOShAWOePay QOUXanU/+8wAUS3u/6w+54a5DUrjmbkSdjAh/e0Tf8fqt0Qh9LNP+3WdB6b4AAzW83 Wc8d4QIEQtAonR+3OBQTiMck4oMUGaeroD5XtXSEjFqKIldJ888J7KUMiCUFYnEbjk KIGAgFw2Y7vTJ9R9WSXDtfcgf5Jj8mK6VWIXFrcd9uKIiO6Yz/a3phHCpPHk7hdwC3 LrkWLwQi/tMXw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 19/21] rcu/context-tracking: Remove unused and/or unecessary middle functions Date: Tue, 3 May 2022 12:00:49 +0200 Message-Id: <20220503100051.2799723-20-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Some eqs functions are now only used internally by context tracking, so their public declarations can be removed. Also middle functions such as rcu_user_*() and rcu_idle_*() which now directly call to rcu_eqs_enter() and rcu_eqs_exit() can be wiped out as well. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- Documentation/RCU/stallwarn.rst | 2 +- include/linux/hardirq.h | 8 -- include/linux/rcupdate.h | 8 -- include/linux/rcutiny.h | 2 - include/linux/rcutree.h | 2 - kernel/context_tracking.c | 147 +++++++++++--------------------- 6 files changed, 50 insertions(+), 119 deletions(-) diff --git a/Documentation/RCU/stallwarn.rst b/Documentation/RCU/stallwarn.= rst index 7858c3afa1f4..bebcaef475a6 100644 --- a/Documentation/RCU/stallwarn.rst +++ b/Documentation/RCU/stallwarn.rst @@ -97,7 +97,7 @@ warnings: which will include additional debugging information. =20 - A low-level kernel issue that either fails to invoke one of the - variants of rcu_user_enter(), rcu_user_exit(), ct_idle_enter(), + variants of rcu_eqs_enter(true), rcu_eqs_exit(true), ct_idle_enter(), ct_idle_exit(), ct_irq_enter(), or ct_irq_exit() on the one hand, or that invokes one of them too many times on the other. Historically, the most frequent issue has been an omission diff --git a/include/linux/hardirq.h b/include/linux/hardirq.h index 345cdbe9c1b7..d57cab4d4c06 100644 --- a/include/linux/hardirq.h +++ b/include/linux/hardirq.h @@ -92,14 +92,6 @@ void irq_exit_rcu(void); #define arch_nmi_exit() do { } while (0) #endif =20 -#ifdef CONFIG_TINY_RCU -static inline void rcu_nmi_enter(void) { } -static inline void rcu_nmi_exit(void) { } -#else -extern void rcu_nmi_enter(void); -extern void rcu_nmi_exit(void); -#endif - /* * NMI vs Tracing * -------------- diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 923e4fa04338..79ea1a45adf0 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -105,14 +105,6 @@ static inline void rcu_sysrq_start(void) { } static inline void rcu_sysrq_end(void) { } #endif /* #else #ifdef CONFIG_RCU_STALL_COMMON */ =20 -#ifdef CONFIG_NO_HZ_FULL -void rcu_user_enter(void); -void rcu_user_exit(void); -#else -static inline void rcu_user_enter(void) { } -static inline void rcu_user_exit(void) { } -#endif /* CONFIG_NO_HZ_FULL */ - #ifdef CONFIG_RCU_NOCB_CPU void rcu_init_nohz(void); int rcu_nocb_cpu_offload(int cpu); diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h index 5921d9ae7ab5..430aac8a7344 100644 --- a/include/linux/rcutiny.h +++ b/include/linux/rcutiny.h @@ -86,8 +86,6 @@ static inline int rcu_needs_cpu(void) static inline void rcu_virt_note_context_switch(int cpu) { } static inline void rcu_cpu_stall_reset(void) { } static inline int rcu_jiffies_till_stall_check(void) { return 21 * HZ; } -static inline void rcu_idle_enter(void) { } -static inline void rcu_idle_exit(void) { } static inline void rcu_irq_exit_check_preempt(void) { } #define rcu_is_idle_cpu(cpu) \ (is_idle_task(current) && !in_nmi() && !in_hardirq() && !in_serving_softi= rq()) diff --git a/include/linux/rcutree.h b/include/linux/rcutree.h index d9e8e6233c47..72131447762e 100644 --- a/include/linux/rcutree.h +++ b/include/linux/rcutree.h @@ -47,8 +47,6 @@ unsigned long start_poll_synchronize_rcu(void); bool poll_state_synchronize_rcu(unsigned long oldstate); void cond_synchronize_rcu(unsigned long oldstate); =20 -void rcu_idle_enter(void); -void rcu_idle_exit(void); bool rcu_is_idle_cpu(int cpu); =20 #ifdef CONFIG_PROVE_RCU diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 279fb5875dd8..b906fb55903b 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -140,52 +140,17 @@ static noinstr void rcu_eqs_enter(bool user) } =20 /** - * rcu_idle_enter - inform RCU that current CPU is entering idle - * - * Enter idle mode, in other words, -leave- the mode in which RCU - * read-side critical sections can occur. (Though RCU read-side - * critical sections can occur in irq handlers in idle, a possibility - * handled by irq_enter() and irq_exit().) - * - * If you add or remove a call to rcu_idle_enter(), be sure to test with - * CONFIG_RCU_EQS_DEBUG=3Dy. - */ -noinstr void rcu_idle_enter(void) -{ - lockdep_assert_irqs_disabled(); - rcu_eqs_enter(false); -} - -#ifdef CONFIG_NO_HZ_FULL -/** - * rcu_user_enter - inform RCU that we are resuming userspace. - * - * Enter RCU idle mode right before resuming userspace. No use of RCU - * is permitted between this call and rcu_user_exit(). This way the - * CPU doesn't need to maintain the tick for RCU maintenance purposes - * when the CPU runs in userspace. - * - * If you add or remove a call to rcu_user_enter(), be sure to test with - * CONFIG_RCU_EQS_DEBUG=3Dy. - */ -noinstr void rcu_user_enter(void) -{ - rcu_eqs_enter(true); -} -#endif /* CONFIG_NO_HZ_FULL */ - -/** - * rcu_nmi_exit - inform RCU of exit from NMI context + * ct_nmi_exit - inform RCU of exit from NMI context * * If we are returning from the outermost NMI handler that interrupted an * RCU-idle period, update ct->dynticks and ct->dynticks_nmi_nesting * to let the RCU grace-period handling know that the CPU is back to * being RCU-idle. * - * If you add or remove a call to rcu_nmi_exit(), be sure to test + * If you add or remove a call to ct_nmi_exit(), be sure to test * with CONFIG_RCU_EQS_DEBUG=3Dy. */ -noinstr void rcu_nmi_exit(void) +noinstr void ct_nmi_exit(void) { struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); =20 @@ -266,46 +231,7 @@ static void noinstr rcu_eqs_exit(bool user) } =20 /** - * rcu_idle_exit - inform RCU that current CPU is leaving idle - * - * Exit idle mode, in other words, -enter- the mode in which RCU - * read-side critical sections can occur. - * - * If you add or remove a call to rcu_idle_exit(), be sure to test with - * CONFIG_RCU_EQS_DEBUG=3Dy. - * - * FIXME: This function should be noinstr but the below local_irq_save() is - * unsafe because it involves illegal RCU uses through tracing and lockdep. - * This must be fixed first. - */ -void rcu_idle_exit(void) -{ - unsigned long flags; - - local_irq_save(flags); - rcu_eqs_exit(false); - local_irq_restore(flags); -} -EXPORT_SYMBOL_GPL(rcu_idle_exit); - -#ifdef CONFIG_NO_HZ_FULL -/** - * rcu_user_exit - inform RCU that we are exiting userspace. - * - * Exit RCU idle mode while entering the kernel because it can - * run a RCU read side critical section anytime. - * - * If you add or remove a call to rcu_user_exit(), be sure to test with - * CONFIG_RCU_EQS_DEBUG=3Dy. - */ -void noinstr rcu_user_exit(void) -{ - rcu_eqs_exit(true); -} -#endif /* ifdef CONFIG_NO_HZ_FULL */ - -/** - * rcu_nmi_enter - inform RCU of entry to NMI context + * ct_nmi_enter - inform RCU of entry to NMI context * * If the CPU was idle from RCU's viewpoint, update ct->dynticks and * ct->dynticks_nmi_nesting to let the RCU grace-period handling know @@ -313,10 +239,10 @@ void noinstr rcu_user_exit(void) * long as the nesting level does not overflow an int. (You will probably * run out of stack space first.) * - * If you add or remove a call to rcu_nmi_enter(), be sure to test + * If you add or remove a call to ct_nmi_enter(), be sure to test * with CONFIG_RCU_EQS_DEBUG=3Dy. */ -noinstr void rcu_nmi_enter(void) +noinstr void ct_nmi_enter(void) { long incby =3D 2; struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); @@ -431,7 +357,13 @@ void noinstr __ct_user_enter(enum ctx_state state) * that will fire and reschedule once we resume in user/guest mode. */ rcu_irq_work_resched(); - rcu_user_enter(); + /* + * Enter RCU idle mode right before resuming userspace. No use of RCU + * is permitted between this call and rcu_eqs_exit(). This way the + * CPU doesn't need to maintain the tick for RCU maintenance purposes + * when the CPU runs in userspace. + */ + rcu_eqs_enter(true); } /* * Even if context tracking is disabled on this CPU, because it's outside @@ -464,7 +396,7 @@ void ct_user_enter(enum ctx_state state) /* * Some contexts may involve an exception occuring in an irq, * leading to that nesting: - * ct_irq_enter() rcu_user_exit() rcu_user_exit() ct_irq_exit() + * ct_irq_enter() rcu_eqs_exit(true) rcu_eqs_enter(true) ct_irq_exit() * This would mess up the dyntick_nesting count though. And rcu_irq_*() * helpers are enough to protect RCU uses inside the exception. So * just return immediately if we detect we are in an IRQ. @@ -513,10 +445,10 @@ void noinstr __ct_user_exit(enum ctx_state state) if (__this_cpu_read(context_tracking.state) =3D=3D state) { if (__this_cpu_read(context_tracking.active)) { /* - * We are going to run code that may use RCU. Inform - * RCU core about that (ie: we may need the tick again). + * Exit RCU idle mode while entering the kernel because it can + * run a RCU read side critical section anytime. */ - rcu_user_exit(); + rcu_eqs_exit(true); if (state =3D=3D CONTEXT_USER) { instrumentation_begin(); vtime_user_exit(current); @@ -606,16 +538,45 @@ DEFINE_PER_CPU(struct context_tracking, context_track= ing) =3D { }; EXPORT_SYMBOL_GPL(context_tracking); =20 - +/** + * ct_idle_enter - inform RCU that current CPU is entering idle + * + * Enter idle mode, in other words, -leave- the mode in which RCU + * read-side critical sections can occur. (Though RCU read-side + * critical sections can occur in irq handlers in idle, a possibility + * handled by irq_enter() and irq_exit().) + * + * If you add or remove a call to ct_idle_enter(), be sure to test with + * CONFIG_RCU_EQS_DEBUG=3Dy. + */ noinstr void ct_idle_enter(void) { - rcu_idle_enter(); + lockdep_assert_irqs_disabled(); + rcu_eqs_enter(false); } EXPORT_SYMBOL_GPL(ct_idle_enter); =20 +/** + * ct_idle_exit - inform RCU that current CPU is leaving idle + * + * Exit idle mode, in other words, -enter- the mode in which RCU + * read-side critical sections can occur. + * + * If you add or remove a call to ct_idle_exit(), be sure to test with + * CONFIG_RCU_EQS_DEBUG=3Dy. + * + * FIXME: This function should be noinstr but the below local_irq_save() is + * unsafe because it involves illegal RCU uses through tracing and lockdep. + * This must be fixed first. + */ void ct_idle_exit(void) { - rcu_idle_exit(); + unsigned long flags; + + local_irq_save(flags); + rcu_eqs_exit(false); + local_irq_restore(flags); + } EXPORT_SYMBOL_GPL(ct_idle_exit); =20 @@ -709,13 +670,3 @@ void ct_irq_exit_irqson(void) ct_irq_exit(); local_irq_restore(flags); } - -noinstr void ct_nmi_enter(void) -{ - rcu_nmi_enter(); -} - -noinstr void ct_nmi_exit(void) -{ - rcu_nmi_exit(); -} --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F356C433FE for ; Tue, 3 May 2022 10:03:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233999AbiECKGr (ORCPT ); Tue, 3 May 2022 06:06:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234232AbiECKGQ (ORCPT ); Tue, 3 May 2022 06:06:16 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DE73396A5 for ; Tue, 3 May 2022 03:02:17 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E1B08B81A67 for ; Tue, 3 May 2022 10:02:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7A0B8C385B2; Tue, 3 May 2022 10:02:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572134; bh=g8ewgdEUW3BVgsPYAoymr2WWvCzaMxjDdepxBcYRDLQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RikaItYA46lI/hRM8m9QydNKuwzOoBIiUPN4jPNiJRGnmyKdfOkJ2uuExcIv2UFQs N7O4BXCJyWKDik8IssVdiAs4Jdx61lsqIIQyRDzDubMo7f0kl4e8Pd2uG1l91p/Xx6 9vsAWd8UvwgFMHjt3m73UoDXiobG/iQg3ms+b62l58ZXIQ9NyLqtRMv/v1ty5L+eSM oeb3WCGn4k9p8dnAxbtmVLkCUHCytajGAY7JuQmP3DmfqIX0akeD7tn7hRxHolwFF+ 9QRAP+POYeFkaK+dspDTRXxwPKcKj9cMq3r+eL4XW6urKcO9w+R8G9fXzX38V8Lkgp gxeLq3TlK0org== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 20/21] context_tracking: Convert state to atomic_t Date: Tue, 3 May 2022 12:00:50 +0200 Message-Id: <20220503100051.2799723-21-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Context tracking's state and dynticks counter are going to be merged in a single field so that both updates can happen atomically and at the same time. Prepare for that with converting the state into an atomic_t. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/context_tracking.h | 18 ++---------- include/linux/context_tracking_state.h | 40 ++++++++++++++++++++++---- kernel/context_tracking.c | 15 ++++++---- 3 files changed, 46 insertions(+), 27 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_track= ing.h index cad8aecb7321..082bf47bbd18 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -56,7 +56,7 @@ static inline enum ctx_state exception_enter(void) !context_tracking_enabled()) return 0; =20 - prev_ctx =3D this_cpu_read(context_tracking.state); + prev_ctx =3D __ct_state(); if (prev_ctx !=3D CONTEXT_KERNEL) ct_user_exit(prev_ctx); =20 @@ -86,26 +86,14 @@ static __always_inline void context_tracking_guest_exit= (void) __ct_user_exit(CONTEXT_GUEST); } =20 -/** - * ct_state() - return the current context tracking state if known - * - * Returns the current cpu's context tracking state if context tracking - * is enabled. If context tracking is disabled, returns - * CONTEXT_DISABLED. This should be used primarily for debugging. - */ -static __always_inline enum ctx_state ct_state(void) -{ - return context_tracking_enabled() ? - this_cpu_read(context_tracking.state) : CONTEXT_DISABLED; -} #else static inline void user_enter(void) { } static inline void user_exit(void) { } static inline void user_enter_irqoff(void) { } static inline void user_exit_irqoff(void) { } -static inline enum ctx_state exception_enter(void) { return 0; } +static inline int exception_enter(void) { return 0; } static inline void exception_exit(enum ctx_state prev_ctx) { } -static inline enum ctx_state ct_state(void) { return CONTEXT_DISABLED; } +static inline int ct_state(void) { return -1; } static __always_inline bool context_tracking_guest_enter(void) { return fa= lse; } static inline void context_tracking_guest_exit(void) { } =20 diff --git a/include/linux/context_tracking_state.h b/include/linux/context= _tracking_state.h index ded257130987..edd9f8f5b753 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -9,6 +9,13 @@ /* Offset to allow distinguishing irq vs. task-based idle entry/exit. */ #define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) =20 +enum ctx_state { + CONTEXT_DISABLED =3D -1, /* returned by ct_state() if unknown */ + CONTEXT_KERNEL =3D 0, + CONTEXT_USER, + CONTEXT_GUEST, +}; + struct context_tracking { #ifdef CONFIG_CONTEXT_TRACKING_USER /* @@ -19,12 +26,7 @@ struct context_tracking { */ bool active; int recursion; - enum ctx_state { - CONTEXT_DISABLED =3D -1, /* returned by ct_state() if unknown */ - CONTEXT_KERNEL =3D 0, - CONTEXT_USER, - CONTEXT_GUEST, - } state; + atomic_t state; #endif atomic_t dynticks; /* Even value for idle, else odd. */ long dynticks_nesting; /* Track process nesting level. */ @@ -34,6 +36,11 @@ struct context_tracking { #ifdef CONFIG_CONTEXT_TRACKING DECLARE_PER_CPU(struct context_tracking, context_tracking); =20 +static __always_inline int __ct_state(void) +{ + return atomic_read(this_cpu_ptr(&context_tracking.state)); +} + static __always_inline int ct_dynticks(void) { return atomic_read(this_cpu_ptr(&context_tracking.dynticks)); @@ -92,6 +99,27 @@ static inline bool context_tracking_enabled_this_cpu(voi= d) return context_tracking_enabled() && __this_cpu_read(context_tracking.act= ive); } =20 +/** + * ct_state() - return the current context tracking state if known + * + * Returns the current cpu's context tracking state if context tracking + * is enabled. If context tracking is disabled, returns + * CONTEXT_DISABLED. This should be used primarily for debugging. + */ +static __always_inline int ct_state(void) +{ + int ret; + + if (!context_tracking_enabled()) + return CONTEXT_DISABLED; + + preempt_disable(); + ret =3D __ct_state(); + preempt_enable(); + + return ret; +} + #else static inline bool context_tracking_enabled(void) { return false; } static inline bool context_tracking_enabled_cpu(int cpu) { return false; } diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index b906fb55903b..acbe96779e8c 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -328,6 +328,7 @@ static __always_inline void context_tracking_recursion_= exit(void) */ void noinstr __ct_user_enter(enum ctx_state state) { + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); lockdep_assert_irqs_disabled(); =20 /* Kernel threads aren't supposed to go to userspace */ @@ -336,8 +337,8 @@ void noinstr __ct_user_enter(enum ctx_state state) if (!context_tracking_recursion_enter()) return; =20 - if ( __this_cpu_read(context_tracking.state) !=3D state) { - if (__this_cpu_read(context_tracking.active)) { + if (__ct_state() !=3D state) { + if (ct->active) { /* * At this stage, only low level arch entry code remains and * then we'll run in userspace. We can assume there won't be @@ -378,7 +379,7 @@ void noinstr __ct_user_enter(enum ctx_state state) * OTOH we can spare the calls to vtime and RCU when context_tracking.ac= tive * is false because we know that CPU is not tickless. */ - __this_cpu_write(context_tracking.state, state); + atomic_set(&ct->state, state); } context_tracking_recursion_exit(); } @@ -439,11 +440,13 @@ NOKPROBE_SYMBOL(user_enter_callable); */ void noinstr __ct_user_exit(enum ctx_state state) { + struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); + if (!context_tracking_recursion_enter()) return; =20 - if (__this_cpu_read(context_tracking.state) =3D=3D state) { - if (__this_cpu_read(context_tracking.active)) { + if (__ct_state() =3D=3D state) { + if (ct->active) { /* * Exit RCU idle mode while entering the kernel because it can * run a RCU read side critical section anytime. @@ -456,7 +459,7 @@ void noinstr __ct_user_exit(enum ctx_state state) instrumentation_end(); } } - __this_cpu_write(context_tracking.state, CONTEXT_KERNEL); + atomic_set(&ct->state, CONTEXT_KERNEL); } context_tracking_recursion_exit(); } --=20 2.25.1 From nobody Sun May 10 14:11:35 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E81C0C433F5 for ; Tue, 3 May 2022 10:03:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234065AbiECKGj (ORCPT ); Tue, 3 May 2022 06:06:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234264AbiECKGR (ORCPT ); Tue, 3 May 2022 06:06:17 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4B1D3A5F3 for ; Tue, 3 May 2022 03:02:21 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id C338ACE09AF for ; Tue, 3 May 2022 10:02:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08259C385B0; Tue, 3 May 2022 10:02:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1651572138; bh=Jcia6O3gdktrxIg42qU22WxLBLJXPgO8D+STnpXEMEc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IhG4VkdtIZ3D5i/wE4p/f3JGzVWtdcBQV/tC6XOgq3IQoTvpbKCRfbi70x44eSsBg ZB/57bYG9LOai7RGyrzeTUf5ufS1+btXdXoiB573nZwIIm2Op8bVGMg0zCoTlRnEKF cB6lIUWefYUvjuJWdvKmHAG50yyaehyQN+UtKuc2YQkSwsFfFoWXAqKTtYh+y+3uHZ TQ++PLSjOzJHqtbVS0cg2N/89tdXe5VIfkKFlE12VYgj0b3iNtp/4v/Kc10AcY3jwV sSfbz1LCInJ9RRxv1CoZiHTHscYGpyOPtJdaUXPUIyhGdgTWagw7Ae7DOBovwkqLXD 4JeL9eLqt27CA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Peter Zijlstra , Phil Auld , Alex Belits , Nicolas Saenz Julienne , Xiongfeng Wang , Neeraj Upadhyay , Thomas Gleixner , Yu Liao , Boqun Feng , "Paul E . McKenney" , Marcelo Tosatti , Paul Gortmaker , Uladzislau Rezki , Joel Fernandes Subject: [PATCH 21/21] rcu/context_tracking: Merge dynticks counter and context tracking states Date: Tue, 3 May 2022 12:00:51 +0200 Message-Id: <20220503100051.2799723-22-frederic@kernel.org> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220503100051.2799723-1-frederic@kernel.org> References: <20220503100051.2799723-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Updating the context tracking state and the RCU dynticks counter atomically in a single operation is a first step towards improving CPU isolation. This makes the context tracking state updates fully ordered and therefore allow for later enhancements such as postponing some work while a task is running isolated in userspace until it ever comes back to the kernel. The state field becomes divided in two parts: 1) Two Lower bits for context tracking state: CONTEXT_KERNEL =3D 0 CONTEXT_IDLE =3D 1, CONTEXT_USER =3D 2, CONTEXT_GUEST =3D 3, 2) Higher bits for RCU eqs dynticks counting: RCU_DYNTICKS_IDX =3D 4 The dynticks counting is always incremented by this value. (state & RCU_DYNTICKS_IDX) means we are NOT in an extended quiescent state. This makes the chance for a collision more likely between two RCU dynticks snapshots but wrapping up 28 bits of eqs dynticks increments still takes some bad luck (also rdp.dynticks_snap could be converted from int to long?) Some RCU eqs functions have been renamed to better reflect their broader scope that now include context tracking state. Signed-off-by: Frederic Weisbecker Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Neeraj Upadhyay Cc: Uladzislau Rezki Cc: Joel Fernandes Cc: Boqun Feng Cc: Nicolas Saenz Julienne Cc: Marcelo Tosatti Cc: Xiongfeng Wang Cc: Yu Liao Cc: Phil Auld Cc: Paul Gortmaker Cc: Alex Belits Reviewed-by: Nicolas Saenz Julienne Tested-by: Nicolas Saenz Julienne --- include/linux/context_tracking.h | 8 +-- include/linux/context_tracking_state.h | 27 +++++--- kernel/context_tracking.c | 88 +++++++++++++------------- kernel/rcu/tree.c | 13 ++-- kernel/rcu/tree_stall.h | 4 +- 5 files changed, 74 insertions(+), 66 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_track= ing.h index 082bf47bbd18..a0c3bf8f3bb1 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -118,16 +118,16 @@ extern void ct_idle_exit(void); */ static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void) { - return !(arch_atomic_read(this_cpu_ptr(&context_tracking.dynticks)) & 0x1= ); + return !(arch_atomic_read(this_cpu_ptr(&context_tracking.state)) & RCU_DY= NTICKS_IDX); } =20 /* - * Increment the current CPU's context_tracking structure's ->dynticks fie= ld + * Increment the current CPU's context_tracking structure's ->state field * with ordering. Return the new value. */ -static __always_inline unsigned long rcu_dynticks_inc(int incby) +static __always_inline unsigned long ct_state_inc(int incby) { - return arch_atomic_add_return(incby, this_cpu_ptr(&context_tracking.dynti= cks)); + return arch_atomic_add_return(incby, this_cpu_ptr(&context_tracking.state= )); } =20 #else diff --git a/include/linux/context_tracking_state.h b/include/linux/context= _tracking_state.h index edd9f8f5b753..0841435620cf 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -10,12 +10,20 @@ #define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) =20 enum ctx_state { - CONTEXT_DISABLED =3D -1, /* returned by ct_state() if unknown */ - CONTEXT_KERNEL =3D 0, - CONTEXT_USER, - CONTEXT_GUEST, + CONTEXT_DISABLED =3D -1, /* returned by ct_state() if unknown */ + CONTEXT_KERNEL =3D 0, + CONTEXT_IDLE =3D 1, + CONTEXT_USER =3D 2, + CONTEXT_GUEST =3D 3, + CONTEXT_MAX =3D 4, }; =20 +/* Even value for idle, else odd. */ +#define RCU_DYNTICKS_IDX CONTEXT_MAX + +#define CT_STATE_MASK (CONTEXT_MAX - 1) +#define CT_DYNTICKS_MASK (~CT_STATE_MASK) + struct context_tracking { #ifdef CONFIG_CONTEXT_TRACKING_USER /* @@ -26,9 +34,8 @@ struct context_tracking { */ bool active; int recursion; +#endif atomic_t state; -#endif - atomic_t dynticks; /* Even value for idle, else odd. */ long dynticks_nesting; /* Track process nesting level. */ long dynticks_nmi_nesting; /* Track irq/NMI nesting level. */ }; @@ -38,24 +45,24 @@ DECLARE_PER_CPU(struct context_tracking, context_tracki= ng); =20 static __always_inline int __ct_state(void) { - return atomic_read(this_cpu_ptr(&context_tracking.state)); + return atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_STATE_MASK; } =20 static __always_inline int ct_dynticks(void) { - return atomic_read(this_cpu_ptr(&context_tracking.dynticks)); + return atomic_read(this_cpu_ptr(&context_tracking.state)) & CT_DYNTICKS_M= ASK; } =20 static __always_inline int ct_dynticks_cpu(int cpu) { struct context_tracking *ct =3D per_cpu_ptr(&context_tracking, cpu); - return atomic_read(&ct->dynticks); + return atomic_read(&ct->state) & CT_DYNTICKS_MASK; } =20 static __always_inline int ct_dynticks_cpu_acquire(int cpu) { struct context_tracking *ct =3D per_cpu_ptr(&context_tracking, cpu); - return atomic_read_acquire(&ct->state); + return atomic_read_acquire(&ct->state) & CT_DYNTICKS_MASK; } =20 static __always_inline long ct_dynticks_nesting(void) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index acbe96779e8c..98341e1f6d1c 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -65,7 +65,7 @@ static __always_inline void rcu_dynticks_task_trace_exit(= void) * RCU is watching prior to the call to this function and is no longer * watching upon return. */ -static noinstr void rcu_dynticks_eqs_enter(void) +static noinstr void ct_kernel_exit_state(int offset) { int seq; =20 @@ -75,9 +75,9 @@ static noinstr void rcu_dynticks_eqs_enter(void) * next idle sojourn. */ rcu_dynticks_task_trace_enter(); // Before ->dynticks update! - seq =3D rcu_dynticks_inc(1); + seq =3D ct_state_inc(offset); // RCU is no longer watching. Better be in extended quiescent state! - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && (seq & 0x1)); + WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && (seq & RCU_DYNTICKS_IDX)= ); } =20 /* @@ -85,7 +85,7 @@ static noinstr void rcu_dynticks_eqs_enter(void) * called from an extended quiescent state, that is, RCU is not watching * prior to the call to this function and is watching upon return. */ -static noinstr void rcu_dynticks_eqs_exit(void) +static noinstr void ct_kernel_enter_state(int offset) { int seq; =20 @@ -94,10 +94,10 @@ static noinstr void rcu_dynticks_eqs_exit(void) * and we also must force ordering with the next RCU read-side * critical section. */ - seq =3D rcu_dynticks_inc(1); + seq =3D ct_state_inc(offset); // RCU is now watching. Better not be in an extended quiescent state! rcu_dynticks_task_trace_exit(); // After ->dynticks update! - WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & 0x1)); + WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & RCU_DYNTICKS_IDX= )); } =20 /* @@ -108,7 +108,7 @@ static noinstr void rcu_dynticks_eqs_exit(void) * the possibility of usermode upcalls having messed up our count * of interrupt nesting level during the prior busy period. */ -static noinstr void rcu_eqs_enter(bool user) +static noinstr void ct_kernel_exit(bool user, int offset) { struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); =20 @@ -128,13 +128,13 @@ static noinstr void rcu_eqs_enter(bool user) WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); rcu_preempt_deferred_qs(current); =20 - // instrumentation for the noinstr rcu_dynticks_eqs_enter() - instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); + // instrumentation for the noinstr ct_kernel_exit_state() + instrument_atomic_write(&ct->state, sizeof(ct->state)); =20 instrumentation_end(); WRITE_ONCE(ct->dynticks_nesting, 0); /* Avoid irq-access tearing. */ // RCU is watching here ... - rcu_dynticks_eqs_enter(); + ct_kernel_exit_state(offset); // ... but is no longer watching here. rcu_dynticks_task_enter(); } @@ -143,7 +143,7 @@ static noinstr void rcu_eqs_enter(bool user) * ct_nmi_exit - inform RCU of exit from NMI context * * If we are returning from the outermost NMI handler that interrupted an - * RCU-idle period, update ct->dynticks and ct->dynticks_nmi_nesting + * RCU-idle period, update ct->state and ct->dynticks_nmi_nesting * to let the RCU grace-period handling know that the CPU is back to * being RCU-idle. * @@ -180,12 +180,12 @@ noinstr void ct_nmi_exit(void) trace_rcu_dyntick(TPS("Startirq"), ct_dynticks_nmi_nesting(), 0, ct_dynti= cks()); WRITE_ONCE(ct->dynticks_nmi_nesting, 0); /* Avoid store tearing. */ =20 - // instrumentation for the noinstr rcu_dynticks_eqs_enter() - instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); + // instrumentation for the noinstr ct_kernel_exit_state() + instrument_atomic_write(&ct->state, sizeof(ct->state)); instrumentation_end(); =20 // RCU is watching here ... - rcu_dynticks_eqs_enter(); + ct_kernel_exit_state(RCU_DYNTICKS_IDX); // ... but is no longer watching here. =20 if (!in_nmi()) @@ -200,7 +200,7 @@ noinstr void ct_nmi_exit(void) * allow for the possibility of usermode upcalls messing up our count of * interrupt nesting level during the busy period that is just now startin= g. */ -static void noinstr rcu_eqs_exit(bool user) +static void noinstr ct_kernel_enter(bool user, int offset) { struct context_tracking *ct =3D this_cpu_ptr(&context_tracking); long oldval; @@ -215,12 +215,12 @@ static void noinstr rcu_eqs_exit(bool user) } rcu_dynticks_task_exit(); // RCU is not watching here ... - rcu_dynticks_eqs_exit(); + ct_kernel_enter_state(offset); // ... but is watching here. instrumentation_begin(); =20 - // instrumentation for the noinstr rcu_dynticks_eqs_exit() - instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); + // instrumentation for the noinstr ct_kernel_enter_state() + instrument_atomic_write(&ct->state, sizeof(ct->state)); =20 trace_rcu_dyntick(TPS("End"), ct_dynticks_nesting(), 1, ct_dynticks()); WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(c= urrent)); @@ -233,7 +233,7 @@ static void noinstr rcu_eqs_exit(bool user) /** * ct_nmi_enter - inform RCU of entry to NMI context * - * If the CPU was idle from RCU's viewpoint, update ct->dynticks and + * If the CPU was idle from RCU's viewpoint, update ct->state and * ct->dynticks_nmi_nesting to let the RCU grace-period handling know * that the CPU is active. This implementation permits nested NMIs, as * long as the nesting level does not overflow an int. (You will probably @@ -264,14 +264,14 @@ noinstr void ct_nmi_enter(void) rcu_dynticks_task_exit(); =20 // RCU is not watching here ... - rcu_dynticks_eqs_exit(); + ct_kernel_enter_state(RCU_DYNTICKS_IDX); // ... but is watching here. =20 instrumentation_begin(); // instrumentation for the noinstr rcu_dynticks_curr_cpu_in_eqs() - instrument_atomic_read(&ct->dynticks, sizeof(ct->dynticks)); - // instrumentation for the noinstr rcu_dynticks_eqs_exit() - instrument_atomic_write(&ct->dynticks, sizeof(ct->dynticks)); + instrument_atomic_read(&ct->state, sizeof(ct->state)); + // instrumentation for the noinstr ct_kernel_enter_state() + instrument_atomic_write(&ct->state, sizeof(ct->state)); =20 incby =3D 1; } else if (!in_nmi()) { @@ -364,22 +364,23 @@ void noinstr __ct_user_enter(enum ctx_state state) * CPU doesn't need to maintain the tick for RCU maintenance purposes * when the CPU runs in userspace. */ - rcu_eqs_enter(true); + ct_kernel_exit(true, RCU_DYNTICKS_IDX + state); + } else { + /* + * Even if context tracking is disabled on this CPU, because it's outsi= de + * the full dynticks mask for example, we still have to keep track of t= he + * context transitions and states to prevent inconsistency on those of + * other CPUs. + * If a task triggers an exception in userspace, sleep on the exception + * handler and then migrate to another CPU, that new CPU must know where + * the exception returns by the time we call exception_exit(). + * This information can only be provided by the previous CPU when it ca= lled + * exception_enter(). + * OTOH we can spare the calls to vtime and RCU when context_tracking.a= ctive + * is false because we know that CPU is not tickless. + */ + atomic_add(state, &ct->state); } - /* - * Even if context tracking is disabled on this CPU, because it's outside - * the full dynticks mask for example, we still have to keep track of the - * context transitions and states to prevent inconsistency on those of - * other CPUs. - * If a task triggers an exception in userspace, sleep on the exception - * handler and then migrate to another CPU, that new CPU must know where - * the exception returns by the time we call exception_exit(). - * This information can only be provided by the previous CPU when it cal= led - * exception_enter(). - * OTOH we can spare the calls to vtime and RCU when context_tracking.ac= tive - * is false because we know that CPU is not tickless. - */ - atomic_set(&ct->state, state); } context_tracking_recursion_exit(); } @@ -451,15 +452,16 @@ void noinstr __ct_user_exit(enum ctx_state state) * Exit RCU idle mode while entering the kernel because it can * run a RCU read side critical section anytime. */ - rcu_eqs_exit(true); + ct_kernel_enter(true, RCU_DYNTICKS_IDX - state); if (state =3D=3D CONTEXT_USER) { instrumentation_begin(); vtime_user_exit(current); trace_user_exit(0); instrumentation_end(); } + } else { + atomic_sub(state, &ct->state); } - atomic_set(&ct->state, CONTEXT_KERNEL); } context_tracking_recursion_exit(); } @@ -537,7 +539,7 @@ void __init context_tracking_init(void) DEFINE_PER_CPU(struct context_tracking, context_tracking) =3D { .dynticks_nmi_nesting =3D DYNTICK_IRQ_NONIDLE, .dynticks_nesting =3D 1, - .dynticks =3D ATOMIC_INIT(1), + .state =3D ATOMIC_INIT(RCU_DYNTICKS_IDX), }; EXPORT_SYMBOL_GPL(context_tracking); =20 @@ -555,7 +557,7 @@ EXPORT_SYMBOL_GPL(context_tracking); noinstr void ct_idle_enter(void) { lockdep_assert_irqs_disabled(); - rcu_eqs_enter(false); + ct_kernel_exit(false, RCU_DYNTICKS_IDX + CONTEXT_IDLE); } EXPORT_SYMBOL_GPL(ct_idle_enter); =20 @@ -577,7 +579,7 @@ void ct_idle_exit(void) unsigned long flags; =20 local_irq_save(flags); - rcu_eqs_exit(false); + ct_kernel_enter(false, RCU_DYNTICKS_IDX - CONTEXT_IDLE); local_irq_restore(flags); =20 } diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 16d601512da7..e04205c1c62d 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -272,9 +272,9 @@ void rcu_softirq_qs(void) */ static void rcu_dynticks_eqs_online(void) { - if (ct_dynticks() & 0x1) + if (ct_dynticks() & RCU_DYNTICKS_IDX) return; - rcu_dynticks_inc(1); + ct_state_inc(RCU_DYNTICKS_IDX); } =20 /* @@ -293,7 +293,7 @@ static int rcu_dynticks_snap(int cpu) */ static bool rcu_dynticks_in_eqs(int snap) { - return !(snap & 0x1); + return !(snap & RCU_DYNTICKS_IDX); } =20 /* Return true if the specified CPU is currently idle from an RCU viewpoin= t. */ @@ -321,8 +321,7 @@ bool rcu_dynticks_zero_in_eqs(int cpu, int *vp) int snap; =20 // If not quiescent, force back to earlier extended quiescent state. - snap =3D ct_dynticks_cpu(cpu) & ~0x1; - + snap =3D ct_dynticks_cpu(cpu) & ~RCU_DYNTICKS_IDX; smp_rmb(); // Order ->dynticks and *vp reads. if (READ_ONCE(*vp)) return false; // Non-zero, so report failure; @@ -348,9 +347,9 @@ notrace void rcu_momentary_dyntick_idle(void) int seq; =20 raw_cpu_write(rcu_data.rcu_need_heavy_qs, false); - seq =3D rcu_dynticks_inc(2); + seq =3D ct_state_inc(2 * RCU_DYNTICKS_IDX); /* It is illegal to call this from idle state. */ - WARN_ON_ONCE(!(seq & 0x1)); + WARN_ON_ONCE(!(seq & RCU_DYNTICKS_IDX)); rcu_preempt_deferred_qs(current); } EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle); diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h index 933445d826f1..f33044fd8dfc 100644 --- a/kernel/rcu/tree_stall.h +++ b/kernel/rcu/tree_stall.h @@ -467,7 +467,7 @@ static void print_cpu_stall_info(int cpu) rcuc_starved =3D rcu_is_rcuc_kthread_starving(rdp, &j); if (rcuc_starved) sprintf(buf, " rcuc=3D%ld jiffies(starved)", j); - pr_err("\t%d-%c%c%c%c: (%lu %s) idle=3D%03x/%ld/%#lx softirq=3D%u/%u fqs= =3D%ld%s%s\n", + pr_err("\t%d-%c%c%c%c: (%lu %s) idle=3D%04x/%ld/%#lx softirq=3D%u/%u fqs= =3D%ld%s%s\n", cpu, "O."[!!cpu_online(cpu)], "o."[!!(rdp->grpmask & rdp->mynode->qsmaskinit)], @@ -476,7 +476,7 @@ static void print_cpu_stall_info(int cpu) rdp->rcu_iw_pending ? (int)min(delta, 9UL) + '0' : "!."[!delta], ticks_value, ticks_title, - rcu_dynticks_snap(cpu) & 0xfff, + rcu_dynticks_snap(cpu) & 0xffff, ct_dynticks_nesting_cpu(cpu), ct_dynticks_nmi_nesting_cpu(cpu), rdp->softirq_snap, kstat_softirqs_cpu(RCU_SOFTIRQ, cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, --=20 2.25.1