From nobody Tue Feb 10 21:39:09 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D902C0015E for ; Tue, 1 Aug 2023 13:24:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234101AbjHANYz (ORCPT ); Tue, 1 Aug 2023 09:24:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230315AbjHANYw (ORCPT ); Tue, 1 Aug 2023 09:24:52 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C1841985 for ; Tue, 1 Aug 2023 06:24:51 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CE895615A4 for ; Tue, 1 Aug 2023 13:24:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4BDFBC433C7; Tue, 1 Aug 2023 13:24:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896290; bh=OCbZZEYY6XG+Gj++glqCcqULb++oCr0LofyrtVtyhEc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oUheRgQ6e3jTLVKzFRNeivu8xi+wx3bSs7luMToq7MPUyeMEOdFV9Z10Xwj52jGBl TsBzJKvb9b+/lHy5AF7lkLk+ev1RoQAc/mtDMsEnAswNdWKiH79GfrtjsA5DmDxD8M GFiG1ohj3mNsLa4alssdvt9LGyjknXtmPAD/p8AhgFnq34OEh2S6c0lToN5VK9gU5j kA/v7zsvQ5CTF3dGiW/7YcQBuJqvH75sOtwB2Kultu/XQULLc5jC2TOqCrm5klZUOs /brfyl+A+H/TH738p7IqIOp+65VenPfrbLtrAiSiK77m2Oh3wDfMsnvm2MzM8Luh4+ u7PdXIokymRMw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 1/6] softirq: Turn set_softirq_pending() to reset_softirq_pending() Date: Tue, 1 Aug 2023 15:24:36 +0200 Message-Id: <20230801132441.559222-2-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" set_softirq_pending() is only ever used to reset the pending vector's mask to 0. Make the function more specialized for that very purpose and rename it accordingly. Signed-off-by: Frederic Weisbecker --- include/linux/interrupt.h | 2 +- kernel/softirq.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index bf82980f569d..2099fe3980bc 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -524,7 +524,7 @@ DECLARE_STATIC_KEY_FALSE(force_irqthreads_key); #endif =20 #define local_softirq_pending() (__this_cpu_read(local_softirq_pending_ref= )) -#define set_softirq_pending(x) (__this_cpu_write(local_softirq_pending_ref= , (x))) +#define reset_softirq_pending() (__this_cpu_write(local_softirq_pending_re= f, 0)) #define or_softirq_pending(x) (__this_cpu_or(local_softirq_pending_ref, (x= ))) =20 #endif /* local_softirq_pending */ diff --git a/kernel/softirq.c b/kernel/softirq.c index 623985f18833..1a3c3fe341ea 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -532,7 +532,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(= void) =20 restart: /* Reset the pending bitmask before enabling irqs */ - set_softirq_pending(0); + reset_softirq_pending(); =20 local_irq_enable(); =20 --=20 2.34.1 From nobody Tue Feb 10 21:39:09 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6689C0015E for ; Tue, 1 Aug 2023 13:24:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234122AbjHANY5 (ORCPT ); Tue, 1 Aug 2023 09:24:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234085AbjHANYz (ORCPT ); Tue, 1 Aug 2023 09:24:55 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E5EE198D for ; Tue, 1 Aug 2023 06:24:53 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3CD1661594 for ; Tue, 1 Aug 2023 13:24:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AEFEDC433CA; Tue, 1 Aug 2023 13:24:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896292; bh=vb0eW648+k9SO6deOUTYMYcSVwD0/SVM0GpxXUYmFVs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DivxzMVQpdG0eYF91RGmaydcwUI6rJqX/Eh1Q2gJdYvQasfpLmBSaExyfOVmDFC/b vTywotqFfXyX6HJyrvhKIrH95pEym5ZDYyujswn+zCZqtV5e726DqIs13NQuU9VAOK RYjntqOs3CqvUP6yDz2oFWbAXSIXBZxtmFyYBo0rzMJVA+o5KikYNYm7fiqfIiU6sG Nai3Tmk+XjdQzGejIVIu6VA3m/Yw1Q0IcKgWx7LW8Ncm9xcaUY5LtB5jZnzpk9h30q ruMyRtUdRgRV0vDp2UaVRvcO67ZkPCdSnxa1CZJ24yPF9vupAlpCkmmqa6zDmX991f yw3xq1ru+dgBQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 2/6] softirq: Make softirq handling entry/exit generally available Date: Tue, 1 Aug 2023 15:24:37 +0200 Message-Id: <20230801132441.559222-3-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In order to prepare for re-enabling softirqs from vector callbacks that are known safe, make the code incrementing the preempt count while servicing softirqs more generally available. No intended behaviour change. Signed-off-by: Frederic Weisbecker --- include/linux/bottom_half.h | 7 +++++++ kernel/softirq.c | 22 ++++++++++++++++------ 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/include/linux/bottom_half.h b/include/linux/bottom_half.h index fc53e0ad56d9..2243c7de4917 100644 --- a/include/linux/bottom_half.h +++ b/include/linux/bottom_half.h @@ -33,6 +33,13 @@ static inline void local_bh_enable(void) __local_bh_enable_ip(_THIS_IP_, SOFTIRQ_DISABLE_OFFSET); } =20 +static inline void local_bh_enter(void) +{ + __local_bh_disable_ip(_RET_IP_, SOFTIRQ_OFFSET); +} + +extern void local_bh_exit(void); + #ifdef CONFIG_PREEMPT_RT extern bool local_bh_blocked(void); #else diff --git a/kernel/softirq.c b/kernel/softirq.c index 1a3c3fe341ea..ba998d572ef4 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -247,21 +247,26 @@ void __local_bh_enable_ip(unsigned long ip, unsigned = int cnt) } EXPORT_SYMBOL(__local_bh_enable_ip); =20 +inline void local_bh_exit(void) +{ + __local_bh_enable(SOFTIRQ_OFFSET, true); + WARN_ON_ONCE(in_interrupt()); +} + /* * Invoked from ksoftirqd_run() outside of the interrupt disabled section * to acquire the per CPU local lock for reentrancy protection. */ static inline void ksoftirqd_run_begin(void) { - __local_bh_disable_ip(_RET_IP_, SOFTIRQ_OFFSET); + local_bh_enter(); local_irq_disable(); } =20 /* Counterpart to ksoftirqd_run_begin() */ static inline void ksoftirqd_run_end(void) { - __local_bh_enable(SOFTIRQ_OFFSET, true); - WARN_ON_ONCE(in_interrupt()); + local_bh_exit(); local_irq_enable(); } =20 @@ -389,15 +394,20 @@ void __local_bh_enable_ip(unsigned long ip, unsigned = int cnt) } EXPORT_SYMBOL(__local_bh_enable_ip); =20 +inline void local_bh_exit(void) +{ + __local_bh_enable(SOFTIRQ_OFFSET); + WARN_ON_ONCE(in_interrupt()); +} + static inline void softirq_handle_begin(void) { - __local_bh_disable_ip(_RET_IP_, SOFTIRQ_OFFSET); + local_bh_enter(); } =20 static inline void softirq_handle_end(void) { - __local_bh_enable(SOFTIRQ_OFFSET); - WARN_ON_ONCE(in_interrupt()); + local_bh_exit(); } =20 static inline void ksoftirqd_run_begin(void) --=20 2.34.1 From nobody Tue Feb 10 21:39:09 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BB1DC001E0 for ; Tue, 1 Aug 2023 13:25:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234172AbjHANZG (ORCPT ); Tue, 1 Aug 2023 09:25:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234114AbjHANY5 (ORCPT ); Tue, 1 Aug 2023 09:24:57 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AF091985 for ; Tue, 1 Aug 2023 06:24:56 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9DD4F61586 for ; Tue, 1 Aug 2023 13:24:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B81DC433C8; Tue, 1 Aug 2023 13:24:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896295; bh=feqQBtIQhDcLNaK5yV5c0SoBbM4IgwK6op+QnaFp2wY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hJ1VvodxVi5EP11Bjtgw6Oj0x5oqLYcCDP6Jfz/HKcjqeloi1SlufqjWZ14f0Wpco 1AUGhZ2yRxdEFjieO2AJkIxl1upO9jbBluXpIcvFnyNSe3D1sPqeiVTFtlraJPeRFj UYCyv4B8zEjo13yezuB34uFq8Q+CNAtxh4bV104867QpLikgOnv2FzxpKbB/vYIg7j R2Hez7VsHhP1DtV/hANNbPwxjqTvqa+nse/MjQWu1+GMAMSF1Ta8dARfDl2oNBoLdv ZTEEPOXW0M/ww/bfXlau7+mEp6IPb/yVTo8/xIQlA78omsqY7eZBrhhWolq81ik3rN pIXQiO8N55kVA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 3/6] softirq: Introduce softirq disabled mask Date: Tue, 1 Aug 2023 15:24:38 +0200 Message-Id: <20230801132441.559222-4-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" (DISCLAIMER: contains -RT bits) When softirq vectors will be able to re-enable softirqs when deemed safe, for example when a timer callback is tagged as soft-interruptible by other softirq vectors, care must be taken to ensure a given vector is not re-entrant. Ie: a vector can be interrupted by others but not by itself. In order to prepare for this, introduce a softirq disabled mask so that vectors can disable themselves before re-enabling softirqs. Signed-off-by: Frederic Weisbecker --- arch/Kconfig | 3 +++ include/linux/bottom_half.h | 2 ++ include/linux/interrupt.h | 15 ++++++++++++--- kernel/softirq.c | 16 +++++++++++++++- 4 files changed, 32 insertions(+), 4 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 205fd23e0cad..d23968860ddf 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1358,6 +1358,9 @@ config RELR config ARCH_HAS_MEM_ENCRYPT bool =20 +config ARCH_HAS_SOFTIRQ_DISABLED_MASK + bool + config ARCH_HAS_CC_PLATFORM bool =20 diff --git a/include/linux/bottom_half.h b/include/linux/bottom_half.h index 2243c7de4917..d5b37b580c79 100644 --- a/include/linux/bottom_half.h +++ b/include/linux/bottom_half.h @@ -42,6 +42,8 @@ extern void local_bh_exit(void); =20 #ifdef CONFIG_PREEMPT_RT extern bool local_bh_blocked(void); +extern void local_bh_vec_enable(int vec); +extern void local_bh_vec_disable(int vec); #else static inline bool local_bh_blocked(void) { return false; } #endif diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index 2099fe3980bc..7819d16d8d6f 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -523,8 +523,16 @@ DECLARE_STATIC_KEY_FALSE(force_irqthreads_key); #define local_softirq_pending_ref irq_stat.__softirq_pending #endif =20 -#define local_softirq_pending() (__this_cpu_read(local_softirq_pending_ref= )) -#define reset_softirq_pending() (__this_cpu_write(local_softirq_pending_re= f, 0)) +#if defined(CONFIG_PREEMPT_RT) && defined(CONFIG_ARCH_HAS_SOFTIRQ_DISABLED= _MASK) +#define local_softirq_disabled() (__this_cpu_read(local_softirq_disabled_r= ef)) +#else +#define local_softirq_disabled() (0) +#endif + +#define local_softirq_pending() (__this_cpu_read(local_softirq_pending_ref= ) & \ + ~local_softirq_disabled()) +#define reset_softirq_pending() (__this_cpu_and(local_softirq_pending_ref,= \ + local_softirq_disabled())) #define or_softirq_pending(x) (__this_cpu_or(local_softirq_pending_ref, (x= ))) =20 #endif /* local_softirq_pending */ @@ -614,7 +622,8 @@ extern void raise_hrtimer_softirq(void); =20 static inline unsigned int local_pending_timers(void) { - return __this_cpu_read(pending_timer_softirq); + return __this_cpu_read(pending_timer_softirq) & + ~local_softirq_disabled(); } =20 #else diff --git a/kernel/softirq.c b/kernel/softirq.c index ba998d572ef4..a394f78de627 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -297,6 +297,18 @@ void do_softirq_post_smp_call_flush(unsigned int was_p= ending) invoke_softirq(); } =20 +#ifdef CONFIG_ARCH_HAS_SOFTIRQ_DISABLED_MASK +void local_bh_vec_enable(int vec) +{ + __this_cpu_and(local_softirq_disabled_ref, ~vec); +} + +void local_bh_vec_disable(int vec) +{ + __this_cpu_or(local_softirq_disabled_ref, vec); +} +#endif + #else /* CONFIG_PREEMPT_RT */ =20 /* @@ -1009,11 +1021,13 @@ static int timersd_should_run(unsigned int cpu) static void run_timersd(unsigned int cpu) { unsigned int timer_si; + unsigned long timersd_vecs =3D (1 << TIMER_SOFTIRQ) | (1 << HRTIMER_SOFTI= RQ); =20 ksoftirqd_run_begin(); =20 timer_si =3D local_pending_timers(); - __this_cpu_write(pending_timer_softirq, 0); + __this_cpu_and(pending_timer_softirq, + local_softirq_disabled() & timersd_vecs); or_softirq_pending(timer_si); =20 __do_softirq(); --=20 2.34.1 From nobody Tue Feb 10 21:39:09 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B9E0C0015E for ; Tue, 1 Aug 2023 13:25:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234201AbjHANZK (ORCPT ); Tue, 1 Aug 2023 09:25:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37012 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234108AbjHANZD (ORCPT ); Tue, 1 Aug 2023 09:25:03 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B379C2116 for ; Tue, 1 Aug 2023 06:24:58 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0E5E8615A3 for ; Tue, 1 Aug 2023 13:24:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7FF58C433C7; Tue, 1 Aug 2023 13:24:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896297; bh=kRD+z973ymfN8CUInsVsbutA6ADoO6ucBF0+2kGIUy4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WQ8ryfpIqxjOXfYyBJzKG4qNR6jyMndhOItuP0gL6Zbgx8+366OyCfJMkg5+BZMFU auaat53rdg3hW7VpAIdGX1pYYWZ4yNc/ppnGpvrqLn4F4ZP4fD1FYymoXgwah8cSMT uYWt+A+CW+hwn1wiSqnpb5qBSDAv+Ibjqz2u5kSPq5VV/H2Gl6jL7YPMeV/6AkfGuA 9XuN+ZIvzLOaoddC29YcQln5X8lCsIrY3xThY7Dd6O09l0/a3Ley0FzhflcQBXdx47 eWINZFRzrKs6nKk3kKDM9viCOE2mroMnZiqreHOWvNJwXxhQX4BTbr6y6LOTwocsVN oUDSA7fzB3cWw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 4/6] x86/softirq: Support softirq disabled mask Date: Tue, 1 Aug 2023 15:24:39 +0200 Message-Id: <20230801132441.559222-5-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Support the new softirq disabled vectors mask and put in the per-cpu hot structure along with the pending vectors mask as both are used closely together. Signed-off-by: Frederic Weisbecker --- arch/x86/Kconfig | 1 + arch/x86/include/asm/current.h | 1 + arch/x86/include/asm/hardirq.h | 1 + 3 files changed, 3 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index a2a410d13e39..f1cd68b672dc 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -93,6 +93,7 @@ config X86 select ARCH_HAS_COPY_MC if X86_64 select ARCH_HAS_SET_MEMORY select ARCH_HAS_SET_DIRECT_MAP + select ARCH_HAS_SOFTIRQ_DISABLED_MASK select ARCH_HAS_STRICT_KERNEL_RWX select ARCH_HAS_STRICT_MODULE_RWX select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index a1168e7b69e5..b86ca9c0ddc2 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -23,6 +23,7 @@ struct pcpu_hot { unsigned long top_of_stack; void *hardirq_stack_ptr; u16 softirq_pending; + u16 softirq_disabled; #ifdef CONFIG_X86_64 bool hardirq_stack_inuse; #else diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index 66837b8c67f1..933cf05e5738 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -61,6 +61,7 @@ extern u64 arch_irq_stat(void); #define arch_irq_stat arch_irq_stat =20 #define local_softirq_pending_ref pcpu_hot.softirq_pending +#define local_softirq_disabled_ref pcpu_hot.softirq_disabled =20 #if IS_ENABLED(CONFIG_KVM_INTEL) static inline void kvm_set_cpu_l1tf_flush_l1d(void) --=20 2.34.1 From nobody Tue Feb 10 21:39:09 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADA13C00528 for ; Tue, 1 Aug 2023 13:25:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234163AbjHANZN (ORCPT ); Tue, 1 Aug 2023 09:25:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233096AbjHANZF (ORCPT ); Tue, 1 Aug 2023 09:25:05 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C036A2683 for ; Tue, 1 Aug 2023 06:25:01 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6F61961594 for ; Tue, 1 Aug 2023 13:25:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E4767C433C9; Tue, 1 Aug 2023 13:24:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896299; bh=rdh2v8ubgWirANjBqQeINfRDtrMu7AmbvfgkqLxk0ZQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eg8Wsrw5hhHyNuQV0Y6edeuyltWCE4/vwspiKaXOia1pjtO644tTGTaOJ4UmbAXoH rhQfXc0vC+L0z6dqLCXrZKJb5TyV0VDLXhQCSqPKPNBhlYp8od3LSXJ7ct6zYYTS+1 bMqrSBbA2Reu++Eulkkw+tVEmB0Xqdhw34BP4GGdfuO5vCW7lmb+XPjShz1/sva7Q6 ESsceLV2WEluWAt56iBfc68isurryOxCW1erhZ+g9K9C9aRh5cSAOTNBRhpKbhrH81 Na3OoRxbeq2wS9Fi3g8nX2KGaI0N8gZuSPBnVkrfDcQjgOaVrOsTq2CvukqOMM1BZh E8dBg1bEfGepQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 5/6] timers: Introduce soft-interruptible timers Date: Tue, 1 Aug 2023 15:24:40 +0200 Message-Id: <20230801132441.559222-6-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Most timers are unrelated to networking or other softirq vectors (RCU, NET_*, HRTIMER, TASKLET, ...). Yet when a timer batch is executing, other softirq vectors have to wait for the timers batch completion even though there is nothing to synchronize against most callbacks. However there is no automatic way to determine if a timer callback is safely soft-interruptible by other vectors. So the only long term viable approach to solve this is to adopt a progressive push down solution similar to the one used for getting rid of the big kernel lock. Introduce a new TIMER_SOFTINTERRUPTIBLE flag which tells the timer subsystem that a callback is safely soft-interruptible by other vectors, either because it's completely unrelated to them or because it uses the appropriate local_bh_disable()/spin_lock_bh() on narrowed-down regions. Once all timers are dealt with after a few years, it will become possible to run timers out of the softirqs processing. It's worth noting though that if the softirq infrastructure supports soft-interruption of a TIMER_SOFTINTERRUPTIBLE timer, it doesn't allow yet a TIMER_SOFTINTERRUPTIBLE timer to soft-interrupt other vectors, even though nothing prevents from it to happen from a correctness point of view, more tweaks are needed to support that. Signed-off-by: Frederic Weisbecker --- include/linux/timer.h | 5 +++-- kernel/time/timer.c | 18 ++++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/include/linux/timer.h b/include/linux/timer.h index 9162f275819a..fbe40bacc8c3 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -61,13 +61,14 @@ struct timer_list { * should be placed on a particular CPU, then add_timer_on() has to be * used. */ -#define TIMER_CPUMASK 0x0003FFFF +#define TIMER_CPUMASK 0x0001FFFF /* If 1 more bit is needed, flags must b= e 64 */ +#define TIMER_SOFTINTERRUPTIBLE 0x00020000 #define TIMER_MIGRATING 0x00040000 #define TIMER_BASEMASK (TIMER_CPUMASK | TIMER_MIGRATING) #define TIMER_DEFERRABLE 0x00080000 #define TIMER_PINNED 0x00100000 #define TIMER_IRQSAFE 0x00200000 -#define TIMER_INIT_FLAGS (TIMER_DEFERRABLE | TIMER_PINNED | TIMER_IRQSAFE) +#define TIMER_INIT_FLAGS (TIMER_SOFTINTERRUPTIBLE | TIMER_DEFERRABLE | TIM= ER_PINNED | TIMER_IRQSAFE) #define TIMER_ARRAYSHIFT 22 #define TIMER_ARRAYMASK 0xFFC00000 =20 diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 7cad6fe3c035..1e43f54def0e 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1676,6 +1676,7 @@ static void call_timer_fn(struct timer_list *timer, unsigned long baseclk) { int count =3D preempt_count(); + bool softinterruptible =3D false; =20 #ifdef CONFIG_LOCKDEP /* @@ -1689,6 +1690,17 @@ static void call_timer_fn(struct timer_list *timer, =20 lockdep_copy_map(&lockdep_map, &timer->lockdep_map); #endif + + if (IS_ENABLED(CONFIG_PREEMPT_RT) && + IS_ENABLED(CONFIG_ARCH_HAS_SOFTIRQ_DISABLED_MASK) && + timer->flags & TIMER_SOFTINTERRUPTIBLE) + softinterruptible =3D true; + + if (softinterruptible) { + local_bh_vec_disable(1 << TIMER_SOFTIRQ); + local_bh_exit(); + } + /* * Couple the lock chain with the lock chain at * timer_delete_sync() by acquiring the lock_map around the fn() @@ -1702,6 +1714,12 @@ static void call_timer_fn(struct timer_list *timer, =20 lock_map_release(&lockdep_map); =20 + if (softinterruptible) { + local_bh_enter(); + local_bh_vec_enable(1 << TIMER_SOFTIRQ); + } + + if (count !=3D preempt_count()) { WARN_ONCE(1, "timer: %pS preempt leak: %08x -> %08x\n", fn, count, preempt_count()); --=20 2.34.1 From nobody Tue Feb 10 21:39:09 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16C52C0015E for ; Tue, 1 Aug 2023 13:25:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234242AbjHANZU (ORCPT ); Tue, 1 Aug 2023 09:25:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234187AbjHANZI (ORCPT ); Tue, 1 Aug 2023 09:25:08 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36825269F for ; Tue, 1 Aug 2023 06:25:03 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E1624615A3 for ; Tue, 1 Aug 2023 13:25:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 509D6C433C8; Tue, 1 Aug 2023 13:25:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896302; bh=rciJFUiyVsvhXbHYtyfujY9ldyQ8u3jv3Unlkm5oJ8Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lnp7XiqARKpzKndW55oU0pFk++KVe7qMUYvK5aKtl891T2uF5+HHyVOFY++Pfj7cb 9hIF71vVnTxKoWv8urpMEhOPN9X+JwnUUTcJ0iKi2dnt6OrQecFIZq/jtkqDOHeIRx 0B6hJ1R3KNBOfN8wHxkzYjsgIXm2rGyxDV2qS5hr6qHZ6cn+mg7ijE4vbVyFziZpGJ 63bdbmru35c+St1REUh6OG8y0Z4iaOlXWpJvmsgypEio81WHULIM+b0B8qX94ygl7H 2ekN/heycVbwDUdNup6qsP/UYHcyO3HEwuovJ3oE71r8F4iINoP3dB1axgSedSDqPd usHOdgqfR2MhA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 6/6] timers: Make process_timeout() soft-interruptible Date: Tue, 1 Aug 2023 15:24:41 +0200 Message-Id: <20230801132441.559222-7-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The most frequent timer can be safely interrupted by other vectors as it's only performing a wake-up. Signed-off-by: Frederic Weisbecker --- kernel/time/timer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 1e43f54def0e..0f6eb9c0901c 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2180,7 +2180,7 @@ signed long __sched schedule_timeout(signed long time= out) expire =3D timeout + jiffies; =20 timer.task =3D current; - timer_setup_on_stack(&timer.timer, process_timeout, 0); + timer_setup_on_stack(&timer.timer, process_timeout, TIMER_SOFTINTERRUPTIB= LE); __mod_timer(&timer.timer, expire, MOD_TIMER_NOTPENDING); schedule(); del_timer_sync(&timer.timer); --=20 2.34.1