From nobody Thu Sep 11 14:38:23 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7782CC04A94 for ; Fri, 4 Aug 2023 11:30:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230055AbjHDLax (ORCPT ); Fri, 4 Aug 2023 07:30:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229599AbjHDLau (ORCPT ); Fri, 4 Aug 2023 07:30:50 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BFA43E7 for ; Fri, 4 Aug 2023 04:30:48 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1691148646; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1T2AeWiya68KJLNtsfeN+eBKy/0/70O9BggovBkInGY=; b=TMbtZKjleCooMYS4uB+lFWSsOnrkfX78pGv4mnOlxMBKg3E2ejw5zXx0u3O6gMNvieET3J 04ATwz6F+0bDOXBWORbVMRd8OJAAccCU4CWW4K6uPliemgFaKdv7BGUUhByp+XLMLA+9ns 1HsIoYmrY1yzqNWCpYmpA9wWKGZWR4rWpp3QgtIN+6EtfYTUakId2RVn6tGcDRz6SK2jb0 cWjoXqT9vE0NmHUsdlBLnGltEBavXcuWF8YCgvOPGbA+azZnx2EtSqfweNQjuPMZg14I9E fWA0NlOc04Zdqf7F5lTvaAQgq+IS1SqEROuTeer8O5CfIbS8fxi8NHdqDXXiWg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1691148646; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1T2AeWiya68KJLNtsfeN+eBKy/0/70O9BggovBkInGY=; b=t4Ak2F4YBaw6B5zQGDc8F5Hl/F8C2wCrFjOfUK3CFag/dBDbS/qZKLFT52bbDHmN1EYuQZ I/hCb6EvOE6K6LAg== To: linux-kernel@vger.kernel.org Cc: Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Frederic Weisbecker , Ingo Molnar , John Stultz , Juri Lelli , Mel Gorman , Peter Zijlstra , Stephen Boyd , Steven Rostedt , Thomas Gleixner , Valentin Schneider , Vincent Guittot , Sebastian Andrzej Siewior Subject: [RFC PATCH 1/3] sched/core: Provide a method to check if a task is PI-boosted. Date: Fri, 4 Aug 2023 13:30:37 +0200 Message-Id: <20230804113039.419794-2-bigeasy@linutronix.de> In-Reply-To: <20230804113039.419794-1-bigeasy@linutronix.de> References: <20230804113039.419794-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Provide a method to check if a task inherited the priority from another task. This happens if a task owns a lock which is requested by a task with higher priority. This can be used as a hint to add a preemption point to the critical section. Provide a function which reports true if the task is PI-boosted. Signed-off-by: Sebastian Andrzej Siewior --- include/linux/sched.h | 1 + kernel/sched/core.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 609bde814cb06..77fd274133750 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1887,6 +1887,7 @@ static inline int dl_task_check_affinity(struct task_= struct *p, const struct cpu } #endif =20 +extern bool task_is_pi_boosted(const struct task_struct *p); extern int yield_to(struct task_struct *p, bool preempt); extern void set_user_nice(struct task_struct *p, long nice); extern int task_prio(const struct task_struct *p); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index c52c2eba7c739..132f06522efa0 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -8886,6 +8886,21 @@ static inline void preempt_dynamic_init(void) { } =20 #endif /* #ifdef CONFIG_PREEMPT_DYNAMIC */ =20 +/* + * task_is_pi_boosted - Check if task has been PI boosted. + * @p: Task to check. + * + * Return true if task is subject to priority inheritance. + */ +bool task_is_pi_boosted(const struct task_struct *p) +{ + int prio =3D p->prio; + + if (!rt_prio(prio)) + return false; + return prio !=3D p->normal_prio; +} + /** * yield - yield the current processor to other threads. * --=20 2.40.1 From nobody Thu Sep 11 14:38:23 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43BCFC001DF for ; Fri, 4 Aug 2023 11:31:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230381AbjHDLbB (ORCPT ); Fri, 4 Aug 2023 07:31:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229790AbjHDLau (ORCPT ); Fri, 4 Aug 2023 07:30:50 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BE67126 for ; Fri, 4 Aug 2023 04:30:49 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1691148647; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pyi0hihZY2jn2pAnNhMOJoJ7pshtJwt3VPheJEyFbQY=; b=T1VtCb5summwfo85838+uP2xHDFvzN6L8pIbSEkwnY/Bt+cKFhCDOm9QsKgH4oyCZ/7jis cdVHWkciqXSseqEOQwiDr+0vLtRzh1TBFxBElnlhE+AhRmfVBFfLxYswhBp7tk/7vWEjdh V71wZ7ucT+37ZAq7H8oGEKgfxsDBzGMFchyUMxZLF8BUr8UQZs5L8MacZjkfP4dueYS5JA eztj6HLBkd3Yy+m3AuayEymhBBb2jgCsRyXAPLSAh/r9pZLBfIG9eV4RsHIOQSVLTE7HgY HRLp90yFXYXk4bKbOByTwwYmEyrZwHwnpwmXfVz6y5HklIJWVAeZp8vFknmC2A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1691148647; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pyi0hihZY2jn2pAnNhMOJoJ7pshtJwt3VPheJEyFbQY=; b=WV6XnA1MUccjmajKxXKw6rjDjDZMqwO4T7Nxv7DOikc6Mw/KSWSxDZE7hZL6AKXlo5BpEg ktbaUx5mCTI4yADw== To: linux-kernel@vger.kernel.org Cc: Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Frederic Weisbecker , Ingo Molnar , John Stultz , Juri Lelli , Mel Gorman , Peter Zijlstra , Stephen Boyd , Steven Rostedt , Thomas Gleixner , Valentin Schneider , Vincent Guittot , Sebastian Andrzej Siewior Subject: [RFC PATCH 2/3] softirq: Add function to preempt serving softirqs. Date: Fri, 4 Aug 2023 13:30:38 +0200 Message-Id: <20230804113039.419794-3-bigeasy@linutronix.de> In-Reply-To: <20230804113039.419794-1-bigeasy@linutronix.de> References: <20230804113039.419794-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add a functionality for the softirq handler to preempt its current work if needed. The softirq core has no particular state. It reads and resets the pending softirq bits and then processes one after the other. It can already be preempted while it invokes a certain softirq handler. By enabling the BH the softirq core releases the per-CPU bh lock which serializes all softirq handler. It is safe to do as long as the code does not expect any serialisation in between. A typical scenarion would after the invocation of callback where no state needs to be preserved before the next callback is invoked. Add functionaliry to preempt the serving softirqs. Signed-off-by: Sebastian Andrzej Siewior --- include/linux/bottom_half.h | 2 ++ kernel/softirq.c | 13 +++++++++++++ 2 files changed, 15 insertions(+) diff --git a/include/linux/bottom_half.h b/include/linux/bottom_half.h index fc53e0ad56d90..448bbef474564 100644 --- a/include/linux/bottom_half.h +++ b/include/linux/bottom_half.h @@ -35,8 +35,10 @@ static inline void local_bh_enable(void) =20 #ifdef CONFIG_PREEMPT_RT extern bool local_bh_blocked(void); +extern void softirq_preempt(void); #else static inline bool local_bh_blocked(void) { return false; } +static inline void softirq_preempt(void) { } #endif =20 #endif /* _LINUX_BH_H */ diff --git a/kernel/softirq.c b/kernel/softirq.c index 807b34ccd7973..dd3307a619af7 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -247,6 +247,19 @@ void __local_bh_enable_ip(unsigned long ip, unsigned i= nt cnt) } EXPORT_SYMBOL(__local_bh_enable_ip); =20 +void softirq_preempt(void) +{ + if (WARN_ON_ONCE(!preemptible())) + return; + + if (WARN_ON_ONCE(__this_cpu_read(softirq_ctrl.cnt) !=3D SOFTIRQ_OFFSET)) + return; + + __local_bh_enable(SOFTIRQ_OFFSET, true); + /* preemption point */ + __local_bh_disable_ip(_RET_IP_, SOFTIRQ_OFFSET); +} + /* * Invoked from ksoftirqd_run() outside of the interrupt disabled section * to acquire the per CPU local lock for reentrancy protection. --=20 2.40.1 From nobody Thu Sep 11 14:38:23 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99D8EC04A94 for ; Fri, 4 Aug 2023 11:30:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230264AbjHDLa5 (ORCPT ); Fri, 4 Aug 2023 07:30:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229658AbjHDLau (ORCPT ); Fri, 4 Aug 2023 07:30:50 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D321122 for ; Fri, 4 Aug 2023 04:30:48 -0700 (PDT) From: Sebastian Andrzej Siewior DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1691148647; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u9DRriWNDJonfqILcAsrrLxxJbWS0AzfymHX8jUCjo8=; b=fk745ondrh8eLE4m+TZCnu5htor3AbCLsT4+gFHOWDRVKIbdOc2TCz6/3Rf3T9RhXPsMNG +IqdHUMAIID8mVdpg3Y9QmcRvEf4at/XN1X9ZCQYAXrVspw6fUL+TdrTIWzjagne9uHgoL R2VhE9VJuQ9xSOqdTCNZYoZ7klEeTxH5DglueMXM8samBWOEi3VcICroY+KLuV0OA/05pJ O+WNMMcWkJHgRfxm2gL3I2O2seurRBpcZSLFdzWrMnWTUAg8jiGVT98xViKDo20s4XLuRw /XTSpZVmctAOtP2fIA5N36/N3omJ/I/ZBa871vz5UAqUaMpfpvZKPcOZTvy+SQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1691148647; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u9DRriWNDJonfqILcAsrrLxxJbWS0AzfymHX8jUCjo8=; b=GmBffpV5Rx7yoUp7D7U90VSB2AeDuKYy9aGkOaDP6w1kbSCJCZkUAhLR0rDVNZamigz879 akaXrATbou6VVCBg== To: linux-kernel@vger.kernel.org Cc: Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Frederic Weisbecker , Ingo Molnar , John Stultz , Juri Lelli , Mel Gorman , Peter Zijlstra , Stephen Boyd , Steven Rostedt , Thomas Gleixner , Valentin Schneider , Vincent Guittot , Sebastian Andrzej Siewior Subject: [RFC PATCH 3/3] time: Allow to preempt after a callback. Date: Fri, 4 Aug 2023 13:30:39 +0200 Message-Id: <20230804113039.419794-4-bigeasy@linutronix.de> In-Reply-To: <20230804113039.419794-1-bigeasy@linutronix.de> References: <20230804113039.419794-1-bigeasy@linutronix.de> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The TIMER_SOFTIRQ handler invokes timer callbacks of the expired timers. Before each invocation the timer_base::lock is dropped. The only lock that is still held is the timer_base::expiry_lock and the per-CPU bh-lock as part of local_bh_disable(). The former is released as part of lock up prevention if the timer is preempted by the caller which is waiting for its completion. Both locks are already released as part of timer_sync_wait_running(). This can be extended by also releasing in bh-lock. The timer core does not rely on any state that is serialized by the bh-lock. The timer callback expects the bh-state to be serialized by the lock but there is no need to keep state synchronized while invoking multiple callbacks. Preempt handling softirqs and release all locks after a timer invocation if the current has inherited priority. Signed-off-by: Sebastian Andrzej Siewior --- kernel/time/timer.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 63a8ce7177dd4..b76856bc2edd2 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1470,9 +1470,16 @@ static inline void timer_base_unlock_expiry(struct t= imer_base *base) */ static void timer_sync_wait_running(struct timer_base *base) { - if (atomic_read(&base->timer_waiters)) { + bool need_preempt; + + need_preempt =3D task_is_pi_boosted(current); + if (need_preempt || atomic_read(&base->timer_waiters)) { raw_spin_unlock_irq(&base->lock); spin_unlock(&base->expiry_lock); + + if (need_preempt) + softirq_preempt(); + spin_lock(&base->expiry_lock); raw_spin_lock_irq(&base->lock); } --=20 2.40.1