From nobody Mon Apr 13 18:44:07 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50697C433FE for ; Wed, 16 Nov 2022 07:59:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232913AbiKPH7m (ORCPT ); Wed, 16 Nov 2022 02:59:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232346AbiKPH7i (ORCPT ); Wed, 16 Nov 2022 02:59:38 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 801F1C7 for ; Tue, 15 Nov 2022 23:59:37 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id om10-20020a17090b3a8a00b002108b078ab1so1167530pjb.9 for ; Tue, 15 Nov 2022 23:59:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7v71ZLMMw9Ypk/qBIbo+9W6doSfdrcneW8rCdbD8mEA=; b=aVyjfoFajZKLOg+r2F+ZQPrKrjySyvIWcFKpJ4d1e/CHJ+bdY58v7gW+f+Uko6nH3q RP7ZRjV924KcAXvTR872pThBb1PBfZDwI7ClhwK6GnQXqxyGtxoLvgHaaQnmdTqo3wuF 0geX7hpH8jUu5+RwxicO37D214kmNIZ77WZQsMgg/OUarnykVJDAR5LSF6U2kPfeyvG8 tTzIZfYtvRtuBhA69pk4ncCYsPHVvjW1YNHQI6BN8e2pKkTolxwRmWGq6FF2A2xn2g96 ysvANC+S6GVrtMlLqB6D/tngj+IkyohmnBIV5/Oxeb1stneOUaM93jc8AJk9FPlW6aYM MSPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7v71ZLMMw9Ypk/qBIbo+9W6doSfdrcneW8rCdbD8mEA=; b=K8RH5Ww4LEbl/qx6nQIRw+h0xgfOtFaQ7Y30nhdxNn6P4xzUKT5Ai2w/TETKeHCWLq U6GVer2wh0S/AE1pf6VOBsJRBTU1O45QnBY0Kt06UXP1PHiZzTvTHeF7HXPsitFcWG++ cBIRSBTOrWf9FxmSszRVuYVGJJBa65OdlIgF3pV9VwOLAbUyJ3HPBoeRZ3H0XqOuYSBJ 2ofWvaB+0vXEv4svKN2s9kKqobPRiHaWNivIFOcaVFTcSbeujmfZ0Hw1EYjginrSxDAZ gJ83hCZOACuu6UakbG4yMJcqt1X5WSAnsbirKzpw9MyCR9YUwtPANysGSuTAVkNNTIQz w9/g== X-Gm-Message-State: ANoB5pmqt+eQEwnAmhl03leP5yijnD5KrSMarJNS7Z6xz8ythv1DZ8G7 vDebokOf0tKy2sOx/7oX7nakB2a0uZb0dX+sBr6TweX8E6ww7NEZHfbGgyG3dC96lT3+KvIumYJ a2pHNaMx89bFDRs5vOiofn4dsANmVQfDXFAJtRDfcp0w1a3sKjTwLkfrR6krSBGEbdSV5AnE= X-Google-Smtp-Source: AA0mqf5zdCB8zDLqALqxeUrqXKtJdklvUaACudcSRvctdG4d1OSjkhAibLqVajh1x9fSGf32ro7DsfCFSxoA X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a17:902:8d90:b0:187:3bc8:fc56 with SMTP id v16-20020a1709028d9000b001873bc8fc56mr7837879plo.83.1668585576869; Tue, 15 Nov 2022 23:59:36 -0800 (PST) Date: Wed, 16 Nov 2022 07:59:26 +0000 In-Reply-To: <20221116075929.453876-1-jstultz@google.com> Mime-Version: 1.0 References: <20221116075929.453876-1-jstultz@google.com> X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221116075929.453876-2-jstultz@google.com> Subject: [PATCH v5 1/3] softirq: Add generic accessor to percpu softirq_pending data From: John Stultz To: LKML Cc: John Stultz , John Dias , "Connor O'Brien" , Rick Yiu , John Kacur , Qais Yousef , Chris Redpath , Abhijeet Dharmapurikar , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Thomas Gleixner , Heiko Carstens , Vasily Gorbik , Joel Fernandes , Alexander Gordeev , kernel-team@android.com, kernel test robot Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In a previous iteration of this patch series, I was checking: per_cpu(irq_stat, cpu).__softirq_pending which resulted in build errors on s390. This patch tries to create a generic accessor to this percpu softirq_pending data. This interface is inherently racy as its reading percpu data without a lock. However, being able to peek at the softirq pending data allows us to make better decisions about rt task placement vs just ignoring it. On s390 this call returns 0, which maybe isn't ideal but results in no functional change from what we do now. TODO: Heiko suggested changing s390 to use a proper per-cpu irqstat variable instead. Feedback or suggestions for better approach here would be welcome! Cc: John Dias Cc: Connor O'Brien Cc: Rick Yiu Cc: John Kacur Cc: Qais Yousef Cc: Chris Redpath Cc: Abhijeet Dharmapurikar Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Thomas Gleixner Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Joel Fernandes Cc: Alexander Gordeev Cc: kernel-team@android.com Reported-by: kernel test robot Signed-off-by: John Stultz --- arch/s390/include/asm/hardirq.h | 6 ++++++ include/linux/interrupt.h | 11 +++++++++++ 2 files changed, 17 insertions(+) diff --git a/arch/s390/include/asm/hardirq.h b/arch/s390/include/asm/hardir= q.h index 58668ffb5488..cd9cc11588ab 100644 --- a/arch/s390/include/asm/hardirq.h +++ b/arch/s390/include/asm/hardirq.h @@ -16,6 +16,12 @@ #define local_softirq_pending() (S390_lowcore.softirq_pending) #define set_softirq_pending(x) (S390_lowcore.softirq_pending =3D (x)) #define or_softirq_pending(x) (S390_lowcore.softirq_pending |=3D (x)) +/* + * Not sure what the right thing is here for s390, + * but returning 0 will result in no logical change + * from what happens now + */ +#define __cpu_softirq_pending(x) (0) =20 #define __ARCH_IRQ_STAT #define __ARCH_IRQ_EXIT_IRQS_DISABLED diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index a92bce40b04b..a749a8663841 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -527,6 +527,17 @@ DECLARE_STATIC_KEY_FALSE(force_irqthreads_key); #define set_softirq_pending(x) (__this_cpu_write(local_softirq_pending_ref= , (x))) #define or_softirq_pending(x) (__this_cpu_or(local_softirq_pending_ref, (x= ))) =20 +/** + * __cpu_softirq_pending() - Checks to see if softirq is pending on a cpu + * + * This helper is inherently racy, as we're accessing per-cpu data w/o loc= ks. + * But peeking at the flag can still be useful when deciding where to plac= e a + * task. + */ +static inline u32 __cpu_softirq_pending(int cpu) +{ + return (u32)per_cpu(local_softirq_pending_ref, cpu); +} #endif /* local_softirq_pending */ =20 /* Some architectures might implement lazy enabling/disabling of --=20 2.38.1.431.g37b22c650d-goog From nobody Mon Apr 13 18:44:07 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E07F2C433FE for ; Wed, 16 Nov 2022 07:59:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232959AbiKPH7t (ORCPT ); Wed, 16 Nov 2022 02:59:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232845AbiKPH7k (ORCPT ); Wed, 16 Nov 2022 02:59:40 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44FA6C7 for ; Tue, 15 Nov 2022 23:59:39 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id x11-20020a056a000bcb00b0056c6ec11eefso9352997pfu.14 for ; Tue, 15 Nov 2022 23:59:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1Yx3hruW+9J71DEKe9UfpU1h/qKMqxTS0D2Vr6oFr1g=; b=BeKPVtw938FgpE2fj5tbX0/xEYCMV7j4HPSZT9yK4HfbwlBapDncQLYGz0imOhWMRo nZ23AgSsBznN+yiJ4hnMJgwNvo8W3lmSmCVppq9zzTAdcjIs9VOHrTDisssRdWupkUiN GfG+lIvkJGiJ3kpfkc2xrnx9AebesJNqVqY6f0ovXKFavDz4SclJCEUeaF3sMtRm6DBh p/0ZhckxElVomontjl4KA9uZ9+y64hN8H9pdQP3FbgiwFsFKpl6rgYL7BSFpw3vBjd/v r4wJ9qGpBddA1NG9ocHfHdaM+6pQVlHzGnlwaxzxbQCIAwssvG2H2QV70xnmXU9luvVn G7jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1Yx3hruW+9J71DEKe9UfpU1h/qKMqxTS0D2Vr6oFr1g=; b=jkruDA4bmNf6P6Kt42Y88A6lACz+5zMJjq2I8B7wkPt7NfDvAt+p/Aoit1g0T46RO7 NoBG5igJpvg0pkj4cKZQLBrhvaNaYFhF++XpdaO+04c7Q2t0lE8ukWdFljs6hC3jVXsn Q4H/FWjEOX9RtDz4VpfgB5DYe/t+yh7BhqJ6ZsGGd3qtMW6MmAfEjy/jE6etB86FRk+Q HuIGE9ggO2QMzRJxl6hcGSGR5RW3AUyXx/lRubxSNzgVztrMCSzo0+SNtcqbEf/3mKR+ nOWukmZrLC9PDD2iGo/bRRDSHZDKqiUqlOSkEv+iSqPMA2U/z0r8btlud3av3tZjzumC yfmQ== X-Gm-Message-State: ANoB5pnika6DibfLvtZiKELRCDCTOw2t4vZ8YhN+aKEwd34xRzhX9/oT nKRq5xSbgP1qeA8YuwzoWaaoSIrOTvmVGyE8NeWTCCyagMpBtMuaIM1Rt/O5jmgfrAf6IMhAd9j Psm2rRL9fiky754gNnrNv2LXEqqnMN8nfr9tHDG1BscInlV3icl9vX/2RgOC5lFRMfL3e9SQ= X-Google-Smtp-Source: AA0mqf5REgd3La0c0BUKsNCQTlYjigyvXiB15pYfzbJ/8LOJjZ9R8v//NstnmmHG+svikd4dvpNnclE2cDnr X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a17:902:b183:b0:186:d5b9:fbcd with SMTP id s3-20020a170902b18300b00186d5b9fbcdmr8083893plr.64.1668585578685; Tue, 15 Nov 2022 23:59:38 -0800 (PST) Date: Wed, 16 Nov 2022 07:59:27 +0000 In-Reply-To: <20221116075929.453876-1-jstultz@google.com> Mime-Version: 1.0 References: <20221116075929.453876-1-jstultz@google.com> X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221116075929.453876-3-jstultz@google.com> Subject: [PATCH v5 2/3] sched: Avoid placing RT threads on cores handling long softirqs From: John Stultz To: LKML Cc: "Connor O'Brien" , John Dias , Rick Yiu , John Kacur , Qais Yousef , Chris Redpath , Abhijeet Dharmapurikar , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Thomas Gleixner , Heiko Carstens , Vasily Gorbik , Joel Fernandes , Alexander Gordeev , kernel-team@android.com, "J . Avila" , John Stultz Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Connor O'Brien In certain audio use cases, scheduling RT threads on cores that are handling softirqs can lead to glitches. Prevent this behavior in cases where the softirq is likely to take a long time. To avoid unnecessary migrations, the old behavior is preserved for RCU, SCHED and TIMER irqs which are expected to be relatively quick. This patch reworks and combines two related changes originally by John Dias Cc: John Dias Cc: Connor O'Brien Cc: Rick Yiu Cc: John Kacur Cc: Qais Yousef Cc: Chris Redpath Cc: Abhijeet Dharmapurikar Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Thomas Gleixner Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Joel Fernandes Cc: Alexander Gordeev Cc: kernel-team@android.com Signed-off-by: John Dias [elavila: Port to mainline, amend commit text] Signed-off-by: J. Avila [connoro: Reworked, simplified, and merged two patches together] Signed-off-by: Connor O'Brien [jstultz: Further simplified and fixed issues, reworded commit message, removed arm64-isms] Signed-off-by: John Stultz --- v2: * Reformatted Kconfig entry to match coding style (Reported-by: Randy Dunlap ) * Made rt_task_fits_capacity_and_may_preempt static to avoid warnings (Reported-by: kernel test robot ) * Rework to use preempt_count and drop kconfig dependency on ARM64 v3: * Use introduced __cpu_softirq_pending() to avoid s390 build issues (Reported-by: kernel test robot ) v4: * Drop TASKLET_SOFTIRQ from LONG_SOFTIRQS (suggested by Qais) * Depend on !PREEMPT_RT (Suggested by Qais) * Larger simplification of logic (suggested by Qais) * Rework LONG_SOFTIRQS to use BIT() macros * Rename task_may_preempt() to cpu_busy_with_softirqs() v5: * Conditionalize active_softirqs handling (suggested by Alexander Gordeev ) * Reorder rt_task_fits_cpu to have the "fast" function first (Suggested by Alexander Gordeev ) * Fix bug I introduced in v2 condensing task_thread_info(task)->preempt_count to preempt_count() (Reported-by: Alexander Gordeev ) * Tweak comment discription to remove the vauge "slow" descriptor of softirqs being run by ksoftirqd (Suggested by Alexander Gordeev ) * Switch to using CONFIG_RT_SOFTIRQ_AWARE_SCHED (suggested by Joel Fernandes ) * Simplify cpu_busy_with_softirqs() logic as pointed out by Alexander Gordeev * Switch to using IS_ENABLED rather then defining my own macro (suggsted by Joel Fernandes ) --- include/linux/interrupt.h | 9 +++++++ init/Kconfig | 10 ++++++++ kernel/sched/rt.c | 49 ++++++++++++++++++++++++++++++++------- kernel/softirq.c | 17 ++++++++++++++ 4 files changed, 76 insertions(+), 9 deletions(-) diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index a749a8663841..7d09eb998d4c 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -582,6 +582,11 @@ enum * _ IRQ_POLL: irq_poll_cpu_dead() migrates the queue */ #define SOFTIRQ_HOTPLUG_SAFE_MASK (BIT(RCU_SOFTIRQ) | BIT(IRQ_POLL_SOFTIRQ= )) +/* Softirq's where the handling might be long: */ +#define LONG_SOFTIRQ_MASK (BIT(NET_TX_SOFTIRQ) | \ + BIT(NET_RX_SOFTIRQ) | \ + BIT(BLOCK_SOFTIRQ) | \ + BIT(IRQ_POLL_SOFTIRQ)) =20 /* map softirq index to softirq name. update 'softirq_to_name' in * kernel/softirq.c when adding a new softirq. @@ -618,6 +623,10 @@ extern void raise_softirq(unsigned int nr); =20 DECLARE_PER_CPU(struct task_struct *, ksoftirqd); =20 +#ifdef CONFIG_RT_SOFTIRQ_AWARE_SCHED +DECLARE_PER_CPU(u32, active_softirqs); +#endif + static inline struct task_struct *this_cpu_ksoftirqd(void) { return this_cpu_read(ksoftirqd); diff --git a/init/Kconfig b/init/Kconfig index abf65098f1b6..ce0f0be5759c 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1291,6 +1291,16 @@ config SCHED_AUTOGROUP desktop applications. Task group autogeneration is currently based upon task session. =20 +config RT_SOFTIRQ_AWARE_SCHED + bool "Improve RT scheduling during long softirq execution" + depends on SMP && !PREEMPT_RT + default n + help + Enable an optimization which tries to avoid placing RT tasks on CPUs + occupied by nonpreemptible tasks, such as a long softirq or CPUs + which may soon block preemptions, such as a CPU running a ksoftirq + thread which handles slow softirqs. + config SYSFS_DEPRECATED bool "Enable deprecated sysfs features to support old userspace tools" depends on SYSFS diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index ed2a47e4ddae..152347c4394c 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1595,6 +1595,32 @@ static void yield_task_rt(struct rq *rq) #ifdef CONFIG_SMP static int find_lowest_rq(struct task_struct *task); =20 +#ifdef CONFIG_RT_SOFTIRQ_AWARE_SCHED +/* + * Return whether the given cpu is currently non-preemptible + * while handling a potentially long softirq, or if the current + * task is likely to block preemptions soon because it is a + * ksoftirq thread that is handling softirqs. + */ +static bool cpu_busy_with_softirqs(int cpu) +{ + u32 softirqs =3D per_cpu(active_softirqs, cpu) | + __cpu_softirq_pending(cpu); + + return softirqs & LONG_SOFTIRQ_MASK; +} +#else +static bool cpu_busy_with_softirqs(int cpu) +{ + return false; +} +#endif /* CONFIG_RT_SOFTIRQ_AWARE_SCHED */ + +static bool rt_task_fits_cpu(struct task_struct *p, int cpu) +{ + return rt_task_fits_capacity(p, cpu) && !cpu_busy_with_softirqs(cpu); +} + static int select_task_rq_rt(struct task_struct *p, int cpu, int flags) { @@ -1633,22 +1659,24 @@ select_task_rq_rt(struct task_struct *p, int cpu, i= nt flags) * This test is optimistic, if we get it wrong the load-balancer * will have to sort it out. * - * We take into account the capacity of the CPU to ensure it fits the - * requirement of the task - which is only important on heterogeneous - * systems like big.LITTLE. + * We use rt_task_fits_cpu() to evaluate if the CPU is busy with + * potentially long-running softirq work, as well as take into + * account the capacity of the CPU to ensure it fits the + * requirement of the task - which is only important on + * heterogeneous systems like big.LITTLE. */ test =3D curr && unlikely(rt_task(curr)) && (curr->nr_cpus_allowed < 2 || curr->prio <=3D p->prio); =20 - if (test || !rt_task_fits_capacity(p, cpu)) { + if (test || !rt_task_fits_cpu(p, cpu)) { int target =3D find_lowest_rq(p); =20 /* * Bail out if we were forcing a migration to find a better * fitting CPU but our search failed. */ - if (!test && target !=3D -1 && !rt_task_fits_capacity(p, target)) + if (!test && target !=3D -1 && !rt_task_fits_cpu(p, target)) goto out_unlock; =20 /* @@ -1890,14 +1918,17 @@ static int find_lowest_rq(struct task_struct *task) return -1; /* No other targets possible */ =20 /* - * If we're on asym system ensure we consider the different capacities - * of the CPUs when searching for the lowest_mask. + * If we're using the softirq optimization or if we are + * on asym system, ensure we consider the softirq processing + * or different capacities of the CPUs when searching for the + * lowest_mask. */ - if (sched_asym_cpucap_active()) { + if (IS_ENABLED(CONFIG_RT_SOFTIRQ_AWARE_SCHED) || + sched_asym_cpucap_active()) { =20 ret =3D cpupri_find_fitness(&task_rq(task)->rd->cpupri, task, lowest_mask, - rt_task_fits_capacity); + rt_task_fits_cpu); } else { =20 ret =3D cpupri_find(&task_rq(task)->rd->cpupri, diff --git a/kernel/softirq.c b/kernel/softirq.c index c8a6913c067d..dd92ce8f771b 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -60,6 +60,21 @@ static struct softirq_action softirq_vec[NR_SOFTIRQS] __= cacheline_aligned_in_smp =20 DEFINE_PER_CPU(struct task_struct *, ksoftirqd); =20 +#ifdef CONFIG_RT_SOFTIRQ_AWARE_SCHED +/* + * active_softirqs -- per cpu, a mask of softirqs that are being handled, + * with the expectation that approximate answers are acceptable and theref= ore + * no synchronization. + */ +DEFINE_PER_CPU(u32, active_softirqs); +static inline void set_active_softirqs(u32 pending) +{ + __this_cpu_write(active_softirqs, pending); +} +#else /* CONFIG_RT_SOFTIRQ_AWARE_SCHED */ +static inline void set_active_softirqs(u32 pending) {}; +#endif /* CONFIG_RT_SOFTIRQ_AWARE_SCHED */ + const char * const softirq_to_name[NR_SOFTIRQS] =3D { "HI", "TIMER", "NET_TX", "NET_RX", "BLOCK", "IRQ_POLL", "TASKLET", "SCHED", "HRTIMER", "RCU" @@ -551,6 +566,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(= void) restart: /* Reset the pending bitmask before enabling irqs */ set_softirq_pending(0); + set_active_softirqs(pending); =20 local_irq_enable(); =20 @@ -580,6 +596,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(= void) pending >>=3D softirq_bit; } =20 + set_active_softirqs(0); if (!IS_ENABLED(CONFIG_PREEMPT_RT) && __this_cpu_read(ksoftirqd) =3D=3D current) rcu_softirq_qs(); --=20 2.38.1.431.g37b22c650d-goog From nobody Mon Apr 13 18:44:07 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99A43C433FE for ; Wed, 16 Nov 2022 08:00:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232970AbiKPIAB (ORCPT ); Wed, 16 Nov 2022 03:00:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232908AbiKPH7m (ORCPT ); Wed, 16 Nov 2022 02:59:42 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10FC5C7 for ; Tue, 15 Nov 2022 23:59:42 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id e128-20020a636986000000b0047075a1c725so8926470pgc.19 for ; Tue, 15 Nov 2022 23:59:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LdOYk9RhgVfGcLslo0PFudrqwmqg3H4kC0w388sZncY=; b=fr/JCq7r62KE0Vzw76H1bT9MaRv3VI3U4VSrGRUA3D9YlXCe2DUz51VM2lWrXOMUxV YVvBZTX2gKnpizrEuUVasgONXKXLzliBHpflI08C9h2pwa2NzO4q064Jgg6ZKBz6igZS I9JwaIyxHq8NVtKRz0QSHSCkkbKYzCXQIsScAG1CA3z51Y67t22Hx7StMioevPluwQDa 882XH8O/VafbYarNIkrtfLoqikiFo3UDW0rnkG0mNz8KsvhslrVhuOyDLWy5QEH8/pzd HHHIfpyQnTGTWeZAjbB1M+aC4xSsTawDgHC1jpHvqvNdu5z+IuWLtIrY3uupyafBSqv0 ifUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LdOYk9RhgVfGcLslo0PFudrqwmqg3H4kC0w388sZncY=; b=dPhPrn8VdVuiKAWl13KtVGuUsWLWjTqgls5xBROGoXBc3RPmZqTDg8ljvpKbaGSqs/ sS51DcUQ3AoiLpsdef3bNdACo3ILbOtQQzLKeLLBVl1ZB1YxUz2br/iY2LImD/hWm15o CP+n9p9Aq1d+3FM0rB1Uxi+SLkyTuKDzC0HG87VCJky1k/HfRV0yWqNX62rpE7vwWAWC rExbZQrDSgWWdBKKaY6FI7L8TXIH7SIYZQKN9iokyx4COp0F42uSVFBUBRXkQijr/QNf IDMbGEKEkWIc+tB3mRQKyl3yurcW6+jSJ90FNLj6YjugvKYPStBJ1KndqvLbMoIpaU2K TCBg== X-Gm-Message-State: ANoB5pmg+IhiwPYYzdjJ4/L7yspZ/g00/XX1XgQIk4RTvZc38jncyC+Z SkCHM7wVArN3RwgknsTpDPBye2hvILPMDONKqu+pe+c3NKxdw+67HKMoXkFyYUuFGHaat6Wjkzg rB5SqVFILSXXFfqPGYkquhwMSv7rX3fi0H9TqwHIdhjzgYZ1HiuzLNwnWhzWMrvSL+73D1lU= X-Google-Smtp-Source: AA0mqf6bh8609Vl5w5K4CFmBHb/c4y1gsGgnU4gCKQV5FPf5yeq4UY8W2edrceQwM+tG3UuG5lmDBiDT0Uax X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a17:90a:a392:b0:20a:fee1:8f69 with SMTP id x18-20020a17090aa39200b0020afee18f69mr225380pjp.0.1668585580398; Tue, 15 Nov 2022 23:59:40 -0800 (PST) Date: Wed, 16 Nov 2022 07:59:28 +0000 In-Reply-To: <20221116075929.453876-1-jstultz@google.com> Mime-Version: 1.0 References: <20221116075929.453876-1-jstultz@google.com> X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221116075929.453876-4-jstultz@google.com> Subject: [PATCH v5 3/3] softirq: defer softirq processing to ksoftirqd if CPU is busy with RT From: John Stultz To: LKML Cc: Pavankumar Kondeti , John Dias , "Connor O'Brien" , Rick Yiu , John Kacur , Qais Yousef , Chris Redpath , Abhijeet Dharmapurikar , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Thomas Gleixner , Heiko Carstens , Vasily Gorbik , Joel Fernandes , Alexander Gordeev , kernel-team@android.com, Satya Durga Srinivasu Prabhala , "J . Avila" , John Stultz Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Pavankumar Kondeti Defer the softirq processing to ksoftirqd if a RT task is running or queued on the current CPU. This complements the RT task placement algorithm which tries to find a CPU that is not currently busy with softirqs. Currently NET_TX, NET_RX, BLOCK and IRQ_POLL softirqs are only deferred as they can potentially run for long time. Additionally, this patch stubs out ksoftirqd_running() logic, in the CONFIG_RT_SOFTIRQ_AWARE_SCHED case, as deferring potentially long-running softirqs will cause the logic to not process shorter-running softirqs immediately. By stubbing it out the potentially long running softirqs are deferred, but the shorter running ones can still run immediately. This patch includes folded-in fixes by: Lingutla Chandrasekhar Satya Durga Srinivasu Prabhala J. Avila Cc: John Dias Cc: Connor O'Brien Cc: Rick Yiu Cc: John Kacur Cc: Qais Yousef Cc: Chris Redpath Cc: Abhijeet Dharmapurikar Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Thomas Gleixner Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Joel Fernandes Cc: Alexander Gordeev Cc: kernel-team@android.com Signed-off-by: Pavankumar Kondeti [satyap@codeaurora.org: trivial merge conflict resolution.] Signed-off-by: Satya Durga Srinivasu Prabhala [elavila: Port to mainline, squash with bugfix] Signed-off-by: J. Avila [jstultz: Rebase to linus/HEAD, minor rearranging of code, included bug fix Reported-by: Qais Yousef ] Signed-off-by: John Stultz --- v4: * Fix commit message to accurately note long-running softirqs (suggested by Qais) * Switch to using rt_task(current) (suggested by Qais) v5: * Switch to using CONFIG_RT_SOFTIRQ_AWARE_SCHED (suggested by Joel Fernandes ) --- kernel/softirq.c | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index dd92ce8f771b..5db2afd0be68 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -95,6 +95,7 @@ static void wakeup_softirqd(void) wake_up_process(tsk); } =20 +#ifndef CONFIG_RT_SOFTIRQ_AWARE_SCHED /* * If ksoftirqd is scheduled, we do not want to process pending softirqs * right now. Let ksoftirqd handle this at its own rate, to get fairness, @@ -109,6 +110,9 @@ static bool ksoftirqd_running(unsigned long pending) return false; return tsk && task_is_running(tsk) && !__kthread_should_park(tsk); } +#else +#define ksoftirqd_running(pending) (false) +#endif /* CONFIG_RT_SOFTIRQ_AWARE_SCHED */ =20 #ifdef CONFIG_TRACE_IRQFLAGS DEFINE_PER_CPU(int, hardirqs_enabled); @@ -540,6 +544,21 @@ static inline bool lockdep_softirq_start(void) { retur= n false; } static inline void lockdep_softirq_end(bool in_hardirq) { } #endif =20 +#ifdef CONFIG_RT_SOFTIRQ_AWARE_SCHED +static __u32 softirq_deferred_for_rt(__u32 *pending) +{ + __u32 deferred =3D 0; + + if (rt_task(current)) { + deferred =3D *pending & LONG_SOFTIRQ_MASK; + *pending &=3D ~LONG_SOFTIRQ_MASK; + } + return deferred; +} +#else +#define softirq_deferred_for_rt(x) (0) +#endif + asmlinkage __visible void __softirq_entry __do_softirq(void) { unsigned long end =3D jiffies + MAX_SOFTIRQ_TIME; @@ -547,6 +566,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(= void) int max_restart =3D MAX_SOFTIRQ_RESTART; struct softirq_action *h; bool in_hardirq; + __u32 deferred; __u32 pending; int softirq_bit; =20 @@ -558,14 +578,16 @@ asmlinkage __visible void __softirq_entry __do_softir= q(void) current->flags &=3D ~PF_MEMALLOC; =20 pending =3D local_softirq_pending(); + deferred =3D softirq_deferred_for_rt(&pending); =20 softirq_handle_begin(); + in_hardirq =3D lockdep_softirq_start(); account_softirq_enter(current); =20 restart: /* Reset the pending bitmask before enabling irqs */ - set_softirq_pending(0); + set_softirq_pending(deferred); set_active_softirqs(pending); =20 local_irq_enable(); @@ -604,13 +626,16 @@ asmlinkage __visible void __softirq_entry __do_softir= q(void) local_irq_disable(); =20 pending =3D local_softirq_pending(); + deferred =3D softirq_deferred_for_rt(&pending); + if (pending) { if (time_before(jiffies, end) && !need_resched() && --max_restart) goto restart; + } =20 + if (pending | deferred) wakeup_softirqd(); - } =20 account_softirq_exit(current); lockdep_softirq_end(in_hardirq); --=20 2.38.1.431.g37b22c650d-goog