From nobody Thu Apr 2 19:49:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 379DBC433FE for ; Mon, 3 Oct 2022 23:20:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229507AbiJCXUs (ORCPT ); Mon, 3 Oct 2022 19:20:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229561AbiJCXUn (ORCPT ); Mon, 3 Oct 2022 19:20:43 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B56E186CB for ; Mon, 3 Oct 2022 16:20:42 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id cp14-20020a056a00348e00b005604b9eb5aaso3917180pfb.7 for ; Mon, 03 Oct 2022 16:20:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=c2cO7Sy64Zhrp4qdC6wFKXRMlV4RIl42WCCMI78jqrM=; b=oKuEPo8fzinBKPuCyPOK02shSwOH4GNl+UA1yIV+WYtbFo9sv0kKty/FVZfrb/wZ3q 9bw+T4ylsRKutVSfSPf9dxrnkxzvLxyHu32/vbeNI0QtvQk1Ukcs0JMd0uPxbbnG9oez pBuCyeyp6+dYZrADXn7FcuDn866q0Ipw3edJQEUPHMfsaa+/Zr+0X/85220MR0QOs6QS zp9oB4+zeFQnGOvOx83+5aY1r5CJuC+/bMw2EvlgQJLtG+qsksXpHwE2YirCkOsrxB3a JddC5PpbRuIc2uXC7o5O+p6/Wjxch1udK5pP5dUfXcAh/9zkaC0UJdW4Q+XxQQEWQj27 LURw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=c2cO7Sy64Zhrp4qdC6wFKXRMlV4RIl42WCCMI78jqrM=; b=FreaxkbBD+zXee7kYG79/5jxbrSWMH0pqvZfulLE2lbxA5IAHUMcrF5Irj6r2Cy8/b 3VRCPPdPCIR/EEqNrPlEkvZPlrYrs47GcmOgF0PWN80+FhSBll536AWa9lDmMYqxvuVb 3+L2K+TGt8voDdP/J9d2HhdMxh/gzWmHii4p5jLY6VVzQ4gm6jwxfwa4b1QfFUqdYK4r TbIdJOpNsNugrLwsjn/lOcbujEb0l0p2Ba1H3mGTpsMY3AtRcBmg6GpoEDzoAz+I0VBh RL3pN6nM12ZHCECcwyesHYsV/mrfVBpOhLahc4eaucctLmp1HJCva0Lp0o0adlGhIDZJ 2fPg== X-Gm-Message-State: ACrzQf2lzPl6Ay0NevSTOMbmJ9lnARQSu3CsjVQ/9SfQGL8IyUnhnAAd LX9M6AOOFZMUgPAGmH9yOJO0+FocW5W1VkdBLmxGd+xs+COIj0iQqAR+GnrdRdH+G+19E5LRyIF v3ZFqr4C9sVJP5klRBhkn58keVw7QdE9KXwun210Z7l+kp6XhT34urSctcRMw6me4Sv/XExo= X-Google-Smtp-Source: AMsMyM4V9u6cPtlnNBhlU7Yzc0gm+3oXE20A2kEVPVUi4odHcUxHLBf8A5t4BxIAbzu+FKRWoI44RRIDH+8j X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a17:90a:cd06:b0:203:ae0e:6a21 with SMTP id d6-20020a17090acd0600b00203ae0e6a21mr86882pju.0.1664839241058; Mon, 03 Oct 2022 16:20:41 -0700 (PDT) Date: Mon, 3 Oct 2022 23:20:31 +0000 In-Reply-To: <20221003232033.3404802-1-jstultz@google.com> Mime-Version: 1.0 References: <20221003232033.3404802-1-jstultz@google.com> X-Mailer: git-send-email 2.38.0.rc1.362.ged0d419d3c-goog Message-ID: <20221003232033.3404802-2-jstultz@google.com> Subject: [RFC PATCH v4 1/3] softirq: Add generic accessor to percpu softirq_pending data From: John Stultz To: LKML Cc: John Stultz , John Dias , "Connor O'Brien" , Rick Yiu , John Kacur , Qais Yousef , Chris Redpath , Abhijeet Dharmapurikar , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Thomas Gleixner , Heiko Carstens , Vasily Gorbik , kernel-team@android.com, kernel test robot Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In a previous iteration of this patch series, I was checking: per_cpu(irq_stat, cpu).__softirq_pending which resulted in build errors on s390. This patch tries to create a generic accessor to this percpu softirq_pending data. This interface is inherently racy as its reading percpu data without a lock. However, being able to peek at the softirq pending data allows us to make better decisions about rt task placement vs just ignoring it. On s390 this call returns 0, which maybe isn't ideal but results in no functional change from what we do now. TODO: Heiko suggested changing s390 to use a proper per-cpu irqstat variable instead. Feedback or suggestions for better approach here would be welcome! Cc: John Dias Cc: Connor O'Brien Cc: Rick Yiu Cc: John Kacur Cc: Qais Yousef Cc: Chris Redpath Cc: Abhijeet Dharmapurikar Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Thomas Gleixner Cc: Heiko Carstens Cc: Vasily Gorbik Cc: kernel-team@android.com Reported-by: kernel test robot Signed-off-by: John Stultz --- arch/s390/include/asm/hardirq.h | 6 ++++++ include/linux/interrupt.h | 11 +++++++++++ 2 files changed, 17 insertions(+) diff --git a/arch/s390/include/asm/hardirq.h b/arch/s390/include/asm/hardir= q.h index 58668ffb5488..cd9cc11588ab 100644 --- a/arch/s390/include/asm/hardirq.h +++ b/arch/s390/include/asm/hardirq.h @@ -16,6 +16,12 @@ #define local_softirq_pending() (S390_lowcore.softirq_pending) #define set_softirq_pending(x) (S390_lowcore.softirq_pending =3D (x)) #define or_softirq_pending(x) (S390_lowcore.softirq_pending |=3D (x)) +/* + * Not sure what the right thing is here for s390, + * but returning 0 will result in no logical change + * from what happens now + */ +#define __cpu_softirq_pending(x) (0) =20 #define __ARCH_IRQ_STAT #define __ARCH_IRQ_EXIT_IRQS_DISABLED diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index a92bce40b04b..a749a8663841 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -527,6 +527,17 @@ DECLARE_STATIC_KEY_FALSE(force_irqthreads_key); #define set_softirq_pending(x) (__this_cpu_write(local_softirq_pending_ref= , (x))) #define or_softirq_pending(x) (__this_cpu_or(local_softirq_pending_ref, (x= ))) =20 +/** + * __cpu_softirq_pending() - Checks to see if softirq is pending on a cpu + * + * This helper is inherently racy, as we're accessing per-cpu data w/o loc= ks. + * But peeking at the flag can still be useful when deciding where to plac= e a + * task. + */ +static inline u32 __cpu_softirq_pending(int cpu) +{ + return (u32)per_cpu(local_softirq_pending_ref, cpu); +} #endif /* local_softirq_pending */ =20 /* Some architectures might implement lazy enabling/disabling of --=20 2.38.0.rc1.362.ged0d419d3c-goog From nobody Thu Apr 2 19:49:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A38BEC433FE for ; Mon, 3 Oct 2022 23:21:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229828AbiJCXVA (ORCPT ); Mon, 3 Oct 2022 19:21:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229614AbiJCXUp (ORCPT ); Mon, 3 Oct 2022 19:20:45 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 850221A07C for ; Mon, 3 Oct 2022 16:20:43 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id cb7-20020a056a00430700b00561b86e0265so737765pfb.13 for ; Mon, 03 Oct 2022 16:20:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=pDv9MbuxBmAy2p481/S839KuNUaJqK+qEEyKDH/P8jg=; b=YSUB9TS7tzwXSKdgDan4fTK5RCStmviw55WtSOVirbTzeNG9zZgycj6x093YZnliQa eKRVGd/QYPTYv+xmQAdqRK4jkf+X9gjBw+JuXexA21V6QeoInDKIwudm9wUDJZgd/Hbb rJoCOE43zTaYeGP2Tu1z5QThKxrTVGugSDoUonSrMdkFvoCH50QOhw4kfWQ0XOZWD1rp XyHGRGdYj37CokAojWtXOFOTiSK+mjo/oKtn+g8351cEgRlowKRRDuGsWa7rekrEgaf3 SdWpCpG39DCBhaSgLEdlciqAA9TamZkAMV18EZovFiXEbONSRqcHC9iwRHiRo+TZLO6S TYwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=pDv9MbuxBmAy2p481/S839KuNUaJqK+qEEyKDH/P8jg=; b=WpCLQJUmTX0RwzSfxCowDweoUNXGs7dwLHiWf9HDYRqX/FeS4EyHyEdJxAGaLP1bJs 7LQHtcoOJR0pczvGhuJiboZpsQ77UAEVqid1/vlicTFQ6vxAif6VmHwp0UBCM3Senrjb z4OEYSKdPcpDYA0whH+xUj2tIayM3hjy06LgbTpUQjys/p7p3lluAp5Pm0dmYmNciYps hhjRsd11kSi1g9/I9LeBIRP3Aqi/KABX9bfet9lvtOXG+2y+OhduwGgRF3k1tLAuUD7N 9BBqQQOSSOOLAbIH8qE+8QTTX2crgTI/SKwetM8BobE2WmPz2tC/zjY3L4fED/jYENd+ z82Q== X-Gm-Message-State: ACrzQf34SXSi98r3aFB/7YqvuxWHfvE7SC0ojOZ+/wzELCczu8n4tCI2 xZVyZLTYwibgROsgol2bO4utNwz0f6SAU+HAxlYpumgSu2eL95BXsga6/re1baeDR1Et2Eip3sK 10Lt1pQtYVoNdangN+daIw2bjdjFvHaK2sUw25AmVvuI0Y1KKDZXADGvHKc3wg4oGnsx3+3g= X-Google-Smtp-Source: AMsMyM7/GszLcL2nhEgb3xESlCVVH8SUddXYPRoi9TxQqvwlmio10xzBe1RFug1usvzBWmxXkdzhKhjbORe7 X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:aa7:8543:0:b0:54b:6ea4:7a12 with SMTP id y3-20020aa78543000000b0054b6ea47a12mr24678596pfn.33.1664839242759; Mon, 03 Oct 2022 16:20:42 -0700 (PDT) Date: Mon, 3 Oct 2022 23:20:32 +0000 In-Reply-To: <20221003232033.3404802-1-jstultz@google.com> Mime-Version: 1.0 References: <20221003232033.3404802-1-jstultz@google.com> X-Mailer: git-send-email 2.38.0.rc1.362.ged0d419d3c-goog Message-ID: <20221003232033.3404802-3-jstultz@google.com> Subject: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs From: John Stultz To: LKML Cc: "Connor O'Brien" , John Dias , Rick Yiu , John Kacur , Qais Yousef , Chris Redpath , Abhijeet Dharmapurikar , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Thomas Gleixner , kernel-team@android.com, "J . Avila" , John Stultz Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Connor O'Brien In certain audio use cases, scheduling RT threads on cores that are handling softirqs can lead to glitches. Prevent this behavior in cases where the softirq is likely to take a long time. To avoid unnecessary migrations, the old behavior is preserved for RCU, SCHED and TIMER irqs which are expected to be relatively quick. This patch reworks and combines two related changes originally by John Dias Cc: John Dias Cc: Connor O'Brien Cc: Rick Yiu Cc: John Kacur Cc: Qais Yousef Cc: Chris Redpath Cc: Abhijeet Dharmapurikar Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Thomas Gleixner Cc: kernel-team@android.com Signed-off-by: John Dias [elavila: Port to mainline, amend commit text] Signed-off-by: J. Avila [connoro: Reworked, simplified, and merged two patches together] Signed-off-by: Connor O'Brien [jstultz: Further simplified and fixed issues, reworded commit message, removed arm64-isms] Signed-off-by: John Stultz --- v2: * Reformatted Kconfig entry to match coding style (Reported-by: Randy Dunlap ) * Made rt_task_fits_capacity_and_may_preempt static to avoid warnings (Reported-by: kernel test robot ) * Rework to use preempt_count and drop kconfig dependency on ARM64 v3: * Use introduced __cpu_softirq_pending() to avoid s390 build issues (Reported-by: kernel test robot ) v4: * Drop TASKLET_SOFTIRQ from LONG_SOFTIRQS (suggested by Qais) * Depend on !PREEMPT_RT (Suggested by Qais) * Larger simplification of logic (suggested by Qais) * Rework LONG_SOFTIRQS to use BIT() macros * Rename task_may_preempt() to cpu_busy_with_softirqs() --- include/linux/interrupt.h | 6 ++++ init/Kconfig | 10 +++++++ kernel/sched/rt.c | 61 +++++++++++++++++++++++++++++++++------ kernel/softirq.c | 9 ++++++ 4 files changed, 77 insertions(+), 9 deletions(-) diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index a749a8663841..e3a4add67e8c 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -582,6 +582,11 @@ enum * _ IRQ_POLL: irq_poll_cpu_dead() migrates the queue */ #define SOFTIRQ_HOTPLUG_SAFE_MASK (BIT(RCU_SOFTIRQ) | BIT(IRQ_POLL_SOFTIRQ= )) +/* Softirq's where the handling might be long: */ +#define LONG_SOFTIRQ_MASK (BIT(NET_TX_SOFTIRQ) | \ + BIT(NET_RX_SOFTIRQ) | \ + BIT(BLOCK_SOFTIRQ) | \ + BIT(IRQ_POLL_SOFTIRQ)) =20 /* map softirq index to softirq name. update 'softirq_to_name' in * kernel/softirq.c when adding a new softirq. @@ -617,6 +622,7 @@ extern void raise_softirq_irqoff(unsigned int nr); extern void raise_softirq(unsigned int nr); =20 DECLARE_PER_CPU(struct task_struct *, ksoftirqd); +DECLARE_PER_CPU(u32, active_softirqs); =20 static inline struct task_struct *this_cpu_ksoftirqd(void) { diff --git a/init/Kconfig b/init/Kconfig index 532362fcfe31..3d1de6edcfa1 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1284,6 +1284,16 @@ config SCHED_AUTOGROUP desktop applications. Task group autogeneration is currently based upon task session. =20 +config RT_SOFTIRQ_OPTIMIZATION + bool "Improve RT scheduling during long softirq execution" + depends on SMP && !PREEMPT_RT + default n + help + Enable an optimization which tries to avoid placing RT tasks on CPUs + occupied by nonpreemptible tasks, such as a long softirq or CPUs + which may soon block preemptions, such as a CPU running a ksoftirq + thread which handles slow softirqs. + config SYSFS_DEPRECATED bool "Enable deprecated sysfs features to support old userspace tools" depends on SYSFS diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 55f39c8f4203..3c628db807c8 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1599,6 +1599,44 @@ static void yield_task_rt(struct rq *rq) #ifdef CONFIG_SMP static int find_lowest_rq(struct task_struct *task); =20 +#ifdef CONFIG_RT_SOFTIRQ_OPTIMIZATION +#define __use_softirq_opt 1 +/* + * Return whether the given cpu is currently non-preemptible + * while handling a potentially long softirq, or if the current + * task is likely to block preemptions soon because it is a + * ksoftirq thread that is handling slow softirq. + */ +static bool cpu_busy_with_softirqs(int cpu) +{ + u32 softirqs =3D per_cpu(active_softirqs, cpu) | + __cpu_softirq_pending(cpu); + struct task_struct *cpu_ksoftirqd =3D per_cpu(ksoftirqd, cpu); + struct task_struct *curr; + struct rq *rq =3D cpu_rq(cpu); + int ret; + + rcu_read_lock(); + curr =3D READ_ONCE(rq->curr); /* unlocked access */ + ret =3D (softirqs & LONG_SOFTIRQ_MASK) && + (curr =3D=3D cpu_ksoftirqd || + preempt_count() & SOFTIRQ_MASK); + rcu_read_unlock(); + return ret; +} +#else +#define __use_softirq_opt 0 +static bool cpu_busy_with_softirqs(int cpu) +{ + return false; +} +#endif /* CONFIG_RT_SOFTIRQ_OPTIMIZATION */ + +static bool rt_task_fits_cpu(struct task_struct *p, int cpu) +{ + return !cpu_busy_with_softirqs(cpu) && rt_task_fits_capacity(p, cpu); +} + static int select_task_rq_rt(struct task_struct *p, int cpu, int flags) { @@ -1637,22 +1675,24 @@ select_task_rq_rt(struct task_struct *p, int cpu, i= nt flags) * This test is optimistic, if we get it wrong the load-balancer * will have to sort it out. * - * We take into account the capacity of the CPU to ensure it fits the - * requirement of the task - which is only important on heterogeneous - * systems like big.LITTLE. + * We use rt_task_fits_cpu() to evaluate if the CPU is busy with + * potentially long-running softirq work, as well as take into + * account the capacity of the CPU to ensure it fits the + * requirement of the task - which is only important on + * heterogeneous systems like big.LITTLE. */ test =3D curr && unlikely(rt_task(curr)) && (curr->nr_cpus_allowed < 2 || curr->prio <=3D p->prio); =20 - if (test || !rt_task_fits_capacity(p, cpu)) { + if (test || !rt_task_fits_cpu(p, cpu)) { int target =3D find_lowest_rq(p); =20 /* * Bail out if we were forcing a migration to find a better * fitting CPU but our search failed. */ - if (!test && target !=3D -1 && !rt_task_fits_capacity(p, target)) + if (!test && target !=3D -1 && !rt_task_fits_cpu(p, target)) goto out_unlock; =20 /* @@ -1894,14 +1934,17 @@ static int find_lowest_rq(struct task_struct *task) return -1; /* No other targets possible */ =20 /* - * If we're on asym system ensure we consider the different capacities - * of the CPUs when searching for the lowest_mask. + * If we're using the softirq optimization or if we are + * on asym system, ensure we consider the softirq processing + * or different capacities of the CPUs when searching for the + * lowest_mask. */ - if (static_branch_unlikely(&sched_asym_cpucapacity)) { + if (__use_softirq_opt || + static_branch_unlikely(&sched_asym_cpucapacity)) { =20 ret =3D cpupri_find_fitness(&task_rq(task)->rd->cpupri, task, lowest_mask, - rt_task_fits_capacity); + rt_task_fits_cpu); } else { =20 ret =3D cpupri_find(&task_rq(task)->rd->cpupri, diff --git a/kernel/softirq.c b/kernel/softirq.c index c8a6913c067d..35ee79dd8786 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -60,6 +60,13 @@ static struct softirq_action softirq_vec[NR_SOFTIRQS] __= cacheline_aligned_in_smp =20 DEFINE_PER_CPU(struct task_struct *, ksoftirqd); =20 +/* + * active_softirqs -- per cpu, a mask of softirqs that are being handled, + * with the expectation that approximate answers are acceptable and theref= ore + * no synchronization. + */ +DEFINE_PER_CPU(u32, active_softirqs); + const char * const softirq_to_name[NR_SOFTIRQS] =3D { "HI", "TIMER", "NET_TX", "NET_RX", "BLOCK", "IRQ_POLL", "TASKLET", "SCHED", "HRTIMER", "RCU" @@ -551,6 +558,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(= void) restart: /* Reset the pending bitmask before enabling irqs */ set_softirq_pending(0); + __this_cpu_write(active_softirqs, pending); =20 local_irq_enable(); =20 @@ -580,6 +588,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(= void) pending >>=3D softirq_bit; } =20 + __this_cpu_write(active_softirqs, 0); if (!IS_ENABLED(CONFIG_PREEMPT_RT) && __this_cpu_read(ksoftirqd) =3D=3D current) rcu_softirq_qs(); --=20 2.38.0.rc1.362.ged0d419d3c-goog From nobody Thu Apr 2 19:49:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C65C3C433F5 for ; Mon, 3 Oct 2022 23:21:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229959AbiJCXVC (ORCPT ); Mon, 3 Oct 2022 19:21:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229671AbiJCXUr (ORCPT ); Mon, 3 Oct 2022 19:20:47 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E56CB1CFE0 for ; Mon, 3 Oct 2022 16:20:45 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id nl8-20020a17090b384800b00205f930565cso5104145pjb.2 for ; Mon, 03 Oct 2022 16:20:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=OpL44AhoDepjhQgCA7XT7WjFsWnCkxKd4LK73L1sblU=; b=i1qeAFa2M86DdbqZK7B4EvOFe4XIKOy/5pQ3Y7flv0mtafGoRocY7Evwe7GHmgQts5 iwl8Xwp/4C3t+JYsBuZv5SUkiEyFhldJKAdy3iDKueeI4VAytlMjeEDYxdm2n6gD8/0p QE5Y21bp3fpZJfYrWMhOsMlCvRccwrskJow/jJOzFbEe68GrmUZ+jPOIqrMJXd6gIh1p YOt/dak6KmXO9QaoDN6bjizpSArmAlXLDRmDr7yh22KK+8GpARiz5zZ7j+ztSU9ZAzm1 3ODNzM04sskJ1Ho6SE9mAX+hKfjk1CtbUgSlfuHjJQsP9B/O7EjerSXnmt1sUPBsC9uu vlBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=OpL44AhoDepjhQgCA7XT7WjFsWnCkxKd4LK73L1sblU=; b=llZWoB35OYGXNPqpfALOmLgrUB/lkhRCitTaE8lS03cyIO3bwarJgigYfjn9t/ryK/ nXnfX3AlKWKYT6VOZHkovAabcKoALuUpQ7Duf0wQzS+6a8RanrqsUAIXBj65reyEsNtN l1M5JdRtRYVOKiucaJSG9U6zpwJZIgpzklEKDuM35Z8QXeCz+Y1rbD01BfaRIP+0Pyr8 U2SgY0UnA2JCkqvmEF8AmCHA1fR3QlZRJvhyVKdviAfK0d0EdMPHgbeQBNpT55pTTHfk bt9PWaezUTuuYYkDFVgugyGoznhyHCzsJgIk9cyEcYoewfYBOAAvWg6er5pXY5GrFm3q AE6g== X-Gm-Message-State: ACrzQf3LKnhPD6cwkUhmL407fmhYOkw/QeFZzNdAFD2SKcgXbSv+Nvtq GfMk+UYW4Y5pCfxMVX3RjFr0HyrrMQNRUmS7F4O+KXBrWzzZbPR5LZJ3RBWylJo3ArMGJbaPFa4 gd1y77Ass/RbulwgkkUakn1uVZrzpWrrYGzbgSXcb3WwjXeMQ7Q9f+NNhY+V8y6pmvUNOqZk= X-Google-Smtp-Source: AMsMyM69kuB/EYt+LGZsbDdPxSn7Kjy2BTNT6+AibEbFFPaee51sYFMff+desTHfSJUV6MCreqmsQzKnnzb0 X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a17:902:d2c6:b0:17f:592b:35dd with SMTP id n6-20020a170902d2c600b0017f592b35ddmr7784704plc.172.1664839244526; Mon, 03 Oct 2022 16:20:44 -0700 (PDT) Date: Mon, 3 Oct 2022 23:20:33 +0000 In-Reply-To: <20221003232033.3404802-1-jstultz@google.com> Mime-Version: 1.0 References: <20221003232033.3404802-1-jstultz@google.com> X-Mailer: git-send-email 2.38.0.rc1.362.ged0d419d3c-goog Message-ID: <20221003232033.3404802-4-jstultz@google.com> Subject: [RFC PATCH v4 3/3] softirq: defer softirq processing to ksoftirqd if CPU is busy with RT From: John Stultz To: LKML Cc: Pavankumar Kondeti , John Dias , "Connor O'Brien" , Rick Yiu , John Kacur , Qais Yousef , Chris Redpath , Abhijeet Dharmapurikar , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Thomas Gleixner , kernel-team@android.com, Satya Durga Srinivasu Prabhala , "J . Avila" , John Stultz Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Pavankumar Kondeti Defer the softirq processing to ksoftirqd if a RT task is running or queued on the current CPU. This complements the RT task placement algorithm which tries to find a CPU that is not currently busy with softirqs. Currently NET_TX, NET_RX, BLOCK and IRQ_POLL softirqs are only deferred as they can potentially run for long time. Additionally, this patch stubs out ksoftirqd_running() logic, in the CONFIG_RT_SOFTIRQ_OPTIMIZATION case, as deferring potentially long-running softirqs will cause the logic to not process shorter-running softirqs immediately. By stubbing it out the potentially long running softirqs are deferred, but the shorter running ones can still run immediately. This patch includes folded-in fixes by: Lingutla Chandrasekhar Satya Durga Srinivasu Prabhala J. Avila Cc: John Dias Cc: Connor O'Brien Cc: Rick Yiu Cc: John Kacur Cc: Qais Yousef Cc: Chris Redpath Cc: Abhijeet Dharmapurikar Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Thomas Gleixner Cc: kernel-team@android.com Signed-off-by: Pavankumar Kondeti [satyap@codeaurora.org: trivial merge conflict resolution.] Signed-off-by: Satya Durga Srinivasu Prabhala [elavila: Port to mainline, squash with bugfix] Signed-off-by: J. Avila [jstultz: Rebase to linus/HEAD, minor rearranging of code, included bug fix Reported-by: Qais Yousef ] Signed-off-by: John Stultz --- v4: * Fix commit message to accurately note long-running softirqs (suggested by Qais) * Switch to using rt_task(current) (suggested by Qais) --- kernel/softirq.c | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index 35ee79dd8786..c8ce12bbab04 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -87,6 +87,7 @@ static void wakeup_softirqd(void) wake_up_process(tsk); } =20 +#ifndef CONFIG_RT_SOFTIRQ_OPTIMIZATION /* * If ksoftirqd is scheduled, we do not want to process pending softirqs * right now. Let ksoftirqd handle this at its own rate, to get fairness, @@ -101,6 +102,9 @@ static bool ksoftirqd_running(unsigned long pending) return false; return tsk && task_is_running(tsk) && !__kthread_should_park(tsk); } +#else +#define ksoftirqd_running(pending) (false) +#endif /* CONFIG_RT_SOFTIRQ_OPTIMIZATION */ =20 #ifdef CONFIG_TRACE_IRQFLAGS DEFINE_PER_CPU(int, hardirqs_enabled); @@ -532,6 +536,21 @@ static inline bool lockdep_softirq_start(void) { retur= n false; } static inline void lockdep_softirq_end(bool in_hardirq) { } #endif =20 +#ifdef CONFIG_RT_SOFTIRQ_OPTIMIZATION +static __u32 softirq_deferred_for_rt(__u32 *pending) +{ + __u32 deferred =3D 0; + + if (rt_task(current)) { + deferred =3D *pending & LONG_SOFTIRQ_MASK; + *pending &=3D ~LONG_SOFTIRQ_MASK; + } + return deferred; +} +#else +#define softirq_deferred_for_rt(x) (0) +#endif + asmlinkage __visible void __softirq_entry __do_softirq(void) { unsigned long end =3D jiffies + MAX_SOFTIRQ_TIME; @@ -539,6 +558,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(= void) int max_restart =3D MAX_SOFTIRQ_RESTART; struct softirq_action *h; bool in_hardirq; + __u32 deferred; __u32 pending; int softirq_bit; =20 @@ -550,14 +570,16 @@ asmlinkage __visible void __softirq_entry __do_softir= q(void) current->flags &=3D ~PF_MEMALLOC; =20 pending =3D local_softirq_pending(); + deferred =3D softirq_deferred_for_rt(&pending); =20 softirq_handle_begin(); + in_hardirq =3D lockdep_softirq_start(); account_softirq_enter(current); =20 restart: /* Reset the pending bitmask before enabling irqs */ - set_softirq_pending(0); + set_softirq_pending(deferred); __this_cpu_write(active_softirqs, pending); =20 local_irq_enable(); @@ -596,13 +618,16 @@ asmlinkage __visible void __softirq_entry __do_softir= q(void) local_irq_disable(); =20 pending =3D local_softirq_pending(); + deferred =3D softirq_deferred_for_rt(&pending); + if (pending) { if (time_before(jiffies, end) && !need_resched() && --max_restart) goto restart; + } =20 + if (pending | deferred) wakeup_softirqd(); - } =20 account_softirq_exit(current); lockdep_softirq_end(in_hardirq); --=20 2.38.0.rc1.362.ged0d419d3c-goog