From nobody Wed Sep 17 03:32:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC13AC4167B for ; Thu, 22 Dec 2022 22:12:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230206AbiLVWMx (ORCPT ); Thu, 22 Dec 2022 17:12:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229627AbiLVWMt (ORCPT ); Thu, 22 Dec 2022 17:12:49 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEB8121E28; Thu, 22 Dec 2022 14:12:48 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6209861DAE; Thu, 22 Dec 2022 22:12:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8D6ABC433EF; Thu, 22 Dec 2022 22:12:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1671747167; bh=6/3Z9VJ1qfYuDDjUhrIE3LXc4NT3VetuZ5KotX+3+58=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=axJZ+bqrgkFrKPJvPBzSSfx/QTBG34Fme1UeVDMTF+N3TEa3nbuov7nBj8Ff9ycRL rKXiubu2M5nCya5ElgsumAzZ8YvjVl9bCqLjXpolV+b+VVJbmqkOW2A/o2XqWbuYev G1cnGBCZtFIj10TMPRE6di3qTSGhHpYUnTfiPSedQO66GibXATNTcr+SXJU9x+9pCx tS8C8JHNfMPDaana1h1HVYkwpMkjKV7pDa7IlxIfK8PjQ5b4/9ViyMcQHqofW4V0Fd AtlnEVtE8ijGsEws3uF2IkfeITjTReDOUXi8bDcQ79F0soE1ZyMy3DPaWJyN6Vs8lf sMTspFS7s76Nw== From: Jakub Kicinski To: peterz@infradead.org, tglx@linutronix.de Cc: jstultz@google.com, edumazet@google.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Jakub Kicinski Subject: [PATCH 1/3] softirq: rename ksoftirqd_running() -> ksoftirqd_should_handle() Date: Thu, 22 Dec 2022 14:12:42 -0800 Message-Id: <20221222221244.1290833-2-kuba@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221222221244.1290833-1-kuba@kernel.org> References: <20221222221244.1290833-1-kuba@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" ksoftirqd_running() takes the high priority softirqs into consideration, so ksoftirqd_should_handle() seems like a better name. Signed-off-by: Jakub Kicinski --- kernel/softirq.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index c8a6913c067d..00b838d566c1 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -86,7 +86,7 @@ static void wakeup_softirqd(void) * unless we're doing some of the synchronous softirqs. */ #define SOFTIRQ_NOW_MASK ((1 << HI_SOFTIRQ) | (1 << TASKLET_SOFTIRQ)) -static bool ksoftirqd_running(unsigned long pending) +static bool ksoftirqd_should_handle(unsigned long pending) { struct task_struct *tsk =3D __this_cpu_read(ksoftirqd); =20 @@ -236,7 +236,7 @@ void __local_bh_enable_ip(unsigned long ip, unsigned in= t cnt) goto out; =20 pending =3D local_softirq_pending(); - if (!pending || ksoftirqd_running(pending)) + if (!pending || ksoftirqd_should_handle(pending)) goto out; =20 /* @@ -432,7 +432,7 @@ static inline bool should_wake_ksoftirqd(void) =20 static inline void invoke_softirq(void) { - if (ksoftirqd_running(local_softirq_pending())) + if (ksoftirqd_should_handle(local_softirq_pending())) return; =20 if (!force_irqthreads() || !__this_cpu_read(ksoftirqd)) { @@ -468,7 +468,7 @@ asmlinkage __visible void do_softirq(void) =20 pending =3D local_softirq_pending(); =20 - if (pending && !ksoftirqd_running(pending)) + if (pending && !ksoftirqd_should_handle(pending)) do_softirq_own_stack(); =20 local_irq_restore(flags); --=20 2.38.1 From nobody Wed Sep 17 03:32:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C466C4167B for ; Thu, 22 Dec 2022 22:12:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235235AbiLVWM4 (ORCPT ); Thu, 22 Dec 2022 17:12:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229666AbiLVWMt (ORCPT ); Thu, 22 Dec 2022 17:12:49 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35BF524BCD; Thu, 22 Dec 2022 14:12:49 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BFA8461DB1; Thu, 22 Dec 2022 22:12:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E8AA8C43392; Thu, 22 Dec 2022 22:12:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1671747168; bh=D5KSQncHbyZRvS0KOGabofpFDBc6Ejaiaupvz+j2wyA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N/XYYW2GuzaLDUnRG8/SOo/WZrhBXMZ9XxOtpoxV40Qx0lWnlbL4ZRqqyp8pyLo9A 2YPpejBNuVp0k0bzm826qR9jwXAuAD5Wq/nBU82scAxNYZf2LgXb5bxUivd7Nh33Rr Z17UpYi+6RtO4litzdZ2sVIcXTgxM0NCNeiSaI8l7Vitnxx0IWACG1bs99DDV9lgg+ +1f1zQ5KTTv8zLk5tarwUYsN4pB9TsaSGPfAOIXbq1T5vvA9MjlF6BF8JN69ZL9w3a dj1CE0klsCxKDMRIErGSwQzwWdY75b//Q1mmg2ra57QjUUOI9GXCTqLDyDksTLHwmi oi+8El0DcWohg== From: Jakub Kicinski To: peterz@infradead.org, tglx@linutronix.de Cc: jstultz@google.com, edumazet@google.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Jakub Kicinski Subject: [PATCH 2/3] softirq: avoid spurious stalls due to need_resched() Date: Thu, 22 Dec 2022 14:12:43 -0800 Message-Id: <20221222221244.1290833-3-kuba@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221222221244.1290833-1-kuba@kernel.org> References: <20221222221244.1290833-1-kuba@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" need_resched() added in commit c10d73671ad3 ("softirq: reduce latencies") does improve latency for real workloads (for example memcache). Unfortunately it triggers quite often even for non-network-heavy apps (~900 times a second on a loaded webserver), and in small fraction of cases whatever the scheduler decided to run will hold onto the CPU for the entire time slice. 10ms+ stalls on a machine which is not actually under overload cause erratic network behavior and spurious TCP retransmits. Typical end-to-end latency in a datacenter is < 200us so its common to set TCP timeout to 10ms or less. The intent of the need_resched() is to let a low latency application respond quickly and yield (to ksoftirqd). Put a time limit on this dance. Ignore the fact that ksoftirqd is RUNNING if we were trying to be nice and the application did not yield quickly. On a webserver loaded at 90% CPU this change reduces the numer of 8ms+ stalls the network softirq processing sees by around 10x (2/sec -> 0.2/sec). It also seems to reduce retransmissions by ~10% but the data is quite noisy. Signed-off-by: Jakub Kicinski --- kernel/softirq.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index 00b838d566c1..ad200d386ec1 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -59,6 +59,7 @@ EXPORT_PER_CPU_SYMBOL(irq_stat); static struct softirq_action softirq_vec[NR_SOFTIRQS] __cacheline_aligned_= in_smp; =20 DEFINE_PER_CPU(struct task_struct *, ksoftirqd); +static DEFINE_PER_CPU(unsigned long, overload_limit); =20 const char * const softirq_to_name[NR_SOFTIRQS] =3D { "HI", "TIMER", "NET_TX", "NET_RX", "BLOCK", "IRQ_POLL", @@ -89,10 +90,15 @@ static void wakeup_softirqd(void) static bool ksoftirqd_should_handle(unsigned long pending) { struct task_struct *tsk =3D __this_cpu_read(ksoftirqd); + unsigned long ov_limit; =20 if (pending & SOFTIRQ_NOW_MASK) return false; - return tsk && task_is_running(tsk) && !__kthread_should_park(tsk); + if (likely(!tsk || !task_is_running(tsk) || __kthread_should_park(tsk))) + return false; + + ov_limit =3D __this_cpu_read(overload_limit); + return time_is_after_jiffies(ov_limit); } =20 #ifdef CONFIG_TRACE_IRQFLAGS @@ -492,6 +498,9 @@ asmlinkage __visible void do_softirq(void) #define MAX_SOFTIRQ_TIME msecs_to_jiffies(2) #define MAX_SOFTIRQ_RESTART 10 =20 +#define SOFTIRQ_OVERLOAD_TIME msecs_to_jiffies(100) +#define SOFTIRQ_DEFER_TIME msecs_to_jiffies(2) + #ifdef CONFIG_TRACE_IRQFLAGS /* * When we run softirqs from irq_exit() and thus on the hardirq stack we n= eed @@ -588,10 +597,16 @@ asmlinkage __visible void __softirq_entry __do_softir= q(void) =20 pending =3D local_softirq_pending(); if (pending) { - if (time_before(jiffies, end) && !need_resched() && - --max_restart) + unsigned long limit; + + if (time_is_before_eq_jiffies(end) || !--max_restart) + limit =3D SOFTIRQ_OVERLOAD_TIME; + else if (need_resched()) + limit =3D SOFTIRQ_DEFER_TIME; + else goto restart; =20 + __this_cpu_write(overload_limit, jiffies + limit); wakeup_softirqd(); } =20 --=20 2.38.1 From nobody Wed Sep 17 03:32:44 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65FFEC4332F for ; Thu, 22 Dec 2022 22:13:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235692AbiLVWNA (ORCPT ); Thu, 22 Dec 2022 17:13:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229754AbiLVWMu (ORCPT ); Thu, 22 Dec 2022 17:12:50 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B86A10FDB; Thu, 22 Dec 2022 14:12:49 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2377C61DB4; Thu, 22 Dec 2022 22:12:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 518A6C433F2; Thu, 22 Dec 2022 22:12:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1671747168; bh=mQD2ZYw4YRduH/pzuxEiNaEpTsT1YNKaQ62QM19DZGQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XQqMj6Z4it5dSFIYft/H2Xzfs/wXS3w0orSN/1N6zaT/JSyNX+MYxUMbSYWJb9qYs MSKBbg7QoMfXqkYroZVfTbb7AAbk30WPU5M1BbT3jayFvOocStnYrnMcrVyj0QPZB0 9wn1QMIuHbt1FRas8Tec/sEx7+abSc2//V5Qaep3OMpz2jNJt4jWu4RjRr1z3kzjcm K6l5kMN0ViZaNaWxzH8bc65XUL49e+TYdhLQnVgqmjx6Op02k6yrBr5FfRDBqPzDBl BopxF+W1aJmuDKVqwz7T09xdXnCm+8FgHIjxNkfLtL8byI++EUno7vjf2agYZZ66yP Z35qEt0tuQneg== From: Jakub Kicinski To: peterz@infradead.org, tglx@linutronix.de Cc: jstultz@google.com, edumazet@google.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Jakub Kicinski Subject: [PATCH 3/3] softirq: don't yield if only expedited handlers are pending Date: Thu, 22 Dec 2022 14:12:44 -0800 Message-Id: <20221222221244.1290833-4-kuba@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221222221244.1290833-1-kuba@kernel.org> References: <20221222221244.1290833-1-kuba@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In networking we try to keep Tx packet queues small, so we limit how many bytes a socket may packetize and queue up. Tx completions (from NAPI) notify the sockets when packets have left the system (NIC Tx completion) and the socket schedules a tasklet to queue the next batch of frames. This leads to a situation where we go thru the softirq loop twice. First round we have pending =3D NET (from the NIC IRQ/NAPI), and the second iteration has pending =3D TASKLET (the socket tasklet). On two web workloads I looked at this condition accounts for 10% and 23% of all ksoftirqd wake ups respectively. We run NAPI which wakes some process up, we hit need_resched() and wake up ksoftirqd just to run the TSQ (TCP small queues) tasklet. Tweak the need_resched() condition to be ignored if all pending softIRQs are "non-deferred". The tasklet would run relatively soon, anyway, but once ksoftirqd is woken we're risking stalls. I did not see any negative impact on the latency in an RR test on a loaded machine with this change applied. Signed-off-by: Jakub Kicinski --- kernel/softirq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index ad200d386ec1..4ac59ffb0d55 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -601,7 +601,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(= void) =20 if (time_is_before_eq_jiffies(end) || !--max_restart) limit =3D SOFTIRQ_OVERLOAD_TIME; - else if (need_resched()) + else if (need_resched() && pending & ~SOFTIRQ_NOW_MASK) limit =3D SOFTIRQ_DEFER_TIME; else goto restart; --=20 2.38.1