From nobody Sun May 10 09:54:16 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9C2AC433EF for ; Fri, 13 May 2022 06:24:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377290AbiEMGYr (ORCPT ); Fri, 13 May 2022 02:24:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240920AbiEMGYl (ORCPT ); Fri, 13 May 2022 02:24:41 -0400 Received: from out30-54.freemail.mail.aliyun.com (out30-54.freemail.mail.aliyun.com [115.124.30.54]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AB50291E76 for ; Thu, 12 May 2022 23:24:38 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R291e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04357;MF=dtcccc@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0VD1tx8B_1652423067; Received: from localhost.localdomain(mailfrom:dtcccc@linux.alibaba.com fp:SMTPD_---0VD1tx8B_1652423067) by smtp.aliyun-inc.com(127.0.0.1); Fri, 13 May 2022 14:24:35 +0800 From: Tianchen Ding To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Cc: linux-kernel@vger.kernel.org Subject: [RFC PATCH] sched: Queue task on wakelist in the same llc if the wakee cpu is idle Date: Fri, 13 May 2022 14:24:27 +0800 Message-Id: <20220513062427.2375743-1-dtcccc@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We notice the commit 518cd6234178 ("sched: Only queue remote wakeups when crossing cache boundaries") disabled queuing tasks on wakelist when the cpus share llc. This is because, at that time, the scheduler must send IPIs to do ttwu_queue_wakelist. Nowadays, ttwu_queue_wakelist also supports TIF_POLLING, so this is not a problem now when the wakee cpu is in idle polling. Benefits: Queuing the task on idle cpu can help improving performance on waker cpu and utilization on wakee cpu, and further improve locality because the wakee cpu can handle its own rq. This patch helps improving rt on our real java workloads where wakeup happens frequently. Does this patch bring IPI flooding? For archs with TIF_POLLING_NRFLAG (e.g., x86), there will be no difference if the wakee cpu is idle polling. If the wakee cpu is idle but not polling, the later check_preempt_curr() will send IPI too. For archs without TIF_POLLING_NRFLAG (e.g., arm64), the IPI is unavoidable, since the later check_preempt_curr() will send IPI when wakee cpu is idle. Benchmark: running schbench -m 2 -t 8 on 8269CY: without patch: Latency percentiles (usec) 50.0000th: 10 75.0000th: 14 90.0000th: 16 95.0000th: 16 *99.0000th: 17 99.5000th: 20 99.9000th: 23 min=3D0, max=3D28 with patch: Latency percentiles (usec) 50.0000th: 6 75.0000th: 8 90.0000th: 9 95.0000th: 9 *99.0000th: 10 99.5000th: 10 99.9000th: 14 min=3D0, max=3D16 We've also tested unixbench and see about 10% improvement on Pipe-based Context Switching, and no performance regression on other test cases. For arm64, we've tested schbench and unixbench on Kunpeng920, the results show that, the improvement is not as obvious as on x86, and there's no performance regression. Signed-off-by: Tianchen Ding --- kernel/sched/core.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 51efaabac3e4..cae5011a8b1f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3820,6 +3820,9 @@ static inline bool ttwu_queue_cond(int cpu, int wake_= flags) if (!cpu_active(cpu)) return false; =20 + if (cpu =3D=3D smp_processor_id()) + return false; + /* * If the CPU does not share cache, then queue the task on the * remote rqs wakelist to avoid accessing remote data. @@ -3827,6 +3830,12 @@ static inline bool ttwu_queue_cond(int cpu, int wake= _flags) if (!cpus_share_cache(smp_processor_id(), cpu)) return true; =20 + /* + * If the CPU is idle, let itself do activation to improve utilization. + */ + if (available_idle_cpu(cpu)) + return true; + /* * If the task is descheduling and the only running task on the * CPU then use the wakelist to offload the task activation to @@ -3842,9 +3851,6 @@ static inline bool ttwu_queue_cond(int cpu, int wake_= flags) static bool ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_f= lags) { if (sched_feat(TTWU_QUEUE) && ttwu_queue_cond(cpu, wake_flags)) { - if (WARN_ON_ONCE(cpu =3D=3D smp_processor_id())) - return false; - sched_clock_cpu(cpu); /* Sync clocks across CPUs */ __ttwu_queue_wakelist(p, cpu, wake_flags); return true; --=20 2.27.0