From nobody Thu Apr 16 05:20:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 331A1C4332F for ; Thu, 24 Nov 2022 09:08:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229641AbiKXJII (ORCPT ); Thu, 24 Nov 2022 04:08:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229569AbiKXJIA (ORCPT ); Thu, 24 Nov 2022 04:08:00 -0500 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60042729B4 for ; Thu, 24 Nov 2022 01:07:59 -0800 (PST) Received: by mail-pj1-x1035.google.com with SMTP id w4-20020a17090ac98400b002186f5d7a4cso4617174pjt.0 for ; Thu, 24 Nov 2022 01:07:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:cc:to:subject:from:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=lpuryGZJKM59gQIs5x5YGpBkqDs6fnHEuG6NexMes78=; b=yph/fRCVfCoEmq/3sBY95VkHR/nOlP3ylkcd0NGrRXBH2BJpQlhfNo4Rl0Dh51OS9I cwOeGCa2cWpUR3RWxVAp8ejWAm9kEkt1h+/m7/3I3wsEKAjaN4ZayHpzURom65QMyuZl 6rVgdzQSl1UJOe5k65wvVw08IEYX//yIP5Ybnq0mWKYIFCzx+SGCHX9ZTO6PL5nD0VVi Loyi9S3pSxFQIykIwXjyGZvQEAL55KgAI4YQ5ynmYmlwR9YwSU5KeFf6AAGnwdPOPurl tRvWJMvuh7G3246tS3vjuZKpqUYcFKfu2YBjr9vFY2uMhR/+ZCh631HOOdR5vKn6SMW8 cZ6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:from:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=lpuryGZJKM59gQIs5x5YGpBkqDs6fnHEuG6NexMes78=; b=6FApLXE6MY9INY8oZLCJylwQOlhvIUOqrY+r/1y7sUyVXKx5tOniNVUk56CPyWnmUs KW/zvMIXpy9FDEuYIcsTvz1sFLBrp2hzX9+3w1BXpI55JFQ5j8vyEU4yIp+tdDPKZ/uB EJ2SbtC6pMAkGu4KVFFdUZ9GzoOeWmdVOSMHE755oUtFjB2bPOX+ZrtmRn+9XCodwtQO JfGTbLe7xxXGOQLIMgDq+X+7qrM5qxjMSfkyLRTjsELYB29Y9CWh5LMzpdRp2MaafYnJ sOvMqmV1P8Ze4uYw8PtBl1oznmQUeXvsLQ8NJymAYRTN3+rjk8NeYLDjMV4WmESZ/bJJ 8v2w== X-Gm-Message-State: ANoB5plfCyjbtmJo1JE4kQmfW7nj6bUwjfLNeEl71CBmxw3rpjMoH8CU avauQHmoVPzwp3rf3TIHh5PFUw== X-Google-Smtp-Source: AA0mqf52kSu03psvPnVxMJ9yyk/4Fi0SHQC5rOF8pPLKol+68U6nU0Uq8541R5NhJOkasNDXNStlfQ== X-Received: by 2002:a17:90a:7896:b0:218:6fd6:b693 with SMTP id x22-20020a17090a789600b002186fd6b693mr34771104pjk.29.1669280878873; Thu, 24 Nov 2022 01:07:58 -0800 (PST) Received: from ?IPV6:2409:8962:ba17:2b8:60e7:3d11:5617:d572? ([2409:8962:ba17:2b8:60e7:3d11:5617:d572]) by smtp.gmail.com with ESMTPSA id x189-20020a6231c6000000b0056cee8af3a6sm686346pfx.54.2022.11.24.01.07.53 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 24 Nov 2022 01:07:58 -0800 (PST) Message-ID: Date: Thu, 24 Nov 2022 17:07:46 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 From: chenying Subject: [PATCH] Reduce rq lock contention in load_balance() To: mingo@redhat.com, Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Benjamin Segall Cc: linux-kernel , Abel Wu Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8"; format="flowed" From: chenying When doing newidle load balancing, we may have lock contention on rq->lock while finding the same busiest rq on multiple cpus. However, it is often the case that after the first load balancing, the busiest-rq may not be the busiest anymore. This may lead to pointless waits for locks. For this case, we want to use trylock to reduce rq lock contention in load_balance(). We add rq->lb_lock for the load balancing path, and uses trylock to try to acquire the busiest rq lb_lock, if it fails, clear the busiest rq's cpu from load_balance_mask and then goto refind. The test results show that this patch brings ~35% rq lock contentions reduced and no scheduling latency reduction. unpatched: lock_stat version 0.4 Suggested-by: Abel Wu ---------------------------------------------------------------------------= ---------------------------------------------------------------------------= ----------------------------------------------------------------------- class name con-bounces contentions=20 waittime-min waittime-max waittime-total waittime-avg=20 acq-bounces acquisitions holdtime-min holdtime-max holdtime-total=20 holdtime-avg ...........................................................................= ...........................................................................= ....................................................................... &rq->lock: 24906 25996=20 0.08 27.77 43122.87 1.66=20 1216316 6601547 0.05 41.59 10224267.38=20 1.55 --------- &rq->lock 1210=20 [<000000000fe88813>] scheduler_tick+0x4f/0xf0 &rq->lock 1885=20 [<00000000de367e3c>] _nohz_idle_balance+0x116/0x250=20 &rq->lock 15111 [<00000000daf6fa95>]=20 update_blocked_averages+0x30/0x6f0 &rq->lock 1156=20 [<00000000d5c71b0e>] __schedule+0xa9/0x800 --------- &rq->lock 15542=20 [<00000000daf6fa95>] update_blocked_averages+0x30/0x6f0 &rq->lock 733=20 [<000000000fe88813>] scheduler_tick+0x4f/0xf0 &rq->lock 3066=20 [<000000000bc2ee47>] try_to_wake_up+0x206/0x710 &rq->lock 1272=20 [<00000000d5c71b0e>] __schedule+0xa9/0x800 patched: lock_stat version 0.4 ---------------------------------------------------------------------------= ---------------------------------------------------------------------------= ----------------------------------------------------------------------- class name con-bounces contentions=20 waittime-min waittime-max waittime-total waittime-avg=20 acq-bounces acquisitions holdtime-min holdtime-max holdtime-total=20 holdtime-avg ...........................................................................= ...........................................................................= ....................................................................... &rq->lock: 16174 17105=20 0.07 33.13 31154.45 1.82=20 1162817 6602803 0.05 64.68 10141979.28=20 1.54 --------- &rq->lock 11665=20 [<00000000ce27c902>] update_blocked_averages+0x30/0x700 &rq->lock 1457=20 [<00000000a6302c24>] try_to_wake_up+0x206/0x710 &rq->lock 1159=20 [<000000009f2bc605>] __schedule+0xa9/0x810 &rq->lock 1411=20 [<00000000aa0a6e31>] _nohz_idle_balance+0x116/0x250 --------- &rq->lock 3032=20 [<00000000a6302c24>] try_to_wake_up+0x206/0x710 &rq->lock 248=20 [<000000008bd7e827>] load_balance+0x571/0xe80 &rq->lock 11502=20 [<00000000ce27c902>] update_blocked_averages+0x30/0x700 &rq->lock 1253=20 [<000000009f2bc605>] __schedule+0xa9/0x810 unpatched: # ./runqlat 60 1 usecs : count distribution 0 -> 1 : 1172 |=20 | 2 -> 3 : 210063 |************************=20 | 4 -> 7 : 337576=20 |****************************************| 8 -> 15 : 24555 |**=20 | 16 -> 31 : 13598 |*=20 | 32 -> 63 : 779 |=20 | 64 -> 127 : 230 |=20 | 128 -> 255 : 83 |=20 | 256 -> 511 : 54 |=20 | 512 -> 1023 : 62 |=20 | 1024 -> 2047 : 123 |=20 | 2048 -> 4095 : 283 |=20 | 4096 -> 8191 : 1362 |=20 | 8192 -> 16383 : 2775 |=20 | 16384 -> 32767 : 52352 |******=20 | 32768 -> 65535 : 14 |=20 | 65536 -> 131071 : 140 |=20 | patched: # ./runqlat 60 1 usecs : count distribution 0 -> 1 : 1091 |=20 | 2 -> 3 : 205259 |***********************=20 | 4 -> 7 : 351620=20 |****************************************| 8 -> 15 : 27812 |***=20 | 16 -> 31 : 13971 |*=20 | 32 -> 63 : 727 |=20 | 64 -> 127 : 198 |=20 | 128 -> 255 : 103 |=20 | 256 -> 511 : 61 |=20 | 512 -> 1023 : 45 |=20 | 1024 -> 2047 : 108 |=20 | 2048 -> 4095 : 271 |=20 | 4096 -> 8191 : 1342 |=20 | 8192 -> 16383 : 2732 |=20 | 16384 -> 32767 : 49367 |*****=20 | 32768 -> 65535 : 8 |=20 | 65536 -> 131071 : 183 |=20 | test script: #!/bin/bash mkdir /sys/fs/cgroup/cpuset/test1 echo 12,14,16,18,20,22 > /sys/fs/cgroup/cpuset/test1/cpuset.cpus echo 0 > /sys/fs/cgroup/cpuset/test1/cpuset.mems mkdir /sys/fs/cgroup/cpuset/test2 echo 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46=20 > /sys/fs/cgroup/cpuset/test2/cpuset.cpus echo 0 > /sys/fs/cgroup/cpuset/test2/cpuset.mems cgexec -g cpuset:test1 sysbench --test=3Dcpu --cpu-max-prime=3D200000 run=20 --num-threads=3D24 --rate=3D100 --time=3D6000 & cgexec -g cpuset:test2 sysbench --test=3Dcpu --cpu-max-prime=3D200000 run=20 --num-threads=3D96 --rate=3D100 --time=3D6000 & Suggested-by: Abel Wu Signed-off-by: chenying --- kernel/sched/core.c | 1 + kernel/sched/fair.c | 12 ++++++++++++ kernel/sched/sched.h | 1 + 3 files changed, 14 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index daff72f00385..d41f1a9c7d5f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -9697,6 +9697,7 @@ void __init sched_init(void) rq =3D cpu_rq(i); raw_spin_lock_init(&rq->__lock); + raw_spin_lock_init(&rq->lb_lock); rq->nr_running =3D 0; rq->calc_load_active =3D 0; rq->calc_load_update =3D jiffies + LOAD_FREQ; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e4a0b8bd941c..d92c42671b99 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10295,6 +10295,7 @@ static int load_balance(int this_cpu, struct rq=20 *this_rq, goto out_balanced; } +refind: busiest =3D find_busiest_queue(&env, group); if (!busiest) { schedstat_inc(sd->lb_nobusyq[idle]); @@ -10303,6 +10304,14 @@ static int load_balance(int this_cpu, struct rq=20 *this_rq, WARN_ON_ONCE(busiest =3D=3D env.dst_rq); + if (!raw_spin_trylock(&busiest->lb_lock)) { + __cpumask_clear_cpu(cpu_of(busiest), cpus); + if (cpumask_intersects(sched_group_span(group), cpus)) + goto refind; + + goto out_balanced; + } + schedstat_add(sd->lb_imbalance[idle], env.imbalance); env.src_cpu =3D busiest->cpu; @@ -10403,6 +10412,8 @@ static int load_balance(int this_cpu, struct rq=20 *this_rq, /* All tasks on this runqueue were pinned by CPU affinity */ if (unlikely(env.flags & LBF_ALL_PINNED)) { + raw_spin_unlock(&busiest->lb_lock); + __cpumask_clear_cpu(cpu_of(busiest), cpus); /* * Attempting to continue load balancing at the current @@ -10420,6 +10431,7 @@ static int load_balance(int this_cpu, struct rq=20 *this_rq, goto out_all_pinned; } } + raw_spin_unlock(&busiest->lb_lock); if (!ld_moved) { schedstat_inc(sd->lb_failed[idle]); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index a4a20046e586..384690bda8c3 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -954,6 +954,7 @@ struct balance_callback { struct rq { /* runqueue lock: */ raw_spinlock_t __lock; + raw_spinlock_t lb_lock; /* * nr_running and cpu_load should be in the same cacheline because --=20 2.11.0