From nobody Wed Feb 11 12:07:24 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E5CDC7EE23 for ; Thu, 8 Jun 2023 22:32:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237202AbjFHWc3 (ORCPT ); Thu, 8 Jun 2023 18:32:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237142AbjFHWcS (ORCPT ); Thu, 8 Jun 2023 18:32:18 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AC03C2 for ; Thu, 8 Jun 2023 15:32:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1686263537; x=1717799537; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bxF4ml+2B8CLxSOfFJcQMgzgf08hlb1xTj0M9Ps/3+U=; b=e3EB2gwQfBe0ZP648wLyitcM+HJdh5obYl70+wPOlAt0M+xn25/YBE8z Adi7lEj/ooIFUCJseHlVmVgZq8EgGHCeQLksPqLWm6YWn20UmsysFvx4J pt8dveoY3OQ1eK6rCCPwrFs7IPFajVsk4xDi+r/w00ahMMyV/86yKz0++ jFmaIKuQIXsi/oaX5OkLT2Hka0L6JbwPrcKa/rSSL/6pHrfNSzom/g454 wabDbYCicYWkUC96JJqxfVZKnUC+041e4w7m1SOYB5Hfht7gctkjuL8uq +WPE6ND795WEqX+y0x9qmi8yCHgMqA4Tqiycvk07Ac5b+t+BeEl8gfSXH A==; X-IronPort-AV: E=McAfee;i="6600,9927,10735"; a="347094742" X-IronPort-AV: E=Sophos;i="6.00,227,1681196400"; d="scan'208";a="347094742" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jun 2023 15:32:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10735"; a="956906448" X-IronPort-AV: E=Sophos;i="6.00,227,1681196400"; d="scan'208";a="956906448" Received: from b04f130c83f2.jf.intel.com ([10.165.154.98]) by fmsmga006.fm.intel.com with ESMTP; 08 Jun 2023 15:32:15 -0700 From: Tim Chen To: Peter Zijlstra Cc: Tim C Chen , Juri Lelli , Vincent Guittot , Ricardo Neri , "Ravi V . Shankar" , Ben Segall , Daniel Bristot de Oliveira , Dietmar Eggemann , Len Brown , Mel Gorman , "Rafael J . Wysocki" , Srinivas Pandruvada , Steven Rostedt , Valentin Schneider , Ionela Voinescu , x86@kernel.org, linux-kernel@vger.kernel.org, Shrikanth Hegde , Srikar Dronamraju , naveen.n.rao@linux.vnet.ibm.com, Yicong Yang , Barry Song , Chen Yu , Hillf Danton Subject: [Patch v2 3/6] sched/fair: Implement prefer sibling imbalance calculation between asymmetric groups Date: Thu, 8 Jun 2023 15:32:29 -0700 Message-Id: X-Mailer: git-send-email 2.32.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Tim C Chen In the current prefer sibling load balancing code, there is an implicit assumption that the busiest sched group and local sched group are equivalent, hence the tasks to be moved is simply the difference in number of tasks between the two groups (i.e. imbalance) divided by two. However, we may have different number of cores between the cluster groups, say when we take CPU offline or we have hybrid groups. In that case, we should balance between the two groups such that #tasks/#cores ratio is the same between the same between both groups. Hence the imbalance computed will need to reflect this. Additionally when we have asymmetric packing, we will need to bias the balancing according to whether the busiest group is favored or the local group if favored. If the destination group is favored and has idle cores, we should move at least one task from the busiest group to avoid leaving favored idle core unused. But if the busiest group is favored, we should limit the number of tasks we move so we will not create idle cores in busiest group, leaving favored cores unused. Adjust the sibling imbalance computation to take into account of the above considerations. Signed-off-by: Tim Chen --- kernel/sched/fair.c | 65 +++++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 5 ++++ 2 files changed, 66 insertions(+), 4 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 03573362274f..0b0904263d51 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9372,6 +9372,65 @@ static inline bool smt_balance(struct lb_env *env, s= truct sg_lb_stats *sgs, return false; } =20 +static inline long sibling_imbalance(struct lb_env *env, + struct sd_lb_stats *sds, + struct sg_lb_stats *busiest, + struct sg_lb_stats *local) +{ + int ncores_busiest, ncores_local; + long imbalance; + + if (env->idle =3D=3D CPU_NOT_IDLE) + return 0; + + ncores_busiest =3D sds->busiest->cores; + ncores_local =3D sds->local->cores; + + if (ncores_busiest =3D=3D ncores_local && + (!(env->sd->flags & SD_ASYM_PACKING) || + sched_asym_equal(env->dst_cpu, + sds->busiest->asym_prefer_cpu))) { + imbalance =3D busiest->sum_nr_running; + lsub_positive(&imbalance, local->sum_nr_running); + return imbalance; + } + + /* Balance such that nr_running/ncores ratio are same on both groups */ + imbalance =3D ncores_local * busiest->sum_nr_running; + lsub_positive(&imbalance, ncores_busiest * local->sum_nr_running); + /* Normalize imbalance to become tasks to be moved to restore balance */ + imbalance /=3D ncores_local + ncores_busiest; + + if (env->sd->flags & SD_ASYM_PACKING) { + int limit; + + if (!busiest->sum_nr_running) + goto out; + + if (sched_asym_prefer(env->dst_cpu, sds->busiest->asym_prefer_cpu)) { + /* Don't leave preferred core idle */ + if (imbalance =3D=3D 0 && local->sum_nr_running < ncores_local) + imbalance =3D 1; + goto out; + } + + /* Limit tasks moved from preferred group, don't leave cores idle */ + limit =3D busiest->sum_nr_running; + lsub_positive(&limit, ncores_busiest); + if (imbalance > limit) + imbalance =3D limit; + + goto out; + } + + /* Take advantage of resource in an empty sched group */ + if (imbalance =3D=3D 0 && local->sum_nr_running =3D=3D 0 && + busiest->sum_nr_running > 1) + imbalance =3D 1; +out: + return imbalance << 1; +} + static inline bool sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) { @@ -10230,14 +10289,12 @@ static inline void calculate_imbalance(struct lb_= env *env, struct sd_lb_stats *s } =20 if (busiest->group_weight =3D=3D 1 || sds->prefer_sibling) { - unsigned int nr_diff =3D busiest->sum_nr_running; /* * When prefer sibling, evenly spread running tasks on * groups. */ env->migration_type =3D migrate_task; - lsub_positive(&nr_diff, local->sum_nr_running); - env->imbalance =3D nr_diff; + env->imbalance =3D sibling_imbalance(env, sds, busiest, local); } else { =20 /* @@ -10424,7 +10481,7 @@ static struct sched_group *find_busiest_group(struc= t lb_env *env) * group's child domain. */ if (sds.prefer_sibling && local->group_type =3D=3D group_has_spare && - busiest->sum_nr_running > local->sum_nr_running + 1) + sibling_imbalance(env, &sds, busiest, local) > 1) goto force_balance; =20 if (busiest->group_type !=3D group_overloaded) { diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 5f7f36e45b87..adffe0894cdb 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -804,6 +804,11 @@ static inline bool sched_asym_prefer(int a, int b) return arch_asym_cpu_priority(a) > arch_asym_cpu_priority(b); } =20 +static inline bool sched_asym_equal(int a, int b) +{ + return arch_asym_cpu_priority(a) =3D=3D arch_asym_cpu_priority(b); +} + struct perf_domain { struct em_perf_domain *em_pd; struct perf_domain *next; --=20 2.32.0