From nobody Mon May 25 00:08:59 2026 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7AC693B19BA; Wed, 20 May 2026 08:34:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779266048; cv=none; b=KbEaFASratVpesdfwr7dE0Pd6+mUEtSU/GWA1POZVUnKJxcSQiF9qNZtAY8Jn7Z6yoQ8ze2Ep2KSbDXPyyrjZ4h/UNkq77vW0A6p4s343Q9XvLeeK1M9IRHpWJ7vrHGm8dgRzRkhiJxK7kyJ7Q6/KgCaXP/elIcB24UxNdZaR98= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779266048; c=relaxed/simple; bh=+WuszxKg++ASmnak0r1D6MaV11EJJy/NiVT6ie/YhPE=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=VQpgDM2ieRRrUaXtp7ykzV31q1biptcHRmaEiErWvJyuL6hYGLArONWxeF9Qcs1Wk0yiTVcjSgeY0NEP8/ybEryWQzihJX47P73Y4BFOEhB1tJTnURzgR/uIrPt5Z+WFoIAjyNHYhqjnr5FLPrLjj9NLydelT0AIWYhj/zmIK5k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=pSKUhwKt; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=ZyXpbub7; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="pSKUhwKt"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ZyXpbub7" Date: Wed, 20 May 2026 08:34:03 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1779266045; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NsxRP++06FMQa9Adja3ePX37ji2Cr3O1rIe4jTqGLSU=; b=pSKUhwKtyqnxB8jbEntGhx2JXFZ+nFENdiPzzkQo/cZW5ACGV8lhIrL9WpiBXo4htFIBaN XJgxPV+kiFJQzhihjF6fSMNZHMOYekcd3G2z/wEw7Sm1Ff2RhnfH5Efa488M4Xhig5YETj 2jv06e1/SqRmcesMU/zwSK4Ts8q99d3tpdNHW9YQ0UlaTJR1h9MoicNz2n4J938T9MJAvE Om/4ZsMNccoU3br+0oCDIVZErI6pOKzeNtG0LmCxE2I0BGCw1qqnlXhX+VB0QTCaQwkGkS xWspzPLOtiDPdyVPXK1nRJE+ttKuX0Br08GjVI6aq7K/mMAaaKN5EjIQED/6eQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1779266045; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NsxRP++06FMQa9Adja3ePX37ji2Cr3O1rIe4jTqGLSU=; b=ZyXpbub7Iy/d6bUSh6PAwqfYnaZXY14x4CqrZPSW9Q9N5BW+uOl/USAqf7Q3j9xXkCKUJO uGC7b61LWiwHj3DQ== From: "tip-bot2 for Andrea Righi" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/core] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection Cc: Felix Abecassis , Andrea Righi , "Peter Zijlstra (Intel)" , Vincent Guittot , K Prateek Nayak , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20260511142502.3873984-1-arighi@nvidia.com> References: <20260511142502.3873984-1-arighi@nvidia.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <177926604351.711.144783779882910093.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the sched/core branch of tip: Commit-ID: 25a32e400a14009601c0a727643057f5515152df Gitweb: https://git.kernel.org/tip/25a32e400a14009601c0a727643057f55= 15152df Author: Andrea Righi AuthorDate: Mon, 11 May 2026 16:25:02 +02:00 Committer: Peter Zijlstra CommitterDate: Tue, 19 May 2026 12:17:38 +02:00 sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses select_idle_capacity() and prioritizes idle CPUs with higher capacity for better task placement. However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone. Introduce SMT awareness in the asym-capacity idle selection policy: when SMT is active, always prefer fully-idle SMT cores over partially-idle ones. Prioritizing fully-idle SMT cores yields better task placement because the effective capacity of partially-idle SMT cores is reduced; always preferring them when available leads to more accurate capacity usage on task wakeup. On an SMT system with asymmetric CPU capacities (NVIDIA Vera Rubin), SMT-aware idle selection has been shown to improve throughput by around 15-18% over NO_ASYM mainline and by around 60% over ASYM mainline, for CPU-bound workloads (NVBLAS) running an amount of tasks equal to the amount of SMT cores. Reported-by: Felix Abecassis Signed-off-by: Andrea Righi Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Vincent Guittot Reviewed-by: K Prateek Nayak Link: https://patch.msgid.link/20260511142502.3873984-1-arighi@nvidia.com --- kernel/sched/fair.c | 120 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 114 insertions(+), 6 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 2637a6f..8854d4d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7951,6 +7951,54 @@ static int select_idle_cpu(struct task_struct *p, st= ruct sched_domain *sd, bool=20 } =20 /* + * Idle-capacity scan converts util_fits_cpu() outcomes into preference ra= nks, + * where lower values indicate a better fit - see select_idle_capacity(). + * + * A CPU that both fits the task and sits on a fully-idle SMT core is retu= rned + * immediately and is never assigned one of these ranks. On !SMT every CPU= is + * its own "core", so the early return covers all fits-and-idle cases and = the + * core-tier ranks below become unreachable. + * + * Rank Val Tier Meaning + * ------------------------------ --- ------ ------------------------= --- + * ASYM_IDLE_UCLAMP_MISFIT -4 core Idle core; capacity fits + * util but uclamp_min miss= es. + * ASYM_IDLE_COMPLETE_MISFIT -3 core Idle core; capacity does + * not fit. Still beats eve= ry + * thread-tier rank: a busy + * sibling cuts effective + * capacity more than a + * misfit hurts a quiet cor= e. + * ASYM_IDLE_THREAD_FITS -2 thread Busy SMT sibling; capaci= ty + * fits util + uclamp. + * ASYM_IDLE_THREAD_UCLAMP_MISFIT -1 thread Busy SMT sibling; capaci= ty + * fits but uclamp_min miss= es + * (native util_fits_cpu() + * return value). + * ASYM_IDLE_THREAD_MISFIT 0 thread Busy SMT sibling; capaci= ty + * does not fit. + * + * ASYM_IDLE_CORE_BIAS (-3) is an offset, not a state. On an idle core, + * fits +=3D ASYM_IDLE_CORE_BIAS rebases thread-tier ranks into the core t= ier: + * + * ASYM_IDLE_THREAD_UCLAMP_MISFIT (-1) + BIAS -> ASYM_IDLE_UCLAMP_MISFIT= (-4) + * ASYM_IDLE_THREAD_MISFIT (0) + BIAS -> ASYM_IDLE_COMPLETE_MISF= IT (-3) + * + * ASYM_IDLE_THREAD_FITS (-2) is never rebased because a fully-fitting idl= e-core + * candidate early-returns from select_idle_capacity(). + */ +enum asym_fits_state { + ASYM_IDLE_UCLAMP_MISFIT =3D -4, + ASYM_IDLE_COMPLETE_MISFIT, + ASYM_IDLE_THREAD_FITS, + ASYM_IDLE_THREAD_UCLAMP_MISFIT, + ASYM_IDLE_THREAD_MISFIT, + + /* util_fits_cpu() bias for idle core */ + ASYM_IDLE_CORE_BIAS =3D -3, +}; + +/* * Scan the asym_capacity domain for idle CPUs; pick the first idle one on= which * the task fits. If no CPU is big enough, but there are idle ones, try to * maximize capacity. @@ -7958,8 +8006,14 @@ static int select_idle_cpu(struct task_struct *p, st= ruct sched_domain *sd, bool=20 static int select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int t= arget) { + /* + * On !SMT systems, has_idle_core is always false and preferred_core + * is always true (CPU =3D=3D core), so the SMT preference logic below + * collapses to the plain capacity scan. + */ + bool has_idle_core =3D sched_smt_active() && test_idle_cores(target); unsigned long task_util, util_min, util_max, best_cap =3D 0; - int fits, best_fits =3D 0; + int fits, best_fits =3D ASYM_IDLE_THREAD_MISFIT; int cpu, best_cpu =3D -1; struct cpumask *cpus; =20 @@ -7971,6 +8025,7 @@ select_idle_capacity(struct task_struct *p, struct sc= hed_domain *sd, int target) util_max =3D uclamp_eff_value(p, UCLAMP_MAX); =20 for_each_cpu_wrap(cpu, cpus, target) { + bool preferred_core =3D !has_idle_core || is_core_idle(cpu); unsigned long cpu_cap =3D capacity_of(cpu); =20 if (!choose_idle_cpu(cpu, p)) @@ -7978,8 +8033,14 @@ select_idle_capacity(struct task_struct *p, struct s= ched_domain *sd, int target) =20 fits =3D util_fits_cpu(task_util, util_min, util_max, cpu); =20 - /* This CPU fits with all requirements */ - if (fits > 0) + /* + * Perfect fit: capacity satisfies util + uclamp and the CPU + * sits on a fully-idle SMT core, this is a !SMT system, or + * there is no idle core to find. + * Short-circuit the rank-based selection and return + * immediately. + */ + if (fits > 0 && preferred_core) return cpu; /* * Only the min performance hint (i.e. uclamp_min) doesn't fit. @@ -7987,9 +8048,33 @@ select_idle_capacity(struct task_struct *p, struct s= ched_domain *sd, int target) */ else if (fits < 0) cpu_cap =3D get_actual_cpu_capacity(cpu); + /* + * fits > 0 implies we are not on a preferred core, but the util + * fits CPU capacity. Set fits to ASYM_IDLE_THREAD_FITS + * so the effective range becomes + * [ASYM_IDLE_THREAD_FITS, ASYM_IDLE_THREAD_MISFIT], where: + * ASYM_IDLE_THREAD_MISFIT - does not fit + * ASYM_IDLE_THREAD_UCLAMP_MISFIT - fits with the exception of UCLAMP= _MIN + * ASYM_IDLE_THREAD_FITS - fits with the exception of preferred_core + */ + else if (fits > 0) + fits =3D ASYM_IDLE_THREAD_FITS; =20 /* - * First, select CPU which fits better (-1 being better than 0). + * If we are on a preferred core, translate the range of fits + * of [ASYM_IDLE_THREAD_UCLAMP_MISFIT, ASYM_IDLE_THREAD_MISFIT] to + * [ASYM_IDLE_UCLAMP_MISFIT, ASYM_IDLE_COMPLETE_MISFIT]. + * This ensures that an idle core is always given priority over + * (partially) busy core. + * + * A fully fitting idle core would have returned early and hence + * fits > 0 for preferred_core need not be dealt with. + */ + if (preferred_core) + fits +=3D ASYM_IDLE_CORE_BIAS; + + /* + * First, select CPU which fits better (lower is more preferred). * Then, select the one with best capacity at same level. */ if ((fits < best_fits) || @@ -8000,6 +8085,19 @@ select_idle_capacity(struct task_struct *p, struct s= ched_domain *sd, int target) } } =20 + /* + * A value in the [ASYM_IDLE_UCLAMP_MISFIT, ASYM_IDLE_COMPLETE_MISFIT] + * range means the chosen CPU is in a fully idle SMT core. Values above + * ASYM_IDLE_COMPLETE_MISFIT mean we never ranked such a CPU best. + * + * The asym-capacity wakeup path returns from select_idle_sibling() + * after this function and never runs select_idle_cpu(), so the usual + * select_idle_cpu() tail that clears idle cores must live here when the + * idle-core preference did not win. + */ + if (has_idle_core && best_fits > ASYM_IDLE_COMPLETE_MISFIT) + set_idle_cores(target, false); + return best_cpu; } =20 @@ -8008,12 +8106,22 @@ static inline bool asym_fits_cpu(unsigned long util, unsigned long util_max, int cpu) { - if (sched_asym_cpucap_active()) + if (sched_asym_cpucap_active()) { /* * Return true only if the cpu fully fits the task requirements * which include the utilization and the performance hints. + * + * When SMT is active, also require that the core has no busy + * siblings. + * + * Note: gating on is_core_idle() also makes the early-bailout + * candidates in select_idle_sibling() (target, prev, + * recent_used_cpu) idle-core-aware on ASYM+SMT, which the + * NO_ASYM path does not do. */ - return (util_fits_cpu(util, util_min, util_max, cpu) > 0); + return (!sched_smt_active() || is_core_idle(cpu)) && + (util_fits_cpu(util, util_min, util_max, cpu) > 0); + } =20 return true; }