From nobody Mon May 25 00:09:07 2026 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B8B23E3D96; Wed, 20 May 2026 08:34:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779266088; cv=none; b=nSJVjd8bhwyvuYTHeIQow6G7kjw+SHaz7DQm07jOF7Div0H9yF6Ua2vY0cIcmZcirdU5J8uOIPvIQeIhFtxVgVUfH2Xknb73GB/uFQk06HnkqjDFEzUxKrn9zmGNmxOkqXlnlppiRlp65v10Isvuv6V13B4qOt+HCRuorkuK51Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779266088; c=relaxed/simple; bh=df5k9ekG878y8pWqaoykQs5ykr35iqSjlsoTtfcvp0E=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=davgrxn5VNdaHwLeP4wwTPxtzDxtou0/nd4ANDrDI3w0XvY4NN75cf6eX6qDZzUciNjL6D2t8yoWndpCE8IXKWDkJNqA8V94rEQw3Rv6MiZ1fDIJ/dE4K6gxBUboaraIOetTfY/SysDV8Wje8ZyLONfRHsM5inCs92CUVnhuWBI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=1CTINnFk; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=uLNg5LT0; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="1CTINnFk"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="uLNg5LT0" Date: Wed, 20 May 2026 08:34:44 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1779266085; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rN/G3GNn8RpeHC9Tw6t0yAj5FQ5TdHxfkmYjvQ327TE=; b=1CTINnFkk2uMJ0a35mlvbCyxiKdDPwSIJ9rkvbRSR35eF8lmzSF6X0ZRNafwknHVpDF5ad iLa2eCztlAZ/rPZRgctupI/9bRzE/iKQW5UbMsqqGONKqyvtkUdY9hCfXPHki59Ur/VrEw +YYdbx3h6+MuZ8qk/Fxp+HeGp18RF/VHxtnb50Mb0qJnut49s6GTNhVhXPRlLPj2jpGLL+ fQ4H3NXVhw4jjGZ6A2KH0TOvWKYUjGp1KBMEnyLL/Rt3m1YXmJR/ZEAwgC6bZvC7fO2HE2 +LOLIFp3hlH+7iSRaiPIJ7a9sTB2V6PgIqrFHrkiz16EVMMEn46VrT3/whSfbQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1779266085; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rN/G3GNn8RpeHC9Tw6t0yAj5FQ5TdHxfkmYjvQ327TE=; b=uLNg5LT09jqVT6j0UJN5pM91HzdqQ585V4wMskjTJqqcgpPSZ7QEfA7u8wjyQsE+Az+T4b nRpeZcez6CoMMrAw== From: "tip-bot2 for Tim Chen" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/core] sched/cache: Respect LLC preference in task migration and detach Cc: "Peter Zijlstra (Intel)" , K Prateek Nayak , Chen Yu , Tim Chen , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: =?utf-8?q?=3C53da65f3d59de31e1a1dc59a4093d8dd9d4dc206=2E1775065?= =?utf-8?q?312=2Egit=2Etim=2Ec=2Echen=40linux=2Eintel=2Ecom=3E?= References: =?utf-8?q?=3C53da65f3d59de31e1a1dc59a4093d8dd9d4dc206=2E17750653?= =?utf-8?q?12=2Egit=2Etim=2Ec=2Echen=40linux=2Eintel=2Ecom=3E?= Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <177926608448.711.7008097872150906619.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the sched/core branch of tip: Commit-ID: 5b1d5e6db20a6c64ffb95d04578db8c4b0228eea Gitweb: https://git.kernel.org/tip/5b1d5e6db20a6c64ffb95d04578db8c4b= 0228eea Author: Tim Chen AuthorDate: Wed, 01 Apr 2026 14:52:27 -07:00 Committer: Peter Zijlstra CommitterDate: Thu, 09 Apr 2026 15:49:51 +02:00 sched/cache: Respect LLC preference in task migration and detach During load balancing, make can_migrate_task() consider a task's LLC preference. Prevent a task from being moved out of its preferred LLC. During the regular load balancing, if the task cannot be migrated due to LLC locality, the nr_balance_failed also should not be increased. Suggested-by: Peter Zijlstra (Intel) Suggested-by: K Prateek Nayak Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/53da65f3d59de31e1a1dc59a4093d8dd9d4dc206.177= 5065312.git.tim.c.chen@linux.intel.com --- kernel/sched/fair.c | 83 ++++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 13 +++++++- 2 files changed, 91 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index bfb6c0c..5f22e5a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9955,6 +9955,7 @@ enum migration_type { #define LBF_DST_PINNED 0x04 #define LBF_SOME_PINNED 0x08 #define LBF_ACTIVE_LB 0x10 +#define LBF_LLC_PINNED 0x20 =20 struct lb_env { struct sched_domain *sd; @@ -10267,8 +10268,8 @@ static enum llc_mig can_migrate_llc(int src_cpu, in= t dst_cpu, * Check if task p can migrate from source LLC to * destination LLC in terms of cache aware load balance. */ -static __maybe_unused enum llc_mig can_migrate_llc_task(int src_cpu, int d= st_cpu, - struct task_struct *p) +static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu, + struct task_struct *p) { struct mm_struct *mm; bool to_pref; @@ -10335,6 +10336,46 @@ alb_break_llc(struct lb_env *env) =20 return false; } + +/* + * Check if migrating task p from env->src_cpu to + * env->dst_cpu breaks LLC localiy. + */ +static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env) +{ + if (!sched_cache_enabled()) + return false; + + if (task_has_sched_core(p)) + return false; + /* + * Skip over tasks that would degrade LLC locality; + * only when nr_balanced_failed is sufficiently high do we + * ignore this constraint. + * + * Threshold of cache_nice_tries is set to 1 higher + * than nr_balance_failed to avoid excessive task + * migration at the same time. + */ + if (env->sd->nr_balance_failed >=3D env->sd->cache_nice_tries + 1) + return false; + + /* + * We know the env->src_cpu has some tasks prefer to + * run on env->dst_cpu, skip the tasks do not prefer + * env->dst_cpu, and find the one that prefers. + */ + if (env->migration_type =3D=3D migrate_llc_task && + READ_ONCE(p->preferred_llc) !=3D llc_id(env->dst_cpu)) + return true; + + if (can_migrate_llc_task(env->src_cpu, + env->dst_cpu, p) !=3D mig_forbid) + return false; + + return true; +} + #else static inline bool get_llc_stats(int cpu, unsigned long *util, unsigned long *cap) @@ -10347,6 +10388,12 @@ alb_break_llc(struct lb_env *env) { return false; } + +static inline bool +migrate_degrades_llc(struct task_struct *p, struct lb_env *env) +{ + return false; +} #endif /* * can_migrate_task - may task p from runqueue rq be migrated to this_cpu? @@ -10444,10 +10491,29 @@ int can_migrate_task(struct task_struct *p, struc= t lb_env *env) return 1; =20 degrades =3D migrate_degrades_locality(p, env); - if (!degrades) + if (!degrades) { + /* + * If the NUMA locality is not broken, + * further check if migration would hurt + * LLC locality. + */ + if (migrate_degrades_llc(p, env)) { + /* + * If regular load balancing fails to pull a task + * due to LLC locality, this is expected behavior + * and we set LBF_LLC_PINNED so we don't increase + * nr_balance_failed unecessarily. + */ + if (env->migration_type !=3D migrate_llc_task) + env->flags |=3D LBF_LLC_PINNED; + + return 0; + } + hot =3D task_hot(p, env); - else + } else { hot =3D degrades > 0; + } =20 if (!hot || env->sd->nr_balance_failed > env->sd->cache_nice_tries) { if (hot) @@ -13067,9 +13133,16 @@ more_balance: * * Similarly for migration_misfit which is not related to * load/util migration, don't pollute nr_balance_failed. + * + * The same for cache aware scheduling's allowance for + * load imbalance. If regular load balance does not + * migrate task due to LLC locality, it is a expected + * behavior and don't pollute nr_balance_failed. + * See can_migrate_task(). */ if (idle !=3D CPU_NEWLY_IDLE && - env.migration_type !=3D migrate_misfit) + env.migration_type !=3D migrate_misfit && + !(env.flags & LBF_LLC_PINNED)) sd->nr_balance_failed++; =20 if (need_active_balance(&env)) { diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 3c9e92b..a56619b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1547,6 +1547,14 @@ extern void sched_core_dequeue(struct rq *rq, struct= task_struct *p, int flags); extern void sched_core_get(void); extern void sched_core_put(void); =20 +static inline bool task_has_sched_core(struct task_struct *p) +{ + if (sched_core_disabled()) + return false; + + return !!p->core_cookie; +} + #else /* !CONFIG_SCHED_CORE: */ =20 static inline bool sched_core_enabled(struct rq *rq) @@ -1587,6 +1595,11 @@ static inline bool sched_group_cookie_match(struct r= q *rq, return true; } =20 +static inline bool task_has_sched_core(struct task_struct *p) +{ + return false; +} + #endif /* !CONFIG_SCHED_CORE */ =20 #ifdef CONFIG_RT_GROUP_SCHED