From nobody Mon May 25 00:08:59 2026 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9717A3EC2CD; Wed, 20 May 2026 08:34:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779266099; cv=none; b=PX2KgFvVri//l1nPal2RYqGx8pbMJdCP8+w40tMRd5vzgTN+jsVwjGycUI5U9WWRyRJfSoCDi3Yd4Nur/mfEgITDX60MWZLCQnptKPZLByEv45EMu0ZDlR3vMH4iKHFUwVMeFgqae+Ox42hNilBMsMAQ0bEO1gzuOOjGRURzvHY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779266099; c=relaxed/simple; bh=F7RvwUemTFZC8avVcdRcjn86CKbw99Yb2GmikllTeD0=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=S1s3CWCnW1mrOd44SB6lrBDC+Y9xJEcdj+4iszBIKCLfzqtWw7cuNdpKihNzZf6LeJP976OcdE6OTwy8d0yTYQTmboCwZm9XLVh4EQlbAv//2iCkhSezzYHk/tSHHPxVrr4Yl+xxhGqgYveabTFbmfbYVoLkcQ4mWyG18qmskPY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=DC1cOx8F; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=twQGXfkH; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="DC1cOx8F"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="twQGXfkH" Date: Wed, 20 May 2026 08:34:54 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1779266096; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CuDU1jYGdgPmWqLNOnhP6pvuqsYOMnVN55z4UO20RWk=; b=DC1cOx8FVySalp+jPGfbLJIs1SdrxC2HkhWgDCBHDG+Vnws6OTbCTI4hGzear+KbcdqAew QJpxIwTfM+3OX71sNA4zrKEB4bSFWD/tbViaVHeJSjR46JuyER1Su9LPdpJB+JLkLhOCex z2EbC343q1Lommh5BeMvkWMzMtTBM/UW/S5eRiicbXxO5O/OSdkyuE8RA8cCgWg+JJJKGo BAOg0BVWYY/nwSjgS2TqMMQH1Ozao0jrSYUF3QkAbbmyW139gQ5apPWOZsfMgX2adtDgpP K+UE3afPw1OoPeTeWd1J3F03hLOdzErEUkXTb+GN0HrN9x6fbAz0oCLYFBu/bw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1779266096; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CuDU1jYGdgPmWqLNOnhP6pvuqsYOMnVN55z4UO20RWk=; b=twQGXfkHau9vqOFUUFBkpSkl56A1gfEk87pN7t+KM5ocvKazfjdqkqSzC/VSlNt1wBlIAx ubeoy7vD/NcI3jDA== From: "tip-bot2 for Tim Chen" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/core] sched/cache: Track LLC-preferred tasks per runqueue Cc: Chen Yu , Tim Chen , "Peter Zijlstra (Intel)" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: =?utf-8?q?=3C459a37102f3d74a4e09ea58401d2094ac731d044=2E1775065?= =?utf-8?q?312=2Egit=2Etim=2Ec=2Echen=40linux=2Eintel=2Ecom=3E?= References: =?utf-8?q?=3C459a37102f3d74a4e09ea58401d2094ac731d044=2E17750653?= =?utf-8?q?12=2Egit=2Etim=2Ec=2Echen=40linux=2Eintel=2Ecom=3E?= Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <177926609471.711.9800945470622390841.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the sched/core branch of tip: Commit-ID: 46afe3af7ead57190b6d362e214814ec804e3b7b Gitweb: https://git.kernel.org/tip/46afe3af7ead57190b6d362e214814ec8= 04e3b7b Author: Tim Chen AuthorDate: Wed, 01 Apr 2026 14:52:19 -07:00 Committer: Peter Zijlstra CommitterDate: Thu, 09 Apr 2026 15:49:49 +02:00 sched/cache: Track LLC-preferred tasks per runqueue For each runqueue, track the number of tasks with an LLC preference and how many of them are running on their preferred LLC. This mirrors nr_numa_running and nr_preferred_running for NUMA balancing, and will be used by cache-aware load balancing in later patches. Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/459a37102f3d74a4e09ea58401d2094ac731d044.177= 5065312.git.tim.c.chen@linux.intel.com --- kernel/sched/core.c | 5 +++++- kernel/sched/fair.c | 47 ++++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 8 +++++++- 3 files changed, 57 insertions(+), 3 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d11e27b..eb542c9 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -539,6 +539,11 @@ void __trace_set_current_state(int state_value) } EXPORT_SYMBOL(__trace_set_current_state); =20 +int task_llc(const struct task_struct *p) +{ + return per_cpu(sd_llc_id, task_cpu(p)); +} + /* * Serialization rules: * diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6e78ecf..e66da7a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1384,6 +1384,30 @@ static int llc_id(int cpu) return per_cpu(sd_llc_id, cpu); } =20 +static void account_llc_enqueue(struct rq *rq, struct task_struct *p) +{ + int pref_llc; + + pref_llc =3D p->preferred_llc; + if (pref_llc < 0) + return; + + rq->nr_llc_running++; + rq->nr_pref_llc_running +=3D (pref_llc =3D=3D task_llc(p)); +} + +static void account_llc_dequeue(struct rq *rq, struct task_struct *p) +{ + int pref_llc; + + pref_llc =3D p->preferred_llc; + if (pref_llc < 0) + return; + + rq->nr_llc_running--; + rq->nr_pref_llc_running -=3D (pref_llc =3D=3D task_llc(p)); +} + void mm_init_sched(struct mm_struct *mm, struct sched_cache_time __percpu *_pcpu_sched) { @@ -1490,6 +1514,8 @@ static int get_pref_llc(struct task_struct *p, struct= mm_struct *mm) return mm_sched_llc; } =20 +static unsigned int task_running_on_cpu(int cpu, struct task_struct *p); + static inline void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) { @@ -1530,8 +1556,13 @@ void account_mm_sched(struct rq *rq, struct task_str= uct *p, s64 delta_exec) =20 mm_sched_llc =3D get_pref_llc(p, mm); =20 - if (READ_ONCE(p->preferred_llc) !=3D mm_sched_llc) + /* task not on rq accounted later in account_entity_enqueue() */ + if (task_running_on_cpu(rq->cpu, p) && + READ_ONCE(p->preferred_llc) !=3D mm_sched_llc) { + account_llc_dequeue(rq, p); WRITE_ONCE(p->preferred_llc, mm_sched_llc); + account_llc_enqueue(rq, p); + } } =20 static void task_tick_cache(struct rq *rq, struct task_struct *p) @@ -1714,6 +1745,10 @@ static inline int get_pref_llc(struct task_struct *p, return -1; } =20 +static void account_llc_enqueue(struct rq *rq, struct task_struct *p) {} + +static void account_llc_dequeue(struct rq *rq, struct task_struct *p) {} + #endif /* CONFIG_SCHED_CACHE */ =20 /* @@ -4200,9 +4235,11 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct= sched_entity *se) { update_load_add(&cfs_rq->load, se->load.weight); if (entity_is_task(se)) { + struct task_struct *p =3D task_of(se); struct rq *rq =3D rq_of(cfs_rq); =20 - account_numa_enqueue(rq, task_of(se)); + account_numa_enqueue(rq, p); + account_llc_enqueue(rq, p); list_add(&se->group_node, &rq->cfs_tasks); } cfs_rq->nr_queued++; @@ -4213,7 +4250,11 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct= sched_entity *se) { update_load_sub(&cfs_rq->load, se->load.weight); if (entity_is_task(se)) { - account_numa_dequeue(rq_of(cfs_rq), task_of(se)); + struct task_struct *p =3D task_of(se); + struct rq *rq =3D rq_of(cfs_rq); + + account_numa_dequeue(rq, p); + account_llc_dequeue(rq, p); list_del_init(&se->group_node); } cfs_rq->nr_queued--; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 3cb3ab0..3c9e92b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1196,6 +1196,12 @@ struct rq { #ifdef CONFIG_NUMA_BALANCING unsigned int numa_migrate_on; #endif + +#ifdef CONFIG_SCHED_CACHE + unsigned int nr_pref_llc_running; + unsigned int nr_llc_running; +#endif + /* * This is part of a global counter where only the total sum * over all CPUs matters. A task can increase this counter on @@ -2077,6 +2083,8 @@ init_numa_balancing(u64 clone_flags, struct task_stru= ct *p) =20 #endif /* !CONFIG_NUMA_BALANCING */ =20 +int task_llc(const struct task_struct *p); + static inline void queue_balance_callback(struct rq *rq, struct balance_callback *head,