From nobody Sun Apr 12 21:00:55 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6FEC739FCD1 for ; Wed, 1 Apr 2026 21:46:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.15 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775080016; cv=none; b=QAGI50wNovJNZYWaBQev0ABXe2gbbo0TN2PcUiNRPselN+XFQYGQD5PmYJj436F2b/pubu2wHTNe32PS/3i7J3a/8XwsO7Hah3VcEgyti/R+LxB+7ZqgeF8+05pkM5iBIbqsIzn8iAXwZbqakGu9jHCeuANxvRXTETODpoZmiGs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775080016; c=relaxed/simple; bh=7oxtfI0KHtwl0BOu+zI/L5CxroANeisK8RbDWCk0BvE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=EnAMNGbk/f90JMEeqtaIqYyL6f5bY4JVo+oCjK77HoV14XmJ0hVl7quJgUjccD5qcBsTg9pEq4gOOchIcNZQfeCs1UmFr6NLsE7Iovz5W5ynCRULKY2fyUGfkulaiWIntMPw6NwPy5nSMPV05LXn2rrmkY8uZFA8xZdnv8AIR5U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=pass smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=aFr7MGrl; arc=none smtp.client-ip=198.175.65.15 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="aFr7MGrl" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1775080014; x=1806616014; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7oxtfI0KHtwl0BOu+zI/L5CxroANeisK8RbDWCk0BvE=; b=aFr7MGrlXZnfXsLF/sCzVnxVA3YasFno2Jbal1BHV958XKyCbXtpxc+X YBUteJSw8ZCI0wTqDUF2p2Zw5KRN2598Od7cGNPqGRkH/f8Kstvl+nnJs LP3SMxUrB4PmWSlORCJrVVPhf522SNdT6y6WCpnmpX2kduNhY3ulVZSF4 to2wn4Yx9lqJG53aZWTtUnP7GHBcTCBcRjUWFJ7GOVhXSYXJMBhw0Vo9X EnWprMoyeCPxnCXRsA5hSWHAxhwun3XmHpCNQHXhOrH3RD15U6hB43DP3 +euLUnsZBfTYhb8y2p/BjXzt3weYMPPpM1AKX9FeXqtMyZuMqzGe4EECm A==; X-CSE-ConnectionGUID: lUliMuvMRMqWfuRnqW0iyg== X-CSE-MsgGUID: bDodznmgTIuiwuRQFY7qGA== X-IronPort-AV: E=McAfee;i="6800,10657,11746"; a="79739836" X-IronPort-AV: E=Sophos;i="6.23,153,1770624000"; d="scan'208";a="79739836" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by orvoesa107.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2026 14:46:53 -0700 X-CSE-ConnectionGUID: nkFdx41sQI+PKGOlsTaApQ== X-CSE-MsgGUID: S6UJa4bHRieAhrGL0GbJ/w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.23,153,1770624000"; d="scan'208";a="249842422" Received: from b04f130c83f2.jf.intel.com ([10.165.154.98]) by fmviesa002.fm.intel.com with ESMTP; 01 Apr 2026 14:46:52 -0700 From: Tim Chen To: Peter Zijlstra , Ingo Molnar , K Prateek Nayak , "Gautham R . Shenoy" , Vincent Guittot Cc: Tim Chen , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Madadi Vineeth Reddy , Hillf Danton , Shrikanth Hegde , Jianyong Wu , Yangyu Chen , Tingyin Duan , Vern Hao , Vern Hao , Len Brown , Aubrey Li , Zhao Liu , Chen Yu , Chen Yu , Adam Li , Aaron Lu , Tim Chen , Josh Don , Gavin Guo , Qais Yousef , Libo Chen , linux-kernel@vger.kernel.org Subject: [Patch v4 07/22] sched/cache: Track LLC-preferred tasks per runqueue Date: Wed, 1 Apr 2026 14:52:19 -0700 Message-Id: <459a37102f3d74a4e09ea58401d2094ac731d044.1775065312.git.tim.c.chen@linux.intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For each runqueue, track the number of tasks with an LLC preference and how many of them are running on their preferred LLC. This mirrors nr_numa_running and nr_preferred_running for NUMA balancing, and will be used by cache-aware load balancing in later patches. Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen --- Notes: v3->v4: Use READ_ONCE() to access p->preferred_llc (Madadi Vineeth Reddy). kernel/sched/core.c | 5 +++++ kernel/sched/fair.c | 47 +++++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 8 ++++++++ 3 files changed, 57 insertions(+), 3 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 1188b5d24933..93a1dbc02667 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -539,6 +539,11 @@ void __trace_set_current_state(int state_value) } EXPORT_SYMBOL(__trace_set_current_state); =20 +int task_llc(const struct task_struct *p) +{ + return per_cpu(sd_llc_id, task_cpu(p)); +} + /* * Serialization rules: * diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1eda689e0136..4b760bd604e7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1291,6 +1291,30 @@ static int llc_id(int cpu) return per_cpu(sd_llc_id, cpu); } =20 +static void account_llc_enqueue(struct rq *rq, struct task_struct *p) +{ + int pref_llc; + + pref_llc =3D p->preferred_llc; + if (pref_llc < 0) + return; + + rq->nr_llc_running++; + rq->nr_pref_llc_running +=3D (pref_llc =3D=3D task_llc(p)); +} + +static void account_llc_dequeue(struct rq *rq, struct task_struct *p) +{ + int pref_llc; + + pref_llc =3D p->preferred_llc; + if (pref_llc < 0) + return; + + rq->nr_llc_running--; + rq->nr_pref_llc_running -=3D (pref_llc =3D=3D task_llc(p)); +} + void mm_init_sched(struct mm_struct *mm, struct sched_cache_time __percpu *_pcpu_sched) { @@ -1397,6 +1421,8 @@ static int get_pref_llc(struct task_struct *p, struct= mm_struct *mm) return mm_sched_llc; } =20 +static unsigned int task_running_on_cpu(int cpu, struct task_struct *p); + static inline void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) { @@ -1439,8 +1465,13 @@ void account_mm_sched(struct rq *rq, struct task_str= uct *p, s64 delta_exec) =20 mm_sched_llc =3D get_pref_llc(p, mm); =20 - if (READ_ONCE(p->preferred_llc) !=3D mm_sched_llc) + /* task not on rq accounted later in account_entity_enqueue() */ + if (task_running_on_cpu(rq->cpu, p) && + READ_ONCE(p->preferred_llc) !=3D mm_sched_llc) { + account_llc_dequeue(rq, p); WRITE_ONCE(p->preferred_llc, mm_sched_llc); + account_llc_enqueue(rq, p); + } } =20 static void task_tick_cache(struct rq *rq, struct task_struct *p) @@ -1620,6 +1651,10 @@ static inline int get_pref_llc(struct task_struct *p, return -1; } =20 +static void account_llc_enqueue(struct rq *rq, struct task_struct *p) {} + +static void account_llc_dequeue(struct rq *rq, struct task_struct *p) {} + #endif /* CONFIG_SCHED_CACHE */ =20 /* @@ -4106,9 +4141,11 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct= sched_entity *se) { update_load_add(&cfs_rq->load, se->load.weight); if (entity_is_task(se)) { + struct task_struct *p =3D task_of(se); struct rq *rq =3D rq_of(cfs_rq); =20 - account_numa_enqueue(rq, task_of(se)); + account_numa_enqueue(rq, p); + account_llc_enqueue(rq, p); list_add(&se->group_node, &rq->cfs_tasks); } cfs_rq->nr_queued++; @@ -4119,7 +4156,11 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct= sched_entity *se) { update_load_sub(&cfs_rq->load, se->load.weight); if (entity_is_task(se)) { - account_numa_dequeue(rq_of(cfs_rq), task_of(se)); + struct task_struct *p =3D task_of(se); + struct rq *rq =3D rq_of(cfs_rq); + + account_numa_dequeue(rq, p); + account_llc_dequeue(rq, p); list_del_init(&se->group_node); } cfs_rq->nr_queued--; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9defeeeb3e8e..081f23a48414 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1195,6 +1195,12 @@ struct rq { #ifdef CONFIG_NUMA_BALANCING unsigned int numa_migrate_on; #endif + +#ifdef CONFIG_SCHED_CACHE + unsigned int nr_pref_llc_running; + unsigned int nr_llc_running; +#endif + /* * This is part of a global counter where only the total sum * over all CPUs matters. A task can increase this counter on @@ -2066,6 +2072,8 @@ init_numa_balancing(u64 clone_flags, struct task_stru= ct *p) =20 #endif /* !CONFIG_NUMA_BALANCING */ =20 +int task_llc(const struct task_struct *p); + static inline void queue_balance_callback(struct rq *rq, struct balance_callback *head, --=20 2.32.0