From nobody Mon May 25 00:09:41 2026 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 036463ED3AE; Wed, 20 May 2026 08:35:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779266101; cv=none; b=fvjXSnx01fuC/BOZBtYqa5RO7I2S66b85f7vUixXyx3BEdyxEaJmKQKm35SKFqBoobqwu2K6oyhkE+PQzbXSLpTB60gr+cszHu+UTSzUwXLmZ6ny2jzn5FcSg9VU54Pyyf8d1M80w7dmKgEIVGqykxZN91asAYprG++DEuE290M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779266101; c=relaxed/simple; bh=+72Q9KEbEoOdOeh7SCd61VOsjsC89uEjT+SqLKg7PHI=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=GbXMmgyaG553Mqzy1mhhPBSA1wj4lUeuCOWzi458h9O/mP91qZzzQtW2qEJRRyaWuLG1WpucZISU7M7VzX4GvaRSGPx4jnPFqYdHZSlyJcUfKmbgbnhxNGZa99zLXvaya+e9K2a9/tEQ//zDsqWj3njoVjKro6ZtGqIlZ7xU0lQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=lcBfxdNW; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=ADEw/Hg1; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="lcBfxdNW"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ADEw/Hg1" Date: Wed, 20 May 2026 08:34:57 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1779266098; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pEqshoOQmf9MUM5O8yOMoqVq6/zEBqNL9HYPkzVnuc0=; b=lcBfxdNWrNvva18FH/hUXUZcW90Ey7bM4SrXgmWvstQnAuj2tqxISMaWSZeZsSmh/eCZtX w8BSzsZxl1RwVXPIT/joz9tpjvWPhgxen59ZzJFhdOBZwHZvcfDqdTINg8DAz6A/6U808W HaIiyXwQacpCEBYYjuhTJavMtW0m7jWblvq+oCU398X+/BJz2lyJ9UfQ+DUVZJrXGNs4G1 rLUSsA+3KijxhYtBOHzE9n1HH+2gyl3NzzEbLBdb4m9FWJWespPVs5DUDi5xKRKzCg0ZtH gTL3LQo434U1xbnhulk3Cu36/6xUiBgwGxUlR0HKhmttdZuzISljM3QZq4+6+Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1779266098; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pEqshoOQmf9MUM5O8yOMoqVq6/zEBqNL9HYPkzVnuc0=; b=ADEw/Hg1XEpUUceZ/BF5a89uWVip5tfdZoonJ7cnYmac+xhYICfx8RTPZA7KnfdJwokNjR B6rWctunH55OtLCw== From: "tip-bot2 for Tim Chen" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/core] sched/cache: Make LLC id continuous Cc: "Peter Zijlstra (Intel)" , K Prateek Nayak , Chen Yu , Tim Chen , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: =?utf-8?q?=3C047ef46339e4db497b54a89940a7ebedf27fcf28=2E1775065?= =?utf-8?q?312=2Egit=2Etim=2Ec=2Echen=40linux=2Eintel=2Ecom=3E?= References: =?utf-8?q?=3C047ef46339e4db497b54a89940a7ebedf27fcf28=2E17750653?= =?utf-8?q?12=2Egit=2Etim=2Ec=2Echen=40linux=2Eintel=2Ecom=3E?= Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <177926609731.711.6243196736222305464.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the sched/core branch of tip: Commit-ID: b5ea300a17e37eada7a98561fbd34a3054578713 Gitweb: https://git.kernel.org/tip/b5ea300a17e37eada7a98561fbd34a305= 4578713 Author: Tim Chen AuthorDate: Wed, 01 Apr 2026 14:52:17 -07:00 Committer: Peter Zijlstra CommitterDate: Thu, 09 Apr 2026 15:49:48 +02:00 sched/cache: Make LLC id continuous Introduce an index mapping between CPUs and their LLCs. This provides a roughly continuous per LLC index needed for cache-aware load balancing in later patches. The existing per_cpu llc_id usually points to the first CPU of the LLC domain, which is sparse and unsuitable as an array index. Using llc_id directly would waste memory. With the new mapping, CPUs in the same LLC share an approximate continuous id: per_cpu(llc_id, CPU=3D0...15) =3D 0 per_cpu(llc_id, CPU=3D16...31) =3D 1 per_cpu(llc_id, CPU=3D32...47) =3D 2 ... Note that the LLC IDs are allocated via bitmask, so the IDs may be reused during CPU offline->online transitions. Suggested-by: Peter Zijlstra (Intel) Originally-by: K Prateek Nayak Co-developed-by: Chen Yu Signed-off-by: Chen Yu Signed-off-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/047ef46339e4db497b54a89940a7ebedf27fcf28.177= 5065312.git.tim.c.chen@linux.intel.com --- kernel/sched/core.c | 2 +- kernel/sched/sched.h | 3 +- kernel/sched/topology.c | 90 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 93 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7e0b55e..d11e27b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -8630,6 +8630,8 @@ int sched_cpu_deactivate(unsigned int cpu) */ synchronize_rcu(); =20 + sched_domains_free_llc_id(cpu); + sched_set_rq_offline(rq, cpu); =20 scx_rq_deactivate(rq); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index f939d45..3cb3ab0 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -4053,6 +4053,9 @@ static inline bool sched_cache_enabled(void) return false; } #endif + +void sched_domains_free_llc_id(int cpu); + extern void init_sched_mm(struct task_struct *p); =20 extern u64 avg_vruntime(struct cfs_rq *cfs_rq); diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 5847b83..1200670 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -19,8 +19,10 @@ void sched_domains_mutex_unlock(void) } =20 /* Protected by sched_domains_mutex: */ +static cpumask_var_t sched_domains_llc_id_allocmask; static cpumask_var_t sched_domains_tmpmask; static cpumask_var_t sched_domains_tmpmask2; +int max_lid; =20 static int __init sched_debug_setup(char *str) { @@ -663,7 +665,7 @@ static void destroy_sched_domains(struct sched_domain *= sd) */ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc); DEFINE_PER_CPU(int, sd_llc_size); -DEFINE_PER_CPU(int, sd_llc_id); +DEFINE_PER_CPU(int, sd_llc_id) =3D -1; DEFINE_PER_CPU(int, sd_share_id); DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa); @@ -692,7 +694,6 @@ static void update_top_cache_domain(int cpu) =20 rcu_assign_pointer(per_cpu(sd_llc, cpu), sd); per_cpu(sd_llc_size, cpu) =3D size; - per_cpu(sd_llc_id, cpu) =3D id; rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds); =20 sd =3D lowest_flag_domain(cpu, SD_CLUSTER); @@ -1790,6 +1791,11 @@ const struct cpumask *tl_mc_mask(struct sched_domain= _topology_level *tl, int cpu { return cpu_coregroup_mask(cpu); } + +#define llc_mask(cpu) cpu_coregroup_mask(cpu) + +#else +#define llc_mask(cpu) cpumask_of(cpu) #endif =20 const struct cpumask *tl_pkg_mask(struct sched_domain_topology_level *tl, = int cpu) @@ -2650,6 +2656,61 @@ static void adjust_numa_imbalance(struct sched_domai= n *sd_llc) } } =20 +static int __sched_domains_alloc_llc_id(void) +{ + int lid, max; + + lockdep_assert_held(&sched_domains_mutex); + + lid =3D cpumask_first_zero(sched_domains_llc_id_allocmask); + /* + * llc_id space should never grow larger than the + * possible number of CPUs in the system. + */ + if (lid >=3D nr_cpu_ids) + return -1; + + __cpumask_set_cpu(lid, sched_domains_llc_id_allocmask); + max =3D cpumask_last(sched_domains_llc_id_allocmask); + if (max > max_lid) + max_lid =3D max; + + return lid; +} + +static void __sched_domains_free_llc_id(int cpu) +{ + int i, lid, max; + + lockdep_assert_held(&sched_domains_mutex); + + lid =3D per_cpu(sd_llc_id, cpu); + if (lid =3D=3D -1 || lid >=3D nr_cpu_ids) + return; + + per_cpu(sd_llc_id, cpu) =3D -1; + + for_each_cpu(i, llc_mask(cpu)) { + /* An online CPU owns the llc_id. */ + if (per_cpu(sd_llc_id, i) =3D=3D lid) + return; + } + + __cpumask_clear_cpu(lid, sched_domains_llc_id_allocmask); + + max =3D cpumask_last(sched_domains_llc_id_allocmask); + /* shrink max lid to save memory */ + if (max < max_lid) + max_lid =3D max; +} + +void sched_domains_free_llc_id(int cpu) +{ + sched_domains_mutex_lock(); + __sched_domains_free_llc_id(cpu); + sched_domains_mutex_unlock(); +} + /* * Build sched domains for a given set of CPUs and attach the sched domains * to the individual CPUs @@ -2675,6 +2736,7 @@ build_sched_domains(const struct cpumask *cpu_map, st= ruct sched_domain_attr *att /* Set up domains for CPUs specified by the cpu_map: */ for_each_cpu(i, cpu_map) { struct sched_domain_topology_level *tl; + int lid; =20 sd =3D NULL; for_each_sd_topology(tl) { @@ -2688,6 +2750,29 @@ build_sched_domains(const struct cpumask *cpu_map, s= truct sched_domain_attr *att if (cpumask_equal(cpu_map, sched_domain_span(sd))) break; } + + lid =3D per_cpu(sd_llc_id, i); + if (lid =3D=3D -1) { + /* try to reuse the llc_id of its siblings */ + for (int j =3D cpumask_first(llc_mask(i)); + j < nr_cpu_ids; + j =3D cpumask_next(j, llc_mask(i))) { + if (i =3D=3D j) + continue; + + lid =3D per_cpu(sd_llc_id, j); + + if (lid !=3D -1) { + per_cpu(sd_llc_id, i) =3D lid; + + break; + } + } + + /* a new LLC is detected */ + if (lid =3D=3D -1) + per_cpu(sd_llc_id, i) =3D __sched_domains_alloc_llc_id(); + } } =20 if (WARN_ON(!topology_span_sane(cpu_map))) @@ -2831,6 +2916,7 @@ int __init sched_init_domains(const struct cpumask *c= pu_map) { int err; =20 + zalloc_cpumask_var(&sched_domains_llc_id_allocmask, GFP_KERNEL); zalloc_cpumask_var(&sched_domains_tmpmask, GFP_KERNEL); zalloc_cpumask_var(&sched_domains_tmpmask2, GFP_KERNEL); zalloc_cpumask_var(&fallback_doms, GFP_KERNEL);