From nobody Mon Jun 8 07:24:34 2026 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2151C3D25C8; Thu, 4 Jun 2026 18:45:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780598735; cv=none; b=LZprAfiU64I5Pianq+xqypRQRETMJCeFLw2f7F6GZZzX1Hz+n0s+xjh0HBI3bILA33W92lVPtOb/4tK0ewe7dYOVzcy4GeKtMJsa14vtvyLJtmswNFcOSZQ5yXistvT8zuojU7pFGAB2hiOHfZHRloqtL+f7H3dWagqBiwO6P0c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780598735; c=relaxed/simple; bh=zRYeRe6INGV111J7QOfm+lb+pkEEqjitMurfOKxt3yo=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=RAi2+P7aegvN3D45TEKEFfOC3tnsImQGN4rKewp7fxMnEh/6GWYFfjNw7tuJD9Cu/fQjI96rWidZu4FxD+pvv0ikn9N5qW+AS5ySr9TPRsZrYtZJdnTpFA+tRjiN5i3CrYCVp6dGyZAbYCymr79y3In/VOMqXxYIqekFmtToZcA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=s+Rei8Gf; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=6reoiwes; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="s+Rei8Gf"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="6reoiwes" Date: Thu, 04 Jun 2026 18:45:31 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1780598732; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ioP/fmBGpT0gDXNnNu9mxvtYF37XFu2/K/9ftyxdwAE=; b=s+Rei8GftruyrVuXbkLwya91dNoc3Th3qV5aNgNXTWuo04jSZy83aAxUCHjg5QI03T9YAF WqqoaC1ke0fnu3/Ve3nll4BknFMsHu096tnMo/jtRY6cDMxCmAzMzoQWbTDcdPk4D29S3G 40Pv3p92+IUOeh8W1T6NwkFms/xvYN7AKcg1Li6AhRLV2SGUlxZhbRifle1GqVwY76AzAo X78uUMgh9t4gOlg0RvhqJeaOMeqaJn7gFy4MgU01kCdGH2qpm9UTMTR9wbshPVwEgR0FfP 2yANj1KSyhoPN2R8cOrwbokbAmZypJi8i3q0Rv8aWQlgNujBg00yL2eOtpbOcQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1780598732; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ioP/fmBGpT0gDXNnNu9mxvtYF37XFu2/K/9ftyxdwAE=; b=6reoiwesMdOJaDqsaXW4p2be0VydTVD3TU8NC9wyd7wczbMuEf8bqjieKUYGceVsCpf8PS 8rWrNusnoqEQ7TDQ== From: "tip-bot2 for Zecheng Li" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: sched/core] sched/fair: Co-locate cfs_rq and sched_entity in cfs_tg_state Cc: Zecheng Li , Zecheng Li , "Peter Zijlstra (Intel)" , K Prateek Nayak , Josh Don , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20260522141623.600235-2-zli94@ncsu.edu> References: <20260522141623.600235-2-zli94@ncsu.edu> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <178059873129.710.15581737529187615443.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the sched/core branch of tip: Commit-ID: dfcfc97b6df0ea8e1b7d3b590022782abbec3389 Gitweb: https://git.kernel.org/tip/dfcfc97b6df0ea8e1b7d3b590022782ab= bec3389 Author: Zecheng Li AuthorDate: Fri, 22 May 2026 10:15:48 -04:00 Committer: Peter Zijlstra CommitterDate: Tue, 02 Jun 2026 12:26:10 +02:00 sched/fair: Co-locate cfs_rq and sched_entity in cfs_tg_state Improve data locality and reduce pointer chasing by allocating struct cfs_rq and struct sched_entity together for non-root task groups. This is achieved by introducing a new combined struct cfs_tg_state that holds both objects in a single allocation. This patch: - Introduces struct cfs_tg_state that embeds cfs_rq, sched_entity, and sched_statistics together in a single structure. - Updates __schedstats_from_se() in stats.h to use cfs_tg_state for access= ing sched_statistics from a group sched_entity. - Modifies alloc_fair_sched_group() and free_fair_sched_group() to allocate and free the new struct as a single unit. - Modifies the per-CPU pointers in task_group->se and task_group->cfs_rq to point to the members in the new combined structure. Signed-off-by: Zecheng Li Signed-off-by: Zecheng Li Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: K Prateek Nayak Reviewed-by: Josh Don Link: https://patch.msgid.link/20260522141623.600235-2-zli94@ncsu.edu --- kernel/sched/fair.c | 18 ++++++------------ kernel/sched/sched.h | 12 ++++++++++++ kernel/sched/stats.h | 9 +-------- 3 files changed, 19 insertions(+), 20 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 7816270..e7d7d47 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -15083,8 +15083,6 @@ void free_fair_sched_group(struct task_group *tg) for_each_possible_cpu(i) { if (tg->cfs_rq) kfree(tg->cfs_rq[i]); - if (tg->se) - kfree(tg->se[i]); } =20 kfree(tg->cfs_rq); @@ -15093,6 +15091,7 @@ void free_fair_sched_group(struct task_group *tg) =20 int alloc_fair_sched_group(struct task_group *tg, struct task_group *paren= t) { + struct cfs_tg_state *state; struct sched_entity *se; struct cfs_rq *cfs_rq; int i; @@ -15109,16 +15108,13 @@ int alloc_fair_sched_group(struct task_group *tg,= struct task_group *parent) init_cfs_bandwidth(tg_cfs_bandwidth(tg), tg_cfs_bandwidth(parent)); =20 for_each_possible_cpu(i) { - cfs_rq =3D kzalloc_node(sizeof(struct cfs_rq), - GFP_KERNEL, cpu_to_node(i)); - if (!cfs_rq) + state =3D kzalloc_node(sizeof(*state), + GFP_KERNEL, cpu_to_node(i)); + if (!state) goto err; =20 - se =3D kzalloc_node(sizeof(struct sched_entity_stats), - GFP_KERNEL, cpu_to_node(i)); - if (!se) - goto err_free_rq; - + cfs_rq =3D &state->cfs_rq; + se =3D &state->se; init_cfs_rq(cfs_rq); init_tg_cfs_entry(tg, cfs_rq, se, i, parent->se[i]); init_entity_runnable_average(se); @@ -15126,8 +15122,6 @@ int alloc_fair_sched_group(struct task_group *tg, s= truct task_group *parent) =20 return 1; =20 -err_free_rq: - kfree(cfs_rq); err: return 0; } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b3aff26..585aba9 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2294,6 +2294,18 @@ static inline struct task_group *task_group(struct t= ask_struct *p) return p->sched_task_group; } =20 +#ifdef CONFIG_FAIR_GROUP_SCHED +/* + * Defined here to be available before stats.h is included, since + * stats.h has dependencies on things defined later in this file. + */ +struct cfs_tg_state { + struct cfs_rq cfs_rq; + struct sched_entity se; + struct sched_statistics stats; +} __no_randomize_layout; +#endif + /* Change a task's cfs_rq and parent entity if it moves across CPUs/groups= */ static inline void set_task_rq(struct task_struct *p, unsigned int cpu) { diff --git a/kernel/sched/stats.h b/kernel/sched/stats.h index a612cf2..ebe0a77 100644 --- a/kernel/sched/stats.h +++ b/kernel/sched/stats.h @@ -89,19 +89,12 @@ static inline void rq_sched_info_depart (struct rq *rq= , unsigned long long delt =20 #endif /* CONFIG_SCHEDSTATS */ =20 -#ifdef CONFIG_FAIR_GROUP_SCHED -struct sched_entity_stats { - struct sched_entity se; - struct sched_statistics stats; -} __no_randomize_layout; -#endif - static inline struct sched_statistics * __schedstats_from_se(struct sched_entity *se) { #ifdef CONFIG_FAIR_GROUP_SCHED if (!entity_is_task(se)) - return &container_of(se, struct sched_entity_stats, se)->stats; + return &container_of(se, struct cfs_tg_state, se)->stats; #endif return &task_of(se)->stats; }