From nobody Sat Sep 26 23:51:52 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C75543E2AB6 for ; Fri, 28 Aug 2026 07:56:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787903818; cv=none; b=pqak1OnVwh1pakR7M64epimGnP8J65cHXwKlyC2VZYLcSa+9ByIs5gNUYHqZWeuamFa7TVMx3W/Xhw8gDGlQVloj6e7JyVnUbkPiuX0dqqi7iDltGOTKblRYWWDaAj++lVmm8k/2RV66sro3FArC35xI9GZp+sGTmHqCk+v2mio= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787903818; c=relaxed/simple; bh=ro634XVMhtw9WYQC2xn1URYgTBufOhqDk9NHwyb7fzE=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=ENb4wKo5uId+IkWqSQAENSxg2MGO0q6eIAZoe2jupSFC2TxUDKt+stAy9ExDEhauPdP+6on8wq2chT7qmVvurKe+7VVC6Y2d7sFfXuMJDwqZCz/ahIpN2HcA8s0TTX75Dthj70BEOm3L+UGnHGqkIlAS8bmJmO6tAzCBs181bAU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=U2QcOiW1; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="U2QcOiW1" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=zsXQHZ11n0M3yUj8f8e+hKw2WpdtXgIQ8/yIqhe1w/A=; b=U2QcOiW14oY1O5rqO4aQ1onuXS SzXS0P5/gknvm6KlHtP3nCHYMZ+5U3Ns8mOyCjWKrXzcCq9GRrx7vPKlhbVSSay24ldsl+J6tLcDE XmvpbNGz4oo0Ms7YBCcyO/32vj7l3fmP9orjRdjuAHjZ7nA2br3U+vTWEEQOyJbvFi3jh3fiFfm+K 1qJqhFlp8yno+oQGe81uJIRfi2mxvroN50ZJuuC9P3ps9FT2JTOHT/vPNfMckSK7LVBvRucdeET0o RGBCJexeitQGzIIRuaEK11dHvJYb6w+V/SPRM1o4EH9z4AbvrNPhgZarZFWcFWFvaWvMP0RWdyjso e3VUBQug==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.99.2 #2 (Red Hat Linux)) id 1wzrS1-000000086Qz-3CVA; Fri, 28 Aug 2026 07:56:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 3E6963006FA; Fri, 28 Aug 2026 09:56:36 +0200 (CEST) Message-ID: <20260828075558.320346065@infradead.org> User-Agent: quilt/0.68 Date: Fri, 28 Aug 2026 09:41:00 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com, tj@kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/4] sched: Rename/clarify sched_class::task_tick(.queued) argument References: <20260828074059.232353141@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For some reason the sched_class::task_tick() argument that indicates it bei= ng an hrtick, is called @queued. I'm sure naming is hard and all, but lets fix this. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/deadline.c | 4 ++-- kernel/sched/ext/ext.c | 2 +- kernel/sched/fair.c | 12 ++++++------ kernel/sched/idle.c | 2 +- kernel/sched/rt.c | 2 +- kernel/sched/sched.h | 2 +- 6 files changed, 12 insertions(+), 12 deletions(-) --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2876,7 +2876,7 @@ static void put_prev_task_dl(struct rq * * and everything must be accessed through the @rq and @curr passed in * parameters. */ -static void task_tick_dl(struct rq *rq, struct task_struct *p, int queued) +static void task_tick_dl(struct rq *rq, struct task_struct *p, int hrtick) { update_curr_dl(rq); =20 @@ -2886,7 +2886,7 @@ static void task_tick_dl(struct rq *rq, * not being the leftmost task anymore. In that case NEED_RESCHED will * be set and schedule() will start a new hrtick for the next task. */ - if (hrtick_enabled_dl(rq) && queued && p->dl.runtime > 0 && + if (hrtick_enabled_dl(rq) && hrtick && p->dl.runtime > 0 && is_leftmost(&p->dl, &rq->dl)) start_hrtick_dl(rq, &p->dl); } --- a/kernel/sched/ext/ext.c +++ b/kernel/sched/ext/ext.c @@ -3787,7 +3787,7 @@ void scx_tick(struct rq *rq) update_other_load_avgs(rq); } =20 -static void task_tick_scx(struct rq *rq, struct task_struct *curr, int que= ued) +static void task_tick_scx(struct rq *rq, struct task_struct *curr, int hrt= ick) { struct scx_sched *sch =3D scx_task_sched(curr); =20 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6542,7 +6542,7 @@ static void put_prev_entity(struct cfs_r } =20 static void -entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int queued) +entity_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr, int hrtick) { /* * Update run-time statistics of the 'current'. @@ -6557,10 +6557,10 @@ entity_tick(struct cfs_rq *cfs_rq, struc =20 #ifdef CONFIG_SCHED_HRTICK /* - * queued ticks are scheduled to match the slice, so don't bother + * hrticks are scheduled to match the slice, so don't bother * validating it and just reschedule. */ - if (queued) { + if (hrtick) { resched_curr(rq_of(cfs_rq)); return; } @@ -15020,7 +15020,7 @@ static inline void task_tick_core(struct * and everything must be accessed through the @rq and @curr passed in * parameters. */ -static void task_tick_fair(struct rq *rq, struct task_struct *curr, int qu= eued) +static void task_tick_fair(struct rq *rq, struct task_struct *curr, int hr= tick) { struct sched_entity *se =3D &curr->se; =20 @@ -15030,7 +15030,7 @@ static void task_tick_fair(struct rq *rq =20 for_each_sched_entity(se) { cfs_rq =3D cfs_rq_of(se); - entity_tick(cfs_rq, se, queued); + entity_tick(cfs_rq, se, hrtick); =20 weight =3D __calc_prop_weight(cfs_rq, se, weight); } @@ -15039,7 +15039,7 @@ static void task_tick_fair(struct rq *rq reweight_eevdf(cfs_rq, se, weight, se->on_rq); } =20 - if (queued) + if (hrtick) return; =20 if (static_branch_unlikely(&sched_numa_balancing)) --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -535,7 +535,7 @@ dequeue_task_idle(struct rq *rq, struct * and everything must be accessed through the @rq and @curr passed in * parameters. */ -static void task_tick_idle(struct rq *rq, struct task_struct *curr, int qu= eued) +static void task_tick_idle(struct rq *rq, struct task_struct *curr, int hr= tick) { update_curr_idle(rq); } --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -2538,7 +2538,7 @@ static inline void watchdog(struct rq *r * and everything must be accessed through the @rq and @curr passed in * parameters. */ -static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued) +static void task_tick_rt(struct rq *rq, struct task_struct *p, int hrtick) { struct sched_rt_entity *rt_se =3D &p->rt; =20 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2718,7 +2718,7 @@ struct sched_class { * sched_tick: rq->lock * sched_tick_remote: rq->lock */ - void (*task_tick)(struct rq *rq, struct task_struct *p, int queued); + void (*task_tick)(struct rq *rq, struct task_struct *p, int hrtick); /* * sched_cgroup_fork: p->pi_lock */ From nobody Sat Sep 26 23:51:52 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C763B3E2ABA for ; Fri, 28 Aug 2026 07:56:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787903816; cv=none; b=r+ucCfMCf4njv9ed6/HccQFN3iJhNRn4ihuuFAeftYiM3GctkHH+kZc8HUy/V0Yno71EQhKFhEAnRqc/bUzq//Ahiyx720qBF5563rf+JlsnSjpjgF41y7bYpTpUUHdBvZCmZX6OiZZ0ZivxdzOyHq2zNHJAN+vM2nMu7a9NpYo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787903816; c=relaxed/simple; bh=SaMMzbjl6vWNTPXE3AZLpzSOl7NrxKo9JJPuMhsHP6A=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=NdR+1cVoKuniChn7BAEeDv2lHCku6AXQ2tcXHLXNGRoWsLbT1CTMR03O3ELAUw546d3oIGUB0LMEHfaRdCOjaSYKboMTAyarrLUB6DlP1oHn6gJiJZiom9V6jKTm/Nzp6Wv6N8FopPnrFxWs1IQ1WTC2qkz8YaZsE9WiNGw31W0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=MDdq5Epw; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="MDdq5Epw" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=LLqdVHIkMiv21YiQRIUu7pIEo2mOwrqXr51hi7vMDCQ=; b=MDdq5EpwwhgNWz2LfuBaaXTuwL lLAt1ecvyKCwSVZkmTz2FNkfbW/gw+/l6epKwsZrZaGT4V/ec74JqvGy4T8hLPdcbogaTivqKSR0w Ge7EnhyjcHU7DkjLuw0jyo9nTQm8P3Cfq1icZ97tx8xgBEOPcDFWWwIIDYO6qdDLolCL11L1TCdjB zFDrg7W0fOsUDFPdjynqX5Sh6mMVrhga/6i9ylwDKnwEueGxTAEcCqkYeI56HnlLRSW63LnIT/E5C z9CoexL9f6/55vVvDk94Zq2z6RgDM4sWcytAVrN8WziFLOViREnDL3h7IeYwvNrsmt0ven54C6tPH /pjbeVCA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.99.2 #2 (Red Hat Linux)) id 1wzrS1-000000086R0-3ECb; Fri, 28 Aug 2026 07:56:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 42DA83008E2; Fri, 28 Aug 2026 09:56:36 +0200 (CEST) Message-ID: <20260828075558.438093356@infradead.org> User-Agent: quilt/0.68 Date: Fri, 28 Aug 2026 09:41:01 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com, tj@kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/4] sched/fair: Fold cfs_rq_of(se) into for_each_sched_entity() References: <20260828074059.232353141@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Pretty much every for_each_sched_entity() loop does: cfs_rq =3D cfs_rq_of(s= e) as the very first thing. Fold it into the for_each_sched_entity() macro. This paves the way to have the macro track a backlink transparantly. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/fair.c | 81 +++++++++++++++++++++++------------------------= ----- 1 file changed, 37 insertions(+), 44 deletions(-) --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -317,8 +317,8 @@ const struct sched_class fair_sched_clas #ifdef CONFIG_FAIR_GROUP_SCHED =20 /* Walk up scheduling entities hierarchy */ -#define for_each_sched_entity(se) \ - for (; se; se =3D se->parent) +#define for_each_sched_entity(se, cfs_rq) \ + for (; (se) && ((cfs_rq) =3D cfs_rq_of(se)); (se) =3D (se)->parent) =20 static inline bool list_add_leaf_cfs_rq(struct cfs_rq *cfs_rq) { @@ -452,8 +452,8 @@ static int se_is_idle(struct sched_entit =20 #else /* !CONFIG_FAIR_GROUP_SCHED: */ =20 -#define for_each_sched_entity(se) \ - for (; se; se =3D NULL) +#define for_each_sched_entity(se, cfs_rq) \ + for (; (se) && ((cfs_rq) =3D cfs_of_of(se)); (se) =3D NULL) =20 static inline bool list_add_leaf_cfs_rq(struct cfs_rq *cfs_rq) { @@ -2075,9 +2075,10 @@ static void update_curr(struct cfs_rq *c static void update_curr_fair(struct rq *rq) { struct sched_entity *se =3D &rq->donor->se; + struct cfs_rq *cfs_rq; =20 - for_each_sched_entity(se) - update_curr(cfs_rq_of(se)); + for_each_sched_entity(se, cfs_rq) + update_curr(cfs_rq); } =20 static inline void @@ -4800,18 +4801,19 @@ static void reweight_task_fair(struct rq { struct sched_entity *se =3D &p->se; unsigned long weight =3D NICE_0_LOAD; + struct cfs_rq *cfs_rq =3D cfs_rq_of(se); =20 if (se->on_rq) update_curr_fair(rq); =20 - reweight_entity(cfs_rq_of(se), se, lw->weight); + reweight_entity(cfs_rq, se, lw->weight); se->load.inv_weight =3D lw->inv_weight; =20 if (!se->on_rq) return; =20 - for_each_sched_entity(se) - weight =3D __calc_prop_weight(cfs_rq_of(se), se, weight); + for_each_sched_entity(se, cfs_rq) + weight =3D __calc_prop_weight(cfs_rq, se, weight); =20 reweight_eevdf(&rq->cfs, &p->se, weight, p->se.on_rq); } @@ -6382,6 +6384,8 @@ static __always_inline void return_cfs_r =20 static void set_delayed(struct sched_entity *se) { + struct cfs_rq *cfs_rq; + se->sched_delayed =3D 1; =20 /* @@ -6392,15 +6396,14 @@ static void set_delayed(struct sched_ent if (!entity_is_task(se)) return; =20 - for_each_sched_entity(se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - + for_each_sched_entity(se, cfs_rq) cfs_rq->h_nr_runnable--; - } } =20 static void clear_delayed(struct sched_entity *se) { + struct cfs_rq *cfs_rq; + se->sched_delayed =3D 0; =20 /* @@ -6412,11 +6415,8 @@ static void clear_delayed(struct sched_e if (!entity_is_task(se)) return; =20 - for_each_sched_entity(se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - + for_each_sched_entity(se, cfs_rq) cfs_rq->h_nr_runnable++; - } } =20 static void @@ -7078,14 +7078,16 @@ void unthrottle_cfs_rq(struct cfs_rq *cf walk_tg_tree_from(cfs_rq->tg, tg_nop, tg_unthrottle_up, (void *)rq); =20 if (!cfs_rq->load.weight) { + struct cfs_rq *cfs_rq_se; + if (!cfs_rq->on_list) return; /* * Nothing to run but something to decay (on_list)? * Complete the branch. */ - for_each_sched_entity(se) { - if (list_add_leaf_cfs_rq(cfs_rq_of(se))) + for_each_sched_entity(se, cfs_rq_se) { + if (list_add_leaf_cfs_rq(cfs_rq_se)) break; } } @@ -7931,13 +7933,12 @@ static unsigned long enqueue_hierarchy(s struct sched_entity *se =3D &p->se; int h_nr_idle =3D task_has_idle_policy(p); int h_nr_runnable =3D 1; + struct cfs_rq *cfs_rq; =20 if (task_new && se->sched_delayed) h_nr_runnable =3D 0; =20 - for_each_sched_entity(se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - + for_each_sched_entity(se, cfs_rq) { update_curr(cfs_rq); =20 if (!se->on_rq) { @@ -8066,16 +8067,15 @@ static void dequeue_hierarchy(struct tas bool task_sleep =3D flags & DEQUEUE_SLEEP; bool task_delayed =3D flags & DEQUEUE_DELAYED; bool task_throttled =3D flags & DEQUEUE_THROTTLE; - int h_nr_runnable =3D 0; int h_nr_idle =3D task_has_idle_policy(p); + int h_nr_runnable =3D 0; + struct cfs_rq *cfs_rq; bool dequeue =3D true; =20 if (task_sleep || task_delayed || !se->sched_delayed) h_nr_runnable =3D 1; =20 - for_each_sched_entity(se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - + for_each_sched_entity(se, cfs_rq) { update_curr(cfs_rq); =20 if (dequeue) { @@ -11301,8 +11301,7 @@ static void update_cfs_rq_h_load(struct return; =20 WRITE_ONCE(cfs_rq->h_load_next, NULL); - for_each_sched_entity(se) { - cfs_rq =3D cfs_rq_of(se); + for_each_sched_entity(se, cfs_rq) { WRITE_ONCE(cfs_rq->h_load_next, se); if (cfs_rq->last_h_load_update =3D=3D now) break; @@ -14948,9 +14947,9 @@ static inline void task_tick_core(struct static void se_fi_update(const struct sched_entity *se, unsigned int fi_se= q, bool forceidle) { - for_each_sched_entity(se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); + struct cfs_rq *cfs_rq; =20 + for_each_sched_entity(se, cfs_rq) { if (forceidle) { if (cfs_rq->forceidle_seq =3D=3D fi_seq) break; @@ -15028,10 +15027,8 @@ static void task_tick_fair(struct rq *rq unsigned long weight =3D NICE_0_LOAD; struct cfs_rq *cfs_rq; =20 - for_each_sched_entity(se) { - cfs_rq =3D cfs_rq_of(se); + for_each_sched_entity(se, cfs_rq) { entity_tick(cfs_rq, se, hrtick); - weight =3D __calc_prop_weight(cfs_rq, se, weight); } =20 @@ -15113,9 +15110,7 @@ static void propagate_entity_cfs_rq(stru /* Start to propagate at parent */ se =3D se->parent; =20 - for_each_sched_entity(se) { - cfs_rq =3D cfs_rq_of(se); - + for_each_sched_entity(se, cfs_rq) { update_load_avg(cfs_rq, se, UPDATE_TG); =20 if (!cfs_rq_pelt_clock_throttled(cfs_rq)) @@ -15218,9 +15213,7 @@ static void set_next_task_fair(struct rq if (on_rq) __dequeue_entity(cfs_rq, se); =20 - for_each_sched_entity(se) { - cfs_rq =3D cfs_rq_of(se); - + for_each_sched_entity(se, cfs_rq) { if (!IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) || !first || !cfs_rq->h_curr) set_next_entity(cfs_rq, se); @@ -15420,13 +15413,14 @@ static int __sched_group_set_shares(stru for_each_possible_cpu(i) { struct rq *rq =3D cpu_rq(i); struct sched_entity *se =3D tg_se(tg, i); + struct cfs_rq *cfs_rq; struct rq_flags rf; =20 /* Propagate contribution to hierarchy */ rq_lock_irqsave(rq, &rf); update_rq_clock(rq); - for_each_sched_entity(se) { - update_load_avg(cfs_rq_of(se), se, UPDATE_TG); + for_each_sched_entity(se, cfs_rq) { + update_load_avg(cfs_rq, se, UPDATE_TG); update_cfs_group(se); } rq_unlock_irqrestore(rq, &rf); @@ -15473,6 +15467,7 @@ int sched_group_set_idle(struct task_gro struct sched_entity *se =3D tg_se(tg, i); struct cfs_rq *grp_cfs_rq =3D tg_cfs_rq(tg, i); bool was_idle =3D cfs_rq_is_idle(grp_cfs_rq); + struct cfs_rq *cfs_rq; long idle_task_delta; struct rq_flags rf; =20 @@ -15487,9 +15482,7 @@ int sched_group_set_idle(struct task_gro if (!cfs_rq_is_idle(grp_cfs_rq)) idle_task_delta *=3D -1; =20 - for_each_sched_entity(se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - + for_each_sched_entity(se, cfs_rq) { if (!se->on_rq) break; From nobody Sat Sep 26 23:51:52 2026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 387173F106A for ; Fri, 28 Aug 2026 07:56:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.50.34 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787903810; cv=none; b=ft8PpLeGNTFja/b5yVJGCSmv0703opdaBBe7pwRlu8fc6opxEuUmTf95pkZBRjQt2UBtfqUlRqeX+Rm4tBmLcWwal3etMvxJRxNGwRkY3PrMdicY81YDKg0B+3h7Z3tJQ0le5wjAi/Pe8v1ltgOGep1HLDG61yRpNqOaN7zdNxU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787903810; c=relaxed/simple; bh=UNcuZVeYp4Bj5abLNKnDw5Ks1ljRuBQVfJiXPxSxGsU=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=YCd8TiscvmU6ntvfYFamb3WudDIaj+ONlo5A03G89S9d7os7KN1NLrbboaFVQ/5Y8zKy0YdUDvz36WmoEZE2EjppVwIqzoIDTzjhi4z5KEnQr15k/n0iXmveKANCRzi5wICNweB/IeVqG5s/xjCm7LkjS2sl+rIeJFa5ep8OmiI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=pmMKbK9Y; arc=none smtp.client-ip=90.155.50.34 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="pmMKbK9Y" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=gKZldRxGMu2/vJWAh+lpOQibZ85xMWnHyys3uDe5DSc=; b=pmMKbK9YJ3BavoN9qXH6udMmRw ZARIgDnbnKhmyPLpMRFOy/u6gitCPeQBBCsb+eVzOa0UGl0xFHhLaOo3bDMUGJWtydEVKIkx0RJvj N/QoCSRLrmJKj27bh3SwgHl+WS8DryA6YlwJQcD5oj4B8pwW645LlTSq+zGdUgc9etdVsXqpmGGbn IviH9RGd0hy08ctJxVDOh2MMxiZZjAg8EC+nrRnBeIxhzevMTyhzc76pXBnpmQIflf4t59ecCrW6n 9ho5x6NfHQtFsIX7gEUlYCyA2xNvTevEbL1hRNs6jx+qZUDEh6KfC4WWFQrlX/sAMNvx+zGtg2ypw XVfik7bA==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.99.1 #2 (Red Hat Linux)) id 1wzrS1-0000000EGKI-0vQr; Fri, 28 Aug 2026 07:56:37 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 4790A30301D; Fri, 28 Aug 2026 09:56:36 +0200 (CEST) Message-ID: <20260828075558.542761795@infradead.org> User-Agent: quilt/0.68 Date: Fri, 28 Aug 2026 09:41:02 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com, tj@kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/4] sched/fair: Extend for_each_sched_entity() with a back-link References: <20260828074059.232353141@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Leave a trail of bread crumbs, such that we can find out way back down the hierarchy. No actual users yet, but split out because its a bit tricky. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/fair.c | 9 +++++++-- kernel/sched/sched.h | 1 + 2 files changed, 8 insertions(+), 2 deletions(-) --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -317,8 +317,13 @@ const struct sched_class fair_sched_clas #ifdef CONFIG_FAIR_GROUP_SCHED =20 /* Walk up scheduling entities hierarchy */ -#define for_each_sched_entity(se, cfs_rq) \ - for (; (se) && ((cfs_rq) =3D cfs_rq_of(se)); (se) =3D (se)->parent) +#define for_each_sched_entity(se, cfs_rq) \ + for (struct sched_entity *_BL =3D NULL; \ + (se) && ((cfs_rq) =3D cfs_rq_of(se), (cfs_rq)->backlink =3D _BL, tru= e);\ + (se) =3D (se)->parent, _BL =3D (se)) + +#define for_each_sched_entity_bl(se, cfs_rq) \ + for (; ((se) =3D (cfs_rq)->backlink); (cfs_rq) =3D group_cfs_rq(se)) =20 static inline bool list_add_leaf_cfs_rq(struct cfs_rq *cfs_rq) { --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -726,6 +726,7 @@ struct cfs_rq { unsigned long tg_runnable_avg_contrib; long propagate; long prop_runnable_sum; + struct sched_entity *backlink; =20 /* * h_load =3D weight * f(tg) From nobody Sat Sep 26 23:51:52 2026 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C77523E2ACA for ; Fri, 28 Aug 2026 07:56:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=90.155.92.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787903818; cv=none; b=WoSldvBT1c3bnBT74hud54bkX8ZLFtUy5cm30sfll/XB2ARjVn6q7+bcluRVTW2wZnrxLlnJIXqsCMDHU9hBySrEIcdD6YYqggXRzWmWBTzd3DgX0iji7pITZdcr4QtLx8+tTyF971HU6fBq2hmMRehkWrIwZMtXiOiAjPx+3hA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787903818; c=relaxed/simple; bh=zRlXqD+YzJf1lXrv65yKym55u+K3iRGHQUtIfc9XQ/4=; h=Message-ID:Date:From:To:Cc:Subject:References:MIME-Version: Content-Type; b=uHiVzG0f+AwQZYnpOMxpYrjMSzY1mRku7tjbTmiAp1eqSe/YDpekskUVvd4fxgzMVfcHYfAgvOKWkHrv4Us/UFUXNSHVtllEIwq5qt7pysDP3oSZrdVT7rEAcwh+BP/v7O0s038u9183luJcaqN5Z+tO3LXlQNdtHkODddklJYM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org; spf=pass smtp.mailfrom=infradead.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b=g7xOlsvL; arc=none smtp.client-ip=90.155.92.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="g7xOlsvL" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=qieUllO7DZXMqKGNoxOhT3YyL0Qj+wnboIe3/JX7lq8=; b=g7xOlsvL/Xx3OLIsGQYntVlzDy O4hDqWqkFnmRudz67ATpBJQivkAvOOeGA/nGyfRxjEd8LumxlqE6mn9mA1oNipm6JjKIrCEXTvihH wmsHNaPJpTrlYmthJQrhrThgpujLSoPU4Vdw9TSV5EFSd0/sHHkU3c4QtoO3/zcmrxDk7VxcTel89 6syjes85X6bBerXh8FEBX4HLfFOa9xUb6juKBXcCPeG2nJYcZxzpIKpxfW2G8UMY97MA9AIABnSWH p560psWEr0dy4pj9ff9T7in8EslRo96ML5wei/2hF4QD17IF5dzi+LTI4yup1GGEsRGauFNDql5Wo QuP/QjXw==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.99.2 #2 (Red Hat Linux)) id 1wzrS1-000000086R1-3AyS; Fri, 28 Aug 2026 07:56:38 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id 4CA40303027; Fri, 28 Aug 2026 09:56:36 +0200 (CEST) Message-ID: <20260828075558.660152190@infradead.org> User-Agent: quilt/0.68 Date: Fri, 28 Aug 2026 09:41:03 +0200 From: Peter Zijlstra To: mingo@kernel.org Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com, tj@kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/4] sched/fair: Rework/fix task_h_load() References: <20260828074059.232353141@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There are a number of issues with task_h_load(): - its hierarchy traversal is racy to the point of being broken; where originally it was meant to be used under rq->lock, but lacking an assert= ion for that fact, its use spread and violated this. The result is that the back-link state is prone to races. - it is rate-limited on jiffies, which is HZ, not the underlying PELT deca= y. - since its update is tied to task_h_load() usage, the cfs_rq->h_load numb= ers are not often 'up-to-date', rendering their output in sched/debug near usele= ss. Rework the whole thing to keep a more up-to-date and less broken cfs_rq->h_= load number. Move the back-link tracking into for_each_sched_entity(), such that any such loop sets up a path back. Use this to (optionally) re-compute cfs_rq->h_loa= d on enqueue, dequeue, set_next and tick, all sites that hold rq->lock. This ensures that 'active' cgroups have reasonably up-to-date cfs_rq->h_loa= d. Additionally, have __update_blocked_fair() update cfs_rq->h_load for all cgroups. Finally, replace the jiffy rate-limit with one that is tied to the PELT dec= ay. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/fair.c | 108 ++++++++++++++++++++++++++++++++--------------= ----- kernel/sched/pelt.h | 7 +++ kernel/sched/sched.h | 1=20 3 files changed, 76 insertions(+), 40 deletions(-) --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5541,6 +5541,44 @@ static inline bool skip_blocked_update(s return true; } =20 +static inline void __update_cfs_rq_h_load(struct cfs_rq *cfs_rq, + struct sched_entity *se, + struct cfs_rq *p_cfs_rq) +{ + unsigned long load =3D cfs_rq->avg.load_avg; + + if (cfs_rq !=3D &cfs_rq->rq->cfs) { + if (!se) + se =3D &container_of(cfs_rq, struct cfs_tg_state, cfs_rq)->se; + if (!p_cfs_rq) + p_cfs_rq =3D cfs_rq_of(se); + + load =3D p_cfs_rq->h_load; + load =3D div64_ul(load * se->avg.load_avg, + p_cfs_rq->avg.load_avg + 1); + } + + WRITE_ONCE(cfs_rq->h_load, load); +} + +static inline bool update_cfs_rq_h_load(struct cfs_rq *cfs_rq, + struct sched_entity *se, + struct cfs_rq *p_cfs_rq) +{ + /* + * Mask out the segment bits, if the remaining bits match, then there + * hasn't been a decay since the last time. + */ + if ((cfs_rq->last_h_load_update & ~PELT_SEGMENT_MASK) =3D=3D + (cfs_rq->avg.last_update_time & ~PELT_SEGMENT_MASK)) + return false; + + __update_cfs_rq_h_load(cfs_rq, se, p_cfs_rq); + + cfs_rq->last_h_load_update =3D cfs_rq->avg.last_update_time; + return true; +} + #else /* !CONFIG_FAIR_GROUP_SCHED: */ =20 static inline void update_tg_load_avg(struct cfs_rq *cfs_rq) {} @@ -5554,6 +5592,10 @@ static inline int propagate_entity_load_ =20 static inline void add_tg_cfs_propagate(struct cfs_rq *cfs_rq, long runnab= le_sum) {} =20 +static inline bool update_cfs_rq_h_load(struct cfs_rq *cfs_rq, + struct sched_entity *se, + struct cfs_rq *p_cfs_rq) { return false; } + #endif /* !CONFIG_FAIR_GROUP_SCHED */ =20 #ifdef CONFIG_NO_HZ_COMMON @@ -7966,6 +8008,9 @@ static unsigned long enqueue_hierarchy(s flags =3D ENQUEUE_WAKEUP; } =20 + for_each_sched_entity_bl(se, cfs_rq) + update_cfs_rq_h_load(group_cfs_rq(se), se, cfs_rq); + return weight; } =20 @@ -8107,6 +8152,9 @@ static void dequeue_hierarchy(struct tas flags |=3D DEQUEUE_SLEEP; flags &=3D ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL); } + + for_each_sched_entity_bl(se, cfs_rq) + update_cfs_rq_h_load(group_cfs_rq(se), se, cfs_rq); } =20 /* @@ -11288,51 +11336,23 @@ static bool __update_blocked_fair(struct *done =3D false; } =20 - return decayed; -} - -/* - * Compute the hierarchical load factor for cfs_rq and all its ascendants. - * This needs to be done in a top-down fashion because the load of a child - * group is a fraction of its parents load. - */ -static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq) -{ - struct sched_entity *se =3D cfs_rq_se(cfs_rq); - unsigned long now =3D jiffies; - unsigned long load; - - if (cfs_rq->last_h_load_update =3D=3D now) - return; - - WRITE_ONCE(cfs_rq->h_load_next, NULL); - for_each_sched_entity(se, cfs_rq) { - WRITE_ONCE(cfs_rq->h_load_next, se); - if (cfs_rq->last_h_load_update =3D=3D now) - break; - } - - if (!se) { - cfs_rq->h_load =3D cfs_rq_load_avg(cfs_rq); - cfs_rq->last_h_load_update =3D now; - } + /* + * The above (forward) leaf_cfs_rq_list traversal will have done + * update_cfs_rq_load_avg() in a bottom-up fashion. Now iterate the + * list backwards, such that we're ensured to have visited every + * parent of the current group to update h_load in a top-down fashion. + */ + list_for_each_entry_reverse(cfs_rq, &rq->leaf_cfs_rq_list, leaf_cfs_rq_li= st) + update_cfs_rq_h_load(cfs_rq, NULL, NULL); =20 - while ((se =3D READ_ONCE(cfs_rq->h_load_next)) !=3D NULL) { - load =3D cfs_rq->h_load; - load =3D div64_ul(load * se->avg.load_avg, - cfs_rq_load_avg(cfs_rq) + 1); - cfs_rq =3D group_cfs_rq(se); - cfs_rq->h_load =3D load; - cfs_rq->last_h_load_update =3D now; - } + return decayed; } =20 static unsigned long task_h_load(struct task_struct *p) { struct cfs_rq *cfs_rq =3D task_cfs_rq(p); =20 - update_cfs_rq_h_load(cfs_rq); - return div64_ul(p->se.avg.load_avg * cfs_rq->h_load, + return div64_ul(p->se.avg.load_avg * READ_ONCE(cfs_rq->h_load), cfs_rq_load_avg(cfs_rq) + 1); } #else /* !CONFIG_FAIR_GROUP_SCHED: */ @@ -14949,7 +14969,7 @@ static inline void task_tick_core(struct /* * se_fi_update - Update the cfs_rq->zero_vruntime_fi in a CFS hierarchy i= f needed. */ -static void se_fi_update(const struct sched_entity *se, unsigned int fi_se= q, +static void se_fi_update(struct sched_entity *se, unsigned int fi_seq, bool forceidle) { struct cfs_rq *cfs_rq; @@ -15039,6 +15059,11 @@ static void task_tick_fair(struct rq *rq =20 se =3D &curr->se; reweight_eevdf(cfs_rq, se, weight, se->on_rq); + + if (!hrtick) { + for_each_sched_entity_bl(se, cfs_rq) + update_cfs_rq_h_load(group_cfs_rq(se), se, cfs_rq); + } } =20 if (hrtick) @@ -15230,6 +15255,9 @@ static void set_next_task_fair(struct rq weight =3D __calc_prop_weight(cfs_rq, se, weight); } =20 + for_each_sched_entity_bl(se, cfs_rq) + update_cfs_rq_h_load(group_cfs_rq(se), se, cfs_rq); + if (throttled) task_throttle_setup_work(p); =20 @@ -15428,6 +15456,8 @@ static int __sched_group_set_shares(stru update_load_avg(cfs_rq, se, UPDATE_TG); update_cfs_group(se); } + for_each_sched_entity_bl(se, cfs_rq) + update_cfs_rq_h_load(group_cfs_rq(se), se, cfs_rq); rq_unlock_irqrestore(rq, &rf); } =20 --- a/kernel/sched/pelt.h +++ b/kernel/sched/pelt.h @@ -5,6 +5,13 @@ =20 #include "sched-pelt.h" =20 +/* + * Pelt uses apprixmate 'us' as ns/1024; and then uses time segments of 10= 24 + * 'us'. As a result each segment is in fact '1<<20' ns. + */ +#define PELT_SEGMENT_NS (1<<20) +#define PELT_SEGMENT_MASK (PELT_SEGMENT_NS-1) + int __update_load_avg_blocked_se(u64 now, struct sched_entity *se); int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_enti= ty *se); int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq); --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -736,7 +736,6 @@ struct cfs_rq { */ unsigned long h_load; u64 last_h_load_update; - struct sched_entity *h_load_next; =20 struct rq *rq; /* CPU runqueue to which this cfs_rq is attached */