From nobody Fri Sep 25 13:25:06 2026 Received: from gentwo.org (gentwo.org [62.72.0.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7E4F4BEE5B for ; Fri, 11 Sep 2026 22:31:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=62.72.0.81 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789165894; cv=none; b=EKk0Se7wNkSy2ys5LcuDOxIwzWdKMQ2t1G0Y+Wunl0FeRLVweYuQGBj+Dm7Kxq1vDfppzy7LGgvu3lwjE7fdiWTxSwbO/NHmWNCR9cnsi/lo7KVf++alecZ40J9J0KYqn7rtvqp/GSGWecadDcVLTc3YUNQQwRB8v7MRodV/AQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789165894; c=relaxed/simple; bh=b5SMf4/pt5Y4MtXnNZ1zNe1Wj7E5xDWMi8TsoorU6rg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=KXWk429hGoFjzQ8YbQOnNtmXur7GEAiNYblaQMdFYJ5BARKAHAhcAxMDej+xhfDXBniGNUdjTM8cNMg3rGeDkdwAGYJcOmwMUmKW7YQZ+Vouspe/Ov0uf6S+GRK0ZfIGFJ/3OtNGXm7qWNdKP7DQBtlsKOohhlaQmk8XhPmaE/c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org; spf=pass smtp.mailfrom=gentwo.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b=TaJymjzK; arc=none smtp.client-ip=62.72.0.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gentwo.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b="TaJymjzK" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gentwo.org; s=default; t=1789164504; bh=b5SMf4/pt5Y4MtXnNZ1zNe1Wj7E5xDWMi8TsoorU6rg=; h=From:Date:Subject:To:Cc:From; b=TaJymjzKIE9yh8oLVLZUPoo7SyumHjd1VkBRUkpnRwq63/BjGD2csygL8RWzrkBK0 T0/tbRkZIuORQI+Natm6EFLDRFExkgDgHGo53YSJoEfL95DgBqO6iSjWPOI9QRd2Bp KagdRyB2LrX6InDwlenT0U541vbOb6RLqQhVY58o= Received: from sut02sys-r112.scc-lab.amperecomputing.com (localhost [127.0.0.1]) by gentwo.org (Postfix) with ESMTPS id B7C1C40110; Fri, 11 Sep 2026 15:08:23 -0700 (PDT) From: "Shubhang Kaushik (Ampere)" Date: Fri, 11 Sep 2026 15:08:17 -0700 Subject: [PATCH v2] sched: Restart hrtick after same-task repicks Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-sched-fair-hrtick-restart-v2-1-0d34db26ecd1@gentwo.org> X-B4-Tracking: v=1; b=H4sIANB7pGoC/42NQQ6CMBBFr0Jm7RhaiKAr7mFYFDqlExNqpg1qS O9u5QQu30v++ztEEqYIt2oHoY0jh7WAPlUwe7MuhGwLg671pe5Vg3H2ZNEZFvSSeH6gUExGEpr pahvbduS6Hsr+KeT4fbTvY2HPMQX5HFeb+tl/qptCha1uaqtI9W6ahoXW9ArnIAuMOecvEo2s2 MMAAAA= X-Change-ID: 20260813-sched-fair-hrtick-restart-ab9d3d47ef78 To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Shubhang Kaushik , Christoph Lameter , Zhan Xusheng , Tejun Heo , David Vernet , Andrea Righi , Changwoo Min Cc: linux-kernel@vger.kernel.org, sched-ext@lists.linux.dev, "Shubhang Kaushik (Ampere)" X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1789164503; l=10808; i=sh@gentwo.org; s=20251010; h=from:subject:message-id; bh=b5SMf4/pt5Y4MtXnNZ1zNe1Wj7E5xDWMi8TsoorU6rg=; b=2QS2NlIbN9Qm4r686ePN3BAfYYjz07PQSzHkjC/Djm1tXFyA6dw9j+1cmkzmk9mg9UjCeIyff q8xb2a0WVYZDoCRVYxWPZF9/GVnOzFc54QSCFwOHz8iRfEMdu3lGGvD X-Developer-Key: i=sh@gentwo.org; a=ed25519; pk=jc8YIRvxPSyJaBRe5y+a4N0RXKBUEcAh8+OFhlROXPY= Fair and DL hrticks are one-shot timers. After an hrtick expires, the next preemption point must be explicitly armed for the task that remains selected. For fair, the missed path is: hrtick() -> rq->donor->sched_class->task_tick(rq, rq->donor, 1) -> task_tick_fair(..., queued=3D1) -> entity_tick() -> resched_curr() -> schedule() -> pick_task_fair() selects the current task again -> put_prev_set_next_task(rq, prev, next) -> next =3D=3D prev -> return put_prev_set_next_task() returns without calling next->sched_class->set_next_task(). Consequently, set_next_task_fair() is not called and does not invoke hrtick_start_fair() to arm the next fair hrtick. The same `next =3D=3D prev` return skips set_next_task_dl() and start_hrtick_dl() when a DL hrtick causes a reschedule that repicks the current DL task. Replace the `bool first` argument of set_next_task() with an enum. SNT_NORMAL and SNT_PICK preserve the existing call sites, while SNT_REPICK identifies the next =3D=3D prev case. On `SNT_REPICK`, fair and = DL skip their normal task-transition work because the task is already current, but restart their respective hrticks. Calling the class-specific rearm from the repick path also preserves the existing hrtick scheduling rules. The call occurs during schedule(), with hrtick programming deferred by rq->hrtick_sched; hrtick_start() records the new delay and hrtick_schedule_exit() rearms the hrtimer after scheduling completes. This replaces the fair specific rq state and runnable count checks with a common scheduler core solution. Signed-off-by: Shubhang Kaushik (Ampere) --- Testing: - Built and booted baseline 08df884136f1 and v2 with CONFIG_HIGH_RES_TIMERS=3Dy and CONFIG_SCHED_HRTICK=3Dy. - Enabled HRTICK and DELAY_DEQUEUE, set base_slice_ns to 3000000, and pinned two CPU-bound nice-0 fair-class tasks plus a periodic sleeper = to CPU 0. - Collected five 10-second `perf sched` traces per kernel. Across the CPU-bound fair-class tasks: baseline: maximum runtime 5.227ms; 6 samples above 4ms. v2: maximum runtime 3.386ms; no samples above 4ms. --- Changes in v2: - Replace the fair-specific hrtick rearm state with SNT_REPICK. - Restart hrticks for both fair and SCHED_DEADLINE same-task repicks. - Remove the delayed-dequeue runnable-count condition. Link to v1: https://lore.kernel.org/r/20260813-sched-fair-hrtick-restart-v1= -1-4230d1e18fbb@gentwo.org --- kernel/sched/core.c | 2 +- kernel/sched/deadline.c | 8 ++++++-- kernel/sched/ext/ext.c | 5 ++++- kernel/sched/fair.c | 15 ++++++++++----- kernel/sched/idle.c | 5 ++++- kernel/sched/rt.c | 7 +++++-- kernel/sched/sched.h | 16 ++++++++++++---- kernel/sched/stop_task.c | 5 ++++- 8 files changed, 46 insertions(+), 17 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index b998ef6b87af4fa60422fe86b3702c43103ea9aa..90607d2f3a93a59c2227ce9775c= e3976054a0b96 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7221,7 +7221,7 @@ static void __sched notrace __schedule(int sched_mode) * on_cpu. */ donor->sched_class->put_prev_task(rq, donor, donor); - donor->sched_class->set_next_task(rq, donor, true); + donor->sched_class->set_next_task(rq, donor, SNT_PICK); } } else { rq_set_donor(rq, next); diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 0663c00c41c04213fda2ebaf16a0bdf34e0849a8..6c17563f42a1d22bae5fc9f85b7= 5497b6b4faccb 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2773,11 +2773,14 @@ static void start_hrtick_dl(struct rq *rq, struct s= ched_dl_entity *dl_se) * DL keeps current in tree, because ->deadline is not typically changed w= hile * a task is runnable. */ -static void set_next_task_dl(struct rq *rq, struct task_struct *p, bool fi= rst) +static void set_next_task_dl(struct rq *rq, struct task_struct *p, enum sn= t_e type) { struct sched_dl_entity *dl_se =3D &p->dl; struct dl_rq *dl_rq =3D &rq->dl; =20 + if (type =3D=3D SNT_REPICK) + goto repick; + p->se.exec_start =3D rq_clock_task(rq); if (on_dl_rq(&p->dl)) update_stats_wait_end_dl(dl_rq, dl_se); @@ -2788,7 +2791,7 @@ static void set_next_task_dl(struct rq *rq, struct ta= sk_struct *p, bool first) WARN_ON_ONCE(dl_rq->curr); dl_rq->curr =3D dl_se; =20 - if (!first) + if (type !=3D SNT_PICK) return; =20 if (rq->donor->sched_class !=3D &dl_sched_class) @@ -2796,6 +2799,7 @@ static void set_next_task_dl(struct rq *rq, struct ta= sk_struct *p, bool first) =20 deadline_queue_push_tasks(rq); =20 +repick: if (hrtick_enabled_dl(rq)) start_hrtick_dl(rq, &p->dl); } diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c index 51de1d8b72a1ede6d16c07b58defb1284285b8a6..31a300f2d3b202d588ca666d6b8= f6d4705ee6c79 100644 --- a/kernel/sched/ext/ext.c +++ b/kernel/sched/ext/ext.c @@ -3003,10 +3003,13 @@ static enum scx_dsp_verdict dispatch_one(struct rq = *rq, struct task_struct *prev return verdict; } =20 -static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool f= irst) +static void set_next_task_scx(struct rq *rq, struct task_struct *p, enum s= nt_e type) { struct scx_sched *sch =3D scx_task_sched(p); =20 + if (type =3D=3D SNT_REPICK) + return; + if (p->scx.flags & SCX_TASK_QUEUED) { /* * Core-sched might decide to execute @p before it is diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ade1eceb39b806e60e44efe54e33a7097d01d1df..4a83ad97544675d52ca6b296a5e= 57356da537293 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -15245,14 +15245,18 @@ static void switched_to_fair(struct rq *rq, struc= t task_struct *p) } } =20 -static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool = first) +static void set_next_task_fair(struct rq *rq, struct task_struct *p, enum = snt_e type) { struct sched_entity *se =3D &p->se; - bool throttled =3D false; struct cfs_rq *cfs_rq =3D &rq->cfs; unsigned long weight =3D NICE_0_LOAD; + bool first =3D type =3D=3D SNT_PICK; + bool throttled =3D false; bool on_rq =3D se->on_rq; =20 + if (type =3D=3D SNT_REPICK) + goto repick; + clear_buddies(cfs_rq, se); =20 if (on_rq) @@ -15296,11 +15300,12 @@ static void set_next_task_fair(struct rq *rq, str= uct task_struct *p, bool first) =20 WARN_ON_ONCE(se->sched_delayed); =20 - if (hrtick_enabled_fair(rq)) - hrtick_start_fair(rq, p); - update_misfit_status(p, rq); sched_fair_update_stop_tick(rq, p); + +repick: + if (hrtick_enabled_fair(rq)) + hrtick_start_fair(rq, p); } =20 void init_cfs_rq(struct cfs_rq *cfs_rq) diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index eb73b65ce6c4714db1e07066c86a8cf39adbc328..76f3c84ca6846afc30609fdc512= 553c6797e9557 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -487,8 +487,11 @@ static void put_prev_task_idle(struct rq *rq, struct t= ask_struct *prev, struct t update_rq_avg_idle(rq); } =20 -static void set_next_task_idle(struct rq *rq, struct task_struct *next, bo= ol first) +static void set_next_task_idle(struct rq *rq, struct task_struct *next, en= um snt_e type) { + if (type =3D=3D SNT_REPICK) + return; + update_idle_core(rq); scx_update_idle(rq, true, true); schedstat_inc(rq->sched_goidle); diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 85303add726df4c2da846c427aab2d627fca9589..1535046a23ffa0705d90dce301f= a4c5b015703a1 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1654,11 +1654,14 @@ static void wakeup_preempt_rt(struct rq *rq, struct= task_struct *p, int flags) check_preempt_equal_prio(rq, p); } =20 -static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, = bool first) +static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, = enum snt_e type) { struct sched_rt_entity *rt_se =3D &p->rt; struct rt_rq *rt_rq =3D &rq->rt; =20 + if (type =3D=3D SNT_REPICK) + return; + p->se.exec_start =3D rq_clock_task(rq); if (on_rt_rq(&p->rt)) update_stats_wait_end_rt(rt_rq, rt_se); @@ -1666,7 +1669,7 @@ static inline void set_next_task_rt(struct rq *rq, st= ruct task_struct *p, bool f /* The running task is never eligible for pushing */ dequeue_pushable_task(rq, p); =20 - if (!first) + if (type !=3D SNT_PICK) return; =20 /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index e656c7059bf864d1ed91d4ec3d4624850aded7e0..defcf7fd023fa5bb3c34c3c29f9= 9afe3c42a87de 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2621,6 +2621,12 @@ struct affinity_context { =20 extern s64 update_curr_common(struct rq *rq); =20 +enum snt_e { + SNT_NORMAL, + SNT_PICK, + SNT_REPICK, +}; + struct sched_class { =20 #ifdef CONFIG_UCLAMP_TASK @@ -2678,7 +2684,7 @@ struct sched_class { * __schedule: rq->lock */ void (*put_prev_task)(struct rq *rq, struct task_struct *p, struct task_s= truct *next); - void (*set_next_task)(struct rq *rq, struct task_struct *p, bool first); + void (*set_next_task)(struct rq *rq, struct task_struct *p, enum snt_e ty= pe); =20 /* * select_task_rq: p->pi_lock @@ -2781,7 +2787,7 @@ static inline void put_prev_task(struct rq *rq, struc= t task_struct *prev) =20 static inline void set_next_task(struct rq *rq, struct task_struct *next) { - next->sched_class->set_next_task(rq, next, false); + next->sched_class->set_next_task(rq, next, SNT_NORMAL); } =20 static inline void @@ -2802,11 +2808,13 @@ static inline void put_prev_set_next_task(struct rq= *rq, =20 __put_prev_set_next_dl_server(rq, prev, next); =20 - if (next =3D=3D prev) + if (next =3D=3D prev) { + next->sched_class->set_next_task(rq, next, SNT_REPICK); return; + } =20 prev->sched_class->put_prev_task(rq, prev, next); - next->sched_class->set_next_task(rq, next, true); + next->sched_class->set_next_task(rq, next, SNT_PICK); } =20 /* diff --git a/kernel/sched/stop_task.c b/kernel/sched/stop_task.c index c909ca0d8c87c1055ad5ffe58ae9f044194ee358..1e0109ec36b3f9b84f6f7290e58= 4e38257db701c 100644 --- a/kernel/sched/stop_task.c +++ b/kernel/sched/stop_task.c @@ -27,8 +27,11 @@ wakeup_preempt_stop(struct rq *rq, struct task_struct *p= , int flags) /* we're never preempted */ } =20 -static void set_next_task_stop(struct rq *rq, struct task_struct *stop, bo= ol first) +static void set_next_task_stop(struct rq *rq, struct task_struct *stop, en= um snt_e type) { + if (type =3D=3D SNT_REPICK) + return; + stop->se.exec_start =3D rq_clock_task(rq); } =20 --- base-commit: 08df884136f1c1197bab2a27814404fd329d9aac change-id: 20260813-sched-fair-hrtick-restart-ab9d3d47ef78 Best regards, --=20 Shubhang Kaushik (Ampere)