From nobody Tue Sep 29 01:20:27 2026 Received: from gentwo.org (gentwo.org [62.72.0.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 144F1381B0C for ; Thu, 13 Aug 2026 21:24:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=62.72.0.81 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786656242; cv=none; b=bE+PXMn98/QP+lBG4M4pMrE7dcizYki4QiXEVBHOotGNRSy2xWjzk8MJnijCFMkwvsSTJ3m3rZgjtjumatCdUDI8tMTSz9ax+cA6/HdMT8V38gAHeQVtKTfqTfQG6WisYljsotlcOFNO3CLIr1x9n9MuQuKw8t0RRDutTBBDGLE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786656242; c=relaxed/simple; bh=3r9nDXH6GG3l6s6q5IGix5ZJRAyvh3QihR3R1yUuqqk=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=Zq82gP+zpqLk0l6fpsp2GMkiC15TOk10h6gbGs+nzqeUTb04+jC4bH8gCGMd6ubEDSfnxWYVn3/EefBtlTH8nONnTaYyH1yLfvPLmyh1KEDjo5hAUt4988MU2ugHhYtsFJlSXAFyEj75R859USnyGqlRhDMykfeCEkWFN/9e0pc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org; spf=pass smtp.mailfrom=gentwo.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b=UG92HR+W; arc=none smtp.client-ip=62.72.0.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gentwo.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b="UG92HR+W" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gentwo.org; s=default; t=1786656234; bh=3r9nDXH6GG3l6s6q5IGix5ZJRAyvh3QihR3R1yUuqqk=; h=From:Date:Subject:To:Cc:From; b=UG92HR+WssfslbbxOTsk3SSzShU+9oX8kFLKrsQMdFEriZAdtfXhJLvmYgudVq38P BP0ygzeaq9AWiAmpNfJI1hTHij5ZznO9UJjpB+JJopNUtQik+pHRsjCZ1aGkqGY2uH uV8BQ+XzaEicf2VUSv7k0g4DZSinuN83Sb1yP3Qw= Received: from sut02sys-r112.scc-lab.amperecomputing.com (localhost [127.0.0.1]) by gentwo.org (Postfix) with ESMTPS id B3432401C8; Thu, 13 Aug 2026 14:23:53 -0700 (PDT) From: "Shubhang Kaushik (Ampere)" Date: Thu, 13 Aug 2026 14:23:48 -0700 Subject: [PATCH] sched/fair: Restart hrtick after same-task repicks Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260813-sched-fair-hrtick-restart-v1-1-4230d1e18fbb@gentwo.org> X-B4-Tracking: v=1; b=H4sIAOM1fmoC/x3MSQqAMAxA0atI1gaccLqKuKhtaoOgkhQRxLtbX L7F/w8oCZPCmD0gdLHysSeUeQY2mH0lZJcMVVG1RV/WqDaQQ29YMEhku6GQRiMRzTK42jUd+a6 H1J9Cnu//Pc3v+wEKjSpvawAAAA== X-Change-ID: 20260813-sched-fair-hrtick-restart-ab9d3d47ef78 To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Shubhang Kaushik , Christoph Lameter Cc: linux-kernel@vger.kernel.org, "Shubhang Kaushik (Ampere)" X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1786656233; l=5449; i=sh@gentwo.org; s=20251010; h=from:subject:message-id; bh=3r9nDXH6GG3l6s6q5IGix5ZJRAyvh3QihR3R1yUuqqk=; b=RfeoSMhVdaDhpXf7qYFjTMSEvk18ONNvkuCVpVKMMuk0XNIZbpuFnnBUJBT5tWtieOn/+U2L2 pw2D5cqE9srC246Ie8Gbc+nRSNltDMhQ/pyI/LcfQWhFGflyjW8Numm X-Developer-Key: i=sh@gentwo.org; a=ed25519; pk=jc8YIRvxPSyJaBRe5y+a4N0RXKBUEcAh8+OFhlROXPY= Fair hrtick is implemented with a one-shot timer, so each precise preemption point has to be programmed explicitly. The usual fair path does this from set_next_task_fair(), which calls hrtick_start_fair(). The missed path is: hrtick -> task_tick_fair(..., queued=3D1) -> entity_tick() -> resched_curr() -> schedule() -> pick_task_fair() picks current again -> put_prev_set_next_task() -> next =3D=3D prev -> return Since set_next_task_fair() is skipped, hrtick_start_fair() is not called and no new fair hrtick is started. Record when a queued fair hrtick may need a restart, and consume that state only from the same-task fast path. Limit this to cases where more than one fair entity is runnable and all queued fair entities are runnable, avoiding extra hrticks for delayed-dequeue and pipe-like cases where queued entities are not all competing for CPU time. Signed-off-by: Shubhang Kaushik (Ampere) --- On v7.2-rc7 mainline (3aa1dcaa4f6f), with HRTICK enabled, base_slice_ns=3D3000000, and two CPU-bound tasks pinned to one CPU, the nice-0 task's runtime intervals above 8ms dropped from 228 in a 10s perf sched capture to 34-38 across repeated runs with this change. A similar missed hrtick start was previously reported for the older pick_next_task_fair() flow: Message-ID: <20241111074841.8802-1-shijie@os.amperecomputing.com> --- kernel/sched/core.c | 2 ++ kernel/sched/fair.c | 29 ++++++++++++++++++++++++++++- kernel/sched/sched.h | 21 ++++++++++++++++++++- 3 files changed, 50 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 96226707c2f6135341aa779b8262f113e103d8ad..5ec8c3f752fa48149469907edb5= 95ede0769e94a 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1013,12 +1013,14 @@ static inline void hrtick_schedule_exit(struct rq *= rq) __hrtimer_rearm_deferred(); =20 rq->hrtick_sched =3D HRTICK_SCHED_NONE; + rq->hrtick_rearm_fair =3D false; } =20 static void hrtick_rq_init(struct rq *rq) { INIT_CSD(&rq->hrtick_csd, __hrtick_start, rq); rq->hrtick_sched =3D HRTICK_SCHED_NONE; + rq->hrtick_rearm_fair =3D false; hrtimer_setup(&rq->hrtick_timer, hrtick, CLOCK_MONOTONIC, HRTIMER_MODE_REL_HARD | HRTIMER_MODE_LAZY_REARM); } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d78467ec6ee1343050fcc2794dafb38ade3599e5..2d90a9a84175833bdb78f6f78d2= 3b124105fcb82 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7681,6 +7681,22 @@ static void hrtick_start_fair(struct rq *rq, struct = task_struct *p) hrtick_start(rq, (scale * delta) / 1024); } =20 +void __hrtick_rearm_fair(struct rq *rq, struct task_struct *p) +{ + rq->hrtick_rearm_fair =3D false; + + if (!hrtick_enabled_fair(rq)) + return; + + if (hrtick_active(rq)) + return; + + if (p->sched_class !=3D &fair_sched_class) + return; + + hrtick_start_fair(rq, p); +} + /* * Called on enqueue to start the hrtick when h_nr_queued becomes more tha= n 1. */ @@ -14858,8 +14874,19 @@ static void task_tick_fair(struct rq *rq, struct t= ask_struct *curr, int queued) entity_tick(cfs_rq, se, queued); } =20 - if (queued) + if (queued) { + /* + * Fair hrtick is one-shot. If this hrtick-triggered + * reschedule picks the same task again, set_next_task_fair() + * will be skipped. Mark that path for a possible restart, but + * avoid delayed-dequeue cases where queued entities are not all + * runnable. + */ + rq->hrtick_rearm_fair =3D hrtick_enabled_fair(rq) && + rq->cfs.h_nr_runnable > 1 && + rq->cfs.h_nr_runnable =3D=3D rq->cfs.h_nr_queued; return; + } =20 if (static_branch_unlikely(&sched_numa_balancing)) task_tick_numa(rq, curr); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 56acf502ba260ab18bacd7a4c2efdec612d50125..faf63eea233981fbd7e0a13b652= f0c37d292ef35 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1313,6 +1313,7 @@ struct rq { ktime_t hrtick_time; ktime_t hrtick_delay; unsigned int hrtick_sched; + bool hrtick_rearm_fair; #endif =20 #ifdef CONFIG_SCHEDSTATS @@ -2745,6 +2746,18 @@ static inline void set_next_task(struct rq *rq, stru= ct task_struct *next) next->sched_class->set_next_task(rq, next, false); } =20 +#ifdef CONFIG_SCHED_HRTICK +void __hrtick_rearm_fair(struct rq *rq, struct task_struct *p); + +static inline void hrtick_rearm_fair(struct rq *rq, struct task_struct *p) +{ + if (rq->hrtick_rearm_fair) + __hrtick_rearm_fair(rq, p); +} +#else +static inline void hrtick_rearm_fair(struct rq *rq, struct task_struct *p)= { } +#endif + static inline void __put_prev_set_next_dl_server(struct rq *rq, struct task_struct *prev, @@ -2763,8 +2776,14 @@ static inline void put_prev_set_next_task(struct rq = *rq, =20 __put_prev_set_next_dl_server(rq, prev, next); =20 - if (next =3D=3D prev) + if (next =3D=3D prev) { + /* + * Same-task repicks skip class callbacks. Restart fair hrtick + * if the queued tick path marked it as needed. + */ + hrtick_rearm_fair(rq, next); return; + } =20 prev->sched_class->put_prev_task(rq, prev, next); next->sched_class->set_next_task(rq, next, true); --- base-commit: 3aa1dcaa4f6f5ae08936491e08bd456f331f2d40 change-id: 20260813-sched-fair-hrtick-restart-ab9d3d47ef78 Best regards, --=20 Shubhang Kaushik (Ampere)