From nobody Fri Sep 25 01:25:57 2026 Received: from gentwo.org (gentwo.org [62.72.0.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BCF6372662 for ; Thu, 17 Sep 2026 23:16:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=62.72.0.81 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789687022; cv=none; b=sohoMcIU4/+lm907WIiFczaSbULBsYD59bkBS+DZEC1Umprt91ay1+76ZLKiOLEMZWSYkpv/F0oX7M5puo3KYoBKjHG/z4e9WcbPvlpdvvPOBqu4+nvPb4s+gSjnWVKDWT04NejynKoKzKEenusP5UI6wKl1v+K6QdEy8mWqmSA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789687022; c=relaxed/simple; bh=ghz8/Rw9QNYJf1qQYDhCazT1pB3J1010HxmP/p/ZygU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=ULAaINEVJi9tZl0LbpOpaDSJVSEJ9WRJzkQkCpm1PLXg2wNoj0DCWC91Dgpp33iXeJstXncRyJN6L4jMMcm1qYHdJTK20rFbE4jEq8cNwhldeHLRx//XCmSeswFr8+576vmeSA2JEjuJmPaEcpojRwIXh56x3pIqiDh+BcZNP2M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org; spf=pass smtp.mailfrom=gentwo.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b=hyNxg5e+; arc=none smtp.client-ip=62.72.0.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=gentwo.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gentwo.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=gentwo.org header.i=@gentwo.org header.b="hyNxg5e+" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gentwo.org; s=default; t=1789686427; bh=ghz8/Rw9QNYJf1qQYDhCazT1pB3J1010HxmP/p/ZygU=; h=From:Date:Subject:To:Cc:From; b=hyNxg5e+F6s+NFz4FJfXFfK2jFI4rda8Xo1v/LokUlB9d0yk1x+Nbx+oxtJ6W8z7N 4c0N1BxW6i4aOseEI76kMig3KZiA9xI6Vluy+n1lG+wVyL1HXw9LcfDRFL18CV7ZZb lIFVCqVnA67/tlcCcooPzN2jyWrWMm5D27EPX8Sk= Received: from sut02sys-r112.scc-lab.amperecomputing.com (localhost [127.0.0.1]) by gentwo.org (Postfix) with ESMTPS id 5468040836; Thu, 17 Sep 2026 16:07:07 -0700 (PDT) From: "Shubhang Kaushik (Ampere)" Date: Thu, 17 Sep 2026 16:06:59 -0700 Subject: [PATCH v4] sched: Restart fair hrtick after same-task repicks Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260917-sched-fair-hrtick-restart-v4-1-4dd1414da81a@gentwo.org> X-B4-Tracking: v=1; b=H4sIAJNyrGoC/43NTQrCMBCG4atI1kYySX9SV95DXLTJpA1CK5NQl dK7m3ajIhSX7wfzzMQCksfAjruJEY4++KFPke13zHR13yL3NjWTQhZCg+LBdGi5qz3xjqI3V04 YYk2R101llc1KdKVm6f5G6Pxjtc+X1J0PcaDn+mqEZf1HHYEDz6QSFhC0a5pTi328D4eBWrawo 3xTFcAWJRMlrMpsIws0Fn4o9UnlW5RKVJlD6ZTWThj9Rc3z/AJVJMAVWQEAAA== X-Change-ID: 20260813-sched-fair-hrtick-restart-ab9d3d47ef78 To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Shubhang Kaushik , Christoph Lameter , Zhan Xusheng , Tejun Heo , David Vernet , Andrea Righi , Changwoo Min Cc: linux-kernel@vger.kernel.org, sched-ext@lists.linux.dev, "Shubhang Kaushik (Ampere)" X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1789686427; l=11342; i=sh@gentwo.org; s=20251010; h=from:subject:message-id; bh=ghz8/Rw9QNYJf1qQYDhCazT1pB3J1010HxmP/p/ZygU=; b=JRhYZVRguBNJauAscWk2XeLIrOlB921hKlHHApT96WGyKSyPV0n6UUHs65dOPhS30Z9C21ewi Bsl6pyGGz6PBYKa8bHDz8EmY/K8YNlk6hCJKoe0NyzyvHNLZpBKo7+3 X-Developer-Key: i=sh@gentwo.org; a=ed25519; pk=jc8YIRvxPSyJaBRe5y+a4N0RXKBUEcAh8+OFhlROXPY= Fair hrticks are one-shot timers. A same-task repick after hrtick expiry skips set_next_task_fair() and leaves no hrtick armed for the next fair preemption point: hrtick() -> task_tick_fair(..., queued=3D1) -> entity_tick() -> resched_curr() -> schedule() -> pick_task_fair() selects current -> put_prev_set_next_task(rq, prev, next) -> next =3D=3D prev -> return Introduce SNT_REPICK for the next =3D=3D prev path: put_prev_set_next_task(rq, prev, next) -> next =3D=3D prev -> next->sched_class->set_next_task(rq, next, SNT_REPICK) -> set_next_task_fair() -> hrtick_start_fair() set_next_task_fair() skips task-transition work for SNT_REPICK and restarts the fair hrtick. hrtick_start() records the delay during schedule(), and hrtick_schedule_exit() rearms the timer. Do not restart the SCHED_DEADLINE hrtick for SNT_REPICK. Same-task repicks skip put_prev_task() for both fair and SCHED_DEADLINE, but pick_task_fair() refreshes the selected fair entity before selecting it again. hrtick_start_fair() reads that refreshed entity, including with group scheduling. The SCHED_DEADLINE picker has no corresponding current-runtime update. Restarting its hrtick on SNT_REPICK could therefore use stale dl_se->runtime and delay the next runtime enforcement point. Signed-off-by: Shubhang Kaushik (Ampere) Reviewed-by: Zhan Xusheng --- Testing: - HRTICK and DELAY_DEQUEUE enabled, base_slice_ns=3D3000000: p90: 3.998ms baseline -> 3.053ms v3 p99: 5.144ms baseline -> 4.533ms v3 >4ms: 7.89% baseline -> 2.26% v3 >6ms: 0.309% baseline -> 0.013% v3 - HRTICK_DL stress-ng smoke test: baseline and v3 completed 2691 bogo ops with no new dmesg warnings. - Also built with CONFIG_HIGH_RES_TIMERS=3Dn, which disables CONFIG_SCHED_HRTICK and CONFIG_HRTIMER_REARM_DEFERRED. The kernel booted successfully; a 30-second stress-ng --cpu 2 run completed without new dmesg warnings. --- Changes in v4: - Clarify why fair can rearm an hrtick on SNT_REPICK while DL cannot. - Document the SNT_NORMAL, SNT_PICK, and SNT_REPICK call site meanings. - Build tested CONFIG_SCHED_CLASS_EXT=3Dy. Link to v3: https://lore.kernel.org/r/20260915-sched-fair-hrtick-restart-v3= -1-7517f388f0c8@gentwo.org Changes in v3: - Drop the SCHED_DEADLINE SNT_REPICK hrtick rearm from v2. Link to v2: https://lore.kernel.org/r/20260911-sched-fair-hrtick-restart-v2= -1-0d34db26ecd1@gentwo.org Changes in v2: - Replace the fair-specific hrtick rearm state with SNT_REPICK. - Restart hrticks for both fair and SCHED_DEADLINE same-task repicks. - Remove the delayed-dequeue runnable-count condition. Link to v1: https://lore.kernel.org/r/20260813-sched-fair-hrtick-restart-v1= -1-4230d1e18fbb@gentwo.org --- kernel/sched/core.c | 2 +- kernel/sched/deadline.c | 7 +++++-- kernel/sched/ext/ext.c | 5 ++++- kernel/sched/fair.c | 21 ++++++++++++++++----- kernel/sched/idle.c | 5 ++++- kernel/sched/rt.c | 7 +++++-- kernel/sched/sched.h | 16 ++++++++++++---- kernel/sched/stop_task.c | 5 ++++- 8 files changed, 51 insertions(+), 17 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7885ff76e69f28e1c81d72d87e5ceef0562ccea9..4f7d2139f29d5e6a9241e0bb873= 69bc20359955c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7222,7 +7222,7 @@ static void __sched notrace __schedule(int sched_mode) * on_cpu. */ donor->sched_class->put_prev_task(rq, donor, donor); - donor->sched_class->set_next_task(rq, donor, true); + donor->sched_class->set_next_task(rq, donor, SNT_PICK); } } else { rq_set_donor(rq, next); diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 0663c00c41c04213fda2ebaf16a0bdf34e0849a8..1502a45d76aaf44079d618d8ae1= 93c9b213fc401 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -2773,11 +2773,14 @@ static void start_hrtick_dl(struct rq *rq, struct s= ched_dl_entity *dl_se) * DL keeps current in tree, because ->deadline is not typically changed w= hile * a task is runnable. */ -static void set_next_task_dl(struct rq *rq, struct task_struct *p, bool fi= rst) +static void set_next_task_dl(struct rq *rq, struct task_struct *p, enum sn= t_e type) { struct sched_dl_entity *dl_se =3D &p->dl; struct dl_rq *dl_rq =3D &rq->dl; =20 + if (type =3D=3D SNT_REPICK) + return; + p->se.exec_start =3D rq_clock_task(rq); if (on_dl_rq(&p->dl)) update_stats_wait_end_dl(dl_rq, dl_se); @@ -2788,7 +2791,7 @@ static void set_next_task_dl(struct rq *rq, struct ta= sk_struct *p, bool first) WARN_ON_ONCE(dl_rq->curr); dl_rq->curr =3D dl_se; =20 - if (!first) + if (type !=3D SNT_PICK) return; =20 if (rq->donor->sched_class !=3D &dl_sched_class) diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c index 3219f0da0fe4e756132d9418874e6d93be501a4e..dd81c73b30ae0adb4083b14187d= bc5b6de651a0f 100644 --- a/kernel/sched/ext/ext.c +++ b/kernel/sched/ext/ext.c @@ -3008,10 +3008,13 @@ static enum scx_dsp_verdict dispatch_one(struct rq = *rq, struct task_struct *prev return verdict; } =20 -static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool f= irst) +static void set_next_task_scx(struct rq *rq, struct task_struct *p, enum s= nt_e type) { struct scx_sched *sch =3D scx_task_sched(p); =20 + if (type =3D=3D SNT_REPICK) + return; + if (p->scx.flags & SCX_TASK_QUEUED) { /* * Core-sched might decide to execute @p before it is diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 7455a83a6a9907b4fd18c3a99df707438818709b..9f2542d8fc6ea5ada813480c9e2= ca329262f0ba3 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -15253,14 +15253,18 @@ static void switched_to_fair(struct rq *rq, struc= t task_struct *p) } } =20 -static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool = first) +static void set_next_task_fair(struct rq *rq, struct task_struct *p, enum = snt_e type) { struct sched_entity *se =3D &p->se; - bool throttled =3D false; struct cfs_rq *cfs_rq =3D &rq->cfs; unsigned long weight =3D NICE_0_LOAD; + bool first =3D type =3D=3D SNT_PICK; + bool throttled =3D false; bool on_rq =3D se->on_rq; =20 + if (type =3D=3D SNT_REPICK) + goto repick; + clear_buddies(cfs_rq, se); =20 if (on_rq) @@ -15304,11 +15308,18 @@ static void set_next_task_fair(struct rq *rq, str= uct task_struct *p, bool first) =20 WARN_ON_ONCE(se->sched_delayed); =20 - if (hrtick_enabled_fair(rq)) - hrtick_start_fair(rq, p); - update_misfit_status(p, rq); sched_fair_update_stop_tick(rq, p); + +repick: + /* + * A same-task repick skips put_prev_task_fair(), but + * pick_task_fair() refreshed the entity hrtick_start_fair() reads + * before selecting it again. rq->cfs.curr identifies that entity, + * including with group scheduling. + */ + if (hrtick_enabled_fair(rq)) + hrtick_start_fair(rq, p); } =20 void init_cfs_rq(struct cfs_rq *cfs_rq) diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index eb73b65ce6c4714db1e07066c86a8cf39adbc328..76f3c84ca6846afc30609fdc512= 553c6797e9557 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -487,8 +487,11 @@ static void put_prev_task_idle(struct rq *rq, struct t= ask_struct *prev, struct t update_rq_avg_idle(rq); } =20 -static void set_next_task_idle(struct rq *rq, struct task_struct *next, bo= ol first) +static void set_next_task_idle(struct rq *rq, struct task_struct *next, en= um snt_e type) { + if (type =3D=3D SNT_REPICK) + return; + update_idle_core(rq); scx_update_idle(rq, true, true); schedstat_inc(rq->sched_goidle); diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 85303add726df4c2da846c427aab2d627fca9589..1535046a23ffa0705d90dce301f= a4c5b015703a1 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1654,11 +1654,14 @@ static void wakeup_preempt_rt(struct rq *rq, struct= task_struct *p, int flags) check_preempt_equal_prio(rq, p); } =20 -static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, = bool first) +static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, = enum snt_e type) { struct sched_rt_entity *rt_se =3D &p->rt; struct rt_rq *rt_rq =3D &rq->rt; =20 + if (type =3D=3D SNT_REPICK) + return; + p->se.exec_start =3D rq_clock_task(rq); if (on_rt_rq(&p->rt)) update_stats_wait_end_rt(rt_rq, rt_se); @@ -1666,7 +1669,7 @@ static inline void set_next_task_rt(struct rq *rq, st= ruct task_struct *p, bool f /* The running task is never eligible for pushing */ dequeue_pushable_task(rq, p); =20 - if (!first) + if (type !=3D SNT_PICK) return; =20 /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index e656c7059bf864d1ed91d4ec3d4624850aded7e0..6c08d5345b0abbd6e407ceb720c= eeaab3ddcdcb2 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2621,6 +2621,12 @@ struct affinity_context { =20 extern s64 update_curr_common(struct rq *rq); =20 +enum snt_e { + SNT_NORMAL, /* set_next_task() */ + SNT_PICK, /* put_prev_set_next_task(): prev !=3D next */ + SNT_REPICK, /* put_prev_set_next_task(): prev =3D=3D next */ +}; + struct sched_class { =20 #ifdef CONFIG_UCLAMP_TASK @@ -2678,7 +2684,7 @@ struct sched_class { * __schedule: rq->lock */ void (*put_prev_task)(struct rq *rq, struct task_struct *p, struct task_s= truct *next); - void (*set_next_task)(struct rq *rq, struct task_struct *p, bool first); + void (*set_next_task)(struct rq *rq, struct task_struct *p, enum snt_e ty= pe); =20 /* * select_task_rq: p->pi_lock @@ -2781,7 +2787,7 @@ static inline void put_prev_task(struct rq *rq, struc= t task_struct *prev) =20 static inline void set_next_task(struct rq *rq, struct task_struct *next) { - next->sched_class->set_next_task(rq, next, false); + next->sched_class->set_next_task(rq, next, SNT_NORMAL); } =20 static inline void @@ -2802,11 +2808,13 @@ static inline void put_prev_set_next_task(struct rq= *rq, =20 __put_prev_set_next_dl_server(rq, prev, next); =20 - if (next =3D=3D prev) + if (next =3D=3D prev) { + next->sched_class->set_next_task(rq, next, SNT_REPICK); return; + } =20 prev->sched_class->put_prev_task(rq, prev, next); - next->sched_class->set_next_task(rq, next, true); + next->sched_class->set_next_task(rq, next, SNT_PICK); } =20 /* diff --git a/kernel/sched/stop_task.c b/kernel/sched/stop_task.c index c909ca0d8c87c1055ad5ffe58ae9f044194ee358..1e0109ec36b3f9b84f6f7290e58= 4e38257db701c 100644 --- a/kernel/sched/stop_task.c +++ b/kernel/sched/stop_task.c @@ -27,8 +27,11 @@ wakeup_preempt_stop(struct rq *rq, struct task_struct *p= , int flags) /* we're never preempted */ } =20 -static void set_next_task_stop(struct rq *rq, struct task_struct *stop, bo= ol first) +static void set_next_task_stop(struct rq *rq, struct task_struct *stop, en= um snt_e type) { + if (type =3D=3D SNT_REPICK) + return; + stop->se.exec_start =3D rq_clock_task(rq); } =20 --- base-commit: b5a051f6b840d48f159166ef073d3021989bfb50 change-id: 20260813-sched-fair-hrtick-restart-ab9d3d47ef78 Best regards, --=20 Shubhang Kaushik (Ampere)