From nobody Fri Dec 19 14:04:58 2025 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8652123B8 for ; Wed, 20 Dec 2023 00:19:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jstultz.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JPTksi+1" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-5c1b986082dso147474a12.0 for ; Tue, 19 Dec 2023 16:19:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1703031545; x=1703636345; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NUHnww5ktcUEm6gbdUuCZMr5JgxeUPAZP1Uwwm4gSJs=; b=JPTksi+1QkpwIp9mYQ42y/GKuFXCfknzAretgW5L6UmlJPxOJIOL0VN+vdKY4nxSKS f1GbT4A9g7hZNUnkn8IKsEWFvhw7WYQtE5SXPgbq9vFPHRoHuKpjjl1poFKG000EcSXV g1N1nBZTfDleN4xBL4n6P2WdBH+XpSg388RjD6w5TiOV2ZiuXSpXRy/x22lInm4iIVbM 1s8qAxjZANGOs3Yd/e4nNgQFu2cQD4O0NJspVoWypSmZhQBAS6Y69ohXpzWEZ2g6c9OC 3zWqA8hRuN14BcZVDc0Dhbo/nlo4L0Xhv8v5Z36CJG3fgJPKiDBg5T2ku8h0D0TcMYCs 9qow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703031545; x=1703636345; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NUHnww5ktcUEm6gbdUuCZMr5JgxeUPAZP1Uwwm4gSJs=; b=tIUSEbxI9Fl+2y+ZBnAHPGpDUfz+4GinBCQnEr9pAmT7iclZIu9qswUqjcp4HQlb/a 1ExLsrrU3g8ttap/bPmLOH4vjPZ5qlBROYuNYHZdjBWFoFZ/n82fpHqtF2rt0rJ2I1zc zLER9WoUGR+1MDOFCyM4sq9GsFBWlU6G1OcKXNMPFPCMo6z1lMpSojtTEAPOb9HXLjAj CWx85YeCfmLRxFcvOlscmd9PUOO30Xy7msXzAq84svzyx3FtHqcHzyu8SI3hNeIpZkh3 G3r+79gJ+PwcwHWe3AbxzxpUB6Mmpt0Ud2D1mQY6STbI1BTcmcFHJzDsLdNDxgVsKUjr ubLg== X-Gm-Message-State: AOJu0Yx9c2cghaR0EhuvjlsTFa2mrBmBGX9Y5B0FvJ2XQ1Q1MT1ncU8s JaX1zuEGyCKJ5PC3jUgF3ZHZwEk/TMrcBhmFG+Xp1DRB4PQWe1DHyie3UdHsm4uOdrpGGpeGFQR y7C8piCYBdVBtXKgM0TvZsqJzBle4LYTBQMx3820sIgiVDC+A0opv71Xh8kpWskuuwQzJ+Oo= X-Google-Smtp-Source: AGHT+IGo5rlZ2EWtcahUdEV5X1N/1i4O/tPYFCTuxXZMdv3cH1dn2fb15eyFgZxhI9LlgzkQX2rpajQ9NpOr X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a05:6a02:6a1:b0:5cd:9acb:a4a2 with SMTP id ca33-20020a056a0206a100b005cd9acba4a2mr42196pgb.2.1703031543770; Tue, 19 Dec 2023 16:19:03 -0800 (PST) Date: Tue, 19 Dec 2023 16:18:12 -0800 In-Reply-To: <20231220001856.3710363-1-jstultz@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20231220001856.3710363-1-jstultz@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231220001856.3710363-2-jstultz@google.com> Subject: [PATCH v7 01/23] sched: Unify runtime accounting across classes From: John Stultz To: LKML Cc: Peter Zijlstra , Joel Fernandes , Qais Yousef , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Youssef Esmat , Mel Gorman , Daniel Bristot de Oliveira , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , kernel-team@android.com, "Connor O'Brien" , John Stultz Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Peter Zijlstra All classes use sched_entity::exec_start to track runtime and have copies of the exact same code around to compute runtime. Collapse all that. Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Youssef Esmat Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E. McKenney" Cc: Metin Kaya Cc: Xuewen Yan Cc: K Prateek Nayak Cc: Thomas Gleixner Cc: kernel-team@android.com Signed-off-by: Peter Zijlstra (Intel) [fix conflicts, fold in update_current_exec_runtime] Signed-off-by: Connor O'Brien [jstultz: rebased, resolving minor conflicts] Signed-off-by: John Stultz --- v7: * Minor typo fixup suggested by Metin Kaya --- include/linux/sched.h | 2 +- kernel/sched/deadline.c | 13 +++------- kernel/sched/fair.c | 56 ++++++++++++++++++++++++++++++---------- kernel/sched/rt.c | 13 +++------- kernel/sched/sched.h | 12 ++------- kernel/sched/stop_task.c | 13 +--------- 6 files changed, 52 insertions(+), 57 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..1e80c330f755 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -523,7 +523,7 @@ struct sched_statistics { u64 block_max; s64 sum_block_runtime; =20 - u64 exec_max; + s64 exec_max; u64 slice_max; =20 u64 nr_migrations_cold; diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index b28114478b82..6140f1f51da1 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -1275,9 +1275,8 @@ static void update_curr_dl(struct rq *rq) { struct task_struct *curr =3D rq->curr; struct sched_dl_entity *dl_se =3D &curr->dl; - u64 delta_exec, scaled_delta_exec; + s64 delta_exec, scaled_delta_exec; int cpu =3D cpu_of(rq); - u64 now; =20 if (!dl_task(curr) || !on_dl_rq(dl_se)) return; @@ -1290,21 +1289,15 @@ static void update_curr_dl(struct rq *rq) * natural solution, but the full ramifications of this * approach need further study. */ - now =3D rq_clock_task(rq); - delta_exec =3D now - curr->se.exec_start; - if (unlikely((s64)delta_exec <=3D 0)) { + delta_exec =3D update_curr_common(rq); + if (unlikely(delta_exec <=3D 0)) { if (unlikely(dl_se->dl_yielded)) goto throttle; return; } =20 - schedstat_set(curr->stats.exec_max, - max(curr->stats.exec_max, delta_exec)); - trace_sched_stat_runtime(curr, delta_exec, 0); =20 - update_current_exec_runtime(curr, now, delta_exec); - if (dl_entity_is_special(dl_se)) return; =20 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d7a3c63a2171..1251fd01a555 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1129,23 +1129,17 @@ static void update_tg_load_avg(struct cfs_rq *cfs_r= q) } #endif /* CONFIG_SMP */ =20 -/* - * Update the current task's runtime statistics. - */ -static void update_curr(struct cfs_rq *cfs_rq) +static s64 update_curr_se(struct rq *rq, struct sched_entity *curr) { - struct sched_entity *curr =3D cfs_rq->curr; - u64 now =3D rq_clock_task(rq_of(cfs_rq)); - u64 delta_exec; - - if (unlikely(!curr)) - return; + u64 now =3D rq_clock_task(rq); + s64 delta_exec; =20 delta_exec =3D now - curr->exec_start; - if (unlikely((s64)delta_exec <=3D 0)) - return; + if (unlikely(delta_exec <=3D 0)) + return delta_exec; =20 curr->exec_start =3D now; + curr->sum_exec_runtime +=3D delta_exec; =20 if (schedstat_enabled()) { struct sched_statistics *stats; @@ -1155,9 +1149,43 @@ static void update_curr(struct cfs_rq *cfs_rq) max(delta_exec, stats->exec_max)); } =20 - curr->sum_exec_runtime +=3D delta_exec; - schedstat_add(cfs_rq->exec_clock, delta_exec); + return delta_exec; +} + +/* + * Used by other classes to account runtime. + */ +s64 update_curr_common(struct rq *rq) +{ + struct task_struct *curr =3D rq->curr; + s64 delta_exec; =20 + delta_exec =3D update_curr_se(rq, &curr->se); + if (unlikely(delta_exec <=3D 0)) + return delta_exec; + + account_group_exec_runtime(curr, delta_exec); + cgroup_account_cputime(curr, delta_exec); + + return delta_exec; +} + +/* + * Update the current task's runtime statistics. + */ +static void update_curr(struct cfs_rq *cfs_rq) +{ + struct sched_entity *curr =3D cfs_rq->curr; + s64 delta_exec; + + if (unlikely(!curr)) + return; + + delta_exec =3D update_curr_se(rq_of(cfs_rq), curr); + if (unlikely(delta_exec <=3D 0)) + return; + + schedstat_add(cfs_rq->exec_clock, delta_exec); curr->vruntime +=3D calc_delta_fair(delta_exec, curr); update_deadline(cfs_rq, curr); update_min_vruntime(cfs_rq); diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index 6aaf0a3d6081..9cdea3ea47da 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1002,24 +1002,17 @@ static void update_curr_rt(struct rq *rq) { struct task_struct *curr =3D rq->curr; struct sched_rt_entity *rt_se =3D &curr->rt; - u64 delta_exec; - u64 now; + s64 delta_exec; =20 if (curr->sched_class !=3D &rt_sched_class) return; =20 - now =3D rq_clock_task(rq); - delta_exec =3D now - curr->se.exec_start; - if (unlikely((s64)delta_exec <=3D 0)) + delta_exec =3D update_curr_common(rq); + if (unlikely(delta_exec < 0)) return; =20 - schedstat_set(curr->stats.exec_max, - max(curr->stats.exec_max, delta_exec)); - trace_sched_stat_runtime(curr, delta_exec, 0); =20 - update_current_exec_runtime(curr, now, delta_exec); - if (!rt_bandwidth_enabled()) return; =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 2e5a95486a42..3e0e4fc8734b 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2212,6 +2212,8 @@ struct affinity_context { unsigned int flags; }; =20 +extern s64 update_curr_common(struct rq *rq); + struct sched_class { =20 #ifdef CONFIG_UCLAMP_TASK @@ -3261,16 +3263,6 @@ extern int sched_dynamic_mode(const char *str); extern void sched_dynamic_update(int mode); #endif =20 -static inline void update_current_exec_runtime(struct task_struct *curr, - u64 now, u64 delta_exec) -{ - curr->se.sum_exec_runtime +=3D delta_exec; - account_group_exec_runtime(curr, delta_exec); - - curr->se.exec_start =3D now; - cgroup_account_cputime(curr, delta_exec); -} - #ifdef CONFIG_SCHED_MM_CID =20 #define SCHED_MM_CID_PERIOD_NS (100ULL * 1000000) /* 100ms */ diff --git a/kernel/sched/stop_task.c b/kernel/sched/stop_task.c index 6cf7304e6449..b1b8fe61c532 100644 --- a/kernel/sched/stop_task.c +++ b/kernel/sched/stop_task.c @@ -70,18 +70,7 @@ static void yield_task_stop(struct rq *rq) =20 static void put_prev_task_stop(struct rq *rq, struct task_struct *prev) { - struct task_struct *curr =3D rq->curr; - u64 now, delta_exec; - - now =3D rq_clock_task(rq); - delta_exec =3D now - curr->se.exec_start; - if (unlikely((s64)delta_exec < 0)) - delta_exec =3D 0; - - schedstat_set(curr->stats.exec_max, - max(curr->stats.exec_max, delta_exec)); - - update_current_exec_runtime(curr, now, delta_exec); + update_curr_common(rq); } =20 /* --=20 2.43.0.472.g3155946c3a-goog