From nobody Sun Apr 12 05:51:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 992EEC00144 for ; Mon, 1 Aug 2022 04:28:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234029AbiHAE2P (ORCPT ); Mon, 1 Aug 2022 00:28:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53178 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239199AbiHAE2H (ORCPT ); Mon, 1 Aug 2022 00:28:07 -0400 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33132DFA6 for ; Sun, 31 Jul 2022 21:28:06 -0700 (PDT) Received: by mail-pg1-x52c.google.com with SMTP id s206so8697010pgs.3 for ; Sun, 31 Jul 2022 21:28:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/w4d2+EaQroyv2P/11Ro6fW4T9xIlpzK4n5YR8bSveQ=; b=SjypgSlEb0OP8iYCGGYtl5NTi7gHq0BX+OwbrP1J8RTxVLJtlwsCt8HHENmT1W/7Hs 5VQtbEpd9Hlt3meP5bennKtCSfMXuZPYRpqBSmSCrbu7vQ636/HoukKS4SqT2UFUVo3g 9QA4Zw0nwRlc15LNoSBT8FHHypdEHJC5OhdNZSyuiUuo0SdlUtNMyJgltYauzKpHmMXf BhOnw5knl1gwCkyGYTHA++UVBKswNn5SghrO1gJOz9Sl7c3wamkXv4B9C0IZmiQad9tB Ys5KnIpDtbG6waWI6t6TjdQXT9JCKtoT6mSsoVNF1L7niH3j2GBH25bxscQrwouYzmcR VEZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/w4d2+EaQroyv2P/11Ro6fW4T9xIlpzK4n5YR8bSveQ=; b=BPNzEWxtTMDvZVAUUDVlSSRJB/JD36IQqIf/buEg7ha3O8bsD6qs3fVvO2kbcd+0zu lJ8ANnFwBE25NGyo1GH9xt0KZ2+hupYxwlN8vsmHEM+vFit2z2GovLKH6f+fd/AHONVB cCrQnTxk1CMNVkUZH0eKakXpAk911kiWLXyS9SJtNkIFigTS+hqMKG2xHan8A5sIMkOl jVzRa38RDm20wycDOlPOWWgc80EtwZW5DyuLAcHYofmTOR+E5Vrz4UfW1zS8TQTQwzko UIiPGxMeNQ1O9R2WxtdeD5fzHOXWKfHmr6iDoNmlVT/WM4zrwy/UiGg0aQcAZHasXpSI o3iQ== X-Gm-Message-State: ACgBeo1QK0RSqZ2ljeJEXAbdB0Mn2VFad6R2GGj6h7DQtqRTJVhMK42P yQXM3bMl+B4JMVBJHrKynhtgvg== X-Google-Smtp-Source: AA6agR495PY4jykHh20wj3uEIXUt+rv6O6xqlNdzX0mVPMpQCfaNl6aE2MCoh6YkWS/xWTHrU47WXA== X-Received: by 2002:aa7:8e91:0:b0:52d:8ebf:29a4 with SMTP id a17-20020aa78e91000000b0052d8ebf29a4mr1706695pfr.1.1659328085577; Sun, 31 Jul 2022 21:28:05 -0700 (PDT) Received: from C02CV1DAMD6P.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id ot10-20020a17090b3b4a00b001f326ead012sm7012202pjb.37.2022.07.31.21.28.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jul 2022 21:28:05 -0700 (PDT) From: Chengming Zhou To: mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Chengming Zhou Subject: [PATCH v3 01/10] sched/fair: maintain task se depth in set_task_rq() Date: Mon, 1 Aug 2022 12:27:36 +0800 Message-Id: <20220801042745.7794-2-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220801042745.7794-1-zhouchengming@bytedance.com> References: <20220801042745.7794-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Previously we only maintain task se depth in task_move_group_fair(), if a !fair task change task group, its se depth will not be updated, so commit eb7a59b2c888 ("sched/fair: Reset se-depth when task switched to F= AIR") fix the problem by updating se depth in switched_to_fair() too. Then commit daa59407b558 ("sched/fair: Unify switched_{from,to}_fair() and task_move_group_fair()") unified these two functions, moved se.depth setting to attach_task_cfs_rq(), which further into attach_entity_cfs_rq() with commit df217913e72e ("sched/fair: Factorize attach/detach entity"). This patch move task se depth maintenance from attach_entity_cfs_rq() to set_task_rq(), which will be called when CPU/cgroup change, so its depth will always be correct. This patch is preparation for the next patch. Signed-off-by: Chengming Zhou Reviewed-by: Dietmar Eggemann Reviewed-by: Vincent Guittot --- kernel/sched/fair.c | 8 -------- kernel/sched/sched.h | 1 + 2 files changed, 1 insertion(+), 8 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 2fc47257ae91..77cd2bad17a8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11566,14 +11566,6 @@ static void attach_entity_cfs_rq(struct sched_enti= ty *se) { struct cfs_rq *cfs_rq =3D cfs_rq_of(se); =20 -#ifdef CONFIG_FAIR_GROUP_SCHED - /* - * Since the real-depth could have been changed (only FAIR - * class maintain depth value), reset depth properly. - */ - se->depth =3D se->parent ? se->parent->depth + 1 : 0; -#endif - /* Synchronize entity with its cfs_rq */ update_load_avg(cfs_rq, se, sched_feat(ATTACH_AGE_LOAD) ? 0 : SKIP_AGE_LO= AD); attach_entity_load_avg(cfs_rq, se); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index aad7f5ee9666..8cc3eb7b86cd 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1940,6 +1940,7 @@ static inline void set_task_rq(struct task_struct *p,= unsigned int cpu) set_task_rq_fair(&p->se, p->se.cfs_rq, tg->cfs_rq[cpu]); p->se.cfs_rq =3D tg->cfs_rq[cpu]; p->se.parent =3D tg->se[cpu]; + p->se.depth =3D tg->se[cpu] ? tg->se[cpu]->depth + 1 : 0; #endif =20 #ifdef CONFIG_RT_GROUP_SCHED --=20 2.36.1 From nobody Sun Apr 12 05:51:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 306CCC3F6B0 for ; Mon, 1 Aug 2022 04:28:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239308AbiHAE2U (ORCPT ); Mon, 1 Aug 2022 00:28:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239227AbiHAE2L (ORCPT ); Mon, 1 Aug 2022 00:28:11 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0EB2DFBD for ; Sun, 31 Jul 2022 21:28:09 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id q22so1235214plr.9 for ; Sun, 31 Jul 2022 21:28:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uFH8NHGJSUxhHtDtnk+1UL0mZ/w91LaurRijcSElzyc=; b=Ha82v9KNGhCiFujRAf0gBjJg8QepWcRRBMiEJ0OWomqfxXW7Uu7JIaHfkhH7v1xyJb 0BimZu2020tux6gYkxv8xAzCWLA5PlFd1MSC1W7X6xkJZKaWvNwn1IPS0UOUWARrOWJp 9ApMIdatqLjpyxQtV6bpwR+umQ7hwy6iYqmCrunhgGi+tVA7lcGabFWWw+WLc8gkSBW/ kMp73uxtzB8EspQwoON/ZsSFdclmGJyctw4l/yZ1iW0aijEgEjLJXvumGVxVA+1FEw8m iN3iUTeDYCYZ4gSWOkzJ8eo0kBD+uBpYy4hzDURn1BnlriCzun8h3GTJ1nkEQz7xHtlZ PYMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uFH8NHGJSUxhHtDtnk+1UL0mZ/w91LaurRijcSElzyc=; b=PZDkxZdWYzg3mNMvz6fY2q2UOU9Ag++A9Sothi11f94WgXwxOGO5OQjizbcHa9K6U5 jbORuvQiqu6t6m0/TQnGWiUIr17xF0g7yEwXYKli3ClnEr1WD/hv+bs510Gdjjfe2kVl KLjAPQY2MC0A+C+IIAtOmW77/JfSyFHznQy0esTxoN5uf5AQWQlfqZTPDhYn2CJJtS+T u2GCJpp92LNsWDoJTOh6Agvlu6fIPP+KxTs96FZmKwWnPPTUKJQoWF5jgaK/+cdnYxMb 2hghxRIz8BHaWnmdK+dRcDiB2xX7XJtI1YIGVMJMd/IBr5l03ONYieS17ovFUptaQxZr BjnQ== X-Gm-Message-State: ACgBeo1OfRt9lS5aI9IWzeE6AvWH3yFXGavaYg2uXyNxbOcOSZmaoKrr rp5ZvQq1agPJG5XITwbb848+bw== X-Google-Smtp-Source: AA6agR4PdlbV0N3lzrlOFesTrwTTeKDgxIxFuz10fcTjT/8h7hJ099PHU6qVgKJQxMVh5hFizv/diA== X-Received: by 2002:a17:902:e750:b0:16c:761:6e96 with SMTP id p16-20020a170902e75000b0016c07616e96mr14716714plf.0.1659328089424; Sun, 31 Jul 2022 21:28:09 -0700 (PDT) Received: from C02CV1DAMD6P.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id ot10-20020a17090b3b4a00b001f326ead012sm7012202pjb.37.2022.07.31.21.28.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jul 2022 21:28:09 -0700 (PDT) From: Chengming Zhou To: mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Chengming Zhou Subject: [PATCH v3 02/10] sched/fair: remove redundant cpu_cgrp_subsys->fork() Date: Mon, 1 Aug 2022 12:27:37 +0800 Message-Id: <20220801042745.7794-3-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220801042745.7794-1-zhouchengming@bytedance.com> References: <20220801042745.7794-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We use cpu_cgrp_subsys->fork() to set task group for the new fair task in cgroup_post_fork(). Since commit b1e8206582f9 ("sched: Fix yet more sched_fork() races") has already set_task_rq() for the new fair task in sched_cgroup_fork(), so cpu_cgrp_subsys->fork() can be removed. cgroup_can_fork() --> pin parent's sched_task_group sched_cgroup_fork() __set_task_cpu() set_task_rq() cgroup_post_fork() ss->fork() :=3D cpu_cgroup_fork() sched_change_group(..., TASK_SET_GROUP) task_set_group_fair() set_task_rq() --> can be removed After this patch's change, task_change_group_fair() only need to care about task cgroup migration, make the code much simplier. Signed-off-by: Chengming Zhou Reviewed-by: Vincent Guittot Reviewed-by: Dietmar Eggemann --- kernel/sched/core.c | 27 ++++----------------------- kernel/sched/fair.c | 23 +---------------------- kernel/sched/sched.h | 5 +---- 3 files changed, 6 insertions(+), 49 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 5555e49c4e12..614d7180c99e 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -481,8 +481,7 @@ sched_core_dequeue(struct rq *rq, struct task_struct *p= , int flags) { } * p->se.load, p->rt_priority, * p->dl.dl_{runtime, deadline, period, flags, bw, density} * - sched_setnuma(): p->numa_preferred_nid - * - sched_move_task()/ - * cpu_cgroup_fork(): p->sched_task_group + * - sched_move_task(): p->sched_task_group * - uclamp_update_active() p->uclamp* * * p->state <- TASK_*: @@ -10125,7 +10124,7 @@ void sched_release_group(struct task_group *tg) spin_unlock_irqrestore(&task_group_lock, flags); } =20 -static void sched_change_group(struct task_struct *tsk, int type) +static void sched_change_group(struct task_struct *tsk) { struct task_group *tg; =20 @@ -10141,7 +10140,7 @@ static void sched_change_group(struct task_struct *= tsk, int type) =20 #ifdef CONFIG_FAIR_GROUP_SCHED if (tsk->sched_class->task_change_group) - tsk->sched_class->task_change_group(tsk, type); + tsk->sched_class->task_change_group(tsk); else #endif set_task_rq(tsk, task_cpu(tsk)); @@ -10172,7 +10171,7 @@ void sched_move_task(struct task_struct *tsk) if (running) put_prev_task(rq, tsk); =20 - sched_change_group(tsk, TASK_MOVE_GROUP); + sched_change_group(tsk); =20 if (queued) enqueue_task(rq, tsk, queue_flags); @@ -10250,23 +10249,6 @@ static void cpu_cgroup_css_free(struct cgroup_subs= ys_state *css) sched_unregister_group(tg); } =20 -/* - * This is called before wake_up_new_task(), therefore we really only - * have to set its group bits, all the other stuff does not apply. - */ -static void cpu_cgroup_fork(struct task_struct *task) -{ - struct rq_flags rf; - struct rq *rq; - - rq =3D task_rq_lock(task, &rf); - - update_rq_clock(rq); - sched_change_group(task, TASK_SET_GROUP); - - task_rq_unlock(rq, task, &rf); -} - static int cpu_cgroup_can_attach(struct cgroup_taskset *tset) { struct task_struct *task; @@ -11132,7 +11114,6 @@ struct cgroup_subsys cpu_cgrp_subsys =3D { .css_released =3D cpu_cgroup_css_released, .css_free =3D cpu_cgroup_css_free, .css_extra_stat_show =3D cpu_extra_stat_show, - .fork =3D cpu_cgroup_fork, .can_attach =3D cpu_cgroup_can_attach, .attach =3D cpu_cgroup_attach, .legacy_cftypes =3D cpu_legacy_files, diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 77cd2bad17a8..89626b115660 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11661,15 +11661,7 @@ void init_cfs_rq(struct cfs_rq *cfs_rq) } =20 #ifdef CONFIG_FAIR_GROUP_SCHED -static void task_set_group_fair(struct task_struct *p) -{ - struct sched_entity *se =3D &p->se; - - set_task_rq(p, task_cpu(p)); - se->depth =3D se->parent ? se->parent->depth + 1 : 0; -} - -static void task_move_group_fair(struct task_struct *p) +static void task_change_group_fair(struct task_struct *p) { detach_task_cfs_rq(p); set_task_rq(p, task_cpu(p)); @@ -11681,19 +11673,6 @@ static void task_move_group_fair(struct task_struc= t *p) attach_task_cfs_rq(p); } =20 -static void task_change_group_fair(struct task_struct *p, int type) -{ - switch (type) { - case TASK_SET_GROUP: - task_set_group_fair(p); - break; - - case TASK_MOVE_GROUP: - task_move_group_fair(p); - break; - } -} - void free_fair_sched_group(struct task_group *tg) { int i; diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 8cc3eb7b86cd..19e0076e4245 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2203,11 +2203,8 @@ struct sched_class { =20 void (*update_curr)(struct rq *rq); =20 -#define TASK_SET_GROUP 0 -#define TASK_MOVE_GROUP 1 - #ifdef CONFIG_FAIR_GROUP_SCHED - void (*task_change_group)(struct task_struct *p, int type); + void (*task_change_group)(struct task_struct *p); #endif }; =20 --=20 2.36.1 From nobody Sun Apr 12 05:51:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20B32C00144 for ; Mon, 1 Aug 2022 04:28:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239346AbiHAE2Z (ORCPT ); Mon, 1 Aug 2022 00:28:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53382 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239256AbiHAE2S (ORCPT ); Mon, 1 Aug 2022 00:28:18 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7FF81DFD2 for ; Sun, 31 Jul 2022 21:28:13 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id e8-20020a17090a280800b001f2fef7886eso10710523pjd.3 for ; Sun, 31 Jul 2022 21:28:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BbAH2OjARx+sGC5hf2wT2Wik++XxhkGMe8jaPc24Anw=; b=IczVJ2eKvmP1G8pKdOptWmdZQ50McO+PEJnY9lhMh9n3yBOci29Hy7BfOVJaA96C33 HIKcza5nQHSjytEj3bEsizWtqvT0dDNntorVhqwoPgzcm1ViswS0EoJ9KDOe19hR0aKC F9MTAobqpUKmNLaxX448BoW9xynm3SDQ+NwFI2Z3sCwNgJSov2CgF6Xid1OU+QMPLDcg yhzFbqgbf26Dv6fYpfnPqi2wFMd7mX1YT/u94gJEpjMaCpAwrJzKtr+nMOk9/xGBGSsx czxVrP9M0yVeJe/bWpT0px2MKBRptjYW0MFTAMxNb8mHgYzA4HzfbstW2W3MsMUWm7+g BLpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BbAH2OjARx+sGC5hf2wT2Wik++XxhkGMe8jaPc24Anw=; b=StQZZofpnWvxJpykH2XQsAD5W6w8qiNc98296GCj3SLsJBe9Gak5z+WTyrhRqZqoKD fUPdY+G8U8WC6crcO08+0V1nyOW+eo4V8K0nICJAMq+6FDi47S4B8h9N1fbNGc1J5XzK 3VWlqecs0dlzQAcul1LDFON5UouUZe1j1BP0dECeAizRtj/JlYjA+z/u5gydAaA2DXv0 D0WupDpr2fvDnRNryLiQPFQVW00bgc3JNweLFZkK/AzgCQKYJ8bsWSXhR9aUjQLv7Oel AJz/ynV/r773mu3mBL3bte8hDYibsZwvuyNohn6fD534Scdwu3w45auSkeH5JsYEcpGZ mrgA== X-Gm-Message-State: ACgBeo1MBU8QUjXxOEYkuTL7fhV1pA82BWX/Gb9gkvVVIWHXgXSSLWfS liaQaP9NlebVMLpEpgJkpovYvw== X-Google-Smtp-Source: AA6agR42VQvDZQLNDhWgGmDrWl43U9P1MPEvqcTr204dMPM1bxfhkCHoUMDjEe1mJQD53KUFgT8eZA== X-Received: by 2002:a17:903:2589:b0:16d:c26c:d641 with SMTP id jb9-20020a170903258900b0016dc26cd641mr14829718plb.8.1659328093055; Sun, 31 Jul 2022 21:28:13 -0700 (PDT) Received: from C02CV1DAMD6P.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id ot10-20020a17090b3b4a00b001f326ead012sm7012202pjb.37.2022.07.31.21.28.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jul 2022 21:28:12 -0700 (PDT) From: Chengming Zhou To: mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Chengming Zhou Subject: [PATCH v3 03/10] sched/fair: reset sched_avg last_update_time before set_task_rq() Date: Mon, 1 Aug 2022 12:27:38 +0800 Message-Id: <20220801042745.7794-4-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220801042745.7794-1-zhouchengming@bytedance.com> References: <20220801042745.7794-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" set_task_rq() -> set_task_rq_fair() will try to synchronize the blocked task's sched_avg when migrate, which is not needed for already detached task. task_change_group_fair() will detached the task sched_avg from prev cfs_rq first, so reset sched_avg last_update_time before set_task_rq() to avoid th= at. Signed-off-by: Chengming Zhou Reviewed-by: Dietmar Eggemann Reviewed-by: Vincent Guittot --- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 89626b115660..948b4cd2a024 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11664,12 +11664,12 @@ void init_cfs_rq(struct cfs_rq *cfs_rq) static void task_change_group_fair(struct task_struct *p) { detach_task_cfs_rq(p); - set_task_rq(p, task_cpu(p)); =20 #ifdef CONFIG_SMP /* Tell se's cfs_rq has been changed -- migrated */ p->se.avg.last_update_time =3D 0; #endif + set_task_rq(p, task_cpu(p)); attach_task_cfs_rq(p); } =20 --=20 2.36.1 From nobody Sun Apr 12 05:51:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEA62C00144 for ; Mon, 1 Aug 2022 04:28:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239394AbiHAE2f (ORCPT ); Mon, 1 Aug 2022 00:28:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238580AbiHAE23 (ORCPT ); Mon, 1 Aug 2022 00:28:29 -0400 Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C09013E0E for ; Sun, 31 Jul 2022 21:28:17 -0700 (PDT) Received: by mail-pg1-x532.google.com with SMTP id f11so8680607pgj.7 for ; Sun, 31 Jul 2022 21:28:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=udDFRX/Zxv8XpR6w6oj8eI1Kuby9UyxSYlMsPVhSx90=; b=4CvWm6wUWPIhkVslOOClBwYVsWG4ehAm7PCpN1HNR0wHswd9vKabaskfLF5F2S9ki3 Q3nt+vF/SP6MhjH7JDIVewtkJfuO1EpPEyFbRdK2gbw8ZhviNJf1T9F39DYfwDT+WFX8 qkLuHCLEM5d5Q1OuZCg/ZZoaF0VB7YOCNb5SUdZe0HNv8Oj8xaYvAJkZpSmX5ZLfwzJL 4oykFqPDCVP2lMteOlo/ynczqhKkGUs5NMIm+tT8ZDd6I3HcZ47ZqUrhH3XRpVPIBLZs ixaIMJK84QM6UAFLNdBF77r+rimM9cQ9f+guqUpFuOwB+B56PWltea0HxvRXbi9TzEaT kK0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=udDFRX/Zxv8XpR6w6oj8eI1Kuby9UyxSYlMsPVhSx90=; b=R2kHtT5fWLekHDF7nPPTWqz3DXecuWA1/2fFh65f7VRRxP5b9DM1cVN+SKVTcAVoD6 q48c+z0lRzfPYn+CNcu1/3F3rHvnQ/pJZjjExx7mD3Ge988/OGWKQATzB5MSOFYnzESP 0EvuI+2k6gRoUwQvTvsq5zmv007WpAeWLOehBlNyKuZlZqgbNcPLM4vvI+Ukz56eqRX6 szjopqUcHs6/fY9iui9ajckdr3doXfLFvDqh8UUmgXFvG/hTnx5WGyTrD0I2f1SWh3Nb 4D22wMi9pfFUgkwhk/mOYT262pVW3HaZ119WoEHTHeC3Y7NKhrZxgEgOMh6ZtmmGE7Xy z+Eg== X-Gm-Message-State: AJIora/R/4jsEy+AmKFms4jwE3cJdo6hdKuoiiyG0NLfp0eAZsF+u9cK nGLRMU4KYblkNKjszTl8HWDJTw== X-Google-Smtp-Source: AGRyM1sWUk4GUtJoZg6ZjANvTTM0kTJ1AA8tbdM1jC4zNYd81f0VkmUfoqud/e1od45z3YsE8tQP+w== X-Received: by 2002:a63:d244:0:b0:41a:6769:104a with SMTP id t4-20020a63d244000000b0041a6769104amr11583612pgi.525.1659328096766; Sun, 31 Jul 2022 21:28:16 -0700 (PDT) Received: from C02CV1DAMD6P.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id ot10-20020a17090b3b4a00b001f326ead012sm7012202pjb.37.2022.07.31.21.28.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jul 2022 21:28:16 -0700 (PDT) From: Chengming Zhou To: mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Chengming Zhou Subject: [PATCH v3 04/10] sched/fair: update comments in enqueue/dequeue_entity() Date: Mon, 1 Aug 2022 12:27:39 +0800 Message-Id: <20220801042745.7794-5-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220801042745.7794-1-zhouchengming@bytedance.com> References: <20220801042745.7794-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When reading the sched_avg related code, I found the comments in enqueue/dequeue_entity() are not updated with the current code. We don't add/subtract entity's runnable_avg from cfs_rq->runnable_avg during enqueue/dequeue_entity(), those are done only for attach/detach. This patch updates the comments to reflect the current code working. Signed-off-by: Chengming Zhou Acked-by: Vincent Guittot --- kernel/sched/fair.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 948b4cd2a024..956aed56ac1e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4434,7 +4434,8 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_en= tity *se, int flags) /* * When enqueuing a sched_entity, we must: * - Update loads to have both entity and cfs_rq synced with now. - * - Add its load to cfs_rq->runnable_avg + * - For group_entity, update its runnable_weight to reflect the new + * h_nr_running of its group cfs_rq. * - For group_entity, update its weight to reflect the new share of * its group cfs_rq * - Add its new weight to cfs_rq->load.weight @@ -4519,7 +4520,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_en= tity *se, int flags) /* * When dequeuing a sched_entity, we must: * - Update loads to have both entity and cfs_rq synced with now. - * - Subtract its load from the cfs_rq->runnable_avg. + * - For group_entity, update its runnable_weight to reflect the new + * h_nr_running of its group cfs_rq. * - Subtract its previous weight from cfs_rq->load.weight. * - For group entity, update its weight to reflect the new share * of its group cfs_rq. --=20 2.36.1 From nobody Sun Apr 12 05:51:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 264A2C00144 for ; Mon, 1 Aug 2022 04:28:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239060AbiHAE2k (ORCPT ); Mon, 1 Aug 2022 00:28:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239287AbiHAE2b (ORCPT ); Mon, 1 Aug 2022 00:28:31 -0400 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5943DFD1 for ; Sun, 31 Jul 2022 21:28:20 -0700 (PDT) Received: by mail-pg1-x52c.google.com with SMTP id s206so8697358pgs.3 for ; Sun, 31 Jul 2022 21:28:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wEr/YAo8Ety4XQg4L8nCVzG/mDAKtkMhP0Ioite8Q7A=; b=XeSrugbfI6aj60VUwq5iG9EJ82tjnNEdsgnUqDJs7DYGkzhraMvOD/HQwnD8utZyyZ UONKm86ENSbC8Y70Y6vIhBuWpYTNlQUHup/hhxc5NPl5wnyyLmHYrcykEtFsrg9C4A5O T00t4t6wlH13+z9q7+56lF8D+jaJ0k7d9Ez3EUtr5LblKKaa2ZKTwI7tKzP7950fvZpb CpFPD2vJeFEjnpfmqbqJ+wQaY2bJ3B7lzW1dgNJ87hq/EajLRCmJ9ssUvIUU4swGQQw+ t3corhQfkEBxS/RhKW+eJmJRsqq5QZYL7ihm97RYrbDUx3l+hCx67mudAe8AOl5lTOqe T/IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wEr/YAo8Ety4XQg4L8nCVzG/mDAKtkMhP0Ioite8Q7A=; b=Z5d8HfBcnmldac7g474n0FWJ6t9bLy+uOaE9iGrUtQKtneaUqta6b4Y5XVLvQGTc9M 2IHHC9uf99SfU/ptuDfsxANaW/cm9HxQaMI29H4rgWPdYIqsirsrDKEwYB95WeTm2APK gF8U16gzt6s6mm6IfeO90P3DzDanKuAiFuIY2S80p8aOI0OGRRSaGN8al41SCQcoXMUG xtJi+nrgA9wawIVg0DNw6OizJ03N+za3sRXElFGqJVid2H6OWHmAfiQZ9yphQgoVSL3A kxl5pNpao4JZkdMfW5zpLSf09Q6jnteKC8+EpHI9nr+JW0xhAasnjAM+os3+lxDNoaAP thUA== X-Gm-Message-State: AJIora/2TBoxOuWvlqSxtop0czWqv7e2eraUQ3M43U5zGPjVRN18mr/3 Qdd/jby/HqMFIoao1kjjNKHTeA== X-Google-Smtp-Source: AGRyM1sTksTYMLANJ9c9Sx+2lC3dOIYFdAde2PUFP39zo101buENMk9SQnevPmmhdmsgrw2jsKqSEQ== X-Received: by 2002:a63:4503:0:b0:419:7b8c:cb11 with SMTP id s3-20020a634503000000b004197b8ccb11mr11771437pga.446.1659328100430; Sun, 31 Jul 2022 21:28:20 -0700 (PDT) Received: from C02CV1DAMD6P.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id ot10-20020a17090b3b4a00b001f326ead012sm7012202pjb.37.2022.07.31.21.28.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jul 2022 21:28:20 -0700 (PDT) From: Chengming Zhou To: mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Chengming Zhou Subject: [PATCH v3 05/10] sched/fair: combine detach into dequeue when migrating task Date: Mon, 1 Aug 2022 12:27:40 +0800 Message-Id: <20220801042745.7794-6-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220801042745.7794-1-zhouchengming@bytedance.com> References: <20220801042745.7794-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When we are migrating task out of the CPU, we can combine detach and propagation into dequeue_entity() to save the detach_entity_cfs_rq() in migrate_task_rq_fair(). This optimization is like combining DO_ATTACH in the enqueue_entity() when migrating task to the CPU. So we don't have to traverse the CFS tree extra time to do the detach_entity_cfs_rq() -> propagate_entity_cfs_rq(), which wouldn't be called anymore with this patch's change. detach_task() deactivate_task() dequeue_task_fair() for_each_sched_entity(se) dequeue_entity() update_load_avg() /* (1) */ detach_entity_load_avg() set_task_cpu() migrate_task_rq_fair() detach_entity_cfs_rq() /* (2) */ update_load_avg(); detach_entity_load_avg(); propagate_entity_cfs_rq(); for_each_sched_entity() update_load_avg() This patch save the detach_entity_cfs_rq() called in (2) by doing the detach_entity_load_avg() for a CPU migrating task inside (1) (the task being the first se in the loop) Signed-off-by: Chengming Zhou Reviewed-by: Vincent Guittot --- kernel/sched/fair.c | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 956aed56ac1e..ba8b937854b4 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4003,6 +4003,7 @@ static void detach_entity_load_avg(struct cfs_rq *cfs= _rq, struct sched_entity *s #define UPDATE_TG 0x1 #define SKIP_AGE_LOAD 0x2 #define DO_ATTACH 0x4 +#define DO_DETACH 0x8 =20 /* Update task and its cfs_rq load average */ static inline void update_load_avg(struct cfs_rq *cfs_rq, struct sched_ent= ity *se, int flags) @@ -4031,7 +4032,13 @@ static inline void update_load_avg(struct cfs_rq *cf= s_rq, struct sched_entity *s */ attach_entity_load_avg(cfs_rq, se); update_tg_load_avg(cfs_rq); - + } else if (flags & DO_DETACH) { + /* + * DO_DETACH means we're here from dequeue_entity() + * and we are migrating task out of the CPU. + */ + detach_entity_load_avg(cfs_rq, se); + update_tg_load_avg(cfs_rq); } else if (decayed) { cfs_rq_util_change(cfs_rq, 0); =20 @@ -4292,6 +4299,7 @@ static inline bool cfs_rq_is_decayed(struct cfs_rq *c= fs_rq) #define UPDATE_TG 0x0 #define SKIP_AGE_LOAD 0x0 #define DO_ATTACH 0x0 +#define DO_DETACH 0x0 =20 static inline void update_load_avg(struct cfs_rq *cfs_rq, struct sched_ent= ity *se, int not_used1) { @@ -4512,6 +4520,11 @@ static __always_inline void return_cfs_rq_runtime(st= ruct cfs_rq *cfs_rq); static void dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) { + int action =3D UPDATE_TG; + + if (entity_is_task(se) && task_on_rq_migrating(task_of(se))) + action |=3D DO_DETACH; + /* * Update run-time statistics of the 'current'. */ @@ -4526,7 +4539,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_en= tity *se, int flags) * - For group entity, update its weight to reflect the new share * of its group cfs_rq. */ - update_load_avg(cfs_rq, se, UPDATE_TG); + update_load_avg(cfs_rq, se, action); se_update_runnable(se); =20 update_stats_dequeue_fair(cfs_rq, se, flags); @@ -7081,8 +7094,6 @@ select_task_rq_fair(struct task_struct *p, int prev_c= pu, int wake_flags) return new_cpu; } =20 -static void detach_entity_cfs_rq(struct sched_entity *se); - /* * Called immediately before a task is migrated to a new CPU; task_cpu(p) = and * cfs_rq_of(p) references at time of call are still valid and identify the @@ -7104,15 +7115,7 @@ static void migrate_task_rq_fair(struct task_struct = *p, int new_cpu) se->vruntime -=3D u64_u32_load(cfs_rq->min_vruntime); } =20 - if (p->on_rq =3D=3D TASK_ON_RQ_MIGRATING) { - /* - * In case of TASK_ON_RQ_MIGRATING we in fact hold the 'old' - * rq->lock and can modify state directly. - */ - lockdep_assert_rq_held(task_rq(p)); - detach_entity_cfs_rq(se); - - } else { + if (!task_on_rq_migrating(p)) { remove_entity_load_avg(se); =20 /* --=20 2.36.1 From nobody Sun Apr 12 05:51:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 698D1C00144 for ; Mon, 1 Aug 2022 04:28:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239395AbiHAE2t (ORCPT ); Mon, 1 Aug 2022 00:28:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239442AbiHAE2c (ORCPT ); Mon, 1 Aug 2022 00:28:32 -0400 Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9F4213E98 for ; Sun, 31 Jul 2022 21:28:24 -0700 (PDT) Received: by mail-pl1-x62d.google.com with SMTP id y15so9378159plp.10 for ; Sun, 31 Jul 2022 21:28:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VD5PsrnG4Eo2WXx0muZETZVU+kr0DPpaUqCGnet6TY0=; b=wg/w1zmVzPXaoyqzfbUFv01AVeSQuUx8CVIjAL6m28X3Udr+v4ZkGmsmO32oYYX60d vh4rpephJtA3bHshRJF9m1uW3FvUACwcJabMVkML3DgIFjcHg63KoStgSjGrgTcKUUXB u4wMFQeb7W8sD+ZeQj44QHlLsw8an7EGlm9YFj2IbLxy7++Ds/g9G4zMOTNCcgxax7eu /a2e5wKtTtEJfmI/9pkBMtR7kw6zaOEHLYNXfMFA0mgJuzwxwlHa2Y52Zsl0XZCZOuja 526qaxH0fVFmVFs9M8UlovuOJXL8nAoaBNjn3uQ7NtEQzWWLkKYi6IC/QaxcarAej7xF tyVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VD5PsrnG4Eo2WXx0muZETZVU+kr0DPpaUqCGnet6TY0=; b=IAn+Xt1UauDdSrXaMEPykm6Jr42U/69LS8iiZrYEGiC05IzzmsYf5bfX/cC6YIxJoI jIzn5sfRPI7JEPScMnrsr+b5w3FiN4vgM6BgbE43dP/ODaOGDazMP82k4/ABEZ/FBc7F ePfnsF89ZEAvcDqinPTdp/li24b4Uk4upKFEHtemrqt/ExdAH7rMsKqL0qR6dqSIZga3 QIN4+o4U9Vo04oXuu2ViZxyU9wUX9RtOWpUTIjsbnFu5nT5ZewYyBGyPNOOXEf3CVS4M +5Gj8b35Z3IxoZiQa6HIDeRgT1fJhJONO8Apz3ZbbS7q7hI6qZtTqgEglihL96zJEvpO +qGg== X-Gm-Message-State: ACgBeo2+Hl92TtuVBj6PC2atNjQL2/J22kJuDzv/GfieOGe7LvlyQCwg 1z7ZVvL4xmghHgxqg0/AOCK+SkwZG16HjA== X-Google-Smtp-Source: AA6agR6P07LcCCzOCMJXZBS47PS1QPJFgLj2NzynLVaUyXNbNegJoEt57FQ2iktN11CeHV2nQ4LQ4A== X-Received: by 2002:a17:902:6943:b0:16c:ba81:1842 with SMTP id k3-20020a170902694300b0016cba811842mr14937571plt.24.1659328104223; Sun, 31 Jul 2022 21:28:24 -0700 (PDT) Received: from C02CV1DAMD6P.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id ot10-20020a17090b3b4a00b001f326ead012sm7012202pjb.37.2022.07.31.21.28.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jul 2022 21:28:24 -0700 (PDT) From: Chengming Zhou To: mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Chengming Zhou Subject: [PATCH v3 06/10] sched/fair: fix another detach on unattached task corner case Date: Mon, 1 Aug 2022 12:27:41 +0800 Message-Id: <20220801042745.7794-7-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220801042745.7794-1-zhouchengming@bytedance.com> References: <20220801042745.7794-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" commit 7dc603c9028e ("sched/fair: Fix PELT integrity for new tasks") fixed two load tracking problems for new task, including detach on unattached new task problem. There still left another detach on unattached task problem for the task which has been woken up by try_to_wake_up() and waiting for actually being woken up by sched_ttwu_pending(). try_to_wake_up(p) cpu =3D select_task_rq(p) if (task_cpu(p) !=3D cpu) set_task_cpu(p, cpu) migrate_task_rq_fair() remove_entity_load_avg() --> unattached se->avg.last_update_time =3D 0; __set_task_cpu() ttwu_queue(p, cpu) ttwu_queue_wakelist() __ttwu_queue_wakelist() task_change_group_fair() detach_task_cfs_rq() detach_entity_cfs_rq() detach_entity_load_avg() --> detach on unattached task set_task_rq() attach_task_cfs_rq() attach_entity_cfs_rq() attach_entity_load_avg() The reason of this problem is similar, we should check in detach_entity_cfs= _rq() that se->avg.last_update_time !=3D 0, before do detach_entity_load_avg(). This patch move detach/attach_entity_cfs_rq() functions upper to be together with other load tracking functions to avoid to use another CONFIG_SMP, which also simplify the code. Signed-off-by: Chengming Zhou --- kernel/sched/fair.c | 133 ++++++++++++++++++++++---------------------- 1 file changed, 68 insertions(+), 65 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ba8b937854b4..a32da4e71ddf 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -874,9 +874,6 @@ void init_entity_runnable_average(struct sched_entity *= se) void post_init_entity_util_avg(struct task_struct *p) { } -static void update_tg_load_avg(struct cfs_rq *cfs_rq) -{ -} #endif /* CONFIG_SMP */ =20 /* @@ -3176,6 +3173,7 @@ void reweight_task(struct task_struct *p, int prio) load->inv_weight =3D sched_prio_to_wmult[prio]; } =20 +static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq); static inline int throttled_hierarchy(struct cfs_rq *cfs_rq); =20 #ifdef CONFIG_FAIR_GROUP_SCHED @@ -4022,7 +4020,6 @@ static inline void update_load_avg(struct cfs_rq *cfs= _rq, struct sched_entity *s decayed |=3D propagate_entity_load_avg(se); =20 if (!se->avg.last_update_time && (flags & DO_ATTACH)) { - /* * DO_ATTACH means we're here from enqueue_entity(). * !last_update_time means we've passed through @@ -4085,6 +4082,71 @@ static void remove_entity_load_avg(struct sched_enti= ty *se) raw_spin_unlock_irqrestore(&cfs_rq->removed.lock, flags); } =20 +#ifdef CONFIG_FAIR_GROUP_SCHED +/* + * Propagate the changes of the sched_entity across the tg tree to make it + * visible to the root + */ +static void propagate_entity_cfs_rq(struct sched_entity *se) +{ + struct cfs_rq *cfs_rq =3D cfs_rq_of(se); + + if (cfs_rq_throttled(cfs_rq)) + return; + + if (!throttled_hierarchy(cfs_rq)) + list_add_leaf_cfs_rq(cfs_rq); + + /* Start to propagate at parent */ + se =3D se->parent; + + for_each_sched_entity(se) { + cfs_rq =3D cfs_rq_of(se); + + update_load_avg(cfs_rq, se, UPDATE_TG); + + if (cfs_rq_throttled(cfs_rq)) + break; + + if (!throttled_hierarchy(cfs_rq)) + list_add_leaf_cfs_rq(cfs_rq); + } +} +#else +static void propagate_entity_cfs_rq(struct sched_entity *se) { } +#endif + +static void detach_entity_cfs_rq(struct sched_entity *se) +{ + struct cfs_rq *cfs_rq =3D cfs_rq_of(se); + + /* + * In case the task sched_avg hasn't been attached: + * - A forked task which hasn't been woken up by wake_up_new_task(). + * - A task which has been woken up by try_to_wake_up() but is + * waiting for actually being woken up by sched_ttwu_pending(). + */ + if (!se->avg.last_update_time) + return; + + /* Catch up with the cfs_rq and remove our load when we leave */ + update_load_avg(cfs_rq, se, 0); + detach_entity_load_avg(cfs_rq, se); + update_tg_load_avg(cfs_rq); + propagate_entity_cfs_rq(se); +} + +static void attach_entity_cfs_rq(struct sched_entity *se) +{ + struct cfs_rq *cfs_rq =3D cfs_rq_of(se); + + /* Synchronize entity with its cfs_rq */ + update_load_avg(cfs_rq, se, sched_feat(ATTACH_AGE_LOAD) ? 0 : SKIP_AGE_LO= AD); + attach_entity_load_avg(cfs_rq, se); + update_tg_load_avg(cfs_rq); + propagate_entity_cfs_rq(se); +} + static inline unsigned long cfs_rq_runnable_avg(struct cfs_rq *cfs_rq) { return cfs_rq->avg.runnable_avg; @@ -4307,11 +4369,8 @@ static inline void update_load_avg(struct cfs_rq *cf= s_rq, struct sched_entity *s } =20 static inline void remove_entity_load_avg(struct sched_entity *se) {} - -static inline void -attach_entity_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) {} -static inline void -detach_entity_load_avg(struct cfs_rq *cfs_rq, struct sched_entity *se) {} +static inline void detach_entity_cfs_rq(struct sched_entity *se) {} +static inline void attach_entity_cfs_rq(struct sched_entity *se) {} =20 static inline int newidle_balance(struct rq *rq, struct rq_flags *rf) { @@ -11522,62 +11581,6 @@ static inline bool vruntime_normalized(struct task= _struct *p) return false; } =20 -#ifdef CONFIG_FAIR_GROUP_SCHED -/* - * Propagate the changes of the sched_entity across the tg tree to make it - * visible to the root - */ -static void propagate_entity_cfs_rq(struct sched_entity *se) -{ - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - - if (cfs_rq_throttled(cfs_rq)) - return; - - if (!throttled_hierarchy(cfs_rq)) - list_add_leaf_cfs_rq(cfs_rq); - - /* Start to propagate at parent */ - se =3D se->parent; - - for_each_sched_entity(se) { - cfs_rq =3D cfs_rq_of(se); - - update_load_avg(cfs_rq, se, UPDATE_TG); - - if (cfs_rq_throttled(cfs_rq)) - break; - - if (!throttled_hierarchy(cfs_rq)) - list_add_leaf_cfs_rq(cfs_rq); - } -} -#else -static void propagate_entity_cfs_rq(struct sched_entity *se) { } -#endif - -static void detach_entity_cfs_rq(struct sched_entity *se) -{ - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - - /* Catch up with the cfs_rq and remove our load when we leave */ - update_load_avg(cfs_rq, se, 0); - detach_entity_load_avg(cfs_rq, se); - update_tg_load_avg(cfs_rq); - propagate_entity_cfs_rq(se); -} - -static void attach_entity_cfs_rq(struct sched_entity *se) -{ - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - - /* Synchronize entity with its cfs_rq */ - update_load_avg(cfs_rq, se, sched_feat(ATTACH_AGE_LOAD) ? 0 : SKIP_AGE_LO= AD); - attach_entity_load_avg(cfs_rq, se); - update_tg_load_avg(cfs_rq); - propagate_entity_cfs_rq(se); -} - static void detach_task_cfs_rq(struct task_struct *p) { struct sched_entity *se =3D &p->se; --=20 2.36.1 From nobody Sun Apr 12 05:51:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8B9CC00144 for ; Mon, 1 Aug 2022 04:28:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239151AbiHAE25 (ORCPT ); Mon, 1 Aug 2022 00:28:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239502AbiHAE2f (ORCPT ); Mon, 1 Aug 2022 00:28:35 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 011F013F1B for ; Sun, 31 Jul 2022 21:28:29 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id x10so8697431plb.3 for ; Sun, 31 Jul 2022 21:28:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qM5t1EVQbnssjElUyFkZNByPxLswIQhwSrHRz48dXWo=; b=xi9UjcUW4iBi+IuxTN6mmZ+i3gBKqsgjnolzaRD3dRQzotiZXRfETFn9aT3Hu4npwy cSI1WasmgaKz0rgy2K0jcHtdHfrFigSYylueA9L6dBqTQEwCv2MRTXqSpZUJGR7ETbOS okqJtj0iYbIv240aKWtH+maYhgiWtcpojYn+1iZjCHMbEUPlwhE9gZT/qeLVccHcf0xw PCuSdigQeaH/l3WTMB3E5IHXOJfBxkwE0bVt1zjuA/9Yb8ALqmq8rkwMJy5ItgXSzAe7 lf5S5q7q6ZS6Di3Aby5HLsKaVxMjmnmjFhSsN9PwQP8tgneqN30yjmaIGCsles6qT4jW lzrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qM5t1EVQbnssjElUyFkZNByPxLswIQhwSrHRz48dXWo=; b=HR6YXjqE/vLCL75GPdvwuw2ZNRqmUnVLL4onX9eKhH6i8zThUFM4FINJnCXDL6QO4Q 76S/Hdr4nSe0AW21BvfHnoOELnLq1H35miHmrBUiZ0Zlp1JYWmL/XQFbtYDgBmd94D16 gW9R7We9RKwfc8n1ys348B/RMrEZQbegxKoTSXPCr/rFpNGBWeAwR65xSaju0UiyPvnJ eARg7GFZoFT5qOg75A0E7f4Q+bzIznD/jZqHZJh1J7CgdSPZRWwD9QECfrC+wAlFFC1P TNFIHK2Q0l3QPfGuZFDHtGnJu4Zde6Ty/AQ6fkWj/aIjw+DIcviGWqNhQP2silSuFojE pVrQ== X-Gm-Message-State: ACgBeo37RVqDwaRCAJuV8bM4QedpnIujW4K253QLJ5FT4GmwSji7RI0+ JiNw2SemJOBUol6eX0i0cZcCyQ== X-Google-Smtp-Source: AA6agR4KB/qhLFEe6ItaP7mbifr1Td54rTx+xkyl6aZ+rbJmPb/C9iyZlCt0Uxq+kHSuOh6IFs9oHg== X-Received: by 2002:a17:902:a502:b0:16b:fbd9:7fc5 with SMTP id s2-20020a170902a50200b0016bfbd97fc5mr15350849plq.112.1659328108379; Sun, 31 Jul 2022 21:28:28 -0700 (PDT) Received: from C02CV1DAMD6P.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id ot10-20020a17090b3b4a00b001f326ead012sm7012202pjb.37.2022.07.31.21.28.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jul 2022 21:28:28 -0700 (PDT) From: Chengming Zhou To: mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Chengming Zhou Subject: [PATCH v3 07/10] sched/fair: allow changing cgroup of new forked task Date: Mon, 1 Aug 2022 12:27:42 +0800 Message-Id: <20220801042745.7794-8-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220801042745.7794-1-zhouchengming@bytedance.com> References: <20220801042745.7794-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" commit 7dc603c9028e ("sched/fair: Fix PELT integrity for new tasks") introduce a TASK_NEW state and an unnessary limitation that would fail when changing cgroup of new forked task. Because at that time, we can't handle task_change_group_fair() for new forked fair task which hasn't been woken up by wake_up_new_task(), which will cause detach on an unattached task sched_avg problem. This patch delete this unnessary limitation by adding check before do attach_entity_cfs_rq(). Signed-off-by: Chengming Zhou --- include/linux/sched.h | 5 ++--- kernel/sched/core.c | 30 +++++++----------------------- kernel/sched/fair.c | 7 ++++++- 3 files changed, 15 insertions(+), 27 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 88b8817b827d..b504e55bbf7a 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -95,10 +95,9 @@ struct task_group; #define TASK_WAKEKILL 0x0100 #define TASK_WAKING 0x0200 #define TASK_NOLOAD 0x0400 -#define TASK_NEW 0x0800 /* RT specific auxilliary flag to mark RT lock waiters */ -#define TASK_RTLOCK_WAIT 0x1000 -#define TASK_STATE_MAX 0x2000 +#define TASK_RTLOCK_WAIT 0x0800 +#define TASK_STATE_MAX 0x1000 =20 /* Convenience macros for the sake of set_current_state: */ #define TASK_KILLABLE (TASK_WAKEKILL | TASK_UNINTERRUPTIBLE) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 614d7180c99e..220bce5e73e0 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4500,11 +4500,11 @@ int sched_fork(unsigned long clone_flags, struct ta= sk_struct *p) { __sched_fork(clone_flags, p); /* - * We mark the process as NEW here. This guarantees that + * We mark the process as running here. This guarantees that * nobody will actually run it, and a signal or other external * event cannot wake it up and insert it on the runqueue either. */ - p->__state =3D TASK_NEW; + p->__state =3D TASK_RUNNING; =20 /* * Make sure we do not leak PI boosting priority to the child. @@ -4622,7 +4622,6 @@ void wake_up_new_task(struct task_struct *p) struct rq *rq; =20 raw_spin_lock_irqsave(&p->pi_lock, rf.flags); - WRITE_ONCE(p->__state, TASK_RUNNING); #ifdef CONFIG_SMP /* * Fork balancing, do it here and not earlier because: @@ -10249,36 +10248,19 @@ static void cpu_cgroup_css_free(struct cgroup_sub= sys_state *css) sched_unregister_group(tg); } =20 +#ifdef CONFIG_RT_GROUP_SCHED static int cpu_cgroup_can_attach(struct cgroup_taskset *tset) { struct task_struct *task; struct cgroup_subsys_state *css; - int ret =3D 0; =20 cgroup_taskset_for_each(task, css, tset) { -#ifdef CONFIG_RT_GROUP_SCHED if (!sched_rt_can_attach(css_tg(css), task)) return -EINVAL; -#endif - /* - * Serialize against wake_up_new_task() such that if it's - * running, we're sure to observe its full state. - */ - raw_spin_lock_irq(&task->pi_lock); - /* - * Avoid calling sched_move_task() before wake_up_new_task() - * has happened. This would lead to problems with PELT, due to - * move wanting to detach+attach while we're not attached yet. - */ - if (READ_ONCE(task->__state) =3D=3D TASK_NEW) - ret =3D -EINVAL; - raw_spin_unlock_irq(&task->pi_lock); - - if (ret) - break; } - return ret; + return 0; } +#endif =20 static void cpu_cgroup_attach(struct cgroup_taskset *tset) { @@ -11114,7 +11096,9 @@ struct cgroup_subsys cpu_cgrp_subsys =3D { .css_released =3D cpu_cgroup_css_released, .css_free =3D cpu_cgroup_css_free, .css_extra_stat_show =3D cpu_extra_stat_show, +#ifdef CONFIG_RT_GROUP_SCHED .can_attach =3D cpu_cgroup_can_attach, +#endif .attach =3D cpu_cgroup_attach, .legacy_cftypes =3D cpu_legacy_files, .dfl_cftypes =3D cpu_files, diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a32da4e71ddf..ad20a939227d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -11603,7 +11603,12 @@ static void attach_task_cfs_rq(struct task_struct = *p) struct sched_entity *se =3D &p->se; struct cfs_rq *cfs_rq =3D cfs_rq_of(se); =20 - attach_entity_cfs_rq(se); + /* + * We couldn't detach or attach a forked task which + * hasn't been woken up by wake_up_new_task(). + */ + if (p->on_rq || se->sum_exec_runtime) + attach_entity_cfs_rq(se); =20 if (!vruntime_normalized(p)) se->vruntime +=3D cfs_rq->min_vruntime; --=20 2.36.1 From nobody Sun Apr 12 05:51:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C498AC00144 for ; Mon, 1 Aug 2022 04:29:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239597AbiHAE3J (ORCPT ); Mon, 1 Aug 2022 00:29:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239212AbiHAE2h (ORCPT ); Mon, 1 Aug 2022 00:28:37 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5E1F13E0E for ; Sun, 31 Jul 2022 21:28:32 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id h21-20020a17090aa89500b001f31a61b91dso11070748pjq.4 for ; Sun, 31 Jul 2022 21:28:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AvQvdg1vIjzkFTrlMknrP1X+3K0scOovc4jDGCZM+94=; b=QIQ9iLPpsRUCeZ1EYh7NYfSGZ48dZtim7NYNjhN4A6WQWl1WLWQuOtjvV5QDVSy79u YbHj7i7Lb6QxXmxljjy9Kbc3pAa3ooyd6me8M7DT5GizI6d5vevGtZkwpXD5IPa1ZxxM PSGlcSQOojomlOorHSY+mcdduFDViTJzMMWbk9P3CB8sVLd3lkHzLKvJ94h4hWfS7vAf Ms0k5S7CYqpVxBvM0Gh7Ohb+XByP4qodVZRydTY1GuhurkhQw8v1GWFQRkUofURVKo7n KQkst9vEfLsr1SJu5YN5PfQzNpl0GctoiKo0KkjXUTUTBTdiNazia2oCD0f61WYhtbvU QxpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AvQvdg1vIjzkFTrlMknrP1X+3K0scOovc4jDGCZM+94=; b=OeFOdqH5HbcfgsjvtfdfzA62EyozNW9V34yNNZ4olp0wYLjRpkd5Zx1vgXKBtTrBN1 gMKTygmSKOj1f+GZzOUumtACr9wPNmnFsfs7v8mnmqrmoh9R5BwI6dBK0+M3Gd9amXqc fGqX6kV1wjf5RuP90ihaWh4sDKvRMR56+vqDqvEg7XJACAG9JdaciRZ2jpWJCF2QTd2f EjrYn0Hd+JTDa3YczF3tpJ8WHiGiSh019uMVThdggrpUwX1wJ3lCx//+OVmAKkMRTZUY rUZbvHJz4XxdYbNJ7KONt91SZm6Q1Rgsq0Mxzgc8a4sC0+7RBC3CwGgypnqsN5c5bFWJ 5CnQ== X-Gm-Message-State: ACgBeo26fxksZa0nxNe02bxidfncjiv6pZ81rqzF1yjkqXHvKL1EtIaP 8UwnWlicb8LxvFwl+TkPylBa3A== X-Google-Smtp-Source: AA6agR4zMUHRA6SGw0k9rjg/qI5h65RxbGjqVMadDKDGilpXXOADiAMOqeRAJ4BIDz8aeCD2uYux3A== X-Received: by 2002:a17:90a:e7d1:b0:1f5:665:4607 with SMTP id kb17-20020a17090ae7d100b001f506654607mr2242827pjb.77.1659328112261; Sun, 31 Jul 2022 21:28:32 -0700 (PDT) Received: from C02CV1DAMD6P.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id ot10-20020a17090b3b4a00b001f326ead012sm7012202pjb.37.2022.07.31.21.28.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jul 2022 21:28:31 -0700 (PDT) From: Chengming Zhou To: mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Chengming Zhou Subject: [PATCH v3 08/10] sched/fair: refactor detach/attach_entity_cfs_rq using update_load_avg() Date: Mon, 1 Aug 2022 12:27:43 +0800 Message-Id: <20220801042745.7794-9-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220801042745.7794-1-zhouchengming@bytedance.com> References: <20220801042745.7794-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Since update_load_avg() now support DO_ATTACH and DO_DETACH flags to attach or detach entity sched_avg to the cfs_rq, we can using it to refactor detach/attach_entity_cfs_rq() functions. Note we can attach a task with last_update_time!=3D0 from switched_to_fair() since we want to decay sched_avg when running in !fair class. So this patch move last_update_time condition check to enqueue_entity() for task which migrate CPU or change cgroup. Signed-off-by: Chengming Zhou --- kernel/sched/fair.c | 68 ++++++++++++++++++--------------------------- 1 file changed, 27 insertions(+), 41 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ad20a939227d..b8cb826bd755 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4019,21 +4019,10 @@ static inline void update_load_avg(struct cfs_rq *c= fs_rq, struct sched_entity *s decayed =3D update_cfs_rq_load_avg(now, cfs_rq); decayed |=3D propagate_entity_load_avg(se); =20 - if (!se->avg.last_update_time && (flags & DO_ATTACH)) { - /* - * DO_ATTACH means we're here from enqueue_entity(). - * !last_update_time means we've passed through - * migrate_task_rq_fair() indicating we migrated. - * - * IOW we're enqueueing a task on a new CPU. - */ + if (flags & DO_ATTACH) { attach_entity_load_avg(cfs_rq, se); update_tg_load_avg(cfs_rq); } else if (flags & DO_DETACH) { - /* - * DO_DETACH means we're here from dequeue_entity() - * and we are migrating task out of the CPU. - */ detach_entity_load_avg(cfs_rq, se); update_tg_load_avg(cfs_rq); } else if (decayed) { @@ -4082,44 +4071,31 @@ static void remove_entity_load_avg(struct sched_ent= ity *se) raw_spin_unlock_irqrestore(&cfs_rq->removed.lock, flags); } =20 -#ifdef CONFIG_FAIR_GROUP_SCHED /* * Propagate the changes of the sched_entity across the tg tree to make it * visible to the root */ -static void propagate_entity_cfs_rq(struct sched_entity *se) +static void propagate_entity_cfs_rq(struct sched_entity *se, int flags) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - - if (cfs_rq_throttled(cfs_rq)) - return; - - if (!throttled_hierarchy(cfs_rq)) - list_add_leaf_cfs_rq(cfs_rq); - - /* Start to propagate at parent */ - se =3D se->parent; + struct cfs_rq *cfs_rq; =20 for_each_sched_entity(se) { cfs_rq =3D cfs_rq_of(se); =20 - update_load_avg(cfs_rq, se, UPDATE_TG); + update_load_avg(cfs_rq, se, flags); =20 if (cfs_rq_throttled(cfs_rq)) break; =20 if (!throttled_hierarchy(cfs_rq)) list_add_leaf_cfs_rq(cfs_rq); + + flags =3D UPDATE_TG; } } -#else -static void propagate_entity_cfs_rq(struct sched_entity *se) { } -#endif =20 static void detach_entity_cfs_rq(struct sched_entity *se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); - /* * In case the task sched_avg hasn't been attached: * - A forked task which hasn't been woken up by wake_up_new_task(). @@ -4130,21 +4106,18 @@ static void detach_entity_cfs_rq(struct sched_entit= y *se) return; =20 /* Catch up with the cfs_rq and remove our load when we leave */ - update_load_avg(cfs_rq, se, 0); - detach_entity_load_avg(cfs_rq, se); - update_tg_load_avg(cfs_rq); - propagate_entity_cfs_rq(se); + propagate_entity_cfs_rq(se, DO_DETACH | UPDATE_TG); } =20 static void attach_entity_cfs_rq(struct sched_entity *se) { - struct cfs_rq *cfs_rq =3D cfs_rq_of(se); + int flags =3D DO_ATTACH | UPDATE_TG; + + if (!sched_feat(ATTACH_AGE_LOAD)) + flags |=3D SKIP_AGE_LOAD; =20 - /* Synchronize entity with its cfs_rq */ - update_load_avg(cfs_rq, se, sched_feat(ATTACH_AGE_LOAD) ? 0 : SKIP_AGE_LO= AD); - attach_entity_load_avg(cfs_rq, se); - update_tg_load_avg(cfs_rq); - propagate_entity_cfs_rq(se); + /* Synchronize entity with its cfs_rq and attach our load */ + propagate_entity_cfs_rq(se, flags); } =20 static inline unsigned long cfs_rq_runnable_avg(struct cfs_rq *cfs_rq) @@ -4479,6 +4452,15 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_e= ntity *se, int flags) { bool renorm =3D !(flags & ENQUEUE_WAKEUP) || (flags & ENQUEUE_MIGRATED); bool curr =3D cfs_rq->curr =3D=3D se; + int action =3D UPDATE_TG; + + /* + * !last_update_time means we've passed through migrate_task_rq_fair() + * or task_change_group_fair() indicating we migrated cfs_rq. IOW we're + * enqueueing a task on a new CPU or moving task to a new cgroup. + */ + if (!se->avg.last_update_time) + action |=3D DO_ATTACH; =20 /* * If we're the current task, we must renormalise before calling @@ -4507,7 +4489,7 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_en= tity *se, int flags) * its group cfs_rq * - Add its new weight to cfs_rq->load.weight */ - update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH); + update_load_avg(cfs_rq, se, action); se_update_runnable(se); update_cfs_group(se); account_entity_enqueue(cfs_rq, se); @@ -4581,6 +4563,10 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_e= ntity *se, int flags) { int action =3D UPDATE_TG; =20 + /* + * When we are migrating task out of the CPU, we should + * detach entity sched_avg from the cfs_rq. + */ if (entity_is_task(se) && task_on_rq_migrating(task_of(se))) action |=3D DO_DETACH; =20 --=20 2.36.1 From nobody Sun Apr 12 05:51:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0668C00144 for ; Mon, 1 Aug 2022 04:29:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239640AbiHAE3P (ORCPT ); Mon, 1 Aug 2022 00:29:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239273AbiHAE2r (ORCPT ); Mon, 1 Aug 2022 00:28:47 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23B2813E25 for ; Sun, 31 Jul 2022 21:28:37 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id d65-20020a17090a6f4700b001f303a97b14so10731542pjk.1 for ; Sun, 31 Jul 2022 21:28:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WKQDPUfMOqNNFX3ZLkp3YqkVikVp2AFXf+ofcvp1J5c=; b=xCo14mSny3rZHNsOFgGw4Mkq+T0dT7zpzcYZyORepoc5/u42hGdysangVWtHvSMnik kd3BQuYTvkg/oeuQdUS6ZJT3ZN8+WnucAnFlsQ18pbaG5lFh9+RIMom7TLe/O9vpFcTb UL5o4SfrSOYML5eNcGLPYIG2lbS6OU9IGNKbtusG6tpmv45IQNFRiaX2ctCuBSOwdoig pfTnOu+2eZSJ0Zyi4x9pS6F97P+H+pdKO1qVCI0P0P2Uw/6aq07Kj9tYWI0oDpgjFnU/ qkTYhGTXuOycgLX0G0EzILNt5rMbdXDRJ8uXU/5SCvU6MhaZuKmSxJgow0rjOtQPLq04 KFcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WKQDPUfMOqNNFX3ZLkp3YqkVikVp2AFXf+ofcvp1J5c=; b=2mj+DZp6yBJHVDbtSOrBOslOlCjGVrEc2xv2GPrsyfT0QgvwxaAKX7GLaWBSH6lqj1 /2LStU4G9LR+mIiiZ8SULruLPmGJ/sv+/3ztmOLnV1pr2QJmCxlVsat+2JsCn3ydOmnl dIW9DLaY0Oyi/Gryg8aS+nNWf/pseekoJ8whbLTwD20b+KN1KPQGu+U8fgOlhdetyMIT 8+J3JBDBc06AnKuJ4K5wmY5jsz2JCMGpaCTuXVcAscumyLaNMTRat95b9BatfIoLyTt6 bHMUTdBc/D7LN5P4dIE2k8GwHvHOAy0q7151m1ndvPzG5pkmg7CjXeWiPyXqDEZzZ28U GFLg== X-Gm-Message-State: ACgBeo2crmySdwKLRGQqgOrAsGsl3tvXL1A5n36In1gYJ7q/zMrA9r2L GH0tzVG4vysBE55n3THt2XF4lHwfBsKJpA== X-Google-Smtp-Source: AA6agR4G9Fl+nArsOolHt1TIH4T/atuWoRK1GVSJgygTE68ffdHM2cygtSYrbBRVPc2Tt+/jyYsj2g== X-Received: by 2002:a17:902:e84e:b0:16c:773:9daf with SMTP id t14-20020a170902e84e00b0016c07739dafmr14999273plg.43.1659328116230; Sun, 31 Jul 2022 21:28:36 -0700 (PDT) Received: from C02CV1DAMD6P.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id ot10-20020a17090b3b4a00b001f326ead012sm7012202pjb.37.2022.07.31.21.28.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jul 2022 21:28:35 -0700 (PDT) From: Chengming Zhou To: mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Chengming Zhou Subject: [PATCH v3 09/10] sched/fair: defer task sched_avg attach to enqueue_entity() Date: Mon, 1 Aug 2022 12:27:44 +0800 Message-Id: <20220801042745.7794-10-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220801042745.7794-1-zhouchengming@bytedance.com> References: <20220801042745.7794-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When wake_up_new_task(), we would use post_init_entity_util_avg() to init util_avg/runnable_avg based on cpu's util_avg at that time, then attach task sched_avg to cfs_rq. Since enqueue_entity() would always attach any unattached task entity, so we can defer this work to enqueue_entity(). post_init_entity_util_avg(p) attach_entity_cfs_rq() --> (1) activate_task(rq, p) enqueue_task() :=3D enqueue_task_fair() enqueue_entity() update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH) if (!se->avg.last_update_time && (flags & DO_ATTACH)) attach_entity_load_avg() --> (2) This patch defer attach from (1) to (2) Signed-off-by: Chengming Zhou --- kernel/sched/fair.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index b8cb826bd755..18e3dff606db 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -799,8 +799,6 @@ void init_entity_runnable_average(struct sched_entity *= se) /* when this task enqueue'ed, it will contribute to its cfs_rq's load_avg= */ } =20 -static void attach_entity_cfs_rq(struct sched_entity *se); - /* * With new tasks being created, their initial util_avgs are extrapolated * based on the cfs_rq's current util_avg: @@ -863,8 +861,6 @@ void post_init_entity_util_avg(struct task_struct *p) se->avg.last_update_time =3D cfs_rq_clock_pelt(cfs_rq); return; } - - attach_entity_cfs_rq(se); } =20 #else /* !CONFIG_SMP */ --=20 2.36.1 From nobody Sun Apr 12 05:51:33 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44D7CC00144 for ; Mon, 1 Aug 2022 04:29:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239700AbiHAE3S (ORCPT ); Mon, 1 Aug 2022 00:29:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239647AbiHAE2s (ORCPT ); Mon, 1 Aug 2022 00:28:48 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8339313F4B for ; Sun, 31 Jul 2022 21:28:40 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id ha11so9797498pjb.2 for ; Sun, 31 Jul 2022 21:28:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MF8vit1pzG5l/SDs5xaftECSlzOvu9Mvi/2q+XylMVk=; b=7+mdjoRv3mAvaUsy/qaAqhIYUHnFhezVR2vZwwAlFIVwXdSwYRO6mwRJo5YwkyX2xm w0nJsBSfCBMbFdMsrc9F1Bye1kKKsSUgknIHjIBqb2KUpeLO2HrNVNX18ADlsMdXV9/u khn3T3GUscsXQ3Ju5KF8pxHrUQHRYdg8WG5qBj1sAsPQQNBbtiu/E3AguAwoKKQde44v uFSG2PMpXEbsOil4TOLca6ibOSf8BUzGWCwOZj9ECc1V7x06Z9nAhxZ60QJOn3kw9f91 6kkNZ6f9Bu5QJBhONo/OBOn+p0nlXs+kD6BNyxknJlo1kfcAkVmvCm+7eVKatdKus8SG FQhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MF8vit1pzG5l/SDs5xaftECSlzOvu9Mvi/2q+XylMVk=; b=p9T18MQfahM8FEYyya8t/BgLM/dKJHUdDK8Hoss+vXH4RxBAGopLkop6+nvRoKpp/o tfDjvmz2RH5aoDqyeqei5FwgBm/tPUqs0RUucBQnRCJ6idy8vPL7g/EaTBGmmAkApf3F +b3ZORS987zKSuNX7w2oaSbKUWTOThDYfQQCSZx3cGrKjzVQwWHWaC2el8WlF2L+aXCB 31d8YdxnMs8p5TwCstcwxt6k4gLBtjAulM3JPF1V4nfajKWwDpPoktBAX0J9jywmqRVb bKIM7ra6QDOnJL+LzgQqC+4N+sguIuSMTt5toy2ZHWj/uiQG3cW478ELfE5LbmOmxFPC UYIA== X-Gm-Message-State: ACgBeo2xDNdGKY5zZJV4yJjZ0nfs86zWA1jM6Lmw/a54m181ZBm/gRCn DQsW4zUSIes68/gvP1AL6quv0Q== X-Google-Smtp-Source: AA6agR7/2W0lgoYct0viSrbCL+9fFWA7Ftr7sCPcpxtGoRtRb7ZnC5KEK1VOskIud253AtdsW4OxqQ== X-Received: by 2002:a17:90b:4b8a:b0:1f5:942:dcc6 with SMTP id lr10-20020a17090b4b8a00b001f50942dcc6mr1595918pjb.78.1659328120088; Sun, 31 Jul 2022 21:28:40 -0700 (PDT) Received: from C02CV1DAMD6P.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id ot10-20020a17090b3b4a00b001f326ead012sm7012202pjb.37.2022.07.31.21.28.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Jul 2022 21:28:39 -0700 (PDT) From: Chengming Zhou To: mingo@redhat.com, peterz@infradead.org, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, vschneid@redhat.com Cc: linux-kernel@vger.kernel.org, Chengming Zhou Subject: [PATCH v3 10/10] sched/fair: don't init util/runnable_avg for !fair task Date: Mon, 1 Aug 2022 12:27:45 +0800 Message-Id: <20220801042745.7794-11-zhouchengming@bytedance.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220801042745.7794-1-zhouchengming@bytedance.com> References: <20220801042745.7794-1-zhouchengming@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" post_init_entity_util_avg() init task util_avg according to the cpu util_avg at the time of fork, which will decay when switched_to_fair() some time lat= er, we'd better to not set them at all in the case of !fair task. Suggested-by: Vincent Guittot Signed-off-by: Chengming Zhou --- kernel/sched/fair.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 18e3dff606db..071c159605e7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -833,20 +833,6 @@ void post_init_entity_util_avg(struct task_struct *p) long cpu_scale =3D arch_scale_cpu_capacity(cpu_of(rq_of(cfs_rq))); long cap =3D (long)(cpu_scale - cfs_rq->avg.util_avg) / 2; =20 - if (cap > 0) { - if (cfs_rq->avg.util_avg !=3D 0) { - sa->util_avg =3D cfs_rq->avg.util_avg * se->load.weight; - sa->util_avg /=3D (cfs_rq->avg.load_avg + 1); - - if (sa->util_avg > cap) - sa->util_avg =3D cap; - } else { - sa->util_avg =3D cap; - } - } - - sa->runnable_avg =3D sa->util_avg; - if (p->sched_class !=3D &fair_sched_class) { /* * For !fair tasks do: @@ -861,6 +847,20 @@ void post_init_entity_util_avg(struct task_struct *p) se->avg.last_update_time =3D cfs_rq_clock_pelt(cfs_rq); return; } + + if (cap > 0) { + if (cfs_rq->avg.util_avg !=3D 0) { + sa->util_avg =3D cfs_rq->avg.util_avg * se->load.weight; + sa->util_avg /=3D (cfs_rq->avg.load_avg + 1); + + if (sa->util_avg > cap) + sa->util_avg =3D cap; + } else { + sa->util_avg =3D cap; + } + } + + sa->runnable_avg =3D sa->util_avg; } =20 #else /* !CONFIG_SMP */ --=20 2.36.1