From nobody Wed Dec 17 10:58:40 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26BCCC4167B for ; Fri, 1 Dec 2023 16:17:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378399AbjLAQRF (ORCPT ); Fri, 1 Dec 2023 11:17:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378341AbjLAQQ4 (ORCPT ); Fri, 1 Dec 2023 11:16:56 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52B171724 for ; Fri, 1 Dec 2023 08:17:02 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id ffacd0b85a97d-332ddd8e0bdso2238415f8f.0 for ; Fri, 01 Dec 2023 08:17:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1701447421; x=1702052221; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XSKzabyf617fM1KjEqCfZfHlIw32BDDg2adJnGl6QJ8=; b=DQ3YFnHounWAqRoT31a5f8sVryNO0uLYEC2wE8ZUzQOYOPszbJfX/sqHsyoEhw4tOp pl/JcCzJZfhLx8yy0SQKRx3bKjmPYFW7gUr9skm+xEMYbAeUTp9fQo5ViDq64Ga1kVaT mLx9kY/PWRB/IoPUQZV16xa9OYNbsT9XGakS2jxDpNJ7991GKHlcRfjLG00N7inK3O6k RL8Z4zxOMljn++e1LEnOPkbCTOQRK1PrD/xtWIeKb+L6irEEDz/6B+Ya9MiQ800fRQSQ LqlTDpFk8n+bUCj6djNyiUzE+Na5dHYVIuO6I5lk7d0RNPso5C9QODkdIcCSmS5TLPMk Mb5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701447421; x=1702052221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XSKzabyf617fM1KjEqCfZfHlIw32BDDg2adJnGl6QJ8=; b=uCBdc89554O/tXN3R8PjkRSSIo7znpKtJNFA+RCkzkM9mrtx6nXc8Gg/cT4x6+ZZdy FLMQ/+vOUBfbQrGmAMOaMIG5cA6lLtEONFmRGum0vnpPnhYDQjgH2DHLAywLYNvEGQ+n 5HREKZA+aB1Y20BsXiXiBoffY64bLbkeyvE8nzpDapIOOr6o8EVwDkzyG6Fq5S+Kgqwm kzbq+WdL02UvlKvEaj2wN4sQCUnotQXjg+zjI8+6clyjzLxqLbsg4rbZD4KPgNikYDJ4 UrfZgGd9Abou3EBiieDdUchu/GgmNYbMFmRFH8wGB+nGUFFd4cpnR3miJ6tQUjGF9C9d 4nVQ== X-Gm-Message-State: AOJu0YxbmQaV9cLkfJt9TEWmreYTW3v/6HtaMJ7mRn7RUg1B9737XWkE pz0R/ceYeLP61zxmGM/ynFZ/wg== X-Google-Smtp-Source: AGHT+IHjqFYkByXc9Gb3pcb1OogPS3f1+e4xDX/ps+5Ye1VAGcstpN9qt433fzLCKLTPnd9WoPHO1g== X-Received: by 2002:a05:600c:1615:b0:40b:5e56:7b5c with SMTP id m21-20020a05600c161500b0040b5e567b5cmr764423wmn.165.1701447420729; Fri, 01 Dec 2023 08:17:00 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:6db5:f04:fbd5:88e1]) by smtp.gmail.com with ESMTPSA id l27-20020a05600c1d1b00b003feae747ff2sm9591398wms.35.2023.12.01.08.16.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 08:16:58 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, corbet@lwn.net, alexs@kernel.org, siyanteng@loongson.cn, qyousef@layalina.io, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org Cc: lukasz.luba@arm.com, hongyan.xia2@arm.com, yizhou.tang@shopee.com, Vincent Guittot Subject: [PATCH v2 1/2] sched/fair: Remove SCHED_FEAT(UTIL_EST_FASTUP, true) Date: Fri, 1 Dec 2023 17:16:51 +0100 Message-Id: <20231201161652.1241695-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231201161652.1241695-1-vincent.guittot@linaro.org> References: <20231201161652.1241695-1-vincent.guittot@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org sched_feat(UTIL_EST_FASTUP) has been added to easily disable the feature in order to check for possibly related regressions. After 3 years, it has never been used and no regression has been reported. Let remove it and make fast increase a permanent behavior. Signed-off-by: Vincent Guittot Reviewed-and-tested-by: Lukasz Luba Reviewed-by: Dietmar Eggemann Reviewed-by: Hongyan Xia Reviewed-by: Tang Yizhou Reviewed-by: Alex Shi Reviewed-by: Yanteng Si --- Documentation/scheduler/schedutil.rst | 7 +++---- Documentation/translations/zh_CN/scheduler/schedutil.rst | 7 +++---- kernel/sched/fair.c | 8 +++----- kernel/sched/features.h | 1 - 4 files changed, 9 insertions(+), 14 deletions(-) diff --git a/Documentation/scheduler/schedutil.rst b/Documentation/schedule= r/schedutil.rst index 32c7d69fc86c..803fba8fc714 100644 --- a/Documentation/scheduler/schedutil.rst +++ b/Documentation/scheduler/schedutil.rst @@ -90,8 +90,8 @@ For more detail see: - Documentation/scheduler/sched-capacity.rst:"1. CPU Capacity + 2. Task u= tilization" =20 =20 -UTIL_EST / UTIL_EST_FASTUP -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D +UTIL_EST +=3D=3D=3D=3D=3D=3D=3D=3D =20 Because periodic tasks have their averages decayed while they sleep, even though when running their expected utilization will be the same, they suff= er a @@ -99,8 +99,7 @@ though when running their expected utilization will be th= e same, they suffer a =20 To alleviate this (a default enabled option) UTIL_EST drives an Infinite Impulse Response (IIR) EWMA with the 'running' value on dequeue -- when it= is -highest. A further default enabled option UTIL_EST_FASTUP modifies the IIR -filter to instantly increase and only decay on decrease. +highest. UTIL_EST filters to instantly increase and only decay on decrease. =20 A further runqueue wide sum (of runnable tasks) is maintained of: =20 diff --git a/Documentation/translations/zh_CN/scheduler/schedutil.rst b/Doc= umentation/translations/zh_CN/scheduler/schedutil.rst index d1ea68007520..7c8d87f21c42 100644 --- a/Documentation/translations/zh_CN/scheduler/schedutil.rst +++ b/Documentation/translations/zh_CN/scheduler/schedutil.rst @@ -89,16 +89,15 @@ r_cpu=E8=A2=AB=E5=AE=9A=E4=B9=89=E4=B8=BA=E5=BD=93=E5= =89=8DCPU=E7=9A=84=E6=9C=80=E9=AB=98=E6=80=A7=E8=83=BD=E6=B0=B4=E5=B9=B3=E4= =B8=8E=E7=B3=BB=E7=BB=9F=E4=B8=AD=E4=BB=BB=E4=BD=95=E5=85=B6=E5=AE=83CPU=E7= =9A=84=E6=9C=80 - Documentation/translations/zh_CN/scheduler/sched-capacity.rst:"1. CPU C= apacity + 2. Task utilization" =20 =20 -UTIL_EST / UTIL_EST_FASTUP -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D +UTIL_EST +=3D=3D=3D=3D=3D=3D=3D=3D =20 =E7=94=B1=E4=BA=8E=E5=91=A8=E6=9C=9F=E6=80=A7=E4=BB=BB=E5=8A=A1=E7=9A=84= =E5=B9=B3=E5=9D=87=E6=95=B0=E5=9C=A8=E7=9D=A1=E7=9C=A0=E6=97=B6=E4=BC=9A=E8= =A1=B0=E5=87=8F=EF=BC=8C=E8=80=8C=E5=9C=A8=E8=BF=90=E8=A1=8C=E6=97=B6=E5=85= =B6=E9=A2=84=E6=9C=9F=E5=88=A9=E7=94=A8=E7=8E=87=E4=BC=9A=E5=92=8C=E7=9D=A1= =E7=9C=A0=E5=89=8D=E7=9B=B8=E5=90=8C=EF=BC=8C =E5=9B=A0=E6=AD=A4=E5=AE=83=E4=BB=AC=E5=9C=A8=E5=86=8D=E6=AC=A1=E8=BF=90= =E8=A1=8C=E5=90=8E=E4=BC=9A=E9=9D=A2=E4=B8=B4=EF=BC=88DVFS=EF=BC=89=E7=9A= =84=E4=B8=8A=E6=B6=A8=E3=80=82 =20 =E4=B8=BA=E4=BA=86=E7=BC=93=E8=A7=A3=E8=BF=99=E4=B8=AA=E9=97=AE=E9=A2=98= =EF=BC=8C=EF=BC=88=E4=B8=80=E4=B8=AA=E9=BB=98=E8=AE=A4=E4=BD=BF=E8=83=BD=E7= =9A=84=E7=BC=96=E8=AF=91=E9=80=89=E9=A1=B9=EF=BC=89UTIL_EST=E9=A9=B1=E5=8A= =A8=E4=B8=80=E4=B8=AA=E6=97=A0=E9=99=90=E8=84=89=E5=86=B2=E5=93=8D=E5=BA=94 =EF=BC=88Infinite Impulse Response=EF=BC=8CIIR=EF=BC=89=E7=9A=84EWMA=EF=BC= =8C=E2=80=9C=E8=BF=90=E8=A1=8C=E2=80=9D=E5=80=BC=E5=9C=A8=E5=87=BA=E9=98=9F= =E6=97=B6=E6=98=AF=E6=9C=80=E9=AB=98=E7=9A=84=E3=80=82 -=E5=8F=A6=E4=B8=80=E4=B8=AA=E9=BB=98=E8=AE=A4=E4=BD=BF=E8=83=BD=E7=9A=84= =E7=BC=96=E8=AF=91=E9=80=89=E9=A1=B9UTIL_EST_FASTUP=E4=BF=AE=E6=94=B9=E4=BA= =86IIR=E6=BB=A4=E6=B3=A2=E5=99=A8=EF=BC=8C=E4=BD=BF=E5=85=B6=E5=85=81=E8=AE= =B8=E7=AB=8B=E5=8D=B3=E5=A2=9E=E5=8A=A0=EF=BC=8C -=E4=BB=85=E5=9C=A8=E5=88=A9=E7=94=A8=E7=8E=87=E4=B8=8B=E9=99=8D=E6=97=B6= =E8=A1=B0=E5=87=8F=E3=80=82 +UTIL_EST=E6=BB=A4=E6=B3=A2=E4=BD=BF=E5=85=B6=E5=9C=A8=E9=81=87=E5=88=B0=E6= =9B=B4=E9=AB=98=E5=80=BC=E6=97=B6=E7=AB=8B=E5=88=BB=E5=A2=9E=E5=8A=A0=EF=BC= =8C=E8=80=8C=E9=81=87=E5=88=B0=E4=BD=8E=E5=80=BC=E6=97=B6=E4=BC=9A=E7=BC=93= =E6=85=A2=E8=A1=B0=E5=87=8F=E3=80=82 =20 =E8=BF=9B=E4=B8=80=E6=AD=A5=EF=BC=8C=E8=BF=90=E8=A1=8C=E9=98=9F=E5=88=97= =E7=9A=84=EF=BC=88=E5=8F=AF=E8=BF=90=E8=A1=8C=E4=BB=BB=E5=8A=A1=E7=9A=84=EF= =BC=89=E5=88=A9=E7=94=A8=E7=8E=87=E4=B9=8B=E5=92=8C=E7=94=B1=E4=B8=8B=E5=BC= =8F=E8=AE=A1=E7=AE=97=EF=BC=9A =20 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index bcea3d55d95d..e94d65da8d66 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4870,11 +4870,9 @@ static inline void util_est_update(struct cfs_rq *cf= s_rq, * to smooth utilization decreases. */ ue.enqueued =3D task_util(p); - if (sched_feat(UTIL_EST_FASTUP)) { - if (ue.ewma < ue.enqueued) { - ue.ewma =3D ue.enqueued; - goto done; - } + if (ue.ewma < ue.enqueued) { + ue.ewma =3D ue.enqueued; + goto done; } =20 /* diff --git a/kernel/sched/features.h b/kernel/sched/features.h index a3ddf84de430..143f55df890b 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -83,7 +83,6 @@ SCHED_FEAT(WA_BIAS, true) * UtilEstimation. Use estimated CPU utilization. */ SCHED_FEAT(UTIL_EST, true) -SCHED_FEAT(UTIL_EST_FASTUP, true) =20 SCHED_FEAT(LATENCY_WARN, false) =20 --=20 2.34.1 From nobody Wed Dec 17 10:58:40 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1808BC4167B for ; Fri, 1 Dec 2023 16:17:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1378442AbjLAQRO (ORCPT ); Fri, 1 Dec 2023 11:17:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1378448AbjLAQQ7 (ORCPT ); Fri, 1 Dec 2023 11:16:59 -0500 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E2C6170F for ; Fri, 1 Dec 2023 08:17:03 -0800 (PST) Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-40b479ec4a3so20813705e9.2 for ; Fri, 01 Dec 2023 08:17:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1701447422; x=1702052222; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E74LvjTxUZWLtJvvaUHypL9+wv4NgOIsNkiFl+I7+yg=; b=MlcwSg2qs1ntSYVGQd/MCxfNhPe3vgD+XBhLpmL/1OrvBtdqd8wea7Xh7kY+91rFbk wfA0udfQ0SAGtnTxH6ISma0+fgeGFZFl11vubdfkJbr3XWMLjyuNbX/PVI0ILaoK4qlZ 7L3CQ2exQCXuF+A47yMr1Y5Mqo+5WF493t13jPQprskHBceXGHM68FHbJdigDSpea7hW bY1Dy0s3eDLdElZ7N7Ybz4YvQNZDqP2x0+o1nDyEMeE7Kz4wR0DNa8c3rQzT6pFvfJRA P7o/NpFZcts5mOj8VBX+9Ku+gd72xlnn1OJZ6lNObt9l5nmfKszoRE+JDY/xckI/w0qN 2xEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701447422; x=1702052222; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E74LvjTxUZWLtJvvaUHypL9+wv4NgOIsNkiFl+I7+yg=; b=s5cF4QXSGOm9JWJbLGWnLQoc+7DIBGSsjPlbs6wy4RDTNL2HQIFv500YcF+SnQuR6m fIOWxkjcnvlxOq+dGyd6SqOGTzDFfI10ohyijy951SMal0evoLayn9PDYDCw8FUVbXOZ EVe2lbkkYAAoL+ZWMNutpUSOq/c3g9Y1eUh5/vaDJKHdMWQmNcsNdkjepLglgnrngUjG TjDZzWNCs+jtxQOjPyciQrTEE9K4arIJ8mcleZ+TcOOeTIt03PaonetAxhkM47LFFuAW jJni2GMtyCtM7aCftwl2wPrsgGv+42ccpwrT9dprVCIWUGKwi7ReIrIeNYj2uR3ShUhV b1hg== X-Gm-Message-State: AOJu0YyRiVsXsFZwHE1A4z6/Le20oDMlrC0QDOjtjgQWVjamZr5alPb2 /ge60h8HCqgsVP1xKwdyOONmcA== X-Google-Smtp-Source: AGHT+IGw9gQNjf3z/miV80OD0PEF8SvC09RO8HQQ/oLaFcUtbEKwH5EunZeAHMp3SHeSv1bdXOq24A== X-Received: by 2002:a05:600c:5247:b0:40b:5e22:962 with SMTP id fc7-20020a05600c524700b0040b5e220962mr727299wmb.81.1701447422401; Fri, 01 Dec 2023 08:17:02 -0800 (PST) Received: from vingu-book.. ([2a01:e0a:f:6020:6db5:f04:fbd5:88e1]) by smtp.gmail.com with ESMTPSA id l27-20020a05600c1d1b00b003feae747ff2sm9591398wms.35.2023.12.01.08.17.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Dec 2023 08:17:01 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, corbet@lwn.net, alexs@kernel.org, siyanteng@loongson.cn, qyousef@layalina.io, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org Cc: lukasz.luba@arm.com, hongyan.xia2@arm.com, yizhou.tang@shopee.com, Vincent Guittot Subject: [PATCH v2 2/2] sched/fair: Simplify util_est Date: Fri, 1 Dec 2023 17:16:52 +0100 Message-Id: <20231201161652.1241695-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231201161652.1241695-1-vincent.guittot@linaro.org> References: <20231201161652.1241695-1-vincent.guittot@linaro.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" With UTIL_EST_FASTUP now being permanent, we can take advantage of the fact that the ewma jumps directly to a higher utilization at dequeue to simplify util_est and remove the enqueued field. Signed-off-by: Vincent Guittot Reviewed-and-tested-by: Lukasz Luba Reviewed-by: Dietmar Eggemann Reviewed-by: Hongyan Xia Reviewed-by: Alex Shi --- include/linux/sched.h | 49 +++++++------------------- kernel/sched/debug.c | 7 ++-- kernel/sched/fair.c | 82 ++++++++++++++++--------------------------- kernel/sched/pelt.h | 4 +-- 4 files changed, 48 insertions(+), 94 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 8d258162deb0..03bfe9ab2951 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -415,42 +415,6 @@ struct load_weight { u32 inv_weight; }; =20 -/** - * struct util_est - Estimation utilization of FAIR tasks - * @enqueued: instantaneous estimated utilization of a task/cpu - * @ewma: the Exponential Weighted Moving Average (EWMA) - * utilization of a task - * - * Support data structure to track an Exponential Weighted Moving Average - * (EWMA) of a FAIR task's utilization. New samples are added to the moving - * average each time a task completes an activation. Sample's weight is ch= osen - * so that the EWMA will be relatively insensitive to transient changes to= the - * task's workload. - * - * The enqueued attribute has a slightly different meaning for tasks and c= pus: - * - task: the task's util_avg at last task dequeue time - * - cfs_rq: the sum of util_est.enqueued for each RUNNABLE task on that C= PU - * Thus, the util_est.enqueued of a task represents the contribution on the - * estimated utilization of the CPU where that task is currently enqueued. - * - * Only for tasks we track a moving average of the past instantaneous - * estimated utilization. This allows to absorb sporadic drops in utilizat= ion - * of an otherwise almost periodic task. - * - * The UTIL_AVG_UNCHANGED flag is used to synchronize util_est with util_a= vg - * updates. When a task is dequeued, its util_est should not be updated if= its - * util_avg has not been updated in the meantime. - * This information is mapped into the MSB bit of util_est.enqueued at deq= ueue - * time. Since max value of util_est.enqueued for a task is 1024 (PELT uti= l_avg - * for a task) it is safe to use MSB. - */ -struct util_est { - unsigned int enqueued; - unsigned int ewma; -#define UTIL_EST_WEIGHT_SHIFT 2 -#define UTIL_AVG_UNCHANGED 0x80000000 -} __attribute__((__aligned__(sizeof(u64)))); - /* * The load/runnable/util_avg accumulates an infinite geometric series * (see __update_load_avg_cfs_rq() in kernel/sched/pelt.c). @@ -505,9 +469,20 @@ struct sched_avg { unsigned long load_avg; unsigned long runnable_avg; unsigned long util_avg; - struct util_est util_est; + unsigned int util_est; } ____cacheline_aligned; =20 +/* + * The UTIL_AVG_UNCHANGED flag is used to synchronize util_est with util_a= vg + * updates. When a task is dequeued, its util_est should not be updated if= its + * util_avg has not been updated in the meantime. + * This information is mapped into the MSB bit of util_est at dequeue time. + * Since max value of util_est for a task is 1024 (PELT util_avg for a tas= k) + * it is safe to use MSB. + */ +#define UTIL_EST_WEIGHT_SHIFT 2 +#define UTIL_AVG_UNCHANGED 0x80000000 + struct sched_statistics { #ifdef CONFIG_SCHEDSTATS u64 wait_start; diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index 168eecc209b4..8d5d98a5834d 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -684,8 +684,8 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct c= fs_rq *cfs_rq) cfs_rq->avg.runnable_avg); SEQ_printf(m, " .%-30s: %lu\n", "util_avg", cfs_rq->avg.util_avg); - SEQ_printf(m, " .%-30s: %u\n", "util_est_enqueued", - cfs_rq->avg.util_est.enqueued); + SEQ_printf(m, " .%-30s: %u\n", "util_est", + cfs_rq->avg.util_est); SEQ_printf(m, " .%-30s: %ld\n", "removed.load_avg", cfs_rq->removed.load_avg); SEQ_printf(m, " .%-30s: %ld\n", "removed.util_avg", @@ -1075,8 +1075,7 @@ void proc_sched_show_task(struct task_struct *p, stru= ct pid_namespace *ns, P(se.avg.runnable_avg); P(se.avg.util_avg); P(se.avg.last_update_time); - P(se.avg.util_est.ewma); - PM(se.avg.util_est.enqueued, ~UTIL_AVG_UNCHANGED); + PM(se.avg.util_est, ~UTIL_AVG_UNCHANGED); #endif #ifdef CONFIG_UCLAMP_TASK __PS("uclamp.min", p->uclamp_req[UCLAMP_MIN].value); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e94d65da8d66..823dd76d0546 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4781,9 +4781,7 @@ static inline unsigned long task_runnable(struct task= _struct *p) =20 static inline unsigned long _task_util_est(struct task_struct *p) { - struct util_est ue =3D READ_ONCE(p->se.avg.util_est); - - return max(ue.ewma, (ue.enqueued & ~UTIL_AVG_UNCHANGED)); + return READ_ONCE(p->se.avg.util_est) & ~UTIL_AVG_UNCHANGED; } =20 static inline unsigned long task_util_est(struct task_struct *p) @@ -4800,9 +4798,9 @@ static inline void util_est_enqueue(struct cfs_rq *cf= s_rq, return; =20 /* Update root cfs_rq's estimated utilization */ - enqueued =3D cfs_rq->avg.util_est.enqueued; + enqueued =3D cfs_rq->avg.util_est; enqueued +=3D _task_util_est(p); - WRITE_ONCE(cfs_rq->avg.util_est.enqueued, enqueued); + WRITE_ONCE(cfs_rq->avg.util_est, enqueued); =20 trace_sched_util_est_cfs_tp(cfs_rq); } @@ -4816,34 +4814,20 @@ static inline void util_est_dequeue(struct cfs_rq *= cfs_rq, return; =20 /* Update root cfs_rq's estimated utilization */ - enqueued =3D cfs_rq->avg.util_est.enqueued; + enqueued =3D cfs_rq->avg.util_est; enqueued -=3D min_t(unsigned int, enqueued, _task_util_est(p)); - WRITE_ONCE(cfs_rq->avg.util_est.enqueued, enqueued); + WRITE_ONCE(cfs_rq->avg.util_est, enqueued); =20 trace_sched_util_est_cfs_tp(cfs_rq); } =20 #define UTIL_EST_MARGIN (SCHED_CAPACITY_SCALE / 100) =20 -/* - * Check if a (signed) value is within a specified (unsigned) margin, - * based on the observation that: - * - * abs(x) < y :=3D (unsigned)(x + y - 1) < (2 * y - 1) - * - * NOTE: this only works when value + margin < INT_MAX. - */ -static inline bool within_margin(int value, int margin) -{ - return ((unsigned int)(value + margin - 1) < (2 * margin - 1)); -} - static inline void util_est_update(struct cfs_rq *cfs_rq, struct task_struct *p, bool task_sleep) { - long last_ewma_diff, last_enqueued_diff; - struct util_est ue; + unsigned int ewma, dequeued, last_ewma_diff; =20 if (!sched_feat(UTIL_EST)) return; @@ -4855,23 +4839,25 @@ static inline void util_est_update(struct cfs_rq *c= fs_rq, if (!task_sleep) return; =20 + /* Get current estimate of utilization */ + ewma =3D READ_ONCE(p->se.avg.util_est); + /* * If the PELT values haven't changed since enqueue time, * skip the util_est update. */ - ue =3D p->se.avg.util_est; - if (ue.enqueued & UTIL_AVG_UNCHANGED) + if (ewma & UTIL_AVG_UNCHANGED) return; =20 - last_enqueued_diff =3D ue.enqueued; + /* Get utilization at dequeue */ + dequeued =3D task_util(p); =20 /* * Reset EWMA on utilization increases, the moving average is used only * to smooth utilization decreases. */ - ue.enqueued =3D task_util(p); - if (ue.ewma < ue.enqueued) { - ue.ewma =3D ue.enqueued; + if (ewma <=3D dequeued) { + ewma =3D dequeued; goto done; } =20 @@ -4879,27 +4865,22 @@ static inline void util_est_update(struct cfs_rq *c= fs_rq, * Skip update of task's estimated utilization when its members are * already ~1% close to its last activation value. */ - last_ewma_diff =3D ue.enqueued - ue.ewma; - last_enqueued_diff -=3D ue.enqueued; - if (within_margin(last_ewma_diff, UTIL_EST_MARGIN)) { - if (!within_margin(last_enqueued_diff, UTIL_EST_MARGIN)) - goto done; - - return; - } + last_ewma_diff =3D ewma - dequeued; + if (last_ewma_diff < UTIL_EST_MARGIN) + goto done; =20 /* * To avoid overestimation of actual task utilization, skip updates if * we cannot grant there is idle time in this CPU. */ - if (task_util(p) > arch_scale_cpu_capacity(cpu_of(rq_of(cfs_rq)))) + if (dequeued > arch_scale_cpu_capacity(cpu_of(rq_of(cfs_rq)))) return; =20 /* * To avoid underestimate of task utilization, skip updates of EWMA if * we cannot grant that thread got all CPU time it wanted. */ - if ((ue.enqueued + UTIL_EST_MARGIN) < task_runnable(p)) + if ((dequeued + UTIL_EST_MARGIN) < task_runnable(p)) goto done; =20 =20 @@ -4907,25 +4888,24 @@ static inline void util_est_update(struct cfs_rq *c= fs_rq, * Update Task's estimated utilization * * When *p completes an activation we can consolidate another sample - * of the task size. This is done by storing the current PELT value - * as ue.enqueued and by using this value to update the Exponential - * Weighted Moving Average (EWMA): + * of the task size. This is done by using this value to update the + * Exponential Weighted Moving Average (EWMA): * * ewma(t) =3D w * task_util(p) + (1-w) * ewma(t-1) * =3D w * task_util(p) + ewma(t-1) - w * ewma(t-1) * =3D w * (task_util(p) - ewma(t-1)) + ewma(t-1) - * =3D w * ( last_ewma_diff ) + ewma(t-1) - * =3D w * (last_ewma_diff + ewma(t-1) / w) + * =3D w * ( -last_ewma_diff ) + ewma(t-1) + * =3D w * (-last_ewma_diff + ewma(t-1) / w) * * Where 'w' is the weight of new samples, which is configured to be * 0.25, thus making w=3D1/4 ( >>=3D UTIL_EST_WEIGHT_SHIFT) */ - ue.ewma <<=3D UTIL_EST_WEIGHT_SHIFT; - ue.ewma +=3D last_ewma_diff; - ue.ewma >>=3D UTIL_EST_WEIGHT_SHIFT; + ewma <<=3D UTIL_EST_WEIGHT_SHIFT; + ewma -=3D last_ewma_diff; + ewma >>=3D UTIL_EST_WEIGHT_SHIFT; done: - ue.enqueued |=3D UTIL_AVG_UNCHANGED; - WRITE_ONCE(p->se.avg.util_est, ue); + ewma |=3D UTIL_AVG_UNCHANGED; + WRITE_ONCE(p->se.avg.util_est, ewma); =20 trace_sched_util_est_se_tp(&p->se); } @@ -7653,16 +7633,16 @@ cpu_util(int cpu, struct task_struct *p, int dst_cp= u, int boost) if (sched_feat(UTIL_EST)) { unsigned long util_est; =20 - util_est =3D READ_ONCE(cfs_rq->avg.util_est.enqueued); + util_est =3D READ_ONCE(cfs_rq->avg.util_est); =20 /* * During wake-up @p isn't enqueued yet and doesn't contribute - * to any cpu_rq(cpu)->cfs.avg.util_est.enqueued. + * to any cpu_rq(cpu)->cfs.avg.util_est. * If @dst_cpu =3D=3D @cpu add it to "simulate" cpu_util after @p * has been enqueued. * * During exec (@dst_cpu =3D -1) @p is enqueued and does - * contribute to cpu_rq(cpu)->cfs.util_est.enqueued. + * contribute to cpu_rq(cpu)->cfs.util_est. * Remove it to "simulate" cpu_util without @p's contribution. * * Despite the task_on_rq_queued(@p) check there is still a diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h index 3a0e0dc28721..9e1083465fbc 100644 --- a/kernel/sched/pelt.h +++ b/kernel/sched/pelt.h @@ -52,13 +52,13 @@ static inline void cfs_se_util_change(struct sched_avg = *avg) return; =20 /* Avoid store if the flag has been already reset */ - enqueued =3D avg->util_est.enqueued; + enqueued =3D avg->util_est; if (!(enqueued & UTIL_AVG_UNCHANGED)) return; =20 /* Reset flag to report util_avg has been updated */ enqueued &=3D ~UTIL_AVG_UNCHANGED; - WRITE_ONCE(avg->util_est.enqueued, enqueued); + WRITE_ONCE(avg->util_est, enqueued); } =20 static inline u64 rq_clock_pelt(struct rq *rq) --=20 2.34.1