From nobody Mon Dec 1 21:32:53 2025 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE5B93002AA for ; Mon, 1 Dec 2025 09:13:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580397; cv=none; b=nUUVMS6C9TCOKz2m+UGQrNtYC9xHoF8rnu8JKymfzHhRo/W6ch4GM3F2HrTCuaIDxV4SWLuG6hNdHTZFTXtam2pLmfCbL3Q4BeIBZc3J7zRWg5ILdCxeTlPuW0YHYWpOTLQ5AcA67vztaEx2pc/gwf3jpdo1po3/Zxr6S1cRcd4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580397; c=relaxed/simple; bh=zV8/cEhPE7YZs10VV/3SNSRu2hQIU2zJ8NKh0SZ9GTw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oCNbKo9ELAkmWUiB82FYNR3DF7zdS1FU/rr7PF6Zho9y02sQnLHM69hVAmruBB84A9ClL9/TaVBcNHU8KeCD3Se9NwHMv7dtuQcG+VzHS8uGvpQ2c687fv0BBugBk2xXZE9LBzeKWPd2TwOOEyip3Ny4Dr88efymiXDuM3FXtmw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=UctolfIt; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="UctolfIt" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-42e2d5e119fso644701f8f.2 for ; Mon, 01 Dec 2025 01:13:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764580393; x=1765185193; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OwX/olriHOAxoNjW5iw4IZ+oz3oFzlu8t83MnxTRx/U=; b=UctolfIttpGws+/h6k+ef5UHIjIeNQczHMQw8TPc4Y/f+EVqqBfhr8s6IcXDSbQVkF 2TEs2Z0r2l1GkZlHQV8Iu6SRCvimnyqYGMKK2VRVfCCB1+ZmJDbMnQ9W4z1eNU79Nmt+ +5S3BFyo57abXDev7KZX++GhDapWoJt349p8F5q9Y7DJC6ZmMFcub7F1dXaO5qxhOcfA w5ieOZ8NKXlBJ0X7gRRaJ/1tgd+bPWA/dVf6GsUv6DFciZX8O6z4lg+UgmzA0g7wuEun QgvBk4/afx2CNOaN/sI+zjy6IK6bwNJUDtwbixH5JFkHiCt3GBBRZj+Q6vJwIRAFrYcf h1Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764580393; x=1765185193; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=OwX/olriHOAxoNjW5iw4IZ+oz3oFzlu8t83MnxTRx/U=; b=OgfThU6nCaEcRW0kdYoZieAdN9WDf6g/zwZDnsLRtnOnEaIxAmzqH8JnPB2npUGeDb Sz6bzH5z17n5Fl5JzGG3NTa5E+yKAj7IT85Ffq/9mbRXJJaGmRfuTEHpRlRWkWmcSwxG EDgMO+U7BgV+ol5qfcuBuRGjbmFdsY5TAMhuTgxRNSS8/P7eaopb5HkknA3umB9575QT 0GkojBLC1RaBjEE9lLTSDX8GsXJiwU3hE+ozfmQlRfO9Tiwz3ZzBtoioKUtXp4R9h4Zj LwXK6AHDihJRYahn0vAHTov6RUJKMkaqbDfZdAnqjmqRD7YByvTu3RwkKoXDCRtbBs5t MqYw== X-Forwarded-Encrypted: i=1; AJvYcCVNSOkjjhoBWlrNGAPkKrWBVZfFjfv2FqsMgLaxHsmwnb1Dkzu3cHM9ln2az2Xietk4zESZiTEyTtaajBE=@vger.kernel.org X-Gm-Message-State: AOJu0YwHPZE/0uT3yOkBuhjrE0hI2PsSrLXPWbfWZoAMVQKagTFoCg0h G7Tr5u/DOr7A7MvIXBSI1oN6jm0Rl1u7Sm00cy/vpn6vJLOUi4iDDeNNuTdLgtVBGIU= X-Gm-Gg: ASbGncswADHLlPB/gQKU+P6VH2dApceEw0Up01yJr2lYXlpq7Ou0RbZs6uy2AM5w8aP fCbjcwzSO8m1y6je3+CwleSclsW6tuL/U9k9ejh9VzTWYXM8BNmPJ7p9pIxCHOb4M1n9qISU4bj yz2hcQ2wurzI7xcJAmI1cYuOa6+PallpexK/cPuF+u9Af/4viZJ0tYiQ5f02fkWXCaRLKO0Rnl1 yQpsmOwRWBuc4aV/BBYRS4YOxbkvtsSqiqj/VNNy3Le7pgP1yEDhQLgAjgV+NEGuo/xwKXRTfrv TUjpGuK6Y9tA49xr+8WT34PERiMvJq0bvmOJx3dcFVPqLGUz016P32KrJnlJkc3UE4B3sxCmXwt WvqvUZBNjcm3wfglCJ6B0hZ2RZI6RZuZ8Xz5SfGWxoX/09c/yuPmr+G8aTcA3QVfxfx7kFWj0wb PHoU0H1MqDmTjVG+jxKUVmKoldOtBQ7g== X-Google-Smtp-Source: AGHT+IEITz2BTVLYYbRegTBFRIeCOG40K9cpljUqZ4AWg5n55qXnDrUrMakteG+U0d0IcxvPwua9Vw== X-Received: by 2002:a05:6000:18a9:b0:42b:3e60:18e9 with SMTP id ffacd0b85a97d-42cc1ab8bf8mr37104043f8f.10.1764580393051; Mon, 01 Dec 2025 01:13:13 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:77d1:da59:4ce1:741c]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1caa767dsm25344110f8f.38.2025.12.01.01.13.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Dec 2025 01:13:11 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [PATCH 1/6 v7] sched/fair: Filter false overloaded_group case for EAS Date: Mon, 1 Dec 2025 10:13:03 +0100 Message-ID: <20251201091308.761711-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251201091308.761711-1-vincent.guittot@linaro.org> References: <20251201091308.761711-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With EAS, a group should be set overloaded if at least 1 CPU in the group is overutilized but it can happen that a CPU is fully utilized by tasks because of clamping the compute capacity of the CPU. In such case, the CPU is not overutilized and as a result should not be set overloaded as well. group_overloaded being a higher priority than group_misfit, such group can be selected as the busiest group instead of a group with a mistfit task and prevents load_balance to select the CPU with the misfit task to pull the latter on a fitting CPU. Signed-off-by: Vincent Guittot Tested-by: Pierre Gondois --- kernel/sched/fair.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1855975b8248..b10f04715251 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9987,6 +9987,7 @@ struct sg_lb_stats { unsigned int group_asym_packing; /* Tasks should be moved to preferred CP= U */ unsigned int group_smt_balance; /* Task on busy SMT be moved */ unsigned long group_misfit_task_load; /* A CPU has a task too big for its= capacity */ + unsigned int group_overutilized; /* At least one CPU is overutilized in t= he group */ #ifdef CONFIG_NUMA_BALANCING unsigned int nr_numa_running; unsigned int nr_preferred_running; @@ -10219,6 +10220,13 @@ group_has_capacity(unsigned int imbalance_pct, str= uct sg_lb_stats *sgs) static inline bool group_is_overloaded(unsigned int imbalance_pct, struct sg_lb_stats *sgs) { + /* + * With EAS and uclamp, 1 CPU in the group must be overutilized to + * consider the group overloaded. + */ + if (sched_energy_enabled() && !sgs->group_overutilized) + return false; + if (sgs->sum_nr_running <=3D sgs->group_weight) return false; =20 @@ -10402,14 +10410,12 @@ sched_reduced_capacity(struct rq *rq, struct sche= d_domain *sd) * @group: sched_group whose statistics are to be updated. * @sgs: variable to hold the statistics for this group. * @sg_overloaded: sched_group is overloaded - * @sg_overutilized: sched_group is overutilized */ static inline void update_sg_lb_stats(struct lb_env *env, struct sd_lb_stats *sds, struct sched_group *group, struct sg_lb_stats *sgs, - bool *sg_overloaded, - bool *sg_overutilized) + bool *sg_overloaded) { int i, nr_running, local_group, sd_flags =3D env->sd->flags; bool balancing_at_rd =3D !env->sd->parent; @@ -10431,7 +10437,7 @@ static inline void update_sg_lb_stats(struct lb_env= *env, sgs->sum_nr_running +=3D nr_running; =20 if (cpu_overutilized(i)) - *sg_overutilized =3D 1; + sgs->group_overutilized =3D 1; =20 /* * No need to call idle_cpu() if nr_running is not 0 @@ -11103,13 +11109,15 @@ static inline void update_sd_lb_stats(struct lb_e= nv *env, struct sd_lb_stats *sd update_group_capacity(env->sd, env->dst_cpu); } =20 - update_sg_lb_stats(env, sds, sg, sgs, &sg_overloaded, &sg_overutilized); + update_sg_lb_stats(env, sds, sg, sgs, &sg_overloaded); =20 if (!local_group && update_sd_pick_busiest(env, sds, sg, sgs)) { sds->busiest =3D sg; sds->busiest_stat =3D *sgs; } =20 + sg_overutilized |=3D sgs->group_overutilized; + /* Now, start updating sd_lb_stats */ sds->total_load +=3D sgs->group_load; sds->total_capacity +=3D sgs->group_capacity; --=20 2.43.0 From nobody Mon Dec 1 21:32:53 2025 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49C673002C5 for ; Mon, 1 Dec 2025 09:13:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580398; cv=none; b=g/1Kr2QwDxAsHdILqKwlAAKM/y6exjLGSDysWZWFtMVQPWSbTnNh3s2rNC6UivraiYjjvLbVxhXmj7i0dzs390cmy/ft3cCW/z5JlahDoz9Z/u0dMWryL4UU+FNSVH0waOrLJrgpoTtF+iFEEaqaTTY3s6jTyTmipuywNf7+iWk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580398; c=relaxed/simple; bh=4TjgOClD9MZ3bpoMuNewJ2AXb9TzO5tUug3pj3x8AyM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=J3Xyu7sjBK/7jUjTN9NM0bsnoHeV9Er1hL/NWrWbRp0fWTRqMpylZ9EKUSOlrSN9peow2paR3NAJfolCnp6T9Pvs/JP2ZyOG5doX51CXeVXdcWP+UJ6sgD6261CwDi6Wk9FrrtMD+crM3zcK7cxxFvsZFsiMaYMJwMUVwFhhs5M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=CN9KPChi; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="CN9KPChi" Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-4779cb0a33fso39944175e9.0 for ; Mon, 01 Dec 2025 01:13:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764580394; x=1765185194; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8fl5Pg9Y8yiY67LalwVtDEwtUsWcIOn3WvNZLWDGeNc=; b=CN9KPChiN1kflbaNNFcVeVPa8NfnnGadZtOe0c6e4bn17kXeN2w4EKsS4fOnGCN4YQ hrcklGLtc/i44+pZUMqyDXeobE8mzbkLEmEkW1twCYlpur8U3xK4qVOcB6VI34olzusm ahzuj04ChhGyr07/IFudNhHhH8C/cDjElMtDPOHPUzp9K+5pAHn0XJPdcs9u2vhkW/+R VxpE/scU+ueUUwTb8MQDrTu2yQwmDjSbQdrwdTqcvJgsKzEk3Vbw+oaFcRpCM00X8FWY V6EYlLXy39cOpi/ksgLyFVge2/Zie6/c9kADOH43XfrETZMOo5M3I3/9DluNiNW3ewWw bfEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764580394; x=1765185194; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8fl5Pg9Y8yiY67LalwVtDEwtUsWcIOn3WvNZLWDGeNc=; b=DxML8DTQlmcUAEZwh8j92yD6uc45tTje1Db+ltLDcYvTkXVI/JDwJNTjlaNTZsP1kp BgFNURCXfzbsz7u/QRQ5uuQLDjBc7pQZRmhAMYLURlJbuK3sJf4cwVbYYaJ1APQJ1125 sj6JX9+2obaUpMXpMD/FNnHQNWcXccJbPSC4YZUbs0VtepB+QN9Y3+kAI1kROs9y0l1P 7T3yoTvDF/8rzzq+iD/WrT2CXGSzBY+k+mWMplDsESulMTUp9Z80c40X8EeDcd+13Bx0 suAa7a7AKND584/sf9wBx+88XQeNyQeKqstheaYS9G4ZnU2R++IgnytvXHMVUwSOlbfv 8kGQ== X-Forwarded-Encrypted: i=1; AJvYcCWWJZr19q3ITz+gzG+7trJpyYUcOmw+AbGZazWSLBuKVPF3OXMudLZMXY6dwQO++opWSJt5ERnPQ4p01S4=@vger.kernel.org X-Gm-Message-State: AOJu0Yz6hOGnG+iPE+Cka4icIsy/Fi8xtSY1fkCvPdl7V7CNg3Mm0eCm 5601fsDFWpyA4yBPOoao5Xzq92LqFlONCdxOB91L2EA/KjyVtAPEX8v2giURDGcsvqQ= X-Gm-Gg: ASbGncuVoPgjXMpi9hZ7xX+Vk7jCbCEkiZLjoUR9wbjX0nvi7EoR1VFSB18WPTKsN1y 7QzRCrrgNJKGzNUjl8S/4VQkifIKQJAtK86XInIwu14n8GLt15EzN1xkSZ5uNQPmLxWDpNiBvux cEi6PzwxMfwwZowin0BCghF19rJpjU25LBCfFPhSIUqJpFstxlrHaL9gAcARezXxqvgMo+IHZ+B lAnn6soW2qCc5KpHBWuTkvQfUgrQ4k2PSFCES7bLcMoHsQW/7bfRk6uvjYZmuUgeh9hz7VBRRdw BMfFZsRKGbQKYajSTuQzOx7mRioIU/h0GRJQpijdMBv9/JFd0pnZiV61FDJhXjLDztRMAcIoMEM fxZoTN0kIIFEup3cNyvT24mkxxqRJuBCf096EQkOyAIn0Q0qpcSDbLqOcisKoM8PbSOavAuFwDK J6bJ4DrhFLWyNwHkYNAvg= X-Google-Smtp-Source: AGHT+IHo7/kZSUWNFwyzin4Mrz7uGa1uyn/rEhoKmv9Y3WFoT0OloGB+9YngizrzX1dTqMMe/edb4A== X-Received: by 2002:a05:6000:2906:b0:42b:41d3:daf1 with SMTP id ffacd0b85a97d-42cc1d1965amr42404387f8f.38.1764580394544; Mon, 01 Dec 2025 01:13:14 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:77d1:da59:4ce1:741c]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1caa767dsm25344110f8f.38.2025.12.01.01.13.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Dec 2025 01:13:13 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [PATCH 2/6 v7] sched/fair: Update overutilized detection Date: Mon, 1 Dec 2025 10:13:04 +0100 Message-ID: <20251201091308.761711-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251201091308.761711-1-vincent.guittot@linaro.org> References: <20251201091308.761711-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Checking uclamp_min is useless and counterproductive for overutilized state as misfit can now happen without being in overutilized state Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index b10f04715251..f430ec890b72 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6785,16 +6785,15 @@ static inline void hrtick_update(struct rq *rq) =20 static inline bool cpu_overutilized(int cpu) { - unsigned long rq_util_min, rq_util_max; + unsigned long rq_util_max; =20 if (!sched_energy_enabled()) return false; =20 - rq_util_min =3D uclamp_rq_get(cpu_rq(cpu), UCLAMP_MIN); rq_util_max =3D uclamp_rq_get(cpu_rq(cpu), UCLAMP_MAX); =20 /* Return true only if the utilization doesn't fit CPU's capacity */ - return !util_fits_cpu(cpu_util_cfs(cpu), rq_util_min, rq_util_max, cpu); + return !util_fits_cpu(cpu_util_cfs(cpu), 0, rq_util_max, cpu); } =20 /* --=20 2.43.0 From nobody Mon Dec 1 21:32:53 2025 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4E262FFF9D for ; Mon, 1 Dec 2025 09:13:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580399; cv=none; b=Vln1vMixEG97dXQNKVkxzYwr0ta1f4bEQXgbM8U0qdE8CAkJQescqGU/Xo9vFenU5V5DAPQ8/9fTmXmbdX78g8KAXiplRPiFBiMK/cRg93dRg8q/VMW9KaGEYPhrEHIQgqIv3EKKY01toYCgxurDHrjIecEOBzYNHtVSSHtDf5Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580399; c=relaxed/simple; bh=FTHun3PMOq+CklxsBPL6XoRjxH2p1DjEDgtWxflTW+A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LMxr+/OeFBnpaw7R7iFDcR0PKjGlcR/Yz7/jNmerIz+kR1qRf9Ox02oVPpl864Z9ioJczBxhEhDrRt5v+eckTfLslePBZpm8C+OAGqmY+aRC01ADzIM2xB5eTswcN4OPUqYXoFdiROT0hIDPueoSi0H/Ptm+SpG3Wad4Jw4oJfY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=Ppreef/i; arc=none smtp.client-ip=209.85.221.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="Ppreef/i" Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-42e2e52cc04so373531f8f.0 for ; Mon, 01 Dec 2025 01:13:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764580396; x=1765185196; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=95eiwIqVqnWVfXykfzvRd1ip/6pVXilQ5gdb8x62/lM=; b=Ppreef/ilPsI2Kk34engXQoV6y37W9iatFh1X+dDIwwmtf0iUNTzfjXfOuc7X+B4F+ 67J2Rp7sckmVQCp1YEcGypFYpKrGze1mZyWDO00gEEGOW1PbXgSdsAmckhaxcNZAzsau X4cRHB6sh3aVzR6XreT6oj+3pEJcCJyQ3YMtBzJBJAjZd/+JjYLYbw8iKcBD3HTpdwNI jXXBCnEnIhwjSrL2Rf7QvQXzlyst0cZ4gM2c3nbKgSyjq/tluBRWlCZNVr5n7mYMXRt2 mL/mMCyr9yBvOsDiig4jPwM9DiVweUNVcU8Vu3jU+AjpvWxBIjY2ili6UztGaWkvzcyX 0PuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764580396; x=1765185196; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=95eiwIqVqnWVfXykfzvRd1ip/6pVXilQ5gdb8x62/lM=; b=Q5XtKSVfxCAIyqysvAoci23mgQlE7S6zhqZOWoNVaybEWp9MKu7Jn9/s5P7Rc1cqwG FnpVveqJ/JcKVekKikXa2ZjaX/8mcEZ1X5VFTbgDQP4fjQ9xpwN9hv3DPKz/kxKfIhM0 kaFoeY+WNUCf4ONApusH9TYGZxENVRmSlSmYNIj0dH0mln/9DTK5F/U6w6YzmG7V94oT klUVF4Zq3xwuRMcOymMa+wgj4XtLiuuGxsh7bnWhiQ0rmDZ7zWHRUYxluRirMPk0zg3i j7qA5rWpBSN/+lPdsUhC7TKnZXClWqRbWG3vFLWFu0cK9KzPbPJ3iH3keSsaBrfCnas1 XG7w== X-Forwarded-Encrypted: i=1; AJvYcCX2AdbN20qDsIq+n7FxXNHlAEf1QerEXIkJZhZLEwoS1F3VMO+i6Je4j/nNGm7vhSDpaJcU+z4I/ATPq4c=@vger.kernel.org X-Gm-Message-State: AOJu0YwvfjWb0UMwg0GhVY3xmm65WF4KmhholdJnOLSpasBg/C+q9Z0W lmCrKkpcDFyvH1H/ZkegGIfQtb6ki0a5tEunlqxa+7qdCMMTwKrJg96JnDIuZczihjQ= X-Gm-Gg: ASbGncvLQH2e1rU/8TnKqhgDbhZYfwCX3T8j/BEYImuMYUI55PFxNn8ppMWttHWtBnD ac8ktYxmCR04O3/kJMt46QOqGjKzS4CYM7gmRg1oBDbg5eUGAhD6DhXBDRwF1niqBO7kpvgP/Th RIx3gb86oUQA3eDB7xWAo70rUv4YlbKKoIfoL8bXdxNXI0X3NhFzny5H5Q5SjpkA38TtNzsE8BN bBvsIGC3SvtdOMvp3DfX4BnyijJjaPnP1lPvKy/RWKxz5L/Vx4BTzMHtUgNvP+ryyD7q7lgMIEq Q+JImkhP1F0ZEIXiXu6FwTRB00Yf6wqEF2QF6r6+5aL6rdfgcm4ukO3YZ2jfxNQQI47bcVRwKwE WMILJEvdaW6dG+lvnLiYDUuNTNVMFvu7WoVyDzKv0D1GXZQ7KOSfPESU97uHHbGD01PAoe5+Z/q bcvClm1NPqaE2Dm51n1XcdXtGX64Ip4Q== X-Google-Smtp-Source: AGHT+IEXL0FW4FA4PxEHTN9REcRL69V6HM5QKNK233vETcLtWanPetqEVi1wgeCdVoKDJu7B6akfxQ== X-Received: by 2002:a5d:588c:0:b0:429:f14a:9807 with SMTP id ffacd0b85a97d-42e0f34fadbmr26712550f8f.40.1764580395918; Mon, 01 Dec 2025 01:13:15 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:77d1:da59:4ce1:741c]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1caa767dsm25344110f8f.38.2025.12.01.01.13.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Dec 2025 01:13:14 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [PATCH 3/6 v7] sched/fair: Prepare select_task_rq_fair() to be called for new cases Date: Mon, 1 Dec 2025 10:13:05 +0100 Message-ID: <20251201091308.761711-4-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251201091308.761711-1-vincent.guittot@linaro.org> References: <20251201091308.761711-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Update select_task_rq_fair() to be called out of the 3 current cases which are : - wake up - exec - fork We wants to select a rq in some new cases like pushing a runnable task on a better CPU than the local one. In such case, it's not a wakeup , nor an exec nor a fork. We make sure to not distrub these cases but still go through EAS and fast-path. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f430ec890b72..80c4131fb35b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8518,6 +8518,7 @@ static int select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags) { int sync =3D (wake_flags & WF_SYNC) && !(current->flags & PF_EXITING); + int want_sibling =3D !(wake_flags & (WF_EXEC | WF_FORK)); struct sched_domain *tmp, *sd =3D NULL; int cpu =3D smp_processor_id(); int new_cpu =3D prev_cpu; @@ -8535,16 +8536,21 @@ select_task_rq_fair(struct task_struct *p, int prev= _cpu, int wake_flags) if ((wake_flags & WF_CURRENT_CPU) && cpumask_test_cpu(cpu, p->cpus_ptr)) return cpu; + } =20 - if (!is_rd_overutilized(this_rq()->rd)) { - new_cpu =3D find_energy_efficient_cpu(p, prev_cpu); - if (new_cpu >=3D 0) - return new_cpu; - new_cpu =3D prev_cpu; - } + /* + * We don't want EAS to be called for exec or fork but it should be + * called for any other case such as wake up or push callback. + */ + if (!is_rd_overutilized(this_rq()->rd) && want_sibling) { + new_cpu =3D find_energy_efficient_cpu(p, prev_cpu); + if (new_cpu >=3D 0) + return new_cpu; + new_cpu =3D prev_cpu; + } =20 + if (wake_flags & WF_TTWU) want_affine =3D !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr); - } =20 rcu_read_lock(); for_each_domain(cpu, tmp) { @@ -8575,7 +8581,7 @@ select_task_rq_fair(struct task_struct *p, int prev_c= pu, int wake_flags) if (unlikely(sd)) { /* Slow path */ new_cpu =3D sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag); - } else if (wake_flags & WF_TTWU) { /* XXX always ? */ + } else if (want_sibling) { /* Fast path */ new_cpu =3D select_idle_sibling(p, prev_cpu, new_cpu); } --=20 2.43.0 From nobody Mon Dec 1 21:32:53 2025 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B823A301035 for ; Mon, 1 Dec 2025 09:13:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580401; cv=none; b=FuXCBO4ZdeL+0bkRqL66t5itArR7dtarr6X7Ql6GN9J0JlIla8lkhGW44gbG9cKNh//IT8wn2ldmued58PlFEoF/C88axvE88rWH8EEBD+mBc2QO/+sKXb4aAnYqTEa/OoEKHS0Vf5odFBixxjtfE4f5U5cFw1+kDUVpEjEvOYM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580401; c=relaxed/simple; bh=TWMuj6O2T1H94T5izSq1oEyI0WoW18DKO4WAteFRkKA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GoHbYGLnryzoMK4FGPaTmTyBkOXT6bM7dP8/qFjKqo4bgff8toq+K+jYSPFxga4oqCGneTtGZzJnxMhnqFfyrEICO3C732UMlj/KTXlPmXgAytZyU445OdbTM9AxqNS04Uiwiy5TCfEh278ZCqhZPvR0hLmYoXL/OgKT62/RKqk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=O1fcz+Nb; arc=none smtp.client-ip=209.85.221.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="O1fcz+Nb" Received: by mail-wr1-f54.google.com with SMTP id ffacd0b85a97d-42e2d44c727so584592f8f.0 for ; Mon, 01 Dec 2025 01:13:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764580398; x=1765185198; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7eIKMzShNanebsz39IRyp+q+2Oqh30CR1zVjIwRlB9s=; b=O1fcz+NbBr4jFxUx6eKHZqow1GcipGbnJ1XN7ohm737XnGXyuUNjypiuO8U+jgfhVm hRVNyYtaCqzSllrsYBnbqcsa+ohp+/rPM0wRWI72ynH30WsMDdKSjYa1v5Dwqn4H2U/p raSNc/4fRm/p+qPM1iuVWld5iqjw/yMqUEf/Ypf+6ZOL6GbBwQNARP1VPqnW9gr0Egu5 VIr4F0OBEXYDNMC2IVxZ/KuWlkp5qAyLRs1YZgSbyh7Jru1etobn3xYoX3zzmD0btih1 m0cwh1pSyRziKDlkr59WYAvhpUizZeM2/taLt4k9FvZ4JAPy+KLsF7ZDPTWXfXzAqI4O CSPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764580398; x=1765185198; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=7eIKMzShNanebsz39IRyp+q+2Oqh30CR1zVjIwRlB9s=; b=qmETZrczpg4QqK80/aoWib5IXcZMLKLxQkkxDGfHDkVxorS6E+MlWN+hecO5wnN34n t4Q+JcW6/3ZMRwgpoE8OS3jIasQQsTtLtKR2cDZIqSJtv5LoZwbQgQYQH1aOXqKhPzFF 2p7XbaMjy8r8LhM3ngkuCTZhW6zYJw44+Qh1WO1tXSwk54PsujybfLK/KXsudtULAzbS oMSsH8BWApoD8GsyMMPe2Qg5vUkgdHR5lD9HukOcnajNbrp5VzgqSXRI11Z0TUb1Due7 hEXEpOQWc6q9xJRuiFwkrh0W6bJltjLtpQZCIHyDUB6PtyPhWkvmZasMJAp/9o6ecxCY 5TgA== X-Forwarded-Encrypted: i=1; AJvYcCUEVs+JxqNuNew0trcu6Deve7zD3nBvVGdpHA0dQT2pt3okPbBh8jX+3Gqijho4hZb9BxIqCV/tCGmpCXY=@vger.kernel.org X-Gm-Message-State: AOJu0YyfSgH/MCIFmMav2qEn3ZnvG5psQaF8VeReQd8cCr0wEH4nm/tO mFY+mV0CPI67FxwjYYrBfPcB07qxgWhcJ5vJiedRFkeBPPbS7Q0JI/CPR0f2XFCub8A= X-Gm-Gg: ASbGncvc9MJ+OY0vzcL3Bc5NslhQL0WEwTy75Q3T3PEE55jFEwmqdbpRmBkNFzHPmK9 tsmwrEpsTaWJdsNmlONJ/dY8eijA6aQVVo+iOiKGF1DlrYT4qjK3t6NMrvMPR5PWgD8InBQaz+d X+tjh4X6KdcjUug46bdiHizHfXj3nNcB5fGmCuzzInRH9LhnHzm66SQ0wc+Ttu1nxJ6V0/B9L7U k/Bo+OE+w1TSJkT8zzZoQcVMMQ91LUT1A1kH6LATj8Qq7yWsX9V86GmlD9ldOoZoPMY840qLfvW 8mdu0HZ5GzggWXhumby+U+jAxCMt13IGQPnLsuD3otMaqLtd2GcVtANUZzMK8FglQogCLVdJWAT /E5nWaje20uyBRI+YQGpnqzcMH3OnQ6KMeZ1MQNjRniywcm81EYYQQ2aEdpUp3+HOO2oyGpilN8 iCZ5W7KwSyCO4wkXGwaxrJXVEYnTOv5g== X-Google-Smtp-Source: AGHT+IEmQNcSzsFsTaxk/7FjXYT4vdoumBVXxltVbe78suF6ZSkZb3LHRiWFkRPxdaa/J2xklFKRsw== X-Received: by 2002:a05:6000:178b:b0:42b:487c:d7cb with SMTP id ffacd0b85a97d-42cc1cf3860mr37808875f8f.34.1764580397731; Mon, 01 Dec 2025 01:13:17 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:77d1:da59:4ce1:741c]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1caa767dsm25344110f8f.38.2025.12.01.01.13.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Dec 2025 01:13:16 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [PATCH 4/6 v7] sched/fair: Add push task mechanism for fair Date: Mon, 1 Dec 2025 10:13:06 +0100 Message-ID: <20251201091308.761711-5-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251201091308.761711-1-vincent.guittot@linaro.org> References: <20251201091308.761711-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" EAS is based on wakeup events to efficiently place tasks on the system, but there are cases where a task doesn't have wakeup events anymore or at a far too low pace. For such situation, we can take advantage of the task being put back in the enqueued list to check if it should be pushed on another CPU. When the task is alone on the CPU, it's never put back in the enqueued list; In this special case, we use the tick to run the check. Add a push task mecanism that enables fair scheduler to push runnable tasks. EAS will be one user but other feature like filling idle CPUs can also take advantage of it. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 211 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 4 + 2 files changed, 213 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 80c4131fb35b..4e94a4cb8caa 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6989,6 +6989,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) hrtick_update(rq); } =20 +static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p= ); /* * Basically dequeue_task_fair(), except it can deal with dequeue_entity() * failing half-way through and resume the dequeue later. @@ -7017,6 +7018,8 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags) h_nr_idle =3D task_has_idle_policy(p); if (task_sleep || task_delayed || !se->sched_delayed) h_nr_runnable =3D 1; + + fair_remove_pushable_task(rq, p); } =20 for_each_sched_entity(se) { @@ -8504,6 +8507,187 @@ static int find_energy_efficient_cpu(struct task_st= ruct *p, int prev_cpu) return target; } =20 +DEFINE_STATIC_KEY_FALSE(sched_push_task); + +static inline bool sched_push_task_enabled(void) +{ + return static_branch_unlikely(&sched_push_task); +} + +static bool fair_push_task(struct rq *rq, struct task_struct *p) +{ + return false; +} + +static inline int has_pushable_tasks(struct rq *rq) +{ + return !plist_head_empty(&rq->cfs.pushable_tasks); +} + +static struct task_struct *pick_next_pushable_fair_task(struct rq *rq) +{ + struct task_struct *p; + + if (!has_pushable_tasks(rq)) + return NULL; + + p =3D plist_first_entry(&rq->cfs.pushable_tasks, + struct task_struct, pushable_tasks); + + WARN_ON_ONCE(rq->cpu !=3D task_cpu(p)); + WARN_ON_ONCE(task_current(rq, p)); + WARN_ON_ONCE(p->nr_cpus_allowed <=3D 1); + WARN_ON_ONCE(!task_on_rq_queued(p)); + + /* + * Remove task from the pushable list as we try only once after that + * the task has been put back in enqueued list. + */ + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); + + return p; +} + +static int +select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags); + +/* + * See if the non running fair tasks on this rq can be sent on other CPUs + * that fits better with their profile. + */ +static bool push_fair_task(struct rq *rq) +{ + struct task_struct *next_task; + int prev_cpu, new_cpu; + struct rq *new_rq; + + next_task =3D pick_next_pushable_fair_task(rq); + if (!next_task) + return false; + + if (is_migration_disabled(next_task)) + return true; + + /* We might release rq lock */ + get_task_struct(next_task); + + prev_cpu =3D rq->cpu; + + new_cpu =3D select_task_rq_fair(next_task, prev_cpu, 0); + + if (new_cpu =3D=3D prev_cpu) + goto out; + + new_rq =3D cpu_rq(new_cpu); + + if (double_lock_balance(rq, new_rq)) { + /* The task has already migrated in between */ + if (task_cpu(next_task) !=3D rq->cpu) { + double_unlock_balance(rq, new_rq); + goto out; + } + + deactivate_task(rq, next_task, 0); + set_task_cpu(next_task, new_cpu); + activate_task(new_rq, next_task, 0); + + resched_curr(new_rq); + + double_unlock_balance(rq, new_rq); + } + +out: + put_task_struct(next_task); + + return true; +} + +static void push_fair_tasks(struct rq *rq) +{ + /* push_fair_task() will return true if it moved a fair task */ + while (push_fair_task(rq)) + ; +} + +static DEFINE_PER_CPU(struct balance_callback, fair_push_head); + +static inline void fair_queue_pushable_tasks(struct rq *rq) +{ + if (!sched_push_task_enabled() || !has_pushable_tasks(rq)) + return; + + queue_balance_callback(rq, &per_cpu(fair_push_head, rq->cpu), push_fair_t= asks); +} + +static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p) +{ + if (sched_push_task_enabled()) + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); +} + +static void fair_add_pushable_task(struct rq *rq, struct task_struct *p) +{ + if (sched_push_task_enabled() && fair_push_task(rq, p)) { + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); + plist_node_init(&p->pushable_tasks, p->prio); + plist_add(&p->pushable_tasks, &rq->cfs.pushable_tasks); + } +} + +static int active_load_balance_cpu_stop(void *data); + +/* + * See if the alone task running on the CPU should migrate on a better than + * the local one. + */ +static inline bool check_pushable_task(struct task_struct *p, struct rq *r= q) +{ + int new_cpu, cpu =3D cpu_of(rq); + + if (!sched_push_task_enabled()) + return false; + + if (WARN_ON(!p)) + return false; + + if (WARN_ON(!task_current(rq, p))) + return false; + + if (is_migration_disabled(p)) + return false; + + /* If there are several task, wait for being put back */ + if (rq->nr_running > 1) + return false; + + if (!fair_push_task(rq, p)) + return false; + + new_cpu =3D select_task_rq_fair(p, cpu, 0); + + if (new_cpu =3D=3D cpu) + return false; + + /* + * ->active_balance synchronizes accesses to + * ->active_balance_work. Once set, it's cleared + * only after active load balance is finished. + */ + if (!rq->active_balance) { + rq->active_balance =3D 1; + rq->push_cpu =3D new_cpu; + } else + return false; + + raw_spin_rq_unlock(rq); + stop_one_cpu_nowait(cpu, + active_load_balance_cpu_stop, rq, + &rq->active_balance_work); + raw_spin_rq_lock(rq); + + return true; +} + /* * select_task_rq_fair: Select target runqueue for the waking task in doma= ins * that have the relevant SD flag set. In practice, this is SD_BALANCE_WAK= E, @@ -8973,6 +9157,12 @@ pick_next_task_fair(struct rq *rq, struct task_struc= t *prev, struct rq_flags *rf put_prev_entity(cfs_rq, pse); set_next_entity(cfs_rq, se); =20 + /* + * The previous task might be eligible for being pushed on + * another cpu if it is still active. + */ + fair_add_pushable_task(rq, prev); + __set_next_task_fair(rq, p, true); } =20 @@ -9036,6 +9226,13 @@ static void put_prev_task_fair(struct rq *rq, struct= task_struct *prev, struct t cfs_rq =3D cfs_rq_of(se); put_prev_entity(cfs_rq, se); } + + /* + * The previous task might be eligible for being pushed on another cpu + * if it is still active. + */ + fair_add_pushable_task(rq, prev); + } =20 /* @@ -13390,8 +13587,10 @@ static void task_tick_fair(struct rq *rq, struct t= ask_struct *curr, int queued) if (static_branch_unlikely(&sched_numa_balancing)) task_tick_numa(rq, curr); =20 - update_misfit_status(curr, rq); - check_update_overutilized_status(task_rq(curr)); + if (!check_pushable_task(curr, rq)) { + update_misfit_status(curr, rq); + check_update_overutilized_status(task_rq(curr)); + } =20 task_tick_core(rq, curr); } @@ -13552,6 +13751,8 @@ static void __set_next_task_fair(struct rq *rq, str= uct task_struct *p, bool firs { struct sched_entity *se =3D &p->se; =20 + fair_remove_pushable_task(rq, p); + if (task_on_rq_queued(p)) { /* * Move the next running task to the front of the list, so our @@ -13567,6 +13768,11 @@ static void __set_next_task_fair(struct rq *rq, st= ruct task_struct *p, bool firs if (hrtick_enabled_fair(rq)) hrtick_start_fair(rq, p); =20 + /* + * Try to push prev task before checking misfit for next task as + * the migration of prev can make next fitting the CPU + */ + fair_queue_pushable_tasks(rq); update_misfit_status(p, rq); sched_fair_update_stop_tick(rq, p); } @@ -13596,6 +13802,7 @@ void init_cfs_rq(struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline =3D RB_ROOT_CACHED; cfs_rq->zero_vruntime =3D (u64)(-(1LL << 20)); + plist_head_init(&cfs_rq->pushable_tasks); raw_spin_lock_init(&cfs_rq->removed.lock); } =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b419a4d98461..697bd654298a 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -711,6 +711,8 @@ struct cfs_rq { unsigned long runnable_avg; } removed; =20 + struct plist_head pushable_tasks; + #ifdef CONFIG_FAIR_GROUP_SCHED u64 last_update_tg_load_avg; unsigned long tg_load_avg_contrib; @@ -3620,6 +3622,8 @@ static inline bool sched_energy_enabled(void) { retur= n false; } =20 #endif /* !(CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL) */ =20 +DECLARE_STATIC_KEY_FALSE(sched_push_task); + #ifdef CONFIG_MEMBARRIER =20 /* --=20 2.43.0 From nobody Mon Dec 1 21:32:53 2025 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B2FA30147A for ; Mon, 1 Dec 2025 09:13:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580402; cv=none; b=KeSPVJoZDHDUPczwezHCr3VuTN926Oo+0Zw5DflPP3OeEo4PnEJoKxs1U80elf9JvhWB/PFYWyYP6TftAS2dvjPBGVDO6/Q0WslmLtQcq5GdFXhO4QjArHJ2gjCIwZ/vW8ks1k456Y8d1ZHXYY7syxH6YOYQ1iqy9DMt1bKiQgA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580402; c=relaxed/simple; bh=CjNnaBgGisRqeIBtu4QTxT/7hU/7LjOKwy33GHb5I/E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N9fq0sal2v4qU7VlK81T6jrUL2kxfdzwJOJ9LiPG2wqcymZHMyeJyM14sovYFgTZYXsVie3vKh/Elj5mQViRPqUYzVlpLLu3tdbT4YUkT6BNPlARd5E/Xbpt/0i4u+Tr7eJaHPmySQvXe8TAL1rE9Qy/+wFK9UJ+EIFNgvHxw+Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=FrH/sAv1; arc=none smtp.client-ip=209.85.221.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="FrH/sAv1" Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-42e2d44c727so584637f8f.0 for ; Mon, 01 Dec 2025 01:13:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764580399; x=1765185199; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R3BOBSKZ2J6e7XAb32c+ir+ZqfdYV3db2IojbkKh5Tk=; b=FrH/sAv1pLxlTmnvcU+Qay3I3QiXz68nev6rTHS0Mu4Vv5swdkbW+am33oKXmCDfcW qsS3V7hiP8W+Krw0vPPs7s1MWthN947dNs51laym4g/wljwk9NXW5Cd3ai+lUWTC6cJb XC6H/bZzx7gBIOk12Nc96q5aJCm8rp5uWrv0Cr+Elmq3bgT7G8wAMir538h7cNujacI3 k5HOYkCd89UImdnnD6RvqJlbXqusVGEM7+LBL64KXjebPF4aw5/EJF3hbOca9Onq+MCF /AMzUN7fvQwzpkT4RKtowFD9+lpoamV/5J+VrsiO3tvfVsAiplMzGy2psX7tcmgpLNEE 5G7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764580399; x=1765185199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=R3BOBSKZ2J6e7XAb32c+ir+ZqfdYV3db2IojbkKh5Tk=; b=ZV3UJ1RmEFdHAmBWMc8sx5obbffz/yojvs2GhQ82dbNrMrbkHr7cQGWNkZWjJnYAZc lX7M8YrZx4CsPd4HMkqWlucLrdgaQq66uk7+ktA/lHzpJjzueBFigXtKN1dstHpPT/QV KhT7R5twiiFLfCxNTvmdozX5Tt/yqO9wKKRH0WYZru1/0Mp4pm92kQFaL/fi/Xd3iZ0U bX+Gr98ncrdj+OWx3uE1fqwh26LGhxQ2pi7eCJ+oiYE7r0I8cYiw8nAA5c1aiAEkeqaN cU+tV5XgHS8Cr3L8cL5DQTNIk8awW9rY5tDCn81zmZDBML3JKPXyIot/8+PDuWArnMcg 8Olg== X-Forwarded-Encrypted: i=1; AJvYcCX8GPZtmo+5FFZQNS990STcL4HI03yZmowTUyTLy7P/KIP3en9o1+QtIPmRHeY5aH8K8/PS0D4oOJcBFoY=@vger.kernel.org X-Gm-Message-State: AOJu0YyO/ui1gmeLrA8V9/H3+RHZYQf2j2oR1hWMmkM6pGggeMex0DU8 JLi7Ub5WTfrZ3JaMHAeRskvj0He+5eLjNO6v4e5qNJFx2JBylKQW33saGM6fswAYPaU= X-Gm-Gg: ASbGnctnOVlkxd5VrCpT1szgniyLq/Z3rzFzpB4F4ZOTE+75ipyVITx2wclxu72JZCM 61vbon7TgZYYagqG5/x/94Q9LyfP/VpaURkB2mmE5uoAOAtav0FFpxUTL99bg6kuoBmp32csFsd CgowdYDOtVG5Y9mft2OxiSxSGnRvgdNsAF4O/8RC8JRZMVh2NMZ5SwWZS1L35vcNP25FCam6Eqe yeCHjFFZmjo7Y2GClm8z16ov9pxkS28B+yf9PXq+ui0nTcS8x/USKOQrkEsJ8ghMCFBbfI6nqNA ek/GYBzNtdHzwPwoKcyQpBehPpWdS7PuELQs/cgvQuQXCIooeU5ytwdrOEv1RVi8n74c1H2F6Wy 8eWVaPXNkHJhc1gO55DE2NG4plvScu0uBwwhLMiO0CguwkrSRz5j/1FenzCfj7Ed43lsWHtJ6Ub noJOVMd7o7R7BKLKqCs5o= X-Google-Smtp-Source: AGHT+IH9fwMbJkaTV4rEU/fI7QYBA/hmyqNQ2s5J6VC3QXwjDINBIstq7NwJbJjJi/Xztvw5A9qcFA== X-Received: by 2002:a5d:5c89:0:b0:42b:5448:7b11 with SMTP id ffacd0b85a97d-42cc1cf3abdmr41231690f8f.33.1764580398584; Mon, 01 Dec 2025 01:13:18 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:77d1:da59:4ce1:741c]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1caa767dsm25344110f8f.38.2025.12.01.01.13.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Dec 2025 01:13:18 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [RFC PATCH 5/6 v7] sched/fair: Enable idle core tracking for !SMT Date: Mon, 1 Dec 2025 10:13:07 +0100 Message-ID: <20251201091308.761711-6-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251201091308.761711-1-vincent.guittot@linaro.org> References: <20251201091308.761711-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable has_idle_cores at llc level feature for !SMT system for which CPU equals core. We don't enable has_idle_core feature of select_idle_cpu to be conservative and don't parse all CPUs of LLC. At now, has_idle_cores can be cleared even if a CPU is idle because of SIS_UTIL but it looks reasonnable as the probablity to get an idle CPU is low anyway. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 29 +++++++---------------------- kernel/sched/sched.h | 42 +++++++++++++++++++++++++++++------------- 2 files changed, 36 insertions(+), 35 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 4e94a4cb8caa..9af8d0a61856 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7500,19 +7500,6 @@ static inline int __select_idle_cpu(int cpu, struct = task_struct *p) return -1; } =20 -#ifdef CONFIG_SCHED_SMT -DEFINE_STATIC_KEY_FALSE(sched_smt_present); -EXPORT_SYMBOL_GPL(sched_smt_present); - -static inline void set_idle_cores(int cpu, int val) -{ - struct sched_domain_shared *sds; - - sds =3D rcu_dereference(per_cpu(sd_llc_shared, cpu)); - if (sds) - WRITE_ONCE(sds->has_idle_cores, val); -} - static inline bool test_idle_cores(int cpu) { struct sched_domain_shared *sds; @@ -7524,6 +7511,10 @@ static inline bool test_idle_cores(int cpu) return false; } =20 +#ifdef CONFIG_SCHED_SMT +DEFINE_STATIC_KEY_FALSE(sched_smt_present); +EXPORT_SYMBOL_GPL(sched_smt_present); + /* * Scans the local SMT mask to see if the entire core is idle, and records= this * information in sd_llc_shared->has_idle_cores. @@ -7611,15 +7602,6 @@ static int select_idle_smt(struct task_struct *p, st= ruct sched_domain *sd, int t =20 #else /* !CONFIG_SCHED_SMT: */ =20 -static inline void set_idle_cores(int cpu, int val) -{ -} - -static inline bool test_idle_cores(int cpu) -{ - return false; -} - static inline int select_idle_core(struct task_struct *p, int core, struct= cpumask *cpus, int *idle_cpu) { return __select_idle_cpu(core, p); @@ -7885,6 +7867,9 @@ static int select_idle_sibling(struct task_struct *p,= int prev, int target) if ((unsigned)i < nr_cpumask_bits) return i; =20 + if (!sched_smt_active()) + set_idle_cores(target, 0); + /* * For cluster machines which have lower sharing cache like L2 or * LLC Tag, we tend to find an idle CPU in the target's cluster diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 697bd654298a..b9e228333d5e 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1591,19 +1591,6 @@ do { \ flags =3D _raw_spin_rq_lock_irqsave(rq); \ } while (0) =20 -#ifdef CONFIG_SCHED_SMT -extern void __update_idle_core(struct rq *rq); - -static inline void update_idle_core(struct rq *rq) -{ - if (static_branch_unlikely(&sched_smt_present)) - __update_idle_core(rq); -} - -#else /* !CONFIG_SCHED_SMT: */ -static inline void update_idle_core(struct rq *rq) { } -#endif /* !CONFIG_SCHED_SMT */ - #ifdef CONFIG_FAIR_GROUP_SCHED =20 static inline struct task_struct *task_of(struct sched_entity *se) @@ -2091,6 +2078,35 @@ static __always_inline bool sched_asym_cpucap_active= (void) return static_branch_unlikely(&sched_asym_cpucapacity); } =20 +static inline void set_idle_cores(int cpu, int val) +{ + struct sched_domain_shared *sds; + + sds =3D rcu_dereference(per_cpu(sd_llc_shared, cpu)); + if (sds) + WRITE_ONCE(sds->has_idle_cores, val); +} + +#ifdef CONFIG_SCHED_SMT +extern void __update_idle_core(struct rq *rq); + +static inline void update_idle_core(struct rq *rq) +{ + if (static_branch_unlikely(&sched_smt_present)) + __update_idle_core(rq); + else + set_idle_cores(cpu_of(rq), 1); + +} + +#else /* !CONFIG_SCHED_SMT: */ +static inline void update_idle_core(struct rq *rq) +{ + set_idle_cores(cpu_of(rq), 1); +} +#endif /* !CONFIG_SCHED_SMT */ + + struct sched_group_capacity { atomic_t ref; /* --=20 2.43.0 From nobody Mon Dec 1 21:32:53 2025 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C79173016E1 for ; Mon, 1 Dec 2025 09:13:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580403; cv=none; b=HmYK/sdXMYa6kA+w9zSJYw4QzhdpCxO4fWbrRvgraGxAGloUzRTf1T1AD9FATjH1EMKvWElw/qT7v0dnDqJIrFo1EuboWRh4eRHdhdgtdWe5lhk1lcglBvjh/k2HCsgQh+TY4UXJhPENdqZb1PTzuR9Dxg/tdNekFXZGP1pdUmo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764580403; c=relaxed/simple; bh=qiH6JApkt155+leNQvk4txPd9WgvnIHl5mspK/fIUvA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YCfmrSUoh1/F43Zpu/bXxLhTOq0jvsOeFQa+cot3XHpi0tIjxT+cIrgY3vLdCUMJBkHeTLd8BZ0KQy8ZLYxXQkGfv5OjUv4OyW1nezf2eRXPZBK2k/ukHTqrtS01tuSJV6Jo8BHyHR3yF3NpSols0cQ9GG3kmVJF0MJO2Ei2BGc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=daetojZz; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="daetojZz" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-42b38693c4dso1358998f8f.3 for ; Mon, 01 Dec 2025 01:13:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764580400; x=1765185200; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HuNRomfv0vCIa2D0+6on2ZVGZKQRa6gA3SVt7VHzktk=; b=daetojZzNyPIMoYKGGQB0rWIaAodZuaTgcTUHNiwESwjVymHhns8kFGkyymsz290qV i3tdGGPGQqAkfxtlVEaqVDAwCpHOlRlirWhRtrL1FRpkdGBBNF50L4ANlBiggP6sYarn sctzfEB2+JUu9b+jzwBhC1aUiSml9Ju0CHF409trN+1+cTDpGkY70WwlzR0hscgkBU/5 BWjgvfhjSFSO8RgMGlAVPb7ARifWSuEvU5SIkoAv03oDgRgOG1GZvxSKtzbyJU2bWADT Stf3vsG9352evCADzdlT5KEeqfvhqyFT8nwRWtKN4duiAp6buu/vKbx88aEC0N6iNDJ+ baNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764580400; x=1765185200; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=HuNRomfv0vCIa2D0+6on2ZVGZKQRa6gA3SVt7VHzktk=; b=Ms5juhN6X5fJ/34gJQrEWs+FznDFP3fyokpoWsEkb57c2GU0Y7zc4EEydTwgrwqviN rhUyjGcJoMZpHrOaBXxjcGRhVmrrkrQf5QX/Dd5YDvq0vjPc2KFCwgHW6CzRGVN+/yKc Afm2TKVclrYqfeWWD1sgxmKWILmM2OUN2LApGQoMlTbeNXqDgZoKDTt/TM7d0z+Gbkzo pd+gIN2/lsxnrEmyyFAWjvHO7GerCkwOs4PMENm6aD83YvHksRsmtqaDvN3zeCSu3uqx sXgVq0Oy9YcQ1OXXnlozyJ81tM8uma7XJGxNqZ3kQwBstU4ULQBU4c0TxTFJF/w3b7L/ FfnA== X-Forwarded-Encrypted: i=1; AJvYcCUkMv3FndrOwme83xWnXB8v8Wx7ZHbtiuQM2n0FCQs5PCpk+Txv/UwiHwWhOWZ/0e3c2ScOwdbKzRfQcB8=@vger.kernel.org X-Gm-Message-State: AOJu0YwmaKECRZjEPzy3SrQMJ8jRany3FE5n1So1y9lLGmOPsJwfmLps 8TXjRTM0ZIdfWqUYIBLR4IRwG7w7yR3XehD+hCxHSkrz5RwqpPwDuysRWE6LMzU0eKw= X-Gm-Gg: ASbGnct283f3VBAQsFYRiJY4YPlaOEkb4uAq3B6Be8Ev/pZ52qVI/EW0vPA3UspR4WU WGaTz1BR8vOk+zzB0AsSMWtK9b1irPkbURVFWKr8riXjYZ/u1TvPiEcPrMMAmR3KeHBon00MTdL 8P3JjRDJO3XcSNG6cU/zSY8iQIfW3t1J/MeBMgkCyWPwJa1AELByeqn0SeJvqzAr3MI1QjbyQK5 sxFgT2dKPqxyfVaojz9j+U/isXrIqslmOWSeDk9A3PFux+0ERFsgb0oC0gkIN4NAP8nGdZzBt6+ A/iAuTojSyY3ZdIU/aGW8DxcJgfrKJ2DiJAnKy2RhuuxdZ8f6F5T0pKNGGhPVneGOERdxvFmMEa rt9bXi+HKClkFb5JM1lKd7oOS4pygBeoa//HOSF5ShFUFmDeahTSXFiY/NVriarrg9IynT3qzN5 mdXMNA0+sLHSC4owwCJ0c= X-Google-Smtp-Source: AGHT+IFg9Xc0ON3PE4VBuS79Bj3CwMQXsVYJVY6JvZMubox5KOFxDBvIFISn4TLDaLNsymYPVz5zzw== X-Received: by 2002:a05:6000:400e:b0:429:ba48:4d8 with SMTP id ffacd0b85a97d-42e0f22ccc9mr26615223f8f.25.1764580400146; Mon, 01 Dec 2025 01:13:20 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:77d1:da59:4ce1:741c]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1caa767dsm25344110f8f.38.2025.12.01.01.13.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Dec 2025 01:13:18 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [RFC PATCH 6/6 v7] sched/fair: Add EAS and idle cpu push trigger Date: Mon, 1 Dec 2025 10:13:08 +0100 Message-ID: <20251201091308.761711-7-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251201091308.761711-1-vincent.guittot@linaro.org> References: <20251201091308.761711-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" EAS is based on wakeup events to efficiently place tasks on the system, but there are cases where a task doesn't have wakeup events anymore or at a far too low pace. For such cases, we check if it's worht pushing hte task on another CPUs instead of putting it back in the enqueued list. Wake up events remain the main way to migrate tasks but we now detect situation where a task is stuck on a CPU by checking that its utilization is larger than the max available compute capacity (max cpu capacity or uclamp max setting) When the system becomes overutilized and some CPUs are idle, we try to push tasks instead of waiting periodic load balance. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 65 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/topology.c | 3 ++ 2 files changed, 68 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9af8d0a61856..e9e1d0c05805 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6990,6 +6990,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) } =20 static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p= ); + /* * Basically dequeue_task_fair(), except it can deal with dequeue_entity() * failing half-way through and resume the dequeue later. @@ -8499,8 +8500,72 @@ static inline bool sched_push_task_enabled(void) return static_branch_unlikely(&sched_push_task); } =20 +static inline bool task_stuck_on_cpu(struct task_struct *p, int cpu) +{ + unsigned long max_capa, util; + + max_capa =3D min(get_actual_cpu_capacity(cpu), + uclamp_eff_value(p, UCLAMP_MAX)); + util =3D max(task_util_est(p), task_runnable(p)); + + /* + * Return true only if the task might not sleep/wakeup because of a low + * compute capacity. Tasks, which wake up regularly, will be handled by + * feec(). + */ + return (util > max_capa); +} + +static inline bool sched_energy_push_task(struct task_struct *p, struct rq= *rq) +{ + if (!sched_energy_enabled()) + return false; + + if (is_rd_overutilized(rq->rd)) + return false; + + if (task_stuck_on_cpu(p, cpu_of(rq))) + return true; + + if (!task_fits_cpu(p, cpu_of(rq))) + return true; + + return false; +} + +static inline bool sched_idle_push_task(struct task_struct *p, struct rq *= rq) +{ + if (rq->nr_running =3D=3D 1) + return false; + + if (!is_rd_overutilized(rq->rd)) + return false; + + /* If there are idle cpus in the llc then try to push the task on it */ + if (test_idle_cores(cpu_of(rq))) + return true; + + return false; +} + + static bool fair_push_task(struct rq *rq, struct task_struct *p) { + if (!task_on_rq_queued(p)) + return false; + + if (p->se.sched_delayed) + return false; + + if (p->nr_cpus_allowed =3D=3D 1) + return false; + + if (sched_energy_push_task(p, rq)) + return true; + + if (sched_idle_push_task(p, rq)) + return true; + return false; } =20 diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index cf643a5ddedd..5edf7b117ed9 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -391,10 +391,13 @@ static void sched_energy_set(bool has_eas) if (sched_debug()) pr_info("%s: stopping EAS\n", __func__); static_branch_disable_cpuslocked(&sched_energy_present); + static_branch_dec_cpuslocked(&sched_push_task); + } else if (has_eas && !sched_energy_enabled()) { } else if (has_eas && !static_branch_unlikely(&sched_energy_present)) { if (sched_debug()) pr_info("%s: starting EAS\n", __func__); static_branch_enable_cpuslocked(&sched_energy_present); + static_branch_inc_cpuslocked(&sched_push_task); } } =20 --=20 2.43.0