From nobody Mon Feb 9 02:40:17 2026 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 864F22BE7DC for ; Tue, 2 Dec 2025 18:12:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699171; cv=none; b=Cb7YrEaubVHGvy91HTNeI4zZMx6yMubYO8ske9WE3j2+ImCfOHaZ2yeAizTsmgiWKi4xd6r+Bl3L7joXkre/gnjNCiTf1W+8jX2ZKaWWLd3eRs6SDB5cWoRP9Q/KWvFwQ7iRyoRiFs+7Y1dyv2ZTX0BNOdjXP0MT4Llcy28pAQ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699171; c=relaxed/simple; bh=zV8/cEhPE7YZs10VV/3SNSRu2hQIU2zJ8NKh0SZ9GTw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fIf9DjPcVRzw/Zr0GLVZOkhjll7Qp2LkmBNzwTUiX8yx8k9m/CCkoI3r1jOhGBM7wowJx53hmQj/yyXnNMl4IphwhjaR8qKGgZsDiyVaB5lDHKXrechBhc3qzMEKnbxM98YFNsX+Menj4WSSGcsbAUwuyMNi5/eUpoTF2Ljo3XE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=b2mCTEIM; arc=none smtp.client-ip=209.85.221.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="b2mCTEIM" Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-42b38693c4dso2217408f8f.3 for ; Tue, 02 Dec 2025 10:12:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764699167; x=1765303967; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OwX/olriHOAxoNjW5iw4IZ+oz3oFzlu8t83MnxTRx/U=; b=b2mCTEIMuyH4X5jUUYz6zROMvwKzaQ4fmIwGm03btKYUuJDcRQAPFIjVGG16NeHyBz 6DlMTzISzrQnsYq0wOm2fdDP7M2/UQ8UfGE6UYaVn2RefvQrnlvgBIxkRd8krfBE+X4T uOlA3vQCP1oIBulwlZCU9eOYEjlE2jkapQIsE0lVQ1Ce7zcv+NvtTIwmXd0YzKc+FqjL eW8jyKozbQ+QmaGAzZfP14YSacRlz+O1+o3oSaTr/pa8Pbm9qILw4VNmiwULpUE6wZx3 CmhUy/mg/egG8YFn9l3RzUMNcdXncVxh9M16VA28XdZJ8FMUU7LPHteg+/7fRZ2VUiC0 9j1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764699167; x=1765303967; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=OwX/olriHOAxoNjW5iw4IZ+oz3oFzlu8t83MnxTRx/U=; b=l2bTptWfFzkzJ/JEWTgy7P+xqvF2rpf+Q6yquZ2CgqLWpitmYajRsEyY2Lft/ko0X1 LngNtw8mFp+5vaU1FhhqTyvM8dO9YOiHKeqG878NKxtLCMqqq4LxEV2GTpuWs1v+Ci2i 1VdyYTmDr/QHirNUr/Bx/5C2SnJe2j7zzBECIx1i/e+fgepjPBJ6DzVkFKSE3ssivuNN mZsBaJcDC3l0HzkfxJBl1hPRAYPei7uUOo1nkwyTUdwRBEoJxezC/wXYO5lk+GIJnDvx igcTsXOUCipwLweTgd2daKpJeM8xqy5vyQe9scGEdUJXcBuyEi+vX/sKSDV0enzlnX3/ 9/8A== X-Forwarded-Encrypted: i=1; AJvYcCUr+hGlqsRfJBFm4fXRolZMkHisotq9gCRA7DmL8tQDBeXiz5SY1TikWQggJg0S71omRCXckHQjHiUFJLs=@vger.kernel.org X-Gm-Message-State: AOJu0YyWv1EAVF04M6VUqdFgjOA9d2xIqPu1hwDdHksQalSASmGpNsdZ g/WaU14HpQCaIDT90WmzWMaYlXdUSsCYx6qxmM7t+IjDBTT3h1jEAVQx3CstT8a8N/E= X-Gm-Gg: ASbGncsUxovMgc2ViIlYbYgEGBPE8dE96j7l3M89Yk1z9PLc491iQ3S7Hmi7zklBdq5 V6fWUU0zCrH7Qht8Hgiu2hQK39oMYFZuHVSKMFNKSjqTbKEuirDP1GvG/NZR4XWHdYPKQHR1Wo9 evOGSm+H5OA6XMw1bfOCddSSgnuV0sBz814bqeAqq1nxh7gtF4frPkMlzclb/upD6p23Fpk6s5+ UwTR2IiaAV/9vvIw577aiVWFLs/KTzakHQ0E6ZS+0Dh9SRi4/P6/g0n/FxIR2aOau/eyq8elXPM ZtLtOBzsWyxbpAtIDs/ss0auTV92dXhPt44w0q5OpKI3pE4GLn6r2v7i/kkkznVmHbGdyyJtbg1 Fr4GTw/nSaJMEJqrCIM2yn7cGRmUBeKL0Wqbz4wFPm11tzljDoJyCynfhHraniBlbSCzQtyiC4/ cOm1dqqPrPLIj2HYhvGQ== X-Google-Smtp-Source: AGHT+IGDsVFt1qYkodvgFgkNaOhx3dvZIwirFdKAvvfWWNtnnxBgxeKL1KOBgSgZl9isMoxZVshncw== X-Received: by 2002:a05:6000:290e:b0:42b:5448:7b07 with SMTP id ffacd0b85a97d-42e0f1df33fmr31745588f8f.2.1764699166647; Tue, 02 Dec 2025 10:12:46 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:cae:51ce:40d7:cfd1]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1c5d613esm34883983f8f.11.2025.12.02.10.12.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Dec 2025 10:12:45 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [PATCH 1/6 v8] sched/fair: Filter false overloaded_group case for EAS Date: Tue, 2 Dec 2025 19:12:37 +0100 Message-ID: <20251202181242.1536213-2-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251202181242.1536213-1-vincent.guittot@linaro.org> References: <20251202181242.1536213-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" With EAS, a group should be set overloaded if at least 1 CPU in the group is overutilized but it can happen that a CPU is fully utilized by tasks because of clamping the compute capacity of the CPU. In such case, the CPU is not overutilized and as a result should not be set overloaded as well. group_overloaded being a higher priority than group_misfit, such group can be selected as the busiest group instead of a group with a mistfit task and prevents load_balance to select the CPU with the misfit task to pull the latter on a fitting CPU. Signed-off-by: Vincent Guittot Tested-by: Pierre Gondois --- kernel/sched/fair.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1855975b8248..b10f04715251 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9987,6 +9987,7 @@ struct sg_lb_stats { unsigned int group_asym_packing; /* Tasks should be moved to preferred CP= U */ unsigned int group_smt_balance; /* Task on busy SMT be moved */ unsigned long group_misfit_task_load; /* A CPU has a task too big for its= capacity */ + unsigned int group_overutilized; /* At least one CPU is overutilized in t= he group */ #ifdef CONFIG_NUMA_BALANCING unsigned int nr_numa_running; unsigned int nr_preferred_running; @@ -10219,6 +10220,13 @@ group_has_capacity(unsigned int imbalance_pct, str= uct sg_lb_stats *sgs) static inline bool group_is_overloaded(unsigned int imbalance_pct, struct sg_lb_stats *sgs) { + /* + * With EAS and uclamp, 1 CPU in the group must be overutilized to + * consider the group overloaded. + */ + if (sched_energy_enabled() && !sgs->group_overutilized) + return false; + if (sgs->sum_nr_running <=3D sgs->group_weight) return false; =20 @@ -10402,14 +10410,12 @@ sched_reduced_capacity(struct rq *rq, struct sche= d_domain *sd) * @group: sched_group whose statistics are to be updated. * @sgs: variable to hold the statistics for this group. * @sg_overloaded: sched_group is overloaded - * @sg_overutilized: sched_group is overutilized */ static inline void update_sg_lb_stats(struct lb_env *env, struct sd_lb_stats *sds, struct sched_group *group, struct sg_lb_stats *sgs, - bool *sg_overloaded, - bool *sg_overutilized) + bool *sg_overloaded) { int i, nr_running, local_group, sd_flags =3D env->sd->flags; bool balancing_at_rd =3D !env->sd->parent; @@ -10431,7 +10437,7 @@ static inline void update_sg_lb_stats(struct lb_env= *env, sgs->sum_nr_running +=3D nr_running; =20 if (cpu_overutilized(i)) - *sg_overutilized =3D 1; + sgs->group_overutilized =3D 1; =20 /* * No need to call idle_cpu() if nr_running is not 0 @@ -11103,13 +11109,15 @@ static inline void update_sd_lb_stats(struct lb_e= nv *env, struct sd_lb_stats *sd update_group_capacity(env->sd, env->dst_cpu); } =20 - update_sg_lb_stats(env, sds, sg, sgs, &sg_overloaded, &sg_overutilized); + update_sg_lb_stats(env, sds, sg, sgs, &sg_overloaded); =20 if (!local_group && update_sd_pick_busiest(env, sds, sg, sgs)) { sds->busiest =3D sg; sds->busiest_stat =3D *sgs; } =20 + sg_overutilized |=3D sgs->group_overutilized; + /* Now, start updating sd_lb_stats */ sds->total_load +=3D sgs->group_load; sds->total_capacity +=3D sgs->group_capacity; --=20 2.43.0 From nobody Mon Feb 9 02:40:17 2026 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 086592C236D for ; Tue, 2 Dec 2025 18:12:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699171; cv=none; b=C0Ya/Sj3E3Ou0iJL6V0rj+uPRvhgBWLS0VbycmvAsXzViYicARQrpOnZfK+e7gKI+IcTq5VOkABYG44W5qFfCq3ZTAWqOnjB/49kEke3Ze7JoeUqKay57lH4+LBL1DkkEcbESrDE57onfm1PxJoqrW0ZRvTEGWuXUv7yQpNdxvo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699171; c=relaxed/simple; bh=4TjgOClD9MZ3bpoMuNewJ2AXb9TzO5tUug3pj3x8AyM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RxwtCJ4oUFpwxB2PAVJzJoap8Yo2vekDVFTD/bGGU+4wvRIs4IXQ1Moucg8jYV/eADPkCqrR8zjSVMiaCAajnRqa2dpu7PXnbnifvp0uoOffl5/kLsVosPBhYsaG0IPpBGYa+KIn27PdIwt2w0QJAnDVQ4vqQ0G54Zf9U79l9t8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=cEw+9h3g; arc=none smtp.client-ip=209.85.128.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="cEw+9h3g" Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-477a2ab455fso66424175e9.3 for ; Tue, 02 Dec 2025 10:12:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764699168; x=1765303968; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8fl5Pg9Y8yiY67LalwVtDEwtUsWcIOn3WvNZLWDGeNc=; b=cEw+9h3gVoSQRuFI2h6ZLBrz6TjmdKiJouNR2Dp7SGKwnGfy/Fo8mO+R0XziFRvDPD H17IHxxRkA6/K74DOt9MxwJL0Qayu4ofVUkeAux43isUMots1txo23vyRPRGA4xl7aTD 5GZSjWXeN5W7FJDK83AszdAhXb/JyKH3Yrw1Pmg7aYn/KNeB1/Tqa2M5fmcnoLqXM8IQ WkIiyFnOyAfih/D8yfuVVfFOE/U588Dxtz2zv4ukhEalxfxWXS7IDrSVkAACrdh6l08m kTXQFlVJGf3F1EYRhUkOYz5vno9aGV4pHEfjyXZ0qTG3eOCwTXYsrBsap0uWL/jKUpIn YrUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764699168; x=1765303968; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8fl5Pg9Y8yiY67LalwVtDEwtUsWcIOn3WvNZLWDGeNc=; b=TiWvWvkwL2ovLHYvQz7tnwDDoZzQCLaKx6pzzlXwllmHAA7ihJCz2NX6efQzxSpUuy eECSkuKOkCEP6sr+WWNiuv3ZbhDyHBW5UVY2POOSRqUWiB2lFIPI84G03evkRSqXT2H5 /sCq/mgQm4bv6ltenm3877yWDlpVfqTUKv6tBiAN1Si5UrJRpLUukb1E4Hg4I75264Ey UGaTyWZOQif3FF+18IjuhpLg8RZJ7Y0ddGV6MXi09xS++Q/XykHbAAin6Ymipbirmtwq Wf0HD4X4uMinQsw8JJK/3fU8m5GOF2JT0Ahqp2snoJEylYNuS0PAAeSzX9lA/9r1BT3V Sahw== X-Forwarded-Encrypted: i=1; AJvYcCVnFCZGJ4oLPDN4s7DbOAAw5MGzi1USDgfvxgOkoTQ5QL7udohb+R0UhFx/aZFQ3KKmpRo4AgkTFllqgSU=@vger.kernel.org X-Gm-Message-State: AOJu0YxTMusvnxAbh8QRHHUgw2QEsB6tNERkpqfamwVeF2PlibBKyO1O Xzah1qDGO6Z0lFj48s5DZEQraA4Ldhkf9hVRyqdkWYDo0PSVKQfqZcRnHmdJuVqRXqKu15LQCis HM9hO X-Gm-Gg: ASbGncs0K8LLUGSWIeDv3RDo7RYBrQnuRyMHgTV9Dx3zjEXXNU1J3IDrT5qp7yyF5kH E79+Ou0GtBi0IchoBMNc2dyIeU4tnwvEXu6NIGTjGdx/YPYdJf8XAt77O1KkMRkwauATdX6LQYX mSXI9dt4J0Rg97/ct1IouVQoH4iV5pWl1un1RM61MQATXKs/zXOVmTxPzb45/LME1OZXZQPHWdb 3pb35+9+7mr4xgSgbBa0cPpc1ALXkLFxfswm88wPfeeJ5y62+4CXNVGvBmcQfR2C5Y78KBm+g3K btIe/Xx5/NcDCIo+o6W7KoVGLSDoopnJiR8m38YjuN6ylU53QKmNPhCSPCnj8p/+usmbu6DcVEb SpkUabFQ3cOP9xTdHZxXbp+Twxr2BZbKBOXYpb6zSecQoMguSTG1qKbbMwI9iOErqYhf9oJB6tZ r2zgMzXhgovPaL+CZriA== X-Google-Smtp-Source: AGHT+IH4mNocjZgiIQYAc4db0y+2GM4Gy8vd5XK7cmb0tmLrp0qqiU2heNb6j2HEO7gkAqLjbqOcXg== X-Received: by 2002:a05:6000:2689:b0:429:c617:a32f with SMTP id ffacd0b85a97d-42f72241102mr88192f8f.52.1764699168363; Tue, 02 Dec 2025 10:12:48 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:cae:51ce:40d7:cfd1]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1c5d613esm34883983f8f.11.2025.12.02.10.12.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Dec 2025 10:12:47 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [PATCH 2/6 v8] sched/fair: Update overutilized detection Date: Tue, 2 Dec 2025 19:12:38 +0100 Message-ID: <20251202181242.1536213-3-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251202181242.1536213-1-vincent.guittot@linaro.org> References: <20251202181242.1536213-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Checking uclamp_min is useless and counterproductive for overutilized state as misfit can now happen without being in overutilized state Signed-off-by: Vincent Guittot Reviewed-by: Qais Yousef --- kernel/sched/fair.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index b10f04715251..f430ec890b72 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6785,16 +6785,15 @@ static inline void hrtick_update(struct rq *rq) =20 static inline bool cpu_overutilized(int cpu) { - unsigned long rq_util_min, rq_util_max; + unsigned long rq_util_max; =20 if (!sched_energy_enabled()) return false; =20 - rq_util_min =3D uclamp_rq_get(cpu_rq(cpu), UCLAMP_MIN); rq_util_max =3D uclamp_rq_get(cpu_rq(cpu), UCLAMP_MAX); =20 /* Return true only if the utilization doesn't fit CPU's capacity */ - return !util_fits_cpu(cpu_util_cfs(cpu), rq_util_min, rq_util_max, cpu); + return !util_fits_cpu(cpu_util_cfs(cpu), 0, rq_util_max, cpu); } =20 /* --=20 2.43.0 From nobody Mon Feb 9 02:40:17 2026 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9712F2C326D for ; Tue, 2 Dec 2025 18:12:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699173; cv=none; b=G+hWd0x+oA+VSrkWiY141tFeFPHkgGk8qFytp0qcb5DKkDWwAjpSl8wd6EGZD6OXJUI4rkXMYLBqilFUJB20n2PTH02lSF73XHmveEvyTsHdV4sg1INPC20C3vyTdZYnHEMyQ8tRnNevEdeK+K792J6fQwzrv9u66hmUWMH2PDo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699173; c=relaxed/simple; bh=FTHun3PMOq+CklxsBPL6XoRjxH2p1DjEDgtWxflTW+A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=l6BU9NC7655zahjqLmq1/6PJ+uSMInRjAEhG454DjSbWckdAu55El2g7S2ZeIVUVeXtCta/NO2yfLHLkA3sfAjIzckNiItHcFD3WaIw9VF7gEIPdrapTaDDD1ibtPVHXgxZbTtkF63LFAUn8gRgsY2Par1c2KNvEfo7c4S2BEGM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=n+NAT8A6; arc=none smtp.client-ip=209.85.128.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="n+NAT8A6" Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-47778b23f64so34115645e9.0 for ; Tue, 02 Dec 2025 10:12:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764699170; x=1765303970; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=95eiwIqVqnWVfXykfzvRd1ip/6pVXilQ5gdb8x62/lM=; b=n+NAT8A6F2N9PQ2aEZ2J05S0OdoKJauI1lV/UpIvmjQWAXuPZp3ynbmFBfON4i0OtX E+O7U05ZlmVIYrD9ivX9z0N3QVBjBV+SVKsnD1Pncl5FT7+XEnFM6pHRd/0BeHJ8Zq0D ua7Ph89APFgJ3hE6rQ01SyTVS0yHEBqI1w8mOcRafYefNwgwQF4C/iCoiBKN+zdWMzDw PqbS8KQK8l+pprsPyRV8AEW/9t+2yxdkyev1P5B3SHY83z/xrNvgh4n/6Am8MnGwpwGp LO2yfcmMQvM9SYVLe/cw5fvmf3s2XkOom/sfZiE0Dw/NWb9aFs1Gr4BHlpCwZrWH6F6j IVPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764699170; x=1765303970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=95eiwIqVqnWVfXykfzvRd1ip/6pVXilQ5gdb8x62/lM=; b=aNOUSJEQu/E35vUbF5wN6LIDdWKCIqSyiYjxJQSnhUTZR7hOWeyUk6cliR11OmiSwQ I2j1Dol17/nA4ehmed7DvFchWrlD6uMl6niiKQhDqRJVelRpDY00KgDZPbK4EC78eioc /oAP+7tS2siebTbUg5PWTh+eEI4nBEEQuuqvj3QFAuIBcFnITrf5xijtFQ4CoBDn2Hwx huuxdmcFaJdZjAHDff9IqhXsnnoz1QuoPEUmCuWloZR/1EUdYIfwW4awY2+VCzhRGFUe rrmjk2JvBuZpkTETUWVv9eF3PSjURS96rHLtXwa2pxaYIU6zicmsLeLB+PrTjChGmVe+ ghXA== X-Forwarded-Encrypted: i=1; AJvYcCXD9Kc0U0yfB68dE4R0u8iVNro1upk1+upOiVaVFPNtk6SgpyX+AbpTkp91DfAli3HSeINuDVtNcm7w3fo=@vger.kernel.org X-Gm-Message-State: AOJu0Yzky7IEfb4FN0DgkmEA/8I7cu38HxB2irPGq7I+8BzPcqSnoG2E Ecfx488YSQ3DJT5g/Xayob4vGgaSf0sZbBsunn1F75ufvZTqgPaPHZoWKnoL42Nj2EU= X-Gm-Gg: ASbGncvi/ljdnrkNzkQKI/E+wKXqgRmmGvnDdYAgnGJFYb2fhfGpfQ1DhsW+RG9fN7x oChohJ79dosgvDcWn6vfTrOsuT/qXNAdfk3d3cjkfCv0S5m5w6h4u9Tp1Sx130O3WmSQ4zXPSf3 9cu9RdcANFzscTm46Po6/WuOI+yC4rovmz/84TRnIuwPFiHR/yhONTtAYdykLpPf9B0P4KruZ/s pn1WYhqro4vF362uQa1L3V/wVyCzL6/9pPau8tvE/9CYJw9r5GMjvBNOBjLIiT8BCtFyqiQ6fcs kR4e9u2lkAQK2eytPhyuT4l4e0KzveI7CD4TP3SPW+e/WNHStKrv2Uuwtqzu/Bvk8Ukf2kUows5 AmxGSmA7EWxW3Lk7B681+Qhjc8gkK9VXzYFolMe+rilQOPJ6NdkwjWsgq9p5XUlS3wWQY47eDTb ZTcFIVN5GeSzqiuzqesw== X-Google-Smtp-Source: AGHT+IHKwsNZQ54WQeh1OkEZp+rHek7cofKh2jF8AvBFF7bo+ipZzX+cmNGZb52xkE/HDIEOGX81tw== X-Received: by 2002:a05:6000:240b:b0:42b:43cc:982e with SMTP id ffacd0b85a97d-42f7222e786mr108447f8f.36.1764699169822; Tue, 02 Dec 2025 10:12:49 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:cae:51ce:40d7:cfd1]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1c5d613esm34883983f8f.11.2025.12.02.10.12.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Dec 2025 10:12:48 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [PATCH 3/6 v8] sched/fair: Prepare select_task_rq_fair() to be called for new cases Date: Tue, 2 Dec 2025 19:12:39 +0100 Message-ID: <20251202181242.1536213-4-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251202181242.1536213-1-vincent.guittot@linaro.org> References: <20251202181242.1536213-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Update select_task_rq_fair() to be called out of the 3 current cases which are : - wake up - exec - fork We wants to select a rq in some new cases like pushing a runnable task on a better CPU than the local one. In such case, it's not a wakeup , nor an exec nor a fork. We make sure to not distrub these cases but still go through EAS and fast-path. Signed-off-by: Vincent Guittot Reviewed-by: Qais Yousef --- kernel/sched/fair.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f430ec890b72..80c4131fb35b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8518,6 +8518,7 @@ static int select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags) { int sync =3D (wake_flags & WF_SYNC) && !(current->flags & PF_EXITING); + int want_sibling =3D !(wake_flags & (WF_EXEC | WF_FORK)); struct sched_domain *tmp, *sd =3D NULL; int cpu =3D smp_processor_id(); int new_cpu =3D prev_cpu; @@ -8535,16 +8536,21 @@ select_task_rq_fair(struct task_struct *p, int prev= _cpu, int wake_flags) if ((wake_flags & WF_CURRENT_CPU) && cpumask_test_cpu(cpu, p->cpus_ptr)) return cpu; + } =20 - if (!is_rd_overutilized(this_rq()->rd)) { - new_cpu =3D find_energy_efficient_cpu(p, prev_cpu); - if (new_cpu >=3D 0) - return new_cpu; - new_cpu =3D prev_cpu; - } + /* + * We don't want EAS to be called for exec or fork but it should be + * called for any other case such as wake up or push callback. + */ + if (!is_rd_overutilized(this_rq()->rd) && want_sibling) { + new_cpu =3D find_energy_efficient_cpu(p, prev_cpu); + if (new_cpu >=3D 0) + return new_cpu; + new_cpu =3D prev_cpu; + } =20 + if (wake_flags & WF_TTWU) want_affine =3D !wake_wide(p) && cpumask_test_cpu(cpu, p->cpus_ptr); - } =20 rcu_read_lock(); for_each_domain(cpu, tmp) { @@ -8575,7 +8581,7 @@ select_task_rq_fair(struct task_struct *p, int prev_c= pu, int wake_flags) if (unlikely(sd)) { /* Slow path */ new_cpu =3D sched_balance_find_dst_cpu(sd, p, cpu, prev_cpu, sd_flag); - } else if (wake_flags & WF_TTWU) { /* XXX always ? */ + } else if (want_sibling) { /* Fast path */ new_cpu =3D select_idle_sibling(p, prev_cpu, new_cpu); } --=20 2.43.0 From nobody Mon Feb 9 02:40:17 2026 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2844F2C325A for ; Tue, 2 Dec 2025 18:12:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699175; cv=none; b=g6qgWPnwgF4+LeWY1YJeTKobChNZ8jc/sixeON+6MZffjHVgYfh/sag0tXtwseTilKSG++scq3nELPEWoJjWvUKZn++Rb5EjrWlopQKopchIk5QaPJN8Bayi0fFUn2qSmR6f86iYiW8U7UjueudeubhUp1ktz0ZImd2cmZljNaE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699175; c=relaxed/simple; bh=yqj7P/HYnOqIGgjnRsxw62yPjJYvMCWDq1nk2Mb/+SI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=q0liCwOTx33AbnhRM+8tQpYyQBqepHNXXnWbkC4B5bmZuBj5AqorUmDgBwuriNmEjhdJz0CNbL2Ch5J5g5f1K/RwUOnx8XCv3rMG0OVA0rNNUW2tnUuYi0uIWZK/UTSO+YxC5/mCOT/Wr0u59EFRVCkkjGbgSkhfWqaefwMSLIk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=X5vhUcvC; arc=none smtp.client-ip=209.85.221.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="X5vhUcvC" Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-42e2b78d45bso1587841f8f.0 for ; Tue, 02 Dec 2025 10:12:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764699171; x=1765303971; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PoJqWP9S1BHvgP8J17UtYpg7LtWNY87dWXfyHa/YHGM=; b=X5vhUcvCqba9+7YxS/XrzmM6L6w3QP18dZWg4FlpACVPOlPg0kGQr24K3LrhUHpm4q h+RVRcDvOk9cd33TF1wAFS0Rc7lejOrLKUs2Ct4llr7s+EozLN47UfE6TrMqx/RoEpmw M22YUOgG0iWiN6ZtNTLi7fMET8S/Ror243gRIZoTsvg7+g92T5agydD19YFoURHQO34f TLrRv5sxm+8HZnnVju4pPXlFQBmFAA8tlLy8aD3UQ60SMgaeBAeinkLNvx5MBBJa5Pgw O0f/nkVYGyFx/IY+nGBYrlPr+y4OliYn9MjNn9zFS2Ac0/SI9kP5BwkPvqaMZTlO9ocX I7vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764699171; x=1765303971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=PoJqWP9S1BHvgP8J17UtYpg7LtWNY87dWXfyHa/YHGM=; b=p0hxfoSOw13Vawzvp8yI84UwuW7gr6kVQf6GxTVWtjo/OXt9Nr3QpciDjvTljb5sOm EHJ+8R4dz19grp6Ed3vw24Huw9CtAjnG1grY7sNrUVCMXToHi11AH1Mc79eEmVCLu800 egX4zdUQrtJh2U2gBqKcgVynEk6VS0y4yaKCgE7Kexr8Df9wK8rnz+cYas8wiTECGrrE tsvB+yq5i4HVV06dQYXHiu/JgX00KEbxlleHLRh3rC1RVlh3wjhXlvmEuS7b5Bqs4eyz spc1n5AnvXapscnPOnP2RHxDdq7g8KewTKG2nbi31Fz9H2A0Kb6vjcXL7QuhmE9U3JQw 07+A== X-Forwarded-Encrypted: i=1; AJvYcCXrTWcvsFse1pA3LCQR9HhfLUR55I34wpRQWQdgH/NZhVMrSrCFYA2vI5lXGrZ96oT8XnZQk81Xh958UPY=@vger.kernel.org X-Gm-Message-State: AOJu0Yy+OXx+rcHy8t8jFmsS5ha7/e4CAIvxXBZiON8wtZOu3LLwQSTG uO6O/CujT43vNOIRq4OilbgHHTUPnKo4QKFSHB4aLmmfaGEkVb4ivu9EN0YK4JIhRn0= X-Gm-Gg: ASbGncs5GE7WHC9/DyeTTKR5ZFzd8rKtE3r2TZ39LpHLa7AhDpIrV9oaM/HpMfnFPgl 8TwVA8kURmVwCzTJ/4+1La+L+VvI9UVAhFyMgaFyl5iVUWyoYj91ctEPnEvFUuSJcwNaQ/PfVbV dB3F59IS20vd9o/NHgYBeAzjeXdzR8ybqMPE/JP+KMOne8Ljuf8Se25jripsY9OeGAq2/O2dbiF UZnIsZ0q7Ing/G7l029IzbVzcIB9QpYl0drvusrSpWHLRs3G4wB2a12mW/FlE8Y05phMRU+Mclh zCnW61WUUg+JIfCRGaq5JzpE4lG+j3KErWnoAP3HbgB3l2mHV5Qjq8N7MA3sGpari6V2ngho/O0 XGl+hT02F4LxA4nVhAz0yX/Juc6xPp22YL70Eo8JVrs+ZIhO5JKC6jrnBQ2hsInaQEqpQJanadk Zmf7gDtUlbGGyQHClSqA== X-Google-Smtp-Source: AGHT+IHl2SR6N56KfwdLejYZ+WFC23ZyhlXXWGFTKodvmAZIOMnaZJtMuwdhh1GUAIaf76TSi+UjFA== X-Received: by 2002:a05:6000:2c0d:b0:42b:3afa:5e1d with SMTP id ffacd0b85a97d-42cc1ceeb13mr47262811f8f.20.1764699171347; Tue, 02 Dec 2025 10:12:51 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:cae:51ce:40d7:cfd1]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1c5d613esm34883983f8f.11.2025.12.02.10.12.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Dec 2025 10:12:50 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [PATCH 4/6 v8] sched/fair: Add push task mechanism for fair Date: Tue, 2 Dec 2025 19:12:40 +0100 Message-ID: <20251202181242.1536213-5-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251202181242.1536213-1-vincent.guittot@linaro.org> References: <20251202181242.1536213-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" EAS is based on wakeup events to efficiently place tasks on the system, but there are cases where a task doesn't have wakeup events anymore or at a far too low pace. For such situation, we can take advantage of the task being put back in the enqueued list to check if it should be pushed on another CPU. When the task is alone on the CPU, it's never put back in the enqueued list; In this special case, we use the tick to run the check. Add a push task mechanism that enables fair scheduler to push runnable tasks. EAS will be one user but other feature like filling idle CPUs can also take advantage of it. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 212 ++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 4 + 2 files changed, 214 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 80c4131fb35b..252254168c92 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6989,6 +6989,8 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) hrtick_update(rq); } =20 +static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p= ); + /* * Basically dequeue_task_fair(), except it can deal with dequeue_entity() * failing half-way through and resume the dequeue later. @@ -7017,6 +7019,8 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags) h_nr_idle =3D task_has_idle_policy(p); if (task_sleep || task_delayed || !se->sched_delayed) h_nr_runnable =3D 1; + + fair_remove_pushable_task(rq, p); } =20 for_each_sched_entity(se) { @@ -8504,6 +8508,187 @@ static int find_energy_efficient_cpu(struct task_st= ruct *p, int prev_cpu) return target; } =20 +DEFINE_STATIC_KEY_FALSE(sched_push_task); + +static inline bool sched_push_task_enabled(void) +{ + return static_branch_unlikely(&sched_push_task); +} + +static bool fair_push_task(struct rq *rq, struct task_struct *p) +{ + return false; +} + +static inline int has_pushable_tasks(struct rq *rq) +{ + return !plist_head_empty(&rq->cfs.pushable_tasks); +} + +static struct task_struct *pick_next_pushable_fair_task(struct rq *rq) +{ + struct task_struct *p; + + if (!has_pushable_tasks(rq)) + return NULL; + + p =3D plist_first_entry(&rq->cfs.pushable_tasks, + struct task_struct, pushable_tasks); + + WARN_ON_ONCE(rq->cpu !=3D task_cpu(p)); + WARN_ON_ONCE(task_current(rq, p)); + WARN_ON_ONCE(p->nr_cpus_allowed <=3D 1); + WARN_ON_ONCE(!task_on_rq_queued(p)); + + /* + * Remove task from the pushable list as we try only once after that + * the task has been put back in enqueued list. + */ + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); + + return p; +} + +static int +select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags); + +/* + * See if the non running fair tasks on this rq can be sent on other CPUs + * that fits better with their profile. + */ +static bool push_fair_task(struct rq *rq) +{ + struct task_struct *next_task; + int prev_cpu, new_cpu; + struct rq *new_rq; + + next_task =3D pick_next_pushable_fair_task(rq); + if (!next_task) + return false; + + if (is_migration_disabled(next_task)) + return true; + + /* We might release rq lock */ + get_task_struct(next_task); + + prev_cpu =3D rq->cpu; + + new_cpu =3D select_task_rq_fair(next_task, prev_cpu, 0); + + if (new_cpu =3D=3D prev_cpu) + goto out; + + new_rq =3D cpu_rq(new_cpu); + + if (double_lock_balance(rq, new_rq)) { + /* The task has already migrated in between */ + if (task_cpu(next_task) !=3D rq->cpu) { + double_unlock_balance(rq, new_rq); + goto out; + } + + deactivate_task(rq, next_task, 0); + set_task_cpu(next_task, new_cpu); + activate_task(new_rq, next_task, 0); + + resched_curr(new_rq); + + double_unlock_balance(rq, new_rq); + } + +out: + put_task_struct(next_task); + + return true; +} + +static void push_fair_tasks(struct rq *rq) +{ + /* push_fair_task() will return true if it moved a fair task */ + while (push_fair_task(rq)) + ; +} + +static DEFINE_PER_CPU(struct balance_callback, fair_push_head); + +static inline void fair_queue_pushable_tasks(struct rq *rq) +{ + if (!sched_push_task_enabled() || !has_pushable_tasks(rq)) + return; + + queue_balance_callback(rq, &per_cpu(fair_push_head, rq->cpu), push_fair_t= asks); +} + +static void fair_remove_pushable_task(struct rq *rq, struct task_struct *p) +{ + if (sched_push_task_enabled()) + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); +} + +static void fair_add_pushable_task(struct rq *rq, struct task_struct *p) +{ + if (sched_push_task_enabled() && fair_push_task(rq, p)) { + plist_del(&p->pushable_tasks, &rq->cfs.pushable_tasks); + plist_node_init(&p->pushable_tasks, p->prio); + plist_add(&p->pushable_tasks, &rq->cfs.pushable_tasks); + } +} + +static int active_load_balance_cpu_stop(void *data); + +/* + * See if the alone task running on the CPU should migrate on a better than + * the local one. + */ +static inline bool check_pushable_task(struct task_struct *p, struct rq *r= q) +{ + int new_cpu, cpu =3D cpu_of(rq); + + if (!sched_push_task_enabled()) + return false; + + if (WARN_ON(!p)) + return false; + + if (WARN_ON(!task_current(rq, p))) + return false; + + if (is_migration_disabled(p)) + return false; + + /* If there are several task, wait for being put back */ + if (rq->nr_running > 1) + return false; + + if (!fair_push_task(rq, p)) + return false; + + new_cpu =3D select_task_rq_fair(p, cpu, 0); + + if (new_cpu =3D=3D cpu) + return false; + + /* + * ->active_balance synchronizes accesses to + * ->active_balance_work. Once set, it's cleared + * only after active load balance is finished. + */ + if (!rq->active_balance) { + rq->active_balance =3D 1; + rq->push_cpu =3D new_cpu; + } else + return false; + + raw_spin_rq_unlock(rq); + stop_one_cpu_nowait(cpu, + active_load_balance_cpu_stop, rq, + &rq->active_balance_work); + raw_spin_rq_lock(rq); + + return true; +} + /* * select_task_rq_fair: Select target runqueue for the waking task in doma= ins * that have the relevant SD flag set. In practice, this is SD_BALANCE_WAK= E, @@ -8973,6 +9158,12 @@ pick_next_task_fair(struct rq *rq, struct task_struc= t *prev, struct rq_flags *rf put_prev_entity(cfs_rq, pse); set_next_entity(cfs_rq, se); =20 + /* + * The previous task might be eligible for being pushed on + * another cpu if it is still active. + */ + fair_add_pushable_task(rq, prev); + __set_next_task_fair(rq, p, true); } =20 @@ -9036,6 +9227,13 @@ static void put_prev_task_fair(struct rq *rq, struct= task_struct *prev, struct t cfs_rq =3D cfs_rq_of(se); put_prev_entity(cfs_rq, se); } + + /* + * The previous task might be eligible for being pushed on another cpu + * if it is still active. + */ + fair_add_pushable_task(rq, prev); + } =20 /* @@ -13390,8 +13588,10 @@ static void task_tick_fair(struct rq *rq, struct t= ask_struct *curr, int queued) if (static_branch_unlikely(&sched_numa_balancing)) task_tick_numa(rq, curr); =20 - update_misfit_status(curr, rq); - check_update_overutilized_status(task_rq(curr)); + if (!check_pushable_task(curr, rq)) { + update_misfit_status(curr, rq); + check_update_overutilized_status(task_rq(curr)); + } =20 task_tick_core(rq, curr); } @@ -13552,6 +13752,8 @@ static void __set_next_task_fair(struct rq *rq, str= uct task_struct *p, bool firs { struct sched_entity *se =3D &p->se; =20 + fair_remove_pushable_task(rq, p); + if (task_on_rq_queued(p)) { /* * Move the next running task to the front of the list, so our @@ -13567,6 +13769,11 @@ static void __set_next_task_fair(struct rq *rq, st= ruct task_struct *p, bool firs if (hrtick_enabled_fair(rq)) hrtick_start_fair(rq, p); =20 + /* + * Try to push prev task before checking misfit for next task as + * the migration of prev can make next fitting the CPU + */ + fair_queue_pushable_tasks(rq); update_misfit_status(p, rq); sched_fair_update_stop_tick(rq, p); } @@ -13596,6 +13803,7 @@ void init_cfs_rq(struct cfs_rq *cfs_rq) { cfs_rq->tasks_timeline =3D RB_ROOT_CACHED; cfs_rq->zero_vruntime =3D (u64)(-(1LL << 20)); + plist_head_init(&cfs_rq->pushable_tasks); raw_spin_lock_init(&cfs_rq->removed.lock); } =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b419a4d98461..697bd654298a 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -711,6 +711,8 @@ struct cfs_rq { unsigned long runnable_avg; } removed; =20 + struct plist_head pushable_tasks; + #ifdef CONFIG_FAIR_GROUP_SCHED u64 last_update_tg_load_avg; unsigned long tg_load_avg_contrib; @@ -3620,6 +3622,8 @@ static inline bool sched_energy_enabled(void) { retur= n false; } =20 #endif /* !(CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL) */ =20 +DECLARE_STATIC_KEY_FALSE(sched_push_task); + #ifdef CONFIG_MEMBARRIER =20 /* --=20 2.43.0 From nobody Mon Feb 9 02:40:17 2026 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E41A92C21CF for ; Tue, 2 Dec 2025 18:12:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699176; cv=none; b=BC8k+AK2XpsNvvX3tvTa/b6shN4a135beMmjLYOqG9AyKU9S98/bw/nnWS1AnKCuDj+aSYC5U6V408ov4BtHmvKb4DgqwDL7ELnjwFnt/IT1VIBjWdDeFXYuDV/6mkcm1okX7lbloAnxoAiHlOwTfYoXECliKBViveUXe577gP0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699176; c=relaxed/simple; bh=gwZbldmSDzyXKgO0g/yzSuOFMNK/YSGgkFmk5lttomQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WxIcNhZRdJ9Yrosf6/yrZ+epg1lxjZIdyE4g98IILC7unE+eeLVU647OU/9L4k8Kj/Ow6UJh5kC6F8NOZpL7v1H+kCvTsE0kuY/AnpmJ9bzkc8894Mx0aFSQsA8azrb+F8+ER7YaPDjWVZuQsqrjmceJ53kA1h+R9Udmx9x9ac8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=HevPo2af; arc=none smtp.client-ip=209.85.221.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="HevPo2af" Received: by mail-wr1-f52.google.com with SMTP id ffacd0b85a97d-42e2e167067so1608908f8f.2 for ; Tue, 02 Dec 2025 10:12:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764699173; x=1765303973; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XcTkkpq4Ew0rdtNEH8zRQFULIsRB0r8eNuPnMor+11A=; b=HevPo2afigoMTHVeuz68e7YHymo99W7fbSWXRSM6Vjj2chFZLwOQEWZv6tGE6CBfsE wyn1q20SquBxlJWJm3RWv9a0Jx8Ily4e8d2XdbdSSZAS2gDdmbBAwaYJSSmd+Cgar+Yh tcVQemgeLWDy85TVEv7SJlQhwva3rC/54EwgiRzVLcdvtXMXtdwPs2Ol+9llbJWhF4Px Pdv2wInlWFIeM1avlxkP58PL8pBs+zzn3xoN/3LFEzHWlJxQGO3Pe6xfGhOrRDOlus28 0CCDcFj/1/OA+fCX0XLppHAI8yfVvtMVLzPsREATZyxnhCtSRu/VDM6LEd8KZKIW/Qxa xmxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764699173; x=1765303973; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=XcTkkpq4Ew0rdtNEH8zRQFULIsRB0r8eNuPnMor+11A=; b=qtO/ZuI6QEZu8fmtzOOaqz6hv0Zmr9EQzoX46EP5O4ksHaHWUwG1hhKO5suW45d0IC Cps+vBG4t+OtjsshxFKiPWVpheoKA4nDh6I9+RwyFBKX7T9rUJkypbgw22Ih/tGzUL8X xofNrvx8RIHZdDS3qhxIOJXhVOl9K9HNtf01OyhftlSXvLaGPWvRX6Gy8SZCfQXTKKYr jGmyARZrhP+71vvc1d1G/uuEw4ByaifxMMtu6CiEM1QPgH8nkUasfIUDEBUzfTU8rOgM FPjY6BwllrKjn1/iwUJPyyNTjx84R4iBRpiDOGa4VHt/qbW5Dv/48gKpL+XNjnhRz4Kw nI6A== X-Forwarded-Encrypted: i=1; AJvYcCUshCtxv/0nMUkTLySo8mmgb7cZlwUi5dZhMpU8PBhdDQkqlIKu/GN/iQ3PrIEjvWWoszQDMh87uuNfqQc=@vger.kernel.org X-Gm-Message-State: AOJu0YwDNb9Jqh86cHERg4FpLxrhYlU+qIlq9W0bKdVO3/MrKXA7I6oe LawOaqaoVp5twS8uMPQ78CTR5AYHPukGJw5NBoJ96QTrFui3yXr95LQbeM6ewQMYmzg= X-Gm-Gg: ASbGnctJHRaB+c9od8nh4N25+0snchrflvIA2dqgJ5MvaBd9wW7H7Ai+j4hRzlE0p5y 1mK1xYkGYYogKLypsdvRyxkqqpw1u+6GzFUofKU4FK5x3TlYdQB3IuKcNJ2HRxB5MrFxLTnYO5c UCQFoqGACnMlvzLBPqeFutrQyXYJTJ/sx8GWWprC42mp7YpxSHEtDVcIeRPMBNI9Ep0zsi/T+go qi9NB+6+/eBW4gzJknasdWtJ/WzBe5jvyVqMt0Oe/+LPhJP2VEJwfKRm+0DGebuKvk6CKR20Qwv iQ4dsJaD20DIjcdPM4KfzDqdX19mgwAgGaC2QfSlrEWmRemKsfFlXB1kXdgBZMPEFUTQNsmLPiM RNfy2RMaK3cDLYKF4NV7635nly7Bh7hYlo8VD6nYxxfqAX0Px5umukJA/MPAkgOE4NUiwEKYow8 3Ti8OfRuKfIBGdnS7Ji3fXYRht3XDc X-Google-Smtp-Source: AGHT+IEyps3iW3EcFe09ECJoqiiA0oFsrcYRToAcvOfKrTjr+Dc5uM3v/bSAgtdWrSXT5STO9xW45w== X-Received: by 2002:a05:6000:1788:b0:42b:36f4:cd21 with SMTP id ffacd0b85a97d-42f7224532cmr92975f8f.40.1764699173164; Tue, 02 Dec 2025 10:12:53 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:cae:51ce:40d7:cfd1]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1c5d613esm34883983f8f.11.2025.12.02.10.12.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Dec 2025 10:12:51 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [RFC PATCH 5/6 v8] sched/fair: Enable idle core tracking for !SMT Date: Tue, 2 Dec 2025 19:12:41 +0100 Message-ID: <20251202181242.1536213-6-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251202181242.1536213-1-vincent.guittot@linaro.org> References: <20251202181242.1536213-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable has_idle_cores at llc level feature for !SMT system for which CPU equals core. We don't enable has_idle_core feature of select_idle_cpu to be conservative and don't parse all CPUs of LLC. At now, has_idle_cores can be cleared even if a CPU is idle because of SIS_UTIL but it looks reasonnable as the probablity to get an idle CPU is low anyway. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 29 +++++++---------------------- kernel/sched/sched.h | 42 +++++++++++++++++++++++++++++------------- 2 files changed, 36 insertions(+), 35 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 252254168c92..0c0c675f39cf 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7501,19 +7501,6 @@ static inline int __select_idle_cpu(int cpu, struct = task_struct *p) return -1; } =20 -#ifdef CONFIG_SCHED_SMT -DEFINE_STATIC_KEY_FALSE(sched_smt_present); -EXPORT_SYMBOL_GPL(sched_smt_present); - -static inline void set_idle_cores(int cpu, int val) -{ - struct sched_domain_shared *sds; - - sds =3D rcu_dereference(per_cpu(sd_llc_shared, cpu)); - if (sds) - WRITE_ONCE(sds->has_idle_cores, val); -} - static inline bool test_idle_cores(int cpu) { struct sched_domain_shared *sds; @@ -7525,6 +7512,10 @@ static inline bool test_idle_cores(int cpu) return false; } =20 +#ifdef CONFIG_SCHED_SMT +DEFINE_STATIC_KEY_FALSE(sched_smt_present); +EXPORT_SYMBOL_GPL(sched_smt_present); + /* * Scans the local SMT mask to see if the entire core is idle, and records= this * information in sd_llc_shared->has_idle_cores. @@ -7612,15 +7603,6 @@ static int select_idle_smt(struct task_struct *p, st= ruct sched_domain *sd, int t =20 #else /* !CONFIG_SCHED_SMT: */ =20 -static inline void set_idle_cores(int cpu, int val) -{ -} - -static inline bool test_idle_cores(int cpu) -{ - return false; -} - static inline int select_idle_core(struct task_struct *p, int core, struct= cpumask *cpus, int *idle_cpu) { return __select_idle_cpu(core, p); @@ -7886,6 +7868,9 @@ static int select_idle_sibling(struct task_struct *p,= int prev, int target) if ((unsigned)i < nr_cpumask_bits) return i; =20 + if (!sched_smt_active()) + set_idle_cores(target, 0); + /* * For cluster machines which have lower sharing cache like L2 or * LLC Tag, we tend to find an idle CPU in the target's cluster diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 697bd654298a..b9e228333d5e 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1591,19 +1591,6 @@ do { \ flags =3D _raw_spin_rq_lock_irqsave(rq); \ } while (0) =20 -#ifdef CONFIG_SCHED_SMT -extern void __update_idle_core(struct rq *rq); - -static inline void update_idle_core(struct rq *rq) -{ - if (static_branch_unlikely(&sched_smt_present)) - __update_idle_core(rq); -} - -#else /* !CONFIG_SCHED_SMT: */ -static inline void update_idle_core(struct rq *rq) { } -#endif /* !CONFIG_SCHED_SMT */ - #ifdef CONFIG_FAIR_GROUP_SCHED =20 static inline struct task_struct *task_of(struct sched_entity *se) @@ -2091,6 +2078,35 @@ static __always_inline bool sched_asym_cpucap_active= (void) return static_branch_unlikely(&sched_asym_cpucapacity); } =20 +static inline void set_idle_cores(int cpu, int val) +{ + struct sched_domain_shared *sds; + + sds =3D rcu_dereference(per_cpu(sd_llc_shared, cpu)); + if (sds) + WRITE_ONCE(sds->has_idle_cores, val); +} + +#ifdef CONFIG_SCHED_SMT +extern void __update_idle_core(struct rq *rq); + +static inline void update_idle_core(struct rq *rq) +{ + if (static_branch_unlikely(&sched_smt_present)) + __update_idle_core(rq); + else + set_idle_cores(cpu_of(rq), 1); + +} + +#else /* !CONFIG_SCHED_SMT: */ +static inline void update_idle_core(struct rq *rq) +{ + set_idle_cores(cpu_of(rq), 1); +} +#endif /* !CONFIG_SCHED_SMT */ + + struct sched_group_capacity { atomic_t ref; /* --=20 2.43.0 From nobody Mon Feb 9 02:40:17 2026 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57C452D47E1 for ; Tue, 2 Dec 2025 18:12:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699178; cv=none; b=mdDluql9Y7RWGjzW++vXgXDQS1d7EmQV48rtvKtZvxVm8xv3E702d/4ek00RzsEZUXTEIJ9s165u/spSVe2vaS3B2x08PgMUwFBspvfXYihC5LcaSj4a0rIn10ZtqVkejLsE+ht3u6SLJt+gTdAUsvZRYvdkTzZdbwOIYNAY4H0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764699178; c=relaxed/simple; bh=UiWqb8MdrW/+gYArt01l6+/JiISmeop4oTulLHurPf4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KnJ+WB4v/5LC65ls/oiySVxmFIOhlwwMPPE8kFElOBNWaTWuFuYAV+wqGui3npi0QOuEIfa/22pYOQkcIwx2MNOQhDHng0Ap6ZXdAtkFjzZ5nqVcqJGEqmd0Po/N5fTlkiu5Zywn5nHNq3FyVeW5bz27OGtwpzNJqUq28yWm3t8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org; spf=pass smtp.mailfrom=linaro.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b=VZ4SAeGr; arc=none smtp.client-ip=209.85.221.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linaro.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="VZ4SAeGr" Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-42e2ce8681eso2466081f8f.0 for ; Tue, 02 Dec 2025 10:12:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1764699175; x=1765303975; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DbERmXmHoO6rYfqQ3qholnukozTs1hnP5I+Qfm+p/v4=; b=VZ4SAeGrczwOSpiTLcVYp3Wrof0QDqrlhtUpTJyqa9AA0p7LxkXjEavRgyNjFbKvNS zt2CJ266uW9Jjo87ZwPZX4eB24TuqgixryEsaY7UZw4Cc5zsXWFa82RlzQ5Of7ERJYsR brB0mgm6wI0u8MZvLiYUm90zqfJL+h3O9DvVxIg+pTPwFPdNUlpNQJDLlNA6SsEejOaR iqeIQfmGLdEjKZa7CpI2qtqW54PiD6OzYNN9JS2if2kVHwClf35SGRVsiI9rP54dhJ16 LgFV3KHtGTWSUwU9aEe/S8oSVBIp0fe8XMs6wGsDPNQDEcQDIl3bEOZxPYRafE0oYWWv gVmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764699175; x=1765303975; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=DbERmXmHoO6rYfqQ3qholnukozTs1hnP5I+Qfm+p/v4=; b=KlU6fzruukFJqh40ctrevV353yN0e3XTaGLlFKiUTpigWQ3ncM+5OdS71MFgRxBQvs b5bWiej4xrzUBKcEvOleDJL10exL3Kb2edf0eeJSWytUSl6fZNAhGxXoaITrKhFuwV/r tVo4IpFe/mYqbdH7yEvRW61zGumtm7QYW0/8492/906o0kg0lq+4U3WHF58+mcX1htHL Dv/pjtbNypfWTR8v1PJkqEUVCNpFILVLW1xOTg/+ax/Pb2JeKqOoWxWjn4o/tvORkkxx nVpN0/OUF+O+oxotm9Kn4lkRhuCeXQFHP0ixUqYTZH0MXA+SSgNebfI4iZ/9DktU4cd/ eANQ== X-Forwarded-Encrypted: i=1; AJvYcCXZ/IVhMCm6x0WeDyPgzaimHYz5bCxTEWgD6MH+ZfDzzGYBk+DSGHNVah/AwoM2aUN+q+TaJMNpRYp14MU=@vger.kernel.org X-Gm-Message-State: AOJu0YwAxAVu/H5yW0EvcdCsLIJQYA7n8+LuKSRT7AAtwkQ/046JENuL Qysfl8hxiNlAXWD0rgtrf/RTmlWRme88sUbH0VQ4c1g7ho17p7fb2tl97Umaf2iXeqA= X-Gm-Gg: ASbGncu6kadE8HsB7ofxgg5fuwx6CUVDLmuDaY1iP8oueMD3VIJQAHQdQ2osI0AZsC+ ZtcoJ8m5VqH/oORR8wmST76BlHyeNoq7pfRO/AsBeyZcTo1hHcKFcQNstMLzcr+GeXwlcGJnecI jfkdyon22bgcoRlAMY5BnmaF+/nV0NF4xuKL4BR+w0S+EIZC+FOmY0v8tr9DvPIJo5rDTGLv/Yc psUVZR13efOFiwDBOEyRJZ4Wn2ppj9JXeMPiJ8htNaFNVgbM3PAliKXaBS0yCbDQDNM6QM4o3Or xW93pqgVrOoMGXUB+cNDYnY6yCIRH+98h3H8NDrJiBKF4+c0pySlkWk6WtXXHaZkjUO7J3Ui0By wnHQrO3WPxmN+2GUB3IQCBLfWh0GMldn08Q7eSuTOM/hq0YkuYSakd5Skmhd1e1TBg+4IXMmpN4 AKPzZXXDoMf6pCDgxKlQ== X-Google-Smtp-Source: AGHT+IHhJ8804qhy5t6IYqIUGUiWOzSd6RcKa6FxOH9QRUq3E+E7EIlTDPnpKUMVaZM3bBVnvr+n+w== X-Received: by 2002:a05:6000:4210:b0:42b:4279:eddd with SMTP id ffacd0b85a97d-42cc1cc3084mr50479477f8f.24.1764699174630; Tue, 02 Dec 2025 10:12:54 -0800 (PST) Received: from vingu-cube.. ([2a01:e0a:f:6020:cae:51ce:40d7:cfd1]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-42e1c5d613esm34883983f8f.11.2025.12.02.10.12.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Dec 2025 10:12:53 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, linux-kernel@vger.kernel.org, pierre.gondois@arm.com, kprateek.nayak@amd.com Cc: qyousef@layalina.io, hongyan.xia2@arm.com, christian.loehle@arm.com, luis.machado@arm.com, Vincent Guittot Subject: [RFC PATCH 6/6 v8] sched/fair: Add EAS and idle cpu push trigger Date: Tue, 2 Dec 2025 19:12:42 +0100 Message-ID: <20251202181242.1536213-7-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251202181242.1536213-1-vincent.guittot@linaro.org> References: <20251202181242.1536213-1-vincent.guittot@linaro.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" EAS is based on wakeup events to efficiently place tasks on the system, but there are cases where a task doesn't have wakeup events anymore or at a far too low pace. For such cases, we check if it's worth pushing the task on another CPUs instead of putting it back in the enqueued list. Wake up events remain the main way to migrate tasks but we now detect situation where a task is stuck on a CPU by checking that its utilization is larger than the max available compute capacity (max cpu capacity or uclamp max setting). When the system becomes overutilized and some CPUs are idle, we try to push tasks instead of waiting periodic load balance. Signed-off-by: Vincent Guittot --- kernel/sched/fair.c | 64 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/topology.c | 2 ++ 2 files changed, 66 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 0c0c675f39cf..e9e1d0c05805 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8500,8 +8500,72 @@ static inline bool sched_push_task_enabled(void) return static_branch_unlikely(&sched_push_task); } =20 +static inline bool task_stuck_on_cpu(struct task_struct *p, int cpu) +{ + unsigned long max_capa, util; + + max_capa =3D min(get_actual_cpu_capacity(cpu), + uclamp_eff_value(p, UCLAMP_MAX)); + util =3D max(task_util_est(p), task_runnable(p)); + + /* + * Return true only if the task might not sleep/wakeup because of a low + * compute capacity. Tasks, which wake up regularly, will be handled by + * feec(). + */ + return (util > max_capa); +} + +static inline bool sched_energy_push_task(struct task_struct *p, struct rq= *rq) +{ + if (!sched_energy_enabled()) + return false; + + if (is_rd_overutilized(rq->rd)) + return false; + + if (task_stuck_on_cpu(p, cpu_of(rq))) + return true; + + if (!task_fits_cpu(p, cpu_of(rq))) + return true; + + return false; +} + +static inline bool sched_idle_push_task(struct task_struct *p, struct rq *= rq) +{ + if (rq->nr_running =3D=3D 1) + return false; + + if (!is_rd_overutilized(rq->rd)) + return false; + + /* If there are idle cpus in the llc then try to push the task on it */ + if (test_idle_cores(cpu_of(rq))) + return true; + + return false; +} + + static bool fair_push_task(struct rq *rq, struct task_struct *p) { + if (!task_on_rq_queued(p)) + return false; + + if (p->se.sched_delayed) + return false; + + if (p->nr_cpus_allowed =3D=3D 1) + return false; + + if (sched_energy_push_task(p, rq)) + return true; + + if (sched_idle_push_task(p, rq)) + return true; + return false; } =20 diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index cf643a5ddedd..00abd01acb84 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -391,10 +391,12 @@ static void sched_energy_set(bool has_eas) if (sched_debug()) pr_info("%s: stopping EAS\n", __func__); static_branch_disable_cpuslocked(&sched_energy_present); + static_branch_dec_cpuslocked(&sched_push_task); } else if (has_eas && !static_branch_unlikely(&sched_energy_present)) { if (sched_debug()) pr_info("%s: starting EAS\n", __func__); static_branch_enable_cpuslocked(&sched_energy_present); + static_branch_inc_cpuslocked(&sched_push_task); } } =20 --=20 2.43.0