From nobody Fri Sep 25 17:49:18 2026 Received: from m16.mail.163.com (m16.mail.163.com [117.135.210.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D33733C536 for ; Thu, 10 Sep 2026 04:31:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.4 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; cv=none; b=OERePTv1qFOhSrWfoXZ06umNS7NODmWeJyJExOYia2topE87+O39Bgmwf3VM8TWVQ2QwCG/0rC4Nkivoa+773FQXbskxDAQyiUy5akMJbDCm7JO2cQf0tNxa2imuuf9M0aTOcPQ7XIHzm+k29h1bq+nADrXyXS+tjpww3FSvnxM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; c=relaxed/simple; bh=YxICK9eRAEZm/b4KxMY6YgmHqrAXs3BXuBQsTXme5fo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=siPLCDXFl3+4/pxGnC0XkuNttD/hAGplcJjqAhXzGaXjHVAuBO+5l7LHFbpjtNnuOBIZjIJeNS5XhUCX5lQeMRGHmpVc/3dE2Rza50f7bnm7kEHeBD9aCjdoSUbiTuzeEgxykqzDS+Jnvp7LsMPlVZpkgm5VE11PkQ1UQcEsqug= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=bl1gqFjE; arc=none smtp.client-ip=117.135.210.4 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="bl1gqFjE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=mw Zx8AptPNgUqEDEdFiFTHEgRLhgM0jjpuxmqGdyRGw=; b=bl1gqFjE7XFJ3+aRq+ s0zknJqTCfYADk5COqLXf/SYES8ImH1ZGaB/YV61im/0dIq/EfYPCkGA+qmlJzO8 KQ42pkPLYdvWoqp2S9PzQ7YJ4860OgcEOpaluMKuRJSJQBG7IQipawDYA0Fg15m0 X4tM64fFFgPC/Lh0SSJh45HGY= Received: from zhaoxin-MS-7E12.. (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wAXgp5AMqJqap0YBw--.51115S3; Thu, 10 Sep 2026 12:29:55 +0800 (CST) From: Xin Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com Cc: linux-kernel@vger.kernel.org, Xin Zhao Subject: [RFC PATCH RESEND 01/10] sched/fair: Do not set_rd_overloaded() if rd->online != env->cpus Date: Thu, 10 Sep 2026 12:29:41 +0800 Message-Id: <20260910042950.1619727-2-jackzxcui1989@163.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260910042950.1619727-1-jackzxcui1989@163.com> References: <20260910042950.1619727-1-jackzxcui1989@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wAXgp5AMqJqap0YBw--.51115S3 X-Coremail-Antispam: 1Uf129KBjvJXoW7ZF1UKr1fArWUAF1UAry7KFg_yoW8Cr4Dpa yDKayIgr48twnxKasrAF4vg3yUJws5J3WYva1ak3yrJF1Yyw1YvrW0v34DCr4rKa4rZ3WS vrWUtFW7u3WjyFJanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0zK-erbUUUUU= X-CM-SenderInfo: pmdfy650fxxiqzyzqiywtou0bp/xtbC5wPf6WqiMkOBYgAA3I Content-Type: text/plain; charset="utf-8" In update_sg_lb_stats(), it only traverses sched_group that belongs to env->cpus, but env->cpus may not necessarily equal rd->online. This can lead to the incorrect clearing of the overloaded flag of rd. For example, if cpuA belongs to the online CPU mask of the rd but does not belong to env->cpus, and cpuA consistently maintains nr_running >=3D 2, while other CPUs in rd->online keep rq->nr_running <=3D 1, the overloaded flag of rd will not be set until next update of update_sd_lb_stats() for that rd. During this period, sched_balance_newidle() will prematurely return due to the incorrect assumption that the rd is in a non-overloaded state. In update_sd_lb_stats(), add a check to verify whether rd->online is equal to env->cpus before calling set_rd_overloaded() to avoid such incorrect settings. Signed-off-by: Xin Zhao --- kernel/sched/fair.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index dcf860c59a14..13e873b1ef58 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12679,8 +12679,12 @@ static inline void update_sd_lb_stats(struct lb_en= v *env, struct sd_lb_stats *sd env->fbq_type =3D fbq_classify_group(&sds->busiest_stat); =20 if (!env->sd->parent) { - /* update overload indicator if we are at root domain */ - set_rd_overloaded(env->dst_rq->rd, sg_overloaded); + /* + * Update overload indicator if we are at root domain. + * Note that env->cpus may change during sched_balance_rq(). + */ + if (cpumask_equal(env->dst_rq->rd->online, env->cpus)) + set_rd_overloaded(env->dst_rq->rd, sg_overloaded); =20 /* Update over-utilization (tipping point, U >=3D 0) indicator */ set_rd_overutilized(env->dst_rq->rd, sg_overutilized); --=20 2.34.1 From nobody Fri Sep 25 17:49:18 2026 Received: from m16.mail.163.com (m16.mail.163.com [117.135.210.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1341A33C536 for ; Thu, 10 Sep 2026 04:31:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014711; cv=none; b=B7TlSBGN3p2Ij1aDbunlLl044Mt2A4bcD3H+dwTeyJ8BPkOBrQ5GEOnrI+KOT26KMbtYU/gNeJCBdt7CtARgq+eAQ7QUyw1KA7DZwtg2FjlEFGrKJDSiNgRY4apZGDiPHoDB7uV21jAHg2ccP86OUU/jrHQ2NR0qy7pNIVxw6S8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014711; c=relaxed/simple; bh=IYwCHGvRCE0IUY7GbipIBjw2hj/OsJ7eLxHeg+OkQyg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tB4OP6VbNeBSakfRdXijYjpMpRemtgnyC2BQ0IWHIjH444ak6pFMVX7mRdhvOLu2hbVT7MHASIg3eKKXrXVxaERXQCc03Y/Uds8nCfEz71Jvu7btlgeY3u8ibhwxhI7TpYW2Lyfj9gCbPTllEnh4uoc/GoVrXnPg8APOrXdPKHQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=SmuMhkLX; arc=none smtp.client-ip=117.135.210.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="SmuMhkLX" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=wn OvNfyp/TALdwlgN291rg+EW9FopoczSlJR5mBkWNc=; b=SmuMhkLXgGfU8H+7mq 5RJ6G17x9h4s2D/kDVPCs/+djAEu0Bassr/gUTt00YfUA1gCwhwid0GZOzWqD7CQ CxHniWcKFSnnRmJe8vs81fwyxV/p6ht4YuGHSD0xkMRRB+Hk1jHqYpo3HY89HFtU GitAQF/HAK6O8w6sPUtpMymWY= Received: from zhaoxin-MS-7E12.. (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wAXgp5AMqJqap0YBw--.51115S4; Thu, 10 Sep 2026 12:29:56 +0800 (CST) From: Xin Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com Cc: linux-kernel@vger.kernel.org, Xin Zhao Subject: [RFC PATCH RESEND 02/10] scbed/fair: Remove duplicate check for busiest_cpu in active_load_balance_cpu_stop() Date: Thu, 10 Sep 2026 12:29:42 +0800 Message-Id: <20260910042950.1619727-3-jackzxcui1989@163.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260910042950.1619727-1-jackzxcui1989@163.com> References: <20260910042950.1619727-1-jackzxcui1989@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wAXgp5AMqJqap0YBw--.51115S4 X-Coremail-Antispam: 1Uf129KBjvdXoWruw1UKFWxuF43KFyrZr1kZrb_yoWkZrX_ur nrurn3Kr1jvFn09ws3CrZ3Xr1F9a4YgF18G3W0gFZrCry0qrZrKrZakFn5Xr93W3yIyFZF vwn0gF1jqw1DujkaLaAFLSUrUUUUjb8apTn2vfkv8UJUUUU8Yxn0WfASr-VFAUDa7-sFnT 9fnUUvcSsGvfC2KfnxnUUI43ZEXa7xREjjg7UUUUU== X-CM-SenderInfo: pmdfy650fxxiqzyzqiywtou0bp/xtbC5wTg6mqiMkSBhwAA3R Content-Type: text/plain; charset="utf-8" The check for cpu_active(busiest_cpu) already ensures that busiest_cpu has not gone down. An additional check for busiest_cpu !=3D smp_processor_id() is redundant. After this modification, busiest_cpu will no longer be bound to smp_processor_id(), allowing the active_load_balance_cpu_stop function to accommodate more scenarios, such as preempt active balancing feature that will be addressed in later patches. Signed-off-by: Xin Zhao --- kernel/sched/fair.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 13e873b1ef58..11c104010b2e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13769,9 +13769,7 @@ static int active_load_balance_cpu_stop(void *data) if (!cpu_active(busiest_cpu) || !cpu_active(target_cpu)) goto out_unlock; =20 - /* Make sure the requested CPU hasn't gone down in the meantime: */ - if (unlikely(busiest_cpu !=3D smp_processor_id() || - !busiest_rq->active_balance)) + if (unlikely(!busiest_rq->active_balance)) goto out_unlock; =20 /* Is there any task to move? */ --=20 2.34.1 From nobody Fri Sep 25 17:49:18 2026 Received: from m16.mail.163.com (m16.mail.163.com [117.135.210.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86E2D37E5CB for ; Thu, 10 Sep 2026 04:31:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.4 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; cv=none; b=KUN24ES+0SuBqEmIKYLVfqwI3xcpo4Jnq/TefkK5YpyrpuyG5EVt1X5UZgE5QaQ+PJpSkpWq4W6qo5r0EAJGBLPVeb1Wm48XLnnH6uk7e4ZPQeUXuQotSwCvTkmVDccaGLCxw5vEi+hBoms6QuRX0LhxeU+NpAMnuGbItu9Dipc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; c=relaxed/simple; bh=MmE42vByQegAZN4de4r4cEy2IgaWKlOiDSWRCGZyQM8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=n9R9V4V2Z9ETdnISBpjl2goIKpRwJtawlCKRxkCbKUfOUCML3Rca0v+zwx5ofAiJZi5egzfryxs8zZqZlA7yAMJzwqwJ264qoKexOznYBFciWJaQfkDnI0nlkfPGuzd8aVzYaqtBNYfvp3DV4cgDI08Z/YMyo2iSuJ/JYsCDJPg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=aMS/tKfb; arc=none smtp.client-ip=117.135.210.4 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="aMS/tKfb" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=A6 5RVDN267qKaanvkAeYvWohUstOEpo7HrQt+WE9L84=; b=aMS/tKfbHEEcW5zvYz TXCClqtH3fNm/0upcqUi/ryL06s2EnPrKAgjsQ4okzRI2aEDYDsHXq9KqptxMdnX 5U4UpRarEK0iwmL/Nk7kMlLW3aHQm7wlQPhRr/H8xRjF7CmJrSIJqjlVKapKbdVh mPDFYpW2FgB2WApMXyIExtcYs= Received: from zhaoxin-MS-7E12.. (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wAXgp5AMqJqap0YBw--.51115S5; Thu, 10 Sep 2026 12:29:58 +0800 (CST) From: Xin Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com Cc: linux-kernel@vger.kernel.org, Xin Zhao Subject: [RFC PATCH RESEND 03/10] sched/fair: Clear active_balance at the end of active_load_balance_cpu_stop() Date: Thu, 10 Sep 2026 12:29:43 +0800 Message-Id: <20260910042950.1619727-4-jackzxcui1989@163.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260910042950.1619727-1-jackzxcui1989@163.com> References: <20260910042950.1619727-1-jackzxcui1989@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wAXgp5AMqJqap0YBw--.51115S5 X-Coremail-Antispam: 1Uf129KBjvJXoW7uFy5CFW5Ar48Zr15JF1DJrb_yoW8ZF1fpr WjyayIgw4ktayYvrZ2ka18ury7uan8Jr4UGrnFqrWrZF15C3s5tw1F934xur45Zrs5CFn0 yay7tr4UCa4rGr7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07UrGYXUUUUU= X-CM-SenderInfo: pmdfy650fxxiqzyzqiywtou0bp/xtbC6Abg6mqiMkaCYgAA34 Content-Type: text/plain; charset="utf-8" The rq->active_balance flag is used to prevent multiple CPUs from simultaneously dispatching active balance stop tasks. Since there can only ever be one consumer of the stop task, it is not strictly necessary to protect the setting of rq->active_balance to 0 with the rq lock in active_load_balance_cpu_stop(). Therefore, we can move the action of clearing rq->active_balance to the end of active_load_balance_cpu_stop(). The benefit of this approach is that the task load of dst_rq will change due to the execution of attach_one_task(), which helps avoid prematurely clearing rq->active_balance before attach_one_task(), thus preventing unnecessary dispatch of duplicate active balance stop tasks. Active balance stop task is triggered only when rq->active_balance flag changes from 0 to 1, and there can be at most one consumer of active balance stop task at any given time. Therefore, we should never see zero rq->active_balance in active_load_balance_cpu_stop(), use WARN_ON_ONCE instead. Signed-off-by: Xin Zhao --- kernel/sched/fair.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 11c104010b2e..20d03ceed9d7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13769,8 +13769,7 @@ static int active_load_balance_cpu_stop(void *data) if (!cpu_active(busiest_cpu) || !cpu_active(target_cpu)) goto out_unlock; =20 - if (unlikely(!busiest_rq->active_balance)) - goto out_unlock; + WARN_ON_ONCE(!busiest_rq->active_balance); =20 /* Is there any task to move? */ if (busiest_rq->nr_running <=3D 1) @@ -13815,13 +13814,13 @@ static int active_load_balance_cpu_stop(void *dat= a) } rcu_read_unlock(); out_unlock: - busiest_rq->active_balance =3D 0; rq_unlock(busiest_rq, &rf); =20 if (p) attach_one_task(target_rq, p); =20 local_irq_enable(); + busiest_rq->active_balance =3D 0; =20 return 0; } --=20 2.34.1 From nobody Fri Sep 25 17:49:18 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B57934AB01 for ; Thu, 10 Sep 2026 04:31:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.3 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; cv=none; b=Tin1c9OXqiwMRclF4zjheDqt8CXuHD2swHwieABfs9lATKIXZqX8im++cmPl1GZH0iqWUR1dVQ9uVxu+kgEnOrZf+iy5NJ3MPmieMsBFAuFm7G3dLCp6lo6WKiaXWPHYH398mmJNv/6xsHrFgH1nIT7dVcRR0xmNf/ANWXQt4QI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; c=relaxed/simple; bh=gQ2GOV8Dmrszsdh+wQhrIb1AXtKM9wHta2VUf59x08M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=p5wlwH+Pvju2LG2+OfJ/Plnpg7i2iK8iEaa+i4x1hpqzGEDT0tLpZEcGsKOloo9k9yVN0EIcr6GfTyMHcApV+OpEMJWHZnaX4N3Wdn51FM0a49UAqgFNlsgITOXA2DCkXkYTi4X2sx+Qbpe+Iz+GrPyd9rt9aDjp+eCl7Ca71lk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=agyzdUVW; arc=none smtp.client-ip=220.197.31.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="agyzdUVW" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=T7 aj3ANDLcZnLgr2CwfVQ6lQY11yYofrsfu5AMmmt8M=; b=agyzdUVW2XyxzCXQ67 HzjkCQoNECrQ/JnFx8HSTsqtOjKg7ealzeBAZrV1AlNoetQqStXSMcWYyFQ9+14z jkT5xJV/Y8lP4K+Hwki4hct6NYH9IVhuckXOJnNJEVqdjGY8mCL8/9Zcxg8SQrPE HuC/stG3UQVqoipirCE8U3k7o= Received: from zhaoxin-MS-7E12.. (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wAXgp5AMqJqap0YBw--.51115S6; Thu, 10 Sep 2026 12:29:59 +0800 (CST) From: Xin Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com Cc: linux-kernel@vger.kernel.org, Xin Zhao Subject: [RFC PATCH RESEND 04/10] sched/fair: Add LB_PROMOTE feature to enhance real-time performance of fair tasks Date: Thu, 10 Sep 2026 12:29:44 +0800 Message-Id: <20260910042950.1619727-5-jackzxcui1989@163.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260910042950.1619727-1-jackzxcui1989@163.com> References: <20260910042950.1619727-1-jackzxcui1989@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wAXgp5AMqJqap0YBw--.51115S6 X-Coremail-Antispam: 1Uf129KBjvJXoWxurWUJw13JF45CFy5Xr13urg_yoWrGrykpa 93Xr4Yyr4DJryIy34fZr4xXr1ru393Cry3tr1kGw1xXas8Xr1ayw4IgF4UKFs3C397Za1j q3W2q343uF1jvaDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0ziqjgfUUUUU= X-CM-SenderInfo: pmdfy650fxxiqzyzqiywtou0bp/xtbC5wfg6mqiMkeBzAAA3a Content-Type: text/plain; charset="utf-8" Embedded platforms commonly use CONFIG_HZ_250, and testing has revealed that there are numerous instances of unreasonable CPU idle events on such platforms. Unreasonable CPU idle refers to situations where the CPU enters an idle state for a duration of time (t), while there are tasks that can run on the idle CPU and are not limited by cgroup constraints, yet these tasks remain unscheduled for a duration greater than (t), t > 2.5 ms. Testing has shown that over 95% of these events last less than 4 ms, but there are still some instances of longer durations between 4-5ms, even occasionally between 5-10 ms. For a real-time system, scheduling delays greater than 4 ms can lead to performance spikes. Enabling this option can effectively reduce the occurrence of unreasonable CPU idle events on low HZ systems like CONFIG_HZ_250, and completely eliminate events exceeding 4 ms. Note that the feature only affects fair tasks. Note that enabling this feature will increase sys%, as it uses CPU time that would have been idle to expedite the scheduling of tasks. There will also be some CPU overhead involved in searching for suitable tasks. This feature has been split into several smaller patches, which will be elaborated on one by one later. Below are some test data: Test one compares the number of unreasonable CPU idle events and their distribution when this feature is enabled versus when it is not, under the same fillback scenario. The test duration was 60 seconds. LB_PROMOTE(on/off) 2.5-3ms 3-4ms 4ms+ index 0 on 0 0 0 index 1 off 4 13 1 index 2 on 0 0 0 index 3 off 6 3 0 index 4 on 0 0 0 index 5 off 1 1 0 Test two compares the performance of the system with and without the feature enabled, based on the same fillback scenario. Each test lasts for 25 minutes, and a total of 15 comparative tests were conducted. The results include a comparison of the maximum and median values of end-to-end latency and sys%. LB_PROMOTE(on/off) on off end-to-end latency(max) 172 180 end-to-end latency(median of avg) 166 167.68 sys%(max) 9.68 9.35 sys%(median of avg) 8.81 8.55 Signed-off-by: Xin Zhao --- kernel/sched/features.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 8f0dee8fc475..4916a4b89ab3 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -142,3 +142,27 @@ SCHED_FEAT(LATENCY_WARN, false) */ SCHED_FEAT(NI_RANDOM, true) SCHED_FEAT(NI_RATE, true) + +/* + * Embedded platforms commonly use CONFIG_HZ_250, and testing has revealed= that + * there are numerous instances of unreasonable CPU idle events on such + * platforms. Unreasonable CPU idle refers to situations where the CPU ent= ers + * an idle state for a duration of time (t), while there are tasks that ca= n run + * on the idle CPU and are not limited by cgroup constraints, yet these ta= sks + * remain unscheduled for a duration greater than (t), t > 2.5 ms. + * + * Testing has shown that over 95% of these events last less than 4 ms, but + * there are still some instances of longer durations between 4-5ms, even + * occasionally between 5-10 ms. For a real-time system, scheduling delays + * greater than 4 ms can lead to performance spikes. + * + * Enabling this option can effectively reduce the occurrence of unreasona= ble + * CPU idle events on low HZ systems like CONFIG_HZ_250, and completely + * eliminate events exceeding 4 ms. Note that the feature only affects fair + * tasks. + * + * Note that enabling this feature will increase sys%, as it uses CPU time= that + * would have been idle to expedite the scheduling of tasks. There will al= so be + * some CPU overhead involved in searching for suitable tasks. + */ +SCHED_FEAT(LB_PROMOTE, false) --=20 2.34.1 From nobody Fri Sep 25 17:49:18 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BFAE38F248 for ; Thu, 10 Sep 2026 04:31:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.2 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; cv=none; b=o08N7pxSLe9bVsycHYlowPoChKdVZmbg80pFX8D0tNrvz66+YVl+vTP7/OwF7D86+XJI7gw3E/aCOPXy3gAlLA6uQRKQhdqzPX1+Ufqhw/HG6jcNik9YmRmun7abWwt4SrAOE53VGXVz3PAQ7OF92PG3HLsQcuxtuMQWNkBm+Sw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; c=relaxed/simple; bh=7rqDzVbAOrvGHPe3+y17ORUypnLLrLSrXFuJ0eFrblU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jJ3Zuhv7R8QCWO9tRaXGBEmcKRETrS/XURv8dEqjLbJ2MNO2e68Ez999qwPmkTbG5nHQI5kO+KsxI+fZPinNTWbOKFf8/D7T8DswoKxWXuXUxRX5QXV+FI2LNai8HAb/Fzb7SpS5UxbWvS2i9Cs9IzRLfoLXZ/FRtQsAlcZdUx8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=Ae3UiVv6; arc=none smtp.client-ip=220.197.31.2 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="Ae3UiVv6" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=4t nSggQ8oN3baWQ2VgtsX9D1878Stv467Btkc2KmqxQ=; b=Ae3UiVv6U5La4d87WL HBy8D0gxOm+OZVpT9V9IA5EFBE9SjQiRM1i0s6JMRSWZjfS/rCFeL/nkpnDObxHL fOJVotGqNGBDGpkd4be3gg6dPdK3LCdeEwHaDSvxZ7/RmjQN0dRTjgXH1ZnUcZOk SmpWuuUFtZ2Ouww83hBkD5NyI= Received: from zhaoxin-MS-7E12.. (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wAXgp5AMqJqap0YBw--.51115S7; Thu, 10 Sep 2026 12:30:00 +0800 (CST) From: Xin Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com Cc: linux-kernel@vger.kernel.org, Xin Zhao Subject: [RFC PATCH RESEND 05/10] sched/fair: Introduce select_task_rq_fair_thin() to select rq when LB_PROMOTE Date: Thu, 10 Sep 2026 12:29:45 +0800 Message-Id: <20260910042950.1619727-6-jackzxcui1989@163.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260910042950.1619727-1-jackzxcui1989@163.com> References: <20260910042950.1619727-1-jackzxcui1989@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wAXgp5AMqJqap0YBw--.51115S7 X-Coremail-Antispam: 1Uf129KBjvJXoWxKF4DZF1rGr4UGr1rtFy3twb_yoW7Xr15pF 4Fqw13trsrAw4Ivw1fCrs7Cr1Yv34rGw47Kr1fJF95Ca45Xryv9F1FgrnxXFyfCr1kZFyj qFW8Kr17CrWqvaUanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0ziDDGcUUUUU= X-CM-SenderInfo: pmdfy650fxxiqzyzqiywtou0bp/xtbC5wjh62qiMkiB6wAA39 Content-Type: text/plain; charset="utf-8" The logic of select_task_rq_fair() is relatively complex, and on typical embedded systems, the number of CPUs in sd_llc domain is often limited to a maximum of 4, and sometimes only 2. This makes the complex logic of select_task_rq_fair() seem less necessary. Additionally, the update logic for the nr_idle_scan value that select_task_rq_fair() relies on is also quite time-consuming. Moreover, embedded systems require better real-time performance, and with fewer CPUs available, it becomes necessary to bind certain tasks across the sd_llc range. The default enabled feature, SD_WAKE_AFFINE, causes select_task_rq_fair() to take the fast path, often overlooking some idle CPUs across sd_llc domain, leading to increased scheduling latency. To address this, we introduce the select_task_rq_fair_thin() function, which serves as a streamlined version of select_task_rq_fair(). It can quickly perform CPU selection while also considering the real-time requirements of embedded systems. select_task_rq_fair_thin() retains the priority selection logic for prev_cpu and recent_used_cpu, and it will prioritize CPUs within the sd_llc. If there are no idle CPUs in the sd_llc, it will then look for other available CPUs to run. When the LB_PROMOTE feature is enabled, select_task_rq_fair_thin() will replace the original select_task_rq_fair(), and the update logic for nr_idle_scan that select_task_rq_fair() relies on will no longer need to be executed. Testing has shown that in our system with 18 CPUs running at 2.1GHz, where the first three sd_llc domains each contains 4 CPUs and the last sd_llc contains 2 CPUs, under same fillback scenario, select_task_rq_fair_thin() executes 25% faster than the original select_task_rq_fair(). It saves 22ms over a 10-second period, with this optimization accounting for 0.174% of total system time. Additionally, we measured the execution time of update_idle_cpu_scan, which took 0.5ms over the same 10-second period. If we use select_task_rq_fair_thin() instead, this time can be eliminated, accounting for 0.04% of total system time. Therefore, the overall optimization contributes to a reduction of 0.214% of total system time. Signed-off-by: Xin Zhao --- kernel/sched/fair.c | 52 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 20d03ceed9d7..f10e709921fd 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9661,6 +9661,51 @@ static int find_energy_efficient_cpu(struct task_str= uct *p, int prev_cpu) return target; } =20 +/* + * A streamlined version of select_task_rq_fair(). + * It runs faster than select_task_rq_fair, especially when there are not + * many CPUs. It will prioritize selecting an idle CPU in the following or= der: + * 1. prev_cpu + * 2. recent_used_cpu + * 3. cpu belongs to intersection of sd_llc and cpus_ptr + * 4. cpu belongs to cpus_ptr but not belongs to sd_llc + * If there is no idle CPU in cpus_ptr, it will select prev_cpu. + */ +static int select_task_rq_fair_thin(struct task_struct *p, int prev_cpu, i= nt wake_flags) +{ + int recent_used_cpu, target, cpu, start =3D nr_cpu_ids; + struct sched_domain *sd; + + if (likely(available_idle_cpu(prev_cpu))) + return prev_cpu; + + recent_used_cpu =3D p->recent_used_cpu; + p->recent_used_cpu =3D prev_cpu; + if (recent_used_cpu !=3D prev_cpu && available_idle_cpu(recent_used_cpu)) + return recent_used_cpu; + + target =3D prev_cpu; + rcu_read_lock(); + + sd =3D rcu_dereference(per_cpu(sd_llc, target)); + if (sd) + start =3D cpumask_first_and(sched_domain_span(sd), p->cpus_ptr); + if (start >=3D nr_cpu_ids) + start =3D cpumask_first(p->cpus_ptr); + + for_each_cpu_wrap(cpu, p->cpus_ptr, start) { + if (available_idle_cpu(cpu)) { + target =3D cpu; + goto unlock; + } + } + +unlock: + rcu_read_unlock(); + + return target; +} + /* * select_task_rq_fair: Select target runqueue for the waking task in doma= ins * that have the relevant SD flag set. In practice, this is SD_BALANCE_WAK= E, @@ -9682,6 +9727,9 @@ select_task_rq_fair(struct task_struct *p, int prev_c= pu, int wake_flags) /* SD_flags and WF_flags share the first nibble */ int sd_flag =3D wake_flags & 0xF; =20 + if (sched_feat(LB_PROMOTE)) + return select_task_rq_fair_thin(p, prev_cpu, wake_flags); + /* * required for stable ->cpus_allowed */ @@ -12564,8 +12612,10 @@ static void update_idle_cpu_scan(struct lb_env *en= v, * So the write of this hint only occurs during periodic load * balancing, rather than CPU_NEWLY_IDLE, because the latter * can fire way more frequently than the former. + * When LB_PROMOTE is enabled, select_task_rq_fair() is no longer + * used, and there is no need to update nr_idle_scan. */ - if (!sched_feat(SIS_UTIL) || env->idle =3D=3D CPU_NEWLY_IDLE) + if (!sched_feat(SIS_UTIL) || env->idle =3D=3D CPU_NEWLY_IDLE || sched_fea= t(LB_PROMOTE)) return; =20 sd_share =3D sd->shared; --=20 2.34.1 From nobody Fri Sep 25 17:49:18 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF6483AE6EB for ; Thu, 10 Sep 2026 04:31:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014681; cv=none; b=QX89vzzmdocMJnHwq+h0MFrvOf69XeTc2RAtIzeGpPB6UudDrJufsCPbfVjaGHkjaVmyNvTwHCELkwfn6AKjChTc7ZHrXcaM8NaPDwguL7Ps7mPu8YqD8F2p6Rw9PHuaFoNKtOfQ0S0rJ+5zwYPx1QI2Q1eZHR3z87zanambzMU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014681; c=relaxed/simple; bh=9kEKGitU0KGtrdmgg+MeYLoaKIghma4epy46ju6KunM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=okmFt54gGaLJDcY7/hbhH3MSxaMjSqL85Vot7gzuHbWECj9ZhddbM49i0ZkWUatlY3uSRZ55gVp/+8H545PQ2o+mqOCpPhyesU/9C8M7SQplNr5osZLQe6taZZMsswOJGvatN2VGyR0Yc06taxTJ4HEzjQHaSdmk2lEasSOcQPI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=ObaLzogi; arc=none smtp.client-ip=220.197.31.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="ObaLzogi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=2p URnNgagNSzAGeA4erfdSykdqLoBlr6NUXAPfZFKc4=; b=ObaLzogiP9kGptYSMD DDWODHSGqZrNkfOyHEQGQQjc/z4+uJGfOHFVgpvz8LIWoqRJRwU3CMqgBT9KCi9D PNAAaD3i3nKpRbDzZdw6riUtWorq1Tzd6cSFxSvr485bZ1hO9qBtZdcsIH84zb13 RhS0SL2zmVu2AUh22RPKSFySE= Received: from zhaoxin-MS-7E12.. (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wAXgp5AMqJqap0YBw--.51115S8; Thu, 10 Sep 2026 12:30:01 +0800 (CST) From: Xin Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com Cc: linux-kernel@vger.kernel.org, Xin Zhao Subject: [RFC PATCH RESEND 06/10] sched/fair: Modify active_load_balance_cpu_stop() to accommodate more scenarios Date: Thu, 10 Sep 2026 12:29:46 +0800 Message-Id: <20260910042950.1619727-7-jackzxcui1989@163.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260910042950.1619727-1-jackzxcui1989@163.com> References: <20260910042950.1619727-1-jackzxcui1989@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wAXgp5AMqJqap0YBw--.51115S8 X-Coremail-Antispam: 1Uf129KBjvJXoWxWr47Gw1xtr4kGrWUAr1rtFb_yoW5KF48pr WUKw4agw4DJayUZ390yF4xur1IgwsxWw47Jan2qF4rCF15G3s5tw1F9r1fuF45uFZ5CFna ya1Dtw47CFWUtFUanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0pEDGYhUUUUU= X-CM-SenderInfo: pmdfy650fxxiqzyzqiywtou0bp/xtbC5wrh62qiMkqCDAAA3Z Content-Type: text/plain; charset="utf-8" Modify active_load_balance_cpu_stop() to make it more generalized, no longer limited to the context of finding the busiest rq in the sched_balance_rq() scenario to migrate to the dst rq. Change the variables that start with 'busiest' to start with 'src'. Additionally, adjust the reference to the active_balance variable from busiest_rq->active_balance to this_rq()->active_balance. With these changes, the CPU where the stop task is located can serve as either a dst CPU or a src CPU, laying the groundwork for the upcoming preempt active balance feature. Signed-off-by: Xin Zhao --- kernel/sched/fair.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f10e709921fd..9ffd01717599 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13796,33 +13796,33 @@ update_next_balance(struct sched_domain *sd, unsi= gned long *next_balance) =20 /* * active_load_balance_cpu_stop is run by the CPU stopper. It pushes - * running tasks off the busiest CPU onto idle CPUs. It requires at + * one running task off the src CPU onto dst CPU. It requires at * least 1 task to be running on each physical CPU where possible, and * avoids physical / logical imbalances. */ static int active_load_balance_cpu_stop(void *data) { - struct rq *busiest_rq =3D data; - int busiest_cpu =3D cpu_of(busiest_rq); - int target_cpu =3D busiest_rq->push_cpu; + struct rq *src_rq =3D data; + int src_cpu =3D cpu_of(src_rq); + int target_cpu =3D this_rq()->push_cpu; struct rq *target_rq =3D cpu_rq(target_cpu); struct sched_domain *sd; struct task_struct *p =3D NULL; struct rq_flags rf; =20 - rq_lock_irq(busiest_rq, &rf); + rq_lock_irq(src_rq, &rf); /* * Between queueing the stop-work and running it is a hole in which * CPUs can become inactive. We should not move tasks from or to * inactive CPUs. */ - if (!cpu_active(busiest_cpu) || !cpu_active(target_cpu)) + if (!cpu_active(src_cpu) || !cpu_active(target_cpu)) goto out_unlock; =20 - WARN_ON_ONCE(!busiest_rq->active_balance); + WARN_ON_ONCE(!this_rq()->active_balance); =20 /* Is there any task to move? */ - if (busiest_rq->nr_running <=3D 1) + if (src_rq->nr_running <=3D 1) goto out_unlock; =20 /* @@ -13830,12 +13830,12 @@ static int active_load_balance_cpu_stop(void *dat= a) * we need to fix it. Originally reported by * Bjorn Helgaas on a 128-CPU setup. */ - WARN_ON_ONCE(busiest_rq =3D=3D target_rq); + WARN_ON_ONCE(src_rq =3D=3D target_rq); =20 /* Search for an sd spanning us and the target CPU. */ rcu_read_lock(); for_each_domain(target_cpu, sd) { - if (cpumask_test_cpu(busiest_cpu, sched_domain_span(sd))) + if (cpumask_test_cpu(src_cpu, sched_domain_span(sd))) break; } =20 @@ -13844,14 +13844,14 @@ static int active_load_balance_cpu_stop(void *dat= a) .sd =3D sd, .dst_cpu =3D target_cpu, .dst_rq =3D target_rq, - .src_cpu =3D busiest_rq->cpu, - .src_rq =3D busiest_rq, + .src_cpu =3D src_rq->cpu, + .src_rq =3D src_rq, .idle =3D CPU_IDLE, .flags =3D LBF_ACTIVE_LB, }; =20 schedstat_inc(sd->alb_count); - update_rq_clock(busiest_rq); + update_rq_clock(src_rq); =20 p =3D detach_one_task(&env); if (p) { @@ -13864,13 +13864,13 @@ static int active_load_balance_cpu_stop(void *dat= a) } rcu_read_unlock(); out_unlock: - rq_unlock(busiest_rq, &rf); + rq_unlock(src_rq, &rf); =20 if (p) attach_one_task(target_rq, p); =20 local_irq_enable(); - busiest_rq->active_balance =3D 0; + this_rq()->active_balance =3D 0; =20 return 0; } --=20 2.34.1 From nobody Fri Sep 25 17:49:18 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 662B93AAF57 for ; Thu, 10 Sep 2026 04:31:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.4 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; cv=none; b=lDmP+JAYPI4/PHUI65yMZ6F8qnRBJHchz/l2RpQH2FFL7yUYhMnPVRGuAt+oTkh3BLOYrePVMeqn5/xb1JY+feHELaF/A9NC9wumz+AZx58CMHsMn1evcCaNTAENDbaWpNkD1PVHisPMasXYA5VWLWptSs9nHFYxt0x1Q8A0I3A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; c=relaxed/simple; bh=wrEPrEhzPywTkV5zFcDwTZrPWZGDw0UPGGLR4mG1UIw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aSUnSVdVBRVdnF+thfr4Bhn2ONVTQ01aL+NG9P8EHruokgCRi5/ogwouHm36HnvYjPZzPqhNS86lX6JIdFVgD4l+GOShQ7QfjD9iHGSdQEsNNQkCxDrnJXweBgHnz49IMtf6ZzuR9B871Vcpkpnr0LHPN8fioXVqPPsjFCKQAjo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=FsIDDZR7; arc=none smtp.client-ip=220.197.31.4 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="FsIDDZR7" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=tX BLjoyVcqoSnCe5hTKnzGr+xFkpChqXdUEH+D46IKk=; b=FsIDDZR71g4x3myDUf mOmBD6PhW1IeOqc4w+O/YzKy0O8EkAP6HC1WxSIRVB1bJHvcDf5S/OTZ0LdMgdfv QSmEAGkKFqiZ+jqZwjT1mOtOMGEW+pXvBVeCSZWTa8M5Mjjd/SGlVx3ODuJ6jfm9 0XBEPl/D/hqsbG9yNXoeVno4Q= Received: from zhaoxin-MS-7E12.. (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wAXgp5AMqJqap0YBw--.51115S9; Thu, 10 Sep 2026 12:30:03 +0800 (CST) From: Xin Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com Cc: linux-kernel@vger.kernel.org, Xin Zhao Subject: [RFC PATCH RESEND 07/10] sched/fair: Trigger active balance if a CFS task is preempted when LB_PROMOTE Date: Thu, 10 Sep 2026 12:29:47 +0800 Message-Id: <20260910042950.1619727-8-jackzxcui1989@163.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260910042950.1619727-1-jackzxcui1989@163.com> References: <20260910042950.1619727-1-jackzxcui1989@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wAXgp5AMqJqap0YBw--.51115S9 X-Coremail-Antispam: 1Uf129KBjvJXoW3XF4fJry5KF15Zry7CFW3Jrb_yoW7trWfpr Wvya4rJa1DJ3W2q3ySkr48Zr13Wwn5Jayxtan7Jw4rAF15K34rt3Zavw13Wr45Crn5uF4a yr1jv3y2k3W8JF7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0ziqYLQUUUUU= X-CM-SenderInfo: pmdfy650fxxiqzyzqiywtou0bp/xtbC6Avh62qiMkuC8AAA3q Content-Type: text/plain; charset="utf-8" When LB_PROMOTE is on, in addition to the scenarios modified in the previous patches regarding .select_task_rq implement, preemption of CFS tasks also needs to be addressed. Once a CFS task is preempted by a real-time task, if that task is executing logic in a critical section that does not support priority inheritance, such as read-write locks or read-write semaphores, it may lead to performance issues. By adding this checkpoint for when a CFS task is preempted, we can migrate the task to an idle CPU, thus alleviating such performance problems. Additionally, the patch set for the LB_PROMOTE feature does not alter the logic of task_hot(), so this patch will not significantly increase the frequency of load balance migration. The newly introduced function preempt_active_balance() will be triggered in __schedule() when it detects that the previous task has been preempted while the LB_PROMOTE feature is enabled. preempt_active_balance() will first check whether the currently preempted task has other CPUs available to run on. If there are idle CPUs available, the preempted task will be migrated to one of those idle CPUs. The implementation of preempt_active_balance() considers the desire not to introduce additional "holes" of rq locks, so the migration triggering action is deferred to the balance_callback which is trigger_preempt_alb(). trigger_preempt_alb() will issue a stop work to allow the idle target CPU to perform the migration. After the modifications made by the previous patches, we can reuse active_load_balance_cpu_stop() to assist in this migration action. In preempt_active_balance(), we use list_move_tail to move the currently preempted task to position where detach_one_task() will first traverse, allowing active_load_balance_cpu_stop() to prioritize finding the task. Signed-off-by: Xin Zhao --- kernel/sched/core.c | 3 ++ kernel/sched/fair.c | 80 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 2 ++ 3 files changed, 85 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 2e7cde033a31..757ea5ef303d 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7227,6 +7227,9 @@ static void __sched notrace __schedule(int sched_mode) =20 trace_sched_switch(preempt, prev, next, prev_state); =20 + if (sched_feat(LB_PROMOTE) && preempt) + preempt_active_balance(prev); + /* Also unlocks the rq: */ rq =3D context_switch(rq, prev, next, &rf); } else { diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9ffd01717599..d2b538abdf3e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13875,6 +13875,86 @@ static int active_load_balance_cpu_stop(void *data) return 0; } =20 +static DEFINE_PER_CPU(struct balance_callback, preempt_push_head); +static DEFINE_PER_CPU(int, push_cpu); + +static void trigger_preempt_alb(struct rq *rq) +{ + unsigned long flags; + int active_balance =3D 0; + int src_cpu =3D rq->cpu; + int idle_cpu =3D per_cpu(push_cpu, src_cpu); + struct rq *dst_rq =3D cpu_rq(idle_cpu); + + preempt_disable(); + raw_spin_rq_unlock(rq); + raw_spin_rq_lock_irqsave(dst_rq, flags); + if (dst_rq->active_balance) + goto unlock_rq; + dst_rq->active_balance =3D 1; + dst_rq->push_cpu =3D idle_cpu; + active_balance =3D 1; + +unlock_rq: + raw_spin_rq_unlock_irqrestore(dst_rq, flags); + if (active_balance) { + stop_one_cpu_nowait(idle_cpu, active_load_balance_cpu_stop, rq, + &dst_rq->active_balance_work); + } + preempt_enable(); + raw_spin_rq_lock(rq); +} + +static void queue_preempt_alb_callback(struct rq *rq, int dst_cpu) +{ + per_cpu(push_cpu, rq->cpu) =3D dst_cpu; + queue_balance_callback(rq, &per_cpu(preempt_push_head, rq->cpu), trigger_= preempt_alb); +} + +void preempt_active_balance(struct task_struct *prev) +{ + struct rq *this_rq =3D this_rq(); + int this_cpu =3D smp_processor_id(), cpu; + int start =3D nr_cpu_ids, idle_cpu =3D nr_cpu_ids; + struct sched_domain *sd; + + if (unlikely(prev->sched_class !=3D &fair_sched_class)) + return; + + if (unlikely(prev->migration_disabled)) + return; + + cpu =3D prev->recent_used_cpu; + if (cpu !=3D this_cpu && available_idle_cpu(cpu) && !cpu_rq(cpu)->active_= balance) { + idle_cpu =3D cpu; + goto active_balance; + } + + rcu_read_lock(); + + sd =3D rcu_dereference(per_cpu(sd_llc, this_cpu)); + if (sd) + start =3D cpumask_first_and(sched_domain_span(sd), prev->cpus_ptr); + if (start >=3D nr_cpu_ids) + start =3D cpumask_first(prev->cpus_ptr); + + for_each_cpu_wrap(cpu, prev->cpus_ptr, start) { + if (cpu !=3D this_cpu && available_idle_cpu(cpu) && !cpu_rq(cpu)->active= _balance) { + idle_cpu =3D cpu; + goto unlock; + } + } + +unlock: + rcu_read_unlock(); + if (idle_cpu =3D=3D nr_cpu_ids) + return; + +active_balance: + list_move_tail(&prev->se.group_node, &this_rq->cfs_tasks); + queue_preempt_alb_callback(this_rq, idle_cpu); +} + /* * Scale the max sched_balance_rq interval with the number of CPUs in the = system. * This trades load-balance latency on larger machines for less cross talk. diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 26ae13c86b69..11848708e5ce 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -4193,6 +4193,8 @@ extern struct balance_callback *splice_balance_callba= cks(struct rq *rq); extern void __balance_callbacks(struct rq *rq, struct rq_flags *rf); extern void balance_callbacks(struct rq *rq, struct balance_callback *head= ); =20 +extern void preempt_active_balance(struct task_struct *prev); + /* * The 'sched_change' pattern is the safe, easy and slow way of changing a * task's scheduling properties. It dequeues a task, such that the schedul= er --=20 2.34.1 From nobody Fri Sep 25 17:49:18 2026 Received: from m16.mail.163.com (m16.mail.163.com [117.135.210.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70EFD3AE1A8 for ; Thu, 10 Sep 2026 04:31:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.3 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; cv=none; b=PyIr2Hw+b6ZCXNG+hgi2XktcMDALQ0+1xhpSGzYv2S360T1/R5YtxbGU+O+01d6FFiInwOxzppmkfIpaiKI3aRXcpvi7hAQIOuOowmisHF9m5fbHl8xMqOjN/2c6ZL9HgEUTGg2eGg+ODrarOKFXnALoCuG0HZioHQSQrKZuL64= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; c=relaxed/simple; bh=tXOb2gM5KmdToeaqxMTKhvq6W4aDrf8KUKnU0DE+TKg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KGpAvDmp33OFphs8cNCuoOtbsQCZDJPV/nhjo6oR4guFeCvmsJb2pBwyiCtgIdrbGFpnQAlBzErsqNLf0mWKNUDTCxJg4vcFCAgaXbmy23iBJqmrkQdnnY0PGy21BpZi3tWFEqkDHhj4i7DGzMvuqXb3VI6JRlS0H163V/LBk4g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=NZcENpY4; arc=none smtp.client-ip=117.135.210.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="NZcENpY4" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=Iz Q9B9qyBge0iH1jK1v8qnI3vqWt1L1DDgzup5RG3Pw=; b=NZcENpY4x4FwZAPicl 0DhL5AdUbSxNYhTOi2PAH6qQQWpchmPhlVDGtyHauIi133JQWl+kxoea9hQr+LAC QBaqFavNmi5bS/sOha574I9jS6mXco2UCCQ3l+lw4YCu49aCygreF0fAk5QCEXGJ 5Tmw3hJ4WagVSmXf0d+8pcwnQ= Received: from zhaoxin-MS-7E12.. (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wAXgp5AMqJqap0YBw--.51115S10; Thu, 10 Sep 2026 12:30:04 +0800 (CST) From: Xin Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com Cc: linux-kernel@vger.kernel.org, Xin Zhao Subject: [RFC PATCH RESEND 08/10] sched/fair: Do not check avg_idle to prematurely exit newly idle when LB_PROMOTE Date: Thu, 10 Sep 2026 12:29:48 +0800 Message-Id: <20260910042950.1619727-9-jackzxcui1989@163.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260910042950.1619727-1-jackzxcui1989@163.com> References: <20260910042950.1619727-1-jackzxcui1989@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wAXgp5AMqJqap0YBw--.51115S10 X-Coremail-Antispam: 1Uf129KBjvJXoW7WF4fCw4xJFW8Xry8Xr18Krg_yoW8Xw4fpr ZYyan5Cr1qq3Z5Ja4kAF4kWw1agwsrJa43WF1Iy343Jr15J34Fv3WSqa43WFWfCryrCF4a vr1jqw12k3W0grJanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0ziDDGcUUUUU= X-CM-SenderInfo: pmdfy650fxxiqzyzqiywtou0bp/xtbC6Azi7GqiMkyDEQAA3O Content-Type: text/plain; charset="utf-8" When LB_PROMOTE is on, there are high real-time requirements, we should avoid prematurely exiting in sched_balance_newidle() due to a short avg_idle. This is because after any newly idle state, if no tasks are found to run, there may be a long-term lack of new incoming tasks to wake up on that CPU. Meanwhile, there may be tasks waiting to run on other cores, leading to unreasonable CPU idleness. The definition of unreasonable CPU idleness can be found in the commit-log of the previous patch that introduced the LB_PROMOTE feature. Signed-off-by: Xin Zhao --- kernel/sched/fair.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d2b538abdf3e..8a1d2763a923 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -14694,7 +14694,7 @@ static int sched_balance_newidle(struct rq *this_rq= , struct rq_flags *rf) goto out; =20 if (!get_rd_overloaded(this_rq->rd) || - this_rq->avg_idle < sd->max_newidle_lb_cost) { + (!sched_feat(LB_PROMOTE) && this_rq->avg_idle < sd->max_newidle_lb_co= st)) { =20 update_next_balance(sd, &next_balance); goto out; @@ -14716,7 +14716,8 @@ static int sched_balance_newidle(struct rq *this_rq= , struct rq_flags *rf) =20 update_next_balance(sd, &next_balance); =20 - if (this_rq->avg_idle < curr_cost + sd->max_newidle_lb_cost) + if (!sched_feat(LB_PROMOTE) && + this_rq->avg_idle < curr_cost + sd->max_newidle_lb_cost) break; =20 if (sd->flags & SD_BALANCE_NEWIDLE) { --=20 2.34.1 From nobody Fri Sep 25 17:49:18 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 616143A0B2E for ; Thu, 10 Sep 2026 04:31:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.2 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; cv=none; b=AsfxjRb0m3WI6qW5GmC5SYDChul0rDEf1ymr++00ft0IitPbfBE10C0Tifowk56J4jLHbUyd9QwN/+6d3q8bVn4RbOj4MxAKqtkK8KwnwtuCXY2o6duRXtjWe2LEHYCabFUsVpUvg9lJmTjBfJVTnDlVzNvYIDoMjQ/Ik7WnjQw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014680; c=relaxed/simple; bh=KsIoKJWnFpTP1QqcLS2ahGyzsxSWbGwq020JU74xWZg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Sbtld54k5wEtPHWhPOTkM3W55UiTdF24RXRX4U2KHKCTnYkLOD+QUr+oOm1I+ksdioclwrNjqqKK+ZskmR6XiA4PDXhg09fFnmfG7PHbPvnbuhiIFj3p1RcpYJBuxobu7pHnFdG/uNTJsMDFs0FlZsdPCa+s8tdok0ffuutxoUE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=FO3VO3u8; arc=none smtp.client-ip=220.197.31.2 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="FO3VO3u8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=KT QU5wwOgoyx1VpyqCpFm3lgO2ZriMIQJJ0cehW7kzQ=; b=FO3VO3u84DpcA5aPQG n04P55yuOE8NfY9B+Q1VqCeVkwP9b4/mEaTmLV1pn1PsY6tLWRMxxpFDIHlSqrOK wdiGbzyDCuqlxOejsXx3dqaLwnngfSDJSnDYrMO3PLUVaYNZ32DiGrkju7m4W+Xg K/OMKbbcbS28hYmP1RwsGXU6U= Received: from zhaoxin-MS-7E12.. (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wAXgp5AMqJqap0YBw--.51115S11; Thu, 10 Sep 2026 12:30:06 +0800 (CST) From: Xin Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com Cc: linux-kernel@vger.kernel.org, Xin Zhao Subject: [RFC PATCH RESEND 09/10] sched/fair: Not goto more_balance if newly idle and has pending task when LBF_NEED_BREAK Date: Thu, 10 Sep 2026 12:29:49 +0800 Message-Id: <20260910042950.1619727-10-jackzxcui1989@163.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260910042950.1619727-1-jackzxcui1989@163.com> References: <20260910042950.1619727-1-jackzxcui1989@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wAXgp5AMqJqap0YBw--.51115S11 X-Coremail-Antispam: 1Uf129KBjvJXoW7KF43KF4rWF1DWF4rXw4UXFb_yoW8Jw4xpr Wv9F45Za1qq345A39ayF48ur15uw1Sk398uFZrArWfJrn0qFWjvFZYga9xWFWjvFykA3Wr ZF1jg3429340yF7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0ziqYLQUUUUU= X-CM-SenderInfo: pmdfy650fxxiqzyzqiywtou0bp/xtbCvw7i7GqiMk7xdgAA3M Content-Type: text/plain; charset="utf-8" When LBF_NEED_BREAK flag is set in env.flags, no longer unconditionally go to more_balance. Instead, we exclude the case when it is newly idle and there are pending tasks. This helps avoid unnecessary CPU wastage caused by repeatedly going to more_balance when the task load is too high during sched_balance_rq(). In another 'goto more_balance' case when LBF_DST_PINNED flag is set in env.flags, we do not need to add the check. Because LBF_DST_PINNED flag is only set within can_migrate_task(). Before setting LBF_DST_PINNED flag in can_migrate_task(), there is a check to see if it is newly idle. can_migrate_task() will exit without setting the LBF_DST_PINNED flag if it is newly idle. Signed-off-by: Xin Zhao --- kernel/sched/fair.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8a1d2763a923..1ae351ea5949 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13561,7 +13561,9 @@ static int sched_balance_rq(int this_cpu, struct rq= *this_rq, =20 if (env.flags & LBF_NEED_BREAK) { env.flags &=3D ~LBF_NEED_BREAK; - goto more_balance; + if (!(env.idle =3D=3D CPU_NEWLY_IDLE && + (env.dst_rq->nr_running > 0 || env.dst_rq->ttwu_pending))) + goto more_balance; } =20 /* --=20 2.34.1 From nobody Fri Sep 25 17:49:18 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 59D75379C24 for ; Thu, 10 Sep 2026 04:31:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014717; cv=none; b=MNILQZIS1bLv2p4AtpOlt2lQO2M/x56TzWy8KZqKLvTMHjHm0CEktLxy6z9OEfqrrI4jeGj/2OAVAMit6ltP31bVQAgIfHxxB0yyEfH1NztrhdmCOkbasiItq3S5RIlq00Olr8eBRuZFtZH36ppFKfVzKrCgBtbiPkzGrV1OVAw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789014717; c=relaxed/simple; bh=JyuRVYUXENtFmSue4479W+r7Lt9Y/8EY2DF4juzQW6k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tdWiHvP/OaamovBcYmSIToo1HKHYgkHvJ6I/8+RRn9anvpRBQmSTPqM7AudviQExfYs4B3jJO05CbgPkRWCoaRrTdldbr+Uc1OUkEsN1f3ACgcm4qdVRoIjf80GvF/isSLcaJvf4zuSL33yLCvLnp89HG6r6tVDUyWA4l4XZbag= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=BEsuT74h; arc=none smtp.client-ip=220.197.31.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="BEsuT74h" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=L7 zCxf1hzQ1p+u7zu+dIehAq8l0mgUUnBoS0+XuNGiQ=; b=BEsuT74h3QY57fEmrt 37jO6gpiixKu3vp/HtiTJMaddvBso/nZPH2cwWkbZnK2FMKOrMIsPuJamDEsJcbL uvBXNItzwq4ZYCg6utM+BhtiRGgmrzmRbdGT2QWh1JIRrCsShW15wM5rntSt2Q2/ SRqnfR8E9OpS8VS5s52e9ma7s= Received: from zhaoxin-MS-7E12.. (unknown []) by gzga-smtp-mtada-g0-3 (Coremail) with SMTP id _____wAXgp5AMqJqap0YBw--.51115S12; Thu, 10 Sep 2026 12:30:07 +0800 (CST) From: Xin Zhao To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, kprateek.nayak@amd.com Cc: linux-kernel@vger.kernel.org, Xin Zhao Subject: [RFC PATCH RESEND 10/10] sched/fair: Strive to find a task to migrate if newly idle when LB_PROMOTE Date: Thu, 10 Sep 2026 12:29:50 +0800 Message-Id: <20260910042950.1619727-11-jackzxcui1989@163.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260910042950.1619727-1-jackzxcui1989@163.com> References: <20260910042950.1619727-1-jackzxcui1989@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wAXgp5AMqJqap0YBw--.51115S12 X-Coremail-Antispam: 1Uf129KBjvJXoWxCr1DWw47WryDtw43GrWDXFb_yoWrAr47pr ZY9a1rKa1Dtw13t3sIkFsrZr1agws7Xr47JFZ7Jr1fCr45J34aqrnaqa43AF4rurs5Zr1a vrnrKw1j9w17trJanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0zigdbnUUUUU= X-CM-SenderInfo: pmdfy650fxxiqzyzqiywtou0bp/xtbC5w-i7GqiMk+CsgAA3j Content-Type: text/plain; charset="utf-8" This patch is a core modification within this patch set. The simple_find label in this patch implements a straightforward logic for finding a migration task. It allows any instances in sched_balance_rq() that are unable to find a migration task for various reasons to fallback to logic executing the simple_find label to find one before exiting newly idle process. This patch addresses the newly idle scenario by preventing early exit in cases when !ld_moved && !active_balance, effectively executing the logic of simple_find label. Testing has shown that the situations listed below, account for a significant proportion of early exits: 1. Failure in sched_balance_find_src_group 2. Failure in sched_balance_find_src_rq 3. ld_moved is 0 and active_balance has not been triggered Of course, even with this change, it is still possible that no migration task can be found. However, it at least ensures that all selectable CPUs within the sched_domain have been thoroughly traversed. Signed-off-by: Xin Zhao --- kernel/sched/fair.c | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1ae351ea5949..10ec7bb9e18c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13470,6 +13470,8 @@ static int sched_balance_rq(int this_cpu, struct rq= *this_rq, struct rq *busiest; struct rq_flags rf; struct cpumask *cpus =3D this_cpu_cpumask_var_ptr(load_balance_mask); + int cpu; + bool sfind =3D false; struct lb_env env =3D { .sd =3D sd, .dst_cpu =3D this_cpu, @@ -13504,15 +13506,41 @@ static int sched_balance_rq(int this_cpu, struct = rq *this_rq, group =3D sched_balance_find_src_group(&env); if (!group) { schedstat_inc(sd->lb_nobusyg[idle]); + if (sched_feat(LB_PROMOTE)) + goto simple_find; goto out_balanced; } =20 busiest =3D sched_balance_find_src_rq(&env, group); if (!busiest) { schedstat_inc(sd->lb_nobusyq[idle]); + if (sched_feat(LB_PROMOTE)) + goto simple_find; goto out_balanced; } + goto begin_balance; + +simple_find: + if (env.idle !=3D CPU_NEWLY_IDLE || + (env.dst_rq->nr_running > 0 || env.dst_rq->ttwu_pending)) + goto out_balanced; =20 + sfind =3D true; + env.migration_type =3D migrate_task; + env.imbalance =3D 1; + + for_each_cpu_andnot(cpu, env.cpus, env.dst_grpmask) { + busiest =3D cpu_rq(cpu); + if (busiest->nr_running <=3D 1) { + __cpumask_clear_cpu(cpu, cpus); + continue; + } + break; + } + if (cpu >=3D nr_cpu_ids) + goto out_balanced; + +begin_balance: WARN_ON_ONCE(busiest =3D=3D env.dst_rq); =20 update_lb_imbalance_stat(&env, sd, idle); @@ -13615,6 +13643,7 @@ static int sched_balance_rq(int this_cpu, struct rq= *this_rq, =20 /* All tasks on this runqueue were pinned by CPU affinity */ if (unlikely(env.flags & LBF_ALL_PINNED)) { +check_redo: __cpumask_clear_cpu(cpu_of(busiest), cpus); /* * Attempting to continue load balancing at the current @@ -13627,6 +13656,8 @@ static int sched_balance_rq(int this_cpu, struct rq= *this_rq, if (!cpumask_subset(cpus, env.dst_grpmask)) { env.loop =3D 0; env.loop_break =3D SCHED_NR_MIGRATE_BREAK; + if (sfind) + goto simple_find; goto redo; } goto out_all_pinned; @@ -13668,8 +13699,11 @@ static int sched_balance_rq(int this_cpu, struct r= q *this_rq, * if the curr task on busiest CPU can't be * moved to this_cpu: */ - if (!cpumask_test_cpu(this_cpu, busiest->curr->cpus_ptr)) + if (!cpumask_test_cpu(this_cpu, busiest->curr->cpus_ptr)) { + if (sched_feat(LB_PROMOTE) && env.idle =3D=3D CPU_NEWLY_IDLE) + goto check_redo; goto out_one_pinned; + } =20 /* Record that we found at least one task that could run on this_cpu */ env.flags &=3D ~LBF_ALL_PINNED; @@ -13703,6 +13737,8 @@ static int sched_balance_rq(int this_cpu, struct rq= *this_rq, preempt_enable(); =20 out_unbalanced: + if (sched_feat(LB_PROMOTE) && !active_balance && env.idle =3D=3D CPU_NEWL= Y_IDLE) + goto check_redo; /* We were unbalanced, so reset the balancing interval */ sd->balance_interval =3D sd->min_interval; goto out; --=20 2.34.1