From nobody Fri Sep 25 05:30:16 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6A4584BD0F2; Wed, 16 Sep 2026 10:01:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789552918; cv=none; b=YAlTfiA25+Pf3fKVi/LdLrUYDZBDo0+3HLIIzOpKLbt9tlLqnwqapb1y+XWZO8dWsizP3E6MXIkg53aCsBPegEx6FyCQFJZj5BViCFrv7QCG/jFIt8/oSvXUv/o+nqD6s+Vto1ZkQcdfHro7z2R9gWrcrEBnSJEupK2unCXwu5Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789552918; c=relaxed/simple; bh=HVVfzed720/rXRspNBfPrMK4Ps/7gqBjL4uqDIo/5Ik=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pz1jZgVcLfMBZ8E8pSAtt/gwnPfarX7/nq767LfFWqH+i+32/pvJOn2h396ig867xA/T9g+pil1otFF4Avgci36x42nL0giG69jYDd/ogckX8VCTZDmOvc4sRvPK/Q8c7IBRXotYUZWD5uO1zbrBDp3FK8MmVj9jlMIJxI9USwk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=ie2R/szZ; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="ie2R/szZ" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AA6A1176A; Wed, 16 Sep 2026 03:01:26 -0700 (PDT) Received: from e127648.carmbridge.arm.com (unknown [10.0.129.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A384E3F86F; Wed, 16 Sep 2026 03:01:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1789552890; bh=HVVfzed720/rXRspNBfPrMK4Ps/7gqBjL4uqDIo/5Ik=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ie2R/szZG9eP9YOERTdXZlTlVdwz/i8uQGEfkM5W28JrcqNOnK68OewUlsmfNOWa4 aPBK+eXibzbo4z9HDDY4gE/+XAXKbaNdaGdnPyM6V/41PLt6KmcbsiA+62PGHINyg6 Ur6hZcMFZuwAw0I2D+l+RJ8Uqc3v39VnwyongDW8= From: Christian Loehle To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Valentin Schneider , K Prateek Nayak , Beata Michalska , Elif Topuz , "Rafael J . Wysocki" , Daniel Lezcano , Shubhang Kaushik , Christoph Lameter , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Christian Loehle Subject: [PATCH 1/2] sched/fair: Drop idle recency from slow-path CPU selection Date: Wed, 16 Sep 2026 11:01:15 +0100 Message-Id: <20260916100116.701206-2-christian.loehle@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260916100116.701206-1-christian.loehle@arm.com> References: <20260916100116.701206-1-christian.loehle@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The slow-path CPU picker favours the most recently idle CPU as a proxy for cache warmth. A more recent idle stamp may make ongoing entry more likely. Among CPUs with equal advertised exit latency, this may favour the one with the highest wakeup cost: if entry cannot be aborted, it must finish entry and then exit, while an already-resident CPU only needs to exit. The same advertised worst-case latency covers both cases. idle_stamp does not track the current CPUIdle entry, so an older scheduler-idle CPU may also be re-entering. A recent scheduler-idle transition may also mark a short gap in recurring task activity, so the CPU may soon be busy again. Drop the timestamp tie-break, retaining the first idle candidate unless a lower advertised exit latency is found. Signed-off-by: Christian Loehle Reviewed-by: Vincent Guittot Tested-by: Muhammad Usama Anjum --- kernel/sched/fair.c | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 7455a83a6a99..ff5793bddc35 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8459,7 +8459,6 @@ sched_balance_find_dst_group_cpu(struct sched_group *= group, struct task_struct * { unsigned long load, min_load =3D ULONG_MAX; unsigned int min_exit_latency =3D UINT_MAX; - u64 latest_idle_timestamp =3D 0; int least_loaded_cpu =3D this_cpu; int shallowest_idle_cpu =3D -1; int i; @@ -8480,23 +8479,11 @@ sched_balance_find_dst_group_cpu(struct sched_group= *group, struct task_struct * =20 if (available_idle_cpu(i)) { struct cpuidle_state *idle =3D idle_get_state(rq); if (idle && idle->exit_latency < min_exit_latency) { - /* - * We give priority to a CPU whose idle state - * has the smallest exit latency irrespective - * of any idle timestamp. - */ min_exit_latency =3D idle->exit_latency; - latest_idle_timestamp =3D rq->idle_stamp; shallowest_idle_cpu =3D i; } else if ((!idle || idle->exit_latency =3D=3D min_exit_latency) && - rq->idle_stamp > latest_idle_timestamp) { - /* - * If equal or no active idle state, then - * the most recently idled CPU might have - * a warmer cache. - */ - latest_idle_timestamp =3D rq->idle_stamp; + shallowest_idle_cpu =3D=3D -1) { shallowest_idle_cpu =3D i; } } else if (shallowest_idle_cpu =3D=3D -1) { --=20 2.34.1 From nobody Fri Sep 25 05:30:16 2026 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3ED37476CDA; Wed, 16 Sep 2026 10:01:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789552912; cv=none; b=XtY8xzH1QflwryLuD9IRVZyKQDO0ow1JUfoh+uL2cxMX+x54H8Ghx9gjJfehEOEiOFmcIMVm6pEHaQC358v63jFodfOmTNvCkWlZJ/bWd9MIxbtEHSU5fM9XVLscvwih3h3dzN/zRUxCPU5zaOjsyigNLlDt+6+XcxTibyY4D8U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789552912; c=relaxed/simple; bh=+U4NUhzKtF3iaqxkjxI+IFGed8Vvw8nCA5q4a9gp7Sw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=koUKPdBbACf4BPC7cyz+xzfe/qvDyPCVx4EnE4UX8paC3ywuPAUzIErdr7AlYA13gvqH0ubCZbEmNZoANGUPKN0I005ZlyJJl8oYQRcaN07UEXZ1+bOFj9xO5JDmTs6j/Wx7QqmX0ovPZ6SosHA4up/fjFgzW/sV0+B0+zNN0wU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=Q4VMMO6a; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="Q4VMMO6a" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CC7D219F0; Wed, 16 Sep 2026 03:01:29 -0700 (PDT) Received: from e127648.carmbridge.arm.com (unknown [10.0.129.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A253C3F86F; Wed, 16 Sep 2026 03:01:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1789552893; bh=+U4NUhzKtF3iaqxkjxI+IFGed8Vvw8nCA5q4a9gp7Sw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Q4VMMO6anAb22G15ijiEgbyXYFZ1L1UIkumiSX+dZ6JZKz5QmxwE2I1xqbxGvlqjk kSKs19t85UI03ZLWHuIjta321yW42GC4oS2tb5zFcOYgmhxaateCZ7BMvX5W+X5deE 3b2/crnShg87pxtwuKizbGf85SQXSalcAbnL9mvE= From: Christian Loehle To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Valentin Schneider , K Prateek Nayak , Beata Michalska , Elif Topuz , "Rafael J . Wysocki" , Daniel Lezcano , Shubhang Kaushik , Christoph Lameter , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, Christian Loehle Subject: [PATCH 2/2] sched/fair: Randomize equally shallow slow-path candidates Date: Wed, 16 Sep 2026 11:01:16 +0100 Message-Id: <20260916100116.701206-3-christian.loehle@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260916100116.701206-1-christian.loehle@arm.com> References: <20260916100116.701206-1-christian.loehle@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Picking the first eligible idle CPU leaves a scan-order bias. Concurrent slow-path selectors can choose the same CPU before either task is enqueued. Use reservoir sampling in the tie branch, resetting the candidate count when a lower advertised exit latency is found. Use the per-CPU scheduler PRNG and reciprocal_scale() to avoid variable division or a second scan. This reduces deterministic convergence without reserving the chosen CPU. Signed-off-by: Christian Loehle Tested-by: Muhammad Usama Anjum --- kernel/sched/fair.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ff5793bddc35..6836a8364440 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -8459,6 +8460,7 @@ sched_balance_find_dst_group_cpu(struct sched_group *= group, struct task_struct * { unsigned long load, min_load =3D ULONG_MAX; unsigned int min_exit_latency =3D UINT_MAX; + unsigned int nr_candidates =3D 0; int least_loaded_cpu =3D this_cpu; int shallowest_idle_cpu =3D -1; int i; @@ -8482,9 +8484,12 @@ sched_balance_find_dst_group_cpu(struct sched_group = *group, struct task_struct * if (idle && idle->exit_latency < min_exit_latency) { min_exit_latency =3D idle->exit_latency; shallowest_idle_cpu =3D i; + nr_candidates =3D 1; - } else if ((!idle || idle->exit_latency =3D=3D min_exit_latency) && - shallowest_idle_cpu =3D=3D -1) { - shallowest_idle_cpu =3D i; + } else if (!idle || idle->exit_latency =3D=3D min_exit_latency) { + nr_candidates++; + if (nr_candidates =3D=3D 1 || + !reciprocal_scale(sched_rng(), nr_candidates)) + shallowest_idle_cpu =3D i; } } else if (shallowest_idle_cpu =3D=3D -1) { load =3D cpu_load(cpu_rq(i)); --=20 2.34.1