From nobody Sat Jun 13 03:29:22 2026 Received: from CY7PR03CU001.outbound.protection.outlook.com (mail-westcentralusazon11010019.outbound.protection.outlook.com [40.93.198.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF6803822A1 for ; Sat, 9 May 2026 18:10:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.198.19 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778350210; cv=fail; b=OAPgOlBT0ibN+9wVNVcFaR/uZB4LFOizXGQb7CFXB2Ucrsbtdq+jKwJwqDhSl43N/AkwSxEmRpF7wND85wQL28spCIZPLnDtV4H1e8NscpKcjAmL5JWYjOu338AmFPwPZf8h9my/ze1AL8++wlygyjBckOsCtoHlXY+wY0CxL3s= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778350210; c=relaxed/simple; bh=NdUoCOMIo85ungML7yHXVApozm7tqzVyq7RkrBBWKp4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=OtokFyN+WJvCe4sZ+YrBBUAo9vtFiJcfErzgBPBbevFC+oplCa/aVvky71hVHlNFa/2fvIQMGwKmndn95uTolh3orVePXbl3Ea8wsfrYxFHbwNUyIS7zB6SoHPwt7T+qQ3EWikrlKI2WsGjGFDWZDmqy5I/ls2npjiCY0cmc0bg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=uaY4JFMU; arc=fail smtp.client-ip=40.93.198.19 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="uaY4JFMU" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kq97avuoI/Pf4ov93mcDSkbjrnhY1MfLeSO/6ewirdn7LjaUnWRiOI0SBKfviUezj/gFz5nlCyBEcga0Sq01hln7AgTRCoMQtxkCVwtu0On1H7pne4L2oqXwGbQjxly1dkoibXj7Dgpi9HOIj5Mm4t16qbY9EenC3ONWyw1iqS1h10CdbSF1Gh3ARsShGAXH98m2/yjGdzKZn25RqMyGnwYovLYxJdtCuwofUOZ5wEXeuYTv/W/HRfQYOvMCYmblUz40Ow+RWAFsFxFL6AnKJ+wMMVClUYxHLUjmuwzkfOK+Ej1VoLG5/F8bn7f2Ee+/nK/mdxX+n1T1j6OSA+znVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=C22u1z1KvbBxI2BfQK3lulG8ioKyClZYVSYhkp6ntXs=; b=XhwPXPGgw2woj/2tFqpi5XiuKdzb26rRd29IjfBXCtpqxe6DtgTJ7a5EZ1Mz+PrnnXMyueq7vNXU/MLAP6raiT3n6qjt1uALuho03eYyO7DD1VevKZbQpk7ExPk/VD+afl19tvdBF00kWm+M4yj2Gt5oUSzjtAyMbGCXl4kYdUo3pShCMbKxZVJaxHNZO41/WwhNLl2Vyl76xPYEpSUhcMzx84SuA8xdKbLmB8a8/zNe4N5yYZczZmzogttNFbWsdzmvNigcobRClTLCdOAXa2YSSnlcBmVNAynBqNJny3erJuh5YethltMtlRE4GvssiJyimFQPEVI/ObGdKA0s8w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=C22u1z1KvbBxI2BfQK3lulG8ioKyClZYVSYhkp6ntXs=; b=uaY4JFMUX3HA16iXqXF6Bdfrb6hNE97fEGlIYZwxJT8W7V8LjA8xfmTnNho/dRtF23bVTHy6gqezEiAy7t7zt2pJ0kjMeCDvcCjJRdtyqk5p36DH7iXNxgPw9MQmsuenqdlzyyxjR3BzzedH/+oM8OOggOoHQ1fPzNhhuxI3YiDG+thU4c7hMdFj68Rwdc9TlTScGYNwd4WVrcfT1qOwZGuxJOLmC0lZbAetxaz1Ki9rAlCygH6wUkc2S81qI1gXEjGDpHYs5vjyUS1qn4ZG2gX7eWsjQeA0Nt3UsHZnlyJAPsz1boiXDWsKLQI20tEB3cHwLrx4chhndUCm++hSWw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by SA5PPF9BB0D8619.namprd12.prod.outlook.com (2603:10b6:80f:fc04::8d8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.25; Sat, 9 May 2026 18:10:04 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9891.019; Sat, 9 May 2026 18:10:04 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Phil Auld , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , linux-kernel@vger.kernel.org Subject: [PATCH 1/5] sched/fair: Drop redundant RCU read lock in NOHZ kick path Date: Sat, 9 May 2026 20:07:25 +0200 Message-ID: <20260509180955.1840064-2-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260509180955.1840064-1-arighi@nvidia.com> References: <20260509180955.1840064-1-arighi@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MI2P293CA0007.ITAP293.PROD.OUTLOOK.COM (2603:10a6:290:45::7) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|SA5PPF9BB0D8619:EE_ X-MS-Office365-Filtering-Correlation-Id: dfdb5e47-5886-4191-996e-08deadf63186 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: p5SLUpZk8Hrji2PRpoPiU5ZEayYKVqpuYOXGYBVdboVTR1EOgcQe8bJF1iEN3WBYvMnGvjQJEFxao7Xa9FJX2JawtoPESqNC+ElMPZa2pL+tU0Od6T+F7QrXnk77sJdCaDAyTGgnnXhs09bzk9ArgecULL3sEODoVOuynI5Ej6dNpFm1TFzm9bdzUU2+JPyFywJVW4Qn+6mPEgpXhJCeJzdXDY5KCGKSGvAz3c0SWTTSWsRQt1/VQwrydU2c2iMbuchsVY9Jm5GUVe6ijl7KIp9YHnMeAJ/FjeJ9Ol+Kp0GtGOcQEE0MxhST9zRoPAHqpB6zLLu/G26cepYdRKUh7ECzk1bWfTI55Bn3QAnGM7S3uoYy4z5Y5vDKYuVnqbjvmW2XrdzAwLNjPFkxrKx3eAIKRG78l24OZNHGSv4AUCX7VTsBh4wnypQmSBiQnYH8aWX7YZc1qaOP2WmOO8XI+A+KtIaETr0QWVRHXCuTXIXugBnrKfb64ZY7IccFLz6Zsne18JdVeZIJ2MqnCxuAHiLcfgDzwlJ8+9nlQWQXp0i3MN4FcicVFxOnkINoTFC/2ziQDVwEEDijX/Mx5Fk09wqJxhFov5rFa8bqekpbh0szhx9vDqtvYZt2IkHl0L5W+bWmFgCTtHWxRQWzEDu34uRYk7e7KH/yMHwCrUtFdoqAKjdiN4h/F1slprjaZW1x X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?M3Jsa0vLCvBUz978xLtND1xExqxAm7Kl/dvxkJcvlkC0dS2PgmLyxilQ8lq8?= =?us-ascii?Q?H15uPbYRESDPby9OY+r9iaFTdJrzcj7dllLsZA2CfLOnX7ZnPN/Ticz64WMl?= =?us-ascii?Q?UVvvSN0Q1sVgZeM+8HTyTkWAQoxKxAPIutsvqTxVJsnwYV4Q5XP26d4GKxSx?= =?us-ascii?Q?YoVoIg03DS7xXkFGowkXZkAGIrDDuJEeEtbHXFJslPSiDp/0ySA/0u80qute?= =?us-ascii?Q?09aocoHgwY6FpPoD98vFvaFct5/9rv4KVeeq7hlCnayy8viNqHRgVWk7XmWQ?= =?us-ascii?Q?NsG/qJdGfaM5x4GyUqK2jsoaQrUEPA5IqktswrNfxsx6893Lnvtx76m4CPbi?= =?us-ascii?Q?d7KeeBq9Wvqzz2BmOdmnyr8rk/fN2oEgMfsWSMzsP5OPul+xJpJv/fiorNP+?= =?us-ascii?Q?9i7mnjT9Xlh5By6lDR74EWHynQzMxtyR2mzC64ED6oLTsQOUkrd/8Wvy3CjJ?= =?us-ascii?Q?o/qX4+VC0UvGco8E1Kr/ewjDZFDRvc3EZu73hQ4mJFB16rTsCAlZe1sAHjyi?= =?us-ascii?Q?77lwKV3+B/ynPi4DmYUA2W2M2fc2NH9c4WybDNvxqx7BuoceyiYZECaRIRmo?= =?us-ascii?Q?dgHvW16oha30vKdyTiuT/yjGL7EWWQI7++bnOJOW8+MhPlEst2b6/zAL78ne?= =?us-ascii?Q?Qc8NOBDQYa9syYvsMhYayB3kJsMrLYhOisio2FTe5+Tyj5i4CC5iDkX/0bw+?= =?us-ascii?Q?mBMbYyMpek19z0rU41TktEewgjV9U/OQbzqc6Q7sadsZmEe0jW76PzQk9OkX?= =?us-ascii?Q?T819QuHzjc7+oYgL2V9fxP85YlHWnjlH2IOX2q8Va+9O0yWN6nMiMP+/LWGN?= =?us-ascii?Q?mKsz8F/KcCgpNlwa3DpRyYv8RMCFE5ryoagBRRTiZvqtlCYh9pimDRpQX17H?= =?us-ascii?Q?+Is9zILlE85UV49cRN5fQtbJ4E3j40M3BcNiyOh0ZdSIKLfEqODqN0tOyWsX?= =?us-ascii?Q?EyC7PeXY1F79zVIDKpSunsJEM1PBWRy27f+nqzI7+hkDNC1rNNwFtmyrleNQ?= =?us-ascii?Q?4oOKGh8KiPkNvkRUYBQIJdnpn1wmuHmDj9Ooh+tgsvCdBTMs62NON+tBlXQc?= =?us-ascii?Q?hko1H+WN3oZBbzs440CJ6s+LdbjBMI1Ww7QkzeTe3gs9Dl0w/HyIxvRH1Kne?= =?us-ascii?Q?frFdKPR5WIcg1DG7uS1IRXldc7Shvq3NKCcMjpW+qnfgZPSZ+VTdR8772Nzd?= =?us-ascii?Q?/hCqbCUD9tw0awjkY7fMp6HwLrYGbPPIVoZewKM3PydtIAs16RD1wsLoEi2Z?= =?us-ascii?Q?SHFgR8ALCflK/AbniYDDfVh/CYK8OvhrUtCCAQf5ll1lUdL3GZzhLLOp/5iV?= =?us-ascii?Q?bQO520a3eGaNQJTb4BUBe/j6unR2Cv7Q4glogKpbcRHWnoL+ew4kNngsVXop?= =?us-ascii?Q?m6MPrEuwHzFEX4iIenOQmMNErw2D3T1VRj/VwLvdp/lQPLd6kIcbIVDSciIP?= =?us-ascii?Q?xgJ/6O1f2OzN9zaaaczGC4+VNXOlDaUzd/D0sA6L5Wye0iy+dRjqzkzT1+qQ?= =?us-ascii?Q?KqKqOaqm8efSz/NwClSTaF59nGyoPcHD2iJeGhFPGRk3uLunG6LgqEUwky7m?= =?us-ascii?Q?X5JpEL5tNxVLjewjltaS2ihGTusPUzywrQMrpB8tb2brO2J37YwgT1l74xpq?= =?us-ascii?Q?gfpG5VtySIIw3UX8DwLEBK1knPObluJ7Yio/7hx6lcWKLRu3Pp0orZZpm1R2?= =?us-ascii?Q?3W/fVcvqua1H6qIBlJcKT7OCERAS+jGgiA80ARIH583GkEV84825OPtDtOvn?= =?us-ascii?Q?z/oPkm81Vg=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: dfdb5e47-5886-4191-996e-08deadf63186 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2026 18:10:04.0924 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 01wHSX+GHOphLbgXH1tweHfKHa6JVs7OqwILmio07lZuAERsWv6XY4CxOpJxYzveklq2y5hJk7LRlmDA+RqKzQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA5PPF9BB0D8619 Content-Type: text/plain; charset="utf-8" nohz_balancer_kick() is reached from sched_balance_trigger(), which is called from sched_tick(). sched_tick() runs with IRQs disabled, so the additional rcu_read_lock/unlock() used around sched_domain accesses in this path is redundant. Rely on the existing IRQ-disabled context (and the rcu_dereference_all() checking) instead. The same applies to set_cpu_sd_state_idle(), called from the idle entry path with IRQs disabled, and to set_cpu_sd_state_busy(), reachable via nohz_balance_exit_idle() from two contexts: nohz_balancer_kick() (IRQs disabled, as above) and sched_cpu_deactivate() (the CPUHP_AP_ACTIVE teardown, which runs under cpus_write_lock(), so it cannot race with sched-domain rebuilds). In both cases the rcu_dereference_all() validation is sufficient. No functional change intended. Cc: Vincent Guittot Cc: Dietmar Eggemann Suggested-by: K Prateek Nayak Reviewed-by: K Prateek Nayak Signed-off-by: Andrea Righi Reviewed-by: Shrikanth Hegde Reviewed-by: Vincent Guittot --- kernel/sched/fair.c | 38 +++++++++++--------------------------- 1 file changed, 11 insertions(+), 27 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3ebec186f9823..6b059ee80b631 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12785,8 +12785,6 @@ static void nohz_balancer_kick(struct rq *rq) goto out; } =20 - rcu_read_lock(); - sd =3D rcu_dereference_all(rq->sd); if (sd) { /* @@ -12794,8 +12792,8 @@ static void nohz_balancer_kick(struct rq *rq) * capacity, kick the ILB to see if there's a better CPU to run on: */ if (rq->cfs.h_nr_runnable >=3D 1 && check_cpu_capacity(rq, sd)) { - flags =3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; - goto unlock; + flags |=3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; + goto out; } } =20 @@ -12811,8 +12809,8 @@ static void nohz_balancer_kick(struct rq *rq) */ for_each_cpu_and(i, sched_domain_span(sd), nohz.idle_cpus_mask) { if (sched_asym(sd, i, cpu)) { - flags =3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; - goto unlock; + flags |=3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; + goto out; } } } @@ -12823,10 +12821,8 @@ static void nohz_balancer_kick(struct rq *rq) * When ASYM_CPUCAPACITY; see if there's a higher capacity CPU * to run the misfit task on. */ - if (check_misfit_status(rq)) { - flags =3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; - goto unlock; - } + if (check_misfit_status(rq)) + flags |=3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; =20 /* * For asymmetric systems, we do not want to nicely balance @@ -12835,7 +12831,7 @@ static void nohz_balancer_kick(struct rq *rq) * * Skip the LLC logic because it's not relevant in that case. */ - goto unlock; + goto out; } =20 sds =3D rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); @@ -12850,13 +12846,9 @@ static void nohz_balancer_kick(struct rq *rq) * like this LLC domain has tasks we could move. */ nr_busy =3D atomic_read(&sds->nr_busy_cpus); - if (nr_busy > 1) { - flags =3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; - goto unlock; - } + if (nr_busy > 1) + flags |=3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; } -unlock: - rcu_read_unlock(); out: if (READ_ONCE(nohz.needs_update)) flags |=3D NOHZ_NEXT_KICK; @@ -12868,17 +12860,13 @@ static void nohz_balancer_kick(struct rq *rq) static void set_cpu_sd_state_busy(int cpu) { struct sched_domain *sd; - - rcu_read_lock(); sd =3D rcu_dereference_all(per_cpu(sd_llc, cpu)); =20 if (!sd || !sd->nohz_idle) - goto unlock; + return; sd->nohz_idle =3D 0; =20 atomic_inc(&sd->shared->nr_busy_cpus); -unlock: - rcu_read_unlock(); } =20 void nohz_balance_exit_idle(struct rq *rq) @@ -12897,17 +12885,13 @@ void nohz_balance_exit_idle(struct rq *rq) static void set_cpu_sd_state_idle(int cpu) { struct sched_domain *sd; - - rcu_read_lock(); sd =3D rcu_dereference_all(per_cpu(sd_llc, cpu)); =20 if (!sd || sd->nohz_idle) - goto unlock; + return; sd->nohz_idle =3D 1; =20 atomic_dec(&sd->shared->nr_busy_cpus); -unlock: - rcu_read_unlock(); } =20 /* --=20 2.54.0 From nobody Sat Jun 13 03:29:22 2026 Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012053.outbound.protection.outlook.com [40.107.209.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC50434750B for ; Sat, 9 May 2026 18:10:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.209.53 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778350214; cv=fail; b=sRBrAen8HnTJZ0OsEhna9FjVPKcsego7xl2JxjPAF3jd6KMis5QbuwoGgjGjTEkOavSXRQtvhdwtquder+azwXSq2vabDs8dc80afb4EmNEMMYCQtRFTelvvLW2jhaQ0IvuwpYhh7orkpr5+g7CYsUuTbW8A6vXolbLOjMV330I= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778350214; c=relaxed/simple; bh=hSNJvbmDFUOwzMAAJSKN4CNIhxyGqVvN9KSqzfRq/2E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=sJDsxY71kT125t53JB58xX0iw6KPtMALUOvVu9ZW+l/YwoL6uSNEZ4VHOwOaqqkR5ufGouDP1kGxmOirZV0vXMW5GP0vfgIJ0qVdb9dSZOrnMRZEOHQ2HsQZC9NQrNt0VuFUh3gEjRgV+dEEXZoxVK1y2xorJfmMdmDT2r8+62A= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=gaOBCshe; arc=fail smtp.client-ip=40.107.209.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="gaOBCshe" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qNrN8aHMZwT7G6pZZI5TfJbw64K/uQ8LGWc/CV8zqQasX3zAWfd+zRVi1EymEmyraPysmOTZIP41O8cpqk4UXesWaaBZYRXAmgsvGm++5eOv6zk0flXVZ9nobFfBDtyqTr3jPrP7hGXxOviHUkp45I+HnLjXErHU9b/dTGyQdqBINxmX1YmLywjUTKdXRIL18dad0NGxitvHI93sb3Hy7uy5Fq7hceBQ7HvdvR6QH0AzQLvyF1p9gcWtbEmrs9IoYXYINgDLVHIouOvoTRw7uC4bs+IVdm/XuZ3gDUx+aD2g5MgR+pEs+6ZHHB68xiTG11NcLjwTQf0L3znrFY3g1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pEV+Lw0HuznHk6rtUeSfTu3qPHh5oXqhxVm1sYCaWMo=; b=BuUFHoiAErdetpG9gYInF15DFeuQGtnqe+Iv4Ttg04R+7gbOzoa+jsbbu78RyuYGHk1jw+4Fkfc6UcoGppvXhx4Yej2YYLW8PmvtEZRbn5L9L5Th3oGR/W1ho8YP8Uj9hZkwkYBo8mHlSIl5NsQkF1Ejpw2QuG0+a+rVBXXaUYCr2qcjhBJIoO5DcW0ozzqRsJkvY5p7zAhN1tSjm3Ztac5xIUgCSja/9OftPxYvJF2jShQJyKYJSBiPgwROe4r3KgSFnd5mbarbb4/WEk+kqZr/PdWwB9hfL0Fwp88YOG5Xo6tcsO6pppR4w0ezXe3JkUrIk9klP870TKnFJsXIRw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pEV+Lw0HuznHk6rtUeSfTu3qPHh5oXqhxVm1sYCaWMo=; b=gaOBCshe8jQqTSjPV/aUUhI4ym6JFwbDAftq8PiNII8xKcP+jzm81Fk6WBOj8BV1AbzVfFQIWvo4W45ss9wUxUx3xmbqX/o5ik9NX2gbyTw2l0cTCAS2gHXwXscSNTpLCxP9X3olaJRwUbFjyYdQJ+IWXh7etsw3XODz0qntIHnn+9IaaN3Wpih+uAIPCU62CIk3bqoRkTrBiciUcyzkmBvUMqy71cgRWpC7M6r4QVp8pcB7Lt4iWGzSLPcuLqltlJURxcljjsq7lc3TtOOOumw4cu6kbjwYYvih1YICI+gjnpGBEMz875ftXYaIp3muJvEOwD1UJlpw1QUyXhFBDw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by SA5PPF9BB0D8619.namprd12.prod.outlook.com (2603:10b6:80f:fc04::8d8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.25; Sat, 9 May 2026 18:10:07 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9891.019; Sat, 9 May 2026 18:10:07 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Phil Auld , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , linux-kernel@vger.kernel.org Subject: [PATCH 2/5] sched/fair: Attach sched_domain_shared to sd_asym_cpucapacity Date: Sat, 9 May 2026 20:07:26 +0200 Message-ID: <20260509180955.1840064-3-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260509180955.1840064-1-arighi@nvidia.com> References: <20260509180955.1840064-1-arighi@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MI1P293CA0020.ITAP293.PROD.OUTLOOK.COM (2603:10a6:290:3::18) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|SA5PPF9BB0D8619:EE_ X-MS-Office365-Filtering-Correlation-Id: c6b979f2-0df7-45bb-6bc3-08deadf633c3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: M7pu++cTBijVoLFo/Nf/VJelokSC0krcbJMkBf30SZ322ev/Vgbqetbta8Slh8VcAM8nYDit0fgdBuLByRuT13Hj+kWmulpaLiVbwPxfRYsBrKuhaYBL1xbYpMTo8feD1/jh39GWaU6iKN7v6ZEIjfEUU2FDeu6VFjaU/T6x+4FViXX3xzULdYlJeX52wPVLKXti3RqufcJub/EwC1RE4UW3MKYDVtYqXnwP1LmUtWuJyKbH7K8WwmdM+cBJWNCVFSuWIvAAV7xOUlAPtl1BgpRyTGbXDLmCDmRbdBqPHfatqlf+//NFWS4OhIq4l1j9qlhkDYS1EiTIIUBkF57HBAEkaUKyp941V2FM+M//7R6PxppummiBLN0dngxLtV+skI34jJev7GWeEG9ZCEIoJQaI+a1C82/b4Z34IwWQ0/BvUvnr299jju+h/OaB541OCsS1pzPuGVKoWeGgTXdDvnFwoC5znyUeNem6+belEaTeccl521P1kasLJ8LW0nOCxcKNY1mQcYtWVuS88LF5jv4Fo+QIukyrXN6DhmpfVCWOmde52WmtVol4VwcB7pBe7KtQpwOUoXJT0M/KSxQFHj6VoZuHdoK8/WZ9n89kj9DphOatUYf0VB8i5YBCnr1/m9Yi3w9M+6WTZOYmUdyCdSXlpiZQSpMRo1NqwGh/xnF0lt3Vh+iKB45/Xm8USDjp X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?cKhA7U8v2PMiJ7pyMmQ042ASr1rLlBpe3ltnP5mr+ChmduPxq5xXQpWG0JaJ?= =?us-ascii?Q?dHEaIafqAZU8PAUvfynvNh32eVudLOjIkImU2Q3jSoOL71fbHZoqDZXcyH/x?= =?us-ascii?Q?sOIIUpuJdOgYbADw/sJnYlf8bofN/tjuZ1j7B7A65bg7J3l9JLEMGqJoaKQf?= =?us-ascii?Q?YqYdVotP6V/E3TQGvbY28Mz7rtAMPUsZG9idmCby/NDNivvC9fdoeT37+btZ?= =?us-ascii?Q?TCPLj4gItlLtJ9h6e21l0noySXc/LOlPBWzh9ixmIjzIra02A/CRcu5XQuJR?= =?us-ascii?Q?CMsMpl15sArG6H+Yrbi+LZZYB1ulxJ9XSGlUKZ7UDT3Cn96PZfgoeL+4J2Cu?= =?us-ascii?Q?S1c601Jdb1f+XhUVmjbSCdjN5Rq1VUhVHChKy7K/mf1QPbL0KXEol7lE5Z92?= =?us-ascii?Q?zBr5ZmgnKEIRTGKwwiayuuE4NR/gIoMlIiTVwkraOfOuUxP0ttPIwRjzktGl?= =?us-ascii?Q?LFj9nYMt5pde5eaklAkz18D2yaTZ2K2XCOMu89t1cLCrhKvW2u8uhKMndGQP?= =?us-ascii?Q?Q5G+FwTqRStuFYrHgKZztUPMXBgQojtatnnJfMtT1STdXcFvgATZZUUtZ/dL?= =?us-ascii?Q?MHY7X29N0hMOoGOtYPYl50NqzHq0dLOW8HXmxgMWAkjYl4AbXTiggoaSJlx1?= =?us-ascii?Q?0mQ0YCzEto7BQUnXDYstvA6d/N44tcOIsfPk5b5EM9bfyhJhDlGWTrX1p7t0?= =?us-ascii?Q?q0Camu6rqrNNWgQYjN7e8SVQg7n0C6gJDMFQpGnZBAcb/YQ7VSF3Ke32HBtE?= =?us-ascii?Q?Nlc6EISFlMpHJm/Ju01aAlYXGWg3U1eoFr9wl6xtP9ar3T497x3sp94NpKp9?= =?us-ascii?Q?kDbBOCeyq2xvw6jnBGp6g1fXwEiC4VBZ8HJKZEKymANZN0FYUom7oSNnZy2+?= =?us-ascii?Q?zWI2gIB7qp75A2Q3LEMc9DC2rwnBx9V0hTqbgOe0mGvf+a/7Rq3W0RVakQHp?= =?us-ascii?Q?2Rj6Z4Y87Xg5T8o31gEznIF/6WIgA0M43KFRV38mflSDTs0b4nfIqG8Yoml0?= =?us-ascii?Q?7JRRnSS3RyM1my27rcLyT3DT3TkMgKC3yooW9R9oPG3ei3gRVs7CqWgxLcfz?= =?us-ascii?Q?SnobfIkEbwC93ewYPgHcYhTJ8VmU0tvA+bUrylmC+TwxFoqL7EutYwJ0gy8S?= =?us-ascii?Q?ksyp+GmvcxjltO16qorJAczOTazq0FBH+df5sllhtZwoWMk6j8tgp70zbRzM?= =?us-ascii?Q?Ou1tVvSrza6oYEZGOunfdcWkAfU5DPLpOQfJKLoxwFJi+XTlsmx1b63A2QIq?= =?us-ascii?Q?wWGJBGklKsRtTsz+1G/N9cIaAwa4QWuPq+JsHFcWTQECM0xUfyHtjcaXFy/H?= =?us-ascii?Q?8FDTMZv61ZztinxvrTciyxX2ESaOVkIkuW+tMtPiHGZrTdqp1AaUlF5VEJeT?= =?us-ascii?Q?6gXI+7gvChJDVXh2AZhA4hZ0nitdHtaQLsBpzPLmMUU9tlQwjjDBOBFTbd0K?= =?us-ascii?Q?wvdFiSAfxT/twx7S5oNTgo9eExCtjurB2pEPj+5SkHMl0/sJ1ka8EsZc/6qT?= =?us-ascii?Q?ZwUt/GxJpfW/wy1QcRzw5cvclSV2RDbSAPfwO1mzMXdMo+0ii9PKlYKOnvz6?= =?us-ascii?Q?w2cBmB2ibOPBjIuj/RkaLP1fqXiN923pSd5uzdynYGfjk/b+/InyaslVvgPi?= =?us-ascii?Q?nTBTW/8HTa+a1msz7ihTt63iHURBbl/TQxy0ZstIZSZBbPof17dPxrJKtvkq?= =?us-ascii?Q?ty/xnXTBS6pfLy5yCn4LiQq/bjfWz6qrNCNpS7EXW9l4aq4DjD0IX9rn5WbP?= =?us-ascii?Q?c8r8JSDM3w=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c6b979f2-0df7-45bb-6bc3-08deadf633c3 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2026 18:10:07.6976 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PD269LhVocBQQyYyLPV3NNNbOogQooTu9O8N1qQjSidbaMauDoEXWbgHhamwafC5QcQGjBfuRNe68eEmWzxV4A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA5PPF9BB0D8619 Content-Type: text/plain; charset="utf-8" From: K Prateek Nayak On asymmetric CPU capacity systems, the wakeup path uses select_idle_capacity(), which scans the span of sd_asym_cpucapacity rather than sd_llc. The has_idle_cores hint however lives on sd_llc->shared, so the wakeup-time read of has_idle_cores operates on an LLC-scoped blob while the actual scan/decision spans the asym domain; nr_busy_cpus also lives in the same shared sched_domain data, but it's never used in the asym CPU capacity scenario. Therefore, move the sched_domain_shared object to sd_asym_cpucapacity whenever the CPU has a SD_ASYM_CPUCAPACITY_FULL ancestor and that ancestor is non-overlapping (i.e., not built from SD_NUMA). In that case the scope of has_idle_cores matches the scope of the wakeup scan. Fall back to attaching the shared object to sd_llc in three cases: 1) plain symmetric systems (no SD_ASYM_CPUCAPACITY_FULL anywhere); 2) CPUs in an exclusive cpuset that carves out a symmetric capacity island: has_asym is system-wide but those CPUs have no SD_ASYM_CPUCAPACITY_FULL ancestor in their hierarchy and follow the symmetric LLC path in select_idle_sibling(); 3) exotic topologies where SD_ASYM_CPUCAPACITY_FULL lands on an SD_NUMA-built domain. init_sched_domain_shared() keys the shared blob off cpumask_first(span), which on overlapping NUMA domains would alias unrelated spans onto the same blob. Keep the shared object on the LLC there; select_idle_capacity() gracefully skips the has_idle_cores preference when sd->shared is NULL. While at it, also rename the per-CPU sd_llc_shared to sd_balance_shared, as it is no longer strictly tied to the LLC. Cc: Vincent Guittot Cc: Dietmar Eggemann Co-developed-by: Andrea Righi Signed-off-by: Andrea Righi Signed-off-by: K Prateek Nayak Acked-by: Vincent Guittot --- kernel/sched/fair.c | 19 ++++++--- kernel/sched/sched.h | 2 +- kernel/sched/topology.c | 95 +++++++++++++++++++++++++++++++++++------ 3 files changed, 95 insertions(+), 21 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6b059ee80b631..960a1a9696b98 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7819,7 +7819,7 @@ static inline void set_idle_cores(int cpu, int val) { struct sched_domain_shared *sds; =20 - sds =3D rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); + sds =3D rcu_dereference_all(per_cpu(sd_balance_shared, cpu)); if (sds) WRITE_ONCE(sds->has_idle_cores, val); } @@ -7828,7 +7828,7 @@ static inline bool test_idle_cores(int cpu) { struct sched_domain_shared *sds; =20 - sds =3D rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); + sds =3D rcu_dereference_all(per_cpu(sd_balance_shared, cpu)); if (sds) return READ_ONCE(sds->has_idle_cores); =20 @@ -7837,7 +7837,7 @@ static inline bool test_idle_cores(int cpu) =20 /* * Scans the local SMT mask to see if the entire core is idle, and records= this - * information in sd_llc_shared->has_idle_cores. + * information in sd_balance_shared->has_idle_cores. * * Since SMT siblings share all cache levels, inspecting this limited remo= te * state should be fairly cheap. @@ -7954,7 +7954,7 @@ static int select_idle_cpu(struct task_struct *p, str= uct sched_domain *sd, bool struct cpumask *cpus =3D this_cpu_cpumask_var_ptr(select_rq_mask); int i, cpu, idle_cpu =3D -1, nr =3D INT_MAX; =20 - if (sched_feat(SIS_UTIL)) { + if (sched_feat(SIS_UTIL) && sd->shared) { /* * Increment because !--nr is the condition to stop scan. * @@ -12834,7 +12834,7 @@ static void nohz_balancer_kick(struct rq *rq) goto out; } =20 - sds =3D rcu_dereference_all(per_cpu(sd_llc_shared, cpu)); + sds =3D rcu_dereference_all(per_cpu(sd_balance_shared, cpu)); if (sds) { /* * If there is an imbalance between LLC domains (IOW we could @@ -12862,7 +12862,11 @@ static void set_cpu_sd_state_busy(int cpu) struct sched_domain *sd; sd =3D rcu_dereference_all(per_cpu(sd_llc, cpu)); =20 - if (!sd || !sd->nohz_idle) + /* + * sd->nohz_idle only pairs with nr_busy_cpus on sd->shared; if this + * domain has no shared object there is nothing to clear or account. + */ + if (!sd || !sd->shared || !sd->nohz_idle) return; sd->nohz_idle =3D 0; =20 @@ -12887,7 +12891,8 @@ static void set_cpu_sd_state_idle(int cpu) struct sched_domain *sd; sd =3D rcu_dereference_all(per_cpu(sd_llc, cpu)); =20 - if (!sd || sd->nohz_idle) + /* See set_cpu_sd_state_busy(): nohz_idle is only used with sd->shared. */ + if (!sd || !sd->shared || sd->nohz_idle) return; sd->nohz_idle =3D 1; =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9f63b15d309d1..330f5893c4561 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2170,7 +2170,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_llc); DECLARE_PER_CPU(int, sd_llc_size); DECLARE_PER_CPU(int, sd_llc_id); DECLARE_PER_CPU(int, sd_share_id); -DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); +DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_balance_shared); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_numa); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 5847b83d9d552..9bc4d11dd6a98 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -665,7 +665,7 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc); DEFINE_PER_CPU(int, sd_llc_size); DEFINE_PER_CPU(int, sd_llc_id); DEFINE_PER_CPU(int, sd_share_id); -DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); +DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_balance_shared); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); @@ -680,20 +680,38 @@ static void update_top_cache_domain(int cpu) int id =3D cpu; int size =3D 1; =20 + sd =3D lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); + /* + * The shared object is attached to sd_asym_cpucapacity only when the + * asym domain is non-overlapping (i.e., not built from SD_NUMA). + * On overlapping (NUMA) asym domains we fall back to letting the + * SD_SHARE_LLC path own the shared object, so sd->shared may be NULL + * here. + */ + if (sd && sd->shared) + sds =3D sd->shared; + + rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); + sd =3D highest_flag_domain(cpu, SD_SHARE_LLC); if (sd) { id =3D cpumask_first(sched_domain_span(sd)); size =3D cpumask_weight(sched_domain_span(sd)); =20 - /* If sd_llc exists, sd_llc_shared should exist too. */ - WARN_ON_ONCE(!sd->shared); - sds =3D sd->shared; + /* + * If sd_asym_cpucapacity didn't claim the shared object, + * sd_llc must have one linked. + */ + if (!sds) { + WARN_ON_ONCE(!sd->shared); + sds =3D sd->shared; + } } =20 rcu_assign_pointer(per_cpu(sd_llc, cpu), sd); per_cpu(sd_llc_size, cpu) =3D size; per_cpu(sd_llc_id, cpu) =3D id; - rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds); + rcu_assign_pointer(per_cpu(sd_balance_shared, cpu), sds); =20 sd =3D lowest_flag_domain(cpu, SD_CLUSTER); if (sd) @@ -711,9 +729,6 @@ static void update_top_cache_domain(int cpu) =20 sd =3D highest_flag_domain(cpu, SD_ASYM_PACKING); rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd); - - sd =3D lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); - rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); } =20 /* @@ -2650,6 +2665,54 @@ static void adjust_numa_imbalance(struct sched_domai= n *sd_llc) } } =20 +static void init_sched_domain_shared(struct s_data *d, struct sched_domain= *sd) +{ + int sd_id =3D cpumask_first(sched_domain_span(sd)); + + sd->shared =3D *per_cpu_ptr(d->sds, sd_id); + /* + * nr_busy_cpus is consumed only by the NOHZ kick path via + * sd_balance_shared; on the asym-capacity path it is initialized but + * never read. + */ + atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); + atomic_inc(&sd->shared->ref); +} + +/* + * For asymmetric CPU capacity, attach sched_domain_shared on the innermost + * SD_ASYM_CPUCAPACITY_FULL ancestor of @cpu's base domain when that ances= tor is + * not an overlapping NUMA-built domain (then LLC should claim shared). + * + * A CPU may lack any FULL ancestor (e.g., exclusive cpuset symmetric isla= nd), + * then LLC must claim shared instead. + * + * Note: SD_ASYM_CPUCAPACITY_FULL is only set when all CPU capacity values + * are present in the domain span, so the asym domain we attach to cannot + * degenerate into a single-capacity group. The relevant edge cases are in= stead + * covered by the caveats above. + * + * Return true if this CPU's asym path claimed sd->shared, false otherwise. + */ +static bool claim_asym_sched_domain_shared(struct s_data *d, int cpu) +{ + struct sched_domain *sd =3D *per_cpu_ptr(d->sd, cpu); + struct sched_domain *sd_asym; + + if (!sd) + return false; + + sd_asym =3D sd; + while (sd_asym && !(sd_asym->flags & SD_ASYM_CPUCAPACITY_FULL)) + sd_asym =3D sd_asym->parent; + + if (!sd_asym || (sd_asym->flags & SD_NUMA)) + return false; + + init_sched_domain_shared(d, sd_asym); + return true; +} + /* * Build sched domains for a given set of CPUs and attach the sched domains * to the individual CPUs @@ -2708,20 +2771,26 @@ build_sched_domains(const struct cpumask *cpu_map, = struct sched_domain_attr *att } =20 for_each_cpu(i, cpu_map) { + bool asym_claimed =3D false; + sd =3D *per_cpu_ptr(d.sd, i); if (!sd) continue; =20 + if (has_asym) + asym_claimed =3D claim_asym_sched_domain_shared(&d, i); + /* First, find the topmost SD_SHARE_LLC domain */ while (sd->parent && (sd->parent->flags & SD_SHARE_LLC)) sd =3D sd->parent; =20 if (sd->flags & SD_SHARE_LLC) { - int sd_id =3D cpumask_first(sched_domain_span(sd)); - - sd->shared =3D *per_cpu_ptr(d.sds, sd_id); - atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); - atomic_inc(&sd->shared->ref); + /* + * Initialize the sd->shared for SD_SHARE_LLC unless + * the asym path above already claimed it. + */ + if (!asym_claimed) + init_sched_domain_shared(&d, sd); =20 /* * In presence of higher domains, adjust the --=20 2.54.0 From nobody Sat Jun 13 03:29:22 2026 Received: from CY7PR03CU001.outbound.protection.outlook.com (mail-westcentralusazon11010027.outbound.protection.outlook.com [40.93.198.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16C833822A1 for ; Sat, 9 May 2026 18:10:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.198.27 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778350217; cv=fail; b=mWGA+Ic+ftyoMopGXckgArKpkvUgKpDOjGZSI3/o4bqtBSZYkptUD+xrl0/ExQ9IeKNpvwNZG4mKkg9PenvethgHtOzpHLNdu8N1tUUSSDVRrqFyVK1RyoFCUdlPyPOYZFXKotX6WtqIo3PTIpnCzCQSu1f/6YSXd8mAMrvQDBo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778350217; c=relaxed/simple; bh=LS/PB4zQ/zpIQJPEfCMrCvuBDAhdGc/oxAoQ3fqCXTQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=chJtgK47R4u/cDYX+gsspDbt9N5WPYvzhFe79PcsCs6Rob/Oj4ahPoPu2MOme4CFjeq1BYRP8HXJuMyupoG/UVeZQa9vHk8GsPtNh1vHd5T7u8a0IVFFQIvYArMhex28DHOtapWEe0ztp9+ci/uS+GZd0S7KKRMpoBnRMQT03qE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=lQYvBVUq; arc=fail smtp.client-ip=40.93.198.27 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="lQYvBVUq" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=IUC7E0BRsdCac3/Qr6VF9WfFsYrLCbTySK24+OyEcaxoZ+vqzF644i8vLzGs8EiVKwyc65AFGzOIHKJ7Mgr/VHqDQCCs2Hu+RYRyjDhPP+Ot7sv12V3pIUjSgFU8vM4TgFFncjaz9QvlAeFtpkolMikZPcolOXEW0/HvnLsrZuIkkQDR8ob2/rzsbwntiyR4x+hf/9lGQmHDxF2ikO4pHUUnncwmIBf706NsPYvJW4RrqOYjI+e8VsVKgwk6/2Byl1IkT977YHVPyT5YpjEA8frMs4SZJNezg/t4k8kvWZbOnfSbJqSQmT/jmQGgkbdTo+AJvQtLTf3Ep2G2+XuJpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Pb8c7z7aU7m/hyREX91UityDyZLAXRi/UWatbMmity8=; b=R57nknM4kHH+4QgNirmMoyi27tddXj6dPvWBi7Uc0Z6KEUEgsK06DEsiJOj41d+D2j2ZfKlV9PDJy0cUpQ6U9TdiSNBvfNrG5Li69BGgAlFyWYmzLKNPs3MpoGKiCicn67+2ZCoustLUGDpCFb0pyaxTRAx6ePw6KexaE0fCNd0Dqk4NppV0XZXMvGdpGHxd5dX+dPHmqr28td5DMw/dv0pyOI/BdCTgEEn0dsDMcxnv39U6HbNtLIgU+fbODl5TDiBIFIFAvjoj8F9mkYH+a1eJAPHEzIHkzDMGi8WCZRkBpFIh2lY1n+9fAqIvjSdB1OazKKilsXVXJb9abHKI3Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Pb8c7z7aU7m/hyREX91UityDyZLAXRi/UWatbMmity8=; b=lQYvBVUq0Kpv7OsiBRY5dRfbuKPNlA1EN+Tu/JLi+19ISKaTdCRlapL6HWq0KFUYW+0LQ2JD8jXhnbN3fCCZ0Y1ttTI2aUblEWzdlbjaVYOqDqPJX/ILRRFAAu/oG3SPgfAss2RARZbHtOqPv7/B56JqX14BYOLGCJ5rRyeLksBul92MyVNmRzSKdzlxS+X5/4UjaCEYpMXNPmvZduyukWaszAPviiFsvYOuyLd9C2wlkvLjuHuwGvbRsq0RTPQGA8nykqlU3zEhO4vQxpsHIcCQnnaNdVRqIX2pdZl5VotmM1a4pLk0Id9WqC+TzQ4ePjYPFcTUfvAx7UDrMlIvDA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by SA5PPF9BB0D8619.namprd12.prod.outlook.com (2603:10b6:80f:fc04::8d8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.25; Sat, 9 May 2026 18:10:11 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9891.019; Sat, 9 May 2026 18:10:11 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Phil Auld , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , linux-kernel@vger.kernel.org Subject: [PATCH 3/5] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection Date: Sat, 9 May 2026 20:07:27 +0200 Message-ID: <20260509180955.1840064-4-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260509180955.1840064-1-arighi@nvidia.com> References: <20260509180955.1840064-1-arighi@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MI1P293CA0021.ITAP293.PROD.OUTLOOK.COM (2603:10a6:290:3::14) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|SA5PPF9BB0D8619:EE_ X-MS-Office365-Filtering-Correlation-Id: 729e459e-36cd-49af-27fc-08deadf635d3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: Oraof3bWQIlnJ2f6VKZyeXuE09I4Q8wUqpm401OqicP4r3Q9ogua92f12P6mTKTJDhe1JSsKYGYs86wk7yqhiSRZE7uPMQbiWhxpqlk1dA0iI0dd/Jkiw5zS85tw2Suibt4L/ZzKLRtnN0b0q0uP9aAojxZj1Ko5jeomP7iMQAK5uQL04TYSUZePQmZ/nFI9e7t9LlNqzRk6Kf6r6M0RcTLaketqh/MRJJP/BbW17pJJq0lNIFoObM95zzUAeOT42JBZw6TUqRwSlCpI5IAXBOUniU2FPViBVxP1JyMuZ6KQfRUwl09rV8TQSAR3FINF0EG7BA6hc+vwBDHcO2uInwbxRF/4oNQ3T877NkeTBmoEin79DUtYozH4xTssrbywqeiGMQT3wQq5UA2srjjgzRqG7zwsb5BmABZ5BG7lcC8bHeNN7kRt2LX3layrIQmW5+aAR8LVJOXm+N9H8Ya2Wp1GKm3rRt89Oe5W3vO9E7aaHFvt6Hh2FVHeEKvecLmFE/DGJXFEgRRFSG5KAPh/IUNObuV/RLwxkkipDoRZe4uPwToknAn0A2yBnjSZPKBr8AX+ws8TbTSBC7tbYQOCyMvxPGWfPKk5f+3y2dr+buNXef0KKwlxxTaVW7ErhPsI2UcmZLaBW09TtzfLu7g3aAETwERyUK7n/WpaNoNoGwCo1gKNdyMySsHkWZPMQs2a X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?8lKub9b2Fb5HuiG/YgTzdOpH5BRzP1tkcU2Lh/1BVy01akxjqK3wsoCQp8fp?= =?us-ascii?Q?D2A1QkGOwjwKzdDUQY480FUY8i92Q3hofznRkrLvGRomAna8zcwQfZNIlz8E?= =?us-ascii?Q?b7KckVdQ6wS0QStP2ZIwJzfbHhwtuYqzB8s3/92CQqg6+6yj6Nvly9FenQC/?= =?us-ascii?Q?I3oIfvktnZdlUBS0FrVO62vPz2RwS2nrCrHTexfPgrFR3lVqo8vCdwbaABHz?= =?us-ascii?Q?tdmovkPc9WuC5HsGV33ioOjhKZuQvj+i5IE4sZfEi4zWLOMlHOgzI9HiUu+u?= =?us-ascii?Q?Ey4TnqD4GbTVKiYUqBDRysygi+RaI1hbKYBNWyGpoRNNgQouIxcRQn9Pbaf9?= =?us-ascii?Q?360PoBNVCQW/YITw1dmHU3LqRt1zuDqSDcfY94qYpYQcC522nodhBWcJ/oZe?= =?us-ascii?Q?vL1/N6FWWHr6F72phDYuQjPDFIc5a8NkYO4jjtTZ0L9i9LZvdvZoTjsKgXKQ?= =?us-ascii?Q?SLITId4/SCQyi44MMzIwuh3gFrP5hVtnoUQ88leQKFtmrsdAVOSdzPBUEflQ?= =?us-ascii?Q?1pvzZesKbSZI7uOvNa/IriUp8LV6U1IbUTP+w+yuM0Fb+HqLlw+W72z5T2KY?= =?us-ascii?Q?/+sb4cjH49iYndWdcNiqGXBbJv7KCxBSzxpmFdlK++KA8uEoJUDsc25CnUqn?= =?us-ascii?Q?yJObbRLKsVqQX+UeBlKFR1clJDcf3QgsBsU40jxHTnlNVwQZ7IpwDZ1S4nPq?= =?us-ascii?Q?se180Hdn+EboZ9mu+SiPnztpjicsnNhBeJDGxh25HuLUv/hJ3RCfW1Ni/mdz?= =?us-ascii?Q?TnOLNANWW+t8ibXCpmIsHvQTZAhOW3S9aP6Ptq+1RoK+4izmtFdA1yGS8BPh?= =?us-ascii?Q?ZSTTo0I7A+X62nXylRU1j03u+WVhDEqBkzJW15VxyJ/IGx3hILzEVdcDh/Xp?= =?us-ascii?Q?0dADeD3j7lhLj0v3M7hbtHejOP5eTMohbkBSEWM9tqFPu2n+NchtsxuK9R5P?= =?us-ascii?Q?ALa1QNQbXx4QfjqDvSkzIP6FtJ8fMCwB3R0RfQbRSkth5u1JJ21vOQ9AuxyS?= =?us-ascii?Q?dWdwoK3a3vI5vSENTLreVhQc+lebxcfUHpTZJIiXMoOvFhUDy0964qCv2YxA?= =?us-ascii?Q?1/1JK6iaxK5JhmpIa53SHP+Ryw13alz/GHH4HRX7ZKiT6X6HPpJ28Wen9OUz?= =?us-ascii?Q?y9N/2SJL54wuhGxBEc1stZXkRrOKkCxaF2/kOQiB4xQyx+dVI7oo2l9idrBJ?= =?us-ascii?Q?9RObkl0+WrYFwTiq305uInHHa3NOOx+7QgMjHo3ZOumCxOdRaz9WqHYBjTaH?= =?us-ascii?Q?HuqBNc8LF1puCW6VJuEEnueJ0ewzppFOIouxq6l+jo0+6qP8G2E8/MbU0NN+?= =?us-ascii?Q?MuArZOxESjCjcfwI1nvXExmYkuQ5L35dXcxSSLQnPHr9X5P9TIwO7NTyOxOs?= =?us-ascii?Q?uLVmKRKzseLhvZem0+Aq3jzyBUXB9vPlxIknDdgj2YyysBviE6ZUPErO2MbE?= =?us-ascii?Q?b47Jb+zHFi3oeHR17l6GZ1/hZGTK57tYxbaAfrOEqfrcwBThMkcwojA+OE7f?= =?us-ascii?Q?KlxAxuhCYGGBr9p02KtjJ7p37H4Y1H3l9ro4qiEghRAtD6JfaAfF2Ke8aOIJ?= =?us-ascii?Q?BOq0aRgw2aoT9TZaRhAkQBWZ7StU0ZDVl9WT6n3LKhLJPfDvM5uZj+aMl/Zw?= =?us-ascii?Q?bWxeZeNTovVBGCa8a2dY0EDdcpluZKOTpNt5ZloqjpuMm3WRvIO2L/LYrEKO?= =?us-ascii?Q?681D93Vvp4NH/kjNdtlrq47F5Jc6m4xW5W+krAchKwebTvk6kJ29FTuh6b3R?= =?us-ascii?Q?uzGSmmMdJg=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 729e459e-36cd-49af-27fc-08deadf635d3 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2026 18:10:11.1654 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: M/CsVlKjcM0yoneEjcdJdOFb3P1HtS1QoRLEjmu9u29pD5acLv6XNQ1hKjWK1OVMWM16qmdmdtAOBA2eP0DO2Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA5PPF9BB0D8619 Content-Type: text/plain; charset="utf-8" On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses select_idle_capacity() and prioritizes idle CPUs with higher capacity for better task placement. However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone. Introduce SMT awareness in the asym-capacity idle selection policy: when SMT is active, always prefer fully-idle SMT cores over partially-idle ones. Prioritizing fully-idle SMT cores yields better task placement because the effective capacity of partially-idle SMT cores is reduced; always preferring them when available leads to more accurate capacity usage on task wakeup. On an SMT system with asymmetric CPU capacities (NVIDIA Vera Rubin), SMT-aware idle selection has been shown to improve throughput by around 15-18% over NO_ASYM mainline and by around 60% over ASYM mainline, for CPU-bound workloads (NVBLAS) running an amount of tasks equal to the amount of SMT cores. Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Christian Loehle Cc: Koba Ko Reviewed-by: K Prateek Nayak Reported-by: Felix Abecassis Signed-off-by: Andrea Righi Reviewed-by: Vincent Guittot --- kernel/sched/fair.c | 119 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 113 insertions(+), 6 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 960a1a9696b98..6f0835c15ee11 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8018,6 +8018,54 @@ static int select_idle_cpu(struct task_struct *p, st= ruct sched_domain *sd, bool return idle_cpu; } =20 +/* + * Idle-capacity scan converts util_fits_cpu() outcomes into preference ra= nks, + * where lower values indicate a better fit - see select_idle_capacity(). + * + * A CPU that both fits the task and sits on a fully-idle SMT core is retu= rned + * immediately and is never assigned one of these ranks. On !SMT every CPU= is + * its own "core", so the early return covers all fits-and-idle cases and = the + * core-tier ranks below become unreachable. + * + * Rank Val Tier Meaning + * ------------------------------ --- ------ ------------------------= --- + * ASYM_IDLE_CORE_UCLAMP_MISFIT -4 core Idle core; capacity fits + * util but uclamp_min miss= es. + * ASYM_IDLE_CORE_COMPLETE_MISFIT -3 core Idle core; capacity does + * not fit. Still beats eve= ry + * thread-tier rank: a busy + * sibling cuts effective + * capacity more than a + * misfit hurts a quiet cor= e. + * ASYM_IDLE_THREAD_FITS -2 thread Busy SMT sibling; capaci= ty + * fits util + uclamp. + * ASYM_IDLE_THREAD_UCLAMP_MISFIT -1 thread Busy SMT sibling; capaci= ty + * fits but uclamp_min miss= es + * (native util_fits_cpu() + * return value). + * ASYM_IDLE_COMPLETE_MISFIT 0 thread Busy SMT sibling; capaci= ty + * does not fit. + * + * ASYM_IDLE_CORE_BIAS (-3) is an offset, not a state. On an idle core, + * fits +=3D ASYM_IDLE_CORE_BIAS rebases thread-tier ranks into the core t= ier: + * + * ASYM_IDLE_THREAD_UCLAMP_MISFIT (-1) + BIAS -> CORE_UCLAMP_MISFIT (-4) + * ASYM_IDLE_COMPLETE_MISFIT (0) + BIAS -> CORE_COMPLETE_MISFIT (-= 3) + * + * ASYM_IDLE_THREAD_FITS (-2) is never rebased because a fully-fitting idl= e-core + * candidate early-returns from select_idle_capacity(). + */ +enum asym_fits_state { + ASYM_IDLE_CORE_UCLAMP_MISFIT =3D -4, + ASYM_IDLE_CORE_COMPLETE_MISFIT, + ASYM_IDLE_THREAD_FITS, + ASYM_IDLE_THREAD_UCLAMP_MISFIT, + ASYM_IDLE_COMPLETE_MISFIT, + + /* util_fits_cpu() bias for idle core */ + ASYM_IDLE_CORE_BIAS =3D -3, +}; + /* * Scan the asym_capacity domain for idle CPUs; pick the first idle one on= which * the task fits. If no CPU is big enough, but there are idle ones, try to @@ -8026,8 +8074,14 @@ static int select_idle_cpu(struct task_struct *p, st= ruct sched_domain *sd, bool static int select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int t= arget) { + /* + * On !SMT systems, has_idle_core is always false and preferred_core + * is always true (CPU =3D=3D core), so the SMT preference logic below + * collapses to the plain capacity scan. + */ + bool has_idle_core =3D sched_smt_active() && test_idle_cores(target); unsigned long task_util, util_min, util_max, best_cap =3D 0; - int fits, best_fits =3D 0; + int fits, best_fits =3D ASYM_IDLE_COMPLETE_MISFIT; int cpu, best_cpu =3D -1; struct cpumask *cpus; =20 @@ -8039,6 +8093,7 @@ select_idle_capacity(struct task_struct *p, struct sc= hed_domain *sd, int target) util_max =3D uclamp_eff_value(p, UCLAMP_MAX); =20 for_each_cpu_wrap(cpu, cpus, target) { + bool preferred_core =3D !has_idle_core || is_core_idle(cpu); unsigned long cpu_cap =3D capacity_of(cpu); =20 if (!choose_idle_cpu(cpu, p)) @@ -8046,8 +8101,13 @@ select_idle_capacity(struct task_struct *p, struct s= ched_domain *sd, int target) =20 fits =3D util_fits_cpu(task_util, util_min, util_max, cpu); =20 - /* This CPU fits with all requirements */ - if (fits > 0) + /* + * Perfect fit: capacity satisfies util + uclamp and the CPU + * sits on a fully-idle SMT core (or this is a !SMT system). + * Short-circuit the rank-based selection and return + * immediately. + */ + if (fits > 0 && preferred_core) return cpu; /* * Only the min performance hint (i.e. uclamp_min) doesn't fit. @@ -8055,9 +8115,33 @@ select_idle_capacity(struct task_struct *p, struct s= ched_domain *sd, int target) */ else if (fits < 0) cpu_cap =3D get_actual_cpu_capacity(cpu); + /* + * fits > 0 implies we are not on a preferred core, but the util + * fits CPU capacity. Set fits to ASYM_IDLE_THREAD_FITS + * so the effective range becomes + * [ASYM_IDLE_THREAD_FITS, ASYM_IDLE_COMPLETE_MISFIT], where: + * ASYM_IDLE_COMPLETE_MISFIT - does not fit + * ASYM_IDLE_THREAD_UCLAMP_MISFIT - fits with the exception of UCLAMP= _MIN + * ASYM_IDLE_THREAD_FITS - fits with the exception of preferred_core + */ + else if (fits > 0) + fits =3D ASYM_IDLE_THREAD_FITS; =20 /* - * First, select CPU which fits better (-1 being better than 0). + * If we are on a preferred core, translate the range of fits + * of [ASYM_IDLE_THREAD_UCLAMP_MISFIT, ASYM_IDLE_COMPLETE_MISFIT] to + * [ASYM_IDLE_CORE_UCLAMP_MISFIT, ASYM_IDLE_CORE_COMPLETE_MISFIT]. + * This ensures that an idle core is always given priority over + * (partially) busy core. + * + * A fully fitting idle core would have returned early and hence + * fits > 0 for preferred_core need not be dealt with. + */ + if (preferred_core) + fits +=3D ASYM_IDLE_CORE_BIAS; + + /* + * First, select CPU which fits better (lower is more preferred). * Then, select the one with best capacity at same level. */ if ((fits < best_fits) || @@ -8068,6 +8152,19 @@ select_idle_capacity(struct task_struct *p, struct s= ched_domain *sd, int target) } } =20 + /* + * A value in the [ASYM_IDLE_CORE_UCLAMP_MISFIT, ASYM_IDLE_CORE_COMPLETE_= MISFIT] + * range means the chosen CPU is in a fully idle SMT core. Values above + * ASYM_IDLE_CORE_COMPLETE_MISFIT mean we never ranked such a CPU best. + * + * The asym-capacity wakeup path returns from select_idle_sibling() + * after this function and never runs select_idle_cpu(), so the usual + * select_idle_cpu() tail that clears idle cores must live here when the + * idle-core preference did not win. + */ + if (has_idle_core && best_fits > ASYM_IDLE_CORE_COMPLETE_MISFIT) + set_idle_cores(target, false); + return best_cpu; } =20 @@ -8076,12 +8173,22 @@ static inline bool asym_fits_cpu(unsigned long util, unsigned long util_max, int cpu) { - if (sched_asym_cpucap_active()) + if (sched_asym_cpucap_active()) { /* * Return true only if the cpu fully fits the task requirements * which include the utilization and the performance hints. + * + * When SMT is active, also require that the core has no busy + * siblings. + * + * Note: gating on is_core_idle() also makes the early-bailout + * candidates in select_idle_sibling() (target, prev, + * recent_used_cpu) idle-core-aware on ASYM+SMT, which the + * NO_ASYM path does not do. */ - return (util_fits_cpu(util, util_min, util_max, cpu) > 0); + return (!sched_smt_active() || is_core_idle(cpu)) && + (util_fits_cpu(util, util_min, util_max, cpu) > 0); + } =20 return true; } --=20 2.54.0 From nobody Sat Jun 13 03:29:22 2026 Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012044.outbound.protection.outlook.com [40.107.209.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1C8A3BC668 for ; Sat, 9 May 2026 18:10:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.209.44 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778350220; cv=fail; b=nrmGXdleX0Hy0WBeE69gnvewbVDdwCu/P54yt1k4AUvGwTJcryPUZfm7y02PED0mKrCey6jCmh+8r9jFsirgIMhnz09HJzWw50DmWQpzkA6cD3C25XLprkXnYtorn7Epvv6tdzrHLhsh1+7lH3MX4iahEMfe2h31DDi5YnX2Ih4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778350220; c=relaxed/simple; bh=sccEN1SyvVGmKrY18Dm5TFmcdNEPwmZSI4z8fA5yv7k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=SAIPvLR+fve/y+66Xstz90+4BILI9qYpNJG5+QO9JtKPfiQN2TRtRMJ6n0krtQ2oKhkRRls2+rZSPevOuvqyblT20ywWGVX6R4Hc6skbHB4mN9aMaJ/tFBSKBCKNsSC7aJksbo9VzzXSgLG2hdyzLKVXUE4xax0+fHR3A7vjZtc= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=hbSmIJKI; arc=fail smtp.client-ip=40.107.209.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="hbSmIJKI" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=aKRu5IrX7ANeVVPZnLn4pPfMYr8GMTXBQe955rXP/wO86DKoHxQ3mPRmIL7ROl9tzWIpXQyL/zazyMSSJl+hcx0Qp9BjAzgpQihkxnVtAFxsNWzuj/8oMuMR3MdAtCXOjvB7zr3sbH/YRUSMfTjLB9cOjxq3JJVX3CnqE6ZfHuhxx1wnVQKg4L0gYAwAVcuvRTPkvbInA3eBJ3s+n0KXUGBlyUxhaHchjKeKCBiEs0yYc0ZRMp6ijcwbu9iNZfsDUKHOP7j0X2Pl+WriRQVph7vfiVIQ09KozLxZNfMppxQwhTczW+zur6Xp+2ypzt+v15F3nT/yGF79rN+LM1eLzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TJyYYFnMhT/fB3OcR/rIPJ0TPlmtMqiWyCqu1pqsSms=; b=nWeJoHoyIz71oI3pgEthYAdHNDooqUNb/2E6qPkHq2UlgOedrid/kmz6P/sfa82AebrjsXN7SdWhH3s2TqUSbs+axWJqtK0kg6reE5KKLzBftCXNMcwP8dTCWmcMqrIGO7n3T+IAnaxAY80f+VlBumUMvmfhYJRHcM4DqCQ/LEWMvxlZmmDGDXTMRxeCEl9xwiHogNpzLrfyuoChmCnjQ+HvuEc3gP3jzdCL6FfIP9yfYTGMVa07GbUEFqpMnCyco04cKqaLbCql83zw9d+ny4wrrqi6qaeW1/Bna/gcpiN0camPEjnAY/LQqU0ibumsa5lbz0FjW1sNY+tDkU0log== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TJyYYFnMhT/fB3OcR/rIPJ0TPlmtMqiWyCqu1pqsSms=; b=hbSmIJKIv3bxyR1XcyIJYBlzxCaZKp/21p7QHhMvKzPoK/UsDD+PQ5RSRXI5t97NCHqk/ERaeWBqYiJnG2IVw5Fv8ouQhSnPNL9qFuqTKvYWYWl7/LV2LF8izhh3KGTKSr/Kc7C7bcCEMjcngJjJW5G2p7qw9M4EbO2jDsPprOCRW1DE3qHyG2IgLWotWJ/lux0k3ZpgZfL0NBQzUygP/akhyPOii7K6uQn2Rj5prZf1p7oosiDqhzGOO+j9Fa+OhsaklkmvCHPzqXxejcd+dT3iINJvk5D44mecLTIGyISh3npkKJl6TEHwaRL0cY4RM6Aql3/Ak4xrsM+LePkfOg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by SA5PPF9BB0D8619.namprd12.prod.outlook.com (2603:10b6:80f:fc04::8d8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.25; Sat, 9 May 2026 18:10:14 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9891.019; Sat, 9 May 2026 18:10:14 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Phil Auld , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , linux-kernel@vger.kernel.org Subject: [PATCH 4/5] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity Date: Sat, 9 May 2026 20:07:28 +0200 Message-ID: <20260509180955.1840064-5-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260509180955.1840064-1-arighi@nvidia.com> References: <20260509180955.1840064-1-arighi@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MI1P293CA0021.ITAP293.PROD.OUTLOOK.COM (2603:10a6:290:3::14) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|SA5PPF9BB0D8619:EE_ X-MS-Office365-Filtering-Correlation-Id: 56b724a4-f99f-49e7-b039-08deadf6377e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: I9lOq/pcs6NHAqnR8fqUzm8LsGsgDH0VEG7zkkZlpAncHpoVlJBhxG9vfT/OWuH6W3G2d9ZfNFUFeW21bmlLHLyBWdS2dkUYvjsxXx7jDxqK+NVdEOj9KgnRCluRThF1yX/6ohw3LgGrtq/9iJ/qfv4WSNVs43En6NjqBN1Nou9VY8MKDFr/hP1aDRaxk1qcD65LIoV5baXnP0jKaTE8grffE5YPXrhLG5sGA+er1wRwSeBKx3QOCC+kedWjkB4RIuTaE0C5bFv3Qw4esgW2L0KuR6R0W7BQIkQ3RL3jxFutdPglvp1BQzfzRylY/Q81vtv2AQf+R/MEil9+VC6WWkKq+iast1XqvIX40nZwP/Y+eIZTw7mGxd8fkRDqJVNbutPP5qPNVRxhbkdxhX2zCAZ50GWV6qDy8U/goZ7REIl3pftTmPBHsbIa7hR4xNrKr5jHzKJUKlVeoMiVwP9XdZT5WT/3c0OBDzo1ey9aTOPAUb0rj2t/HR92g7QHOa5GfnSzYcWbKFoYHwcfT16KZ8RQHkNcSLsEblNmMBMhFKMP3fpmMnic3A6Yhhg6+qTZVwCzJuiOJi/dW4MCDktYtKHNZLdg891IKK5U4cuKvzwp1dKHpF3xL76uzCKkspBs8ZbmTWf93qRD4aBNTHgKfIu9J4I7yQst+s1Mz2+yYkZf+YZ3j8UYSRcPXrKqWvSd X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Ri+faBLXW7HkAXfazKSa4BRGdrmIYFrvv6m6NNjxHF8nSlDX7GT65nFCczO0?= =?us-ascii?Q?PGXEpxjXgDAoW7LnsD+wrZUb/20GuCfKoQlJ4V6BLHmaqbaczkzNn/E/rleZ?= =?us-ascii?Q?P72FpcGXSd4+5wgPDKCwMCfIwSoVULe2U1tJ1jUqHrU0+XC4ru4mFyYT9HNg?= =?us-ascii?Q?1Rk9p6DEwciTY4GmQDThP6ptmnSD0okpifp3V0pZMszK856HgHce+KXMsheO?= =?us-ascii?Q?fkkmlifSfCT4DnD9WzT+E7YeZUDPTf1ZQk4plmlsRRiuClOX5t3rszwUayTU?= =?us-ascii?Q?0+QBZu2Xp0wG5LF3FnThcQgaUa5jno+BBOMBC9mTgYaFHyl1DD3KKEblUCp4?= =?us-ascii?Q?KQzaNL3Z5kC8oknUrdSFPu/h2QEUsA17q6uHnWRE9qbg048ayV74M7eXuO1T?= =?us-ascii?Q?Z0maOB7wkLfnETsfHLwAc/MjqHEmpZ0xaHzuT9ve3muX+FhhjlMlOf+qRH+v?= =?us-ascii?Q?kRqLmDrQ4d9dwzZe9TXkq178MVhI37Bz7VynMakwsmmWnnlgCohoov9LgdUc?= =?us-ascii?Q?wr03P1Pluy2q24lDhbtmqyOsnYbwOqIypZrNq4eENIcgUihIBxDpMP4Mcqse?= =?us-ascii?Q?cMiPr5g9hkY5hTi70/JBjydjBMmxiDBt/2Uw/TtEa4Oq2ZI/vy6xAHaXp4Jd?= =?us-ascii?Q?C6q0LA8Vw8EMCsSwJtoMVJc/79GymwkfZMwuqd/QfEcnR5GAvTPScuj+yNDQ?= =?us-ascii?Q?P2ABbqp9Cnr6Z7Ps/+DkfQHg2nztgUXqbb2tBLojkaLtwB563fri0cg+c8+R?= =?us-ascii?Q?vLeVd/q8ysUZMczgnjcSMh1ebV0LwWWMC0YRj7Hy/R3QXWoDNoJOqmDKZ8yP?= =?us-ascii?Q?jutGZSOXkxinFfSgoTL/InhqL7AoTcy/6oThHoNjn5NeEt3l6/KDlAGDwM/c?= =?us-ascii?Q?Qwj9uCwE5J/7UT1VKkhtdEbsgmZ5MEUZ1xr7EDzAPvW8qgVfwP4kRsfwMJ7t?= =?us-ascii?Q?VxRFd1+Oc56sDXGfvj5UuSwsvY8CPA/LnvcuqJvmzjMTL2j0Sye2vNgcZTaN?= =?us-ascii?Q?X0+gZWCN4ADT3FGtYTX4Ccv3PdscWFSNvEiBkUpyREu4iXaLca5sAJuHDn0n?= =?us-ascii?Q?l/YdYVFoBvLTsayL8j5FidsDG9pNY32qilzBeWhlnD/ZxWn3Erujc9mGg8PI?= =?us-ascii?Q?FvUKpMQXflucILxVEZrE0qQv/jgC0DCeARLGMekzgC8tZIlLuC7JJX4aKnXK?= =?us-ascii?Q?7XQPHOSgBE3sR/Y+OYfEcoWeh7Q9OvKqAinrV4/plOkn7SXuVDnAYdPZcBbX?= =?us-ascii?Q?ArY37R99cm8hSImz8FnMnWE1P0O5Gp4SsrruWOxh4s/SCGgbzycvC9fIglth?= =?us-ascii?Q?Nkh46ufj8Sdsj8dPxI5fm4Yy8yXSR0GV49L+4MycP7bF8rTLynxiOI6ACS7l?= =?us-ascii?Q?C9s9RwsOE+Sn/lyJvKp5c5ifXrEEwRj7gJPDV9f/X3Y7MNSx3y1fVnXZFoHB?= =?us-ascii?Q?YwtsWYMXkkAC0P9HU+Nwx6IcVPU3GZx94x6R+F2s25yj/5uPOuuc+XvwkTuz?= =?us-ascii?Q?TBvR6zpQ6ILvL62KOjaHeTKfjYNo69MO4VbFz5Lbtaw1VcEw3YRuzK7UygXd?= =?us-ascii?Q?SfdpsQh1gT3LYk7C2TVUZQLXs3wE2X1mzSRWMCTuon1PuVvOF8s4ZMWw+rZk?= =?us-ascii?Q?iNYcHvUf2fijA4+6MPBh8k0CNm0H8pwZ7+0wbAI8iOrkK6mN9c2pYSnEvkHY?= =?us-ascii?Q?P2xEpU/zCdz4GtqigtVqpZOUEQO5LqJSiTKE28Jk6jjr0/CBPUK87D2uBHKu?= =?us-ascii?Q?qv4H4Gn94w=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 56b724a4-f99f-49e7-b039-08deadf6377e X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2026 18:10:13.9310 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JnwInhaaaWToCkX1+7rnI2jg6fKv25dIvvQkxF/4vMCo9FLIZVQXj59YJnCq879CatangxI+qjzX8KOiI5Q7Uw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA5PPF9BB0D8619 Content-Type: text/plain; charset="utf-8" When SD_ASYM_CPUCAPACITY load balancing considers pulling a misfit task, capacity_of(dst_cpu) can overstate available compute if the SMT sibling is busy: the core does not deliver its full nominal capacity. If SMT is active and dst_cpu is not on a fully idle core, skip this destination so we do not migrate a misfit expecting a capacity upgrade we cannot actually provide. Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Christian Loehle Cc: Koba Ko Cc: K Prateek Nayak Reported-by: Felix Abecassis Signed-off-by: Andrea Righi Reviewed-by: Vincent Guittot --- kernel/sched/fair.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 6f0835c15ee11..2ddba8bd27e59 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9693,6 +9693,7 @@ struct lb_env { =20 int dst_cpu; struct rq *dst_rq; + bool dst_core_idle; =20 struct cpumask *dst_grpmask; int new_dst_cpu; @@ -10918,10 +10919,16 @@ static bool update_sd_pick_busiest(struct lb_env = *env, * We can use max_capacity here as reduction in capacity on some * CPUs in the group should either be possible to resolve * internally or be covered by avg_load imbalance (eventually). + * + * When SMT is active, only pull a misfit to dst_cpu if it is on a + * fully idle core; otherwise the effective capacity of the core is + * reduced and we may not actually provide more capacity than the + * source. */ if ((env->sd->flags & SD_ASYM_CPUCAPACITY) && (sgs->group_type =3D=3D group_misfit_task) && - (!capacity_greater(capacity_of(env->dst_cpu), sg->sgc->max_capacity) = || + (!env->dst_core_idle || + !capacity_greater(capacity_of(env->dst_cpu), sg->sgc->max_capacity) = || sds->local_stat.group_type !=3D group_has_spare)) return false; =20 @@ -11485,6 +11492,8 @@ static inline void update_sd_lb_stats(struct lb_env= *env, struct sd_lb_stats *sd unsigned long sum_util =3D 0; bool sg_overloaded =3D 0, sg_overutilized =3D 0; =20 + env->dst_core_idle =3D !sched_smt_active() || is_core_idle(env->dst_cpu); + do { struct sg_lb_stats *sgs =3D &tmp_sgs; int local_group; --=20 2.54.0 From nobody Sat Jun 13 03:29:23 2026 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012024.outbound.protection.outlook.com [52.101.48.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6EBB2253EE for ; Sat, 9 May 2026 18:10:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.24 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778350223; cv=fail; b=OFYHlTVKIJxj4foWYicrAU7W6RnqtQwvKBrUf8+LW/gjj8XbbdKg5Dt+Zm8gl9yW5/WoQE78jfrMMC1mZrDdPBeCMmRcQUoZGc7h1sleqA76g7gWdvwpidq4U+7FHdnCkVFtqXqFv9EhOK1g5n3OD5aCuNtKHKbsUMKCJzyIMwI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778350223; c=relaxed/simple; bh=Vk3Qh9ofzVe6R8gN9UXWeYLJ8OquMIKkCg9ytcpJWgo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Qn3kwOKOLGBgyW2+6sdPybD9+p/NeHIY5H2W8Awepptb3K23ANWyVx5TLUi10eZQaW4CPf8wV/AJDw5ShM5A/hIrAuYd1+I3bDpAuXNtBDjr4L+DahYDchZT2IuMsp+mrz7hG2SVPFMnT57Wq6zjoNMz6e7hiML+QJokER9Mya4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=LF/GV7vy; arc=fail smtp.client-ip=52.101.48.24 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="LF/GV7vy" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KJK1XEaxNYIJU7FI8HSz+4Mg650H3lLkIHQuq4ha4kaa0gUi3XxcC8rZ5LRsfMiLulg0bqP47EnanLeH8Hy+IwozA4KpKM8wQt9kh5ShDazoJJuY1lf7Mm5hxJOHhKbYEJ/jWdzbJ+UGvM/IDe04MHlPzSBmywGrimdULMUPrt/NAejdQawULM1KBFCfDReF+8UuRpGybH4ILzuwYtGoxs366b3nInx19xaFep2rkQV3NXumfwxTxSsH/2G9NMCuL0SaOaVivIr5T/eCU4b3dz1W574B3b5kFLhnLKAEen69NOEz43wKdtTftukL3Mp3PjIXNqTxcYRVBiai21lF1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qQuQNDdKy2tH8Hqy87irccrzQr2UCUY1IscMjupCit4=; b=fK8FpXLR4OHgWL5FeaLHm8qaQOOLcdqJIlBFa3livb2dnVlDN4zPi6oIodnXUQc9C1zSKu0cUQTTHU/G8sD6fm0YVFshmuPwIhQh3+ZiUBs5rQt2QLoFCLZP8wKAh76EJ2uxqydC1gk+iB/0paWxjaoVAe9Py5SwaYVDjG3b8oCHN6By8hHDz5J0NgmMmowYlQ7OMw2YpNzUBEoIexcoQXTTn061/XclwmyA7TwhMcwBIRIEWRPXHortzsMvHr/L+NQv7B/bNwhh2IYQ3WOZEvIunL3UUYbAO6B5mxLUSqLGyhIGvMpibaPJ3dEqNv4SSVHMalo9YTrivuO+p3a+Yg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qQuQNDdKy2tH8Hqy87irccrzQr2UCUY1IscMjupCit4=; b=LF/GV7vyhk+Gt0NP5MdpgJ4kZIJSZHz5NTtDvzkQIlpieovFLUokfUOIQDmmXRF59B7J6RfhNr8cl7/MPidGkMTTxZ0Gnow7LsSgcMUD79dN2WFfmKZI/7Mc4XDj3S3NNaNF5Gjoer+H/dI6SK7vYukEIUCup2mdg/k9efd2m+r3rOlQzPLZOt+lmN+DKXap6GdRzarIlPb8bdQmaVvItGoioyml344fbc313RCk45PBJxna+QvthHTai4WRREBZo98K8gIwBKlBJ1J29KV3iRJMaOGqGAl3uiAsH8c3GR+1n05kxWRAMND4mFFnLGzbnWqrpUX2CC6WAyMul4bZgg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by SA5PPF9BB0D8619.namprd12.prod.outlook.com (2603:10b6:80f:fc04::8d8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.25; Sat, 9 May 2026 18:10:17 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9891.019; Sat, 9 May 2026 18:10:17 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Phil Auld , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , linux-kernel@vger.kernel.org Subject: [PATCH 5/5] sched/fair: Add SIS_UTIL support to select_idle_capacity() Date: Sat, 9 May 2026 20:07:29 +0200 Message-ID: <20260509180955.1840064-6-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260509180955.1840064-1-arighi@nvidia.com> References: <20260509180955.1840064-1-arighi@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MI2PEPF00000B80.ITAP293.PROD.OUTLOOK.COM (2603:10a6:298:1::417) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|SA5PPF9BB0D8619:EE_ X-MS-Office365-Filtering-Correlation-Id: 3237cacf-30ba-47d6-33a1-08deadf63995 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: JpJk0niDdJhOhhzGAo0xxb/Uf8hiU3u39AlV7t5hH5PGXKcvHrgGQZQK5GiHjsoQnW5DLvBZsNySx3NnD5i5Au/q+vIAxaW3gQx7hGyfDhEmCf38hHSzc/adhKJeo42euafpnz5C9Opw/fAZVZJXrfHAMwW1npUuvLDN3kaT/GyoLIHMZGL9mb7DcdHKMR1mDJZYvLylYnBfigIfJV5HBPqM1bUqBu2SB5hLzogO0TUgrVmv/xX1H7Y8xnsGtatZE0Bin1oXcD+Yd4cIZQnjMHjONbj1BYooU/9N6hfzrWFGPpFvRc0+TKVovSqKLBHkxfaKdEvXw4KQ1USBEYgZy9/FlKMsgYit1pwYfBLMkPQBeNuCpVC62DLLbFWq+Un9RW0Av+4OnkXoX11yPd28Cp0UkXi5W53uOrA0/WLuDb8dJrtOhzTZADNI0D7qZXod+8zg6o71Qj/N+yrdY3Eye//7LpN/MMwi8z2TBf68fAZXkg7P/bsBp/7jQaLQlpKvmHslemo/SUYYNbCwAJ1eM0m2YDiOgfdigN/4rKV0GYh3T2UIa6QUlKin1Vk/v4Xh72lx+AbVYCshHumNigjfOBr3YfX3onnNs2AdiWP2O0x2XtASFqBr4O8/XDu4yZUmk6BmAVdOStg/GQ7OiuBm0F5BdZhqrpix8sfj6SHVVBrAUd8cqnKL8EslfKlVqiQS X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?A3/Ti91kRkoYgWoql7Y2Qkps9Nkb8A81N1mO+ebGu/jn+OHx4avWZR3diHLJ?= =?us-ascii?Q?JyR7GQpZwqoCAUiUR7N6k0NKOacyma1b8DOeYUCuca+Cd8yHuM3OfFRMqVdv?= =?us-ascii?Q?6Bj7UlCz7471R5dyrBLTiCoQpO/+gqPgPBMxiuTeICi9MLbioJzs78COEQMJ?= =?us-ascii?Q?xpEbqfCrhw4U0ZO0UcwIPvSaZ5WRMBxA9a2EzFzKlJB+yg7zr9W5SsACUczA?= =?us-ascii?Q?sHrRFYrupLa+H4YUrUQrVur9TwtyDGX5utvmQbwKQFyFpltHbb+CoM69nELE?= =?us-ascii?Q?u9QcR9MZzdcSvac7OOlaeYrw2ZTpLsb+bTSpxx4biZNg5slUybglJ+DnnD3s?= =?us-ascii?Q?5+EY+SyTqWwyVr8dbkFe9uTzPPp8rAmcymwK4Nl8UE1Me4DC27BjQjWWU7Aq?= =?us-ascii?Q?EtIgpYc38Ezvmmo9TE8+M4cAS3i53ISEhq6mf6AdGLn/kR+dnfg3ow4Zdjek?= =?us-ascii?Q?QoyAL82O0xnyOV8ehsDnMIFZIRodprOjaU8aICGHrPr0ex/S9tBnY5ttx4H0?= =?us-ascii?Q?AHm6ZxcXXvVy5+a1fhWOX0R8/yQ2w7WdHR5XlQtBQ4G4+L7geSRUHtS1/pcB?= =?us-ascii?Q?/HZkjkuwIuQXlNkIJ7rMi0HBBSXNkAz/u2FoOcssp0lUY5/lgExtxsVFCzbq?= =?us-ascii?Q?1genwNEUMGhjzprZWxyBG37glWnMvCgxO+lkjf7Ne9kxP9d5+5QbVq1aB+JS?= =?us-ascii?Q?8PVEAQxufYQriHuxcXR7Gtw+ox/RYlbNmJoCRklJQQ10+DF5Et9ZDgKDiu/E?= =?us-ascii?Q?LdRS0MQiqXN+vmSf09L95VNrsYwUsN0I/4Xb5FIRt4gzvdB6v28zB2eJDkhz?= =?us-ascii?Q?x1OsQ/koHfNuCI8gKyW3/km60w8xZzfmF/bbYcow2Q2XHVqV2YldtQTMYW4c?= =?us-ascii?Q?8R5xb/2AQwHF4tppWXYakQgkBAlNbjIOdzlkjGsszDer0BG0rwfYN8QQDoMq?= =?us-ascii?Q?A7FjmVpBs6cm4BooYsIy0taO4jbAGKo/9Vrd+ZD4rOLJHXGEF2fuk4j2GAmM?= =?us-ascii?Q?qBf7/Vuly688YycfADgGRlHrt5A5i0RekDtnreLViv+gdWxdWdGNIXD7vlq5?= =?us-ascii?Q?jblVk7lcIRBA8Onrlq4bJCL98YoxoTiah7p4MIimSbgGlJ5Cfnj2pAjeuE99?= =?us-ascii?Q?vgjBdJgAd6xG4VmWZPBSkLZZQMA/RxMNO/gOp496yTDhULF1Zw4z36/VOBtS?= =?us-ascii?Q?pVt0ON3KrxVU3uWnDbv4yMXMMJOKAxQ6Mz4kvi9JAlHWqNQLB3d5OUggFu/1?= =?us-ascii?Q?wYuZTrBbOaiEGR4sgh3RSoVfWdRpgbz2//ag/gQHuWiLSCvGN8QjboiE5nba?= =?us-ascii?Q?GhHHUTVKtJrHWGd5dEV+OATa1LLbhEYU/IBRFP0mbhce2Nh1Zd2Ms1rb4J19?= =?us-ascii?Q?fh+tbM8JxrydQcWt9WzyE/n48dZA5Ys11s2UOXKIUe23S3wZCN566UP6H2pV?= =?us-ascii?Q?qjTUK48dujXMuQdeZyL2xeBfD7/jlXgJmkLIrVFaERDbCFwyJWZ3G+gR2o9v?= =?us-ascii?Q?YqT6HI4pxKYO4t94lGpw7187JCe82yoFgFJXI6ELNnUN1X8aNWJ8Qrth2Prf?= =?us-ascii?Q?6uDrgqJ64kTdoV+dex9n6OmlZpX4E9/LJnzGEy0PLlRK+Xo4fFawAFK85ic+?= =?us-ascii?Q?VcdlHIQwoeNVy46hs975tVW8vKexzjdvPJa606gTSyS3weqFQtMVP+adepu5?= =?us-ascii?Q?uOZ+6s7X56sqeoYaaV2OA+qd8NS1OtsfUxsDaGyjKaf8D4okDdLJJ6xZ/fci?= =?us-ascii?Q?U4MgsmTkYA=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3237cacf-30ba-47d6-33a1-08deadf63995 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2026 18:10:17.4338 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3LkfdyebIjOOsywbYNsgTC64lQJBnK6U+A0imTwVZxUKAM1uO15sjQkrYgqdFJjePXU9DbUahYyFX3dElfthbA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA5PPF9BB0D8619 Content-Type: text/plain; charset="utf-8" From: K Prateek Nayak Add to select_idle_capacity() the same SIS_UTIL-controlled idle-scan mechanism, already used by select_idle_cpu(): when sched_feat(SIS_UTIL) is enabled and the LLC domain has sched_domain_shared data, derive the per-attempt scan limit from sd->shared->nr_idle_scan. That bounds the walk on large LLCs: once nr_idle_scan is exhausted, return the best CPU seen so far. The early exit is gated on !has_idle_core so an active idle-core search (SMT with idle cores reported by test_idle_cores()) isn't cut short before it gets a chance to find one. Cc: Vincent Guittot Cc: Dietmar Eggemann Co-developed-by: Andrea Righi Signed-off-by: Andrea Righi Signed-off-by: K Prateek Nayak Reviewed-by: Vincent Guittot --- kernel/sched/fair.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 2ddba8bd27e59..494149f14d98f 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8084,6 +8084,7 @@ select_idle_capacity(struct task_struct *p, struct sc= hed_domain *sd, int target) int fits, best_fits =3D ASYM_IDLE_COMPLETE_MISFIT; int cpu, best_cpu =3D -1; struct cpumask *cpus; + int nr =3D INT_MAX; =20 cpus =3D this_cpu_cpumask_var_ptr(select_rq_mask); cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); @@ -8092,10 +8093,28 @@ select_idle_capacity(struct task_struct *p, struct = sched_domain *sd, int target) util_min =3D uclamp_eff_value(p, UCLAMP_MIN); util_max =3D uclamp_eff_value(p, UCLAMP_MAX); =20 + if (sched_feat(SIS_UTIL) && sd->shared) { + /* + * Same nr_idle_scan hint as select_idle_cpu(), nr only limits + * the scan when not preferring an idle core. + */ + nr =3D READ_ONCE(sd->shared->nr_idle_scan) + 1; + /* overloaded domain is unlikely to have idle cpu/core */ + if (nr =3D=3D 1) + return -1; + } + for_each_cpu_wrap(cpu, cpus, target) { bool preferred_core =3D !has_idle_core || is_core_idle(cpu); unsigned long cpu_cap =3D capacity_of(cpu); =20 + /* + * Stop when the nr_idle_scan is exhausted (mirrors + * select_idle_cpu() logic). + */ + if (!has_idle_core && --nr <=3D 0) + return best_cpu; + if (!choose_idle_cpu(cpu, p)) continue; =20 --=20 2.54.0