From nobody Fri Dec 19 13:09:33 2025 Received: from CH4PR04CU002.outbound.protection.outlook.com (mail-northcentralusazon11013026.outbound.protection.outlook.com [40.107.201.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 152EE32254E for ; Mon, 8 Dec 2025 09:31:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.201.26 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765186300; cv=fail; b=NF1i/l14Kb6jR2GDS68ZBMU4wfv6CG0JQbCPOCrUeZ5fwziUGJ2gwmHBMr+Ft9UhzH0xbHM+FEL9g9ovBhreicjHVVua0AXHpe9qaGBinu8xlWWgz3U9gv30R03tglm1uoJQ4iyA162qZ3dGeeyou+MJWYcW3T0fBvq8TqdejD0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765186300; c=relaxed/simple; bh=4PUPooYWuhFAUPn0T2v/5u/QTeZlSnr7s9dYWKknHYU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=MZpjdCcAlxM90BcrPbvU6ceOHI38ojEtmrUQ4FIlHDPTU6cBlYSDQfbxVLQgQ9kYhjZm6WTs6yIpwJx9P0SEtLzEhqQ4ho9baVcaLzMDSF+Z/dsKPkCqZVjTig1Nl5oDvfo7ukADHp/eN/ldiUj0mFS23ctISdhexIDYmAYN7hU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=pP6kKjgY; arc=fail smtp.client-ip=40.107.201.26 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="pP6kKjgY" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=alWPyNBTv1tZ21T9Ga90jg8En//4WvNOEWtJeWlqvDGU1TAVPf3RAX8Z4o8tujQ+gvgjod51cFH1nCJiw+WVgaPrToe2aM9/NxRHxMVXd35YSK+/9lYnDeBeLP6vUc50CHP2ssILj8sMJi/PnCn1vFlQgcs4BTS1H90pNJZSWNuic9rMBDpueURe0IQ/SYGrOoZMq/rjRR7tzGunyFr6tw6Hvcyf8ecBnapt3/0wKYUspjdpcgJpVLj0RmpA3rWJ0i5uf1nwhMFy5nweRNXo3yvzmTR8hvXI/k9MA8bTYdlo6+pqDrfclj4w1F4AtggJYaCfFHRqyLDk8oxCXM+/qQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gSJQSjljO133eWXVP72D3R9DU+5pI2DfeR7H7T27l/g=; b=WQiJMexzVwVzjI1RCmLsI5A533G4L0h2fT6fsx94lwFZa9TZEj5cYgjw1mQ4oFttlgxCfd/wcd5jI4S5d8EzB7DPOPZqdvgnbl3PWa9I5MXWBi/D1+hcZCQ3KUSKdZlwqaZS9zbQFRxYeVn/kJ5uANOHzokQib7NUFzu9b8Xk/5KnsYLGy6GlS3sXO06BQUenKiYwgGhEISawd7O7dVwWqsv4iVzYb9rowT22uMyiYVDWr901M0nec79PgW/BaWl3bRRAOIo7motfV2OkgN9xpHVkSzX4B7JsiPtYfy76QAmoQ0DoIcQSigLfBGecrLGFkfZV7DbHi8/bozH4PA+lA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gSJQSjljO133eWXVP72D3R9DU+5pI2DfeR7H7T27l/g=; b=pP6kKjgYpnGVy4wVWBOme4rY0dLNztJI0G6GJVTF2LuQgWJIAKrdHuG8N3Y+mLrYcFSJS86z8Rcl5jnl1gP6uQW/xmPKZkWEEeeNR+BpmlZdEchEE7PnPj5WgUbJSutZ/GQBNBcWr8zMu/YY75cBEniz36sVLuTDGtObDsvZfvA= Received: from DM6PR01CA0013.prod.exchangelabs.com (2603:10b6:5:296::18) by CH2PR12MB4200.namprd12.prod.outlook.com (2603:10b6:610:ac::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9388.14; Mon, 8 Dec 2025 09:31:32 +0000 Received: from DS3PEPF0000C37F.namprd04.prod.outlook.com (2603:10b6:5:296:cafe::52) by DM6PR01CA0013.outlook.office365.com (2603:10b6:5:296::18) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9388.14 via Frontend Transport; Mon, 8 Dec 2025 09:31:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by DS3PEPF0000C37F.mail.protection.outlook.com (10.167.23.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9412.4 via Frontend Transport; Mon, 8 Dec 2025 09:31:32 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Mon, 8 Dec 2025 03:31:26 -0600 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner CC: , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal , Shrikanth Hegde , Chen Yu Subject: [RESEND RFC PATCH v2 13/29] sched/fair: Account idle cpus instead of busy cpus in sd->shared Date: Mon, 8 Dec 2025 09:26:59 +0000 Message-ID: <20251208092744.32737-13-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251208083602.31898-1-kprateek.nayak@amd.com> References: <20251208083602.31898-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37F:EE_|CH2PR12MB4200:EE_ X-MS-Office365-Filtering-Correlation-Id: 516dda95-2b04-463d-3eff-08de363c92ed X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|82310400026|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?gn5GYMqHF2dg3YAN24pf9232Hh6iii/5thspwIs9DfciRgNJA9RR0T1XUW0l?= =?us-ascii?Q?MaEX5jq4XOQ5Sc2Woa96T5KaHj2Qy1gozLnvyidC+FsYiiOAH2iDmwK4BO7X?= =?us-ascii?Q?J917YworC9z/zsYziQFtRvbcGd0hjdSzph2nGAzssY/eCLqR8eK/9fksjHEZ?= =?us-ascii?Q?8dBdk03TCm+EvUVZevLxE8Z4WDmPT7fyhbcbGDZB3YxU98ocAUVt4OM1PD9O?= =?us-ascii?Q?vStQL08X8jlKxsmCeSdE6l1TIW4m8WpT0L7fi9fnQ2KqRKTBTLg2/5Bqz7Ij?= =?us-ascii?Q?7Se6+9Q+LCcoDxeth7Ze/DlNIZabmHF9htEuFfdwVs+ZQe0kUd0LtMIpewO4?= =?us-ascii?Q?tB4sobA1tSX4nu6WKlPWfY//KbAMPp74gNF8utErr1W4mlz9lwzv7i/FO8Ko?= =?us-ascii?Q?WSd+cLad4tpisgD4fH6b/HyYePpvkKFOxchzgT38TGfUg6sZZljPlcE6AU3B?= =?us-ascii?Q?SGFUjBkGxIY2bu1/4YVadvd1RN/SyM3o2tYt6NeDxcn/u+W8TSiV1wCZDx4V?= =?us-ascii?Q?7/S65zaRXwGtQUJ6GWz7s4ywDjDtwpJ+u4/e4M1Zm6PH11fSPCZo9WV5K0Pa?= =?us-ascii?Q?6dYJh9hfSK3fKW/wQlb1quL5pYF61YrGlOnhALA2icrUdOj0CptNNMxtVqAO?= =?us-ascii?Q?FawDI46bbiAp+1Eic49amyXOQichARjOx6BC5G4P0zMl4fyqfFaarOeTS2hO?= =?us-ascii?Q?ByGK3z/v6Qnd+f9L4mmfZKEsQ0rlHNcU9Xg1Wx3RTso9Fwna6HEpYZKfHJLs?= =?us-ascii?Q?kt8x4886IqwumN0LPkis0v38GHYaIBC2i3CrYpoKc5ciyotxrgwCYgiRolta?= =?us-ascii?Q?rbJvCPq9MeBF7/FMonei7HWX/m5RiOkF+5iVQ76AMfksl192LzovMW+VavE/?= =?us-ascii?Q?QppvtdW1C5gJ7j/xBwS8K0wAzUa6hmq0KRIiBTypI8rzyIv47BgnlOMWAmeG?= =?us-ascii?Q?cqmqJ3eJnd2zwMMi/IU0bOXg5xMH1tllc9SehRgV0lwORctCLrbppbnR4YRJ?= =?us-ascii?Q?ABUKbAt4eWXt9inwovPXb9V0hRd/SvUn2orjkB62a73+aaqxEGX3paSqO2Tb?= =?us-ascii?Q?Zzgm6F+2ArqAyMFUB3lfnKSL15zzapmdG/mK4kbYUqopYMrqDttE6URSL2PH?= =?us-ascii?Q?27ISQNhx5hPzcqFf4pRnG/4+MJdwOtgj94YDXaODD26QxG2xiFPEKKe2dvDK?= =?us-ascii?Q?NoXEVv5yu/DUocQXj6Hk3ab0X6G9vXxDDqU7jyGYUvu2coDXko0ucKlBvtAp?= =?us-ascii?Q?xIJreDc4H0d7UudxWX83rd8MOEnS4VHcPDFEiGiyDHCp7RinwSbv2bW+CVyU?= =?us-ascii?Q?JNw8NeFGxCyJnbXjWYHDMQZ0tcBuqD1tlPw828GLeu6H7/sKXDs+SCpYneXt?= =?us-ascii?Q?GBqx85k30LFsdcq+ENLYYvoFZQOLZ+3tkKJ6uUJXRtnj6T7AsIutKDkdZ39H?= =?us-ascii?Q?gCuW6HgVLz68l1/sYUOJAisUr+ODKJAaY5Y8lAiSwI9SI43OEvY9TbrQRlpL?= =?us-ascii?Q?btYb+MA7FYmI0KU0UytMmRPXkgbZ8kIgY5PlyCovcsHvdhmJSu2iwMb52mL9?= =?us-ascii?Q?4nhVqkRJVo0IudxWBFk=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(82310400026)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Dec 2025 09:31:32.3539 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 516dda95-2b04-463d-3eff-08de363c92ed X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37F.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4200 Content-Type: text/plain; charset="utf-8" Switch to keeping track of "sd->shared->nr_idle_cpus" instead of "nr_busy_cpus". Since previous commit corrected the "sd->nohz_idle" state during sched domain rebuild, the nr_idle_cpus will reflect the correct number of idle CPUs. The idle CPUs accounting will be used for nohz idle balance in the subsequent commits. Races are possible during hotplug / cpuset where "nr_idle_cpus" might be incorrectly accounted if the CPU enters exits out of nohz idle state between the read of "rq->nohz_tick_stopped" and the subsequent update of "sd->nohz_idle" in the hotplug path but these inaccuracies are transient and will be corrected when the CPU enters idle or receives a tick. CPU0 (hotplug) CPU1 (exits nohz idle) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D online() if (rq->nohz_tick_stopped) /* True */ ... rq->nohz_tick_stopped = =3D 0 ... set_cpu_sd_state_busy() ... set_cpu_sd_state_idle() These situations are rare and should not have any long-term effect on the nohz idle balancing since there isn't a case where a nohz idle CPU is not set on the mask - either the hotplug thread sees that "rq->nohz_tick_stopped" is set or the CPU going idle sees the updated sched_domain hierarchy. After the conversion, all the bits that use "nr_idle_cpus" are already guarded behind CONFIG_NO_HZ_COMMON which makes it convenient to put the declaration behind CONFIG_NO_HZ_COMMON as well. Signed-off-by: K Prateek Nayak --- include/linux/sched/topology.h | 4 +++- kernel/sched/fair.c | 10 +++++----- kernel/sched/topology.c | 1 - 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index fc3d89160513..15c61aed1b5c 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -65,7 +65,9 @@ struct sched_group; =20 struct sched_domain_shared { atomic_t ref; - atomic_t nr_busy_cpus; +#ifdef CONFIG_NO_HZ_COMMON + atomic_t nr_idle_cpus; +#endif int has_idle_cores; int nr_idle_scan; }; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index de9e81eeb93d..fef3826a258f 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12533,7 +12533,7 @@ static void nohz_balancer_kick(struct rq *rq) * the others are - so just get a NOHZ balance going if it looks * like this LLC domain has tasks we could move. */ - nr_busy =3D atomic_read(&sds->nr_busy_cpus); + nr_busy =3D per_cpu(sd_llc_size, cpu) - atomic_read(&sds->nr_idle_cpus); if (nr_busy > 1) { flags =3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; goto unlock; @@ -12562,7 +12562,7 @@ static void set_cpu_sd_state_busy(int cpu) if (!xchg(&sd->nohz_idle, 0)) return; =20 - atomic_inc(&sd->shared->nr_busy_cpus); + atomic_dec(&sd->shared->nr_idle_cpus); } =20 void nohz_balance_exit_idle(struct rq *rq) @@ -12592,7 +12592,7 @@ static void set_cpu_sd_state_idle(int cpu) if (xchg(&sd->nohz_idle, 1)) return; =20 - atomic_dec(&sd->shared->nr_busy_cpus); + atomic_inc(&sd->shared->nr_idle_cpus); } =20 static void cpu_sd_exit_nohz_balance(struct rq *rq) @@ -13075,7 +13075,7 @@ static void rq_online_fair(struct rq *rq) =20 update_runtime_enabled(rq); =20 - /* Fixup nr_busy_cpus and nohz stats. */ + /* Fixup nr_idle_cpus and nohz stats. */ cpu_sd_reenter_nohz_balance(rq); } =20 @@ -13089,7 +13089,7 @@ static void rq_offline_fair(struct rq *rq) /* Ensure that we remove rq contribution to group share: */ clear_tg_offline_cfs_rqs(rq); =20 - /* Fixup nr_busy_cpus and nohz stats. */ + /* Fixup nr_idle_cpus and nohz stats. */ cpu_sd_exit_nohz_balance(rq); } =20 diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index a212ae52cdac..6b14c7db3e35 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -2656,7 +2656,6 @@ build_sched_domains(const struct cpumask *cpu_map, st= ruct sched_domain_attr *att int llc_id =3D cpumask_first(sched_domain_span(sd)); =20 sd->shared =3D *per_cpu_ptr(d.sds, llc_id); - atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); atomic_inc(&sd->shared->ref); } =20 --=20 2.43.0