From nobody Fri Oct 3 06:35:26 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2062.outbound.protection.outlook.com [40.107.236.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD8F11CEAC2 for ; Thu, 4 Sep 2025 04:15:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.62 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959359; cv=fail; b=cawdTlifYWoPfvafDahtceeELXdfWZ4aXu3VXHNz+DmbSIP0X4kICvezlH5F3uTgWRcvsXy06yNQAz4nAXIGf5e7bMm67mQIpw0R8WId/EvNnBW6i2SvSZkNr7Suk8MpPxP4uNDqZsBdf4A81/2Vli1/Gn3GKS+vAyttPfc6F2Q= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959359; c=relaxed/simple; bh=lVd/Q0UzB5u+uCv1OfF+NpEYbjWt9DS1IhGsvc+i5cA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=OG28TtpMcHXzMnrOWY1VELoojgJulaEFhtKbbDk/SXtvJKrVeI9eKVoiJ3lG+ZjLLRF6mK/8Ka9nuhTQWzUWg3hxmOvRHEeAxel2kmBNKDqVemTUM7wGOojWAlDX3wxA9D0fxPqKWzvjzNIO+qVbGZc8JJf1hnEYgJfhvJ3APXY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=UJdplAaJ; arc=fail smtp.client-ip=40.107.236.62 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="UJdplAaJ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fDW/vclnPNv5P3RCUfwnxqnAuB/f0tX8hziOCs/qp39++SIcalpvJ4JklSwkxgLmhsZDe739DKG4qfmmDJ+XU1s6kgXXLj+FALh5OvroIUvmJ95r/qS3OTKatqqyYVbB5l/wCydcxPOzvjXvtcG8nObRaN1yPmXAPuWuREKXyJkvLbRAKCkaDb4G9WAGcx+rVnFVORGkcnJMUw7ZgnrRk9S+pIzdF+qBRisfllHJDZ5AxiSXrfKXpXAK9gRlyAqoSCOUB0bOpinojb9kCU/9TJYVyXy/U+quSfXLxXPffqEu8TxJAIP5F1Vz0NhNoQMU8nRvEzmXDZC1L1s/Ht0dYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZDw1dWtOCM7UJwk1ahY7d7+pDXERIyhgrIaZuD8fGC8=; b=nxliWrg/D5AFNsDAlwLK1rHjpb9rcw0Dvg+9mOE6C2IL1dB/D2Mu9gC+SzCy7TLND1bgqtpwNOxiXfifw42YvN4W0fMUjZdln/fNV0r7T7k+X+P5of3s6bnFfHa3Bp+OOTFM0rUYETTewfWzui3dVV8S4zQfDe87JsvX2DkHvlgBzbmUJDyMAUGlXCRaAjOye2rgsIClQgy83vW4XErEBPhyEycMbJWJV2giDlFcs9PHmCLTDFfphi/2B5l+9B6NW0zh/+iAEH+JV0hiEaOsVSZTL8C3B2GaBBkJoRP9Nx3X1lXKf0k9iUUE0J/0BMY402tAwFfxFNsvC9PupLgPJA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZDw1dWtOCM7UJwk1ahY7d7+pDXERIyhgrIaZuD8fGC8=; b=UJdplAaJLledK7QULYGRRj0Bhfg05N6n/2LvMSlxTrzmHV+QGVAKoJTmCaxCRBAGHlvX5khQVTjsucceWSBhnsO5MFwXSHLNcAkBowq46LhD7mLuDCc7fx4IzruoQoSbTJ5jUSiWqUxkFnWVtYuzBjO2clwCGwylfWMkX0vMj4g= Received: from SA0PR11CA0144.namprd11.prod.outlook.com (2603:10b6:806:131::29) by SA1PR12MB9472.namprd12.prod.outlook.com (2603:10b6:806:45b::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9094.16; Thu, 4 Sep 2025 04:15:51 +0000 Received: from SA2PEPF00001504.namprd04.prod.outlook.com (2603:10b6:806:131:cafe::a8) by SA0PR11CA0144.outlook.office365.com (2603:10b6:806:131::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.17 via Frontend Transport; Thu, 4 Sep 2025 04:15:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SA2PEPF00001504.mail.protection.outlook.com (10.167.242.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:15:51 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:15:51 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:15:46 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 01/19] sched/fair: Simplify set_cpu_sd_state_*() with guards Date: Thu, 4 Sep 2025 04:14:57 +0000 Message-ID: <20250904041516.3046-2-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001504:EE_|SA1PR12MB9472:EE_ X-MS-Office365-Filtering-Correlation-Id: ca12334e-5a69-4c80-939f-08ddeb69bc18 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|376014|7416014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?F0v6tci659NoL1CglMBFye4twJGKfxkNBrwyu9qzCh7trZKbmARP+2SitiWC?= =?us-ascii?Q?gR7GOEDSn1NyAaSudIcyu3ReBJ3IaHvGpwaXWwyAD0it0VeBLkIdQwEl7yuk?= =?us-ascii?Q?t1lQx6OR9cncEHEJe8t9wLDfmWXsHtjB6R2WTdenywVIcU6CQvn9K5xJfJMq?= =?us-ascii?Q?q9RJZis6nnXUasVPJw9rtWw1n0VK6uxeTSboL4qd7SWyT/tTmesX0MyZEs/H?= =?us-ascii?Q?bDgVO3pDkMbjhryAIFfhp2mGKxUvTsbp+Su05L8tpsOZpSxPmFL9/X6PKv9a?= =?us-ascii?Q?sxogRHySyRwj34hjBTX6SfncNLQriX8YKCFtchcXtntG8Uc5+krqXDP7Qf2o?= =?us-ascii?Q?I0L44Ny9Wu2KKsPluF4x8VKrFiEwwzZEwNNJ9wIy8t67x7Mgk+bf9eV09Hhl?= =?us-ascii?Q?y18g+qlFtHwR+LjTf0XuYMlMF/38ZzjXjc48zXAhfihJVJoQyPBsqbq7M+J7?= =?us-ascii?Q?SnpMVi1xhthXX9w30NYn1UJIu1w4WTqMyFfIg04/xxUtXUCssbI5i7INDCNB?= =?us-ascii?Q?lbmrhJH1tfDuhb1j/EK/M3Mbi9mjBC6TIYP3zj5Wi7OszI6zZBytEc9z/ja5?= =?us-ascii?Q?7frXdBY3qcZhWjNZ4ItV1bLFJnVOG0Z4d5xph/p+gX7hXsiDtO2TVukwP5Pd?= =?us-ascii?Q?SyBcGMDqMELyHtJzIn9qjfDoCMRNFwrBAo2qJQH6/AUQXFOPSPLLkUFzBflQ?= =?us-ascii?Q?lIH1QYyaMZcogQHd/8r1tWjHGu1lNkgAPkTrs/VN25kX4N2O0FO7nAA8Orbh?= =?us-ascii?Q?L4KJ7QKiosuDBAS23tyAKwOCRyWtBJyM08XvSIdo0tIhF/gnbsSE2pmYWbgT?= =?us-ascii?Q?61evkYE5Oi7aAGJgSvj8cNZBuW+ZXIRweClVu9dtyZAk8rLNjwgqa523sp/n?= =?us-ascii?Q?2C+58FLh/zD6ORsra9swWsWJ8Mggob6bcMUBB9NXivtbd+y1Q9Ew1BLrbTio?= =?us-ascii?Q?nSGTmszeZoQ68rSM+ftNjOIxtKM4OKfJMtwzfrhocJP6p2hqwGEad4TPPVn0?= =?us-ascii?Q?QioroSIWk6+uOJd4T5ekTXCF83/Xve6vZeccu214LM4CF7n2jSAUHQV033bi?= =?us-ascii?Q?LKKQ0PM73BrJANxCpGEp7wOsHeaIcE/3Nlho49U8kbL05T2g6XTZSvRKujcN?= =?us-ascii?Q?1XRXZpQ6kjo3kmsqDYlSvkmpdlqMyGT+oZ3ytA7cK8ObXPp9qUKsGky98hYw?= =?us-ascii?Q?E3FgUMXt3/rX4ZQx6wYGHc3KX7VJJkgJh/gAhL06fQQ+CPtUPGlgIcdP4LJs?= =?us-ascii?Q?gWcCCChXXORCeLcO8xmRphPWIp2OYvB0V4La+tS9BRJ+LZ0GD42eAKGv7Ycm?= =?us-ascii?Q?ytLZgMau9aB/z0+iVS+up5e1XdAbcxNIXeD9ENOTfzclKWpo6PCyckoFqt7d?= =?us-ascii?Q?YIaknozNPTUUUV2FXpQf/ZNaGMC7EXrWKEYTKEZGn4WnO4sYHlExsQsBvWkz?= =?us-ascii?Q?Gn+6lgrDBCdgxS2zL707QfsPHG2FGMnjs4dpATA/zg5ThCYAhZ7xapYIaAuO?= =?us-ascii?Q?bibLVshu106PEkyBrOK1hjxPva/+klK3kfAh?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(376014)(7416014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:15:51.5675 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ca12334e-5a69-4c80-939f-08ddeb69bc18 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001504.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB9472 Content-Type: text/plain; charset="utf-8" Simplify set_cpu_sd_state_{busy,idle}() using guard(rcu). Reviewed-by: Gautham R. Shenoy Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index df8dc389af8e..61b59fd75ced 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12449,16 +12449,14 @@ static void set_cpu_sd_state_busy(int cpu) { struct sched_domain *sd; =20 - rcu_read_lock(); - sd =3D rcu_dereference(per_cpu(sd_llc, cpu)); + guard(rcu)(); =20 + sd =3D rcu_dereference(per_cpu(sd_llc, cpu)); if (!sd || !sd->nohz_idle) - goto unlock; - sd->nohz_idle =3D 0; + return; =20 + sd->nohz_idle =3D 0; atomic_inc(&sd->shared->nr_busy_cpus); -unlock: - rcu_read_unlock(); } =20 void nohz_balance_exit_idle(struct rq *rq) @@ -12479,16 +12477,14 @@ static void set_cpu_sd_state_idle(int cpu) { struct sched_domain *sd; =20 - rcu_read_lock(); - sd =3D rcu_dereference(per_cpu(sd_llc, cpu)); + guard(rcu)(); =20 + sd =3D rcu_dereference(per_cpu(sd_llc, cpu)); if (!sd || sd->nohz_idle) - goto unlock; - sd->nohz_idle =3D 1; + return; =20 + sd->nohz_idle =3D 1; atomic_dec(&sd->shared->nr_busy_cpus); -unlock: - rcu_read_unlock(); } =20 /* --=20 2.34.1 From nobody Fri Oct 3 06:35:26 2025 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2041.outbound.protection.outlook.com [40.107.96.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 982EA78F43 for ; Thu, 4 Sep 2025 04:16:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.41 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959376; cv=fail; b=NUcxeClpgTPlgdBKxBcQ7lCBexaPPrkYM8I0uWZeZym9SnPXi2/vgfIwuDv21CAwLcxCy2IXEBXvZTTnPqjdrn+or0DevKWZh8G/zVTBqoTX1Iy2JKqN9TqIKJAgwZkIUK2cZl0tDpItVTO2o3aHzeTUEK8dw5JKlZXArRf18gE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959376; c=relaxed/simple; bh=5CMoVeo+lSr2x/LAY3GtJa9hw+58CYHL1tEr8k0q2r4=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=g+dGrFmw9I4ibvjcSTc7vW2jRCoZplrF5GajKeZ3udhuRdAtsOPsc3FgD5rebVP/w30Ltp/GMgwBFV3ElCq5Fb7zeGIq4RAcm/ryBOg6ulSBfSOWIJ2sxmIxzaka+n6yBZAnBldRiMK9SPw03oCwyfO1ffrYLdzLUZKFLdZN4fM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=M6SLL1mM; arc=fail smtp.client-ip=40.107.96.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="M6SLL1mM" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MIANTpL60iB+Yipy/YAYwDefWTQYt9Wk+m4sUEQEu5WnCYSW1mybiMRyUqQF1kJFdLa2oZtQvSG/79usW9LaVANaJNaBmkU77hvGPpo6Gtqqs1EDpllMu9iCvXNBhovpaCpURoWlkYYahy9Q97ViaFPtT66c4pz4NqNKL4Ip9BK1JAuovhbynGvQHE1EvyBjXPzg907Ezja0ZnxYnXATfNNe+s4vqFrJgFH00fYUzXyVfpNuavunX7Yn+7F2XCdvUoIZU+OjUsnhRoqxmSxx+OvIXm3RpahNyxOcTw5dO5SAYCW3e4gDCMwPEe8fpNbm0BQxb/cCvwKbVV5tQHeN5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KZm5hOD2EeqPbqgTIlFeXFjiImSIw5UtCsLCiAciRA8=; b=UXq0v7PVrv9GAn97FZULM5PA9zTnyqPtH9ICYbwlvORLsZZUVikoyMzPntc0X0NhM72VndLkvTCOOr94R65TaelQKkYMfa5HXCarjbR9rwVO8ScqCwCoUlH/JUFfqhGJbKJkQ3rwwtT1mf0gJHghxIuNO2XLdLffTRlel9uVA6l+auMFiz65lEfWSZfcxHCjHrPaZ0S1BwUmcY70rjp2Ir7xGdV8OKWTuEwDpzjvDaI5DiMt5lLuEzt4MQllBBcGxLO++1py6DX5eBSXME1pscC+UK9JXUzC384GkLXjfcrMeS65L2Ktrox5kDDwmInC3EImFuNKFpoivG3O1/zsXQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KZm5hOD2EeqPbqgTIlFeXFjiImSIw5UtCsLCiAciRA8=; b=M6SLL1mMSv/q3d6DY71eWqfnHWZcdVj0bZltfwTX1G/TcVjmbFoNrwKY6XYis/1F8ELtvcaE12wA2BcmOcDTz9zPkDj2D5jUS+rJQs2FlfvvWo9jOhNTz9jAv2BF/aBO6iS0Kc0LP+oIJ2XQg+bUWEdLecy2VfL+0+owszIap7g= Received: from BY1P220CA0011.NAMP220.PROD.OUTLOOK.COM (2603:10b6:a03:59d::11) by PH8PR12MB6818.namprd12.prod.outlook.com (2603:10b6:510:1c9::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9073.27; Thu, 4 Sep 2025 04:16:09 +0000 Received: from SJ1PEPF00002322.namprd03.prod.outlook.com (2603:10b6:a03:59d:cafe::15) by BY1P220CA0011.outlook.office365.com (2603:10b6:a03:59d::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.17 via Frontend Transport; Thu, 4 Sep 2025 04:16:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C Received: from SATLEXMB03.amd.com (165.204.84.17) by SJ1PEPF00002322.mail.protection.outlook.com (10.167.242.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:16:09 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:16:08 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:16:04 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 02/19] sched/topology: Optimize sd->shared allocation and assignment Date: Thu, 4 Sep 2025 04:14:58 +0000 Message-ID: <20250904041516.3046-3-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ1PEPF00002322:EE_|PH8PR12MB6818:EE_ X-MS-Office365-Filtering-Correlation-Id: 35ca6ff4-8702-4b44-d31d-08ddeb69c6c7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|376014|36860700013|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?B/LZSVNZwfDsiUSzn07+TzFc3PAXM+80n5L7JawkW/Hash8c19BtNKdtbGEB?= =?us-ascii?Q?kwwteHyv5+LTqas1jIzQO/F5VwL4q9E8gQRWPMEb5wMcyro0Y496xp1SdcBx?= =?us-ascii?Q?m0+ANfAEUQ855il9CcyjBCI6FnKVhDuF2WOIm0JKpdm5i1AuCyH7+EWS7RKx?= =?us-ascii?Q?nyCw70DaOw4Bm6Z7iqznRZ84rQDPDAhD4lqxehdc7h1PrmrnKq5nRhOPp+zT?= =?us-ascii?Q?m27OYLvin07b/AFonUkObVlBySlnD+1UFQTbQzjlSrpRqW6PdHQjmg/9oTpS?= =?us-ascii?Q?RFZYLsFBXKGyHay3iqth8ohSGm2g3PNpTm/CGEY+f4pl9z74+Yq7PbXUmOZf?= =?us-ascii?Q?ydA4QhHq/0b2+fQwKNozeU80eN5HauMLflx1J6Xp/+txZn6/COoke3qyaQ4B?= =?us-ascii?Q?3oQ2l1Xxk4C6zPkFI3aWWXiRPlCUBfZH0PNjJJWP+V6R7Mav0gaKjWWB2IfO?= =?us-ascii?Q?ThbE6m0326L279/fdFzf9CoqxTMBnDZWsGmjbWgW632ujqCsXNuQUeh6wtQ3?= =?us-ascii?Q?90PSL/m4/+4rYk3VZv/VQbICYrB2BEb6FpI6YKkPVx5p+ia3zshTcnYqg8Jj?= =?us-ascii?Q?Zipmw5UFLFph6iDMCIpjLsSppldf0+3so8459CBB0DEqg/LTXwWVpaJcIgyw?= =?us-ascii?Q?CmQ24qcDs0mWdM1XajiDzARZKVwK5Vw/92z9H8mskYscEjaf9gzUuMzIPTls?= =?us-ascii?Q?7QsOqJih3NdHASFqplEneLvHn7pfd5sTvBuyz+ADo1qfOIQM1WVflKzUMYhx?= =?us-ascii?Q?UcVj/dB2RHrkO2tXyk8noMIRQPJT90bg7rwbDRnOM/qIqXrhGrai2ZhAZ0eA?= =?us-ascii?Q?t8chYiXwj0h2LT1Bf2m+PF+C5RR6+CHS8+wGCP6uf8Tfjs1kOe+wQ3JQ4xnC?= =?us-ascii?Q?loWJXyMgOnKyfHvmg8f53H1v14La27gD7CqR1VeY0jtgu1rD4JxGxqanRixI?= =?us-ascii?Q?IIrEiRzOrsHHVxupo71cxP/ZBchT69qHtWMDh+FDEoB7mG6m0iIBNC5AWI4k?= =?us-ascii?Q?STumVL+uucRcg1GtGscDGqfSMnCx81KmL3Yjk4k7e5pih3S8iehGd23/Yn3v?= =?us-ascii?Q?08zY5tPjVliM9np86Hz33tst3vzFDXk3pyfyoe07Fo3PGsjm2uq25Hhxgo8c?= =?us-ascii?Q?q6RWgyvLpzIO/BTGg/wKHX686Yyuky/TmN5/UTQWPxCWAcLQUPAvmGBKfGoi?= =?us-ascii?Q?+gdFo3FoxilNGslMsxmWQGhpWHUCVnTrAtQACoIO/9DDCmAgaOLLWjZRe0hd?= =?us-ascii?Q?bf72EuO9gORaatmGJENquJm1O89RrU1tuklYsS/ODoCxWffanzJUlFSuHpU7?= =?us-ascii?Q?sm4gPKM7BX5i7GTMrNj1UWPNMiTX4eNZDHWNi4L2Uin6cI7LiCh1MTr7q/w3?= =?us-ascii?Q?M/VNlyaBvrn7ZJOB9iHiuVu8c3wfUy5iFfh4aEV2uCHjqtN00AP47XXHEHZs?= =?us-ascii?Q?ATYBeEoH0GglgbYl0SSthSqH2UxdfYzdZY6F3SV7XcNgqhkG2WBQPMkVL7U0?= =?us-ascii?Q?H94aSKNy0WQNU3GZD06mNuj49mafrEajH673?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(376014)(36860700013)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:16:09.4212 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 35ca6ff4-8702-4b44-d31d-08ddeb69c6c7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SJ1PEPF00002322.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6818 Content-Type: text/plain; charset="utf-8" sd->shared is ever only used for sd_llc_shared and the sd->shared of lower domains are never referenced by the scheduler. Subsequent optimization will bloat the size of sd->shared and allocation of redundant "sd->shared" can be expensive. sd_llc is always a !SD_OVERLAP domain and it's children are always a subset of the domain, thus the degeneration of a SD_SHARE_LLC domain implies that it either contains a single CPU, or its span matches with that of a its child domain. Initialize a single level of per-CPU sched_domain_shared objects instead of doing it for each topology level. Assign sd->shared to the highest SD_SHARE_LLC domain and rely on the degeneration path to pass it to the correct topology level. Reviewed-by: Gautham R. Shenoy Signed-off-by: K Prateek Nayak --- Valentin, Peter, Was this ever attempted in the past? Is there some interesting topology out there that I'm missing where this wouldn't work? --- include/linux/sched/topology.h | 1 - kernel/sched/topology.c | 101 +++++++++++++++++++++++---------- 2 files changed, 71 insertions(+), 31 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index bbcfdf12aa6e..d816911de435 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -168,7 +168,6 @@ typedef int (*sched_domain_flags_f)(void); =20 struct sd_data { struct sched_domain *__percpu *sd; - struct sched_domain_shared *__percpu *sds; struct sched_group *__percpu *sg; struct sched_group_capacity *__percpu *sgc; }; diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 1104d931c015..f3ebb9e8fe00 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -679,6 +679,9 @@ static void update_top_cache_domain(int cpu) if (sd) { id =3D cpumask_first(sched_domain_span(sd)); size =3D cpumask_weight(sched_domain_span(sd)); + + /* If sd_llc exists, sd_llc_shared should exist too. */ + WARN_ON_ONCE(!sd->shared); sds =3D sd->shared; } =20 @@ -727,6 +730,13 @@ cpu_attach_domain(struct sched_domain *sd, struct root= _domain *rd, int cpu) if (sd_parent_degenerate(tmp, parent)) { tmp->parent =3D parent->parent; =20 + /* Pick reference to parent->shared. */ + if (parent->shared) { + WARN_ON_ONCE(tmp->shared); + tmp->shared =3D parent->shared; + parent->shared =3D NULL; + } + if (parent->parent) { parent->parent->child =3D tmp; parent->parent->groups->flags =3D tmp->flags; @@ -776,6 +786,7 @@ cpu_attach_domain(struct sched_domain *sd, struct root_= domain *rd, int cpu) } =20 struct s_data { + struct sched_domain_shared * __percpu *sds; struct sched_domain * __percpu *sd; struct root_domain *rd; }; @@ -783,6 +794,7 @@ struct s_data { enum s_alloc { sa_rootdomain, sa_sd, + sa_sd_shared, sa_sd_storage, sa_none, }; @@ -1529,6 +1541,9 @@ static void set_domain_attribute(struct sched_domain = *sd, static void __sdt_free(const struct cpumask *cpu_map); static int __sdt_alloc(const struct cpumask *cpu_map); =20 +static void __sds_free(struct s_data *d, const struct cpumask *cpu_map); +static int __sds_alloc(struct s_data *d, const struct cpumask *cpu_map); + static void __free_domain_allocs(struct s_data *d, enum s_alloc what, const struct cpumask *cpu_map) { @@ -1540,6 +1555,9 @@ static void __free_domain_allocs(struct s_data *d, en= um s_alloc what, case sa_sd: free_percpu(d->sd); fallthrough; + case sa_sd_shared: + __sds_free(d, cpu_map); + fallthrough; case sa_sd_storage: __sdt_free(cpu_map); fallthrough; @@ -1555,9 +1573,11 @@ __visit_domain_allocation_hell(struct s_data *d, con= st struct cpumask *cpu_map) =20 if (__sdt_alloc(cpu_map)) return sa_sd_storage; + if (__sds_alloc(d, cpu_map)) + return sa_sd_shared; d->sd =3D alloc_percpu(struct sched_domain *); if (!d->sd) - return sa_sd_storage; + return sa_sd_shared; d->rd =3D alloc_rootdomain(); if (!d->rd) return sa_sd; @@ -1577,9 +1597,6 @@ static void claim_allocations(int cpu, struct sched_d= omain *sd) WARN_ON_ONCE(*per_cpu_ptr(sdd->sd, cpu) !=3D sd); *per_cpu_ptr(sdd->sd, cpu) =3D NULL; =20 - if (atomic_read(&(*per_cpu_ptr(sdd->sds, cpu))->ref)) - *per_cpu_ptr(sdd->sds, cpu) =3D NULL; - if (atomic_read(&(*per_cpu_ptr(sdd->sg, cpu))->ref)) *per_cpu_ptr(sdd->sg, cpu) =3D NULL; =20 @@ -1709,16 +1726,6 @@ sd_init(struct sched_domain_topology_level *tl, sd->cache_nice_tries =3D 1; } =20 - /* - * For all levels sharing cache; connect a sched_domain_shared - * instance. - */ - if (sd->flags & SD_SHARE_LLC) { - sd->shared =3D *per_cpu_ptr(sdd->sds, sd_id); - atomic_inc(&sd->shared->ref); - atomic_set(&sd->shared->nr_busy_cpus, sd_weight); - } - sd->private =3D sdd; =20 return sd; @@ -2295,10 +2302,6 @@ static int __sdt_alloc(const struct cpumask *cpu_map) if (!sdd->sd) return -ENOMEM; =20 - sdd->sds =3D alloc_percpu(struct sched_domain_shared *); - if (!sdd->sds) - return -ENOMEM; - sdd->sg =3D alloc_percpu(struct sched_group *); if (!sdd->sg) return -ENOMEM; @@ -2309,7 +2312,6 @@ static int __sdt_alloc(const struct cpumask *cpu_map) =20 for_each_cpu(j, cpu_map) { struct sched_domain *sd; - struct sched_domain_shared *sds; struct sched_group *sg; struct sched_group_capacity *sgc; =20 @@ -2320,13 +2322,6 @@ static int __sdt_alloc(const struct cpumask *cpu_map) =20 *per_cpu_ptr(sdd->sd, j) =3D sd; =20 - sds =3D kzalloc_node(sizeof(struct sched_domain_shared), - GFP_KERNEL, cpu_to_node(j)); - if (!sds) - return -ENOMEM; - - *per_cpu_ptr(sdd->sds, j) =3D sds; - sg =3D kzalloc_node(sizeof(struct sched_group) + cpumask_size(), GFP_KERNEL, cpu_to_node(j)); if (!sg) @@ -2368,8 +2363,6 @@ static void __sdt_free(const struct cpumask *cpu_map) kfree(*per_cpu_ptr(sdd->sd, j)); } =20 - if (sdd->sds) - kfree(*per_cpu_ptr(sdd->sds, j)); if (sdd->sg) kfree(*per_cpu_ptr(sdd->sg, j)); if (sdd->sgc) @@ -2377,8 +2370,6 @@ static void __sdt_free(const struct cpumask *cpu_map) } free_percpu(sdd->sd); sdd->sd =3D NULL; - free_percpu(sdd->sds); - sdd->sds =3D NULL; free_percpu(sdd->sg); sdd->sg =3D NULL; free_percpu(sdd->sgc); @@ -2386,6 +2377,42 @@ static void __sdt_free(const struct cpumask *cpu_map) } } =20 +static int __sds_alloc(struct s_data *d, const struct cpumask *cpu_map) +{ + int j; + + d->sds =3D alloc_percpu(struct sched_domain_shared *); + if (!d->sds) + return -ENOMEM; + + for_each_cpu(j, cpu_map) { + struct sched_domain_shared *sds; + + sds =3D kzalloc_node(sizeof(struct sched_domain_shared), + GFP_KERNEL, cpu_to_node(j)); + if (!sds) + return -ENOMEM; + + *per_cpu_ptr(d->sds, j) =3D sds; + } + + return 0; +} + +static void __sds_free(struct s_data *d, const struct cpumask *cpu_map) +{ + int j; + + if (!d->sds) + return; + + for_each_cpu(j, cpu_map) + kfree(*per_cpu_ptr(d->sds, j)); + + free_percpu(d->sds); + d->sds =3D NULL; +} + static struct sched_domain *build_sched_domain(struct sched_domain_topolog= y_level *tl, const struct cpumask *cpu_map, struct sched_domain_attr *attr, struct sched_domain *child, int cpu) @@ -2537,8 +2564,19 @@ build_sched_domains(const struct cpumask *cpu_map, s= truct sched_domain_attr *att unsigned int imb_span =3D 1; =20 for (sd =3D *per_cpu_ptr(d.sd, i); sd; sd =3D sd->parent) { + struct sched_domain *parent =3D sd->parent; struct sched_domain *child =3D sd->child; =20 + /* Attach sd->shared to the topmost SD_SHARE_LLC domain. */ + if ((sd->flags & SD_SHARE_LLC) && + (!parent || !(parent->flags & SD_SHARE_LLC))) { + int llc_id =3D cpumask_first(sched_domain_span(sd)); + + sd->shared =3D *per_cpu_ptr(d.sds, llc_id); + atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); + atomic_inc(&sd->shared->ref); + } + if (!(sd->flags & SD_SHARE_LLC) && child && (child->flags & SD_SHARE_LLC)) { struct sched_domain __rcu *top_p; @@ -2591,6 +2629,9 @@ build_sched_domains(const struct cpumask *cpu_map, st= ruct sched_domain_attr *att if (!cpumask_test_cpu(i, cpu_map)) continue; =20 + if (atomic_read(&(*per_cpu_ptr(d.sds, i))->ref)) + *per_cpu_ptr(d.sds, i) =3D NULL; + for (sd =3D *per_cpu_ptr(d.sd, i); sd; sd =3D sd->parent) { claim_allocations(i, sd); init_sched_groups_capacity(i, sd); --=20 2.34.1 From nobody Fri Oct 3 06:35:26 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2059.outbound.protection.outlook.com [40.107.223.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 730031A9F88 for ; Thu, 4 Sep 2025 04:16:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.59 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959392; cv=fail; b=OXLBy3gq5UWPWLefF1Bt6vzLnUWOO/Hmbq+sJ047qZxJxW/e5w5eptvxkO34mejxOmsmoDP01dDjhiZjG/HoQFxPpaQc7N8enhTDueolAWwC8SsVebu/GFU2+cyVXnDX7X6ugjmb9pRJCH7fo48RcF5t9FMCmaanLqFcDSIueXI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959392; c=relaxed/simple; bh=Qr1WRu9cKpi/2lsCmhnKlIDO48HzMf0bYugwkA84dPQ=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ikG/MgeQHHFrMmRvah0X3r8B2M00LBBDNCEhFMPrPtidYIiv67/VA7yy2BSllZX643X839/XipPSZudPOqa4cXEZwRy6PA1Mz2mXadkUbEnX6eipdXs3+wryUs18Mp0VI+tBGDmEUK6EMlJVgoPlyrMjSnbCj3HFrgbjuveM6rk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=UDe/kQ4W; arc=fail smtp.client-ip=40.107.223.59 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="UDe/kQ4W" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=zHD/QzSXF0mWMxu08Frwc8vZQOtFrriow5dYANVcWyskJNIJ7txm8ohOGox5qJBCnFIa9AZrAOG0XubSZ92qd1KyI/lPerWJ0GW6geXuMgWVcLCrOYUFX9RpbjRZKRM+BT7KTib2yzWcNMNZO4BWknAbm78iL0GkjIYA8BDyhOPotcpomC7nlQgZI4/rOSrf0rd6hsv1FbJ+xFV4plxthOZ+BVMGz/gw0uI1pcLwWDuXxrfDD+/MbzYUeEq/YUaioYCGIBRoJhOUryNKTq/F29XjkjuPzHbVTGvrsYDUKSCBKs/r14ZU7G5QKoIpjYolBQfl8xVwqAuvJEKyFShRzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Li9NRlPeSt6+5NzF4+XuWwqei7SmM/TmdtydgazBA2k=; b=WTD2KZHqmqucrYNA1L0hW9lS7utKtF9u3akRxPA9afbaH0b7RjBu6l/93wAaiMeNpTd/Hs17PW47kCAhjIdQA6JOrnCh8foLpC/uCdlMOZOk2FtXEddCLCTkIbe3m1TSlfNVx9c2vZOzWAZYrbol2r2IZEXBpd82476UzDlMlwLT6+jrmgXaqW2g1FDVm+J5FfCkFXfrk8QZDoULlNa5imFQGYznxePBPkK9D47E2zbJI3qeVakFHWfRnVrYahpW679J/gO6Wo2qL7NdKVKSYGiLS7GcaisqUcwmM5Bun2o97qwyFuW7dAGmjDObNLdPZ58jpa9N/TfQrsulzNm9Xw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Li9NRlPeSt6+5NzF4+XuWwqei7SmM/TmdtydgazBA2k=; b=UDe/kQ4WznmNQu1G8ffJpOJLvHw1oQKDMDK7PclZ6fbiVUj8/R5OnkVss+PGEKI8ZqU25bP7JtofydO8aWkBnRZDdOIEIO5VE/eW6v0s3QcOCd0rsve5HHHm49gIreDtM8RMTtT3l8oBPWQSCiLY44VHgM3pA2tKKIDQ8dfENbw= Received: from DS7PR03CA0314.namprd03.prod.outlook.com (2603:10b6:8:2b::26) by DS5PPF8002542C7.namprd12.prod.outlook.com (2603:10b6:f:fc00::657) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9073.27; Thu, 4 Sep 2025 04:16:27 +0000 Received: from CY4PEPF0000E9D7.namprd05.prod.outlook.com (2603:10b6:8:2b:cafe::e1) by DS7PR03CA0314.outlook.office365.com (2603:10b6:8:2b::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.16 via Frontend Transport; Thu, 4 Sep 2025 04:16:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C Received: from SATLEXMB03.amd.com (165.204.84.17) by CY4PEPF0000E9D7.mail.protection.outlook.com (10.167.241.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:16:26 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:16:25 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:16:21 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 03/19] sched/fair: Use rq->nohz_tick_stopped in update_nohz_stats() Date: Thu, 4 Sep 2025 04:14:59 +0000 Message-ID: <20250904041516.3046-4-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D7:EE_|DS5PPF8002542C7:EE_ X-MS-Office365-Filtering-Correlation-Id: f2072109-96ff-4545-052e-08ddeb69d11a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?0NJ2Hi2jGJSoIKhqtB4/UMCKSjVwK/hAx4bhj1aLKpZSea0g26gBCp+rzEK/?= =?us-ascii?Q?cJLEO+/mRfYfXsBQFG4JCmwQz1QAK+9/64soshrOiJkU/kSodDt+aoeDGDg7?= =?us-ascii?Q?LLMcKVfiJ3fPRcNJuqxER5mCnQQEAaKFJOaG5MwpBO7Qe6z9cfzvFZxcQbgi?= =?us-ascii?Q?GTbexLRg1UcVMjsnPxue1EkGg+g/vcZGy+flVvk/9J//e0dGb6H5b7Ll43cB?= =?us-ascii?Q?1jHM2SaZEdJolJow5WMOb6nh6gI4UaIEpAeHtocIlNfc2Jto+EBtMRg/cUq4?= =?us-ascii?Q?gcmFc0vVPHJX5igBo3szAfUDlZWgHUP8jmBEP+xbZd1Kupn3M8JlnHJvdXlT?= =?us-ascii?Q?nlJqy0QEB8gGBizcuhBfblBHfY3Eo2VWu06XuGl0l7uJnzCZD9YIosy6JRhY?= =?us-ascii?Q?6g66eFlv/KT8lJxEMXRbzAkikxmF4AfPIuGGiPj7rUKvPBVIVHgtb5uP5uEZ?= =?us-ascii?Q?Cv+nBPjmuLMRoSTH7YSHq8iDGOIvqFP7JSNcJUL7QxhJb0bG5bqa5LYttdJB?= =?us-ascii?Q?W/5ozuAKEJsLcgeisNWOJiVA/gdUlDS06E1uM9PXxD+UhD0L3sWRxO+OA2sD?= =?us-ascii?Q?Br2e5gcAR6WjnG2x6XkXk7ZOSYETcddfrWclpxmLAYsBHL9vEHkhYHm0aBCh?= =?us-ascii?Q?zTuwx24bGz9DDWjLEAcbnEPI+/oCR5cOC5tcbuUaZcFb4uqLMZnPLEaT+qx8?= =?us-ascii?Q?Wh+QcpZe8IVhqO5P3356UCYmJ3RXjASPGAcLUvUC7F/lcYG3YlTX8Fcme3jr?= =?us-ascii?Q?NBnJO1oK7yT/nB+fqUJW20p7dBptIY95mT9dIekPsk2HZWsdyshP8GPBalWG?= =?us-ascii?Q?v/PoJixUb/lae+W2LeMHGWpthaweuVDveXKlC7EiJTdukJuecvUype1hfcCV?= =?us-ascii?Q?AT1R615ek7ZYLgBk44bdtLaSUc6QtLxlaTXKw0i+F8MhooFiU7nrxVTbgydw?= =?us-ascii?Q?XR/lW6xRXG1FV1YsF2RsTqwqrvgdxykAUlM/NfbpTjgi9MMFbVjgE9M7GPMn?= =?us-ascii?Q?laLmIYv00vCFb82k8mnqeHV+X1WXldDxEjmzcgk1pHJwKha6e43x+sP1/bX1?= =?us-ascii?Q?ZVlGAaHAAXvya8B2RRtRspZ+tZcaQw/B5I2RBHSI3XSPiUl9RWinysuymjEA?= =?us-ascii?Q?jLbVIaMmGovisGe+lLR1qldDE3HTyBLf6dyDOCFq9e1LsARafgyGahNuepRk?= =?us-ascii?Q?IA5UCLWn8Po5kEeoLnGx806N1NBciJCQ3lM7CWBrjWFGmammWFle76WhNwMc?= =?us-ascii?Q?fW068dUzg+Zdk5wrPwbbZLOXKbf6FMUeStnBQcvbRtynoVAXpA7WijrlASzd?= =?us-ascii?Q?KEVZ20J3EsjGULqlKsAHxFyDsyruRa19vCn19Tl1Nv1vTeNweoacSY5xeDC6?= =?us-ascii?Q?XsHXDaB+Gxy8Qtm7jOdTxcfH/xZ+tnblFnDDjARG/Z4oqKFJANfmuoDcapSP?= =?us-ascii?Q?YjQK/wJV9L0tVBlPnP05Zjla+XnQ/Qsmr0WvqzxG3kBp1WxWiB3RPjGwdFfJ?= =?us-ascii?Q?IusiljHE7+GDbmXKYa4Y7gUmg3RJfHJKsY6d?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:16:26.7826 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f2072109-96ff-4545-052e-08ddeb69d11a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D7.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS5PPF8002542C7 Content-Type: text/plain; charset="utf-8" "rq->nohz_tick_stopped" always follows the state of CPU on "nohz.idle_cpus_mask". Use the local rq indicator instead of checking for the CPU on the "idle_cpus_mask". Use READ_ONCE() and WRITE_ONCE() for "rq->nohz_tick_stopped" to ensure update_nohz_stats() always sees the latest value. This cleanup is necessary to avoid the number of references to the global "nohz.idle_cpus_mask" to ease the transition to a distributed nohz idle tracking strategy. No functional changes intended. Reviewed-by: Gautham R. Shenoy Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 61b59fd75ced..af9f2c1e93f8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12463,10 +12463,10 @@ void nohz_balance_exit_idle(struct rq *rq) { WARN_ON_ONCE(rq !=3D this_rq()); =20 - if (likely(!rq->nohz_tick_stopped)) + if (likely(!READ_ONCE(rq->nohz_tick_stopped))) return; =20 - rq->nohz_tick_stopped =3D 0; + WRITE_ONCE(rq->nohz_tick_stopped, 0); cpumask_clear_cpu(rq->cpu, nohz.idle_cpus_mask); atomic_dec(&nohz.nr_cpus); =20 @@ -12514,14 +12514,14 @@ void nohz_balance_enter_idle(int cpu) * *_avg. The CPU is already part of nohz.idle_cpus_mask so the clear * of nohz.has_blocked can only happen after checking the new load */ - if (rq->nohz_tick_stopped) + if (READ_ONCE(rq->nohz_tick_stopped)) goto out; =20 /* If we're a completely isolated CPU, we don't play: */ if (on_null_domain(rq)) return; =20 - rq->nohz_tick_stopped =3D 1; + WRITE_ONCE(rq->nohz_tick_stopped, 1); =20 cpumask_set_cpu(cpu, nohz.idle_cpus_mask); atomic_inc(&nohz.nr_cpus); @@ -12551,7 +12551,7 @@ static bool update_nohz_stats(struct rq *rq) if (!rq->has_blocked_load) return false; =20 - if (!cpumask_test_cpu(cpu, nohz.idle_cpus_mask)) + if (!READ_ONCE(rq->nohz_tick_stopped)) return false; =20 if (!time_after(jiffies, READ_ONCE(rq->last_blocked_load_update_tick))) --=20 2.34.1 From nobody Fri Oct 3 06:35:26 2025 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2054.outbound.protection.outlook.com [40.107.94.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1126F1BC58 for ; Thu, 4 Sep 2025 04:16:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.54 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959408; cv=fail; b=XPSCIKdwIDbnYdj9SZjmwjhFqFL51bZOvkxHiv/hNA21pkOSdxHd8LLwa0UHwILqYLFHt6IOu8UiE+au8EKLDFrsIxuD8/bbxGq0hYKfDJ9jLgr761v3afxhyAeNW4wpdkf7NDdHhfzXx5bTdeUnv6OsXer9yA037h9I4dS1Cro= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959408; c=relaxed/simple; bh=4X1hzMA9LoxxJLjHySPzE9yRiy/3COiZd5nNx6dHtu0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=d9QU0vipYxj6N9XEn7FlqLw9BDuACoxM0z+xlfZgpGIHnKl7+lP/BXoCbbZ06eC+lz8IwOdImO2vuz5eMVS5YJhi05IMlU3rBXxBWYI4OXbYvfo3mlVDA9EffwvR3unizb6ObDjo1c3Wy55bavfF7jArejm6+DlIwQFg5OGM0rs= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=V/38qBk7; arc=fail smtp.client-ip=40.107.94.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="V/38qBk7" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=zEOlVUwjlcBHw1hhzoVBSq3/gB8CfuoUCFqRA/i46TCN9QWAJ6w8PR2CDRGS2RVUKXztzFARLZI2d8C19HH7S3iU5z34GM3vRxB3oW/uDhrsGw5VnBUCPK9vXQSHMe+B5bK3uFjzwAYDnvUE7ZJlM9LYjxETfbXRXUNi6pJZXfgu+YvWl8+6M6LqQwZWEN8Cugr3voY8AOp3ugad/Z8TAIRLggd1X8ZbxycmS37wksaPmlZftrv7GYzBuD7cxdPIlwY1vZ0STs//kQhf9sExuKkGp8btBZn58Q+xQgHfR5LSZivxTfjlhxHR6+KrVav4Lk5gmHxm8VQw8+S0QMfKiw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=H0s3Ux2Bq68yPW2jHFW5iBJUmDlzPyQHlo4vWJiZn5U=; b=tMxnapLneCCfVBUCfmE8yiXANie2cyEM5DMTD1wfdqrmydeSwTVBlIVKSQ4hIJuM296Z99pJnbvGcpTXIQNhSHenwHh8lyNJrLIqKQNlsCs/rvxo85bM+dfu6XlICLSIc5OlwRmcmFL8IHkYsmYRufZpj/Bjf865ATLtBMitVdYeUpSi7aE5XdTiyladHjTY89xxCEPsU/zY6x5vBLipsmsoS8fKE634Nzpi3kGDJEjtCS+VfmDQe95wThcsNZwFiUWva6s58mW9i0PnPuatNyXNcpHE1YXrtxCi2aGr0UIegfPnUO5j8oUtJmzMWUBGqqsAHYtRUfVeRmsV0r5CHA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=H0s3Ux2Bq68yPW2jHFW5iBJUmDlzPyQHlo4vWJiZn5U=; b=V/38qBk7MvP963SkR5ga4HWrpJvYKAZg/cv9W4a2EibbQXDwiP4N5mXx1a4RGFfhv5UwtqIzsok0PYh3+2lfHZ4VGKCZunvQrib0/Zkykn0XVv2ijbEPG+qg/B7XL995AK+jxbJXou6nRVqYnClZC4/EjVfPj7gOz0bG5EVohtY= Received: from SN7PR04CA0186.namprd04.prod.outlook.com (2603:10b6:806:126::11) by BL1PR12MB5874.namprd12.prod.outlook.com (2603:10b6:208:396::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9073.27; Thu, 4 Sep 2025 04:16:43 +0000 Received: from SA2PEPF0000150B.namprd04.prod.outlook.com (2603:10b6:806:126:cafe::a5) by SN7PR04CA0186.outlook.office365.com (2603:10b6:806:126::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.17 via Frontend Transport; Thu, 4 Sep 2025 04:16:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SA2PEPF0000150B.mail.protection.outlook.com (10.167.242.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:16:43 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:16:42 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:16:38 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 04/19] sched/fair: Use xchg() to set sd->nohz_idle state Date: Thu, 4 Sep 2025 04:15:00 +0000 Message-ID: <20250904041516.3046-5-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF0000150B:EE_|BL1PR12MB5874:EE_ X-MS-Office365-Filtering-Correlation-Id: 68947a0e-bd88-44d9-0b09-08ddeb69dad4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|7416014|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?DoF+/udD/rmKRjyZOwiIkCQAOTpLUv3MPwIMP5xdpJPXNux1fwqyM9O5Bzgr?= =?us-ascii?Q?9FDHUsNE5dUe3jg207h/rTmz6zUbLd0CfYbZg5PMVjurkRtp3bVB45uZPXkH?= =?us-ascii?Q?4Dxk69wHJh3bDlSPeGiLGohu3iIFdyELJXAVbWHl6MNBaG4qwlfGWCDkk+qn?= =?us-ascii?Q?3N+BSdDcZ2M21/h0cs2p7tae9qiEli0e4zXsc6GsofeAEutREnUbEu8T4XCU?= =?us-ascii?Q?jE0ASzf7eRnjE2xZHhzbK8vo4qPNpyieeo9JnE776EtMt0JIdFBm0hQ8yxML?= =?us-ascii?Q?D1Y/TQszT1Sx61fPvgJB6vEn5goA04wjL+HPMGyLxKyNyeMI52NFWz9tLiwz?= =?us-ascii?Q?+ToFTs0SBv9GOBRIT1jGJdIfjQAD+UvQl2ppyhcT0LeHFHA7L30PaMiUYlnb?= =?us-ascii?Q?/mxYHOfQX+8k1Yq+s7pz/0qUiD+DJNDn70/PbDkz7OOm/JLZqZOrCXcRDq07?= =?us-ascii?Q?B4/VMWQfrfv8NUolqZKU1+d562f9gY5KuT1ou6ZoZ4E7ldXqiBqhyX82c4uP?= =?us-ascii?Q?Ae679IGzua34E6RtbgVObMLg9A91jOoHj48SdBDHuSAr9bHXVq8Z5M63jyLN?= =?us-ascii?Q?XejZFXXynilVg8AYMbjEXQaRoiI1ZRWs4YGDzOtBR7NXTsd3yaIiwtT/hJtr?= =?us-ascii?Q?/QQ1keDP2tX3VK+5qXThAvvUuShqW1y+bMICBy+TvmF81X22GBtf324Frilh?= =?us-ascii?Q?dvDJ5KiUWHybNl/0L/LEv4nu5sowa1A2/FsYDaH8pruZMm82rp56Y3lLPxTH?= =?us-ascii?Q?fYqg3ctzCzeee0EIQSXstnyawS3kz0U8vI5YLHb0CrdCetpn7VQFYq52KOYS?= =?us-ascii?Q?9bCQfJJRis5w7a0urIbGH7fr5hOPtNzNW6kdDkFj7FXu7RQKYdKkrhRScOM5?= =?us-ascii?Q?cH9omqlBgOpj7h/yQ4WLc+v8i/KzV1JH43mNhAlqU7efufmPmtijBXffA02M?= =?us-ascii?Q?nFNr0/Mq9NHN2/KWCxJK4eUqvPsrr98EMM9ehD+MWAeIuMXqtfx7UWuREJ4F?= =?us-ascii?Q?/Bze6I8y4um5SAdQihmoP+8pMbAg07IJCWeI5suxxyuiaYZ74nAz6OnEnBjn?= =?us-ascii?Q?Rjs1QWv9qeC/fAiUvMFnbiqd3YVwzvq5EnvBUjBt6kaWOULKGb7ZtGnvMinz?= =?us-ascii?Q?14iXDIV8Fz1Z2gIPscWCVZ6QYzXWdDyDyNLWOlgHzpBd70PfycbOzckVtdeX?= =?us-ascii?Q?z1GJumwlkClrvVFBERaDsXN0f+7423v+hLzWrhkjOo8l52tamlUy6P/A4yxW?= =?us-ascii?Q?eQ1sn76VY6xZG54grM/fVGodR0h9WGxGUEKjaXm+j7kV1iv8dc7WwyoE5uPt?= =?us-ascii?Q?OjMG3QihF5X1T81+DX4HfKlhKNIy8SoZWEAGV/cbfrvGBdhPQDZ1dxlG+AmU?= =?us-ascii?Q?NZAmKM8U4l+GTDUnqfp3knzZhcwOQZ7dNZII0I0druT0TPmgQOxrkPpqPEdj?= =?us-ascii?Q?vB+urgH0z6uOiZVvmy7iWOnqL1SXqSOtoA70Q5lNUOhLsVghHJfgnfbqHuq8?= =?us-ascii?Q?r35wa/scVQubKZf8bZMbXZEqsyzFHu+FyFPm?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(7416014)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:16:43.1269 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 68947a0e-bd88-44d9-0b09-08ddeb69dad4 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF0000150B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5874 Content-Type: text/plain; charset="utf-8" "sd->nohz_idle" currently relies on the fact that it is always modified from the local CPU to keep consistency however, the subsequent commits will use the rq->online() callback to correct the "sd->nohz_idle" status from a remote CPU during sched domain rebuild. Use xchg() instead to prevent double accounting of "nr_busy_cpus" if the status update on the local CPU races with a correction from a remote CPU. No functional changes intended. Reviewed-by: Gautham R. Shenoy Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index af9f2c1e93f8..0e68371365a5 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12455,7 +12455,9 @@ static void set_cpu_sd_state_busy(int cpu) if (!sd || !sd->nohz_idle) return; =20 - sd->nohz_idle =3D 0; + if (!xchg(&sd->nohz_idle, 0)) + return; + atomic_inc(&sd->shared->nr_busy_cpus); } =20 @@ -12483,7 +12485,9 @@ static void set_cpu_sd_state_idle(int cpu) if (!sd || sd->nohz_idle) return; =20 - sd->nohz_idle =3D 1; + if (xchg(&sd->nohz_idle, 1)) + return; + atomic_dec(&sd->shared->nr_busy_cpus); } =20 --=20 2.34.1 From nobody Fri Oct 3 06:35:26 2025 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2057.outbound.protection.outlook.com [40.107.96.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CFE1B1B5EB5 for ; Thu, 4 Sep 2025 04:17:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959427; cv=fail; b=lsQkRHdSeaOlCX/w00jjhSJj4pt3cETSSMaXZzVlPIhINAy/CSBp7kG3FHuYizjP7p6AHNM3eFltyqvL2rsFpPJInz/eMYRKNIRFhaxBGLS21rOFYoItTf15Gx2wnH8iRAE0PL/s/EdfB+4uJWWxaXN7rrP56Q59mMHtYfMqIVc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959427; c=relaxed/simple; bh=X34noNQN9mCC9jCbTMYj0D5mEZF04aqfSRYeFAIQJnY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=V1mvW3Sm4qItFeKzZjUJmt5TW7envEZB+8j/83Py0z40wne7lbFSFU0sD73lHPJ0EWXuShuuqc5wpyQ2v3RhGL0eslIfsMBeew8ovN0puCPII3bklwQr4jRX4pE64lay6EqOxxF2ziX5Z4plYx3hF9Ray0VkkUVfchViSzPP2YI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Zzucrq9e; arc=fail smtp.client-ip=40.107.96.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Zzucrq9e" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=RwArl+otluOUi0BUHXej3j2wwkeXvVec2zGMFK5ohs5m3doemEf5MCH30NnYIo9DtKQUrgrom5Uz2JfD1wKGu6Ao5foUDh8g6UlLiVx6pehTSUMU8wKUfSz6hA7M7g4RmHKku11KKxpeJA8BVIx1OU5fEjM7KgsFFwpY5zW//SWgYDjr/tHsmatPWQ6UpSBvYiMUu58oxxvy+UABW0uFA5DmRFtmT1MWHROYshhGAtIi7b6yTxjkBA9MXgBCQtn357e/0MRecZfKzE3HKli6H3ZmT1xOe3dKJoGm97ea7OXZbHXh9dK9ClUPFco3x5MJmfaaHdIxH1vWzf8OPYLDGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Wc0WjoBf3x2IP2fefgHf3CXq8vQRZN8N8Xx2UulOBtM=; b=YKpqEqU2TalNSRi+19RCSK1YFq+Py20MVlIjwXCXiQO/6a2IbSLJjmKIdWu0OFmsq9D7vHtf5iYoCn2gTKlKGewMCD1VcdFZrKDeODdsQZFYZ317HZO2+qwN/oP6GjqJ0wZ23WqJAzVaAE63fIE7yZnx3yhlqxdBUDZhoJBM1t0/PeX/hPU374thXW6P/M3FGDsZ1upNPuVd9yip4oxPudm2ET+W3FWnkVu+B3HxqNJUjQjjdVpAQos5qnMucL4Ht55SyUVPR9rue4iobwSS3xXAQGUJirl9h/IIRsR9pFBxs96uQKpL0KT6YOOv0rZvdlOGJgMFPi22spjCiZiJvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Wc0WjoBf3x2IP2fefgHf3CXq8vQRZN8N8Xx2UulOBtM=; b=Zzucrq9eV/9G2XApvzZ2jgUFvD1qGvjtMOWoMulRQmvCnQdNykTrNCdqQPnEqFrYrNc04x2SnZWhJXI8as8KKxfNI5NzSInJGP6zlGescn5EuOD/9hl0nxe5W/dibjmGkJWYWMcmcY4m/gf1xHB/ofd0SLuDtkCTfmHIUNgAn6k= Received: from DS7PR03CA0316.namprd03.prod.outlook.com (2603:10b6:8:2b::17) by SJ2PR12MB7918.namprd12.prod.outlook.com (2603:10b6:a03:4cc::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9073.27; Thu, 4 Sep 2025 04:17:00 +0000 Received: from CY4PEPF0000E9D7.namprd05.prod.outlook.com (2603:10b6:8:2b:cafe::c) by DS7PR03CA0316.outlook.office365.com (2603:10b6:8:2b::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.18 via Frontend Transport; Thu, 4 Sep 2025 04:17:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C Received: from SATLEXMB03.amd.com (165.204.84.17) by CY4PEPF0000E9D7.mail.protection.outlook.com (10.167.241.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:17:00 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:16:59 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:16:55 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 05/19] sched/topology: Attach new hierarchy in rq_attach_root() Date: Thu, 4 Sep 2025 04:15:01 +0000 Message-ID: <20250904041516.3046-6-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D7:EE_|SJ2PR12MB7918:EE_ X-MS-Office365-Filtering-Correlation-Id: 6596bd8b-bad1-4762-c779-08ddeb69e521 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?0a5XeIyW9pIgEHBVwth1GCUynV6kNjG1pOSAsHUmwcYz3YQn2nv38OmpUGeU?= =?us-ascii?Q?nqC8o6Ylr5+STRVRhUkbanfbgxmEm/zepBvtV8ZHq2sG2D24QSIdBbwyCTrZ?= =?us-ascii?Q?XgfZizCAWJgxb4QROUb93xUZOPa86H3aDMweLDLiGzuFQW/5q+00O9P7t/Tg?= =?us-ascii?Q?/HojKdLQvyWsM8pYghJkf0wjoCfBg1yFuifsVOnjlgJoCzxeVyiPU7U5pFwT?= =?us-ascii?Q?4ITJ4F6FsvGdAG9rrBh35bsKc3wpR7aDtxNJiFd7fDO1pLvCV63k1r7lYOCe?= =?us-ascii?Q?oRiT4p+WVnZiP1BzRV3fpmNqzx4I42O/sPeRjCYUzcRe5xDsXyYJheq8o9mr?= =?us-ascii?Q?zi1XbGqUfonmwAcj4tKF+2xgYa+SZ13osJbREKvvhRYdwASuk9jglsPK1zBF?= =?us-ascii?Q?EMdjUX8W1PYu8duuyo3OWOaEPf4AZeXCvn1kx4OoB668beMapJDcR2SbnG+u?= =?us-ascii?Q?6PFX26xEgHZL/Da2XImYusQsa6veACbLrVnKGZxYL+m8gvDuwp5SDHwlaCa6?= =?us-ascii?Q?JH2OyB1X8XK20KMz335OeP+9CXXdZ8eKAbSUj6MCwYhPP3grPzdFxhulTUrV?= =?us-ascii?Q?7MyHa1m28uUR52HW8FPX6AAxpNrauTK/dv8nb/bQsHg23qMKTINTBMT8xhUW?= =?us-ascii?Q?R8rGsRK/N58RH3qP/SS7a0VJaUfjz+OTeH40p/1fynjTlj+u7b3T8O5H9j77?= =?us-ascii?Q?QUxNU8ElqiCpXA0R8tLnKpFeW4N6GWC633lNVBOfNo9NJ7jVpMJO/8UWuedi?= =?us-ascii?Q?D/Xr1LIYheKz3AADk5LqWXVkM1NkfsDlmnlQAOaBhkIPyXMmsz77w/eLO2G/?= =?us-ascii?Q?DzpAKesaUX3hrAuP6btLasK6gpdF9rcd/1Ml26R7+7aYqgdrXGRwRpKrgCK+?= =?us-ascii?Q?I3xg4Fv969YZefEjkeNUStJS8tR1VoPw8VVdUnL+63fEuLi8GvjFPqiid8Cl?= =?us-ascii?Q?/sfK5DMhlMQZDACKR1Q3hdVHP/LsXmdl3WKkfQOcYwW7Fkm0mmXg6GjWa0Dp?= =?us-ascii?Q?NtjPTnRMgNyXXGdDI1XFiozCzIGneH2crbg6/iuQdY0aFlQVgsqC6RoMFKIT?= =?us-ascii?Q?7i0ARujCO8XhsvVbZUsMFkfaOcSuAIeQJIXhZLWD5T9egWGN8efW09h2dyAR?= =?us-ascii?Q?kpwZN5om/yVJaHcn/2JtOnEuuqxLp7TUPtQ0mPPxGGqLRNxtbFZg25wvU/DE?= =?us-ascii?Q?nXTIp7p8mS9pp0dpW8+eN1N+tFKytTjhVhLdox3EpmwNpY4xduAZyURxL+gp?= =?us-ascii?Q?VsWNP1B6+TVn+DSKqZXO9CRforrG+48AwccvDQbCnSHkN+T8ZiSjmMveBJDD?= =?us-ascii?Q?D/6mSe/jX4pYxe935o/cjFPoG4EW2p6tFo3XQOV0/G03MNNoIZHsRW2Ya1f/?= =?us-ascii?Q?fy7lix39KCrM107Z+PEQj2xXhVR5iADSRocgTZKhniVmPvf7fU6f/gU0XutY?= =?us-ascii?Q?aO2OOWYqksuVijJTLIXwZ/IOdxUWCb8Ar2e8fz4RQ43tHZtbDCgOxKTOCnaX?= =?us-ascii?Q?uvov3I/3AWhbbNUeFkfSLHnFV8Si2Zj0N3Fa?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:17:00.3882 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6596bd8b-bad1-4762-c779-08ddeb69e521 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D7.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7918 Content-Type: text/plain; charset="utf-8" Gregory Haskins introduced the concept of root domain in commit 57d885fea0da ("sched: add sched-domain roots") which put rq_attach_root() before rcu_assign_pointer(rq->sd, sd). Further development around dynamic sched domain allocation, and update_top_cache_domain() that came later relied on the new sched domain hierarchy and was naturaturally done after rq->sd was updated. Since subsequent changes around nohz idle CPU tracking creates a dependency for the fair_sched_class->offline() callback to use the old hierarchy and the fair_sched_class->online() callback to use the updated hierarchy, update rq->sd and the top cache domains within rq_attach_root() between the two callbacks. This modifies the flow as follows: old flow new flow =3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D= =3D=3D=3D=3D rq_attach_root() rq_attach_root() set_rq_offline(); set_rq_offline(); /* Operate on old = sd_llc */ set_rq_online(); rq->sd =3D sd; rq->sd =3D sd; update_top_cache_domain() update_top_cache_domain() set_rq_online(); /* Operate on new s= d_llc */ No functional changes intended. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 2 +- kernel/sched/sched.h | 2 +- kernel/sched/topology.c | 38 ++++++++++++++++++++++++-------------- 3 files changed, 26 insertions(+), 16 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index feb750aae71b..771fc6239386 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -8798,7 +8798,7 @@ void __init sched_init(void) =20 INIT_LIST_HEAD(&rq->cfs_tasks); =20 - rq_attach_root(rq, &def_root_domain); + rq_attach_root(rq, &def_root_domain, NULL); #ifdef CONFIG_NO_HZ_COMMON rq->last_blocked_load_update_tick =3D jiffies; atomic_set(&rq->nohz_flags, 0); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b5367c514c14..9ae3e2993641 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1020,7 +1020,7 @@ struct root_domain { =20 extern void init_defrootdomain(void); extern int sched_init_domains(const struct cpumask *cpu_map); -extern void rq_attach_root(struct rq *rq, struct root_domain *rd); +extern void rq_attach_root(struct rq *rq, struct root_domain *rd, struct s= ched_domain *sd); extern void sched_get_rd(struct root_domain *rd); extern void sched_put_rd(struct root_domain *rd); =20 diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index f3ebb9e8fe00..a059641e12e5 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -469,9 +469,12 @@ static void free_rootdomain(struct rcu_head *rcu) kfree(rd); } =20 -void rq_attach_root(struct rq *rq, struct root_domain *rd) +static void update_top_cache_domain(int cpu); + +void rq_attach_root(struct rq *rq, struct root_domain *rd, struct sched_do= main *sd) { struct root_domain *old_rd =3D NULL; + int cpu =3D cpu_of(rq); struct rq_flags rf; =20 rq_lock_irqsave(rq, &rf); @@ -494,6 +497,8 @@ void rq_attach_root(struct rq *rq, struct root_domain *= rd) } =20 atomic_inc(&rd->refcount); + rcu_assign_pointer(rq->sd, sd); + update_top_cache_domain(cpu); rq->rd =3D rd; =20 cpumask_set_cpu(rq->cpu, rd->span); @@ -711,16 +716,27 @@ static void update_top_cache_domain(int cpu) rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); } =20 +struct s_data { + struct sched_domain_shared * __percpu *sds; + struct sched_domain * __percpu *sd; + struct root_domain *rd; +}; + /* * Attach the domain 'sd' to 'cpu' as its base domain. Callers must * hold the hotplug lock. */ static void -cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu) +cpu_attach_domain(struct s_data *d, int cpu) { + struct root_domain *rd =3D d->rd; + struct sched_domain *sd =3D NULL; struct rq *rq =3D cpu_rq(cpu); struct sched_domain *tmp; =20 + if (d->sd) + sd =3D *per_cpu_ptr(d->sd, cpu); + /* Remove the sched domains which do not contribute to scheduling. */ for (tmp =3D sd; tmp; ) { struct sched_domain *parent =3D tmp->parent; @@ -776,21 +792,13 @@ cpu_attach_domain(struct sched_domain *sd, struct roo= t_domain *rd, int cpu) =20 sched_domain_debug(sd, cpu); =20 - rq_attach_root(rq, rd); tmp =3D rq->sd; - rcu_assign_pointer(rq->sd, sd); + rq_attach_root(rq, rd, sd); dirty_sched_domain_sysctl(cpu); destroy_sched_domains(tmp); =20 - update_top_cache_domain(cpu); } =20 -struct s_data { - struct sched_domain_shared * __percpu *sds; - struct sched_domain * __percpu *sd; - struct root_domain *rd; -}; - enum s_alloc { sa_rootdomain, sa_sd, @@ -2642,9 +2650,8 @@ build_sched_domains(const struct cpumask *cpu_map, st= ruct sched_domain_attr *att rcu_read_lock(); for_each_cpu(i, cpu_map) { rq =3D cpu_rq(i); - sd =3D *per_cpu_ptr(d.sd, i); =20 - cpu_attach_domain(sd, d.rd, i); + cpu_attach_domain(&d, i); =20 if (lowest_flag_domain(i, SD_CLUSTER)) has_cluster =3D true; @@ -2749,6 +2756,9 @@ int __init sched_init_domains(const struct cpumask *c= pu_map) static void detach_destroy_domains(const struct cpumask *cpu_map) { unsigned int cpu =3D cpumask_any(cpu_map); + struct s_data d =3D { + .rd =3D &def_root_domain, + }; int i; =20 if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu))) @@ -2759,7 +2769,7 @@ static void detach_destroy_domains(const struct cpuma= sk *cpu_map) =20 rcu_read_lock(); for_each_cpu(i, cpu_map) - cpu_attach_domain(NULL, &def_root_domain, i); + cpu_attach_domain(&d, i); rcu_read_unlock(); } =20 --=20 2.34.1 From nobody Fri Oct 3 06:35:26 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2085.outbound.protection.outlook.com [40.107.243.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0576F78F43 for ; Thu, 4 Sep 2025 04:17:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.85 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959451; cv=fail; b=YmRn+hrIgaWJjg7YL/wdwGl92oUWK9NLg075tRZ6jyhPtBNh5AcQ2qrxF6TXREoTgN19m1SOEk+87SfCxqh6BZ39yWmQue3FvapsF+F7ng7/fybkZ652EEKXKl7E1HR+Y5I7IOh79xsahjYDd31OrW2RANHuVRJZX89nIKIk/Vo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959451; c=relaxed/simple; bh=GHKWNttjmgO6HBz0z21b5sn4FzPQlp7mD1Hfv0XtmiM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=W3mTcftSLi3zF7DZz7s8Mez8qAk2gFJZZpv5KygKgh+aDXH03glxmGBcIQa7unN6Un4Wl2RWmeB5naUl+M6Gdbe20TDHFiz3gLtAnx6anmu9V/2rDqerVfHaBaFixVHCh8vHaEiASQE0sn8ShjHSHmvPy7k8es2ROPgqaI43Qa0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=B+mm/QTL; arc=fail smtp.client-ip=40.107.243.85 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="B+mm/QTL" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=mmJ3K85oSJN+onjPYdXGQ3e5OTBPF8VHGZJuhWfAt39kkHlKcyrWYWIbQVeHm13l91pvKUhYtgyai8IWcgOWL6CoWnQ1/LM68pbF/IiN5Fnot40wTweIvR0BWmFZy/nsk1TvHEKZXhm3zMyCH6UlFt3uLAtZFi4/bvJtKnbWIyEByiKq8wMKkJ++nWUVXRBtaE4IY52mmfERYSxTpgcgphlUgArH/mGVUd/goDTdN5jnUcUlBlJ+U/VQsCqRELUjDnJL/wLowtY80idwb1AQZFsHMjMlj5d4qAq2fW7jbxjiZ/gMHsqrqLu8j2zpEHA4glshpUXiNLPlDKiiqhnzUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=a/aJryXXOgpMbyU6L9uyW2Xw08Ihh1aMO5fgxttBzTc=; b=AfNGKnEOLO+GVd16bit8sRURdXcSE5p6+eNhZlkqNoUz8SOFf/PNrgcrw2w+DEZkQX8eMKBFAPgOQ4109HvJFwTSLnTbFSsFRRC3HR0sgtA1Gebr8NPTWWXMyepph80X3MBvmnimczS0yRN1Skd8LLO6nPacwLDoQcnbl1/kCS1CU/LyNsOik1FGd1y6vTe3UYXt6T5Yu/65zmWYEOH/Kuy/4lZMaRHqr6VEyq5e91ZRPj9mw299eVOPKvo7HD8gZBrf285485bJpgcgMYpKmUPKIosWHj6BAUGfjPBK8IrOl1xwUcjQJEIf+G+Cmb6vdZ+QClOU8aoV0meOooHsSQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a/aJryXXOgpMbyU6L9uyW2Xw08Ihh1aMO5fgxttBzTc=; b=B+mm/QTLEuNMBut/WDJsUNAj9oCwRnV2xB3TXQha/X+SPJ+7fXc3AR7yxBGIdoKGLCWaiPXr0avNIwVtncIbk+Y9g1K9zGiFqQVktSLQp4tEPaoOOngtT6LYh8Vo7TMRIOpdrsjN75CIwoIcJbGyIbYbMw4ghWQzo3p0AKlYAfc= Received: from SA0PR11CA0064.namprd11.prod.outlook.com (2603:10b6:806:d2::9) by DS5PPF2FA070BDF.namprd12.prod.outlook.com (2603:10b6:f:fc00::649) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9052.15; Thu, 4 Sep 2025 04:17:23 +0000 Received: from SA2PEPF00001509.namprd04.prod.outlook.com (2603:10b6:806:d2:cafe::3c) by SA0PR11CA0064.outlook.office365.com (2603:10b6:806:d2::9) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.16 via Frontend Transport; Thu, 4 Sep 2025 04:17:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SA2PEPF00001509.mail.protection.outlook.com (10.167.242.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:17:22 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:17:22 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:17:17 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 06/19] sched/fair: Fixup sd->nohz_idle state during hotplug / cpuset Date: Thu, 4 Sep 2025 04:15:02 +0000 Message-ID: <20250904041516.3046-7-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001509:EE_|DS5PPF2FA070BDF:EE_ X-MS-Office365-Filtering-Correlation-Id: 3257cafa-fdc0-4c92-6206-08ddeb69f25d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|7416014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?GVcyFjv4Y9bd8+V4mH6BpKhBLbkZQ/bbTIKt4iVYOy7g5Bw+aLZmNsG4AriB?= =?us-ascii?Q?mSNrb+cUWhu8TBxi5aQ0fClUuTkXlc1rcCBMe6iS/73cV3HbIyjVcdeTRyCn?= =?us-ascii?Q?9+yckekjFFUu3ASFQ/BBw7Tl0GwIy9mP8tlh3X3h5adaHVoDlof9dy8f3Psp?= =?us-ascii?Q?JP3HL9ChP/DnsCOzjxuJZ6g+mrfGRAcAgJMuRmLNXXvt7kcjYSgvJTx9jc5j?= =?us-ascii?Q?gsysIZHtu3RPSlAjpteFonEGSQB4MjNzOF6OpZu99jPB/MRGlxUaI1RiFUTL?= =?us-ascii?Q?0Z4kzpcE4gpVDpyW4qb/VKCyJ6YryhJeFz0Z1BMDwF7bFGRZqq/pXWo1QjOS?= =?us-ascii?Q?cxfFvLnpa0+YUdqzsZQ68GwLRyvavtgepSoG5cFa1HD/O0wo9EeLgMIDGt8A?= =?us-ascii?Q?bwY3epw6brtp0SF1VT13i5Mo6gD1gk/hK/BsgMQ/Q9GFxIhzU7iCSNWJt/O+?= =?us-ascii?Q?zFEAqpTCzr3Hsn6g9H3V0Fb9pWDllik+8F4C3HolkuluqRyMqt3yf++jZ7VZ?= =?us-ascii?Q?WrccM69+EsE+ujO7EQhnZYK1dR0LGjMVQFwisbNZG+g6wFt/Gh0XKKj17/Bw?= =?us-ascii?Q?FOQFQtjMPjfdf//WXflu8aXHzuQbpjF8ly3UyPjmdPFuATKAK8Dh+tZ4MseP?= =?us-ascii?Q?cTl4dUuwRzMZJT6lLuGw5EKA/Kgfi6qY+VdkdlETbBkXW9EEDZciY+TpKC9d?= =?us-ascii?Q?T9cfWbcAqKwQNhc5a3Xt09ju3D6JcF2raOEgFidRgip2KJ2HkSYedXumWqgz?= =?us-ascii?Q?ovSRJ9xrT+JbPAJseO2rjrDtEBLK4tztbHwZAi+cjo5vfx/OiaCP97G1nAWK?= =?us-ascii?Q?azSXLMaw9scWSD7Zxt9fz9ce5FtDR2ZrbGjpEz19kvs9wd1KrDvRw/Jl9z0n?= =?us-ascii?Q?297EcuSpqkJbOOXxD5aMobEHZYoYY11KD4oi7vDz+lIdKUvNKLtM/QWgXnY8?= =?us-ascii?Q?cajytL3y+ECqJGnhmIwJQswiyDYZnEgnXs4Io2dY8wg80HUhWqpT8Bj4x6Nd?= =?us-ascii?Q?YzZ7pJ3k7+wQVs+7TnLlBqjjWFzDzggaSzi2/ScC6HdOYv5uxuLHuEddoHyC?= =?us-ascii?Q?EU3Onagw/tM6UWTuRPCZw5D7+gQlwYKN0BSMC0eW1yWg0bd91kE5/ejqUm+B?= =?us-ascii?Q?Y+WStQZ0vLIIBuLyvHa80k4zZo4x1HSMOzbncnD3w5YMESSiPtv1bGBsV1e8?= =?us-ascii?Q?Vm/Ep8dxUctjHAp5dG3x2jIYAHelxVqSCL7QjJWZhlOD8v5i5nSQ1p3SSwm2?= =?us-ascii?Q?ncztc91Gcnd+/02EoueA0w/raughGwQWM3oPjWnlhmCRZLPw00IHGw6mi+pL?= =?us-ascii?Q?tc7g2KCAFlGnZ5VgGBKCguyrbsWP5ZShQ8bOxPHntsb+dfOd8xm9DjsASlHG?= =?us-ascii?Q?ooQ+l7ORIuUT52gmno/gXF+aGbKCFU5PHOAgQnp89VA+OVkB4nzKRIXTursK?= =?us-ascii?Q?0lBeEapJhSxNs9NdblMLNoXPLqMI056npcgQhobSm2my3rOmARYTzw2xSYq9?= =?us-ascii?Q?/JE0zZ7PzXAIdCQyPw6YbNAvkG2exFLYvIUw?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(376014)(7416014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:17:22.6077 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3257cafa-fdc0-4c92-6206-08ddeb69f25d X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001509.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS5PPF2FA070BDF Content-Type: text/plain; charset="utf-8" When CPUs are transitioned offline, set_cpu_sd_state_busy() is called to exclude the CPU from nohz idle balancing. Currently, it is done independently via sched_cpu_deactivate() but subsequent transition of the nohz idle tracking into the sched domain hierarchy would require each CPU's status to be updated in the new hierarchy generated during hotplug / cpuset. Call set_cpu_sd_state_{idle,busy}() from the fair_sched_class' offline() and online() callback respectively to ensure "sd->nohz_idle" of the updated hierarchy reflects the same state as "rq->nohz_tick_stopped". The changes from the prior commit to always update the rq->sd and the top cache domains from rq_attach_root() ensures that the offline() callback works on the older hierarchy and the online() callback works on the updated hierarchy. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 0e68371365a5..61e1b4deb3e8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12491,6 +12491,18 @@ static void set_cpu_sd_state_idle(int cpu) atomic_dec(&sd->shared->nr_busy_cpus); } =20 +static void cpu_sd_exit_nohz_balance(struct rq *rq) +{ + if (READ_ONCE(rq->nohz_tick_stopped)) + set_cpu_sd_state_busy(cpu_of(rq)); +} + +static void cpu_sd_reenter_nohz_balance(struct rq *rq) +{ + if (READ_ONCE(rq->nohz_tick_stopped)) + set_cpu_sd_state_idle(cpu_of(rq)); +} + /* * This routine will record that the CPU is going idle with tick stopped. * This info will be used in performing idle load balancing in the future. @@ -12741,6 +12753,8 @@ static void nohz_newidle_balance(struct rq *this_rq) } =20 #else /* !CONFIG_NO_HZ_COMMON: */ +static inline void cpu_sd_exit_nohz_balance(struct rq *rq) { } +static inline void cpu_sd_reenter_nohz_balance(struct rq *rq) { } static inline void nohz_balancer_kick(struct rq *rq) { } =20 static inline bool nohz_idle_balance(struct rq *this_rq, enum cpu_idle_typ= e idle) @@ -12940,6 +12954,9 @@ static void rq_online_fair(struct rq *rq) update_sysctl(); =20 update_runtime_enabled(rq); + + /* Fixup nr_busy_cpus and nohz stats. */ + cpu_sd_reenter_nohz_balance(rq); } =20 static void rq_offline_fair(struct rq *rq) @@ -12951,6 +12968,9 @@ static void rq_offline_fair(struct rq *rq) =20 /* Ensure that we remove rq contribution to group share: */ clear_tg_offline_cfs_rqs(rq); + + /* Fixup nr_busy_cpus and nohz stats. */ + cpu_sd_exit_nohz_balance(rq); } =20 #ifdef CONFIG_SCHED_CORE --=20 2.34.1 From nobody Fri Oct 3 06:35:26 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2059.outbound.protection.outlook.com [40.107.244.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8DBE1A9F88 for ; Thu, 4 Sep 2025 04:17:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.59 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959468; cv=fail; b=PGoEatedq/5M4rTZlcWE6UYFKKjuBPB8Pe6lHkgAUn23iyW5JmFyt1dJH0Ab7ULGC0mE9hy7yQH4/HcvVyYtv79bJf/1Fp559kwFmb8zXFnULSAdlhpklHRw3enLqPpuZLtH583nJateJZnc9rIfoHSz0yyfQd9eUeiazcLk9pU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959468; c=relaxed/simple; bh=KV17BgO5tesOUBSvGkSR6Vqjs7Zr7ODWAyCNzKz/XtE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ZJuiQHkOvrC7S3iW91IN6dbYgFaetdzh/TlqbyE3hzjVXRIFy9B4zICaMWuwTrhpatIDpnYiDuLB4w8ZmTZQWFSwckNg9yjvjNWENkb+KLZ8Eqcehmlej1D4NxGPU9eHn/NOzUGswBocan90hD3I5poeFF9Rla8iJVdYJ74TCOk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=gAlbyeqp; arc=fail smtp.client-ip=40.107.244.59 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="gAlbyeqp" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ZMnC/7OEqTZ6jJAXB3wL8dxWf+fuXGt2NrExqnzFNrIhVf8DIzZviojDFqXs+4W+OSpTEn1IHTOH0EMKAuJf8vPPB8QaTWrtLbtj7pGmTC+jjtTX2EzoiqjZAAD4liiMvsIJEuXlYxLp2ceXv+k9wIJo0yooqYzIJ2SDZ2LvhgPWQ0/BZEfk66wnQ9qoCpWxyxOAPuqOu3KDCMsxw/+sjZc4KjAWZhaRlvFTKLmMHDZmP75K1FTFRJN3gwPotOFyppQ5rHMI4woh1c/5MQwYmuKBLzvzeAcInHrw103G/oiyuGFEG6ncbd7IkqTxt20IVLQdQ41CqgFlXDZNFo7cNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dqCcg5TpGZ9gEDApA/kQbKXVkLe96lj0Fj0Wtg0Nq+w=; b=YCqviRZy7CeKtxgkBhov9IVvwmywjPbQGc9F0DamH7MNC5wRfu67N0Prh0C4XZAFwAqL6Vl9Lb78vJxw+nGs7z7+lAyeK35UCUgGGKu0mMk+gUlkjWxskw7NPkvqEDhn6b+jbWcGQUsv9uAqsL/bK2ROwa7n7q6m0naIG1F+QEI+Ok8S55fHvwswHmgr1nRZiCg4cdinhygpXY5TtZ7KPAhgHuesbzDHF6X/Z3lncoZScP/ZtnVoqaWDotZMpMajVRq7e33TbPR21vFiQX4rSCEDTTzkLe/vOwHO0MbsLxBoTpjqODDl5oqivrSwafIxa2uJHoyotkkKcVK9OlVcPw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dqCcg5TpGZ9gEDApA/kQbKXVkLe96lj0Fj0Wtg0Nq+w=; b=gAlbyeqpfSvckT6v5YEJAOEni+/YjLAtUDE8D3l2yfv5txPd2FhrVO1lynn4Ck+CgJw1/lxagUma6Vf4lY6O0JNA7GSb3C8IgcC6um5Q7Az23obIroIoFGUj7+gbAZbjzAJVb7NsTwLfODMAvnNRb0Nkby50selOki9wgJMr+XI= Received: from SJ0PR03CA0342.namprd03.prod.outlook.com (2603:10b6:a03:39c::17) by DS7PR12MB8232.namprd12.prod.outlook.com (2603:10b6:8:e3::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9094.17; Thu, 4 Sep 2025 04:17:40 +0000 Received: from CY4PEPF0000E9D6.namprd05.prod.outlook.com (2603:10b6:a03:39c:cafe::6e) by SJ0PR03CA0342.outlook.office365.com (2603:10b6:a03:39c::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.17 via Frontend Transport; Thu, 4 Sep 2025 04:17:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C Received: from SATLEXMB03.amd.com (165.204.84.17) by CY4PEPF0000E9D6.mail.protection.outlook.com (10.167.241.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:17:39 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:17:39 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:17:35 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 07/19] sched/fair: Account idle cpus instead of busy cpus in sd->shared Date: Thu, 4 Sep 2025 04:15:03 +0000 Message-ID: <20250904041516.3046-8-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D6:EE_|DS7PR12MB8232:EE_ X-MS-Office365-Filtering-Correlation-Id: 952afbab-01a2-4505-90e3-08ddeb69fcaf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?SwVUQ9PmA3zHH2ByurCpDGyd4J6+WV8zhBngjMRPTK5Uv6ixmCkOkx3FmktB?= =?us-ascii?Q?FTf5TrBHoOgW2glabQnezE/RQmF9YaqguzjrDTzkaIqy268I6GvxdCuhv5Lx?= =?us-ascii?Q?GtIjx2ZBZwWOVXdSX2T9/u6846VxW1H0pljGSru8HdUsR88LnJZcDPmfDmlK?= =?us-ascii?Q?84oCBUKzz4ekqdFa6FN9aEezywOdrgNwVyPsNT4wyHlNR73lHdT+UOR2PblZ?= =?us-ascii?Q?nd5O1y5dvIivPS5s5Ynt9rI+NibOCQ6t/CSMpp6IcSK3JU7YvljpF9VF5Ajn?= =?us-ascii?Q?8yA4W4C1Fs228OJA+ID+gIKm5UnNLoBZ5ZenjKPT61KdvcfrDWzUDut0JkwT?= =?us-ascii?Q?Sb5iIqlFdGmvc9tudfLB9KcccOE51+soJJDmu2wavxC/WlS6WYot1v53iXma?= =?us-ascii?Q?pnzXveVgfE7BE29drz2jx88UMcXmTu3nitUs/UnquQiimmR78HWnitS1squp?= =?us-ascii?Q?WK9KrZEEYFvelunDFXSYP6ZAVYo4aI/xIj6PYAkGBDXUj8rsGp3tmioIDii6?= =?us-ascii?Q?2Pyo5KYAajYLXKap/CTzoPfYtNGgjccxHftP5UzIr8loB7dMCp1h4ZrOoOB5?= =?us-ascii?Q?e3cdYyqP4GqsVFfvfg6pde3BqX24LS2Rhxme5a8WlTpxPJ5DFZAm/3NeDlOR?= =?us-ascii?Q?Vm2ppHutcWgmqowOKOkC6a2mgM4BtLx5/KHX7OZGPSjsrIOHthtjTq7zv17H?= =?us-ascii?Q?zDZoVOJ8nnb6aBvAMkq3/LJ9a4TA4jmYKQYIBWRS8Trrr0caSQP9/SKcEI78?= =?us-ascii?Q?/otEqjt+Gpn/HWr/vk8tPFgz4jpyMs2fXSnrXV6bqhXLO8ukQ6kKqHRCEseO?= =?us-ascii?Q?DW43mVHziS4hfaozHU3jsaVSzRZVKvCPEbnP1bM+fqtL5+rcONhP+cw2YGW8?= =?us-ascii?Q?tJ/jg8dli3/SUF9x7jmX5BkjxNZ6/OugInYcxafV9shS5TZK64sLIsNRzbOQ?= =?us-ascii?Q?hU4Mr2PtqGou8a0SbjsBiDJLqEj11WZEEk/uLB2i/NPS5pkewOTEofkhCYu3?= =?us-ascii?Q?LhCmY+dCh1sYiUJyKO/kocYkRBQ2nqsVfMhrcQZu1BKNBPV/yuPBvsNUvaBK?= =?us-ascii?Q?zdFD4Qs8rgU0+Y4bGfLEJ0TDvu59iw+HPkN1/IZUd1iPWZ6XD3DtlXxXVc6E?= =?us-ascii?Q?Ct9f/61k4wW3dKIoaPPQMPlFG0wxIIdr0QQWQAxPn7S4hl6TU4blj9G6AZDs?= =?us-ascii?Q?rFA4Z4YkiFn/ljw1bBU/LQozGaWXAv7V+W20Ye+wS2ZXwfOlrxFmvIkYdTlN?= =?us-ascii?Q?wRCgIY04rpX4EK2tmN677MFhi09fTQ+biOjqkISh4Kv8/HxspAjZaXGaXWCe?= =?us-ascii?Q?C8oClkHYARd5VbWxx0WdQ0KbVTfANgzmYnp2VKi5ZliFRhgiGjslRYUWEN4T?= =?us-ascii?Q?ysoDjDYiEQeOIwFAI6NDcwgQEc2EWTif+aM7Sk01N+duBqrI99w9iiImYPW3?= =?us-ascii?Q?wM10Tf/cw025JVK2aONXeMj2Hia+mGNNqNG0G2CEescT9MCtyn05lScqY2m2?= =?us-ascii?Q?+KODXxmdJOKWjesvu0i/q8hOzMxI1hm3wZG8?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:17:39.9059 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 952afbab-01a2-4505-90e3-08ddeb69fcaf X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D6.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB8232 Content-Type: text/plain; charset="utf-8" Switch to keeping track of "sd->shared->nr_idle_cpus" instead of "nr_busy_cpus". Since previous commit corrected the "sd->nohz_idle" state during sched domain rebuild, the nr_idle_cpus will reflect the correct number of idle CPUs. The idle CPUs accounting will be used for nohz idle balance in the subsequent commits. Races are possible during hotplug / cpuset where "nr_idle_cpus" might be incorrectly accounted if the CPU enters exits out of nohz idle state between the read of "rq->nohz_tick_stopped" and the subsequent update of "sd->nohz_idle" in the hotplug path but these inaccuracies are transient and will be corrected when the CPU enters idle or receives a tick. CPU0 (hotplug) CPU1 (exits nohz idle) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D online() if (rq->nohz_tick_stopped) /* True */ ... rq->nohz_tick_stopped = =3D 0 ... set_cpu_sd_state_busy() ... set_cpu_sd_state_idle() These situations are rare and should not have any long-term effect on the nohz idle balancing since there isn't a case where a nohz idle CPU is not set on the mask - either the hotplug thread sees that "rq->nohz_tick_stopped" is set or the CPU going idle sees the updated sched_domain hierarchy. After the conversion, all the bits that use "nr_idle_cpus" are already guarded behind CONFIG_NO_HZ_COMMON which makes it convenient to put the declaration behind CONFIG_NO_HZ_COMMON as well. Signed-off-by: K Prateek Nayak --- include/linux/sched/topology.h | 4 +++- kernel/sched/fair.c | 10 +++++----- kernel/sched/topology.c | 1 - 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index d816911de435..2f0d8ecea427 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -65,7 +65,9 @@ struct sched_group; =20 struct sched_domain_shared { atomic_t ref; - atomic_t nr_busy_cpus; +#ifdef CONFIG_NO_HZ_COMMON + atomic_t nr_idle_cpus; +#endif int has_idle_cores; int nr_idle_scan; }; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 61e1b4deb3e8..dee0ded7f40d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12429,7 +12429,7 @@ static void nohz_balancer_kick(struct rq *rq) * the others are - so just get a NOHZ balance going if it looks * like this LLC domain has tasks we could move. */ - nr_busy =3D atomic_read(&sds->nr_busy_cpus); + nr_busy =3D per_cpu(sd_llc_size, cpu) - atomic_read(&sds->nr_idle_cpus); if (nr_busy > 1) { flags =3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; goto unlock; @@ -12458,7 +12458,7 @@ static void set_cpu_sd_state_busy(int cpu) if (!xchg(&sd->nohz_idle, 0)) return; =20 - atomic_inc(&sd->shared->nr_busy_cpus); + atomic_dec(&sd->shared->nr_idle_cpus); } =20 void nohz_balance_exit_idle(struct rq *rq) @@ -12488,7 +12488,7 @@ static void set_cpu_sd_state_idle(int cpu) if (xchg(&sd->nohz_idle, 1)) return; =20 - atomic_dec(&sd->shared->nr_busy_cpus); + atomic_inc(&sd->shared->nr_idle_cpus); } =20 static void cpu_sd_exit_nohz_balance(struct rq *rq) @@ -12955,7 +12955,7 @@ static void rq_online_fair(struct rq *rq) =20 update_runtime_enabled(rq); =20 - /* Fixup nr_busy_cpus and nohz stats. */ + /* Fixup nr_idle_cpus and nohz stats. */ cpu_sd_reenter_nohz_balance(rq); } =20 @@ -12969,7 +12969,7 @@ static void rq_offline_fair(struct rq *rq) /* Ensure that we remove rq contribution to group share: */ clear_tg_offline_cfs_rqs(rq); =20 - /* Fixup nr_busy_cpus and nohz stats. */ + /* Fixup nr_idle_cpus and nohz stats. */ cpu_sd_exit_nohz_balance(rq); } =20 diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index a059641e12e5..0b0257937a97 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -2581,7 +2581,6 @@ build_sched_domains(const struct cpumask *cpu_map, st= ruct sched_domain_attr *att int llc_id =3D cpumask_first(sched_domain_span(sd)); =20 sd->shared =3D *per_cpu_ptr(d.sds, llc_id); - atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); atomic_inc(&sd->shared->ref); } =20 --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2049.outbound.protection.outlook.com [40.107.223.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23ECA1BC58 for ; Thu, 4 Sep 2025 04:18:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.49 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959485; cv=fail; b=W9JHtNy+JhuXis8c4CvgiTtURQ8fpjOYCMnFAsMT6QGwrQq4PTDLIDMVJWiWFRlZr8BagieAQb5hCZ2ijtm0reKlcqp9+6jYpQGzmNZ1+sDaYgDM3iM+gna4nblhEEiRb94eQH11IblunLWXOA5PLC+Iu+uGzkgogWNCypF5pvI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959485; c=relaxed/simple; bh=FMew96lDjvFWNRmBL380KbU2s36MMaSFDFrK5Iczs3M=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=UmjjQIBFcIntOvwpv+IxSLw7B0s+MSwGvE+pUehHJGmGUQRqlCFRUAEjWr4TGKHaJg9zHlDQUR4Bp1SfhG+Slwy7dU2yeXYomfKRjxUaf2nH41ZQ84tDBXb8qgiILkreylgElho4g12lYp8eFz+tWkxgm6QN3uOFV05CVNn+e9g= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Ii+YSO6h; arc=fail smtp.client-ip=40.107.223.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Ii+YSO6h" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vscVgU2glzJBHW8tXqAELLDoJ7k71p7kZKl/xbeANy6BjM2P7XweG1YSQX8SPO5ZGaKqQzbj22hfFpaveypSyw8QEaWDwDcP1dvY28wIQdJFpbaftQXpVg4+Tv1VPNCf2b7PGbEROIvXqQx3Vtc9x3X1Cl5RIOGOgcntE4U56V+XMkgcOVM7UNTKyC7fvt6NICbh0P7c6Wy7lF2X3pb31MqFt5Pif4YkkySAKGWThpF+kMEpjjNR5y1f7px1qD/bljBRzY2ESAIpKXsWg2CFi/So+J4V75tbZMWqjiyrFaEZF+HCdrvUpgFbWVOncwPKqTRVUYbdeiCNl3MSDzEPKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NxjoBt4iA7Xgr2dFS05ErOIFnuKyixnANaUEZp/wI4Y=; b=ZaOnNeieViXIGKu6vepBaP0TL509W2szheB2FH76GYqBcepZ8liJJfzE8cIRGqvIDZ66+NMGbzDHtnl229kVOHP8EOjfpUWor8se5Gv1nC6tiPcPU7MjCajTpLjin9xKHc6gUxKPVEpCrD2nJXw91sfPjOHO5GbrMAyupXOtLLjjSZxfNdLMk2pSK92T+qwMIPjZgevujCeLYvQi8W6MyeQDZkWQW7F8h3ZSHgJYn6YE2ueeUxRpzniroZGpQSxWRSxY0NYRs3sEFnSK6zq0qrEyIec6e5ZDw41pCf7KUEyxhuop2VsoMRNV6GSfjG1kmwk9So5gSaKRY7WWTwEE3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NxjoBt4iA7Xgr2dFS05ErOIFnuKyixnANaUEZp/wI4Y=; b=Ii+YSO6hikpn7XwF98Bz0LdHeC1Ei3KmIRJALLjKZxATkAN5i+JsOzxu/EoDsiXzmTb2QhBQwQzsuts5bri0fMj5q9sT8J1vD7PmBIMt2bXGATx23+oXlfJPoIqg/EpYkkSquRv52CfKJApdikOZ6mH03B57ttqDUmEBvENf42o= Received: from DM6PR07CA0098.namprd07.prod.outlook.com (2603:10b6:5:337::31) by BL1PR12MB5947.namprd12.prod.outlook.com (2603:10b6:208:39a::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9094.17; Thu, 4 Sep 2025 04:17:57 +0000 Received: from CY4PEPF0000E9DC.namprd05.prod.outlook.com (2603:10b6:5:337:cafe::ec) by DM6PR07CA0098.outlook.office365.com (2603:10b6:5:337::31) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.16 via Frontend Transport; Thu, 4 Sep 2025 04:17:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C Received: from SATLEXMB03.amd.com (165.204.84.17) by CY4PEPF0000E9DC.mail.protection.outlook.com (10.167.241.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:17:57 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:17:56 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:17:52 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 08/19] sched/topology: Introduce fallback sd->shared assignment Date: Thu, 4 Sep 2025 04:15:04 +0000 Message-ID: <20250904041516.3046-9-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9DC:EE_|BL1PR12MB5947:EE_ X-MS-Office365-Filtering-Correlation-Id: 4fc1ce7e-6d8e-4227-fb8d-08ddeb6a06e0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|1800799024|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?C6s1F37DBprAgDADoNr3cz4uFa/MMOkfmWtXqwQSw62m4PCVRz5EFVGE+qMe?= =?us-ascii?Q?gC5nlf+ozSZ3AWIJS3bYQHlkzBP34UwqC4gF72LqFuVIeYFT273SrnHQy6IA?= =?us-ascii?Q?VdEADTxo3ffM4PpYyg6XbYKzxuuWQDuZnM0Rh3QCtRIfCja6X90zUvm6tEYd?= =?us-ascii?Q?z1tgbhdahh0tVRi5UHuOwGg+pC6a7P7Bp5N1dXHF32Hh3/Zm1wbx4NbLz+Cj?= =?us-ascii?Q?MAPtE0lduS32xCz5d1FhlVVsvoG8o3yll0NPq3LSWsqZ1XTotGvuLBj9Y6sV?= =?us-ascii?Q?s8OMZ1xvmFestl5vLs9uVP4euckCQutO7B9LcEOWraB3Lx9nMIODXhlxx3gw?= =?us-ascii?Q?MlP5m29i8vDCI9iM4x+m4WZXIFp/Y0SIoyA4XgTIMrCA5Hm3casqC0rn0Ct6?= =?us-ascii?Q?g+5b2WdFjaUr9sAgcsbFRB3I8wmLSDXLQYdjLIhg25AtdB6DFH+r4GLAVgPG?= =?us-ascii?Q?mq0aNrC/KROtgWM3xRkwrwzVOepT8EENYstlCQi6080VjKeQ3pio6COMxQM+?= =?us-ascii?Q?SvNa9KR8kW5vvEwdf7zsjJofr5U+RkQfGLgkN4/IbF372NGOo9u/lGqbMZu1?= =?us-ascii?Q?c33xOlD8dOBRcPRFsUHJXW+rjuEG1GIP+gYuQCswkB9NOZB2/BdI82I8VKOk?= =?us-ascii?Q?o/apUxu3j5Wi47xzXhL6Fv2ym3GowUGUMc4LWKFDD/Rabxh5kgM/HNcsyKjo?= =?us-ascii?Q?+o5t1ay8HOyONUGmfTCM22i+wbAUZeD+Qdynvsdlp6FvQXgiHfvfuGk/SYe4?= =?us-ascii?Q?KvU8LgmLjimN4CJQ+eHgg2aiqou7Hl4n5he2S+TtP0HvkeuUu5PLzYI/191Z?= =?us-ascii?Q?CpmG7m5Wlxdy4DmT+/nQxJof+ujBaaq3khWeo2bJ56pzmHxqEwAKKxLcw/jj?= =?us-ascii?Q?8m6yMsKoBDTXUNG0wbINPTK+rd3r6a8DQ54NSBzdoI3TyhxMPf4tPlysU2cd?= =?us-ascii?Q?8R/c4tO11NvUC6Xh3tmMfAY6tl7eI+fSVZZhlt/A5cNJO6VgRrcOJhgaWytv?= =?us-ascii?Q?z5a5GvpU9LJfqlgNtL7UFEOPWW07WfH2BTtq4vrb/panb4lVG+PzxFFJ9nIC?= =?us-ascii?Q?u+ZyYJw8IKp6cqIZxJGU8+vwraEBQ5nznSlqVaY2JcMwLHGYvbyEK8RdjXA1?= =?us-ascii?Q?yA4UQo7DiL7WHOdqVvH5fl8hs3gHheBa6/hqRqklag5E8Qz8baF6roZHOCo6?= =?us-ascii?Q?aEKd7PteesprFqm5kxLEaEdtSR4GnEPJsQ6qcjgUErk7IBjzbzVRnjyp9jGD?= =?us-ascii?Q?CxaxH31LKtzdxG2WCqMSLCFMrdRLhhle658/4JVjwVcIAn/DEMW+Kul94X3w?= =?us-ascii?Q?p0+ucN73Mltrz917qB3ypP8R02uYicxz8k9/R6Avb0qS+dPB/DWmIxRDUXzV?= =?us-ascii?Q?FIYOvXRHLCIX30b22Tb4PP9R8xb4na513fB4mC6Bj86bXv0JAkpewxIzphkD?= =?us-ascii?Q?VXqYNu2dY7PEkYjUXdE7GqvfJB5nzg5iJG/Hx2MGp9y4fMGmNK0pj1WXn7Vk?= =?us-ascii?Q?yIJGewSqqGXdmCGFHoHZiHEv+UEa6ucsX4TF?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(1800799024)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:17:57.0060 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4fc1ce7e-6d8e-4227-fb8d-08ddeb6a06e0 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9DC.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5947 Content-Type: text/plain; charset="utf-8" Going forward, tying the nohz balancing to "sd->shared" will require each CPU's hierarchy to have at least one "sd->shared" object tracking its idle status. If the lowest domain of the hierarchy after degeneration does not have the SD_SHARE_LLC flag set, assign a per-Node fallback shared object to the lowest domain on CONFIG_NO_HZ_COMMON. !CONFIG_NO_HZ_COMMON kernels will always have tick enabled on idle CPUs and will not require nohz idle tracking. An example scenario where the fallback shared object is used is as follows - Consider a cpuset with 17CPUs where 16CPUs are from the same LLC and a singleton CPU from another LLC: CPU0: domain0: MC {0-15} groups: {0} {1} ... {15} domain1: PKG {0-16} groups: {0-15} {16} ... CPU15: domain0: MC {0-15} groups: {15} {0} {1} ... {14} domain1: PKG {0-16} groups: {0-15} {16} CPU16: # MC is degenerated since {16} is the only CPU is the domain domain0: PKG {0-15} groups: {16} {0-15} # Assign sd[PKG]->shared =3D fallback_dhared[cpu_to_node(16)] If the lowest domain is a SD_OVERLAP domain, "sd->shared" is only shared by the CPUs on the same node and not the entire domain. This is acceptable since the fallback shared object is only keeping track of the CPU's idle status unlike sd_llc_shared which also tracks "has_idle_cores" and "nr_idle_scan". Signed-off-by: K Prateek Nayak --- kernel/sched/topology.c | 123 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 117 insertions(+), 6 deletions(-) diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 0b0257937a97..d71c60d99313 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -455,6 +455,96 @@ static bool build_perf_domains(const struct cpumask *c= pu_map) static void free_pd(struct perf_domain *pd) { } #endif /* !(CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL) */ =20 +struct s_data { +#ifdef CONFIG_NO_HZ_COMMON + struct sched_domain_shared **fallback_nohz_sds; +#endif + struct sched_domain_shared * __percpu *sds; + struct sched_domain * __percpu *sd; + struct root_domain *rd; +}; + +#ifdef CONFIG_NO_HZ_COMMON + +static int __fallback_sds_alloc(struct s_data *d, unsigned long *visited_n= odes) +{ + int j; + + d->fallback_nohz_sds =3D kcalloc(nr_node_ids, + sizeof(*d->fallback_nohz_sds), GFP_KERNEL); + if (!d->fallback_nohz_sds) + return -ENOMEM; + + /* + * Allocate a fallback sd->shared object + * for each node covered by the cpu_map. + */ + for_each_set_bit(j, visited_nodes, nr_node_ids) { + struct sched_domain_shared *sds; + + sds =3D kzalloc_node(sizeof(struct sched_domain_shared), + GFP_KERNEL, j); + if (!sds) + return -ENOMEM; + + d->fallback_nohz_sds[j] =3D sds; + } + + return 0; +} + +static void __fallback_sds_free(struct s_data *d) +{ + int j; + + if (!d->fallback_nohz_sds) + return; + + for (j =3D 0; j < nr_node_ids; ++j) + kfree(d->fallback_nohz_sds[j]); + + kfree(d->fallback_nohz_sds); + d->fallback_nohz_sds =3D NULL; +} + +static void assign_fallback_sds(struct s_data *d, struct sched_domain *sd,= int cpu) +{ + struct sched_domain_shared *sds; + + sds =3D d->fallback_nohz_sds[cpu_to_node(cpu)]; + sd->shared =3D sds; + atomic_inc(&sd->shared->ref); +} + +static void claim_fallback_sds(struct s_data *d) +{ + int j; + + /* + * Claim allocations for the fallback shared objects + * if they were assigned during cpu_attach_domain(). + */ + for (j =3D 0; j < nr_node_ids; ++j) { + struct sched_domain_shared *sds =3D d->fallback_nohz_sds[j]; + + if (sds && atomic_read(&sds->ref)) + d->fallback_nohz_sds[j] =3D NULL; + } +} + +#else /* !CONFIG_NO_HZ_COMMON */ + +static inline int __fallback_sds_alloc(struct s_data *d, unsigned long *vi= sited_nodes) +{ + return 0; +} + +static inline void __fallback_sds_free(struct s_data *d) { } +static inline void assign_fallback_sds(struct s_data *d, struct sched_doma= in *sd, int cpu) { } +static inline void claim_fallback_sds(struct s_data *d) { } + +#endif /* CONFIG_NO_HZ_COMMON */ + static void free_rootdomain(struct rcu_head *rcu) { struct root_domain *rd =3D container_of(rcu, struct root_domain, rcu); @@ -716,12 +806,6 @@ static void update_top_cache_domain(int cpu) rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); } =20 -struct s_data { - struct sched_domain_shared * __percpu *sds; - struct sched_domain * __percpu *sd; - struct root_domain *rd; -}; - /* * Attach the domain 'sd' to 'cpu' as its base domain. Callers must * hold the hotplug lock. @@ -790,6 +874,14 @@ cpu_attach_domain(struct s_data *d, int cpu) } } =20 + /* + * Ensure there is at least one domain in the + * hierarchy with sd->shared attached to + * ensure participation in nohz balancing. + */ + if (sd && !(sd->flags & SD_SHARE_LLC)) + assign_fallback_sds(d, sd, cpu); + sched_domain_debug(sd, cpu); =20 tmp =3D rq->sd; @@ -2387,12 +2479,19 @@ static void __sdt_free(const struct cpumask *cpu_ma= p) =20 static int __sds_alloc(struct s_data *d, const struct cpumask *cpu_map) { + unsigned long *visited_nodes; int j; =20 + visited_nodes =3D bitmap_alloc(nr_node_ids, GFP_KERNEL); + if (!visited_nodes) + return -ENOMEM; + d->sds =3D alloc_percpu(struct sched_domain_shared *); if (!d->sds) return -ENOMEM; =20 + bitmap_zero(visited_nodes, nr_node_ids); + for_each_cpu(j, cpu_map) { struct sched_domain_shared *sds; =20 @@ -2401,9 +2500,13 @@ static int __sds_alloc(struct s_data *d, const struc= t cpumask *cpu_map) if (!sds) return -ENOMEM; =20 + bitmap_set(visited_nodes, cpu_to_node(j), 1); *per_cpu_ptr(d->sds, j) =3D sds; } =20 + if (__fallback_sds_alloc(d, visited_nodes)) + return -ENOMEM; + return 0; } =20 @@ -2417,6 +2520,8 @@ static void __sds_free(struct s_data *d, const struct= cpumask *cpu_map) for_each_cpu(j, cpu_map) kfree(*per_cpu_ptr(d->sds, j)); =20 + __fallback_sds_free(d); + free_percpu(d->sds); d->sds =3D NULL; } @@ -2655,6 +2760,12 @@ build_sched_domains(const struct cpumask *cpu_map, s= truct sched_domain_attr *att if (lowest_flag_domain(i, SD_CLUSTER)) has_cluster =3D true; } + + /* + * Claim allocations for the fallback shared objects + * if they were assigned during cpu_attach_domain(). + */ + claim_fallback_sds(&d); rcu_read_unlock(); =20 if (has_asym) --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2049.outbound.protection.outlook.com [40.107.243.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 706E91BC58 for ; Thu, 4 Sep 2025 04:18:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.49 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959501; cv=fail; b=BfimGFOTpXKiNc1U1rlIGOhJXDxDVYv4dhV7D7PD2rLk/a9TuHxikiprGu2hhbhkcfXC6ZKNqGS3acFjUJD7oaCB41v4KgNyCYc94E9hg0bPbrlwMZ7efJhwcQX13haVVfs8wfcOlQFGLpl5/5bWoPh4l4vDO/o0ufIFUhEPwWc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959501; c=relaxed/simple; bh=u2rNDdKkP0SQ0ttI1YOun7g1iJXrLG5/q0Gf95Y4H3c=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=FnPb0h7JgLGczMMC9JV5+me1l3zx8CUO2pQDQuuq2iUL+2mCxP92cnyCLlwgCujFN7hI+Qj5lOFBzozSsOL9adklSmMTEvBXX43nw7r+b3kMrEWVdgxtSeMBICheHTV01zTAVUNqmzLl4v79n2x9D9d8Z4oDfVKkLf0a8NmIIC8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=4s/JzYjr; arc=fail smtp.client-ip=40.107.243.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="4s/JzYjr" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uKzN6czCfgYnrh6YB/TZNDdlgjsEi8TpbtCJpyuONI6edDpp8B2e9FlDY1LKHt9eOZxukYGBLtQY1NyNDFA3G33Eis5XsFmr8+I+/Uj+CIfYOfU0WQx74g3rECt064D6nbfdNku9bu7i4LD+AfAFD4AX/ivts4GDP0toS93xWVQIkTV55b16DPVdsL10qqr3nqJi6hT4nIp3bviHbHHtsXtBccpdvBhJXn6VwYN53asmeDFQ7BRzQa6iU9/ZlRy4XEWNTymyjv5PpgkI/pR9K7hYBfNdoXPU7CtkjIHwv7+xMvjbDqQXFkz+YX+mYwVBszEytKG8XtRPwAK/5L2EEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xAfcEeA5Mf6rl7zoeQLTaVVhpW5455LMFPvJ1SEQB1g=; b=WM0abD+SmF04Bwz5dJPsSrt6j3OmLirtwQc/q+fGCS99GG/d8SRBZZ7TwgK9xx8RJsNjqug9RGG1zPLToVl8Uf3DNVD3OhQiYtB5PxzqL5S+CZAwJ8pEdO/NDBnPtZnUy7Xtn9xUxVG7t8jvw7XD74s8Jp/TR1S0nVGXs1YoOq9zXFiDP63MytFWMJwv3ZeAcasMOeSLqPR0ggkgmU5FazHmu+koc2qv4mPqSNdD8HvNg1rnS9t0k7OBN/MpEsulvxP7p8khZjSpZg/O40Nq5dCwU/9vQDOQAhDHRqHVoeVOKLZ2gvG2d4dHZ0x9sgc7afMiZ8iFf4E8i92moKZv3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xAfcEeA5Mf6rl7zoeQLTaVVhpW5455LMFPvJ1SEQB1g=; b=4s/JzYjrcEpfMuO1Z/FQif07Id3/WGP7uQj8U033xUL3pAtIPAPO2V7s2B/RfF2ohsi5YilRi0vhOPyLgYW491NhIZ7DngYY07ICNRfv8fDfwV/PrFhuPXqrlPLIXpwxdk0gbC7p0H5FNY/FJZAeMnnFrJt4sTnkkQvXOw21fLM= Received: from SN4PR0501CA0066.namprd05.prod.outlook.com (2603:10b6:803:41::43) by DS0PR12MB7927.namprd12.prod.outlook.com (2603:10b6:8:147::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9094.17; Thu, 4 Sep 2025 04:18:14 +0000 Received: from SA2PEPF00001506.namprd04.prod.outlook.com (2603:10b6:803:41:cafe::2) by SN4PR0501CA0066.outlook.office365.com (2603:10b6:803:41::43) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9115.6 via Frontend Transport; Thu, 4 Sep 2025 04:18:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SA2PEPF00001506.mail.protection.outlook.com (10.167.242.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:18:14 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:18:13 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:18:09 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 09/19] sched/topology: Introduce percpu sd_nohz for nohz state tracking Date: Thu, 4 Sep 2025 04:15:05 +0000 Message-ID: <20250904041516.3046-10-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001506:EE_|DS0PR12MB7927:EE_ X-MS-Office365-Filtering-Correlation-Id: 7686ed8d-2667-4cbf-6804-08ddeb6a1117 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?nyuhlNXy+NXArKsGa3HF07VAS9ajcp7aj7jgaw6udApppjZJT2J0rypBIhGQ?= =?us-ascii?Q?GjoWVAVZbTtMAygJPfvjKxXGs6/vn8SXUxPN64d9BMVT5u7WKS3gr3WJTbnl?= =?us-ascii?Q?gUGGmPp9r1KCHL+zegMMHAaXm6UZrGuPbn/Pmr5JHqYTIJn34ve+EkfSBA77?= =?us-ascii?Q?CDbYZEim7TssFpoftgloUpC9+wMWt4Ml5y3khU2U6dFg0U8YID3ctlkrMsr/?= =?us-ascii?Q?QX7xgwv94+Knyovx4Hy9RmJ6epnRGMmY8qIYSxapNPhhNT8WRHBI8wxOLFDF?= =?us-ascii?Q?D8SlaO5zXPIwOEuuNl8aEFXqvxELq2u8WJxeh4ohhQVwW/YMU3m+DLPsbYdI?= =?us-ascii?Q?TkwIg7rMOS9joaTANym4jnZjCuXltFi7F0rlPNMkj+BhxPnhyNufGfgl2O3Q?= =?us-ascii?Q?+jcPQ6HGGRKpeJQ1SjZua47xEvwOKG+iYF7qrhvu2NO7UMvssvCePo4Ykc1i?= =?us-ascii?Q?heoP8XIv1csgqFKVyYc5Ewuikn6VSE+ilkk40AJD6N4Y5Xn/FmnQQYxM+v9f?= =?us-ascii?Q?8pQ18vNOgv8Fte4pllQELXP+eEXARsJKeqNL8YF95oXqjxYSQ4aojVQSM/Ty?= =?us-ascii?Q?IF5hZzT0Rv6yFY1hBfa2QR6xUaw+EEViTGhSFOoAo2o3/AJsvolnFiXnuutg?= =?us-ascii?Q?tqTvRZqUUPRcFjuWczIAMSxOaCcSr07WZaF/D23GH1d3CcmurXcmCUmCSalx?= =?us-ascii?Q?PQ27bKt3Pwo6f1R/j9fw4cQrO0GD4ZQxtrz0lVHx4qFybNMG5By8EVCa3buP?= =?us-ascii?Q?8tXtaH5NWluy2fPzvgWWduU84okoYEi4Y1pZqUiLn1nzKOOz4wZjgDY7XYCu?= =?us-ascii?Q?ASo9pwRCZisa2kzEqLFq9sx0YkO8EZymiDLSFT9KfzDNc20IVR4Sk1hX8/lT?= =?us-ascii?Q?5def8wylbtVjeC1VZtI3IawUUGxPDp5mg+FdeAo9xF4auBtVkszgoleEW3PX?= =?us-ascii?Q?Bn5PFY5RT0/GjYXwtGXayMyul9dYmWcTeFokgbZJ2YpgtD3PFxB8fwjD+NZW?= =?us-ascii?Q?odWo6Q/Ha6p3zK95cI3BZpmeOpTyB8jkpVSratoO+RkYgHam1T0tn7eVvWq6?= =?us-ascii?Q?Q+4MXsLMWjxrq8gfdyZ7EMmaGM2k7E+GPVIMfZKrNi7TKaKYZgYFeT0FeFx2?= =?us-ascii?Q?xdEp4u3qcgVgT+h1356shB490GxhToLks9zpoWPxDw/HtwCVOEMyeeSylGvz?= =?us-ascii?Q?x1dK6tERF4Thqqyf/lZ/KY/olkEqEcCDCX+qhEFbvnXLyN9GbHmX5sHtM6lK?= =?us-ascii?Q?5SDaWVsuLuQe0s9CWVwf2ASgvZKFecfUu4/JAi46UevkzPQVDg2Me5i/hVU/?= =?us-ascii?Q?KhkXXIkxFJqYIakX4pOOIE2DbyrH7ZRPHKqqTSnZ7fSYtqlX1/+lgzfHEaQz?= =?us-ascii?Q?iupGAwsirjoNEh5m3F98tjduTRIuUOZU3RDZ7Wj9wHL3eFrHagQz7W/idw7g?= =?us-ascii?Q?ju3p1dEJ38TIxvNZyVclJYiik+YEWmTf7zLKUoUyhG+WgQR0XvYZno1+kFJQ?= =?us-ascii?Q?3xPTZRytDvc4q7O2ZqMZ82q7WCp2jblN+NFr?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:18:14.1547 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7686ed8d-2667-4cbf-6804-08ddeb6a1117 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001506.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7927 Content-Type: text/plain; charset="utf-8" Add a per-CPU "sd_nohz" variable to track domains in the local CPU's hierarchy that have "sd->shared" set for nohz idle tracking. Convert cpu_set_sd_state_*() to use "sd_nohz" instead of "sd_llc" increasing the scope of tracking to all CPUs with non-NULL sd hierarchy attached as opposed to just those CPUs with a "sd_llc" domain. "sd_nohz" will be used in a subsequent commit to transition nohz_balancer_kick() to use the distributed nohz idle tracking. Reviewed-by: Gautham R. Shenoy Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 4 ++-- kernel/sched/sched.h | 2 ++ kernel/sched/topology.c | 22 ++++++++++++++++++++++ 3 files changed, 26 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index dee0ded7f40d..624b3753c818 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12451,7 +12451,7 @@ static void set_cpu_sd_state_busy(int cpu) =20 guard(rcu)(); =20 - sd =3D rcu_dereference(per_cpu(sd_llc, cpu)); + sd =3D rcu_dereference(per_cpu(sd_nohz, cpu)); if (!sd || !sd->nohz_idle) return; =20 @@ -12481,7 +12481,7 @@ static void set_cpu_sd_state_idle(int cpu) =20 guard(rcu)(); =20 - sd =3D rcu_dereference(per_cpu(sd_llc, cpu)); + sd =3D rcu_dereference(per_cpu(sd_nohz, cpu)); if (!sd || sd->nohz_idle) return; =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9ae3e2993641..35ffb3926334 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3097,6 +3097,8 @@ extern void cfs_bandwidth_usage_dec(void); =20 #define nohz_flags(cpu) (&cpu_rq(cpu)->nohz_flags) =20 +DECLARE_PER_CPU(struct sched_domain __rcu *, sd_nohz); + extern void nohz_balance_exit_idle(struct rq *rq); #else /* !CONFIG_NO_HZ_COMMON: */ static inline void nohz_balance_exit_idle(struct rq *rq) { } diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index d71c60d99313..f230892528c7 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -466,6 +466,8 @@ struct s_data { =20 #ifdef CONFIG_NO_HZ_COMMON =20 +DEFINE_PER_CPU(struct sched_domain __rcu *, sd_nohz); + static int __fallback_sds_alloc(struct s_data *d, unsigned long *visited_n= odes) { int j; @@ -532,6 +534,23 @@ static void claim_fallback_sds(struct s_data *d) } } =20 +static void update_nohz_domain(int cpu) +{ + struct sched_domain *sd =3D highest_flag_domain(cpu, SD_SHARE_LLC); + + /* + * If sd_llc doesn't exist, use the lowest sd for nohz idle + * tracking. cpu_attach_domain() already ensures sd->shared is + * assigned for the lowest domain if sd_llc doesn't exist after + * degeneration of the hierarchy. + */ + if (!sd) + sd =3D rcu_dereference(cpu_rq(cpu)->sd); + + WARN_ON_ONCE(sd && !sd->shared); + rcu_assign_pointer(per_cpu(sd_nohz, cpu), sd); +} + #else /* !CONFIG_NO_HZ_COMMON */ =20 static inline int __fallback_sds_alloc(struct s_data *d, unsigned long *vi= sited_nodes) @@ -542,6 +561,7 @@ static inline int __fallback_sds_alloc(struct s_data *d= , unsigned long *visited_ static inline void __fallback_sds_free(struct s_data *d) { } static inline void assign_fallback_sds(struct s_data *d, struct sched_doma= in *sd, int cpu) { } static inline void claim_fallback_sds(struct s_data *d) { } +static inline void update_nohz_domain(int cpu) { } =20 #endif /* CONFIG_NO_HZ_COMMON */ =20 @@ -804,6 +824,8 @@ static void update_top_cache_domain(int cpu) =20 sd =3D lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); + + update_nohz_domain(cpu); } =20 /* --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2056.outbound.protection.outlook.com [40.107.223.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 878B978F43 for ; Thu, 4 Sep 2025 04:18:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.56 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959522; cv=fail; b=cEpo7ZyiumjJxj6uCjfZlEOFqjP91Oey81/XRkviWitmPm6RoWhs7M8hiecZfY5rLih1UzHsZFG1sNxulKvt1nHKg4Fd0SNlJYug3SsVKmyJYsv173zODpQEmMAIVUmgLyFc+W7PN9UrY8L/VeQ+w8SobRIZBAwUn+5cDzLgl60= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959522; c=relaxed/simple; bh=pJy7wI1F71X+agKOfh19SZeOvb2WnGFidEWwCxybO4U=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=dnFyL62xKd+i8NpZ4jdYkG6rqHAMrA/Ti/5hFad7ZDOLqpKLMEv5I34o6Khh+Z1lC+iJNa6eZWd64tG/Q4ngHM0MLhG6MLErWqnbN+jNOmAjx5CTwjVinLBmobBz50KD+O6iz6gZwSJH+/vpx9DiDQxAF6bIiL6gt4I4lOe9pus= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=W9K+24vI; arc=fail smtp.client-ip=40.107.223.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="W9K+24vI" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vocmWAJ+YBMuIiRbh84mwhsn4/iqRbAJRtV1iN56bGMWMUQqKpBv6m9pZy/cdtvOiQqFcr4IZeA995gPw3zCXWIGhhAEPm9A120ZY4SJI+nFNxNmBCI+T8loJSdQq8TGWobzwxdB+MWI4kXhZoatXH7MumAQb206rAEOCy2KE5lU73gmXJc6G8a3ts8uEqQMUd/Y9C+LCWesuqQViYnoAaTnvZmlZtg06XT94kjynYsb6eVriiRnrPaIu4KkYjl9iXM2Rp/3U6a3xFZvZZNuRxck3iU4RgsvwbELne9Jbw5JxXmnzacVywGHPrZzn6FOJFAF9aNcKCuth8307xMJEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RKTbjyymihfZs+a1XKLkRhsraQkuiKw2YH1W29dLrr4=; b=DB8Fv2X5sBA6DZBbYgBxaKxnw7cwj/PKXjiayvAGrZZ1qsmB7UhalHrzLNBk0B+I7+yeUOvdT5BP4VUae43YLiQvTb4IqKFYemGryQmBv6oq+nWsnnJiVdyu1SlWqpOLm6DDZO2nD4a+5nDTTJEGyz008iSSIR3eQkZfvSHMJZH3TfTVqg8H+bhCKGdc0uh+wotPCOXJ46xrvEBE+m1ZKSg7LCxt0dDQBd44DJegDxokRioC3crvYL4kJjWkoMImNZBMVxCK8+gbBuzSH+pnP91Cc82FNipiMrqgdCVti2s6BrkzSD+T9wBsRNc5Q09svy6wAF02YbTMLs1aAEPbKA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RKTbjyymihfZs+a1XKLkRhsraQkuiKw2YH1W29dLrr4=; b=W9K+24vILXz8PYK5+YC/u4V3LbU8fjWHgYZM6HQy7yogZh2arYRY5wLf2G0iG127ncQZDXwRBEU0Z/eZSP7XX/FCLRCcZ0BsbJ6EgfqP6KzywC/jW8BT3Voxt8shE6oRyI20TyGkNmvoksox4/AMu8vJy6rU5QOoLV+tdGP/oXs= Received: from DS7PR03CA0262.namprd03.prod.outlook.com (2603:10b6:5:3b3::27) by DS7PR12MB5837.namprd12.prod.outlook.com (2603:10b6:8:78::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9073.27; Thu, 4 Sep 2025 04:18:36 +0000 Received: from CY4PEPF0000E9D9.namprd05.prod.outlook.com (2603:10b6:5:3b3:cafe::92) by DS7PR03CA0262.outlook.office365.com (2603:10b6:5:3b3::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.17 via Frontend Transport; Thu, 4 Sep 2025 04:18:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C Received: from SATLEXMB03.amd.com (165.204.84.17) by CY4PEPF0000E9D9.mail.protection.outlook.com (10.167.241.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:18:36 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:18:35 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:18:31 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 10/19] sched/topology: Introduce "idle_cpus_mask" in sd->shared Date: Thu, 4 Sep 2025 04:15:06 +0000 Message-ID: <20250904041516.3046-11-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D9:EE_|DS7PR12MB5837:EE_ X-MS-Office365-Filtering-Correlation-Id: 6544d648-d94e-41a5-2cbb-08ddeb6a1e5a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|7416014|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?0+Y5DOmsopDjHBf3Zjrip0m08/4MDP/w/gwMz4A2A4I9norcW+hnAKPsdICq?= =?us-ascii?Q?sfyJrWmav5gD///QjSM9cHWafRtZe554CFUgvrLJ/HsNRUQyeEY7zLMEgUMn?= =?us-ascii?Q?M/evHvB2OXdzD/diIq0d1r+hHOF1pEZ2VPlW/c9xlLDzz2N/XnVA8AkQ8ybX?= =?us-ascii?Q?lZ02sle358LtRO6pEPSYzel9U/kX2H1jZ4aih9BrhfUjMV9pJnmQ+dKTR9dT?= =?us-ascii?Q?HOMWnlyzBKMiGofvzKy1rEY+LEjQz+JIi9skaYacrhYV1Tx/R5oqlpxT8nl0?= =?us-ascii?Q?lSz/7SLYe7rIFi63yEjc/WcqXm80aYcoqtT+n8PwX3Tyg7q1Q4EYTVMFvWtQ?= =?us-ascii?Q?tjuCOAMbeGxWsF37Mf9n/EAe1TU2gJBLGWmWB/Oo2pBuu711e4iQcTRIJiFw?= =?us-ascii?Q?oN4vIFg2+FvyvUZ6GrDO4XyNP2TQDRtXjEygqIs7rFexYloPHfS3F/bOeGqC?= =?us-ascii?Q?4bUzIWJXixw1zPR9UtbbwFJcr+QKBKwJw9M0njSn/kNeEmUU2ajJvPedIu8r?= =?us-ascii?Q?KQkZ8bGZ7EuhJcjmTzsOfMhaTdv3VBj86Y0xBwmw1C2DDPZYddmn6TgnE4fe?= =?us-ascii?Q?KmwINQPKlXVInWDk3vUT2xknYtkS8zLutdfiP4ckOTIWYKk/S+l1cFqQ/6jd?= =?us-ascii?Q?JWGA6NZ8TweaJa5tDB+hKSiX+Z3dqYN/NZc/orMZHYuTwnyE4QWILOMZJdS/?= =?us-ascii?Q?z3bGYwxUqbg7jbOf6whq7BAKC4fPSa83UiPl+LWbdaHCp2YCFAZkNfAE4tEp?= =?us-ascii?Q?CC1EG7YntuxsSxmelTX8saP9ohHc7dmafla7ksZuVgsh7WtU0BUp72OKPvsu?= =?us-ascii?Q?R5HNJymopftZqEarP0sMfbEbuZarFFrNoMkwSZ80QAS5fF7gr3R1X/a5Xv9n?= =?us-ascii?Q?6cBhRUhwaGjLq/NOPXGA4emWtDa+iPRzmcMVP7W7BaGgHJ/Di1dPdyHKavdd?= =?us-ascii?Q?V9UVLkwQpwzOM/J5AiAHlA6cz9tou0xtzpuqWwHTRNONaV1/9KTmldK6YJre?= =?us-ascii?Q?xVJagvxP/514v/+JC8hihMNj6PzDUQv79Hhdvzq+QzIwOrIQG2fRPk9r8UM+?= =?us-ascii?Q?FhqIM7sfefEzm7CnNUddIMlGRGyGPjbhF0NFbEcDRgjcKus0/8D5nVLi/v4C?= =?us-ascii?Q?Pd2Sq2DRL+JucPq+iiPLtqV4S3BBpti3dB3xhWtFlVxHCIcWbN+1085BtXrP?= =?us-ascii?Q?8NJwTyl/hafwyla6yWDdM1Wp7muVUEDd7N/8qU4a8vwTUAv/GcTJdnH2RAgA?= =?us-ascii?Q?aOVFYRfBOibPdeS2Zu3IWPt90tiG8n4PsIab8J1dRwXW0JrLbgFqghIBhBH8?= =?us-ascii?Q?0p/xg5cPHH0lvJxrE+rJn4DNzssV4nKjIlVxfSCWR+JDM15wmOHpcq/+Bkbh?= =?us-ascii?Q?Ofp1erFIWf2zrTtTjnGNxwcU4JT/T4nBUTZryuGtM6DcPcKe1quWSFViApcm?= =?us-ascii?Q?a3A3y9+2y3YVDwc9BQsHMpbJlykvOmtu+dMgs8Y8LrOjv9y4hdOymnDkmcUD?= =?us-ascii?Q?+ishk4DyCNx5Do6FG1aXsPRTxR7beyPHWb8d?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(7416014)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:18:36.3895 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6544d648-d94e-41a5-2cbb-08ddeb6a1e5a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D9.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5837 Content-Type: text/plain; charset="utf-8" Introduce "idle_cpus_mask" to track idle CPUs within the sd_nohz domains. This mask will eventually replace the need for the global "nohz.idle_cpus" mask for nohz idle balancing. Convert cpu_set_sd_state_*() to set / clear the CPU from the mask when transitioning to / out of nohz idle state. Reviewed-by: Gautham R. Shenoy Signed-off-by: K Prateek Nayak --- include/linux/sched/topology.h | 1 + kernel/sched/fair.c | 2 ++ kernel/sched/topology.c | 48 +++++++++++++++++++++++++++++++--- 3 files changed, 47 insertions(+), 4 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 2f0d8ecea427..6db3448e2f00 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -67,6 +67,7 @@ struct sched_domain_shared { atomic_t ref; #ifdef CONFIG_NO_HZ_COMMON atomic_t nr_idle_cpus; + struct cpumask *idle_cpus_mask; #endif int has_idle_cores; int nr_idle_scan; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 624b3753c818..c8226520758d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12458,6 +12458,7 @@ static void set_cpu_sd_state_busy(int cpu) if (!xchg(&sd->nohz_idle, 0)) return; =20 + cpumask_clear_cpu(cpu, sd->shared->idle_cpus_mask); atomic_dec(&sd->shared->nr_idle_cpus); } =20 @@ -12488,6 +12489,7 @@ static void set_cpu_sd_state_idle(int cpu) if (xchg(&sd->nohz_idle, 1)) return; =20 + cpumask_set_cpu(cpu, sd->shared->idle_cpus_mask); atomic_inc(&sd->shared->nr_idle_cpus); } =20 diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index f230892528c7..c2832445c578 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -468,6 +468,24 @@ struct s_data { =20 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_nohz); =20 +static int __sds_nohz_idle_alloc(struct sched_domain_shared *sds, int node) +{ + sds->idle_cpus_mask =3D kzalloc_node(cpumask_size(), GFP_KERNEL, node); + + if (!sds->idle_cpus_mask) + return -ENOMEM; + + return 0; +} + +static void __sds_nohz_idle_free(struct sched_domain_shared *sds) +{ + if (!sds) + return; + + kfree(sds->idle_cpus_mask); +} + static int __fallback_sds_alloc(struct s_data *d, unsigned long *visited_n= odes) { int j; @@ -490,6 +508,9 @@ static int __fallback_sds_alloc(struct s_data *d, unsig= ned long *visited_nodes) return -ENOMEM; =20 d->fallback_nohz_sds[j] =3D sds; + + if (__sds_nohz_idle_alloc(sds, j)) + return -ENOMEM; } =20 return 0; @@ -502,8 +523,10 @@ static void __fallback_sds_free(struct s_data *d) if (!d->fallback_nohz_sds) return; =20 - for (j =3D 0; j < nr_node_ids; ++j) + for (j =3D 0; j < nr_node_ids; ++j) { + __sds_nohz_idle_free(d->fallback_nohz_sds[j]); kfree(d->fallback_nohz_sds[j]); + } =20 kfree(d->fallback_nohz_sds); d->fallback_nohz_sds =3D NULL; @@ -553,6 +576,13 @@ static void update_nohz_domain(int cpu) =20 #else /* !CONFIG_NO_HZ_COMMON */ =20 +static int __sds_nohz_idle_alloc(struct sched_domain_shared *sds, int node) +{ + return 0; +} + +static void __sds_nohz_idle_free(struct sched_domain_shared *sds) { } + static inline int __fallback_sds_alloc(struct s_data *d, unsigned long *vi= sited_nodes) { return 0; @@ -740,8 +770,10 @@ static void destroy_sched_domain(struct sched_domain *= sd) */ free_sched_groups(sd->groups, 1); =20 - if (sd->shared && atomic_dec_and_test(&sd->shared->ref)) + if (sd->shared && atomic_dec_and_test(&sd->shared->ref)) { + __sds_nohz_idle_free(sd->shared); kfree(sd->shared); + } kfree(sd); } =20 @@ -2524,6 +2556,9 @@ static int __sds_alloc(struct s_data *d, const struct= cpumask *cpu_map) =20 bitmap_set(visited_nodes, cpu_to_node(j), 1); *per_cpu_ptr(d->sds, j) =3D sds; + + if (__sds_nohz_idle_alloc(sds, cpu_to_node(j))) + return -ENOMEM; } =20 if (__fallback_sds_alloc(d, visited_nodes)) @@ -2539,8 +2574,13 @@ static void __sds_free(struct s_data *d, const struc= t cpumask *cpu_map) if (!d->sds) return; =20 - for_each_cpu(j, cpu_map) - kfree(*per_cpu_ptr(d->sds, j)); + for_each_cpu(j, cpu_map) { + struct sched_domain_shared *sds; + + sds =3D *per_cpu_ptr(d->sds, j); + __sds_nohz_idle_free(sds); + kfree(sds); + } =20 __fallback_sds_free(d); =20 --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2045.outbound.protection.outlook.com [40.107.93.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0AA4A3AC1C for ; Thu, 4 Sep 2025 04:18:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.45 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959539; cv=fail; b=sRp78qSgwsZSUoe5jqggxYtD8bkNuNYNNxo3e0QWZyOiwtgnH6XedPs+B+YC7BymqC23KLvwl81+8xfc3Zaq39ZZaUI0rhzwMJ0ThbBc/E4LU1R8mbrR1PtHatFx8atbpn3IwzxNe7RzRVLZ/0FyiSMI6ln9IHi8wOQi9dkBWvo= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959539; c=relaxed/simple; bh=lAuwVVELkkpVc3SQWpma2f1S3qy8Nx6YBjKyWBEwovw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=AtJn5uecz7b3zCG+TRmoUe30esur1n2B9J6+ldE3DmplDKkuurz97t+yTvVUbQSe3f8Nb3F5TPWL8TnrWcSMA/my2vskAt8kGA1Cb2vQSGbo2xRMmncQfxj3pkbntsaOJnj0uRkhS/1XYdNwzjK4R46OwJUMqDBu5cAArImgxFU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=HJynRKVz; arc=fail smtp.client-ip=40.107.93.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="HJynRKVz" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=h/KElIIcj7P559fVRZpqJu3phnxQJfgACCaUWyMAq7FFnzubbdLLD4YgYSic59zleREm9Hp+1ojXdimcn+hmYj+Vudu0Hv7rUajFg8tVzs7kl01G9BHRvKQJS+FYfie8g1JqMzLXdE4F+3bfGftgrMUwYtCFjlDAQzBp/AdxEq1ML2E6KloQ8ys3OhBHApq4tHqq6UTFz7H2spr59PVMz+r9grgmORGHBDlEKS3zM7qcOxdK1kth8GxdeD5BZQAaR67nyv4Iq2IlAN+00LHcDggf5R/kJ7aCz1+ilygzRZziMyOKMNJfyhSqK8oFwnT0y70ZXRKV3XXGerzsQ5hTag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LZYVaoL89n3tfHmcj34wYeNmwbgqIJz/jop3Fy3ncV4=; b=wuAbqbibSLsbD8qJdvLo2a8VbPeE3irc54bTEbYf1N/KONu9Wvilm2clHf1RgkSCT21YqP6OC4uVcnoN1bo2AKxxoDzjtpkMWxNwB/MZBqiyoCkMYk1YvYlRH2smtMZ/W458qeYmKtpMFf+3S0Sye7YT072EPv5GS2O4UDsTJw2ddw5dNcfNwGetNQVWA3wKxxDxmmWT4ov99saMB73CzQYx+JZS8zCun3iuJUxK0CqRBEzXB59go4zwLobQNYuCTKwLhQ36KI80fO0toiQED59xxVfAUJ5T+6a1g14dPTdS4UGhXfCZSJAWh7oFTalJE3WTJnhV4kt9Qjrx//1sFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LZYVaoL89n3tfHmcj34wYeNmwbgqIJz/jop3Fy3ncV4=; b=HJynRKVzmBprrs0kSFCIOWEo13XqUAqe9NbSRJ3pnmqvSE1KzQyzMQs44yvZZQypU48yXYWmpirIxC6ap2j02RTGzCP/67jvxeBzsJi69/NpEFl8l5B7cKP/GRFEIPw3FIQ/4eIjBgdCJBngI4RVpNEY91SiAf4LMOhX/DoQydg= Received: from CY5PR13CA0026.namprd13.prod.outlook.com (2603:10b6:930::20) by CH3PR12MB8353.namprd12.prod.outlook.com (2603:10b6:610:12c::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9094.17; Thu, 4 Sep 2025 04:18:54 +0000 Received: from CY4PEPF0000E9DB.namprd05.prod.outlook.com (2603:10b6:930:0:cafe::85) by CY5PR13CA0026.outlook.office365.com (2603:10b6:930::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9115.6 via Frontend Transport; Thu, 4 Sep 2025 04:18:53 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C Received: from SATLEXMB03.amd.com (165.204.84.17) by CY4PEPF0000E9DB.mail.protection.outlook.com (10.167.241.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:18:53 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:18:53 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:18:48 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 11/19] sched/topology: Introduce "nohz_shared_list" to keep track of sd->shared Date: Thu, 4 Sep 2025 04:15:07 +0000 Message-ID: <20250904041516.3046-12-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9DB:EE_|CH3PR12MB8353:EE_ X-MS-Office365-Filtering-Correlation-Id: 68b7d49a-2074-4bca-679d-08ddeb6a28b4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ibvqfUQ89E6tm/cbsI1XSUPYVczYZRkRiyljg9/VAnE805uQyowAXyLiPynW?= =?us-ascii?Q?1Qkv2z6pH1OGId835rRirEoW6eldabePp0/xprfHbn66W4K5/rukiT6jsTzV?= =?us-ascii?Q?fpQOpaEJvy97W02FklyITannGguerbWn9QmSOx+gk9Jyr4DE7DjNCXb22i5C?= =?us-ascii?Q?9jckpD2MuMjo1o5xMhxwqQngcyE9mZdswCioJa0X9N1x7CitfpuqvmeT3TVC?= =?us-ascii?Q?jBfHCJYl7O+meaNsHES2tJr+2HGZAbNdZX9pZJfr0uwDtKJsOgXy+vh+PN6G?= =?us-ascii?Q?S/0XUed7JdL5MbOu8OEMP9Js0Q1qlPZwgPI4JN03zW8l3l4XlFCvr8gOt0HF?= =?us-ascii?Q?Vho+wMu64zhfSxKpkriFvZ1xreVpxQ3domYazdiKPJFqvZveQtWQNxkFwrlg?= =?us-ascii?Q?HtjBmLoCispJB6ZJj8ySLYIESMJOOWoiQXnYM1G7/lWFD3lh5UoiULGFriif?= =?us-ascii?Q?2EAh5+zfUXY8+aoze6/t8Uo2oIyruVaQ8A9JOGdv48pY+8pJGmtZppKYqiJf?= =?us-ascii?Q?ea6FHltbmaUzJoqR7J6kiQGvfNy2FBOqtqxuv4xsHLwmf21QSDriU6AWDMTY?= =?us-ascii?Q?M+RL9FYguAew3kkpBTPJEDk0nfzz42fSLy40nws4n1weD6qkDU9qQxonUGtm?= =?us-ascii?Q?U5DiltTMyHdj5F/6dXMVZzSWNP3XJlipF1kXGpjI+kuy+qVl3jhi+BiCU4Y9?= =?us-ascii?Q?sSXAifYsdugdbNFkdjfKe6YszXLm82h0SbgcacFeMG9kjGGdxcVfoDyI62qt?= =?us-ascii?Q?LeKggAfglJ28UKcerk9IVNYKhT1he7O44wUgzWQ5CkSS408YjJAM/8es2Oml?= =?us-ascii?Q?PRByrQTBN1DRn1O94q+7wj0iscqmYTvzyFFJNGzqD39obeC+u9q9q5SbemWn?= =?us-ascii?Q?JCXLYkUnTcslkAqoobmErK5ECLPekMRVN70AKdySCTPzNEsoMh712js9dbT7?= =?us-ascii?Q?Egj5H4Vj/nZJJV6R2EG6HdX8Ju0UjhW8WhfcyMLBdRUbCWWhowmB4wusPGv9?= =?us-ascii?Q?SsghFU98mhPRRM6Hcm13J4kNi78S5dkCnIzJdPyEKai910feapmdJirTAcpB?= =?us-ascii?Q?qvZhhtM7HzhSTRyrTlyiQQeZNIe77jRCjeUja1w1Xs5LsyYHM2SHYSf5jdYV?= =?us-ascii?Q?c4FqsLpJJbLzUI2UMV6kI5Ep+L7BsgvvTufrJnelDvzwvwaxfX+jatFYXVhh?= =?us-ascii?Q?6cnzcOzopycatKsz4cjSGNZRiTjuJUOQXsppo6xFw7SJZgxzY3pwPbp+DGlf?= =?us-ascii?Q?iFuHwF7lmxyt/3FHh6IErSbvcgJ1lbHIq0NSf/8/KpVFUzMKNCuRbPSdfNWN?= =?us-ascii?Q?eeA4xmh/aFZ+btw47fj1C3q+LWH92iCb5SSznZj7R535Msz8yYKOm1mIlA3U?= =?us-ascii?Q?D5e8Ej6/EB5EneFM0yUiKUeUxHiChtn0n0063C7GFVMnO8EnpAIcZd6kYnJR?= =?us-ascii?Q?35vtoTtXJS60c8wP8sq17APhs9qrHQsjEPGiTfeGWOWSUqjYW9eeaZWtmKi8?= =?us-ascii?Q?/2mMTQkAQxc9vy32YrLAMrj+TOtgLKYsXMcV?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:18:53.7572 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 68b7d49a-2074-4bca-679d-08ddeb6a28b4 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9DB.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8353 Content-Type: text/plain; charset="utf-8" Chain together all the "sd_nohz->shared" objects in a RCU protected "nohz_shared_list" to get the full view of system for nohz idle balancing. Since __sds_nohz_idle_alloc() is also responsible for initializing the list_head, rename it to __sds_nohz_idle_alloc_init() to fit the bill. List modifications are protected by the "nohz_shared_list_lock" and freeing of sd->shared on the list is delayed by one grace period to ensure all the untracked references to the shared object under RCU are dropped. Users of "nohz_shared_list" will be added in subsequent patches where all the current users of "nohz.idle_cpus" will be converted to use the distributed tracking. Reviewed-by: Gautham R. Shenoy Signed-off-by: K Prateek Nayak --- include/linux/sched/topology.h | 12 ++++--- kernel/sched/sched.h | 1 + kernel/sched/topology.c | 62 +++++++++++++++++++++++++++++++--- 3 files changed, 65 insertions(+), 10 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 6db3448e2f00..8400961c1c61 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -64,13 +64,15 @@ extern int sched_domain_level_max; struct sched_group; =20 struct sched_domain_shared { - atomic_t ref; + atomic_t ref; #ifdef CONFIG_NO_HZ_COMMON - atomic_t nr_idle_cpus; - struct cpumask *idle_cpus_mask; + atomic_t nr_idle_cpus; + struct cpumask *idle_cpus_mask; + struct list_head nohz_list_node; + struct rcu_head rcu; #endif - int has_idle_cores; - int nr_idle_scan; + int has_idle_cores; + int nr_idle_scan; }; =20 struct sched_domain { diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 35ffb3926334..9cffcfbef1ae 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3098,6 +3098,7 @@ extern void cfs_bandwidth_usage_dec(void); #define nohz_flags(cpu) (&cpu_rq(cpu)->nohz_flags) =20 DECLARE_PER_CPU(struct sched_domain __rcu *, sd_nohz); +extern struct list_head nohz_shared_list; =20 extern void nohz_balance_exit_idle(struct rq *rq); #else /* !CONFIG_NO_HZ_COMMON: */ diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index c2832445c578..86e33ed07254 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -468,8 +468,12 @@ struct s_data { =20 DEFINE_PER_CPU(struct sched_domain __rcu *, sd_nohz); =20 -static int __sds_nohz_idle_alloc(struct sched_domain_shared *sds, int node) +static DEFINE_RAW_SPINLOCK(nohz_shared_list_lock); +LIST_HEAD(nohz_shared_list); + +static int __sds_nohz_idle_alloc_init(struct sched_domain_shared *sds, int= node) { + sds->nohz_list_node =3D (struct list_head)LIST_HEAD_INIT(sds->nohz_list_n= ode); sds->idle_cpus_mask =3D kzalloc_node(cpumask_size(), GFP_KERNEL, node); =20 if (!sds->idle_cpus_mask) @@ -483,6 +487,7 @@ static void __sds_nohz_idle_free(struct sched_domain_sh= ared *sds) if (!sds) return; =20 + WARN_ON_ONCE(!list_empty(&sds->nohz_list_node)); kfree(sds->idle_cpus_mask); } =20 @@ -509,7 +514,7 @@ static int __fallback_sds_alloc(struct s_data *d, unsig= ned long *visited_nodes) =20 d->fallback_nohz_sds[j] =3D sds; =20 - if (__sds_nohz_idle_alloc(sds, j)) + if (__sds_nohz_idle_alloc_init(sds, j)) return -ENOMEM; } =20 @@ -560,6 +565,7 @@ static void claim_fallback_sds(struct s_data *d) static void update_nohz_domain(int cpu) { struct sched_domain *sd =3D highest_flag_domain(cpu, SD_SHARE_LLC); + struct sched_domain_shared *sds =3D NULL; =20 /* * If sd_llc doesn't exist, use the lowest sd for nohz idle @@ -570,13 +576,52 @@ static void update_nohz_domain(int cpu) if (!sd) sd =3D rcu_dereference(cpu_rq(cpu)->sd); =20 - WARN_ON_ONCE(sd && !sd->shared); + if (sd) + sds =3D sd->shared; + + if (sds && list_empty(&sds->nohz_list_node)) { + /* + * IRQs should be disabled by the caller since they + * hold the rq_lock. + */ + lockdep_assert_irqs_disabled(); + + guard(raw_spinlock)(&nohz_shared_list_lock); + list_add_rcu(&sds->nohz_list_node, &nohz_shared_list); + } + + WARN_ON_ONCE(sd && !sds); rcu_assign_pointer(per_cpu(sd_nohz, cpu), sd); } =20 +static void destroy_sched_domain_shared_rcu(struct rcu_head *rcu) +{ + struct sched_domain_shared *sds =3D container_of(rcu, struct sched_domain= _shared, rcu); + + kfree(sds->idle_cpus_mask); + kfree(sds); +} + +/* + * If sd->shared is on the rcu protected nohz_shared_list, + * remove it the list and wait once grace period before + * freeing. + */ +static int sds_delayed_free(struct sched_domain_shared *sds) +{ + if (list_empty(&sds->nohz_list_node)) + return 0; + + scoped_guard(raw_spinlock_irqsave, &nohz_shared_list_lock) + list_del_rcu(&sds->nohz_list_node); + + call_rcu(&sds->rcu, destroy_sched_domain_shared_rcu); + return 1; +} + #else /* !CONFIG_NO_HZ_COMMON */ =20 -static int __sds_nohz_idle_alloc(struct sched_domain_shared *sds, int node) +static int __sds_nohz_idle_alloc_init(struct sched_domain_shared *sds, int= node) { return 0; } @@ -592,6 +637,7 @@ static inline void __fallback_sds_free(struct s_data *d= ) { } static inline void assign_fallback_sds(struct s_data *d, struct sched_doma= in *sd, int cpu) { } static inline void claim_fallback_sds(struct s_data *d) { } static inline void update_nohz_domain(int cpu) { } +static inline int sds_delayed_free(struct sched_domain_shared *sds) { retu= rn 0; } =20 #endif /* CONFIG_NO_HZ_COMMON */ =20 @@ -771,9 +817,15 @@ static void destroy_sched_domain(struct sched_domain *= sd) free_sched_groups(sd->groups, 1); =20 if (sd->shared && atomic_dec_and_test(&sd->shared->ref)) { + if (sds_delayed_free(sd->shared)) { + sd->shared =3D NULL; + goto out; + } + __sds_nohz_idle_free(sd->shared); kfree(sd->shared); } +out: kfree(sd); } =20 @@ -2557,7 +2609,7 @@ static int __sds_alloc(struct s_data *d, const struct= cpumask *cpu_map) bitmap_set(visited_nodes, cpu_to_node(j), 1); *per_cpu_ptr(d->sds, j) =3D sds; =20 - if (__sds_nohz_idle_alloc(sds, cpu_to_node(j))) + if (__sds_nohz_idle_alloc_init(sds, cpu_to_node(j))) return -ENOMEM; } =20 --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2077.outbound.protection.outlook.com [40.107.212.77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3246084039 for ; Thu, 4 Sep 2025 04:19:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.212.77 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959561; cv=fail; b=Zjunqlex8opYshYCKBFAO+zxRdwkn1XqK5vz/6Bn8NRYl8emvAPVT/SYkPsOgCbSIJb9sNcvgKqwUpOORlD3ZF5ih3wR+NqOrZYY0DPGVQoUmFprLrDu7KuX0mDxLekiUxAUy8mh0h9DvbqrZd8b4XNaeBgsrFYNVqhPhRzu6go= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959561; c=relaxed/simple; bh=UbqNP7yzxxuzDE2T8HMFDkKIIvDtIBhspd5b+NsRs/I=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=uj90DQGYTqMaxCw/brcY0kjsQ1m+CvsOPufmCjy4DvDX3ILmAfxeoXHx4LeFyYxizXVgRqPLOO23JAh0SzsrFY18GCpL7nH3fF2tDse3X/tR3QkcxE1cpK3seJzOJJ1eKzNWOeUscXUb9zTWQEG/N90tB1VByik5Hgd/UEuxwzQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=U5mkfzaL; arc=fail smtp.client-ip=40.107.212.77 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="U5mkfzaL" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=WIjZN1lJcR273b5RnA+OpNiCC9gx29FbYZmUTK4Mew/8pVjVbjRaeO+o1IZ+i2zV1FoS0+q1MSCVqcChXk1+TXL7wIkJJbS9bl71BXN3D4gAzZP90ELoXrj1y49P5+rATey0h1i8PMee44uNvTZUWB6AtV6T1wYRYBbog71wyRTNI1OZmtdZSGAZrLxqb9hlijgIP1heTH/UlrADhRykzKZKE/WHj0+Ehjq6ANsXW5dqKt8O0hRABec2HK6LuJBkRq0YHo186aPqCouZZA+FqeIxl+4KjhW/q/Kw0rplhL+QiMsFc0MYJ9ChjRteaFQZkKQ2/kxcQN+TKpF+TRsfEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xzBm82KMAqIwzEzZLuqU9WZuysTXUdwisIWz+YG+CYg=; b=OnbKv9j1ra7Y63smaPZbCRtqhERL/DUXaIKZQGuxVvJMr0JDr6ZXFIhfMuVCrXAm7jPV3uPTVQhUzYTVgGW+A7zWiZcnjYrfs/w3F5vrPC2+oY9XqN0bqmOeWRXCEmTnrSeRKLK89kFi4xeQXRjOlGQVaMNlA6Bc7W5oFHlW+l0lfVcRMZe38SLskxJnLQW35v4dvxt99v3/EtjmN2gI9PTrqH5QEiKuUDOvGb70/JKwHaXjqQj2HBZZ3kfv/pq6nKtlgn8uLj64uGryHsJRKgiJzcTtcKWKdcQIsH8uEti7MO4gyWWv37dfFG0ZL7AN2Os9OJpaZN0IfecgP2qW2A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xzBm82KMAqIwzEzZLuqU9WZuysTXUdwisIWz+YG+CYg=; b=U5mkfzaLEZffYQEmvYKhkMQPxuTl1efsK6qaW8ZwPRpR4ysXgLXeDYFuXYcOfSdAEmnzy46s0dTFyYxDLQqL+G8gV0VWdlT958anqLcnDoFwM53P+UhWxro7UFOcmwTQWJ0BovDP+MjgwhX78HE0KFpK8AZ9y1F7HIhfAk9oHsE= Received: from DS7PR03CA0260.namprd03.prod.outlook.com (2603:10b6:5:3b3::25) by PH8PR12MB8431.namprd12.prod.outlook.com (2603:10b6:510:25a::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9052.20; Thu, 4 Sep 2025 04:19:13 +0000 Received: from CY4PEPF0000E9D9.namprd05.prod.outlook.com (2603:10b6:5:3b3:cafe::27) by DS7PR03CA0260.outlook.office365.com (2603:10b6:5:3b3::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.16 via Frontend Transport; Thu, 4 Sep 2025 04:19:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C Received: from SATLEXMB03.amd.com (165.204.84.17) by CY4PEPF0000E9D9.mail.protection.outlook.com (10.167.241.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:19:12 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:19:10 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:19:06 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 12/19] sched/fair: Reorder the barrier in nohz_balance_enter_idle() Date: Thu, 4 Sep 2025 04:15:08 +0000 Message-ID: <20250904041516.3046-13-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9D9:EE_|PH8PR12MB8431:EE_ X-MS-Office365-Filtering-Correlation-Id: e8fa31cd-6fa6-429a-c12a-08ddeb6a3403 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vsdjT5snjYEg+jL4k88Q19uzwFWiru0JDIA+l7XqVv13JOS731oaFRMHSuuA?= =?us-ascii?Q?UMhF6lmcua8eo2PFerMmgdt25M2ctg/OGnDYRdBoG09o1SrS6LTMOCu4GEqA?= =?us-ascii?Q?AUVxbugvXiQMDX8L60t33ixAh3fPs4QhQrs+jS539jYX31vEbmIA1iDhoWFq?= =?us-ascii?Q?hT6umqTTytpgZulJqUNSLB3eGa462y0rh3FZP3SaoY8edukGa99aC9scF/j0?= =?us-ascii?Q?s0xmT65EjfuX2AA2dgj3oIEtYoLSjU4k9fIKvDGbq6AEKj/Bad3Ntgg6nEa4?= =?us-ascii?Q?ngxMVSuCpBBMiGJdwQNQ7JlHEbZ9cgMhxV6+s+qHQ+B6ZU8mCtn0k6zgsqnt?= =?us-ascii?Q?IhP7xF5EyRQFa/KXsmAb0mYbd3Nq6PbbsBTRqKXOWqpIRnEuRSxvU5Fn4k3n?= =?us-ascii?Q?1eE011iNQAsDF5vq0y+9fCyO/JEdFF9ePD/UNRyXJdMlHSCHUj9Am2Ig7DHu?= =?us-ascii?Q?GBzoQPRn/R9CJPA/m2nnMz1QNxOpfbv+oMU8EWeRcK6Y+eOFNSHqw+/cwgbf?= =?us-ascii?Q?80L4siUDyyEsyfM8fJTQc/LmsPSDBJpddkNObzvDCrxwDkWQV6MMvpshSLMb?= =?us-ascii?Q?ZGiUQGxth34HyUQBi/eJvav1wLEt/O4Xo2l0IKAvewU+crHxhiIg7jjJmQ8r?= =?us-ascii?Q?HfOyTqsQLTMXIld/C5AfQTiBKHyInQZII51oz8ETY/BpUg4XxamwOoF3w5Xr?= =?us-ascii?Q?m5VV6At0iVFP2lCPh1549oJuZLiRCi/dh0lFWKmF0RVFcDRyaAxMgi5ryXUp?= =?us-ascii?Q?va2IH7D6SSdNfK4hv90szcdd36p4GfR1hyM/pZnjaEGQCjDwVNtwxMh61IJM?= =?us-ascii?Q?cxh41/dfdEr6OHIG1jlMUAgKiHmTAj8w69veKeXrRMsmWYdems1nPxxr96uK?= =?us-ascii?Q?niK7sT5J3/m1mwIj05StOUr7zXizpKYscp6wNHI1Bag3BBehxdUZmWbev/gp?= =?us-ascii?Q?Vco9Ouw8fX+AY1tG0kOcbv4cA23FHSdC5xsTqsOu5QoOekMPUdKxqfFqykmn?= =?us-ascii?Q?24ZJyNNcq99kfXnLJyTfz8RNMKMa1X9ltBfe79gIQmXaC/kH4TpJ2caaiJ0L?= =?us-ascii?Q?I7/bbyRBOzOnpXrUajpQV0zR5M78UpNUQeP9Xy//CIlvyFTf7rqhvnBPT709?= =?us-ascii?Q?yH7a8qhP1c04Qca+VYaG/shP+XQFqg8/SpXCxMEOr+Vq6wV6afvWhZmp0HUY?= =?us-ascii?Q?JEXMJXScTC4Pzm34BDvvC0zh/XV8P6Lx36WG+jPbHMBgKhUf0HeJDAX9HAmw?= =?us-ascii?Q?30PAlLKEKskHkV1LC05p1WSsdQ/214lBgCl3Z3NksuhrGWhWOvrLDd+T+fx4?= =?us-ascii?Q?nQp00aivX7c19jhri4GaZbVlnXxSyJeLlbRSD31HhhKzEl7F+G6Lxms3iBuT?= =?us-ascii?Q?rpctvQ32lrQL42RosBNzvaTOX+0k4pjiPrsIOfG9vQ75Rf0sJ2yEfItDH374?= =?us-ascii?Q?BF/PPJbQfYILosKuHBGPsjvN6f1D22S/Yodrl2MjLv6DIc/O10+kYdVfg4eV?= =?us-ascii?Q?h7YvzEyigdcW90l2zWcwnageaS2cdtXG4wKS?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:19:12.7324 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e8fa31cd-6fa6-429a-c12a-08ddeb6a3403 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9D9.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB8431 Content-Type: text/plain; charset="utf-8" The smp_mb__after_atomic() in nohz_balance_enter_idle() is used to order the setting of the nohz idle CPU on the "nohz.idle_cpus_mask" with the write to "nohz.needs_update" as per the comment. Since set_cpu_sd_state_idle() is now responsible for idle tracking using "sd_nohz->shared", reorder the call to set_cpu_sd_state_idle() to come before the barrier ensuring the CPU is tracked on the idle mask before the update to "nohz.needs_update" is observed. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index c8226520758d..d13de6feb467 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12544,6 +12544,8 @@ void nohz_balance_enter_idle(int cpu) cpumask_set_cpu(cpu, nohz.idle_cpus_mask); atomic_inc(&nohz.nr_cpus); =20 + set_cpu_sd_state_idle(cpu); + /* * Ensures that if nohz_idle_balance() fails to observe our * @idle_cpus_mask store, it must observe the @has_blocked @@ -12551,8 +12553,6 @@ void nohz_balance_enter_idle(int cpu) */ smp_mb__after_atomic(); =20 - set_cpu_sd_state_idle(cpu); - WRITE_ONCE(nohz.needs_update, 1); out: /* --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2056.outbound.protection.outlook.com [40.107.236.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E3D63AC1C for ; Thu, 4 Sep 2025 04:19:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.56 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959577; cv=fail; b=RkV6cNFH0WIcda+eIHrcb9kUVca/Uc/Yx+Yr1LX16zjFB9F/j2s8wLgitbKDvzl87aLaEFTFUJgJ2Mf+K91TWn6XrostNUy4z/6YxrCi5C+upyfcXo5esKbVcEBgZSsnw3yGMozwydrCSBf6Z2juHgH/qo2kYbKoPMb2iP3n27g= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959577; c=relaxed/simple; bh=+MBlvRSUrxHwtgwrrTINLPkcr6Me5Ls2RofZtN38uDE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QV7zND6PfU9m3aVBBS8DitXn3Gzfy7CAj3Qm/RTXETOqTxssl1hHAtGViLBLsAS1x+WEmplUZr3WZ64TbGr2UBduWK01Hy8KEDZqJILq08ofQjt0S1ijoy0zdW4CTrJBMKrh/SPFoV1qQ19XdHAfMwTQiyHZiiGtS8AzAnVmf1s= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=09QWxHS6; arc=fail smtp.client-ip=40.107.236.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="09QWxHS6" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GaICA3io8uoZIrnvtKtbMAq70sCDgS0A+FfFCrIdxPb6Z4O9FICVrZkDlT+fM+/8X0kRQjZGTpjm06/xZI9Vs4L0qW5XPTf6F/ufu9Em/3jktUAddvTcl8eUklNiB99K+KSwV7HkpiO8NBOdzoGh3sdSeu7aQcILLa4UsRdNU0Lk8GM4k8RU6MNyjQN9gT2inHD8FhInuf8OT+0/X7tmL0PnHd4mUSfJqjOkSEhSZL73HYAu6E3VfNEuI4cbMpn5H4z7ZifOYVlvjGKLJYlK1PNwzOkRvbga3ot6Y8ImXgnEZBabjepZe4jGN+mJlZTc9oXZ4Z0ECT9UQE4Sm1kBeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=y4kRASR9h/lsud7FqOIhspyz8BQBgl63gSC5rs21WTw=; b=nMocNtQcnqJ3s6014LR4fVbBHJUF1FQw+og97bs8FQdfYF9GKFeZGqMC6rfLLaDiwjRFtujVDLtDA6R9meLKpf0XxooJuZZMndDpsHeZAGv0QK67A+EdZQ1Air6ZYD9ZYlUT6M3Qf1Q6KTaXqtS4tE1Q5jD4b71I1iDt9V/H69QUci+eiU1P9ywFX/0Pgt780vA4s1YEvGpJWnKaqbgt5/6AcqqCehOyWdjNw6YQQcwIUFhor5YNI/Y9gbxGwlOxgbGT/odPsajD42a2CY6VfmRIT4ijQxcHSKqzdSn4sI4+uVRITvQoxtno7C5ek0nHzHu+pINCDDbH/DCxolxuQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y4kRASR9h/lsud7FqOIhspyz8BQBgl63gSC5rs21WTw=; b=09QWxHS6L1E1yNcpm1R6b/zI4wqEHGEFMCwH32ctU4mjivMh0AZ7ZC2tz5atBa97XikKDorwa8kjih1X6pAlsafNURzcAx6UrTm8Diqz36s15e9psmcvGySJz4ChiAeiWsMWK1LDyNOI+qVWq7v2j4MZ1SF/zkpaNuiQCxLPIMs= Received: from SN7PR04CA0186.namprd04.prod.outlook.com (2603:10b6:806:126::11) by DM6PR12MB4107.namprd12.prod.outlook.com (2603:10b6:5:218::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9073.27; Thu, 4 Sep 2025 04:19:28 +0000 Received: from SA2PEPF0000150B.namprd04.prod.outlook.com (2603:10b6:806:126:cafe::e3) by SN7PR04CA0186.outlook.office365.com (2603:10b6:806:126::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.17 via Frontend Transport; Thu, 4 Sep 2025 04:19:28 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SA2PEPF0000150B.mail.protection.outlook.com (10.167.242.43) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:19:28 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:19:27 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:19:23 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 13/19] sched/fair: Extract the main _nohz_idle_balance() loop into a helper Date: Thu, 4 Sep 2025 04:15:09 +0000 Message-ID: <20250904041516.3046-14-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF0000150B:EE_|DM6PR12MB4107:EE_ X-MS-Office365-Filtering-Correlation-Id: 9a4c066c-e5dc-4338-6a12-08ddeb6a3d35 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|30052699003|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?OiFO0qQcHvmw3qabiYE7FLazWdvi9Us1JDphIbDHOy3oMKaIfCI5zqihcsQC?= =?us-ascii?Q?poWYmh6VZdwNJgdbeHsd4C56duEmTCzi7aH629jZmjaTtL8j9KoNkJGTQ6oV?= =?us-ascii?Q?lH0rHyUK/y5Bolaol9WL+/tf0QOKzejT9ZNFuoWyhAKD8c5t6SUxJDmpvuX5?= =?us-ascii?Q?c4Hw8IzH9Lj6+T6TGYy2/icyEfQrNzwKcDv/YFQRmkrIVFLmAyWcixnczLSw?= =?us-ascii?Q?56a8Q44gFPg2kqV6OA//hQKk1rurCM8LrGw8ZHSU78Ihvj1xWPLJBBrLTkCJ?= =?us-ascii?Q?fMEwe599/HOk7KozY0k3tCSxlYKOZ1shNwFNXfZl53Qde9uy2iNT8rF20BTS?= =?us-ascii?Q?DMN6BeF0TZBnkDvQ891YVXzrj9E9oZNYwzWv+s2aBI/S+yVD2jQYv66hNWTg?= =?us-ascii?Q?HlCNmUxWb+HaD+xi6PkVK8UOZVOvcj2Fc5z6VHDcf/PL+5jBjlInm+th9NHl?= =?us-ascii?Q?NyFhCz1tsmrIiVfSCma8gh5WwOJ+jPo22ae3tVAWGCK/P0bF3Wlwv79nlxpI?= =?us-ascii?Q?jEDU78LpIpNkxcDt+ZhGAeCPt8x+XodT2+64JFq3pJGqlyzAxXgiknAnUioT?= =?us-ascii?Q?EbBXsJRPBV6YH7VDxkxRcDqfA8D5nEs+2Mc4F7qNRlLLzqqRehv59JBxOGue?= =?us-ascii?Q?t3Ekuf41yGYRXH5f8MIsnsry8Gac285o9BR37SzM1nS79sr+S9x1LpzjrXyW?= =?us-ascii?Q?BC9hijWfSFNpEFIQiTarmApwY1AuNEzUcjO/XO70J7W2OcyFsJm/J1u27nKx?= =?us-ascii?Q?hqiFG1MviDIlSYkgVyxo/Tm75h5xxV1gQODV0ebSobPtw9Q3NyicZ28Av4Av?= =?us-ascii?Q?6V/dEnedp2zJ1KzDLJdE6h7dEJr/r/4iV/vWeMpo6P2V/3GP5/sL2qacjOtu?= =?us-ascii?Q?+dmUEv+iLxGHxZxk0XwZ0wMoJahOVmIszzFhOAfkCSVpVfhpZ7Xsc1q5dypD?= =?us-ascii?Q?023u0cT6hppwGqj2FHWZIj09avEDjRWqUe37OoEzwTKOwgDd/2UT0XR5qPY0?= =?us-ascii?Q?DD7rKXF3pcV6UkODnBxXTs1qOIjbiH45XZrgA5hNrjsw6H8L63z3BSyZiYZ9?= =?us-ascii?Q?j5Svr0iSCPLLappb/iOnBMV/W+U/+o1jHU+hikiqpAEm5ksDz1xwOjkP7TY6?= =?us-ascii?Q?BWB505OVu1c/Z23qnDjTin/LibRctyjpvbpwVC47PkFTSLOaoRGRqKKvXA3h?= =?us-ascii?Q?L7jyZWVf6uLtLnRBjd3WJtG9kQcNxXURysjzd+rGtQuBbQxb8DPCcpYmGAIl?= =?us-ascii?Q?1H/YUjh8GmIsTpCqsBscEWDzWK990odQ3ZdXxvs4+YmdaEGk1MvHYzJs5nA+?= =?us-ascii?Q?ITlF41tAG7XLD9kexGMl6/o/lkZLWIML3Fr+Gy164tVs75uFXZR5T1ESQzSW?= =?us-ascii?Q?PZtGBP2E4OTpNhIuaQxZsBIYkI4RqctXtZ2B2cQRUKVNKyNJ5fh2A2QQi9lS?= =?us-ascii?Q?L3rROSwZx1NV3z3Qy/HWc5r3dcboe+0HOJV0dGfdnd8Wy0Qk7DN5Q+bzlVar?= =?us-ascii?Q?HZNBxQzeIj61PB4wnOEUw3ePKydKqYsaygdK?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(30052699003)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:19:28.1742 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9a4c066c-e5dc-4338-6a12-08ddeb6a3d35 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF0000150B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4107 Content-Type: text/plain; charset="utf-8" With the distributed nohz idle CPU tracking using the "nohz_shared_list", the trick to do the load balancing for the CPU in charge of the nohz idle balance at the very end using for_each_cpu_warp() will no longer work. Extract the main loop and the loop body doing the load balancing on behalf of an idle target into two separate helpers - sched_balance_nohz_idle() and sched_balance_idle_rq() respectively keeping consistent with the naming convention in fair.c. This will help transitioning _nohz_idle_balance() to use the "nohz_shared_list" in the subsequent commit. No functional changes intended. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 161 ++++++++++++++++++++++++++++---------------- 1 file changed, 102 insertions(+), 59 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d13de6feb467..ad6e97be97d5 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12580,6 +12580,97 @@ static bool update_nohz_stats(struct rq *rq) return rq->has_blocked_load; } =20 +/* + * sched_balance_idle_rq(): Balance a target idle rq. + * @rq: rq of the idle CPU that may require stats update or balancing. + * @flags: nohz flags of the balancing CPU. + * @next_balance: Pointer to variable storing the time in jiffies to + * trigger the next nohz idle balancing. + * + * Returns: Update flags for caller to take appropriate actions. + * NOHZ_STATS_KICK indicates rq still has blocked load after the update. + * NOHZ_NEXT_KICK indicates that *next_balance was updated to rq->next_bal= ance. + */ +static unsigned int sched_balance_idle_rq(struct rq *rq, + unsigned int flags, + unsigned long *next_balance) +{ + unsigned int update_flags =3D 0; + + /* If rq has blocked load, indicate via NOHZ_STATS_KICK. */ + if ((flags & NOHZ_STATS_KICK) && update_nohz_stats(rq)) + update_flags |=3D NOHZ_STATS_KICK; + + /* If time for next balance is due, do the balance. */ + if (time_after_eq(jiffies, rq->next_balance)) { + struct rq_flags rf; + + rq_lock_irqsave(rq, &rf); + update_rq_clock(rq); + rq_unlock_irqrestore(rq, &rf); + + if (flags & NOHZ_BALANCE_KICK) + sched_balance_domains(rq, CPU_IDLE); + } + + /* Indicate update to next_balance via NOHZ_NEXT_KICK. */ + if (time_after(*next_balance, rq->next_balance)) { + *next_balance =3D rq->next_balance; + update_flags |=3D NOHZ_NEXT_KICK; + } + + return update_flags; +} + +/* + * sched_balance_nohz_idle(): Core nohz idle balancing loop. + * @balancing_cpu: CPU doing the balancing on behalf of all nohz idle CPUs. + * @flags: nohz flags of the balancing CPU. + * @start: Time in jiffies when nohz indicators were cleared. + * + * Returns: + * < 0 - The balancing CPU turned busy while balancing. + * 0 - All CPUs were balanced; No blocked load if @flags had NOHZ_STATS_= KICK. + * > 0 - One or more idle CPUs still have blocked load if @flags had + * NOHZ_STATS_KICK. + */ +static int sched_balance_nohz_idle(int balancing_cpu, unsigned int flags, = unsigned long start) +{ + /* Earliest time when we have to do rebalance again */ + unsigned long next_balance =3D start + 60*HZ; + unsigned int update_flags =3D 0; + int target_cpu; + + /* + * Start with the next CPU after the balancing CPU so we will end with + * balancing CPU and let a chance for other idle cpu to pull load. + */ + for_each_cpu_wrap(target_cpu, nohz.idle_cpus_mask, balancing_cpu + 1) { + if (!idle_cpu(target_cpu)) + continue; + + /* + * If balancing CPU gets work to do, stop the load balancing + * work being done for other CPUs. Next load balancing owner + * will pick it up. + */ + if (!idle_cpu(balancing_cpu) && need_resched()) + return -EBUSY; + + update_flags |=3D sched_balance_idle_rq(cpu_rq(target_cpu), flags, &next= _balance); + } + + /* + * next_balance will be updated only when there is a need. + * When the CPU is attached to null domain for ex, it will not be + * updated. + */ + if (likely(update_flags & NOHZ_NEXT_KICK)) + nohz.next_balance =3D next_balance; + + return update_flags & NOHZ_STATS_KICK; +} + /* * Internal function that runs load balance for all idle CPUs. The load ba= lance * can be a simple update of blocked load or a complete load balance with @@ -12587,14 +12678,8 @@ static bool update_nohz_stats(struct rq *rq) */ static void _nohz_idle_balance(struct rq *this_rq, unsigned int flags) { - /* Earliest time when we have to do rebalance again */ unsigned long now =3D jiffies; - unsigned long next_balance =3D now + 60*HZ; - bool has_blocked_load =3D false; - int update_next_balance =3D 0; - int this_cpu =3D this_rq->cpu; - int balance_cpu; - struct rq *rq; + int ret; =20 WARN_ON_ONCE((flags & NOHZ_KICK_MASK) =3D=3D NOHZ_BALANCE_KICK); =20 @@ -12619,60 +12704,18 @@ static void _nohz_idle_balance(struct rq *this_rq= , unsigned int flags) */ smp_mb(); =20 - /* - * Start with the next CPU after this_cpu so we will end with this_cpu an= d let a - * chance for other idle cpu to pull load. - */ - for_each_cpu_wrap(balance_cpu, nohz.idle_cpus_mask, this_cpu+1) { - if (!idle_cpu(balance_cpu)) - continue; - - /* - * If this CPU gets work to do, stop the load balancing - * work being done for other CPUs. Next load - * balancing owner will pick it up. - */ - if (!idle_cpu(this_cpu) && need_resched()) { - if (flags & NOHZ_STATS_KICK) - has_blocked_load =3D true; - if (flags & NOHZ_NEXT_KICK) - WRITE_ONCE(nohz.needs_update, 1); - goto abort; - } - - rq =3D cpu_rq(balance_cpu); - - if (flags & NOHZ_STATS_KICK) - has_blocked_load |=3D update_nohz_stats(rq); - - /* - * If time for next balance is due, - * do the balance. - */ - if (time_after_eq(jiffies, rq->next_balance)) { - struct rq_flags rf; - - rq_lock_irqsave(rq, &rf); - update_rq_clock(rq); - rq_unlock_irqrestore(rq, &rf); - - if (flags & NOHZ_BALANCE_KICK) - sched_balance_domains(rq, CPU_IDLE); - } - - if (time_after(next_balance, rq->next_balance)) { - next_balance =3D rq->next_balance; - update_next_balance =3D 1; - } - } + ret =3D sched_balance_nohz_idle(cpu_of(this_rq), flags, now); =20 /* - * next_balance will be updated only when there is a need. - * When the CPU is attached to null domain for ex, it will not be - * updated. + * The balancing CPU turned busy. Set nohz.{needs_update,has_blocked} + * indicators to ensure next CPU observing them triggers nohz idle + * balance again. */ - if (likely(update_next_balance)) - nohz.next_balance =3D next_balance; + if (ret < 0) { + if (flags & NOHZ_NEXT_KICK) + WRITE_ONCE(nohz.needs_update, 1); + goto abort; + } =20 if (flags & NOHZ_STATS_KICK) WRITE_ONCE(nohz.next_blocked, @@ -12680,7 +12723,7 @@ static void _nohz_idle_balance(struct rq *this_rq, = unsigned int flags) =20 abort: /* There is still blocked load, enable periodic update */ - if (has_blocked_load) + if (ret) WRITE_ONCE(nohz.has_blocked, 1); } =20 --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2064.outbound.protection.outlook.com [40.107.223.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11AB21BC58 for ; Thu, 4 Sep 2025 04:19:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.64 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959593; cv=fail; b=P7edYPDu3nkXCPMsQrOkwrALU6wwXcwm5tVmkb2hAHfc9VJtgrsP7RZidDFW2upJ/0iLXDDraufa/MXp/2a9Y4WpXHZM3rMacFed0Wgr3NZj92onH9ttRbYY8feCg87JjR44Wrfpp7lahJN4HGfayjOIqlK/vHyT9mW294YLQd0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959593; c=relaxed/simple; bh=WqdP7UJjy009eMZBoQZNzqqvQDceWZ9gL3Ezc8E+4ko=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nmjr9OOOrG/5S+amJnN4Gt9/C+3SYLN6wWFmbROE7lmfj/JuS32/LTA1vjziousevLN/EmweTdxFJDhF7TKYpIm+/cMW7iMXCDHzqQ4Ud8AV6ogq/beuu02Bs0iyXbyDjm95tL2CTR0AasU1pVqVF1HJmLnwVut7eVyaMOzn2tM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=N+dH5AoI; arc=fail smtp.client-ip=40.107.223.64 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="N+dH5AoI" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=E4QhUsbmet9W2iT2I+x6DFdJFhGFgcWauQTBy/YyzSn/GoeRSAKcgiveU32VULg/VocDUEAj3XxGZgpXLXpR+gVoLdq19RxHNJpnjsfz6ksGO2DdGKPox/YxHMS4rRWPI0Hz29C0YE5RXzTDGZZvRsdY2PyZHOw984SaNWc2mxfZ6esL4OxdKyxATBIQBiafaRglLmryeXgLtUh6zNbcdQkmufMOnQQ5n6yEIJdVKEYGv1E4mICNM9BBWZv9hSDqJvlp1Igdb72PNh0URKr5xEPlGONAcxOk+ealUAdpwGAU0DJwXiunuALh78IbKCd/0t0Ckk4/vGBkJfn6GjIO6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9LKqP2qbbpTwvj2ay+7dEWLveJOHWMQlyYSE1RzOPic=; b=uyqkzUoNzk7czIFey9Njg46K4MpWZgsLR74AehGV4yDdWOsWaXlLiAta6vvKkxeDVs5M8x+hpidloAOENl9Ldmssn5s+CZJr6z5GyaffixVO1pWbQ1uc6AhKNU4J73pA6zqT0x/ozddk4tASZ1UXvRRSaAwWBopORhdgJTy4Fs6zkjsUrsj8KuIHt9aqcq+oPgT5D7zx7E0kXnJurPRbfZ8iFkOi9pqquBXWsSyI8lQB/l4GBczTjUJYGiFh0OCAYUyczbY4fNtiLC8Ham3fBbPdszCSjPdy1H/4vk6kQecKc8C+cy1raRd7yZn0fHxVQGpmQWisqS0VzF+1hQF/iQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9LKqP2qbbpTwvj2ay+7dEWLveJOHWMQlyYSE1RzOPic=; b=N+dH5AoIs4NVCrjvVdS5EPNnmKIbQ0AIuk0hsDdNGH6mWU5MkdNR6bU7E8MGipB99Ta1mco8uscriRutRn3AyoiWIKcOKUO4gk2sSUWxaBs6KSiXt416QFDcfgofODFQ9dwPKgL/jxj9wuX/6UdZLxMuOb0TGNTdtT9+TVtaQAM= Received: from SN7PR04CA0212.namprd04.prod.outlook.com (2603:10b6:806:127::7) by DS0PR12MB8044.namprd12.prod.outlook.com (2603:10b6:8:148::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9052.29; Thu, 4 Sep 2025 04:19:46 +0000 Received: from SA2PEPF0000150A.namprd04.prod.outlook.com (2603:10b6:806:127:cafe::4c) by SN7PR04CA0212.outlook.office365.com (2603:10b6:806:127::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.17 via Frontend Transport; Thu, 4 Sep 2025 04:19:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SA2PEPF0000150A.mail.protection.outlook.com (10.167.242.42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:19:45 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:19:45 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:19:40 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 14/19] sched/fair: Convert find_new_ilb() to use nohz_shared_list Date: Thu, 4 Sep 2025 04:15:10 +0000 Message-ID: <20250904041516.3046-15-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF0000150A:EE_|DS0PR12MB8044:EE_ X-MS-Office365-Filtering-Correlation-Id: 506b5b1a-5b19-414f-4971-08ddeb6a47b6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Z2YNYiQe/a8nFVucLrCEdxtB4N6ZMI+RRIRV2QklQ52wRgS+H7t7BfuW6j1c?= =?us-ascii?Q?eWHvjFxZ7tN3961wtIAGuU8dJew6pa/SEu2I1oABreMInCr41cD7S1CrJrzO?= =?us-ascii?Q?fyxZEw3yaKteRBl94k4FnSAyjypPZNyROcHAoDaEymFMQMRO7cms2fWbz461?= =?us-ascii?Q?CgOsuQ778OYrLgJgL53wF7K3tW377vMulsyQBN+dylLSgQ+yPAZ100fq6BOY?= =?us-ascii?Q?QfkTiHYAuLu+u4QK/IBZqshMuOQRzykCxjZli4eH4KiQTmBOcsFZeWqKJGFu?= =?us-ascii?Q?6qUC20IBSZPhfV4gFzz1tQh6jWdT5CGO79EFNq0H67cUDtpfifH97SO3EuqF?= =?us-ascii?Q?uwB6gtqciZyH8/b1Vd5njPj26wwslkslO62Xa74LsZb/ZIzac0rRqEHvO1+9?= =?us-ascii?Q?yxSc6P0XqazIa41dL58taKBR5PDroMy6tAx/dwnWgKMwJIjxcmhsxOYIyhvV?= =?us-ascii?Q?X9NpHA+3LnpNTh7OYmCo2Q223rbq78VLFG4JYXhJEyJfITY+r//OCNfE0QVR?= =?us-ascii?Q?pJsFn5nA5hWVtzb6Ky//159W1pBCJMoUibPeK34LEDxHs2bIR9hBwEQgBfk5?= =?us-ascii?Q?aTovEZYNnL2ubkJuLGD2DN8ZJ3D14SMs5iBciP/CrMv0TLxHWJw0zG9FAEZC?= =?us-ascii?Q?V1JnuN+NV3GkC46A8Wek186Rzvrd1sjGfIA76aM8JG0MePxQZOB5d8JyTRhk?= =?us-ascii?Q?Vq5nSGnXclBACUtINtlbA0y/hMWGa3QFgrH1+egXJofn0foHANhvDdU1Je37?= =?us-ascii?Q?VER9NNR0wT6dr8b2miEIjDFLekGA1Owet9uIuBU5864+gpdnKXiTiQvlu3ZS?= =?us-ascii?Q?d4Cs1dUFMzStMzkR7cgQUbdt4h7r96NdLMC0MlQMyZgjFFrr5wwolcsWSmM+?= =?us-ascii?Q?6A59Yfv1qt6/4y56XiccemTT38wvGborPmEgRNNzpKfhehpM7iazib5MfUnC?= =?us-ascii?Q?x035nRrqLJh1Xkt5rMIGNEn/V5IcEmnIP8neq+z0RJJ8lFZptT1VH7O1gCMf?= =?us-ascii?Q?VaY1UXXWF9UPPD2vLs6cMqZ6Te5drrv4y59ECezo0rUK0QSxNnZmAIfLv1q1?= =?us-ascii?Q?zMT7IDYZcZiwKYqAtvQEjNdFi1isSgUZ56Y6CZN76Ka480xY3J3djkwEk5Kq?= =?us-ascii?Q?fJI3JTvcIYcPOH8iyXt7rTNPS90fcKoAX4nxjI75BnNe49s2NJyAwvG5eH77?= =?us-ascii?Q?D1dkfR6zzChS2VM5n91d1NuShYcdFN8yEn8RiOvkkiRda4WBWHHFQp/T6Chd?= =?us-ascii?Q?q16buUMgulLY/i2aoUbCJiTFytiufO50sfHad31qA5itI5drun1uhARjco4d?= =?us-ascii?Q?u0xNMjBI90Xipqp26rL+cf8fHBjuFd89tDmrwGD4kYEuP/N0IvazeqOVRKyA?= =?us-ascii?Q?rDKObZ/rOkKMxtGj4DU778NvGSji3tMxnjVKuwP/ikOF8mmwkStfZWODQVFJ?= =?us-ascii?Q?tFrMp4rS8wlLyWdDa6I4oXAjTyrLTrIFUah5LrmWFpdcOhhxpqZJxcU7Kk5V?= =?us-ascii?Q?Z5+zqQBv73V+L+I4swM+YDjLCcG4wkMwkks7?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:19:45.7959 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 506b5b1a-5b19-414f-4971-08ddeb6a47b6 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF0000150A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8044 Content-Type: text/plain; charset="utf-8" Use "nohz_shared_list" in find_new_ilb() instead of the global "nohz.idle_cpus_mask". Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ad6e97be97d5..5e7198078d19 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12263,18 +12263,27 @@ static inline int on_null_domain(struct rq *rq) */ static inline int find_new_ilb(void) { + struct sched_domain_shared *sds; const struct cpumask *hk_mask; - int ilb_cpu; =20 hk_mask =3D housekeeping_cpumask(HK_TYPE_KERNEL_NOISE); =20 - for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) { + guard(rcu)(); + + list_for_each_entry_rcu(sds, &nohz_shared_list, nohz_list_node) { + int ilb_cpu; =20 - if (ilb_cpu =3D=3D smp_processor_id()) + /* No idle CPUs in this domain */ + if (!atomic_read(&sds->nr_idle_cpus)) continue; =20 - if (idle_cpu(ilb_cpu)) - return ilb_cpu; + for_each_cpu_and(ilb_cpu, sds->idle_cpus_mask, hk_mask) { + if (ilb_cpu =3D=3D smp_processor_id()) + continue; + + if (idle_cpu(ilb_cpu)) + return ilb_cpu; + } } =20 return -1; --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2065.outbound.protection.outlook.com [40.107.223.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2F683AC1C for ; Thu, 4 Sep 2025 04:20:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.65 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959614; cv=fail; b=peupvB6XuvOdwp/CjfFraVe235bTGXp2M+52PIEbDiAFQUArdj1nopmuHEpvN99QMaq9udI71VL/XKwUsoJcLYzGGDoPrK2qDZq6fBC+oNtyRJ4WbFgDFk/Qm3noG9qJy5eXl56IFv9o1XyE1o8fuPmxrJHPdCQurahPNi+q+KE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959614; c=relaxed/simple; bh=YMWp1oAGAUdWbY6heSmOcfug0aiKkQSvG+khnJXT8hk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=K194IX/tDqRusvHodeLlvpnXi3OkGwgFlwi7/Z8ijnjrsB4Nl1r7P1dOkZaGQhG4e35ohLjWs1WRwFGphf/KUYAbZeIHZ3D+rh5Xtp5G9rpDTGV7yJLCF/pUAJbD8jOXbDIi1GAC9YxPwNTQOmEai6axyvDgF/ddakCTbndxufY= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=sUQ/G012; arc=fail smtp.client-ip=40.107.223.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="sUQ/G012" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=wikI6gprCm7salNKLXorcoquHuArAQPwgraH7nE/kdy4S57UvZ51G5tXfks06Ydd4hp9441A3/d04wr3Rfd/uwqfITKSx7/cxdfLh/B6b1SEjxpMegTd/OOn1EnwNrWYiGtuVxpWUDK8RLoVjYPsjDJvGVzFh846o6OYouxa+Bv7dyGhKcTX7bThDtmnW32Gta5SN5mZFoChY0Q0sie5kTI7G1qF/yGQKZtpsD3IHLHZVVd8T+MJv4DI2HlINCNOOQNjnkztTyC0dPQW1ZhBKZPxu8k9u4ANtVjM9CQs11YHhWUNqU59AigB3VWm6ULGfaSzQAQMnWEkQOpWvIDfSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=J6ZH8cUYPxMEF6gJh4Qvdm7EFKssCP9kvh6CreJngeo=; b=H8d+8H7JQjVv8JGEWk6tWXVH6saR326PRsKxZDnkqNSnuLUH1a4gML3hMlFDu8sCaDvkSptAyge4vnsU/CJx5l4ayUpWnYk76oHxYP7sI5ao1StlIO7y3C7INVdjfAHo3UK25fiZ7xmWF8QaZqzxxnbn7JfnbtEJSkHyLfnZ7DR+ofKDQhHWsgU8TsKUPHBrir/+vLIwmOAeXZo/z7VjMujLQtQ6gLuw3VidsKYgBRJSoqvSdTF5skRlM9nELeGiTFtk47JktIpXa+5t7GVzxuUO1ohrfLhiLtUPSkPA3/+wN+c4Gp7SL4RJTD1b3905mJVo147knZDfmsLp25gZVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J6ZH8cUYPxMEF6gJh4Qvdm7EFKssCP9kvh6CreJngeo=; b=sUQ/G012CaTUsfdx2weybWBSt537MBtoae5Plu4v7ZEpAkbqLcAZkFh7pRfo1nFOztCkFj40JpCgqYQ1wnfWCI1I18Rug3151iShbJImPfwuD+gIjzlAzunqymcckWHgFfaeX6PLMEysYKvK8/I+xkf2i19/+PzFZJwpy5zAMpw= Received: from SA9PR13CA0143.namprd13.prod.outlook.com (2603:10b6:806:27::28) by MN2PR12MB4111.namprd12.prod.outlook.com (2603:10b6:208:1de::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9073.27; Thu, 4 Sep 2025 04:20:07 +0000 Received: from SA2PEPF00001506.namprd04.prod.outlook.com (2603:10b6:806:27:cafe::48) by SA9PR13CA0143.outlook.office365.com (2603:10b6:806:27::28) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.10 via Frontend Transport; Thu, 4 Sep 2025 04:20:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SA2PEPF00001506.mail.protection.outlook.com (10.167.242.38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:20:07 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:20:02 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:19:58 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 15/19] sched/fair: Introduce sched_asym_prefer_idle() for ILB kick Date: Thu, 4 Sep 2025 04:15:11 +0000 Message-ID: <20250904041516.3046-16-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001506:EE_|MN2PR12MB4111:EE_ X-MS-Office365-Filtering-Correlation-Id: 8ce241c9-19c2-4949-31f7-08ddeb6a549f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?0NMpEag1ccNi1xzst5ezdXxnVyFijKuardFZQq5poZJruT3OS7+L79iZwaRF?= =?us-ascii?Q?qUOM6CcKRdR0n0jk0jhUg5CVKd0+UahysgxF5Q74/5Pjr5FceQm6CASZFOJm?= =?us-ascii?Q?gwSms4o3zr9eKsYIZNEfXA+jG2UKlAyT/sXyCYoKWBtAneIHYGw53qigIv6Z?= =?us-ascii?Q?p7SewT3sNo0nDbfQQ/UmmMj3zURkEIcguFqYWxgfwv3+cbxZBHeCG9m6jXfQ?= =?us-ascii?Q?qzy2X0SBf74mhPlkcWicN24oH+aelRbgNs4UG8zpfaD2bqRrHX7Uv8cQP9s1?= =?us-ascii?Q?jaQzx5CNw9GQIABXI5XmT/Z5Pc7m7k/0aRCG34Ncjui4V6q4h6DxedtRIs4t?= =?us-ascii?Q?xAyqEmb05K7AfDCgOvrddwsk3+Qttr+ppkBSU89/oZzSHTkIYi2k/JzeWSyT?= =?us-ascii?Q?QdteZK400Ig8QWvYvyl5eRMSG+EtPvRV21c9RIs+Rj2AFmK4KyVb0wRPf+E+?= =?us-ascii?Q?jTjNqbRaOCjbMJWYTMEi+uups6VUHTaqbAOuk0YkOs9MyKNfKdzzAp4dhgsG?= =?us-ascii?Q?bVgNNrfiGwanZJrNtZ46bWlzhZR4bvc4FUkcbOoGEL+RsxsMgWizI2vlvEUq?= =?us-ascii?Q?11KStMPv1uyJORnPtiEVirzYFEvtDutdtiyYg1kcf7rSSAV7nKts0/Oz6dHy?= =?us-ascii?Q?RyFYdzebUQUvqy84XvlYuyypRnoJfSSUchozWidbxBu3Ik1ORuad9X2QfFmU?= =?us-ascii?Q?QBtkz1MN2Oq1po5GHjfy75i6nTKRyQIoIcUm7adv8zqb6/BkmwzhZce5BF85?= =?us-ascii?Q?XBYa88jVriZv3wENE75A7DSOP1R3BM6SwP3y0cbAA+mHlGydXqGYiYOin/9g?= =?us-ascii?Q?+9W8PfSFDiZM+m9vt3HLH1MerqDlr3Js49rGTdIcbc0EB5AIkF1aOjkiO7Gj?= =?us-ascii?Q?30giAEyC3m/O+ayZ0zO1DdMnnLlGzcFj2/3PoC9jMtcAamMDLNobnTSqe6+n?= =?us-ascii?Q?joQPfqZhCN5Jy00bKe+h6KX4VUUwuXkNtK8F95njHV6yVcK1ARi2TGEAnalg?= =?us-ascii?Q?h9fydgab3Ab8Acpzlkme/9ikB0W9mB2VG72Y05Ms9b81Qgeahj0xj0tETTLM?= =?us-ascii?Q?GyHMP59EOzW0Vajlevn/BZEDFnSwQmPTYH21A4vVV5vphZjfOIDQzS4hcEaQ?= =?us-ascii?Q?9dMm1KHeJKcm+WVHTjz766EHbFrlQ0+RkizdjSNAAc01XSudrCaPicoMNHr4?= =?us-ascii?Q?3/+E1U+ronGDDB9O5BXCTEfnhZUs571OUdaJw4HNkTn01E8nlS1f9bwr/rFq?= =?us-ascii?Q?8Zgk91mVZ5aQ43W1zFxegx541WKLZ0JwS4YCt3tQUxsS4Jaz2ROIWPvJ3lGf?= =?us-ascii?Q?t8+Kwek7WfuJrvlIvEhzKBvKUshGRg2fjJqUEwqnGqbZiX4LOrRbI7zHOYnX?= =?us-ascii?Q?YQ2XC3CB+IulWjlN1RTEO6JG9ctC50quM369OMl0K7R1zfRdjp13jt9YVt8q?= =?us-ascii?Q?j3hSKmRAzEm2um5jIv42dIDsc8p2im4xPvVGzxDN1IyDx3MSf40GVnMasrL0?= =?us-ascii?Q?+zZubzn0yOr0VYourjC1fNoAQ3UXbb+McfdE?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(7416014)(1800799024)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:20:07.4603 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8ce241c9-19c2-4949-31f7-08ddeb6a549f X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001506.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4111 Content-Type: text/plain; charset="utf-8" nohz_balancer_kick() currently depends on "nohz.idle_cpus_maks" to kick off an idle balance if it finds a more preferred CPU in nohz idle state. Move the detection logic into sched_asym_prefer_idle() and use the distributed "sd->shared" based nohz tracking to kick ILB if a more preferred CPU is found idle. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 70 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 54 insertions(+), 16 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 5e7198078d19..d309cb73d428 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12289,6 +12289,56 @@ static inline int find_new_ilb(void) return -1; } =20 +static inline int +__asym_prefer_idle(int cpu, struct sched_domain *sd_asym, struct cpumask *= idle_mask) +{ + int i; + + for_each_cpu_and(i, idle_mask, sched_domain_span(sd_asym)) { + if (sched_asym(sd_asym, i, cpu)) + return 1; + } + + return 0; +} + +static int sched_asym_prefer_idle(int cpu, struct sched_domain *sd_asym) +{ + struct sched_domain *sd =3D rcu_dereference(per_cpu(sd_nohz, cpu)); + struct sched_domain_shared *sds; + + if (!sd) + return 0; + + /* + * If sd_asym_packing is a subset of sd_llc, don't traverse + * the nohz_shared_list and directly use the mask from + * sd_nohz->shared to find it a preferred CPU is idling. + */ + if (sd_asym->flags & SD_SHARE_LLC) { + sds =3D sd->shared; + + if (sds && + atomic_read(&sds->nr_idle_cpus) && + __asym_prefer_idle(cpu, sd_asym, sds->idle_cpus_mask)) + return 1; + + return 0; + } + + /* Try the entire list id sd_asym_packing is larger than sd_nohz. */ + list_for_each_entry_rcu(sds, &nohz_shared_list, nohz_list_node) { + /* No idle CPUs in this domain */ + if (!atomic_read(&sds->nr_idle_cpus)) + continue; + + if (__asym_prefer_idle(cpu, sd_asym, sds->idle_cpus_mask)) + return 1; + } + + return 0; +} + /* * Kick a CPU to do the NOHZ balancing, if it is time for it, via a cross-= CPU * SMP function call (IPI). @@ -12343,7 +12393,7 @@ static void nohz_balancer_kick(struct rq *rq) unsigned long now =3D jiffies; struct sched_domain_shared *sds; struct sched_domain *sd; - int nr_busy, i, cpu =3D rq->cpu; + int nr_busy, cpu =3D rq->cpu; unsigned int flags =3D 0; =20 if (unlikely(rq->idle_balance)) @@ -12389,21 +12439,9 @@ static void nohz_balancer_kick(struct rq *rq) } =20 sd =3D rcu_dereference(per_cpu(sd_asym_packing, cpu)); - if (sd) { - /* - * When ASYM_PACKING; see if there's a more preferred CPU - * currently idle; in which case, kick the ILB to move tasks - * around. - * - * When balancing between cores, all the SMT siblings of the - * preferred CPU must be idle. - */ - for_each_cpu_and(i, sched_domain_span(sd), nohz.idle_cpus_mask) { - if (sched_asym(sd, i, cpu)) { - flags =3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; - goto unlock; - } - } + if (sd && sched_asym_prefer_idle(cpu, sd)) { + flags =3D NOHZ_STATS_KICK | NOHZ_BALANCE_KICK; + goto unlock; } =20 sd =3D rcu_dereference(per_cpu(sd_asym_cpucapacity, cpu)); --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2077.outbound.protection.outlook.com [40.107.100.77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1FE081DED4C for ; Thu, 4 Sep 2025 04:20:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.77 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959635; cv=fail; b=KeuOEjASPvYydWwdF2DQhKWKbDvfq6UnuaVzrs0mX/3sthX5I/abyMBQOPyNjdNxo2uCa/Q36sO7BKYeZCqfU9Hg3IalrvwIwjTHDpvbwCJYcQWQSfC/rTLYGacZlH+fepy3B7jLS5mzCo6sP3PW+Yi6uuBUV9ikC3EhDveVj0I= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959635; c=relaxed/simple; bh=F3E3zYBbHyopIXwqIxcJek4bfuJ7LiBQO4n/7nIjFdw=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=YcJ+3wlMxirOk7M2kktqxAQnRNemK34jv1+xbgkw+MG9S7DXfkfZOFFZCy8KMPaQZgGCf4jqKWQSTd6Dc7paw5bjCNtxXzjiP0pOh89+CMBUDpiRfB2zC90PydfcMDDgOFYEHih17vCb2BhhLX+7z7geCemgVdadg7L5oW0OTXU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=GGx92Cln; arc=fail smtp.client-ip=40.107.100.77 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="GGx92Cln" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fpQG77PaqLcA5N8lvC2T+VxPXnzgMd22FGBOd1IgDkUsdy8BA1taXFVTs/quiKj8hFffOC/5mslM94vqXvOaAMJTjcy+wqK+CZD/fX1+n2dCLLe/418/kvlAG181JydBc+RJ9r+F9YYw04EgHUGM0m4VLxJW7qNFgSFW79yvxIcv8G86smfWVBrvsq4xsk8fkSLyimo3/Y9mEuwncVru7IoNOK0qbfVzu+wO8/7LYcQlZauexUfwAlQEhUOkyaOQAjsaw+lKsSoBkvN1mAJTyTBZ2HKRuJ6N2cux15o8jHUlCuj4KJmygGRjiYhNJCMhsKKcEeBe0j3TL2gpBoATug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3YoPxriIVapTGkqMB7FGQ9QL2f5SA2AMi4KllrwrYZc=; b=TQTA+XF0OQW+WtxVByYqtxuczrh5+m50q+Gl07ysieiG9c1LqtdWWzXhcHOQoretZf5UUykHTkx1aJXi9zcCl+Qxc3rxZ2z4pgtp0SZqJCvkzwqomOFxlMSR4AI9ssKOOjSo45IYJUUTqfsPGeQpUuFLMgo7wIZkjM/6/lT77QWhFh7gOA2uHGWqnhucMEhyP95GrICT9oAb6VHcTjEvWF7BvBoDLwdbL7i91GX0EAE8a9t4WOudyYZHn5XjqA06nFoxnIh0XEVK1CMVSvo1S6Oz7jiFAOH6YT9oz3wUTHGwFZK5rriS3EUOxZDr5cUVPrh1nTk57cGeY7R0oLzjIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=temperror (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=temperror action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3YoPxriIVapTGkqMB7FGQ9QL2f5SA2AMi4KllrwrYZc=; b=GGx92ClnJNpz5vi4QwXZH8ewgoz5wiZda1pFhv6ZEJNkjhzpAipRo+PwqCOQJ4cqYNjCQoUryMf4hpPB7Hakk3BGjxRvmbaASLK+GzrKoG+uxsUyJw8VjmFSnloHcpBabv521CcsBq53GOwXWEM3bB1mWTh2SEEmCUPcr3H8ZBY= Received: from BN1PR14CA0027.namprd14.prod.outlook.com (2603:10b6:408:e3::32) by PH7PR12MB5619.namprd12.prod.outlook.com (2603:10b6:510:136::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9073.27; Thu, 4 Sep 2025 04:20:27 +0000 Received: from BL6PEPF0001AB4B.namprd04.prod.outlook.com (2603:10b6:408:e3:cafe::51) by BN1PR14CA0027.outlook.office365.com (2603:10b6:408:e3::32) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.18 via Frontend Transport; Thu, 4 Sep 2025 04:20:27 +0000 X-MS-Exchange-Authentication-Results: spf=temperror (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=temperror action=none header.from=amd.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of amd.com: DNS Timeout) Received: from SATLEXMB04.amd.com (165.204.84.17) by BL6PEPF0001AB4B.mail.protection.outlook.com (10.167.242.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:20:26 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:20:24 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:20:20 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 16/19] sched/fair: Convert sched_balance_nohz_idle() to use nohz_shared_list Date: Thu, 4 Sep 2025 04:15:12 +0000 Message-ID: <20250904041516.3046-17-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB4B:EE_|PH7PR12MB5619:EE_ X-MS-Office365-Filtering-Correlation-Id: 75c34108-41e9-4754-699d-08ddeb6a5fe9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|376014|7416014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?w86NuQ/cutBXzR4nOZZEhUXgDgZfduaghkzEw9XEQWWP8xHJdJMAcE1mcW61?= =?us-ascii?Q?FMw+zu2+LCmDAOEGcIGv0g4vT66QerJ4kgTxXLYaN3bjudmgtHSlw3RNOjYu?= =?us-ascii?Q?0H+mZPCzAcSvb8CV5t3ZA9pn430+p03zyyIFlsCaTYK4fh+7vwQ1oKeAae+e?= =?us-ascii?Q?0NQ2Psw062UTFFL+ZuXG5k30N7kelesYLgcIfEs/mqfD0DxyKNUlEh3SRlLr?= =?us-ascii?Q?Fks+nVlYE3cHJCGWeFAhTsyPQO9soIibyuZhnw+34H3jpKXQbTMWa3Tyzjn2?= =?us-ascii?Q?L7oRqn4CRYJGBUolouuTL1QR/1aGE9D1CAPU8h7wdy2iLH3TuQRwOqrTDd6E?= =?us-ascii?Q?8u0DRxKKiSeVgrkIZFrNGqM1jMZc9kD1cFGzIg46/2S+T/S+bV3tLefyadAr?= =?us-ascii?Q?nMGITvd5sxeoOE3PtPrwrqFABtDwuxwRrK+FRoCoPngrNSigASgwUNRqkXp5?= =?us-ascii?Q?CiL1g7nwt1K7qMpDkkWb80QlJrxFm0vpVdG3sSAC/4VI5y7EEFDZuQVbDlt8?= =?us-ascii?Q?eSs3SZTZRGWWFSVVhE65l7W7L93KRwv30/hHtiwo1NYyYqqIEDXgOf5hDHwh?= =?us-ascii?Q?Ex6kkliK6vEt/uJIzvQhXVgxjwNobnLuZTJHRZh9uIHvBY5422kvs6QCjcnk?= =?us-ascii?Q?DKMV+sKg+PWqjKXBvcX0vGnZDaYKqcBLJ8grNRlkkMtPtvthhZ1oPFtc3S3s?= =?us-ascii?Q?LrUhiDPMq1QrgsaWQq/28hyU8VDem+pw5nUiB1C3iCsJDprLO8eXlWU4HkVC?= =?us-ascii?Q?KOrBzgEV7eI7HVTmiJcaj434roYbyCu0/BQi1W5IGfzEUxJTuklyUebHIrko?= =?us-ascii?Q?5YD/6jW0mv2h39BVNEkqd8RIYu7tCxZ11jHqvsEYjCgutP523VrC/5eGw4qM?= =?us-ascii?Q?I5x7IWJb/VvbPcuOWn3OG7e5emzWy+mikV2xeRRrQ8xm3e4AMjG6/C/9CU3J?= =?us-ascii?Q?HNWe4IDsBqqRr53roIsX9rbrOez0d/XiohjvHrckDRpha2cYHQYAK6hU9P8S?= =?us-ascii?Q?XOiv5JmtyE4Et2zlX5No/N03jKUhwKoBiMPotWcKIIcI/3DsCI5W2EZxoZCo?= =?us-ascii?Q?7xHEXgHp5blh4Gu5A8oUlt2fJ1Sx0zmUcw35ZS4FaC8qyG72WQ+3r96pU8Z3?= =?us-ascii?Q?h9P7HAH5b+P3R0ejfbDS8/GCbDINGBVN705soglFMuSUu7LX4V3IZDtDhQku?= =?us-ascii?Q?2/HWZ3xjKnHHhTETv97gb8N/muIqGfB9S8Blcot/8Ef8l+6RHrYcUq91N7Wt?= =?us-ascii?Q?hISRvchwqyLrq27fDSZBO72LxPsqB1sBxvxKbSVIerOIT9cjwutRmd8Kgd1E?= =?us-ascii?Q?J1x92JrnRTP7TOoFr6XpoVNtEhcZV2qFV/DBmF9j4MmFauhI/9VG23MC1FwF?= =?us-ascii?Q?uF3AQXvdyIcDuscYvIPh8siIGif6osTo2opMusn3Ez0FbU5zxzfCI7ioqHDq?= =?us-ascii?Q?nkxYBxuBAhnTzZdcuI0cSOpL8toqiwFl2UD/wSqVN1GLgx2Zd+jJzzNIAS7L?= =?us-ascii?Q?EbzDnWKiJSOU/fyHFkqAHz6fkjLSj/tIuxi3?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(376014)(7416014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:20:26.4229 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 75c34108-41e9-4754-699d-08ddeb6a5fe9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB4B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5619 Content-Type: text/plain; charset="utf-8" Convert the main nohz idle load balancing loop in sched_balance_nohz_idle() to use the distributed nohz idle tracking mechanism via "nohz_shared_list". The nifty trick to balance the nohz owner at the very end using for_each_cpu_wrap() is lost during this transition. Special care is taken to ensure nohz.{needs_update,has_blocked} are set correctly for a reattempt if the balance_cpu turns bust towards the end of nohz balancing preserving the current behavior. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 62 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 47 insertions(+), 15 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index d309cb73d428..c7ac8e7094ed 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12685,27 +12685,59 @@ static int sched_balance_nohz_idle(int balancing_= cpu, unsigned int flags, unsign { /* Earliest time when we have to do rebalance again */ unsigned long next_balance =3D start + 60*HZ; + struct sched_domain_shared *sds; unsigned int update_flags =3D 0; - int target_cpu; =20 - /* - * Start with the next CPU after the balancing CPU so we will end with - * balancing CPU and let a chance for other idle cpu to pull load. - */ - for_each_cpu_wrap(target_cpu, nohz.idle_cpus_mask, balancing_cpu + 1) { - if (!idle_cpu(target_cpu)) + rcu_read_lock(); + list_for_each_entry_rcu(sds, &nohz_shared_list, nohz_list_node) { + int target_cpu; + + /* No idle CPUs in this domain */ + if (!atomic_read(&sds->nr_idle_cpus)) continue; =20 - /* - * If balancing CPU gets work to do, stop the load balancing - * work being done for other CPUs. Next load balancing owner - * will pick it up. - */ - if (!idle_cpu(balancing_cpu) && need_resched()) - return -EBUSY; + for_each_cpu(target_cpu, sds->idle_cpus_mask) { + /* Deal with the balancing CPU at the end. */ + if (balancing_cpu =3D=3D target_cpu) + continue; + + if (!idle_cpu(target_cpu)) + continue; =20 - update_flags |=3D sched_balance_idle_rq(cpu_rq(target_cpu), flags, &next= _balance); + /* + * If balancing CPU gets work to do, stop the load balancing + * work being done for other CPUs. Next load balancing owner + * will pick it up. + */ + if (!idle_cpu(balancing_cpu) && need_resched()) { + rcu_read_unlock(); + return -EBUSY; + } + + update_flags |=3D sched_balance_idle_rq(cpu_rq(target_cpu), + flags, &next_balance); + } } + rcu_read_unlock(); + + /* + * If we reach here, all CPUs have been balance and it is time + * to balance the balancing_cpu. + * + * If coincidentally the balancing CPU turns busy at this point + * and is the only nohz idle CPU, we still need to set + * nohz.{needs_update,has_blocked} since the CPU can transition + * back to nohz idle before the tick hits. + * + * In the above case, rq->nohz_tick_stopped is never cleared and + * nohz_balance_enter_idle() skips setting nohz.has_blocked. + * Return -EBUSY instructing the caller to reset the nohz + * signals allowing a reattempt. + */ + if (!idle_cpu(balancing_cpu) && need_resched()) + return -EBUSY; + + update_flags |=3D sched_balance_idle_rq(cpu_rq(balancing_cpu), flags, &ne= xt_balance); =20 /* * next_balance will be updated only when there is a need. --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2059.outbound.protection.outlook.com [40.107.223.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18FD684039 for ; Thu, 4 Sep 2025 04:20:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.59 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959647; cv=fail; b=vAvXhgWaB2yqjEPX5/0Rsd0i7Ii+3wm/1r4Wrk59pTXz+o3KXdNzJDwWjVZUbb3k0JFt3ps9AYFkJGxoMIyProxV5yABXx+BPUiCKQtQNuYcHGlIertsNsl+0HytxD5Wzf6BvXoFVU4gc3HIDmBU8qty90r+uaQpmOm6Pq9pNRU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959647; c=relaxed/simple; bh=hKyvYl5PTn64DXXq3r80m8W91MQZPadx1o9nxdUukA0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ITGT2kaPQ7389jpqGtl2e5jbrwHXkHUbKDRkqo0hcobDbwH43O0hz7oI6av7LVFB9gaU5QLM9/VTPT53KmaT43xZc96k792SBEVBC6Gb3miT1cwenYhZoLwC1en0BC6EgthX4e4LGG0reO2EGb7bBx13/4o7rSEdE6T9NDK/yhg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=oAJw9VpN; arc=fail smtp.client-ip=40.107.223.59 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="oAJw9VpN" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ja/gDHhcyftlx6ez1Wasvuv+fMlNGf5IJJcRcT2Ylk5GUxY4z3R8aitHgQ/cpMrSF+f9QKzdytCyuy474LdBjo/kXgM+9fVq+HApBwz0Ki6kdBFWAfdw+hnMB6Q16STo7erRTlZZFSRL2nXfPQzhsxPpwZ6tPyTHCYOMGrby4gfFmdcXU38jnI4rxE6MDp9GwxwzNkwD2lcxr5FSJ2y31JChgxOAhNzrIRogsmMF/pMO7Pnjx19jnBXr2KE3psgNjozR6cuJFAl2IoE2ij36WriKutA/7uDsiq9bF4uVvIiqCf/eRBUuh0908xjZSS0F3GQrOV0T6vwZpxUzwypjKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=K/twyjNUGVacluBbzZ/TgA9cups/thiEJLF5ODhc6u8=; b=xOaMObeL1T+6dHbqal6w7Xz8m/r6Pxt5tF/mZSSOA4azY00sUvI/DPOvFa04ZfYEI35v4IPGfb3YGu0iZpMHyB+Jj3IPme24NISx6AoAeWTNd0iOaglVAmO2wwJPs/Gq/yfq1M+3mFz/3fJc1qnDODFUipfKnttxMrtKe2UhT1Rt+nrDaLmFGriqYuK6RC3e07ZesFfmtLuDH5Bgcomd5z/6bIP+XyKoePgmlYUdQRPSKvXovyjfK4mAeYt0eimSgtaFhfbhypRoSiwi9lMnrDUeVOo/9bHOrAF5S75QOwWyuz1i232liJRIHYASu7xqsIeZnzRbI1JIN3kJM0fywA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=K/twyjNUGVacluBbzZ/TgA9cups/thiEJLF5ODhc6u8=; b=oAJw9VpNNLb0gxwQ+hMpe91zbSh6UJ3RuLtMmd1hffPBpUBkBhM2E5uLoTljh5eYteJrEKubyMeUaksJcgdD+V862m4oTtys7M5fDPl+Veh566SKTpJFS4JC0cIHMvUYPRNNZgxZr0XOcO2Esfu1PyhRkMQfuFZ/Rm8mtz5FzPA= Received: from DM6PR07CA0067.namprd07.prod.outlook.com (2603:10b6:5:74::44) by BY5PR12MB4147.namprd12.prod.outlook.com (2603:10b6:a03:205::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9073.27; Thu, 4 Sep 2025 04:20:42 +0000 Received: from CY4PEPF0000E9DA.namprd05.prod.outlook.com (2603:10b6:5:74:cafe::bf) by DM6PR07CA0067.outlook.office365.com (2603:10b6:5:74::44) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.17 via Frontend Transport; Thu, 4 Sep 2025 04:20:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C Received: from SATLEXMB03.amd.com (165.204.84.17) by CY4PEPF0000E9DA.mail.protection.outlook.com (10.167.241.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:20:42 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:20:41 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:20:37 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 17/19] sched/fair: Remove "nohz.idle_cpus_mask" Date: Thu, 4 Sep 2025 04:15:13 +0000 Message-ID: <20250904041516.3046-18-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000E9DA:EE_|BY5PR12MB4147:EE_ X-MS-Office365-Filtering-Correlation-Id: a5aee6b6-14ea-42c6-9bd4-08ddeb6a694a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|82310400026|7416014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?YQOo/4bH00sLg/wmd0KXttX4Jb32jap1rpH0RjBoScPsb7uytSk7LqwCKQ7I?= =?us-ascii?Q?Cy5p4p4qOHn1ipz8JPrFW7WBH9mgHtxpSAr+U8Iy6Wj3ucilqs+CYqaCo0RM?= =?us-ascii?Q?WxI1d8QHJcmtw0kVBTbYWDv4+yJMLsX/5hMP6E8xWhs4xIB89jtielGkwvvw?= =?us-ascii?Q?HRbXGPE3XA8G7xpbIoE+sJOIvOWkOwaQzF6yOdZJ0sNnmj8SjW924FrNbU6k?= =?us-ascii?Q?VXwW7wG42ngDCmN1fd2vxAjUeXl/+mHrsT40diCXt3Rbe4fXQwa+Bo0lLDTv?= =?us-ascii?Q?0bzMeAUfu648TmeO7BmxA4mwMhMgiTghGEVHPXhcpeidaNBDXfQeKGBMu6Rv?= =?us-ascii?Q?OQkmQGrOlQy9XbU2U4yXCHJd+B5AFmOxGPhYiuZN5cZ+A56eO/g8A7GUlwVr?= =?us-ascii?Q?cwmUv9IuPQLqbEWwKhJTmfZjMQHhBF7SvbRx7ob39eYOJO3zB2dGe8LgThU4?= =?us-ascii?Q?MD/IVOwqhxJCxUpZwotD4mJLkZtsioq1aXr1gfsWLyTHmdr8ItncHQpcKdso?= =?us-ascii?Q?7cJIUwFTiwRPofaj4rDqHtv3KTmL60WKNit1JoMCvoJKdnJhlfLakS6p3eDs?= =?us-ascii?Q?0Sy4dVBhksGrK/NbLVK0yNfPJvMphuV3drACkxhmNJIpcDmOcBqcd3vDS9yy?= =?us-ascii?Q?QetjuWxFDtDrX9prsEd/Vl76c9GjKtNWk5qf4NRsGzHsMS+rL0qROr+wKb4G?= =?us-ascii?Q?I8gwgkccxRSZgu6kzpABWR7cWVRnIVtB3sjt8NWdVCL5PejYT/7K3JT/5+vq?= =?us-ascii?Q?jkY96mX8OA0OTdVQ4k3Jwv4n+3dHL0gQuUv4Ft5JKHD3NBVVGtq/n6WkDTcz?= =?us-ascii?Q?q91hcNAUG1JsPwL0LQb/dyy26pm5PTKTwMSwPAn26A+vLPHFRTer6KIogCqF?= =?us-ascii?Q?EOt3ywuj1jxpNBMYGJZ1vBHEyVDbbSA+dTrpYpgBO+BSPsWxld2f5Tp7eRtV?= =?us-ascii?Q?5G4qbylc14KPDkQ4P4HB2lzxIAcZbrL0rNkM/1hq/OCXUZtnDOW9no2uv0J2?= =?us-ascii?Q?dbvyNG/Itm2vfQ1zcuI8/PvaZZJkc65evaSqj6XJtcMXGD1rJgepjGhiOuOG?= =?us-ascii?Q?EpxwJUZh3Xn0/0PtQWJ1EpjnLVq05HvPBCu3I09kluECeI7VAEKq8q0/Y3CK?= =?us-ascii?Q?/zEKvvEDetkoR68DI+LkoshWGSVTSzALId5hFldlfV7TgQrgFTINW/iTTw38?= =?us-ascii?Q?riGF856jILQo7KKLsBnGbDWX31LJw9Ezxl75G7BX0SVqXWhJivTUuWZ6FuNv?= =?us-ascii?Q?/UQBoc4Mo0udrag44XGYhHg+yZ1nRw7/rR9TQt5EQj6XKoKdwD8yAR0Rbyjp?= =?us-ascii?Q?hpTCkzRR9jgTvIoiTvqszd0YzIH+ixPm1HwSqx6tuOUCQDGbS3Na/rHI3BTK?= =?us-ascii?Q?KEpldIZBsDNYPx4iyHU15r5VfHMWKX/dVB9aKLuhHbBXgPLLb6eFBIAJ1ilH?= =?us-ascii?Q?BxTkGJnOs/VE4zsl3x2RN9npCAaaRIcJBYMjxSd7mpHub15kE0FWFyPmMpGX?= =?us-ascii?Q?//V9odBbW18kX9KclqoKi67M8Y55s12N5/40?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(82310400026)(7416014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:20:42.1148 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a5aee6b6-14ea-42c6-9bd4-08ddeb6a694a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000E9DA.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4147 Content-Type: text/plain; charset="utf-8" All users of "nohz.idle_cpus_mask" have been converted to use the distributed nohz idle tracking using "nohz_shared_list". Get rid of the centralized "nohz.idle_cpus_mask" based tracking. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index c7ac8e7094ed..5b693bd0fab4 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7169,7 +7169,6 @@ static DEFINE_PER_CPU(cpumask_var_t, should_we_balanc= e_tmpmask); #ifdef CONFIG_NO_HZ_COMMON =20 static struct { - cpumask_var_t idle_cpus_mask; atomic_t nr_cpus; int has_blocked; /* Idle CPUS has blocked load */ int needs_update; /* Newly idle CPUs need their next_balance collated */ @@ -12517,7 +12516,6 @@ void nohz_balance_exit_idle(struct rq *rq) return; =20 WRITE_ONCE(rq->nohz_tick_stopped, 0); - cpumask_clear_cpu(rq->cpu, nohz.idle_cpus_mask); atomic_dec(&nohz.nr_cpus); =20 set_cpu_sd_state_busy(rq->cpu); @@ -12576,8 +12574,9 @@ void nohz_balance_enter_idle(int cpu) /* * The tick is still stopped but load could have been added in the * meantime. We set the nohz.has_blocked flag to trig a check of the - * *_avg. The CPU is already part of nohz.idle_cpus_mask so the clear - * of nohz.has_blocked can only happen after checking the new load + * *_avg. The CPU is already part of sd_nohz->idle_cpus_mask so the + * clear of nohz.has_blocked can only happen after checking the new + * load */ if (READ_ONCE(rq->nohz_tick_stopped)) goto out; @@ -12588,7 +12587,6 @@ void nohz_balance_enter_idle(int cpu) =20 WRITE_ONCE(rq->nohz_tick_stopped, 1); =20 - cpumask_set_cpu(cpu, nohz.idle_cpus_mask); atomic_inc(&nohz.nr_cpus); =20 set_cpu_sd_state_idle(cpu); @@ -12832,15 +12830,15 @@ static bool nohz_idle_balance(struct rq *this_rq,= enum cpu_idle_type idle) * entering idle state. Here we run ILB directly without issuing IPIs. * * Note that when this function is called, the tick may not yet be stopped= on - * this CPU yet. nohz.idle_cpus_mask is updated only when tick is stopped = and - * cleared on the next busy tick. In other words, nohz.idle_cpus_mask upda= tes - * don't align with CPUs enter/exit idle to avoid bottlenecks due to high = idle - * entry/exit rate (usec). So it is possible that _nohz_idle_balance() is - * called from this function on (this) CPU that's not yet in the mask. Tha= t's - * OK because the goal of nohz_run_idle_balance() is to run ILB only for - * updating the blocked load of already idle CPUs without waking up one of - * those idle CPUs and outside the preempt disable / IRQ off phase of the = local - * cpu about to enter idle, because it can take a long time. + * this CPU yet. sd_nohz->idle_cpus_mask is updated only when tick is stop= ped + * and cleared on the next busy tick. In other words, sd_nohz->idle_cpus_m= ask + * updates don't align with CPUs enter/exit idle to avoid bottlenecks due = to + * high idle entry/exit rate (usec). So it is possible that + * _nohz_idle_balance() is called from this function on (this) CPU that's = not + * yet in the mask. That's OK because the goal of nohz_run_idle_balance() = is to + * run ILB only for updating the blocked load of already idle CPUs without + * waking up one of those idle CPUs and outside the preempt disable / IRQ = off + * phase of the local cpu about to enter idle, because it can take a long = time. */ void nohz_run_idle_balance(int cpu) { @@ -13841,6 +13839,5 @@ __init void init_sched_fair_class(void) #ifdef CONFIG_NO_HZ_COMMON nohz.next_balance =3D jiffies; nohz.next_blocked =3D jiffies; - zalloc_cpumask_var(&nohz.idle_cpus_mask, GFP_NOWAIT); #endif } --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2062.outbound.protection.outlook.com [40.107.236.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB7B31D5CC7 for ; Thu, 4 Sep 2025 04:21:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.62 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959666; cv=fail; b=opuKk/li8tfq2PwyQx0NRDJ0Wuz5CIhtVTLrzE56QEe2tMBGIVEsmdOJH1T2UmCgVekWHBTAIzkPfVSvBDYddD+dYH6v5dLRVC/ELwMKFWlGxW0az93gh731q+54emR2hPFWCABJJUvkApy713pOa2URHJ+xYCQVpOQ2Pc/qh9k= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959666; c=relaxed/simple; bh=nr17AjcVe1HiZTAB3y0lXa8nBZWVA5M1AJIb8/yEQwM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=nAkxh+eRr/UhGPlv2a9f/8SO5OVqIRjFb2N/d8cWhdAPktn49SdmVoVfU5UVf63/l7apDz2Y8Y1v62fFgDJpXmC5qRfBzh3tH0h6XNiks28ii2uYDjM3uz7mEHPKIslHPa3JzbKeKd4yBKG7aYC7EFyYb3i42TnIrRH6cR3+Dkw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=zKFTV3zg; arc=fail smtp.client-ip=40.107.236.62 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="zKFTV3zg" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=E9cvvz5XUu+ryNAOoTBjT5xVAl2IPs9yF1AjZbv1fwBEQ7/WcZAnyTUYDMz/eD2ImKbNW98r3D0JjtG1fXvLuudCiEL7pdalqOiE/Emvv2jF4hiR+C4j/xfvOlHQGoRH1jkQnMNVG7z10s6/dF/gc3lXEH7SAwWKaQNDuHrl+4DrUnIP2oRYpu7gBd1IrOEkshPNt5FejWE4QVcYZ06GwRx9GJXSXFn9xQ+dmVp+EfzGMcOpilXDbiQhdKrHZ7qKHhlZ+Wt35Nnny7qMs1rT7Fgz3nmXNVjCS+Q2azMz4R/24+mIC+Sdm0Fl3RJa1ghTN3encuY6/2ZG6MqpT5Oozw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vv1JvgBBI7j1I+4EsOc/0xkP3+S5VtDpVyOtvLrYIIw=; b=Seni9SA1lLsVokRL/BuY2daHiZhUchdlsj7gMBqBQk5gbINnfIng9EKYdBgiExwYzSGb3gQinwELHsJ5Q/hcXMoTepW7Im75XfXVGJZ4X07GvI1H7SPlqornv8RNlUJE61Fw7QNRHSa0A23FWQhMjoeGdKC/mAi3bmNjahDYrPJchaDWa1oKUzfh9kCKnuUAnU0ThdfiaBGqCaTvui/pxdZI14xKLr23EbPxF3uwak9FenOp1VlMf4w9digDnmmUemZQfAxTAwjU7TDw44oQme1x8w1hkeTIHua7gMADiyZU9rgcJU0m8E+lBatUnMrTcutpiJVEqXx0Jvuv/KLRXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vv1JvgBBI7j1I+4EsOc/0xkP3+S5VtDpVyOtvLrYIIw=; b=zKFTV3zgdqY1tDAoG/tjd3PImBidn945ZtRi2YWhPSd+76rrzdEpD8rd+iTJzAzWVuACmWqYIjZ0q5ZchkckYTVvX0Y6ijXUzYAooO8BCLJHujMNhk+NeY9C8CaRAX250vLwDx1t/4yhrHiXBC3J4TZ5y0p/Z03GmWx5kvjhyJI= Received: from BL1P221CA0013.NAMP221.PROD.OUTLOOK.COM (2603:10b6:208:2c5::11) by PH7PR12MB6610.namprd12.prod.outlook.com (2603:10b6:510:212::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9094.16; Thu, 4 Sep 2025 04:20:59 +0000 Received: from BL6PEPF0001AB4D.namprd04.prod.outlook.com (2603:10b6:208:2c5:cafe::39) by BL1P221CA0013.outlook.office365.com (2603:10b6:208:2c5::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.17 via Frontend Transport; Thu, 4 Sep 2025 04:20:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BL6PEPF0001AB4D.mail.protection.outlook.com (10.167.242.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:20:59 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:20:58 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:20:54 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 18/19] sched/fair: Optimize global "nohz.nr_cpus" tracking Date: Thu, 4 Sep 2025 04:15:14 +0000 Message-ID: <20250904041516.3046-19-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB4D:EE_|PH7PR12MB6610:EE_ X-MS-Office365-Filtering-Correlation-Id: 7c05353d-f4e8-40b7-0d42-08ddeb6a737b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|30052699003|36860700013|1800799024|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?2VOnH9mwmPdHIqDXNxEhQXwZrzyCC4OSGcPlnwjsv94UTySd5LbU4FD0Rih7?= =?us-ascii?Q?y84mzmJthnM7jlGnzWkT4HUwQ3SjtBs2QclvmEVAsbkp+P1shzn17hP3ECZG?= =?us-ascii?Q?746WZRN5+wRr1e5wJj6JRgClWUBuB6y/UulIfmrvdRP4Ktw3e7TtZN22Q8ma?= =?us-ascii?Q?N0CypAscZQJUxCr6OCQDYNPtOXHm8pI7KZtdQvSsxryPHeOyWotPy3KDyjsC?= =?us-ascii?Q?irKZjJQLq957vmZvPWLfGhQ4x94G0oKepA+eOa523kgP73C+Cer70MHh6+ca?= =?us-ascii?Q?rBmey7DpT/1n+wKcAgjrVYMSLlJ0DXdJWr31SFSyi3gr8rCjPYwUJPbMkASj?= =?us-ascii?Q?UpZ1OBBriXaVEgdg/JVqfLKX0Vx1B+uQ+43uxMAjq2d0hiukOcsjXG4KqoIb?= =?us-ascii?Q?sL4QnqI214Q2g7kRL9CY2mSxHVR5BYR7ZeqjJT+05WppPSJpTB9nmyLY/5XK?= =?us-ascii?Q?X7z2fkoFTF72dVfcncr3NjoFU/xJi7OZ1gLZ9hLZ3A+d7bt1Iz+wzN5iIu/l?= =?us-ascii?Q?LZtyLhMFkPYbebYeNXHAXE47IB6RW3g2vBrh4OIDhVbchUDcB2RyzMjwRlzC?= =?us-ascii?Q?ysYR+roh27iybncACvTpzpE6FCQ1WPmiFjn3LqGlGOHp4uXSYbwKMoyICzqh?= =?us-ascii?Q?mIJJuiD8kqdLVCzJDHiYRnkrrZ8nP5JfTgLWPrrmnfLrHvIbWu8aqq+65Ypf?= =?us-ascii?Q?VCYzPjQ3qTBy2xf1kprssgLVELz1GN6/AESdTI9S1FmbRZozi4yLFqR4oMPn?= =?us-ascii?Q?Q8UkHEVhWKlRrzeRw+A/1rteVf/NyyOuhYiE5U+Ru6ATdxvkbTTQ2YrJVxbn?= =?us-ascii?Q?ogcVhK+M6q6lZsYwIA5li/Cxsqx9TSc6VxmSenGpIIkIF0XDTZpI1cNuDMJm?= =?us-ascii?Q?GB0TAqWDf3YJ+HWCfcLieqMnG1WRSdyhafbHFaC8YRPpqy8MWsM2M75E3z7R?= =?us-ascii?Q?VLLy+/U6HR22JG6vbjhu5jevoLCzSl9ueblKwQC7//rB8r+tdTz6c6M6x+VU?= =?us-ascii?Q?iomkzRlRUBl+JY6O/0WcctNlHmiDZcHSGupaT2ZpHM5ExgrPm8BERT6pQgTj?= =?us-ascii?Q?WZnXGmr5rbKH4tqaSpNnZ2MWFvCdDTFkzzvbuwqlx88vLOLM5WeBwxzhJgQ+?= =?us-ascii?Q?BQjtekcpV5vJhkvC6T5c4qnTcdzF/LFVTPzy+Q13XkH/G0cNrE2AIcqKF8Hr?= =?us-ascii?Q?WShD2/AjzAqW7ISLp1olgaD7OfuM8hi3e4X+WylkKpng+yJQ8fVpohzlIpT5?= =?us-ascii?Q?24GMDBB7R5a2jbNUTvAobTBowgrhRvaiiEjLhPySiW1OLln/hfVOicoOrLLC?= =?us-ascii?Q?hRBJpkIuIsm7f9zA7PPlB4ZC2uO7qjq8G30WYGz4sh/86nGk0LyRvPWxs60Z?= =?us-ascii?Q?GpaRWwT5AhjAbhZCL8MUgMCUg7NYz6ozoLp7SSwsnTng53TYSCr/JeBogxck?= =?us-ascii?Q?7XEOZn0SAd3QKXKBBDOScGCXtqUV7jMXjUXyUkLAZrMSGVsOLPjqwO52KKwi?= =?us-ascii?Q?Mmv42UI3yGWkWukDPktIX8I+eFlkM5hafQGl?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(30052699003)(36860700013)(1800799024)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:20:59.2547 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7c05353d-f4e8-40b7-0d42-08ddeb6a737b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB4D.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6610 Content-Type: text/plain; charset="utf-8" Optimize "nohz.nr_cpus" by tracking number of "sd_nohz->shared" with non-zero "nr_idle_cpus" count via "nohz.nr_doms" and only updating at the boundary of "sd_nohz->shared->nr_idle_cpus" going from 0 -> 1 and back from 1 -> 0. This also introduces a chance of double accounting when a nohz idle entry or the tick races with hotplug or cpuset as described in __nohz_exit_idle_tracking(). __nohz_exit_idle_tracking() called when the sched_domain_shared nodes tracking idle CPUs are freed is used to correct any potential double accounting which can unnecessarily trigger nohz idle balances even when all the CPUs have tick enabled. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 1 + kernel/sched/topology.c | 1 + 3 files changed, 58 insertions(+), 7 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 5b693bd0fab4..d65acf7ea12e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7169,7 +7169,7 @@ static DEFINE_PER_CPU(cpumask_var_t, should_we_balanc= e_tmpmask); #ifdef CONFIG_NO_HZ_COMMON =20 static struct { - atomic_t nr_cpus; + atomic_t nr_doms; int has_blocked; /* Idle CPUS has blocked load */ int needs_update; /* Newly idle CPUs need their next_balance collated */ unsigned long next_balance; /* in jiffy units */ @@ -12408,7 +12408,7 @@ static void nohz_balancer_kick(struct rq *rq) * None are in tickless mode and hence no need for NOHZ idle load * balancing: */ - if (likely(!atomic_read(&nohz.nr_cpus))) + if (likely(!atomic_read(&nohz.nr_doms))) return; =20 if (READ_ONCE(nohz.has_blocked) && @@ -12505,7 +12505,8 @@ static void set_cpu_sd_state_busy(int cpu) return; =20 cpumask_clear_cpu(cpu, sd->shared->idle_cpus_mask); - atomic_dec(&sd->shared->nr_idle_cpus); + if (!atomic_dec_return(&sd->shared->nr_idle_cpus)) + atomic_dec(&nohz.nr_doms); } =20 void nohz_balance_exit_idle(struct rq *rq) @@ -12516,7 +12517,6 @@ void nohz_balance_exit_idle(struct rq *rq) return; =20 WRITE_ONCE(rq->nohz_tick_stopped, 0); - atomic_dec(&nohz.nr_cpus); =20 set_cpu_sd_state_busy(rq->cpu); } @@ -12535,7 +12535,58 @@ static void set_cpu_sd_state_idle(int cpu) return; =20 cpumask_set_cpu(cpu, sd->shared->idle_cpus_mask); - atomic_inc(&sd->shared->nr_idle_cpus); + if (!atomic_fetch_inc(&sd->shared->nr_idle_cpus)) + atomic_inc(&nohz.nr_doms); +} + +/* + * Correct nohz.nr_doms if sd_nohz->shared was found to have non-zero + * nr_idle_cpus when freeing. No local references to sds remain at + * this point and the only reference possible via "nohz_shared_list" + * will be dropped after the grace period. + */ +void __nohz_exit_idle_tracking(struct sched_domain_shared *sds) +{ + + /* + * It is possible for a idle entry to race with sched domain rebuild like: + * + * CPU0 (hotplug) CPU1 (nohz idle) + * + * rq->offline(CPU1) + * set_cpu_sd_state_busy() + * rq->sd =3D sdd; # Processes IPI, re-enters nohz idle + * ... # For old sd_nohz + * ... atomic_fetch_inc(&sd_nohz->shared->nr_idle_cpus); + * ... atomic_inc(&nohz.nr_doms); # XXX: Accounted once + * update_top_cache_domains() + * rq->online(CPU1) + * # rq->nohz_tick_stopped is true + * set_cpu_sd_state_idle() + * # For new sd_nohz + * atomic_fetch_inc(&sd_nohz->shared->nr_idle_cpus); + * atomic_inc(&nohz.nr_doms); # XXX: Accounted twice + * ... + * + * "nohz.nr_doms" is used as an entry criteria in nohz_balancer_kick() + * and this double accounting can lead to wasted idle balancing + * triggers. Use this path to correct the accounting: + * + * # In sds_delayed_free() + * __nohz_exit_idle_tracking(sds) + * # sd->shared->nr_idle_cpus is !=3D 0 + * atomic_dec(&nohz.nr_doms); # XXX: Fixes nohz.nr_doms + */ + if (atomic_read(&sds->nr_idle_cpus)) { + /* + * Reset the "nr_idle_cpus" indicator to prevent + * existing readers from traversing the idle mask + * to reduce chances of traversing the same CPU + * twice. + */ + atomic_set(&sds->nr_idle_cpus, 0); + atomic_dec(&nohz.nr_doms); + } } =20 static void cpu_sd_exit_nohz_balance(struct rq *rq) @@ -12587,8 +12638,6 @@ void nohz_balance_enter_idle(int cpu) =20 WRITE_ONCE(rq->nohz_tick_stopped, 1); =20 - atomic_inc(&nohz.nr_cpus); - set_cpu_sd_state_idle(cpu); =20 /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9cffcfbef1ae..fcf4503caada 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -3100,6 +3100,7 @@ extern void cfs_bandwidth_usage_dec(void); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_nohz); extern struct list_head nohz_shared_list; =20 +extern void __nohz_exit_idle_tracking(struct sched_domain_shared *sds); extern void nohz_balance_exit_idle(struct rq *rq); #else /* !CONFIG_NO_HZ_COMMON: */ static inline void nohz_balance_exit_idle(struct rq *rq) { } diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 86e33ed07254..ee9eed8470ba 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -615,6 +615,7 @@ static int sds_delayed_free(struct sched_domain_shared = *sds) scoped_guard(raw_spinlock_irqsave, &nohz_shared_list_lock) list_del_rcu(&sds->nohz_list_node); =20 + __nohz_exit_idle_tracking(sds); call_rcu(&sds->rcu, destroy_sched_domain_shared_rcu); return 1; } --=20 2.34.1 From nobody Fri Oct 3 06:35:27 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2067.outbound.protection.outlook.com [40.107.237.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29D971B5EB5 for ; Thu, 4 Sep 2025 04:21:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.67 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959688; cv=fail; b=jHbT9PLS7zMzhg2XtYEZaJHVg8wDDpJKAvEbpLSbPM8zJvldZ+f6N5Y9DLekPse+C2rlS8RK/hOd0LMdcV+Mfed2bTsXSROyVnBrmb0w1PkxYfWYR/awPE/KxtvsBUFktfuTB0oFJwmNdYzY1rAeV2SWdj7MxVtjaNL4pGHXNYw= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756959688; c=relaxed/simple; bh=Bewtx85+0zjdl6Egzr/A2j83dx2EPEC6rSvrfNgC+4k=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=tObUloobTReh0DHtYQd9ofyMUq35hxAClvUxl2aNtUFojdHY8aPJ7YuljiGMA9gtoNlhMQfEy21xL/4viMXS48KZqz9Ce0t+D1inIVaOQBeSbidwzylHKPgWZo+Bak0Xzc4GcCF/bbwG0wPpPzdeM38hsgG1eSeesth+E1RRobk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=E/A2TRjW; arc=fail smtp.client-ip=40.107.237.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="E/A2TRjW" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=i1uRHUHEGwHag0eGh6O1kUewFcScY1VZwV2WjFCvGJKr0GOUxmPp07N+sAq4j2nicOt4jL/lBaSBCeZuMqgSQJ2D0toYyHm4XJlcSDO0cHQLZEwJheHgzd9+pT4t9k+WI+pAcCuyeHifKePb2ctRVm28HBcS8LBOOk0UyEzO5J+59djB1SZUAU/2X9igP0z0nZ/DC6FaGBH/QFj5AuxjdxvPJxc6Psl35qsddbyZ1q6nSXiuLfdn7sx2uuYSogzQ7qEQNjTJvOwva6iXCYMMpGWvkxq3N5im77xqw7OwecKjJms9ydjI0yeD5WY8hs2HBbFKYaWfb3GUK0T/CnXb9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zt/hmsChhZytJbqIcMuGUymGW7RVj96CW6rPAK1mA4w=; b=NAkTIyrtOjLnByltu5S64oix7g7EeXmM13gA8QnfBbKP5xX5y2MXceSsOgkmJi3hxHwnKAbuqxqP6/i/EJU7HSz4p30/QxNxlBphUF6DdTckW7WeIVV5QGDU4P4Zoa2NwgcxQM8QjD9JaOy1uOGqcDegBTFAGl/YIWpGeobnYh6oimBuhRR4Y+dgcc/QOfsdJXXNEIS5Dlj/ztDp+DvzeEI74UA5Jxg84w7YmPIi2tzCg5NzRqOI/OVGyxn3F+CO16QdD3yx9YclwYRnAn8u/KfMIAhgMyRvZ/QrjnKm5rtIyidAEScvMdoOBXWV6GmbDLqAcSDGMcUiOnNol+875Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zt/hmsChhZytJbqIcMuGUymGW7RVj96CW6rPAK1mA4w=; b=E/A2TRjWLHgv4xj1wh4uclivkULD18D45eO/XbnBFsTG55ppFzpdZxRByxvVu973c8Cp/pLfXrvEcPiuOWSFOzmaIX55+lB27f0WW4J9HziDk9EAM0Uw7vwl4PSkpeH0DhhnzqbFTSHTg7kfVs0m8t9OhVvF9adjOlCcId4z5vI= Received: from SJ0PR13CA0051.namprd13.prod.outlook.com (2603:10b6:a03:2c2::26) by MW6PR12MB8952.namprd12.prod.outlook.com (2603:10b6:303:246::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9052.29; Thu, 4 Sep 2025 04:21:22 +0000 Received: from SA2PEPF000015C6.namprd03.prod.outlook.com (2603:10b6:a03:2c2:cafe::aa) by SJ0PR13CA0051.outlook.office365.com (2603:10b6:a03:2c2::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9094.16 via Frontend Transport; Thu, 4 Sep 2025 04:21:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB03.amd.com; pr=C Received: from SATLEXMB03.amd.com (165.204.84.17) by SA2PEPF000015C6.mail.protection.outlook.com (10.167.241.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.9094.14 via Frontend Transport; Thu, 4 Sep 2025 04:21:21 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by SATLEXMB03.amd.com (10.181.40.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 3 Sep 2025 23:21:21 -0500 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.1748.10; Wed, 3 Sep 2025 21:21:16 -0700 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner , CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 19/19] sched/topology: Add basic debug information for "nohz_shared_list" Date: Thu, 4 Sep 2025 04:15:15 +0000 Message-ID: <20250904041516.3046-20-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250904041516.3046-1-kprateek.nayak@amd.com> References: <20250904041516.3046-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To satlexmb09.amd.com (10.181.42.218) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF000015C6:EE_|MW6PR12MB8952:EE_ X-MS-Office365-Filtering-Correlation-Id: 4219fe04-eb74-40ac-0e69-08ddeb6a80d9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|376014|7416014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?deQX2YvzHrGm/jswrAGEl9pTAkTBEQow87XnL/4W9/2pL85ld620GsQHyC4g?= =?us-ascii?Q?t4BRvRjtY9LrNjnHk6cCQWOvzPRzYZrW3QWGHt8pkC4xqBisMMJ3cdQ4szyt?= =?us-ascii?Q?iozTF5QZoSAyTnBlo+zYDaNrF2OOGkcz/9X9xtZzfpWKcH6royHd2aL/FekZ?= =?us-ascii?Q?Ow8wzlhedSyXZW2X+vxoRPeektDzfF6pWsnZe6rB4f3SpUIl9Bn0cFLcyg84?= =?us-ascii?Q?petaNu6iI0DzuzDyYgT+u3GTXAeW22rpmXUc9bKMkDtKC+uxZIknX9TX+cU9?= =?us-ascii?Q?XMHOV/BDiS0L+NX7csEI7PhmWAmsZGPiW4W140YP+/h5zkMoxlgaJlozwvMU?= =?us-ascii?Q?Bjxh/j76UOai6fjclXLgo1VoZmd64cFRPRQJBvJjCXJ22gtaSfp+ZQhR2G4R?= =?us-ascii?Q?DcIN3JWJL0bYyFQn5P8e3etsmVeRuE0l/FvOg+yHr/e29Hc5AdIqTftuBvmk?= =?us-ascii?Q?3LwMSdzCgvdJwHMCeRFK9YOPlh6x41vmzCNtWJad01iS5vHckftZrDKiyVBt?= =?us-ascii?Q?Fe4hCOJLwSqriw6gD/nSosO/wJzl91CrYWIBuOK1KqigHIfHaq33X+8X3Put?= =?us-ascii?Q?Vlf08ZVrDK95ZTSi2N9V4B60E1biU9xRwMKMDVtXPW2tkz3QLoQPvqg71O+D?= =?us-ascii?Q?pGQ622hQ299nNqadjHT2fF07RgP90D41Gk1y8Livu8xOIoMbNkDIuZSuN1hl?= =?us-ascii?Q?7Epjo6Yewun/hhPeHL4Y/wqYuaHvHm7/dVMlg8JmSZs7nzD3bUnprtxXn7/q?= =?us-ascii?Q?i+QUWdzkEq5s/+dHbnXw62cWHP2zmYuJGOAmpZEIKVWlN29DvTG5lIYNPJb7?= =?us-ascii?Q?g+9O4Ekt5Z5CchotFbv5QozBzulSZ+jk7Puafhqq95UfpVyYt497bi30LmP1?= =?us-ascii?Q?ElECoZUeoasNgsZpD/SHkS8Rht4lc7Yu5LyK/y5Uze06KhibsWIoAMsQ0L+V?= =?us-ascii?Q?6g49X/ygEB8d80bVbJiKdI552m0CDXGE4PL6+Ww8wRFi0tOCyc6TqSSSe5Hn?= =?us-ascii?Q?LOv2/zTMfw/2YqvfYhQFUxukqU5RI7Yo2ln4ZNdTS+Oh/ra67qe3IMIDVVqs?= =?us-ascii?Q?MXZIKjQ1RR34goqfl58XWoamqDbByXMBvRXiXFdQMNyPnLnY+fuC8m4+f43v?= =?us-ascii?Q?eGc+2NcN/oLrLa1/SdTu0fPPe6XU+fQwwVZK2BT0CpHRX6JDkCbnR8q+9kUX?= =?us-ascii?Q?Es21Z/qJYp3XZOeVAsU46ObWDQTt1/eV0tNnGl5JQVzVfGsp6qLyl/kY/ld2?= =?us-ascii?Q?q6uh8NNsueNZZEQx7t+Ci1XzILbFlObrdfIe511TFFOzR7kw28MoXNdKigql?= =?us-ascii?Q?pdEg7B01n6o8Q4j5m04ZRXp0Wa5XtwUGRefXo4jj/EkgCphHPhTH5NnCmYnJ?= =?us-ascii?Q?O1uemCmhpBCLTX2vqJO8hqYo+Jqx4uVqieOrZBA5r2ZjBB3TVkkXpe+WTlsr?= =?us-ascii?Q?QB4cUIiCuyWtVR336TRLXgvtr6TQNOgTDq0wSsYJhzAnpaB6KAyn3ih2C/Nv?= =?us-ascii?Q?2Lx9W1eis78TQcZa1jZICd3PhJ6fJ7Vkwpn4?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB03.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(376014)(7416014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Sep 2025 04:21:21.6499 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4219fe04-eb74-40ac-0e69-08ddeb6a80d9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB03.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF000015C6.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB8952 Content-Type: text/plain; charset="utf-8" Introduce debug_nohz_shared_list_update() to count the number of entries in "nohz_shared_list" after each list modification. XXX: There isn't a great way to jump from a sched_domain_shared object to the sched_domain struct that references it which prevents printing more information about the sched domain that was linked with the shared object. Signed-off-by: K Prateek Nayak --- Note: Something in this patch upsets the LOCKDEP + sched_verbose cmdline combo and hangs my system soon into boot. LOCKDEP alone or sched_verbose cmdline alone doesn't have this effect and until Patch 18 even the offending combo runs fine. DEBUG_LIST didn't point to anything and I'm still scrathing my head on why this may be the case (maybe I need some glasses). --- kernel/sched/topology.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index ee9eed8470ba..85b46ba594a3 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -471,6 +471,20 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_nohz); static DEFINE_RAW_SPINLOCK(nohz_shared_list_lock); LIST_HEAD(nohz_shared_list); =20 +static void debug_nohz_shared_list_update(void) +{ + struct sched_domain_shared *sds; + int count =3D 0; + + if (!sched_debug()) + return; + + list_for_each_entry(sds, &nohz_shared_list, nohz_list_node) + count++; + + pr_info("%s: %d nohz_shared_list entries found.\n", __func__, count); +} + static int __sds_nohz_idle_alloc_init(struct sched_domain_shared *sds, int= node) { sds->nohz_list_node =3D (struct list_head)LIST_HEAD_INIT(sds->nohz_list_n= ode); @@ -588,6 +602,7 @@ static void update_nohz_domain(int cpu) =20 guard(raw_spinlock)(&nohz_shared_list_lock); list_add_rcu(&sds->nohz_list_node, &nohz_shared_list); + debug_nohz_shared_list_update(); } =20 WARN_ON_ONCE(sd && !sds); @@ -612,8 +627,10 @@ static int sds_delayed_free(struct sched_domain_shared= *sds) if (list_empty(&sds->nohz_list_node)) return 0; =20 - scoped_guard(raw_spinlock_irqsave, &nohz_shared_list_lock) + scoped_guard(raw_spinlock_irqsave, &nohz_shared_list_lock) { list_del_rcu(&sds->nohz_list_node); + debug_nohz_shared_list_update(); + } =20 __nohz_exit_idle_tracking(sds); call_rcu(&sds->rcu, destroy_sched_domain_shared_rcu); --=20 2.34.1