From nobody Fri Dec 19 13:09:44 2025 Received: from DM5PR21CU001.outbound.protection.outlook.com (mail-centralusazon11011037.outbound.protection.outlook.com [52.101.62.37]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 788EC329C7A for ; Mon, 8 Dec 2025 09:33:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.62.37 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765186406; cv=fail; b=B/CuccDBGmKdgKgURcgY+6MaXvvXsmPZlvcr6MV2CrfAdWzTVbkfIWwcQRwAIH78Oj9eWeO9EIa8Y1bDG/OsoKNyXYN9l0Ma4Uedu7lBK1dnUeP2yVS4aJW90lFTxz1R2qAYvl2rXJRvg0IiK3htAVG9sV3gB0NIX13k/S3b8DI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765186406; c=relaxed/simple; bh=zue4ToYN19MsjuVpmShe2C3de7WnA5UiINr7sNjhFQo=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Xv9zSum8sWG205zQYzCkXOLxH51Lj0W1QM6vz6PUuhXfWdBbYbOFykNIfynJPIWHPswU1vbB0Gkn6gC+L+TxPFm1PI1ywuZziG7qSExeJhcgW1jcP0pp4wsmVnNDuWjVT94tsqu+RlWVQ2LDOVtNMR9DeoMqG1I/VHslFxdV86s= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=b8iSzcfv; arc=fail smtp.client-ip=52.101.62.37 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="b8iSzcfv" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KbUPZJm1q2vjCyFSdk6cBDJ0dcq2aO1bvWUCpNMfPaGqscr68guzNFcZ/jUioCJ7PVdXenKZIcWqDkpotdArAVQI3HfINRXwwxP7hUvmqzsTqAt8BBrayi5AivMzRToQJV8KORX8rI+aHaIXUIXAZuHsYLWf5QBVAt5A0qy6AXQhBXYcaMfyQwfR6prexBe16m+FcPLPbK4NY7J7JIhE43R6hg+NRTCwMj2atbWvr1ZAaGBzXvQR/KQOSE6t7gbC8OpgVr0edVvWTKBd8Kg19qdXTqe360vwdxfghVGhNj2LGvcO4GS7a/uMMtdxxjuqBCK02JtuwTLFe0us0D2dIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UpHwopGmPFtIgk/lzd/wgJFYTKkNfz53eyPZhhicuy8=; b=ASNH6ScE6qMHJzScPMsLSLrXafXW18P//CsvNAvkv30NCB1tBwceFZnNqewdyiqvZaX+Jy6RtNFMvz3m+EgyoX4+CnMRDp5ATSYnZfUvYBb69MNBLZKOQFqWXoTmZ+XoKW4cGjzc6/eiD0vtyGaGJFKlMkSpOFxG7ipVqtApZZvgo3gcd6UH+kBmYiSgFKandhHxQ48lpV0o2/GPCBq24byNLjS1rBgh5v7QSoppBOhjhGJ3I3kh3OCn0Ma/GLBd4zjmAZsOtABYqTmoQ4SKrcvHrvZMWF2l0SzVEO8r07B7YZ/SgqY6CFnDAsT4U9z/QaPloG71kEvx7IhW+gTBYQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UpHwopGmPFtIgk/lzd/wgJFYTKkNfz53eyPZhhicuy8=; b=b8iSzcfvcm5iAvff5HmwhchkxkSaV2qjbbjT/eJ62Idkdkmvsy1Is5ppPwU8a4DjK8cwPBHV1CpR+N9k3cz8rE7YUrU0xOpyi0Z9+9eydCbuZdxblw+zkMMqseNP3+SnvwUN5Hx0phtTHZUo2Op7zSVSfbTFVrg6Qs/Bb+k+zXE= Received: from DM6PR11CA0040.namprd11.prod.outlook.com (2603:10b6:5:14c::17) by SJ2PR12MB8781.namprd12.prod.outlook.com (2603:10b6:a03:4d0::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9388.14; Mon, 8 Dec 2025 09:33:19 +0000 Received: from DS3PEPF0000C37C.namprd04.prod.outlook.com (2603:10b6:5:14c:cafe::12) by DM6PR11CA0040.outlook.office365.com (2603:10b6:5:14c::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9388.14 via Frontend Transport; Mon, 8 Dec 2025 09:33:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by DS3PEPF0000C37C.mail.protection.outlook.com (10.167.23.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9412.4 via Frontend Transport; Mon, 8 Dec 2025 09:33:19 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Mon, 8 Dec 2025 03:33:13 -0600 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Anna-Maria Behnsen , Frederic Weisbecker , Thomas Gleixner CC: , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal , Shrikanth Hegde , Chen Yu Subject: [RESEND RFC PATCH v2 19/29] sched/fair: Extract the main _nohz_idle_balance() loop into a helper Date: Mon, 8 Dec 2025 09:27:05 +0000 Message-ID: <20251208092744.32737-19-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20251208083602.31898-1-kprateek.nayak@amd.com> References: <20251208083602.31898-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37C:EE_|SJ2PR12MB8781:EE_ X-MS-Office365-Filtering-Correlation-Id: c88f34fe-c9d7-49ff-dfaf-08de363cd2a4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|30052699003|36860700013|1800799024|82310400026|376014|7416014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vqPliZAo9vbJ9tQhnCZOePTwZrytwJ+OTWiVHthR1Jap/g0967zEJgSbp0Rb?= =?us-ascii?Q?KDAfEnqJVfyG53XrAOXiOxVK1LdrJI12a6iaYICW94hHSawiiyqQG2IYiTyg?= =?us-ascii?Q?XlIotmT5OGBoeVwys/NEcEQ7PwDwZ/pWOU1oq49hGgKpQczz40vZniQCNDhk?= =?us-ascii?Q?fQKlIAhmjRC0j/hJi0JsbXb8LM9fwPMhRNUBQUUGX0mjzGatZ+smMejOE7+9?= =?us-ascii?Q?w+b6ToQ7R7hky+bqHXIVqNqVfAduHZ7MAbxOzKqRCGiNbqXCjCGBdIf/Ll3L?= =?us-ascii?Q?WBtmkr/VfEqWkNBtdzOPvEWX2aXgPer468JDTwnVaW1r4Lt+rndNa2IZL86J?= =?us-ascii?Q?PFFccSqbYx1L7aH69q5BC56Wqbf69Cv9YYFZI2iW4OX2wMlfJPwyl7JWY8YH?= =?us-ascii?Q?G7E/RIyExM97IXoJuO8ftqwxoPrmpZdZSclG7ewouXgQvPAxowC68wDFrhqA?= =?us-ascii?Q?h4PRX9QaaJ1j7KeO66iKBOzIt6AGtR7Nz8qQ+9tF5NPjhNMU4UDN1dBkfwVc?= =?us-ascii?Q?PN0q47L62n8koshqfBobklP11lLjKyA0x4jSHWGo32cw2NZVJhX8vtiY9nBX?= =?us-ascii?Q?YDRI7KTul51UqSNhbNquaddYUfSRL5Cbi38zEAbf1cKd5BDpVFISwD/S5gl8?= =?us-ascii?Q?7/Mwl9EexIvwDBrGWavi7JAlga01bX4t22wWECCXq6rRw77RfOPzB/X0QynX?= =?us-ascii?Q?RzP8r2piWn/cY/Tic1M9EQrxfX+INB7/WFNYMzNdwXZKpYmLu3sezqEieNYV?= =?us-ascii?Q?+dzDRBim927L5qLr110MUQwMu+3CH8rsHOmSSi+sxDykCwbCWQ52MWBLsSLl?= =?us-ascii?Q?AHOrSHGuylCCO43HEEXN/LmDOPTbGL7ZNSG0irQbqFKT/BnfDCFrVMpPA+yd?= =?us-ascii?Q?OGzd5Xaxv4jf5ivUSrnuJV/5B/xzJrwHG7feRJgbdjoY+o4nrgObQQx525Xd?= =?us-ascii?Q?TJatLJtASH2JG2lZ+v+JVqC2iCtFGXyYfXrr+ACxZEfiOj2UUte6RjMcmRBa?= =?us-ascii?Q?busUBcZWu7PTaOYn8b04JvlVLGl/RLiNbKdovDXjG0C2/1RzTXyvPJtsFw7U?= =?us-ascii?Q?1AjY/sNQYVG6Xw8gW3AHldpS2eWXBfmPD5UIXsx9auYPAR4tvnqnZ0Cprh2s?= =?us-ascii?Q?SQZvJDzFWUUwR8V7cT7Gu6vXk8YLQtEi0RhiTQCnEts67+RERWrTRXa/mOiW?= =?us-ascii?Q?7dqa8+yFJq1UnRBEG6ZR/pux/o9FZPDTRvjOczS4RNC4ttjpLHzZErZ4TDiI?= =?us-ascii?Q?AhC9ym1stfbGAHVzncCLeXAT+btO3I2VkpptTLgiB7VSeMsfRXq4B2LZ6QmN?= =?us-ascii?Q?/4Jxtmi7wEfyo/N9bVRPXnn0qg6N0UdJIV/3JDyciiW7cT6QHB2gEhGrcboo?= =?us-ascii?Q?UEtuAvbLNHRKfFNw/pXq8aZSfdVPbD365FwNq8HrKp2B5kkd7Loj50p5s50Q?= =?us-ascii?Q?QXuyNDK/13N4n+fnapj3t4a5gVsmr35Vmy4uXG7KbtwoY8h1kcDWpM7yMCsu?= =?us-ascii?Q?foU3z0PFyByKxrdBUAmnMTeojnq5mKnMHwqn0uwUXdGOC9rIdSnP1/ns0cbR?= =?us-ascii?Q?2IBqfvfPvdd8y08+jMs=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(30052699003)(36860700013)(1800799024)(82310400026)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Dec 2025 09:33:19.2346 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c88f34fe-c9d7-49ff-dfaf-08de363cd2a4 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37C.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8781 Content-Type: text/plain; charset="utf-8" With the distributed nohz idle CPU tracking using the "nohz_shared_list", the trick to do the load balancing for the CPU in charge of the nohz idle balance at the very end using for_each_cpu_warp() will no longer work. Extract the main loop and the loop body doing the load balancing on behalf of an idle target into two separate helpers - sched_balance_nohz_idle() and sched_balance_idle_rq() respectively keeping consistent with the naming convention in fair.c. This will help transitioning _nohz_idle_balance() to use the "nohz_shared_list" in the subsequent commit. No functional changes intended. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 161 ++++++++++++++++++++++++++++---------------- 1 file changed, 102 insertions(+), 59 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 5e5db24e1384..5bbd56161838 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12684,6 +12684,97 @@ static bool update_nohz_stats(struct rq *rq) return rq->has_blocked_load; } =20 +/* + * sched_balance_idle_rq(): Balance a target idle rq. + * @rq: rq of the idle CPU that may require stats update or balancing. + * @flags: nohz flags of the balancing CPU. + * @next_balance: Pointer to variable storing the time in jiffies to + * trigger the next nohz idle balancing. + * + * Returns: Update flags for caller to take appropriate actions. + * NOHZ_STATS_KICK indicates rq still has blocked load after the update. + * NOHZ_NEXT_KICK indicates that *next_balance was updated to rq->next_bal= ance. + */ +static unsigned int sched_balance_idle_rq(struct rq *rq, + unsigned int flags, + unsigned long *next_balance) +{ + unsigned int update_flags =3D 0; + + /* If rq has blocked load, indicate via NOHZ_STATS_KICK. */ + if ((flags & NOHZ_STATS_KICK) && update_nohz_stats(rq)) + update_flags |=3D NOHZ_STATS_KICK; + + /* If time for next balance is due, do the balance. */ + if (time_after_eq(jiffies, rq->next_balance)) { + struct rq_flags rf; + + rq_lock_irqsave(rq, &rf); + update_rq_clock(rq); + rq_unlock_irqrestore(rq, &rf); + + if (flags & NOHZ_BALANCE_KICK) + sched_balance_domains(rq, CPU_IDLE); + } + + /* Indicate update to next_balance via NOHZ_NEXT_KICK. */ + if (time_after(*next_balance, rq->next_balance)) { + *next_balance =3D rq->next_balance; + update_flags |=3D NOHZ_NEXT_KICK; + } + + return update_flags; +} + +/* + * sched_balance_nohz_idle(): Core nohz idle balancing loop. + * @balancing_cpu: CPU doing the balancing on behalf of all nohz idle CPUs. + * @flags: nohz flags of the balancing CPU. + * @start: Time in jiffies when nohz indicators were cleared. + * + * Returns: + * < 0 - The balancing CPU turned busy while balancing. + * 0 - All CPUs were balanced; No blocked load if @flags had NOHZ_STATS_= KICK. + * > 0 - One or more idle CPUs still have blocked load if @flags had + * NOHZ_STATS_KICK. + */ +static int sched_balance_nohz_idle(int balancing_cpu, unsigned int flags, = unsigned long start) +{ + /* Earliest time when we have to do rebalance again */ + unsigned long next_balance =3D start + 60*HZ; + unsigned int update_flags =3D 0; + int target_cpu; + + /* + * Start with the next CPU after the balancing CPU so we will end with + * balancing CPU and let a chance for other idle cpu to pull load. + */ + for_each_cpu_wrap(target_cpu, nohz.idle_cpus_mask, balancing_cpu + 1) { + if (!idle_cpu(target_cpu)) + continue; + + /* + * If balancing CPU gets work to do, stop the load balancing + * work being done for other CPUs. Next load balancing owner + * will pick it up. + */ + if (!idle_cpu(balancing_cpu) && need_resched()) + return -EBUSY; + + update_flags |=3D sched_balance_idle_rq(cpu_rq(target_cpu), flags, &next= _balance); + } + + /* + * next_balance will be updated only when there is a need. + * When the CPU is attached to null domain for ex, it will not be + * updated. + */ + if (likely(update_flags & NOHZ_NEXT_KICK)) + nohz.next_balance =3D next_balance; + + return update_flags & NOHZ_STATS_KICK; +} + /* * Internal function that runs load balance for all idle CPUs. The load ba= lance * can be a simple update of blocked load or a complete load balance with @@ -12691,14 +12782,8 @@ static bool update_nohz_stats(struct rq *rq) */ static void _nohz_idle_balance(struct rq *this_rq, unsigned int flags) { - /* Earliest time when we have to do rebalance again */ unsigned long now =3D jiffies; - unsigned long next_balance =3D now + 60*HZ; - bool has_blocked_load =3D false; - int update_next_balance =3D 0; - int this_cpu =3D this_rq->cpu; - int balance_cpu; - struct rq *rq; + int ret; =20 WARN_ON_ONCE((flags & NOHZ_KICK_MASK) =3D=3D NOHZ_BALANCE_KICK); =20 @@ -12723,60 +12808,18 @@ static void _nohz_idle_balance(struct rq *this_rq= , unsigned int flags) */ smp_mb(); =20 - /* - * Start with the next CPU after this_cpu so we will end with this_cpu an= d let a - * chance for other idle cpu to pull load. - */ - for_each_cpu_wrap(balance_cpu, nohz.idle_cpus_mask, this_cpu+1) { - if (!idle_cpu(balance_cpu)) - continue; - - /* - * If this CPU gets work to do, stop the load balancing - * work being done for other CPUs. Next load - * balancing owner will pick it up. - */ - if (!idle_cpu(this_cpu) && need_resched()) { - if (flags & NOHZ_STATS_KICK) - has_blocked_load =3D true; - if (flags & NOHZ_NEXT_KICK) - WRITE_ONCE(nohz.needs_update, 1); - goto abort; - } - - rq =3D cpu_rq(balance_cpu); - - if (flags & NOHZ_STATS_KICK) - has_blocked_load |=3D update_nohz_stats(rq); - - /* - * If time for next balance is due, - * do the balance. - */ - if (time_after_eq(jiffies, rq->next_balance)) { - struct rq_flags rf; - - rq_lock_irqsave(rq, &rf); - update_rq_clock(rq); - rq_unlock_irqrestore(rq, &rf); - - if (flags & NOHZ_BALANCE_KICK) - sched_balance_domains(rq, CPU_IDLE); - } - - if (time_after(next_balance, rq->next_balance)) { - next_balance =3D rq->next_balance; - update_next_balance =3D 1; - } - } + ret =3D sched_balance_nohz_idle(cpu_of(this_rq), flags, now); =20 /* - * next_balance will be updated only when there is a need. - * When the CPU is attached to null domain for ex, it will not be - * updated. + * The balancing CPU turned busy. Set nohz.{needs_update,has_blocked} + * indicators to ensure next CPU observing them triggers nohz idle + * balance again. */ - if (likely(update_next_balance)) - nohz.next_balance =3D next_balance; + if (ret < 0) { + if (flags & NOHZ_NEXT_KICK) + WRITE_ONCE(nohz.needs_update, 1); + goto abort; + } =20 if (flags & NOHZ_STATS_KICK) WRITE_ONCE(nohz.next_blocked, @@ -12784,7 +12827,7 @@ static void _nohz_idle_balance(struct rq *this_rq, = unsigned int flags) =20 abort: /* There is still blocked load, enable periodic update */ - if (has_blocked_load) + if (ret) WRITE_ONCE(nohz.has_blocked, 1); } =20 --=20 2.43.0