From nobody Fri Dec 19 04:18:28 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2043.outbound.protection.outlook.com [40.107.244.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A3CED1EBFFD for ; Thu, 20 Feb 2025 09:33:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.43 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044030; cv=fail; b=bKkWFXVGiwfFkXcB7LoL/QAUb8Fj9SpRwlh6BU2Fpd8Y8u8YgxyinzFmgYQ+3XXYS1Nuy4HGOSP8I4LRaoTbSXCQUAXPnzBVDfb1BMa04oLyepx83goDmK9A0bxLuU4lnZlhpd8DcjZaiLa9zaEABqataWyd4Aal5AIYpgrDNuM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044030; c=relaxed/simple; bh=EY+ZPV2UDAjmwvuZF/12xh8zj5ylqkGD63zKcIAWjLg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=GKtRg5dbEYFdipcbL1NifrfnB1oz9kcNw6mO0hExvRCgiKRo6BQpc91DuMmo/3EosRwmkvoIFEn6Nd0rpbavQYiyMv92NQe4MLLbb/Wi3U8aaYgbmhXQpFXiwO4GZm5GwH0LAlR1IMFL2QjBOxDkKOGRGB7yeMgUFhyuROmpj90= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=XVig+p7w; arc=fail smtp.client-ip=40.107.244.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="XVig+p7w" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=zSnU1FnvCkq+CO3hWBmKiZgamGhlL96SaDwTAEXiQK5hTXeKKOOvqXFF9iTQfkChiveef8Bd86/7nCiHGwBr/xmHCpT0eWAcw/ZpcvnS94bU0n2zArz77sqoJhpzCu/wFdOnUN8SDU55ElqpH/kuoyX2JbTql6lftKBKqdNdRt0emNNMlDtDt4V9IeUqzERh51/JmdruK0juIMkgZweiiCL0A8oYcTPljud+/kyvCe4tCdrFEgkbQAl4gMQM7LA5s5QZqXLJPx7wMsiLo1QcBlpVKAVqaszHx1pMQWksvJva97BVUW8nGtoNgksi42iB3tRhJfdCzRJU8MAi+z2Axg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/mgLUF5MCAG2c+vMAAZrM47apPNBTI4L3mAjcPGfHXY=; b=XREX+YGBIbI6YWyY6Okav8xF4Fb+d7QhSSRwRW+ntSJHHqAA15vnNdKK95PgPlnFNY5hGLkg+pDpU31C/L+xOmm0XuzsauIEBVJ8GfULEZpEYxjCRUnTdgDT0bYY+/nei/HRehb9etXRVRgaJIwBAihYVBOZeAH2I2PYJDjIGRn21uUgP1sD6uEnr2cZhcHvfSM9TnWanrKH2i6brfoWFSaQ6J9xgXKWCTmWzwCcErRslekD+Ia1PJtDOgcRuyxYH83bydKpc633MeWphlCmd19dx/3+BkxKeztP0lH7MghJBAb6TH4jGVOJV4tw9OjnwEDjJspfoLKdthqnRY61DA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/mgLUF5MCAG2c+vMAAZrM47apPNBTI4L3mAjcPGfHXY=; b=XVig+p7wqYsQkZHdKFi/p4X/AcUqWbSwoTMrdMfu5PXI4+6NrpE3z5w3OncMkCZDn+xfSnwUa0Fy0+DUeyybIHcoIEcixhFo8zSMWwy2MwW/rcQcLMVuGUh+kddmx/YkpiEJKah7jSEAOQkOGVsT8yllSmxByAFQ8PREGeVsbEI= Received: from DM6PR11CA0006.namprd11.prod.outlook.com (2603:10b6:5:190::19) by SJ2PR12MB7990.namprd12.prod.outlook.com (2603:10b6:a03:4c3::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.17; Thu, 20 Feb 2025 09:33:45 +0000 Received: from DS3PEPF000099DC.namprd04.prod.outlook.com (2603:10b6:5:190:cafe::97) by DM6PR11CA0006.outlook.office365.com (2603:10b6:5:190::19) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.15 via Frontend Transport; Thu, 20 Feb 2025 09:33:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DC.mail.protection.outlook.com (10.167.17.198) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:33:45 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:33:36 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 01/22] kernel/entry/common: Move syscall_enter_from_user_mode_work() out of header Date: Thu, 20 Feb 2025 09:32:36 +0000 Message-ID: <20250220093257.9380-2-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DC:EE_|SJ2PR12MB7990:EE_ X-MS-Office365-Filtering-Correlation-Id: 33ee9450-b483-437f-d1ef-08dd5191abdb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?/KHhypdHr4SdziSCL8oBmv57iljV3mnTOTSzhiw44T3EtqlPggjXF3Q8qoyk?= =?us-ascii?Q?ifU8K3eJB23BaORTum4xQ59NcePlcBSqSq1aRwSvFnb70GDU+yiIa0nmhzFv?= =?us-ascii?Q?Y2f/JRUeByBvmhZnSye2fjEAV8UhAqMMoLXsniPLoowwCjHFjg1oGrEFottK?= =?us-ascii?Q?3vl/iBqMjPcocusXgvvoS1C3oKAItQlBdxylamuQklq9OY4JIccCposOMhD8?= =?us-ascii?Q?UCnSqYvuJxyGHAtrYwz5AFgokFVjf6aP4hIAnLwsHiCAI5COFOeC9oN+ao5O?= =?us-ascii?Q?l4KNNx7mXyLlItNXlvQ1btJ/qyvNoclLz/UNjtuY4WbOYMfMGyQCb2w7VKmh?= =?us-ascii?Q?kIrVXHMLw3Bt0VgBCLOTnY2g1IFcSTOvdaMt1IANskHyXmHFUUHuQD1mXy0t?= =?us-ascii?Q?Ywpa83gt2SrutpkXvOIRupOVY9G+OSK4DF7Vo7/u4jSuLwWxc9yuQqL5kvue?= =?us-ascii?Q?2d4L+kPrEkdnUMbPRpeA14RjQ/RBeH0AHJl+IfLY9mfQK1uu+U0Yzi9ipR+5?= =?us-ascii?Q?D04iyD0ps6WNvxGrI5nImfvpMJxCO8yB0R8fucdOdh7qQkDcadC5lASlO7IT?= =?us-ascii?Q?1VwQ1yejNuHl6eN1PWG6mUsaU86dZW05R/V/h0hDotp1/stNGXYg34bdgEag?= =?us-ascii?Q?fAPg0ihrElEhb9M9ORvpTcz3FouOqAz1VeV/Zph3DqjlATS4B8FLW2+X6aAv?= =?us-ascii?Q?dfHv/mc0RPGEDRW0UphDyUBW+xhgBqD37613HH4ZJPDN6EPsa4T5SGVQEiOy?= =?us-ascii?Q?M870AJ/TcAlU5x2ZLdO80kqK20LKnDDLwmDZPnO9tM0RymALhcPVd10zBCwN?= =?us-ascii?Q?cjkB8IP7nG/Y3DS2MHTF1aAX+MBIHFcqfVChj89ThHIcJNKrntno3XhmlMOC?= =?us-ascii?Q?iWs4BgtLFml6ptOj6S6SJA+fpNqFREquIrbPfIK+LcOY+viA6NiedL4QAaxJ?= =?us-ascii?Q?v8LmHL0DQGg8tksRiKNEXuzCw2E18DgufN8c+7Ro/ANzX15xw4fk68S0TWKM?= =?us-ascii?Q?xwanB7DoSaNA9AuvAWK8J0vwaOKLJxCwpYuQ7XBdMdnA3KOUkK4j6lqwMHoA?= =?us-ascii?Q?DFsxrRW0cwZXY7cAU0L9lDz3EHeuqwBC5NYVgpJQAduxwii+GgW/kYmr+LSp?= =?us-ascii?Q?QsgyaYWOGwTW+OvkKsPx58zkdGWKbxrv3qKKDt7RnmIKiegTKiPAsCeXkRbZ?= =?us-ascii?Q?RP68mL/nb8byMqCTpMMZIYBedtlrcBn85nwSg1KS17LnC66OFV/enzMxwkhm?= =?us-ascii?Q?yNRIJQWoBpsMNqNG5lYZmxp4ThzVuzdbCZ9FyVcx8z+SHSjdRVcxi8kBqFBt?= =?us-ascii?Q?lpoUudruV4RokZc+DGjFyZU05DeE6FIT9IHUt5CNZm5p1m5R9A5wf/65HUhI?= =?us-ascii?Q?p9gD2waYmZdBOhB5AdXgby22GIR6LDW6CGEm9cPa0v5/fMvusPI7VrL7VzxY?= =?us-ascii?Q?m8/FpJrE3gKfNjeaLQjJpyJCFguXLIZNbkR22S1kuoHf7WV8hZ8I432ExEVQ?= =?us-ascii?Q?J+zchvXQoi2Plrs=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:33:45.0802 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 33ee9450-b483-437f-d1ef-08dd5191abdb X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DC.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB7990 Content-Type: text/plain; charset="utf-8" Retain the prototype of syscall_enter_from_user_mode_work() in linux/entry-common.h and move the function definition to kernel/entry/common.c in preparation to notify the scheduler of task entering and exiting kernel mode for syscall. The two architectures that use it directly (x86, s390) and the four that call it via syscall_enter_from_user_mode() (x86, riscv, loongarch, s390) end up selecting GENERIC_ENTRY, hence, no functional changes are intended. syscall_enter_from_user_mode_work() will end up calling function whose visibility needs to be limited for kernel/* use only for cfs throttling deferral. Signed-off-by: K Prateek Nayak --- include/linux/entry-common.h | 10 +--------- kernel/entry/common.c | 10 ++++++++++ 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/include/linux/entry-common.h b/include/linux/entry-common.h index fc61d0205c97..7569a49cf7a0 100644 --- a/include/linux/entry-common.h +++ b/include/linux/entry-common.h @@ -161,15 +161,7 @@ long syscall_trace_enter(struct pt_regs *regs, long sy= scall, * ptrace_report_syscall_entry(), __secure_computing(), trace_sys_ente= r() * 2) Invocation of audit_syscall_entry() */ -static __always_inline long syscall_enter_from_user_mode_work(struct pt_re= gs *regs, long syscall) -{ - unsigned long work =3D READ_ONCE(current_thread_info()->syscall_work); - - if (work & SYSCALL_WORK_ENTER) - syscall =3D syscall_trace_enter(regs, syscall, work); - - return syscall; -} +long syscall_enter_from_user_mode_work(struct pt_regs *regs, long syscall); =20 /** * syscall_enter_from_user_mode - Establish state and check and handle work diff --git a/kernel/entry/common.c b/kernel/entry/common.c index e33691d5adf7..cc93cdcc36d0 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -79,6 +79,16 @@ noinstr void syscall_enter_from_user_mode_prepare(struct= pt_regs *regs) instrumentation_end(); } =20 +__always_inline long syscall_enter_from_user_mode_work(struct pt_regs *reg= s, long syscall) +{ + unsigned long work =3D READ_ONCE(current_thread_info()->syscall_work); + + if (work & SYSCALL_WORK_ENTER) + syscall =3D syscall_trace_enter(regs, syscall, work); + + return syscall; +} + /* Workaround to allow gradual conversion of architecture code */ void __weak arch_do_signal_or_restart(struct pt_regs *regs) { } =20 base-commit: d34e798094ca7be935b629a42f8b237d4d5b7f1d --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2056.outbound.protection.outlook.com [40.107.243.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C34021EBFFD for ; Thu, 20 Feb 2025 09:34:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.56 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044057; cv=fail; b=eoo99BTFIsE1fmGh7Dy7nC3V4wWu2/QfemhvxMqdqvppdjcyeMmcaz6RN9q/KhqhvxKDY5ZowcoDUUgn1EAuLqdwhWTxv84pUrqq9DDJBUDzTP4NQYM/ePpV/I6JHDpJ3fPysr/elsz8IUF/0FaHYJjwknYWDBSMl9rA3+JGjoI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044057; c=relaxed/simple; bh=UIPxSCJkI06Infgf3LtjZGZLjH0nyNtzfUngkyYS9wY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=bnOJJabTzpFzmxHQJgUkt6q6/IZF8DbjByolfabdscds9i/TT4rcCK6fHjBX8u4qxX/m6Kk/cerEMY+F4s0HURPRLnVyxx512LJcA7xxZ6t6DaL1caVT77sXYlU/O/Oh0n4VZvDJK1zrHUe1ungJtTe9p3dAbAXiBoQh+YELKPk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=OLTbf3WQ; arc=fail smtp.client-ip=40.107.243.56 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="OLTbf3WQ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lF6HtKz5skpercyepTyzsiu/BJ9BGyZwGRctuWFHiHqYh0ARCU0tWfrI6kgceG5Gb35M3yLugfbkD4HIc0TRlbNUKhKbm8LhuCADVJOuGfSBki1EXUugWVckl1yaE871RI9RZfjhusRmo4wGQdDmlIcfxOhO20FwtSZVsJKt3xqHW5q0BqNciMUzhPiyYUEI2YZw4E3LaCDTxd1Z4oDV+FZcJDreiKcwdF8fmUS/z9kEiyZZNEVaANLQHaUwpcVT8OcVFKFjQlJ9OgGd4wUbpoOFrGZyQVsI0uYvbXM3V7A2djXB8AGberHozOOXXzwTWIdsBlAwt+h9cd+o6C7r4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kBFdh1DRzZ9UpkhGFDk3x4FN526UFdoen+gchhsFbEM=; b=GwXYotqGlZVxF0tynzSQMcmTTRMNJllh0IcTTjSGpRAZ158IAoYRCG2FNMPQSzGlePp2bzXPLCn3V23mk26oAKQ+2pBxplhxQqJDzQb1A8VbGJwbSAGuwcs7cK4H73ImwtdMskEWqeYI34kAMxjCLPSfnweVJDRu/yqRMjJB64l9qYsxmUUo/lKRC1m00vUeHaqKBZKLnwK50CtVreG8+Z/1m4R2ri37A4X7B9DnABwJMN3GG58gIprqQgTfkk4HhGULPXF5vvTpmIplceP9lNqb+217rMqKfvQOIT3h2qlNfY4cdhNRTqv7wJnz+wadz+ZDGKhduBI6BbLU6cAS2A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kBFdh1DRzZ9UpkhGFDk3x4FN526UFdoen+gchhsFbEM=; b=OLTbf3WQILbm0uTS4Ch5mM8UNsuZURvFomyhzMef+jYIwTV3bHc6Vbi4OjyzJxgu1qMCJR/JiYRle6DSBdKXun3c5m+qcMa9IJiL/dPVrsYU2fbEU2oiVi162Z1f2jMLCRp2vMPT4i/fh2Q92q1juwb5JDsmuUnjvKWFTj7vb+0= Received: from CH2PR12CA0011.namprd12.prod.outlook.com (2603:10b6:610:57::21) by PH0PR12MB8823.namprd12.prod.outlook.com (2603:10b6:510:28e::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.16; Thu, 20 Feb 2025 09:34:08 +0000 Received: from DS3PEPF000099DB.namprd04.prod.outlook.com (2603:10b6:610:57:cafe::df) by CH2PR12CA0011.outlook.office365.com (2603:10b6:610:57::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.15 via Frontend Transport; Thu, 20 Feb 2025 09:34:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DB.mail.protection.outlook.com (10.167.17.197) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:34:08 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:33:57 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 02/22] sched/fair: Convert "se->runnable_weight" to unsigned int and pack the struct Date: Thu, 20 Feb 2025 09:32:37 +0000 Message-ID: <20250220093257.9380-3-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DB:EE_|PH0PR12MB8823:EE_ X-MS-Office365-Filtering-Correlation-Id: 7c893e11-182c-4c08-1535-08dd5191b9a2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|7416014|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?T2wZKpaPn7tjPK0GM/+VhoSfhGc9G5Gg/mcnrZd5a8oDc/IFLN07L3Jyfg6+?= =?us-ascii?Q?f5oNYncdkQe/EoPotQpW/dtPC4vxZTJSufA5RK62J5qI+yQv9JvCB+os2GGy?= =?us-ascii?Q?LEZ2/Xrab7QtgBT5fdf0XFJtIz7j2zP0YPUOFfz5CfRkX8MlRog/nvgVwuq9?= =?us-ascii?Q?LPgdW7ItpVAAs9ZskGTtjSMQgzQaUG/qQx+HeL4DZDCp0QdwArrQAUjiWO3x?= =?us-ascii?Q?NxeDamkkDpT62/LKV3tCVovUwDVd8ENfIjZbugrHy5XtklCNHO/cvRess0IZ?= =?us-ascii?Q?/3k8qwWjKe7S4KIll6r1b0MRv7YjI2d78NlYtq/4UltU0pJzRD+yxjvTr/tE?= =?us-ascii?Q?EM7oP7wgYB7wOEQ210uJFhtaK2nC5M8BuD9SY5/hNvhJWrXUs1zn32lfT1a9?= =?us-ascii?Q?gmmNpykpsmNDGCNuSXtyyH8m+ScjzAXguIQzGdZeMJ8pl4m5ldNEctksA9OM?= =?us-ascii?Q?FbUKsdWMd+rbrBKaa1fb1+1t6ZNE+QJTIsrOf7004c5ilOH8vaiMTUWMVFMv?= =?us-ascii?Q?SXF73T+21XinHskD1FaM7BE38lRRnFoTDxIshokqURNxitimgPV1MTrN/N8Q?= =?us-ascii?Q?Ol0xH3VXIJC/luBTDq9qAvUSVO6nm5pzd1Tk8SDshxV7S0iMXOVIsIE+KEo4?= =?us-ascii?Q?kv4t1a64fhCHaCon5GQ1DLB43rGy8cOaeDYjqg2kLJlLzT0EBtMvNsI0pDgx?= =?us-ascii?Q?aHnlW/7ezLOczvWHgYcb9dK/wgNxbm2ryA/fs4/Ezc08QmvpiJ3x39+i1Qi/?= =?us-ascii?Q?9REPMakc7CMH0TI28q0lYYZbKohlEPOTWmu3CXRmkthy7FwT7QrswacW/EHq?= =?us-ascii?Q?tawJD2devn4z5t4sbb9HIEKvhmmMSVaWntQPFWfQvFPb5Tm2Oz302gCdZJwb?= =?us-ascii?Q?jbdcJtCbLCv86l03YieRUVYoFZOVICLL2ls0Qf2Dst7KqunDAwcjv+j+53PM?= =?us-ascii?Q?i8gRIRewdYjjKCqzG9xo4S9CJowc5bo9BSjn7NuTbUANu3FKzTDO2VXH2vIT?= =?us-ascii?Q?Ryx6bPMPZCE4LbVpqtNS11DP5Dq2bBY/0rjmvYwFkcAufb20rzyO+wMimzpH?= =?us-ascii?Q?98nIaapMXOYkz1F2bjZsrFK0VE1owI4EUkBObVKeAz0nJv+9q5+HLJr4RsOR?= =?us-ascii?Q?t0dQhNqSUZaivcv1l9rMcJMbm+11L5EjMa2kUujwSmvNExEqxz0JazLnkq9d?= =?us-ascii?Q?CK4kTS3qrC8sG+m682QpjaBlAVohXD66KR825YhBoEfKpOmNR/PauXy8Sv4G?= =?us-ascii?Q?gGrYXNq9KmIu2m4NqJiufsxJV+BnZMGf9Z8v7mq39aZ71ytD52QzQYRlRvua?= =?us-ascii?Q?yfayBQ+YVZsaBlUAzRYG5LyLWvtDVbnAMMEmWr6jDkGEr4ibvA2nnT63+W69?= =?us-ascii?Q?bPznhZwyr9AFUCZxFFxvgKw0GTE0piYNkzpLbc30d1c0Mur4IaupHqgE26AW?= =?us-ascii?Q?Z2OjfzZ/UqQsZR4/B+Q0L5twHDntie2+RXMXaXr9TGIT1wNHKbiPtNVFcTeN?= =?us-ascii?Q?R5+b4u3rVrgytNI=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(376014)(7416014)(1800799024)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:34:08.2055 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7c893e11-182c-4c08-1535-08dd5191b9a2 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DB.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8823 Content-Type: text/plain; charset="utf-8" se->runnable_weight tracks group_cfs_rq(se)->h_nr_running which is an unsigned int. Convert runnable_weight to an unsigned int to match the types. pahole notes that there is a 4 byte hole after se's "depth" member since depth is an integer. Move depth below runnable_weight to prevent the 4 byte hole and reclaim 8 bytes of space in total. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 9632e3318e0d..34862d904ea3 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -569,14 +569,14 @@ struct sched_entity { u64 nr_migrations; =20 #ifdef CONFIG_FAIR_GROUP_SCHED - int depth; struct sched_entity *parent; /* rq on which this entity is (to be) queued: */ struct cfs_rq *cfs_rq; /* rq "owned" by this entity/group: */ struct cfs_rq *my_q; /* cached value of my_q->h_nr_running */ - unsigned long runnable_weight; + unsigned int runnable_weight; + int depth; #endif =20 #ifdef CONFIG_SMP --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2066.outbound.protection.outlook.com [40.107.244.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCEE41EB188 for ; Thu, 20 Feb 2025 09:34:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.66 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044071; cv=fail; b=OOvocxLRnoTsQFmwk3ZGCdLQR8Iks58ymHfwOqc3y3jnJYgWExc6tEADzyrTIBpBrmYKzMeXyR9z+NcoCO2epUGmmzWt19hd9Oz4UWCSvrlhlu/ChDiCpTL1gImXJL5WAizwYfNQIDewSegWo4SaG9ESkO/eyIgqgOMDtZ8QeeY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044071; c=relaxed/simple; bh=PXBnxvdSR08axr/5ulrd+i+xg/LYAleBs9/oKfVm9Wk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QkX9oPdPR/Z2V0nAEjam9kBY5E2M+2voavISAePFelekYWAmBmXGux1TT5OMauHIIKusLffK6Dl7WO0w2C5ltLo/oWV79LzIyyebCtwKzGCjXhiudR/Ps9mOyCqyZlDn+Ioeio+fxO491UQSUSvtufdxHJRCHK2Es6kj8jzxEKE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Tb7dstII; arc=fail smtp.client-ip=40.107.244.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Tb7dstII" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BdKs+oDO11Ui3JfDruSnl8pdOe5lTDEyz1PoVvIOFZIRjd6I+2Y3TSKlaU9qRSvdiczip96KyqiF1qdALF70dSYHQgoKkJNb+eDOea2asEThjPP8KItUTNu8ehnkXJyn8qBqOfLHCAWzLkGKZtRQdEZwp8/QsLqTO8tAfIRmlwvQUFNcvW12Bw0GcXoX2Cp92VS9CHzAJe2+bCD/KNZpvIwQu9LLFRRTVG5NTxmDfJpH0ToE/cQsXnuJ4bvToZU94nJr5n0sfVlD+Imis7DOixcWSa1W0i1SHH/fL8Sz7rUA8Gj3TAzjTh9tCtcnw1JSaG9SGkvrniVOOJmAGpwXbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xo5YEpkAzZH6uuGbbFuGZkNa23/Az+vfC+Spr7Rc8Tw=; b=ujLOyWLheJJSUJs+Cl4sccCsyp+jCwtjggnICekeY6CGEYZ+ZUXGxughesvAYOcyyRiSZjC9xjG6C40xZV4S6AVumeuPyBgr/brz7q3pXZ777Ir0cxB4zFyxxzDuwBHISK4td0mGkD4nhC/Goz1zw+R8B6DR1ymBt7wG3AGU1XTe3l7ji+4ltuMpWX0BvYIytndFoSsmj+YFpCH38mS+d0rTyUEESUHYF7jMlMcIs0RBcahJ+uAm1kLJN3Ix4eYbyICERQSXnJfqhJf7P8TNeQICxDx1vBj4i9L1EPXUWK3YOhd3u4ULi9Hotdp4IyV2EqPyMaCMiW3yTTYoSNPviw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xo5YEpkAzZH6uuGbbFuGZkNa23/Az+vfC+Spr7Rc8Tw=; b=Tb7dstIIgBVStolojTQV1oI0vJpvDpr9UfY93ZHXL+027AONcX81voAWT/n3PSVB9YBQke6FiYd7xI7xEz0J2zN56PdL+UZvA4395BhxZCKf2kJDkMZw4n7EvWh+miknu+Wl+pHVVYopmmXadveHb7GvdBpdDLysZrO0Z+ytKFc= Received: from DS7PR03CA0204.namprd03.prod.outlook.com (2603:10b6:5:3b6::29) by PH0PR12MB7488.namprd12.prod.outlook.com (2603:10b6:510:1e9::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.15; Thu, 20 Feb 2025 09:34:27 +0000 Received: from DS3PEPF000099DF.namprd04.prod.outlook.com (2603:10b6:5:3b6:cafe::78) by DS7PR03CA0204.outlook.office365.com (2603:10b6:5:3b6::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.16 via Frontend Transport; Thu, 20 Feb 2025 09:34:27 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DF.mail.protection.outlook.com (10.167.17.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:34:27 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:34:18 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 03/22] [PoC] kernel/entry/common: Mark syscall as a kernel critical section Date: Thu, 20 Feb 2025 09:32:38 +0000 Message-ID: <20250220093257.9380-4-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DF:EE_|PH0PR12MB7488:EE_ X-MS-Office365-Filtering-Correlation-Id: 4c1d4e04-5742-47af-83fc-08dd5191c4e3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|376014|7416014|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?N8Z6UVSx5YEnxe2nusjEETGfDPJ1d74q5pRLnseupmCfTMsqp/3oQwDoz5YE?= =?us-ascii?Q?6QygecdpZeonSBz0A2Magw9m1mDcaWJ5PlWcRt0ZVFsQSGvpgCvRoibJYHvX?= =?us-ascii?Q?DKsfLmkWAOh9kPNukDvzr639lBLuabS6DAz0fGxHOK1auN2oz2m5qRU/nnPK?= =?us-ascii?Q?PV0WR2vyDMfYfXq5XNDTevrSYTQkwmCU++1Tgpd9bRUJuB54/zEctHd/uo+x?= =?us-ascii?Q?WnL5ydEnCN2E11ZIZvRniS8jyicbTvc5avqIhpEq8UEKOeaBmwfDIuwANz+c?= =?us-ascii?Q?T7TY+NeJ+fxk1ozCZPMUo7oAoUvCliAc+asR+Lu984hybXjbYKWbbQmDE9Kz?= =?us-ascii?Q?2qD6aJfm9YBw35GyYZbvt8Fo0SD1z884jSg9pDxg8W10UcXjfuszzRXgSurK?= =?us-ascii?Q?tRdIflieD9fm+jGXBmCzGFwauBYUx5l7/+k8wU8vw1BicCzQh1W0fBXbZyWW?= =?us-ascii?Q?oNaKEb2UBmt1b9KKs0SyDLcazoq1SqvBs3YF18V3mrilQ720V/rmfuk+IDbc?= =?us-ascii?Q?TctGT3qLvFuFwpOv7oaGgcJscNLtb42L6LLepPVQ65EPhgsIRq4QzCalkffR?= =?us-ascii?Q?xV86kzUYRTWZIH4M/lBHQVJm82X6d7OqgJmH9iwe8d8z+OOVXz3g+SlYLdSO?= =?us-ascii?Q?hhRdzu3M3zo0rnglzKS5dofWB1ob9fQlfm8GVUEB0nhPsslab6MrrVFqQl7T?= =?us-ascii?Q?ILRs0z15NEr7F+xXQNvZr3/EhNsiW3duCoE1+VSbx9fgHRAfYkC4QP2QDN3N?= =?us-ascii?Q?dzPIdibnxSgxpXK7cKaC6syMxUfz37GCEv8IcQrP1tO+UhhEDVYYhKPOJ9GS?= =?us-ascii?Q?ifclIQLvnwKd3AfN+g0pFX89BvLfJ5K4/TBlawcqEBJX93sHwL9LqIJFBu7o?= =?us-ascii?Q?u9fV+yS78rHkA9r6027Anr+7xcUXqUonNfNMmG4yoJRK5+hNpu+jg4KBUBsT?= =?us-ascii?Q?bP9jaMUgdewaBUr62+wcwFvCkndjTy46ByFDBVAzysR1UDPbK1smQ1cXj8Pq?= =?us-ascii?Q?DOWTqy1ny+rMgT/khIs18PaNOeQ2x6TQ5OVzg1ubZgxBshPr9YmVixZH59ro?= =?us-ascii?Q?DuFc2ODet0mY+CEIudxdmqrUtIzRwS9Ss9g/OqM14J8TMWbBDqf+ajzzOAFw?= =?us-ascii?Q?ZVvC/XPiYfr9Isz7M/QIHSPfxDQyMdFqhniYsDevQfptDj8ykkCLA445+/5m?= =?us-ascii?Q?lTuwJCP/f4d+kqs/bjXIwuO1LQVFnXtt6cwZwTaeM07HAZb2kuruIt1Ngsxk?= =?us-ascii?Q?/6hinmZJfu2hiKtkLvoTEoSR+tjS/1qvgf7jNHMpnW8SUdxDiFQAIfZZkxAx?= =?us-ascii?Q?ON0ItBE5LYnD5NZbLDHqRxmQT+s5Jn7Yx1iy6+FKAy+rool/JgatUhlOqYXd?= =?us-ascii?Q?bNDq00YDnLy0EB6clOO/yoiDBmSbRQi6VFhsuicoi/5/kM/oPLsbxqDaikUy?= =?us-ascii?Q?c3R90xC5waJsR93O2NDQiQykMQUnWrogbEq7sWni7p1gayzr0yYnxi+8zZ00?= =?us-ascii?Q?hOF4u7e7oofLsK0=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(376014)(7416014)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:34:27.0737 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4c1d4e04-5742-47af-83fc-08dd5191c4e3 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DF.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7488 Content-Type: text/plain; charset="utf-8" Mark the syscall boundary as a kernel critical section. Use a per-task "kernel_cs_count" to track task's entry from userspace and exit to userspace. When "kernel_cs_count" is non-zero, the task is executing in kernel mode. For this Proof-of-Concept, "kernel_cs_count" can only be 1 or 0 for a tasks and the implementation will run with the same assumption. The critical section is defined as an integer count to allow fine grained control in the future where certain boundaries within the kernel can be marked as resource holding critical sections. For the sake of simplicity, the whole kernel mode is marked as a critical section in this PoC. For future extensibility, sched_notify_critical_sction{entry,exit}() helpers are defined to mark boundaries of kernel critical section and is similar to preempt_count() mechanism. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 19 ++++++++++++++++++- kernel/entry/common.c | 7 +++++++ kernel/entry/common.h | 4 ++++ kernel/sched/fair.c | 20 ++++++++++++++++++++ 4 files changed, 49 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 34862d904ea3..63f3f235a5c1 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -577,7 +577,24 @@ struct sched_entity { /* cached value of my_q->h_nr_running */ unsigned int runnable_weight; int depth; -#endif + +#ifdef CONFIG_CFS_BANDWIDTH + /* + * Keep track of tasks, and cfs_rq(s) that contains tasks + * running in kernel mode. Any throttling event for the + * cfs_rq will be deferred until this count hits 0. + * + * Semantics: + * + * - For task: It represents if the task is currently + * running in kernel mode. It is always 0 or 1. + * + * TODO: Describe for sched_entity when implementing. + */ + int kernel_cs_count; + /* hole */ +#endif /* CONFIG_CFS_BANDWIDTH */ +#endif /* CONFIG_FAIR_GROUP_SCHED */ =20 #ifdef CONFIG_SMP /* diff --git a/kernel/entry/common.c b/kernel/entry/common.c index cc93cdcc36d0..b132b96e2b96 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -83,6 +83,8 @@ __always_inline long syscall_enter_from_user_mode_work(st= ruct pt_regs *regs, lon { unsigned long work =3D READ_ONCE(current_thread_info()->syscall_work); =20 + sched_notify_critical_section_entry(); + if (work & SYSCALL_WORK_ENTER) syscall =3D syscall_trace_enter(regs, syscall, work); =20 @@ -214,6 +216,11 @@ static __always_inline void __syscall_exit_to_user_mod= e_work(struct pt_regs *reg { syscall_exit_to_user_mode_prepare(regs); local_irq_disable_exit_to_user(); + /* + * Notify scheduler that the task is exiting to userspace after a + * syscall. Must be called before checking for NEED_RESCHED work. + */ + sched_notify_critical_section_exit(); exit_to_user_mode_prepare(regs); } =20 diff --git a/kernel/entry/common.h b/kernel/entry/common.h index f6e6d02f07fe..73e699a4c3e9 100644 --- a/kernel/entry/common.h +++ b/kernel/entry/common.h @@ -4,4 +4,8 @@ =20 bool syscall_user_dispatch(struct pt_regs *regs); =20 +/* sched notifiers for CFS bandwidth deferral */ +extern void sched_notify_critical_section_entry(void); +extern void sched_notify_critical_section_exit(void); + #endif diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 857808da23d8..becf2d35f35a 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -58,6 +58,8 @@ #include "stats.h" #include "autogroup.h" =20 +#include "../entry/common.h" /* critical section entry / exit notifiers */ + /* * The initial- and re-scaling of tunables is configurable * @@ -6704,6 +6706,20 @@ bool cfs_task_bw_constrained(struct task_struct *p) return false; } =20 +__always_inline void sched_notify_critical_section_entry(void) +{ + current->se.kernel_cs_count++; + /* + * Post this point, the task is considered to be in a kernel + * critical section and will defer bandwidth throttling. + */ +} + +__always_inline void sched_notify_critical_section_exit(void) +{ + current->se.kernel_cs_count--; +} + #ifdef CONFIG_NO_HZ_FULL /* called from pick_next_task_fair() */ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct = *p) @@ -6772,6 +6788,10 @@ bool cfs_task_bw_constrained(struct task_struct *p) return false; } #endif + +__always_inline void sched_notify_critical_section_entry(void) {} +__always_inline void sched_notify_critical_section_exit(void) {} + #endif /* CONFIG_CFS_BANDWIDTH */ =20 #if !defined(CONFIG_CFS_BANDWIDTH) || !defined(CONFIG_NO_HZ_FULL) --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2087.outbound.protection.outlook.com [40.107.95.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 99F651EDA08 for ; Thu, 20 Feb 2025 09:34:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.95.87 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044093; cv=fail; b=hmcqJTPYFd4ZtfdfRRRTSQBVztI/+cstsNGppOUyIqpVXVyotfR7WpeQ0udfGTXwHncMy0ob9pL4KpBuTwVSRGsy5nh2etZeES5KCh+w+dbGYNI7HUpZoKhlJXlgdNc/gcu0GbcxQF+O5VUTM1NMFE6kfPltdJygbs5euo4XzT8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044093; c=relaxed/simple; bh=tmeAss37SoNQc9EHx3hocH+nRv7twvt5nQlycP+Sk8w=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=pUVIpqR5rf268wlNh0aK7rsGcc7X9HVkWd6rpRscll3lP1q3exkGoynGVYVoCKvCuonXgCByjhHtCTmAmLHtVePuehoRh3wlxRv96CvA9LDcaoBsWG2Aj7plSrsfFQWGx3sFCUplF4UcVwKVXckIpboTW/Z5Lt9a2hXYGlWwuGQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=hmSexrhP; arc=fail smtp.client-ip=40.107.95.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="hmSexrhP" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xRznBieZjjazhJVDUXCpdzMc2Z6pKpik2gE3ZqnVTQQhMuZIPkKkZdfR5N+a3YeAjSQOXf7Inc6YXAIf+IwXliZqpaPz/rrDP9uQYxrU8+DLcwQjXwagX9OqW/5jfmtFJS3zOAa1p4R9Hy/yWz8FBoYzBmLwcsrLIazI00V0dwhtfnaYNuynz8/XGcV7ufVYqbr6ZW3xZ7/T4qAfqDWuPxAgFGbKy5xvonmTmF1de6PUaQ1Zh+/Ml76Z/zcx+CFyhZ+kTZyseoIgEAWuDt5C9V4hEZQVPPDq9XBfRefWkLeYgQmA+zMclrB1bC4RYEKDV5Z2l/EwJaSg/jpdcXYDfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CGuJkEZJ1okMvOGxv+Hz5pvHTp/JxaKF5Hqh6V76DJg=; b=SG72a8pOkU9IDq3fZpXd3LtIxApjLOcJxKOL+f2BUQYF/L6v4BcA1qlicuyLYtHzNDvyE4lcTCNzyIfcwfVnxLbLn8dgkjW9YwjkU9xcT9fdi6ad1esG/SMEa7nJPFzayLn/AApAwTx/6PZI3wgR28vn4E4euKCEafdRs1JqNrHFioMYXCzbW1/Wx59ZJPdPRGpvg/s4bVRtni6sBw5DxOJHl4ZMeLPDglT3tI/nrAcc6fbZ4D/e5TGHhtpqXPfnWBwgbL1t6ippE8nZnU/tLJFP09dDQObBBApou+VhNJyxb91z9U+bd9y/K6FLzO5XfCDqQgjsox2bgKRdRIZ9Dw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CGuJkEZJ1okMvOGxv+Hz5pvHTp/JxaKF5Hqh6V76DJg=; b=hmSexrhP97HN5Bqt6Fqety3nSSn7Qd4T8qQ44YC8rCbDezdtEvt7UNYAUJPOruAZ7mo3VC2ABYLL5PRqfPDVHO446MpT1Z553DkSACm7DclQiKl//C3VNV5KPGdOMFqiVCPBldx3cRRrPgRF1osiKdtd7RoakHlIwkqZG7SHf+s= Received: from DM6PR11CA0032.namprd11.prod.outlook.com (2603:10b6:5:190::45) by IA1PR12MB7759.namprd12.prod.outlook.com (2603:10b6:208:420::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Thu, 20 Feb 2025 09:34:48 +0000 Received: from DS3PEPF000099DC.namprd04.prod.outlook.com (2603:10b6:5:190:cafe::35) by DM6PR11CA0032.outlook.office365.com (2603:10b6:5:190::45) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.15 via Frontend Transport; Thu, 20 Feb 2025 09:34:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DC.mail.protection.outlook.com (10.167.17.198) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:34:48 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:34:39 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 04/22] [PoC] kernel/sched: Inititalize "kernel_cs_count" for new tasks Date: Thu, 20 Feb 2025 09:32:39 +0000 Message-ID: <20250220093257.9380-5-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DC:EE_|IA1PR12MB7759:EE_ X-MS-Office365-Filtering-Correlation-Id: 0a3acc76-aa93-4692-50a6-08dd5191d16b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|376014|7416014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?KR2FQmxrpZCgdDCfLscv5/bhwMX+80eCDgVu9ATUGVIEEyUNuWR2fbcAnuk9?= =?us-ascii?Q?qvjXc05e23om97qutRjPy7s+AZVr+CYbJWluFvJ5do9IlgIzKQXZp4LbrdSO?= =?us-ascii?Q?JKdptEVKygSRC1rxKQfSPjZ060bHqkYPjN4U4W0X45aKeGAQKIfvxUznF43D?= =?us-ascii?Q?lGTvxOI6Nq5ydsMW326gC50cRufL6jDlmyCcI92mbTs6YuKLgBhwufyNIz1s?= =?us-ascii?Q?brqFeJQ5foVHkShOiIK6q3Uyesw1VrMik6wp/ujybOiKsvh8LIEMDAQZZ3xR?= =?us-ascii?Q?mcMJSzLWPVEUcW0ISvnqovJ5sChKqUXnL6W0ClJMN4/QZhFxKvd269gGQpMS?= =?us-ascii?Q?YpW3LISib1weKOxqCa5swprE+9Fce72Akq1W7voWy74KCOLU7wBcAqwOwVrz?= =?us-ascii?Q?kdwDGiwLionX01h1wAtpCUiZ08+6CD0l0G8QAKY9QuFjcELD4a5UhDJRlWNG?= =?us-ascii?Q?09mWiBPMf5UaZtjhOj2d510PRYnJ4LmPnlcWV28YE7AtIvbRmy58QAj3oaFS?= =?us-ascii?Q?yz3H2+Qt9Sy4OKCpep1u69qNGHVxZxput5sn5GjPICZVqi9SG6SpedtUf9lS?= =?us-ascii?Q?uBwVKfFpWliGPnp64UXzUTv/DEWRry1lwad8RgbAlBz24Nfgf820mP/yVqVZ?= =?us-ascii?Q?/hDMvYLelVZf0Vby2fN5ENwBO340mESA3qQZ9aorjAwXUhAysq4IOCuyYQBk?= =?us-ascii?Q?Yt7y8EsZPG5++zu0ajHOZHj6C4hQiS46YMJQBULNwLaseb1TA3PHtxXEet9F?= =?us-ascii?Q?1A3LtI3+FjrNr+Q0Yw/k5tL+5KSUaImAkWBMWReiTvLlVcMdpU+SwgAP4d86?= =?us-ascii?Q?SMcCvOLrDDD9oCsDUbF04qsz4gJKrEbYD2LTEbehutQ/6+zgfuQBV3OyEwzE?= =?us-ascii?Q?d8HdyZPArmkv85f2ycx0kuHPEgeank/E85rv+Fi65xGTY3gAXZ6Bi35CJMWv?= =?us-ascii?Q?YAuiSN72yXaPCp1RIXguUcE2xt2/v+Cd/SkE+RJSkPARxG/EM5tHrzDzBTrO?= =?us-ascii?Q?7EQjbZ+7uM6PYQV5hwg51YRql6s9WwTW8a5tfjtxp0KfOnJLiYlp6vyU/OW6?= =?us-ascii?Q?6KJRfyVIv46U280t+qhSrpTQBydwOwJDSsVMocHijKoxv0xH1nFcJ2DkzhNO?= =?us-ascii?Q?R4VqwSZjkS2PI1pAZakQeL0zfFyXdh5mHIh0Wh7sa2jRd5k3pRk7JKrFEeLE?= =?us-ascii?Q?L9xJqWqfO1Ua/ZAkbNN5SsOeZ7/X2l6EjjU/lzpnNw469/eDOxM15ANbZEUk?= =?us-ascii?Q?xbwj2N2Jq0Df15XlW3n0gt/aCNllpp4M6rp85Q6rOvpFBEUxmm1Snfan1aBD?= =?us-ascii?Q?o1W1s7ROUVBbbQNiLqBRAqyT7VvNkSYObYZWzMd0qDH8d/eFIRQfVLTdiQbc?= =?us-ascii?Q?HOsuAm2HSXWffHxg22aAXEtC7YknDnlaOOjvBMctTdNGUyMHGxxi+z11cYrq?= =?us-ascii?Q?3hIOwwpjJYiOoVT344rcDZmuzBokh+ZFggxqsgBs5W7h4uT+ZpFYkk58NOD1?= =?us-ascii?Q?YjhY8h+Qo3fdkjc=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(376014)(7416014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:34:48.1119 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0a3acc76-aa93-4692-50a6-08dd5191d16b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DC.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7759 Content-Type: text/plain; charset="utf-8" Since only archs that select GENERIC_ENTRY can track a syscall entry and exit for userspace tasks, appropriately set the "kernel_cs_count" depending on the arch. For arch that select GENERIC_ENTRY, the "kernel_cs_count" is initialized the 1 since the task starts running by exiting out of syscall without a matching syscall entry. For any future fine-grained tracking, the initial count must be adjusted appropriately. XXX: A kernel thread will always appear to be running a kernel critical section. Is this desirable? Signed-off-by: K Prateek Nayak --- init/init_task.c | 5 ++++- kernel/sched/core.c | 6 +++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/init/init_task.c b/init/init_task.c index e557f622bd90..90abbd248c6a 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -88,7 +88,10 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = =3D { .fn =3D do_no_restart_syscall, }, .se =3D { - .group_node =3D LIST_HEAD_INIT(init_task.se.group_node), + .group_node =3D LIST_HEAD_INIT(init_task.se.group_node), +#ifdef CONFIG_CFS_BANDWIDTH + .kernel_cs_count =3D (IS_ENABLED(CONFIG_GENERIC_ENTRY)) ? 1 : 0, +#endif }, .rt =3D { .run_list =3D LIST_HEAD_INIT(init_task.rt.run_list), diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 165c90ba64ea..0851cdad9242 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4493,7 +4493,11 @@ static void __sched_fork(unsigned long clone_flags, = struct task_struct *p) =20 #ifdef CONFIG_FAIR_GROUP_SCHED p->se.cfs_rq =3D NULL; -#endif +#ifdef CONFIG_CFS_BANDWIDTH + /* Only the arch that select GENERIC_ENTRY can defer throttling */ + p->se.kernel_cs_count =3D (IS_ENABLED(CONFIG_GENERIC_ENTRY)) ? 1 : 0; +#endif /* CONFIG_CFS_BANDWIDTH */ +#endif /* CONFIG_FAIR_GROUP_SCHED */ =20 #ifdef CONFIG_SCHEDSTATS /* Even if schedstat is disabled, there should not be garbage */ --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2058.outbound.protection.outlook.com [40.107.220.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D2571EE019 for ; Thu, 20 Feb 2025 09:35:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.58 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044113; cv=fail; b=imGaZZsjhgE6wBzuUBZOOqnDrG2q5COwTpoG9KvXn4R2NdOiYgZWUzy0e83CelYtkzB/zaTmB1lDUSr2KvUQYcU4Lq3C25UCqYj5YuMoB1VXYaCY7Ehg5PXHTSrlv6HidrNVS/SQWszwPffB1GGlQT8VzNse62gz9k8wMgZbTgM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044113; c=relaxed/simple; bh=5Ax0oXVg4YtlxWii2Fu6KwnGPqUqITcXJuiS6WlNDOU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=mbgUQ7zJkYnIaZRurdF0ix7iL9pNR+ZjOOKNDiCUVKybahE51PeDn5ROvtNVsEwjA1BsNhRP4y4NSBjtn5QVjSCRgjtHMeWaQ5aUWxTLiYjqIWDkUmauPZ6P+6/0UNi2a5EmfxukOSUK5hIOxvgpu9EuJpenxjH7sl1jDEY0jUo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=PJ5//6gp; arc=fail smtp.client-ip=40.107.220.58 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="PJ5//6gp" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=umrfExN8wrkpTknAByMCCpnk/B2mC3GeYqOyRULRzX5AJfZ9MdfFafAP/tlEtTY/K7D7I1E16pdHGAih9eghWVS9GFJCJzPlBSMbnPGuchjkNsxHdYGWJ45kYjX48DIsHDeaWDIBw7/B4syIS7/QIsD1uDTW9qg/fTb6eObbCmNbnXdYFTXf1prnS+IknrqyXA68qbfYeK9MHa42yQ6Jei4XcrjtjtPt5eDYVk/AOwilXWFggsaqe9edguHTgvAqxdWbVbFoj+KC5C3pGef0xfizsQQ7Ht/xVszR5+pDBo1wwazMJ3nn4bVkdE4WNdh8Ap8Ha78HPWB38XNBUzI0Ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=93U530oj7EinGA8waDM6NqWD74Bp8HhoLE/SJzu35Vs=; b=pEicvJHhy75H8TIPHlTp2eybZbZIQZwK/BMK/g/8ji1QaISKTDUrgg45C7iok0kIxq8Q2en2mzQtvoOA16qEy+FK89IIlGeUQ5Gck4oE7dD8sWIY/rm8C/etS5bCVveZ4Hs+uSX846Ud74ZJcKAOkeefk8pqBSAKTbU9s6/MYttZn5xh+AZkquxI5pDV+HT1hY2polx1LFmzxtPnAU337Vwz/ctrgLFZczg5F0nE9Pou7M0ge9ukE3EhMlMnKGy7I1XJuHiBUOYBu0P8isUsf/PuU8mA0kyJ0n5koV/cD7d9b0AfUiD2jY5COmAC4zy0a82o9FC32l2ckR9X4dh7bg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=93U530oj7EinGA8waDM6NqWD74Bp8HhoLE/SJzu35Vs=; b=PJ5//6gpM8Yhg4UtKIZz35c3m84pvLq0Zni6sHDNibCrd8e7X8ItD+AvdeRMxvQSMK+eAVYBXXsXD+wKxPG2709FhxD9oIcxe8XHVOQw4qlu8F6FPxDwMSlnJDyKWldQ6v3ez4KvhMWlDC7xlvONKHYqpqGTeEhJA0MEYmhMOPM= Received: from CH2PR19CA0016.namprd19.prod.outlook.com (2603:10b6:610:4d::26) by CH2PR12MB4135.namprd12.prod.outlook.com (2603:10b6:610:7c::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.15; Thu, 20 Feb 2025 09:35:09 +0000 Received: from DS3PEPF000099DD.namprd04.prod.outlook.com (2603:10b6:610:4d:cafe::ed) by CH2PR19CA0016.outlook.office365.com (2603:10b6:610:4d::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.14 via Frontend Transport; Thu, 20 Feb 2025 09:35:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DD.mail.protection.outlook.com (10.167.17.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:35:08 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:35:00 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 05/22] sched/fair: Track EEVDF stats for entities preempted in kernel mode Date: Thu, 20 Feb 2025 09:32:40 +0000 Message-ID: <20250220093257.9380-6-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DD:EE_|CH2PR12MB4135:EE_ X-MS-Office365-Filtering-Correlation-Id: 701cbfb9-f947-490b-9754-08dd5191ddbe X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|36860700013|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?9sVtrYVW0FP+j+WwVFd0Du4YmmvcqhUQck3rXy3IPkbEDKZVc9mJsffNtFR0?= =?us-ascii?Q?Zg5L7Wv1pyQwpO+WqPC63eluIB+pACsYNDoAaRe6Q9Rlg8lOAnKXJlnXRjxa?= =?us-ascii?Q?GFIsmYXAVYH7+jbnuNd/fTbNsj9H8daMaMAJaI//xiPp8C2jlhPqWXWWuODb?= =?us-ascii?Q?4x15KJzRJ2irdE/jRqvtOrCv1aAXJu0KOFajKnaB7GX2CkTJ5ycKOnpp/GRv?= =?us-ascii?Q?uWcfJh/fQOkvJV4/DrEKjM8Zts2jFkzJ2QZoGOQYn5HJd2QrHA7vDBveg3KE?= =?us-ascii?Q?NxJ4aT8VHHodTXWgZOrsD9bAJhG7TMH032fZUY2CL7vtW25f2MQdQLRtO9un?= =?us-ascii?Q?V6J8dPLugmucePzRlNj2llI0iqjosTAfzSGOQiVuoBlTMyYNYAkM83AymPMA?= =?us-ascii?Q?JHkmlu7Q4PCTDsezEaw5OPpFtBsGdP5kvHLRwulybwuZySZQE88HrSnz4/rD?= =?us-ascii?Q?5TWuLlp0Q1+FtXF1WnDxxzWz0aArB301rxX5Zs8EqpprF7hGsag4pviD+wOA?= =?us-ascii?Q?9UlNmyI7deZJDZonVkBhOi8x9z+ad9LkwyCZipnBxdXyD7TsGsDPDwtrKGku?= =?us-ascii?Q?1ZpyqBylM8Iabqi7uI/auhGsRHCv8ymcHBPtSex3OC7rKHweDJGf2r6zFbAS?= =?us-ascii?Q?7zbm3fMPMUrsZrQFLAi+hqflgf6KqjSD0TQG5zGp5Hup3179J2l3U0sfrfrs?= =?us-ascii?Q?SCF+spS68ArkAAp86QRgkvZZ+DtWT/S4pideamzPWYGvIeCuoqMwyh50GL/r?= =?us-ascii?Q?LxEK2ZLy5cy3+6tFaP2qWSFWqZcHaBtQk7paThfyzP4Qq4ItJsj2MpPL+27m?= =?us-ascii?Q?4+oejUtTVzl0p0KGxNwM3KvAtWaTFmWjUdbAm8pHOm28qlsi/Y30zFRejjsI?= =?us-ascii?Q?0V96Gmy3NI9X0NbcPxKrDagd5qAdCl6uWRWg/O1H6Ca//fcwDbZS8FOBTa9Q?= =?us-ascii?Q?4NZ2fmiA2WdwC5WT/72LTGMoxWRVIce8o1/SzBD3S58c6jgw31OaV7O1vGN1?= =?us-ascii?Q?HpXxB1LZ9qaB9BL9Rs9HxS1w38kGrp6cNNjB4bHqgNro4Ca8hoqZGcN7+xTu?= =?us-ascii?Q?7SETHgGXQmGqLYV8Tcj70Mo8r+HLr4CvN59ssn2yiNHBoaCha2oA941SPzhf?= =?us-ascii?Q?6oS3Z375rCGDdd2FPZt5wrVoKRgafZxQNXneUIDqc7NfOfkBu7hp9siXRzqK?= =?us-ascii?Q?h0zou65ANDRg1jwZcJFulAKa9HyX2ckAQeyrS5e/9A4mkJxCqLIWvbMp/K0+?= =?us-ascii?Q?I0xcb1bGb/yxiu7UGNDZBifIB9vQinqtJ2ciep2brI1FeVdRQ/oge0DJMAfx?= =?us-ascii?Q?Zjgq/JL6uvgoR2xqNGtMVhNiqEswQGc8mxzskvjrDrjCXRMnt/cWh26WWKYq?= =?us-ascii?Q?cMdSulV0F+V+YkQ2MOIFOGfJWne2zpkwqUrp1WuagescmxXp0oAXsPOpluUx?= =?us-ascii?Q?MP9Bs+RUPfa3adtCTnh7A6oM1NYy7kx0yJTtXqfZYD0M8hTAZ+6VrA4ZOP+v?= =?us-ascii?Q?hQ13TU26hklF/IA=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(36860700013)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:35:08.7904 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 701cbfb9-f947-490b-9754-08dd5191ddbe X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DD.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4135 Content-Type: text/plain; charset="utf-8" Throttled hierarchies will require only picking between kernel mode preempted entities queued on them with throttle deferral. Track EEVDF stats of kernel mode preempted entities in avg_kcs_vruntime and avg_kcs_load which is the same as avg_vruntime and avg_load respectively, but only contains stats for kernel mode preempted entities queued on the rbtree. Since all the checks for eligibility are entity_key() based, also update avg_kcs_vruntime when min_vruntime of the cfs_rq changes. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 62 ++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 6 +++++ 2 files changed, 68 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index becf2d35f35a..cbb7a227afe7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -523,6 +523,9 @@ static int se_is_idle(struct sched_entity *se) =20 static __always_inline void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec); +static __always_inline void avg_kcs_vruntime_add(struct cfs_rq *cfs_rq, st= ruct sched_entity *se); +static __always_inline void avg_kcs_vruntime_sub(struct cfs_rq *cfs_rq, st= ruct sched_entity *se); +static __always_inline void avg_kcs_vruntime_update(struct cfs_rq *cfs_rq,= s64 delta); =20 /************************************************************** * Scheduling class tree data structure manipulation methods: @@ -630,6 +633,7 @@ avg_vruntime_add(struct cfs_rq *cfs_rq, struct sched_en= tity *se) =20 cfs_rq->avg_vruntime +=3D key * weight; cfs_rq->avg_load +=3D weight; + avg_kcs_vruntime_add(cfs_rq, se); } =20 static void @@ -640,6 +644,7 @@ avg_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_en= tity *se) =20 cfs_rq->avg_vruntime -=3D key * weight; cfs_rq->avg_load -=3D weight; + avg_kcs_vruntime_sub(cfs_rq, se); } =20 static inline @@ -649,6 +654,7 @@ void avg_vruntime_update(struct cfs_rq *cfs_rq, s64 del= ta) * v' =3D v + d =3D=3D> avg_vruntime' =3D avg_runtime - d*avg_load */ cfs_rq->avg_vruntime -=3D cfs_rq->avg_load * delta; + avg_kcs_vruntime_update(cfs_rq, delta); } =20 /* @@ -6720,6 +6726,58 @@ __always_inline void sched_notify_critical_section_e= xit(void) current->se.kernel_cs_count--; } =20 +static inline int se_in_kernel(struct sched_entity *se) +{ + return se->kernel_cs_count; +} + +/* + * Same as avg_vruntime_add() except avg_kcs_vruntime_add() only adjusts t= he avg_kcs_vruntime + * and avg_kcs_load of kernel mode preempted entity when it joins the rbtr= ee. + */ +static __always_inline void avg_kcs_vruntime_add(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) +{ + unsigned long weight; + s64 key; + + if (!se_in_kernel(se)) + return; + + weight =3D scale_load_down(se->load.weight); + key =3D entity_key(cfs_rq, se); + + cfs_rq->avg_kcs_vruntime +=3D key * weight; + cfs_rq->avg_kcs_load +=3D weight; +} + +/* + * Same as avg_vruntime_sub() except avg_kcs_vruntime_sub() only adjusts t= he avg_kcs_vruntime + * and avg_kcs_load of kernel mode preempted entity when it leaves the rbt= ree. + */ +static __always_inline void avg_kcs_vruntime_sub(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) +{ + unsigned long weight; + s64 key; + + if (!se_in_kernel(se)) + return; + + weight =3D scale_load_down(se->load.weight); + key =3D entity_key(cfs_rq, se); + + cfs_rq->avg_kcs_vruntime -=3D key * weight; + cfs_rq->avg_kcs_load -=3D weight; +} + +/* + * Same as avg_vruntime_update() except it adjusts avg_kcs_vruntime based = on avg_kcs_load + * when min_vruntime of the cfs_rq changes. + */ +static __always_inline void avg_kcs_vruntime_update(struct cfs_rq *cfs_rq,= s64 delta) +{ + cfs_rq->avg_kcs_vruntime -=3D cfs_rq->avg_kcs_load * delta; +} + #ifdef CONFIG_NO_HZ_FULL /* called from pick_next_task_fair() */ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct = *p) @@ -6792,6 +6850,10 @@ bool cfs_task_bw_constrained(struct task_struct *p) __always_inline void sched_notify_critical_section_entry(void) {} __always_inline void sched_notify_critical_section_exit(void) {} =20 +static __always_inline void avg_kcs_vruntime_add(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) {} +static __always_inline void avg_kcs_vruntime_sub(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) {} +static __always_inline void avg_kcs_vruntime_update(struct cfs_rq *cfs_rq,= s64 delta) {} + #endif /* CONFIG_CFS_BANDWIDTH */ =20 #if !defined(CONFIG_CFS_BANDWIDTH) || !defined(CONFIG_NO_HZ_FULL) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index ab16d3d0e51c..22567d236f82 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -658,6 +658,12 @@ struct cfs_rq { s64 avg_vruntime; u64 avg_load; =20 +#ifdef CONFIG_CFS_BANDWIDTH + /* EEVDF stats of entities preempted in kernel mode */ + s64 avg_kcs_vruntime; + u64 avg_kcs_load; +#endif + u64 min_vruntime; #ifdef CONFIG_SCHED_CORE unsigned int forceidle_seq; --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2081.outbound.protection.outlook.com [40.107.92.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A4FA1EB188 for ; Thu, 20 Feb 2025 09:35:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.81 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044136; cv=fail; b=BQuLeR9R+g2rbcUlhJ4FymNi29H8QSS+NT6FVE743nm7alTMRNZ1zq+/rzALkZ7gi422ZZV+kGHT9TToQPxsG9GMAfF4qRYdrGoSvJYACbw1Y/qLCt/eMx6qhLRI6Otl1gaTlMLvA6CrWa17jmwPYRCL9/sBe7ZNcUYgOyCF/Ac= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044136; c=relaxed/simple; bh=GCFiVpZ/z4C+JvopM7kOFTVVu+xLGhSyVX+GUz4m2OM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=psnLRVHF6h1WumDR4MXMRPzPR/NMGuYM774i2paVpFAxb7+FLqITyr6eOTlY1ppkgKQYmkXm+kLCSV2NnXCmJKnElqS5PhcJiwb8J2vdv734I7P3CZW92oB88P4F+7mQ3TA610Enlks6X7LbjHQx1+D1f+JOvmjFU6ImS5NG/AU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=QhXceTQ6; arc=fail smtp.client-ip=40.107.92.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="QhXceTQ6" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vOXkZn/SHjhpTNvk/9Wv2fYUPnALhhwPlzIIK3KiwveVQS9bMvHxsn5xp9Ohxk2MlbGr0SS9na+fmtXOS9azH751/tcxMzbLShhH2fexMPY7Nsou8yXzcl3LTme36so0ns5j1TruVtrv8apooxuLDDYJMh8ExiMNdWrOJSwdK/gikIX0FAoCd9H6l3qHTPFQQmDOZDI0+IlWQzZk8+aLCl7q08dNA2wvjWY7sX6rtaTvL/moHsZcceAAUEP/gBEf11lmGI0Ai+P49KxLkU2l4Ri+1Bf7MRZI5ej2Dan8gPA4Yt6fIj6Q98c1aJ0XJUaFzmgglvEx3p/2lfqExLlNyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=X0T1MRunkRoyKdShMPDMeq1UnsU+Q8JfvCgI6ycNwe4=; b=rXCIXE6/2nyA9aZVPC3umNAT+jP2oStzpEPNhIhnaheXmvyCTyJlehyjwYMRWbFJabOiItjaFxlsbjbFsR6/68Dgk+D7Ic8PAG7+D1Vw/EZODDMcmoUud9K7Sxo2kk2wmmpFuRZarxFWKVcVDvxlqrfNqOniR5rTp9F9wjI+CbG6eg5LV8BdTjnqAckDIm/BXfJrPgTLlow1yV+XOSpYzRc5TLb6+xMs1oimAxfu84qoJuxyz0+NlnMIHwzf2Avsm90tgSsZO/8KllYZGmNOQASVYhTxT+S1JYfMGjLJ/HUIec9eCW2QFJAPigE6YDGR4Oi3ksQOqBX3azSPDP5X7w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X0T1MRunkRoyKdShMPDMeq1UnsU+Q8JfvCgI6ycNwe4=; b=QhXceTQ628XGHrAylEnoZGYx8Tu8VT94RJ7MGR6sRCAc3nvPdEWZjF/jxf9gZBWvphTQq3VfJX34hnsVHh6ZjhauuhWEpTzo7IBrsQc7eKdYKvahxu63EYUCysAY0KJUZJBBFLhlYK1G4ALIDlmOhzc/kWHM5ZlwrY3clFqEZ7I= Received: from DS0PR17CA0015.namprd17.prod.outlook.com (2603:10b6:8:191::23) by CH1PPF2D39B31FF.namprd12.prod.outlook.com (2603:10b6:61f:fc00::60a) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.16; Thu, 20 Feb 2025 09:35:30 +0000 Received: from DS3PEPF000099E1.namprd04.prod.outlook.com (2603:10b6:8:191:cafe::3b) by DS0PR17CA0015.outlook.office365.com (2603:10b6:8:191::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.14 via Frontend Transport; Thu, 20 Feb 2025 09:35:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099E1.mail.protection.outlook.com (10.167.17.196) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:35:30 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:35:21 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 06/22] sched/fair: Propagate the min_vruntime of kernel mode preempted entity Date: Thu, 20 Feb 2025 09:32:41 +0000 Message-ID: <20250220093257.9380-7-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099E1:EE_|CH1PPF2D39B31FF:EE_ X-MS-Office365-Filtering-Correlation-Id: 21dfb24f-206d-4044-e499-08dd5191eab9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|7416014|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?SS5tJfmnCMPcbmV8LZFbJRSLhSSpMCYHzwofrfgbZ7LP2ZE+ZbYrErH4/brY?= =?us-ascii?Q?CLsFAp6MKTt9RorVJQpmDMqnf9i97gaiw5AtQc7ovVKDQSPal7YBQPzWpyeP?= =?us-ascii?Q?sJGDgpH/7wAAEbgXDcxPm302yYAPFFWgYlQutCM9QjXJRY5LQQMTlWLAdNF5?= =?us-ascii?Q?QJ4++3vWZTGBpiRxFfDGreez+kpxzE+uTcDwIeycoUTyf+1PRSmYurW0azDE?= =?us-ascii?Q?EinITQOBg37D1JauRIochMWsSXtbGCn8yeak10m1U75HPJmU3KzRCxfYWyhF?= =?us-ascii?Q?ypwB7AmTURUZvk/i/8HKFFcyrhY4xom74cHZkriltCRE5cA4HeTqLJfz6VTZ?= =?us-ascii?Q?mdDNd/B7NTAXPh6adw1CvCr+jXtMFLCOF5ujrlwDXjb5wZe2oB7KMS1l5wC3?= =?us-ascii?Q?4dQ0MJ+aP+bmo3nFsGHkaLdFNYkOBAZPnYN/MX2K8wYJAniSvUVTDBQiuF0E?= =?us-ascii?Q?8mjYsdAjP6TtZXCj0sb3oSeccWeE2RVfi/m7Ewxha23zKpjDYwAjcCiW8L70?= =?us-ascii?Q?WlRlfoSMlZIlRJPlrF7FU4yOUnnvTuzJSnzJFLBKyiOmFzrT+HHk4BiTIVuL?= =?us-ascii?Q?LA0DeL5DJgR78NulFKJGN98JSRKySBHUvfn53ETr2LfQSp8erjcFbz2jRxjN?= =?us-ascii?Q?wzqRUVjaNjYdmSwjeVHa0/LN8cs9k8irOVpMSblJYjQB92st+sYLtozI1sqG?= =?us-ascii?Q?GzFaempYbuTn/17GrYDM9WIabsIMRoSlk8fZTn/rnbSG0QUMnveMOX/YQyeb?= =?us-ascii?Q?wsXQbMTB5ZrbcOvnSFNOyiBe1yz8j9CINpugOrge/vWZDrzFlM8AtFols7uJ?= =?us-ascii?Q?Nmohxm9sZ2cEu7gjkWDBG2pww1UlgLWqsPbR7woWeKmbZolflFkiYL90sMI7?= =?us-ascii?Q?KHncjbnIpUj3lUIxsJs/6UAKAHCA77lanz9qX178/QLL08t9tVlRJmb4AIXq?= =?us-ascii?Q?iTA8t4ACHC/J0FDoFRC43qDou9MpdCH3KXGD5MWMImukmNfjlwMaQvV1mBt8?= =?us-ascii?Q?N4IHLj/qzu9JVVnvf1h7lG80ffQ1kB512MDmf2Zt0J5zMKZU7oYKSac6shUG?= =?us-ascii?Q?1gJXLp2phHjfqmBR2e3dO2PNFdCNwZYFn1IFynpzypheEGOPoGXNOetSRqIc?= =?us-ascii?Q?dNuW+LM5ynecPvOc7kjCgeu+9shoViR8HFmBAdbhpnQU6B7QQ/B8k6UBCp2Y?= =?us-ascii?Q?FjX996JUmCnNtJdpy0LaGdmU+ZjVLGL2l2rnVfn9Exr9usb16vNP9shn311Q?= =?us-ascii?Q?MmV3hiqiq5kOWdGnPz/9X6abEGTqhbpDtS17h5LT28OgsLLDbDWLClJqW480?= =?us-ascii?Q?pQTv4wmUoF9U09XLAQ3A4TvZUMw8DEUsuTgY17QsIE1CVdJ3zD0rfy2cyZKV?= =?us-ascii?Q?urIhpISdeB5kJIblgl7v24pKlbh3MiQEmZFpk48leFlQNWk6wUfJKIlRW3g4?= =?us-ascii?Q?VyTYHKOO0+7295vVkRQUxavqub/RqajB7lW99sMfsAf199krUyylgI3VzGZS?= =?us-ascii?Q?TzUxHznHPKG8Nc4=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(7416014)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:35:30.5856 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 21dfb24f-206d-4044-e499-08dd5191eab9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099E1.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH1PPF2D39B31FF Content-Type: text/plain; charset="utf-8" Propagate the min_vruntime of the kernel mode preempted entity to the root of the cfs_rq's rbtree. This will be soon used to pick amongst the kernel mode entities on a throttled hierarchy using the similar min-heap approach that pick_eevdf() currently implements. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 6 ++++++ kernel/sched/fair.c | 47 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 52 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 63f3f235a5c1..4bb7e45758f4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -593,6 +593,12 @@ struct sched_entity { */ int kernel_cs_count; /* hole */ + + /* + * min_vruntime of the kernel mode preempted entities + * in the subtree of this sched entity. + */ + s64 min_kcs_vruntime; #endif /* CONFIG_CFS_BANDWIDTH */ #endif /* CONFIG_FAIR_GROUP_SCHED */ =20 diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index cbb7a227afe7..ba1bd60ce433 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -828,6 +828,9 @@ static inline void __min_slice_update(struct sched_enti= ty *se, struct rb_node *n } } =20 +static __always_inline void init_se_kcs_stats(struct sched_entity *se); +static inline bool min_kcs_vruntime_update(struct sched_entity *se); + /* * se->min_vruntime =3D min(se->vruntime, {left,right}->min_vruntime) */ @@ -836,6 +839,7 @@ static inline bool min_vruntime_update(struct sched_ent= ity *se, bool exit) u64 old_min_vruntime =3D se->min_vruntime; u64 old_min_slice =3D se->min_slice; struct rb_node *node =3D &se->run_node; + bool kcs_stats_unchanged =3D min_kcs_vruntime_update(se); =20 se->min_vruntime =3D se->vruntime; __min_vruntime_update(se, node->rb_right); @@ -846,7 +850,8 @@ static inline bool min_vruntime_update(struct sched_ent= ity *se, bool exit) __min_slice_update(se, node->rb_left); =20 return se->min_vruntime =3D=3D old_min_vruntime && - se->min_slice =3D=3D old_min_slice; + se->min_slice =3D=3D old_min_slice && + kcs_stats_unchanged; } =20 RB_DECLARE_CALLBACKS(static, min_vruntime_cb, struct sched_entity, @@ -858,6 +863,7 @@ RB_DECLARE_CALLBACKS(static, min_vruntime_cb, struct sc= hed_entity, static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *s= e) { avg_vruntime_add(cfs_rq, se); + init_se_kcs_stats(se); se->min_vruntime =3D se->vruntime; se->min_slice =3D se->slice; rb_add_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline, @@ -6778,6 +6784,39 @@ static __always_inline void avg_kcs_vruntime_update(= struct cfs_rq *cfs_rq, s64 d cfs_rq->avg_kcs_vruntime -=3D cfs_rq->avg_kcs_load * delta; } =20 +static __always_inline void init_se_kcs_stats(struct sched_entity *se) +{ + /* + * With the introduction of EEVDF, the vruntime of entities can go negati= ve when + * a lagging entity joins a runqueue with avg_vruntime < vlag. Use LLONG_= MAX as + * the upper bound to differentiate the case where no kernel mode preempt= ed + * entities are queued on the subtree. + */ + se->min_kcs_vruntime =3D (se_in_kernel(se)) ? se->vruntime : LLONG_MAX; +} + +static inline void __min_kcs_vruntime_update(struct sched_entity *se, stru= ct rb_node *node) +{ + if (node) { + struct sched_entity *rse =3D __node_2_se(node); + + if (rse->min_kcs_vruntime < se->min_kcs_vruntime) + se->min_kcs_vruntime =3D rse->min_kcs_vruntime; + } +} + +static inline bool min_kcs_vruntime_update(struct sched_entity *se) +{ + u64 old_min_kcs_vruntime =3D se->min_kcs_vruntime; + struct rb_node *node =3D &se->run_node; + + init_se_kcs_stats(se); + __min_kcs_vruntime_update(se, node->rb_right); + __min_kcs_vruntime_update(se, node->rb_left); + + return se->min_kcs_vruntime =3D=3D old_min_kcs_vruntime; +} + #ifdef CONFIG_NO_HZ_FULL /* called from pick_next_task_fair() */ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct = *p) @@ -6853,6 +6892,12 @@ __always_inline void sched_notify_critical_section_e= xit(void) {} static __always_inline void avg_kcs_vruntime_add(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) {} static __always_inline void avg_kcs_vruntime_sub(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) {} static __always_inline void avg_kcs_vruntime_update(struct cfs_rq *cfs_rq,= s64 delta) {} +static __always_inline void init_se_kcs_stats(struct sched_entity *se) {} + +static inline bool min_kcs_vruntime_update(struct sched_entity *se) +{ + return true; +} =20 #endif /* CONFIG_CFS_BANDWIDTH */ =20 --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2071.outbound.protection.outlook.com [40.107.94.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 016921EDA08 for ; Thu, 20 Feb 2025 09:35:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.71 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044158; cv=fail; b=Xdys68/sNGjbVlL2ieVBfTS551ySlbix4SFZN9q0TrjAXZW/0bX1GKDyUfBMH8/b2PND1qycCtg4SMshBNUtU76DAThKs0AoNmzf7FW71DkZIE2aRoTKAwUSbaj7LZqb569/N7jrtl8TMG5h57KAdGiSWoXj7AXO1FLlxlKekhc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044158; c=relaxed/simple; bh=gEJNMxBDe3sWUsSn19wU+R+KS/4ddbI8seGEJVwGxpU=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qkYC11PY0Hr0f4xS8+KNOBVghLywQXmiZyGS3cGPD09XC8EtBVxR3ThX+aj2UHwMKQO7V35vZ03XtcOgwD7ce4BzzzNvpQY+0jdhbYwjuxDIrFdsveEprc76DL5t+mqdtizlTX4a+O1hctU9Z7xx2DzlxhoyGzr5Nh7zdyMP99M= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=OI+HP+dp; arc=fail smtp.client-ip=40.107.94.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="OI+HP+dp" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=irl6I4I4T7iSK4T18wn2tFDRe1VvZHoJyRJcCHYv/p7PMEhIR2pSUiuhKWOSLNv4njX1r+T1ebIQQmpOV5SNfakU2MSYXxw0bb9FCTdUb6Qhzb0xazstwYc7t5uvWHhZuJXVJR4cOBZARxf+aNpV5a7WLUpz9DtfSaazN1ZoqWwWEq2FADLtMktpRLbfnTG3dKs/JOiHhU/k/gchOScsLLtvpJMo8N+YE6hvP2qB4e+Egz5J4KyNtTEJvNFO21ptspwicIhJLGoj+tFMTQt3tc5DiGC5HG8+Ja/+FIKlsKS6WV60xYceZxE1Rp2iZFGotn/OvjgE4HOFPrSlCZeIPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RHWX5d7o1X/WWHhVJpbpRN159IorwOAvTto4D1H/Nho=; b=ftEEEIcEX1CwUUz0pXZlbiZDsLvNgkPL0QneAWqdXyQZP1NVFO2+lpUb/gIkUS4z1PG9dSSuuFd7+oCxvtS/27fVQ2fcmGFLCNG5imxBGdISNt4291r6Bgc2Y4cWI1yupjiGWzzXFbxFAt/MGKabiHQtLzMyR7hJnfwt7Dwmh+uKI0PgzHtIid5uzqWXNa6Gl2T81FU2eXv2EKnPjDg1SFi6qasxDZA+y6dH9cAcuj3UHRJMY+xzAYGSeoMpn193W1zEDrUa5DgHtQX0WQ2iapb8MYBl+l3W0v+MOFZNiWGglONMVPzcj++F88Jl2agPYa39HmESsy4SDX5r6uTTWQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RHWX5d7o1X/WWHhVJpbpRN159IorwOAvTto4D1H/Nho=; b=OI+HP+dp26RCEoOGRnpR5e4psFESXr7xvbSuGTaYyJlgu/Dt1AuW5aJQZAiTCfbzLPzv+ILDppYB43taKNN+Ixq16sBunnphRbISJJciEkofiwBGZRTC5GEoyksqvqGLfqRjcR59MBmkigbN/a8biNaD7ro8eUGvoy7woLSI11I= Received: from DS7PR03CA0204.namprd03.prod.outlook.com (2603:10b6:5:3b6::29) by MN0PR12MB6001.namprd12.prod.outlook.com (2603:10b6:208:37d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Thu, 20 Feb 2025 09:35:52 +0000 Received: from DS3PEPF000099DF.namprd04.prod.outlook.com (2603:10b6:5:3b6:cafe::e4) by DS7PR03CA0204.outlook.office365.com (2603:10b6:5:3b6::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.16 via Frontend Transport; Thu, 20 Feb 2025 09:35:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DF.mail.protection.outlook.com (10.167.17.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:35:51 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:35:43 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 07/22] sched/fair: Propagate preempted entity information up cgroup hierarchy Date: Thu, 20 Feb 2025 09:32:42 +0000 Message-ID: <20250220093257.9380-8-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DF:EE_|MN0PR12MB6001:EE_ X-MS-Office365-Filtering-Correlation-Id: 26d1426f-96a8-4178-4a7a-08dd5191f767 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?EHIYfl/HanvZQ83qOmPsSvqXheiF/0AFRfjtnxUz8kl08btcbLsAGkwXSX9l?= =?us-ascii?Q?jlQ5blFOZpGK33DFXOeOmYh/mFM/Bg2qLhpOEc9d1Hj0U3PzxRzvJoa0zjMN?= =?us-ascii?Q?gCGryXiY40s1Y2eWBbN2eNlDaCcVYwsfKGU2S1qmBiOIXZs8dFAyqt4zJzIJ?= =?us-ascii?Q?BiBA1keQWyzhTfociuCRCEyro4ZeRAWeGEifBtGZxTqKkzyr4qtv1zskr1xA?= =?us-ascii?Q?Ob4i3XSXkE0Rk9YKk+tKn4VL3nZu4bSU1/bK8Gkvnr0hZWzcmjquaxT4Aks8?= =?us-ascii?Q?xFHORi7e442hI8t6gI26Si/2cS66Y443FEpHgDHZS3/C5ulQiWX6gX1eZFN+?= =?us-ascii?Q?3ieyh0iPM5e7mrgN7j5JaEai2EYJwavg9yJmBJHjIGH6f2+K3gNH67DKMT3W?= =?us-ascii?Q?AyuLjoAD1YP+mpoO5VNlXpstPGdu3PRPOanqgs2sBpQq8mcVyvbMfrn41h5A?= =?us-ascii?Q?MuEh+vx0ab3DJjlMU6CXsdoOfFKvjkmJity1wnRMYT3TTIIJeX+3fEgXFyTz?= =?us-ascii?Q?MaommAfQyItPtI/hvfv2HDwG4659tRvi2bv+6LByUrCGRfwPgs8Vhc4itMVH?= =?us-ascii?Q?SGwOWhOefNegOORmHc/rDOTea3GDXqHN+CfWxpEXv2HU91vasJrMklNAYIM7?= =?us-ascii?Q?UV7dzrHwdu7OdHc0mr5aodS5yl/rtHyl+3jNaw0C4pHDoWco6VmXOg+NYNLf?= =?us-ascii?Q?GmTpj7whQ0ibMKmEcyYoVCoZf2Yjx2tKPKDJYg5YCOFTEwYG18sv07Nl//Ez?= =?us-ascii?Q?AcVpj2E5wOElyGiHsQoyCshnTuK024LvSObMx/58s/cAUm243ZONvKop+fvC?= =?us-ascii?Q?A/zlfmleSUkdaUQZwBgdS4vXKFZk4pGObB6lVARsyzh9nXKhyDV12Xy16J7C?= =?us-ascii?Q?ugf+48qL+qA5/iTFUe1qzVAg+uyt+C1bLrA5ARaAEK3P/YNMmVW/aMe6uyPx?= =?us-ascii?Q?B/HlFChuzrSOVjTDu4pgDrc587Bv7V/8SHwexRg6FEVH/agcJZJgoBCm4Byq?= =?us-ascii?Q?XiZeSVCUB6hPD6SHXwFIkASZe13/ZZo0laQOZzSiqBSSPLNWSGAX3L6Tnnm+?= =?us-ascii?Q?9urpZ3NckQujc3ABcj+DdHGXYDnD8UpYzgwiiw5DlpeP3wItbKMlrrCChI/m?= =?us-ascii?Q?2IBeAsVa7soCh5qb7TBvKBs4YSqK0PBWN00b6EQzo270Hfvf8B684zr+AKrr?= =?us-ascii?Q?nHJPF66x6el1q0sXvoWMa7l5utI1QkzYZc6yr36kml/TTsJWSJgvaPYvAqUV?= =?us-ascii?Q?E75j0jNhbnMUs67w0NVaTb88awlCl8yIB8UKabf/lONoLud2Jmhf3xSb4nfB?= =?us-ascii?Q?u+OJdllpVhX0VrQxtUinr0d25SzjvGFIPxmbyVQOitN6nFlAmv5mheHOYtRJ?= =?us-ascii?Q?Lg7j4pEqvsGyE4wj7T11cCT1GraUpoDSs7Vw6Cq/TdbcjQSh0mbHPecK50XB?= =?us-ascii?Q?j5CpHMToVxLViUroELXXBRUsatM2qXCinu/Mnq4GLg12ALGSnCXSNxb3iTl1?= =?us-ascii?Q?C+6KX0drdtLXfbM=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:35:51.8390 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 26d1426f-96a8-4178-4a7a-08dd5191f767 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DF.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6001 Content-Type: text/plain; charset="utf-8" Use "kernel_cs_count" in cfs_rq's sched entity to track the number of preempted entities queued on the cfs_rq. Since the current task can frequently switch in and out of kernel mode, "cfs_rq->curr" and the whole sched entity hierarchy of the current task is treated special. This is similar to "runnable_sum" except "kernel_cs_count" does not have a corresponding count in the in struct cfs_rq. The counts are managed at enqueue, dequeue, and task pick boundaries since the entities on rbtree have a stable "kernel_cs_count". Use min_vruntime_cb_propagate() generated from RB_DECLARE_CALLBACKS() to propagate the adjustments to the root of the rbtree. Since propagations require the state of the task being enqueued / dequeued / put / set, expose se_in_kernel() to generic code to streamline the propagation. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 7 ++- kernel/sched/fair.c | 121 ++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 124 insertions(+), 4 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 4bb7e45758f4..48115de839a7 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -589,7 +589,12 @@ struct sched_entity { * - For task: It represents if the task is currently * running in kernel mode. It is always 0 or 1. * - * TODO: Describe for sched_entity when implementing. + * - For cfs_rq: It represents the sum of the + * kernel_cs_count of the entities queued below it + * except for the "current". Since the current task can + * frequently switch in and out of kernel mode, its + * hierarchy is treated special and its contribution + * is accounted at key decision points. */ int kernel_cs_count; /* hole */ diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ba1bd60ce433..9f28624e4442 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6732,7 +6732,7 @@ __always_inline void sched_notify_critical_section_ex= it(void) current->se.kernel_cs_count--; } =20 -static inline int se_in_kernel(struct sched_entity *se) +static __always_inline int se_in_kernel(struct sched_entity *se) { return se->kernel_cs_count; } @@ -6817,6 +6817,76 @@ static inline bool min_kcs_vruntime_update(struct sc= hed_entity *se) return se->min_kcs_vruntime =3D=3D old_min_kcs_vruntime; } =20 +static inline void account_kcs_enqueue(struct cfs_rq *gcfs_rq, bool in_ker= nel) +{ + struct sched_entity *se; + struct cfs_rq *cfs_rq; + + if (!in_kernel) + return; + + se =3D gcfs_rq->tg->se[cpu_of(rq_of(gcfs_rq))]; + if (!se) + return; + + cfs_rq =3D cfs_rq_of(se); + se->kernel_cs_count++; + + /* se has transitioned into a kernel mode preempted entity */ + if (se->kernel_cs_count =3D=3D 1 && se !=3D cfs_rq->curr && se->on_rq) { + /* + * Must be done after "kernel_cs_count" has been + * incremented since avg_kcs_vruntime_add() only + * adjusts the stats if it believes the entity is in + * the kernel mode. + */ + avg_kcs_vruntime_add(cfs_rq, se); + + /* Propagate min_kcs_vruntime_update() till rb_root */ + min_vruntime_cb_propagate(&se->run_node, NULL); + } + + /* Sanity check */ + SCHED_WARN_ON(se->kernel_cs_count > gcfs_rq->h_nr_queued); +} + +static inline void account_kcs_dequeue(struct cfs_rq *gcfs_rq, bool in_ker= nel) +{ + struct sched_entity *se; + struct cfs_rq *cfs_rq; + bool transition_out; + + if (!in_kernel) + return; + + se =3D gcfs_rq->tg->se[cpu_of(rq_of(gcfs_rq))]; + if (!se) + return; + + cfs_rq =3D cfs_rq_of(se); + transition_out =3D se->kernel_cs_count =3D=3D 1; + + /* + * Discount the load and avg_kcs_vruntime contribution if the + * entity is transitioning out. Must be done before + * "kernel_cs_count" is decremented as avg_kcs_vruntime_sub() + * should still consider it to be in kernel mode to adjust + * the stats. + */ + if (transition_out && se !=3D cfs_rq->curr && se->on_rq) + avg_kcs_vruntime_sub(cfs_rq, se); + + se->kernel_cs_count--; + + /* Propagate min_kcs_vruntime_update() till rb_root */ + if (transition_out && se !=3D cfs_rq->curr && se->on_rq) + min_vruntime_cb_propagate(&se->run_node, NULL); + + /* Sanity checks */ + SCHED_WARN_ON(se->kernel_cs_count > gcfs_rq->h_nr_queued); + SCHED_WARN_ON(se->kernel_cs_count < 0); +} + #ifdef CONFIG_NO_HZ_FULL /* called from pick_next_task_fair() */ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct = *p) @@ -6889,6 +6959,11 @@ bool cfs_task_bw_constrained(struct task_struct *p) __always_inline void sched_notify_critical_section_entry(void) {} __always_inline void sched_notify_critical_section_exit(void) {} =20 +static __always_inline int se_in_kernel(struct sched_entity *se) +{ + return false; +} + static __always_inline void avg_kcs_vruntime_add(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) {} static __always_inline void avg_kcs_vruntime_sub(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) {} static __always_inline void avg_kcs_vruntime_update(struct cfs_rq *cfs_rq,= s64 delta) {} @@ -6899,6 +6974,9 @@ static inline bool min_kcs_vruntime_update(struct sch= ed_entity *se) return true; } =20 +static inline void account_kcs_enqueue(struct cfs_rq *gcfs_rq, bool in_ker= nel) {} +static inline void account_kcs_dequeue(struct cfs_rq *gcfs_rq, bool in_ker= nel) {} + #endif /* CONFIG_CFS_BANDWIDTH */ =20 #if !defined(CONFIG_CFS_BANDWIDTH) || !defined(CONFIG_NO_HZ_FULL) @@ -7056,6 +7134,8 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) { struct cfs_rq *cfs_rq; struct sched_entity *se =3D &p->se; + /* A task on CPU has its in-kernel stats discounted */ + bool task_in_kernel =3D !task_on_cpu(rq, p) && se_in_kernel(se); int h_nr_idle =3D task_has_idle_policy(p); int h_nr_runnable =3D 1; int task_new =3D !(flags & ENQUEUE_WAKEUP); @@ -7110,6 +7190,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) cfs_rq->h_nr_runnable +=3D h_nr_runnable; cfs_rq->h_nr_queued++; cfs_rq->h_nr_idle +=3D h_nr_idle; + account_kcs_enqueue(cfs_rq, task_in_kernel); =20 if (cfs_rq_is_idle(cfs_rq)) h_nr_idle =3D 1; @@ -7136,6 +7217,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) cfs_rq->h_nr_runnable +=3D h_nr_runnable; cfs_rq->h_nr_queued++; cfs_rq->h_nr_idle +=3D h_nr_idle; + account_kcs_enqueue(cfs_rq, task_in_kernel); =20 if (cfs_rq_is_idle(cfs_rq)) h_nr_idle =3D 1; @@ -7196,6 +7278,7 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags) bool task_sleep =3D flags & DEQUEUE_SLEEP; bool task_delayed =3D flags & DEQUEUE_DELAYED; struct task_struct *p =3D NULL; + bool task_in_kernel =3D false; int h_nr_idle =3D 0; int h_nr_queued =3D 0; int h_nr_runnable =3D 0; @@ -7205,6 +7288,8 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags) if (entity_is_task(se)) { p =3D task_of(se); h_nr_queued =3D 1; + /* A task on CPU has its in-kernel stats discounted */ + task_in_kernel =3D !task_on_cpu(rq, p) && se_in_kernel(se); h_nr_idle =3D task_has_idle_policy(p); if (task_sleep || task_delayed || !se->sched_delayed) h_nr_runnable =3D 1; @@ -7226,6 +7311,7 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags) cfs_rq->h_nr_runnable -=3D h_nr_runnable; cfs_rq->h_nr_queued -=3D h_nr_queued; cfs_rq->h_nr_idle -=3D h_nr_idle; + account_kcs_dequeue(cfs_rq, task_in_kernel); =20 if (cfs_rq_is_idle(cfs_rq)) h_nr_idle =3D h_nr_queued; @@ -7267,6 +7353,7 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags) cfs_rq->h_nr_runnable -=3D h_nr_runnable; cfs_rq->h_nr_queued -=3D h_nr_queued; cfs_rq->h_nr_idle -=3D h_nr_idle; + account_kcs_dequeue(cfs_rq, task_in_kernel); =20 if (cfs_rq_is_idle(cfs_rq)) h_nr_idle =3D h_nr_queued; @@ -9029,6 +9116,8 @@ pick_next_task_fair(struct rq *rq, struct task_struct= *prev, struct rq_flags *rf */ if (prev !=3D p) { struct sched_entity *pse =3D &prev->se; + bool prev_in_kernel =3D task_on_rq_queued(prev) && se_in_kernel(pse); + bool next_in_kernel =3D se_in_kernel(se); struct cfs_rq *cfs_rq; =20 while (!(cfs_rq =3D is_same_group(se, pse))) { @@ -9036,18 +9125,35 @@ pick_next_task_fair(struct rq *rq, struct task_stru= ct *prev, struct rq_flags *rf int pse_depth =3D pse->depth; =20 if (se_depth <=3D pse_depth) { - put_prev_entity(cfs_rq_of(pse), pse); + cfs_rq =3D cfs_rq_of(pse); + + account_kcs_enqueue(cfs_rq, prev_in_kernel); + put_prev_entity(cfs_rq, pse); pse =3D parent_entity(pse); } if (se_depth >=3D pse_depth) { - set_next_entity(cfs_rq_of(se), se); + cfs_rq =3D cfs_rq_of(se); + + set_next_entity(cfs_rq, se); se =3D parent_entity(se); + account_kcs_dequeue(cfs_rq, next_in_kernel); } } =20 put_prev_entity(cfs_rq, pse); set_next_entity(cfs_rq, se); =20 + if (prev_in_kernel !=3D next_in_kernel) { + for_each_sched_entity(se) { + cfs_rq =3D cfs_rq_of(se); + + if (prev_in_kernel) + account_kcs_enqueue(cfs_rq, prev_in_kernel); + else + account_kcs_dequeue(cfs_rq, next_in_kernel); + } + } + __set_next_task_fair(rq, p, true); } =20 @@ -9114,10 +9220,17 @@ void fair_server_init(struct rq *rq) static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, st= ruct task_struct *next) { struct sched_entity *se =3D &prev->se; + /* + * When next is NULL, it is a save-restore operation. If the task is no + * longer queued on the rq, the stats have been already discounted at + * pick and should not be adjusted here. + */ + bool task_in_kernel =3D next && task_on_rq_queued(prev) && se_in_kernel(s= e); struct cfs_rq *cfs_rq; =20 for_each_sched_entity(se) { cfs_rq =3D cfs_rq_of(se); + account_kcs_enqueue(cfs_rq, task_in_kernel); put_prev_entity(cfs_rq, se); } } @@ -13424,11 +13537,13 @@ static void __set_next_task_fair(struct rq *rq, s= truct task_struct *p, bool firs static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool = first) { struct sched_entity *se =3D &p->se; + bool task_in_kernel =3D !task_on_cpu(rq, p) && se_in_kernel(se); =20 for_each_sched_entity(se) { struct cfs_rq *cfs_rq =3D cfs_rq_of(se); =20 set_next_entity(cfs_rq, se); + account_kcs_dequeue(cfs_rq, task_in_kernel); /* ensure bandwidth has been allocated on our new cfs_rq */ account_cfs_rq_runtime(cfs_rq, 0); } --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2076.outbound.protection.outlook.com [40.107.93.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF9861F03D4 for ; Thu, 20 Feb 2025 09:36:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.76 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044179; cv=fail; b=L8PHzlHcqU+DUNVFujvdyyTe3TEcXzpjtl5krO/gfQIA0OX+8E3N94C0yv7UohFDOIJutMioq5LPdM3ElbMshS+LqOe+oziG1KS+V+H6R8HQEJL9O3fpZN2O0CK0G6EOtlGHNSUOQZwoVnFN8NAYOpE6T9szkXXlkbNbE7O9KRU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044179; c=relaxed/simple; bh=p1LMJdt8iWYYnZE2IuN8pGcIx+K4RX1RZN0a8NXhk5U=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=SAHSIgB00hHliE0eCYHMdW2GmOE1/6WKZ1vR82L+tBgoyRdZJcEtjgSeeWr6CaeNWq1S7QcHbM4ZwFZ80QQX1Z1oCup9pVe5elHzWJEONh0gEgWmBcntetg4moncx8fAqQcRDzQ6LAtM0m+fwFNx6a6sVO+OBbzhflHb9wYj4UI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=M/ekUtoc; arc=fail smtp.client-ip=40.107.93.76 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="M/ekUtoc" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=JLgbFvVIUxER6zQh7fAGdY6wxGvrwu5h5iglXrx5k2Hoo2jNEdR7OUSHFqf2uvgs0/4feF7Iynf0Cn6S6nRxJe4EHt9Gy9QDPht2gqtrXNECawc63KzQxBIyA2MTV6xJYWG4DdaR6SXGsPrquXTV5y7XIGTbd9xRKh/2tjSVnNgbAmHIIidPAG4Jk1FvYGj8zx7oiwVy7syuXTtyOeDEBr0KwBg3zIYttFm1MgxBV/qKl+HULY69UMnB6JdbcLZxMPjI7Z40rs7KjC6+Xs4RwgDA/+55ytNP9EC80/hkbjOCbSeLidC6uKJWpdf1tbqdqXtBwRnKHo1WFCsBP5spUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wsyHr8OIJjB23ZLAsncS91MPiMVjz60/hNOFvMm07Xk=; b=xIWXaGKeBOmZlHPIBI1UniMI2RlJimBWaQ4MHpRgmcpqgqUSjmzUSA6TW52aYLUaOPy3ATXPcOWzbVeCqhdKdv4KcietIJi8HOpN+62HSIcJ+jcD95cZzzQevIz8GD+/2V4Qdf37iJ9niYem7HmuWhFSc2NwbGp9IoWvk0oQSo8xXCIFVc1A2gVmYumwn1Xmx0wE/9MnIZBYJKDBd1L+iwVQG1iUhN55TGXoz8890uzNhJ4t1XzSCzXnC3qvVWohII7D6G6LhmTNLNGKU3QOwpLu2klvCN7GUsfM1IMc58oOMoUXNKQAVz3uE+dRpFGYc+X/v41z7u3y/us4qL8ZPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wsyHr8OIJjB23ZLAsncS91MPiMVjz60/hNOFvMm07Xk=; b=M/ekUtoc5eJun7fLbMiS6unni5ZdMGBWT6tndU2Z2h+uk/jb2PzaWIhujomE5w2Z1chBJv39sn/hZI5apaoqTLtKrHTLnZGX6lagtNg7oOfQBiAi1RmQEhyHHgymon4VEKwfPdPjYPZUL2i7VhhKiBcYvrbaWDo3HcLKPM82ioQ= Received: from CH0PR13CA0007.namprd13.prod.outlook.com (2603:10b6:610:b1::12) by DS7PR12MB6093.namprd12.prod.outlook.com (2603:10b6:8:9e::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Thu, 20 Feb 2025 09:36:13 +0000 Received: from DS3PEPF000099E2.namprd04.prod.outlook.com (2603:10b6:610:b1:cafe::c7) by CH0PR13CA0007.outlook.office365.com (2603:10b6:610:b1::12) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.14 via Frontend Transport; Thu, 20 Feb 2025 09:36:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099E2.mail.protection.outlook.com (10.167.17.201) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:36:12 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:36:04 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 08/22] sched/fair: Allow pick_eevdf() to pick in-kernel entities on throttled hierarchy Date: Thu, 20 Feb 2025 09:32:43 +0000 Message-ID: <20250220093257.9380-9-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099E2:EE_|DS7PR12MB6093:EE_ X-MS-Office365-Filtering-Correlation-Id: 929b1135-e3d9-4546-4ea6-08dd519203d8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?mCq2HbXEpp2UCt2sBn7CkbhkwQbE6wCAtwM9BP+gfXmFSDArZ+cXqM44Tst1?= =?us-ascii?Q?PuMpKCRF5e0YPK7+aUZ2knFj1iC4OhZODlSWRcP3d4oG46jdfvYE950hVJ1O?= =?us-ascii?Q?qp3oyq3c2020sTIk6x/ma4Q9N9boElPX9U1bp2nw6HY+6+ancPXfP6EcMdDf?= =?us-ascii?Q?riSmlFW1/zxOeAuZX4sQ8V2igqb3H+t/XjwVOMZoqpSTtWSEoHbEFCjnOrfB?= =?us-ascii?Q?NKT4VPsSuxic/jmM2JLQb12shvM5PB6nvPx8X8S5E/AyThpdkHqEU+IxiE27?= =?us-ascii?Q?aZqkUH4jVLueNyGGy3no4fajYtrPomUjlHY5xjbSHWt0oBeym+niJepmFAKc?= =?us-ascii?Q?x1iAHluiEHUlRAGi+q/WJElbuZfX2szCDGxkpmTf7lDvI1BsMBggCjjAtwAx?= =?us-ascii?Q?xbtxpVrrmSfnP9WPJ518W8IhwRFPX0fXlWk61kubobdazQxaEmhtrq5yHBdf?= =?us-ascii?Q?nQR2YwZdoUENYLzMtMjjFp0chIOfACiWy8FvMZsXm2JbAHdyVu4gRnvf518h?= =?us-ascii?Q?2+42JAulbIbVL5F1JrywD4dsUnHJrlaiRl/VV3PY/vA6oqOlZ/NwDraaoRWo?= =?us-ascii?Q?9vbmVHx+6ym+U6GhxhKRt6KDfE/LPkYjrdBvr5waw+8M5ecSL/iv3YvZPCub?= =?us-ascii?Q?jaL3FseI0i4kZilBDw8f9w4Z2C7PAyuFk+CaI3JGcqQHVzk+twZAVSO6qgwQ?= =?us-ascii?Q?QsOfLYPXcXYWwhzr3nMfS9qsynxEZa3UA7hEgzSJJw01O2hdLnJOye9PK6+E?= =?us-ascii?Q?QAZ/dN4FaYiSRGpd7olqmuQKYrU1fL80OGuGeAcTON9osSds1mKQM44KAevS?= =?us-ascii?Q?s3zSvx3t1Q6zs5CeFr44XuhTqHl95BEhK5fakoHwyf/fO22w0tdtX04x31Gt?= =?us-ascii?Q?T1xBrFYOoKZaApF97+hfQETh9dRyrCyoRYm0hlBBFKcG8JnW8L7X+Y2nwoLi?= =?us-ascii?Q?sPjGSdDpuE11fOoELs+dPvBnOdzG10xJ8cjzRNiEWDqYWet/dF4bvQ/9P6vZ?= =?us-ascii?Q?ZUiw45AUHSejKLLBPoS01Drm8yyrInvc2mXBjheE0kTDWPa/qmT7qwXPEPeq?= =?us-ascii?Q?FwpC8lqVyQPwkHt/3KUN3VWYVHCeMT/221IdLsw0ETqan9CYottJFMhS1Wea?= =?us-ascii?Q?eJOMRo7VpOp+F1ZuKm8Xi9IdNJbwFn8+vwODGJ4W/7RgLzYGOolQaXW5F8gU?= =?us-ascii?Q?gJrxIDA4wCOOBTPv/YKq+0vDayTJISD2GWkgGITaVCtpNNkqx0hN7s+vyq+m?= =?us-ascii?Q?KdPQHRcojb87G5RLBv+gAfXwsgz6UDoxSVFtTKy5BlMh4pGKWsdVTwTJpny/?= =?us-ascii?Q?mK96R7B1o5SKqW/qoARR3H0ywn97EOUarN1MPCCyqPnE8TAGcFPigh11enQ/?= =?us-ascii?Q?9Kj0Kzji2Cu1z5I/hXIYDF4zlYxGsK3rl6uzUuj31Reb8rYqNrskmYDwXZkO?= =?us-ascii?Q?iYkoMEOPuvAc0M9MU1a4eR50VsHdckUjODctX2xXgdEUSwVfT3ErpHvF6b6o?= =?us-ascii?Q?CcQn8kFj3qA4Eq8=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(7416014)(1800799024)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:36:12.7123 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 929b1135-e3d9-4546-4ea6-08dd519203d8 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099E2.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6093 Content-Type: text/plain; charset="utf-8" pick_task_fair() makes sure that the a throttled cfs_rq is dequeued and unreachable to pick_eevdf() when it encounters one during pick. With deferred throttling, it is possible for a throttled cfs_rq to have kernel mode preempted entities queued on it making it runnable. Allow pick_eevdf() to find and return only the kernel mode preempted entities when picking on a throttled hierarchy. Introduce two new wrappers - pick_entity() and pick_subtree() around entity_eligible() to abstract away the nuaces of picking on a throttled hierarchy. Introduce pick_se_on_throttled() and pick_subtree_on_throttled() to find the eligibility of kernel mode preempted entity or its subtree amongst all kernel mode preempted entities still queued on the rbtree using the EEVDF stats. Since current task is outside the "kernel_cs_count" tracking, take special care when accounting for it in pick_*_on_throttled(). Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 134 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 124 insertions(+), 10 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9f28624e4442..4fd70012b479 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -526,6 +526,9 @@ void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 = delta_exec); static __always_inline void avg_kcs_vruntime_add(struct cfs_rq *cfs_rq, st= ruct sched_entity *se); static __always_inline void avg_kcs_vruntime_sub(struct cfs_rq *cfs_rq, st= ruct sched_entity *se); static __always_inline void avg_kcs_vruntime_update(struct cfs_rq *cfs_rq,= s64 delta); +static __always_inline int pick_se_on_throttled(struct cfs_rq *cfs_rq, str= uct sched_entity *se); +static __always_inline +int pick_subtree_on_throttled(struct cfs_rq *cfs_rq, struct sched_entity *= se); =20 /************************************************************** * Scheduling class tree data structure manipulation methods: @@ -750,6 +753,24 @@ int entity_eligible(struct cfs_rq *cfs_rq, struct sche= d_entity *se) return vruntime_eligible(cfs_rq, se->vruntime); } =20 +static __always_inline +int pick_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool h_thr= ottled) +{ + if (unlikely(h_throttled)) + return pick_se_on_throttled(cfs_rq, se); + + return vruntime_eligible(cfs_rq, se->vruntime); +} + +static __always_inline +int pick_subtree(struct cfs_rq *cfs_rq, struct sched_entity *se, bool h_th= rottled) +{ + if (unlikely(h_throttled)) + return pick_subtree_on_throttled(cfs_rq, se); + + return vruntime_eligible(cfs_rq, se->min_vruntime); +} + static u64 __update_min_vruntime(struct cfs_rq *cfs_rq, u64 vruntime) { u64 min_vruntime =3D cfs_rq->min_vruntime; @@ -936,7 +957,7 @@ static inline void cancel_protect_slice(struct sched_en= tity *se) * * Which allows tree pruning through eligibility. */ -static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq) +static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq, bool h_throt= tled) { struct rb_node *node =3D cfs_rq->tasks_timeline.rb_root.rb_node; struct sched_entity *se =3D __pick_first_entity(cfs_rq); @@ -950,14 +971,14 @@ static struct sched_entity *pick_eevdf(struct cfs_rq = *cfs_rq) if (cfs_rq->nr_queued =3D=3D 1) return curr && curr->on_rq ? curr : se; =20 - if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) + if (curr && (!curr->on_rq || !pick_entity(cfs_rq, curr, h_throttled))) curr =3D NULL; =20 if (sched_feat(RUN_TO_PARITY) && curr && protect_slice(curr)) return curr; =20 /* Pick the leftmost entity if it's eligible */ - if (se && entity_eligible(cfs_rq, se)) { + if (se && pick_entity(cfs_rq, se, h_throttled)) { best =3D se; goto found; } @@ -970,8 +991,7 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *c= fs_rq) * Eligible entities in left subtree are always better * choices, since they have earlier deadlines. */ - if (left && vruntime_eligible(cfs_rq, - __node_2_se(left)->min_vruntime)) { + if (left && pick_subtree(cfs_rq, __node_2_se(left), h_throttled)) { node =3D left; continue; } @@ -983,7 +1003,7 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *= cfs_rq) * entity, so check the current node since it is the one * with earliest deadline that might be eligible. */ - if (entity_eligible(cfs_rq, se)) { + if (pick_entity(cfs_rq, se, h_throttled)) { best =3D se; break; } @@ -5601,14 +5621,14 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_= rq) /* * Picking the ->next buddy will affect latency but not fairness. */ - if (sched_feat(PICK_BUDDY) && - cfs_rq->next && entity_eligible(cfs_rq, cfs_rq->next)) { + if (sched_feat(PICK_BUDDY) && cfs_rq->next && + pick_entity(cfs_rq, cfs_rq->next, throttled_hierarchy(cfs_rq))) { /* ->next will never be delayed */ SCHED_WARN_ON(cfs_rq->next->sched_delayed); return cfs_rq->next; } =20 - se =3D pick_eevdf(cfs_rq); + se =3D pick_eevdf(cfs_rq, throttled_hierarchy(cfs_rq)); if (se->sched_delayed) { dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED); /* @@ -6795,6 +6815,89 @@ static __always_inline void init_se_kcs_stats(struct= sched_entity *se) se->min_kcs_vruntime =3D (se_in_kernel(se)) ? se->vruntime : LLONG_MAX; } =20 +/* + * Current task is outside the hierarchy during pick_eevdf(). Since + * "kernel_cs_count" has not been adjusted yet with put_prev_entity(), + * its preempted status is not accounted to the hierarchy. Check the + * status of current task too when accounting the contribution of + * cfs_rq->curr during the pick. + */ +static inline bool curr_h_is_throttled(struct cfs_rq *cfs_rq) +{ + struct sched_entity *curr =3D cfs_rq->curr; + struct task_struct *p; + + /* Current hierarchy has been dequeued. */ + if (!curr || !curr->on_rq) + return false; + + /* + * There are kernel mode preempted tasks + * queued below this cfs_rq. + */ + if (se_in_kernel(cfs_rq->curr)) + return true; + + p =3D rq_of(cfs_rq)->curr; + /* Current task has been dequeued. */ + if (!task_on_rq_queued(p)) + return false; + + /* Current task is still in kernel mode. */ + return se_in_kernel(&p->se); +} + +/* Same as vruntime eligible except this works with avg_kcs_vruntime and a= vg_kcs_load. */ +static __always_inline +int throttled_vruntime_eligible(struct cfs_rq *cfs_rq, u64 vruntime, bool = account_curr) +{ + s64 avg =3D cfs_rq->avg_kcs_vruntime; + long load =3D cfs_rq->avg_kcs_load; + + if (account_curr) { + struct sched_entity *curr =3D cfs_rq->curr; + unsigned long weight =3D scale_load_down(curr->load.weight); + + avg +=3D entity_key(cfs_rq, curr) * weight; + load +=3D weight; + } + + return avg >=3D (s64)(vruntime - cfs_rq->min_vruntime) * load; +} + +/* Same as entity_eligible() but for throttled hierarchy. */ +static __always_inline int pick_se_on_throttled(struct cfs_rq *cfs_rq, str= uct sched_entity *se) +{ + bool account_curr =3D curr_h_is_throttled(cfs_rq); + + if (se =3D=3D cfs_rq->curr) { + /* + * If cfs_rq->curr is not accountable, it implies there + * are no more kernel mode preempted tasks below it. + */ + if (!account_curr) + return false; + } else if (!se_in_kernel(se)) + return false; + + return throttled_vruntime_eligible(cfs_rq, + se->vruntime, + account_curr); +} + +/* Similar to entity_eligible(cfs_rq, se->min_vruntime) but for throttled = hierarchy. */ +static __always_inline +int pick_subtree_on_throttled(struct cfs_rq *cfs_rq, struct sched_entity *= se) +{ + /* There are no kernel mode preempted entities in the subtree. */ + if (se->min_kcs_vruntime =3D=3D LLONG_MAX) + return false; + + return throttled_vruntime_eligible(cfs_rq, + se->min_kcs_vruntime, + curr_h_is_throttled(cfs_rq)); +} + static inline void __min_kcs_vruntime_update(struct sched_entity *se, stru= ct rb_node *node) { if (node) { @@ -6969,6 +7072,17 @@ static __always_inline void avg_kcs_vruntime_sub(str= uct cfs_rq *cfs_rq, struct s static __always_inline void avg_kcs_vruntime_update(struct cfs_rq *cfs_rq,= s64 delta) {} static __always_inline void init_se_kcs_stats(struct sched_entity *se) {} =20 +static __always_inline int pick_se_on_throttled(struct cfs_rq *cfs_rq, str= uct sched_entity *se) +{ + return vruntime_eligible(cfs_rq, se->vruntime); +} + +static __always_inline +int pick_subtree_on_throttled(struct cfs_rq *cfs_rq, struct sched_entity *= se) +{ + return vruntime_eligible(cfs_rq, se->min_vruntime); +} + static inline bool min_kcs_vruntime_update(struct sched_entity *se) { return true; @@ -9045,7 +9159,7 @@ static void check_preempt_wakeup_fair(struct rq *rq, = struct task_struct *p, int /* * If @p has become the most eligible task, force preemption. */ - if (pick_eevdf(cfs_rq) =3D=3D pse) + if (pick_eevdf(cfs_rq, throttled_hierarchy(cfs_rq)) =3D=3D pse) goto preempt; =20 return; --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2072.outbound.protection.outlook.com [40.107.93.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2AAFC1E04AE for ; Thu, 20 Feb 2025 09:36:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.72 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044200; cv=fail; b=AyFbswe78dJBxtXBRyBth85ZwdPEet6g5JC1fppkVy9s9kW8OntEtyXE1FXtdPmLU+PqeN/oIEzHyVsI8Kny8NgaMDsFO39VezLJIN8RtumUeGH8nF21OfOb0q9RzOxBoVv0GV777bthx+wzou9lDnTyY3zIrI0uzSGmhDPSaI4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044200; c=relaxed/simple; bh=6iR4/BgPsAII7PNwhW2SB9bzncZ4b9IL78w7NlC7Kww=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KrYqdqk5CEAzp1x4c00loNQkegrNpgkRhejRYQlDO1d24r/COTjIn/ODWUtffL89l2c0T0lbd7/IDRbgQGEXHMg6lDHdthyZ4QoHTDhP4oDlX12ZqMfkghBiHEyWs6TOzJEPnDh7pvIb0UwFv+Dmp0bTi2ta9Z2cg7+zwGLOiDk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=RGzO8FWr; arc=fail smtp.client-ip=40.107.93.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="RGzO8FWr" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=f4Brs+TZK3Uv+R0PSxzjjUAyTkHpV+Zv4WDWEc2VYqJwHCnxwW0qBtqqXAiercmZSuGqCl3ZZ4y0GZKfUe+XucDCYq1IU2jv2WvTIWlKzsil24971N4xdGyNSaVPzTPrd8qunntYt9WmoWRSytkO0qaZI4g0g4/rHRjHIKIQH1gQK9YDI7DiyNNEn4crUJHSwWyqFNwUbb5VePTIvMRhy//6rjI3+LhHXws/K+FnxBwsDA7KzOxPOF6+N+x47scy+gJbLVuXzzC7bZiIOfvZzFHps8fpZu2H4Lo+O43Ehw4s2Q3XbRzu0CaOItLv+HjS+/VIf+UizPYzJX5MLgHZaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=j04M7FpuFUmb8dwl4D/pdyEWwDHPVe4vDdGMefBm8AE=; b=dHddA/0UMGCrVYD6ZUVcZVK6BzqcKbPDt8BHqdjctZ8J43nJKcK5LrQ/4CVYkVSHgx7awA49A6YwPZ7GNNws5Yicky3Uo/M6oxwRfw+Iv6ydo1pNVw/uwQMmxD6uuY08rP4tAo6FznmwxRhIbrpgcsWFblDX5+BROot7KjgoZujq5H2gcfnGITsN0YWqp0skOAlSlDLox5kom/6DNrw5x5Mxt9dIwfyP6YfGUTXoFsuTR+5GIgtk7aT8Ti0hnbCBOjjvJhI8u81FpLsUFPg8faUOGqAM6j6qLqVnVgIHSExgb3u1V2k+1wKQhzBE7fZp8YacAF4s1sCySAlIJY7t+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j04M7FpuFUmb8dwl4D/pdyEWwDHPVe4vDdGMefBm8AE=; b=RGzO8FWrnZLdHu911eiU7mMLkQMeOen8hk2IjFqr4sXHzaGV/2e54Cw2IMGEJLIolF0SBWPY+joJCcH8f1xAs8HvtU4LrwA0RRbJao41ySjhAIXkI2s/HLK1iJDjAgMEYngcSikrmN/Tz/cV4oJbhjKDY5S+GcRFqxThA2hP3S0= Received: from DM6PR02CA0163.namprd02.prod.outlook.com (2603:10b6:5:332::30) by SA1PR12MB6918.namprd12.prod.outlook.com (2603:10b6:806:24d::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.20; Thu, 20 Feb 2025 09:36:33 +0000 Received: from DS3PEPF000099DD.namprd04.prod.outlook.com (2603:10b6:5:332:cafe::11) by DM6PR02CA0163.outlook.office365.com (2603:10b6:5:332::30) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.15 via Frontend Transport; Thu, 20 Feb 2025 09:36:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DD.mail.protection.outlook.com (10.167.17.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:36:33 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:36:25 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 09/22] sched/fair: Introduce cfs_rq throttled states in preparation for partial throttling Date: Thu, 20 Feb 2025 09:32:44 +0000 Message-ID: <20250220093257.9380-10-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DD:EE_|SA1PR12MB6918:EE_ X-MS-Office365-Filtering-Correlation-Id: bb1c438d-8660-4e5f-4814-08dd5192103b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?kXL6aTFi+OTbVShSjtT5T1xbMXeQV6/M9zfOli8tDWZL2FwWsiggJPMh9CyV?= =?us-ascii?Q?4bjtJWdgRSDvIPXXMuy+H/je8vtkfMrb2/HnWO7HanPHOeHQpOqe6Nzg7rbY?= =?us-ascii?Q?GUce8c8+oCtFeIdGz3RR7ccNlDWoKq5xBkQKn7jdfoJwDcvLBudkY2lwdrMa?= =?us-ascii?Q?9lZZD1s2tRb64VAjtbcJ+VRSUGvrsv0sDA/doRv+b/7YBgbdaJnDhxeCB2AZ?= =?us-ascii?Q?RboGYuN91WRRpkfhn5vS4zCOPQCr4c9U78QGTKcaa8L8tGkumGoeNperupHK?= =?us-ascii?Q?ePxtu4sxx83IUhMXmW3IKziZ+p6P2+56CrcKBsYRJomltWAX6cjFKRuxWJeu?= =?us-ascii?Q?1DzdC5ohdn9jdzUuhzvlfz8rOs+x2A+egJVOsBP3jVIH7S3BXs9JXJ4V82ZT?= =?us-ascii?Q?GqhxnvGJqAWpOOuUJF/TW+9Sy4HQl0RkCoD4aVSSqmLva1HCHd16o7hkTsoT?= =?us-ascii?Q?wlptcO88vhnQLuEVUBqyO+K9IeJQyHZHnw5XgD0tAdX/r6zWoHtRzXvYts2c?= =?us-ascii?Q?UlsREiSUNY94ebMJrkgL/Nx90VyhjljrDWorv5nFEna50Uyy26ShJvXJIDkV?= =?us-ascii?Q?Coq5ksim+cP+53bh+xS9gXrVhrxOndLGfESZe8KJFimlXF5Z0liJeIPx0c/C?= =?us-ascii?Q?hPxY3slhtIzdYa1iQcYF+pnIDf5IlgycK93mCt31d+E6qcT/TIjjPIXyrLmg?= =?us-ascii?Q?mANvjGACYbDsXJu3TrHOsgMzKN8BG0LwEe+qe5Hkbh0h2S6dWD40AzwwZSZr?= =?us-ascii?Q?1nEpDIp2C2rgs5ABioOJLcq1ftXPZhmmpZXw8olN5QCVYTX8AcfOAeaBci+F?= =?us-ascii?Q?tfjIXoKDfkkG1J07hMBqmsn38qVy/TlPX5GKM5A0EiYxauq3+jNoZbZqwqou?= =?us-ascii?Q?eduTOi04UrqAd6YhYKky8kkF1XB/OObMVol7d6DHyI8tbSjYdIcUCahSQcX6?= =?us-ascii?Q?/i8tK1ONbTRwNQ8yOd2g5cPbUk3W/61nvWwxc6ML6PAD+03FZS7IYH2DhzeS?= =?us-ascii?Q?1sv9i+LJbbx1UyuyLgwH8/WwKpKmFGt8kUXacQMYttwGnOF0gJ2hTtX9rDDH?= =?us-ascii?Q?1lJsDCXCHk/YsMkrstVZUmeKiQ9dVyyMkPVxU3TAdyMMlNvl1cQ+WB6La3i3?= =?us-ascii?Q?Hy9tlAYz7G5Uq7a45wvhQEoix9sPc3QNwxP/bfQ5lwKANmO5IG9x9g6gNTnE?= =?us-ascii?Q?mcpwzru+Mb8J0876/srlQE8zEvwuX3hz9JPAF2Ivn2nAnUqr8hRObNS2RYbi?= =?us-ascii?Q?hujqRS9A8md/G5p6noo5wCATqCzKIqjsQEZvoDq8RsdWmCWkEP7N3l8gEq4o?= =?us-ascii?Q?XGdyKfjL7SZ5PJEaQXwsD5Yb8RZqqaFLU5Waq3D24czbxsQ0rjS+1OjAgQcZ?= =?us-ascii?Q?95eyB0ptJHj1VxeZWjoQks0xFPVGk6jKpVFEk3yzJMbPWHNR9V0dGwtpOkgU?= =?us-ascii?Q?IlpSW6y/986G67Pc+5XQ47kQHWwEK5UJYfX5ffd4s0Jsm8EoYfzip8R0NBbY?= =?us-ascii?Q?+1YGG5uZPEUxt6E=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:36:33.5092 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bb1c438d-8660-4e5f-4814-08dd5192103b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DD.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6918 Content-Type: text/plain; charset="utf-8" On introduction of throttle deferral, cfs_rq on encountering a throttle event with kernel mode preempted entities will be marked partially throttled allowing only these kernel mode entities to run until they sleep or exit to userspace. Introduce "throttled_state" enum to define the three throttle states - CFS_UNTHROTTLED, CFS_THROTTLED_PARTIAL, and CFS_THROTTLED. In addition to cfs_rq_throttled() which will now track both partial and complete throttle, introduce a new helper cfs_rq_h_throttled() to detect a completely throttled hierarchy. Convert appropriate cfs_rq_throttled() checks to cfs_rq_h_throttled() to guard logic that only concerns with a complete throttle. Also take the opportunity to convert any open coded references to cfs_tr->throttled to use one of the two helpers. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 67 ++++++++++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 16 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 4fd70012b479..c84cd2d92343 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5340,7 +5340,11 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_ent= ity *se, int flags) } =20 static void check_enqueue_throttle(struct cfs_rq *cfs_rq); + +/* cfs_rq is throttled either completely or partially */ static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq); +/* cfs_rq is throttled completely and the hierarchy is frozen */ +static inline int cfs_rq_h_throttled(struct cfs_rq *cfs_rq); =20 static void requeue_delayed_entity(struct sched_entity *se); @@ -5404,7 +5408,7 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_en= tity *se, int flags) #ifdef CONFIG_CFS_BANDWIDTH struct rq *rq =3D rq_of(cfs_rq); =20 - if (cfs_rq_throttled(cfs_rq) && !cfs_rq->throttled_clock) + if (cfs_rq_h_throttled(cfs_rq) && !cfs_rq->throttled_clock) cfs_rq->throttled_clock =3D rq_clock(rq); if (!cfs_rq->throttled_clock_self) cfs_rq->throttled_clock_self =3D rq_clock(rq); @@ -5448,7 +5452,7 @@ static void set_delayed(struct sched_entity *se) struct cfs_rq *cfs_rq =3D cfs_rq_of(se); =20 cfs_rq->h_nr_runnable--; - if (cfs_rq_throttled(cfs_rq)) + if (cfs_rq_h_throttled(cfs_rq)) break; } } @@ -5470,7 +5474,7 @@ static void clear_delayed(struct sched_entity *se) struct cfs_rq *cfs_rq =3D cfs_rq_of(se); =20 cfs_rq->h_nr_runnable++; - if (cfs_rq_throttled(cfs_rq)) + if (cfs_rq_h_throttled(cfs_rq)) break; } } @@ -5817,7 +5821,7 @@ static void __account_cfs_rq_runtime(struct cfs_rq *c= fs_rq, u64 delta_exec) if (likely(cfs_rq->runtime_remaining > 0)) return; =20 - if (cfs_rq->throttled) + if (cfs_rq_throttled(cfs_rq)) return; /* * if we're unable to extend our runtime we resched so that the active @@ -5836,11 +5840,37 @@ void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, = u64 delta_exec) __account_cfs_rq_runtime(cfs_rq, delta_exec); } =20 +/* cfs_rq->throttled states */ +enum throttle_state { + /* + * cfs_rq is unthrottled; All the queued entities + * can be picked to run. + */ + CFS_UNTHROTTLED =3D 0, + /* + * cfs_rq is only marked throttled. There are kernel + * mode preempted entities that are still runnable. + * PELT is not frozen yet. + */ + CFS_THROTTLED_PARTIAL, + /* + * cfs_rq is fully throttled with PELT frozen. There + * are no entities that are considered runnable under + * throttle. + */ + CFS_THROTTLED +}; + static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq) { return cfs_bandwidth_used() && cfs_rq->throttled; } =20 +static inline int cfs_rq_h_throttled(struct cfs_rq *cfs_rq) +{ + return cfs_bandwidth_used() && (cfs_rq->throttled =3D=3D CFS_THROTTLED); +} + /* check whether cfs_rq, or any parent, is throttled */ static inline int throttled_hierarchy(struct cfs_rq *cfs_rq) { @@ -6011,7 +6041,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) * Note: distribution will already see us throttled via the * throttled-list. rq->lock protects completion. */ - cfs_rq->throttled =3D 1; + cfs_rq->throttled =3D CFS_THROTTLED; SCHED_WARN_ON(cfs_rq->throttled_clock); if (cfs_rq->nr_queued) cfs_rq->throttled_clock =3D rq_clock(rq); @@ -6028,7 +6058,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) =20 se =3D cfs_rq->tg->se[cpu_of(rq)]; =20 - cfs_rq->throttled =3D 0; + cfs_rq->throttled =3D CFS_UNTHROTTLED; =20 update_rq_clock(rq); =20 @@ -6080,7 +6110,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) qcfs_rq->h_nr_idle +=3D idle_delta; =20 /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(qcfs_rq)) + if (cfs_rq_h_throttled(qcfs_rq)) goto unthrottle_throttle; } =20 @@ -6098,7 +6128,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) qcfs_rq->h_nr_idle +=3D idle_delta; =20 /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(qcfs_rq)) + if (cfs_rq_h_throttled(qcfs_rq)) goto unthrottle_throttle; } =20 @@ -6503,7 +6533,7 @@ static bool check_cfs_rq_runtime(struct cfs_rq *cfs_r= q) * it's possible for a throttled entity to be forced into a running * state (e.g. set_curr_task), in this case we're finished. */ - if (cfs_rq_throttled(cfs_rq)) + if (cfs_rq_h_throttled(cfs_rq)) return true; =20 return throttle_cfs_rq(cfs_rq); @@ -7029,6 +7059,11 @@ static inline int cfs_rq_throttled(struct cfs_rq *cf= s_rq) return 0; } =20 +static inline int cfs_rq_h_throttled(struct cfs_rq *cfs_rq) +{ + return 0; +} + static inline int throttled_hierarchy(struct cfs_rq *cfs_rq) { return 0; @@ -7310,7 +7345,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) h_nr_idle =3D 1; =20 /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) + if (cfs_rq_h_throttled(cfs_rq)) goto enqueue_throttle; =20 flags =3D ENQUEUE_WAKEUP; @@ -7337,7 +7372,7 @@ enqueue_task_fair(struct rq *rq, struct task_struct *= p, int flags) h_nr_idle =3D 1; =20 /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) + if (cfs_rq_h_throttled(cfs_rq)) goto enqueue_throttle; } =20 @@ -7431,7 +7466,7 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags) h_nr_idle =3D h_nr_queued; =20 /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) + if (cfs_rq_h_throttled(cfs_rq)) return 0; =20 /* Don't dequeue parent if it has other entities besides us */ @@ -7472,8 +7507,8 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags) if (cfs_rq_is_idle(cfs_rq)) h_nr_idle =3D h_nr_queued; =20 - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) + /* end evaluation on encountering a throttled cfs_rq hierarchy */ + if (cfs_rq_h_throttled(cfs_rq)) return 0; } =20 @@ -13519,7 +13554,7 @@ static void propagate_entity_cfs_rq(struct sched_en= tity *se) { struct cfs_rq *cfs_rq =3D cfs_rq_of(se); =20 - if (cfs_rq_throttled(cfs_rq)) + if (cfs_rq_h_throttled(cfs_rq)) return; =20 if (!throttled_hierarchy(cfs_rq)) @@ -13533,7 +13568,7 @@ static void propagate_entity_cfs_rq(struct sched_en= tity *se) =20 update_load_avg(cfs_rq, se, UPDATE_TG); =20 - if (cfs_rq_throttled(cfs_rq)) + if (cfs_rq_h_throttled(cfs_rq)) break; =20 if (!throttled_hierarchy(cfs_rq)) --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2087.outbound.protection.outlook.com [40.107.236.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 39EBB1F03CA for ; Thu, 20 Feb 2025 09:36:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.87 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044219; cv=fail; b=VuDd1N/QzanMFjyXW7kRxDdTtvZt/oW/Acwe7OH091EIyw3Un+Shdgc236KxlU/Hw8lvrDvK+Euf84/yQGBy+e6Z2qFOD7pfuYMkq2hmOfk5Le21eC0BFPt9Mz1ts22Tqv7M2LQjM1qjIPrlx466GVGpauf5sUHaStylkNfIFRQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044219; c=relaxed/simple; bh=UPIjRl6gD5NpWUvhpP6l3S3+onK6Ku7QuvRooI1n/uk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=fNy1nRy69cgsukcvcxDadlZHMDYpxstErttXqdJ0ntld/6pcWgr8meYGigNFk/GQlXCLTazD7NSImyPXCGaxgFVvQfPeachKsvORQsjmD7gyhzTkh2aGjyXC1YlMCpSExoRYIuV76Si9u16WPWhNuEq2Sn4nZBGBrKYpBYqtdTo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Q7g83zb3; arc=fail smtp.client-ip=40.107.236.87 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Q7g83zb3" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=icpag6sP5GQHJuaEm0Un1psv7+yxAABtoJhERwcFpvWGgFBhdVrRf3cMhp8TMHVmYRK2wuyLD9Ilm/oTvV/wQwLaTjP5s82NUCyqfKICvGnVTRy4gyQkd2qfL1dWM4tJGjSjr6CTeWGwv7dqKxzuNlPdsWGkJFHijUm1ZPbjGStlLQGBGj7pXeJFGfXoZ6oTPOks1OVfBnrVhxYWoCny+kISw0XYFZD5AvxdmEzCJQaxbcjVGK3aav5McSTtl/5Gl/+GCmZ5qnm1Kr4A5CCjy+YLm5X5l8IUXSzBKJQuPL3r3e+8d51qtum97m+4fuL+bZSi+9T27t797+UrKarbHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xMQc69NMTfLvejTsmV7srFHuaWBqlTGgDjfjlm5JO4Y=; b=wPxEv61HU5R5g/wIqnYK7alxeyMNsiKDqQ0ZQ/EnRsVfhlRVu2yvxNrox5x+7cBmrmUNtW/b5xXihOfj53dmGt2QO2CniwCTiDNnAH7zit2G9KrhgIV9qRBVHieb3uqx5ZdlUEJ6rdeMJdtSyN6F7E/YGzzZlJIyvddtze6XH0uo3mpkloXMevAS67DySQuJIVSalim1YDlW7kN2n56nQts5gbDFoJNTp8O1FTjZOgSSbz0muPg5uPjWzik1jHCubD5vW2ZdM7P87LGdHZlHMb1eJKwfMOh3zT/Eeb+CoixqtrfTwtVHvr61Zd5v3vM99awAXbbsaL9VqwKu8JEtzA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xMQc69NMTfLvejTsmV7srFHuaWBqlTGgDjfjlm5JO4Y=; b=Q7g83zb3wSho3JpmIwja5bDglPMHvFb+GoheF1QMoPnNx6OLEvqq1eh6lF7PvzeE6U0YB++vOau3pP/OTi/QljbreO9L0zNigktgv3TxITB6n6NzIWoHKUut3tKTRMIPRBZc1fnc3gOStAdyNX4L4WhN/AwZKHGvILy2b5Fv7LA= Received: from DS7PR03CA0210.namprd03.prod.outlook.com (2603:10b6:5:3b6::35) by CH3PR12MB8281.namprd12.prod.outlook.com (2603:10b6:610:128::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.19; Thu, 20 Feb 2025 09:36:54 +0000 Received: from DS3PEPF000099DF.namprd04.prod.outlook.com (2603:10b6:5:3b6:cafe::9) by DS7PR03CA0210.outlook.office365.com (2603:10b6:5:3b6::35) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.15 via Frontend Transport; Thu, 20 Feb 2025 09:36:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DF.mail.protection.outlook.com (10.167.17.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:36:54 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:36:46 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 10/22] sched/fair: Prepare throttle_cfs_rq() to allow partial throttling Date: Thu, 20 Feb 2025 09:32:45 +0000 Message-ID: <20250220093257.9380-11-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DF:EE_|CH3PR12MB8281:EE_ X-MS-Office365-Filtering-Correlation-Id: e161d82e-e05e-404e-268a-08dd51921caf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|7416014|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?9a8zlsbyGcu47n7fQab1cNr33fweZ3Rfg2iJ7igs7ypBRVOw0+30nIgpksA0?= =?us-ascii?Q?b3+0+ov3q9nP0b14SND3pWbhN28y3LncZDo2eP4q1qT2+oqEiqLnb14S81aG?= =?us-ascii?Q?d4AZ6PLMjQfOrOvyFtkjVj/mDqhBT+o9y+9A2YbY2huowr7h27697IpPxdgL?= =?us-ascii?Q?wWvsbqJocV6vDrsmibWbmXfOLQO5uBGyf6tCVC41KI+GSpJSORALHOU1U/aG?= =?us-ascii?Q?o2i7xVd4B539C9BfvMxCEhvbdEWRrDUv6n+hbzhYmZk1yXgj5mFN0qEA6BCf?= =?us-ascii?Q?t6vw6OMcYBcF8kuQivyD0TBup+TJHd8qBHWO4tf87YmYrAPF2Z4lHH7O2kMl?= =?us-ascii?Q?HSiKB1Q6TcyUC+kP1FVTbK9UGMUd9VkRqD9Ucu3hc8n5NNe8IstRc/X8qerj?= =?us-ascii?Q?oZmqUGEmJtOkn3lE3U5zwiEQFupjyuUT8ZxdqgvsVjuGnRzoGwiyHe7IM1b7?= =?us-ascii?Q?015pX7vKEVOYj1Us66R7/WMv7WcXxP5TC3t6YppSUOpiHhgb6dS/BoNrCngb?= =?us-ascii?Q?6UGQBdhBsaw1cr9kzPt96A++TtFQTwsJRCG/D+vHonP0CdGupB4DlhHo7Uir?= =?us-ascii?Q?avK/WaCzzv9pSkJvd6edU7nxSiI5GnCCudE0Y+VBNcYS2g0sOQvzADaA7lJ2?= =?us-ascii?Q?46oauOXwqRqwJEkyYQPsQnuNPlLBkF9nFhe+67rDbwfaGn9zaC1vC8e61NrX?= =?us-ascii?Q?Klkdiw45i7EgHG6Lz3M0nT6zv2CxCW1ZKq9sfu2VvKzGlyInP+yCCbcnOpyi?= =?us-ascii?Q?MqH+U3pdr3ii7lwnpXvlPjD+NMsuZ/9zK+8/gq7eO00zsRQ4yHnQs10WyEtv?= =?us-ascii?Q?cgQpM3zhfPi2ghjvGIanOtwMyl9kBpxwL+vww+g2igwya/jB4RQC58swxzwe?= =?us-ascii?Q?GdsT15imNo1tUaVyMMBR+im8+INw2MWJ2MPR/uapbVWiv4bJusCzIkLWZHh8?= =?us-ascii?Q?41L8Ozefo/CmTeJuo29nME6tSbplm7pc3FweKUIvdV56SEMF7tW3j6423XGp?= =?us-ascii?Q?6SQrBWp05nvi7gHo3xBdF4S81HmPeLUfda345MMmmuZrpAm9NGb61Y/Ig1CP?= =?us-ascii?Q?aCDFrxU9Ils8zZfrI8MaUcsEsUTeM04yWEuYzcMFW213o6nYCAzXk5etVyd4?= =?us-ascii?Q?EmETzZbbXk8N5opWuUFRIFE+YmhDJ/AinihTMmy0k6zF7ptT4gHzbGfvqLRL?= =?us-ascii?Q?DuT7jLlHahNaz7ehB6b17GFjhtPrbKC5TF9Rb3AYY+2G+EjP9nxc2z0P7MUN?= =?us-ascii?Q?XIoI8hdDn/RAnOxsDrTigLwI1m6wEY2Q7WTNzNdxUMsSQcNYDlHTU2aeMZQ4?= =?us-ascii?Q?4g0pry8QN+MmcQN4EJAMROb1wHdDSEBYRsw8Hkv2mt8Q04rr70sbgNIXcdB5?= =?us-ascii?Q?mg0pPnqi9s/+pdUMePcdtct1ggjVcEnbKQ54HOYQ75aA/aB09I4AKl0TppZq?= =?us-ascii?Q?DyXy5T7DHMeIqsXutMY+ROxzBkzjTZ9s7QX9MWT+lZzi9sPRGPIQ4LetgCvw?= =?us-ascii?Q?QarrcuKTEZO/Oc4=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(376014)(7416014)(1800799024)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:36:54.3856 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e161d82e-e05e-404e-268a-08dd51921caf X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DF.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8281 Content-Type: text/plain; charset="utf-8" throttle_cfs_rq() will determine the throttle state first time it is called using the not yet fully implemented determine_throttle_state() helper to completely or partially throttle a cfs_rq when it has run out of bandwidth. For a partial throttle, throttle_cfs_rq() will only set cfs_rq->throttled to CFS_THROTTLED_PARTIAL, leaving the rest same. When a partially throttled hierarchy needs to be upgraded once all the kernel mode preempted entities yield or exit to userspace, throttle_cfs_rq() will be called again to upgrade the throttle status. determine_throttle_state() currently only returns CFS_THROTTLED and will be implemented in the subsequent commits once the foundation is laid. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 55 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 47 insertions(+), 8 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index c84cd2d92343..8e1df614e82f 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5894,6 +5894,16 @@ static inline int throttled_lb_pair(struct task_grou= p *tg, throttled_hierarchy(dest_cfs_rq); } =20 +static enum throttle_state +determine_throttle_state(struct cfs_rq *gcfs_rq, struct sched_entity *se) +{ + /* + * TODO: Implement rest once plumbing for + * CFS_THROTTLED_PARTIAL is done. + */ + return CFS_THROTTLED; +} + static int tg_unthrottle_up(struct task_group *tg, void *data) { struct rq *rq =3D data; @@ -5946,9 +5956,25 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) { struct rq *rq =3D rq_of(cfs_rq); struct cfs_bandwidth *cfs_b =3D tg_cfs_bandwidth(cfs_rq->tg); - struct sched_entity *se; long queued_delta, runnable_delta, idle_delta, dequeue =3D 1; + struct sched_entity *se =3D cfs_rq->tg->se[cpu_of(rq)]; long rq_h_nr_queued =3D rq->cfs.h_nr_queued; + int new_state =3D determine_throttle_state(cfs_rq, se); + int prev_state =3D cfs_rq->throttled; + + /* Fully throttled cfs_rq should not reach here */ + SCHED_WARN_ON(cfs_rq_h_throttled(cfs_rq)); + + /* Nothing to do */ + if (new_state =3D=3D prev_state) + return false; + + /* + * We've been upgraded! Just dequeue since we are already on the + * throttled_list. Let distribution unthrottle us. + */ + if (prev_state) + goto throttle_dequeue; =20 raw_spin_lock(&cfs_b->lock); /* This will start the period timer if necessary */ @@ -5969,10 +5995,13 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) raw_spin_unlock(&cfs_b->lock); =20 if (!dequeue) - return false; /* Throttle no longer required. */ + new_state =3D CFS_UNTHROTTLED; =20 - se =3D cfs_rq->tg->se[cpu_of(rq_of(cfs_rq))]; + /* If the cfs_rq is only partially throttled, we are done. */ + if (new_state < CFS_THROTTLED) + goto done; =20 +throttle_dequeue: /* freeze hierarchy runnable averages while throttled */ rcu_read_lock(); walk_tg_tree_from(cfs_rq->tg, tg_throttle_down, tg_nop, (void *)rq); @@ -6041,11 +6070,16 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) * Note: distribution will already see us throttled via the * throttled-list. rq->lock protects completion. */ - cfs_rq->throttled =3D CFS_THROTTLED; - SCHED_WARN_ON(cfs_rq->throttled_clock); - if (cfs_rq->nr_queued) - cfs_rq->throttled_clock =3D rq_clock(rq); - return true; + cfs_rq->throttled =3D new_state; + if (new_state =3D=3D CFS_THROTTLED) { + SCHED_WARN_ON(cfs_rq->throttled_clock); + if (cfs_rq->nr_queued) + cfs_rq->throttled_clock =3D rq_clock(rq); + + return true; + } + + return false; } =20 void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) @@ -6536,6 +6570,11 @@ static bool check_cfs_rq_runtime(struct cfs_rq *cfs_= rq) if (cfs_rq_h_throttled(cfs_rq)) return true; =20 + /* + * throttle_cfs_rq() will reevaluate the throttle status for + * partially throttled hierarchy and upgrade to a full throttle + * once all the kernel mode entities leave or exit to user mode. + */ return throttle_cfs_rq(cfs_rq); } =20 --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2079.outbound.protection.outlook.com [40.107.236.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0568D1F03C8 for ; Thu, 20 Feb 2025 09:37:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.79 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044240; cv=fail; b=PgWOOSLoDgTDxFVXcA4IK3Yjo6qbReq17gjLGosRyrO0NQ1z9rcZMInW0rzkooB/xlWVWkb0J/tpvLWkw5tLKaGPVarLaWHg680vZywQ8y4bEg+ZhV1zAe76JvSxUB0jT8MVikg6maJPBvanRVPNsYs0lRMlE7T2/XaK7zdIiz0= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044240; c=relaxed/simple; bh=sbZhAAHiTVfsHVPifa6mzEHQs+ZPUQL/KLrZ3xD3+co=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=avu7jU3/PicMo5Z9YPp6HzHo8Sh0rG3GAF5Y/RsQHN/pWBB7IYvjBqSfGeK/KtD7TAqxrn06e/eYOU+U4k30a4wc+Bc/7AHTjyCQplXTfmcTM9aDRqwV5E1Q2eH+Sjzgu9kK8t6BP8sx82Err3NNfFkqLkIwDaIoe5e2mNBxH1Y= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=LEMS2rUv; arc=fail smtp.client-ip=40.107.236.79 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="LEMS2rUv" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=g8v8uqmEw4QCiiEnCv4GCtbJ2+1rHq803jqubVG/lWf9w+bBDC4EDvlgnlXSgKZo9rqzEB4jlywQNptnMGgdPqn402aSs3s5xsf2g2yCFGTvTK25YAv/MzGf+VNeMVFwmoV7ABu4dE8YAYIf+3owKziGYqT3zAgn4DGMXqM9L+F+qdnF9iHg8r9hrs7SD3OwhOeZgdfQeLF5fW9Dlcu83dMVDO2wYvD5qK5ZxUWqrmJe/dawN+x6svzL9emTZlp2iWqy2/rD9TqAZxoXsA/MKfpnr18f+Rd4TMVLGsPY+RqWiLOdRi3LSAT5e2rnvY8U1WgsiGu3yaMTY9VVjZ+4Mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=J8vqVAnvAG5sapvjR2GbHSLHilK7UfH5obQ+j0sIp3E=; b=adDa8A4mH8JQxcWhV7kvOTwY77u0UtSeU9tFWtzivi6oAIJRDXr68XEfnGQivhku2QJOZW6RRrpfj/+uVx0QWgrK5qAmsW1t2FoDXPFizNm00wQxePx98bqdlNEyV3+A1kmYoqWaf6YiJf2zRMCTRt7+MJOcdo8ALIzCFc69lvZu8P2vhvWwT9HMvLSZBZY7MQRmKcog5XT/pZCYZ9kpC1lqPVGwKrXmQ5+FwedDQsXL/KXt3pGUAoWBJY4hW38moVM/noaZuZuEVhiWl/MhpflxwZK2vHcq10P8Y5A5rT4aY+L2/BPnO7QG9hpui5nfFMeJC90qG2M6Kg5TTSKHWA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J8vqVAnvAG5sapvjR2GbHSLHilK7UfH5obQ+j0sIp3E=; b=LEMS2rUvOuesAxM0AY7D8lNMeUqWg4+bUEijErfFA1UNEbfFgEYbWXlkNFl3tGQa1654CzNNwDbOB5E6UBq42ct1GCo4aXR9+9rSuxC7gBCgfWIQcgBxodmS78fEaHzTMaUBYQ3oGwEBAxq9JmrGbEwDPsOko3uDMgvGytsY3+s= Received: from DM6PR02CA0168.namprd02.prod.outlook.com (2603:10b6:5:332::35) by CH2PR12MB4261.namprd12.prod.outlook.com (2603:10b6:610:a9::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Thu, 20 Feb 2025 09:37:15 +0000 Received: from DS3PEPF000099DD.namprd04.prod.outlook.com (2603:10b6:5:332:cafe::61) by DM6PR02CA0168.outlook.office365.com (2603:10b6:5:332::35) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.16 via Frontend Transport; Thu, 20 Feb 2025 09:37:15 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DD.mail.protection.outlook.com (10.167.17.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:37:15 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:37:07 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 11/22] sched/fair: Prepare unthrottle_cfs_rq() to demote throttle status Date: Thu, 20 Feb 2025 09:32:46 +0000 Message-ID: <20250220093257.9380-12-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DD:EE_|CH2PR12MB4261:EE_ X-MS-Office365-Filtering-Correlation-Id: 64b547db-5823-43bf-e8b2-08dd51922930 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|7416014|1800799024|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?SbbeaAIwYMdDQhH+siHPIoMt0jCw/eGMgWjoABaL9h6rZt6C4OJUlS5iLh+c?= =?us-ascii?Q?5pxFs5TxhIahp277oXS0t3JHnlS38sPa3yQNMXv3tKH4HRk8hJFTydJl/Va0?= =?us-ascii?Q?LMG2f7ua5ZArDw58LLQvRJi9t+RHjPKi7ncxJxF8LTt7zJ0T+/Tq0FDqM+NR?= =?us-ascii?Q?YBzv/oGaoPt/QdLC/Cj81fIAwRYhPtn/oZgN1rd+nUcgLIl+J9UBSz9zKFCi?= =?us-ascii?Q?s694ehrHHfLeedSwzzyXE61AT9/icl1k06HoFb09PH41mwgBAJdQSU27eP0c?= =?us-ascii?Q?m5Cl7DdlPgeNP1MJsLPmeP2FflfBvvkhkduDBbaB+5VC9G2f5VmsUEvYC3y6?= =?us-ascii?Q?RIl13EnRNdbu76iXPquelK2bkmOxGVF+vnQbaG7b0HfgzsLgv4V79kKOqsFY?= =?us-ascii?Q?Xlm3mDkDBTbmWC3DWIcYOyfueJf4QRQuR2YrbDKTQd/ztDRYIosCRQOtwz2x?= =?us-ascii?Q?j5rUmrX57o7zhfadLIASJfMjOc2ZVPHQU/F+3GU3vUpKcxrXFpjji8VNPgfr?= =?us-ascii?Q?1GoKrDDQo+zTmsiamBaDhumoWqiHw6VryIl3m6wb6/eNkXNVH6Z+wq51gayK?= =?us-ascii?Q?kwmjxgL3Qbsmpxtxrs8V3kqVPVgJtJyJ6kNsGBSMZtPyRfe0Nn9CslQAk7UX?= =?us-ascii?Q?FUVey6z5YhknNf4vHwGP1zr27SUnVKqe6RM9BpgEOnUhVh1NyyPGPfGfYphf?= =?us-ascii?Q?RsxknF+WZtB9hRuY33pOMQ/qu0WeNKyUAQ0qRlVS2zWQJKMuDp9WQhxyzteT?= =?us-ascii?Q?qvglb6TDawhGO4fOIsPlfgR99qm4yfBqNf53Te5Ngc2cY78RRWgIYORqn152?= =?us-ascii?Q?+shRbU7wWrqv09lEPZfaAuC68SAiirI6kkLbrUqxWXpkELMEukPobhv0sJDD?= =?us-ascii?Q?uKsNDIpalU1uPgPZaM8L+LNViTMQgQt8we6o+wywjOOYuMvFXppoOI3ZrN8u?= =?us-ascii?Q?zudEmpaPRW0tJyVbysdHz7mlZksC7KcCSZ75xZy6heb0TTw+sWLyVRHuUpb8?= =?us-ascii?Q?OuS9KUwDZKibMTnPNkMrxu8HZZerZEgvol6Pt0QsN2w65/IVjZyaBfavQEhP?= =?us-ascii?Q?WoJLvlT/NkBCLpyQbdlFY+GH6684og1X1VPWyCtcJMXOK0IbPUT1OuI2qU8O?= =?us-ascii?Q?aLY9H9CJT69Jff0fftnDJ4HO8ZPU7PJyp3mSYYQpkEmSSC28l/871eMxZl02?= =?us-ascii?Q?+TeenBWplDmuHxoUEaZklw2vjghEP65ForSlDHoCtFdwtdThIsjAzy0+ZKBD?= =?us-ascii?Q?E/Djs0OdLBhBtxrUiVDrfey9KxFpz+PSb3a6mXdBeLIH83wEk6ZHvr5Bg3Hz?= =?us-ascii?Q?p1+d/VcnVVkeg3mmAkl9+cIuDbbaIVz7WBtT5S9TCSk/dMesqZP3441yMfaL?= =?us-ascii?Q?/CpTVdZmEdKEOqpfAT1eFcofil9VxZ+WORP1Ebo9Fj2LQo3zgObCYmSIxm6A?= =?us-ascii?Q?BdaHJNXih7yKwkQtGV0KqBQ37N24C4bqKL36kJAaTAkwxP1sCd4cPP738gAh?= =?us-ascii?Q?7MBDUB4AKNxOQY8=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(7416014)(1800799024)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:37:15.3687 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 64b547db-5823-43bf-e8b2-08dd51922930 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DD.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4261 Content-Type: text/plain; charset="utf-8" If an entity which was blocked when running in kernel mode wakes up, a fully throttled hierarchy needs to be demoted to a partially throttled one. Prepare unthrottle_cfs_rq() to demote the status where the caller explicitly specifies via "demote_to_partial" indicator that it is requesting the throttle status to be demoted. Modify all current callers of unthrottle_cfs_rq()() setting "demote_to_partial" as false since all the existing scenarios completely unthrottles a cfs_rq. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 2 +- kernel/sched/fair.c | 39 +++++++++++++++++++++++++++++---------- kernel/sched/sched.h | 2 +- 3 files changed, 31 insertions(+), 12 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 0851cdad9242..a797517d3dcf 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -9488,7 +9488,7 @@ static int tg_set_cfs_bandwidth(struct task_group *tg= , u64 period, u64 quota, cfs_rq->runtime_remaining =3D 0; =20 if (cfs_rq->throttled) - unthrottle_cfs_rq(cfs_rq); + unthrottle_cfs_rq(cfs_rq, false); } =20 if (runtime_was_enabled && !runtime_enabled) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8e1df614e82f..091493bc8506 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6082,17 +6082,26 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) return false; } =20 -void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) +void unthrottle_cfs_rq(struct cfs_rq *cfs_rq, bool demote_to_partial) { struct rq *rq =3D rq_of(cfs_rq); struct cfs_bandwidth *cfs_b =3D tg_cfs_bandwidth(cfs_rq->tg); struct sched_entity *se; long queued_delta, runnable_delta, idle_delta; long rq_h_nr_queued =3D rq->cfs.h_nr_queued; + int throttled_state =3D cfs_rq->throttled; =20 se =3D cfs_rq->tg->se[cpu_of(rq)]; =20 - cfs_rq->throttled =3D CFS_UNTHROTTLED; + if (demote_to_partial) { + /* + * A demotion to partially throttled state can only be + * requested on a fully throttled hierarchy. + */ + SCHED_WARN_ON(!cfs_rq_h_throttled(cfs_rq)); + cfs_rq->throttled =3D CFS_THROTTLED_PARTIAL; + } else + cfs_rq->throttled =3D CFS_UNTHROTTLED; =20 update_rq_clock(rq); =20 @@ -6101,9 +6110,16 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) cfs_b->throttled_time +=3D rq_clock(rq) - cfs_rq->throttled_clock; cfs_rq->throttled_clock =3D 0; } - list_del_rcu(&cfs_rq->throttled_list); + + /* Partial throttle should retain itself in the throttled_list */ + if (!demote_to_partial) + list_del_rcu(&cfs_rq->throttled_list); raw_spin_unlock(&cfs_b->lock); =20 + /* If cfs_rq was partially throttled, we have nothing to do */ + if (throttled_state =3D=3D CFS_THROTTLED_PARTIAL) + goto unthrottle_throttle; + /* update hierarchical throttle state */ walk_tg_tree_from(cfs_rq->tg, tg_nop, tg_unthrottle_up, (void *)rq); =20 @@ -6176,8 +6192,11 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) unthrottle_throttle: assert_list_leaf_cfs_rq(rq); =20 - /* Determine whether we need to wake up potentially idle CPU: */ - if (rq->curr =3D=3D rq->idle && rq->cfs.nr_queued) + /* + * Determine whether we need to wake up potentially idle CPU or + * reevalutate our pick on the throttled hierarchy. + */ + if (cfs_rq->curr || (rq->curr =3D=3D rq->idle && rq->cfs.nr_queued)) resched_curr(rq); } =20 @@ -6212,7 +6231,7 @@ static void __cfsb_csd_unthrottle(void *arg) list_del_init(&cursor->throttled_csd_list); =20 if (cfs_rq_throttled(cursor)) - unthrottle_cfs_rq(cursor); + unthrottle_cfs_rq(cursor, false); } =20 rcu_read_unlock(); @@ -6227,7 +6246,7 @@ static inline void __unthrottle_cfs_rq_async(struct c= fs_rq *cfs_rq) bool first; =20 if (rq =3D=3D this_rq()) { - unthrottle_cfs_rq(cfs_rq); + unthrottle_cfs_rq(cfs_rq, false); return; } =20 @@ -6243,7 +6262,7 @@ static inline void __unthrottle_cfs_rq_async(struct c= fs_rq *cfs_rq) #else static inline void __unthrottle_cfs_rq_async(struct cfs_rq *cfs_rq) { - unthrottle_cfs_rq(cfs_rq); + unthrottle_cfs_rq(cfs_rq, false); } #endif =20 @@ -6329,7 +6348,7 @@ static bool distribute_cfs_runtime(struct cfs_bandwid= th *cfs_b) list_del_init(&cfs_rq->throttled_csd_list); =20 if (cfs_rq_throttled(cfs_rq)) - unthrottle_cfs_rq(cfs_rq); + unthrottle_cfs_rq(cfs_rq, false); =20 rq_unlock_irqrestore(rq, &rf); } @@ -6786,7 +6805,7 @@ static void __maybe_unused unthrottle_offline_cfs_rqs= (struct rq *rq) * there's some valid quota amount */ cfs_rq->runtime_remaining =3D 1; - unthrottle_cfs_rq(cfs_rq); + unthrottle_cfs_rq(cfs_rq, false); } rcu_read_unlock(); =20 diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 22567d236f82..bd43271fa166 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -554,7 +554,7 @@ extern void init_cfs_bandwidth(struct cfs_bandwidth *cf= s_b, struct cfs_bandwidth =20 extern void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b); extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b); -extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq); +extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq, bool demote_to_partia= l); extern bool cfs_task_bw_constrained(struct task_struct *p); =20 extern void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq, --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2054.outbound.protection.outlook.com [40.107.243.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 041EA1E9B19 for ; Thu, 20 Feb 2025 09:37:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.54 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044260; cv=fail; b=GYGd68/9US2g/IpALhzELqM1ypsEtmyxG8HSZ5Cnpi7DjlGHVp+qLxN7kbf5bNumR20qr0aXGxj0yfsGe92BpEUV4sYbIS3ryScaRsO2u7FeTdMMd8a8iH0KgHqkfjfnV5zFMKLV5sgxEAl3oXzoWomuIfcDLGDxVr5Q4pYCX/w= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044260; c=relaxed/simple; bh=PcaYm2DMYRsNqifk66WL6QIBqUjg0Wsrul82jbOHI3g=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=UKmvcqlE8ALs9umTmkejRewYtk5HF4bhrj83V1PWKFKa7KMn2LYSehJ6Vjgfd30AbGuNxnX8r3gnSynNiyDKuPbCRBQy2h0g/SzDfZMf5btk6EtyarnM0m3uNS9cegPGiWZMmFkkzJJaa3ozWFqcE1ifefCYdcAYawe/walYGRU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=28sIhw1w; arc=fail smtp.client-ip=40.107.243.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="28sIhw1w" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DzUZaWl5RvwEFqZVqsum7F1TlauwkPYrGLzFziOtYGR7/2LaxAxOWqcyPeoaROrhZW1ZCaIrrSQgiJuZKjKNeeOB9oyveK1rOv7okn6dM6f5iSVhu8tHYaGFovKNU1Ia/mRiOkl2O0kFxJoj+fbip5Mq4yr9JlsNrgJqSQRbnRKb3AR3L/SkZ2xgPy5+MKS0wyrmhFD0ZofrjNLNHZJhUY3P5BiBae2zmKNpINL6f5sOyexZIFKsfSVKM0O8/aB9Pv4wvuLKIepXb3sLJzxQ1fH6iZIbISnXC1/EDnyJJOUU0qx+Zt3Rz4tg5t6Dcm/l3K86qIbmgzMZRPknUrLOKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iH1v9eBHjwLdmS8FXD8Br5M04r9aHqs2WF3ePVt2370=; b=uAzy3C4T+3AqJoM9WBNlyUXiBf2XsDP2QZHo4RwRO62BzXj4aVaKqkQvlxc1y1Tgwng2VOo4cC7KXGADzm1e8nd79Iw3/GGcZh4khYXhUh3VdDzZQz0Z3AaU3hT3Fh59TV8sHNDl1Ozk3SICF1fkxUZ/ekhjw23k+V/xJFZjoTdS1LgRv9WWway3GlrBTTFtsLxMMUNImFUL8FbStVRp1ZJcJ1R6yRB6AaOV3eaabv70kPvuBCm6xKEZMQf6P1KgadjGaL0aAaNEFZDH44TAWJ4QV92kc/RDtYw2dHO+IWdGuR6hW4oiu5NJ2kGjnqVHe0vn9ag2NM095B5s1tQrjw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iH1v9eBHjwLdmS8FXD8Br5M04r9aHqs2WF3ePVt2370=; b=28sIhw1w4rZyyN+5Wruo6NCBpwv7Tuoc7V0FIKWjDFOqjdrxETjpljfHw6usJ/7YcrDdWcT2Ept0Wo6a0EHhT8TRFEuUVoVGg3K92BcwIx2mXjru46tffNkij7GapQUT8QvKxpIOiTbKCrFySkgnI2Gqi21HkN4PJaY4orS0yHU= Received: from CH2PR12CA0029.namprd12.prod.outlook.com (2603:10b6:610:57::39) by DM4PR12MB7599.namprd12.prod.outlook.com (2603:10b6:8:109::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.15; Thu, 20 Feb 2025 09:37:36 +0000 Received: from DS3PEPF000099DB.namprd04.prod.outlook.com (2603:10b6:610:57:cafe::b9) by CH2PR12CA0029.outlook.office365.com (2603:10b6:610:57::39) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.14 via Frontend Transport; Thu, 20 Feb 2025 09:37:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DB.mail.protection.outlook.com (10.167.17.197) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:37:36 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:37:28 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 12/22] sched/fair: Prepare bandwidth distribution to unthrottle partial throttles right away Date: Thu, 20 Feb 2025 09:32:47 +0000 Message-ID: <20250220093257.9380-13-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DB:EE_|DM4PR12MB7599:EE_ X-MS-Office365-Filtering-Correlation-Id: e2324ae2-6293-4fd5-3f39-08dd5192359a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?I5DuPx6xi1sht0Cqh8Pozyj7dulx+0nxgMqOKjKFEqdjYBKM/7TYAyDnqlcg?= =?us-ascii?Q?V8eI+7oQhxobP9vX9FgVoPNS3G7kEQ4xPsYS38/r9cJ8P46tTrVputTxKlJA?= =?us-ascii?Q?x6KnqIIAZqvyWW9hPX8xBQ/qZ/6+EWnB2YkSKWoQ3C/nS8ZRIFGK+jWSEjwn?= =?us-ascii?Q?7Mm9foR1ZZJqTcs9l9823EL3Zbd8HUFMJwjQEbmsk1gJiDs1AuLteXvDbTEY?= =?us-ascii?Q?RoyTZzOGt1kFHCKER729mdAsNaQUbM2mrbMai9Rp+nm/lq4/heOXFWORKNeP?= =?us-ascii?Q?9cur5QmwfsfM40/9ATSP8C6ASqQ8L0gn4GZQClCWIew5EMJajZFjs9aaARn9?= =?us-ascii?Q?2Ucv6GK1xfRA0svXb3GyuRdGwDC8n8dvwhvNnxgtB1HnxhiMw0BSf6WTd72b?= =?us-ascii?Q?pEY9/LRVKzWP6gOXGRER+70wtHhn1OyTlWTXSGpr+LKSi7zLNzoPOdf5iGsL?= =?us-ascii?Q?ztkOvlx58NMGXPBnd9VvJeVMl6Go4DaDmYQmcSc3U1q93TXz7mXEZY+7R/oj?= =?us-ascii?Q?TKUU9aqYkJifsr1m2WIvCwMiGTTNz6Tjyewaa6LIteKYC/8rzTR+8BIAbmj9?= =?us-ascii?Q?d5kdPu4EOCfSYssIDtKF4WjJ4uAVGuDXjQ0qY0Fin5w+dYqhct1ga5El2iZR?= =?us-ascii?Q?dcCbVa22H2OdkAl6tegRXf/l754r5Vz0fGOGjt6tux46sm2THgJilAXn0oPu?= =?us-ascii?Q?nGthu1MbiR88XVeV7Mn6WwmTQpN01cu3Gopz1SWITU/w2ICsoyFIc4EBtz+9?= =?us-ascii?Q?wfcfwY4zi83ZY3QIdJN81Wtfy1dDCFjEWOoWidV+dqNsKkhwqp5afPQ1xUWN?= =?us-ascii?Q?scp+3nENFevVqAvMUcWBjerbWDArPgSTu01ShATMMQkPgsMPv5Aza9OaVAOu?= =?us-ascii?Q?ESFWNLdvqAkosfSiBV71RUH3GawpzFastaQYCgWe/Kx3nQ6/uffFWCcfdfbe?= =?us-ascii?Q?B9f5sNXjYrr37dMe9NqG1BE9mE1MOTsRtQOICXI/4QrXZPaJ6rjPTeW+w85c?= =?us-ascii?Q?nZQzowNpJUSm7vxGw47CvO5Z72r9d+omrEEPHuRs7xaloxRfyfoF0p+LKwIa?= =?us-ascii?Q?cUkYP+0ZxXNpr5lz76DFCOQ0uZvj8iLHL5njUcipDVyf7TRTpjjUy1CsbJDN?= =?us-ascii?Q?HZvU33bo8EUB3HhymWJvEhYFWCVpr+WvuOFf2Gf0HhPwPto8X6cjcI1EQ20y?= =?us-ascii?Q?eTmfvXOuoKz3segyGhmd3fRJrFfxJJVWZxHvgf5NU5jEApWhB0z0X3oiJ2Wl?= =?us-ascii?Q?nILSLJgHMjHBLI8J/Pr/TQM8JzI1IE21PiHx4zXcTLDq90CX/m4/wbULU/Vk?= =?us-ascii?Q?GMZAZ5H9vI7Hve8bqQGhMeRh8uebjgv0ylTvMeXLDbrul87Zbc+YbZ8XoUJV?= =?us-ascii?Q?Bw7D71i+mPzOKkwjkgmc1ycZgVuerxNkswo4O0z9jYASOQTitLBVEtylsEW3?= =?us-ascii?Q?k+4yjM06nF1OCmL98lFLpbCWg5MbM5aeuirHVwGeaBb/1HEiFSLSdD+E5F8L?= =?us-ascii?Q?DSh/SpNWX7MdZzI=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:37:36.1917 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e2324ae2-6293-4fd5-3f39-08dd5192359a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DB.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB7599 Content-Type: text/plain; charset="utf-8" Unthrottling a partially throttled hierarchy is cheap since it only involves updating the cfs_rq's throttle status and does not involve modifing the hierarchical indicators via tg_tree walks or enqueuing the cfs_rq back. Unthrottle a partially throttled cfs_rq right away. This also helps avoiding a subtle race with async throttling, specifically: distribute_cfs_runtime() rq_lock_irqsave(rq) # Replenish bandwidth schedule() unthrottle_cfs_rq_async() local_irq_disable() smp_call_function_single_async() -> # IPI pending rq_unlock_irqrestore() rq_lock() pick_task_fair() check_cfs_rq_runtime() # runtime_remaining > 0 but # cfs_rq->throttled !=3D 0 The above can lead pick_eevdf() failing to find a sched_entity which is now avoided. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 091493bc8506..8824e89a3ede 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6320,6 +6320,17 @@ static bool distribute_cfs_runtime(struct cfs_bandwi= dth *cfs_b) =20 /* we check whether we're throttled above */ if (cfs_rq->runtime_remaining > 0) { + /* + * Partially throttled cfs_rq is cheap to unthrottle + * since they only require updating the throttled + * status of cfs_rq. Save the need for reacquiring the + * rq_lock and a possible IPI by unthrottling it here. + */ + if (!cfs_rq_h_throttled(cfs_rq)) { + unthrottle_cfs_rq(cfs_rq, false); + goto next; + } + if (cpu_of(rq) !=3D this_cpu) { unthrottle_cfs_rq_async(cfs_rq); } else { --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2075.outbound.protection.outlook.com [40.107.96.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F2F51EE032 for ; Thu, 20 Feb 2025 09:38:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.96.75 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044282; cv=fail; b=RLfpOTMJ3LmRtM1tVUczZFbxKkVZt/nfxN+vgJqOrI4iSf8U6gFtwUP1YtulTejOKTb3wfjypeibe7+ToZu5mnR26FVobaapQ3RNkpN0EHvBMWjxYhqiVdnjD7QDT6mM9mBwYKGZ6dxVRO4mqt78lYASZCpEC7ZXs4EEM2wwxXY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044282; c=relaxed/simple; bh=TcEor6K/5IJGnDlcRajSnsACMAxcQHuMu0F7nrgJc2o=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ViOCvU3diYyd0CCdV1Wp/nbhWze5JCP5GisB+DrH3LFLe+7XNmr1pTOMY/e2b+R5Dj7CwP6AfQyKglK3jq/+ZVBPargv4diCCM5uUJf7L9Wk463+an+/0N8K+pj0R+ZLi56cofWmyV/9RwUQqR/oU6YP7uz+EH4mEJ/urrDhJbo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=SpT7deOR; arc=fail smtp.client-ip=40.107.96.75 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="SpT7deOR" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CUDsPuGkYUNLlSochvzII3gm953Yk+YhxwKiehEquP5Z1rCP3Y/WR+07lEidWd+5hrWfyr2HJODmtpGjJ5mblmfjJW6EFmAKU8jhwyAZyELPBIVL3I3owiOOtKZeG1V3kPySiUX79frkwaCTLkt9aroWmMkMc7tXMik0V6AYEySb/BEBh0GZvrfPuebnUMGKfU2qbZ7w+IhqwcHY4hFUYW060l9+fRgcQVpFmhYP94+Yq9Kjpuf5cMVEPpZVV5vRMj3xIXx9nSBAeDTVoHuvj0l9gt9jmR1OsHqHKtrWvPiATN6rNMHXG2o2rqQhwyRogWOHlNniySkdDDx3v+EjrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mpeMy8jI0CI1cCJJtAuSHemhXiOxLJ+rQiLNO+zDyhc=; b=SOlVJ96r1OtW1PkGp0ZFWR7EMJ+PjoQDSj33cF/fmRnkXjpiXVJWLFC6PNANpkwXIVjenS+5EH7soH08COhWpbU08YJ/0RP/G3vSmUrjpPO0A3SP8+wnXyGO93Kbn/RvQkmomKnEoRL4z7KWJFAT8LEBOFt4kyAGVqwNhfXlRp83WTWk/X73NkSdrqQuYKCnTzR8qXcYUNbV67WOyLQZEVCZn2NfVgzLHI8VsvMk7GkQzUGO2SBsw8OWNtS2S7eK5slBCKu8vyWW2XY2VvojkXsXisWaWjusGXQtyKCSZ3Cf25eNcCImGSAu31/rl091/QAMesqeBVNOZM235dXQ7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mpeMy8jI0CI1cCJJtAuSHemhXiOxLJ+rQiLNO+zDyhc=; b=SpT7deORVpbOY+0FT9AMVSIF1HnOL8TpEooTIkmeST4nBKXZFog3qSrHPcyFb+aK0/LZrsTKudpyTY4FdLm1Byub9zObsFzjkuDkJXP3SaPIS9VDbtCp/ytbj9z/4Mw1NfrRGyCYlaU+/gSYyPV3lAwuvHHAHjYN6Sy6sFHrOTM= Received: from DM6PR02CA0143.namprd02.prod.outlook.com (2603:10b6:5:332::10) by CH3PR12MB7691.namprd12.prod.outlook.com (2603:10b6:610:151::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.15; Thu, 20 Feb 2025 09:37:57 +0000 Received: from DS3PEPF000099DD.namprd04.prod.outlook.com (2603:10b6:5:332:cafe::f6) by DM6PR02CA0143.outlook.office365.com (2603:10b6:5:332::10) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.15 via Frontend Transport; Thu, 20 Feb 2025 09:37:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS3PEPF000099DD.mail.protection.outlook.com (10.167.17.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:37:57 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:37:49 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 13/22] sched/fair: Correct the throttle status supplied to pick_eevdf() Date: Thu, 20 Feb 2025 09:32:48 +0000 Message-ID: <20250220093257.9380-14-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DD:EE_|CH3PR12MB7691:EE_ X-MS-Office365-Filtering-Correlation-Id: caa4601b-a1f6-423d-6238-08dd5192421f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?qdMPnhCXQ4wVYv0BHIL4AHFfWswa/svYt8yupSCoMz0W5Cf9kgUhTkSK0sdY?= =?us-ascii?Q?6cJyA99KT4gy215S8xq7X9+sjg1rOrDlmgt24SZ4L6jgWW4jbQtnMXMEPPGg?= =?us-ascii?Q?+vJXe8kKcxMd6Tti8WD4Gr7Y6iA0HLTmKcOp/pSmRMLZTZgtiEJkp7qPKQ9a?= =?us-ascii?Q?01ZdwZy2tWzC0kjZ11BnRxkaK/G+1HfswHWgFhfscv2uPsYG7v5sf7mOT6Wy?= =?us-ascii?Q?WQSvsvo5/auDswpO0HM5LwU/RPSrcw0NZDpx0dx63VzCoUXwRuMKVJHq8SDf?= =?us-ascii?Q?ESVUt+Xjn1GIGp2Al7ug+OB4mYcQ1DwbnbVwO/3Q0hfyllsJi8Ic8Mbzy4ae?= =?us-ascii?Q?DRhPIjcRubU7khXbl5Q9Rzsv8b3spaIvqIl6wva7qVE7ysCKpjAKtXB40xdI?= =?us-ascii?Q?/QMXIcr/nze3S8hnq51C3VXNnSfZqWGANcIJ5sO3WAog2l2sXG/Nu/hsBHu+?= =?us-ascii?Q?ZeZ/l42nEYRqSv477QE4S2GgOSyuA9Esd7Bqt5pvopD4DxESMqeEosRI7sU5?= =?us-ascii?Q?6MCPKHLhAzuWYVx/a/g4NuXuuhGDy94nKU7g4W1ztFGlSRlCKYfn0N8piszW?= =?us-ascii?Q?OqYDiG9ozjMjhJf4oZKjBs4bfRNpxPEIhOFhIGeDKy67ieq3abIxu9q77jR7?= =?us-ascii?Q?qq+zsagGPUUKvOTj+WJPtaE4s/GUgWAOMeBXlVamgJbTn9TOmV+RtMLdGD3w?= =?us-ascii?Q?X2IxJeKKXs8XYLiWUuawPcIkMdqi2FjD7dPxF4zGX8EJDRF2xNbXnyXJieZb?= =?us-ascii?Q?V7geTwRkbG1qFCBsawYilgULlyVGIXZTR5uoNJ80WnsWDUJBGjqYbJJ70Bo8?= =?us-ascii?Q?tyxiJZVhPb7d9S4AIg499gEmEibi8EfLNFhNsYAzG6+K5gHCKRVpTeGNaBS6?= =?us-ascii?Q?ZqmfO1K0cshaKvTJm5yk0OOYTcRsUoDXV8gKSsGBfDFioqDZw500hoeSXMPJ?= =?us-ascii?Q?gBBXkzyg2BYwm63gPVCWieFiH8b4IZuTDm4J9lTFfSalR5qlUfsIaMT8nhJv?= =?us-ascii?Q?QWFu0/HcIe6fDQ1gZDe3z9yfuYvyuntOmk1ouOj5eNQwh/k9UZ8yYe9Gi+CT?= =?us-ascii?Q?ZytTf6Ur8x8Kl3dE05Dr5RZ67jXYJAlgHCC2ibVwfFBKxt/Aqq3tM+t0CZjb?= =?us-ascii?Q?AyRuvvJipV+J4qDHxNkXCjD9AV7ySvfomyQ/8gFI04J6+DH1czeFoiRcEeke?= =?us-ascii?Q?8QRtizC7QkB8JhZhfpdnAD3jmTTDtPooas+lENvunU2FEqw2fCm9LQQbtUC8?= =?us-ascii?Q?ivY38CFKnAC9m/BOeMMSQmpLxWZ3kcgT0s79IuhodzJwTf0Pn0nKWhLB9Bm+?= =?us-ascii?Q?TLv3EYccQO5jtwamy8Bu2g8YTkOXdjaI5+K2nxjoTPXOdRczabKWb87oEu1+?= =?us-ascii?Q?Uxy4YzdHBS4C0RaRh59gOH3LNs0WQK+VLIrEqB3/Y04yxR+BVnVyEllMijgQ?= =?us-ascii?Q?RXIYNu1nsjShEMElZtn8x92o3crmVLaLACC7T1Q+JKGn0xAXq89zhoTZlcW8?= =?us-ascii?Q?Vng4ZTSyr5HAd54=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:37:57.1969 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: caa4601b-a1f6-423d-6238-08dd5192421f X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DD.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7691 Content-Type: text/plain; charset="utf-8" A partially throttled hierarchy does not set the hierarchical indicator. Find the throttled status of the hierarchy when traversing down the tree in pick_task_fair() and pass the correct throttle status to pick_eevdf(). Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8824e89a3ede..1d871509b246 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5618,7 +5618,7 @@ static int dequeue_entities(struct rq *rq, struct sch= ed_entity *se, int flags); * 4) do not run the "skip" process, if something else is available */ static struct sched_entity * -pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq) +pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool h_throttled) { struct sched_entity *se; =20 @@ -5626,13 +5626,13 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_= rq) * Picking the ->next buddy will affect latency but not fairness. */ if (sched_feat(PICK_BUDDY) && cfs_rq->next && - pick_entity(cfs_rq, cfs_rq->next, throttled_hierarchy(cfs_rq))) { + pick_entity(cfs_rq, cfs_rq->next, h_throttled)) { /* ->next will never be delayed */ SCHED_WARN_ON(cfs_rq->next->sched_delayed); return cfs_rq->next; } =20 - se =3D pick_eevdf(cfs_rq, throttled_hierarchy(cfs_rq)); + se =3D pick_eevdf(cfs_rq, h_throttled); if (se->sched_delayed) { dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED); /* @@ -9187,6 +9187,7 @@ static void check_preempt_wakeup_fair(struct rq *rq, = struct task_struct *p, int struct sched_entity *se =3D &donor->se, *pse =3D &p->se; struct cfs_rq *cfs_rq =3D task_cfs_rq(donor); int cse_is_idle, pse_is_idle; + bool h_throttled =3D false; =20 if (unlikely(se =3D=3D pse)) return; @@ -9260,10 +9261,16 @@ static void check_preempt_wakeup_fair(struct rq *rq= , struct task_struct *p, int if (do_preempt_short(cfs_rq, pse, se)) cancel_protect_slice(se); =20 + for_each_sched_entity(se) { + h_throttled =3D h_throttled || cfs_rq_throttled(cfs_rq_of(se)); + if (h_throttled) + break; + } + /* * If @p has become the most eligible task, force preemption. */ - if (pick_eevdf(cfs_rq, throttled_hierarchy(cfs_rq)) =3D=3D pse) + if (pick_eevdf(cfs_rq, h_throttled) =3D=3D pse) goto preempt; =20 return; @@ -9276,12 +9283,15 @@ static struct task_struct *pick_task_fair(struct rq= *rq) { struct sched_entity *se; struct cfs_rq *cfs_rq; + bool h_throttled; =20 again: cfs_rq =3D &rq->cfs; if (!cfs_rq->nr_queued) return NULL; =20 + h_throttled =3D false; + do { /* Might not have done put_prev_entity() */ if (cfs_rq->curr && cfs_rq->curr->on_rq) @@ -9290,7 +9300,8 @@ static struct task_struct *pick_task_fair(struct rq *= rq) if (unlikely(check_cfs_rq_runtime(cfs_rq))) goto again; =20 - se =3D pick_next_entity(rq, cfs_rq); + h_throttled =3D h_throttled || cfs_rq_throttled(cfs_rq); + se =3D pick_next_entity(rq, cfs_rq, h_throttled); if (!se) goto again; cfs_rq =3D group_cfs_rq(se); --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2081.outbound.protection.outlook.com [40.107.236.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9529A1EE7B7 for ; Thu, 20 Feb 2025 09:38:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.236.81 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044305; cv=fail; b=r7YKBlv0sF1J5ttYKih/iWGzA2LPJs1XGUxHYQQSrGpaaq8L1rn+v5Mi3SERZcDqjlXbHcesUmbrT3btT8lSS9Ygcy+SemXPSpUI4WtAW7Hko4CHlb7w05v7js22EX+Z7xKoJdIcsqYfZz4PETu0wVC3GAhzxuRztOiQMQcDVqA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044305; c=relaxed/simple; bh=7DsmJ9pJi2aGFH+fxFfh7c6Zm4gRix7r/JLV0o+xCDg=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Tdgbau7AMZJiWgncMyWBBsIR/MSSk56yJyiCesR0XiAnH7HEuG0W38dvcFCoyVhBnkCRmz79B2jtuzUuz5pRg2DzrhgUADUHCzZGypa7Jj+6hT+tCUPLdOyLwn1IFwRt9EnZeM6HAnH/R4EQEoOCRiOw6ZoVCzt/KLC+bjLL0Cg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=bxR3nvb3; arc=fail smtp.client-ip=40.107.236.81 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="bxR3nvb3" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=hFethb5edMfYvTTRjqFTcx1rLIgqljNkknazoX0wn7kBPLxaL2BJAOhy428i5aPjy+BEYwIsVZZB3zaj8zneSAX0uzu1m9k/Pa8N4vTcvxR7RwJqdmfCVObC4nJxMmPL/SZ73tiaxpUIxuxGb+6H1SkhJsDT6N342bK/O7cz1bqFO51vSLSz8bv5HtCnc8X8pFxeFkr3PNQbCXMXK0Q6a5bepOstAIJ2/ZkHRi82tLJKYvF1vxxjUfWq8FftGoYnPOUZciYpmQ6dmjdEkhl5fdvDKAx3bdlgWSiEmQ6yaXWHvM7bIa9mhGoq9M1MieCHbl/8YUOzvpPCEvd1+yLkRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=O/pCy8bdFTh/kb/9k+el28421njpzz0Pzj1UasNPQ9s=; b=HiYt9g5ZaV5M9iN0DGK+q1Tdq5hje2TaQVWDg0G75nVO0ZYMNJ5nfF9PUwqIc1KWAgEWahcXiH2LUyPYuCYkVe3aLQObMk8xu6aZlMVzupMAfDi7hSNAseMSokSlA8jyT7kmJ+01/xAVwFo7vWeBoX6En/Tru1zbWrWSjP3ihc/BIb1JAsDRpb5Sv5AoBzkBGaX9vVHoHVx5F1ih40YHfI5CjkIAbyaR7TZ4ptT1iwinFunMZpd53I6IFJmoC2Eg7um+PzHsK+IXiGTJd1sF4HprtbmGf4yC2g3YJ6nUwKh4aPs9T/E2tEToAvsqv/KLsYbRdOfgoFRkXHspgXIXLQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=temperror (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=temperror action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=O/pCy8bdFTh/kb/9k+el28421njpzz0Pzj1UasNPQ9s=; b=bxR3nvb3fRCifxDR1Ng0WG9CYJzPZnOTgs+C8KMs7oLgc4HPKI9KUs55Dukcfde45v1SgBI18IL/GrGnGUhJRsDivCgGfBhhGmJTZONzpQc3RWNeRR52R+KsyGdO/HLcZNqCcmr1rJHUFojFqkD6/LM+oYr8ZdWvhOiSdqBdcBQ= Received: from BN0PR04CA0024.namprd04.prod.outlook.com (2603:10b6:408:ee::29) by SN7PR12MB7809.namprd12.prod.outlook.com (2603:10b6:806:34e::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.15; Thu, 20 Feb 2025 09:38:19 +0000 Received: from BN1PEPF00005FFF.namprd05.prod.outlook.com (2603:10b6:408:ee:cafe::41) by BN0PR04CA0024.outlook.office365.com (2603:10b6:408:ee::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.14 via Frontend Transport; Thu, 20 Feb 2025 09:38:19 +0000 X-MS-Exchange-Authentication-Results: spf=temperror (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=temperror action=none header.from=amd.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of amd.com: DNS Timeout) Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00005FFF.mail.protection.outlook.com (10.167.243.231) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:38:18 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:38:10 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 14/22] sched/fair: Mark a task if it was picked on a partially throttled hierarchy Date: Thu, 20 Feb 2025 09:32:49 +0000 Message-ID: <20250220093257.9380-15-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00005FFF:EE_|SN7PR12MB7809:EE_ X-MS-Office365-Filtering-Correlation-Id: b9061e71-6b4e-444f-7a56-08dd51924e8c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|376014|7416014|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?fUwNCtlKaimXtmIxeG+durD0bY3nt3HsYemhAm2e/wIKsOG0Sj3o+aA6Q6c9?= =?us-ascii?Q?qQLpsemCt89L7QNnbR3ivkklQPNt/8Y3WajSBBbv7L1EIPHxYjOH4Pl2bLnT?= =?us-ascii?Q?X1OLTnbtwccVkKw7ObfPs4e2/4J/RsZPyJwvMRng9We1sUPIKTiGEW3xl1S0?= =?us-ascii?Q?/WLCdNXzZ7e3+CioF56322EByQ4Jaw0NLE8aU+VcZeCGDNxPKRx5JluGvfrV?= =?us-ascii?Q?Mf0PRDgRkywNIj4evRT0fEBBFVIOa5hEYeUE4WIb2m3yX5Fm8swPzO+RBCyK?= =?us-ascii?Q?dHDbkieXIkdXUs1rbJHgQneoYd3pFFRxJrDkgK1+UidHnCZDwI41Q0jVvi4h?= =?us-ascii?Q?wQF2FJ9v2MbD6h74QT2//ZurPaDK/llZab1xJ7BD24iZctTudWZ7h51Esd7L?= =?us-ascii?Q?oBM6uyLZcjExP+6Ytx5OyeRhvnvGO1jmm9fRgQZNhND0qin6k1pX+ObYRlRS?= =?us-ascii?Q?yKs9kdCwmuQVBmLIf7PJeSW2jVTMDs/dvM+Ok2dnLht3UBcUTQKSm4dKOtca?= =?us-ascii?Q?z5Up69RlP1koQQys48jvOPLX1I9CWDkKkFJ/0/tuQU6zJlweSks1nijrwue/?= =?us-ascii?Q?KC8y0BERotmV4bKVIX1i8/k4r9BZ+5Vodmr4hcYABRljnK5tPpRFS77fcCzN?= =?us-ascii?Q?OnoOqs2EysQsng0Hv/R1IN8IhT1TorZ0Mvs5L6ZWi/WxKGqqjd0XfQG1M9GD?= =?us-ascii?Q?MG4EFE02zJm/98lNJQjEgs3pqZetIpT4xjd9PjLvhJq9qOq/QBoeLmuJqdoM?= =?us-ascii?Q?W20rBkBVoHaMRBTGhp/tvJsL7RYRAx30637bhfC3BohgRAVBkOTUvH6UnCWx?= =?us-ascii?Q?IQEalXUyfy7HimDjx1Q/vMhlRbpARNe7UXO/6V9wydvczTk6yNNm64kJWINH?= =?us-ascii?Q?YA0GwN2X/UU2/w/2DvqreBOA260s3VK+z5j3ywqhkjF7/W3c0nzJx8drQENg?= =?us-ascii?Q?9Aj45xtJmxDHM49Q/EW0I2YZby2n0Ly22fcxcHAvJZLW6GoXs5GxcjK47iqO?= =?us-ascii?Q?XWN+2XN9L3d6/dtVwTssT8ECAY3xi9y+JSR/FXLvIZ9w4EkithM7P1htPOX8?= =?us-ascii?Q?7g49bz8Z9riweYYgXLycAAeBr64G9oFQy/7xKMLsbkBA7PbPoYxplr2Yqi/J?= =?us-ascii?Q?W37AKrtRlHdzB8ESxggEpSFUUpjK5pelCh4zC/EQTslcp2B+qviWUEYRSqFH?= =?us-ascii?Q?7MHYVh8LMXQYdcdx9u6noLXzp45A116J/bL4qu6NUSABpmXbWX4Bnqkg7bKF?= =?us-ascii?Q?gxTmAU7ofj1jMtpfPI13ogPSo5UMCACNTMV9a6dylYYmuFmypH+y6lfp90AM?= =?us-ascii?Q?qO5Dz+/8clBWWTgYsPMwbEPr8HdCMG4FiXTHo74x3UrPocDpxV4drVGNKfsY?= =?us-ascii?Q?FfUKv6B4DnHG55ad00NDESsOi4sqEUY4OCXjwsbF7OkuiBifabLBATjYdRzT?= =?us-ascii?Q?d0gVlAyjo2vhVyZw4GtfCCiBkJtS+FZUx05YcldUpD7LmfeqTkBt5FVa9G5F?= =?us-ascii?Q?tYBYshitpfgVvfA=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(376014)(7416014)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:38:18.0912 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b9061e71-6b4e-444f-7a56-08dd51924e8c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00005FFF.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7809 Content-Type: text/plain; charset="utf-8" With partial throttle, the hierarchical throttle indicator, namely "cfs_rq->throttle_count" will not be set. To detect a task picked on a partially throttled hierarchy, set a "sched_throttled" indicator in its sched_entity to trigger a schedule() when "kernel_cs_count" hits 0. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 3 +++ kernel/sched/fair.c | 30 +++++++++++++++++++++++++++++- 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 48115de839a7..200cc086e121 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -597,6 +597,9 @@ struct sched_entity { * is accounted at key decision points. */ int kernel_cs_count; + + /* Entity picked on a throttled hierarchy */ + unsigned char sched_throttled; /* hole */ =20 /* diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1d871509b246..68c194169c00 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6856,6 +6856,17 @@ static __always_inline int se_in_kernel(struct sched= _entity *se) return se->kernel_cs_count; } =20 +/* se picked on a partially throttled hierarchy. */ +static inline void task_mark_throttled(struct task_struct *p) +{ + p->se.sched_throttled =3D 1; +} + +static inline void task_clear_throttled(struct task_struct *p) +{ + p->se.sched_throttled =3D 0; +} + /* * Same as avg_vruntime_add() except avg_kcs_vruntime_add() only adjusts t= he avg_kcs_vruntime * and avg_kcs_load of kernel mode preempted entity when it joins the rbtr= ee. @@ -7171,6 +7182,8 @@ static __always_inline int se_in_kernel(struct sched_= entity *se) return false; } =20 +static __always_inline void task_mark_throttled(struct task_struct *p) {} +static __always_inline void task_clear_throttled(struct task_struct *p) {} static __always_inline void avg_kcs_vruntime_add(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) {} static __always_inline void avg_kcs_vruntime_sub(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) {} static __always_inline void avg_kcs_vruntime_update(struct cfs_rq *cfs_rq,= s64 delta) {} @@ -9281,6 +9294,7 @@ static void check_preempt_wakeup_fair(struct rq *rq, = struct task_struct *p, int =20 static struct task_struct *pick_task_fair(struct rq *rq) { + struct task_struct *next; struct sched_entity *se; struct cfs_rq *cfs_rq; bool h_throttled; @@ -9307,7 +9321,11 @@ static struct task_struct *pick_task_fair(struct rq = *rq) cfs_rq =3D group_cfs_rq(se); } while (cfs_rq); =20 - return task_of(se); + next =3D task_of(se); + if (h_throttled) + task_mark_throttled(next); + + return next; } =20 static void __set_next_task_fair(struct rq *rq, struct task_struct *p, boo= l first); @@ -9349,6 +9367,8 @@ pick_next_task_fair(struct rq *rq, struct task_struct= *prev, struct rq_flags *rf bool next_in_kernel =3D se_in_kernel(se); struct cfs_rq *cfs_rq; =20 + task_clear_throttled(prev); + while (!(cfs_rq =3D is_same_group(se, pse))) { int se_depth =3D se->depth; int pse_depth =3D pse->depth; @@ -9457,6 +9477,14 @@ static void put_prev_task_fair(struct rq *rq, struct= task_struct *prev, struct t bool task_in_kernel =3D next && task_on_rq_queued(prev) && se_in_kernel(s= e); struct cfs_rq *cfs_rq; =20 + /* + * Clear the pick on throttled indicator only if + * another task was picked and not for a save / + * restore operation for the task. + */ + if (next) + task_clear_throttled(prev); + for_each_sched_entity(se) { cfs_rq =3D cfs_rq_of(se); account_kcs_enqueue(cfs_rq, task_in_kernel); --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2051.outbound.protection.outlook.com [40.107.223.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60A8F1EE032 for ; Thu, 20 Feb 2025 09:38:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.51 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044325; cv=fail; b=bmaWBde85OE7byO//p5xMJqiwC+KTpAxjFGCYmj8AXjnqUCVu8mj/27agSMKUhCXsaIjQAbvC/rTyKAKnoUVdPvHCH0R/MYEorWJ00LPOF+d2xTrwEJ80I1KFUZe4quh7cN6xCI1u8EqX1r/dpellNKZ7TpfF4SJEKdDxPAObvY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044325; c=relaxed/simple; bh=uOMAFWC0FfqVuawHy7qpWWvZgglj88NuSg2svmpxMLk=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=oKEXF9JFkqfTzGOVzZaIv9L9bbANfH+ri90S8lgWczjfIDgIbsb12F8+fpzNZQPj1+1ShUiakjqPHWGq5QBlYfDJmQxnRqSzaLygxW8AQBbI/FgF6wLsopG316pXk9Za9SGHUabHt7rocPXETIwtFKksVErnv40OVutaMin3SbI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=mK8T7fHL; arc=fail smtp.client-ip=40.107.223.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="mK8T7fHL" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=N9eXf7bmnq7nFrVCQdkow5alAZPgDezODB2d3BN7qSHc8uSY6MNASQf+W0UhTNsujM4ikeQWnHQbLlQea96nXFM/8cqcos2ViQhuPCtAqFxS9gPRYqPX1m7894ADKd2eTuNS8LIfGVMFLSOiHdrmTwhJfpN25TWoFehbCaDwK/9YwZ8I/a3A/fFqxcN1Ml79jbfRgQmOuluJGr3vdz8jkCAvryPivhveCAbnMeS/ubZTvxJ6Bkdr8hQQQR6yH12pIM1THit8hY502C/F+NGT0R9ZTM+ErRDk5dq+rf149lsnXgNXJW3jy4oH55iSWYV/SetfIjGUfSkj5rgm09RF0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=t+VOB9lL33fkfvsjnklkiZYZ+D3epSxl6jgPoBUqQe0=; b=sqN83jewZGv1l+HAsELUWahax8g5NAuFiSKKlFY7kg2urbTfTQ9wm87Kx7VoMFI9oHvY5lpUuxajZaMDRKET9RWVKksvyQ/7CTlsnrnEDKjHrf5Hgo4bq2WHg6jNixY3pXX9aK9V1cAlnHqape1wCrW+by44L27J06OKaVnMPkSLFr3NpAwx7gDaz5X2w+h/QVE6vd5IIpdwGL+K5hq/CJXWR+IpWoAJ7FBD7SUzJhXeg+DNqJCQ6Tfw9rp0J0LH2IcHiITkIl7qg5Vaw+2+Mh3z48OCHvrOO/oOn/O9fR21lkKcjVmmkHyIVgLYv4ALRiztTxU2Vs09NRcr5DnLXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=t+VOB9lL33fkfvsjnklkiZYZ+D3epSxl6jgPoBUqQe0=; b=mK8T7fHLXGL7o8CQ6JvqGKOLxc099xSe5oLsr1prr4/OOVgMLvIMIfNyi+6qtstIuMLwjbvVUOirzMc+sB4xLIMA59zW435KxRYbUb5v/j2gQckEwVg+dUf/6cyh36Ikm/S54DfCLMn9mr2h2o5Fxq4B8SEWfOwQecHv4wUeG4E= Received: from BN8PR12CA0020.namprd12.prod.outlook.com (2603:10b6:408:60::33) by IA0PR12MB9047.namprd12.prod.outlook.com (2603:10b6:208:402::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.16; Thu, 20 Feb 2025 09:38:39 +0000 Received: from BN1PEPF00006000.namprd05.prod.outlook.com (2603:10b6:408:60:cafe::a6) by BN8PR12CA0020.outlook.office365.com (2603:10b6:408:60::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.14 via Frontend Transport; Thu, 20 Feb 2025 09:38:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00006000.mail.protection.outlook.com (10.167.243.232) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:38:39 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:38:31 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 15/22] sched/fair: Call resched_curr() from sched_notify_syscall_exit() Date: Thu, 20 Feb 2025 09:32:50 +0000 Message-ID: <20250220093257.9380-16-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00006000:EE_|IA0PR12MB9047:EE_ X-MS-Office365-Filtering-Correlation-Id: e60fb49d-7932-4404-5bec-08dd51925b08 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?dhFX8o8x8r1fhpg+uLiAlpWi07HiZ9m5QUvIucwryjgHGJ90A+KKoMpHD+21?= =?us-ascii?Q?tmXHSVF4/g6ucVhYd2JDe9wwX3YumipeS8g/VrD40c+fEqvNxoKNkfQYE4yd?= =?us-ascii?Q?/4ikoUhdb7Im9MJoNKHt95S70QFfP/4j0T8YylaskQooxKPJVsmiRNU9nkRP?= =?us-ascii?Q?vXwWrxznIHeUwZ3vhrdpq3qwwBMw6EROByuxiOwfPaRiG6f+prxa8nOo37VV?= =?us-ascii?Q?DIuZhVYU3a4ua6FNQyJGoqvr26bym/oDJARUPE0N2fU5cxMHnO3LaIF4Z60V?= =?us-ascii?Q?KSREl07PDikj9oQYwwujs1yu9b2WLBmp6kxTS9TymGUHJoWayFJ5GByXwBC/?= =?us-ascii?Q?XkBauzywBFkmXOlblmjHEddhhSkxOHHdV2Vvo268V8remqWcDyTkwgx+2ceT?= =?us-ascii?Q?vcfy9roJLMq405akevKcJHP2ck5SpIOZsWagG9TOyV3ZkAh8Xa5hHylr4aWv?= =?us-ascii?Q?jOuHLAmgEYqmGr68P3oiCTsNrXxzOYZYznuu+yTcvDRJFWHUKZlRMuHBMvr9?= =?us-ascii?Q?vrGDsAcrQeCHvOzoU4H0RpVTAXHawGyzr6SrLMde6r+eAMEaF0ggfD3rAzdn?= =?us-ascii?Q?sCE6aHuDGxA21N+vCikyVyHo5+6hYL5RAEmSID0u84gtXgdBZ3KXkhzLncPh?= =?us-ascii?Q?+QVegKqolAdQ5ylLfwCkBXcuwsxgyC7QBtd1W8PJBshRp58ycPFo3VY8xeyk?= =?us-ascii?Q?ZxsCmfzeCGoKXUHDgU63vg3Fura4RriSZQIdFHtzECzhmW2uf7nt576aj1YL?= =?us-ascii?Q?gTIUy4nzPrR+Tk7BbT7fx2QREeKIziryfFVkQcTVvgrd60DpNViWBq6bBA8N?= =?us-ascii?Q?gfEifDe2Mv3UaiR3FxFu/8szSiHWSg/rwugXeeb2KYH2pR3wV32Obh8192BU?= =?us-ascii?Q?sSJa9iz+Ez95dN46fjyVc/hRdVtXXYZcaa7f+DrVNqizqR4o2T9NbGlsldIV?= =?us-ascii?Q?dZCjBIk+PZ0QjHoVC/yqYtStUZlveB0NRdfEotmtQQe6vNy7LWMKwTuwjRP1?= =?us-ascii?Q?l5xzV5xPQok6lhRGuBrq0k8cNia7ZCDB4YbY9zujIIQMU/rHi915qzRo01C+?= =?us-ascii?Q?BV4LzjHIszTsOxI/qW6fp1XU+4ynFITcWPJ9WDeyRytQcJQoPL7xmjbuZoZf?= =?us-ascii?Q?3+Qc+TaU3ezp/uZVOdvvHhI07mzVTnQS2Ka4grNWiTuCtWTrNnLD5EfmYQn/?= =?us-ascii?Q?ARFdcffsII2FTvoO24ZsHhC08okVOTZWynHMl14EucI6V4daP8TF+DcP580Z?= =?us-ascii?Q?5fzJ7saQYBBgtMVo0v1YOGN2JiLucGiSRsb87RFEaxOlVXnKdPNjsYsCK8RU?= =?us-ascii?Q?asNkVPk1ZvYHAsdA9/JsliTcLqJI/eMa7NYijCSY3GTJqMmjuFZGSy/3VXDj?= =?us-ascii?Q?iH4ZLumWXgBDWm8zgqxtTnFqx2pu3OZnu9LO7KZE5VeyhtcyuPTzuRCJnsFa?= =?us-ascii?Q?XIDZ2S9nceXBVb6uAkrWbEKRWPhE5B0P5ApKHviewWwwhKvPZMvq5Erejkl1?= =?us-ascii?Q?iyihuekjRl0cEtk=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:38:39.0359 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e60fb49d-7932-4404-5bec-08dd51925b08 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00006000.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB9047 Content-Type: text/plain; charset="utf-8" With pick_eevdf() marking a pick on throttled hierarchy with "sched_throttled", reschedule the current when the "kernel_cs_count" hits 0 if it was picked on a throttled hierarchy. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 68c194169c00..0332e95d36b5 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6839,6 +6839,7 @@ bool cfs_task_bw_constrained(struct task_struct *p) =20 __always_inline void sched_notify_critical_section_entry(void) { + SCHED_WARN_ON(current->se.kernel_cs_count); current->se.kernel_cs_count++; /* * Post this point, the task is considered to be in a kernel @@ -6848,7 +6849,23 @@ __always_inline void sched_notify_critical_section_e= ntry(void) =20 __always_inline void sched_notify_critical_section_exit(void) { + lockdep_assert_irqs_disabled(); + current->se.kernel_cs_count--; + SCHED_WARN_ON(current->se.kernel_cs_count); + + /* + * XXX: Can we get away with using set_thread_flag() + * and not grabbing the rq_lock since we'll call + * schedule() soon after enabling interrupts again in + * exit_to_user_mode_loop()? + */ + if (!current->se.kernel_cs_count && current->se.sched_throttled) { + struct rq *rq =3D this_rq(); + + guard(rq_lock_irqsave)(rq); + resched_curr(rq); + } } =20 static __always_inline int se_in_kernel(struct sched_entity *se) --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2077.outbound.protection.outlook.com [40.107.237.77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 526471EE006 for ; Thu, 20 Feb 2025 09:39:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.77 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044347; cv=fail; b=byX8eU5CJ9WARI5VtMADr22aFeiW28QI3Gk9YjiWIfpmcoHUam9a3EpDe9v1c9/zSTu+lyh3S2ta01ZQseNcKFOeuilgm7hx5lzZZh4PP6EzxrmAjsz1zbCXk1+8EdS+cv1zimYT9N8gbzYjE3F7+NdBl7J1aD2ZlNA9ccGcJpc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044347; c=relaxed/simple; bh=2JsbmonhzJebd+yJNYZDQ4SlblGVT7Wq0RF80rcD48U=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=BkVkA2rLRPvvC3enWgChOOFigiHiS4h6EnLB13EjBa2xJUHZNPNkA/PrSuLCH9WAMfIpEVCdq/Cx8AjSbNnadHft/+KC1YXPeiY08lXA77sE4bcewnvfQgcpTUD7f2ST0mmN8dtDwRe2rh9HP0PhvbWRAj6bqG+NRCaMpHpqmI8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=ZAhFHFLI; arc=fail smtp.client-ip=40.107.237.77 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="ZAhFHFLI" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BagdjJeWYxehshpho/ZCUU1ve/5w9sKn90wVhKCjEqGHu1Jy97uWyiWjftZrSx9nt7po/eTrExEuZrTNqQCET3IKh6yhBXskWL8Zs1Ul3z4eeQRN7EUZVCPiiajIVunPbmHhkexC30RaQmU43DAttXbnvDc7/vzLJ+GVw/i/DnR3KRgAlnVM4ppLmg8q7RRMq4khzH/blLNeoFU1FR5k3/Xhudf2/94FvFXpHm5qR8tXNUohes18BItP7Tb3Gp2rTEW9pGG9NkblCG3Q6yhlYVyvvKhcS6esoaFAwKiCPb9eQCxXVJZGg9/GNndoDZfkjROxxrsRXUKM8ItOQuIeqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=X1UqyyqS99cOtGpEyaC22+Cd0Yn9s3uScy+YAJ1jqz8=; b=oZQRreDIx0fApnKWA/bARAje6jq3svlpPi0O4Wa9x4yWkrNk+GET0JtJrOhrTzJireGhPiBhElJMNdp6o4gFApsCmxHJr1cW5oQJwpjP9nSnve9I+gZraYwTWsRwm1z9p1WLRbEKIEfkhkv0ZSMP+7h2dKkENn/CuwzYKxMfdL7MadBpSHOh45kHnLH8X4cCKmTgl5V4adV71cT+J77UEWglLFzSRiGi8pxQ3eR1r1CmNAHjoCkFqg95l/IWN6iMea+wmmBirUNXyKifwr+V8TE2Q/1tWR83SIGe1hmM9TypF50D8mwzbBjc0rc4BNhy0dsaKfIuXCKZjw3E+hZkww== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X1UqyyqS99cOtGpEyaC22+Cd0Yn9s3uScy+YAJ1jqz8=; b=ZAhFHFLISDF7RjAg3a2jQnksRtoA6Cmsa+fyjmmtBkBEJT77siKaOHMW+PCl3Plf6BX5eMp0YQpAp0HBgizKotuwDUawud89DaHYNUXQ504ney2hYMp2llUZ+eDA8sbhsVaY7J38px9XGKSvxDNbQ1IefJz6Jx/EZV/0I0A4Owc= Received: from BN0PR04CA0020.namprd04.prod.outlook.com (2603:10b6:408:ee::25) by SA3PR12MB7974.namprd12.prod.outlook.com (2603:10b6:806:307::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.15; Thu, 20 Feb 2025 09:39:00 +0000 Received: from BN1PEPF00005FFF.namprd05.prod.outlook.com (2603:10b6:408:ee:cafe::fa) by BN0PR04CA0020.outlook.office365.com (2603:10b6:408:ee::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.15 via Frontend Transport; Thu, 20 Feb 2025 09:39:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00005FFF.mail.protection.outlook.com (10.167.243.231) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:39:00 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:38:52 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 16/22] sched/fair: Prepare enqueue to partially unthrottle cfs_rq Date: Thu, 20 Feb 2025 09:32:51 +0000 Message-ID: <20250220093257.9380-17-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00005FFF:EE_|SA3PR12MB7974:EE_ X-MS-Office365-Filtering-Correlation-Id: 63693129-f655-4846-99b6-08dd5192678b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?w5q/n6RJdzrNDYwUY5GkLYSSQizF4bII2L5V2jEROW1oCsolW9RTXkeQLdAo?= =?us-ascii?Q?+JvQpxl8Dlf1Y43uFim2IZsUlwQE3TykHt5qyyfEeYTJ72uMsz6OJrlDhv3R?= =?us-ascii?Q?piFEi1OC/E4i12UBmqAcBBlc7QfLaGLaOzBKkA15p9JgDzc6fmtBC4izGDFZ?= =?us-ascii?Q?xVme2qfF/huzh0KQvT0Iw/fDDc3UsAfoyC286IotKrxvKjBH3I553nChQt4L?= =?us-ascii?Q?dhe1PptoFqG/ELw81ZsRzL8rkXZNaMiuOz2oKyItkMe0gXPTD7f7TaMLERcd?= =?us-ascii?Q?t4XdWECyAj/xavGn3UkaafsmOJZ00BjtbgV2MCURw1Hury9ZJ2D4gHo+APdO?= =?us-ascii?Q?b0oPXG1IsxeTz7unRW9pyuG6Afhqn5UEN1M411ulcKLfmOhgKUMffFy19rv7?= =?us-ascii?Q?VIJcPaOg2zNTUBrtMRabp6qFKGXy6sGcDwi2sHE/Z6n7StKsH8XI3FdaTg67?= =?us-ascii?Q?5zev1A364X7NjtcFIJL3rhbNS19zXdiU7sqHO3ixlMWIDH/d/gOtlXY1Gamv?= =?us-ascii?Q?yg0G0L1TeA7QqBewHMxwA83r6yRg2W5LvD05eJdrdJk2ere8fIah8lkrpj0G?= =?us-ascii?Q?MIPFnbCDnIHlgozTaOQqLsLRB6I8L6NZTk0kUAR8CRRRo6QpeYNO0+hUEy37?= =?us-ascii?Q?rPhglEMgTvypM2DnTPaGw4EwQeb83T/O0yvU4mSw53LfDJ9rPmhCLJpVlOlj?= =?us-ascii?Q?Hr60Mf/2Q1rfkM/lC0qAasNCbcVFTzSanlSO6e1bA1bUM+534FJ5X8cE2kI9?= =?us-ascii?Q?vBIGEzSW5WgD3v24p6yJuGHpvN1lsulEq22bc3uJ+/O7iwCfcTrESBQCXqrw?= =?us-ascii?Q?GuQQmzTHv+mvCPODlkBBa2sBkoU22PVth0BZnW/2M5AFHHNkMfkNPWrcE2kW?= =?us-ascii?Q?PSDlVA4Z6DoTk67W6sX85ZNdBjYRkIN51aS3lQllhdsrrEMVvRs/cZhqy8eT?= =?us-ascii?Q?SL8IbtFVqTFAOkfUJr8jTRk1ZtDkZLlr12/LBLi+XqbmZz9XZiok153n7/M+?= =?us-ascii?Q?8fkDXm41/iki3ea1t7/2N03B5ZqN16YzNRgQHlxJMIyXfYajDaKI0dlejSYz?= =?us-ascii?Q?QyZ6jj6GIEGxW8mwPs9djOuFTuQ6buGsPSNIss1CkILDLG3OUpJsEG21Gkbu?= =?us-ascii?Q?zwv1RqlelL+6QMZmWFdHDoK+xeRuPCjWYltfghl/HDGXJRgn32IbmVB/OUfJ?= =?us-ascii?Q?q6KBPFvq04kk1wkkH55xuDNZgrUucxu1YvZiXY776c728gAdGl9AssMEfxke?= =?us-ascii?Q?MLFxDFSvBXtjnKxRZ4se8qQ8K7/TTNCzwh1a0M4ngAU4Kjp6yRG/8q9yw3fo?= =?us-ascii?Q?sxsrt7ZfIPNhxlKbLB2+iXafxpEYO150HFUBHVhFWW0kL+/cVX+db6EF0kVo?= =?us-ascii?Q?L7ts8KeYkuM7D7Pixay9z2rw0IiqEbZBObSZpKGMIUs1cTU1INVHVupl14v6?= =?us-ascii?Q?kk4j9uzzmuaycenvX6H4iIMAzEtyPmwqhfVTHQh5++Y7vleQ2McmgEdfeKIS?= =?us-ascii?Q?keCGt0+1n5Eb5CM=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:39:00.0130 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 63693129-f655-4846-99b6-08dd5192678b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00005FFF.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7974 Content-Type: text/plain; charset="utf-8" Partially unthrottle frozen hierarchy when a kernel mode preempted entity is enqueued on a fully throttled cfs_rq. unthrottle_throttle() will be implemented once the plumbing for partial throttle is complete. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 56 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 54 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 0332e95d36b5..3bcb56a62266 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7117,6 +7117,43 @@ static inline void account_kcs_dequeue(struct cfs_rq= *gcfs_rq, bool in_kernel) SCHED_WARN_ON(se->kernel_cs_count < 0); } =20 +/* + * Unthrottle a fully throttled hierarchy when a kernel mode task + * joins the hierarchy. + */ +static void unthrottle_throttled(struct cfs_rq *gcfs_rq, bool in_kernel) +{ + struct rq *rq =3D rq_of(gcfs_rq); + struct sched_entity *se =3D gcfs_rq->tg->se[cpu_of(rq)]; + + /* TODO: Remove this early return once plumbing is done */ + return; + + /* + * Demoting a cfs_rq to partial throttle will trigger a + * rq_clock update. Skip all the updates and use the + * last updated time. + */ + rq_clock_start_loop_update(rq); + unthrottle_cfs_rq(gcfs_rq, true); + + for_each_sched_entity(se) { + struct cfs_rq *cfs_rq =3D cfs_rq_of(se); + + /* + * Fixup what is missed by unthrottle_cfs_rq() that + * enqueue_task_fair() would have done. + */ + update_cfs_group(se); + account_kcs_enqueue(cfs_rq, in_kernel); + + if (cfs_rq_h_throttled(cfs_rq)) + unthrottle_cfs_rq(cfs_rq, true); + } + + rq_clock_stop_loop_update(rq); +} + #ifdef CONFIG_NO_HZ_FULL /* called from pick_next_task_fair() */ static void sched_fair_update_stop_tick(struct rq *rq, struct task_struct = *p) @@ -7224,6 +7261,7 @@ static inline bool min_kcs_vruntime_update(struct sch= ed_entity *se) =20 static inline void account_kcs_enqueue(struct cfs_rq *gcfs_rq, bool in_ker= nel) {} static inline void account_kcs_dequeue(struct cfs_rq *gcfs_rq, bool in_ker= nel) {} +static __always_inline void unthrottle_throttled(struct cfs_rq *cfs_rq, bo= ol in_kernel) {} =20 #endif /* CONFIG_CFS_BANDWIDTH */ =20 @@ -7444,8 +7482,18 @@ enqueue_task_fair(struct rq *rq, struct task_struct = *p, int flags) h_nr_idle =3D 1; =20 /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_h_throttled(cfs_rq)) + if (cfs_rq_h_throttled(cfs_rq)) { + /* + * Since a kernel mode preempted entity has + * joined a fully throttled hierarchy, unfreeze + * it. Since unthrottle_cfs_rq() adjusts the + * h_nr_* stats and the averages internally, + * skip to the end. + */ + if (task_in_kernel) + unthrottle_throttled(cfs_rq, task_in_kernel); goto enqueue_throttle; + } =20 flags =3D ENQUEUE_WAKEUP; } @@ -7471,8 +7519,12 @@ enqueue_task_fair(struct rq *rq, struct task_struct = *p, int flags) h_nr_idle =3D 1; =20 /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_h_throttled(cfs_rq)) + if (cfs_rq_h_throttled(cfs_rq)) { + /* Ditto as above */ + if (task_in_kernel) + unthrottle_throttled(cfs_rq, task_in_kernel); goto enqueue_throttle; + } } =20 if (!rq_h_nr_queued && rq->cfs.h_nr_queued) { --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2072.outbound.protection.outlook.com [40.107.100.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E40F51EE7D6 for ; Thu, 20 Feb 2025 09:39:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.100.72 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044365; cv=fail; b=kPIVzvXhksOb0M+GpZUv0C0w2lklJ3y2cGPqS9iC2RmdRkO71hmSoJH9v+KKOZkjAdIb6c2fWnvyzBgLQhgTXp/o8V0ihy3yUYVzriOSRNIU25eAOuemPs+FUV0Nizh4meNvNWsaRhveCQsrRYFyf9uYzUmHvtwzAuPsd/Ih9/U= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044365; c=relaxed/simple; bh=FlZa9MtP1VmQZ6vjSiokY21w30KRVVVwZ6V+WOLP7Ug=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=a5kc+kv6MEM6kSzQifcpJKJ3ho46HxyL4ZSKsY1bwx0nT5w9yH62bVYXn6FY8QtWBWDWzMgHwdEEunIV/k9gnlh+O8uTVBH6tRWmWRQ+tySVBg1ePW2DuUloryYo4O+TF6IY0QUpINmqvPSIYNJYTAdJPg1F7Xq1xy46EfVN3wk= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=0GJ1RqFd; arc=fail smtp.client-ip=40.107.100.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="0GJ1RqFd" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gLgYrI2xWYUoI0DuqUEFT8x4lNUiS1c6sjRboQwjJYI4ll+pTmCJAWnXDEp3P4+REiRl9lFuUyULu3FEBeCf7lRGeJamB1G0dWALXNdRmqbUhdgog8FCSdJhlMYVp48lElh/Cepr/LKIArFOBTj/SrLBpfYSvwOgFQ17+hS1XCDykRhOljAqk85+NZRPrtJrbNsIJOZDhPOyiUzTEYsB7uz9GJcJj6K3GpORCNC2DBQa/Uf/5Mn0yo4PFO1JAC/j8+ms0pec2x1DlukKke0tg8naeV3zdhtPG2TG6ubIA5TGsgeHFz6LY6H1OKX1H4M6LiAfkd5VluaDd3FPTuYKLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aXc+PBLmf9U1pnN5cFTU9KvcxVe9yZz/72X97T40q8Y=; b=U1IFfg5DS4+7L2raTp3eV1QezfiAmNk4AqRJoeY3Y94SVeXsiyZrawO1MMkcUF02twylc+8WQ+tTDKTShnKzfq+M6ubYMLVS8usE0Gz4IIGN/sQD5T4yb9H6GNJcMzQmPpVIVWmz3MJi0wTHTl85zWpOr+UT0tEJviy6c0dPtOzDjDypRx+7kOQZpl02ko76D2lQv5oM5yBXDJlpEOC9agqjKoIvgBsPM+Pz7QfSqLelVsAoPxoQT5Ta7kdts53g5rZrKK8knsYLWJu5aGM8Y8xrJ9DMklI3MXDWpv/SVu8C9fw+hMaJ4ul6E6lahod9ZnZCIgYjYMc1nm8HTTbBaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aXc+PBLmf9U1pnN5cFTU9KvcxVe9yZz/72X97T40q8Y=; b=0GJ1RqFd9/46wMuDv5woekLJdHhwoIhdA27+NbbXXiWuYiyc/6k0UjDDdNpG202p/JavdQCCzFGAKMdHGED/dCoioadRp5/tWI/GZ03xlzJdbgNB1iT4X7OKfMJsSa+i9rP7biB5fyvAtfK+k1kFwFTfHPo0n1TVHWdDJ73wAO4= Received: from MN0PR04CA0026.namprd04.prod.outlook.com (2603:10b6:208:52d::17) by IA1PR12MB8468.namprd12.prod.outlook.com (2603:10b6:208:445::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Thu, 20 Feb 2025 09:39:21 +0000 Received: from BN1PEPF00006003.namprd05.prod.outlook.com (2603:10b6:208:52d:cafe::61) by MN0PR04CA0026.outlook.office365.com (2603:10b6:208:52d::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.15 via Frontend Transport; Thu, 20 Feb 2025 09:39:21 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00006003.mail.protection.outlook.com (10.167.243.235) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:39:20 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:39:12 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 17/22] sched/fair: Clear pick on throttled indicator when task leave fair class Date: Thu, 20 Feb 2025 09:32:52 +0000 Message-ID: <20250220093257.9380-18-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00006003:EE_|IA1PR12MB8468:EE_ X-MS-Office365-Filtering-Correlation-Id: 4793534b-5a3e-4dc3-4ae8-08dd519273f3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|7416014|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?02TV1xn3jo/OhxExNoBxh1Pa4iVWO2Ce4OeHnpLn9fz1e+b7XB5erKdrI8ml?= =?us-ascii?Q?yBT38VtCMCXmTSzcBtGyEsdM7CSShFMiB84x89oNQOqGxdMQ4ibmShg9fNhX?= =?us-ascii?Q?jbHOiBeTcoci5hDZoqU90U3iGS6SzqOGb19K0QAPHOFUgsWfVNtIV/DfsOCm?= =?us-ascii?Q?5SokJaobuzG/DHlxMaGd49kv2gYNjE1Ns3NcIrw+wYaS6mkHyxiQI6dZwJLK?= =?us-ascii?Q?1RKlhdeBDvEgFrKY93uRW/1BFgR/qtjFDOPSj2IUueeeAJlPJ/igvgguPlcD?= =?us-ascii?Q?oscliGKjixemx0LlFkwgdGsV+89d6/v5Jjr2/lILaGBuPuloNCHEVzQN+5Sc?= =?us-ascii?Q?mMculfsoiu2NpX64Yo1gS4yWVeKyKVRI0U2VtFojB5WH7GdhVYQRhbA8cRFV?= =?us-ascii?Q?4sL4VpJF711KS7b5uL60KuegFSNf41jk2lKoFHHHRCoSpnWVejcFH3Mn9+eg?= =?us-ascii?Q?DV/vmI6nrOaN5c0/YhwHLBQLa4Q0hqQstmGqhqtrB8Rlnudk+Rv5QduwXXiL?= =?us-ascii?Q?iMMjHQGJR+JNK0McgXLBaB/JnKlalPOp7OeXU0MmJctNex6IEkHAApvnAdjz?= =?us-ascii?Q?b7DXM61/u8AtA9hoi+aSn0ic610NWSgJfYxkF8PWF5hAlswWqvKDlI1Gq8Je?= =?us-ascii?Q?tNhy/ZAMYPkzmhFyYUyr71b4V0mMRIEL0pXZ2cVhK4F1rr5TuNlk2Yvt7fkc?= =?us-ascii?Q?kpSbyv3/+oybf+qSf7SRaHMhlUN191Y4/nJZquHGogNnZ9nUGo2cL+HY5x5K?= =?us-ascii?Q?Ow6zsyaySwZW64adQzJucDxbNKHsWVVYejGLnRzaJejw5rXhx49dJUEst8zQ?= =?us-ascii?Q?+wQ8J5dyZGPAhnC0dHGJsytpJITROYyjznPU0S504ihKTcms2gx0yOHfnzXx?= =?us-ascii?Q?e5SSxhRBL4rREZXGs67ilokf0YqojTWKtC80BHOxeOBybaZvQS+8N/DkGpo4?= =?us-ascii?Q?d4oWAFKs88mwRq0gAsmBDMWpvu9RSKrQ37ClPMKl5VIJ3Gqyeew0hnkCvLvk?= =?us-ascii?Q?GKOHwIahBaj/C5jK9oTQ0bNghAq9HE3n2sMtIcNff/x2jTxzBz6spECe6wkC?= =?us-ascii?Q?1JDzD1iE5+x4PmNkILBANXUn9A4Nsk0+/ELYoavbznXaiK2xXIV1pcr7F52z?= =?us-ascii?Q?DrxnCOmK75tBamCSmr1LCZrboSTTxxxEI/EMXdGVQJ9nFeOlwgqdeDFZWNEA?= =?us-ascii?Q?6IoiSQsT8qKEvQwXMPQchJB61RjJrFTtJzYssy57B+NpzogdjtYiABwgJrzi?= =?us-ascii?Q?eNFxyGosMWlJn8scLCVu2ycU26lbB/5K/yFaHsLzmPk8C+kUHM+qM5uoHs9M?= =?us-ascii?Q?D9S2jHB7vQx1h/OrIrpdEx535gHyoljEiJ6cjfFYMUP6Q1uTqxRzsyRx2EMB?= =?us-ascii?Q?ER3GFjdhxa4rPBp7pbtVPykDdW5thOtkxyPn5iGfde4qRa+citlTdMiocOCj?= =?us-ascii?Q?JI7u06tpP9HW6cpbJUvhXRrIi/z9QvYsHR6tEeyO+iJ9ArGiINhEMnlpZQBv?= =?us-ascii?Q?+Nt22V2vGtR9e/k=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:39:20.7627 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4793534b-5a3e-4dc3-4ae8-08dd519273f3 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00006003.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8468 Content-Type: text/plain; charset="utf-8" Prepare switched_from_fair() to clear the "sched_throttled" indicator when task leaves the fair class. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3bcb56a62266..423c5a95989e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13805,6 +13805,7 @@ static void attach_task_cfs_rq(struct task_struct *= p) =20 static void switched_from_fair(struct rq *rq, struct task_struct *p) { + task_clear_throttled(p); detach_task_cfs_rq(p); } =20 --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2043.outbound.protection.outlook.com [40.107.244.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4AFD61F03C2 for ; Thu, 20 Feb 2025 09:39:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.43 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044386; cv=fail; b=DRnbnTbjA0ccYRMq1fhDlEufARxaYWworIXNkg5JLR/y+7QkqGR+CAH6AvKf+uNHNYOobnfS3ktLPT3RbKlQUaMCDPxr4shvV0g2hrqxjrqR3MyQ6Uamx5TLQl6nxBt6hEFtGuadGo6Fn7wo4ezZ1qtkJXM8hW+dpVODLpxgZdY= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044386; c=relaxed/simple; bh=xSOHQHHR9j/ry/OSXFar0Zwu35XBN0HUUnugv1teHd8=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=bK+0v2J1lJd6tzGK0wFe4oEHnr5u4J39pRyYvImH9fsFvakwGeQhFlvSIOQSdBot6QK5HLV0DC2pHHJNG5xRENWLzihMRm0Pw1RYQ513nmV1egiXoRvBL9VkXOyDfUBW74VshSVCwUjwj3qRxotoF3PDi6pIxeJk9eCQlbTWTQg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=bcUq7QW+; arc=fail smtp.client-ip=40.107.244.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="bcUq7QW+" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gd4Q4nw7v2zwyWG22vLCL2t7IPkEzq4o9lJIJ4v1sAnabTV6abOqovZ0TbgR2n9SKvvcgOgjyRLZXzXRHzJHBij8D+TrtypD+cQVmPFI6hHk3mZq6l7fTwJiXJi+V0dWaFvzsY6SbpmKdqcqhLHzr/xcIXT9W9JU3M1hKcemWKUZuisg8sCcAAqj0AOhM3Az8Q++p225sBKOGZeTzl9PSAGkVW2EjMTWQ4IXC4aJKAgzZclEEuqPITh+y++nishqsFlgMVY/LqHkxZVHR3wFQedxhj2YRC2Gd8Jlw8jCMvSKNHbmIvBycL5GG00/5QjHWEAg9uHNdE4Ohl9HA+yDog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CXG+Lw8fG34xio6CV1T4mzUP43ASj9YircPN3WuDJmI=; b=scEZt0/AiYrPBwIA1FZRtB3+kTA1btSWG0eXoiwS0W8z1Bi6UZlGKfF7K2P5mNKM80JL2JHTdx6RpGY8hcTTUCADL0uwLx2PoNmVGipzhUk4j0l+pNOClZlc7ocrTrUvK0TVHINn2hhwjv7yJc3Z/38jcJvKh9Ea9kM/v8vncJjiKSx4fuuQLuA6iaNfSbNZ3QfZsaghAQCCa6O8fzQP6/FDnt8t+Ct4SEhOuainW/MG3j7fawOBgZE0EPCkyjx5ieLvm5C8qgKoEoZAo50xI2qW6jPwmuWPnsN2+8lr+/WY8v3UBcLNJapLucUz1CluqQtethX0TmSzHfEXyKwkeA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CXG+Lw8fG34xio6CV1T4mzUP43ASj9YircPN3WuDJmI=; b=bcUq7QW+RnYeEdo4V0S5Ub5hWVhFx2l+sSFfWrbjcSAu1FbQIVGHIc1KQIDNW6fDVAEK2Kjmn+nU4p/KMtHzRvZ+6XjBafa+N0JGChsWcCoXenK3a9bA/zCYdC61bqT/cF1BT5pZE/P3bMp2MIZEtstP44VCxne44aGftlCCUGg= Received: from BN8PR12CA0025.namprd12.prod.outlook.com (2603:10b6:408:60::38) by DM4PR12MB6374.namprd12.prod.outlook.com (2603:10b6:8:a3::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.19; Thu, 20 Feb 2025 09:39:42 +0000 Received: from BN1PEPF00006000.namprd05.prod.outlook.com (2603:10b6:408:60:cafe::9) by BN8PR12CA0025.outlook.office365.com (2603:10b6:408:60::38) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.16 via Frontend Transport; Thu, 20 Feb 2025 09:39:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00006000.mail.protection.outlook.com (10.167.243.232) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:39:41 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:39:33 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 18/22] sched/fair: Prepare pick_next_task_fair() to unthrottle a throttled hierarchy Date: Thu, 20 Feb 2025 09:32:53 +0000 Message-ID: <20250220093257.9380-19-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00006000:EE_|DM4PR12MB6374:EE_ X-MS-Office365-Filtering-Correlation-Id: 67f52316-b2f8-46ae-9544-08dd5192805e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|376014|7416014|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?CxQW7bwgpfXg25aL97GPifQFRmHzaduCZFDv4TiMr+EIHjSrU7YrDm3zaRPO?= =?us-ascii?Q?n2A94ZBhQKmF2kCAQa2l9xM9lGGYai6IIuNSesT95EhM/bzLDKLuzAYDKLIk?= =?us-ascii?Q?SMI2zGTnQ3kByKxq204r/L2wFwyy6qwMGp+aeeLxpHIAywD7mK/OzatJSpkY?= =?us-ascii?Q?fefZO83DpspZ5C65/RuAT/hEUU0voEbJwDXiYlcb+GPZGz3lNXJiJALupApk?= =?us-ascii?Q?SHVvFn9M0eUyqM9JHfi8umkchztkwczGqpO/xYGk/E/NooyguWE/zEY427E2?= =?us-ascii?Q?AmkeMqyo0zVo8B185uUElp+M7TChom3oARD4vu2859abRCw79SXHwwIW4lPk?= =?us-ascii?Q?z87VKm11Wckg8N5kCDKd9v+1IGAgTkRdJ9ROrwKkacPcyjbt0VCcgQaS+1K7?= =?us-ascii?Q?fXs4Vmh4iSXtqCPLNs2tugkPd3us35m8puGoi5CuLoeyasKoninTdt6O8W3H?= =?us-ascii?Q?g2Q7gtagTGg6iDt2UH5/WTOgU1hVLpkWaFxwHG4/1RT2EjoNLGxK7LDUzBUT?= =?us-ascii?Q?FwuIW/0b2X9KOOHypOsNYTXmqp2tF6U/+8GRR3M4NtR+B/tGNq0epLRmB8Ka?= =?us-ascii?Q?PZE1HEXcSGYduoaEUgu5ed5GZ6RY0d6pQQ4YrEWFyQT3olpjQ3yCGCtFI70k?= =?us-ascii?Q?bxPDvwvrNbcNaQnq5UQLqiEUQPPH4eqR1OhIdJtqRh0mSp6+lPLCRiuniDb7?= =?us-ascii?Q?QbsgsYyiRGwmdPGgaEp1R3N8iO1GHEw4LS1C7RGH6hBCJCP8wh4mUIxay+sd?= =?us-ascii?Q?IOZ5b4sBkib183AgQB1NdlyVQd8NJOnz7dzJcaXWmHF+G81qRVk877pyEekM?= =?us-ascii?Q?LbMMTbTJH0gFjDTs11JCCfc/TjYOXKq4DfNJ9N+cICxJR4T99ohw2wu47N2X?= =?us-ascii?Q?NIGOLddyxz2aWZr5LC7QC01b011mhsJnbom5hcNSogoaigTRCooHNtEr9p37?= =?us-ascii?Q?70s9L08w0s+7T8AzQXWXj/gtMg1dxWzzC9cyeReL1J/oVe7JTxnapNHgqxpa?= =?us-ascii?Q?dLDFnulPONIonz69jRksmGbtnE7fVfArkPJsi5X53puJLx4vUyMxboAH4xEL?= =?us-ascii?Q?HDeaFwV/3jrjx9SUCImNRXDh3kd23ShYuPLm5aFqU96tYq/6ZO/JPlyrB70J?= =?us-ascii?Q?ciKGXAD5f0pEcXNbsmTpQGyyqsCdUceiYDznRvQZBs+r2pBBYgZi/gGka3NF?= =?us-ascii?Q?Uz+DwHyGbO0AW+7fjxBo/NAmJthK9cmA7UCYF50QKnv+YqIlVcOYJH5xclY0?= =?us-ascii?Q?jXzgs+LYtOJzHZp/JH/zbV6qBpzcj0fNaaY6Q43msh8ChwRjilw1rvx4tR44?= =?us-ascii?Q?wkCdsfYExVwFCO39TtpALZalHJYsNke02C5+3aFNb7QdE4It1ZYLDqX3+zjH?= =?us-ascii?Q?1l/ASBsXNFV3Jt5BRmKPN6LJhPXKh7mvLEviMZKmSV0TeEwYSqRzyfTCd1jj?= =?us-ascii?Q?3OdC9RsTRNOOsSCnluS5t27YatH17EjCve2KCiPWND4haB5iuEfBVaDBAZu/?= =?us-ascii?Q?RjT9NUcMf3sT6Lc=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(376014)(7416014)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:39:41.6767 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 67f52316-b2f8-46ae-9544-08dd5192805e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00006000.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6374 Content-Type: text/plain; charset="utf-8" Kernel mode preempted tasks being put back on a throttled hierarchy needs to be reachable during pick. Demote the throttle status to partial if pick_next_task_fair() finds that the previous task was preempted in kernel mode but is on a fully throttled hierarchy. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 423c5a95989e..1497b0aed1c2 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -9407,6 +9407,38 @@ pick_next_task_fair(struct rq *rq, struct task_struc= t *prev, struct rq_flags *rf struct task_struct *p; int new_tasks; =20 +#ifdef CONFIG_CFS_BANDWIDTH + se =3D &prev->se; + + /* + * A task on throttled hierarchy was forced into running state. + * Recheck throttle status when the task hits schedule since the + * "kernel_cs_count" is stable now. If task is preempted in + * kernel mode, partially unthrottle the hierarchy now for it to + * be reachable for pick_task_fair() that follows later. + */ + if (prev->sched_class =3D=3D &fair_sched_class && + task_on_rq_queued(prev) && + se_in_kernel(se) && + throttled_hierarchy(cfs_rq_of(se))) { + struct cfs_rq *cfs_rq =3D cfs_rq_of(se); + + for_each_sched_entity(se) { + cfs_rq =3D cfs_rq_of(se); + + /* There is at least one fully throttled cfs_rq */ + if (cfs_rq_h_throttled(cfs_rq)) + break; + } + + /* + * Only unthrottle; Do not adjust "kernel_cs_count" yet + * since account_kcs_enqueue() below will adjust it. + */ + unthrottle_throttled(cfs_rq, false); + } +#endif + again: p =3D pick_task_fair(rq); if (!p) --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2042.outbound.protection.outlook.com [40.107.94.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57C1B1EE032 for ; Thu, 20 Feb 2025 09:40:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.42 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044410; cv=fail; b=rvJ9BTUzD2xLoJG4LLZtqmm9WNPKHjoVPbkRm09r8xIgfjDwV4U5JJK79n+igIl0nlWXbMXJjw0aD2RbhUwCHam01xowPX0MHuefjCKLwqm+3OOQ4mxn7flrL/3XTvj9HrZtwKb4aSvj0qIe3UOWYJJBICxzUDKCDtxEFSRmn94= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044410; c=relaxed/simple; bh=WfEPtldfjL99H30wLhN2r3QFN0VEJvoj4Xqn6AQ7i8I=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=IYSGLlCg/94ymTpcM9TV+HywMtwZr+5Z1edrWicTKnzDKTPYC6SUKUWzgthk6T50lTh73Cltjnl+tAujMKAPlI+wt+RZ9k3gS+SgTXxjvRqNKwijihcqfbXqEMKHQOatD5yRWvjxW/gilqV6z0+7v68qh2w06K/9paUNyBg189Q= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=dT/xppOJ; arc=fail smtp.client-ip=40.107.94.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="dT/xppOJ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=dw4C7H0/VcdQjw4uqCXturVd0vDXtBfAhkGETx6bxmHL0xQ0N2FIk6QAqUpvSOu5bE5anMN9FG7qmwS9TdyPsVOjBidfSyPIbrODEMD7rE8l6iyr+WT9QdsBWXQgcNm+8v+8MOFlw+moCBXUAPaBpOh6HVKEhAt8T89dudTnuOQPT85vwGySunhR45Sipn+pVjsdQT2bL0fHGnABmjVmNIVF6l/4vOHgP8j99vK29L/m4xurWyAo+Q7wLsLWFgcbJ/aDFJV3nipQHwZS+82E1vXooCjmmKdpmzelraOLviqHiE4Ar7Cs4PRvgSKHC3lKQ0pNpr0660G8GPB5z2GpVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IaW/47/AixTTJrYcn0CU1cejnepHc91zlsswqtw+Bw8=; b=NJBfW/9pdzyZIxBWlRiGw/09IbMUJBygASaBMv5ECmmIZH2FjAO0eCrj0FdmXDeFb4XVxxfIQIsdShr38UJ5AUD5C5LNv0n8rJrfg9EF6Cljo0W7uAXLuSN4JVs3fB444XBc8lSG9AVQ4/5S2+1q/uP3QzMu7CHRaZk7yca3z7sIowltltl3fELhtwEu6OfacwaAe+l5asuhq2EPXPG/p0qN4aq3aRKBxIpUkLKIpiNdOErxvyg/lRYhV6Kq74FnLxChdHQP4wiDCVNfo5d4c4cFaDlQJEt4zJd6UbWiazdueTVhA/5xCjwYHLtTesXVqwgXK5dzI65EWW/kp0zmoA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IaW/47/AixTTJrYcn0CU1cejnepHc91zlsswqtw+Bw8=; b=dT/xppOJuMxWQNFR1zXgHTGqRPt+6LHX5latq4VkJ8bpNKY9f5DhgBpXZA7igFiLhOdljpi10z69EJ92/LqxudjIX78GONPcK3iKE42QmgYW/8ngcAhb9bB6G04aEyPa6BDZRJaVYibFdGFrREIDKeY2xxOeeso+2upkTfOG7Qc= Received: from BLAPR03CA0082.namprd03.prod.outlook.com (2603:10b6:208:329::27) by PH7PR12MB7965.namprd12.prod.outlook.com (2603:10b6:510:270::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.16; Thu, 20 Feb 2025 09:40:03 +0000 Received: from BN1PEPF00006002.namprd05.prod.outlook.com (2603:10b6:208:329:cafe::5c) by BLAPR03CA0082.outlook.office365.com (2603:10b6:208:329::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.15 via Frontend Transport; Thu, 20 Feb 2025 09:40:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00006002.mail.protection.outlook.com (10.167.243.234) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:40:02 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:39:54 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 19/22] sched/fair: Ignore in-kernel indicators for running task outside of schedule() Date: Thu, 20 Feb 2025 09:32:54 +0000 Message-ID: <20250220093257.9380-20-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00006002:EE_|PH7PR12MB7965:EE_ X-MS-Office365-Filtering-Correlation-Id: ba7c6387-d35d-4ebd-0a50-08dd51928cdb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?hj5MlYNgu4z8Vav1xDyQ61VyK/zwwSKMKF5s5YsfaZO3c8N/Rfgyk4sFCgcl?= =?us-ascii?Q?Y5KyUsPH23cGrRcjQ5pOMEfs21N2xwiqWbWHD9kp2KOOq931bB1x/YwjyJKs?= =?us-ascii?Q?aHzKywfOLog2Bx0/DstEDnC+wjha87EWGotDM+UqMPMAPPrxZ4j6pYMTr1ra?= =?us-ascii?Q?3y5ZHy/7lBPND9pacS0J5MDxrLvYwou+xyUOXskKC2t9qgPvI/pi9qsKF5fx?= =?us-ascii?Q?FMSsOFYt1s656zy+zaGcwJ5tQMFgxchYD6vec+bWdamWznhD697rB64BdBaT?= =?us-ascii?Q?dVutr2kk+JjWEU7kMV3HzZZHchMoWfDf/BvtyJYE4Bt/L6vnYhKUCKtn7Kqj?= =?us-ascii?Q?yiKQ33jA8pVy0sZZ7hwds58lxOKk39+gqjEqFK/BUbLTHmrrncRIbXRWm2nr?= =?us-ascii?Q?Vl9cOniXtbYp/p1xvM3+RxJBgHcw3yuEdsTy85649QGS0ThPWl5+v4cxqdoB?= =?us-ascii?Q?5hyTdwZ1EDR7kZFrr/4R2TcxfSejSPJ4qQuFBT6TUtZkenqF3BYkqAbDrOEF?= =?us-ascii?Q?U2kTf6eDbNuYxF1M240bkl+Rvm2+sPZ9rwINKzt6MWDHSbu/STZvaKklwEi6?= =?us-ascii?Q?lng2inoeLY1Kv3lFJZ9WN+gI+CPeAR9weO0G+kyS5UQYn6LtEszXHnGQG1Z/?= =?us-ascii?Q?Kf+/J0xHVavgI3PKryreZvrB7cuf8mkMmyHXxHbeUvFcC9Us3U+GDlo1uVCf?= =?us-ascii?Q?bHUZcm5eLGqOw+5I9ZfZWNBDqXxJ/cy8Flnd6WS1JSe19hbvm4rSlJrMTiSa?= =?us-ascii?Q?NrlquhEwCnQZ6L7vmAmQMMIBVeD8SAXb/QmNA0G3gPOfuyRsHC20oo7++yOh?= =?us-ascii?Q?vADlO9TUWy2KHg2gNDo6rs4ulFws4zmPRHwJ1B8NU7vlelH9gpw2K5QHrfqQ?= =?us-ascii?Q?NMKBnRNoivKDk1SOeXW0RUPbgmTZl147SKhFYr510m1jHgjpskxKS1WS+B6c?= =?us-ascii?Q?Z5sL2p4QTrODJaf1vQMyJXhSushI2w6veCVC69K5nsGgQVT+acD9EoCuHnel?= =?us-ascii?Q?puxhh6mFiHGB83G2PJeiWBp+RSG7zWjX6sPgwzaDO7mOj0cuevwFGMe6sW8o?= =?us-ascii?Q?6q4YdDBbVN7YfgvlwMhZAKOah3vRPP54rFjp4PPbzrv+30wC0bT3d+PjmUUF?= =?us-ascii?Q?Pe7IB8BD8mtjpYLM+AQ64z21hJ6bQ0MNQcCSj/D7pkUWisdX2xQRPnDJcMxc?= =?us-ascii?Q?QQE41FBc0e7FuuD5wlIVJrFaw1n69PGm+5v3B87CsizJtd+5WLWp/KWcWnMF?= =?us-ascii?Q?SZLGwvl73ZLPX7yO4Xyk8vKUuqO7kD5IuEtKeG4dwHqnHWcvfwra3V4iG19t?= =?us-ascii?Q?03VUP4Sj7jvo2ViFyq9nuG26+70JkS7RbdmazPV3C/NqthXZLX9GUoeA5g7a?= =?us-ascii?Q?RapyKzvkER5zX+PP7GGCQvyi5Y7/FZc6VMu9Hxl7Moa/20vaeY3TMOxse4wV?= =?us-ascii?Q?1V+5XeImGeslZ0HvLhVPCNIh4GTYigycoxZQi4ZuFntnhF/PvnYkW6cncXac?= =?us-ascii?Q?4P/tlDpFf1qBBgE=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:40:02.6093 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ba7c6387-d35d-4ebd-0a50-08dd51928cdb X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00006002.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7965 Content-Type: text/plain; charset="utf-8" A running task can go a dequeue_task() -> put_prev_task() -> enqueue_task() -> set_next_task() cycle when on_cpu for a number of operations. Since the task is running on a remote CPU, its "kernel_cs_count" is unstable and looking at it can lead to wrong outcomes (imbalanced accounting at __enqueue_entity() and __dequeue_entity()) Use "sched_throttled" to indicate that the task's kernel mode indicators should be ignored. put_prev_task() is called with prev set to NULL only for save restore operations and is used as an indicator to set the ignore bit. Subsequent call to set_next_task_fair() will clear this indicator. There are cases where the save restore cycle can fully throttle the task's hierarchy. One such condition is: dequeue_task() put_prev_task() # Sets cfs_rq->curr to NULL ... enqueue_task() enqueue_task_fair() check_enqueue_throttle() # cfs_rq->curr is NULL so goes ahead # Full throttle set_next_task() # Sets cfs_rq->curr back If set_next_task_fair() finds a task marked for ignoring but has been forced into running on a throttled hierarchy, request resched for schedule() to partially unthrottle the hierarchy if required before going ahead with the pick. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 115 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 111 insertions(+), 4 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 1497b0aed1c2..55e53db8da45 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6837,6 +6837,21 @@ bool cfs_task_bw_constrained(struct task_struct *p) return false; } =20 +/* se->sched_throttled states: Modifications are serialized by rq_lock */ +enum sched_throttled_states { + /* No throttle status */ + PICK_ON_UNTHROTTLED =3D 0x0, + /* Task was picked on a throttled hierarchy */ + PICK_ON_THROTTLED =3D 0x1, + /* + * Ignore tasks's "kernel_cs_count". Used for save / restore + * operations on running tasks from a remote CPU. Prevents + * enqueue and set_next_entity() from adjusting the stats and + * puts it off until schedule() is called on the CPU. + */ + PICK_IGNORE =3D 0x2, +}; + __always_inline void sched_notify_critical_section_entry(void) { SCHED_WARN_ON(current->se.kernel_cs_count); @@ -6847,6 +6862,8 @@ __always_inline void sched_notify_critical_section_en= try(void) */ } =20 +static inline int task_picked_on_throttled(struct task_struct *p); + __always_inline void sched_notify_critical_section_exit(void) { lockdep_assert_irqs_disabled(); @@ -6860,7 +6877,7 @@ __always_inline void sched_notify_critical_section_ex= it(void) * schedule() soon after enabling interrupts again in * exit_to_user_mode_loop()? */ - if (!current->se.kernel_cs_count && current->se.sched_throttled) { + if (!current->se.kernel_cs_count && task_picked_on_throttled(current)) { struct rq *rq =3D this_rq(); =20 guard(rq_lock_irqsave)(rq); @@ -6873,15 +6890,40 @@ static __always_inline int se_in_kernel(struct sche= d_entity *se) return se->kernel_cs_count; } =20 -/* se picked on a partially throttled hierarchy. */ +/* task picked on a partially throttled hierarchy. */ static inline void task_mark_throttled(struct task_struct *p) { - p->se.sched_throttled =3D 1; + WRITE_ONCE(p->se.sched_throttled, PICK_ON_THROTTLED); +} + +static inline void task_mark_ignore(struct task_struct *p) +{ + WRITE_ONCE(p->se.sched_throttled, READ_ONCE(p->se.sched_throttled) | PICK= _IGNORE); } =20 static inline void task_clear_throttled(struct task_struct *p) { - p->se.sched_throttled =3D 0; + WRITE_ONCE(p->se.sched_throttled, PICK_ON_UNTHROTTLED); +} + +static inline void task_clear_ignore(struct task_struct *p) +{ + WRITE_ONCE(p->se.sched_throttled, READ_ONCE(p->se.sched_throttled) & ~PIC= K_IGNORE); +} + +static inline int __kcs_ignore_entity(struct sched_entity *se) +{ + return READ_ONCE(se->sched_throttled) & PICK_IGNORE; +} + +static inline int task_picked_on_throttled(struct task_struct *p) +{ + return READ_ONCE(p->se.sched_throttled) & PICK_ON_THROTTLED; +} + +static inline int ignore_task_kcs_stats(struct task_struct *p) +{ + return __kcs_ignore_entity(&p->se); } =20 /* @@ -6893,6 +6935,10 @@ static __always_inline void avg_kcs_vruntime_add(str= uct cfs_rq *cfs_rq, struct s unsigned long weight; s64 key; =20 + /* See avg_kcs_vruntime_sub() */ + if (__kcs_ignore_entity(se)) + return; + if (!se_in_kernel(se)) return; =20 @@ -6912,6 +6958,15 @@ static __always_inline void avg_kcs_vruntime_sub(str= uct cfs_rq *cfs_rq, struct s unsigned long weight; s64 key; =20 + /* + * A remote running task in being enqueued for a restore operation. + * Since it has an unstable "kernel_cs_count" being in a running state, + * do not account its stats yet. A set_next_task() -> schedule() will + * follow on the CPU to adjust it. + */ + if (__kcs_ignore_entity(se)) + return; + if (!se_in_kernel(se)) return; =20 @@ -7238,6 +7293,20 @@ static __always_inline int se_in_kernel(struct sched= _entity *se) =20 static __always_inline void task_mark_throttled(struct task_struct *p) {} static __always_inline void task_clear_throttled(struct task_struct *p) {} +static __always_inline void task_mark_ignore(struct task_struct *p) {} +static __always_inline void task_clear_ignore(struct task_struct *p) {} + +static inline int task_picked_on_throttled(struct task_struct *p) +{ + return 0; +} + +static inline int ignore_task_kcs_stats(struct task_struct *p) +{ + return 0; +} + + static __always_inline void avg_kcs_vruntime_add(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) {} static __always_inline void avg_kcs_vruntime_sub(struct cfs_rq *cfs_rq, st= ruct sched_entity *se) {} static __always_inline void avg_kcs_vruntime_update(struct cfs_rq *cfs_rq,= s64 delta) {} @@ -7437,6 +7506,13 @@ enqueue_task_fair(struct rq *rq, struct task_struct = *p, int flags) if (!(p->se.sched_delayed && (task_on_rq_migrating(p) || (flags & ENQUEUE= _RESTORE)))) util_est_enqueue(&rq->cfs, p); =20 + /* + * Running task has just moved to the fair class. + * Ignore "kernel_cs_count" until set_next_task_fair() + */ + if (task_on_cpu(rq, p) && (flags & ENQUEUE_MOVE)) + task_mark_ignore(p); + if (flags & ENQUEUE_DELAYED) { requeue_delayed_entity(se); return; @@ -9582,9 +9658,23 @@ static void put_prev_task_fair(struct rq *rq, struct= task_struct *prev, struct t * Clear the pick on throttled indicator only if * another task was picked and not for a save / * restore operation for the task. + * + * For a save / restore operation of a running task, + * mark the task to be ignored for "kernel_cs_stats" + * adjustment. Either set_next_task_fair() or + * switched_from_fair() will clear these idicators. */ if (next) task_clear_throttled(prev); + else { + /* + * put_prev_task_fair() is only called with next as NULL + * during save / restore operations. Since idle thread + * is always runnable, all other cases will have a valid + * prev task set. + */ + task_mark_ignore(prev); + } =20 for_each_sched_entity(se) { cfs_rq =3D cfs_rq_of(se); @@ -13897,6 +13987,7 @@ static void set_next_task_fair(struct rq *rq, struc= t task_struct *p, bool first) { struct sched_entity *se =3D &p->se; bool task_in_kernel =3D !task_on_cpu(rq, p) && se_in_kernel(se); + bool h_throttled =3D false; =20 for_each_sched_entity(se) { struct cfs_rq *cfs_rq =3D cfs_rq_of(se); @@ -13905,6 +13996,22 @@ static void set_next_task_fair(struct rq *rq, stru= ct task_struct *p, bool first) account_kcs_dequeue(cfs_rq, task_in_kernel); /* ensure bandwidth has been allocated on our new cfs_rq */ account_cfs_rq_runtime(cfs_rq, 0); + /* Check if task is on a partially throttled hierarchy */ + h_throttled =3D h_throttled || cfs_rq_throttled(cfs_rq); + } + + /* Mark the end of save / restore operation. */ + if (ignore_task_kcs_stats(p)) { + task_clear_ignore(p); + + /* + * If the hierarchy is throttled but the task was not picked on + * a throttled hierarchy, the hierarchy was throttled during the + * course of a save / restore operation. Request a resched for + * pick_next_task_fair() to reevaluate the throttle status. + */ + if (h_throttled && !task_picked_on_throttled(p)) + resched_curr(rq); } =20 __set_next_task_fair(rq, p, first); --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2083.outbound.protection.outlook.com [40.107.243.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C65D21E9B19 for ; Thu, 20 Feb 2025 09:40:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.83 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044431; cv=fail; b=pHxv5jh13Hy3pzzk1VLPdz7xV54bc8JG4CNhOTBq7gErdNJN6ndm0FzWmFc2QUrPT4gtnwlZExUxdTPg3z38HsGT2fmdIehFq1SYNQySkXC+WnZpA0qDgcxhpy+RvH3xdWV4/d63/MoaRNatbcatHwOtPS6S4U9cBd7i4o6uiQA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044431; c=relaxed/simple; bh=Kvlp1mWOGLv3Yn/66hKbibE6vRHgTRbx3xmsKxkIZiA=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=XwSpfNt1Ya1GhzHqyTOpPO+mSoeIffUwXLhN+RsB+mjfBfC+9OvaaSYWzLaP7Z6EDxbEumtqXymkbD9bEKFbN6PVM/tjwQO5AeC9vA/2+qpvlqs67x83EIm5d7tmhqKCcb+M5LkXSPreBCKBlIQ+FXI9Zz9zGdbuJCoYaFsa++U= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=RnNNGvTA; arc=fail smtp.client-ip=40.107.243.83 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="RnNNGvTA" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=bROhVNr9G34NzpDMym9t7q3F3yQaptCvsSKlE2SjJDB1laco/rQz45fdUqa5Retqa3Z28lfcjeGF6DjcRGNVlitoloztXp5AOBx/cPHMJhWt84Vd49dJjNca1xyjSB3JDjP06fvDrxG/1MEXBmYlN+YBGB4VuqmQrctoqXGfuBKDzf0KZ21Iaw73gUZlzR2cMc1K9/ya5jpYVzAp+WOsopbOsjaJCHP7zUrJMXRqHJeBqqRjNW3lEauKjqrR6pso5YRALK+rUMljbIlKx5GQ97lELH8eYwH34co9jOZm1dDemzwEcbEBZCzC7RTwpeGJn3jw5270z2Olms8kFNVt2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=y3jrsS4C5/vYW+2SYje4Nn0Qo8aHbolgkpREtLkFtJU=; b=rVYkMd25476Har9nQqZfCH9+7wF/zPr7tO8qFghrB0PYPJePUgEh+bSgns1kgq3ikcxa7AQ0CtNA2ESwHyaro/fKqd27s5myU07GhX11r7ZqDSlMjB+7a3s5iMbhXbu8P9/a+Bu1GmhQBBrI5GoihO1LExEEJL3MzC2Ea+IFcsgy0QJahz7z1vhxmygKUpb0tr0zrLvN0Ot/7zEmNy1+Aj+S2x7ymO2HjJYMWUkpCz4S9mx6jMYAMm9zdhG6ID5MWvTTgf9ndjHE/4Y8TvntaqM0I5uGPr7eITNn9LAIGlxzkekLsnG4YJg3KY6TWUEh9UePuBsokm0uYU3jhsEPXQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y3jrsS4C5/vYW+2SYje4Nn0Qo8aHbolgkpREtLkFtJU=; b=RnNNGvTAw39xFBVCFhKOgzX5bSTNUJdkJX4YJpT2V8bkrpN8aas9bGLiaMMPkoRaXe2KuL7wFJlb+p4CH+DlkJ+DtR40oYLkbSGQQtNtnhCHQRc45ICYr45xcVGw4KUBfUHfytEMsCVl4hGcs9FJQfkPyH/2nSOgx5Wz1Ne/0lk= Received: from BL1PR13CA0409.namprd13.prod.outlook.com (2603:10b6:208:2c2::24) by PH0PR12MB5629.namprd12.prod.outlook.com (2603:10b6:510:141::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.23; Thu, 20 Feb 2025 09:40:24 +0000 Received: from BN1PEPF00005FFF.namprd05.prod.outlook.com (2603:10b6:208:2c2:cafe::92) by BL1PR13CA0409.outlook.office365.com (2603:10b6:208:2c2::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8466.15 via Frontend Transport; Thu, 20 Feb 2025 09:40:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00005FFF.mail.protection.outlook.com (10.167.243.231) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:40:23 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:40:15 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 20/22] sched/fair: Implement determine_throttle_state() for partial throttle Date: Thu, 20 Feb 2025 09:32:55 +0000 Message-ID: <20250220093257.9380-21-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00005FFF:EE_|PH0PR12MB5629:EE_ X-MS-Office365-Filtering-Correlation-Id: 2cc24148-58bf-46ae-1f23-08dd51929947 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|36860700013|82310400026|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?VS4in7vJe8AoRIupVCV8AxHaoJJYh+1yOigCzrRlsFANtep5qGGBbsD7z3DM?= =?us-ascii?Q?E7c04ls7MKlKivdNxWM0AybNp74v5BDpd7B3J7gyNlN4el2lZPecS3eHM09t?= =?us-ascii?Q?a5OI1k9fYjzoMNqKFhrJrL3iE12SyjIbH6MwLERNfqSKODEbdM6BumkWV03y?= =?us-ascii?Q?anCPBthDQPR16RW6+5Qu0RZArA/0lSpv6EQJBQSHQq79Ss/RGF9G18/1G9lG?= =?us-ascii?Q?XiVVV46TsUo6tKEpdRtcV6iff5nEajFkynZ5MUi8DFj/dGMhDJwN88+84n1m?= =?us-ascii?Q?YsNEjEuw4NsodXWkAphOUhgvcoUYYqCUtR4V3eJkWWuEkSUJ1S360y3z7HVM?= =?us-ascii?Q?6NzzhesnYj02RIBAAA/8locmXetIz8vFmdy9e4dz9rc8v27lEDkdCzgidtLq?= =?us-ascii?Q?Q8dG/7OaFYTB0LnTvugpKR70Z5wCVL32Y8NZSUC5bbv8epjBA2ofEkQJx0IL?= =?us-ascii?Q?hdkx34Yczaj/GOz42P5HpyINrQTPvaZc7YJD9i2NiKiyTOsryJbi9BUy+RE9?= =?us-ascii?Q?2Kiuz7z3ZSQUTjPiCIdNpyu9OUT0tttsbqPFvtzDOR2BJnlSBBxglP5qRbNx?= =?us-ascii?Q?ipG41UMrYMBk4Y5A/8x9FKOw4XxfYsi9lA3yu144yIObIqWpa1PxNywRtqjJ?= =?us-ascii?Q?b385V+Mevh2Fxv20lsSav9hlSqJU73r7XHi7196diDg2jAB00bz48opJ5eDw?= =?us-ascii?Q?XMpkgPUmCO51RyM5eveg4p0ZUAdY2IUsuE3r0GZM0dfX7vxwAuk5YAigTzAf?= =?us-ascii?Q?sSxW55/wx7HScojtiR+3YavnDGiRzFadZmld+w8yvE+tBxaV6DIvHRfNZBeV?= =?us-ascii?Q?Kt7ml5kl5hylQIdZqdFuww9CPR1QiAx5KDIq4fYlknpkZH3xxp/x2Tjl276g?= =?us-ascii?Q?gcsQ4QI7XLfIXviv2YQFNHA61rRIStGtTDz/qQZuqXQgFncZ4MpSIU3DxYjz?= =?us-ascii?Q?09OQUo8Xe4Z/TFjVtlV1nFbEhUjGKmOo1W5z3aCbJmjo1TFKGEAwWa+3adge?= =?us-ascii?Q?aZQ/oVSgca/uWCoLLwt7jR22WkF+Uke0y2V8U7NS8W5UgS/ds75T2oHwFKfF?= =?us-ascii?Q?jbVitPSUNFRTjGXqVStKg1eXdmyeOmMfXIJlR3X3xr+Fwdo/18/0RFZYdaf0?= =?us-ascii?Q?DdzKQWwq6JupdU6kR6+3Qfk4XJKNYoWtFC+YbZsGPIjX54fmIQIoFx5QStfd?= =?us-ascii?Q?7CAUWuyz37W+65EjcPS9qf607djOzIVERVVUyatIch8HcGGvzVPCnEq6F95k?= =?us-ascii?Q?G25KU9IQzcMgCneXRh9oLfp0X1vtuc9GRHPFddrIAlYDb1jT3hg5ZThP1eDx?= =?us-ascii?Q?vm83HIN4Wt224xkKTzUgR7PZ0yP+W+Ir0YTjdxGMfMqhD87Grr+JX94lEZNY?= =?us-ascii?Q?ET9s1ph0+K8Ir26aUv8A6sbIOL3iBYucP/C4ToMtAmG7+dQEJFaFVfLp9L8P?= =?us-ascii?Q?IMXuowVGuUGmdm1HOB4y17Ipeb5aSac/os9pvbQH+jJDKoObMCbFzXslqLgG?= =?us-ascii?Q?pB63wC1OrUVHvGA=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(36860700013)(82310400026)(1800799024)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:40:23.4503 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2cc24148-58bf-46ae-1f23-08dd51929947 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00005FFF.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB5629 Content-Type: text/plain; charset="utf-8" With the plumbing for partial throttle in place, implement determine_throttle_state() for partial throttle when it finds cfs_rq with kernel mode preempted entities on it. Also remove the early return in unthrottle_throttled() "Let it rip" Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 55e53db8da45..39c7e8f548ca 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5894,13 +5894,37 @@ static inline int throttled_lb_pair(struct task_gro= up *tg, throttled_hierarchy(dest_cfs_rq); } =20 +static __always_inline int se_in_kernel(struct sched_entity *se); +static inline int ignore_task_kcs_stats(struct task_struct *p); + static enum throttle_state determine_throttle_state(struct cfs_rq *gcfs_rq, struct sched_entity *se) { + struct sched_entity *curr =3D gcfs_rq->curr; + + if (se_in_kernel(se)) + return CFS_THROTTLED_PARTIAL; + /* - * TODO: Implement rest once plumbing for - * CFS_THROTTLED_PARTIAL is done. + * Check if current task's hierarchy needs throttle deferral. + * For save / restore operations, cfs_rq->curr could still be + * set but the task has already been dequeued by the time + * put_prev_task() is called. Only check if gcfs_rq->curr is + * set to check the current task's indicator. If the hierarchy + * leads to a queued task executing in kernel or is having its + * stats ignored, request a partial throttle. + * + * set_nex_task_fair() will request resched if throttle status + * changes once stats are reconsidred. */ + if (curr) { + struct task_struct *p =3D rq_of(gcfs_rq)->curr; + + if (task_on_rq_queued(p) && + (ignore_task_kcs_stats(p) || se_in_kernel(&p->se))) + return CFS_THROTTLED_PARTIAL; + } + return CFS_THROTTLED; } =20 @@ -7181,9 +7205,6 @@ static void unthrottle_throttled(struct cfs_rq *gcfs_= rq, bool in_kernel) struct rq *rq =3D rq_of(gcfs_rq); struct sched_entity *se =3D gcfs_rq->tg->se[cpu_of(rq)]; =20 - /* TODO: Remove this early return once plumbing is done */ - return; - /* * Demoting a cfs_rq to partial throttle will trigger a * rq_clock update. Skip all the updates and use the --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2060.outbound.protection.outlook.com [40.107.92.60]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A05B11E9B36 for ; Thu, 20 Feb 2025 09:40:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.60 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044450; cv=fail; b=hC2MMBLT5rUKm3CtFPc55eC+FbS2AO6HyJ3C8XqwxdRrCRQwvNHNubNOKgiq6FYjv23LBLmUHHUbqMCq1Te4Now0fYryu3MJRyonRQrKr8kko5k6WNGhE0B76MODHCBTR0sDVzvK/bSODe5JLjM+R3V/auctD/UyUwPkZQ+1MoM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044450; c=relaxed/simple; bh=+mXG5MujdqBeiUXgOLCAQbqFT+uMKDK4kY1LFAgTEug=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=UPw/mtOJ7V4KzYJIU/H07terqOqVKuUEGuKOVUo2aSMaLIhNqg/InKuaf+xh7CviNtlf0h0CkcVEdq6vUQW5sC5sDU3u/BnxPGt8ZCAqTZ3beDHHNPYNdB55QI11NXB+pGwZ8Uc9/nFlFBJ0pSKy/G35lA+5wBiejKm6XC8bLFQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Qh2AcYXJ; arc=fail smtp.client-ip=40.107.92.60 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Qh2AcYXJ" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=caK0HXsaaQX2ah9VL6EQeyfeUPv+o18GiyLDgaE15XivP/eoHG2pwv7+4T7CyWRjdQBCiFY8z/gKvbKDhjRF9OaFKgJYZjf5H1zSfiG8eeYd8p1yNnjOfThzWF8tB+gkUjZQ/YPhRlmi7jJZauewASsGWCpjujaWahTVb8ZqIBHATKkZ5OOxjluy/YIPkPWhmjAEqTZipqs8q0CJ+D3MDZzH0pEYQC3Aofh+5pXlzgu1aTOmjbk7Q9MSCDbcmgrk2kChv+YnPp8yuRuNf+MynxsfIwYSOyf1NbD6758LYM2fk2jdKQPotKzmlwefUWlryS9XSML9vU0rntVGeYbnWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Uj5KCx9jgyfW3Aj/ZNL+dU8cihzFhF3M/F05npibLf8=; b=OGHaLb40EAwVzVfe6UC3dJMpj7fvjdNhZtR3cczDdQls7tLpQvfamGNEddvCbrXbT2yyYEYxFmi15fndy3KVkuFWRPlePZTNbB3mkjSH+FDFL9G1dxF+a737XoNzSyrWJTknGvbR7hU4ndHSL7nsTyxgTznn/0awxIC/q8O9tk/vANS+FM1ITWW2tDZTsWHilwUa/+1P4VhsQpNkOxnYDaz6ZQG8TB233H8YhHVlJLp3IesVCb+q5lupgr4AYWN2K5GBz9W+Og+AMMAO7W/EcFJ4HjAOtiwIYWHt24M6QLEnVjvLcezqkS/mTUZxiUO9cdCQfFhTeMPPj/bSOP97jA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Uj5KCx9jgyfW3Aj/ZNL+dU8cihzFhF3M/F05npibLf8=; b=Qh2AcYXJaqLwYfpY61ymx+Efv1KqhK4cUcEsIzKbA5JCcmVAb08cL68tFWzVkLq1cqophLNFRQSWNFJc7cv/rGkMbudAmbOZ2kAEHmQRGcclAQ3rqU0b9S1EwvVQm1YZ0VQlMkKkoFd35/LVPzIiBpQE8+p4Lz2AwOGlqoJ73Nk= Received: from MN2PR08CA0024.namprd08.prod.outlook.com (2603:10b6:208:239::29) by DS0PR12MB7702.namprd12.prod.outlook.com (2603:10b6:8:130::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.24; Thu, 20 Feb 2025 09:40:44 +0000 Received: from BN1PEPF00005FFE.namprd05.prod.outlook.com (2603:10b6:208:239:cafe::fe) by MN2PR08CA0024.outlook.office365.com (2603:10b6:208:239::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8445.20 via Frontend Transport; Thu, 20 Feb 2025 09:40:44 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00005FFE.mail.protection.outlook.com (10.167.243.230) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:40:44 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:40:36 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 21/22] [MAYBE BUGFIX] sched/fair: Group all the se->min_* members together for propagation Date: Thu, 20 Feb 2025 09:32:56 +0000 Message-ID: <20250220093257.9380-22-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00005FFE:EE_|DS0PR12MB7702:EE_ X-MS-Office365-Filtering-Correlation-Id: 3555af5c-c788-42cc-e00a-08dd5192a5a7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|82310400026|1800799024|376014|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?tMS7d9sFJyJ0IUxPJA3iiYOFunEqAiZmEzog742MCb4hwEcQXOM68iOPmg9W?= =?us-ascii?Q?Iw/I/BZ0ljujcEtYTmx3KxsvO/2e42/N8+vas6U5AMSDUtRGMtNGNdSDUiDq?= =?us-ascii?Q?bdgyR5aGVYTalaMFtHN5KfzqIu3fqYGn+ZjT/4trTBGWrwuhoylvpxda+RcZ?= =?us-ascii?Q?IlOZgXIvAFduqo6/6bjcnP/U+FSxT0Tou0yhELs/gL2nYIHqifcaGbdxYzo7?= =?us-ascii?Q?VhLwf5P9XCs5V/8DZ65lRPFeX0HOAv54m4RpY+YSt9UlfV3OK+9h/2lusfDw?= =?us-ascii?Q?20WrFoDBEZykJUpOmo1BOfn8iV1FUVFJdgVa2TokY0y/A0IRJR8m/3fu66Vc?= =?us-ascii?Q?PvnyGOY95VGIT37JMGi4BG0UemaotoVqrP8XwLFuwFb1GZ++wVz3n7Od3I7G?= =?us-ascii?Q?1eqcsNz6VLZh/oo/ZB4IOhvbbnmMOGP6oItHE9HSXGC7EMJrIP8zR/IhHYkR?= =?us-ascii?Q?hphsyH/0nKY440W5bhmhR1PT673vWca76P9ySC0/DEQAEhF2+NH6Ba3gSuhP?= =?us-ascii?Q?engjWpQX46XdE1x6NOmVhWTjY+ayfJ4NzoaT2ON7HiqxCQotcIrh8Bxj1n3+?= =?us-ascii?Q?l1Kmjrfpv6nV2IXX3qW0zLRjbYTYRHRisPMvNRdjaKwV2TDatWOL1GptSkOP?= =?us-ascii?Q?ILA4q8cc1YL25Qgd3OWCQ0LbDXtpegEa/bHsjn6dJoVwJJg7QILOmOeO6ZgC?= =?us-ascii?Q?y8L9OPnD4UYOBIemjttqkc6ww6wjp5hxzUh4DVev1zo6KExYIkIFgqfcJoX/?= =?us-ascii?Q?bckAFCmzqkq/vXib8Bwklko7OqO3O3xtaJUkW0fYCayr6db2zOLUecGM6Gpp?= =?us-ascii?Q?AK10rhY/940+2h3QFJpRUJeoC+RiObSbaxHnhObLoQC7lEPKymfxFpyt6Umz?= =?us-ascii?Q?cmcRsjKbvIUJTP5tvUoZ0ZdbIPKIVsX6qYxnUlCNKB4jPCnfR20eY86/7CLn?= =?us-ascii?Q?c2BoSqsm0TUqKPKIn3oBysSNV/DO6oVirhG7EpUpLS3atpzILxW9iIB+Kfxn?= =?us-ascii?Q?iZyhVyzveNJWFCo+KLZkb/g4x8uDaBj8X1somvdf40ovfMKr2i3KceGtVnVf?= =?us-ascii?Q?xy21N/e5ifh0dWPmWs00fDDPhuVLPKYnxdJXeDNdxMAYZcEvhMnR9q04z5mz?= =?us-ascii?Q?iG+lFMuIGqsMFAYEff0IjWGsM2N4UTPA7Xpk6MdMfliK4TKkQCtAWdyHiQnv?= =?us-ascii?Q?aJOa25YwvnfhZ8tZWXNeLiOr94PwE/6+tlRDZGyF3jyVhpRow6J+QpjRYbA7?= =?us-ascii?Q?xr0V7ZdMU3D9bF3NypALs/OZydYdy7R6q+yJU1Fc274kSptWtE6rWfQxHy/4?= =?us-ascii?Q?zlxnqJPTyl+MK/rIXkqgP4eL0hsI8w+T0hPxvSKjOayfWN51aAH1vNvbPZN+?= =?us-ascii?Q?Qnm4j94t7kT2If8+eHS7+qZWZgUSu0+OSWPi+8mKB7gKDI8uglZVoIaWz4p9?= =?us-ascii?Q?z7VFPGfhZplEaE3hFt6/S56KtHs6RA6MfRkjL3kLV3F0qGfBJ+8iD/Qmz9N/?= =?us-ascii?Q?04WD2hGNI9XvNcM=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(82310400026)(1800799024)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:40:44.2327 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3555af5c-c788-42cc-e00a-08dd5192a5a7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00005FFE.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7702 Content-Type: text/plain; charset="utf-8" When working on the prototype, at some point, min_kcs_vruntime heap integrity issues were encountered. Since then many bugs were ironed out but when auditing rb_{add,erase}_augmented_cached(), it was noticed that only "min_vruntime" was considered as RBAUGMENTED for RB_DECLARE_CALLBACKS(). The augment->copy() function generated only copies RBAUGMENTED member. It is very likely that augment->propagate() is always called after augment->copy() similar to the convention in augment->rotate() which reconstructs all of min_{vruntime,slice,kcs_vruntime} before comparing if the values have changed however this author was not sure if augment->propagate() indeed reaches until the copied node for Case 2. in __rb_erase_augmented(). Group all the min_* members together that need propagation to rb_root and pass the new "min" struct as RBAUGMENTED to ease the author's paranoia. XXX: Reverting this bit reintroduces the heap integrity issue. It does not happen immediately into the run and the amount of traces are overwhelming to analyze which series of rbtree operations lead to this but with this patch I haven't run into the heap integrity issue so far. The cover letter should have more details. Signed-off-by: K Prateek Nayak --- include/linux/sched.h | 20 +++++++++------- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 54 +++++++++++++++++++++---------------------- 3 files changed, 40 insertions(+), 36 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 200cc086e121..6fd5cee3a0f5 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -549,8 +549,18 @@ struct sched_entity { struct load_weight load; struct rb_node run_node; u64 deadline; - u64 min_vruntime; - u64 min_slice; + struct { + u64 vruntime; + u64 slice; + +#ifdef CONFIG_FAIR_GROUP_SCHED + /* + * min_vruntime of the kernel mode preempted + * entities in the subtree of this sched entity. + */ + s64 kcs_vruntime; +#endif + } min; =20 struct list_head group_node; unsigned char on_rq; @@ -601,12 +611,6 @@ struct sched_entity { /* Entity picked on a throttled hierarchy */ unsigned char sched_throttled; /* hole */ - - /* - * min_vruntime of the kernel mode preempted entities - * in the subtree of this sched entity. - */ - s64 min_kcs_vruntime; #endif /* CONFIG_CFS_BANDWIDTH */ #endif /* CONFIG_FAIR_GROUP_SCHED */ =20 diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index ef047add7f9e..37bca53f109f 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -821,7 +821,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct c= fs_rq *cfs_rq) raw_spin_rq_lock_irqsave(rq, flags); root =3D __pick_root_entity(cfs_rq); if (root) - left_vruntime =3D root->min_vruntime; + left_vruntime =3D root->min.vruntime; first =3D __pick_first_entity(cfs_rq); if (first) left_deadline =3D first->deadline; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 39c7e8f548ca..97566a043398 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -768,7 +768,7 @@ int pick_subtree(struct cfs_rq *cfs_rq, struct sched_en= tity *se, bool h_throttle if (unlikely(h_throttled)) return pick_subtree_on_throttled(cfs_rq, se); =20 - return vruntime_eligible(cfs_rq, se->min_vruntime); + return vruntime_eligible(cfs_rq, se->min.vruntime); } =20 static u64 __update_min_vruntime(struct cfs_rq *cfs_rq, u64 vruntime) @@ -800,9 +800,9 @@ static void update_min_vruntime(struct cfs_rq *cfs_rq) =20 if (se) { if (!curr) - vruntime =3D se->min_vruntime; + vruntime =3D se->min.vruntime; else - vruntime =3D min_vruntime(vruntime, se->min_vruntime); + vruntime =3D min_vruntime(vruntime, se->min.vruntime); } =20 /* ensure we never gain time by being placed backwards. */ @@ -819,7 +819,7 @@ static inline u64 cfs_rq_min_slice(struct cfs_rq *cfs_r= q) min_slice =3D curr->slice; =20 if (root) - min_slice =3D min(min_slice, root->min_slice); + min_slice =3D min(min_slice, root->min.slice); =20 return min_slice; } @@ -835,8 +835,8 @@ static inline void __min_vruntime_update(struct sched_e= ntity *se, struct rb_node { if (node) { struct sched_entity *rse =3D __node_2_se(node); - if (vruntime_gt(min_vruntime, se, rse)) - se->min_vruntime =3D rse->min_vruntime; + if (vruntime_gt(min.vruntime, se, rse)) + se->min.vruntime =3D rse->min.vruntime; } } =20 @@ -844,8 +844,8 @@ static inline void __min_slice_update(struct sched_enti= ty *se, struct rb_node *n { if (node) { struct sched_entity *rse =3D __node_2_se(node); - if (rse->min_slice < se->min_slice) - se->min_slice =3D rse->min_slice; + if (rse->min.slice < se->min.slice) + se->min.slice =3D rse->min.slice; } } =20 @@ -853,30 +853,30 @@ static __always_inline void init_se_kcs_stats(struct = sched_entity *se); static inline bool min_kcs_vruntime_update(struct sched_entity *se); =20 /* - * se->min_vruntime =3D min(se->vruntime, {left,right}->min_vruntime) + * se->min.vruntime =3D min(se->vruntime, {left,right}->min_vruntime) */ static inline bool min_vruntime_update(struct sched_entity *se, bool exit) { - u64 old_min_vruntime =3D se->min_vruntime; - u64 old_min_slice =3D se->min_slice; + u64 old_min_vruntime =3D se->min.vruntime; + u64 old_min_slice =3D se->min.slice; struct rb_node *node =3D &se->run_node; bool kcs_stats_unchanged =3D min_kcs_vruntime_update(se); =20 - se->min_vruntime =3D se->vruntime; + se->min.vruntime =3D se->vruntime; __min_vruntime_update(se, node->rb_right); __min_vruntime_update(se, node->rb_left); =20 - se->min_slice =3D se->slice; + se->min.slice =3D se->slice; __min_slice_update(se, node->rb_right); __min_slice_update(se, node->rb_left); =20 - return se->min_vruntime =3D=3D old_min_vruntime && - se->min_slice =3D=3D old_min_slice && + return se->min.vruntime =3D=3D old_min_vruntime && + se->min.slice =3D=3D old_min_slice && kcs_stats_unchanged; } =20 RB_DECLARE_CALLBACKS(static, min_vruntime_cb, struct sched_entity, - run_node, min_vruntime, min_vruntime_update); + run_node, min, min_vruntime_update); =20 /* * Enqueue an entity into the rb-tree: @@ -885,8 +885,8 @@ static void __enqueue_entity(struct cfs_rq *cfs_rq, str= uct sched_entity *se) { avg_vruntime_add(cfs_rq, se); init_se_kcs_stats(se); - se->min_vruntime =3D se->vruntime; - se->min_slice =3D se->slice; + se->min.vruntime =3D se->vruntime; + se->min.slice =3D se->slice; rb_add_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline, __entity_less, &min_vruntime_cb); } @@ -953,7 +953,7 @@ static inline void cancel_protect_slice(struct sched_en= tity *se) * tree keeps the entries sorted on deadline, but also functions as a * heap based on the vruntime by keeping: * - * se->min_vruntime =3D min(se->vruntime, se->{left,right}->min_vruntime) + * se->min.vruntime =3D min(se->vruntime, se->{left,right}->min_vruntime) * * Which allows tree pruning through eligibility. */ @@ -7018,7 +7018,7 @@ static __always_inline void init_se_kcs_stats(struct = sched_entity *se) * the upper bound to differentiate the case where no kernel mode preempt= ed * entities are queued on the subtree. */ - se->min_kcs_vruntime =3D (se_in_kernel(se)) ? se->vruntime : LLONG_MAX; + se->min.kcs_vruntime =3D (se_in_kernel(se)) ? se->vruntime : LLONG_MAX; } =20 /* @@ -7096,11 +7096,11 @@ static __always_inline int pick_subtree_on_throttled(struct cfs_rq *cfs_rq, struct sched_entity *= se) { /* There are no kernel mode preempted entities in the subtree. */ - if (se->min_kcs_vruntime =3D=3D LLONG_MAX) + if (se->min.kcs_vruntime =3D=3D LLONG_MAX) return false; =20 return throttled_vruntime_eligible(cfs_rq, - se->min_kcs_vruntime, + se->min.kcs_vruntime, curr_h_is_throttled(cfs_rq)); } =20 @@ -7109,21 +7109,21 @@ static inline void __min_kcs_vruntime_update(struct= sched_entity *se, struct rb_ if (node) { struct sched_entity *rse =3D __node_2_se(node); =20 - if (rse->min_kcs_vruntime < se->min_kcs_vruntime) - se->min_kcs_vruntime =3D rse->min_kcs_vruntime; + if (rse->min.kcs_vruntime < se->min.kcs_vruntime) + se->min.kcs_vruntime =3D rse->min.kcs_vruntime; } } =20 static inline bool min_kcs_vruntime_update(struct sched_entity *se) { - u64 old_min_kcs_vruntime =3D se->min_kcs_vruntime; + u64 old_min_kcs_vruntime =3D se->min.kcs_vruntime; struct rb_node *node =3D &se->run_node; =20 init_se_kcs_stats(se); __min_kcs_vruntime_update(se, node->rb_right); __min_kcs_vruntime_update(se, node->rb_left); =20 - return se->min_kcs_vruntime =3D=3D old_min_kcs_vruntime; + return se->min.kcs_vruntime =3D=3D old_min_kcs_vruntime; } =20 static inline void account_kcs_enqueue(struct cfs_rq *gcfs_rq, bool in_ker= nel) @@ -7341,7 +7341,7 @@ static __always_inline int pick_se_on_throttled(struc= t cfs_rq *cfs_rq, struct sc static __always_inline int pick_subtree_on_throttled(struct cfs_rq *cfs_rq, struct sched_entity *= se) { - return vruntime_eligible(cfs_rq, se->min_vruntime); + return vruntime_eligible(cfs_rq, se->min.vruntime); } =20 static inline bool min_kcs_vruntime_update(struct sched_entity *se) --=20 2.43.0 From nobody Fri Dec 19 04:18:28 2025 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2076.outbound.protection.outlook.com [40.107.220.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F2501EE7D6 for ; Thu, 20 Feb 2025 09:41:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.220.76 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044472; cv=fail; b=QLwgLYCgR6BCiEKYAKwFZPbbT45agjT70JvEImNw4eFD37FI8Q81IC/ozO+z0ruOuQRNLWmsv4g1ngv9eKOTEMBXY+I6EwWphHTh1yTjvZaYwU/MKbR8bHG9JLdriD/aVOCneNLdyajW1JuLQnGMbXZf712Q/uYgsqyuxL7vkwQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740044472; c=relaxed/simple; bh=f+rub2I1ceHEufMM+Pq3RFHwT8dL31PEHEVJKWpBgO0=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Byr+Z1L4sQPa5+wmfggDFZjWRVZpZ8xar68cVo7WpND9QGRA33Wffb8RgS5VipZzX++R9H/T9bz+ul3rk6zgLcAe1twlLue43nfKmJEbHi944nzcvCf6cQ2gNnKukYosWB4af2VBQh7NbdQISQ2DuMugAwXbTCacERFDSoeQsE8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=pnniTs1K; arc=fail smtp.client-ip=40.107.220.76 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="pnniTs1K" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OXzOwT1+8HIickO102oS+VrJhNUPOtOqCJVdd++/V9djJY1NSqzXtzOZ8asfqZ0kSJ80zJruKhKfZZ6GeKPjagl4ZzqSuGCIiuydM+MZCiD8HYTgz+JG4sCj0TYjSBtEjX4SRswb3PlgC1Gxs0Xcg8Q0FI1Zplgx3wHn/QWS01sGzf5MBshama3CHCPAMuI6RmZQclRdP6lXy/VbjIHjJDWMiD51hgXzfgSYpX4geLRscHwNrmmZlO3DZNFad0ecb219t7YWNtGVwVK9drJvMDqiDne+lCreVuuotWcd6YbOMb9BVcs4C76Xem0HQhON4ZWpkiBhs8b9P8uzdxIC7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XvVuFHTJROWVbhlzzqo+qUY7y97z8qAQmAld3AquM2Y=; b=IAKvJpsFfpQrtZXoWAVukmp83ZxPTQ4h7hkEuJt4+EjQv8OSFVXTkKAIRLNkHB8Y22Z9DUbY26PpBn80hP1rHwhPiLXJGAahTL3JoTiTCxxZx/RVjoSTBMgmFaRFZipCdkPD5As2C6fFQPLCIRNRXqQ8AZ90aCzfMyFBnIjBYVt7ADA49CDPTMSkten9eQ5P8mXwqfiIT65P4OmvKnwnD2hfEy8tA7n3jgYNl/kdrB1CFOisl7+S9MsZMy4QFNqzsPg8KXbwMNz+JqFb8B+q8AaRHMrxtHY5xZOajIdElhHNXmFKoKYx4ab5m3/kkG9vRKBiXfug/tLTWKiF9opc8Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XvVuFHTJROWVbhlzzqo+qUY7y97z8qAQmAld3AquM2Y=; b=pnniTs1KmN6o2fJYG1Kaj3mRaziVrv66OpnyBxp70u2JCy+y0C5vcoPnh6ETm9/MOrih+XjVgqxJLIt0Vx3+oCl/LN1LgEWNX3PKdFsGQPKOGXtx/Aonr4Qut3YzZBqvaWniiT9oQ+jjulHXRnHMwYboneY2oXb7YlifdA/DMJs= Received: from BL6PEPF00013DFD.NAMP222.PROD.OUTLOOK.COM (2603:10b6:22e:400:0:1001:0:1d) by DM6PR12MB4372.namprd12.prod.outlook.com (2603:10b6:5:2af::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Thu, 20 Feb 2025 09:41:05 +0000 Received: from BN1PEPF00006000.namprd05.prod.outlook.com (2a01:111:f403:c803::2) by BL6PEPF00013DFD.outlook.office365.com (2603:1036:903:4::4) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8377.24 via Frontend Transport; Thu, 20 Feb 2025 09:41:05 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN1PEPF00006000.mail.protection.outlook.com (10.167.243.232) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8466.11 via Frontend Transport; Thu, 20 Feb 2025 09:41:05 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Thu, 20 Feb 2025 03:40:57 -0600 From: K Prateek Nayak To: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Valentin Schneider , "Ben Segall" , Thomas Gleixner , "Andy Lutomirski" , CC: Dietmar Eggemann , Steven Rostedt , Mel Gorman , "Sebastian Andrzej Siewior" , Clark Williams , , Tejun Heo , "Frederic Weisbecker" , Barret Rhoden , "Petr Mladek" , Josh Don , Qais Yousef , "Paul E. McKenney" , David Vernet , K Prateek Nayak , "Gautham R. Shenoy" , Swapnil Sapkal Subject: [RFC PATCH 22/22] [DEBUG] sched/fair: Debug pick_eevdf() returning NULL! Date: Thu, 20 Feb 2025 09:32:57 +0000 Message-ID: <20250220093257.9380-23-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250220093257.9380-1-kprateek.nayak@amd.com> References: <20250220093257.9380-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN1PEPF00006000:EE_|DM6PR12MB4372:EE_ X-MS-Office365-Filtering-Correlation-Id: 31e09937-1c8d-4255-8d5d-08dd5192b21c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|36860700013|82310400026; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?zXLIAuh/XE/hNLK4veFHF2IiKQG/5Bxy1Ef30e2s5rOJqmmCXQXu2YkbCnzN?= =?us-ascii?Q?GZsuC8I21L3bCc7dIAKdhp8yT7XLhEV3ED/pToqmgY09Nao3oEHbpx8SlXbQ?= =?us-ascii?Q?YJ9AkG45oRjx9BplBQJZfZx3svOkdnny/s4UzUDy0g238umrqmx8f1BPkq3D?= =?us-ascii?Q?Ts3wDm70t0TbeQ4It6thghzckypM8FFImIB8mSLpEuI91juSFNx5kxTNy0Sw?= =?us-ascii?Q?MumuwqtFP9S2Z0wnpVUXoqmWEionGcyW6PHs2U/v/lA6UFCoXHq7+pD0P0Pf?= =?us-ascii?Q?yg/pM6DvjpKWOMc851b0S8xG1dzfNggPd8HkPRJWWH+66E0JsQtwgONo5k/k?= =?us-ascii?Q?HQRzrWkq+TvkO66IstR96Ctk1Zt/xP5/jDLMi4PrNq80csTDLgAAvrCYlJHV?= =?us-ascii?Q?DZvbX/4lGcYNtTJF/mp5oCwZuJv7s4kO4aATk6LFGG96AYHyeieoH3/B5b1Z?= =?us-ascii?Q?OjfaFv0X9SOjMmDHSPzcZ3blcmdgj57TsQVhYoVg1I+3IkeclYKqXngQt7ot?= =?us-ascii?Q?q7YM5umLrHrKpRMd1sQBDeOmQFbquQkjmuea+9+W2Owx0AOnYyqjoEWA9x2j?= =?us-ascii?Q?u5dzPKKRv1dtyvBe+i9R/PqcRId88s97UH3bf8dypzMtG+CdKY/sRUFfwn/2?= =?us-ascii?Q?2+kwP2oPSOq58wsYkj/mtr8gI1fHeF4SOLdqnyT7AfflEdI7h7VffzQmKXVh?= =?us-ascii?Q?qYq25/qb0TWAI5pztiKsGg0J0OGjCw+GIKC0xKkZH0oEUZjE28qJKP4T3CDC?= =?us-ascii?Q?8PAZx7GbI+rQTGeJulSvl6I8v2j05jvyxmxAkP6ThNEMgiGBl30Q8qoNdAql?= =?us-ascii?Q?1dO2Ky4C80x0gkBMqFhaGNQl0X7XvJeYSjzPuoJBGMfqKeLJcVD3iXuFaf40?= =?us-ascii?Q?dQHSkDso5wy7Vl+dwKjwc2ko74wOjVOytLoTTTGgQ2bndmtvG5aRC1FDREv9?= =?us-ascii?Q?jSLvrY86Fr6jw2/aMt2R9fMYwKLGS3Li4WSLxKROqH1Hyuvf3M9UmHVC5+4Q?= =?us-ascii?Q?zzDLlx6Qbe8tJcLeQJoj24m21qPTTzytq9sA8Dn89Jjm6WC5TE1tiTV1FIQF?= =?us-ascii?Q?IosYtKmFOoD3riy/hMYGdx6zxELtqwcXU9CPU7nnb9Ruyar0dBc53vYBULlY?= =?us-ascii?Q?p8PFDpcK7fEziMPkyaukW9WGI2j6D/DlL1q8EYmTse+vmvsOIUal0UXkOxYe?= =?us-ascii?Q?Z6IU1Aq03f7QDwDsMybZJOuDcb+8Ymxf87hSvQ27Qc8FmLIoz5XmWkPoTaxD?= =?us-ascii?Q?bf11kskCpm8SPvaIpFZhfiez70jz6kpZM8/V/CyH1AXMbfMHhfyQlDQGHu4A?= =?us-ascii?Q?6SdNe7qe13JokYbuWZym24FxWyp7JLt/8fFZ4T3QnDDFfb47PbVoJZZ8tKdh?= =?us-ascii?Q?+khRf2jTIfCsa2XvJU+QjhpbtAFei3J+r27kQHr60tp13pzmEZftO1K9nbWd?= =?us-ascii?Q?pByMxf95f4WbhJWBFhRsslsV+5F8cowUQHh5IIDQa+t88hO+WanCFPv4QZuK?= =?us-ascii?Q?1EgClHHdoG0U6DM=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(36860700013)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2025 09:41:05.1300 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 31e09937-1c8d-4255-8d5d-08dd5192b21c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN1PEPF00006000.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4372 Content-Type: text/plain; charset="utf-8" Dump the stats of cfs_rq and the entities queued on it when pick_eevdf() fails to find a runnable entity. Take the panic that follows since this scenario implies breakdown of the scheduling algorithm. XXX: This will only build with CONFIG_CFS_BANDWIDTH enabled. Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 75 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 97566a043398..270e5f4b2741 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5610,6 +5610,78 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_= entity *se) =20 static int dequeue_entities(struct rq *rq, struct sched_entity *se, int fl= ags); =20 +static void debug_print_se(int cpu, struct sched_entity *se, bool h_thrott= led) +{ + struct cfs_rq *cfs_rq =3D cfs_rq_of(se); + + pr_warn("CPU%d: se: load(%lu) vruntime(%lld) entity_key(%lld) deadline(%l= ld) min_vruntime(%lld) on_rq(%d)\n", cpu, scale_load_down(se->load.weight),= se->vruntime, entity_key(cfs_rq, se), se->deadline, se->min.vruntime, se->= on_rq); + pr_warn("CPU%d: se kcs: kernel_cs_count(%d) min_kcs_vruntime(%lld) pick_e= ntity(%d)\n", cpu, se->kernel_cs_count, se->min.kcs_vruntime, pick_entity(c= fs_rq, se, h_throttled)); +} + +static void debug_print_cfs_rq(int cpu, struct cfs_rq *cfs_rq, struct sche= d_entity *se, bool h_throttled) +{ + pr_warn("CPU%d: ----- cfs_rq ----\n", cpu); + pr_warn("CPU%d: cfs_rq: throttled?(%d) cfs_rq->throttled(%d) h_nr_queued(= %d) h_nr_runnable(%d) nr_queued(%d) gse->kernel_cs_count(%d)\n", cpu, h_thr= ottled, cfs_rq->throttled, cfs_rq->h_nr_queued, cfs_rq->h_nr_runnable, cfs_= rq->nr_queued, (se)? se->kernel_cs_count: -1); + pr_warn("CPU%d: cfs_rq EEVDF: avg_vruntime(%lld) avg_load(%lld) avg_kcs_v= runtime(%lld) avg_kcs_load(%lld) \n", cpu, cfs_rq->avg_vruntime, cfs_rq->av= g_load, cfs_rq->avg_kcs_vruntime, cfs_rq->avg_kcs_load); + + if (cfs_rq->curr) { + pr_warn("CPU%d: ----- cfs_rq->curr ----\n", cpu); + debug_print_se(cpu, cfs_rq->curr, h_throttled); + } + pr_warn("CPU%d: ----- cfs_rq done ----\n", cpu); +} + +static void debug_recursive(int cpu, struct rb_node *node, bool h_throttle= d) +{ + debug_print_se(cpu, __node_2_se(node), h_throttled); + + if (node->rb_left) { + pr_warn("CPU%d: ----- Left Subtree ----\n", cpu); + debug_recursive(cpu, node->rb_left, h_throttled); + pr_warn("CPU%d: ----- Left Subtree Done ----\n", cpu); + } + + if (node->rb_right) { + pr_warn("CPU%d: ----- Right Subtree ----\n", cpu); + debug_recursive(cpu, node->rb_right, h_throttled); + pr_warn("CPU%d: ----- Right Subtree Done ----\n", cpu); + } +} + +static void debug_pick_next_entity(struct cfs_rq *cfs_rq, bool h_throttled) +{ + struct sched_entity *se =3D cfs_rq->tg->se[cpu_of(rq_of(cfs_rq))]; + struct rb_node *node =3D cfs_rq->tasks_timeline.rb_root.rb_node; + struct task_struct *p =3D rq_of(cfs_rq)->curr; + int cpu =3D smp_processor_id(); + + if (p) { + pr_warn("CPU%d: ----- current task ----\n", cpu); + pr_warn("CPU%d: pid(%d) comm(%s) task_cpu(%d) task_on_rq_queued(%d) task= _on_rq_migrating(%d) normal_policy(%d) idle_policy(%d)\n", cpu, p->pid, p->= comm, task_cpu(p), task_on_rq_queued(p), task_on_rq_migrating(p), normal_po= licy(p->policy), idle_policy(p->policy)); + pr_warn("CPU%d: ----- current task done ----\n", cpu); + } + + debug_print_cfs_rq(cpu, cfs_rq, se, h_throttled); + + if (node) { + pr_warn("CPU%d: ----- rbtree traversal: root ----\n", cpu); + debug_recursive(cpu, node, h_throttled); + pr_warn("CPU%d: ----- rbtree done ----\n", cpu); + } + + cfs_rq =3D cfs_rq_of(se); + se =3D parent_entity(se); + + for_each_sched_entity(se) { + pr_warn("CPU%d: ----- parent cfs_rq ----\n", cpu); + debug_print_cfs_rq(cpu, cfs_rq, se, h_throttled); + + cfs_rq =3D cfs_rq_of(se); + } + + debug_print_cfs_rq(cpu, cfs_rq, NULL, false); +} + /* * Pick the next process, keeping these things in mind, in this order: * 1) keep things fair between processes/task groups @@ -5633,6 +5705,9 @@ pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq= , bool h_throttled) } =20 se =3D pick_eevdf(cfs_rq, h_throttled); + if (!se) + debug_pick_next_entity(cfs_rq, h_throttled); + if (se->sched_delayed) { dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED); /* --=20 2.43.0