From nobody Sat Jul 25 05:29:19 2026 Received: from BL2PR02CU003.outbound.protection.outlook.com (mail-eastusazon11011063.outbound.protection.outlook.com [52.101.52.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4D1F38AC75 for ; Fri, 17 Jul 2026 11:16:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.52.63 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784287019; cv=fail; b=FDmnzy/UPzwHY4brsvIsDDDEGnBi8m5EI3iH6XfSXIu9v0YpvfpkDmxJ5Q+8m3kz2wiiZzlmNovOeprT9vkfo4sP/0oc6S1yZiU0CWxNvfIQpo2QWFrmBGH+R7Pw2dswAZJtcTW3/DSVYFf6lF1uCpDUd3Go+9kc2gc16oAdkKU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784287019; c=relaxed/simple; bh=DWsM/Vh956eCKK9yVZeJwsMMF/vTb7axoSCyoM9Ot/Y=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=W8IiTK+bq4pNvTWSoiuWprrtFXDpQwDyXV33jTZeuz9Pb8/MSFdJXT+Up6xjf/BYPHmOU3rWOS+YEuJ8ekt45swLsNWMLOgXL27C0Hpv9N69XLY+7+2uLgQpY2duAKpjex0pE6DVpkwfOJ0qg6fvSZlIJzOz5HaK9yh6U8OoVoM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=PwTZDwf6; arc=fail smtp.client-ip=52.101.52.63 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="PwTZDwf6" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KEn3QlXTNt5kwyWjvP25/fPpr8uPeUFoqm7Y04uQ6g1DbOtAkt8hhioDkj5JdJGuVi4lvrirPCV/Z6BoDbNbClJrBUhiayLCBTxLBtcedXVZPVtwkGWy7PDi67zaIkdHlLWa7InZPNWDZ27F6U2TSQ7VxYtdVRCtj31vrPM+FpgSzvn54G5vr1DEq++D/TnWhJhs7+hgY7idHZ8PzqwMHmSYrG5sFeI6WmPHwHvmORnRLa33VtDsVSCqIP/ZYho81KV4K1BvqNBE/ux3MDG5S5QpaI9rorxFcu+/MKWbeYLlwHHJuQ6T8ptg4Xmson14R4K5oqOFAnz4l5iSGjko+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1/4kLRVGGRJ2QGthgr5yrDLkJ+uRNvjRFsFWoBJHU4w=; b=Ev7oEvl5mAhuPhL/1/nf6Tk9SMib7JXgs0OuxDaF0gnbCQ8vs4dv8CQ9hP4sWANiAr9LASm8iq4y6OMFlPMLsHQS4ytBg/zKwIc/tjuV9TM/5kuHU7rvyu++pQLJb0C5shDYHo6esJRTw2Pgr+BuGsiwudtLOALvXX4jALF5TYdW/au6+6xC/nfkq43K2eu/Sn0e6BFKwX9fyq9Wqsn+3n0pJvcE10TYstUup0ie38ET1zcLYfcIBXPUk2n/irZlLsU7di2iS7txDPk9cKpLnlVF/HdhBdg/Y+5UFBSkujQr4UHLq1Xqf0f6AVAHV1oyGKC5YMWMdpqfbgm8Qu5Ltg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1/4kLRVGGRJ2QGthgr5yrDLkJ+uRNvjRFsFWoBJHU4w=; b=PwTZDwf6KVzduKr9f9E9GPDCmmBpgfp2jlbVt9oobhxYbV5PHwd+2E4pSUn1uikVG3f4MpUeq7rzmjvQvcX4KhyXR84JIGMeEUEcKT7qMoizpn2pSLaw8heGQG4W1qO+sVBydeM24r0gQ8IEU005kkfuyImZJTP2Lv0TGs4EGjw= Received: from BN0PR10CA0011.namprd10.prod.outlook.com (2603:10b6:408:143::13) by SA0PR12MB7074.namprd12.prod.outlook.com (2603:10b6:806:2d5::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.10; Fri, 17 Jul 2026 11:16:45 +0000 Received: from BN2PEPF00004FBE.namprd04.prod.outlook.com (2603:10b6:408:143:cafe::1) by BN0PR10CA0011.outlook.office365.com (2603:10b6:408:143::13) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.223.12 via Frontend Transport; Fri, 17 Jul 2026 11:16:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN2PEPF00004FBE.mail.protection.outlook.com (10.167.243.184) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.245.5 via Frontend Transport; Fri, 17 Jul 2026 11:16:45 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Fri, 17 Jul 2026 06:16:37 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , John Stultz , CC: Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Joel Fernandes , Qais Yousef , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , "Daniel Lezcano" , Suleiman Souhlal , kuyo chang , hupu , Vasily Gorbik , Subject: [RFC PATCH 1/3] sched/core: Track the core-wide pick leader Date: Fri, 17 Jul 2026 11:15:59 +0000 Message-ID: <20260717111601.6017-2-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717111601.6017-1-kprateek.nayak@amd.com> References: <20260717111601.6017-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF00004FBE:EE_|SA0PR12MB7074:EE_ X-MS-Office365-Filtering-Correlation-Id: 24b6a22e-fa57-4c17-94fd-08dee3f4e306 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700016|23010399003|82310400026|376014|7416014|22082099003|18002099003|11063799006|4143699003|56012099006|10067099003; X-Microsoft-Antispam-Message-Info: lXPsgS83pL7CvMpUjMp4Ep4HX3fK3iDr/bmLnb9He4mqWhdJeOxAx4Z3Dm0vNJz9L/6x0nwJsbWzfAteLSLP+OJn9nMwD9sxSpJ/4/ZCFowNSAZP1rgZI55N81Y/eRZ7fTUW1IdZPMlUTVLF9+WN95yZ5wF1RoSXt7u3QUKEaHC6YnxYO+Pjr9JwvSewAyaQf+jwy6GL0PRtrFjPhjy/Ek7w+uAZLjSU2bi04zFY4cB1sfb/iBYWiTDGXpr3bJgtRIOn8O3xwYkLuhfDaD9WeEL/d+IvcWOaTNsCbwD/VAJ9VNTmH6Lfl2ZK/mbcBxH5gGc01k/tB+m52jstssEEqetqzX5gghHbEbL6J/oFeZqA46s6QIdgrPacVCoKkRdHt4mwnKn7iKNoAwXvUTpsXycx6qxRR8x+I4nmWRE26zpf7m4mpw7J4B5hJ3MzYdhG/vx2LcmflB9SOtPvqEHWgMQYm4+hH6QxAudsUmp42SlKlwsGNGfVBwjNLkHUqh1FEQN2TcvmVlBWUTKCesAqz7J1RQduxFLUmy90uv4ysE248MdKxWqHH73H43imdf0IlIkpY3YWYAnpQvsUFkfTVjxaH8/7aaEuNjwsW3adL7fVxokwRxjGXNb+9jrR5zVJpBivNgRg1M8dmhOJODHLBQVNOgHik3fLXhqkrAfXGutmnyg3Jb1UhjYsxpmdK/DjkDktYSH90rnW9TiY6ikMAA== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700016)(23010399003)(82310400026)(376014)(7416014)(22082099003)(18002099003)(11063799006)(4143699003)(56012099006)(10067099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: gmveSdmwWYFDyMFWGAW8QJV5H7da0Ws2Vj4RE9h8Ak0yfng0bW8VTp/EWHX/QKzniVbwMwbpVltdr0WQj+8Z9N7p0edN0eRFYHvGG4FQAKAqwJpz3O46zztjMnrVq+HwqXTo1IXmtd2VO8h3nQ6ynjjnkvoK5k1Tj/BdQtgYLdr49VWmzLhuUPjFR2qNXjoAQAtxCRqyymfD8Ds1qmOmP9T7MAdNap92MnqrqpaDPnB+rwN5lJoT8zd7sinRBZQ50TY5km9V4n3eLYbLJHps/o/7W8uYXgnYHwIxEGD7DFgjBKjrVpi/n+dlg67MXiUf4mbfB5a7xiYrdQj6xUWidHIGnX2+Ln4QhFOpq5Z75c4UHNrWu4wcsKeNteFzZSQZ8PrUWmlVyMBaB2sFJlAGOZNaw/BW4NmxCG0TrMpL3csCfNufzmXDSvI9oCIFJ9Wx X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jul 2026 11:16:45.3311 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 24b6a22e-fa57-4c17-94fd-08dee3f4e306 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF00004FBE.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB7074 Content-Type: text/plain; charset="utf-8" Track the rq whose task is dictating the core-wide cookie. If no core-cookie is set (!needs_sync), every rq on the core is considered a pick leader as it can individually try to dictate a core cookie later. This detail will be used in subsequent commits to extend proxy execution to remain compatible with core scheduling. No functional changes intended. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 11 +++++++++-- kernel/sched/sched.h | 3 +++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 25851bb5489d..31fa6bcee616 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6649,9 +6649,9 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) const struct cpumask *smt_mask; bool fi_before =3D false; bool core_clock_updated =3D (rq =3D=3D rq->core); + struct rq *rq_i, *rq_max; unsigned long cookie; int i, cpu, occ =3D 0; - struct rq *rq_i; bool need_sync; =20 if (!sched_core_enabled(rq)) @@ -6668,6 +6668,7 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) */ rq->core_pick =3D NULL; rq->core_dl_server =3D NULL; + rq->core_pick_leader =3D true; return __pick_next_task(rq, rf); } =20 @@ -6737,6 +6738,7 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) if (!next->core_cookie) { rq->core_pick =3D NULL; rq->core_dl_server =3D NULL; + rq->core_pick_leader =3D true; /* * For robustness, update the min_vruntime_fi for * unconstrained picks as well. @@ -6755,6 +6757,7 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) */ restart_multi: max =3D NULL; + rq_max =3D NULL; for_each_cpu_wrap(i, smt_mask, cpu) { rq_i =3D cpu_rq(i); =20 @@ -6772,11 +6775,15 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) =20 rq_i->core_pick =3D p; rq_i->core_dl_server =3D rq_i->dl_server; + rq_i->core_pick_leader =3D false; =20 - if (!max || prio_less(max, p, fi_before)) + if (!max || prio_less(max, p, fi_before)) { max =3D p; + rq_max =3D rq_i; + } } =20 + rq_max->core_pick_leader =3D true; cookie =3D rq->core->core_cookie =3D max->core_cookie; =20 /* diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 9539867e9e0f..b80a5eb384f0 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1350,6 +1350,9 @@ struct rq { unsigned int core_sched_seq; struct rb_root core_tree; =20 + /* private state for sched_proxy_exec() */ + bool core_pick_leader; + /* shared state -- careful with sched_core_cpu_deactivate() */ unsigned int core_task_seq; unsigned int core_pick_seq; --=20 2.43.0 From nobody Sat Jul 25 05:29:19 2026 Received: from BN1PR04CU002.outbound.protection.outlook.com (mail-eastus2azon11010047.outbound.protection.outlook.com [52.101.56.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66DD12EEE93 for ; Fri, 17 Jul 2026 11:17:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.56.47 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784287030; cv=fail; b=Zw9V0j1V/2FEnsjkeJDVJ3be6oE7DUchIvXyvoS8FL+e5P533zZDOXQ3NeZT+TScTcD7bJ/x2A7lWjzHzbU+1QXP1tNgSAiJvEw5usDPHvYwC//g3NXib23XTZNOq586fyWYdnXrwdegUyytzYdSfwBpNAefeLDCUEFgXN6gqDQ= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784287030; c=relaxed/simple; bh=+5KVOf4fR8BdFfrxH4LQ2mWkxcb2dmR8g8EX1yp+/YY=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=e7qiOnIirZskp4B6ZTCwliHVuC8cbKUTEbulKTKGM51cW4rFn7k+E6dAMOVNQT8mkxDyU3r5ed4celucn9xCnTaoDAZ9twTjC9NQ7tx1lz4M59zRoZjk07joobcJ+Rp87x2dDC+9Sskc7RY4Q5qZ6adGRweMKyGyebs8x1QEs08= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=BuJaTs+O; arc=fail smtp.client-ip=52.101.56.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="BuJaTs+O" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=REOWB42TLiBJriAVp6P/MV7SnO51UlkSprg5GM9Ty7fGfa3Xj10PrFfEaT0Xd/jILgFy02NlUXypz4X3t5XRuKPgL9Zj45zLu1bO+C20AjXHqBEDnc9mlaw/qeszPcKq27rC480mvnfei98k+LbGlxs/Iw97/Q6BhHQuAR0huMpjYmIoh7dneg4d2YYaMvNKg1gn8B1fi9himvuEHYDfebjY+1dg+HPG16T9jqyJytiLR9p16S9ns8/qWpEa0Z/7kBRpjGlpsHUhvsz3mFWka8EbbvvN7ljCNfVokQ/raTTFPWc+7aPYIhlwY7lLVyVh1z/9tQ6hSjwgebDOmlLQug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UFjzCKhyrAA+nimRDsZl0rtJQZGbuT3JVI9WQ3UgPhE=; b=IpZG2r+qYibrAtyh0aH2si3LkBpguhGDmvLlBqsYvAO4AFmEkxA+V8qNl7IEu25An41+o/46rsj7gQJxqxfjM+9fgJ8coz1Boj8eC7wk1mq/dpBP3Y45M8/U3RaEjRtW2weSAark89WYKBOW4YuRWsnWqf13Q5IvVz9nAokt48giDLw7wsfiSQ7M1PfxJ6h/OQK2aHlb6/ek0tU85rd8KCUwflnateE/NCDLNwGEeokcC6OBcqq4N+tb4Qp/a3rTmvl4ANpsq0UgPv1uIir1n+47cCWGWiItxO7jJYrHHR0Yw9W9CM/DOpCuoHthO2wfh0R7Sa5V2F6ZtZmhjJ4zyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UFjzCKhyrAA+nimRDsZl0rtJQZGbuT3JVI9WQ3UgPhE=; b=BuJaTs+OYLB1mnXH7IC+XMK1puR7qfOvVic7zmix+hFWeho85JOz6jCmIoJfLTAiLk9uffgMkdJ4kL5yttitWzK6J/gTLimj2DLI+BEp0jiZaBKb4vrc1MhRblpm5bsomexscblysKN1y9s+CK+RKIbZ+lP/U6P5cJ9xk8i1q3I= Received: from BN0PR10CA0012.namprd10.prod.outlook.com (2603:10b6:408:143::10) by DM4PR12MB7576.namprd12.prod.outlook.com (2603:10b6:8:10c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.202.18; Fri, 17 Jul 2026 11:17:04 +0000 Received: from BN2PEPF00004FBE.namprd04.prod.outlook.com (2603:10b6:408:143:cafe::96) by BN0PR10CA0012.outlook.office365.com (2603:10b6:408:143::10) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.223.12 via Frontend Transport; Fri, 17 Jul 2026 11:17:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN2PEPF00004FBE.mail.protection.outlook.com (10.167.243.184) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.245.5 via Frontend Transport; Fri, 17 Jul 2026 11:17:03 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Fri, 17 Jul 2026 06:16:56 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , John Stultz , CC: Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Joel Fernandes , Qais Yousef , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , "Daniel Lezcano" , Suleiman Souhlal , kuyo chang , hupu , Vasily Gorbik , Subject: [RFC PATCH 2/3] sched/core: Fix forceidle when lock owner has mismatchd cookie Date: Fri, 17 Jul 2026 11:16:00 +0000 Message-ID: <20260717111601.6017-3-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717111601.6017-1-kprateek.nayak@amd.com> References: <20260717111601.6017-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF00004FBE:EE_|DM4PR12MB7576:EE_ X-MS-Office365-Filtering-Correlation-Id: 9d906380-3def-4b43-a84c-08dee3f4ee1d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|23010399003|376014|7416014|1800799024|82310400026|56012099006|4143699003|18002099003|22082099003|11063799006|10067099003; X-Microsoft-Antispam-Message-Info: +ZPgU1OQ0rwJrb3hVSfc8waFv92ooDlnVvbqdX1ZcTYKrbp38ilLhYpnfsKDr1zVLs+Co1qSj2S8H1oh+mURtRNQUyOR1zElBeQ9oQXALQMXSd1/+IRdFvnvKUM8fO9YVtTkpj2rUvhSSw1244nPoyP1xZPH3zeTKMTX+xtOWnFcajhllkTml5v+G5S9Y1FUX+dTKBsE8ijkqwrYDPAzMibcv2L+zMxZt1Iu+ZEcixP/m5EKE1p1zQ8hN8/ZswNiNNUEvbRlZCO7MlE8p/TfDF0cMP75hF1m3ZypcyZRSrtbqtf0BrSuPMm0ki4roJg9CvkMteiYmCjs93cTwml4s95Pg82zsB8u1oqjuY1NPyRBQg3rjziTkh0TWvxwnje+EkhSp0pT7GNRguspkHS1/uF2xTeTngHo7g2/2UoxQlQulOYJOwZFr76bmnK4/Ukyontm0PD4DvMJ1AECqjEv6gYPvW3KKkKW0C+tx4McVSXFQLwNZaSqmuu9G0x9z3p4tnfTsgzWzeg1jLl2N9Z/GMbJ7ZEvKSQwr30cp8/4OlLtv5vJc3qhbnR83RG2LwYQeo2GI9g3+TvSjyqHCYZHzWOAa6XwjA4otZvBpF7zQf7lKV1eax8Xr8X5wYm0wQoiinJcViQPLzIUIcyPEdhq3ytkKwBxJyfwE7RkTWlHiKuKPuIT+aEDGpUVJzT5hNp7+WtNaemvH3yPBQpaFXoIfw== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(23010399003)(376014)(7416014)(1800799024)(82310400026)(56012099006)(4143699003)(18002099003)(22082099003)(11063799006)(10067099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: hO2dlruuWrfwVjDfYs/d3GryLl8IyUo7Wn3AEayWJJCZiEIv7l4QwReiJdYoH282R1q6ulLrfJPQk6tOhkI+e5NBiqeealVGKwY3ryFug8GhFEhGzjhEpuON14XxyERAfgRvhLNZg/OhqemPcKq5xJsVGlWfk/m5ePyiqH36EWgd0/kRhW16YcglewJZTeAjRJZwvIYzpBOiCZzuw4xZavTC7bhh0ywfPbbPxQxpssHoMjDEhxQgp94JVbn14BbpCl6ftXBrwQ6VGATep4KtlnyITs/zGlSfFVbyZ0xhmljk3p6YzajAO87VObvdNBCiovtD9wQIb9Z+iimcdeatv/uWCGlLZj8anLmy5Nx7IhHlDBQMRtKRtHoSHLQv9zZDGSdMYL6RiXPDts29Tz7Q3KouSps0JUUxUkav3p234nh6Ly+X+MRipRzS/ZCFsjW6 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jul 2026 11:17:03.9233 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9d906380-3def-4b43-a84c-08dee3f4ee1d X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF00004FBE.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB7576 Content-Type: text/plain; charset="utf-8" Move Vasily's fix from 7cd4088f70e0 ("sched/core: Don't proxy-exec unmatched cookie lock owners") into a separate helper and fix the forceidle stats, balance callback queuing when rq is not the one dictating the core cookie. The helper will be extended in subsequent commits to allow for re-pick with cookie from the lock owner if rq is the pick leader. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 31fa6bcee616..ed11410f1416 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6598,6 +6598,8 @@ __pick_next_task(struct rq *rq, struct rq_flags *rf) BUG(); /* The idle class should always have a runnable task. */ } =20 +static void proxy_deactivate(struct rq *rq, struct task_struct *donor); + #ifdef CONFIG_SCHED_CORE static inline bool is_task_rq_idle(struct task_struct *t) { @@ -7100,6 +7102,23 @@ static inline void sched_core_cpu_dying(unsigned int= cpu) rq->core =3D rq; } =20 +static struct task_struct * +sched_core_swap_pick(struct rq *rq, struct task_struct *next) +{ + if (!rq->core_pick_leader) { + if (!rq->core->core_forceidle_count++) + rq->core->core_forceidle_seq++; + + proxy_resched_idle(rq); + queue_core_balance(rq); + return rq->idle; + } + + clear_task_blocked_on(rq->donor, NULL); + proxy_deactivate(rq, rq->donor); + return RETRY_TASK; +} + #else /* !CONFIG_SCHED_CORE: */ =20 static inline void sched_core_cpu_starting(unsigned int cpu) {} @@ -7113,6 +7132,18 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) return __pick_next_task(rq, rf); } =20 +static struct task_struct * +sched_core_swap_pick(struct rq *rq, struct task_struct *next) +{ + /* + * Caller should always check !sched_cpu_cookie_match(rq, next) + * which is always false for !CONFIG_SCHED_CORE and execution + * should never reach here. + */ + BUG_ON(); + return next; +} + #endif /* !CONFIG_SCHED_CORE */ =20 /* @@ -7512,13 +7543,6 @@ find_proxy_task(struct rq *rq, struct task_struct *d= onor, struct rq_flags *rf) } WARN_ON_ONCE(owner && !owner->on_rq); =20 - if (owner && !sched_cpu_cookie_match(rq, owner)) { - if (curr_in_chain) - return proxy_resched_idle(rq); - p =3D donor; /* Deactivate the donor, not the runnable owner */ - clear_task_blocked_on(p, NULL); - goto deactivate; - } return owner; =20 deactivate: @@ -7686,6 +7710,11 @@ static void __sched notrace __schedule(int sched_mod= e) zap_balance_callbacks(rq); goto keep_resched; } + if (!sched_cpu_cookie_match(rq, next)) { + next =3D sched_core_swap_pick(rq, next); + if (next =3D=3D RETRY_TASK) + goto pick_again; + } trace_sched_found_proxy_task(rq->donor, next, cpu); } if (rq->donor =3D=3D prev_donor && prev !=3D next) { --=20 2.43.0 From nobody Sat Jul 25 05:29:19 2026 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012004.outbound.protection.outlook.com [52.101.48.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 965003AB26B for ; Fri, 17 Jul 2026 11:17:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.4 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784287051; cv=fail; b=eROoffsO0XvEg/mUBdSMz+WnSgqLlV2jexWk5rxY7yBoOmFhNwc3CLugRgfCH/L0PbomgCbQimVnJbeNxssoyhsSh1iwz0S9HgcRayM96IGj2JWOzK0/8r+cSKH7D68E/V01pRKdBNbX9bn1WIvZ0a750X8dJ/x+hQdF72iP7BM= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784287051; c=relaxed/simple; bh=hSFS3eBjoTASMIPuZcq6mS9DXvgZMQIGpZtCV0G16BI=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=pdeSoZOeEuByrvoCmCef/YiF8fWHHAvWcEgNvQm6Xr7nM2htPiG8xv0MjT75dYw22OKKIFS8laepEYBTv4crm+8lfFA3KOGsstQLdS3w3eZbC6+v02Q2iF6UJF44pkVyTUtDYzaIMHpYd5E42hgx5mDORvpYfSZcPbNLscoaNkw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=vgxt7rV0; arc=fail smtp.client-ip=52.101.48.4 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="vgxt7rV0" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Ie+VdbzCabXykwC4pTjMFdZ+SJaq7voI//PRsD83zZNl5pn9430Mm2+Pca0A0ixT3Cb2k1ZMqcm9GkoabejUZKfVzvEFOu+sXDvM1KPR626CEGgPkht1lUnGoK4gHLcCdXR3RW4Wy4OEQJg/Jg70Ko1ieONwldoDprrrH5ivfetxAAc22eqYYRZtQJQBq00l/u6vLcGfglzoGaYe73l8djv+OLMbyuRUjGzvZA0Jt5Cwtkq46UPGcobJy+nsV1A3YzJTNxXjC9/++O1KV939CtslrzR8oL9f5iDrJ+gI3/iZZj3x3nyPo0V0kQer2Q9JWeJGOnqesyrtVTURGDgvtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sGUydlTqwMgDD3KWMmE0rls83fUeJnK1mGwwzo5Zj7Q=; b=KekiqcM1/uApc9HHtKxkk6dAteK+kC+p4CEzFdlkqJDVCOBJbRPl3nCFoZJbPUpWqI1KRV6w3kEhQpwmCj3LwFUeEEH/fWd1NBvoOINHjRfOZp2oGYFncslKN8Iyod4XErkN+eIWOPj5IVG71xIVU2JfQuKWbGhiXsg/X1mhD798KOLigH8+CqtrUxyu5Y3DL+sFw33zNXYYcKeQmDjoIfjFByNsLplrswz8a9iBKyKByVV2CPm6wOt6g7jKB8JBbBj2hJq1EEQMO4gphoAwlQyUcWoFqOyD+IS/WvfFrDO1VUojizuPuSyeJl5RdxRYNI3l/whp8YRi1Z9UfGDJVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=redhat.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sGUydlTqwMgDD3KWMmE0rls83fUeJnK1mGwwzo5Zj7Q=; b=vgxt7rV0NXZvuaKGTWj1d0CKxPoRgHALa6igpEjg4rI+v/+6KAZEgOB6iX9h3x3GuG3g/6JCTC3tv/C1PI7TllSL67W4ukJjl0mwZSDSZmgGMXu1Tx7UQKebwwKeGOUBduFvsPhION+/u0ApO92go7cXkXwc2qC1GSNMecfO/1k= Received: from BN0PR10CA0009.namprd10.prod.outlook.com (2603:10b6:408:143::6) by PH7PR12MB6634.namprd12.prod.outlook.com (2603:10b6:510:211::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.223.13; Fri, 17 Jul 2026 11:17:25 +0000 Received: from BN2PEPF00004FBE.namprd04.prod.outlook.com (2603:10b6:408:143:cafe::d) by BN0PR10CA0009.outlook.office365.com (2603:10b6:408:143::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.223.13 via Frontend Transport; Fri, 17 Jul 2026 11:17:23 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN2PEPF00004FBE.mail.protection.outlook.com (10.167.243.184) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.245.5 via Frontend Transport; Fri, 17 Jul 2026 11:17:23 +0000 Received: from BLRKPRNAYAK.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.41; Fri, 17 Jul 2026 06:17:16 -0500 From: K Prateek Nayak To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , John Stultz , CC: Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Joel Fernandes , Qais Yousef , Zimuzo Ezeozue , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , K Prateek Nayak , Thomas Gleixner , "Daniel Lezcano" , Suleiman Souhlal , kuyo chang , hupu , Vasily Gorbik , Subject: [RFC PATCH 3/3] sched/core: Swap to lock owner for core-wide pick on core-cookie mismatch Date: Fri, 17 Jul 2026 11:16:01 +0000 Message-ID: <20260717111601.6017-4-kprateek.nayak@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717111601.6017-1-kprateek.nayak@amd.com> References: <20260717111601.6017-1-kprateek.nayak@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF00004FBE:EE_|PH7PR12MB6634:EE_ X-MS-Office365-Filtering-Correlation-Id: 533a5bea-2c91-43da-e808-08dee3f4f9fc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|7416014|376014|23010399003|1800799024|82310400026|10067099003|56012099006|4143699003|11063799006|5023799004|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: 2ec4IuBjndIPSJ/tTtnb9765dYCAy2Oh6JWkE2LJ7cipqu12yKJY0xiNNTzf/xNNydkREkH/9FwqnhAYNQuF3JOQQiCsW6yUgYlF8CK1gdkNxakHC6Vx2eU1MfGPr3l4Iqitz15rh169qmA3cYNjj0Mpbrc3qZSuETgpJJeyVoAFqLNW7Yp4rNIpzbpjpNko27cKVudLLmID4sJWQQatgcXSPYb586oSvclJLfOTboglXsR+2sJdwcERte8f85ChlPXuwSDGTucmO/jixUX7KdsbRKL+bfnEhOjI4WSp70CEMSMq8DOsVTKZaEy5k9CMCK3QasiUmkXXlxd6lggAv25Gqe+6CqFRATEUhiAD4xcxTfRx9TRyBw5AOYnDAtUUunCv++eZJ9IOOviNlkHcpGtcBmbrySwhjcIpDjwuFpSamJh+NEM6Bt/3urHO3/rg7QOa5rBVpAyuJO8lkkKAtw5b+YARRKtzNkl1rCqmEcEOGyS2fFR/1fxiSEl/bNVnk91vf394nYxJpXfPzVHvqJQE3vg7pJHARRQWUxlsGimssjJPJW5WamaybiCp64XvDtEvbeRVc609+UHpVtDbqcYCA2GhsRLncHEbL0v1yS51dLdiYBagJMgRM7cc7gISeODhi9dcMWa/cEEnc6xiNlCRRH5nAZ/CbPjdNKIb8n2j/IEkpHqeWIKowqH2a6C4NVpRSzeTBtl68D2dy5xadg== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(7416014)(376014)(23010399003)(1800799024)(82310400026)(10067099003)(56012099006)(4143699003)(11063799006)(5023799004)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 3w+wX2ZFQ3+BvLyTua482NGor42faLJlB/XDJ7e9xu2VINOeULYYY7epM3TZxouU7DfzgAdXM0fLoWaf1z2yyXVRqHZqkxu8x7Ea0UU/kJdzXc0mFzioyYfd0Kf/+xF7/KvAipREklYV+JsJl/u98JVVFFI0j3wE0SEtK593PjZPDZTp8j9XLi6CWxAG6aaVM94R6o75Wd2ECmnImplWamwxhoxg+mVjfiu0nMIHIQ3eQWz2ZHGbOQLj3YNvKE2JagceWHT6J6DHEJPSACkRX5UhgWevqjg/Z+1N6w+20POMKTq9zSPkUzW9twJAaDAZOrcTALOyYecsjQB1d1w7SkaUT9urL1PGOrX7TNoaMZ2KmiBbaeQjp/foKkO4m7fVcV2Wt5yPz8V0Q4Af/kslw2rH9oZ4HrFRL6myJzYj+MuvB+hnQUYbNtIwvXdM6Hc7 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jul 2026 11:17:23.8552 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 533a5bea-2c91-43da-e808-08dee3f4f9fc X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF00004FBE.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6634 Content-Type: text/plain; charset="utf-8" Proxy donor can have a different core cookie compared to the lock owner. Simply running a lock owner with a mismatched core-cookie in not an option as this breaks the core scheduling guarantees. To prevent a cookie mismatch, use the "rq->core_pick_leader" indicator to know if the CPU with the blocked donor is the one with the highest priority task. If the rq contains the highest priority task, it is that rq's pick which dictates the core-cookie. Once the find_proxy_task() chain stabilizes on the rq (find_proxy_task() returns the valid lock owner queued on the same rq), set a second "rq->core_pick_blocked_donor" and retry the pick. The flattened flow looks as follows: donor =3D pick_next_task() rq_set_donor(rq, donor) if (donor->is_blocked) next =3D find_proxy_task() /* * next has a different cookie compared to doner and * rq->core_pick_leader is set. */ sched_core_swap_task() rq->core_pick_blocked_donor =3D 1; /* rq->donor was the last pick */ return RETRY_TASK; pick_next_task() donor =3D rq->donor; owner =3D NULL; if (rq->core_pick_blocked_donor) owner =3D find_proxy_task(donor); goto restart_multi; /* * Finish the core-pick and validate that this rq * is still the pick leaader. */ if (rq->core_pick_blocked_donor && rq->core_pick_leader && rq->core_pick =3D=3D donor) rq->core_pick =3D owner; /* Swapped donor with owner. */ rq->core->coore_cookie =3D rq->core_pick->cookie; /* owner cookie */ /* Continue with the rest */ return rq->core_pick; /* Owner */ if (sched_core_retain_donor(rq)) rq->sched_core_proxy_pick =3D 0 /* Consume */ Since the owner is not blocked, it continues to run and rq->donor is preserved while the "rq->core_pick_blocked_donor" gets consumed. Signed-off-by: K Prateek Nayak --- kernel/sched/core.c | 81 +++++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 1 + 2 files changed, 74 insertions(+), 8 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ed11410f1416..ab3b3104cd86 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6598,7 +6598,8 @@ __pick_next_task(struct rq *rq, struct rq_flags *rf) BUG(); /* The idle class should always have a runnable task. */ } =20 -static void proxy_deactivate(struct rq *rq, struct task_struct *donor); +static struct task_struct * +find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags = *rf); =20 #ifdef CONFIG_SCHED_CORE static inline bool is_task_rq_idle(struct task_struct *t) @@ -6643,10 +6644,16 @@ extern void task_vruntime_update(struct rq *rq, str= uct task_struct *p, bool in_f =20 static void queue_core_balance(struct rq *rq); =20 +static __always_inline bool sched_core_proxy_pick(struct rq *rq) +{ + return sched_proxy_exec() && unlikely(rq->core_pick_blocked_donor); +} + static struct task_struct * pick_next_task(struct rq *rq, struct rq_flags *rf) __must_hold(__rq_lockp(rq)) { + struct task_struct *owner =3D NULL, *donor =3D rq->donor; struct task_struct *next, *p, *max; const struct cpumask *smt_mask; bool fi_before =3D false; @@ -6728,6 +6735,24 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) */ rq->core->core_task_seq++; =20 + /* Last core_pick resolved to a blocked_donor! */ + if (sched_core_proxy_pick(rq)) { + WARN_ON_ONCE(!donor->is_blocked); + + donor->blocked_donor =3D NULL; + owner =3D find_proxy_task(rq, donor, rf); + if (owner && owner !=3D rq->idle) + goto restart_multi; + /* + * Something changed in the proxy chain! + * Retry pick like normal. Increment core_task_seq + * since the core-wide lock might have been dropped + * during proxy-migration in find_proxy_task(). + */ + rq->core_pick_blocked_donor =3D false; + rq->core->core_task_seq++; + } + /* * Optimize for common case where this CPU has no cookies * and there are no cookied tasks running on siblings. @@ -6786,6 +6811,21 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) } =20 rq_max->core_pick_leader =3D true; + + if (sched_core_proxy_pick(rq)) { + WARN_ON_ONCE(!owner); + + /* + * Core-wide pick resolved to the same state as last time. + * Swap the donor with the lock owner and continue the + * rest of the pick sequence. + */ + if (rq->core_pick_leader && rq->core_pick =3D=3D donor) { + rq_max->core_pick =3D owner; + max =3D owner; + } + } + cookie =3D rq->core->core_cookie =3D max->core_cookie; =20 /* @@ -6883,9 +6923,16 @@ pick_next_task(struct rq *rq, struct rq_flags *rf) } =20 out_set_next: - put_prev_set_next_task(rq, rq->donor, next); - if (rq->core->core_forceidle_count && next =3D=3D rq->idle) - queue_core_balance(rq); + /* + * If this is a redo for setting the owner's core-cookie, + * put_prev_set_next_task() was already done during the + * last pick and there is nothing to do now. + */ + if (!sched_core_proxy_pick(rq)) { + put_prev_set_next_task(rq, rq->donor, next); + if (rq->core->core_forceidle_count && next =3D=3D rq->idle) + queue_core_balance(rq); + } =20 return next; } @@ -7114,11 +7161,21 @@ sched_core_swap_pick(struct rq *rq, struct task_str= uct *next) return rq->idle; } =20 - clear_task_blocked_on(rq->donor, NULL); - proxy_deactivate(rq, rq->donor); + rq->core_pick_blocked_donor =3D true; return RETRY_TASK; } =20 +static bool sched_core_retain_donor(struct rq *rq) +{ + bool retain =3D rq->core_pick_blocked_donor; + + if (!sched_core_enabled(rq)) + return false; + + rq->core_pick_blocked_donor =3D false; + return retain; +} + #else /* !CONFIG_SCHED_CORE: */ =20 static inline void sched_core_cpu_starting(unsigned int cpu) {} @@ -7144,6 +7201,11 @@ sched_core_swap_pick(struct rq *rq, struct task_stru= ct *next) return next; } =20 +static bool sched_core_retain_donor(struct rq *rq) +{ + return false; +} + #endif /* !CONFIG_SCHED_CORE */ =20 /* @@ -7694,8 +7756,11 @@ static void __sched notrace __schedule(int sched_mod= e) if (sched_proxy_exec()) { struct task_struct *prev_donor =3D rq->donor; =20 - rq_set_donor(rq, next); - next->blocked_donor =3D NULL; + if (!sched_core_retain_donor(rq)) { + rq_set_donor(rq, next); + next->blocked_donor =3D NULL; + } + if (unlikely(next->is_blocked)) { next =3D find_proxy_task(rq, next, &rf); if (!next) { diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index b80a5eb384f0..541b2ae673d8 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1352,6 +1352,7 @@ struct rq { =20 /* private state for sched_proxy_exec() */ bool core_pick_leader; + bool core_pick_blocked_donor; =20 /* shared state -- careful with sched_core_cpu_deactivate() */ unsigned int core_task_seq; --=20 2.43.0